From 038f3cea91ddc48cc31b14f66a28c558f3358cdd Mon Sep 17 00:00:00 2001 From: Ryan Chen Date: Tue, 13 Oct 2026 08:10:04 -0400 Subject: [PATCH] Add local LLM efficiency tickets Tickets for per-call thinking control, call-kind logging, a model A/B eval script and fewer discovery round-trips, written for handoff. Co-Authored-By: Claude Opus 5.5 --- docs/efficiency/tickets.md | 618 +++++++++++++++++++++++++++++++++++++ 1 file changed, 618 insertions(+) create mode 100644 docs/efficiency/tickets.md diff --git a/docs/efficiency/tickets.md b/docs/efficiency/tickets.md new file mode 100644 index 0000000..8e1da36 --- /dev/null +++ b/docs/efficiency/tickets.md @@ -0,0 +1,618 @@ +# Local LLM efficiency tickets + +Work items for making SimbaRAG's agent answer faster on the local llama-server +without losing answer quality. Each ticket is written to be picked up cold: it +says what to change, where, why, how to test it, and what "done" means. + +Do the tickets in order. EFF-1 is a prerequisite for measuring EFF-2 and EFF-4, +and EFF-3 is the safety net that tells you whether EFF-2 made answers worse. + +--- + +## Background: read this first + +### The model server + +- llama-server runs on a separate GPU box at `http://192.168.1.7:8085` + (OpenAI-compatible API under `/v1`). The `.env` / `.env.example` addresses + are stale; do not trust them. +- Model: `Qwen3.8-27B-UD-Q4_K_XL.gguf` (unsloth GGUF) on one RTX 3090 (24 GB), + llama.cpp build `b10884` (unsloth fork), vision enabled. +- As of 2026-10-12 the server runs with: + `--parallel 2` (two slots, 32,768 tokens each), KV cache `q8_0`/`q8_0`, + flash attention on, and **MTP speculative decoding** + (`--spec-type draft-mtp --spec-draft-n-max 3`). +- The server-side config is **not** in this repo (it is a systemd unit on the + GPU box). Do not try to change it. Tickets here are app-side only. + +### What we measured (so you understand the goal) + +- Before MTP: ~34 tokens/s generation. After MTP: ~84 tokens/s on tool-call + output, ~57 tokens/s on prose. +- A real chat turn: 7k-13.5k prompt tokens, ~4 model calls per turn, + 0.8-3 s prompt processing per call, **~16 s before the first answer + token**. Prompt processing is NOT the bottleneck. Generated tokens are: + every model call first generates hidden "thinking" tokens, then either a + tool call or the answer. With 4 calls per turn, that thinking adds up. +- Hard-capping thinking hurts reasoning. Same arithmetic question + ("cat eats 3/8 cup twice a day, 14-cup bag opened Sept 3, when does it run + out?", correct answer **Sept 21**): + + | Request setting | Time thinking | Answer | + |---|---|---| + | `reasoning_effort: "low"` (what the app sends today) | 5.9 s | Sept 21 (correct) | + | + `reasoning_budget_tokens: 64` | 0.97 s | Sept 20 (wrong) | + | `chat_template_kwargs: {"enable_thinking": false}` | 0.14 s | Sept 28 (wrong) | + + So the goal is NOT "turn thinking off". It is "turn thinking off or down only + on calls where it is useless, and keep it where the model actually reasons". + +### llama-server request fields you will use + +All verified against the live server on build b10884: + +| Field (top level of the JSON request body) | Effect | +|---|---| +| `reasoning_effort: "low" \| "medium" \| "xhigh"` | Soft hint. The chat template adds a sentence like "Reasoning effort is set to low" to the system prompt. The model usually but not always obeys. The app already sends `"low"` (see `model_kwargs` in `blueprints/conversation/agents.py`). | +| `reasoning_budget_tokens: ` | **Hard cap** on thinking tokens for this request. When hit, llama-server forces the thinking block closed and the model must answer. (`thinking_budget_tokens` is an alias; use `reasoning_budget_tokens`.) | +| `chat_template_kwargs: {"enable_thinking": false}` | No thinking at all for this request. The template pre-fills an empty `` block. | + +These are NOT OpenAI API parameters. The Python `openai` client (which +LangChain's `ChatOpenAI` uses under the hood) rejects unknown keyword +arguments, so they must be sent via **`extra_body`**, e.g. +`llama_chat.bind_tools(tools, extra_body={"reasoning_budget_tokens": 512})`. + +Also note: llama-server's `usage` block does **not** include +`completion_tokens_details.reasoning_tokens`. That is why `reasoning_tokens` +in the app's `stream_request` log line is always `null`. Don't try to "fix" +that in the app; it is a server limitation. Use `output_tokens` per call +instead (thinking tokens are included in it). + +### How the agent is wired (the parts you will touch) + +- `blueprints/conversation/agents.py` + - Near the top: `llama_chat = ChatOpenAI(base_url=..., model_kwargs={"reasoning_effort": "low"}, max_tokens=RESPONSE_RESERVE_TOKENS, ...)` + and `openai_fallback = ChatOpenAI(model="gpt-5-mini", ...)`. + - `build_agent()` at the bottom of the file. Inside it, `routed_model(messages, config)` + runs **once per model call**. It picks the tools, binds them, and returns + `prompt | bound_model`. **This is the hook for per-call settings.** + - `TOOL_ROUTING_MODE` env var (default `static`): + - `static`: every tool the user may use is bound on every call + (`router.select_static`). There is no `discover_tool_groups` tool. + - `dynamic`: `router.select(messages)` picks tool groups from keywords in + the user's message, and the model can call `discover_tool_groups` to + unlock more groups on the next call. + - Every ticket below must work in **both** modes. Tests should cover both + where it matters. +- `blueprints/conversation/tool_routing.py`: `ToolRouter`, `DISCOVERY_NAME = "discover_tool_groups"`, + `GROUP_TERMS` keyword lists, `select()` / `select_static()`. +- `blueprints/conversation/__init__.py`: the streaming endpoint. `_chat_run_stream` + → `event_generator` tracks every model call in the `model_calls` dict + (keyed by LangChain `run_id`) from `on_chat_model_start` / `_stream` / `_end` + events, and `_log_summary()` writes one JSON log line with + `"event": "stream_request"` per user turn. `_model_call_summary()` builds the + per-call dict (`prefill_ms`, `ms`, `prompt_tokens`, `output_tokens`, ...). +- `blueprints/conversation/context.py`: prompt budgeting (`CONTEXT_WINDOW_TOKENS`, + `PROMPT_BUDGET_TOKENS`, `RESPONSE_RESERVE_TOKENS=4096`). You should not need + to change it. + +### The message sequence inside one user turn + +LangGraph's ReAct agent loops: model call → (tools run) → model call → ... +until the model answers without tool calls. The `messages` list that +`routed_model` receives grows like this: + +``` +... earlier turns ... +HumanMessage("how much did we spend on groceries last month?") <- call 1 sees this as the last message +AIMessage(tool_calls=[ynab_spending_report(...)]) <- call 1 output +ToolMessage(name="ynab_spending_report", content="{...}") <- call 2 sees this as the last message +AIMessage("You spent $612 on groceries...") <- call 2 output (final answer) +``` + +In `dynamic` mode there can be a discovery hop: + +``` +HumanMessage("...") +AIMessage(tool_calls=[discover_tool_groups(groups=["finance"])]) +ToolMessage(name="discover_tool_groups", content='{"enabled_tool_groups": ["finance"], ...}') +AIMessage(tool_calls=[ynab_spending_report(...)]) <- this call needs no real thinking +ToolMessage(name="ynab_spending_report", ...) +AIMessage("final answer") +``` + +The model can also make **several tool calls in one AIMessage** (parallel +calls), which produces several `ToolMessage`s in a row. `ToolMessage.name` is +set to the tool's name by LangChain/LangGraph for both success and error +results. `ToolMessage.status` is `"success"` or `"error"`. + +### Ground rules for every ticket + +1. **One ticket = one branch = one PR.** Branch from up-to-date `main`, e.g. + `git checkout main && git pull && git checkout -b feat/eff-1-call-kind-logging`. +2. Line numbers drift. Find code by function/constant name, not by line number. +3. Match surrounding style: plain functions, short docstrings, comments only + where the "why" isn't obvious. Don't reformat unrelated code. +4. Run the unit tests before pushing: + `.venv/bin/pytest tests/unit -q` (or `make test`, which needs `uv`). + All tests must pass, including ones you didn't touch. +5. If you add an environment variable you must add it to **all four** of: + `.env.example`, `docker-compose.yml` (the `environment:` list of the + raggr services), `config/validation.py` (defaults/validation), and the + real `.env` if present. Prefer module constants unless the ticket says + otherwise; fewer knobs is better. +6. Do not write DB migrations by hand. None of these tickets should need a + migration; if you think one does, stop and ask. +7. PRs go to Gitea, not GitHub: `git push -u origin ` then + `tea pr create -t "title" -d "body"`. +8. Never point tests at the live llama-server. Unit tests must run offline. + Only EFF-3's script talks to the real server, and it is not a pytest test. +9. Don't commit `.DS_Store`, `chroma_db/`, `chromadb/`, `.planning/`. +10. **You probably cannot reach the llama-server or production.** It lives on + Ryan's home LAN (`192.168.1.7`), and production logs/DB are on his server. + Don't try to work around that, and don't block on it. Every step below + that needs the live server or prod data is marked **[Ryan runs]**: write + the code so Ryan can run that step with one copy-pasteable command, put + that exact command in the PR description under a "For Ryan to run" + heading, and finish the rest of the ticket. If you *can* reach + `http://192.168.1.7:8085/health` (e.g. running locally on his network), + you may run those steps yourself; they only generate text and change + nothing on the server. + +--- + +## EFF-1: Classify each model call and log the classification + +**Size:** small. **Depends on:** nothing. **Blocks:** EFF-2, EFF-4. + +### Why + +We want to know where the generated tokens go: is it the first call of a turn, +the "now call the tool you just unlocked" call after `discover_tool_groups`, +or the call that writes the answer from tool results? Today the +`stream_request` log has per-call `output_tokens` and `ms` but nothing says +which kind of call each one was. EFF-2 also needs this exact classification to +pick thinking settings, so it lives in one shared function. + +### What to build + +1. **New module** `blueprints/conversation/thinking.py` with: + + ```python + """Per-call classification used to size the model's hidden thinking.""" + + from langchain_core.messages import HumanMessage, ToolMessage + + from blueprints.conversation.tool_routing import DISCOVERY_NAME + + FIRST = "first" # last message is the user's question + AFTER_DISCOVERY = "after_discovery" # only discover_tool_groups results since the model last spoke + AFTER_TOOLS = "after_tools" # real tool results to read (or anything else) + + + def classify_call(messages) -> str: + ... + ``` + + Rules, in this order: + - Collect the **trailing run** of `ToolMessage`s at the end of `messages` + (walk backwards from the end while `isinstance(m, ToolMessage)`; stop at + the first non-ToolMessage). + - If that run is non-empty **and every** message in it has + `m.name == DISCOVERY_NAME` **and** `m.status == "success"` → return + `AFTER_DISCOVERY`. + - Mixed run (discovery + a real tool in the same parallel batch) → + `AFTER_TOOLS`, because there is real data to read. + - A discovery result with `status == "error"` → `AFTER_TOOLS` (something + went wrong; let the model think). + - Else, if `messages` is non-empty and `messages[-1]` is a `HumanMessage` + → `FIRST`. + - Else → `AFTER_TOOLS`. This is the safe default (it gets the most + thinking in EFF-2), so anything unexpected, including an empty list, + lands here. + - `messages` may contain a `SystemMessage` at the start; it does not matter + for these rules. `messages` might also be passed as a LangGraph state + dict `{"messages": [...]}` in some code paths. Look at how + `_build_prompt` in `agents.py` handles that + (`state["messages"] if isinstance(state, dict) else state`) and do the + same at the top of `classify_call`. + +2. **Attach the classification to each model call** in `routed_model` + (`blueprints/conversation/agents.py`, inside `build_agent`): + - Compute `kind = classify_call(messages)`. + - Attach it as LangChain run metadata so it shows up in the streaming + events: wrap the returned chain with + `.with_config(metadata={"call_kind": kind})`. Example: + + ```python + chain = RunnableLambda(partial(_build_prompt, tool_schemas=schemas)) | bound + return chain.with_config(metadata={"call_kind": kind}) + ``` + + LangChain merges config metadata into child runs, so the + `on_chat_model_start` event for this call should carry + `event["metadata"]["call_kind"]`. **Verify this** in a test (see below); + if metadata does not propagate in the installed LangChain version, attach + it to `bound` instead (`bound.with_config(...)`), and re-check. + +3. **Record it in the log line** in `blueprints/conversation/__init__.py`: + - In the `on_chat_model_start` branch where `model_calls[event["run_id"]] = {...}` + is created, add `"call_kind": metadata.get("call_kind")`. + - In `_model_call_summary()`, add `"call_kind": call.get("call_kind")` to + the returned dict. + - Do not change any other field names in the log line; dashboards or + scripts may read them. + +### Tests (new file `tests/unit/test_thinking.py`) + +Write a pure-function test for each rule. Build messages with +`HumanMessage`, `AIMessage(content="", tool_calls=[{"id": "c1", "name": "...", "args": {}}])`, +`ToolMessage(content="...", tool_call_id="c1", name="...", status="success")`. + +- `[HumanMessage]` → `FIRST`. +- `[Human, AI(tool_call discover), Tool(name=discover, success)]` → `AFTER_DISCOVERY`. +- `[Human, AI(discover + ynab), Tool(discover), Tool(ynab)]` → `AFTER_TOOLS` (mixed parallel batch). +- `[Human, AI(discover), Tool(discover, status="error")]` → `AFTER_TOOLS`. +- `[Human, AI(ynab), Tool(ynab)]` → `AFTER_TOOLS`. +- Two discovery results in a row, both success → `AFTER_DISCOVERY`. +- An older discovery earlier in the turn but the last messages are a real + tool result → `AFTER_TOOLS` (only the trailing run counts). +- A previous turn ended in a tool result, then a new `HumanMessage` → `FIRST`. +- `[]` → `AFTER_TOOLS`. +- The state-dict form `{"messages": [HumanMessage(...)]}` → `FIRST`. + +Plus one wiring test that the metadata reaches the streaming events. Look at +`tests/unit/test_tool_routing.py`: `import_agents()` imports `agents.py` with +all external services stubbed, and +`test_real_graph_binds_discovered_schemas_and_accounts_same_subset` builds the +real graph with a `FakeModel`. Copy that pattern: build the agent, run +`agent.astream_events(..., version="v2")` (or whatever the app uses; check +`_chat_run_stream`), collect `on_chat_model_start` events, and assert their +`metadata["call_kind"]` values are `["first", "after_discovery", ...]` for a +dynamic-mode run. If the `FakeModel` there is a plain `RunnableLambda` (not a +chat model), you may not get `on_chat_model_start` events at all. In that +case use `langchain_core.language_models.fake_chat_models.GenericFakeChatModel` +(or `FakeMessagesListChatModel`) with a `bind_tools` that returns itself, so +real chat-model events fire. Keep the test offline. + +### Done when + +- `classify_call` exists with the rules above and the tests pass. +- **[Ryan runs]** After deploying, a `stream_request` log line has a + `call_kind` on every entry in `model_calls`: + `make logs | grep stream_request | tail -5`. +- No behavior change for users (this ticket only adds logging). + +--- + +## EFF-2: Size thinking per call kind (the actual speedup) + +**Size:** small-medium. **Depends on:** EFF-1 (uses `classify_call`). +Ideally EFF-3 exists so you can measure before/after. + +### Why + +Thinking is the biggest chunk of time before the first answer token, but the +experiment in the Background section shows that killing it everywhere makes +math and date answers wrong. Some calls don't need it: + +- **`AFTER_DISCOVERY`** (dynamic mode only): the model just unlocked a tool + group; all it has to do now is call the tool. Thinking here is waste. +- **`FIRST`**: usually "pick a tool and fill in arguments", but sometimes the + model answers directly (chit-chat, or a math question like the cat-food one) + with no tool. So we cap it moderately instead of turning it off. +- **`AFTER_TOOLS`**: the model reads tool results and writes the answer + (YNAB totals, comparing dates). This is where reasoning matters. We only + add a generous cap so one runaway call can't burn the whole 4,096-token + `max_tokens` on thinking and leave nothing for the answer. + +### What to build + +1. In `blueprints/conversation/thinking.py` (from EFF-1), add: + + ```python + FIRST_CALL_THINKING_BUDGET = 512 + AFTER_TOOLS_THINKING_BUDGET = 1024 + + + def thinking_options(kind: str) -> dict: + """llama-server request fields (sent via extra_body) for one call kind.""" + if kind == AFTER_DISCOVERY: + return {"chat_template_kwargs": {"enable_thinking": False}} + if kind == FIRST: + return {"reasoning_budget_tokens": FIRST_CALL_THINKING_BUDGET} + return {"reasoning_budget_tokens": AFTER_TOOLS_THINKING_BUDGET} + ``` + + Keep these as module constants, not environment variables (see ground rule 5). + +2. In `routed_model` (`agents.py`, inside `build_agent`), pass the options to + the **llama** model only: + + ```python + kind = classify_call(messages) # already added in EFF-1 + fallback = openai_fallback.bind_tools(selected) + bound = ( + llama_chat.bind_tools(selected, extra_body=thinking_options(kind)).with_fallbacks([fallback]) + if llama_chat + else fallback + ) + ``` + + - **Do not** pass `extra_body` to `openai_fallback`. OpenAI's API rejects + unknown fields like `reasoning_budget_tokens`, and the fallback is what + saves us when the GPU box is down. + - Keep the existing `model_kwargs={"reasoning_effort": "low"}` on + `llama_chat`. The soft hint and the hard cap work together. + - `bind_tools(tools, **kwargs)` forwards extra kwargs into the request. + Confirm with a test (below) that `extra_body` ends up in the bound + kwargs. If `ChatOpenAI` in the installed version does not accept + `extra_body` through `bind_tools`, use + `llama_chat.bind_tools(selected).bind(extra_body=...)`. + +3. Add the options to the log line so we can correlate: in `routed_model`, + extend the EFF-1 metadata to + `metadata={"call_kind": kind, "thinking": thinking_options(kind)}` and copy + `"thinking"` into `model_calls[...]` / `_model_call_summary()` the same way + as `call_kind`. (If the fallback model handled the call, the field is still + logged; that's fine, `provider` tells you which model ran.) + +### Things that will break and how to fix them + +- `tests/unit/test_tool_routing.py` has fake models with + `def bind_tools(self, tools):` (two places). They will now get an + unexpected `extra_body=` keyword and raise `TypeError`. Change those fakes to + `def bind_tools(self, tools, **kwargs):`. Don't change what the tests assert. +- If you used `.bind(extra_body=...)` instead, the fakes need a `.bind` + method, or make `bind_tools` return a real `Runnable` (they currently return + `RunnableLambda`, which has `.bind`). Check before assuming. + +### Tests (add to `tests/unit/test_thinking.py`) + +- `thinking_options(FIRST) == {"reasoning_budget_tokens": 512}`. +- `thinking_options(AFTER_DISCOVERY) == {"chat_template_kwargs": {"enable_thinking": False}}`. +- `thinking_options(AFTER_TOOLS) == {"reasoning_budget_tokens": 1024}`. +- `thinking_options("something unknown") == {"reasoning_budget_tokens": 1024}` + (safe default). +- Wiring test, modeled on `test_real_graph_binds_discovered_schemas_and_accounts_same_subset`: + a fake llama model whose `bind_tools(self, tools, **kwargs)` records + `kwargs`. Run a dynamic-mode turn that does discovery → tool → answer and + assert the recorded `extra_body` values were, in order: + `{"reasoning_budget_tokens": 512}`, `{"chat_template_kwargs": {"enable_thinking": False}}`, + `{"reasoning_budget_tokens": 1024}`. +- Same wiring test in static mode (no discovery): expect `512` then `1024`. +- Fallback test: with the primary fake set to fail, assert the fallback + fake's `bind_tools` was **never** given `extra_body`. + +### Manual check against the real server (**[Ryan runs]**, optional, read-only, not a test) + +You can confirm llama-server honors the fields with curl. This only generates +text; it changes nothing on the server: + +```bash +curl -s http://192.168.1.7:8085/v1/chat/completions -H 'content-type: application/json' -d '{ + "model": "qwen3.8-27b", "max_tokens": 1024, "reasoning_effort": "low", + "reasoning_budget_tokens": 64, + "messages": [{"role": "user", "content": "What is 17*23?"}] +}' | python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print(len(m.get("reasoning_content") or ""), d["timings"]["predicted_n"], m["content"])' +``` + +A small `reasoning_content` length and `predicted_n` around the budget means +the cap works. + +### Done when + +- Tests above pass, and the whole unit suite passes. +- **[Ryan runs]** EFF-3's evaluation shows: median time-to-first-answer-token lower than + before, **no drop** in tool-choice accuracy, **zero** malformed tool calls, + and no new wrong answers on the arithmetic/date cases. If the arithmetic + cases regress, raise `FIRST_CALL_THINKING_BUDGET` (try 1024) before anything + else, and say so in the PR. +- The PR description includes the exact commands for Ryan to produce a + before/after comparison: EFF-3 run on `main` with `--label before`, then on + this branch with `--label after`, then the compare command. If EFF-3 isn't + merged yet, instead give him the log query for ~10 real chats (`ttft_ms` and + per-call `output_tokens` from `stream_request` lines, grouped by + `call_kind`). Ryan merges only after those numbers look right, so don't + merge it yourself. + +--- + +## EFF-3: Model-level A/B evaluation script + +**Size:** medium. **Depends on:** nothing (but EFF-1/2 make it more useful). + +### Why + +`scripts/evaluate_context.py` only tests history trimming. It never calls a +model, so it can't tell us whether a model or server change made answers +slower or worse. We need a repeatable script that sends realistic agent +requests to whatever `LLAMA_SERVER_URL` points at and reports speed and +correctness, so any change (EFF-2, new llama-server flags, a different model) +can be judged with numbers. + +### What to build + +1. **Fixture file** `tests/fixtures/model_eval/cases.json`: a list of cases. + Each case is one **single model call** (not a full agent loop; that would + need the DB, Paperless, YNAB, etc.). Shape: + + ```json + { + "name": "ynab_groceries_last_month", + "kind": "tool_choice", + "messages": [ + {"role": "user", "content": "How much did we spend on groceries last month?"} + ], + "expect_tool": "ynab_spending_report", + "expect_args_subset": {"category": "Groceries"} + } + ``` + + Three kinds of case: + - `"tool_choice"`: the user message alone; pass if the model's first + response calls `expect_tool`, and `expect_args_subset` (optional) keys + match the call's arguments (string compare, case-insensitive). + - `"answer"`: messages include a canned assistant tool call **and** a canned + tool result (role `"tool"`, realistic JSON copied from a real tool's + output shape; look at the YNAB tools in `blueprints/ynab/tools.py` and + the artifact shapes in `tests/fixtures/context/scenarios.json`); pass if + the final answer text contains every string in `expect_contains` + (case-insensitive) and calls no tool. + - `"direct"`: no tool should be called; pass if the answer contains + `expect_contains`. Include the cat-food question from the Background + section (`expect_contains: ["21"]`; the correct date is Sept 21). + + Write **at least 25 cases** covering: pets/Simba (`simba_search`, + `simbakit_*`), meals (`mealie_*`), notes/journal (`obsidian_*`, + `journal_*`), YNAB spending/balances (`ynab_*`), restaurants (`beli_*`), + web search, date questions ("what's on Friday"), 2-3 follow-up questions + that only make sense with a previous turn ("what about last month?"; put + the previous turn in `messages`), at least 3 `"answer"` cases with + arithmetic over tool results, and 2 `"direct"` reasoning questions. Use + the real tool names: get them from `agents.py` (`@tool` functions) and + `blueprints/ynab/tools.py`. No real personal data in fixtures; invent + plausible numbers. + +2. **Script** `scripts/evaluate_model.py`, run as + `.venv/bin/python -m scripts.evaluate_model [--cases PATH] [--repeat N] [--label NAME] [--out results.json]`: + - Reads `LLAMA_SERVER_URL` and `LLAMA_MODEL_NAME` from the environment + (`load_dotenv()` like the rest of the app). Allow `--base-url` to + override, because the `.env` value is stale; the real server is + `http://192.168.1.7:8085/v1`. + - For the system prompt and tool schemas, use the **real ones** so the + eval matches production: `SIMBA_SYSTEM_PROMPT` from + `blueprints/conversation/prompts.py`, and tool schemas via + `convert_to_openai_tool(t)` for the tools in `agents.tools` (plus YNAB + tools if they're in that list). Importing `agents.py` initializes + services; use the same stubbing trick as `import_agents()` in + `tests/unit/test_tool_routing.py` so the import works offline. Bind all + tools (static mode) unless `--routing dynamic` is given, in which case + use `ToolRouter(...).select(messages)` to pick them like production does. + - If `blueprints/conversation/thinking.py` exists (EFF-2 merged), apply + `thinking_options(classify_call(...))` per case so the eval reflects + production; add `--no-thinking-policy` to compare against today's + behavior. + - Call the server with the plain `openai` client (`OpenAI(base_url=..., api_key="not-needed")`), + **streaming**, with `stream_options={"include_usage": True}`, + `reasoning_effort="low"`, `max_tokens=4096`, and any thinking fields in + `extra_body`. Streaming lets you measure: + - `ttft_ms`: time from request to the first **content or tool-call** + delta (ignore deltas that only carry `reasoning_content`). + - `total_ms`, `completion_tokens` and `prompt_tokens` (from `usage`). + - llama-server also sends a non-standard `timings` object on the final + chunk with `predicted_per_second`, `prompt_per_second`, `draft_n`, + `draft_n_accepted`. Read it from the raw chunk + (`chunk.model_extra.get("timings")` with the openai client) and record + it if present; don't fail if absent. + - Grade each case per its `kind` (rules above). A tool call whose + arguments aren't valid JSON counts as **malformed**: record it + separately, it's the worst failure. + - Output: a table to stdout (one row per case: name, pass/fail, ttft_ms, + total_ms, completion_tokens, tok/s, draft acceptance) and a summary: + pass rate per kind, malformed count, median and p90 of ttft_ms and + total_ms, median completion_tokens. With `--out`, write all raw + results plus the summary as JSON, including `--label`, the base URL, + the model name, and a timestamp. + - `--repeat N` runs each case N times (default 1). Sampling is random + (server default temperature 1.0), so use `--repeat 3` for real + comparisons and report pass rate as passes/(cases×N). + - Add `scripts/compare_model_eval.py a.json b.json` (or a `--compare` + flag) that prints the two summaries side by side with deltas. + - Exit code 0 always (it's a report, not a gate), except exit 2 on + connection errors. + +3. **Docs:** add a short section to `docs/efficiency/README.md` (create it) + explaining how to run it, with an example command and how to read the + output. One paragraph plus commands is enough. + +### Tests + +The script itself talks to a live server, so it is **not** a unit test. But +add `tests/unit/test_model_eval.py` for the pure parts: +- The grading function for each kind, including malformed JSON arguments and + case-insensitive substring matching. +- Loading `cases.json` validates required keys per kind (fail fast with a + clear message naming the bad case). +- The summary math (median/p90) on a small hand-made list. +Keep these offline: no `OpenAI` client construction in tests. + +### Done when + +- The unit tests pass. +- The script runs end to end against a **fake** server so you can prove it + works without the LAN: add `--base-url` support and test it manually + against a tiny local stub (a ~40-line `http.server` or `aiohttp` script + that returns a canned streaming chat-completion with one tool call, a + `usage` chunk and a `timings` object). Don't commit the stub unless it's + clean enough to live in `tests/fixtures/model_eval/` and is used by a test. +- **[Ryan runs]** The PR description ends with the real command for Ryan: + `.venv/bin/python -m scripts.evaluate_model --base-url http://192.168.1.7:8085/v1 --repeat 3 --label baseline --out /tmp/baseline.json` + +--- + +## EFF-4: Fewer discovery round-trips in dynamic routing mode + +**Size:** small. **Depends on:** EFF-1 (to measure). **Only matters if +production runs `TOOL_ROUTING_MODE=dynamic`.** You can't see the production +`.env`; ask Ryan which mode production uses before starting. If production is `static`, skip this +ticket and note that in the PR list. + +### Why + +Every `discover_tool_groups` call is a full extra model round-trip: prompt +processing + generated tokens + the tool call. And because the template puts +the tool definitions at the very top of the prompt, the next call has a +different tool list, so llama-server's prompt cache misses and it reprocesses +the whole prompt. Even with EFF-2 making that call cheaper, it's cheapest not +to make it. Most discovery calls happen because the user's words didn't match +`GROUP_TERMS` in `tool_routing.py`. + +### What to do + +1. **Collect data first. [Ryan runs]** After EFF-1 is deployed for a few days, pull the + user messages that led to discovery. The `stream_request` log has + `tools` (tool spans, with names) and `model_calls` (with `call_kind`). + A turn where a `discover_tool_groups` tool span appears is a miss. The + user's message text is in the DB (`conversation` messages), matched by + the `conv` id in the log line. Write a small script (Ryan runs it on the + server; commit it under `scripts/` only if it's clean and reusable) that + lists, per missed turn: the + user message and the groups the model asked for. +2. For each miss, add the missing everyday words to the right group's string + in `GROUP_TERMS`. Example: if "what did the paycheck cover this month?" + triggered discovery of `finance`, add `paycheck paychecks` to the + `finance` string. Before adding a word, ask: "would this word show up in + questions that have nothing to do with this group?" If yes, don't add it, + because extra tools on every call cost prompt tokens and confuse the model. + Rules of thumb: + - Add nouns and verbs specific to the domain ("groceries", "paycheck", + "litter"); avoid generic words ("left", "show", "get", "much"). + - A word can be in two groups if it really belongs to both. + - Remember matching is on whole lowercase words (`words()` splits on + non-alphanumerics), so add plural forms separately ("bill bills"). +3. Add a test per new word family in `tests/unit/test_tool_routing.py` in the + existing style: `names(router, [HumanMessage(content="")])` + includes the expected tool. + +### Done when + +- Every miss you collected now routes without discovery (tests prove it). +- No existing routing test changed its expectations. +- PR description lists the misses found and the words added. + +--- + +## Manual / ops follow-ups (for Ryan, not for Codex) + +These need the GPU box and are listed so they aren't forgotten: + +- Try `--spec-draft-n-max 4` (tool-call output accepted ~92% of drafts at 3; + depth 4 sometimes wins on JSON-heavy output). Compare with EFF-3, + `--repeat 3`. +- Try `--spec-type draft-mtp,ngram-mod` (n-gram drafting could help when the + model copies IDs from tool results into arguments). Unverified; A/B with + EFF-3. +- Record the final llama-server unit (flags and why) in `docs/deployment.md` + so the server config isn't only on the GPU box. +- Old unit is backed up at `/etc/systemd/system/llama-server.service.bak`.