Merge branch 'main' into docs/efficiency-tickets

Tickets for per-call thinking control, call-kind logging, a model A/B
eval script and fewer discovery round-trips, written for handoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/efficiency/tickets.md

@@ -0,0 +1,618 @@
+# Local LLM efficiency tickets
+
+Work items for making SimbaRAG's agent answer faster on the local llama-server
+without losing answer quality. Each ticket is written to be picked up cold: it
+says what to change, where, why, how to test it, and what "done" means.
+
+Do the tickets in order. EFF-1 is a prerequisite for measuring EFF-2 and EFF-4,
+and EFF-3 is the safety net that tells you whether EFF-2 made answers worse.
+
+---
+
+## Background: read this first
+
+### The model server
+
+- llama-server runs on a separate GPU box at `http://192.168.1.7:8085`
+  (OpenAI-compatible API under `/v1`). The `.env` / `.env.example` addresses
+  are stale; do not trust them.
+- Model: `Qwen3.8-27B-UD-Q4_K_XL.gguf` (unsloth GGUF) on one RTX 3090 (24 GB),
+  llama.cpp build `b10884` (unsloth fork), vision enabled.
+- As of 2026-10-12 the server runs with:
+  `--parallel 2` (two slots, 32,768 tokens each), KV cache `q8_0`/`q8_0`,
+  flash attention on, and **MTP speculative decoding**
+  (`--spec-type draft-mtp --spec-draft-n-max 3`).
+- The server-side config is **not** in this repo (it is a systemd unit on the
+  GPU box). Do not try to change it. Tickets here are app-side only.
+
+### What we measured (so you understand the goal)
+
+- Before MTP: ~34 tokens/s generation. After MTP: ~84 tokens/s on tool-call
+  output, ~57 tokens/s on prose.
+- A real chat turn: 7k-13.5k prompt tokens, ~4 model calls per turn,
+  0.8-3 s prompt processing per call, **~16 s before the first answer
+  token**. Prompt processing is NOT the bottleneck. Generated tokens are:
+  every model call first generates hidden "thinking" tokens, then either a
+  tool call or the answer. With 4 calls per turn, that thinking adds up.
+- Hard-capping thinking hurts reasoning. Same arithmetic question
+  ("cat eats 3/8 cup twice a day, 14-cup bag opened Sept 3, when does it run
+  out?", correct answer **Sept 21**):
+
+  | Request setting | Time thinking | Answer |
+  |---|---|---|
+  | `reasoning_effort: "low"` (what the app sends today) | 5.9 s | Sept 21 (correct) |
+  | + `reasoning_budget_tokens: 64` | 0.97 s | Sept 20 (wrong) |
+  | `chat_template_kwargs: {"enable_thinking": false}` | 0.14 s | Sept 28 (wrong) |
+
+  So the goal is NOT "turn thinking off". It is "turn thinking off or down only
+  on calls where it is useless, and keep it where the model actually reasons".
+
+### llama-server request fields you will use
+
+All verified against the live server on build b10884:
+
+| Field (top level of the JSON request body) | Effect |
+|---|---|
+| `reasoning_effort: "low" \| "medium" \| "xhigh"` | Soft hint. The chat template adds a sentence like "Reasoning effort is set to low" to the system prompt. The model usually but not always obeys. The app already sends `"low"` (see `model_kwargs` in `blueprints/conversation/agents.py`). |
+| `reasoning_budget_tokens: <int>` | **Hard cap** on thinking tokens for this request. When hit, llama-server forces the thinking block closed and the model must answer. (`thinking_budget_tokens` is an alias; use `reasoning_budget_tokens`.) |
+| `chat_template_kwargs: {"enable_thinking": false}` | No thinking at all for this request. The template pre-fills an empty `<think></think>` block. |
+
+These are NOT OpenAI API parameters. The Python `openai` client (which
+LangChain's `ChatOpenAI` uses under the hood) rejects unknown keyword
+arguments, so they must be sent via **`extra_body`**, e.g.
+`llama_chat.bind_tools(tools, extra_body={"reasoning_budget_tokens": 512})`.
+
+Also note: llama-server's `usage` block does **not** include
+`completion_tokens_details.reasoning_tokens`. That is why `reasoning_tokens`
+in the app's `stream_request` log line is always `null`. Don't try to "fix"
+that in the app; it is a server limitation. Use `output_tokens` per call
+instead (thinking tokens are included in it).
+
+### How the agent is wired (the parts you will touch)
+
+- `blueprints/conversation/agents.py`
+  - Near the top: `llama_chat = ChatOpenAI(base_url=..., model_kwargs={"reasoning_effort": "low"}, max_tokens=RESPONSE_RESERVE_TOKENS, ...)`
+    and `openai_fallback = ChatOpenAI(model="gpt-5-mini", ...)`.
+  - `build_agent()` at the bottom of the file. Inside it, `routed_model(messages, config)`
+    runs **once per model call**. It picks the tools, binds them, and returns
+    `prompt | bound_model`. **This is the hook for per-call settings.**
+  - `TOOL_ROUTING_MODE` env var (default `static`):
+    - `static`: every tool the user may use is bound on every call
+      (`router.select_static`). There is no `discover_tool_groups` tool.
+    - `dynamic`: `router.select(messages)` picks tool groups from keywords in
+      the user's message, and the model can call `discover_tool_groups` to
+      unlock more groups on the next call.
+    - Every ticket below must work in **both** modes. Tests should cover both
+      where it matters.
+- `blueprints/conversation/tool_routing.py`: `ToolRouter`, `DISCOVERY_NAME = "discover_tool_groups"`,
+  `GROUP_TERMS` keyword lists, `select()` / `select_static()`.
+- `blueprints/conversation/__init__.py`: the streaming endpoint. `_chat_run_stream`
+  → `event_generator` tracks every model call in the `model_calls` dict
+  (keyed by LangChain `run_id`) from `on_chat_model_start` / `_stream` / `_end`
+  events, and `_log_summary()` writes one JSON log line with
+  `"event": "stream_request"` per user turn. `_model_call_summary()` builds the
+  per-call dict (`prefill_ms`, `ms`, `prompt_tokens`, `output_tokens`, ...).
+- `blueprints/conversation/context.py`: prompt budgeting (`CONTEXT_WINDOW_TOKENS`,
+  `PROMPT_BUDGET_TOKENS`, `RESPONSE_RESERVE_TOKENS=4096`). You should not need
+  to change it.
+
+### The message sequence inside one user turn
+
+LangGraph's ReAct agent loops: model call → (tools run) → model call → ...
+until the model answers without tool calls. The `messages` list that
+`routed_model` receives grows like this:
+
+```
+... earlier turns ...
+HumanMessage("how much did we spend on groceries last month?")   <- call 1 sees this as the last message
+AIMessage(tool_calls=[ynab_spending_report(...)])                 <- call 1 output
+ToolMessage(name="ynab_spending_report", content="{...}")         <- call 2 sees this as the last message
+AIMessage("You spent $612 on groceries...")                       <- call 2 output (final answer)
+```
+
+In `dynamic` mode there can be a discovery hop:
+
+```
+HumanMessage("...")
+AIMessage(tool_calls=[discover_tool_groups(groups=["finance"])])
+ToolMessage(name="discover_tool_groups", content='{"enabled_tool_groups": ["finance"], ...}')
+AIMessage(tool_calls=[ynab_spending_report(...)])   <- this call needs no real thinking
+ToolMessage(name="ynab_spending_report", ...)
+AIMessage("final answer")
+```
+
+The model can also make **several tool calls in one AIMessage** (parallel
+calls), which produces several `ToolMessage`s in a row. `ToolMessage.name` is
+set to the tool's name by LangChain/LangGraph for both success and error
+results. `ToolMessage.status` is `"success"` or `"error"`.
+
+### Ground rules for every ticket
+
+1. **One ticket = one branch = one PR.** Branch from up-to-date `main`, e.g.
+   `git checkout main && git pull && git checkout -b feat/eff-1-call-kind-logging`.
+2. Line numbers drift. Find code by function/constant name, not by line number.
+3. Match surrounding style: plain functions, short docstrings, comments only
+   where the "why" isn't obvious. Don't reformat unrelated code.
+4. Run the unit tests before pushing:
+   `.venv/bin/pytest tests/unit -q` (or `make test`, which needs `uv`).
+   All tests must pass, including ones you didn't touch.
+5. If you add an environment variable you must add it to **all four** of:
+   `.env.example`, `docker-compose.yml` (the `environment:` list of the
+   raggr services), `config/validation.py` (defaults/validation), and the
+   real `.env` if present. Prefer module constants unless the ticket says
+   otherwise; fewer knobs is better.
+6. Do not write DB migrations by hand. None of these tickets should need a
+   migration; if you think one does, stop and ask.
+7. PRs go to Gitea, not GitHub: `git push -u origin <branch>` then
+   `tea pr create -t "title" -d "body"`.
+8. Never point tests at the live llama-server. Unit tests must run offline.
+   Only EFF-3's script talks to the real server, and it is not a pytest test.
+9. Don't commit `.DS_Store`, `chroma_db/`, `chromadb/`, `.planning/`.
+10. **You probably cannot reach the llama-server or production.** It lives on
+    Ryan's home LAN (`192.168.1.7`), and production logs/DB are on his server.
+    Don't try to work around that, and don't block on it. Every step below
+    that needs the live server or prod data is marked **[Ryan runs]**: write
+    the code so Ryan can run that step with one copy-pasteable command, put
+    that exact command in the PR description under a "For Ryan to run"
+    heading, and finish the rest of the ticket. If you *can* reach
+    `http://192.168.1.7:8085/health` (e.g. running locally on his network),
+    you may run those steps yourself; they only generate text and change
+    nothing on the server.
+
+---
+
+## EFF-1: Classify each model call and log the classification
+
+**Size:** small. **Depends on:** nothing. **Blocks:** EFF-2, EFF-4.
+
+### Why
+
+We want to know where the generated tokens go: is it the first call of a turn,
+the "now call the tool you just unlocked" call after `discover_tool_groups`,
+or the call that writes the answer from tool results? Today the
+`stream_request` log has per-call `output_tokens` and `ms` but nothing says
+which kind of call each one was. EFF-2 also needs this exact classification to
+pick thinking settings, so it lives in one shared function.
+
+### What to build
+
+1. **New module** `blueprints/conversation/thinking.py` with:
+
+   ```python
+   """Per-call classification used to size the model's hidden thinking."""
+
+   from langchain_core.messages import HumanMessage, ToolMessage
+
+   from blueprints.conversation.tool_routing import DISCOVERY_NAME
+
+   FIRST = "first"                    # last message is the user's question
+   AFTER_DISCOVERY = "after_discovery"  # only discover_tool_groups results since the model last spoke
+   AFTER_TOOLS = "after_tools"        # real tool results to read (or anything else)
+
+
+   def classify_call(messages) -> str:
+       ...
+   ```
+
+   Rules, in this order:
+   - Collect the **trailing run** of `ToolMessage`s at the end of `messages`
+     (walk backwards from the end while `isinstance(m, ToolMessage)`; stop at
+     the first non-ToolMessage).
+   - If that run is non-empty **and every** message in it has
+     `m.name == DISCOVERY_NAME` **and** `m.status == "success"` → return
+     `AFTER_DISCOVERY`.
+     - Mixed run (discovery + a real tool in the same parallel batch) →
+       `AFTER_TOOLS`, because there is real data to read.
+     - A discovery result with `status == "error"` → `AFTER_TOOLS` (something
+       went wrong; let the model think).
+   - Else, if `messages` is non-empty and `messages[-1]` is a `HumanMessage`
+     → `FIRST`.
+   - Else → `AFTER_TOOLS`. This is the safe default (it gets the most
+     thinking in EFF-2), so anything unexpected, including an empty list,
+     lands here.
+   - `messages` may contain a `SystemMessage` at the start; it does not matter
+     for these rules. `messages` might also be passed as a LangGraph state
+     dict `{"messages": [...]}` in some code paths. Look at how
+     `_build_prompt` in `agents.py` handles that
+     (`state["messages"] if isinstance(state, dict) else state`) and do the
+     same at the top of `classify_call`.
+
+2. **Attach the classification to each model call** in `routed_model`
+   (`blueprints/conversation/agents.py`, inside `build_agent`):
+   - Compute `kind = classify_call(messages)`.
+   - Attach it as LangChain run metadata so it shows up in the streaming
+     events: wrap the returned chain with
+     `.with_config(metadata={"call_kind": kind})`. Example:
+
+     ```python
+     chain = RunnableLambda(partial(_build_prompt, tool_schemas=schemas)) | bound
+     return chain.with_config(metadata={"call_kind": kind})
+     ```
+
+     LangChain merges config metadata into child runs, so the
+     `on_chat_model_start` event for this call should carry
+     `event["metadata"]["call_kind"]`. **Verify this** in a test (see below);
+     if metadata does not propagate in the installed LangChain version, attach
+     it to `bound` instead (`bound.with_config(...)`), and re-check.
+
+3. **Record it in the log line** in `blueprints/conversation/__init__.py`:
+   - In the `on_chat_model_start` branch where `model_calls[event["run_id"]] = {...}`
+     is created, add `"call_kind": metadata.get("call_kind")`.
+   - In `_model_call_summary()`, add `"call_kind": call.get("call_kind")` to
+     the returned dict.
+   - Do not change any other field names in the log line; dashboards or
+     scripts may read them.
+
+### Tests (new file `tests/unit/test_thinking.py`)
+
+Write a pure-function test for each rule. Build messages with
+`HumanMessage`, `AIMessage(content="", tool_calls=[{"id": "c1", "name": "...", "args": {}}])`,
+`ToolMessage(content="...", tool_call_id="c1", name="...", status="success")`.
+
+- `[HumanMessage]` → `FIRST`.
+- `[Human, AI(tool_call discover), Tool(name=discover, success)]` → `AFTER_DISCOVERY`.
+- `[Human, AI(discover + ynab), Tool(discover), Tool(ynab)]` → `AFTER_TOOLS` (mixed parallel batch).
+- `[Human, AI(discover), Tool(discover, status="error")]` → `AFTER_TOOLS`.
+- `[Human, AI(ynab), Tool(ynab)]` → `AFTER_TOOLS`.
+- Two discovery results in a row, both success → `AFTER_DISCOVERY`.
+- An older discovery earlier in the turn but the last messages are a real
+  tool result → `AFTER_TOOLS` (only the trailing run counts).
+- A previous turn ended in a tool result, then a new `HumanMessage` → `FIRST`.
+- `[]` → `AFTER_TOOLS`.
+- The state-dict form `{"messages": [HumanMessage(...)]}` → `FIRST`.
+
+Plus one wiring test that the metadata reaches the streaming events. Look at
+`tests/unit/test_tool_routing.py`: `import_agents()` imports `agents.py` with
+all external services stubbed, and
+`test_real_graph_binds_discovered_schemas_and_accounts_same_subset` builds the
+real graph with a `FakeModel`. Copy that pattern: build the agent, run
+`agent.astream_events(..., version="v2")` (or whatever the app uses; check
+`_chat_run_stream`), collect `on_chat_model_start` events, and assert their
+`metadata["call_kind"]` values are `["first", "after_discovery", ...]` for a
+dynamic-mode run. If the `FakeModel` there is a plain `RunnableLambda` (not a
+chat model), you may not get `on_chat_model_start` events at all. In that
+case use `langchain_core.language_models.fake_chat_models.GenericFakeChatModel`
+(or `FakeMessagesListChatModel`) with a `bind_tools` that returns itself, so
+real chat-model events fire. Keep the test offline.
+
+### Done when
+
+- `classify_call` exists with the rules above and the tests pass.
+- **[Ryan runs]** After deploying, a `stream_request` log line has a
+  `call_kind` on every entry in `model_calls`:
+  `make logs | grep stream_request | tail -5`.
+- No behavior change for users (this ticket only adds logging).
+
+---
+
+## EFF-2: Size thinking per call kind (the actual speedup)
+
+**Size:** small-medium. **Depends on:** EFF-1 (uses `classify_call`).
+Ideally EFF-3 exists so you can measure before/after.
+
+### Why
+
+Thinking is the biggest chunk of time before the first answer token, but the
+experiment in the Background section shows that killing it everywhere makes
+math and date answers wrong. Some calls don't need it:
+
+- **`AFTER_DISCOVERY`** (dynamic mode only): the model just unlocked a tool
+  group; all it has to do now is call the tool. Thinking here is waste.
+- **`FIRST`**: usually "pick a tool and fill in arguments", but sometimes the
+  model answers directly (chit-chat, or a math question like the cat-food one)
+  with no tool. So we cap it moderately instead of turning it off.
+- **`AFTER_TOOLS`**: the model reads tool results and writes the answer
+  (YNAB totals, comparing dates). This is where reasoning matters. We only
+  add a generous cap so one runaway call can't burn the whole 4,096-token
+  `max_tokens` on thinking and leave nothing for the answer.
+
+### What to build
+
+1. In `blueprints/conversation/thinking.py` (from EFF-1), add:
+
+   ```python
+   FIRST_CALL_THINKING_BUDGET = 512
+   AFTER_TOOLS_THINKING_BUDGET = 1024
+
+
+   def thinking_options(kind: str) -> dict:
+       """llama-server request fields (sent via extra_body) for one call kind."""
+       if kind == AFTER_DISCOVERY:
+           return {"chat_template_kwargs": {"enable_thinking": False}}
+       if kind == FIRST:
+           return {"reasoning_budget_tokens": FIRST_CALL_THINKING_BUDGET}
+       return {"reasoning_budget_tokens": AFTER_TOOLS_THINKING_BUDGET}
+   ```
+
+   Keep these as module constants, not environment variables (see ground rule 5).
+
+2. In `routed_model` (`agents.py`, inside `build_agent`), pass the options to
+   the **llama** model only:
+
+   ```python
+   kind = classify_call(messages)          # already added in EFF-1
+   fallback = openai_fallback.bind_tools(selected)
+   bound = (
+       llama_chat.bind_tools(selected, extra_body=thinking_options(kind)).with_fallbacks([fallback])
+       if llama_chat
+       else fallback
+   )
+   ```
+
+   - **Do not** pass `extra_body` to `openai_fallback`. OpenAI's API rejects
+     unknown fields like `reasoning_budget_tokens`, and the fallback is what
+     saves us when the GPU box is down.
+   - Keep the existing `model_kwargs={"reasoning_effort": "low"}` on
+     `llama_chat`. The soft hint and the hard cap work together.
+   - `bind_tools(tools, **kwargs)` forwards extra kwargs into the request.
+     Confirm with a test (below) that `extra_body` ends up in the bound
+     kwargs. If `ChatOpenAI` in the installed version does not accept
+     `extra_body` through `bind_tools`, use
+     `llama_chat.bind_tools(selected).bind(extra_body=...)`.
+
+3. Add the options to the log line so we can correlate: in `routed_model`,
+   extend the EFF-1 metadata to
+   `metadata={"call_kind": kind, "thinking": thinking_options(kind)}` and copy
+   `"thinking"` into `model_calls[...]` / `_model_call_summary()` the same way
+   as `call_kind`. (If the fallback model handled the call, the field is still
+   logged; that's fine, `provider` tells you which model ran.)
+
+### Things that will break and how to fix them
+
+- `tests/unit/test_tool_routing.py` has fake models with
+  `def bind_tools(self, tools):` (two places). They will now get an
+  unexpected `extra_body=` keyword and raise `TypeError`. Change those fakes to
+  `def bind_tools(self, tools, **kwargs):`. Don't change what the tests assert.
+- If you used `.bind(extra_body=...)` instead, the fakes need a `.bind`
+  method, or make `bind_tools` return a real `Runnable` (they currently return
+  `RunnableLambda`, which has `.bind`). Check before assuming.
+
+### Tests (add to `tests/unit/test_thinking.py`)
+
+- `thinking_options(FIRST) == {"reasoning_budget_tokens": 512}`.
+- `thinking_options(AFTER_DISCOVERY) == {"chat_template_kwargs": {"enable_thinking": False}}`.
+- `thinking_options(AFTER_TOOLS) == {"reasoning_budget_tokens": 1024}`.
+- `thinking_options("something unknown") == {"reasoning_budget_tokens": 1024}`
+  (safe default).
+- Wiring test, modeled on `test_real_graph_binds_discovered_schemas_and_accounts_same_subset`:
+  a fake llama model whose `bind_tools(self, tools, **kwargs)` records
+  `kwargs`. Run a dynamic-mode turn that does discovery → tool → answer and
+  assert the recorded `extra_body` values were, in order:
+  `{"reasoning_budget_tokens": 512}`, `{"chat_template_kwargs": {"enable_thinking": False}}`,
+  `{"reasoning_budget_tokens": 1024}`.
+- Same wiring test in static mode (no discovery): expect `512` then `1024`.
+- Fallback test: with the primary fake set to fail, assert the fallback
+  fake's `bind_tools` was **never** given `extra_body`.
+
+### Manual check against the real server (**[Ryan runs]**, optional, read-only, not a test)
+
+You can confirm llama-server honors the fields with curl. This only generates
+text; it changes nothing on the server:
+
+```bash
+curl -s http://192.168.1.7:8085/v1/chat/completions -H 'content-type: application/json' -d '{
+  "model": "qwen3.8-27b", "max_tokens": 1024, "reasoning_effort": "low",
+  "reasoning_budget_tokens": 64,
+  "messages": [{"role": "user", "content": "What is 17*23?"}]
+}' | python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print(len(m.get("reasoning_content") or ""), d["timings"]["predicted_n"], m["content"])'
+```
+
+A small `reasoning_content` length and `predicted_n` around the budget means
+the cap works.
+
+### Done when
+
+- Tests above pass, and the whole unit suite passes.
+- **[Ryan runs]** EFF-3's evaluation shows: median time-to-first-answer-token lower than
+  before, **no drop** in tool-choice accuracy, **zero** malformed tool calls,
+  and no new wrong answers on the arithmetic/date cases. If the arithmetic
+  cases regress, raise `FIRST_CALL_THINKING_BUDGET` (try 1024) before anything
+  else, and say so in the PR.
+- The PR description includes the exact commands for Ryan to produce a
+  before/after comparison: EFF-3 run on `main` with `--label before`, then on
+  this branch with `--label after`, then the compare command. If EFF-3 isn't
+  merged yet, instead give him the log query for ~10 real chats (`ttft_ms` and
+  per-call `output_tokens` from `stream_request` lines, grouped by
+  `call_kind`). Ryan merges only after those numbers look right, so don't
+  merge it yourself.
+
+---
+
+## EFF-3: Model-level A/B evaluation script
+
+**Size:** medium. **Depends on:** nothing (but EFF-1/2 make it more useful).
+
+### Why
+
+`scripts/evaluate_context.py` only tests history trimming. It never calls a
+model, so it can't tell us whether a model or server change made answers
+slower or worse. We need a repeatable script that sends realistic agent
+requests to whatever `LLAMA_SERVER_URL` points at and reports speed and
+correctness, so any change (EFF-2, new llama-server flags, a different model)
+can be judged with numbers.
+
+### What to build
+
+1. **Fixture file** `tests/fixtures/model_eval/cases.json`: a list of cases.
+   Each case is one **single model call** (not a full agent loop; that would
+   need the DB, Paperless, YNAB, etc.). Shape:
+
+   ```json
+   {
+     "name": "ynab_groceries_last_month",
+     "kind": "tool_choice",
+     "messages": [
+       {"role": "user", "content": "How much did we spend on groceries last month?"}
+     ],
+     "expect_tool": "ynab_spending_report",
+     "expect_args_subset": {"category": "Groceries"}
+   }
+   ```
+
+   Three kinds of case:
+   - `"tool_choice"`: the user message alone; pass if the model's first
+     response calls `expect_tool`, and `expect_args_subset` (optional) keys
+     match the call's arguments (string compare, case-insensitive).
+   - `"answer"`: messages include a canned assistant tool call **and** a canned
+     tool result (role `"tool"`, realistic JSON copied from a real tool's
+     output shape; look at the YNAB tools in `blueprints/ynab/tools.py` and
+     the artifact shapes in `tests/fixtures/context/scenarios.json`); pass if
+     the final answer text contains every string in `expect_contains`
+     (case-insensitive) and calls no tool.
+   - `"direct"`: no tool should be called; pass if the answer contains
+     `expect_contains`. Include the cat-food question from the Background
+     section (`expect_contains: ["21"]`; the correct date is Sept 21).
+
+   Write **at least 25 cases** covering: pets/Simba (`simba_search`,
+   `simbakit_*`), meals (`mealie_*`), notes/journal (`obsidian_*`,
+   `journal_*`), YNAB spending/balances (`ynab_*`), restaurants (`beli_*`),
+   web search, date questions ("what's on Friday"), 2-3 follow-up questions
+   that only make sense with a previous turn ("what about last month?"; put
+   the previous turn in `messages`), at least 3 `"answer"` cases with
+   arithmetic over tool results, and 2 `"direct"` reasoning questions. Use
+   the real tool names: get them from `agents.py` (`@tool` functions) and
+   `blueprints/ynab/tools.py`. No real personal data in fixtures; invent
+   plausible numbers.
+
+2. **Script** `scripts/evaluate_model.py`, run as
+   `.venv/bin/python -m scripts.evaluate_model [--cases PATH] [--repeat N] [--label NAME] [--out results.json]`:
+   - Reads `LLAMA_SERVER_URL` and `LLAMA_MODEL_NAME` from the environment
+     (`load_dotenv()` like the rest of the app). Allow `--base-url` to
+     override, because the `.env` value is stale; the real server is
+     `http://192.168.1.7:8085/v1`.
+   - For the system prompt and tool schemas, use the **real ones** so the
+     eval matches production: `SIMBA_SYSTEM_PROMPT` from
+     `blueprints/conversation/prompts.py`, and tool schemas via
+     `convert_to_openai_tool(t)` for the tools in `agents.tools` (plus YNAB
+     tools if they're in that list). Importing `agents.py` initializes
+     services; use the same stubbing trick as `import_agents()` in
+     `tests/unit/test_tool_routing.py` so the import works offline. Bind all
+     tools (static mode) unless `--routing dynamic` is given, in which case
+     use `ToolRouter(...).select(messages)` to pick them like production does.
+   - If `blueprints/conversation/thinking.py` exists (EFF-2 merged), apply
+     `thinking_options(classify_call(...))` per case so the eval reflects
+     production; add `--no-thinking-policy` to compare against today's
+     behavior.
+   - Call the server with the plain `openai` client (`OpenAI(base_url=..., api_key="not-needed")`),
+     **streaming**, with `stream_options={"include_usage": True}`,
+     `reasoning_effort="low"`, `max_tokens=4096`, and any thinking fields in
+     `extra_body`. Streaming lets you measure:
+     - `ttft_ms`: time from request to the first **content or tool-call**
+       delta (ignore deltas that only carry `reasoning_content`).
+     - `total_ms`, `completion_tokens` and `prompt_tokens` (from `usage`).
+     - llama-server also sends a non-standard `timings` object on the final
+       chunk with `predicted_per_second`, `prompt_per_second`, `draft_n`,
+       `draft_n_accepted`. Read it from the raw chunk
+       (`chunk.model_extra.get("timings")` with the openai client) and record
+       it if present; don't fail if absent.
+   - Grade each case per its `kind` (rules above). A tool call whose
+     arguments aren't valid JSON counts as **malformed**: record it
+     separately, it's the worst failure.
+   - Output: a table to stdout (one row per case: name, pass/fail, ttft_ms,
+     total_ms, completion_tokens, tok/s, draft acceptance) and a summary:
+     pass rate per kind, malformed count, median and p90 of ttft_ms and
+     total_ms, median completion_tokens. With `--out`, write all raw
+     results plus the summary as JSON, including `--label`, the base URL,
+     the model name, and a timestamp.
+   - `--repeat N` runs each case N times (default 1). Sampling is random
+     (server default temperature 1.0), so use `--repeat 3` for real
+     comparisons and report pass rate as passes/(cases×N).
+   - Add `scripts/compare_model_eval.py a.json b.json` (or a `--compare`
+     flag) that prints the two summaries side by side with deltas.
+   - Exit code 0 always (it's a report, not a gate), except exit 2 on
+     connection errors.
+
+3. **Docs:** add a short section to `docs/efficiency/README.md` (create it)
+   explaining how to run it, with an example command and how to read the
+   output. One paragraph plus commands is enough.
+
+### Tests
+
+The script itself talks to a live server, so it is **not** a unit test. But
+add `tests/unit/test_model_eval.py` for the pure parts:
+- The grading function for each kind, including malformed JSON arguments and
+  case-insensitive substring matching.
+- Loading `cases.json` validates required keys per kind (fail fast with a
+  clear message naming the bad case).
+- The summary math (median/p90) on a small hand-made list.
+Keep these offline: no `OpenAI` client construction in tests.
+
+### Done when
+
+- The unit tests pass.
+- The script runs end to end against a **fake** server so you can prove it
+  works without the LAN: add `--base-url` support and test it manually
+  against a tiny local stub (a ~40-line `http.server` or `aiohttp` script
+  that returns a canned streaming chat-completion with one tool call, a
+  `usage` chunk and a `timings` object). Don't commit the stub unless it's
+  clean enough to live in `tests/fixtures/model_eval/` and is used by a test.
+- **[Ryan runs]** The PR description ends with the real command for Ryan:
+  `.venv/bin/python -m scripts.evaluate_model --base-url http://192.168.1.7:8085/v1 --repeat 3 --label baseline --out /tmp/baseline.json`
+
+---
+
+## EFF-4: Fewer discovery round-trips in dynamic routing mode
+
+**Size:** small. **Depends on:** EFF-1 (to measure). **Only matters if
+production runs `TOOL_ROUTING_MODE=dynamic`.** You can't see the production
+`.env`; ask Ryan which mode production uses before starting. If production is `static`, skip this
+ticket and note that in the PR list.
+
+### Why
+
+Every `discover_tool_groups` call is a full extra model round-trip: prompt
+processing + generated tokens + the tool call. And because the template puts
+the tool definitions at the very top of the prompt, the next call has a
+different tool list, so llama-server's prompt cache misses and it reprocesses
+the whole prompt. Even with EFF-2 making that call cheaper, it's cheapest not
+to make it. Most discovery calls happen because the user's words didn't match
+`GROUP_TERMS` in `tool_routing.py`.
+
+### What to do
+
+1. **Collect data first. [Ryan runs]** After EFF-1 is deployed for a few days, pull the
+   user messages that led to discovery. The `stream_request` log has
+   `tools` (tool spans, with names) and `model_calls` (with `call_kind`).
+   A turn where a `discover_tool_groups` tool span appears is a miss. The
+   user's message text is in the DB (`conversation` messages), matched by
+   the `conv` id in the log line. Write a small script (Ryan runs it on the
+   server; commit it under `scripts/` only if it's clean and reusable) that
+   lists, per missed turn: the
+   user message and the groups the model asked for.
+2. For each miss, add the missing everyday words to the right group's string
+   in `GROUP_TERMS`. Example: if "what did the paycheck cover this month?"
+   triggered discovery of `finance`, add `paycheck paychecks` to the
+   `finance` string. Before adding a word, ask: "would this word show up in
+   questions that have nothing to do with this group?" If yes, don't add it,
+   because extra tools on every call cost prompt tokens and confuse the model.
+   Rules of thumb:
+   - Add nouns and verbs specific to the domain ("groceries", "paycheck",
+     "litter"); avoid generic words ("left", "show", "get", "much").
+   - A word can be in two groups if it really belongs to both.
+   - Remember matching is on whole lowercase words (`words()` splits on
+     non-alphanumerics), so add plural forms separately ("bill bills").
+3. Add a test per new word family in `tests/unit/test_tool_routing.py` in the
+   existing style: `names(router, [HumanMessage(content="<the real user phrasing>")])`
+   includes the expected tool.
+
+### Done when
+
+- Every miss you collected now routes without discovery (tests prove it).
+- No existing routing test changed its expectations.
+- PR description lists the misses found and the words added.
+
+---
+
+## Manual / ops follow-ups (for Ryan, not for Codex)
+
+These need the GPU box and are listed so they aren't forgotten:
+
+- Try `--spec-draft-n-max 4` (tool-call output accepted ~92% of drafts at 3;
+  depth 4 sometimes wins on JSON-heavy output). Compare with EFF-3,
+  `--repeat 3`.
+- Try `--spec-type draft-mtp,ngram-mod` (n-gram drafting could help when the
+  model copies IDs from tool results into arguments). Unverified; A/B with
+  EFF-3.
+- Record the final llama-server unit (flags and why) in `docs/deployment.md`
+  so the server config isn't only on the GPU box.
+- Old unit is backed up at `/etc/systemd/system/llama-server.service.bak`.