simbarag/docs/efficiency/tickets.md

Local LLM efficiency tickets

Work items for making SimbaRAG's agent answer faster on the local llama-server without losing answer quality. Each ticket is written to be picked up cold: it says what to change, where, why, how to test it, and what "done" means.

Do the tickets in order. EFF-1 is a prerequisite for measuring EFF-2 and EFF-4, and EFF-3 is the safety net that tells you whether EFF-2 made answers worse.

---

Background: read this first

The model server

• llama-server runs on a separate GPU box at http://192.168.1.7:8085 (OpenAI-compatible API under /v1). The .env / .env.example addresses are stale; do not trust them.
• Model: Qwen3.8-27B-UD-Q4_K_XL.gguf (unsloth GGUF) on one RTX 3090 (24 GB), llama.cpp build b10884 (unsloth fork), vision enabled.
• As of 2026-10-12 the server runs with: --parallel 2 (two slots, 32,768 tokens each), KV cache q8_0/q8_0, flash attention on, and MTP speculative decoding (--spec-type draft-mtp --spec-draft-n-max 3).
• The server-side config is not in this repo (it is a systemd unit on the GPU box). Do not try to change it. Tickets here are app-side only.

What we measured (so you understand the goal)

• Before MTP: ~34 tokens/s generation. After MTP: ~84 tokens/s on tool-call output, ~57 tokens/s on prose.

• A real chat turn: 7k-13.5k prompt tokens, ~4 model calls per turn, 0.8-3 s prompt processing per call, ~16 s before the first answer token. Prompt processing is NOT the bottleneck. Generated tokens are: every model call first generates hidden "thinking" tokens, then either a tool call or the answer. With 4 calls per turn, that thinking adds up.

• Hard-capping thinking hurts reasoning. Same arithmetic question ("cat eats 3/8 cup twice a day, 14-cup bag opened Sept 3, when does it run out?", correct answer Sept 21):

  Request setting	Time thinking	Answer
  reasoning_effort: "low" (what the app sends today)	5.9 s	Sept 21 (correct)
  + reasoning_budget_tokens: 64	0.97 s	Sept 20 (wrong)
  chat_template_kwargs: {"enable_thinking": false}	0.14 s	Sept 28 (wrong)

  So the goal is NOT "turn thinking off". It is "turn thinking off or down only on calls where it is useless, and keep it where the model actually reasons".

llama-server request fields you will use

All verified against the live server on build b10884:

Field (top level of the JSON request body)	Effect
reasoning_effort: "low" | "medium" | "xhigh"	Soft hint. The chat template adds a sentence like "Reasoning effort is set to low" to the system prompt. The model usually but not always obeys. The app already sends "low" (see model_kwargs in blueprints/conversation/agents.py).
reasoning_budget_tokens: <int>	Hard cap on thinking tokens for this request. When hit, llama-server forces the thinking block closed and the model must answer. (thinking_budget_tokens is an alias; use reasoning_budget_tokens.)
chat_template_kwargs: {"enable_thinking": false}	No thinking at all for this request. The template pre-fills an empty <think></think> block.

These are NOT OpenAI API parameters. The Python openai client (which LangChain's ChatOpenAI uses under the hood) rejects unknown keyword arguments, so they must be sent via extra_body, e.g. llama_chat.bind_tools(tools, extra_body={"reasoning_budget_tokens": 512}).

Also note: llama-server's usage block does not include completion_tokens_details.reasoning_tokens. That is why reasoning_tokens in the app's stream_request log line is always null. Don't try to "fix" that in the app; it is a server limitation. Use output_tokens per call instead (thinking tokens are included in it).

How the agent is wired (the parts you will touch)

• blueprints/conversation/agents.py
  • Near the top: llama_chat = ChatOpenAI(base_url=..., model_kwargs={"reasoning_effort": "low"}, max_tokens=RESPONSE_RESERVE_TOKENS, ...) and openai_fallback = ChatOpenAI(model="gpt-5-mini", ...).
  • build_agent() at the bottom of the file. Inside it, routed_model(messages, config) runs once per model call. It picks the tools, binds them, and returns prompt | bound_model. This is the hook for per-call settings.
  • TOOL_ROUTING_MODE env var (default static):
    • static: every tool the user may use is bound on every call (router.select_static). There is no discover_tool_groups tool.
    • dynamic: router.select(messages) picks tool groups from keywords in the user's message, and the model can call discover_tool_groups to unlock more groups on the next call.
    • Every ticket below must work in both modes. Tests should cover both where it matters.
• blueprints/conversation/tool_routing.py: ToolRouter, DISCOVERY_NAME = "discover_tool_groups", GROUP_TERMS keyword lists, select() / select_static().
• blueprints/conversation/__init__.py: the streaming endpoint. _chat_run_stream → event_generator tracks every model call in the model_calls dict (keyed by LangChain run_id) from on_chat_model_start / _stream / _end events, and _log_summary() writes one JSON log line with "event": "stream_request" per user turn. _model_call_summary() builds the per-call dict (prefill_ms, ms, prompt_tokens, output_tokens, ...).
• blueprints/conversation/context.py: prompt budgeting (CONTEXT_WINDOW_TOKENS, PROMPT_BUDGET_TOKENS, RESPONSE_RESERVE_TOKENS=4096). You should not need to change it.

The message sequence inside one user turn

LangGraph's ReAct agent loops: model call → (tools run) → model call → ... until the model answers without tool calls. The messages list that routed_model receives grows like this:

... earlier turns ...
HumanMessage("how much did we spend on groceries last month?")   <- call 1 sees this as the last message
AIMessage(tool_calls=[ynab_spending_report(...)])                 <- call 1 output
ToolMessage(name="ynab_spending_report", content="{...}")         <- call 2 sees this as the last message
AIMessage("You spent $612 on groceries...")                       <- call 2 output (final answer)

In dynamic mode there can be a discovery hop:

HumanMessage("...")
AIMessage(tool_calls=[discover_tool_groups(groups=["finance"])])
ToolMessage(name="discover_tool_groups", content='{"enabled_tool_groups": ["finance"], ...}')
AIMessage(tool_calls=[ynab_spending_report(...)])   <- this call needs no real thinking
ToolMessage(name="ynab_spending_report", ...)
AIMessage("final answer")

The model can also make several tool calls in one AIMessage (parallel calls), which produces several ToolMessages in a row. ToolMessage.name is set to the tool's name by LangChain/LangGraph for both success and error results. ToolMessage.status is "success" or "error".

Ground rules for every ticket

1. One ticket = one branch = one PR. Branch from up-to-date main, e.g. git checkout main && git pull && git checkout -b feat/eff-1-call-kind-logging.
2. Line numbers drift. Find code by function/constant name, not by line number.
3. Match surrounding style: plain functions, short docstrings, comments only where the "why" isn't obvious. Don't reformat unrelated code.
4. Run the unit tests before pushing: .venv/bin/pytest tests/unit -q (or make test, which needs uv). All tests must pass, including ones you didn't touch.
5. If you add an environment variable you must add it to all four of: .env.example, docker-compose.yml (the environment: list of the raggr services), config/validation.py (defaults/validation), and the real .env if present. Prefer module constants unless the ticket says otherwise; fewer knobs is better.
6. Do not write DB migrations by hand. None of these tickets should need a migration; if you think one does, stop and ask.
7. PRs go to Gitea, not GitHub: git push -u origin <branch> then tea pr create -t "title" -d "body".
8. Never point tests at the live llama-server. Unit tests must run offline. Only EFF-3's script talks to the real server, and it is not a pytest test.
9. Don't commit .DS_Store, chroma_db/, chromadb/, .planning/.
10. You probably cannot reach the llama-server or production. It lives on Ryan's home LAN (192.168.1.7), and production logs/DB are on his server. Don't try to work around that, and don't block on it. Every step below that needs the live server or prod data is marked [Ryan runs]: write the code so Ryan can run that step with one copy-pasteable command, put that exact command in the PR description under a "For Ryan to run" heading, and finish the rest of the ticket. If you can reach http://192.168.1.7:8085/health (e.g. running locally on his network), you may run those steps yourself; they only generate text and change nothing on the server.

---

EFF-1: Classify each model call and log the classification

Size: small. Depends on: nothing. Blocks: EFF-2, EFF-4.

Why

We want to know where the generated tokens go: is it the first call of a turn, the "now call the tool you just unlocked" call after discover_tool_groups, or the call that writes the answer from tool results? Today the stream_request log has per-call output_tokens and ms but nothing says which kind of call each one was. EFF-2 also needs this exact classification to pick thinking settings, so it lives in one shared function.

What to build

1. New module blueprints/conversation/thinking.py with:

   """Per-call classification used to size the model's hidden thinking."""
   
   from langchain_core.messages import HumanMessage, ToolMessage
   
   from blueprints.conversation.tool_routing import DISCOVERY_NAME
   
   FIRST = "first"                    # last message is the user's question
   AFTER_DISCOVERY = "after_discovery"  # only discover_tool_groups results since the model last spoke
   AFTER_TOOLS = "after_tools"        # real tool results to read (or anything else)
   
   
   def classify_call(messages) -> str:
       ...
   

   Rules, in this order:

   • Collect the trailing run of ToolMessages at the end of messages (walk backwards from the end while isinstance(m, ToolMessage); stop at the first non-ToolMessage).
   • If that run is non-empty and every message in it has m.name == DISCOVERY_NAME and m.status == "success" → return AFTER_DISCOVERY.
     • Mixed run (discovery + a real tool in the same parallel batch) → AFTER_TOOLS, because there is real data to read.
     • A discovery result with status == "error" → AFTER_TOOLS (something went wrong; let the model think).
   • Else, if messages is non-empty and messages[-1] is a HumanMessage → FIRST.
   • Else → AFTER_TOOLS. This is the safe default (it gets the most thinking in EFF-2), so anything unexpected, including an empty list, lands here.
   • messages may contain a SystemMessage at the start; it does not matter for these rules. messages might also be passed as a LangGraph state dict {"messages": [...]} in some code paths. Look at how _build_prompt in agents.py handles that (state["messages"] if isinstance(state, dict) else state) and do the same at the top of classify_call.

2. Attach the classification to each model call in routed_model (blueprints/conversation/agents.py, inside build_agent):

   • Compute kind = classify_call(messages).

   • Attach it as LangChain run metadata so it shows up in the streaming events: wrap the returned chain with .with_config(metadata={"call_kind": kind}). Example:

     chain = RunnableLambda(partial(_build_prompt, tool_schemas=schemas)) | bound
     return chain.with_config(metadata={"call_kind": kind})
     

     LangChain merges config metadata into child runs, so the on_chat_model_start event for this call should carry event["metadata"]["call_kind"]. Verify this in a test (see below); if metadata does not propagate in the installed LangChain version, attach it to bound instead (bound.with_config(...)), and re-check.

3. Record it in the log line in blueprints/conversation/__init__.py:

   • In the on_chat_model_start branch where model_calls[event["run_id"]] = {...} is created, add "call_kind": metadata.get("call_kind").
   • In _model_call_summary(), add "call_kind": call.get("call_kind") to the returned dict.
   • Do not change any other field names in the log line; dashboards or scripts may read them.

Tests (new file tests/unit/test_thinking.py)

Write a pure-function test for each rule. Build messages with HumanMessage, AIMessage(content="", tool_calls=[{"id": "c1", "name": "...", "args": {}}]), ToolMessage(content="...", tool_call_id="c1", name="...", status="success").

• [HumanMessage] → FIRST.
• [Human, AI(tool_call discover), Tool(name=discover, success)] → AFTER_DISCOVERY.
• [Human, AI(discover + ynab), Tool(discover), Tool(ynab)] → AFTER_TOOLS (mixed parallel batch).
• [Human, AI(discover), Tool(discover, status="error")] → AFTER_TOOLS.
• [Human, AI(ynab), Tool(ynab)] → AFTER_TOOLS.
• Two discovery results in a row, both success → AFTER_DISCOVERY.
• An older discovery earlier in the turn but the last messages are a real tool result → AFTER_TOOLS (only the trailing run counts).
• A previous turn ended in a tool result, then a new HumanMessage → FIRST.
• [] → AFTER_TOOLS.
• The state-dict form {"messages": [HumanMessage(...)]} → FIRST.

Plus one wiring test that the metadata reaches the streaming events. Look at tests/unit/test_tool_routing.py: import_agents() imports agents.py with all external services stubbed, and test_real_graph_binds_discovered_schemas_and_accounts_same_subset builds the real graph with a FakeModel. Copy that pattern: build the agent, run agent.astream_events(..., version="v2") (or whatever the app uses; check _chat_run_stream), collect on_chat_model_start events, and assert their metadata["call_kind"] values are ["first", "after_discovery", ...] for a dynamic-mode run. If the FakeModel there is a plain RunnableLambda (not a chat model), you may not get on_chat_model_start events at all. In that case use langchain_core.language_models.fake_chat_models.GenericFakeChatModel (or FakeMessagesListChatModel) with a bind_tools that returns itself, so real chat-model events fire. Keep the test offline.

Done when

• classify_call exists with the rules above and the tests pass.
• [Ryan runs] After deploying, a stream_request log line has a call_kind on every entry in model_calls: make logs | grep stream_request | tail -5.
• No behavior change for users (this ticket only adds logging).

---

EFF-2: Size thinking per call kind (the actual speedup)

Size: small-medium. Depends on: EFF-1 (uses classify_call). Ideally EFF-3 exists so you can measure before/after.

Why

Thinking is the biggest chunk of time before the first answer token, but the experiment in the Background section shows that killing it everywhere makes math and date answers wrong. Some calls don't need it:

• AFTER_DISCOVERY (dynamic mode only): the model just unlocked a tool group; all it has to do now is call the tool. Thinking here is waste.
• FIRST: usually "pick a tool and fill in arguments", but sometimes the model answers directly (chit-chat, or a math question like the cat-food one) with no tool. So we cap it moderately instead of turning it off.
• AFTER_TOOLS: the model reads tool results and writes the answer (YNAB totals, comparing dates). This is where reasoning matters. We only add a generous cap so one runaway call can't burn the whole 4,096-token max_tokens on thinking and leave nothing for the answer.

What to build

1. In blueprints/conversation/thinking.py (from EFF-1), add:

   FIRST_CALL_THINKING_BUDGET = 512
   AFTER_TOOLS_THINKING_BUDGET = 1024
   
   
   def thinking_options(kind: str) -> dict:
       """llama-server request fields (sent via extra_body) for one call kind."""
       if kind == AFTER_DISCOVERY:
           return {"chat_template_kwargs": {"enable_thinking": False}}
       if kind == FIRST:
           return {"reasoning_budget_tokens": FIRST_CALL_THINKING_BUDGET}
       return {"reasoning_budget_tokens": AFTER_TOOLS_THINKING_BUDGET}
   

   Keep these as module constants, not environment variables (see ground rule 5).

2. In routed_model (agents.py, inside build_agent), pass the options to the llama model only:

   kind = classify_call(messages)          # already added in EFF-1
   fallback = openai_fallback.bind_tools(selected)
   bound = (
       llama_chat.bind_tools(selected, extra_body=thinking_options(kind)).with_fallbacks([fallback])
       if llama_chat
       else fallback
   )
   

   • Do not pass extra_body to openai_fallback. OpenAI's API rejects unknown fields like reasoning_budget_tokens, and the fallback is what saves us when the GPU box is down.
   • Keep the existing model_kwargs={"reasoning_effort": "low"} on llama_chat. The soft hint and the hard cap work together.
   • bind_tools(tools, **kwargs) forwards extra kwargs into the request. Confirm with a test (below) that extra_body ends up in the bound kwargs. If ChatOpenAI in the installed version does not accept extra_body through bind_tools, use llama_chat.bind_tools(selected).bind(extra_body=...).

3. Add the options to the log line so we can correlate: in routed_model, extend the EFF-1 metadata to metadata={"call_kind": kind, "thinking": thinking_options(kind)} and copy "thinking" into model_calls[...] / _model_call_summary() the same way as call_kind. (If the fallback model handled the call, the field is still logged; that's fine, provider tells you which model ran.)

Things that will break and how to fix them

• tests/unit/test_tool_routing.py has fake models with def bind_tools(self, tools): (two places). They will now get an unexpected extra_body= keyword and raise TypeError. Change those fakes to def bind_tools(self, tools, **kwargs):. Don't change what the tests assert.
• If you used .bind(extra_body=...) instead, the fakes need a .bind method, or make bind_tools return a real Runnable (they currently return RunnableLambda, which has .bind). Check before assuming.

Tests (add to tests/unit/test_thinking.py)

• thinking_options(FIRST) == {"reasoning_budget_tokens": 512}.
• thinking_options(AFTER_DISCOVERY) == {"chat_template_kwargs": {"enable_thinking": False}}.
• thinking_options(AFTER_TOOLS) == {"reasoning_budget_tokens": 1024}.
• thinking_options("something unknown") == {"reasoning_budget_tokens": 1024} (safe default).
• Wiring test, modeled on test_real_graph_binds_discovered_schemas_and_accounts_same_subset: a fake llama model whose bind_tools(self, tools, **kwargs) records kwargs. Run a dynamic-mode turn that does discovery → tool → answer and assert the recorded extra_body values were, in order: {"reasoning_budget_tokens": 512}, {"chat_template_kwargs": {"enable_thinking": False}}, {"reasoning_budget_tokens": 1024}.
• Same wiring test in static mode (no discovery): expect 512 then 1024.
• Fallback test: with the primary fake set to fail, assert the fallback fake's bind_tools was never given extra_body.

Manual check against the real server ([Ryan runs], optional, read-only, not a test)

You can confirm llama-server honors the fields with curl. This only generates text; it changes nothing on the server:

curl -s http://192.168.1.7:8085/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "qwen3.8-27b", "max_tokens": 1024, "reasoning_effort": "low",
  "reasoning_budget_tokens": 64,
  "messages": [{"role": "user", "content": "What is 17*23?"}]
}' | python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print(len(m.get("reasoning_content") or ""), d["timings"]["predicted_n"], m["content"])'

A small reasoning_content length and predicted_n around the budget means the cap works.

Done when

• Tests above pass, and the whole unit suite passes.
• [Ryan runs] EFF-3's evaluation shows: median time-to-first-answer-token lower than before, no drop in tool-choice accuracy, zero malformed tool calls, and no new wrong answers on the arithmetic/date cases. If the arithmetic cases regress, raise FIRST_CALL_THINKING_BUDGET (try 1024) before anything else, and say so in the PR.
• The PR description includes the exact commands for Ryan to produce a before/after comparison: EFF-3 run on main with --label before, then on this branch with --label after, then the compare command. If EFF-3 isn't merged yet, instead give him the log query for ~10 real chats (ttft_ms and per-call output_tokens from stream_request lines, grouped by call_kind). Ryan merges only after those numbers look right, so don't merge it yourself.

---

EFF-3: Model-level A/B evaluation script

Size: medium. Depends on: nothing (but EFF-1/2 make it more useful).

Why

scripts/evaluate_context.py only tests history trimming. It never calls a model, so it can't tell us whether a model or server change made answers slower or worse. We need a repeatable script that sends realistic agent requests to whatever LLAMA_SERVER_URL points at and reports speed and correctness, so any change (EFF-2, new llama-server flags, a different model) can be judged with numbers.

What to build

1. Fixture file tests/fixtures/model_eval/cases.json: a list of cases. Each case is one single model call (not a full agent loop; that would need the DB, Paperless, YNAB, etc.). Shape:

   {
     "name": "ynab_groceries_last_month",
     "kind": "tool_choice",
     "messages": [
       {"role": "user", "content": "How much did we spend on groceries last month?"}
     ],
     "expect_tool": "ynab_spending_report",
     "expect_args_subset": {"category": "Groceries"}
   }
   

   Three kinds of case:

   • "tool_choice": the user message alone; pass if the model's first response calls expect_tool, and expect_args_subset (optional) keys match the call's arguments (string compare, case-insensitive).
   • "answer": messages include a canned assistant tool call and a canned tool result (role "tool", realistic JSON copied from a real tool's output shape; look at the YNAB tools in blueprints/ynab/tools.py and the artifact shapes in tests/fixtures/context/scenarios.json); pass if the final answer text contains every string in expect_contains (case-insensitive) and calls no tool.
   • "direct": no tool should be called; pass if the answer contains expect_contains. Include the cat-food question from the Background section (expect_contains: ["21"]; the correct date is Sept 21).

   Write at least 25 cases covering: pets/Simba (simba_search, simbakit_*), meals (mealie_*), notes/journal (obsidian_*, journal_*), YNAB spending/balances (ynab_*), restaurants (beli_*), web search, date questions ("what's on Friday"), 2-3 follow-up questions that only make sense with a previous turn ("what about last month?"; put the previous turn in messages), at least 3 "answer" cases with arithmetic over tool results, and 2 "direct" reasoning questions. Use the real tool names: get them from agents.py (@tool functions) and blueprints/ynab/tools.py. No real personal data in fixtures; invent plausible numbers.

2. Script scripts/evaluate_model.py, run as .venv/bin/python -m scripts.evaluate_model [--cases PATH] [--repeat N] [--label NAME] [--out results.json]:

   • Reads LLAMA_SERVER_URL and LLAMA_MODEL_NAME from the environment (load_dotenv() like the rest of the app). Allow --base-url to override, because the .env value is stale; the real server is http://192.168.1.7:8085/v1.
   • For the system prompt and tool schemas, use the real ones so the eval matches production: SIMBA_SYSTEM_PROMPT from blueprints/conversation/prompts.py, and tool schemas via convert_to_openai_tool(t) for the tools in agents.tools (plus YNAB tools if they're in that list). Importing agents.py initializes services; use the same stubbing trick as import_agents() in tests/unit/test_tool_routing.py so the import works offline. Bind all tools (static mode) unless --routing dynamic is given, in which case use ToolRouter(...).select(messages) to pick them like production does.
   • If blueprints/conversation/thinking.py exists (EFF-2 merged), apply thinking_options(classify_call(...)) per case so the eval reflects production; add --no-thinking-policy to compare against today's behavior.
   • Call the server with the plain openai client (OpenAI(base_url=..., api_key="not-needed")), streaming, with stream_options={"include_usage": True}, reasoning_effort="low", max_tokens=4096, and any thinking fields in extra_body. Streaming lets you measure:
     • ttft_ms: time from request to the first content or tool-call delta (ignore deltas that only carry reasoning_content).
     • total_ms, completion_tokens and prompt_tokens (from usage).
     • llama-server also sends a non-standard timings object on the final chunk with predicted_per_second, prompt_per_second, draft_n, draft_n_accepted. Read it from the raw chunk (chunk.model_extra.get("timings") with the openai client) and record it if present; don't fail if absent.
   • Grade each case per its kind (rules above). A tool call whose arguments aren't valid JSON counts as malformed: record it separately, it's the worst failure.
   • Output: a table to stdout (one row per case: name, pass/fail, ttft_ms, total_ms, completion_tokens, tok/s, draft acceptance) and a summary: pass rate per kind, malformed count, median and p90 of ttft_ms and total_ms, median completion_tokens. With --out, write all raw results plus the summary as JSON, including --label, the base URL, the model name, and a timestamp.
   • --repeat N runs each case N times (default 1). Sampling is random (server default temperature 1.0), so use --repeat 3 for real comparisons and report pass rate as passes/(cases×N).
   • Add scripts/compare_model_eval.py a.json b.json (or a --compare flag) that prints the two summaries side by side with deltas.
   • Exit code 0 always (it's a report, not a gate), except exit 2 on connection errors.

3. Docs: add a short section to docs/efficiency/README.md (create it) explaining how to run it, with an example command and how to read the output. One paragraph plus commands is enough.

Tests

The script itself talks to a live server, so it is not a unit test. But add tests/unit/test_model_eval.py for the pure parts:

• The grading function for each kind, including malformed JSON arguments and case-insensitive substring matching.
• Loading cases.json validates required keys per kind (fail fast with a clear message naming the bad case).
• The summary math (median/p90) on a small hand-made list. Keep these offline: no OpenAI client construction in tests.

Done when

• The unit tests pass.
• The script runs end to end against a fake server so you can prove it works without the LAN: add --base-url support and test it manually against a tiny local stub (a ~40-line http.server or aiohttp script that returns a canned streaming chat-completion with one tool call, a usage chunk and a timings object). Don't commit the stub unless it's clean enough to live in tests/fixtures/model_eval/ and is used by a test.
• [Ryan runs] The PR description ends with the real command for Ryan: .venv/bin/python -m scripts.evaluate_model --base-url http://192.168.1.7:8085/v1 --repeat 3 --label baseline --out /tmp/baseline.json

---

EFF-4: Fewer discovery round-trips in dynamic routing mode

Size: small. Depends on: EFF-1 (to measure). Only matters if production runs TOOL_ROUTING_MODE=dynamic. You can't see the production .env; ask Ryan which mode production uses before starting. If production is static, skip this ticket and note that in the PR list.

Why

Every discover_tool_groups call is a full extra model round-trip: prompt processing + generated tokens + the tool call. And because the template puts the tool definitions at the very top of the prompt, the next call has a different tool list, so llama-server's prompt cache misses and it reprocesses the whole prompt. Even with EFF-2 making that call cheaper, it's cheapest not to make it. Most discovery calls happen because the user's words didn't match GROUP_TERMS in tool_routing.py.

What to do

1. Collect data first. [Ryan runs] After EFF-1 is deployed for a few days, pull the user messages that led to discovery. The stream_request log has tools (tool spans, with names) and model_calls (with call_kind). A turn where a discover_tool_groups tool span appears is a miss. The user's message text is in the DB (conversation messages), matched by the conv id in the log line. Write a small script (Ryan runs it on the server; commit it under scripts/ only if it's clean and reusable) that lists, per missed turn: the user message and the groups the model asked for.
2. For each miss, add the missing everyday words to the right group's string in GROUP_TERMS. Example: if "what did the paycheck cover this month?" triggered discovery of finance, add paycheck paychecks to the finance string. Before adding a word, ask: "would this word show up in questions that have nothing to do with this group?" If yes, don't add it, because extra tools on every call cost prompt tokens and confuse the model. Rules of thumb:
   • Add nouns and verbs specific to the domain ("groceries", "paycheck", "litter"); avoid generic words ("left", "show", "get", "much").
   • A word can be in two groups if it really belongs to both.
   • Remember matching is on whole lowercase words (words() splits on non-alphanumerics), so add plural forms separately ("bill bills").
3. Add a test per new word family in tests/unit/test_tool_routing.py in the existing style: names(router, [HumanMessage(content="<the real user phrasing>")]) includes the expected tool.

Done when

• Every miss you collected now routes without discovery (tests prove it).
• No existing routing test changed its expectations.
• PR description lists the misses found and the words added.

---

Manual / ops follow-ups (for Ryan, not for Codex)

These need the GPU box and are listed so they aren't forgotten:

• Try --spec-draft-n-max 4 (tool-call output accepted ~92% of drafts at 3; depth 4 sometimes wins on JSON-heavy output). Compare with EFF-3, --repeat 3.
• Try --spec-type draft-mtp,ngram-mod (n-gram drafting could help when the model copies IDs from tool results into arguments). Unverified; A/B with EFF-3.
• Record the final llama-server unit (flags and why) in docs/deployment.md so the server config isn't only on the GPU box.
• Old unit is backed up at /etc/systemd/system/llama-server.service.bak.