ryan/simbarag
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Add local LLM efficiency tickets #87

Merged
ryan merged 2 commits from docs/efficiency-tickets into main 2026-10-13 21:21:45 -04:00
Conversation 0 Commits 2 Files Changed 1
+618
1 changed files with 618 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 038f3cea91 - Show all commits
docs/efficiency/tickets.md
+618
View File
@@ -0,0 +1,618 @@
# Local LLM efficiency tickets
Work items for making SimbaRAG's agent answer faster on the local llama-server
without losing answer quality. Each ticket is written to be picked up cold: it
says what to change, where, why, how to test it, and what "done" means.
Do the tickets in order. EFF-1 is a prerequisite for measuring EFF-2 and EFF-4,
and EFF-3 is the safety net that tells you whether EFF-2 made answers worse.
---
## Background: read this first
### The model server
- llama-server runs on a separate GPU box at `http://192.168.1.7:8085`
(OpenAI-compatible API under `/v1`). The `.env` / `.env.example` addresses
are stale; do not trust them.
- Model: `Qwen3.8-27B-UD-Q4_K_XL.gguf` (unsloth GGUF) on one RTX 3090 (24 GB),
llama.cpp build `b10884` (unsloth fork), vision enabled.
- As of 2026-10-12 the server runs with:
`--parallel 2` (two slots, 32,768 tokens each), KV cache `q8_0`/`q8_0`,
flash attention on, and **MTP speculative decoding**
(`--spec-type draft-mtp --spec-draft-n-max 3`).
- The server-side config is **not** in this repo (it is a systemd unit on the
GPU box). Do not try to change it. Tickets here are app-side only.
### What we measured (so you understand the goal)
- Before MTP: ~34 tokens/s generation. After MTP: ~84 tokens/s on tool-call
output, ~57 tokens/s on prose.
- A real chat turn: 7k-13.5k prompt tokens, ~4 model calls per turn,
0.8-3 s prompt processing per call, **~16 s before the first answer
token**. Prompt processing is NOT the bottleneck. Generated tokens are:
every model call first generates hidden "thinking" tokens, then either a
tool call or the answer. With 4 calls per turn, that thinking adds up.
- Hard-capping thinking hurts reasoning. Same arithmetic question
("cat eats 3/8 cup twice a day, 14-cup bag opened Sept 3, when does it run
out?", correct answer **Sept 21**):
| Request setting | Time thinking | Answer |
|---|---|---|
| `reasoning_effort: "low"` (what the app sends today) | 5.9 s | Sept 21 (correct) |
| + `reasoning_budget_tokens: 64` | 0.97 s | Sept 20 (wrong) |
| `chat_template_kwargs: {"enable_thinking": false}` | 0.14 s | Sept 28 (wrong) |
So the goal is NOT "turn thinking off". It is "turn thinking off or down only
on calls where it is useless, and keep it where the model actually reasons".
### llama-server request fields you will use
All verified against the live server on build b10884:
| Field (top level of the JSON request body) | Effect |
|---|---|
| `reasoning_effort: "low" \| "medium" \| "xhigh"` | Soft hint. The chat template adds a sentence like "Reasoning effort is set to low" to the system prompt. The model usually but not always obeys. The app already sends `"low"` (see `model_kwargs` in `blueprints/conversation/agents.py`). |
| `reasoning_budget_tokens: <int>` | **Hard cap** on thinking tokens for this request. When hit, llama-server forces the thinking block closed and the model must answer. (`thinking_budget_tokens` is an alias; use `reasoning_budget_tokens`.) |
| `chat_template_kwargs: {"enable_thinking": false}` | No thinking at all for this request. The template pre-fills an empty `<think></think>` block. |
These are NOT OpenAI API parameters. The Python `openai` client (which
LangChain's `ChatOpenAI` uses under the hood) rejects unknown keyword
arguments, so they must be sent via **`extra_body`**, e.g.
`llama_chat.bind_tools(tools, extra_body={"reasoning_budget_tokens": 512})`.
Also note: llama-server's `usage` block does **not** include
`completion_tokens_details.reasoning_tokens`. That is why `reasoning_tokens`
in the app's `stream_request` log line is always `null`. Don't try to "fix"
that in the app; it is a server limitation. Use `output_tokens` per call
instead (thinking tokens are included in it).
### How the agent is wired (the parts you will touch)
- `blueprints/conversation/agents.py`
- Near the top: `llama_chat = ChatOpenAI(base_url=..., model_kwargs={"reasoning_effort": "low"}, max_tokens=RESPONSE_RESERVE_TOKENS, ...)`
and `openai_fallback = ChatOpenAI(model="gpt-5-mini", ...)`.
- `build_agent()` at the bottom of the file. Inside it, `routed_model(messages, config)`
runs **once per model call**. It picks the tools, binds them, and returns
`prompt | bound_model`. **This is the hook for per-call settings.**
- `TOOL_ROUTING_MODE` env var (default `static`):
- `static`: every tool the user may use is bound on every call
(`router.select_static`). There is no `discover_tool_groups` tool.
- `dynamic`: `router.select(messages)` picks tool groups from keywords in
the user's message, and the model can call `discover_tool_groups` to
unlock more groups on the next call.
- Every ticket below must work in **both** modes. Tests should cover both
where it matters.
- `blueprints/conversation/tool_routing.py`: `ToolRouter`, `DISCOVERY_NAME = "discover_tool_groups"`,
`GROUP_TERMS` keyword lists, `select()` / `select_static()`.
- `blueprints/conversation/__init__.py`: the streaming endpoint. `_chat_run_stream`
→ `event_generator` tracks every model call in the `model_calls` dict
(keyed by LangChain `run_id`) from `on_chat_model_start` / `_stream` / `_end`
events, and `_log_summary()` writes one JSON log line with
`"event": "stream_request"` per user turn. `_model_call_summary()` builds the
per-call dict (`prefill_ms`, `ms`, `prompt_tokens`, `output_tokens`, ...).
- `blueprints/conversation/context.py`: prompt budgeting (`CONTEXT_WINDOW_TOKENS`,
`PROMPT_BUDGET_TOKENS`, `RESPONSE_RESERVE_TOKENS=4096`). You should not need
to change it.
### The message sequence inside one user turn
LangGraph's ReAct agent loops: model call → (tools run) → model call → ...
until the model answers without tool calls. The `messages` list that
`routed_model` receives grows like this:
```
... earlier turns ...
HumanMessage("how much did we spend on groceries last month?") <- call 1 sees this as the last message
AIMessage(tool_calls=[ynab_spending_report(...)]) <- call 1 output
ToolMessage(name="ynab_spending_report", content="{...}") <- call 2 sees this as the last message
AIMessage("You spent $612 on groceries...") <- call 2 output (final answer)
```
In `dynamic` mode there can be a discovery hop:
```
HumanMessage("...")
AIMessage(tool_calls=[discover_tool_groups(groups=["finance"])])
ToolMessage(name="discover_tool_groups", content='{"enabled_tool_groups": ["finance"], ...}')
AIMessage(tool_calls=[ynab_spending_report(...)]) <- this call needs no real thinking
ToolMessage(name="ynab_spending_report", ...)
AIMessage("final answer")
```
The model can also make **several tool calls in one AIMessage** (parallel
calls), which produces several `ToolMessage`s in a row. `ToolMessage.name` is
set to the tool's name by LangChain/LangGraph for both success and error
results. `ToolMessage.status` is `"success"` or `"error"`.
### Ground rules for every ticket
1. **One ticket = one branch = one PR.** Branch from up-to-date `main`, e.g.
`git checkout main && git pull && git checkout -b feat/eff-1-call-kind-logging`.
2. Line numbers drift. Find code by function/constant name, not by line number.
3. Match surrounding style: plain functions, short docstrings, comments only
where the "why" isn't obvious. Don't reformat unrelated code.
4. Run the unit tests before pushing:
`.venv/bin/pytest tests/unit -q` (or `make test`, which needs `uv`).
All tests must pass, including ones you didn't touch.
5. If you add an environment variable you must add it to **all four** of:
`.env.example`, `docker-compose.yml` (the `environment:` list of the
raggr services), `config/validation.py` (defaults/validation), and the
real `.env` if present. Prefer module constants unless the ticket says
otherwise; fewer knobs is better.
6. Do not write DB migrations by hand. None of these tickets should need a
migration; if you think one does, stop and ask.
7. PRs go to Gitea, not GitHub: `git push -u origin <branch>` then
`tea pr create -t "title" -d "body"`.
8. Never point tests at the live llama-server. Unit tests must run offline.
Only EFF-3's script talks to the real server, and it is not a pytest test.
9. Don't commit `.DS_Store`, `chroma_db/`, `chromadb/`, `.planning/`.
10. **You probably cannot reach the llama-server or production.** It lives on
Ryan's home LAN (`192.168.1.7`), and production logs/DB are on his server.
Don't try to work around that, and don't block on it. Every step below
that needs the live server or prod data is marked **[Ryan runs]**: write
the code so Ryan can run that step with one copy-pasteable command, put
that exact command in the PR description under a "For Ryan to run"
heading, and finish the rest of the ticket. If you *can* reach
`http://192.168.1.7:8085/health` (e.g. running locally on his network),
you may run those steps yourself; they only generate text and change
nothing on the server.
---
## EFF-1: Classify each model call and log the classification
**Size:** small. **Depends on:** nothing. **Blocks:** EFF-2, EFF-4.
### Why
We want to know where the generated tokens go: is it the first call of a turn,
the "now call the tool you just unlocked" call after `discover_tool_groups`,
or the call that writes the answer from tool results? Today the
`stream_request` log has per-call `output_tokens` and `ms` but nothing says
which kind of call each one was. EFF-2 also needs this exact classification to
pick thinking settings, so it lives in one shared function.
### What to build
1. **New module** `blueprints/conversation/thinking.py` with:
```python
"""Per-call classification used to size the model's hidden thinking."""
from langchain_core.messages import HumanMessage, ToolMessage
from blueprints.conversation.tool_routing import DISCOVERY_NAME
FIRST = "first" # last message is the user's question
AFTER_DISCOVERY = "after_discovery" # only discover_tool_groups results since the model last spoke
AFTER_TOOLS = "after_tools" # real tool results to read (or anything else)
def classify_call(messages) -> str:
...
```
Rules, in this order:
- Collect the **trailing run** of `ToolMessage`s at the end of `messages`
(walk backwards from the end while `isinstance(m, ToolMessage)`; stop at
the first non-ToolMessage).
- If that run is non-empty **and every** message in it has
`m.name == DISCOVERY_NAME` **and** `m.status == "success"` → return
`AFTER_DISCOVERY`.
- Mixed run (discovery + a real tool in the same parallel batch) →
`AFTER_TOOLS`, because there is real data to read.
- A discovery result with `status == "error"` → `AFTER_TOOLS` (something
went wrong; let the model think).
- Else, if `messages` is non-empty and `messages[-1]` is a `HumanMessage`
→ `FIRST`.
- Else → `AFTER_TOOLS`. This is the safe default (it gets the most
thinking in EFF-2), so anything unexpected, including an empty list,
lands here.
- `messages` may contain a `SystemMessage` at the start; it does not matter
for these rules. `messages` might also be passed as a LangGraph state
dict `{"messages": [...]}` in some code paths. Look at how
`_build_prompt` in `agents.py` handles that
(`state["messages"] if isinstance(state, dict) else state`) and do the
same at the top of `classify_call`.
2. **Attach the classification to each model call** in `routed_model`
(`blueprints/conversation/agents.py`, inside `build_agent`):
- Compute `kind = classify_call(messages)`.
- Attach it as LangChain run metadata so it shows up in the streaming
events: wrap the returned chain with
`.with_config(metadata={"call_kind": kind})`. Example:
```python
chain = RunnableLambda(partial(_build_prompt, tool_schemas=schemas)) | bound
return chain.with_config(metadata={"call_kind": kind})
```
LangChain merges config metadata into child runs, so the
`on_chat_model_start` event for this call should carry
`event["metadata"]["call_kind"]`. **Verify this** in a test (see below);
if metadata does not propagate in the installed LangChain version, attach
it to `bound` instead (`bound.with_config(...)`), and re-check.
3. **Record it in the log line** in `blueprints/conversation/__init__.py`:
- In the `on_chat_model_start` branch where `model_calls[event["run_id"]] = {...}`
is created, add `"call_kind": metadata.get("call_kind")`.
- In `_model_call_summary()`, add `"call_kind": call.get("call_kind")` to
the returned dict.
- Do not change any other field names in the log line; dashboards or
scripts may read them.
### Tests (new file `tests/unit/test_thinking.py`)
Write a pure-function test for each rule. Build messages with
`HumanMessage`, `AIMessage(content="", tool_calls=[{"id": "c1", "name": "...", "args": {}}])`,
`ToolMessage(content="...", tool_call_id="c1", name="...", status="success")`.
- `[HumanMessage]` → `FIRST`.
- `[Human, AI(tool_call discover), Tool(name=discover, success)]` → `AFTER_DISCOVERY`.
- `[Human, AI(discover + ynab), Tool(discover), Tool(ynab)]` → `AFTER_TOOLS` (mixed parallel batch).
- `[Human, AI(discover), Tool(discover, status="error")]` → `AFTER_TOOLS`.
- `[Human, AI(ynab), Tool(ynab)]` → `AFTER_TOOLS`.
- Two discovery results in a row, both success → `AFTER_DISCOVERY`.
- An older discovery earlier in the turn but the last messages are a real
tool result → `AFTER_TOOLS` (only the trailing run counts).
- A previous turn ended in a tool result, then a new `HumanMessage` → `FIRST`.
- `[]` → `AFTER_TOOLS`.
- The state-dict form `{"messages": [HumanMessage(...)]}` → `FIRST`.
Plus one wiring test that the metadata reaches the streaming events. Look at
`tests/unit/test_tool_routing.py`: `import_agents()` imports `agents.py` with
all external services stubbed, and
`test_real_graph_binds_discovered_schemas_and_accounts_same_subset` builds the
real graph with a `FakeModel`. Copy that pattern: build the agent, run
`agent.astream_events(..., version="v2")` (or whatever the app uses; check
`_chat_run_stream`), collect `on_chat_model_start` events, and assert their
`metadata["call_kind"]` values are `["first", "after_discovery", ...]` for a
dynamic-mode run. If the `FakeModel` there is a plain `RunnableLambda` (not a
chat model), you may not get `on_chat_model_start` events at all. In that
case use `langchain_core.language_models.fake_chat_models.GenericFakeChatModel`
(or `FakeMessagesListChatModel`) with a `bind_tools` that returns itself, so
real chat-model events fire. Keep the test offline.
### Done when
- `classify_call` exists with the rules above and the tests pass.
- **[Ryan runs]** After deploying, a `stream_request` log line has a
`call_kind` on every entry in `model_calls`:
`make logs | grep stream_request | tail -5`.
- No behavior change for users (this ticket only adds logging).
---
## EFF-2: Size thinking per call kind (the actual speedup)
**Size:** small-medium. **Depends on:** EFF-1 (uses `classify_call`).
Ideally EFF-3 exists so you can measure before/after.
### Why
Thinking is the biggest chunk of time before the first answer token, but the
experiment in the Background section shows that killing it everywhere makes
math and date answers wrong. Some calls don't need it:
- **`AFTER_DISCOVERY`** (dynamic mode only): the model just unlocked a tool
group; all it has to do now is call the tool. Thinking here is waste.
- **`FIRST`**: usually "pick a tool and fill in arguments", but sometimes the
model answers directly (chit-chat, or a math question like the cat-food one)
with no tool. So we cap it moderately instead of turning it off.
- **`AFTER_TOOLS`**: the model reads tool results and writes the answer
(YNAB totals, comparing dates). This is where reasoning matters. We only
add a generous cap so one runaway call can't burn the whole 4,096-token
`max_tokens` on thinking and leave nothing for the answer.
### What to build
1. In `blueprints/conversation/thinking.py` (from EFF-1), add:
```python
FIRST_CALL_THINKING_BUDGET = 512
AFTER_TOOLS_THINKING_BUDGET = 1024
def thinking_options(kind: str) -> dict:
"""llama-server request fields (sent via extra_body) for one call kind."""
if kind == AFTER_DISCOVERY:
return {"chat_template_kwargs": {"enable_thinking": False}}
if kind == FIRST:
return {"reasoning_budget_tokens": FIRST_CALL_THINKING_BUDGET}
return {"reasoning_budget_tokens": AFTER_TOOLS_THINKING_BUDGET}
```
Keep these as module constants, not environment variables (see ground rule 5).
2. In `routed_model` (`agents.py`, inside `build_agent`), pass the options to
the **llama** model only:
```python
kind = classify_call(messages) # already added in EFF-1
fallback = openai_fallback.bind_tools(selected)
bound = (
llama_chat.bind_tools(selected, extra_body=thinking_options(kind)).with_fallbacks([fallback])
if llama_chat
else fallback
)
```
- **Do not** pass `extra_body` to `openai_fallback`. OpenAI's API rejects
unknown fields like `reasoning_budget_tokens`, and the fallback is what
saves us when the GPU box is down.
- Keep the existing `model_kwargs={"reasoning_effort": "low"}` on
`llama_chat`. The soft hint and the hard cap work together.
- `bind_tools(tools, **kwargs)` forwards extra kwargs into the request.
Confirm with a test (below) that `extra_body` ends up in the bound
kwargs. If `ChatOpenAI` in the installed version does not accept
`extra_body` through `bind_tools`, use
`llama_chat.bind_tools(selected).bind(extra_body=...)`.
3. Add the options to the log line so we can correlate: in `routed_model`,
extend the EFF-1 metadata to
`metadata={"call_kind": kind, "thinking": thinking_options(kind)}` and copy
`"thinking"` into `model_calls[...]` / `_model_call_summary()` the same way
as `call_kind`. (If the fallback model handled the call, the field is still
logged; that's fine, `provider` tells you which model ran.)
### Things that will break and how to fix them
- `tests/unit/test_tool_routing.py` has fake models with
`def bind_tools(self, tools):` (two places). They will now get an
unexpected `extra_body=` keyword and raise `TypeError`. Change those fakes to
`def bind_tools(self, tools, **kwargs):`. Don't change what the tests assert.
- If you used `.bind(extra_body=...)` instead, the fakes need a `.bind`
method, or make `bind_tools` return a real `Runnable` (they currently return
`RunnableLambda`, which has `.bind`). Check before assuming.
### Tests (add to `tests/unit/test_thinking.py`)
- `thinking_options(FIRST) == {"reasoning_budget_tokens": 512}`.
- `thinking_options(AFTER_DISCOVERY) == {"chat_template_kwargs": {"enable_thinking": False}}`.
- `thinking_options(AFTER_TOOLS) == {"reasoning_budget_tokens": 1024}`.
- `thinking_options("something unknown") == {"reasoning_budget_tokens": 1024}`
(safe default).
- Wiring test, modeled on `test_real_graph_binds_discovered_schemas_and_accounts_same_subset`:
a fake llama model whose `bind_tools(self, tools, **kwargs)` records
`kwargs`. Run a dynamic-mode turn that does discovery → tool → answer and
assert the recorded `extra_body` values were, in order:
`{"reasoning_budget_tokens": 512}`, `{"chat_template_kwargs": {"enable_thinking": False}}`,
`{"reasoning_budget_tokens": 1024}`.
- Same wiring test in static mode (no discovery): expect `512` then `1024`.
- Fallback test: with the primary fake set to fail, assert the fallback
fake's `bind_tools` was **never** given `extra_body`.
### Manual check against the real server (**[Ryan runs]**, optional, read-only, not a test)
You can confirm llama-server honors the fields with curl. This only generates
text; it changes nothing on the server:
```bash
curl -s http://192.168.1.7:8085/v1/chat/completions -H 'content-type: application/json' -d '{
"model": "qwen3.8-27b", "max_tokens": 1024, "reasoning_effort": "low",
"reasoning_budget_tokens": 64,
"messages": [{"role": "user", "content": "What is 17*23?"}]
}' | python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print(len(m.get("reasoning_content") or ""), d["timings"]["predicted_n"], m["content"])'
```
A small `reasoning_content` length and `predicted_n` around the budget means
the cap works.
### Done when
- Tests above pass, and the whole unit suite passes.
- **[Ryan runs]** EFF-3's evaluation shows: median time-to-first-answer-token lower than
before, **no drop** in tool-choice accuracy, **zero** malformed tool calls,
and no new wrong answers on the arithmetic/date cases. If the arithmetic
cases regress, raise `FIRST_CALL_THINKING_BUDGET` (try 1024) before anything
else, and say so in the PR.
- The PR description includes the exact commands for Ryan to produce a
before/after comparison: EFF-3 run on `main` with `--label before`, then on
this branch with `--label after`, then the compare command. If EFF-3 isn't
merged yet, instead give him the log query for ~10 real chats (`ttft_ms` and
per-call `output_tokens` from `stream_request` lines, grouped by
`call_kind`). Ryan merges only after those numbers look right, so don't
merge it yourself.
---
## EFF-3: Model-level A/B evaluation script
**Size:** medium. **Depends on:** nothing (but EFF-1/2 make it more useful).
### Why
`scripts/evaluate_context.py` only tests history trimming. It never calls a
model, so it can't tell us whether a model or server change made answers
slower or worse. We need a repeatable script that sends realistic agent
requests to whatever `LLAMA_SERVER_URL` points at and reports speed and
correctness, so any change (EFF-2, new llama-server flags, a different model)
can be judged with numbers.
### What to build
1. **Fixture file** `tests/fixtures/model_eval/cases.json`: a list of cases.
Each case is one **single model call** (not a full agent loop; that would
need the DB, Paperless, YNAB, etc.). Shape:
```json
{
"name": "ynab_groceries_last_month",
"kind": "tool_choice",
"messages": [
{"role": "user", "content": "How much did we spend on groceries last month?"}
],
"expect_tool": "ynab_spending_report",
"expect_args_subset": {"category": "Groceries"}
}
```
Three kinds of case:
- `"tool_choice"`: the user message alone; pass if the model's first
response calls `expect_tool`, and `expect_args_subset` (optional) keys
match the call's arguments (string compare, case-insensitive).
- `"answer"`: messages include a canned assistant tool call **and** a canned
tool result (role `"tool"`, realistic JSON copied from a real tool's
output shape; look at the YNAB tools in `blueprints/ynab/tools.py` and
the artifact shapes in `tests/fixtures/context/scenarios.json`); pass if
the final answer text contains every string in `expect_contains`
(case-insensitive) and calls no tool.
- `"direct"`: no tool should be called; pass if the answer contains
`expect_contains`. Include the cat-food question from the Background
section (`expect_contains: ["21"]`; the correct date is Sept 21).
Write **at least 25 cases** covering: pets/Simba (`simba_search`,
`simbakit_*`), meals (`mealie_*`), notes/journal (`obsidian_*`,
`journal_*`), YNAB spending/balances (`ynab_*`), restaurants (`beli_*`),
web search, date questions ("what's on Friday"), 2-3 follow-up questions
that only make sense with a previous turn ("what about last month?"; put
the previous turn in `messages`), at least 3 `"answer"` cases with
arithmetic over tool results, and 2 `"direct"` reasoning questions. Use
the real tool names: get them from `agents.py` (`@tool` functions) and
`blueprints/ynab/tools.py`. No real personal data in fixtures; invent
plausible numbers.
2. **Script** `scripts/evaluate_model.py`, run as
`.venv/bin/python -m scripts.evaluate_model [--cases PATH] [--repeat N] [--label NAME] [--out results.json]`:
- Reads `LLAMA_SERVER_URL` and `LLAMA_MODEL_NAME` from the environment
(`load_dotenv()` like the rest of the app). Allow `--base-url` to
override, because the `.env` value is stale; the real server is
`http://192.168.1.7:8085/v1`.
- For the system prompt and tool schemas, use the **real ones** so the
eval matches production: `SIMBA_SYSTEM_PROMPT` from
`blueprints/conversation/prompts.py`, and tool schemas via
`convert_to_openai_tool(t)` for the tools in `agents.tools` (plus YNAB
tools if they're in that list). Importing `agents.py` initializes
services; use the same stubbing trick as `import_agents()` in
`tests/unit/test_tool_routing.py` so the import works offline. Bind all
tools (static mode) unless `--routing dynamic` is given, in which case
use `ToolRouter(...).select(messages)` to pick them like production does.
- If `blueprints/conversation/thinking.py` exists (EFF-2 merged), apply
`thinking_options(classify_call(...))` per case so the eval reflects
production; add `--no-thinking-policy` to compare against today's
behavior.
- Call the server with the plain `openai` client (`OpenAI(base_url=..., api_key="not-needed")`),
**streaming**, with `stream_options={"include_usage": True}`,
`reasoning_effort="low"`, `max_tokens=4096`, and any thinking fields in
`extra_body`. Streaming lets you measure:
- `ttft_ms`: time from request to the first **content or tool-call**
delta (ignore deltas that only carry `reasoning_content`).
- `total_ms`, `completion_tokens` and `prompt_tokens` (from `usage`).
- llama-server also sends a non-standard `timings` object on the final
chunk with `predicted_per_second`, `prompt_per_second`, `draft_n`,
`draft_n_accepted`. Read it from the raw chunk
(`chunk.model_extra.get("timings")` with the openai client) and record
it if present; don't fail if absent.
- Grade each case per its `kind` (rules above). A tool call whose
arguments aren't valid JSON counts as **malformed**: record it
separately, it's the worst failure.
- Output: a table to stdout (one row per case: name, pass/fail, ttft_ms,
total_ms, completion_tokens, tok/s, draft acceptance) and a summary:
pass rate per kind, malformed count, median and p90 of ttft_ms and
total_ms, median completion_tokens. With `--out`, write all raw
results plus the summary as JSON, including `--label`, the base URL,
the model name, and a timestamp.
- `--repeat N` runs each case N times (default 1). Sampling is random
(server default temperature 1.0), so use `--repeat 3` for real
comparisons and report pass rate as passes/(cases×N).
- Add `scripts/compare_model_eval.py a.json b.json` (or a `--compare`
flag) that prints the two summaries side by side with deltas.
- Exit code 0 always (it's a report, not a gate), except exit 2 on
connection errors.
3. **Docs:** add a short section to `docs/efficiency/README.md` (create it)
explaining how to run it, with an example command and how to read the
output. One paragraph plus commands is enough.
### Tests
The script itself talks to a live server, so it is **not** a unit test. But
add `tests/unit/test_model_eval.py` for the pure parts:
- The grading function for each kind, including malformed JSON arguments and
case-insensitive substring matching.
- Loading `cases.json` validates required keys per kind (fail fast with a
clear message naming the bad case).
- The summary math (median/p90) on a small hand-made list.
Keep these offline: no `OpenAI` client construction in tests.
### Done when
- The unit tests pass.
- The script runs end to end against a **fake** server so you can prove it
works without the LAN: add `--base-url` support and test it manually
against a tiny local stub (a ~40-line `http.server` or `aiohttp` script
that returns a canned streaming chat-completion with one tool call, a
`usage` chunk and a `timings` object). Don't commit the stub unless it's
clean enough to live in `tests/fixtures/model_eval/` and is used by a test.
- **[Ryan runs]** The PR description ends with the real command for Ryan:
`.venv/bin/python -m scripts.evaluate_model --base-url http://192.168.1.7:8085/v1 --repeat 3 --label baseline --out /tmp/baseline.json`
---
## EFF-4: Fewer discovery round-trips in dynamic routing mode
**Size:** small. **Depends on:** EFF-1 (to measure). **Only matters if
production runs `TOOL_ROUTING_MODE=dynamic`.** You can't see the production
`.env`; ask Ryan which mode production uses before starting. If production is `static`, skip this
ticket and note that in the PR list.
### Why
Every `discover_tool_groups` call is a full extra model round-trip: prompt
processing + generated tokens + the tool call. And because the template puts
the tool definitions at the very top of the prompt, the next call has a
different tool list, so llama-server's prompt cache misses and it reprocesses
the whole prompt. Even with EFF-2 making that call cheaper, it's cheapest not
to make it. Most discovery calls happen because the user's words didn't match
`GROUP_TERMS` in `tool_routing.py`.
### What to do
1. **Collect data first. [Ryan runs]** After EFF-1 is deployed for a few days, pull the
user messages that led to discovery. The `stream_request` log has
`tools` (tool spans, with names) and `model_calls` (with `call_kind`).
A turn where a `discover_tool_groups` tool span appears is a miss. The
user's message text is in the DB (`conversation` messages), matched by
the `conv` id in the log line. Write a small script (Ryan runs it on the
server; commit it under `scripts/` only if it's clean and reusable) that
lists, per missed turn: the
user message and the groups the model asked for.
2. For each miss, add the missing everyday words to the right group's string
in `GROUP_TERMS`. Example: if "what did the paycheck cover this month?"
triggered discovery of `finance`, add `paycheck paychecks` to the
`finance` string. Before adding a word, ask: "would this word show up in
questions that have nothing to do with this group?" If yes, don't add it,
because extra tools on every call cost prompt tokens and confuse the model.
Rules of thumb:
- Add nouns and verbs specific to the domain ("groceries", "paycheck",
"litter"); avoid generic words ("left", "show", "get", "much").
- A word can be in two groups if it really belongs to both.
- Remember matching is on whole lowercase words (`words()` splits on
non-alphanumerics), so add plural forms separately ("bill bills").
3. Add a test per new word family in `tests/unit/test_tool_routing.py` in the
existing style: `names(router, [HumanMessage(content="<the real user phrasing>")])`
includes the expected tool.
### Done when
- Every miss you collected now routes without discovery (tests prove it).
- No existing routing test changed its expectations.
- PR description lists the misses found and the words added.
---
## Manual / ops follow-ups (for Ryan, not for Codex)
These need the GPU box and are listed so they aren't forgotten:
- Try `--spec-draft-n-max 4` (tool-call output accepted ~92% of drafts at 3;
depth 4 sometimes wins on JSON-heavy output). Compare with EFF-3,
`--repeat 3`.
- Try `--spec-type draft-mtp,ngram-mod` (n-gram drafting could help when the
model copies IDs from tool results into arguments). Unverified; A/B with
EFF-3.
- Record the final llama-server unit (flags and why) in `docs/deployment.md`
so the server config isn't only on the GPU box.
- Old unit is backed up at `/etc/systemd/system/llama-server.service.bak`.