Keep the chat prompt prefix stable for llama-server prompt caching #77

2026-10-12T23:05:53-04:00

ryan commented

2026-10-12 23:05:53 -04:00

llama-server reuses a prompt prefix only when it is byte-identical. Simulating conversations through the real build_context / ToolRouter / SummaryStore showed ~80% of prompt tokens re-prefilled every turn on main.

Changes:

Memories: rank still decides which memories fit the budget, but they're emitted in stored order, so the system prompt no longer changes per query.
Tools (TOOL_ROUTING_MODE, default static): bind the full role-appropriate tool set in a fixed order, with no discover_tool_groups round trip. Admin gating is unchanged. dynamic keeps the keyword router for comparison. Costs ~1.7k more (cached) tool tokens per call.
History trimming: on overflow, drop old turns in blocks down to a 65% watermark, so the cut point (and the retained prefix) stays put across several turns.
Summary ledger: only included when turns were actually dropped (it duplicated retained turns before). It's placed just before the current turn instead of at the head, with a fixed 2,000-token reserve so its size can't move the cut. It supersedes the dropped-turn excerpts as before.
Logging: stream_request model_calls now record cached_prompt_tokens when the provider reports them, to confirm the win in production.

Re-prefilled share of turn-start prompts (simulation):

scenario	main	this PR
8 turns	80.1%	2.3%
30 turns, no summary store	84.2%	3.3%
30 turns, with summary store (prod)	85.5%	17.8%

Tests: 367 passed.

🤖 Generated with Claude Code

llama-server reuses a prompt prefix only when it is byte-identical. Simulating conversations through the real build_context / ToolRouter / SummaryStore showed ~80% of prompt tokens re-prefilled every turn on main. Changes: - **Memories**: rank still decides which memories fit the budget, but they're emitted in stored order, so the system prompt no longer changes per query. - **Tools** (TOOL_ROUTING_MODE, default `static`): bind the full role-appropriate tool set in a fixed order, with no discover_tool_groups round trip. Admin gating is unchanged. `dynamic` keeps the keyword router for comparison. Costs ~1.7k more (cached) tool tokens per call. - **History trimming**: on overflow, drop old turns in blocks down to a 65% watermark, so the cut point (and the retained prefix) stays put across several turns. - **Summary ledger**: only included when turns were actually dropped (it duplicated retained turns before). It's placed just before the current turn instead of at the head, with a fixed 2,000-token reserve so its size can't move the cut. It supersedes the dropped-turn excerpts as before. - **Logging**: stream_request model_calls now record cached_prompt_tokens when the provider reports them, to confirm the win in production. Re-prefilled share of turn-start prompts (simulation): | scenario | main | this PR | |---|---|---| | 8 turns | 80.1% | 2.3% | | 30 turns, no summary store | 84.2% | 3.3% | | 30 turns, with summary store (prod) | 85.5% | 17.8% | Tests: 367 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ryan added 3 commits 2026-10-12 23:05:53 -04:00

Keep the chat prompt prefix stable for llama-server prompt caching 891f44c1a9

llama-server only reuses a byte-identical prompt prefix. Several things
changed the prefix every turn, forcing most of the prompt to be re-prefilled:

- Memories: rank still chooses which memories fit the budget, but the chosen
  ones are emitted in stored order, so the block is identical across queries.
- History trimming: the conversation-evidence summary is only included when
  turns were actually dropped (it duplicated retained turns otherwise). On
  overflow, old turns are dropped at stable cut points down to ~65% of the
  budget, so the retained prefix holds for several turns instead of shifting
  by one turn every request.
- Tools: TOOL_ROUTING_MODE=static (default) binds the full role-appropriate
  tool set in registration order with no discovery tool; admin gating is
  unchanged. TOOL_ROUTING_MODE=dynamic keeps the keyword router.
- Observability: per-call model_calls in the stream_request log now include
  cached_prompt_tokens when the provider reports cache reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Place conversation evidence before the current turn for prompt caching 1e638e1e3b

The summary ledger is regenerated every turn. At the head of history it
invalidated the cached retained turns on every request once trimming
started. Insert it immediately before the current turn's HumanMessage so
system, tools and retained history stay byte-identical; excerpts of dropped
turns stay at the head since they only change when the cut moves.

Reserve a fixed ledger ceiling (SUMMARY_RESERVE_TOKENS) rather than its
current size, so a shrinking ledger cannot pull the cut back and change the
retained prefix; fall back to the actual size for tight budgets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Drop dropped-turn excerpts when the summary ledger is included 266c6a8d79

The ledger supersedes the "Earlier conversation excerpts" message, as on
main; sending both duplicated dropped-turn content. Removal is consistent
whenever the ledger is included, so the cached prefix stays stable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ryan merged commit e0ab2657ed into main

2026-10-12 23:06:08 -04:00

ryan referenced this pull request

2026-10-13 03:40:40 -04:00

Answer concisely by default and drop prompt text the tool schemas repeat #82

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ryan/simbarag#77