Route tools and retain structured conversation context with durable evidence

docs/context-management.md

@@ -1,43 +1,180 @@
 # Conversation context management
 
-The LangGraph checkpoint retains original messages and artifacts. Each model
-call constructs a bounded view; compaction never rewrites the checkpoint.
+LangGraph checkpoints retain original messages and artifacts. `context.py`
+constructs a bounded model view with copies; it does not rewrite checkpoints.
 
-The prompt budget includes system instructions, saved memories, tool schemas
-(including admin tools), message content and tool arguments. The default target
-is 16,000 estimated tokens. A 4,096-token generation limit and 2,048-token safety
-reserve also constrain the budget against the configured 32,768-token window.
+The prompt budget includes system instructions, saved memories, selected tool
+schemas (including admin tools), message content, tool arguments, and any durable
+summary. Defaults are a 16,000-token prompt target, 4,096-token response reserve,
+2,048-token safety reserve, and 32,768-token context window. Estimates use UTF-8
+bytes divided by three, message overhead, and a fixed image allowance, rather
+than a model-specific tokenizer. Compare estimates with actual model usage.
 
-Estimates use UTF-8 bytes divided by three, message overhead, and a fixed image
-allowance. These are estimates rather than model-specific tokenizer counts.
-Compare `prompt_budget.estimated_prompt_tokens` with actual `stream_request`
-model-call usage and adjust the target/window for the deployed backend.
+## Structured tool compaction
 
-The current user request is always retained. Oversized tool results become
-marked excerpts preserving their beginning and end; all tool IDs and arguments
-remain. If necessary, intermediate assistant text is also shortened. If the
-mandatory request and exchange structure cannot fit, the request fails with an
-explicit error rather than silently dropping the question.
+`compact_tool_result(message, limit)` prefers `ToolMessage.artifact` dictionaries
+or lists, then parseable JSON content. Small payloads remain whole. Large
+payloads retain metadata and compact recognized row collections (`transactions`,
+`rows`, `results`, `documents`, `categories`, `accounts`, `payees`) and text fields
+(`markdown`, `snippet`, `content`, `text`, `memo`, `description`). Output remains
+valid JSON. Rows are sampled in source order without inventing aggregate totals.
 
-Recent history is selected as complete turns. Older user messages, assistant
-answers and tool results are capped; earlier images become placeholders.
-Incomplete tool exchanges from cancelled calls and orphan results are removed
-from the prompt. A small budget is reserved for bounded question/answer excerpts
-from older turns. These excerpts are not an LLM-generated rolling summary and do
-not guarantee retention of every historical decision. No extra model call is
-added to the request path. Saved memories use a separate 1,000-token estimated
-budget, preferring recent entries without deleting omitted memories.
+Each shortened collection reports the returned, retained, and omitted row counts,
+a notice to consult the original artifact/source tool, and omitted source
+references. IDs, paths, dates and source references remain available even when
+row details are omitted. Top-level totals, dates, counts, filters, query, source,
+source references, and note frontmatter remain intact. Text fields use explicit
+prefix excerpts with omitted character counts. Counts describe rows in the
+artifact; a separate upstream `count` can describe a larger query result.
 
-Obsidian reads accept `start_line` (one-based content line, excluding frontmatter)
-and `max_lines`, allowing focused follow-up reads. Full artifact content remains
-available to the interface and checkpoint.
+The artifact is authoritative for this projection. Tools must include relevant
+filters, dates, totals and source IDs in their artifacts or call arguments;
+compaction cannot reconstruct information absent from those inputs. Unknown
+non-JSON output falls back to marked beginning/end excerpts. Structured metadata
+has a minimum size: if required references and exchange structure exceed the
+budget, trimming raises `ValueError` instead of cutting JSON or hiding omissions.
 
-Configuration:
+The current user request and tool arguments remain intact. All retained exchanges
+include the calling assistant and every parallel result. Orphan results and
+unfinished exchanges from cancellations are excluded from the model view.
+Intermediate assistant prose can be shortened. Older turns are selected whole;
+older message text is bounded and images become placeholders. If no durable
+summary is supplied, bounded user/assistant excerpts provide limited continuity.
+Historical excerpts use the assistant role, never the system role.
+
+## Persistent extractive evidence
+
+`summary.py` provides `SummaryStore(path)` using the existing `aiosqlite`
+dependency. Pass a persistent SQLite file on a writable local volume; its parent
+directory must exist. The store creates its own table. `UserMemory` was inspected:
+it stores user-wide explicit facts and has no conversation key or cursor. It is
+not reused for this state. ORM registration already includes
+`blueprints.conversation.models`; this sidecar requires no ORM model or migration.
+
+The store is keyed by `(user_id, conversation_id)` and provides:
+
+```python
+store = SummaryStore("/persistent-data/conversation-summaries.sqlite3")
+state = await store.update(user_id, conversation_id, checkpoint_messages)
+# Or reuse already persisted evidence:
+state = await store.load(user_id, conversation_id)
+summary_message = state.as_message()  # AIMessage, or None for empty evidence
+prompt = build_context(
+    checkpoint_messages, system_prompt, selected_tool_schemas,
+    summary_message=summary_message,
+)
+```
+
+`update` requires a full, stable, append-only snapshot of checkpoint messages,
+including the new current HumanMessage. It summarizes only messages preceding
+that newest user turn; the current request is preserved in full context. The
+last completed turn enters evidence when the next user request arrives. Only
+user text and assistant text without tool calls enter evidence. Tool content,
+artifacts, and assistant tool-call prose never become summary instructions.
+
+The state contains a version, incremental message-index cursor, checkpoint-prefix
+SHA-256 digest, omitted-entry count, and four evidence buckets: constraints,
+decisions, open work, and recent context. Labels use simple keyword heuristics;
+they do not assert that a task was resolved or a decision approved. Each excerpt
+retains its original role, message ID/index and an excerpt marker. Maximums are
+900 UTF-8 bytes per excerpt, six entries per bucket, and 1,800 estimated tokens
+for serialized state. Old entries are evicted from the largest bucket to respect
+the total bound. The assistant-role JSON envelope adds a small charged overhead
+and explicitly labels the contents untrusted historical evidence. Historical
+text is never promoted to privileged system instructions; role separation is
+not a guarantee that a model will resist every prompt injection.
+
+SQLite transactions serialize concurrent processes. Repeated and stale updates
+cannot duplicate evidence or move the cursor backward. A changed summarized
+prefix raises `ValueError`, leaving persisted state unchanged. Checkpoint resets,
+branching, or forks need a new conversation key; do not silently reuse old state.
+Snapshots shorter than the saved cursor are treated as stale; the store cannot
+distinguish such a snapshot from a reset until enough history is available for
+prefix validation. The caller must enforce append-only checkpoint lineage.
+Prefix hashing runs in a thread so large artifact serialization does not block
+the async event loop. It still reads the historical prefix to validate lineage;
+incremental extraction does not make validation constant-time. Failed or
+cancelled transactions roll back on connection close.
+
+This is bounded extractive evidence, not a semantic rolling summary. It can lose
+old decisions, miss constraints in long attachments, retain outdated open-work
+phrases, and label text incorrectly. Keep the checkpoints as the complete source
+of truth. No extra remote/model calls are made, and tests use only local SQLite.
+
+## Request integration
+
+Initialize a store with a durable path and call `update` after obtaining the full
+checkpoint plus the new user message. Pass `state.as_message()` in request-local
+agent config and forward it to `build_context(..., summary_message=...)` from the
+prompt adapter. Do not append this synthetic message to checkpoint history or
+share mutable summary state on a global agent. Scope keys to the authenticated
+user and conversation. Empty evidence yields `None`. Summary tokens are charged
+before history selection. Optional summary evidence is omitted when it would
+prevent the mandatory current request from fitting.
+
+Request handlers can await the local async update, or schedule it as a tracked
+background task and use `load` for the current request (allowing one request of
+lag). The caller must own task lifetime, failure logging, and shutdown draining for
+background updates. Run updates on full checkpoint snapshots, not model-trimmed
+history or `ConversationMessage` display text, which can be truncated. Attach a
+current human request before updating; otherwise the last user turn is deliberately
+left unsummarized. The app initializes the sidecar next to the checkpoint database (override with
+`CONVERSATION_SUMMARY_DB_PATH`). Authorized conversation deletion also deletes
+its summary. Include the sidecar in checkpoint backups. SQLite is suited to workers sharing a local
+volume; replicas without shared storage need an equivalent transactional database
+adapter, not independent sidecars. No change to `models.py` is required for the
+implemented store.
+
+## Configuration and observability
 
 - `CONTEXT_WINDOW_TOKENS`: smallest supported context window across backends.
-- `PROMPT_BUDGET_TOKENS`: prompt target, including fixed instructions and tools.
+- `PROMPT_BUDGET_TOKENS`: prompt target including fixed instructions and tools.
 - `RESPONSE_RESERVE_TOKENS`: maximum generated tokens and reserved answer space.
 
-`context_budget` logs history estimates, message counts, dropped turns and excerpt
+`context_budget` logs estimates, counts, dropped turns and transient excerpt
 inclusion. `prompt_budget` logs fixed overhead and the complete prompt estimate.
-Neither log includes message content.
+Neither logs message content. Saved user memories have a separate estimated
+1,000-token budget without deleting omitted memories. Obsidian reads accept
+one-based `start_line` (excluding frontmatter) and `max_lines` for focused reads;
+full artifacts remain available to the interface and checkpoint.
+
+## Tool routing and saved memories
+
+Each model call binds tool groups selected from the current query and relevant
+follow-up context. A small `discover_tool_groups` tool can list available groups
+or enable additional groups for the next call. Discovery grants are stored in
+that turn's tool results, never global mutable state. Primary and fallback models
+receive the same selected schemas, and prompt budgeting charges that same set.
+Admin-only groups remain subject to authenticated invocation permissions.
+Routing is lexical; unknown requests can use discovery to recover.
+
+Saved memories rank query-word overlap ahead of a small recency bonus, retaining
+a bounded injected block. Selection does not delete stored memories. Requests
+pass the user's query (rather than full attachment contents) into retrieval.
+
+## Stream measurements
+
+`stream_request.tokens` is the sum of actual backend output-token usage only
+when usage is available for every observed model call. Otherwise it is null;
+`completed_output_tokens` reports the known subtotal and `usage_complete` is false.
+`content_chunks` counts emitted content segments, not model tokens.
+`model_to_first_content_ms` is elapsed time including tools and model calls, not
+a measurement of reasoning alone. Each model call includes its model, provider,
+and reasoning-token usage when available. Prometheus records success, error,
+stalled and cancelled outcomes, including failures during final persistence.
+
+## Offline context regression evaluation
+
+Run `.venv/bin/python -m scripts.evaluate_context` to evaluate representative
+finance, note and transaction-search fixtures. The report includes estimated
+input/output tokens, required-fact retention, budget compliance and local
+compaction time. CI tests enforce those facts and budget bounds. These are
+context-retention checks, not measurements of generated-answer accuracy or
+backend latency; live conversation quality and actual usage still need to be
+compared after deployment.
+
+Focused Obsidian reads provide a `model_context` projection so the requested
+line range survives compaction while the full artifact remains in the interface.
+Note excerpts include bounded heading indexes with content line numbers for
+follow-up reads. Repeated structured compaction accumulates omission counts and
+source references without overwriting earlier omissions.