Conversation context management

LangGraph checkpoints retain original messages and artifacts. context.py constructs a bounded model view with copies; it does not rewrite checkpoints.

The prompt budget includes system instructions, saved memories, selected tool schemas (including admin tools), message content, tool arguments, and any durable summary. Defaults are a 16,000-token prompt target, 4,096-token response reserve, 2,048-token safety reserve, and 32,768-token context window. Estimates use UTF-8 bytes divided by three, message overhead, and a fixed image allowance, rather than a model-specific tokenizer. Compare estimates with actual model usage.

Structured tool compaction

compact_tool_result(message, limit) prefers ToolMessage.artifact dictionaries or lists, then parseable JSON content. Small payloads remain whole. Large payloads retain metadata and compact recognized row collections (transactions, rows, results, documents, categories, accounts, payees) and text fields (markdown, snippet, content, text, memo, description). Output remains valid JSON. Rows are sampled in source order without inventing aggregate totals.

Each shortened collection reports the returned, retained, and omitted row counts, a notice to consult the original artifact/source tool, and omitted source references. IDs, paths, dates and source references remain available even when row details are omitted. Top-level totals, dates, counts, filters, query, source, source references, and note frontmatter remain intact. Text fields use explicit prefix excerpts with omitted character counts. Counts describe rows in the artifact; a separate upstream count can describe a larger query result.

The artifact is authoritative for this projection. Tools must include relevant filters, dates, totals and source IDs in their artifacts or call arguments; compaction cannot reconstruct information absent from those inputs. Unknown non-JSON output falls back to marked beginning/end excerpts. Structured metadata has a minimum size: if required references and exchange structure exceed the budget, trimming raises ValueError instead of cutting JSON or hiding omissions.

The current user request and tool arguments remain intact. All retained exchanges include the calling assistant and every parallel result. Orphan results and unfinished exchanges from cancellations are excluded from the model view. Intermediate assistant prose can be shortened. Older turns are selected whole; older message text is bounded and images become placeholders. If no durable summary is supplied, bounded user/assistant excerpts provide limited continuity. Historical excerpts use the assistant role, never the system role.

Persistent extractive evidence

summary.py provides SummaryStore(path) using the existing aiosqlite dependency. Pass a persistent SQLite file on a writable local volume; its parent directory must exist. The store creates its own table. UserMemory was inspected: it stores user-wide explicit facts and has no conversation key or cursor. It is not reused for this state. ORM registration already includes blueprints.conversation.models; this sidecar requires no ORM model or migration.

The store is keyed by (user_id, conversation_id) and provides:

store = SummaryStore("/persistent-data/conversation-summaries.sqlite3")
state = await store.update(user_id, conversation_id, checkpoint_messages)
# Or reuse already persisted evidence:
state = await store.load(user_id, conversation_id)
summary_message = state.as_message()  # AIMessage, or None for empty evidence
prompt = build_context(
    checkpoint_messages, system_prompt, selected_tool_schemas,
    summary_message=summary_message,
)

update requires a full, stable, append-only snapshot of checkpoint messages, including the new current HumanMessage. It summarizes only messages preceding that newest user turn; the current request is preserved in full context. The last completed turn enters evidence when the next user request arrives. Only user text and assistant text without tool calls enter evidence. Tool content, artifacts, and assistant tool-call prose never become summary instructions.

The state contains a version, incremental message-index cursor, checkpoint-prefix SHA-256 digest, omitted-entry count, and four evidence buckets: constraints, decisions, open work, and recent context. Labels use simple keyword heuristics; they do not assert that a task was resolved or a decision approved. Each excerpt retains its original role, message ID/index and an excerpt marker. Maximums are 900 UTF-8 bytes per excerpt, six entries per bucket, and 1,800 estimated tokens for serialized state. Old entries are evicted from the largest bucket to respect the total bound. The assistant-role JSON envelope adds a small charged overhead and explicitly labels the contents untrusted historical evidence. Historical text is never promoted to privileged system instructions; role separation is not a guarantee that a model will resist every prompt injection.

SQLite transactions serialize concurrent processes. Repeated and stale updates cannot duplicate evidence or move the cursor backward. A changed summarized prefix raises ValueError, leaving persisted state unchanged. Checkpoint resets, branching, or forks need a new conversation key; do not silently reuse old state. Snapshots shorter than the saved cursor are treated as stale; the store cannot distinguish such a snapshot from a reset until enough history is available for prefix validation. The caller must enforce append-only checkpoint lineage. Prefix hashing runs in a thread so large artifact serialization does not block the async event loop. It still reads the historical prefix to validate lineage; incremental extraction does not make validation constant-time. Failed or cancelled transactions roll back on connection close.

This is bounded extractive evidence, not a semantic rolling summary. It can lose old decisions, miss constraints in long attachments, retain outdated open-work phrases, and label text incorrectly. Keep the checkpoints as the complete source of truth. No extra remote/model calls are made, and tests use only local SQLite.

Request integration

Initialize a store with a durable path and call update after obtaining the full checkpoint plus the new user message. Pass state.as_message() in request-local agent config and forward it to build_context(..., summary_message=...) from the prompt adapter. Do not append this synthetic message to checkpoint history or share mutable summary state on a global agent. Scope keys to the authenticated user and conversation. Empty evidence yields None. Summary tokens are charged before history selection. Optional summary evidence is omitted when it would prevent the mandatory current request from fitting.

Request handlers can await the local async update, or schedule it as a tracked background task and use load for the current request (allowing one request of lag). The caller must own task lifetime, failure logging, and shutdown draining for background updates. Run updates on full checkpoint snapshots, not model-trimmed history or ConversationMessage display text, which can be truncated. Attach a current human request before updating; otherwise the last user turn is deliberately left unsummarized. The app initializes the sidecar next to the checkpoint database (override with CONVERSATION_SUMMARY_DB_PATH). Authorized conversation deletion also deletes its summary. Include the sidecar in checkpoint backups. SQLite is suited to workers sharing a local volume; replicas without shared storage need an equivalent transactional database adapter, not independent sidecars. No change to models.py is required for the implemented store.

Configuration and observability

CONTEXT_WINDOW_TOKENS: smallest supported context window across backends.
PROMPT_BUDGET_TOKENS: prompt target including fixed instructions and tools.
RESPONSE_RESERVE_TOKENS: maximum generated tokens and reserved answer space.

context_budget logs estimates, counts, dropped turns and transient excerpt inclusion. prompt_budget logs fixed overhead and the complete prompt estimate. Neither logs message content. Saved user memories have a separate estimated 1,000-token budget without deleting omitted memories. Obsidian reads accept one-based start_line (excluding frontmatter) and max_lines for focused reads; full artifacts remain available to the interface and checkpoint.

Tool routing and saved memories

Each model call binds tool groups selected from the current query and relevant follow-up context. A small discover_tool_groups tool can list available groups or enable additional groups for the next call. Discovery grants are stored in that turn's tool results, never global mutable state. Primary and fallback models receive the same selected schemas, and prompt budgeting charges that same set. Admin-only groups remain subject to authenticated invocation permissions. Routing is lexical; unknown requests can use discovery to recover.

Saved memories rank query-word overlap ahead of a small recency bonus, retaining a bounded injected block. Selection does not delete stored memories. Requests pass the user's query (rather than full attachment contents) into retrieval.

Stream measurements

stream_request.tokens is the sum of actual backend output-token usage only when usage is available for every observed model call. Otherwise it is null; completed_output_tokens reports the known subtotal and usage_complete is false. content_chunks counts emitted content segments, not model tokens. model_to_first_content_ms is elapsed time including tools and model calls, not a measurement of reasoning alone. Each model call includes its model, provider, and reasoning-token usage when available. Prometheus records success, error, stalled and cancelled outcomes, including failures during final persistence.

Offline context regression evaluation

Run .venv/bin/python -m scripts.evaluate_context to evaluate representative finance, note and transaction-search fixtures. The report includes estimated input/output tokens, required-fact retention, budget compliance and local compaction time. CI tests enforce those facts and budget bounds. These are context-retention checks, not measurements of generated-answer accuracy or backend latency; live conversation quality and actual usage still need to be compared after deployment.

Focused Obsidian reads provide a model_context projection so the requested line range survives compaction while the full artifact remains in the interface. Note excerpts include bounded heading indexes with content line numbers for follow-up reads. Repeated structured compaction accumulates omission counts and source references without overwriting earlier omissions.

11 KiB Raw Blame History