181 lines
11 KiB
Markdown
181 lines
11 KiB
Markdown
# Conversation context management
|
|
|
|
LangGraph checkpoints retain original messages and artifacts. `context.py`
|
|
constructs a bounded model view with copies; it does not rewrite checkpoints.
|
|
|
|
The prompt budget includes system instructions, saved memories, selected tool
|
|
schemas (including admin tools), message content, tool arguments, and any durable
|
|
summary. Defaults are a 16,000-token prompt target, 4,096-token response reserve,
|
|
2,048-token safety reserve, and 32,768-token context window. Estimates use UTF-8
|
|
bytes divided by three, message overhead, and a fixed image allowance, rather
|
|
than a model-specific tokenizer. Compare estimates with actual model usage.
|
|
|
|
## Structured tool compaction
|
|
|
|
`compact_tool_result(message, limit)` prefers `ToolMessage.artifact` dictionaries
|
|
or lists, then parseable JSON content. Small payloads remain whole. Large
|
|
payloads retain metadata and compact recognized row collections (`transactions`,
|
|
`rows`, `results`, `documents`, `categories`, `accounts`, `payees`) and text fields
|
|
(`markdown`, `snippet`, `content`, `text`, `memo`, `description`). Output remains
|
|
valid JSON. Rows are sampled in source order without inventing aggregate totals.
|
|
|
|
Each shortened collection reports the returned, retained, and omitted row counts,
|
|
a notice to consult the original artifact/source tool, and omitted source
|
|
references. IDs, paths, dates and source references remain available even when
|
|
row details are omitted. Top-level totals, dates, counts, filters, query, source,
|
|
source references, and note frontmatter remain intact. Text fields use explicit
|
|
prefix excerpts with omitted character counts. Counts describe rows in the
|
|
artifact; a separate upstream `count` can describe a larger query result.
|
|
|
|
The artifact is authoritative for this projection. Tools must include relevant
|
|
filters, dates, totals and source IDs in their artifacts or call arguments;
|
|
compaction cannot reconstruct information absent from those inputs. Unknown
|
|
non-JSON output falls back to marked beginning/end excerpts. Structured metadata
|
|
has a minimum size: if required references and exchange structure exceed the
|
|
budget, trimming raises `ValueError` instead of cutting JSON or hiding omissions.
|
|
|
|
The current user request and tool arguments remain intact. All retained exchanges
|
|
include the calling assistant and every parallel result. Orphan results and
|
|
unfinished exchanges from cancellations are excluded from the model view.
|
|
Intermediate assistant prose can be shortened. Older turns are selected whole;
|
|
older message text is bounded and images become placeholders. If no durable
|
|
summary is supplied, bounded user/assistant excerpts provide limited continuity.
|
|
Historical excerpts use the assistant role, never the system role.
|
|
|
|
## Persistent extractive evidence
|
|
|
|
`summary.py` provides `SummaryStore(path)` using the existing `aiosqlite`
|
|
dependency. Pass a persistent SQLite file on a writable local volume; its parent
|
|
directory must exist. The store creates its own table. `UserMemory` was inspected:
|
|
it stores user-wide explicit facts and has no conversation key or cursor. It is
|
|
not reused for this state. ORM registration already includes
|
|
`blueprints.conversation.models`; this sidecar requires no ORM model or migration.
|
|
|
|
The store is keyed by `(user_id, conversation_id)` and provides:
|
|
|
|
```python
|
|
store = SummaryStore("/persistent-data/conversation-summaries.sqlite3")
|
|
state = await store.update(user_id, conversation_id, checkpoint_messages)
|
|
# Or reuse already persisted evidence:
|
|
state = await store.load(user_id, conversation_id)
|
|
summary_message = state.as_message() # AIMessage, or None for empty evidence
|
|
prompt = build_context(
|
|
checkpoint_messages, system_prompt, selected_tool_schemas,
|
|
summary_message=summary_message,
|
|
)
|
|
```
|
|
|
|
`update` requires a full, stable, append-only snapshot of checkpoint messages,
|
|
including the new current HumanMessage. It summarizes only messages preceding
|
|
that newest user turn; the current request is preserved in full context. The
|
|
last completed turn enters evidence when the next user request arrives. Only
|
|
user text and assistant text without tool calls enter evidence. Tool content,
|
|
artifacts, and assistant tool-call prose never become summary instructions.
|
|
|
|
The state contains a version, incremental message-index cursor, checkpoint-prefix
|
|
SHA-256 digest, omitted-entry count, and four evidence buckets: constraints,
|
|
decisions, open work, and recent context. Labels use simple keyword heuristics;
|
|
they do not assert that a task was resolved or a decision approved. Each excerpt
|
|
retains its original role, message ID/index and an excerpt marker. Maximums are
|
|
900 UTF-8 bytes per excerpt, six entries per bucket, and 1,800 estimated tokens
|
|
for serialized state. Old entries are evicted from the largest bucket to respect
|
|
the total bound. The assistant-role JSON envelope adds a small charged overhead
|
|
and explicitly labels the contents untrusted historical evidence. Historical
|
|
text is never promoted to privileged system instructions; role separation is
|
|
not a guarantee that a model will resist every prompt injection.
|
|
|
|
SQLite transactions serialize concurrent processes. Repeated and stale updates
|
|
cannot duplicate evidence or move the cursor backward. A changed summarized
|
|
prefix raises `ValueError`, leaving persisted state unchanged. Checkpoint resets,
|
|
branching, or forks need a new conversation key; do not silently reuse old state.
|
|
Snapshots shorter than the saved cursor are treated as stale; the store cannot
|
|
distinguish such a snapshot from a reset until enough history is available for
|
|
prefix validation. The caller must enforce append-only checkpoint lineage.
|
|
Prefix hashing runs in a thread so large artifact serialization does not block
|
|
the async event loop. It still reads the historical prefix to validate lineage;
|
|
incremental extraction does not make validation constant-time. Failed or
|
|
cancelled transactions roll back on connection close.
|
|
|
|
This is bounded extractive evidence, not a semantic rolling summary. It can lose
|
|
old decisions, miss constraints in long attachments, retain outdated open-work
|
|
phrases, and label text incorrectly. Keep the checkpoints as the complete source
|
|
of truth. No extra remote/model calls are made, and tests use only local SQLite.
|
|
|
|
## Request integration
|
|
|
|
Initialize a store with a durable path and call `update` after obtaining the full
|
|
checkpoint plus the new user message. Pass `state.as_message()` in request-local
|
|
agent config and forward it to `build_context(..., summary_message=...)` from the
|
|
prompt adapter. Do not append this synthetic message to checkpoint history or
|
|
share mutable summary state on a global agent. Scope keys to the authenticated
|
|
user and conversation. Empty evidence yields `None`. Summary tokens are charged
|
|
before history selection. Optional summary evidence is omitted when it would
|
|
prevent the mandatory current request from fitting.
|
|
|
|
Request handlers can await the local async update, or schedule it as a tracked
|
|
background task and use `load` for the current request (allowing one request of
|
|
lag). The caller must own task lifetime, failure logging, and shutdown draining for
|
|
background updates. Run updates on full checkpoint snapshots, not model-trimmed
|
|
history or `ConversationMessage` display text, which can be truncated. Attach a
|
|
current human request before updating; otherwise the last user turn is deliberately
|
|
left unsummarized. The app initializes the sidecar next to the checkpoint database (override with
|
|
`CONVERSATION_SUMMARY_DB_PATH`). Authorized conversation deletion also deletes
|
|
its summary. Include the sidecar in checkpoint backups. SQLite is suited to workers sharing a local
|
|
volume; replicas without shared storage need an equivalent transactional database
|
|
adapter, not independent sidecars. No change to `models.py` is required for the
|
|
implemented store.
|
|
|
|
## Configuration and observability
|
|
|
|
- `CONTEXT_WINDOW_TOKENS`: smallest supported context window across backends.
|
|
- `PROMPT_BUDGET_TOKENS`: prompt target including fixed instructions and tools.
|
|
- `RESPONSE_RESERVE_TOKENS`: maximum generated tokens and reserved answer space.
|
|
|
|
`context_budget` logs estimates, counts, dropped turns and transient excerpt
|
|
inclusion. `prompt_budget` logs fixed overhead and the complete prompt estimate.
|
|
Neither logs message content. Saved user memories have a separate estimated
|
|
1,000-token budget without deleting omitted memories. Obsidian reads accept
|
|
one-based `start_line` (excluding frontmatter) and `max_lines` for focused reads;
|
|
full artifacts remain available to the interface and checkpoint.
|
|
|
|
## Tool routing and saved memories
|
|
|
|
Each model call binds tool groups selected from the current query and relevant
|
|
follow-up context. A small `discover_tool_groups` tool can list available groups
|
|
or enable additional groups for the next call. Discovery grants are stored in
|
|
that turn's tool results, never global mutable state. Primary and fallback models
|
|
receive the same selected schemas, and prompt budgeting charges that same set.
|
|
Admin-only groups remain subject to authenticated invocation permissions.
|
|
Routing is lexical; unknown requests can use discovery to recover.
|
|
|
|
Saved memories rank query-word overlap ahead of a small recency bonus, retaining
|
|
a bounded injected block. Selection does not delete stored memories. Requests
|
|
pass the user's query (rather than full attachment contents) into retrieval.
|
|
|
|
## Stream measurements
|
|
|
|
`stream_request.tokens` is the sum of actual backend output-token usage only
|
|
when usage is available for every observed model call. Otherwise it is null;
|
|
`completed_output_tokens` reports the known subtotal and `usage_complete` is false.
|
|
`content_chunks` counts emitted content segments, not model tokens.
|
|
`model_to_first_content_ms` is elapsed time including tools and model calls, not
|
|
a measurement of reasoning alone. Each model call includes its model, provider,
|
|
and reasoning-token usage when available. Prometheus records success, error,
|
|
stalled and cancelled outcomes, including failures during final persistence.
|
|
|
|
## Offline context regression evaluation
|
|
|
|
Run `.venv/bin/python -m scripts.evaluate_context` to evaluate representative
|
|
finance, note and transaction-search fixtures. The report includes estimated
|
|
input/output tokens, required-fact retention, budget compliance and local
|
|
compaction time. CI tests enforce those facts and budget bounds. These are
|
|
context-retention checks, not measurements of generated-answer accuracy or
|
|
backend latency; live conversation quality and actual usage still need to be
|
|
compared after deployment.
|
|
|
|
Focused Obsidian reads provide a `model_context` projection so the requested
|
|
line range survives compaction while the full artifact remains in the interface.
|
|
Note excerpts include bounded heading indexes with content line numbers for
|
|
follow-up reads. Repeated structured compaction accumulates omission counts and
|
|
source references without overwriting earlier omissions.
|