`reindex` deleted every chunk before contacting Paperless, so an unreachable
or empty Paperless left the store empty (which happened when Paperless was
crash-looping during its v3 upgrade). It now fetches first, aborts without
deleting if that fails or returns nothing, and indexes the fetched documents
instead of fetching twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`make reindex` and `make reindex-obsidian` built a `python -c` string with
backslash continuations, which kept the tab indentation inside the code, so
Python failed with IndentationError. The targets now call the existing
`scripts.manage_vectorstore` CLI. Its `reindex` wiped everything but only
re-indexed Paperless, dropping Obsidian notes; it now indexes both, and a new
`reindex-obsidian` command re-embeds only the vault.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The container ran Obsidian Sync in pull-only mode, so notes, tasks and journal
entries the agent wrote stayed in its local copy. OBSIDIAN_SYNC_MODE now
defaults to bidirectional (merge on conflict); pull-only and mirror-remote
remain available.
Blue and green share the vault volume, and two sync clients writing one vault
and its sync state would race during a deploy. Setup and sync now run under a
flock on a file in the vault, so a new container starts syncing only after the
old container's sync process exits. Clearing Obsidian's stale sync lock is safe
inside that lock. Setup runs in the background, so startup no longer waits for
Obsidian login.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every graph step stores a full snapshot of the conversation, nothing pruned
old ones, and deleting a conversation left its thread behind (about 1.6 MB per
checkpoint at 200 turns). After each turn, only the newest
CHECKPOINT_KEEP_PER_THREAD (default 20) checkpoints of that thread are kept,
plus their pending writes; state reads only use the newest, and pruning a
200-checkpoint thread takes about 2 ms. Deleting a conversation also deletes
its thread, without failing the request if the checkpoint store errors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The periodic sync walked the vault, read files and ran synchronous database
deletes on the event loop, freezing active chat streams while it ran. Those
blocking steps now run in threads. Deletes use JSONB containment, which the
cmetadata GIN index (jsonb_path_ops) supports, and are batched into one
transaction per pass. Deleted files leave the index cache only after their
chunks are deleted, so a failed delete is retried on the next sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compaction replaces a tool's text with its artifact, and the search artifact
only carries UI snippets, so the full chunk the retriever found never reached
the model. Search tools now attach the full text as model_context, which
compaction prefers and shortens under budget pressure while keeping source
ids; the persisted artifact and UI are unchanged. The per-chunk header drops
the raw metadata dict (about a fifth of search output tokens) for title, date
and source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local model is decode-bound, so output length drives latency. The system
prompt asked for "thorough, detailed answers" and to "think things through",
which lengthened both answers and reasoning. It now leads with the answer and
expands only on request. Per-integration tool lists that duplicated the bound
tool docstrings collapse to one paragraph of routing and duplicate-avoidance
rules. The prompt drops from about 1,446 to 897 tokens and stays static, so
the llama-server prompt cache still hits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/conversation/image/<key> presigned any S3 key for any logged-in
user, and /upload-image wrote into whatever conversation_id the form named.
Both now require the conversation in the key (or form) to belong to the
caller and return 404 otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Shared runtime
- One agent entry point (runtime.invoke_agent_for_user / prepare_agent_turn) for
web, email, iMessage, WhatsApp, scheduled and cron, so identity, permissions,
memories, summaries and metrics match across channels.
- Serialize turns per checkpoint thread; /query validates input and returns 409
while a durable run owns the conversation.
Reconnectable chat (durable runs)
- POST /runs creates an idempotent, owner-scoped run; GET .../events replays an
ordered SSE log by cursor, so generation survives refresh and device switches.
Stop cancels; disconnect only detaches. /stream-query is a compatibility adapter.
- Coalesce streamed deltas (150 ms) and wake same-process subscribers instead of
polling; keep buffered text when a run is stopped.
- Drain runs on shutdown; serve via Hypercorn with a shutdown timeout covering
the drain (and no longer in debug mode). CHAT_RUN_* settings are forwarded.
Scheduled messages and cron
- Atomic delivery claims, attempt history and explicit retries; ambiguous sends
become "uncertain" and are never resent automatically.
- Cron triggers need a run_id/Idempotency-Key; legacy jobs need an explicit owner.
- Cancelling never rewrites a started delivery; series stop via tombstones.
Legacy sent rows are not forked into new series. Malformed bodies return 400.
History, integrations and retrieval
- Message-content search, pinned conversations, JSON/Markdown export and a
memory editor.
- Integration status panel and YNAB data-freshness provenance.
- Date-filtered document/note search with stable citations and a source viewer.
Migration 20 is generated by aerich.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Fail startup on unsafe or malformed production settings (scripts/validate_config.py),
run it before migrations, and stop on a failed `aerich upgrade`.
- Forward the context-budget, timeout and metrics settings through Compose; accept
the historical TAVILIY_API_KEY spelling.
- Build the image from uv.lock and yarn.lock in a multi-stage Dockerfile.
- Add Gitea Actions CI: Ruff, unit tests, context evaluation, frontend tests,
typecheck and build.
- Update README and deployment/development docs to match the current stack.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ledger supersedes the "Earlier conversation excerpts" message, as on
main; sending both duplicated dropped-turn content. Removal is consistent
whenever the ledger is included, so the cached prefix stays stable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The summary ledger is regenerated every turn. At the head of history it
invalidated the cached retained turns on every request once trimming
started. Insert it immediately before the current turn's HumanMessage so
system, tools and retained history stay byte-identical; excerpts of dropped
turns stay at the head since they only change when the cut moves.
Reserve a fixed ledger ceiling (SUMMARY_RESERVE_TOKENS) rather than its
current size, so a shrinking ledger cannot pull the cut back and change the
retained prefix; fall back to the actual size for tight budgets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama-server only reuses a byte-identical prompt prefix. Several things
changed the prefix every turn, forcing most of the prompt to be re-prefilled:
- Memories: rank still chooses which memories fit the budget, but the chosen
ones are emitted in stored order, so the block is identical across queries.
- History trimming: the conversation-evidence summary is only included when
turns were actually dropped (it duplicated retained turns otherwise). On
overflow, old turns are dropped at stable cut points down to ~65% of the
budget, so the retained prefix holds for several turns instead of shifting
by one turn every request.
- Tools: TOOL_ROUTING_MODE=static (default) binds the full role-appropriate
tool set in registration order with no discovery tool; admin gating is
unchanged. TOOL_ROUTING_MODE=dynamic keeps the keyword router.
- Observability: per-call model_calls in the stream_request log now include
cached_prompt_tokens when the provider reports cache reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
process_image takes ~560ms on a 12MP photo and image_content_block
decodes HEIC/JPEG with PIL; both ran synchronously inside async
handlers, freezing every concurrent chat stream. Run them via
asyncio.to_thread in the upload handlers, the chat stream, and iMessage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
index_documents sent one embedding request and one insert per chunk.
Send chunks in batches of 32 and, only when a batch fails, retry it one
chunk at a time so the rejected chunk is still named in the logs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 30s time-to-first-token guard measured from the turn's first model call
to the first answer token, so thinking, tool execution and the second call's
prefill all counted against it. Long reasoning on a long conversation was
cancelled mid-generation (llama-server: stop: cancel task at n_gen=1287).
- stream_query: the timeout now fires only when an in-flight model call goes
silent for TTFT_TIMEOUT_SECONDS. Any chunk (thinking or empty) resets the
clock, each model call starts a fresh one, and time spent in tools between
calls doesn't count.
- Trim replayed history (blueprints/conversation/context.py): tool results
and user messages from earlier turns are capped at 4000 chars (attachments
included) and their images replaced by a placeholder. The current turn is
untouched. The window no longer starts with orphaned ToolMessages.
- Log per-call prefill time, duration and prompt/output tokens
(stream_usage=True) plus max_prompt_tokens in the stream_request line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- extract_text (PyMuPDF for PDFs) now runs via asyncio.to_thread in both
/upload and stream-query, so a large PDF doesn't stall other streams.
- /upload returns 422 when extraction yields no text: a scanned PDF
without a text layer, or an empty text file.
- Mock server: *scan*.pdf uploads return that 422.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/conversation/upload takes an image (via process_image) or a
PDF/txt/md/csv file, checks the conversation belongs to the caller, and
stores files at conversations/<id>/files/<uuid>.<ext> with the original
name in S3 metadata. Anything else gets a 415.
stream-query accepts file_key (must sit under this conversation's
files/ prefix), extracts its text capped at 12k chars, appends it to the
agent's user turn as <attached file> and records 📎 name on the saved
message.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Convert ynab_budget_summary, ynab_search_transactions, ynab_spending_report
and ynab_category_budget to content_and_artifact (ynab.summary,
ynab.transactions, ynab.spending_report, ynab.category_budget). The text
the LLM sees is unchanged. Not-synced, bad-input, empty and error paths
return no artifact; Decimals become floats and dates ISO strings.
ynab_search_transactions now fetches up to 500 rows in one query (with an
id tie-break so ordering is stable) and lists the first `limit` in the
text; the artifact carries all of them. ynab_account_balances,
ynab_recent_changes and ynab_sql stay text-only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Convert mealie_get_recipe, mealie_meal_plans, mealie_todays_meals,
mealie_shopping_lists, obsidian_read_note, journal_get_today,
obsidian_search_notes and simba_search to content_and_artifact. The text
the LLM sees is unchanged; the artifact is a JSON-safe ToolArtifact dict
per the Cowork UI contract (mealie.recipe, mealie.meal_plan,
mealie.shopping_lists, obsidian.note, search.documents). Error, empty and
not-configured paths return no artifact.
simba_search's artifact changes from a list of Documents (unused) to a
search.documents dict.
New optional env vars: MEALIE_GROUP_SLUG (default "home") for Mealie v2
recipe links, OBSIDIAN_VAULT_NAME for obsidian://open links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve config/db.py (artifacts + ynab models) and regenerate the YNAB
migration with aerich as 18 on top of main's 17_add_artifacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the Makefile .PHONY conflict (desktop + ynab targets) and
regenerate the YNAB migration with aerich as 17 on top of main's
16_add_oidc_states, replacing the colliding 16_add_ynab_mirror.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Failure backoff is re-checked under the sync lock, so parallel tool
calls share one failed YNAB request instead of retrying in turn.
- Budget id: YNAB_BUDGET_ID aliases ("last-used", "default") count as
unset; state is always keyed by the real budget id, a mismatched
response errors instead of silently skipping every sync, and more
than one stored budget is an error rather than an arbitrary pick.
- Change log renames only id fields that resolve to names (account,
payee, category, group); subtransaction transaction_id stays as is.
- ynab_recent_changes: correct singular for "categories", multi-word
types, and the "month" type.
- The SQL wrap test now parses the wrapped query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Full resyncs mark mirror rows missing from the download as deleted;
YNAB omits deleted entities from non-delta responses, so deletions
made just before a full sync never reached the mirror.
- YNAB tools return errors as text instead of raising, since ToolNode
re-raises and would fail the whole chat turn. ynab_sql catches all
sqlglot errors (e.g. an unbalanced quote) and runs the SQL regenerated
from the validated tree, which also drops comments that broke the
LIMIT wrapper. Results keep duplicate column names.
- Sync fetches from YNAB before opening the DB transaction, discards the
payload if the other container synced meanwhile, re-checks freshness
under the lock so parallel tool calls share one sync, and backs off
for 60s after a failure.
- The mirror is pinned to one budget; a changed YNAB_BUDGET_ID errors
instead of mixing budgets, and an unset id stays on the budget the
first sync resolved from "last-used".
- Change log: no entries for empty month rollover rows, and id fields
are logged as names (category, payee, account). ynab_recent_changes
accepts the names it prints ("assignment") and plurals.
- Outputs sized to the 4000-char tool result cap; ynab_sql reuses the
shared admin message.
- YNAB API client moves back to utils/ynab_service.py, and the frontend
tool status labels list the new tools.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
asyncpg raises OperationalError (not ValueError) on malformed UUIDs, so
bad artifact ids now 404 and a bad conversation_id 400s before any query.
get_conversation keys artifacts by str(message_id) so matching doesn't
depend on the driver's UUID types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stream_query emits the new tool_start/tool_end payloads, maps
on_tool_error to tool_end status=error, and saves an artifact linked to
the tool message. GET /api/conversation/<id> adds tool, label and
artifact to tool messages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pure functions build tool_start/tool_end payloads with run_id, friendly
label, truncated input (config dropped), status, duration and artifact ref.
Labels cover every agent tool, both old and #55 YNAB names, and the
homebox_/bookstack_ MCP prefixes, falling back to "Using <tool>…".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add transitional @jwt_access_required (accepts access tokens, and for now
also refresh tokens) and use it on every protected route and inside
admin_required. /api/user/refresh stays refresh-only.
- Move pending OIDC state/PKCE verifiers from an in-process dict to an
oidc_states table (10 min TTL, single-use) so logins survive a blue/green
flip. Drop the unused redirect_after_login.
- Unit tests for the decorator and the state store.
Deploy this commit on its own, before the access-token frontend: the old
backend answers access tokens with 422, so the new frontend only works once
this backend is live on both colors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YNAB has no event feed, so keep a local copy in ynab_* tables synced via
server_knowledge delta requests: full download on first run, deltas every
YNAB_SYNC_INTERVAL seconds and on demand before each tool call, full
resync daily. Each sync diffs against the mirror and appends real edits
(transactions, assignments, renames) to ynab_changes, the edit history
YNAB doesn't keep.
Agent YNAB tools now answer from the mirror: budget summary (spending
computed from transactions, since YNAB's month activity nets in credit
card payment categories), account balances, transaction search with
splits expanded, spending reports by category/payee/account/month,
category budget, and recent changes. Adds an admin-only read-only
ynab_sql tool guarded by a sqlglot allowlist (scope-resolved tables, no
dangerous functions), a READ ONLY transaction, a timeout, and a row cap.
Replaces utils/ynab_service.py and drops the unused ynab SDK.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allowlist homebox-mcp's general item editor so the agent can rename items
and edit description, notes, quantity, purchase info, flags and custom
fields. Only passed args change; the rest of the item is preserved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load an allowlisted subset of bookstack-mcp tools (search/read plus
create/update books, chapters, pages and comments; no deletes) over stdio
and add them to the admin agent alongside Homebox. Extract the shared
allowlist + is_admin wrapping into utils/mcp_tools.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load an allowlisted subset of homebox-mcp tools (read + safe writes, no
deletes) over stdio and compile them into a separate admin agent, so
non-admins never see them. Each tool also re-checks is_admin per call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tracks end-to-end response time and request counts (by stream mode and
result) via prometheus_client. Exposes GET /metrics behind a bearer
token (METRICS_TOKEN env var) so Prometheus can scrape it without
exposing raw data publicly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Upgrades pypetkitapi to 1.28.0, which adds the Eversweet Ultra (W7H
fountain with AI camera). The poller now extracts per-pet drinking
events (drink_over/pet_detect records) into a new drinking_events
table, plus Ultra-specific alerts for waste tank full and filter
expiry. Uncertain state fields (cwt/wt/pump) are logged for threshold
tuning against real data.
Exposes the data via a new /api/pets/drinking-events endpoint, a
Fountain Activity dashboard section, and a simbakit_drinking_activity
LangChain tool so the agent can answer hydration questions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update from journal/YYYY/YYYY-MM-DD.md to
50 - Journal/YYYY/MM/YYYY-MM-DD.md to match the actual Obsidian vault
folder layout.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate onto PostgreSQL by using pgvector instead of a separate
ChromaDB instance. This removes a Docker volume, a large dependency,
and simplifies the stack without meaningful performance impact at
our document scale.
- Swap langchain-chroma for langchain-postgres (PGVector)
- Use pgvector/pgvector:pg16 Docker image with init script
- Lazy-initialize vector store to avoid eager DB connections
- Add SQL helpers for stats/delete/list (replacing _collection access)
- Remove legacy main.py, chunker, petmd scraper, and /api/query endpoint
Re-index required after deploy (POST /api/rag/index + /index-obsidian).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ryan Chen