Commit Graph

41 Commits

Author SHA1 Message Date
Ryan Chen 5809ad3adb Add durable chat runs, shared agent runtime, reliable jobs, history tools and citations
CI / backend (pull_request) Successful in 42s
CI / frontend (pull_request) Failing after 7s
CI / backend (push) Successful in 41s
CI / frontend (push) Failing after 7s
Shared runtime
- One agent entry point (runtime.invoke_agent_for_user / prepare_agent_turn) for
  web, email, iMessage, WhatsApp, scheduled and cron, so identity, permissions,
  memories, summaries and metrics match across channels.
- Serialize turns per checkpoint thread; /query validates input and returns 409
  while a durable run owns the conversation.

Reconnectable chat (durable runs)
- POST /runs creates an idempotent, owner-scoped run; GET .../events replays an
  ordered SSE log by cursor, so generation survives refresh and device switches.
  Stop cancels; disconnect only detaches. /stream-query is a compatibility adapter.
- Coalesce streamed deltas (150 ms) and wake same-process subscribers instead of
  polling; keep buffered text when a run is stopped.
- Drain runs on shutdown; serve via Hypercorn with a shutdown timeout covering
  the drain (and no longer in debug mode). CHAT_RUN_* settings are forwarded.

Scheduled messages and cron
- Atomic delivery claims, attempt history and explicit retries; ambiguous sends
  become "uncertain" and are never resent automatically.
- Cron triggers need a run_id/Idempotency-Key; legacy jobs need an explicit owner.
- Cancelling never rewrites a started delivery; series stop via tombstones.
  Legacy sent rows are not forked into new series. Malformed bodies return 400.

History, integrations and retrieval
- Message-content search, pinned conversations, JSON/Markdown export and a
  memory editor.
- Integration status panel and YNAB data-freshness provenance.
- Date-filtered document/note search with stable citations and a source viewer.

Migration 20 is generated by aerich.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 23:11:04 -04:00
Ryan Chen 4645920c64 Route tools and retain structured conversation context with durable evidence 2026-10-12 20:19:12 -04:00
Ryan Chen 0fa30edb82 Merge origin/main into feat/ynab-mirror
Resolve config/db.py (artifacts + ynab models) and regenerate the YNAB
migration with aerich as 18 on top of main's 17_add_artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:07:26 -04:00
Ryan Chen 10f3f1bcab Merge origin/main into feat/ynab-mirror
Resolve the Makefile .PHONY conflict (desktop + ynab targets) and
regenerate the YNAB migration with aerich as 17 on top of main's
16_add_oidc_states, replacing the colliding 16_add_ynab_mirror.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:06:47 -04:00
Ryan Chen 0cf6229441 Add Artifact model and owner-scoped artifacts API
GET /api/artifacts?conversation_id= lists ArtifactRefs without data;
GET /api/artifacts/<id> returns the full artifact. Both 404 for other
users. create_artifact_from_output persists ToolArtifact dicts and drops
ones over 256 KB serialised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:53:22 -04:00
Ryan Chen bc3e3d2797 Accept access tokens on protected routes; persist OIDC state
- Add transitional @jwt_access_required (accepts access tokens, and for now
  also refresh tokens) and use it on every protected route and inside
  admin_required. /api/user/refresh stays refresh-only.
- Move pending OIDC state/PKCE verifiers from an in-process dict to an
  oidc_states table (10 min TTL, single-use) so logins survive a blue/green
  flip. Drop the unused redirect_after_login.
- Unit tests for the decorator and the state store.

Deploy this commit on its own, before the access-token frontend: the old
backend answers access tokens with 422, so the new frontend only works once
this backend is live on both colors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:44:34 -04:00
Ryan Chen 8f02318542 Mirror YNAB budget into Postgres with delta sync and change log
YNAB has no event feed, so keep a local copy in ynab_* tables synced via
server_knowledge delta requests: full download on first run, deltas every
YNAB_SYNC_INTERVAL seconds and on demand before each tool call, full
resync daily. Each sync diffs against the mirror and appends real edits
(transactions, assignments, renames) to ynab_changes, the edit history
YNAB doesn't keep.

Agent YNAB tools now answer from the mirror: budget summary (spending
computed from transactions, since YNAB's month activity nets in credit
card payment categories), account balances, transaction search with
splits expanded, spending reports by category/payee/account/month,
category budget, and recent changes. Adds an admin-only read-only
ynab_sql tool guarded by a sqlglot allowlist (scope-resolved tables, no
dangerous functions), a READ ONLY transaction, a timeout, and a row cap.

Replaces utils/ynab_service.py and drops the unused ynab SDK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 17:32:38 -04:00
Ryan Chen 0b8fe7b2df Add admin-only BookStack MCP connector
Load an allowlisted subset of bookstack-mcp tools (search/read plus
create/update books, chapters, pages and comments; no deletes) over stdio
and add them to the admin agent alongside Homebox. Extract the shared
allowlist + is_admin wrapping into utils/mcp_tools.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 16:54:01 -04:00
Ryan Chen 9858e06b73 Add admin-only Homebox MCP connector
Load an allowlisted subset of homebox-mcp tools (read + safe writes, no
deletes) over stdio and compile them into a separate admin agent, so
non-admins never see them. Each tool also re-checks is_admin per call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:01:19 -04:00
Ryan Chen d23273590f Add Beli restaurant recommendation integration
Multi-tenant system to fetch and store restaurant lists from Beli accounts.
Each user can connect up to 3 Beli profiles; a nightly Cronicle job (1–5 AM,
randomised) refreshes the 3 stalest profiles using the reverse-engineered Beli
API (mirrored at git.torrtle.co/ryan/beli-api). Rankings (been) and bookmarks
(want-to-try) are stored in Postgres and queryable via a new LangChain agent
tool. Beli passwords and refresh tokens are stored Fernet-encrypted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-17 19:36:16 -04:00
Ryan Chen 790a7059ce Per-message Cronicle events with recurrence-aware timing
Instead of an every-minute poller, each scheduled message gets its own
Cronicle event created at schedule time:
- One-time: year/month/day/hour/minute one-shot
- Daily/weekly/monthly: recurring timing matching the recurrence setting

Retry (PUT with new scheduled_at on failed messages) also creates a new
Cronicle event and resets status to pending. Recurring messages reuse
their existing Cronicle event across occurrences.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-17 08:32:22 -04:00
Ryan Chen 6e72d5f95a Auto-register scheduled messages event in Cronicle on startup
On startup raggr calls the Cronicle API (CRONICLE_URL + CRONICLE_API_KEY)
and creates a 'Scheduled Messages Trigger' shell event if it doesn't exist,
replacing the need to configure it manually in the Cronicle UI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-17 08:09:27 -04:00
Ryan Chen 35b0e74a41 Migrate scheduled messages from internal poll loop to Cronicle trigger
Replaces the asyncio background loop (15s polling) with a POST
/api/scheduled-messages/trigger endpoint authenticated by X-Cron-Secret,
matching the existing cron jobs blueprint pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-17 07:55:31 -04:00
Ryan Chen 810d32dcc3 Fix: store agent on app object, access via current_app
Global module variable was unreliable across import boundaries. Storing
the compiled agent on app.main_agent and reading it via current_app is
the idiomatic Quart pattern and guaranteed to reflect the initialized value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-16 22:55:39 -04:00
Ryan Chen f1ab1accaf Add LangGraph SQLite checkpointing and cron_jobs migration
Replaces per-request history reconstruction with a persistent AsyncSqliteSaver
checkpointer keyed by conversation ID, so agent state (tool calls, intermediate
steps) carries forward between turns without manual message list building.
Also includes the missing migration for the CronJob model added in bb4619a.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-16 22:34:04 -04:00
Ryan Chen bb4619aefe Add Cronicle-driven cron jobs with local migration workflow
Adds a new cron blueprint with UUID-keyed jobs that Cronicle triggers via
HTTP POST with a shared secret header. Also adds scripts/migrate-new.sh so
migrations can be generated locally against a Docker postgres when no prod
container is running.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-16 19:10:15 -04:00
Ryan Chen 0478ee7568 Add Prometheus metrics endpoint for chat response observability
Tracks end-to-end response time and request counts (by stream mode and
result) via prometheus_client. Exposes GET /metrics behind a bearer
token (METRICS_TOKEN env var) so Prometheus can scrape it without
exposing raw data publicly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-09-15 22:57:12 -04:00
Ryan Chen d8d01a5fff Zero-downtime blue-green deploys behind Caddy health checks
Split the raggr service into raggr-blue (8080) and raggr-green (8081)
behind compose profiles, add a /api/health endpoint and container
healthcheck, and add scripts/deploy.sh to build the inactive color,
wait for it to become healthy, then gracefully stop the old one.
Makefile targets resolve the active color automatically.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-09-10 01:41:37 -04:00
Ryan Chen f5203e0466 Add scheduled messages and strip markdown from iMessage responses
Strip markdown formatting (bold, italic, headers, code, links, lists) from
LLM responses before sending via iMessage. Add scheduled messages feature
with CRUD API, background scheduler loop, and admin frontend panel.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-03 23:25:10 -04:00
Ryan Chen 20576cabf3 Add SendBlue iMessage integration with admin-only access
- New imessage blueprint: webhook receives inbound iMessages, runs through
  LangChain agent, replies via SendBlue REST API
- Admin-only: only users with lldap_admin group can use iMessage channel
- Admin endpoints to link/unlink imessage_number on user accounts
- Add imessage_number field to User model (needs aerich migration)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-03 19:22:15 -04:00
Ryan Chen 869de1c250 Add incremental Obsidian-to-pgvector sync with background watcher
Replace full delete-and-reindex with mtime-based incremental sync that
only re-indexes changed/new files and removes deleted ones. A background
polling task keeps the vector store up-to-date automatically when
OBSIDIAN_CONTINUOUS_SYNC=true.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-31 07:05:48 -04:00
Ryan Chen 47238f8567 Fix Obsidian sync race condition and block credentials.json from being served
Run ob login and sync-setup in foreground before backgrounding sync to
prevent "Another sync instance is already running" error. Restrict the
catch-all route to only serve whitelisted static file extensions to
prevent sensitive files like credentials.json from being exposed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-30 23:59:29 -04:00
Ryan Chen 438399646f Replace ChromaDB with pgvector for vector storage
Consolidate onto PostgreSQL by using pgvector instead of a separate
ChromaDB instance. This removes a Docker volume, a large dependency,
and simplifies the stack without meaningful performance impact at
our document scale.

- Swap langchain-chroma for langchain-postgres (PGVector)
- Use pgvector/pgvector:pg16 Docker image with init script
- Lazy-initialize vector store to avoid eager DB connections
- Add SQL helpers for stats/delete/list (replacing _collection access)
- Remove legacy main.py, chunker, petmd scraper, and /api/query endpoint

Re-index required after deploy (POST /api/rag/index + /index-obsidian).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-24 08:43:52 -04:00
Ryan Chen 500c44feb1 Simplify conversation naming to truncate first message
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-09 22:02:10 -04:00
Ryan Chen 07c272c96a Extend JWT token expiration times
Access tokens now last 1 hour (up from default 15 min) and refresh
tokens last 30 days, reducing frequent re-authentication.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-05 10:10:47 -04:00
Ryan Chen 18ef611134 Merge origin/main: resolve conflicts keeping both email/Mealie and WhatsApp/Mailgun/Obsidian work
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-04 08:19:50 -04:00
Ryan Chen 0415610d64 Add image upload and vision analysis to Ask Simba chat
Users can now attach images in the web chat for Simba to analyze using
Ollama's gemma3 vision model. Images are stored in Garage (S3-compatible)
and displayed in chat history.

Also fixes aerich migration config by extracting TORTOISE_CONFIG into a
standalone config/db.py module, removing the stale aerich_config.py, and
adding missing MODELS_STATE to migration 3.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-04 08:03:19 -04:00
ryan 6a7b1369ad Add email channel via Mailgun for Ask Simba
Users can now receive a unique email address (ask+<token>@domain) and
interact with Simba by sending emails. Inbound emails hit a Mailgun
webhook, are authenticated via HMAC token lookup, processed through the
LangChain agent, and replied to via the Mailgun API.

- Extract shared SIMBA_SYSTEM_PROMPT to blueprints/conversation/prompts.py
- Add email_enabled and email_hmac_token fields to User model
- Create blueprints/email with webhook, signature validation, rate limiting
- Add admin endpoints to enable/disable email per user
- Update AdminPanel with Email column, toggle, and copy-address button
- Add Mailgun env vars to .env.example
- Include database migration for new fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-13 16:21:18 -04:00
ryan 86cc269b3a yeet 2026-03-03 08:23:31 -05:00
Ryan Chen 43dd05f9d5 chore(01-01): add FERNET_KEY config and email tables migration
- Add FERNET_KEY to .env.example with generation instructions
- Register email.models in aerich_config.py and app.py
- Register email blueprint in app.py
- Create database migration for email_accounts, email_sync_status, emails
- Migration includes proper foreign keys and indexes
2026-02-08 09:15:23 -05:00
Ryan Chen 6ae36b51a0 ynab update 2026-01-31 22:47:43 -05:00
Ryan Chen ad39904dda reorganization 2026-01-31 17:13:27 -05:00
Ryan Chen 70799ffb7d refactor 2025-11-10 15:51:13 -05:00
Ryan Chen 29f8894e4a Add configurable database path via environment variables
- Add DATABASE_PATH environment variable support in app.py and aerich_config.py
- DATABASE_PATH: For simple relative/absolute paths (default: database/raggr.db)
- DATABASE_URL: For full connection strings (overrides DATABASE_PATH if set)
- Create .env.example with all configuration options documented
- Maintains backward compatibility with default database location

Usage:
  # Use default path
  python app.py

  # Use custom path for development
  DATABASE_PATH=dev.db python app.py

  # Use full connection string
  DATABASE_URL=sqlite://custom/path.db python app.py

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 12:01:16 -04:00
Ryan Chen e8264e80ce Changing DB thing 2025-10-26 09:36:33 -04:00
Ryan Chen f16e13fccc big uplift 2025-10-26 09:25:17 -04:00
Ryan Chen 29ac724d50 enabling login btw users 2025-10-25 09:30:54 -04:00
Ryan Chen 45a5e92aee Added conversation history (#4)
Reviewed-on: #4
Co-authored-by: Ryan Chen <ryan@torrtle.co>
Co-committed-by: Ryan Chen <ryan@torrtle.co>
2025-10-23 22:29:12 -04:00
Ryan Chen 0bb3e3172b adding image processing pipeline immich -> paperless 2025-10-04 08:54:10 -04:00
Ryan Chen a640ae5fed Docker stuff 2025-10-02 20:21:48 -04:00
Ryan Chen 99c98b7e42 yeet 2025-10-02 19:21:24 -04:00