ryan
9249f92db8
- Full resyncs mark mirror rows missing from the download as deleted;
YNAB omits deleted entities from non-delta responses, so deletions
made just before a full sync never reached the mirror.
- YNAB tools return errors as text instead of raising, since ToolNode
re-raises and would fail the whole chat turn. ynab_sql catches all
sqlglot errors (e.g. an unbalanced quote) and runs the SQL regenerated
from the validated tree, which also drops comments that broke the
LIMIT wrapper. Results keep duplicate column names.
- Sync fetches from YNAB before opening the DB transaction, discards the
payload if the other container synced meanwhile, re-checks freshness
under the lock so parallel tool calls share one sync, and backs off
for 60s after a failure.
- The mirror is pinned to one budget; a changed YNAB_BUDGET_ID errors
instead of mixing budgets, and an unset id stays on the budget the
first sync resolved from "last-used".
- Change log: no entries for empty month rollover rows, and id fields
are logged as names (category, payee, account). ynab_recent_changes
accepts the names it prints ("assignment") and plurals.
- Outputs sized to the 4000-char tool result cap; ynab_sql reuses the
shared admin message.
- YNAB API client moves back to utils/ynab_service.py, and the frontend
tool status labels list the new tools.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
167 lines
8.5 KiB
Markdown
167 lines
8.5 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
SimbaRAG is a RAG (Retrieval-Augmented Generation) conversational AI system for querying information about Simba (a cat) and the household. It ingests documents from Paperless-NGX, Obsidian, and Immich, stores embeddings in PostgreSQL via pgvector (Chroma client), uses LLMs (Ollama or OpenAI) to answer questions with a tool-using LangChain agent, and exposes chat over web, iMessage, and WhatsApp.
|
|
|
|
## Commands
|
|
|
|
### Development
|
|
|
|
```bash
|
|
# Start environment (raggr runs as raggr-blue/raggr-green behind compose profiles)
|
|
docker compose --profile blue up --build
|
|
|
|
# View logs (Makefile resolves the active color)
|
|
make logs
|
|
|
|
# Reindex vector store (all sources / Obsidian only)
|
|
make reindex
|
|
make reindex-obsidian
|
|
```
|
|
|
|
### Database Migrations (Aerich/Tortoise ORM)
|
|
|
|
```bash
|
|
# Generate migration (must run in Docker with DB access)
|
|
make migrate-new
|
|
|
|
# Apply migrations (auto-runs on startup, manual if needed)
|
|
make migrate
|
|
|
|
# View migration history
|
|
make migrate-history
|
|
```
|
|
|
|
Migrations must be backward-compatible: blue-green deploys run `aerich upgrade` in the new container while the old code still serves.
|
|
|
|
### Frontend
|
|
|
|
```bash
|
|
cd raggr-frontend
|
|
yarn install
|
|
yarn build # Production build
|
|
yarn dev # Dev server (rarely needed, backend serves frontend)
|
|
```
|
|
|
|
### Production
|
|
|
|
Zero-downtime blue-green deploy: `raggr-blue` (port 8080) and `raggr-green` (port 8081) alternate behind Caddy health checks. See `docs/deployment.md`.
|
|
|
|
```bash
|
|
make redeploy # git pull + scripts/deploy.sh (build new color, wait healthy, stop old)
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Docker Compose │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ raggr-blue/green (8080/8081) │ postgres (port 5432) │
|
|
│ ├── Quart backend │ PostgreSQL 16 + pgvector│
|
|
│ ├── React frontend (served) │ │
|
|
│ └── Twilio webhooks (iMessage/ │ │
|
|
│ WhatsApp) │ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Backend** (root directory):
|
|
- `app.py` - Quart application entry, serves API and static frontend
|
|
- `main.py` - RAG logic, document indexing, LLM interaction, LangChain agent
|
|
- `llm.py` - LLM client with Ollama primary, OpenAI fallback
|
|
- `blueprints/` - API routes organized as Quart blueprints
|
|
- `users/` - OIDC auth, JWT tokens, RBAC with LDAP groups
|
|
- `conversation/` - Chat conversations, message history, streamed answers
|
|
- `rag/` - Document indexing endpoints (admin-only); logic in `rag/logic.py`
|
|
- `email/` - Email integration (Mailgun webhooks, Fernet encryption, IMAP, parsing)
|
|
- `imessage/` - Twilio webhook receiver for iMessage (HMAC-validated, number allowlist)
|
|
- `whatsapp/` - Twilio webhook receiver for WhatsApp (TWIML replies, SendBlue send support)
|
|
- `scheduled_messages/` - Scheduled message delivery with background scheduler
|
|
- `ynab/` - Postgres mirror of the YNAB budget (`ynab_*` tables), delta-synced via `server_knowledge` (background loop + on-demand before each tool call, full resync daily that also marks rows missing from the download as deleted); `ynab_changes` is the edit history YNAB itself doesn't keep. Agent tools in `tools.py`; admin-only read-only `ynab_sql` in `sql_tool.py`. Manual sync: `make ynab-sync` / `make ynab-resync`
|
|
- `config/` - Configuration modules
|
|
- `db.py` - Database setup
|
|
- `oidc_config.py` - OIDC authentication configuration
|
|
- `utils/` - Reusable utilities and household integrations
|
|
- `chunker.py` - Document chunking for embeddings
|
|
- `cleaner.py` - PDF cleaning and summarization
|
|
- `image_process.py` - Image description with LLM
|
|
- `request.py` - Paperless-NGX API client
|
|
- `obsidian_service.py`, `mealie_service.py`, `ynab_service.py`, `s3_client.py`, `simbakit_service.py` - integrations with Obsidian notes, Mealie recipes, YNAB budgets (API client for the `blueprints/ynab/` mirror), S3, and SimbaKit
|
|
- `homebox_mcp.py` - Homebox inventory tools via the `homebox-mcp` stdio MCP server; admin-only (compiled into `app.admin_agent`, picked per user by `get_agent_for`)
|
|
- `bookstack_mcp.py` - BookStack wiki tools (read/create/update, no deletes) via the `bookstack-mcp` npm stdio server; admin-only like Homebox
|
|
- `mcp_tools.py` - shared allowlist + admin-gate wrapping for stdio MCP integrations
|
|
- `scripts/` - Administrative and utility scripts
|
|
- `add_user.py` - Create users manually
|
|
- `user_message_stats.py` - User message statistics
|
|
- `manage_vectorstore.py` / `inspect_vector_store.py` - Vector store CLI
|
|
- `index_immich.py` - Index Immich photos into the vector store
|
|
- `link_imessage.py` - Link iMessage contacts to users
|
|
- `deploy.sh` - Blue-green deployment script
|
|
- `migrations/` - Database migration files
|
|
- `tests/` - pytest test suite (`tests/unit/` covers email/crypto, YNAB mirror, Obsidian, SimbaKit, rate limiting, user model)
|
|
|
|
**simbakit/** (standalone subproject):
|
|
- Self-hosted PetKit device poller + web dashboard for Simba's smart devices (feeder, litter box, fountain, purifier)
|
|
- Own `pyproject.toml`, `uv.lock`, and Dockerfile; Python 3.14 + uv
|
|
- Polls PetKit cloud API every 2 min into SQLite; Chart.js dashboard; docs in `simbakit/docs/`
|
|
- Queried from the main app via `utils/simbakit_service.py`
|
|
|
|
**Frontend** (`raggr-frontend/`):
|
|
- React 19 with Rsbuild bundler
|
|
- Tailwind CSS for styling
|
|
- Built to `dist/`, served by backend at `/`
|
|
- Key components: `ChatScreen`, `ConversationsPage`, `LoginScreen`, `AdminPanel`, `ScheduledMessagesPanel`, `ToolBubble` (agent tool-call display)
|
|
|
|
**Auth Flow**: LLDAP → Authelia (OIDC) → Backend JWT → Frontend localStorage
|
|
|
|
**Data Sources**: Paperless-NGX documents, Obsidian vault, Immich photos (indexed via `scripts/index_immich.py` / `make reindex`)
|
|
|
|
## Testing
|
|
|
|
Always run `make test` before pushing code to ensure all tests pass.
|
|
|
|
```bash
|
|
make test # Run tests
|
|
make test-cov # Run tests with coverage
|
|
```
|
|
|
|
Note: `make test` may fail locally (no pytest on PATH); use `.venv/bin/pytest tests/ -q` instead.
|
|
|
|
## Git Workflow
|
|
|
|
- The host runs **Gitea** (`git.torrtle.co`), not GitHub, despite the remote name. Use the **`tea` CLI** for PRs (Gitea flavor: `tea pr create`, `tea pr list`, etc. — no `tea create`).
|
|
- Typical flow: commit locally, `git push -u origin <branch>`, then `tea pr create -t "title" -d "body"`.
|
|
- "kermit" is the user's slang for **commit** (e.g. "kermit and push" = commit and push). There is no kermit tool.
|
|
- Keep one logical change per branch/PR; don't ride along on stale feature branches.
|
|
- Don't commit `.DS_Store`, `chroma_db/`, `chromadb/`, `.planning/` (local artifacts).
|
|
|
|
## Key Patterns
|
|
|
|
- All endpoints are async (`async def`)
|
|
- Use `@jwt_refresh_token_required` for authenticated endpoints
|
|
- Use `@admin_required` for admin-only endpoints (checks `lldap_admin` group)
|
|
- Tortoise ORM models in `blueprints/*/models.py`
|
|
- External integrations are service classes in `utils/*_service.py`
|
|
- Frontend API services in `raggr-frontend/src/api/`
|
|
|
|
## Environment Variables
|
|
|
|
See `.env.example`. Key ones:
|
|
- `DATABASE_URL` - PostgreSQL connection
|
|
- `OIDC_*` - Authelia OIDC configuration
|
|
- `LLAMA_SERVER_URL` / `LLAMA_MODEL_NAME` - Local Llama/Ollama LLM server
|
|
- `EMBEDDING_SERVER_URL` / `EMBEDDING_MODEL_NAME` - Embedding model
|
|
- `OPENAI_API_KEY` - Fallback LLM; `TAVILY_API_KEY` - web search tool
|
|
- `PAPERLESS_TOKEN` / `BASE_URL` - Document source
|
|
- `TWILIO_*`, `ALLOWED_IMESSAGE_NUMBERS`, `ALLOWED_WHATSAPP_NUMBERS` - iMessage/WhatsApp webhooks
|
|
- `MAILGUN_*`, `EMAIL_HMAC_SECRET`, `FERNET_KEY` - email integration
|
|
- `SENDBLUE_*` - SMS sending for scheduled messages
|
|
- `YNAB_*`, `MEALIE_*`, `OBSIDIAN_*`, `IMMICH_*`, `SIMBAKIT_URL`, `HOMEBOX_URL`/`HOMEBOX_TOKEN`, `BOOKSTACK_BASE_URL`/`BOOKSTACK_TOKEN_ID`/`BOOKSTACK_TOKEN_SECRET` - household integrations
|
|
|
|
## Documentation
|
|
|
|
Project docs live in `docs/` (mkdocs): `deployment.md`, `development.md`, `authentication.md`, `MIGRATIONS.md`, `VECTORSTORE.md`, `TASKS.md`, `ynab_integration/`.
|