simbarag/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

SimbaRAG is a RAG (Retrieval-Augmented Generation) conversational AI system for querying information about Simba (a cat) and the household. It ingests documents from Paperless-NGX, Obsidian, and Immich, stores embeddings in PostgreSQL via pgvector (Chroma client), uses LLMs (Ollama or OpenAI) to answer questions with a tool-using LangChain agent, and exposes chat over web, iMessage, and WhatsApp.

## Commands

### Development

```bash
# Start environment (raggr runs as raggr-blue/raggr-green behind compose profiles)
docker compose --profile blue up --build

# View logs (Makefile resolves the active color)
make logs

# Reindex vector store (all sources / Obsidian only)
make reindex
make reindex-obsidian
```

### Database Migrations (Aerich/Tortoise ORM)

```bash
# Generate migration (must run in Docker with DB access)
make migrate-new

# Apply migrations (auto-runs on startup, manual if needed)
make migrate

# View migration history
make migrate-history
```

Migrations must be backward-compatible: blue-green deploys run `aerich upgrade` in the new container while the old code still serves.

### Frontend

```bash
cd raggr-frontend
yarn install
yarn build      # Production build
yarn dev        # Dev server (rarely needed, backend serves frontend)
```

### Production

Zero-downtime blue-green deploy: `raggr-blue` (port 8080) and `raggr-green` (port 8081) alternate behind Caddy health checks. See `docs/deployment.md`.

```bash
make redeploy    # git pull + scripts/deploy.sh (build new color, wait healthy, stop old)
```

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     Docker Compose                          │
├─────────────────────────────────────────────────────────────┤
│  raggr-blue/green (8080/8081)   │  postgres (port 5432)    │
│  ├── Quart backend              │  PostgreSQL 16 + pgvector│
│  ├── React frontend (served)    │                          │
│  └── Twilio webhooks (iMessage/ │                          │
│      WhatsApp)                  │                          │
└─────────────────────────────────────────────────────────────┘
```

**Backend** (root directory):
- `app.py` - Quart application entry, serves API and static frontend
- `main.py` - RAG logic, document indexing, LLM interaction, LangChain agent
- `llm.py` - LLM client with Ollama primary, OpenAI fallback
- `blueprints/` - API routes organized as Quart blueprints
  - `users/` - OIDC auth, JWT tokens, RBAC with LDAP groups
  - `conversation/` - Chat conversations, message history, streamed answers
  - `rag/` - Document indexing endpoints (admin-only); logic in `rag/logic.py`
  - `email/` - Email integration (Mailgun webhooks, Fernet encryption, IMAP, parsing)
  - `imessage/` - Twilio webhook receiver for iMessage (HMAC-validated, number allowlist)
  - `whatsapp/` - Twilio webhook receiver for WhatsApp (TWIML replies, SendBlue send support)
  - `scheduled_messages/` - Scheduled message delivery with background scheduler
  - `ynab/` - Postgres mirror of the YNAB budget (`ynab_*` tables), delta-synced via `server_knowledge` (background loop + on-demand before each tool call, full resync daily); `ynab_changes` is the edit history YNAB itself doesn't keep. Agent tools in `tools.py`; admin-only read-only `ynab_sql` in `sql_tool.py`. Manual sync: `make ynab-sync` / `make ynab-resync`
- `config/` - Configuration modules
  - `db.py` - Database setup
  - `oidc_config.py` - OIDC authentication configuration
- `utils/` - Reusable utilities and household integrations
  - `chunker.py` - Document chunking for embeddings
  - `cleaner.py` - PDF cleaning and summarization
  - `image_process.py` - Image description with LLM
  - `request.py` - Paperless-NGX API client
  - `obsidian_service.py`, `mealie_service.py`, `s3_client.py`, `simbakit_service.py` - integrations with Obsidian notes, Mealie recipes, YNAB budgets, S3, and SimbaKit
  - `homebox_mcp.py` - Homebox inventory tools via the `homebox-mcp` stdio MCP server; admin-only (compiled into `app.admin_agent`, picked per user by `get_agent_for`)
  - `bookstack_mcp.py` - BookStack wiki tools (read/create/update, no deletes) via the `bookstack-mcp` npm stdio server; admin-only like Homebox
  - `mcp_tools.py` - shared allowlist + admin-gate wrapping for stdio MCP integrations
- `scripts/` - Administrative and utility scripts
  - `add_user.py` - Create users manually
  - `user_message_stats.py` - User message statistics
  - `manage_vectorstore.py` / `inspect_vector_store.py` - Vector store CLI
  - `index_immich.py` - Index Immich photos into the vector store
  - `link_imessage.py` - Link iMessage contacts to users
  - `deploy.sh` - Blue-green deployment script
- `migrations/` - Database migration files
- `tests/` - pytest test suite (`tests/unit/` covers email/crypto, YNAB mirror, Obsidian, SimbaKit, rate limiting, user model)

**simbakit/** (standalone subproject):
- Self-hosted PetKit device poller + web dashboard for Simba's smart devices (feeder, litter box, fountain, purifier)
- Own `pyproject.toml`, `uv.lock`, and Dockerfile; Python 3.14 + uv
- Polls PetKit cloud API every 2 min into SQLite; Chart.js dashboard; docs in `simbakit/docs/`
- Queried from the main app via `utils/simbakit_service.py`

**Frontend** (`raggr-frontend/`):
- React 19 with Rsbuild bundler
- Tailwind CSS for styling
- Built to `dist/`, served by backend at `/`
- Key components: `ChatScreen`, `ConversationsPage`, `LoginScreen`, `AdminPanel`, `ScheduledMessagesPanel`, `ToolBubble` (agent tool-call display)

**Auth Flow**: LLDAP → Authelia (OIDC) → Backend JWT → Frontend localStorage

**Data Sources**: Paperless-NGX documents, Obsidian vault, Immich photos (indexed via `scripts/index_immich.py` / `make reindex`)

## Testing

Always run `make test` before pushing code to ensure all tests pass.

```bash
make test       # Run tests
make test-cov   # Run tests with coverage
```

Note: `make test` may fail locally (no pytest on PATH); use `.venv/bin/pytest tests/ -q` instead.

## Git Workflow

- The host runs **Gitea** (`git.torrtle.co`), not GitHub, despite the remote name. Use the **`tea` CLI** for PRs (Gitea flavor: `tea pr create`, `tea pr list`, etc. — no `tea create`).
- Typical flow: commit locally, `git push -u origin <branch>`, then `tea pr create -t "title" -d "body"`.
- "kermit" is the user's slang for **commit** (e.g. "kermit and push" = commit and push). There is no kermit tool.
- Keep one logical change per branch/PR; don't ride along on stale feature branches.
- Don't commit `.DS_Store`, `chroma_db/`, `chromadb/`, `.planning/` (local artifacts).

## Key Patterns

- All endpoints are async (`async def`)
- Use `@jwt_refresh_token_required` for authenticated endpoints
- Use `@admin_required` for admin-only endpoints (checks `lldap_admin` group)
- Tortoise ORM models in `blueprints/*/models.py`
- External integrations are service classes in `utils/*_service.py`
- Frontend API services in `raggr-frontend/src/api/`

## Environment Variables

See `.env.example`. Key ones:
- `DATABASE_URL` - PostgreSQL connection
- `OIDC_*` - Authelia OIDC configuration
- `LLAMA_SERVER_URL` / `LLAMA_MODEL_NAME` - Local Llama/Ollama LLM server
- `EMBEDDING_SERVER_URL` / `EMBEDDING_MODEL_NAME` - Embedding model
- `OPENAI_API_KEY` - Fallback LLM; `TAVILY_API_KEY` - web search tool
- `PAPERLESS_TOKEN` / `BASE_URL` - Document source
- `TWILIO_*`, `ALLOWED_IMESSAGE_NUMBERS`, `ALLOWED_WHATSAPP_NUMBERS` - iMessage/WhatsApp webhooks
- `MAILGUN_*`, `EMAIL_HMAC_SECRET`, `FERNET_KEY` - email integration
- `SENDBLUE_*` - SMS sending for scheduled messages
- `YNAB_*`, `MEALIE_*`, `OBSIDIAN_*`, `IMMICH_*`, `SIMBAKIT_URL`, `HOMEBOX_URL`/`HOMEBOX_TOKEN`, `BOOKSTACK_BASE_URL`/`BOOKSTACK_TOKEN_ID`/`BOOKSTACK_TOKEN_SECRET` - household integrations

## Documentation

Project docs live in `docs/` (mkdocs): `deployment.md`, `development.md`, `authentication.md`, `MIGRATIONS.md`, `VECTORSTORE.md`, `TASKS.md`, `ynab_integration/`.