diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md new file mode 100644 index 0000000..98ade32 --- /dev/null +++ b/.planning/codebase/ARCHITECTURE.md @@ -0,0 +1,184 @@ +# Architecture + +**Analysis Date:** 2026-02-04 + +## Pattern Overview + +**Overall:** RAG (Retrieval-Augmented Generation) system with multi-agent conversational AI architecture + +**Key Characteristics:** +- RAG pattern with vector database for document retrieval +- LangChain agent-based orchestration with tool calling +- Blueprint-based API organization (Quart framework) +- Asynchronous request handling throughout +- OIDC authentication with RBAC via LDAP groups +- Streaming SSE responses for real-time chat + +## Layers + +**API Layer (Quart Blueprints):** +- Purpose: HTTP request handling and route organization +- Location: `blueprints/*/` +- Contains: Blueprint definitions, route handlers, request/response serialization +- Depends on: Logic layer, models, JWT middleware +- Used by: Frontend (React SPA), external clients + +**Logic Layer:** +- Purpose: Business logic and domain operations +- Location: `blueprints/*/logic.py`, `blueprints/*/agents.py`, `main.py` +- Contains: Conversation management, RAG indexing, agent orchestration, tool execution +- Depends on: Models, external services, LLM clients +- Used by: API layer + +**Model Layer (Tortoise ORM):** +- Purpose: Database schema and data access +- Location: `blueprints/*/models.py` +- Contains: ORM model definitions, Pydantic serializers, database relationships +- Depends on: PostgreSQL database +- Used by: Logic layer, API layer + +**Integration Layer:** +- Purpose: External service communication +- Location: `utils/`, `config/` +- Contains: Service clients (YNAB, Mealie, Paperless-NGX, OIDC) +- Depends on: External APIs +- Used by: Logic layer, tools + +**Tool Layer (LangChain Tools):** +- Purpose: Agent-callable functions for extended capabilities +- Location: `blueprints/conversation/agents.py` +- Contains: `@tool` decorated functions for document search, web search, YNAB, Mealie +- Depends on: Integration layer, RAG logic +- Used by: LangChain agent + +**Frontend (React SPA):** +- Purpose: User interface +- Location: `raggr-frontend/` +- Contains: React components, API service clients, authentication context +- Depends on: Backend API endpoints +- Used by: End users + +## Data Flow + +**Chat Query Flow:** + +1. User submits query in frontend (`raggr-frontend/src/components/ChatScreen.tsx`) +2. Frontend calls `/api/conversation/query` with SSE streaming (`raggr-frontend/src/api/conversationService.ts`) +3. API endpoint validates JWT, fetches user and conversation (`blueprints/conversation/__init__.py`) +4. User message saved to database via Tortoise ORM (`blueprints/conversation/models.py`) +5. Recent conversation history (last 10 messages) loaded and formatted +6. LangChain agent invoked with messages payload (`blueprints/conversation/agents.py`) +7. Agent decides which tools to call based on query (simba_search, ynab_*, mealie_*, web_search) +8. Tools execute: RAG query (`blueprints/rag/logic.py`), API calls (`utils/*.py`) +9. LLM generates response using tool results +10. Response streamed back via SSE events (status updates, content chunks) +11. Complete response saved to database +12. Frontend renders streaming response in real-time + +**RAG Document Flow:** + +1. Admin triggers indexing via `/api/rag/index` or `/api/rag/reindex` +2. RAG logic fetches documents from Paperless-NGX (`blueprints/rag/fetchers.py`) +3. Documents chunked using LangChain text splitter (1000 chars, 200 overlap) +4. Embeddings generated using OpenAI embedding model (text-embedding-3-small) +5. Vectors stored in ChromaDB persistent collection (`chroma_db/`) +6. Query time: embeddings generated for query, similarity search retrieves top 2 docs +7. Documents serialized and passed to LLM as context + +**State Management:** +- Conversation state: PostgreSQL via Tortoise ORM +- Vector embeddings: ChromaDB persistent storage +- User sessions: JWT tokens in frontend localStorage +- Authentication: OIDC state in-memory (production should use Redis) + +## Key Abstractions + +**Conversation:** +- Purpose: Represents a chat thread with message history +- Examples: `blueprints/conversation/models.py` +- Pattern: Aggregate root with message collection, foreign key to User + +**ConversationMessage:** +- Purpose: Individual message in conversation (user or assistant) +- Examples: `blueprints/conversation/models.py` +- Pattern: Entity with enum speaker type, foreign key to Conversation + +**User:** +- Purpose: Authenticated user with OIDC or local credentials +- Examples: `blueprints/users/models.py` +- Pattern: Entity with bcrypt password hashing, LDAP group membership, admin check method + +**LangChain Agent:** +- Purpose: Orchestrates LLM calls with tool selection +- Examples: `blueprints/conversation/agents.py` (main_agent) +- Pattern: ReAct agent pattern with function calling via OpenAI-compatible API + +**Tool Functions:** +- Purpose: Discrete capabilities callable by the agent +- Examples: `simba_search`, `ynab_budget_summary`, `mealie_shopping_list` in `blueprints/conversation/agents.py` +- Pattern: Decorated functions with docstrings that become tool descriptions + +**LLMClient:** +- Purpose: Abstraction over LLM providers with fallback +- Examples: `llm.py`, `blueprints/conversation/agents.py` +- Pattern: Primary llama-server with OpenAI fallback, OpenAI-compatible interface + +**Service Clients:** +- Purpose: External API integration wrappers +- Examples: `utils/ynab_service.py`, `utils/mealie_service.py`, `utils/request.py` +- Pattern: Class-based clients with async methods, relative date parsing + +## Entry Points + +**Web Application:** +- Location: `app.py` +- Triggers: `python app.py` or Docker container startup +- Responsibilities: Initialize Quart app, register blueprints, configure Tortoise ORM, serve React frontend + +**CLI Indexing:** +- Location: `main.py` (when run as script) +- Triggers: `python main.py --reindex` or `--query ` +- Responsibilities: Document indexing, direct RAG queries without API + +**Database Migrations:** +- Location: `aerich_config.py` +- Triggers: `aerich migrate`, `aerich upgrade` +- Responsibilities: Schema migration generation and application + +**Admin Scripts:** +- Location: `scripts/add_user.py`, `scripts/user_message_stats.py`, `scripts/manage_vectorstore.py` +- Triggers: Manual execution +- Responsibilities: User management, analytics, vector store inspection + +**React Frontend:** +- Location: `raggr-frontend/src/index.tsx` +- Triggers: Bundle served at `/` by backend +- Responsibilities: Initialize React app, authentication context, routing + +## Error Handling + +**Strategy:** Try-catch with logging at service boundaries, HTTP status codes for client errors + +**Patterns:** +- API routes: Return JSON error responses with appropriate HTTP status codes (400, 401, 403, 500) +- Example: `blueprints/rag/__init__.py` line 26-27 +- Async operations: Try-except blocks with logger.error for traceability +- Example: `blueprints/conversation/agents.py` line 142-145 (YNAB tool error handling) +- JWT validation: Decorator-based authentication with 401 response on failure +- Example: `@jwt_refresh_token_required` in all protected routes +- Frontend: Error callbacks in streaming service, redirect to login on session expiry +- Example: `raggr-frontend/src/components/ChatScreen.tsx` line 234-237 +- Agent tool failures: Return error string to agent for recovery or user messaging +- Example: `blueprints/conversation/agents.py` line 384-385 + +## Cross-Cutting Concerns + +**Logging:** Python logging module with INFO level, structured with logger names by module (utils.ynab_service, blueprints.conversation.agents) + +**Validation:** Pydantic models for serialization, Tortoise ORM field constraints, JWT token validation via quart-jwt-extended + +**Authentication:** OIDC (Authelia) with PKCE flow → JWT tokens → RBAC via LDAP groups. Decorators: `@jwt_refresh_token_required` for auth, `@admin_required` for admin-only endpoints (`blueprints/users/decorators.py`) + +--- + +*Architecture analysis: 2026-02-04* diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md new file mode 100644 index 0000000..42b7ffe --- /dev/null +++ b/.planning/codebase/CONCERNS.md @@ -0,0 +1,265 @@ +# Codebase Concerns + +**Analysis Date:** 2026-02-04 + +## Tech Debt + +**Duplicate system prompts in streaming and non-streaming endpoints:** +- Issue: Large system prompt (112 lines) duplicated verbatim in two endpoints +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (lines 56-111 and 206-261) +- Impact: Changes to prompt must be made in two places, increasing maintenance burden and risk of inconsistency +- Fix approach: Extract system prompt to a constant or configuration file + +**SQLite database for indexing tracking alongside PostgreSQL:** +- Issue: Uses SQLite (`database/visited.db`) to track indexed Paperless documents while main data is in PostgreSQL +- Files: `/Users/ryanchen/Programs/raggr/main.py` (lines 73, 212, 226), `/Users/ryanchen/Programs/raggr/scripts/index_immich.py` (line 33) +- Impact: Two database systems to manage, no transactions across databases, deployment complexity +- Fix approach: Migrate indexing tracking to PostgreSQL table using Tortoise ORM + +**Broad exception catching throughout codebase:** +- Issue: 35+ instances of `except Exception as e` catching all exceptions indiscriminately +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (12 instances), `/Users/ryanchen/Programs/raggr/utils/ynab_service.py` (7 instances), `/Users/ryanchen/Programs/raggr/utils/mealie_service.py` (7 instances), `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 171), `/Users/ryanchen/Programs/raggr/blueprints/rag/__init__.py` (lines 26, 46) +- Impact: Masks programming errors, makes debugging difficult, catches system exceptions that shouldn't be caught +- Fix approach: Replace with specific exception types (ValueError, KeyError, HTTPException, etc.) + +**Legacy main.py RAG logic not used by application:** +- Issue: `/Users/ryanchen/Programs/raggr/main.py` contains 275 lines of RAG logic including `consult_oracle()`, `classify_query()`, `consult_simba_oracle()` but app uses LangChain agents instead +- Files: `/Users/ryanchen/Programs/raggr/main.py`, `/Users/ryanchen/Programs/raggr/app.py` (imports `consult_simba_oracle` but endpoint is commented/unused) +- Impact: Dead code increases maintenance burden, confuses new developers about which code path is active +- Fix approach: Archive or remove unused code after verifying no production dependencies + +**Environment variable typo in docker-compose:** +- Issue: Docker compose uses `TAVILIY_KEY` instead of `TAVILY_API_KEY` +- Files: `/Users/ryanchen/Programs/raggr/docker-compose.yml` (line 41), `/Users/ryanchen/Programs/raggr/docker-compose.dev.yml` (line 44) +- Impact: Tavily web search won't work in production Docker deployment +- Fix approach: Standardize on `TAVILY_API_KEY` throughout + +**Hardcoded OpenAI model in conversation rename logic:** +- Issue: Uses `gpt-4o-mini` without environment variable configuration +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/logic.py` (line 72) +- Impact: Cannot switch models, will fail if OpenAI key not configured even when using local LLM +- Fix approach: Make model configurable via environment variable, use same fallback pattern as main agent + +**Debug mode enabled in production app entry:** +- Issue: `debug=True` hardcoded in app.run() +- Files: `/Users/ryanchen/Programs/raggr/app.py` (line 165) +- Impact: Exposes stack traces and sensitive information if run directly (mitigated by Docker CMD using startup.sh) +- Fix approach: Use environment variable for debug flag + +## Known Bugs + +**Empty returns in PDF cleaner error handling:** +- Issue: Error handlers return None or empty lists without logging context +- Files: `/Users/ryanchen/Programs/raggr/utils/cleaner.py` (lines 58, 74, 81) +- Symptoms: Silent failures during PDF processing, no indication why document wasn't indexed +- Trigger: PDF processing errors (malformed PDFs, image conversion failures) +- Workaround: Check logs at DEBUG level, manually test PDF processing + +**Console debug statements left in production code:** +- Issue: print() statements instead of logging in multiple locations +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (lines 109-113), `/Users/ryanchen/Programs/raggr/blueprints/conversation/logic.py` (line 20), `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 311), `/Users/ryanchen/Programs/raggr/raggr-frontend/src/components/ChatScreen.tsx` (lines 99-100, 132-133) +- Symptoms: Unstructured output mixed with proper logs, no log levels +- Fix approach: Replace with structured logging + +**Conversation name timestamp method incorrect:** +- Issue: Uses `.timestamp` property instead of `.timestamp()` method +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 330) +- Symptoms: Conversation name will be method reference string instead of timestamp +- Fix approach: Change to `datetime.datetime.now().timestamp()` + +## Security Considerations + +**JWT secret key has weak default:** +- Risk: Default JWT_SECRET_KEY is "SECRET_KEY" if environment variable not set +- Files: `/Users/ryanchen/Programs/raggr/app.py` (line 39) +- Current mitigation: Documentation requires setting environment variable +- Recommendations: Fail fast on startup if JWT_SECRET_KEY is default value, generate random key on first run + +**Hardcoded API key placeholder in llama-server configuration:** +- Risk: API key set to "not-needed" for local llama-server +- Files: `/Users/ryanchen/Programs/raggr/llm.py` (line 16), `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (line 28) +- Current mitigation: Only used for local trusted network LLM servers +- Recommendations: Document that llama-server should be on trusted network only, consider basic authentication + +**No rate limiting on streaming endpoints:** +- Risk: Users can spawn unlimited concurrent streaming requests +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 29) +- Current mitigation: None +- Recommendations: Add per-user rate limiting, request queue, or connection limit + +**Sensitive data in error messages:** +- Risk: Full exception details returned to client in tool error messages +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (lines 145, 219, 280, etc.) +- Current mitigation: Only exposed to authenticated users +- Recommendations: Sanitize error messages, return generic errors to client, log full details server-side + +## Performance Bottlenecks + +**Large conversation history loaded on every query:** +- Problem: Fetches all messages then slices to last 10 in memory +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (lines 38, 47-50, 188, 197-200) +- Cause: No database-level limit on message fetch +- Improvement path: Add database query limit, use `.order_by('-created_at').limit(10)` at query level + +**Sequential document indexing:** +- Problem: Documents indexed one at a time in loop +- Files: `/Users/ryanchen/Programs/raggr/main.py` (lines 67-96) +- Cause: No parallel processing or batching +- Improvement path: Use asyncio.gather() for concurrent PDF processing, batch ChromaDB inserts + +**No caching for YNAB API calls:** +- Problem: Every query makes fresh API calls even for recently accessed data +- Files: `/Users/ryanchen/Programs/raggr/utils/ynab_service.py` (all methods) +- Cause: No caching layer +- Improvement path: Add Redis/in-memory cache with TTL for budget data, cache budget summaries for 5-15 minutes + +**Frontend loads all conversations on mount:** +- Problem: Fetches all conversations without pagination +- Files: `/Users/ryanchen/Programs/raggr/raggr-frontend/src/components/ChatScreen.tsx` (lines 89-104) +- Cause: No pagination in API or frontend +- Improvement path: Add cursor-based pagination, lazy load older conversations + +**ChromaDB persistence path creates I/O bottleneck:** +- Problem: All embedding queries/inserts hit disk-backed SQLite database +- Files: `/Users/ryanchen/Programs/raggr/main.py` (line 19) +- Cause: Uses PersistentClient without in-memory optimization +- Improvement path: Consider ChromaDB server mode for production, add memory-backed cache layer + +## Fragile Areas + +**LangChain agent tool calling depends on exact model support:** +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (line 733) +- Why fragile: Comment says "Llama 3.1 supports native function calling" but not all local models do +- Test coverage: No automated tests for tool calling +- Safe modification: Always test with target model before deploying, add fallback for models without tool support + +**OIDC user provisioning auto-migrates local users:** +- Files: `/Users/ryanchen/Programs/raggr/blueprints/users/oidc_service.py` (lines 42-53) +- Why fragile: Automatically converts local auth users to OIDC based on email match, clears passwords +- Test coverage: No tests detected +- Safe modification: Add dry-run mode, require admin confirmation for migrations, back up user table first + +**Streaming response parsing relies on specific line format:** +- Files: `/Users/ryanchen/Programs/raggr/raggr-frontend/src/api/conversationService.ts` (lines 95-135) +- Why fragile: Assumes SSE format with `data: ` prefix and JSON, buffer handling for incomplete lines +- Test coverage: No tests for edge cases (connection drops mid-stream, malformed JSON, large chunks) +- Safe modification: Add comprehensive error handling, test with slow connections and large responses + +**Vector store query uses unvalidated metadata filters:** +- Files: `/Users/ryanchen/Programs/raggr/main.py` (lines 133-155) +- Why fragile: Metadata filters from QueryGenerator passed directly to ChromaDB without validation +- Test coverage: None detected +- Safe modification: Validate filter structure before query, whitelist allowed filter keys + +**Document chunking without validation:** +- Files: `/Users/ryanchen/Programs/raggr/utils/chunker.py` referenced in `/Users/ryanchen/Programs/raggr/main.py` (line 69) +- Why fragile: No validation of chunk size, overlap, or content before embedding +- Test coverage: None detected +- Safe modification: Add max chunk length validation, handle empty documents gracefully + +## Scaling Limits + +**Single PostgreSQL connection per request:** +- Current capacity: Depends on PostgreSQL max_connections (default ~100) +- Limit: Connection exhaustion under high concurrent load +- Scaling path: Implement connection pooling with Tortoise ORM pool settings, increase PostgreSQL max_connections + +**ChromaDB local persistence not horizontally scalable:** +- Current capacity: Single-node file-based storage +- Limit: Cannot distribute across multiple app instances, I/O bound on single disk +- Scaling path: Migrate to ChromaDB server mode with shared storage or dedicated vector DB (Qdrant, Pinecone, Weaviate) + +**Server-sent events keep connections open:** +- Current capacity: Limited by web server worker count and file descriptor limits +- Limit: Each streaming query holds connection open for full duration (10-60+ seconds) +- Scaling path: Use message queue (Redis Streams, RabbitMQ) for response streaming, implement connection pooling + +**No horizontal scaling for background indexing:** +- Current capacity: Single process indexes documents sequentially +- Limit: Cannot parallelize across multiple workers/containers +- Scaling path: Implement task queue (Celery, RQ) for distributed indexing, use message broker to coordinate + +**Frontend state management in React useState:** +- Current capacity: Works for single user, no persistence +- Limit: State lost on refresh, no offline support, memory growth with long conversations +- Scaling path: Migrate to Redux/Zustand with persistence, implement virtual scrolling for long conversations + +## Dependencies at Risk + +**ynab Python package is community-maintained:** +- Risk: Unofficial YNAB API wrapper, last update may lag behind API changes +- Impact: YNAB features break if API changes +- Migration plan: Monitor YNAB API changelog, consider switching to direct httpx/aiohttp requests for control + +**LangChain rapid version changes:** +- Risk: Frequent breaking changes between minor versions in LangChain ecosystem +- Impact: Upgrades require code changes, agent patterns deprecated +- Migration plan: Pin specific versions in pyproject.toml, test thoroughly before upgrading + +**Quart framework less mature than Flask:** +- Risk: Smaller community, fewer third-party extensions, async bugs less documented +- Impact: Harder to find solutions for edge cases +- Migration plan: Consider FastAPI as alternative (better async support, more active), or Flask with async support + +## Missing Critical Features + +**No observability/monitoring:** +- Problem: No structured logging, metrics, or tracing +- Blocks: Understanding production issues, performance debugging, user behavior analysis +- Priority: High + +**No backup strategy for ChromaDB vector store:** +- Problem: Vector embeddings not backed up, expensive to regenerate +- Blocks: Disaster recovery, migrating instances +- Priority: High + +**No API versioning:** +- Problem: Breaking API changes will break existing clients +- Blocks: Frontend/backend independent deployment +- Priority: Medium + +**No health check endpoints:** +- Problem: Container orchestration cannot verify service health +- Blocks: Proper Kubernetes deployment, load balancer integration +- Priority: Medium + +**No user quotas or resource limits:** +- Problem: Users can consume unlimited API calls, storage, compute +- Blocks: Cost control, fair resource allocation +- Priority: Medium + +## Test Coverage Gaps + +**No tests for LangChain agent tools:** +- What's not tested: All 15 tools in `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` +- Files: No test files detected for agents module +- Risk: Tool failures not caught until production, parameter handling bugs +- Priority: High + +**No tests for streaming SSE implementation:** +- What's not tested: Server-sent events parsing, partial message handling, error recovery +- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (streaming endpoints), `/Users/ryanchen/Programs/raggr/raggr-frontend/src/api/conversationService.ts` +- Risk: Connection drops, malformed responses cause undefined behavior +- Priority: High + +**No tests for OIDC authentication flow:** +- What's not tested: User provisioning, group claims parsing, token validation +- Files: `/Users/ryanchen/Programs/raggr/blueprints/users/oidc_service.py`, `/Users/ryanchen/Programs/raggr/blueprints/users/__init__.py` +- Risk: Auth bypass, user migration bugs, group permission issues +- Priority: High + +**No integration tests for RAG pipeline:** +- What's not tested: End-to-end document indexing, query, and response generation +- Files: `/Users/ryanchen/Programs/raggr/blueprints/rag/logic.py`, `/Users/ryanchen/Programs/raggr/main.py` +- Risk: Embedding model changes, ChromaDB version changes break retrieval +- Priority: Medium + +**No tests for external service integrations:** +- What's not tested: YNAB API error handling, Mealie API error handling, Tavily search failures +- Files: `/Users/ryanchen/Programs/raggr/utils/ynab_service.py`, `/Users/ryanchen/Programs/raggr/utils/mealie_service.py` +- Risk: API changes break features silently, rate limits not handled +- Priority: Medium + +--- + +*Concerns audit: 2026-02-04* diff --git a/.planning/codebase/CONVENTIONS.md b/.planning/codebase/CONVENTIONS.md new file mode 100644 index 0000000..30b18eb --- /dev/null +++ b/.planning/codebase/CONVENTIONS.md @@ -0,0 +1,333 @@ +# Coding Conventions + +**Analysis Date:** 2026-02-04 + +## Naming Patterns + +**Files:** +- Python: `snake_case.py` - `ynab_service.py`, `mealie_service.py`, `oidc_service.py` +- TypeScript/React: `PascalCase.tsx` for components, `camelCase.ts` for services + - Components: `ChatScreen.tsx`, `AnswerBubble.tsx`, `QuestionBubble.tsx` + - Services: `conversationService.ts`, `userService.ts`, `oidcService.ts` +- Config files: `snake_case.py` - `aerich_config.py`, `oidc_config.py` + +**Functions:** +- Python: `snake_case` - `get_budget_summary()`, `parse_relative_date()`, `consult_simba_oracle()` +- TypeScript: `camelCase` - `handleQuestionSubmit()`, `sendQueryStream()`, `fetchWithRefreshToken()` + +**Variables:** +- Python: `snake_case` - `budget_id`, `access_token`, `llama_url`, `current_user_uuid` +- TypeScript: `camelCase` - `conversationId`, `streamingContent`, `isLoading` + +**Types:** +- Python classes: `PascalCase` - `YNABService`, `MealieService`, `LLMClient`, `User`, `Conversation` +- Python enums: `PascalCase` with SCREAMING_SNAKE_CASE values - `Speaker.USER`, `Speaker.SIMBA` +- TypeScript interfaces: `PascalCase` - `Message`, `Conversation`, `QueryResponse`, `StreamEvent` +- TypeScript types: `PascalCase` - `ChatScreenProps`, `QuestionAnswer` + +**Constants:** +- Python: `SCREAMING_SNAKE_CASE` - `DATABASE_URL`, `TORTOISE_CONFIG`, `PROVIDER` +- TypeScript: `camelCase` - `baseUrl`, `conversationBaseUrl` + +## Code Style + +**Formatting:** +- Python: No explicit formatter configured (no Black, autopep8, or yapf config detected) +- Manual formatting observed: 4-space indentation, line length ~88-100 chars +- TypeScript: Biome 2.3.10 configured in `raggr-frontend/package.json` + - No explicit biome.json found, using defaults + +**Linting:** +- Python: No linter config detected (no pylint, flake8, ruff config) +- TypeScript: Biome handles linting via `@biomejs/biome` package + +**Imports:** +- Python: Standard library first, then third-party, then local imports + ```python + import os + import logging + from datetime import datetime + + from dotenv import load_dotenv + from quart import Blueprint + + from .models import User + from .logic import get_conversation + ``` +- TypeScript: React imports, then third-party, then local (relative) + ```typescript + import { useEffect, useState } from "react"; + import { conversationService } from "../api/conversationService"; + import { QuestionBubble } from "./QuestionBubble"; + ``` + +## Import Organization + +**Order:** +1. Standard library imports +2. Third-party framework imports (Flask/Quart/React/etc) +3. Local application imports (blueprints, utils, models) + +**Path Aliases:** +- None detected in TypeScript - uses relative imports (`../api/`, `./components/`) +- Python uses absolute imports for blueprints and utils modules + +**Absolute vs Relative:** +- Python: Absolute imports for cross-module (`from utils.ynab_service import YNABService`) +- TypeScript: Relative imports (`../api/conversationService`) + +## Error Handling + +**Patterns:** +- Python: Try/except with detailed logging + ```python + try: + # operation + logger.info("[SERVICE] Operation details") + except httpx.HTTPStatusError as e: + logger.error(f"[SERVICE] HTTP error: {e.response.status_code}") + raise + except Exception as e: + logger.error(f"[SERVICE] Error: {type(e).__name__}: {str(e)}") + logger.exception("[SERVICE] Full traceback:") + raise + ``` +- TypeScript: Try/catch with console.error, re-throw or handle gracefully + ```typescript + try { + const response = await fetch(); + } catch (error) { + console.error("Failed to fetch:", error); + if (error.message.includes("Session expired")) { + setAuthenticated(false); + } + } + ``` + +**Async Error Handling:** +- Python: `async def` functions use try/except blocks +- TypeScript: `async` functions use try/catch blocks +- Both propagate errors upward with `raise` (Python) or `throw` (TypeScript) + +**HTTP Errors:** +- Python Quart: Return `jsonify({"error": "message"}), status_code` +- Python httpx: Raise HTTPStatusError, log response text +- TypeScript: Throw Error with descriptive message + +## Logging + +**Framework:** +- Python: Standard `logging` module +- TypeScript: `console.log()`, `console.error()` + +**Patterns:** +- Python: Structured logging with prefixes + ```python + logger = logging.getLogger(__name__) + logger.info("[SERVICE] Operation started") + logger.error(f"[SERVICE] Error: {details}") + logger.exception("[SERVICE] Full traceback:") # After except + ``` +- Logging levels: INFO for operations, ERROR for failures, DEBUG for detailed data +- Service-specific prefixes: `[YNAB]`, `[MEALIE]`, `[YNAB TOOLS]` + +**When to Log:** +- Entry/exit of major operations (API calls, database queries) +- Error conditions with full context +- Configuration/initialization status +- Performance metrics (timing critical operations) + +**Examples from codebase:** +```python +logger.info(f"[YNAB] get_budget_summary() called for budget_id: {self.budget_id}") +logger.info(f"[YNAB] Total budgeted: ${total_budgeted:.2f}") +logger.error(f"[YNAB] Error in get_budget_summary(): {type(e).__name__}: {str(e)}") +``` + +## Comments + +**When to Comment:** +- Complex business logic (date parsing, budget calculations) +- Non-obvious workarounds or API quirks +- Important configuration decisions +- Docstrings for all public functions/methods + +**JSDoc/TSDoc:** +- Python: Docstrings with Args/Returns sections + ```python + def get_transactions(self, start_date: Optional[str] = None) -> dict[str, Any]: + """Get transactions filtered by date range. + + Args: + start_date: Start date in YYYY-MM-DD or relative ('this_month') + + Returns: + Dictionary containing matching transactions and summary. + """ + ``` +- TypeScript: Inline comments, no formal JSDoc detected + ```typescript + // Stream events back to client as they happen + async function generate() { + // ... + } + ``` + +**Comment Style:** +- Python: `# Single line` or `"""Docstring"""` +- TypeScript: `// Single line` or `/* Multi-line */` +- No TODOs/FIXMEs in project code (only in node_modules) + +## Function Design + +**Size:** +- Python: 20-100 lines typical, some reach 150+ (service methods with error handling) +- TypeScript: 10-50 lines for React components, 20-80 for service methods +- Large functions acceptable when handling complex workflows (streaming, API interactions) + +**Parameters:** +- Python: Explicit typing with `Optional[type]`, defaults for optional params + ```python + def get_transactions( + self, + start_date: Optional[str] = None, + end_date: Optional[str] = None, + limit: int = 50, + ) -> dict[str, Any]: + ``` +- TypeScript: Interfaces for complex parameter objects + ```typescript + async sendQueryStream( + query: string, + conversation_id: string, + callbacks: StreamCallbacks, + signal?: AbortSignal, + ): Promise + ``` + +**Return Values:** +- Python: Explicit return type hints - `-> dict[str, Any]`, `-> str`, `-> bool` +- TypeScript: Explicit types - `: Promise`, `: void` +- Dictionary/object returns for complex data (not tuples in Python) + +**Async/Await:** +- Python Quart: All route handlers are `async def` +- Python services: Database queries and external API calls are `async` +- TypeScript: All API calls use `async/await` pattern + +## Module Design + +**Exports:** +- Python: No explicit `__all__`, classes/functions imported directly +- TypeScript: Named exports for classes/functions, default export for singleton services + ```typescript + export const conversationService = new ConversationService(); + ``` + +**Barrel Files:** +- Python: `blueprints/__init__.py` defines blueprints, re-exported +- TypeScript: No barrel files, direct imports + +**Structure:** +- Python blueprints: `__init__.py` contains routes, `models.py` for ORM, `logic.py` for business logic +- Services in separate modules: `utils/ynab_service.py`, `utils/mealie_service.py` +- Separation of concerns: routes, models, business logic, utilities + +## Decorators + +**Authentication:** +- `@jwt_refresh_token_required` - Standard auth requirement +- `@admin_required` - Custom decorator for admin-only routes (wraps `@jwt_refresh_token_required`) + +**Route Decorators:** +- `@app.route()` or `@blueprint.route()` with HTTP method +- Async routes: `async def` function signature + +**Tool Decorators (LangChain):** +- `@tool` - Mark functions as LangChain tools +- `@tool(response_format="content_and_artifact")` - Specialized tool responses + +**Pattern:** +```python +@conversation_blueprint.post("/query") +@jwt_refresh_token_required +async def query(): + current_user_uuid = get_jwt_identity() + # ... +``` + +## Type Hints + +**Python:** +- Modern type hints throughout: `dict[str, Any]`, `Optional[str]`, `list[str]` +- Tortoise ORM types: `fields.ForeignKeyRelation` +- No legacy typing module usage (using built-in generics) + +**TypeScript:** +- Strict typing with interfaces +- Union types for variants: `"user" | "simba"`, `'status' | 'content' | 'done' | 'error'` +- Generic types: `Promise`, `React.ChangeEvent` + +## State Management + +**Python (Backend):** +- Database: Tortoise ORM async models +- In-memory: Module-level variables for services (`ynab_service`, `mealie_service`) +- Session: JWT tokens, in-memory dict for OIDC sessions (`_oidc_sessions`) + +**TypeScript (Frontend):** +- React hooks: `useState`, `useEffect`, `useRef` +- localStorage for JWT tokens (via `userService`) +- No global state management library (no Redux/Zustand) + +**Pattern:** +```typescript +const [isLoading, setIsLoading] = useState(false); +const abortControllerRef = useRef(null); +``` + +## Database Conventions + +**ORM:** +- Tortoise ORM with Aerich for migrations +- Models inherit from `Model` base class +- Field definitions: `fields.UUIDField`, `fields.CharField`, `fields.ForeignKeyField` + +**Naming:** +- Table names: Lowercase plural (`users`, `conversations`, `conversation_messages`) +- Foreign keys: Singular model name (`user`, `conversation`) +- Related names: Plural (`conversations`, `messages`) + +**Pattern:** +```python +class Conversation(Model): + id = fields.UUIDField(primary_key=True) + name = fields.CharField(max_length=255) + user: fields.ForeignKeyRelation = fields.ForeignKeyField( + "models.User", related_name="conversations", null=True + ) + + class Meta: + table = "conversations" +``` + +## API Conventions + +**REST Endpoints:** +- Prefix: `/api/{resource}` +- Blueprints: `/api/user`, `/api/conversation`, `/api/rag` +- CRUD patterns: GET for fetch, POST for create/actions, PUT for update, DELETE for remove + +**Request/Response:** +- JSON payloads: `await request.get_json()` +- Responses: `jsonify({...})` with optional status code +- Streaming: Server-Sent Events (SSE) with `text/event-stream` mimetype + +**Authentication:** +- JWT in Authorization header (managed by `quart-jwt-extended`) +- Refresh tokens for long-lived sessions +- OIDC flow for external authentication + +--- + +*Convention analysis: 2026-02-04* diff --git a/.planning/codebase/INTEGRATIONS.md b/.planning/codebase/INTEGRATIONS.md new file mode 100644 index 0000000..5becfaf --- /dev/null +++ b/.planning/codebase/INTEGRATIONS.md @@ -0,0 +1,182 @@ +# External Integrations + +**Analysis Date:** 2026-02-04 + +## APIs & External Services + +**Document Management:** +- Paperless-NGX - Document ingestion and retrieval + - SDK/Client: Custom client in `utils/request.py` using `httpx` + - Auth: `PAPERLESS_TOKEN` (bearer token) + - Base URL: `BASE_URL` environment variable + - Purpose: Fetch documents for indexing, download PDFs, retrieve document metadata and types + +**LLM Services:** +- llama-server (primary) - Local LLM inference via OpenAI-compatible API + - SDK/Client: `openai` Python package (v2.0.1+) + - Connection: `LLAMA_SERVER_URL` (e.g., `http://192.168.1.213:8080/v1`) + - Model: `LLAMA_MODEL_NAME` (e.g., `llama-3.1-8b-instruct`) + - Implementation: `llm.py` creates OpenAI client with custom base_url + - LangChain: `langchain-openai.ChatOpenAI` with custom base_url for agent framework + +- OpenAI (fallback) - Cloud LLM service + - SDK/Client: `openai` Python package + - Auth: `OPENAI_API_KEY` + - Models: `gpt-4o-mini` (embeddings and chat), `gpt-5-mini` (fallback for agents) + - Implementation: Automatic fallback when `LLAMA_SERVER_URL` not configured + - Used for: Chat completions, embeddings via ChromaDB embedding function + +**Web Search:** +- Tavily - Web search API for real-time information retrieval + - SDK/Client: `tavily-python` (v0.7.17+) + - Auth: `TAVILY_API_KEY` + - Implementation: `blueprints/conversation/agents.py` - `AsyncTavilyClient` + - Used in: LangChain agent tool for web searches + +**Budget Tracking:** +- YNAB (You Need A Budget) - Personal finance and budget management + - SDK/Client: `ynab` Python package (v1.3.0+) + - Auth: `YNAB_ACCESS_TOKEN` (Personal Access Token from YNAB settings) + - Budget Selection: `YNAB_BUDGET_ID` (optional, auto-detects first budget if not set) + - Implementation: `utils/ynab_service.py` - `YNABService` class + - Features: Budget summary, transaction search, category spending, spending insights + - API Endpoints: Budgets API, Transactions API, Months API, Categories API + - Used in: LangChain agent tools for financial queries + +**Meal Planning:** +- Mealie - Self-hosted meal planning and recipe management + - SDK/Client: Custom async client using `httpx` in `utils/mealie_service.py` + - Auth: `MEALIE_API_TOKEN` (Bearer token) + - Base URL: `MEALIE_BASE_URL` (e.g., `http://192.168.1.5:9000`) + - Implementation: `MealieService` class with async methods + - Features: Shopping lists, meal plans, today's meals, recipe details, CRUD operations on meal plans + - API Endpoints: `/api/households/shopping/*`, `/api/households/mealplans/*`, `/api/households/self/recipes/*` + - Used in: LangChain agent tools for meal planning queries + +**Photo Management (referenced but not actively used):** +- Immich - Photo library management + - Connection: `IMMICH_URL`, `IMMICH_API_KEY` + - Search: `SEARCH_QUERY`, `DOWNLOAD_DIR` + - Note: Environment variables defined but service implementation not found in current code + +## Data Storage + +**Databases:** +- PostgreSQL 16 + - Connection: `DATABASE_URL` (format: `postgres://user:pass@host:port/db`) + - Container: `postgres:16-alpine` image + - Client: Tortoise ORM (async ORM with Pydantic models) + - Models: User management, conversations, messages, OIDC state + - Migrations: Aerich tool in `migrations/` directory + - Volume: `postgres_data` persistent volume + +**Vector Store:** +- ChromaDB + - Type: Embedded vector database (PersistentClient) + - Path: `CHROMADB_PATH` (Docker: `/app/data/chromadb`, local: `./data/chromadb`) + - Collections: `simba_docs2` (main RAG documents), `feline_vet_lookup` (veterinary knowledge) + - Embedding Function: OpenAI embeddings via `chromadb.utils.embedding_functions.openai_embedding_function` + - Integration: LangChain via `langchain-chroma` for vector store queries + - Volume: `chromadb_data` persistent volume + +**File Storage:** +- Local filesystem only + - PDF downloads: Temporary files for processing + - Image conversion: Temporary files from PDF to image conversion + - Database tracking: `database/visited.db` SQLite for tracking indexed documents + +**Caching:** +- None - No explicit caching layer configured + +## Authentication & Identity + +**Auth Provider:** +- Authelia (OIDC) - Self-hosted authentication and authorization server + - Implementation: Custom OIDC client in `config/oidc_config.py` + - Discovery: `.well-known/openid-configuration` endpoint (configurable via `OIDC_USE_DISCOVERY`) + - Environment Variables: + - `OIDC_ISSUER` (e.g., `https://auth.example.com`) + - `OIDC_CLIENT_ID` (e.g., `simbarag`) + - `OIDC_CLIENT_SECRET` + - `OIDC_REDIRECT_URI` (default: `http://localhost:8080/`) + - Manual endpoint override: `OIDC_AUTHORIZATION_ENDPOINT`, `OIDC_TOKEN_ENDPOINT`, `OIDC_USERINFO_ENDPOINT`, `OIDC_JWKS_URI` + - Token Verification: JWT verification using `authlib.jose.jwt` with JWKS + - LDAP Integration: LLDAP groups for RBAC (checks `lldap_admin` group for admin permissions) + +**Session Management:** +- JWT tokens via `quart-jwt-extended` + - Secret: `JWT_SECRET_KEY` environment variable + - Storage: Frontend localStorage + - Decorators: `@jwt_refresh_token_required` for protected endpoints, `@admin_required` for admin routes + +## Monitoring & Observability + +**Error Tracking:** +- None - No external error tracking service configured + +**Logs:** +- Standard Python logging to stdout/stderr + - Format: `%(asctime)s - %(name)s - %(levelname)s - %(message)s` + - Level: INFO (configurable via logging module) + - Special loggers: `utils.ynab_service`, `utils.mealie_service`, `blueprints.conversation.agents` set to INFO level + - Docker: Logs accessible via `docker compose logs` + +**Metrics:** +- None - No metrics collection configured + +## CI/CD & Deployment + +**Hosting:** +- Docker Compose - Self-hosted container deployment + - Production: `docker-compose.yml` + - Development: `docker-compose.dev.yml` with volume mounts for hot reload + - Image: `torrtle/simbarag:latest` (custom build) + +**CI Pipeline:** +- None - No automated CI/CD configured + - Manual builds: `docker compose build raggr` + - Manual deploys: `docker compose up -d` + +**Container Registry:** +- Docker Hub (inferred from image name `torrtle/simbarag:latest`) + +## Environment Configuration + +**Required env vars:** +- `DATABASE_URL` - PostgreSQL connection string +- `JWT_SECRET_KEY` - JWT token signing key +- `PAPERLESS_TOKEN` - Paperless-NGX API token +- `BASE_URL` - Paperless-NGX instance URL + +**LLM configuration (choose one):** +- `LLAMA_SERVER_URL` + `LLAMA_MODEL_NAME` - Local llama-server (primary) +- `OPENAI_API_KEY` - OpenAI API (fallback) + +**Optional integrations:** +- `YNAB_ACCESS_TOKEN`, `YNAB_BUDGET_ID` - YNAB budget integration +- `MEALIE_BASE_URL`, `MEALIE_API_TOKEN` - Mealie meal planning +- `TAVILY_API_KEY` - Web search capability +- `IMMICH_URL`, `IMMICH_API_KEY`, `SEARCH_QUERY`, `DOWNLOAD_DIR` - Immich photos + +**OIDC authentication:** +- `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_REDIRECT_URI` +- `OIDC_USE_DISCOVERY` - Enable/disable OIDC discovery (default: true) + +**Secrets location:** +- `.env` file in project root (not committed to git) +- Docker Compose reads from `.env` file automatically +- Example file: `.env.example` with placeholder values + +## Webhooks & Callbacks + +**Incoming:** +- `/api/user/oidc/callback` - OIDC authorization code callback from Authelia + - Method: GET with `code` and `state` query parameters + - Flow: Authorization code → token exchange → user info → JWT creation + +**Outgoing:** +- None - No webhook subscriptions to external services + +--- + +*Integration audit: 2026-02-04* diff --git a/.planning/codebase/STACK.md b/.planning/codebase/STACK.md new file mode 100644 index 0000000..d780667 --- /dev/null +++ b/.planning/codebase/STACK.md @@ -0,0 +1,107 @@ +# Technology Stack + +**Analysis Date:** 2026-02-04 + +## Languages + +**Primary:** +- Python 3.13 - Backend application, RAG logic, API endpoints, utilities + +**Secondary:** +- TypeScript 5.9.2 - Frontend React application with type safety +- JavaScript - Build tooling and configuration + +## Runtime + +**Environment:** +- Python 3.13-slim (Docker container) +- Node.js 20.x (for frontend builds) + +**Package Manager:** +- uv - Python dependency management (Astral's fast installer) +- Yarn - Frontend package management +- Lockfiles: `uv.lock` and `raggr-frontend/yarn.lock` present + +## Frameworks + +**Core:** +- Quart 0.20.0 - Async Python web framework (Flask-like API with async support) +- React 19.1.1 - Frontend UI library +- Rsbuild 1.5.6 - Modern frontend build tool (Rspack-based) + +**Testing:** +- Not explicitly configured in dependencies + +**Build/Dev:** +- Rsbuild 1.5.6 - Frontend bundler with React plugin +- Black 25.9.0 - Python code formatter +- Biome 2.3.10 - Frontend linter and formatter (replaces ESLint/Prettier) +- Pre-commit 4.3.0 - Git hooks for code quality +- Docker Compose - Container orchestration (dev and prod configurations) + +## Key Dependencies + +**Critical:** +- `chromadb>=1.1.0` - Vector database for document embeddings and similarity search +- `openai>=2.0.1` - LLM client library (used for both OpenAI and llama-server via OpenAI-compatible API) +- `langchain>=1.2.0` - LLM application framework with agent and tool support +- `langchain-openai>=1.1.6` - LangChain integration for OpenAI/llama-server +- `langchain-chroma>=1.0.0` - LangChain integration for ChromaDB +- `tortoise-orm>=0.25.1` - Async ORM for PostgreSQL database operations +- `quart-jwt-extended>=0.1.0` - JWT authentication for Quart +- `authlib>=1.3.0` - OIDC/OAuth2 client library + +**Infrastructure:** +- `httpx>=0.28.1` - Async HTTP client for API integrations +- `asyncpg>=0.30.0` - PostgreSQL async driver +- `aerich>=0.8.0` - Database migration tool for Tortoise ORM +- `pymupdf>=1.24.0` - PDF processing (fitz) +- `pillow>=10.0.0` - Image processing +- `pillow-heif>=1.1.1` - HEIF/HEIC image format support +- `bcrypt>=5.0.0` - Password hashing +- `python-dotenv>=1.0.0` - Environment variable management + +**External Service Integrations:** +- `tavily-python>=0.7.17` - Web search API client +- `ynab>=1.3.0` - YNAB (budgeting app) API client +- `axios^1.12.2` - Frontend HTTP client +- `react-markdown^10.1.0` - Markdown rendering in React +- `marked^16.3.0` - Markdown parser + +## Configuration + +**Environment:** +- `.env` files for environment-specific configuration +- Required vars: `DATABASE_URL`, `JWT_SECRET_KEY`, `PAPERLESS_TOKEN`, `BASE_URL` +- Optional LLM: `LLAMA_SERVER_URL`, `LLAMA_MODEL_NAME` (primary) or `OPENAI_API_KEY` (fallback) +- Optional integrations: `YNAB_ACCESS_TOKEN`, `MEALIE_BASE_URL`, `MEALIE_API_TOKEN`, `TAVILY_API_KEY` +- OIDC auth: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_REDIRECT_URI` +- ChromaDB: `CHROMADB_PATH` (defaults to `/app/data/chromadb` in Docker) + +**Build:** +- `pyproject.toml` - Python project metadata and dependencies +- `rsbuild.config.ts` - Frontend build configuration +- `tsconfig.json` - TypeScript compiler configuration +- `Dockerfile` - Multi-stage build (Python + Node.js) +- `docker-compose.yml` - Production container setup +- `docker-compose.dev.yml` - Development with hot reload +- `aerich_config.py` - Database migration configuration +- `.pre-commit-config.yaml` - Git hooks for code quality + +## Platform Requirements + +**Development:** +- Python 3.13+ +- Node.js 20.x +- PostgreSQL 16+ (via Docker or local) +- Docker and Docker Compose (recommended) + +**Production:** +- Docker environment +- PostgreSQL 16-alpine container +- Persistent volumes for ChromaDB and PostgreSQL data +- Network access to external APIs (Paperless-NGX, YNAB, Mealie, Tavily, OpenAI, llama-server) + +--- + +*Stack analysis: 2026-02-04* diff --git a/.planning/codebase/STRUCTURE.md b/.planning/codebase/STRUCTURE.md new file mode 100644 index 0000000..afcac0d --- /dev/null +++ b/.planning/codebase/STRUCTURE.md @@ -0,0 +1,237 @@ +# Codebase Structure + +**Analysis Date:** 2026-02-04 + +## Directory Layout + +``` +raggr/ +├── blueprints/ # API route modules (Quart blueprints) +│ ├── conversation/ # Chat conversation endpoints and logic +│ ├── rag/ # Document indexing and retrieval endpoints +│ └── users/ # Authentication and user management +├── config/ # Configuration modules +├── utils/ # Reusable service clients and utilities +├── scripts/ # Administrative CLI scripts +├── migrations/ # Database schema migrations (Aerich) +├── raggr-frontend/ # React SPA frontend +│ ├── src/ +│ │ ├── components/ # React UI components +│ │ ├── api/ # Frontend API service clients +│ │ ├── contexts/ # React contexts (Auth) +│ │ └── assets/ # Static images +│ └── dist/ # Built frontend (served by backend) +├── chroma_db/ # ChromaDB persistent vector store +├── chromadb/ # Alternate ChromaDB path (legacy) +├── docs/ # Documentation files +├── app.py # Quart application entry point +├── main.py # RAG logic and CLI entry point +├── llm.py # LLM client with provider fallback +└── aerich_config.py # Database migration configuration +``` + +## Directory Purposes + +**blueprints/** +- Purpose: API route organization using Quart blueprint pattern +- Contains: Python packages with `__init__.py` (routes), `models.py` (ORM), `logic.py` (business logic) +- Key files: `conversation/__init__.py` (chat API), `rag/__init__.py` (indexing API), `users/__init__.py` (auth API) + +**blueprints/conversation/** +- Purpose: Chat conversation management +- Contains: Streaming chat endpoints, message persistence, conversation CRUD, agent orchestration +- Key files: `__init__.py` (endpoints), `agents.py` (LangChain agent + tools), `logic.py` (conversation operations), `models.py` (Conversation, ConversationMessage) + +**blueprints/rag/** +- Purpose: Document indexing and vector search +- Contains: Admin-only indexing endpoints, vector store operations, Paperless-NGX integration +- Key files: `__init__.py` (endpoints), `logic.py` (indexing + query), `fetchers.py` (Paperless client) + +**blueprints/users/** +- Purpose: User authentication and authorization +- Contains: OIDC login flow, JWT token management, RBAC decorators +- Key files: `__init__.py` (auth endpoints), `models.py` (User model), `decorators.py` (@admin_required), `oidc_service.py` (user provisioning) + +**config/** +- Purpose: Configuration modules for external integrations +- Contains: OIDC configuration with JWKS verification +- Key files: `oidc_config.py` + +**utils/** +- Purpose: Reusable utilities and external service clients +- Contains: Chunking, cleaning, API clients for YNAB/Mealie/Paperless +- Key files: `chunker.py`, `cleaner.py`, `ynab_service.py`, `mealie_service.py`, `request.py` (Paperless client), `image_process.py` + +**scripts/** +- Purpose: Administrative and maintenance CLI tools +- Contains: User management, statistics, vector store inspection +- Key files: `add_user.py`, `user_message_stats.py`, `manage_vectorstore.py`, `inspect_vector_store.py`, `query.py` + +**migrations/** +- Purpose: Database schema version control (Aerich/Tortoise ORM) +- Contains: SQL migration files generated by `aerich migrate` +- Generated: Yes +- Committed: Yes + +**raggr-frontend/** +- Purpose: React single-page application +- Contains: React 19 components, Rsbuild bundler config, Tailwind CSS, TypeScript +- Key files: `src/App.tsx` (root), `src/index.tsx` (entry), `src/components/ChatScreen.tsx` (main UI) + +**raggr-frontend/src/components/** +- Purpose: React UI components +- Contains: Chat interface, login, conversation list, message bubbles +- Key files: `ChatScreen.tsx`, `LoginScreen.tsx`, `ConversationList.tsx`, `AnswerBubble.tsx`, `QuestionBubble.tsx`, `MessageInput.tsx` + +**raggr-frontend/src/api/** +- Purpose: Frontend service layer for API communication +- Contains: TypeScript service clients with axios/fetch +- Key files: `conversationService.ts` (SSE streaming), `userService.ts`, `oidcService.ts` + +**raggr-frontend/src/contexts/** +- Purpose: React contexts for global state +- Contains: Authentication context +- Key files: `AuthContext.tsx` + +**raggr-frontend/dist/** +- Purpose: Built frontend assets served by backend +- Contains: Bundled JS, CSS, HTML +- Generated: Yes (by Rsbuild) +- Committed: No + +**chroma_db/** and **chromadb/** +- Purpose: ChromaDB persistent vector store data +- Contains: SQLite database files and vector indices +- Generated: Yes (at runtime) +- Committed: No + +**docs/** +- Purpose: Project documentation +- Contains: Integration documentation, technical specs +- Key files: `ynab_integration/` + +## Key File Locations + +**Entry Points:** +- `app.py`: Web server entry point (Quart application) +- `main.py`: CLI entry point for RAG operations +- `raggr-frontend/src/index.tsx`: Frontend entry point + +**Configuration:** +- `.env`: Environment variables (not committed, see `.env.example`) +- `aerich_config.py`: Database migration configuration +- `config/oidc_config.py`: OIDC authentication configuration +- `raggr-frontend/rsbuild.config.ts`: Frontend build configuration + +**Core Logic:** +- `blueprints/conversation/agents.py`: LangChain agent with tool definitions +- `blueprints/rag/logic.py`: Vector store indexing and query operations +- `main.py`: Original RAG implementation (legacy, partially superseded by blueprints) +- `llm.py`: LLM client abstraction with fallback logic + +**Testing:** +- Not detected (no test files found) + +## Naming Conventions + +**Files:** +- Snake_case for Python modules: `ynab_service.py`, `oidc_config.py` +- PascalCase for React components: `ChatScreen.tsx`, `AnswerBubble.tsx` +- Lowercase for config files: `docker-compose.yml`, `pyproject.toml` + +**Directories:** +- Lowercase with underscores for Python packages: `blueprints/conversation/`, `utils/` +- Kebab-case for frontend: `raggr-frontend/` + +**Python Classes:** +- PascalCase: `User`, `Conversation`, `ConversationMessage`, `LLMClient`, `YNABService` + +**Python Functions:** +- Snake_case: `get_conversation_by_id`, `query_vector_store`, `add_message_to_conversation` + +**React Components:** +- PascalCase: `ChatScreen`, `LoginScreen`, `ConversationList` + +**API Routes:** +- Kebab-case: `/api/conversation/query`, `/api/user/oidc/callback` + +**Environment Variables:** +- SCREAMING_SNAKE_CASE: `DATABASE_URL`, `YNAB_ACCESS_TOKEN`, `LLAMA_SERVER_URL` + +## Where to Add New Code + +**New API Endpoint:** +- Primary code: Create or extend blueprint in `blueprints//__init__.py` +- Business logic: Add functions to `blueprints//logic.py` +- Database models: Add to `blueprints//models.py` +- Tests: Not established (no test directory exists) + +**New LangChain Tool:** +- Implementation: Add `@tool` decorated function in `blueprints/conversation/agents.py` +- Service client: If calling external API, create client in `utils/_service.py` +- Add to tools list: Append to `tools` list at bottom of `agents.py` (line 709+) + +**New External Service Integration:** +- Service client: Create `utils/_service.py` with async methods +- Tool wrapper: Add tool function in `blueprints/conversation/agents.py` +- Configuration: Add env vars to `.env.example` + +**New React Component:** +- Component file: `raggr-frontend/src/components/.tsx` +- API service: If needs backend, add methods to `raggr-frontend/src/api/Service.ts` +- Import in: `raggr-frontend/src/App.tsx` or parent component + +**New Database Table:** +- Model: Add Tortoise model to `blueprints//models.py` +- Migration: Run `docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name ` +- Apply: Run `docker compose -f docker-compose.dev.yml exec raggr aerich upgrade` (or restart container) + +**Utilities:** +- Shared helpers: `utils/.py` for Python utilities +- Frontend utilities: `raggr-frontend/src/utils/` (not currently used, would need creation) + +## Special Directories + +**.git/** +- Purpose: Git version control metadata +- Generated: Yes +- Committed: No (automatically handled by git) + +**.venv/** +- Purpose: Python virtual environment +- Generated: Yes (local dev only) +- Committed: No + +**node_modules/** +- Purpose: NPM dependencies for frontend +- Generated: Yes (npm/yarn install) +- Committed: No + +**__pycache__/** +- Purpose: Python bytecode cache +- Generated: Yes (Python runtime) +- Committed: No + +**.planning/** +- Purpose: GSD (Get Stuff Done) codebase documentation +- Generated: Yes (by GSD commands) +- Committed: Yes (intended for project documentation) + +**.claude/** +- Purpose: Claude Code session data +- Generated: Yes +- Committed: No + +**.ruff_cache/** +- Purpose: Ruff linter cache +- Generated: Yes +- Committed: No + +**.ropeproject/** +- Purpose: Rope Python refactoring library cache +- Generated: Yes +- Committed: No + +--- + +*Structure analysis: 2026-02-04* diff --git a/.planning/codebase/TESTING.md b/.planning/codebase/TESTING.md new file mode 100644 index 0000000..aa75c24 --- /dev/null +++ b/.planning/codebase/TESTING.md @@ -0,0 +1,290 @@ +# Testing Patterns + +**Analysis Date:** 2026-02-04 + +## Test Framework + +**Runner:** +- None detected +- No pytest.ini, pytest.toml, jest.config.js, or vitest.config.ts found +- No test files in codebase (no `test_*.py`, `*_test.py`, `*.test.ts`, `*.spec.ts`) + +**Assertion Library:** +- Not applicable (no tests present) + +**Run Commands:** +```bash +# No test commands configured in package.json or standard Python test runners +``` + +## Test File Organization + +**Location:** +- No test files detected in the project + +**Naming:** +- Not established (no existing test files to analyze) + +**Structure:** +``` +# No test directory structure present +``` + +## Test Structure + +**Suite Organization:** +Not applicable - no tests exist in the codebase. + +**Expected Pattern (based on project structure):** +```python +# Python tests would likely use pytest with async support +import pytest +from quart import Quart + +@pytest.mark.asyncio +async def test_endpoint(): + # Test Quart async endpoints + pass +``` + +**TypeScript Pattern (if implemented):** +```typescript +// Would likely use Vitest (matches Rsbuild ecosystem) +import { describe, it, expect } from 'vitest'; + +describe('conversationService', () => { + it('should fetch conversations', async () => { + // Test API service methods + }); +}); +``` + +## Mocking + +**Framework:** +- Not established (no tests present) + +**Likely Approach:** +- Python: `pytest-mock` or `unittest.mock` for services/API calls +- TypeScript: Vitest mocking utilities + +**What to Mock:** +- External API calls (YNAB, Mealie, Paperless-NGX, Tavily) +- LLM interactions (OpenAI/llama-server) +- Database queries (Tortoise ORM) +- Authentication/JWT verification + +**What NOT to Mock:** +- Business logic functions (these should be tested directly) +- Data transformations +- Utility functions without side effects + +## Fixtures and Factories + +**Test Data:** +Not established - would need fixtures for: +- User objects with various authentication states +- Conversation and Message objects +- Mock YNAB/Mealie API responses +- Mock ChromaDB query results + +**Expected Pattern:** +```python +# Python fixtures with pytest +@pytest.fixture +async def test_user(): + """Create a test user.""" + user = await User.create( + username="testuser", + email="test@example.com", + auth_provider="local" + ) + yield user + await user.delete() + +@pytest.fixture +def mock_ynab_response(): + """Mock YNAB API budget response.""" + return { + "budget_name": "Test Budget", + "to_be_budgeted": 100.00, + "total_budgeted": 2000.00, + } +``` + +## Coverage + +**Requirements:** +- No coverage requirements configured +- No `.coveragerc` or coverage configuration in `pyproject.toml` + +**Current State:** +- **0% test coverage** (no tests exist) + +**View Coverage:** +```bash +# Would use pytest-cov for Python +pytest --cov=. --cov-report=html + +# Would use Vitest coverage for TypeScript +npx vitest --coverage +``` + +## Test Types + +**Unit Tests:** +- Not present +- Should test: Service methods, utility functions, data transformations, business logic + +**Integration Tests:** +- Not present +- Should test: API endpoints, database operations, authentication flows, external service integrations + +**E2E Tests:** +- Not present +- Could use: Playwright or Cypress for frontend testing + +## Common Patterns + +**Async Testing:** +Expected pattern for Quart/async Python: +```python +import pytest +from httpx import AsyncClient +from app import app + +@pytest.mark.asyncio +async def test_query_endpoint(): + async with AsyncClient(app=app, base_url="http://test") as client: + response = await client.post( + "/api/conversation/query", + json={"query": "test", "conversation_id": "uuid"} + ) + assert response.status_code == 200 +``` + +**Error Testing:** +Expected pattern: +```python +@pytest.mark.asyncio +async def test_unauthorized_access(): + async with AsyncClient(app=app, base_url="http://test") as client: + response = await client.post("/api/conversation/query") + assert response.status_code == 401 + assert "error" in response.json() +``` + +## Testing Gaps + +**Critical Areas Without Tests:** + +1. **Authentication & Authorization:** + - OIDC flow (`blueprints/users/__init__.py` - 188 lines) + - JWT token refresh + - Admin authorization decorator + - PKCE verification + +2. **Core RAG Functionality:** + - Document indexing (`main.py` - 274 lines) + - Vector store queries (`blueprints/rag/logic.py`) + - LLM agent tools (`blueprints/conversation/agents.py` - 733 lines) + - Query classification + +3. **External Service Integrations:** + - YNAB API client (`utils/ynab_service.py` - 576 lines) + - Mealie API client (`utils/mealie_service.py` - 477 lines) + - Paperless-NGX API client (`utils/request.py`) + - Tavily web search + +4. **Streaming Responses:** + - Server-Sent Events in `/api/conversation/query` + - Frontend SSE parsing (`conversationService.sendQueryStream()`) + +5. **Database Operations:** + - Conversation creation and retrieval + - Message persistence + - User CRUD operations + +6. **Frontend Components:** + - ChatScreen streaming state (`ChatScreen.tsx` - 386 lines) + - Message bubbles rendering + - Authentication context + +## Recommended Testing Strategy + +**Phase 1: Critical Path Tests** +- Authentication endpoints (login, callback, token refresh) +- Conversation query endpoint (non-streaming) +- User creation and retrieval +- Basic YNAB/Mealie service methods + +**Phase 2: Integration Tests** +- Full OIDC authentication flow +- Conversation with messages persistence +- RAG document indexing and retrieval +- External API error handling + +**Phase 3: Frontend Tests** +- Component rendering tests +- API service method tests +- Streaming response handling +- Authentication state management + +**Phase 4: E2E Tests** +- Complete user journey (login → query → response) +- Conversation management +- Admin operations + +## Testing Dependencies to Add + +**Python:** +```toml +# Add to pyproject.toml [tool.poetry.group.dev.dependencies] or requirements-dev.txt +pytest = "^7.0" +pytest-asyncio = "^0.21" +pytest-cov = "^4.0" +pytest-mock = "^3.10" +httpx = "^0.24" # For testing async HTTP +``` + +**TypeScript:** +```json +// Add to raggr-frontend/package.json devDependencies +"@vitest/ui": "^1.0.0", +"vitest": "^1.0.0", +"@testing-library/react": "^14.0.0", +"@testing-library/jest-dom": "^6.0.0" +``` + +## Testing Best Practices (Not Yet Implemented) + +**Database Tests:** +- Use separate test database +- Reset database state between tests +- Use Aerich to apply migrations in test environment + +**Async Tests:** +- Mark all async tests with `@pytest.mark.asyncio` +- Use `AsyncClient` for Quart endpoint testing +- Properly await all async operations + +**Mocking External Services:** +- Mock all HTTP calls to external APIs +- Use `httpx.MockTransport` or `responses` library +- Return realistic mock data based on actual API responses + +**Frontend Testing:** +- Mock API services in component tests +- Test loading/error states +- Test user interactions (clicks, form submissions) +- Verify SSE stream handling + +--- + +*Testing analysis: 2026-02-04* + +**CRITICAL NOTE:** This codebase currently has **no automated tests**. All functionality relies on manual testing. Implementing a test suite should be a high priority, especially for: +- Authentication flows (security-critical) +- External API integrations (reliability-critical) +- Database operations (data integrity-critical) +- Streaming responses (complexity-critical)