ryan/simbarag
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: map existing codebase

Browse Source 
- STACK.md - Technologies and dependencies
- ARCHITECTURE.md - System design and patterns
- STRUCTURE.md - Directory layout
- CONVENTIONS.md - Code style and patterns
- TESTING.md - Test structure
- INTEGRATIONS.md - External services
- CONCERNS.md - Technical debt and issues
This commit is contained in:
Ryan Chen
2026-02-04 16:53:27 -05:00
parent 6ae36b51a0
commit b0b02d24f4
7 changed files with 1598 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

184
.planning/codebase/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,184 @@
# Architecture
**Analysis Date:** 2026-02-04
## Pattern Overview
**Overall:** RAG (Retrieval-Augmented Generation) system with multi-agent conversational AI architecture
**Key Characteristics:**
- RAG pattern with vector database for document retrieval
- LangChain agent-based orchestration with tool calling
- Blueprint-based API organization (Quart framework)
- Asynchronous request handling throughout
- OIDC authentication with RBAC via LDAP groups
- Streaming SSE responses for real-time chat
## Layers
**API Layer (Quart Blueprints):**
- Purpose: HTTP request handling and route organization
- Location: `blueprints/*/`
- Contains: Blueprint definitions, route handlers, request/response serialization
- Depends on: Logic layer, models, JWT middleware
- Used by: Frontend (React SPA), external clients
**Logic Layer:**
- Purpose: Business logic and domain operations
- Location: `blueprints/*/logic.py`, `blueprints/*/agents.py`, `main.py`
- Contains: Conversation management, RAG indexing, agent orchestration, tool execution
- Depends on: Models, external services, LLM clients
- Used by: API layer
**Model Layer (Tortoise ORM):**
- Purpose: Database schema and data access
- Location: `blueprints/*/models.py`
- Contains: ORM model definitions, Pydantic serializers, database relationships
- Depends on: PostgreSQL database
- Used by: Logic layer, API layer
**Integration Layer:**
- Purpose: External service communication
- Location: `utils/`, `config/`
- Contains: Service clients (YNAB, Mealie, Paperless-NGX, OIDC)
- Depends on: External APIs
- Used by: Logic layer, tools
**Tool Layer (LangChain Tools):**
- Purpose: Agent-callable functions for extended capabilities
- Location: `blueprints/conversation/agents.py`
- Contains: `@tool` decorated functions for document search, web search, YNAB, Mealie
- Depends on: Integration layer, RAG logic
- Used by: LangChain agent
**Frontend (React SPA):**
- Purpose: User interface
- Location: `raggr-frontend/`
- Contains: React components, API service clients, authentication context
- Depends on: Backend API endpoints
- Used by: End users
## Data Flow
**Chat Query Flow:**
1. User submits query in frontend (`raggr-frontend/src/components/ChatScreen.tsx`)
2. Frontend calls `/api/conversation/query` with SSE streaming (`raggr-frontend/src/api/conversationService.ts`)
3. API endpoint validates JWT, fetches user and conversation (`blueprints/conversation/__init__.py`)
4. User message saved to database via Tortoise ORM (`blueprints/conversation/models.py`)
5. Recent conversation history (last 10 messages) loaded and formatted
6. LangChain agent invoked with messages payload (`blueprints/conversation/agents.py`)
7. Agent decides which tools to call based on query (simba_search, ynab_*, mealie_*, web_search)
8. Tools execute: RAG query (`blueprints/rag/logic.py`), API calls (`utils/*.py`)
9. LLM generates response using tool results
10. Response streamed back via SSE events (status updates, content chunks)
11. Complete response saved to database
12. Frontend renders streaming response in real-time
**RAG Document Flow:**
1. Admin triggers indexing via `/api/rag/index` or `/api/rag/reindex`
2. RAG logic fetches documents from Paperless-NGX (`blueprints/rag/fetchers.py`)
3. Documents chunked using LangChain text splitter (1000 chars, 200 overlap)
4. Embeddings generated using OpenAI embedding model (text-embedding-3-small)
5. Vectors stored in ChromaDB persistent collection (`chroma_db/`)
6. Query time: embeddings generated for query, similarity search retrieves top 2 docs
7. Documents serialized and passed to LLM as context
**State Management:**
- Conversation state: PostgreSQL via Tortoise ORM
- Vector embeddings: ChromaDB persistent storage
- User sessions: JWT tokens in frontend localStorage
- Authentication: OIDC state in-memory (production should use Redis)
## Key Abstractions
**Conversation:**
- Purpose: Represents a chat thread with message history
- Examples: `blueprints/conversation/models.py`
- Pattern: Aggregate root with message collection, foreign key to User
**ConversationMessage:**
- Purpose: Individual message in conversation (user or assistant)
- Examples: `blueprints/conversation/models.py`
- Pattern: Entity with enum speaker type, foreign key to Conversation
**User:**
- Purpose: Authenticated user with OIDC or local credentials
- Examples: `blueprints/users/models.py`
- Pattern: Entity with bcrypt password hashing, LDAP group membership, admin check method
**LangChain Agent:**
- Purpose: Orchestrates LLM calls with tool selection
- Examples: `blueprints/conversation/agents.py` (main_agent)
- Pattern: ReAct agent pattern with function calling via OpenAI-compatible API
**Tool Functions:**
- Purpose: Discrete capabilities callable by the agent
- Examples: `simba_search`, `ynab_budget_summary`, `mealie_shopping_list` in `blueprints/conversation/agents.py`
- Pattern: Decorated functions with docstrings that become tool descriptions
**LLMClient:**
- Purpose: Abstraction over LLM providers with fallback
- Examples: `llm.py`, `blueprints/conversation/agents.py`
- Pattern: Primary llama-server with OpenAI fallback, OpenAI-compatible interface
**Service Clients:**
- Purpose: External API integration wrappers
- Examples: `utils/ynab_service.py`, `utils/mealie_service.py`, `utils/request.py`
- Pattern: Class-based clients with async methods, relative date parsing
## Entry Points
**Web Application:**
- Location: `app.py`
- Triggers: `python app.py` or Docker container startup
- Responsibilities: Initialize Quart app, register blueprints, configure Tortoise ORM, serve React frontend
**CLI Indexing:**
- Location: `main.py` (when run as script)
- Triggers: `python main.py --reindex` or `--query <text>`
- Responsibilities: Document indexing, direct RAG queries without API
**Database Migrations:**
- Location: `aerich_config.py`
- Triggers: `aerich migrate`, `aerich upgrade`
- Responsibilities: Schema migration generation and application
**Admin Scripts:**
- Location: `scripts/add_user.py`, `scripts/user_message_stats.py`, `scripts/manage_vectorstore.py`
- Triggers: Manual execution
- Responsibilities: User management, analytics, vector store inspection
**React Frontend:**
- Location: `raggr-frontend/src/index.tsx`
- Triggers: Bundle served at `/` by backend
- Responsibilities: Initialize React app, authentication context, routing
## Error Handling
**Strategy:** Try-catch with logging at service boundaries, HTTP status codes for client errors
**Patterns:**
- API routes: Return JSON error responses with appropriate HTTP status codes (400, 401, 403, 500)
- Example: `blueprints/rag/__init__.py` line 26-27
- Async operations: Try-except blocks with logger.error for traceability
- Example: `blueprints/conversation/agents.py` line 142-145 (YNAB tool error handling)
- JWT validation: Decorator-based authentication with 401 response on failure
- Example: `@jwt_refresh_token_required` in all protected routes
- Frontend: Error callbacks in streaming service, redirect to login on session expiry
- Example: `raggr-frontend/src/components/ChatScreen.tsx` line 234-237
- Agent tool failures: Return error string to agent for recovery or user messaging
- Example: `blueprints/conversation/agents.py` line 384-385
## Cross-Cutting Concerns
**Logging:** Python logging module with INFO level, structured with logger names by module (utils.ynab_service, blueprints.conversation.agents)
**Validation:** Pydantic models for serialization, Tortoise ORM field constraints, JWT token validation via quart-jwt-extended
**Authentication:** OIDC (Authelia) with PKCE flow → JWT tokens → RBAC via LDAP groups. Decorators: `@jwt_refresh_token_required` for auth, `@admin_required` for admin-only endpoints (`blueprints/users/decorators.py`)
---
*Architecture analysis: 2026-02-04*

265
.planning/codebase/CONCERNS.md Normal file
View File

@@ -0,0 +1,265 @@
# Codebase Concerns
**Analysis Date:** 2026-02-04
## Tech Debt
**Duplicate system prompts in streaming and non-streaming endpoints:**
- Issue: Large system prompt (112 lines) duplicated verbatim in two endpoints
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (lines 56-111 and 206-261)
- Impact: Changes to prompt must be made in two places, increasing maintenance burden and risk of inconsistency
- Fix approach: Extract system prompt to a constant or configuration file
**SQLite database for indexing tracking alongside PostgreSQL:**
- Issue: Uses SQLite (`database/visited.db`) to track indexed Paperless documents while main data is in PostgreSQL
- Files: `/Users/ryanchen/Programs/raggr/main.py` (lines 73, 212, 226), `/Users/ryanchen/Programs/raggr/scripts/index_immich.py` (line 33)
- Impact: Two database systems to manage, no transactions across databases, deployment complexity
- Fix approach: Migrate indexing tracking to PostgreSQL table using Tortoise ORM
**Broad exception catching throughout codebase:**
- Issue: 35+ instances of `except Exception as e` catching all exceptions indiscriminately
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (12 instances), `/Users/ryanchen/Programs/raggr/utils/ynab_service.py` (7 instances), `/Users/ryanchen/Programs/raggr/utils/mealie_service.py` (7 instances), `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 171), `/Users/ryanchen/Programs/raggr/blueprints/rag/__init__.py` (lines 26, 46)
- Impact: Masks programming errors, makes debugging difficult, catches system exceptions that shouldn't be caught
- Fix approach: Replace with specific exception types (ValueError, KeyError, HTTPException, etc.)
**Legacy main.py RAG logic not used by application:**
- Issue: `/Users/ryanchen/Programs/raggr/main.py` contains 275 lines of RAG logic including `consult_oracle()`, `classify_query()`, `consult_simba_oracle()` but app uses LangChain agents instead
- Files: `/Users/ryanchen/Programs/raggr/main.py`, `/Users/ryanchen/Programs/raggr/app.py` (imports `consult_simba_oracle` but endpoint is commented/unused)
- Impact: Dead code increases maintenance burden, confuses new developers about which code path is active
- Fix approach: Archive or remove unused code after verifying no production dependencies
**Environment variable typo in docker-compose:**
- Issue: Docker compose uses `TAVILIY_KEY` instead of `TAVILY_API_KEY`
- Files: `/Users/ryanchen/Programs/raggr/docker-compose.yml` (line 41), `/Users/ryanchen/Programs/raggr/docker-compose.dev.yml` (line 44)
- Impact: Tavily web search won't work in production Docker deployment
- Fix approach: Standardize on `TAVILY_API_KEY` throughout
**Hardcoded OpenAI model in conversation rename logic:**
- Issue: Uses `gpt-4o-mini` without environment variable configuration
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/logic.py` (line 72)
- Impact: Cannot switch models, will fail if OpenAI key not configured even when using local LLM
- Fix approach: Make model configurable via environment variable, use same fallback pattern as main agent
**Debug mode enabled in production app entry:**
- Issue: `debug=True` hardcoded in app.run()
- Files: `/Users/ryanchen/Programs/raggr/app.py` (line 165)
- Impact: Exposes stack traces and sensitive information if run directly (mitigated by Docker CMD using startup.sh)
- Fix approach: Use environment variable for debug flag
## Known Bugs
**Empty returns in PDF cleaner error handling:**
- Issue: Error handlers return None or empty lists without logging context
- Files: `/Users/ryanchen/Programs/raggr/utils/cleaner.py` (lines 58, 74, 81)
- Symptoms: Silent failures during PDF processing, no indication why document wasn't indexed
- Trigger: PDF processing errors (malformed PDFs, image conversion failures)
- Workaround: Check logs at DEBUG level, manually test PDF processing
**Console debug statements left in production code:**
- Issue: print() statements instead of logging in multiple locations
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (lines 109-113), `/Users/ryanchen/Programs/raggr/blueprints/conversation/logic.py` (line 20), `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 311), `/Users/ryanchen/Programs/raggr/raggr-frontend/src/components/ChatScreen.tsx` (lines 99-100, 132-133)
- Symptoms: Unstructured output mixed with proper logs, no log levels
- Fix approach: Replace with structured logging
**Conversation name timestamp method incorrect:**
- Issue: Uses `.timestamp` property instead of `.timestamp()` method
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 330)
- Symptoms: Conversation name will be method reference string instead of timestamp
- Fix approach: Change to `datetime.datetime.now().timestamp()`
## Security Considerations
**JWT secret key has weak default:**
- Risk: Default JWT_SECRET_KEY is "SECRET_KEY" if environment variable not set
- Files: `/Users/ryanchen/Programs/raggr/app.py` (line 39)
- Current mitigation: Documentation requires setting environment variable
- Recommendations: Fail fast on startup if JWT_SECRET_KEY is default value, generate random key on first run
**Hardcoded API key placeholder in llama-server configuration:**
- Risk: API key set to "not-needed" for local llama-server
- Files: `/Users/ryanchen/Programs/raggr/llm.py` (line 16), `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (line 28)
- Current mitigation: Only used for local trusted network LLM servers
- Recommendations: Document that llama-server should be on trusted network only, consider basic authentication
**No rate limiting on streaming endpoints:**
- Risk: Users can spawn unlimited concurrent streaming requests
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (line 29)
- Current mitigation: None
- Recommendations: Add per-user rate limiting, request queue, or connection limit
**Sensitive data in error messages:**
- Risk: Full exception details returned to client in tool error messages
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (lines 145, 219, 280, etc.)
- Current mitigation: Only exposed to authenticated users
- Recommendations: Sanitize error messages, return generic errors to client, log full details server-side
## Performance Bottlenecks
**Large conversation history loaded on every query:**
- Problem: Fetches all messages then slices to last 10 in memory
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (lines 38, 47-50, 188, 197-200)
- Cause: No database-level limit on message fetch
- Improvement path: Add database query limit, use `.order_by('-created_at').limit(10)` at query level
**Sequential document indexing:**
- Problem: Documents indexed one at a time in loop
- Files: `/Users/ryanchen/Programs/raggr/main.py` (lines 67-96)
- Cause: No parallel processing or batching
- Improvement path: Use asyncio.gather() for concurrent PDF processing, batch ChromaDB inserts
**No caching for YNAB API calls:**
- Problem: Every query makes fresh API calls even for recently accessed data
- Files: `/Users/ryanchen/Programs/raggr/utils/ynab_service.py` (all methods)
- Cause: No caching layer
- Improvement path: Add Redis/in-memory cache with TTL for budget data, cache budget summaries for 5-15 minutes
**Frontend loads all conversations on mount:**
- Problem: Fetches all conversations without pagination
- Files: `/Users/ryanchen/Programs/raggr/raggr-frontend/src/components/ChatScreen.tsx` (lines 89-104)
- Cause: No pagination in API or frontend
- Improvement path: Add cursor-based pagination, lazy load older conversations
**ChromaDB persistence path creates I/O bottleneck:**
- Problem: All embedding queries/inserts hit disk-backed SQLite database
- Files: `/Users/ryanchen/Programs/raggr/main.py` (line 19)
- Cause: Uses PersistentClient without in-memory optimization
- Improvement path: Consider ChromaDB server mode for production, add memory-backed cache layer
## Fragile Areas
**LangChain agent tool calling depends on exact model support:**
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py` (line 733)
- Why fragile: Comment says "Llama 3.1 supports native function calling" but not all local models do
- Test coverage: No automated tests for tool calling
- Safe modification: Always test with target model before deploying, add fallback for models without tool support
**OIDC user provisioning auto-migrates local users:**
- Files: `/Users/ryanchen/Programs/raggr/blueprints/users/oidc_service.py` (lines 42-53)
- Why fragile: Automatically converts local auth users to OIDC based on email match, clears passwords
- Test coverage: No tests detected
- Safe modification: Add dry-run mode, require admin confirmation for migrations, back up user table first
**Streaming response parsing relies on specific line format:**
- Files: `/Users/ryanchen/Programs/raggr/raggr-frontend/src/api/conversationService.ts` (lines 95-135)
- Why fragile: Assumes SSE format with `data: ` prefix and JSON, buffer handling for incomplete lines
- Test coverage: No tests for edge cases (connection drops mid-stream, malformed JSON, large chunks)
- Safe modification: Add comprehensive error handling, test with slow connections and large responses
**Vector store query uses unvalidated metadata filters:**
- Files: `/Users/ryanchen/Programs/raggr/main.py` (lines 133-155)
- Why fragile: Metadata filters from QueryGenerator passed directly to ChromaDB without validation
- Test coverage: None detected
- Safe modification: Validate filter structure before query, whitelist allowed filter keys
**Document chunking without validation:**
- Files: `/Users/ryanchen/Programs/raggr/utils/chunker.py` referenced in `/Users/ryanchen/Programs/raggr/main.py` (line 69)
- Why fragile: No validation of chunk size, overlap, or content before embedding
- Test coverage: None detected
- Safe modification: Add max chunk length validation, handle empty documents gracefully
## Scaling Limits
**Single PostgreSQL connection per request:**
- Current capacity: Depends on PostgreSQL max_connections (default ~100)
- Limit: Connection exhaustion under high concurrent load
- Scaling path: Implement connection pooling with Tortoise ORM pool settings, increase PostgreSQL max_connections
**ChromaDB local persistence not horizontally scalable:**
- Current capacity: Single-node file-based storage
- Limit: Cannot distribute across multiple app instances, I/O bound on single disk
- Scaling path: Migrate to ChromaDB server mode with shared storage or dedicated vector DB (Qdrant, Pinecone, Weaviate)
**Server-sent events keep connections open:**
- Current capacity: Limited by web server worker count and file descriptor limits
- Limit: Each streaming query holds connection open for full duration (10-60+ seconds)
- Scaling path: Use message queue (Redis Streams, RabbitMQ) for response streaming, implement connection pooling
**No horizontal scaling for background indexing:**
- Current capacity: Single process indexes documents sequentially
- Limit: Cannot parallelize across multiple workers/containers
- Scaling path: Implement task queue (Celery, RQ) for distributed indexing, use message broker to coordinate
**Frontend state management in React useState:**
- Current capacity: Works for single user, no persistence
- Limit: State lost on refresh, no offline support, memory growth with long conversations
- Scaling path: Migrate to Redux/Zustand with persistence, implement virtual scrolling for long conversations
## Dependencies at Risk
**ynab Python package is community-maintained:**
- Risk: Unofficial YNAB API wrapper, last update may lag behind API changes
- Impact: YNAB features break if API changes
- Migration plan: Monitor YNAB API changelog, consider switching to direct httpx/aiohttp requests for control
**LangChain rapid version changes:**
- Risk: Frequent breaking changes between minor versions in LangChain ecosystem
- Impact: Upgrades require code changes, agent patterns deprecated
- Migration plan: Pin specific versions in pyproject.toml, test thoroughly before upgrading
**Quart framework less mature than Flask:**
- Risk: Smaller community, fewer third-party extensions, async bugs less documented
- Impact: Harder to find solutions for edge cases
- Migration plan: Consider FastAPI as alternative (better async support, more active), or Flask with async support
## Missing Critical Features
**No observability/monitoring:**
- Problem: No structured logging, metrics, or tracing
- Blocks: Understanding production issues, performance debugging, user behavior analysis
- Priority: High
**No backup strategy for ChromaDB vector store:**
- Problem: Vector embeddings not backed up, expensive to regenerate
- Blocks: Disaster recovery, migrating instances
- Priority: High
**No API versioning:**
- Problem: Breaking API changes will break existing clients
- Blocks: Frontend/backend independent deployment
- Priority: Medium
**No health check endpoints:**
- Problem: Container orchestration cannot verify service health
- Blocks: Proper Kubernetes deployment, load balancer integration
- Priority: Medium
**No user quotas or resource limits:**
- Problem: Users can consume unlimited API calls, storage, compute
- Blocks: Cost control, fair resource allocation
- Priority: Medium
## Test Coverage Gaps
**No tests for LangChain agent tools:**
- What's not tested: All 15 tools in `/Users/ryanchen/Programs/raggr/blueprints/conversation/agents.py`
- Files: No test files detected for agents module
- Risk: Tool failures not caught until production, parameter handling bugs
- Priority: High
**No tests for streaming SSE implementation:**
- What's not tested: Server-sent events parsing, partial message handling, error recovery
- Files: `/Users/ryanchen/Programs/raggr/blueprints/conversation/__init__.py` (streaming endpoints), `/Users/ryanchen/Programs/raggr/raggr-frontend/src/api/conversationService.ts`
- Risk: Connection drops, malformed responses cause undefined behavior
- Priority: High
**No tests for OIDC authentication flow:**
- What's not tested: User provisioning, group claims parsing, token validation
- Files: `/Users/ryanchen/Programs/raggr/blueprints/users/oidc_service.py`, `/Users/ryanchen/Programs/raggr/blueprints/users/__init__.py`
- Risk: Auth bypass, user migration bugs, group permission issues
- Priority: High
**No integration tests for RAG pipeline:**
- What's not tested: End-to-end document indexing, query, and response generation
- Files: `/Users/ryanchen/Programs/raggr/blueprints/rag/logic.py`, `/Users/ryanchen/Programs/raggr/main.py`
- Risk: Embedding model changes, ChromaDB version changes break retrieval
- Priority: Medium
**No tests for external service integrations:**
- What's not tested: YNAB API error handling, Mealie API error handling, Tavily search failures
- Files: `/Users/ryanchen/Programs/raggr/utils/ynab_service.py`, `/Users/ryanchen/Programs/raggr/utils/mealie_service.py`
- Risk: API changes break features silently, rate limits not handled
- Priority: Medium
---
*Concerns audit: 2026-02-04*

333
.planning/codebase/CONVENTIONS.md Normal file
View File

@@ -0,0 +1,333 @@
# Coding Conventions
**Analysis Date:** 2026-02-04
## Naming Patterns
**Files:**
- Python: `snake_case.py` - `ynab_service.py`, `mealie_service.py`, `oidc_service.py`
- TypeScript/React: `PascalCase.tsx` for components, `camelCase.ts` for services
- Components: `ChatScreen.tsx`, `AnswerBubble.tsx`, `QuestionBubble.tsx`
- Services: `conversationService.ts`, `userService.ts`, `oidcService.ts`
- Config files: `snake_case.py` - `aerich_config.py`, `oidc_config.py`
**Functions:**
- Python: `snake_case` - `get_budget_summary()`, `parse_relative_date()`, `consult_simba_oracle()`
- TypeScript: `camelCase` - `handleQuestionSubmit()`, `sendQueryStream()`, `fetchWithRefreshToken()`
**Variables:**
- Python: `snake_case` - `budget_id`, `access_token`, `llama_url`, `current_user_uuid`
- TypeScript: `camelCase` - `conversationId`, `streamingContent`, `isLoading`
**Types:**
- Python classes: `PascalCase` - `YNABService`, `MealieService`, `LLMClient`, `User`, `Conversation`
- Python enums: `PascalCase` with SCREAMING_SNAKE_CASE values - `Speaker.USER`, `Speaker.SIMBA`
- TypeScript interfaces: `PascalCase` - `Message`, `Conversation`, `QueryResponse`, `StreamEvent`
- TypeScript types: `PascalCase` - `ChatScreenProps`, `QuestionAnswer`
**Constants:**
- Python: `SCREAMING_SNAKE_CASE` - `DATABASE_URL`, `TORTOISE_CONFIG`, `PROVIDER`
- TypeScript: `camelCase` - `baseUrl`, `conversationBaseUrl`
## Code Style
**Formatting:**
- Python: No explicit formatter configured (no Black, autopep8, or yapf config detected)
- Manual formatting observed: 4-space indentation, line length ~88-100 chars
- TypeScript: Biome 2.3.10 configured in `raggr-frontend/package.json`
- No explicit biome.json found, using defaults
**Linting:**
- Python: No linter config detected (no pylint, flake8, ruff config)
- TypeScript: Biome handles linting via `@biomejs/biome` package
**Imports:**
- Python: Standard library first, then third-party, then local imports
```python
import os
import logging
from datetime import datetime
from dotenv import load_dotenv
from quart import Blueprint
from .models import User
from .logic import get_conversation
```
- TypeScript: React imports, then third-party, then local (relative)
```typescript
import { useEffect, useState } from "react";
import { conversationService } from "../api/conversationService";
import { QuestionBubble } from "./QuestionBubble";
```
## Import Organization
**Order:**
1. Standard library imports
2. Third-party framework imports (Flask/Quart/React/etc)
3. Local application imports (blueprints, utils, models)
**Path Aliases:**
- None detected in TypeScript - uses relative imports (`../api/`, `./components/`)
- Python uses absolute imports for blueprints and utils modules
**Absolute vs Relative:**
- Python: Absolute imports for cross-module (`from utils.ynab_service import YNABService`)
- TypeScript: Relative imports (`../api/conversationService`)
## Error Handling
**Patterns:**
- Python: Try/except with detailed logging
```python
try:
# operation
logger.info("[SERVICE] Operation details")
except httpx.HTTPStatusError as e:
logger.error(f"[SERVICE] HTTP error: {e.response.status_code}")
raise
except Exception as e:
logger.error(f"[SERVICE] Error: {type(e).__name__}: {str(e)}")
logger.exception("[SERVICE] Full traceback:")
raise
```
- TypeScript: Try/catch with console.error, re-throw or handle gracefully
```typescript
try {
const response = await fetch();
} catch (error) {
console.error("Failed to fetch:", error);
if (error.message.includes("Session expired")) {
setAuthenticated(false);
}
}
```
**Async Error Handling:**
- Python: `async def` functions use try/except blocks
- TypeScript: `async` functions use try/catch blocks
- Both propagate errors upward with `raise` (Python) or `throw` (TypeScript)
**HTTP Errors:**
- Python Quart: Return `jsonify({"error": "message"}), status_code`
- Python httpx: Raise HTTPStatusError, log response text
- TypeScript: Throw Error with descriptive message
## Logging
**Framework:**
- Python: Standard `logging` module
- TypeScript: `console.log()`, `console.error()`
**Patterns:**
- Python: Structured logging with prefixes
```python
logger = logging.getLogger(__name__)
logger.info("[SERVICE] Operation started")
logger.error(f"[SERVICE] Error: {details}")
logger.exception("[SERVICE] Full traceback:") # After except
```
- Logging levels: INFO for operations, ERROR for failures, DEBUG for detailed data
- Service-specific prefixes: `[YNAB]`, `[MEALIE]`, `[YNAB TOOLS]`
**When to Log:**
- Entry/exit of major operations (API calls, database queries)
- Error conditions with full context
- Configuration/initialization status
- Performance metrics (timing critical operations)
**Examples from codebase:**
```python
logger.info(f"[YNAB] get_budget_summary() called for budget_id: {self.budget_id}")
logger.info(f"[YNAB] Total budgeted: ${total_budgeted:.2f}")
logger.error(f"[YNAB] Error in get_budget_summary(): {type(e).__name__}: {str(e)}")
```
## Comments
**When to Comment:**
- Complex business logic (date parsing, budget calculations)
- Non-obvious workarounds or API quirks
- Important configuration decisions
- Docstrings for all public functions/methods
**JSDoc/TSDoc:**
- Python: Docstrings with Args/Returns sections
```python
def get_transactions(self, start_date: Optional[str] = None) -> dict[str, Any]:
"""Get transactions filtered by date range.
Args:
start_date: Start date in YYYY-MM-DD or relative ('this_month')
Returns:
Dictionary containing matching transactions and summary.
"""
```
- TypeScript: Inline comments, no formal JSDoc detected
```typescript
// Stream events back to client as they happen
async function generate() {
// ...
}
```
**Comment Style:**
- Python: `# Single line` or `"""Docstring"""`
- TypeScript: `// Single line` or `/* Multi-line */`
- No TODOs/FIXMEs in project code (only in node_modules)
## Function Design
**Size:**
- Python: 20-100 lines typical, some reach 150+ (service methods with error handling)
- TypeScript: 10-50 lines for React components, 20-80 for service methods
- Large functions acceptable when handling complex workflows (streaming, API interactions)
**Parameters:**
- Python: Explicit typing with `Optional[type]`, defaults for optional params
```python
def get_transactions(
self,
start_date: Optional[str] = None,
end_date: Optional[str] = None,
limit: int = 50,
) -> dict[str, Any]:
```
- TypeScript: Interfaces for complex parameter objects
```typescript
async sendQueryStream(
query: string,
conversation_id: string,
callbacks: StreamCallbacks,
signal?: AbortSignal,
): Promise<void>
```
**Return Values:**
- Python: Explicit return type hints - `-> dict[str, Any]`, `-> str`, `-> bool`
- TypeScript: Explicit types - `: Promise<Conversation>`, `: void`
- Dictionary/object returns for complex data (not tuples in Python)
**Async/Await:**
- Python Quart: All route handlers are `async def`
- Python services: Database queries and external API calls are `async`
- TypeScript: All API calls use `async/await` pattern
## Module Design
**Exports:**
- Python: No explicit `__all__`, classes/functions imported directly
- TypeScript: Named exports for classes/functions, default export for singleton services
```typescript
export const conversationService = new ConversationService();
```
**Barrel Files:**
- Python: `blueprints/__init__.py` defines blueprints, re-exported
- TypeScript: No barrel files, direct imports
**Structure:**
- Python blueprints: `__init__.py` contains routes, `models.py` for ORM, `logic.py` for business logic
- Services in separate modules: `utils/ynab_service.py`, `utils/mealie_service.py`
- Separation of concerns: routes, models, business logic, utilities
## Decorators
**Authentication:**
- `@jwt_refresh_token_required` - Standard auth requirement
- `@admin_required` - Custom decorator for admin-only routes (wraps `@jwt_refresh_token_required`)
**Route Decorators:**
- `@app.route()` or `@blueprint.route()` with HTTP method
- Async routes: `async def` function signature
**Tool Decorators (LangChain):**
- `@tool` - Mark functions as LangChain tools
- `@tool(response_format="content_and_artifact")` - Specialized tool responses
**Pattern:**
```python
@conversation_blueprint.post("/query")
@jwt_refresh_token_required
async def query():
current_user_uuid = get_jwt_identity()
# ...
```
## Type Hints
**Python:**
- Modern type hints throughout: `dict[str, Any]`, `Optional[str]`, `list[str]`
- Tortoise ORM types: `fields.ForeignKeyRelation`
- No legacy typing module usage (using built-in generics)
**TypeScript:**
- Strict typing with interfaces
- Union types for variants: `"user" | "simba"`, `'status' | 'content' | 'done' | 'error'`
- Generic types: `Promise<T>`, `React.ChangeEvent<HTMLTextAreaElement>`
## State Management
**Python (Backend):**
- Database: Tortoise ORM async models
- In-memory: Module-level variables for services (`ynab_service`, `mealie_service`)
- Session: JWT tokens, in-memory dict for OIDC sessions (`_oidc_sessions`)
**TypeScript (Frontend):**
- React hooks: `useState`, `useEffect`, `useRef`
- localStorage for JWT tokens (via `userService`)
- No global state management library (no Redux/Zustand)
**Pattern:**
```typescript
const [isLoading, setIsLoading] = useState<boolean>(false);
const abortControllerRef = useRef<AbortController | null>(null);
```
## Database Conventions
**ORM:**
- Tortoise ORM with Aerich for migrations
- Models inherit from `Model` base class
- Field definitions: `fields.UUIDField`, `fields.CharField`, `fields.ForeignKeyField`
**Naming:**
- Table names: Lowercase plural (`users`, `conversations`, `conversation_messages`)
- Foreign keys: Singular model name (`user`, `conversation`)
- Related names: Plural (`conversations`, `messages`)
**Pattern:**
```python
class Conversation(Model):
id = fields.UUIDField(primary_key=True)
name = fields.CharField(max_length=255)
user: fields.ForeignKeyRelation = fields.ForeignKeyField(
"models.User", related_name="conversations", null=True
)
class Meta:
table = "conversations"
```
## API Conventions
**REST Endpoints:**
- Prefix: `/api/{resource}`
- Blueprints: `/api/user`, `/api/conversation`, `/api/rag`
- CRUD patterns: GET for fetch, POST for create/actions, PUT for update, DELETE for remove
**Request/Response:**
- JSON payloads: `await request.get_json()`
- Responses: `jsonify({...})` with optional status code
- Streaming: Server-Sent Events (SSE) with `text/event-stream` mimetype
**Authentication:**
- JWT in Authorization header (managed by `quart-jwt-extended`)
- Refresh tokens for long-lived sessions
- OIDC flow for external authentication
---
*Convention analysis: 2026-02-04*

182
.planning/codebase/INTEGRATIONS.md Normal file
View File

@@ -0,0 +1,182 @@
# External Integrations
**Analysis Date:** 2026-02-04
## APIs & External Services
**Document Management:**
- Paperless-NGX - Document ingestion and retrieval
- SDK/Client: Custom client in `utils/request.py` using `httpx`
- Auth: `PAPERLESS_TOKEN` (bearer token)
- Base URL: `BASE_URL` environment variable
- Purpose: Fetch documents for indexing, download PDFs, retrieve document metadata and types
**LLM Services:**
- llama-server (primary) - Local LLM inference via OpenAI-compatible API
- SDK/Client: `openai` Python package (v2.0.1+)
- Connection: `LLAMA_SERVER_URL` (e.g., `http://192.168.1.213:8080/v1`)
- Model: `LLAMA_MODEL_NAME` (e.g., `llama-3.1-8b-instruct`)
- Implementation: `llm.py` creates OpenAI client with custom base_url
- LangChain: `langchain-openai.ChatOpenAI` with custom base_url for agent framework
- OpenAI (fallback) - Cloud LLM service
- SDK/Client: `openai` Python package
- Auth: `OPENAI_API_KEY`
- Models: `gpt-4o-mini` (embeddings and chat), `gpt-5-mini` (fallback for agents)
- Implementation: Automatic fallback when `LLAMA_SERVER_URL` not configured
- Used for: Chat completions, embeddings via ChromaDB embedding function
**Web Search:**
- Tavily - Web search API for real-time information retrieval
- SDK/Client: `tavily-python` (v0.7.17+)
- Auth: `TAVILY_API_KEY`
- Implementation: `blueprints/conversation/agents.py` - `AsyncTavilyClient`
- Used in: LangChain agent tool for web searches
**Budget Tracking:**
- YNAB (You Need A Budget) - Personal finance and budget management
- SDK/Client: `ynab` Python package (v1.3.0+)
- Auth: `YNAB_ACCESS_TOKEN` (Personal Access Token from YNAB settings)
- Budget Selection: `YNAB_BUDGET_ID` (optional, auto-detects first budget if not set)
- Implementation: `utils/ynab_service.py` - `YNABService` class
- Features: Budget summary, transaction search, category spending, spending insights
- API Endpoints: Budgets API, Transactions API, Months API, Categories API
- Used in: LangChain agent tools for financial queries
**Meal Planning:**
- Mealie - Self-hosted meal planning and recipe management
- SDK/Client: Custom async client using `httpx` in `utils/mealie_service.py`
- Auth: `MEALIE_API_TOKEN` (Bearer token)
- Base URL: `MEALIE_BASE_URL` (e.g., `http://192.168.1.5:9000`)
- Implementation: `MealieService` class with async methods
- Features: Shopping lists, meal plans, today's meals, recipe details, CRUD operations on meal plans
- API Endpoints: `/api/households/shopping/*`, `/api/households/mealplans/*`, `/api/households/self/recipes/*`
- Used in: LangChain agent tools for meal planning queries
**Photo Management (referenced but not actively used):**
- Immich - Photo library management
- Connection: `IMMICH_URL`, `IMMICH_API_KEY`
- Search: `SEARCH_QUERY`, `DOWNLOAD_DIR`
- Note: Environment variables defined but service implementation not found in current code
## Data Storage
**Databases:**
- PostgreSQL 16
- Connection: `DATABASE_URL` (format: `postgres://user:pass@host:port/db`)
- Container: `postgres:16-alpine` image
- Client: Tortoise ORM (async ORM with Pydantic models)
- Models: User management, conversations, messages, OIDC state
- Migrations: Aerich tool in `migrations/` directory
- Volume: `postgres_data` persistent volume
**Vector Store:**
- ChromaDB
- Type: Embedded vector database (PersistentClient)
- Path: `CHROMADB_PATH` (Docker: `/app/data/chromadb`, local: `./data/chromadb`)
- Collections: `simba_docs2` (main RAG documents), `feline_vet_lookup` (veterinary knowledge)
- Embedding Function: OpenAI embeddings via `chromadb.utils.embedding_functions.openai_embedding_function`
- Integration: LangChain via `langchain-chroma` for vector store queries
- Volume: `chromadb_data` persistent volume
**File Storage:**
- Local filesystem only
- PDF downloads: Temporary files for processing
- Image conversion: Temporary files from PDF to image conversion
- Database tracking: `database/visited.db` SQLite for tracking indexed documents
**Caching:**
- None - No explicit caching layer configured
## Authentication & Identity
**Auth Provider:**
- Authelia (OIDC) - Self-hosted authentication and authorization server
- Implementation: Custom OIDC client in `config/oidc_config.py`
- Discovery: `.well-known/openid-configuration` endpoint (configurable via `OIDC_USE_DISCOVERY`)
- Environment Variables:
- `OIDC_ISSUER` (e.g., `https://auth.example.com`)
- `OIDC_CLIENT_ID` (e.g., `simbarag`)
- `OIDC_CLIENT_SECRET`
- `OIDC_REDIRECT_URI` (default: `http://localhost:8080/`)
- Manual endpoint override: `OIDC_AUTHORIZATION_ENDPOINT`, `OIDC_TOKEN_ENDPOINT`, `OIDC_USERINFO_ENDPOINT`, `OIDC_JWKS_URI`
- Token Verification: JWT verification using `authlib.jose.jwt` with JWKS
- LDAP Integration: LLDAP groups for RBAC (checks `lldap_admin` group for admin permissions)
**Session Management:**
- JWT tokens via `quart-jwt-extended`
- Secret: `JWT_SECRET_KEY` environment variable
- Storage: Frontend localStorage
- Decorators: `@jwt_refresh_token_required` for protected endpoints, `@admin_required` for admin routes
## Monitoring & Observability
**Error Tracking:**
- None - No external error tracking service configured
**Logs:**
- Standard Python logging to stdout/stderr
- Format: `%(asctime)s - %(name)s - %(levelname)s - %(message)s`
- Level: INFO (configurable via logging module)
- Special loggers: `utils.ynab_service`, `utils.mealie_service`, `blueprints.conversation.agents` set to INFO level
- Docker: Logs accessible via `docker compose logs`
**Metrics:**
- None - No metrics collection configured
## CI/CD & Deployment
**Hosting:**
- Docker Compose - Self-hosted container deployment
- Production: `docker-compose.yml`
- Development: `docker-compose.dev.yml` with volume mounts for hot reload
- Image: `torrtle/simbarag:latest` (custom build)
**CI Pipeline:**
- None - No automated CI/CD configured
- Manual builds: `docker compose build raggr`
- Manual deploys: `docker compose up -d`
**Container Registry:**
- Docker Hub (inferred from image name `torrtle/simbarag:latest`)
## Environment Configuration
**Required env vars:**
- `DATABASE_URL` - PostgreSQL connection string
- `JWT_SECRET_KEY` - JWT token signing key
- `PAPERLESS_TOKEN` - Paperless-NGX API token
- `BASE_URL` - Paperless-NGX instance URL
**LLM configuration (choose one):**
- `LLAMA_SERVER_URL` + `LLAMA_MODEL_NAME` - Local llama-server (primary)
- `OPENAI_API_KEY` - OpenAI API (fallback)
**Optional integrations:**
- `YNAB_ACCESS_TOKEN`, `YNAB_BUDGET_ID` - YNAB budget integration
- `MEALIE_BASE_URL`, `MEALIE_API_TOKEN` - Mealie meal planning
- `TAVILY_API_KEY` - Web search capability
- `IMMICH_URL`, `IMMICH_API_KEY`, `SEARCH_QUERY`, `DOWNLOAD_DIR` - Immich photos
**OIDC authentication:**
- `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_REDIRECT_URI`
- `OIDC_USE_DISCOVERY` - Enable/disable OIDC discovery (default: true)
**Secrets location:**
- `.env` file in project root (not committed to git)
- Docker Compose reads from `.env` file automatically
- Example file: `.env.example` with placeholder values
## Webhooks & Callbacks
**Incoming:**
- `/api/user/oidc/callback` - OIDC authorization code callback from Authelia
- Method: GET with `code` and `state` query parameters
- Flow: Authorization code → token exchange → user info → JWT creation
**Outgoing:**
- None - No webhook subscriptions to external services
---
*Integration audit: 2026-02-04*

107
.planning/codebase/STACK.md Normal file
View File

@@ -0,0 +1,107 @@
# Technology Stack
**Analysis Date:** 2026-02-04
## Languages
**Primary:**
- Python 3.13 - Backend application, RAG logic, API endpoints, utilities
**Secondary:**
- TypeScript 5.9.2 - Frontend React application with type safety
- JavaScript - Build tooling and configuration
## Runtime
**Environment:**
- Python 3.13-slim (Docker container)
- Node.js 20.x (for frontend builds)
**Package Manager:**
- uv - Python dependency management (Astral's fast installer)
- Yarn - Frontend package management
- Lockfiles: `uv.lock` and `raggr-frontend/yarn.lock` present
## Frameworks
**Core:**
- Quart 0.20.0 - Async Python web framework (Flask-like API with async support)
- React 19.1.1 - Frontend UI library
- Rsbuild 1.5.6 - Modern frontend build tool (Rspack-based)
**Testing:**
- Not explicitly configured in dependencies
**Build/Dev:**
- Rsbuild 1.5.6 - Frontend bundler with React plugin
- Black 25.9.0 - Python code formatter
- Biome 2.3.10 - Frontend linter and formatter (replaces ESLint/Prettier)
- Pre-commit 4.3.0 - Git hooks for code quality
- Docker Compose - Container orchestration (dev and prod configurations)
## Key Dependencies
**Critical:**
- `chromadb>=1.1.0` - Vector database for document embeddings and similarity search
- `openai>=2.0.1` - LLM client library (used for both OpenAI and llama-server via OpenAI-compatible API)
- `langchain>=1.2.0` - LLM application framework with agent and tool support
- `langchain-openai>=1.1.6` - LangChain integration for OpenAI/llama-server
- `langchain-chroma>=1.0.0` - LangChain integration for ChromaDB
- `tortoise-orm>=0.25.1` - Async ORM for PostgreSQL database operations
- `quart-jwt-extended>=0.1.0` - JWT authentication for Quart
- `authlib>=1.3.0` - OIDC/OAuth2 client library
**Infrastructure:**
- `httpx>=0.28.1` - Async HTTP client for API integrations
- `asyncpg>=0.30.0` - PostgreSQL async driver
- `aerich>=0.8.0` - Database migration tool for Tortoise ORM
- `pymupdf>=1.24.0` - PDF processing (fitz)
- `pillow>=10.0.0` - Image processing
- `pillow-heif>=1.1.1` - HEIF/HEIC image format support
- `bcrypt>=5.0.0` - Password hashing
- `python-dotenv>=1.0.0` - Environment variable management
**External Service Integrations:**
- `tavily-python>=0.7.17` - Web search API client
- `ynab>=1.3.0` - YNAB (budgeting app) API client
- `axios^1.12.2` - Frontend HTTP client
- `react-markdown^10.1.0` - Markdown rendering in React
- `marked^16.3.0` - Markdown parser
## Configuration
**Environment:**
- `.env` files for environment-specific configuration
- Required vars: `DATABASE_URL`, `JWT_SECRET_KEY`, `PAPERLESS_TOKEN`, `BASE_URL`
- Optional LLM: `LLAMA_SERVER_URL`, `LLAMA_MODEL_NAME` (primary) or `OPENAI_API_KEY` (fallback)
- Optional integrations: `YNAB_ACCESS_TOKEN`, `MEALIE_BASE_URL`, `MEALIE_API_TOKEN`, `TAVILY_API_KEY`
- OIDC auth: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`, `OIDC_REDIRECT_URI`
- ChromaDB: `CHROMADB_PATH` (defaults to `/app/data/chromadb` in Docker)
**Build:**
- `pyproject.toml` - Python project metadata and dependencies
- `rsbuild.config.ts` - Frontend build configuration
- `tsconfig.json` - TypeScript compiler configuration
- `Dockerfile` - Multi-stage build (Python + Node.js)
- `docker-compose.yml` - Production container setup
- `docker-compose.dev.yml` - Development with hot reload
- `aerich_config.py` - Database migration configuration
- `.pre-commit-config.yaml` - Git hooks for code quality
## Platform Requirements
**Development:**
- Python 3.13+
- Node.js 20.x
- PostgreSQL 16+ (via Docker or local)
- Docker and Docker Compose (recommended)
**Production:**
- Docker environment
- PostgreSQL 16-alpine container
- Persistent volumes for ChromaDB and PostgreSQL data
- Network access to external APIs (Paperless-NGX, YNAB, Mealie, Tavily, OpenAI, llama-server)
---
*Stack analysis: 2026-02-04*

237
.planning/codebase/STRUCTURE.md Normal file
View File

@@ -0,0 +1,237 @@
# Codebase Structure
**Analysis Date:** 2026-02-04
## Directory Layout
```
raggr/
├── blueprints/ # API route modules (Quart blueprints)
│ ├── conversation/ # Chat conversation endpoints and logic
│ ├── rag/ # Document indexing and retrieval endpoints
│ └── users/ # Authentication and user management
├── config/ # Configuration modules
├── utils/ # Reusable service clients and utilities
├── scripts/ # Administrative CLI scripts
├── migrations/ # Database schema migrations (Aerich)
├── raggr-frontend/ # React SPA frontend
│ ├── src/
│ │ ├── components/ # React UI components
│ │ ├── api/ # Frontend API service clients
│ │ ├── contexts/ # React contexts (Auth)
│ │ └── assets/ # Static images
│ └── dist/ # Built frontend (served by backend)
├── chroma_db/ # ChromaDB persistent vector store
├── chromadb/ # Alternate ChromaDB path (legacy)
├── docs/ # Documentation files
├── app.py # Quart application entry point
├── main.py # RAG logic and CLI entry point
├── llm.py # LLM client with provider fallback
└── aerich_config.py # Database migration configuration
```
## Directory Purposes
**blueprints/**
- Purpose: API route organization using Quart blueprint pattern
- Contains: Python packages with `__init__.py` (routes), `models.py` (ORM), `logic.py` (business logic)
- Key files: `conversation/__init__.py` (chat API), `rag/__init__.py` (indexing API), `users/__init__.py` (auth API)
**blueprints/conversation/**
- Purpose: Chat conversation management
- Contains: Streaming chat endpoints, message persistence, conversation CRUD, agent orchestration
- Key files: `__init__.py` (endpoints), `agents.py` (LangChain agent + tools), `logic.py` (conversation operations), `models.py` (Conversation, ConversationMessage)
**blueprints/rag/**
- Purpose: Document indexing and vector search
- Contains: Admin-only indexing endpoints, vector store operations, Paperless-NGX integration
- Key files: `__init__.py` (endpoints), `logic.py` (indexing + query), `fetchers.py` (Paperless client)
**blueprints/users/**
- Purpose: User authentication and authorization
- Contains: OIDC login flow, JWT token management, RBAC decorators
- Key files: `__init__.py` (auth endpoints), `models.py` (User model), `decorators.py` (@admin_required), `oidc_service.py` (user provisioning)
**config/**
- Purpose: Configuration modules for external integrations
- Contains: OIDC configuration with JWKS verification
- Key files: `oidc_config.py`
**utils/**
- Purpose: Reusable utilities and external service clients
- Contains: Chunking, cleaning, API clients for YNAB/Mealie/Paperless
- Key files: `chunker.py`, `cleaner.py`, `ynab_service.py`, `mealie_service.py`, `request.py` (Paperless client), `image_process.py`
**scripts/**
- Purpose: Administrative and maintenance CLI tools
- Contains: User management, statistics, vector store inspection
- Key files: `add_user.py`, `user_message_stats.py`, `manage_vectorstore.py`, `inspect_vector_store.py`, `query.py`
**migrations/**
- Purpose: Database schema version control (Aerich/Tortoise ORM)
- Contains: SQL migration files generated by `aerich migrate`
- Generated: Yes
- Committed: Yes
**raggr-frontend/**
- Purpose: React single-page application
- Contains: React 19 components, Rsbuild bundler config, Tailwind CSS, TypeScript
- Key files: `src/App.tsx` (root), `src/index.tsx` (entry), `src/components/ChatScreen.tsx` (main UI)
**raggr-frontend/src/components/**
- Purpose: React UI components
- Contains: Chat interface, login, conversation list, message bubbles
- Key files: `ChatScreen.tsx`, `LoginScreen.tsx`, `ConversationList.tsx`, `AnswerBubble.tsx`, `QuestionBubble.tsx`, `MessageInput.tsx`
**raggr-frontend/src/api/**
- Purpose: Frontend service layer for API communication
- Contains: TypeScript service clients with axios/fetch
- Key files: `conversationService.ts` (SSE streaming), `userService.ts`, `oidcService.ts`
**raggr-frontend/src/contexts/**
- Purpose: React contexts for global state
- Contains: Authentication context
- Key files: `AuthContext.tsx`
**raggr-frontend/dist/**
- Purpose: Built frontend assets served by backend
- Contains: Bundled JS, CSS, HTML
- Generated: Yes (by Rsbuild)
- Committed: No
**chroma_db/** and **chromadb/**
- Purpose: ChromaDB persistent vector store data
- Contains: SQLite database files and vector indices
- Generated: Yes (at runtime)
- Committed: No
**docs/**
- Purpose: Project documentation
- Contains: Integration documentation, technical specs
- Key files: `ynab_integration/`
## Key File Locations
**Entry Points:**
- `app.py`: Web server entry point (Quart application)
- `main.py`: CLI entry point for RAG operations
- `raggr-frontend/src/index.tsx`: Frontend entry point
**Configuration:**
- `.env`: Environment variables (not committed, see `.env.example`)
- `aerich_config.py`: Database migration configuration
- `config/oidc_config.py`: OIDC authentication configuration
- `raggr-frontend/rsbuild.config.ts`: Frontend build configuration
**Core Logic:**
- `blueprints/conversation/agents.py`: LangChain agent with tool definitions
- `blueprints/rag/logic.py`: Vector store indexing and query operations
- `main.py`: Original RAG implementation (legacy, partially superseded by blueprints)
- `llm.py`: LLM client abstraction with fallback logic
**Testing:**
- Not detected (no test files found)
## Naming Conventions
**Files:**
- Snake_case for Python modules: `ynab_service.py`, `oidc_config.py`
- PascalCase for React components: `ChatScreen.tsx`, `AnswerBubble.tsx`
- Lowercase for config files: `docker-compose.yml`, `pyproject.toml`
**Directories:**
- Lowercase with underscores for Python packages: `blueprints/conversation/`, `utils/`
- Kebab-case for frontend: `raggr-frontend/`
**Python Classes:**
- PascalCase: `User`, `Conversation`, `ConversationMessage`, `LLMClient`, `YNABService`
**Python Functions:**
- Snake_case: `get_conversation_by_id`, `query_vector_store`, `add_message_to_conversation`
**React Components:**
- PascalCase: `ChatScreen`, `LoginScreen`, `ConversationList`
**API Routes:**
- Kebab-case: `/api/conversation/query`, `/api/user/oidc/callback`
**Environment Variables:**
- SCREAMING_SNAKE_CASE: `DATABASE_URL`, `YNAB_ACCESS_TOKEN`, `LLAMA_SERVER_URL`
## Where to Add New Code
**New API Endpoint:**
- Primary code: Create or extend blueprint in `blueprints/<domain>/__init__.py`
- Business logic: Add functions to `blueprints/<domain>/logic.py`
- Database models: Add to `blueprints/<domain>/models.py`
- Tests: Not established (no test directory exists)
**New LangChain Tool:**
- Implementation: Add `@tool` decorated function in `blueprints/conversation/agents.py`
- Service client: If calling external API, create client in `utils/<service>_service.py`
- Add to tools list: Append to `tools` list at bottom of `agents.py` (line 709+)
**New External Service Integration:**
- Service client: Create `utils/<service>_service.py` with async methods
- Tool wrapper: Add tool function in `blueprints/conversation/agents.py`
- Configuration: Add env vars to `.env.example`
**New React Component:**
- Component file: `raggr-frontend/src/components/<ComponentName>.tsx`
- API service: If needs backend, add methods to `raggr-frontend/src/api/<domain>Service.ts`
- Import in: `raggr-frontend/src/App.tsx` or parent component
**New Database Table:**
- Model: Add Tortoise model to `blueprints/<domain>/models.py`
- Migration: Run `docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name <description>`
- Apply: Run `docker compose -f docker-compose.dev.yml exec raggr aerich upgrade` (or restart container)
**Utilities:**
- Shared helpers: `utils/<utility_name>.py` for Python utilities
- Frontend utilities: `raggr-frontend/src/utils/` (not currently used, would need creation)
## Special Directories
**.git/**
- Purpose: Git version control metadata
- Generated: Yes
- Committed: No (automatically handled by git)
**.venv/**
- Purpose: Python virtual environment
- Generated: Yes (local dev only)
- Committed: No
**node_modules/**
- Purpose: NPM dependencies for frontend
- Generated: Yes (npm/yarn install)
- Committed: No
**__pycache__/**
- Purpose: Python bytecode cache
- Generated: Yes (Python runtime)
- Committed: No
**.planning/**
- Purpose: GSD (Get Stuff Done) codebase documentation
- Generated: Yes (by GSD commands)
- Committed: Yes (intended for project documentation)
**.claude/**
- Purpose: Claude Code session data
- Generated: Yes
- Committed: No
**.ruff_cache/**
- Purpose: Ruff linter cache
- Generated: Yes
- Committed: No
**.ropeproject/**
- Purpose: Rope Python refactoring library cache
- Generated: Yes
- Committed: No
---
*Structure analysis: 2026-02-04*

290
.planning/codebase/TESTING.md Normal file
View File

@@ -0,0 +1,290 @@
# Testing Patterns
**Analysis Date:** 2026-02-04
## Test Framework
**Runner:**
- None detected
- No pytest.ini, pytest.toml, jest.config.js, or vitest.config.ts found
- No test files in codebase (no `test_*.py`, `*_test.py`, `*.test.ts`, `*.spec.ts`)
**Assertion Library:**
- Not applicable (no tests present)
**Run Commands:**
```bash
# No test commands configured in package.json or standard Python test runners
```
## Test File Organization
**Location:**
- No test files detected in the project
**Naming:**
- Not established (no existing test files to analyze)
**Structure:**
```
# No test directory structure present
```
## Test Structure
**Suite Organization:**
Not applicable - no tests exist in the codebase.
**Expected Pattern (based on project structure):**
```python
# Python tests would likely use pytest with async support
import pytest
from quart import Quart
@pytest.mark.asyncio
async def test_endpoint():
# Test Quart async endpoints
pass
```
**TypeScript Pattern (if implemented):**
```typescript
// Would likely use Vitest (matches Rsbuild ecosystem)
import { describe, it, expect } from 'vitest';
describe('conversationService', () => {
it('should fetch conversations', async () => {
// Test API service methods
});
});
```
## Mocking
**Framework:**
- Not established (no tests present)
**Likely Approach:**
- Python: `pytest-mock` or `unittest.mock` for services/API calls
- TypeScript: Vitest mocking utilities
**What to Mock:**
- External API calls (YNAB, Mealie, Paperless-NGX, Tavily)
- LLM interactions (OpenAI/llama-server)
- Database queries (Tortoise ORM)
- Authentication/JWT verification
**What NOT to Mock:**
- Business logic functions (these should be tested directly)
- Data transformations
- Utility functions without side effects
## Fixtures and Factories
**Test Data:**
Not established - would need fixtures for:
- User objects with various authentication states
- Conversation and Message objects
- Mock YNAB/Mealie API responses
- Mock ChromaDB query results
**Expected Pattern:**
```python
# Python fixtures with pytest
@pytest.fixture
async def test_user():
"""Create a test user."""
user = await User.create(
username="testuser",
email="test@example.com",
auth_provider="local"
)
yield user
await user.delete()
@pytest.fixture
def mock_ynab_response():
"""Mock YNAB API budget response."""
return {
"budget_name": "Test Budget",
"to_be_budgeted": 100.00,
"total_budgeted": 2000.00,
}
```
## Coverage
**Requirements:**
- No coverage requirements configured
- No `.coveragerc` or coverage configuration in `pyproject.toml`
**Current State:**
- **0% test coverage** (no tests exist)
**View Coverage:**
```bash
# Would use pytest-cov for Python
pytest --cov=. --cov-report=html
# Would use Vitest coverage for TypeScript
npx vitest --coverage
```
## Test Types
**Unit Tests:**
- Not present
- Should test: Service methods, utility functions, data transformations, business logic
**Integration Tests:**
- Not present
- Should test: API endpoints, database operations, authentication flows, external service integrations
**E2E Tests:**
- Not present
- Could use: Playwright or Cypress for frontend testing
## Common Patterns
**Async Testing:**
Expected pattern for Quart/async Python:
```python
import pytest
from httpx import AsyncClient
from app import app
@pytest.mark.asyncio
async def test_query_endpoint():
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post(
"/api/conversation/query",
json={"query": "test", "conversation_id": "uuid"}
)
assert response.status_code == 200
```
**Error Testing:**
Expected pattern:
```python
@pytest.mark.asyncio
async def test_unauthorized_access():
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post("/api/conversation/query")
assert response.status_code == 401
assert "error" in response.json()
```
## Testing Gaps
**Critical Areas Without Tests:**
1. **Authentication & Authorization:**
- OIDC flow (`blueprints/users/__init__.py` - 188 lines)
- JWT token refresh
- Admin authorization decorator
- PKCE verification
2. **Core RAG Functionality:**
- Document indexing (`main.py` - 274 lines)
- Vector store queries (`blueprints/rag/logic.py`)
- LLM agent tools (`blueprints/conversation/agents.py` - 733 lines)
- Query classification
3. **External Service Integrations:**
- YNAB API client (`utils/ynab_service.py` - 576 lines)
- Mealie API client (`utils/mealie_service.py` - 477 lines)
- Paperless-NGX API client (`utils/request.py`)
- Tavily web search
4. **Streaming Responses:**
- Server-Sent Events in `/api/conversation/query`
- Frontend SSE parsing (`conversationService.sendQueryStream()`)
5. **Database Operations:**
- Conversation creation and retrieval
- Message persistence
- User CRUD operations
6. **Frontend Components:**
- ChatScreen streaming state (`ChatScreen.tsx` - 386 lines)
- Message bubbles rendering
- Authentication context
## Recommended Testing Strategy
**Phase 1: Critical Path Tests**
- Authentication endpoints (login, callback, token refresh)
- Conversation query endpoint (non-streaming)
- User creation and retrieval
- Basic YNAB/Mealie service methods
**Phase 2: Integration Tests**
- Full OIDC authentication flow
- Conversation with messages persistence
- RAG document indexing and retrieval
- External API error handling
**Phase 3: Frontend Tests**
- Component rendering tests
- API service method tests
- Streaming response handling
- Authentication state management
**Phase 4: E2E Tests**
- Complete user journey (login → query → response)
- Conversation management
- Admin operations
## Testing Dependencies to Add
**Python:**
```toml
# Add to pyproject.toml [tool.poetry.group.dev.dependencies] or requirements-dev.txt
pytest = "^7.0"
pytest-asyncio = "^0.21"
pytest-cov = "^4.0"
pytest-mock = "^3.10"
httpx = "^0.24" # For testing async HTTP
```
**TypeScript:**
```json
// Add to raggr-frontend/package.json devDependencies
"@vitest/ui": "^1.0.0",
"vitest": "^1.0.0",
"@testing-library/react": "^14.0.0",
"@testing-library/jest-dom": "^6.0.0"
```
## Testing Best Practices (Not Yet Implemented)
**Database Tests:**
- Use separate test database
- Reset database state between tests
- Use Aerich to apply migrations in test environment
**Async Tests:**
- Mark all async tests with `@pytest.mark.asyncio`
- Use `AsyncClient` for Quart endpoint testing
- Properly await all async operations
**Mocking External Services:**
- Mock all HTTP calls to external APIs
- Use `httpx.MockTransport` or `responses` library
- Return realistic mock data based on actual API responses
**Frontend Testing:**
- Mock API services in component tests
- Test loading/error states
- Test user interactions (clicks, form submissions)
- Verify SSE stream handling
---
*Testing analysis: 2026-02-04*
**CRITICAL NOTE:** This codebase currently has **no automated tests**. All functionality relies on manual testing. Implementing a test suite should be a high priority, especially for:
- Authentication flows (security-critical)
- External API integrations (reliability-critical)
- Database operations (data integrity-critical)
- Streaming responses (complexity-critical)