Compare commits
8 Commits
f68a79bdb7
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
6ae36b51a0 | ||
|
|
f0f72cce36 | ||
|
|
32020a6c60 | ||
|
|
713a058c4f | ||
|
|
12f7d9ead1 | ||
|
|
ad39904dda | ||
|
|
1fd2e860b2 | ||
|
|
7cfad5baba |
16
.env.example
16
.env.example
@@ -14,9 +14,10 @@ JWT_SECRET_KEY=your-secret-key-here
|
||||
PAPERLESS_TOKEN=your-paperless-token
|
||||
BASE_URL=192.168.1.5:8000
|
||||
|
||||
# Ollama Configuration
|
||||
OLLAMA_URL=http://192.168.1.14:11434
|
||||
OLLAMA_HOST=http://192.168.1.14:11434
|
||||
# llama-server Configuration (OpenAI-compatible API)
|
||||
# If set, uses llama-server as the primary LLM backend with OpenAI as fallback
|
||||
LLAMA_SERVER_URL=http://192.168.1.213:8080/v1
|
||||
LLAMA_MODEL_NAME=llama-3.1-8b-instruct
|
||||
|
||||
# ChromaDB Configuration
|
||||
# For Docker: This is automatically set to /app/data/chromadb
|
||||
@@ -26,6 +27,9 @@ CHROMADB_PATH=./data/chromadb
|
||||
# OpenAI Configuration
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# Tavily Configuration (for web search)
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
|
||||
# Immich Configuration
|
||||
IMMICH_URL=http://192.168.1.5:2283
|
||||
IMMICH_API_KEY=your-immich-api-key
|
||||
@@ -44,3 +48,9 @@ OIDC_USE_DISCOVERY=true
|
||||
# OIDC_TOKEN_ENDPOINT=https://auth.example.com/api/oidc/token
|
||||
# OIDC_USERINFO_ENDPOINT=https://auth.example.com/api/oidc/userinfo
|
||||
# OIDC_JWKS_URI=https://auth.example.com/api/oidc/jwks
|
||||
|
||||
# YNAB Configuration
|
||||
# Get your Personal Access Token from https://app.ynab.com/settings/developer
|
||||
YNAB_ACCESS_TOKEN=your-ynab-personal-access-token
|
||||
# Optional: Specify a budget ID, or leave empty to use the default/first budget
|
||||
YNAB_BUDGET_ID=
|
||||
|
||||
6
.pre-commit-config.yaml
Normal file
6
.pre-commit-config.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
repos:
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
rev: v0.8.2
|
||||
hooks:
|
||||
- id: ruff # Linter
|
||||
- id: ruff-format # Formatter
|
||||
109
CLAUDE.md
Normal file
109
CLAUDE.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
SimbaRAG is a RAG (Retrieval-Augmented Generation) conversational AI system for querying information about Simba (a cat). It ingests documents from Paperless-NGX, stores embeddings in ChromaDB, and uses LLMs (Ollama or OpenAI) to answer questions.
|
||||
|
||||
## Commands
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Start dev environment with hot reload
|
||||
docker compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# View logs
|
||||
docker compose -f docker-compose.dev.yml logs -f raggr
|
||||
```
|
||||
|
||||
### Database Migrations (Aerich/Tortoise ORM)
|
||||
|
||||
```bash
|
||||
# Generate migration (must run in Docker with DB access)
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name describe_change
|
||||
|
||||
# Apply migrations (auto-runs on startup, manual if needed)
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich upgrade
|
||||
|
||||
# View migration history
|
||||
docker compose exec raggr aerich history
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
yarn install
|
||||
yarn build # Production build
|
||||
yarn dev # Dev server (rarely needed, backend serves frontend)
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
docker compose build raggr
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Docker Compose │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ raggr (port 8080) │ postgres (port 5432) │
|
||||
│ ├── Quart backend │ PostgreSQL 16 │
|
||||
│ ├── React frontend (served) │ │
|
||||
│ └── ChromaDB (volume) │ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Backend** (root directory):
|
||||
- `app.py` - Quart application entry, serves API and static frontend
|
||||
- `main.py` - RAG logic, document indexing, LLM interaction, LangChain agent
|
||||
- `llm.py` - LLM client with Ollama primary, OpenAI fallback
|
||||
- `aerich_config.py` - Database migration configuration
|
||||
- `blueprints/` - API routes organized as Quart blueprints
|
||||
- `users/` - OIDC auth, JWT tokens, RBAC with LDAP groups
|
||||
- `conversation/` - Chat conversations and message history
|
||||
- `rag/` - Document indexing endpoints (admin-only)
|
||||
- `config/` - Configuration modules
|
||||
- `oidc_config.py` - OIDC authentication configuration
|
||||
- `utils/` - Reusable utilities
|
||||
- `chunker.py` - Document chunking for embeddings
|
||||
- `cleaner.py` - PDF cleaning and summarization
|
||||
- `image_process.py` - Image description with LLM
|
||||
- `request.py` - Paperless-NGX API client
|
||||
- `scripts/` - Administrative and utility scripts
|
||||
- `add_user.py` - Create users manually
|
||||
- `user_message_stats.py` - User message statistics
|
||||
- `manage_vectorstore.py` - Vector store management CLI
|
||||
- `inspect_vector_store.py` - Inspect ChromaDB contents
|
||||
- `query.py` - Query generation utilities
|
||||
- `migrations/` - Database migration files
|
||||
|
||||
**Frontend** (`raggr-frontend/`):
|
||||
- React 19 with Rsbuild bundler
|
||||
- Tailwind CSS for styling
|
||||
- Built to `dist/`, served by backend at `/`
|
||||
|
||||
**Auth Flow**: LLDAP → Authelia (OIDC) → Backend JWT → Frontend localStorage
|
||||
|
||||
## Key Patterns
|
||||
|
||||
- All endpoints are async (`async def`)
|
||||
- Use `@jwt_refresh_token_required` for authenticated endpoints
|
||||
- Use `@admin_required` for admin-only endpoints (checks `lldap_admin` group)
|
||||
- Tortoise ORM models in `blueprints/*/models.py`
|
||||
- Frontend API services in `raggr-frontend/src/api/`
|
||||
|
||||
## Environment Variables
|
||||
|
||||
See `.env.example`. Key ones:
|
||||
- `DATABASE_URL` - PostgreSQL connection
|
||||
- `OIDC_*` - Authelia OIDC configuration
|
||||
- `OLLAMA_URL` - Local LLM server
|
||||
- `OPENAI_API_KEY` - Fallback LLM
|
||||
- `PAPERLESS_TOKEN` / `BASE_URL` - Document source
|
||||
110
DEV-README.md
110
DEV-README.md
@@ -1,110 +0,0 @@
|
||||
# Development Environment Setup
|
||||
|
||||
This guide explains how to run the application in development mode with hot reload enabled.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Development Mode (Hot Reload)
|
||||
|
||||
```bash
|
||||
# Start all services in development mode
|
||||
docker-compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# Or run in detached mode
|
||||
docker-compose -f docker-compose.dev.yml up -d --build
|
||||
```
|
||||
|
||||
### Production Mode
|
||||
|
||||
```bash
|
||||
# Start production services
|
||||
docker-compose up --build
|
||||
```
|
||||
|
||||
## What's Different in Dev Mode?
|
||||
|
||||
### Backend (Quart/Flask)
|
||||
- **Hot Reload**: Python code changes are automatically detected and the server restarts
|
||||
- **Source Mounted**: Your local `services/raggr` directory is mounted as a volume
|
||||
- **Debug Mode**: Flask runs with `debug=True` for better error messages
|
||||
- **Environment**: `FLASK_ENV=development` and `PYTHONUNBUFFERED=1` for immediate log output
|
||||
|
||||
### Frontend (React + rsbuild)
|
||||
- **Auto Rebuild**: Frontend automatically rebuilds when files change
|
||||
- **Watch Mode**: rsbuild runs in watch mode, rebuilding to `dist/` on save
|
||||
- **Source Mounted**: Your local `services/raggr/raggr-frontend` directory is mounted as a volume
|
||||
- **Served by Backend**: Built files are served by the backend, no separate dev server
|
||||
|
||||
## Ports
|
||||
|
||||
- **Application**: 8080 (accessible at `http://localhost:8080` or `http://YOUR_IP:8080`)
|
||||
|
||||
The backend serves both the API and the auto-rebuilt frontend, making it accessible from other machines on your network.
|
||||
|
||||
## Useful Commands
|
||||
|
||||
```bash
|
||||
# View logs
|
||||
docker-compose -f docker-compose.dev.yml logs -f
|
||||
|
||||
# View logs for specific service
|
||||
docker-compose -f docker-compose.dev.yml logs -f raggr-backend
|
||||
docker-compose -f docker-compose.dev.yml logs -f raggr-frontend
|
||||
|
||||
# Rebuild after dependency changes
|
||||
docker-compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# Stop all services
|
||||
docker-compose -f docker-compose.dev.yml down
|
||||
|
||||
# Stop and remove volumes (fresh start)
|
||||
docker-compose -f docker-compose.dev.yml down -v
|
||||
```
|
||||
|
||||
## Making Changes
|
||||
|
||||
### Backend Changes
|
||||
1. Edit any Python file in `services/raggr/`
|
||||
2. Save the file
|
||||
3. The Quart server will automatically restart
|
||||
4. Check logs to confirm reload
|
||||
|
||||
### Frontend Changes
|
||||
1. Edit any file in `services/raggr/raggr-frontend/src/`
|
||||
2. Save the file
|
||||
3. The browser will automatically refresh (Hot Module Replacement)
|
||||
4. No need to rebuild
|
||||
|
||||
### Dependency Changes
|
||||
|
||||
**Backend** (pyproject.toml):
|
||||
```bash
|
||||
# Rebuild the backend service
|
||||
docker-compose -f docker-compose.dev.yml up --build raggr-backend
|
||||
```
|
||||
|
||||
**Frontend** (package.json):
|
||||
```bash
|
||||
# Rebuild the frontend service
|
||||
docker-compose -f docker-compose.dev.yml up --build raggr-frontend
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port Already in Use
|
||||
If you see port binding errors, make sure no other services are running on ports 8080 or 3000.
|
||||
|
||||
### Changes Not Reflected
|
||||
1. Check if the file is properly mounted (check docker-compose.dev.yml volumes)
|
||||
2. Verify the file isn't in an excluded directory (node_modules, __pycache__)
|
||||
3. Check container logs for errors
|
||||
|
||||
### Frontend Not Connecting to Backend
|
||||
Make sure your frontend API calls point to the correct backend URL. If accessing from the same machine, use `http://localhost:8080`. If accessing from another device on the network, use `http://YOUR_IP:8080`.
|
||||
|
||||
## Notes
|
||||
|
||||
- Both services bind to `0.0.0.0` and expose ports, making them accessible on your network
|
||||
- Node modules and Python cache are excluded from volume mounts to use container versions
|
||||
- Database and ChromaDB data persist in Docker volumes across restarts
|
||||
- Access the app from any device on your network using your host machine's IP address
|
||||
@@ -25,6 +25,9 @@ RUN uv pip install --system -e .
|
||||
COPY *.py ./
|
||||
COPY blueprints ./blueprints
|
||||
COPY migrations ./migrations
|
||||
COPY utils ./utils
|
||||
COPY config ./config
|
||||
COPY scripts ./scripts
|
||||
COPY startup.sh ./
|
||||
RUN chmod +x startup.sh
|
||||
|
||||
371
README.md
371
README.md
@@ -1,7 +1,370 @@
|
||||
# simbarag
|
||||
# SimbaRAG 🐱
|
||||
|
||||
**Goal:** Learn how retrieval-augmented generation works and also create a neat little tool to ask about Simba's health.
|
||||
A Retrieval-Augmented Generation (RAG) conversational AI system for querying information about Simba the cat. Built with LangChain, ChromaDB, and modern web technologies.
|
||||
|
||||
**Current objectives:**
|
||||
## Features
|
||||
|
||||
- [ ] Successfully use RAG to ask a question about existing information (e.g. how many teeth has Simba had extracted)
|
||||
- 🤖 **Intelligent Conversations** - LangChain-powered agent with tool use and memory
|
||||
- 📚 **Document Retrieval** - RAG system using ChromaDB vector store
|
||||
- 🔍 **Web Search** - Integrated Tavily API for real-time web searches
|
||||
- 🔐 **OIDC Authentication** - Secure auth via Authelia with LDAP group support
|
||||
- 💬 **Multi-Conversation** - Manage multiple conversation threads per user
|
||||
- 🎨 **Modern UI** - React 19 frontend with Tailwind CSS
|
||||
- 🐳 **Docker Ready** - Containerized deployment with Docker Compose
|
||||
|
||||
## System Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Client Layer"
|
||||
Browser[Web Browser]
|
||||
end
|
||||
|
||||
subgraph "Frontend - React"
|
||||
UI[React UI<br/>Tailwind CSS]
|
||||
Auth[Auth Service]
|
||||
API[API Client]
|
||||
end
|
||||
|
||||
subgraph "Backend - Quart/Python"
|
||||
App[Quart App<br/>app.py]
|
||||
|
||||
subgraph "Blueprints"
|
||||
Users[Users Blueprint<br/>OIDC + JWT]
|
||||
Conv[Conversation Blueprint<br/>Chat Management]
|
||||
RAG[RAG Blueprint<br/>Document Indexing]
|
||||
end
|
||||
|
||||
Agent[LangChain Agent<br/>main.py]
|
||||
LLM[LLM Client<br/>llm.py]
|
||||
end
|
||||
|
||||
subgraph "Tools & Utilities"
|
||||
Search[Simba Search Tool]
|
||||
Web[Web Search Tool<br/>Tavily]
|
||||
end
|
||||
|
||||
subgraph "Data Layer"
|
||||
Postgres[(PostgreSQL<br/>Users & Conversations)]
|
||||
Chroma[(ChromaDB<br/>Vector Store)]
|
||||
end
|
||||
|
||||
subgraph "External Services"
|
||||
Authelia[Authelia<br/>OIDC Provider]
|
||||
LLDAP[LLDAP<br/>User Directory]
|
||||
Ollama[Ollama<br/>Local LLM]
|
||||
OpenAI[OpenAI<br/>Fallback LLM]
|
||||
Paperless[Paperless-NGX<br/>Documents]
|
||||
TavilyAPI[Tavily API<br/>Web Search]
|
||||
end
|
||||
|
||||
Browser --> UI
|
||||
UI --> Auth
|
||||
UI --> API
|
||||
API --> App
|
||||
|
||||
App --> Users
|
||||
App --> Conv
|
||||
App --> RAG
|
||||
|
||||
Conv --> Agent
|
||||
Agent --> Search
|
||||
Agent --> Web
|
||||
Agent --> LLM
|
||||
|
||||
Search --> Chroma
|
||||
Web --> TavilyAPI
|
||||
RAG --> Chroma
|
||||
RAG --> Paperless
|
||||
|
||||
Users --> Postgres
|
||||
Conv --> Postgres
|
||||
|
||||
Users --> Authelia
|
||||
Authelia --> LLDAP
|
||||
|
||||
LLM --> Ollama
|
||||
LLM -.Fallback.-> OpenAI
|
||||
|
||||
style Browser fill:#e1f5ff
|
||||
style UI fill:#fff3cd
|
||||
style App fill:#d4edda
|
||||
style Agent fill:#d4edda
|
||||
style Postgres fill:#f8d7da
|
||||
style Chroma fill:#f8d7da
|
||||
style Ollama fill:#e2e3e5
|
||||
style OpenAI fill:#e2e3e5
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker & Docker Compose
|
||||
- PostgreSQL (or use Docker)
|
||||
- Ollama (optional, for local LLM)
|
||||
- Paperless-NGX instance (for document source)
|
||||
|
||||
### Installation
|
||||
|
||||
1. **Clone the repository**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/yourusername/simbarag.git
|
||||
cd simbarag
|
||||
```
|
||||
|
||||
2. **Configure environment variables**
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your configuration
|
||||
```
|
||||
|
||||
3. **Start the services**
|
||||
|
||||
```bash
|
||||
# Development (local PostgreSQL only)
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# Or full Docker deployment
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. **Access the application**
|
||||
|
||||
Open `http://localhost:8080` in your browser.
|
||||
|
||||
## Development
|
||||
|
||||
### Local Development Setup
|
||||
|
||||
```bash
|
||||
# 1. Start PostgreSQL
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# 2. Set environment variables
|
||||
export DATABASE_URL="postgres://raggr:raggr_dev_password@localhost:5432/raggr"
|
||||
export CHROMADB_PATH="./chromadb"
|
||||
export $(grep -v '^#' .env | xargs)
|
||||
|
||||
# 3. Install dependencies
|
||||
pip install -r requirements.txt
|
||||
cd raggr-frontend && yarn install && yarn build && cd ..
|
||||
|
||||
# 4. Run migrations
|
||||
aerich upgrade
|
||||
|
||||
# 5. Start the server
|
||||
python app.py
|
||||
```
|
||||
|
||||
See [docs/development.md](docs/development.md) for detailed development guide.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
simbarag/
|
||||
├── app.py # Quart application entry point
|
||||
├── main.py # RAG logic & LangChain agent
|
||||
├── llm.py # LLM client with Ollama/OpenAI
|
||||
├── aerich_config.py # Database migration configuration
|
||||
│
|
||||
├── blueprints/ # API route blueprints
|
||||
│ ├── users/ # Authentication & authorization
|
||||
│ ├── conversation/ # Chat conversations
|
||||
│ └── rag/ # Document indexing
|
||||
│
|
||||
├── config/ # Configuration modules
|
||||
│ └── oidc_config.py # OIDC authentication settings
|
||||
│
|
||||
├── utils/ # Reusable utilities
|
||||
│ ├── chunker.py # Document chunking for embeddings
|
||||
│ ├── cleaner.py # PDF cleaning and summarization
|
||||
│ ├── image_process.py # Image description with LLM
|
||||
│ └── request.py # Paperless-NGX API client
|
||||
│
|
||||
├── scripts/ # Administrative scripts
|
||||
│ ├── add_user.py
|
||||
│ ├── user_message_stats.py
|
||||
│ ├── manage_vectorstore.py
|
||||
│ └── inspect_vector_store.py
|
||||
│
|
||||
├── raggr-frontend/ # React frontend
|
||||
│ └── src/
|
||||
│
|
||||
├── migrations/ # Database migrations
|
||||
│
|
||||
├── docs/ # Documentation
|
||||
│ ├── index.md # Documentation hub
|
||||
│ ├── development.md # Development guide
|
||||
│ ├── deployment.md # Deployment & migrations
|
||||
│ ├── VECTORSTORE.md # Vector store management
|
||||
│ ├── MIGRATIONS.md # Migration reference
|
||||
│ └── authentication.md # Authentication setup
|
||||
│
|
||||
├── docker-compose.yml # Production compose
|
||||
├── docker-compose.dev.yml # Development compose
|
||||
├── Dockerfile # Production Dockerfile
|
||||
├── Dockerfile.dev # Development Dockerfile
|
||||
├── CLAUDE.md # AI assistant instructions
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Key Technologies
|
||||
|
||||
### Backend
|
||||
- **Quart** - Async Python web framework
|
||||
- **LangChain** - Agent framework with tool use
|
||||
- **Tortoise ORM** - Async ORM for PostgreSQL
|
||||
- **Aerich** - Database migration tool
|
||||
- **ChromaDB** - Vector database for embeddings
|
||||
- **OpenAI** - Embeddings & LLM (fallback)
|
||||
- **Ollama** - Local LLM (primary)
|
||||
|
||||
### Frontend
|
||||
- **React 19** - UI framework
|
||||
- **Rsbuild** - Fast bundler
|
||||
- **Tailwind CSS** - Utility-first styling
|
||||
- **Axios** - HTTP client
|
||||
|
||||
### Authentication
|
||||
- **Authelia** - OIDC provider
|
||||
- **LLDAP** - Lightweight LDAP server
|
||||
- **JWT** - Token-based auth
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Authentication
|
||||
- `GET /api/user/oidc/login` - Initiate OIDC login
|
||||
- `GET /api/user/oidc/callback` - OIDC callback handler
|
||||
- `POST /api/user/refresh` - Refresh JWT token
|
||||
|
||||
### Conversations
|
||||
- `POST /api/conversation/` - Create conversation
|
||||
- `GET /api/conversation/` - List conversations
|
||||
- `GET /api/conversation/<id>` - Get conversation with messages
|
||||
- `POST /api/conversation/query` - Send message and get response
|
||||
|
||||
### RAG (Admin Only)
|
||||
- `GET /api/rag/stats` - Vector store statistics
|
||||
- `POST /api/rag/index` - Index new documents
|
||||
- `POST /api/rag/reindex` - Clear and reindex all
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `DATABASE_URL` | PostgreSQL connection string | `postgres://...` |
|
||||
| `CHROMADB_PATH` | ChromaDB storage path | `./chromadb` |
|
||||
| `OLLAMA_URL` | Ollama server URL | `http://localhost:11434` |
|
||||
| `OPENAI_API_KEY` | OpenAI API key | - |
|
||||
| `PAPERLESS_TOKEN` | Paperless-NGX API token | - |
|
||||
| `BASE_URL` | Paperless-NGX base URL | - |
|
||||
| `OIDC_ISSUER` | OIDC provider URL | - |
|
||||
| `OIDC_CLIENT_ID` | OIDC client ID | - |
|
||||
| `OIDC_CLIENT_SECRET` | OIDC client secret | - |
|
||||
| `JWT_SECRET_KEY` | JWT signing key | - |
|
||||
| `TAVILY_KEY` | Tavily web search API key | - |
|
||||
|
||||
See `.env.example` for full list.
|
||||
|
||||
## Scripts
|
||||
|
||||
### User Management
|
||||
```bash
|
||||
# Add a new user
|
||||
python scripts/add_user.py
|
||||
|
||||
# View message statistics
|
||||
python scripts/user_message_stats.py
|
||||
```
|
||||
|
||||
### Vector Store Management
|
||||
```bash
|
||||
# Show vector store statistics
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index new documents from Paperless
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Clear and reindex everything
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
|
||||
# Inspect vector store contents
|
||||
python scripts/inspect_vector_store.py
|
||||
```
|
||||
|
||||
See [docs/vectorstore.md](docs/vectorstore.md) for details.
|
||||
|
||||
## Database Migrations
|
||||
|
||||
```bash
|
||||
# Generate a new migration
|
||||
aerich migrate --name "describe_your_changes"
|
||||
|
||||
# Apply pending migrations
|
||||
aerich upgrade
|
||||
|
||||
# View migration history
|
||||
aerich history
|
||||
|
||||
# Rollback last migration
|
||||
aerich downgrade
|
||||
```
|
||||
|
||||
See [docs/deployment.md](docs/deployment.md) for detailed migration workflows.
|
||||
|
||||
## LangChain Agent
|
||||
|
||||
The conversational agent has access to two tools:
|
||||
|
||||
1. **simba_search** - Query the vector store for Simba's documents
|
||||
- Used for: Medical records, veterinary history, factual information
|
||||
|
||||
2. **web_search** - Search the web via Tavily API
|
||||
- Used for: Recent events, external knowledge, general questions
|
||||
|
||||
The agent automatically selects the appropriate tool based on the user's query.
|
||||
|
||||
## Authentication Flow
|
||||
|
||||
```
|
||||
User → Authelia (OIDC) → Backend (JWT) → Frontend (localStorage)
|
||||
↓
|
||||
LLDAP
|
||||
```
|
||||
|
||||
1. User clicks "Login"
|
||||
2. Frontend redirects to Authelia
|
||||
3. User authenticates via Authelia (backed by LLDAP)
|
||||
4. Authelia redirects back with authorization code
|
||||
5. Backend exchanges code for OIDC tokens
|
||||
6. Backend issues JWT tokens
|
||||
7. Frontend stores tokens in localStorage
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Make your changes
|
||||
4. Run tests and linting
|
||||
5. Submit a pull request
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Development Guide](docs/development.md) - Setup and development workflow
|
||||
- [Deployment Guide](docs/deployment.md) - Deployment and migrations
|
||||
- [Vector Store Guide](docs/vectorstore.md) - Managing the vector database
|
||||
- [Authentication Guide](docs/authentication.md) - OIDC and LDAP setup
|
||||
|
||||
## License
|
||||
|
||||
[Your License Here]
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- Built for Simba, the most important cat in the world 🐱
|
||||
- Powered by LangChain, ChromaDB, and the open-source community
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
# Database configuration with environment variable support
|
||||
# Use DATABASE_PATH for relative paths or DATABASE_URL for full connection strings
|
||||
@@ -1,5 +1,6 @@
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from quart import Quart, jsonify, render_template, request, send_from_directory
|
||||
from quart_jwt_extended import JWTManager, get_jwt_identity, jwt_refresh_token_required
|
||||
from tortoise.contrib.quart import register_tortoise
|
||||
@@ -11,6 +12,9 @@ import blueprints.users
|
||||
import blueprints.users.models
|
||||
from main import consult_simba_oracle
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
app = Quart(
|
||||
__name__,
|
||||
static_folder="raggr-frontend/dist/static",
|
||||
@@ -84,7 +84,15 @@ Upcoming Appointments:
|
||||
- Routine Examination: Due 6/1/2026
|
||||
- FVRCP-3yr Vaccine: Due 10/2/2026
|
||||
|
||||
IMPORTANT: When users ask factual questions about Simba's health, medical history, veterinary visits, medications, weight, or any information that would be in documents, you MUST use the simba_search tool to retrieve accurate information before answering. Do not rely on general knowledge - always search the documents for factual questions.""",
|
||||
IMPORTANT: When users ask factual questions about Simba's health, medical history, veterinary visits, medications, weight, or any information that would be in documents, you MUST use the simba_search tool to retrieve accurate information before answering. Do not rely on general knowledge - always search the documents for factual questions.
|
||||
|
||||
BUDGET & FINANCE (YNAB Integration):
|
||||
You have access to Ryan's budget data through YNAB (You Need A Budget). When users ask about financial matters, use the appropriate YNAB tools:
|
||||
- Use ynab_budget_summary for overall budget health and status questions
|
||||
- Use ynab_search_transactions to find specific purchases or spending at particular stores
|
||||
- Use ynab_category_spending to analyze spending by category for a month
|
||||
- Use ynab_insights to provide spending trends, patterns, and recommendations
|
||||
Always use these tools when asked about budgets, spending, transactions, or financial health.""",
|
||||
}
|
||||
]
|
||||
|
||||
295
blueprints/conversation/agents.py
Normal file
295
blueprints/conversation/agents.py
Normal file
@@ -0,0 +1,295 @@
|
||||
import os
|
||||
from typing import cast
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from langchain.agents import create_agent
|
||||
from langchain.chat_models import BaseChatModel
|
||||
from langchain.tools import tool
|
||||
from langchain_openai import ChatOpenAI
|
||||
from tavily import AsyncTavilyClient
|
||||
|
||||
from blueprints.rag.logic import query_vector_store
|
||||
from utils.ynab_service import YNABService
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
# Configure LLM with llama-server or OpenAI fallback
|
||||
llama_url = os.getenv("LLAMA_SERVER_URL")
|
||||
if llama_url:
|
||||
llama_chat = ChatOpenAI(
|
||||
base_url=llama_url,
|
||||
api_key="not-needed",
|
||||
model=os.getenv("LLAMA_MODEL_NAME", "llama-3.1-8b-instruct"),
|
||||
)
|
||||
else:
|
||||
llama_chat = None
|
||||
|
||||
openai_fallback = ChatOpenAI(model="gpt-5-mini")
|
||||
model_with_fallback = cast(
|
||||
BaseChatModel,
|
||||
llama_chat.with_fallbacks([openai_fallback]) if llama_chat else openai_fallback,
|
||||
)
|
||||
client = AsyncTavilyClient(api_key=os.getenv("TAVILY_API_KEY", ""))
|
||||
|
||||
# Initialize YNAB service (will only work if YNAB_ACCESS_TOKEN is set)
|
||||
try:
|
||||
ynab_service = YNABService()
|
||||
ynab_enabled = True
|
||||
except (ValueError, Exception) as e:
|
||||
print(f"YNAB service not initialized: {e}")
|
||||
ynab_enabled = False
|
||||
|
||||
|
||||
@tool
|
||||
async def web_search(query: str) -> str:
|
||||
"""Search the web for current information using Tavily.
|
||||
|
||||
Use this tool when you need to:
|
||||
- Find current information not in the knowledge base
|
||||
- Look up recent events, news, or updates
|
||||
- Verify facts or get additional context
|
||||
- Search for information outside of Simba's documents
|
||||
|
||||
Args:
|
||||
query: The search query to look up on the web
|
||||
|
||||
Returns:
|
||||
Search results from the web with titles, content, and source URLs
|
||||
"""
|
||||
response = await client.search(query=query, search_depth="basic")
|
||||
results = response.get("results", [])
|
||||
|
||||
if not results:
|
||||
return "No results found for the query."
|
||||
|
||||
formatted = "\n\n".join(
|
||||
[
|
||||
f"**{result['title']}**\n{result['content']}\nSource: {result['url']}"
|
||||
for result in results[:5]
|
||||
]
|
||||
)
|
||||
return formatted
|
||||
|
||||
|
||||
@tool(response_format="content_and_artifact")
|
||||
async def simba_search(query: str):
|
||||
"""Search through Simba's medical records, veterinary documents, and personal information.
|
||||
|
||||
Use this tool whenever the user asks questions about:
|
||||
- Simba's health history, medical records, or veterinary visits
|
||||
- Medications, treatments, or diagnoses
|
||||
- Weight, diet, or physical characteristics over time
|
||||
- Veterinary recommendations or advice
|
||||
- Ryan's (the owner's) information related to Simba
|
||||
- Any factual information that would be found in documents
|
||||
|
||||
Args:
|
||||
query: The user's question or information need about Simba
|
||||
|
||||
Returns:
|
||||
Relevant information from Simba's documents
|
||||
"""
|
||||
print(f"[SIMBA SEARCH] Tool called with query: {query}")
|
||||
serialized, docs = await query_vector_store(query=query)
|
||||
print(f"[SIMBA SEARCH] Found {len(docs)} documents")
|
||||
print(f"[SIMBA SEARCH] Serialized result length: {len(serialized)}")
|
||||
print(f"[SIMBA SEARCH] First 200 chars: {serialized[:200]}")
|
||||
return serialized, docs
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_budget_summary() -> str:
|
||||
"""Get overall budget summary and health status from YNAB.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Overall budget health or status
|
||||
- How much money is to be budgeted
|
||||
- Total budget amounts or spending
|
||||
- General budget overview questions
|
||||
|
||||
Returns:
|
||||
Summary of budget health, to-be-budgeted amount, total budgeted,
|
||||
total activity, and available amounts.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
summary = ynab_service.get_budget_summary()
|
||||
return summary["summary"]
|
||||
except Exception as e:
|
||||
return f"Error fetching budget summary: {str(e)}"
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_search_transactions(
|
||||
start_date: str = "",
|
||||
end_date: str = "",
|
||||
category_name: str = "",
|
||||
payee_name: str = "",
|
||||
) -> str:
|
||||
"""Search YNAB transactions by date range, category, or payee.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Specific transactions or purchases
|
||||
- Spending at a particular store or payee
|
||||
- Transactions in a specific category
|
||||
- What was spent during a time period
|
||||
|
||||
Args:
|
||||
start_date: Start date in YYYY-MM-DD format (optional, defaults to 30 days ago)
|
||||
end_date: End date in YYYY-MM-DD format (optional, defaults to today)
|
||||
category_name: Filter by category name (optional, partial match)
|
||||
payee_name: Filter by payee/store name (optional, partial match)
|
||||
|
||||
Returns:
|
||||
List of matching transactions with dates, amounts, categories, and payees.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
result = ynab_service.get_transactions(
|
||||
start_date=start_date or None,
|
||||
end_date=end_date or None,
|
||||
category_name=category_name or None,
|
||||
payee_name=payee_name or None,
|
||||
)
|
||||
|
||||
if result["count"] == 0:
|
||||
return "No transactions found matching the specified criteria."
|
||||
|
||||
# Format transactions for readability
|
||||
txn_list = []
|
||||
for txn in result["transactions"][:10]: # Limit to 10 for readability
|
||||
txn_list.append(
|
||||
f"- {txn['date']}: {txn['payee']} - ${abs(txn['amount']):.2f} ({txn['category'] or 'Uncategorized'})"
|
||||
)
|
||||
|
||||
return (
|
||||
f"Found {result['count']} transactions from {result['start_date']} to {result['end_date']}. "
|
||||
f"Total: ${abs(result['total_amount']):.2f}\n\n"
|
||||
+ "\n".join(txn_list)
|
||||
+ (
|
||||
f"\n\n(Showing first 10 of {result['count']} transactions)"
|
||||
if result["count"] > 10
|
||||
else ""
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
return f"Error searching transactions: {str(e)}"
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_category_spending(month: str = "") -> str:
|
||||
"""Get spending breakdown by category for a specific month.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Spending by category
|
||||
- What categories were overspent
|
||||
- Monthly spending breakdown
|
||||
- Budget vs actual spending for a month
|
||||
|
||||
Args:
|
||||
month: Month in YYYY-MM format (optional, defaults to current month)
|
||||
|
||||
Returns:
|
||||
Spending breakdown by category with budgeted, spent, and available amounts.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
result = ynab_service.get_category_spending(month=month or None)
|
||||
|
||||
summary = (
|
||||
f"Budget spending for {result['month']}:\n"
|
||||
f"Total budgeted: ${result['total_budgeted']:.2f}\n"
|
||||
f"Total spent: ${result['total_spent']:.2f}\n"
|
||||
f"Total available: ${result['total_available']:.2f}\n"
|
||||
)
|
||||
|
||||
if result["overspent_categories"]:
|
||||
summary += (
|
||||
f"\nOverspent categories ({len(result['overspent_categories'])}):\n"
|
||||
)
|
||||
for cat in result["overspent_categories"][:5]:
|
||||
summary += f"- {cat['name']}: Budgeted ${cat['budgeted']:.2f}, Spent ${cat['spent']:.2f}, Over by ${cat['overspent_by']:.2f}\n"
|
||||
|
||||
# Add top spending categories
|
||||
summary += "\nTop spending categories:\n"
|
||||
for cat in result["categories"][:10]:
|
||||
if cat["activity"] < 0: # Only show spending (negative activity)
|
||||
summary += f"- {cat['category']}: ${abs(cat['activity']):.2f} (budgeted: ${cat['budgeted']:.2f}, available: ${cat['available']:.2f})\n"
|
||||
|
||||
return summary
|
||||
except Exception as e:
|
||||
return f"Error fetching category spending: {str(e)}"
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_insights(months_back: int = 3) -> str:
|
||||
"""Generate insights about spending patterns and budget health over time.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Spending trends or patterns
|
||||
- Budget recommendations
|
||||
- Which categories are frequently overspent
|
||||
- How current spending compares to past months
|
||||
- Overall budget health analysis
|
||||
|
||||
Args:
|
||||
months_back: Number of months to analyze (default 3, max 6)
|
||||
|
||||
Returns:
|
||||
Insights about spending trends, frequently overspent categories,
|
||||
and personalized recommendations.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
# Limit to reasonable range
|
||||
months_back = min(max(1, months_back), 6)
|
||||
result = ynab_service.get_spending_insights(months_back=months_back)
|
||||
|
||||
if "error" in result:
|
||||
return result["error"]
|
||||
|
||||
summary = (
|
||||
f"Spending insights for the last {months_back} months:\n\n"
|
||||
f"Average monthly spending: ${result['average_monthly_spending']:.2f}\n"
|
||||
f"Current month spending: ${result['current_month_spending']:.2f}\n"
|
||||
f"Spending trend: {result['spending_trend']}\n"
|
||||
)
|
||||
|
||||
if result["frequently_overspent_categories"]:
|
||||
summary += "\nFrequently overspent categories:\n"
|
||||
for cat in result["frequently_overspent_categories"][:5]:
|
||||
summary += f"- {cat['category']}: overspent in {cat['months_overspent']} of {months_back} months\n"
|
||||
|
||||
if result["recommendations"]:
|
||||
summary += "\nRecommendations:\n"
|
||||
for rec in result["recommendations"]:
|
||||
summary += f"- {rec}\n"
|
||||
|
||||
return summary
|
||||
except Exception as e:
|
||||
return f"Error generating insights: {str(e)}"
|
||||
|
||||
|
||||
# Create tools list based on what's available
|
||||
tools = [simba_search, web_search]
|
||||
if ynab_enabled:
|
||||
tools.extend(
|
||||
[
|
||||
ynab_budget_summary,
|
||||
ynab_search_transactions,
|
||||
ynab_category_spending,
|
||||
ynab_insights,
|
||||
]
|
||||
)
|
||||
|
||||
# Llama 3.1 supports native function calling via OpenAI-compatible API
|
||||
main_agent = create_agent(model=model_with_fallback, tools=tools)
|
||||
@@ -2,6 +2,7 @@ from quart import Blueprint, jsonify
|
||||
from quart_jwt_extended import jwt_refresh_token_required
|
||||
|
||||
from .logic import get_vector_store_stats, index_documents, vector_store
|
||||
from blueprints.users.decorators import admin_required
|
||||
|
||||
rag_blueprint = Blueprint("rag_api", __name__, url_prefix="/api/rag")
|
||||
|
||||
@@ -15,9 +16,9 @@ async def get_stats():
|
||||
|
||||
|
||||
@rag_blueprint.post("/index")
|
||||
@jwt_refresh_token_required
|
||||
@admin_required
|
||||
async def trigger_index():
|
||||
"""Trigger indexing of documents from Paperless-NGX."""
|
||||
"""Trigger indexing of documents from Paperless-NGX. Admin only."""
|
||||
try:
|
||||
await index_documents()
|
||||
stats = get_vector_store_stats()
|
||||
@@ -27,9 +28,9 @@ async def trigger_index():
|
||||
|
||||
|
||||
@rag_blueprint.post("/reindex")
|
||||
@jwt_refresh_token_required
|
||||
@admin_required
|
||||
async def trigger_reindex():
|
||||
"""Clear and reindex all documents."""
|
||||
"""Clear and reindex all documents. Admin only."""
|
||||
try:
|
||||
# Clear existing documents
|
||||
collection = vector_store._collection
|
||||
@@ -1,8 +1,12 @@
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
from dotenv import load_dotenv
|
||||
import httpx
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class PaperlessNGXService:
|
||||
def __init__(self):
|
||||
@@ -1,6 +1,7 @@
|
||||
import datetime
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from langchain_chroma import Chroma
|
||||
from langchain_core.documents import Document
|
||||
from langchain_openai import OpenAIEmbeddings
|
||||
@@ -8,6 +9,9 @@ from langchain_text_splitters import RecursiveCharacterTextSplitter
|
||||
|
||||
from .fetchers import PaperlessNGXService
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
|
||||
|
||||
vector_store = Chroma(
|
||||
@@ -7,7 +7,7 @@ from quart_jwt_extended import (
|
||||
)
|
||||
from .models import User
|
||||
from .oidc_service import OIDCUserService
|
||||
from oidc_config import oidc_config
|
||||
from config.oidc_config import oidc_config
|
||||
import secrets
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
@@ -60,7 +60,7 @@ async def oidc_login():
|
||||
"client_id": oidc_config.client_id,
|
||||
"response_type": "code",
|
||||
"redirect_uri": oidc_config.redirect_uri,
|
||||
"scope": "openid email profile",
|
||||
"scope": "openid email profile groups",
|
||||
"state": state,
|
||||
"code_challenge": code_challenge,
|
||||
"code_challenge_method": "S256",
|
||||
@@ -115,7 +115,9 @@ async def oidc_callback():
|
||||
token_response = await client.post(token_endpoint, data=token_data)
|
||||
|
||||
if token_response.status_code != 200:
|
||||
return jsonify({"error": f"Failed to exchange code for token: {token_response.text}"}), 400
|
||||
return jsonify(
|
||||
{"error": f"Failed to exchange code for token: {token_response.text}"}
|
||||
), 400
|
||||
|
||||
tokens = token_response.json()
|
||||
|
||||
@@ -141,7 +143,13 @@ async def oidc_callback():
|
||||
return jsonify(
|
||||
access_token=access_token,
|
||||
refresh_token=refresh_token,
|
||||
user={"id": str(user.id), "username": user.username, "email": user.email},
|
||||
user={
|
||||
"id": str(user.id),
|
||||
"username": user.username,
|
||||
"email": user.email,
|
||||
"groups": user.ldap_groups,
|
||||
"is_admin": user.is_admin(),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
26
blueprints/users/decorators.py
Normal file
26
blueprints/users/decorators.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""
|
||||
Authentication decorators for role-based access control.
|
||||
"""
|
||||
|
||||
from functools import wraps
|
||||
from quart import jsonify
|
||||
from quart_jwt_extended import jwt_refresh_token_required, get_jwt_identity
|
||||
from .models import User
|
||||
|
||||
|
||||
def admin_required(fn):
|
||||
"""
|
||||
Decorator that requires the user to be an admin (member of lldap_admin group).
|
||||
Must be used on async route handlers.
|
||||
"""
|
||||
|
||||
@wraps(fn)
|
||||
@jwt_refresh_token_required
|
||||
async def wrapper(*args, **kwargs):
|
||||
user_id = get_jwt_identity()
|
||||
user = await User.get_or_none(id=user_id)
|
||||
if not user or not user.is_admin():
|
||||
return jsonify({"error": "Admin access required"}), 403
|
||||
return await fn(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
@@ -12,8 +12,13 @@ class User(Model):
|
||||
email = fields.CharField(max_length=100, unique=True)
|
||||
|
||||
# OIDC fields
|
||||
oidc_subject = fields.CharField(max_length=255, unique=True, null=True, index=True) # "sub" claim from OIDC
|
||||
auth_provider = fields.CharField(max_length=50, default="local") # "local" or "oidc"
|
||||
oidc_subject = fields.CharField(
|
||||
max_length=255, unique=True, null=True, index=True
|
||||
) # "sub" claim from OIDC
|
||||
auth_provider = fields.CharField(
|
||||
max_length=50, default="local"
|
||||
) # "local" or "oidc"
|
||||
ldap_groups = fields.JSONField(default=[]) # LDAP groups from OIDC claims
|
||||
|
||||
created_at = fields.DatetimeField(auto_now_add=True)
|
||||
updated_at = fields.DatetimeField(auto_now=True)
|
||||
@@ -21,6 +26,14 @@ class User(Model):
|
||||
class Meta:
|
||||
table = "users"
|
||||
|
||||
def has_group(self, group: str) -> bool:
|
||||
"""Check if user belongs to a specific LDAP group."""
|
||||
return group in (self.ldap_groups or [])
|
||||
|
||||
def is_admin(self) -> bool:
|
||||
"""Check if user is an admin (member of lldap_admin group)."""
|
||||
return self.has_group("lldap_admin")
|
||||
|
||||
def set_password(self, plain_password: str):
|
||||
self.password = bcrypt.hashpw(
|
||||
plain_password.encode("utf-8"),
|
||||
@@ -1,6 +1,7 @@
|
||||
"""
|
||||
OIDC User Management Service
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional
|
||||
from uuid import uuid4
|
||||
from .models import User
|
||||
@@ -31,10 +32,10 @@ class OIDCUserService:
|
||||
# Update user info from latest claims (optional)
|
||||
user.email = claims.get("email", user.email)
|
||||
user.username = (
|
||||
claims.get("preferred_username")
|
||||
or claims.get("name")
|
||||
or user.username
|
||||
claims.get("preferred_username") or claims.get("name") or user.username
|
||||
)
|
||||
# Update LDAP groups from claims
|
||||
user.ldap_groups = claims.get("groups", [])
|
||||
await user.save()
|
||||
return user
|
||||
|
||||
@@ -47,6 +48,7 @@ class OIDCUserService:
|
||||
user.oidc_subject = oidc_subject
|
||||
user.auth_provider = "oidc"
|
||||
user.password = None # Clear password
|
||||
user.ldap_groups = claims.get("groups", [])
|
||||
await user.save()
|
||||
return user
|
||||
|
||||
@@ -58,14 +60,17 @@ class OIDCUserService:
|
||||
or f"user_{oidc_subject[:8]}"
|
||||
)
|
||||
|
||||
# Extract LDAP groups from claims
|
||||
groups = claims.get("groups", [])
|
||||
|
||||
user = await User.create(
|
||||
id=uuid4(),
|
||||
username=username,
|
||||
email=email
|
||||
or f"{oidc_subject}@oidc.local", # Fallback if no email claim
|
||||
email=email or f"{oidc_subject}@oidc.local", # Fallback if no email claim
|
||||
oidc_subject=oidc_subject,
|
||||
auth_provider="oidc",
|
||||
password=None,
|
||||
ldap_groups=groups,
|
||||
)
|
||||
|
||||
return user
|
||||
@@ -1,13 +0,0 @@
|
||||
import os
|
||||
|
||||
from llm import LLMClient
|
||||
|
||||
USE_OPENAI = os.getenv("OLLAMA_URL")
|
||||
|
||||
|
||||
class Classifier:
|
||||
def __init__(self):
|
||||
self.llm_client = LLMClient()
|
||||
|
||||
def classify_query_by_action(self, query):
|
||||
_prompt = "Classify the query into one of the following options: "
|
||||
0
config/__init__.py
Normal file
0
config/__init__.py
Normal file
@@ -1,11 +1,16 @@
|
||||
"""
|
||||
OIDC Configuration for Authelia Integration
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Dict, Any
|
||||
from authlib.jose import jwt
|
||||
from authlib.jose.errors import JoseError
|
||||
import httpx
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class OIDCConfig:
|
||||
@@ -15,52 +15,56 @@ services:
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
raggr:
|
||||
build:
|
||||
context: ./services/raggr
|
||||
dockerfile: Dockerfile.dev
|
||||
image: torrtle/simbarag:dev
|
||||
ports:
|
||||
- "8080:8080"
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
- PAPERLESS_TOKEN=${PAPERLESS_TOKEN}
|
||||
- BASE_URL=${BASE_URL}
|
||||
- OLLAMA_URL=${OLLAMA_URL:-http://localhost:11434}
|
||||
- CHROMADB_PATH=/app/data/chromadb
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
|
||||
- OIDC_ISSUER=${OIDC_ISSUER}
|
||||
- OIDC_CLIENT_ID=${OIDC_CLIENT_ID}
|
||||
- OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET}
|
||||
- OIDC_REDIRECT_URI=${OIDC_REDIRECT_URI}
|
||||
- OIDC_USE_DISCOVERY=${OIDC_USE_DISCOVERY:-true}
|
||||
- DATABASE_URL=postgres://raggr:raggr_dev_password@postgres:5432/raggr
|
||||
- FLASK_ENV=development
|
||||
- PYTHONUNBUFFERED=1
|
||||
- NODE_ENV=development
|
||||
- TAVILY_KEY=${TAVILIY_KEY}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- chromadb_data:/app/data/chromadb
|
||||
develop:
|
||||
watch:
|
||||
# Sync+restart on any file change under services/raggr
|
||||
- action: sync+restart
|
||||
path: ./services/raggr
|
||||
target: /app
|
||||
ignore:
|
||||
- __pycache__/
|
||||
- "*.pyc"
|
||||
- "*.pyo"
|
||||
- "*.pyd"
|
||||
- .git/
|
||||
- chromadb/
|
||||
- node_modules/
|
||||
- raggr-frontend/dist/
|
||||
# raggr service disabled - run locally for development
|
||||
# raggr:
|
||||
# build:
|
||||
# context: .
|
||||
# dockerfile: Dockerfile.dev
|
||||
# image: torrtle/simbarag:dev
|
||||
# ports:
|
||||
# - "8080:8080"
|
||||
# env_file:
|
||||
# - .env
|
||||
# environment:
|
||||
# - PAPERLESS_TOKEN=${PAPERLESS_TOKEN}
|
||||
# - BASE_URL=${BASE_URL}
|
||||
# - OLLAMA_URL=${OLLAMA_URL:-http://localhost:11434}
|
||||
# - CHROMADB_PATH=/app/data/chromadb
|
||||
# - OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
# - JWT_SECRET_KEY=${JWT_SECRET_KEY}
|
||||
# - OIDC_ISSUER=${OIDC_ISSUER}
|
||||
# - OIDC_CLIENT_ID=${OIDC_CLIENT_ID}
|
||||
# - OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET}
|
||||
# - OIDC_REDIRECT_URI=${OIDC_REDIRECT_URI}
|
||||
# - OIDC_USE_DISCOVERY=${OIDC_USE_DISCOVERY:-true}
|
||||
# - DATABASE_URL=postgres://raggr:raggr_dev_password@postgres:5432/raggr
|
||||
# - FLASK_ENV=development
|
||||
# - PYTHONUNBUFFERED=1
|
||||
# - NODE_ENV=development
|
||||
# - TAVILY_KEY=${TAVILIY_KEY}
|
||||
# depends_on:
|
||||
# postgres:
|
||||
# condition: service_healthy
|
||||
# volumes:
|
||||
# - chromadb_data:/app/data/chromadb
|
||||
# - ./migrations:/app/migrations # Bind mount for migrations (bidirectional)
|
||||
# develop:
|
||||
# watch:
|
||||
# # Sync+restart on any file change in root directory
|
||||
# - action: sync+restart
|
||||
# path: .
|
||||
# target: /app
|
||||
# ignore:
|
||||
# - __pycache__/
|
||||
# - "*.pyc"
|
||||
# - "*.pyo"
|
||||
# - "*.pyd"
|
||||
# - .git/
|
||||
# - chromadb/
|
||||
# - node_modules/
|
||||
# - raggr-frontend/dist/
|
||||
# - docs/
|
||||
# - .venv/
|
||||
|
||||
volumes:
|
||||
chromadb_data:
|
||||
|
||||
@@ -20,7 +20,7 @@ services:
|
||||
|
||||
raggr:
|
||||
build:
|
||||
context: ./services/raggr
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
image: torrtle/simbarag:latest
|
||||
ports:
|
||||
|
||||
53
docs/TASKS.md
Normal file
53
docs/TASKS.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Tasks & Feature Requests
|
||||
|
||||
## Feature Requests
|
||||
|
||||
### YNAB Integration (Admin-Only)
|
||||
- **Description**: Integration with YNAB (You Need A Budget) API to enable financial data queries and insights
|
||||
- **Requirements**:
|
||||
- Admin-guarded endpoint (requires `lldap_admin` group)
|
||||
- YNAB API token configuration in environment variables
|
||||
- Sync budget data, transactions, and categories
|
||||
- Store YNAB data for RAG queries
|
||||
- **Endpoints**:
|
||||
- `POST /api/admin/ynab/sync` - Trigger YNAB data sync
|
||||
- `GET /api/admin/ynab/status` - Check sync status and last update
|
||||
- `GET /api/admin/ynab/budgets` - List available budgets
|
||||
- **Implementation Notes**:
|
||||
- Use YNAB API v1 (https://api.youneedabudget.com/v1)
|
||||
- Consider rate limiting (200 requests per hour)
|
||||
- Store transaction data in PostgreSQL with appropriate indexing
|
||||
- Index transaction descriptions and categories in ChromaDB for RAG queries
|
||||
|
||||
### Money Insights
|
||||
- **Description**: AI-powered financial insights and analysis based on YNAB data
|
||||
- **Features**:
|
||||
- Spending pattern analysis
|
||||
- Budget vs. actual comparisons
|
||||
- Category-based spending trends
|
||||
- Anomaly detection (unusual transactions)
|
||||
- Natural language queries like "How much did I spend on groceries last month?"
|
||||
- Month-over-month and year-over-year comparisons
|
||||
- **Implementation Notes**:
|
||||
- Leverage existing LangChain agent architecture
|
||||
- Add custom tools for financial calculations
|
||||
- Use LLM to generate insights and summaries
|
||||
- Create visualizations or data exports for frontend display
|
||||
|
||||
## Backlog
|
||||
|
||||
- [ ] YNAB API client module
|
||||
- [ ] YNAB data models (Budget, Transaction, Category, Account)
|
||||
- [ ] Database schema for financial data
|
||||
- [ ] YNAB sync background job/scheduler
|
||||
- [ ] Financial insights LangChain tools
|
||||
- [ ] Admin UI for YNAB configuration
|
||||
- [ ] Frontend components for money insights display
|
||||
|
||||
## Technical Debt
|
||||
|
||||
_To be added_
|
||||
|
||||
## Bugs
|
||||
|
||||
_To be added_
|
||||
@@ -13,21 +13,21 @@ The vector store location is controlled by the `CHROMADB_PATH` environment varia
|
||||
|
||||
### CLI (Command Line)
|
||||
|
||||
Use the `manage_vectorstore.py` script for vector store operations:
|
||||
Use the `scripts/manage_vectorstore.py` script for vector store operations:
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
python manage_vectorstore.py stats
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index documents from Paperless-NGX (incremental)
|
||||
python manage_vectorstore.py index
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Clear and reindex all documents
|
||||
python manage_vectorstore.py reindex
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
|
||||
# List documents
|
||||
python manage_vectorstore.py list 10
|
||||
python manage_vectorstore.py list 20 --show-content
|
||||
python scripts/manage_vectorstore.py list 10
|
||||
python scripts/manage_vectorstore.py list 20 --show-content
|
||||
```
|
||||
|
||||
### Docker
|
||||
@@ -36,10 +36,10 @@ Run commands inside the Docker container:
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
docker compose -f docker-compose.dev.yml exec -T raggr python manage_vectorstore.py stats
|
||||
docker compose exec raggr python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Reindex all documents
|
||||
docker compose -f docker-compose.dev.yml exec -T raggr python manage_vectorstore.py reindex
|
||||
docker compose exec raggr python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
@@ -65,7 +65,7 @@ The following authenticated endpoints are available:
|
||||
This indicates a corrupted index. Solution:
|
||||
|
||||
```bash
|
||||
python manage_vectorstore.py reindex
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
### Empty results
|
||||
@@ -73,20 +73,20 @@ python manage_vectorstore.py reindex
|
||||
Check if documents are indexed:
|
||||
|
||||
```bash
|
||||
python manage_vectorstore.py stats
|
||||
python scripts/manage_vectorstore.py stats
|
||||
```
|
||||
|
||||
If count is 0, run:
|
||||
|
||||
```bash
|
||||
python manage_vectorstore.py index
|
||||
python scripts/manage_vectorstore.py index
|
||||
```
|
||||
|
||||
### Different results in Docker vs local
|
||||
|
||||
Docker and local environments use separate ChromaDB instances. To sync:
|
||||
|
||||
1. Index inside Docker: `docker compose exec -T raggr python manage_vectorstore.py reindex`
|
||||
1. Index inside Docker: `docker compose exec raggr python scripts/manage_vectorstore.py reindex`
|
||||
2. Or mount the same volume for both environments
|
||||
|
||||
## Production Considerations
|
||||
274
docs/authentication.md
Normal file
274
docs/authentication.md
Normal file
@@ -0,0 +1,274 @@
|
||||
# Authentication Architecture
|
||||
|
||||
This document describes the authentication stack for SimbaRAG: LLDAP → Authelia → OAuth2/OIDC.
|
||||
|
||||
## Overview
|
||||
|
||||
```
|
||||
┌─────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────┐
|
||||
│ LLDAP │────▶│ Authelia │────▶│ OAuth2/OIDC │────▶│ SimbaRAG │
|
||||
│ (Users) │ │ (IdP) │ │ (Flow) │ │ (App) │
|
||||
└─────────┘ └──────────┘ └──────────────┘ └──────────┘
|
||||
```
|
||||
|
||||
| Component | Role |
|
||||
|-----------|------|
|
||||
| **LLDAP** | Lightweight LDAP server storing users and groups |
|
||||
| **Authelia** | Identity provider that authenticates against LLDAP and issues OIDC tokens |
|
||||
| **SimbaRAG** | Relying party that consumes OIDC tokens and manages sessions |
|
||||
|
||||
## OIDC Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `OIDC_ISSUER` | Authelia server URL | Required |
|
||||
| `OIDC_CLIENT_ID` | Client ID registered in Authelia | Required |
|
||||
| `OIDC_CLIENT_SECRET` | Client secret for token exchange | Required |
|
||||
| `OIDC_REDIRECT_URI` | Callback URL after authentication | Required |
|
||||
| `OIDC_USE_DISCOVERY` | Enable automatic discovery | `true` |
|
||||
| `JWT_SECRET_KEY` | Secret for signing backend JWTs | Required |
|
||||
|
||||
### Discovery
|
||||
|
||||
When `OIDC_USE_DISCOVERY=true`, the application fetches endpoints from:
|
||||
|
||||
```
|
||||
{OIDC_ISSUER}/.well-known/openid-configuration
|
||||
```
|
||||
|
||||
This provides:
|
||||
|
||||
- Authorization endpoint
|
||||
- Token endpoint
|
||||
- JWKS URI for signature verification
|
||||
- Supported scopes and claims
|
||||
|
||||
## Authentication Flow
|
||||
|
||||
### 1. Login Initiation
|
||||
|
||||
```
|
||||
GET /api/user/oidc/login
|
||||
```
|
||||
|
||||
1. Generate PKCE code verifier and challenge (S256)
|
||||
2. Generate CSRF state token
|
||||
3. Store state in session storage
|
||||
4. Return authorization URL for frontend redirect
|
||||
|
||||
### 2. Authorization
|
||||
|
||||
User is redirected to Authelia where they:
|
||||
|
||||
1. Enter LDAP credentials
|
||||
2. Complete MFA if configured
|
||||
3. Consent to requested scopes
|
||||
|
||||
### 3. Callback
|
||||
|
||||
```
|
||||
GET /api/user/oidc/callback?code=...&state=...
|
||||
```
|
||||
|
||||
1. Validate state matches stored value (CSRF protection)
|
||||
2. Exchange authorization code for tokens using PKCE verifier
|
||||
3. Verify ID token signature using JWKS
|
||||
4. Validate claims (issuer, audience, expiration)
|
||||
5. Create or update user in database
|
||||
6. Issue backend JWT tokens (access + refresh)
|
||||
|
||||
### 4. Token Refresh
|
||||
|
||||
```
|
||||
POST /api/user/refresh
|
||||
Authorization: Bearer <refresh_token>
|
||||
```
|
||||
|
||||
Issues a new access token without re-authentication.
|
||||
|
||||
## User Model
|
||||
|
||||
```python
|
||||
class User(Model):
|
||||
id = UUIDField(primary_key=True)
|
||||
username = CharField(max_length=255)
|
||||
password = BinaryField(null=True) # Nullable for OIDC-only users
|
||||
email = CharField(max_length=100, unique=True)
|
||||
|
||||
# OIDC fields
|
||||
oidc_subject = CharField(max_length=255, unique=True, null=True)
|
||||
auth_provider = CharField(max_length=50, default="local") # "local" or "oidc"
|
||||
ldap_groups = JSONField(default=[]) # LDAP groups from OIDC claims
|
||||
|
||||
created_at = DatetimeField(auto_now_add=True)
|
||||
updated_at = DatetimeField(auto_now=True)
|
||||
|
||||
def has_group(self, group: str) -> bool:
|
||||
"""Check if user belongs to a specific LDAP group."""
|
||||
return group in (self.ldap_groups or [])
|
||||
|
||||
def is_admin(self) -> bool:
|
||||
"""Check if user is an admin (member of lldap_admin group)."""
|
||||
return self.has_group("lldap_admin")
|
||||
```
|
||||
|
||||
### User Provisioning
|
||||
|
||||
The `OIDCUserService` handles automatic user creation:
|
||||
|
||||
1. Extract claims from ID token (`sub`, `email`, `preferred_username`)
|
||||
2. Check if user exists by `oidc_subject`
|
||||
3. If not, check by email for migration from local auth
|
||||
4. Create new user or update existing
|
||||
|
||||
## JWT Tokens
|
||||
|
||||
Backend issues its own JWTs after OIDC authentication:
|
||||
|
||||
| Token Type | Purpose | Typical Lifetime |
|
||||
|------------|---------|------------------|
|
||||
| Access Token | API authorization | 15 minutes |
|
||||
| Refresh Token | Obtain new access tokens | 7 days |
|
||||
|
||||
### Claims
|
||||
|
||||
```json
|
||||
{
|
||||
"identity": "<user-uuid>",
|
||||
"type": "access|refresh",
|
||||
"exp": 1234567890,
|
||||
"iat": 1234567890
|
||||
}
|
||||
```
|
||||
|
||||
## Protected Endpoints
|
||||
|
||||
All API endpoints use the `@jwt_refresh_token_required` decorator for basic authentication:
|
||||
|
||||
```python
|
||||
@blueprint.route("/example")
|
||||
@jwt_refresh_token_required
|
||||
async def protected_endpoint():
|
||||
user_id = get_jwt_identity()
|
||||
# ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Role-Based Access Control (RBAC)
|
||||
|
||||
RBAC is implemented using LDAP groups passed through Authelia as OIDC claims. Users in the `lldap_admin` group have admin privileges.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ LLDAP │
|
||||
│ Groups: lldap_admin, lldap_user │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Authelia │
|
||||
│ Scope: groups → Claim: groups = ["lldap_admin"] │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ SimbaRAG │
|
||||
│ 1. Extract groups from ID token │
|
||||
│ 2. Store in User.ldap_groups │
|
||||
│ 3. Check membership with @admin_required decorator │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Authelia Configuration
|
||||
|
||||
Ensure Authelia is configured to pass the `groups` claim:
|
||||
|
||||
```yaml
|
||||
identity_providers:
|
||||
oidc:
|
||||
clients:
|
||||
- client_id: simbarag
|
||||
scopes:
|
||||
- openid
|
||||
- profile
|
||||
- email
|
||||
- groups # Required for RBAC
|
||||
```
|
||||
|
||||
### Admin-Only Endpoints
|
||||
|
||||
The `@admin_required` decorator protects privileged endpoints:
|
||||
|
||||
```python
|
||||
from blueprints.users.decorators import admin_required
|
||||
|
||||
@blueprint.post("/admin-action")
|
||||
@admin_required
|
||||
async def admin_only_endpoint():
|
||||
# Only users in lldap_admin group can access
|
||||
...
|
||||
```
|
||||
|
||||
**Protected endpoints:**
|
||||
|
||||
| Endpoint | Access | Description |
|
||||
|----------|--------|-------------|
|
||||
| `POST /api/rag/index` | Admin | Trigger document indexing |
|
||||
| `POST /api/rag/reindex` | Admin | Clear and reindex all documents |
|
||||
| `GET /api/rag/stats` | All users | View vector store statistics |
|
||||
|
||||
### User Response
|
||||
|
||||
The OIDC callback returns group information:
|
||||
|
||||
```json
|
||||
{
|
||||
"access_token": "...",
|
||||
"refresh_token": "...",
|
||||
"user": {
|
||||
"id": "uuid",
|
||||
"username": "john",
|
||||
"email": "john@example.com",
|
||||
"groups": ["lldap_admin", "lldap_user"],
|
||||
"is_admin": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Current Gaps
|
||||
|
||||
| Issue | Risk | Mitigation |
|
||||
|-------|------|------------|
|
||||
| In-memory session storage | State lost on restart, not scalable | Use Redis for production |
|
||||
| No token revocation | Tokens valid until expiry | Implement blacklist or short expiry |
|
||||
| No audit logging | Cannot track auth events | Add event logging |
|
||||
| Single JWT secret | Compromise affects all tokens | Rotate secrets, use asymmetric keys |
|
||||
|
||||
### Recommendations
|
||||
|
||||
1. **Use Redis** for OIDC state storage in production
|
||||
2. **Implement logout** with token blacklisting
|
||||
3. **Add audit logging** for authentication events
|
||||
4. **Rotate JWT secrets** regularly
|
||||
5. **Use short-lived access tokens** (15 min) with refresh
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `services/raggr/oidc_config.py` | OIDC client configuration and discovery |
|
||||
| `services/raggr/blueprints/users/models.py` | User model definition with group helpers |
|
||||
| `services/raggr/blueprints/users/oidc_service.py` | User provisioning from OIDC claims |
|
||||
| `services/raggr/blueprints/users/__init__.py` | Auth endpoints and flow |
|
||||
| `services/raggr/blueprints/users/decorators.py` | Auth decorators (`@admin_required`) |
|
||||
188
docs/deployment.md
Normal file
188
docs/deployment.md
Normal file
@@ -0,0 +1,188 @@
|
||||
# Deployment & Migrations Guide
|
||||
|
||||
This document covers database migrations and deployment workflows for SimbaRAG.
|
||||
|
||||
## Migration Workflow
|
||||
|
||||
Migrations are managed by [Aerich](https://github.com/tortoise/aerich), the migration tool for Tortoise ORM.
|
||||
|
||||
### Key Principles
|
||||
|
||||
1. **Generate migrations in Docker** - Aerich needs database access to detect schema changes
|
||||
2. **Migrations auto-apply on startup** - Both `startup.sh` and `startup-dev.sh` run `aerich upgrade`
|
||||
3. **Commit migrations to git** - Migration files must be in the repo for production deploys
|
||||
|
||||
### Generating a New Migration
|
||||
|
||||
#### Development (Recommended)
|
||||
|
||||
With `docker-compose.dev.yml`, your local `services/raggr` directory is synced to the container. Migrations generated inside the container appear on your host automatically.
|
||||
|
||||
```bash
|
||||
# 1. Start the dev environment
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# 2. Generate migration (runs inside container, syncs to host)
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name describe_your_change
|
||||
|
||||
# 3. Verify migration was created
|
||||
ls services/raggr/migrations/models/
|
||||
|
||||
# 4. Commit the migration
|
||||
git add services/raggr/migrations/
|
||||
git commit -m "Add migration: describe_your_change"
|
||||
```
|
||||
|
||||
#### Production Container
|
||||
|
||||
For production, migration files are baked into the image. You must generate migrations in dev first.
|
||||
|
||||
```bash
|
||||
# If you need to generate a migration from production (not recommended):
|
||||
docker compose exec raggr aerich migrate --name describe_your_change
|
||||
|
||||
# Copy the file out of the container
|
||||
docker cp $(docker compose ps -q raggr):/app/migrations/models/ ./services/raggr/migrations/
|
||||
```
|
||||
|
||||
### Applying Migrations
|
||||
|
||||
Migrations apply automatically on container start via the startup scripts.
|
||||
|
||||
**Manual application (if needed):**
|
||||
|
||||
```bash
|
||||
# Dev
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich upgrade
|
||||
|
||||
# Production
|
||||
docker compose exec raggr aerich upgrade
|
||||
```
|
||||
|
||||
### Checking Migration Status
|
||||
|
||||
```bash
|
||||
# View applied migrations
|
||||
docker compose exec raggr aerich history
|
||||
|
||||
# View pending migrations
|
||||
docker compose exec raggr aerich heads
|
||||
```
|
||||
|
||||
### Rolling Back
|
||||
|
||||
```bash
|
||||
# Downgrade one migration
|
||||
docker compose exec raggr aerich downgrade
|
||||
|
||||
# Downgrade to specific version
|
||||
docker compose exec raggr aerich downgrade -v 1
|
||||
```
|
||||
|
||||
## Deployment Workflows
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Start with watch mode (auto-restarts on file changes)
|
||||
docker compose -f docker-compose.dev.yml up
|
||||
|
||||
# Or with docker compose watch (requires Docker Compose v2.22+)
|
||||
docker compose -f docker-compose.dev.yml watch
|
||||
```
|
||||
|
||||
The dev environment:
|
||||
- Syncs `services/raggr/` to `/app` in the container
|
||||
- Rebuilds frontend on changes
|
||||
- Auto-applies migrations on startup
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
# Build and deploy
|
||||
docker compose build raggr
|
||||
docker compose up -d
|
||||
|
||||
# View logs
|
||||
docker compose logs -f raggr
|
||||
|
||||
# Verify migrations applied
|
||||
docker compose exec raggr aerich history
|
||||
```
|
||||
|
||||
### Fresh Deploy (New Database)
|
||||
|
||||
On first deploy with an empty database, `startup-dev.sh` runs `aerich init-db` instead of `aerich upgrade`. This creates all tables from the current models.
|
||||
|
||||
For production (`startup.sh`), ensure the database exists and run:
|
||||
|
||||
```bash
|
||||
# If aerich table doesn't exist yet
|
||||
docker compose exec raggr aerich init-db
|
||||
|
||||
# Or if migrating from existing schema
|
||||
docker compose exec raggr aerich upgrade
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No migrations found" on startup
|
||||
|
||||
The `migrations/models/` directory is empty or not copied into the image.
|
||||
|
||||
**Fix:** Ensure migrations are committed and the Dockerfile copies them:
|
||||
```dockerfile
|
||||
COPY migrations ./migrations
|
||||
```
|
||||
|
||||
### Migration fails with "relation already exists"
|
||||
|
||||
The database has tables but aerich doesn't know about them (fresh aerich setup on existing DB).
|
||||
|
||||
**Fix:** Fake the initial migration:
|
||||
```bash
|
||||
# Mark initial migration as applied without running it
|
||||
docker compose exec raggr aerich upgrade --fake
|
||||
```
|
||||
|
||||
### Model changes not detected
|
||||
|
||||
Aerich compares models against the last migration's state. If state is out of sync:
|
||||
|
||||
```bash
|
||||
# Regenerate migration state (dangerous - review carefully)
|
||||
docker compose exec raggr aerich migrate --name fix_state
|
||||
```
|
||||
|
||||
### Database connection errors
|
||||
|
||||
Ensure PostgreSQL is healthy before running migrations:
|
||||
|
||||
```bash
|
||||
# Check postgres status
|
||||
docker compose ps postgres
|
||||
|
||||
# Wait for postgres then run migrations
|
||||
docker compose exec raggr bash -c "sleep 5 && aerich upgrade"
|
||||
```
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `pyproject.toml` | Aerich config (`[tool.aerich]` section) |
|
||||
| `migrations/models/` | Migration files |
|
||||
| `startup.sh` | Production startup (runs `aerich upgrade`) |
|
||||
| `startup-dev.sh` | Dev startup (runs `aerich upgrade` or `init-db`) |
|
||||
| `app.py` | Contains `TORTOISE_CONFIG` |
|
||||
| `aerich_config.py` | Aerich initialization configuration |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Generate migration | `docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name name` |
|
||||
| Apply migrations | `docker compose exec raggr aerich upgrade` |
|
||||
| View history | `docker compose exec raggr aerich history` |
|
||||
| Rollback | `docker compose exec raggr aerich downgrade` |
|
||||
| Fresh init | `docker compose exec raggr aerich init-db` |
|
||||
258
docs/development.md
Normal file
258
docs/development.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# Development Guide
|
||||
|
||||
This guide explains how to run SimbaRAG in development mode.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Option 1: Local Development (Recommended)
|
||||
|
||||
Run PostgreSQL in Docker and the application locally for faster iteration:
|
||||
|
||||
```bash
|
||||
# 1. Start PostgreSQL
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# 2. Set environment variables
|
||||
export DATABASE_URL="postgres://raggr:raggr_dev_password@localhost:5432/raggr"
|
||||
export CHROMADB_PATH="./chromadb"
|
||||
export $(grep -v '^#' .env | xargs) # Load other vars from .env
|
||||
|
||||
# 3. Install dependencies (first time)
|
||||
pip install -r requirements.txt
|
||||
cd raggr-frontend && yarn install && yarn build && cd ..
|
||||
|
||||
# 4. Run migrations
|
||||
aerich upgrade
|
||||
|
||||
# 5. Start the server
|
||||
python app.py
|
||||
```
|
||||
|
||||
The application will be available at `http://localhost:8080`.
|
||||
|
||||
### Option 2: Full Docker Development
|
||||
|
||||
Run everything in Docker with hot reload (slower, but matches production):
|
||||
|
||||
```bash
|
||||
# Uncomment the raggr service in docker-compose.dev.yml first!
|
||||
|
||||
# Start all services
|
||||
docker compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# View logs
|
||||
docker compose -f docker-compose.dev.yml logs -f raggr
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
raggr/
|
||||
├── app.py # Quart application entry point
|
||||
├── main.py # RAG logic and LangChain agent
|
||||
├── llm.py # LLM client (Ollama + OpenAI fallback)
|
||||
├── aerich_config.py # Database migration configuration
|
||||
│
|
||||
├── blueprints/ # API route blueprints
|
||||
│ ├── users/ # Authentication (OIDC, JWT, RBAC)
|
||||
│ ├── conversation/ # Chat conversations and messages
|
||||
│ └── rag/ # Document indexing (admin only)
|
||||
│
|
||||
├── config/ # Configuration modules
|
||||
│ └── oidc_config.py # OIDC authentication settings
|
||||
│
|
||||
├── utils/ # Reusable utilities
|
||||
│ ├── chunker.py # Document chunking for embeddings
|
||||
│ ├── cleaner.py # PDF cleaning and summarization
|
||||
│ ├── image_process.py # Image description with LLM
|
||||
│ └── request.py # Paperless-NGX API client
|
||||
│
|
||||
├── scripts/ # Administrative scripts
|
||||
│ ├── add_user.py # Create users manually
|
||||
│ ├── user_message_stats.py # User message statistics
|
||||
│ ├── manage_vectorstore.py # Vector store management
|
||||
│ ├── inspect_vector_store.py # Inspect ChromaDB contents
|
||||
│ └── query.py # Query generation utilities
|
||||
│
|
||||
├── raggr-frontend/ # React frontend
|
||||
│ └── src/ # Frontend source code
|
||||
│
|
||||
├── migrations/ # Database migrations
|
||||
└── docs/ # Documentation
|
||||
```
|
||||
|
||||
## Making Changes
|
||||
|
||||
### Backend Changes
|
||||
|
||||
**Local development:**
|
||||
1. Edit Python files
|
||||
2. Save
|
||||
3. Restart `python app.py` (or use a tool like `watchdog` for auto-reload)
|
||||
|
||||
**Docker development:**
|
||||
1. Edit Python files
|
||||
2. Files are synced via Docker watch mode
|
||||
3. Container automatically restarts
|
||||
|
||||
### Frontend Changes
|
||||
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
|
||||
# Development mode with hot reload
|
||||
yarn dev
|
||||
|
||||
# Production build (for testing)
|
||||
yarn build
|
||||
```
|
||||
|
||||
The backend serves built files from `raggr-frontend/dist/`.
|
||||
|
||||
### Database Model Changes
|
||||
|
||||
When you modify Tortoise ORM models:
|
||||
|
||||
```bash
|
||||
# Generate migration
|
||||
aerich migrate --name "describe_your_change"
|
||||
|
||||
# Apply migration
|
||||
aerich upgrade
|
||||
|
||||
# View history
|
||||
aerich history
|
||||
```
|
||||
|
||||
See [deployment.md](deployment.md) for detailed migration workflows.
|
||||
|
||||
### Adding Dependencies
|
||||
|
||||
**Backend:**
|
||||
```bash
|
||||
# Add to requirements.txt or use uv
|
||||
pip install package-name
|
||||
pip freeze > requirements.txt
|
||||
```
|
||||
|
||||
**Frontend:**
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
yarn add package-name
|
||||
```
|
||||
|
||||
## Useful Commands
|
||||
|
||||
### Database
|
||||
|
||||
```bash
|
||||
# Connect to PostgreSQL
|
||||
docker compose -f docker-compose.dev.yml exec postgres psql -U raggr -d raggr
|
||||
|
||||
# Reset database
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
aerich init-db
|
||||
```
|
||||
|
||||
### Vector Store
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index new documents from Paperless
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Clear and reindex everything
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
See [vectorstore.md](vectorstore.md) for details.
|
||||
|
||||
### Scripts
|
||||
|
||||
```bash
|
||||
# Add a new user
|
||||
python scripts/add_user.py
|
||||
|
||||
# View message statistics
|
||||
python scripts/user_message_stats.py
|
||||
|
||||
# Inspect vector store contents
|
||||
python scripts/inspect_vector_store.py
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Copy `.env.example` to `.env` and configure:
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `DATABASE_URL` | PostgreSQL connection | `postgres://user:pass@localhost:5432/db` |
|
||||
| `CHROMADB_PATH` | ChromaDB storage path | `./chromadb` |
|
||||
| `OLLAMA_URL` | Ollama server URL | `http://localhost:11434` |
|
||||
| `OPENAI_API_KEY` | OpenAI API key (fallback LLM) | `sk-...` |
|
||||
| `PAPERLESS_TOKEN` | Paperless-NGX API token | `...` |
|
||||
| `BASE_URL` | Paperless-NGX URL | `https://paperless.example.com` |
|
||||
| `OIDC_ISSUER` | OIDC provider URL | `https://auth.example.com` |
|
||||
| `OIDC_CLIENT_ID` | OIDC client ID | `simbarag` |
|
||||
| `OIDC_CLIENT_SECRET` | OIDC client secret | `...` |
|
||||
| `JWT_SECRET_KEY` | JWT signing key | `random-secret` |
|
||||
| `TAVILY_KEY` | Tavily web search API key | `tvly-...` |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
```bash
|
||||
# Find and kill process on port 8080
|
||||
lsof -ti:8080 | xargs kill -9
|
||||
|
||||
# Or change the port in app.py
|
||||
```
|
||||
|
||||
### Database Connection Errors
|
||||
|
||||
```bash
|
||||
# Check if PostgreSQL is running
|
||||
docker compose -f docker-compose.dev.yml ps postgres
|
||||
|
||||
# View PostgreSQL logs
|
||||
docker compose -f docker-compose.dev.yml logs postgres
|
||||
```
|
||||
|
||||
### Frontend Not Building
|
||||
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
rm -rf node_modules dist
|
||||
yarn install
|
||||
yarn build
|
||||
```
|
||||
|
||||
### ChromaDB Errors
|
||||
|
||||
```bash
|
||||
# Clear and recreate ChromaDB
|
||||
rm -rf chromadb/
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
### Import Errors After Reorganization
|
||||
|
||||
Ensure you're in the project root directory when running scripts, or use:
|
||||
|
||||
```bash
|
||||
# Add project root to Python path
|
||||
export PYTHONPATH="${PYTHONPATH}:$(pwd)"
|
||||
python scripts/your_script.py
|
||||
```
|
||||
|
||||
## Hot Tips
|
||||
|
||||
- Use `python -m pdb app.py` for debugging
|
||||
- Enable Quart debug mode in `app.py`: `app.run(debug=True)`
|
||||
- Check API logs: They appear in the terminal running `python app.py`
|
||||
- Frontend logs: Open browser DevTools console
|
||||
- Use `docker compose -f docker-compose.dev.yml down -v` for a clean slate
|
||||
203
docs/index.md
Normal file
203
docs/index.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# SimbaRAG Documentation
|
||||
|
||||
Welcome to the SimbaRAG documentation! This guide will help you understand, develop, and deploy the SimbaRAG conversational AI system.
|
||||
|
||||
## Getting Started
|
||||
|
||||
New to SimbaRAG? Start here:
|
||||
|
||||
1. Read the main [README](../README.md) for project overview and architecture
|
||||
2. Follow the [Development Guide](development.md) to set up your environment
|
||||
3. Learn about [Authentication](authentication.md) setup with OIDC and LDAP
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
### Core Guides
|
||||
|
||||
- **[Development Guide](development.md)** - Local development setup, project structure, and workflows
|
||||
- **[Deployment Guide](deployment.md)** - Database migrations, deployment workflows, and troubleshooting
|
||||
- **[Vector Store Guide](VECTORSTORE.md)** - Managing ChromaDB, indexing documents, and RAG operations
|
||||
- **[Migrations Guide](MIGRATIONS.md)** - Database migration reference
|
||||
- **[Authentication Guide](authentication.md)** - OIDC, Authelia, LLDAP configuration and user management
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| Task | Documentation |
|
||||
|------|---------------|
|
||||
| Set up local dev environment | [Development Guide → Quick Start](development.md#quick-start) |
|
||||
| Run database migrations | [Deployment Guide → Migration Workflow](deployment.md#migration-workflow) |
|
||||
| Index documents | [Vector Store Guide → Management Commands](VECTORSTORE.md#management-commands) |
|
||||
| Configure authentication | [Authentication Guide](authentication.md) |
|
||||
| Run administrative scripts | [Development Guide → Scripts](development.md#scripts) |
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Start local development
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
export DATABASE_URL="postgres://raggr:raggr_dev_password@localhost:5432/raggr"
|
||||
export CHROMADB_PATH="./chromadb"
|
||||
python app.py
|
||||
```
|
||||
|
||||
### Database Migrations
|
||||
|
||||
```bash
|
||||
# Generate migration
|
||||
aerich migrate --name "your_change"
|
||||
|
||||
# Apply migrations
|
||||
aerich upgrade
|
||||
|
||||
# View history
|
||||
aerich history
|
||||
```
|
||||
|
||||
### Vector Store Management
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index new documents
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Reindex everything
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
SimbaRAG is built with:
|
||||
|
||||
- **Backend**: Quart (async Python), LangChain, Tortoise ORM
|
||||
- **Frontend**: React 19, Rsbuild, Tailwind CSS
|
||||
- **Database**: PostgreSQL (users, conversations)
|
||||
- **Vector Store**: ChromaDB (document embeddings)
|
||||
- **LLM**: Ollama (primary), OpenAI (fallback)
|
||||
- **Auth**: Authelia (OIDC), LLDAP (user directory)
|
||||
|
||||
See the [README](../README.md#system-architecture) for detailed architecture diagram.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
simbarag/
|
||||
├── app.py # Quart app entry point
|
||||
├── main.py # RAG & LangChain agent
|
||||
├── llm.py # LLM client
|
||||
├── blueprints/ # API routes
|
||||
├── config/ # Configuration
|
||||
├── utils/ # Utilities
|
||||
├── scripts/ # Admin scripts
|
||||
├── raggr-frontend/ # React UI
|
||||
├── migrations/ # Database migrations
|
||||
├── docs/ # This documentation
|
||||
├── docker-compose.yml # Production Docker setup
|
||||
└── docker-compose.dev.yml # Development Docker setup
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### RAG (Retrieval-Augmented Generation)
|
||||
|
||||
SimbaRAG uses RAG to answer questions about Simba:
|
||||
|
||||
1. Documents are fetched from Paperless-NGX
|
||||
2. Documents are chunked and embedded using OpenAI
|
||||
3. Embeddings are stored in ChromaDB
|
||||
4. User queries are embedded and matched against the store
|
||||
5. Relevant chunks are passed to the LLM for context
|
||||
6. LLM generates an answer using retrieved context
|
||||
|
||||
### LangChain Agent
|
||||
|
||||
The conversational agent has two tools:
|
||||
|
||||
- **simba_search**: Queries the vector store for Simba's documents
|
||||
- **web_search**: Searches the web via Tavily API
|
||||
|
||||
The agent automatically selects tools based on the query.
|
||||
|
||||
### Authentication Flow
|
||||
|
||||
1. User initiates OIDC login via Authelia
|
||||
2. Authelia authenticates against LLDAP
|
||||
3. Backend receives OIDC tokens and issues JWT
|
||||
4. Frontend stores JWT in localStorage
|
||||
5. Subsequent requests use JWT for authorization
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Key environment variables (see `.env.example` for complete list):
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| `DATABASE_URL` | PostgreSQL connection |
|
||||
| `CHROMADB_PATH` | Vector store location |
|
||||
| `OLLAMA_URL` | Local LLM server |
|
||||
| `OPENAI_API_KEY` | OpenAI for embeddings/fallback |
|
||||
| `PAPERLESS_TOKEN` | Document source API |
|
||||
| `OIDC_*` | Authentication configuration |
|
||||
| `TAVILY_KEY` | Web search API |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Authentication
|
||||
- `GET /api/user/oidc/login` - Start OIDC flow
|
||||
- `GET /api/user/oidc/callback` - OIDC callback
|
||||
- `POST /api/user/refresh` - Refresh JWT
|
||||
|
||||
### Conversations
|
||||
- `POST /api/conversation/` - Create conversation
|
||||
- `GET /api/conversation/` - List conversations
|
||||
- `POST /api/conversation/query` - Chat message
|
||||
|
||||
### RAG (Admin Only)
|
||||
- `GET /api/rag/stats` - Vector store stats
|
||||
- `POST /api/rag/index` - Index documents
|
||||
- `POST /api/rag/reindex` - Reindex all
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| Port already in use | Check if services are running: `lsof -ti:8080` |
|
||||
| Database connection error | Ensure PostgreSQL is running: `docker compose ps` |
|
||||
| ChromaDB errors | Clear and reindex: `python scripts/manage_vectorstore.py reindex` |
|
||||
| Import errors | Check you're in `services/raggr/` directory |
|
||||
| Frontend not building | `cd raggr-frontend && yarn install && yarn build` |
|
||||
|
||||
See individual guides for detailed troubleshooting.
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Read the [Development Guide](development.md)
|
||||
2. Set up your local environment
|
||||
3. Make changes and test locally
|
||||
4. Generate migrations if needed
|
||||
5. Submit a pull request
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [LangChain Documentation](https://python.langchain.com/)
|
||||
- [ChromaDB Documentation](https://docs.trychroma.com/)
|
||||
- [Quart Documentation](https://quart.palletsprojects.com/)
|
||||
- [Tortoise ORM Documentation](https://tortoise.github.io/)
|
||||
- [Authelia Documentation](https://www.authelia.com/)
|
||||
|
||||
## Need Help?
|
||||
|
||||
- Check the relevant guide in this documentation
|
||||
- Review troubleshooting sections
|
||||
- Check application logs: `docker compose logs -f`
|
||||
- Inspect database: `docker compose exec postgres psql -U raggr`
|
||||
|
||||
---
|
||||
|
||||
**Documentation Version**: 1.0
|
||||
**Last Updated**: January 2026
|
||||
0
docs/ynab_integration/specification.md
Normal file
0
docs/ynab_integration/specification.md
Normal file
81
index.html
Normal file
81
index.html
Normal file
@@ -0,0 +1,81 @@
|
||||
<!doctype html>
|
||||
|
||||
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||||
<meta name="author" content="Paperless-ngx project and contributors">
|
||||
<meta name="robots" content="noindex,nofollow">
|
||||
|
||||
<title>
|
||||
|
||||
Paperless-ngx sign in
|
||||
|
||||
</title>
|
||||
<link href="/static/bootstrap.min.css" rel="stylesheet">
|
||||
<link href="/static/base.css" rel="stylesheet">
|
||||
</head>
|
||||
|
||||
<body class="text-center">
|
||||
<div class="position-absolute top-50 start-50 translate-middle">
|
||||
<form class="form-accounts" id="form-account" method="post">
|
||||
<input type="hidden" name="csrfmiddlewaretoken" value="KLQ3mMraTFHfK9sMmc6DJcNIS6YixeHnSJiT3A12LYB49HeEXOpx5RnY9V6uPSrD">
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 2897.4 896.6" width='300' class='logo mb-4'>
|
||||
<path class="leaf" d="M140,713.7c-3.4-16.4-10.3-49.1-11.2-49.1c-145.7-87.1-128.4-238-80.2-324.2C59,449,251.2,524,139.1,656.8 c-0.9,1.7,5.2,22.4,10.3,41.4c22.4-37.9,56-83.6,54.3-87.9C65.9,273.9,496.9,248.1,586.6,39.4c40.5,201.8-20.7,513.9-367.2,593.2 c-1.7,0.9-62.9,108.6-65.5,109.5c0-1.7-25.9-0.9-22.4-9.5C133.1,727.4,136.6,720.6,140,713.7L140,713.7z M135.7,632.6 c44-50.9-7.8-137.9-38.8-166.4C149.5,556.7,146,609.3,135.7,632.6L135.7,632.6z" transform="translate(0)" style="fill:#17541f"/>
|
||||
<g class="text" style="fill:#000">
|
||||
<path d="M1022.3,428.7c-17.8-19.9-42.7-29.8-74.7-29.8c-22.3,0-42.4,5.7-60.5,17.3c-18.1,11.6-32.3,27.5-42.5,47.8 s-15.3,42.9-15.3,67.8c0,24.9,5.1,47.5,15.3,67.8c10.3,20.3,24.4,36.2,42.5,47.8c18.1,11.5,38.3,17.3,60.5,17.3 c32,0,56.9-9.9,74.7-29.8v20.4v0.2h84.5V408.3h-84.5V428.7z M1010.5,575c-10.2,11.7-23.6,17.6-40.2,17.6s-29.9-5.9-40-17.6 s-15.1-26.1-15.1-43.3c0-17.1,5-31.6,15.1-43.3s23.4-17.6,40-17.6c16.6,0,30,5.9,40.2,17.6s15.3,26.1,15.3,43.3 S1020.7,563.3,1010.5,575z" transform="translate(0)"/>
|
||||
<path d="M1381,416.1c-18.1-11.5-38.3-17.3-60.5-17.4c-32,0-56.9,9.9-74.7,29.8v-20.4h-84.5v390.7h84.5v-164 c17.8,19.9,42.7,29.8,74.7,29.8c22.3,0,42.4-5.7,60.5-17.3s32.3-27.5,42.5-47.8c10.2-20.3,15.3-42.9,15.3-67.8s-5.1-47.5-15.3-67.8 C1413.2,443.6,1399.1,427.7,1381,416.1z M1337.9,575c-10.1,11.7-23.4,17.6-40,17.6s-29.9-5.9-40-17.6s-15.1-26.1-15.1-43.3 c0-17.1,5-31.6,15.1-43.3s23.4-17.6,40-17.6s29.9,5.9,40,17.6s15.1,26.1,15.1,43.3S1347.9,563.3,1337.9,575z" transform="translate(0)"/>
|
||||
<path d="M1672.2,416.8c-20.5-12-43-18-67.6-18c-24.9,0-47.6,5.9-68,17.6c-20.4,11.7-36.5,27.7-48.2,48s-17.6,42.7-17.6,67.3 c0.3,25.2,6.2,47.8,17.8,68c11.5,20.2,28,36,49.3,47.6c21.3,11.5,45.9,17.3,73.8,17.3c48.6,0,86.8-14.7,114.7-44l-52.5-48.9 c-8.6,8.3-17.6,14.6-26.7,19c-9.3,4.3-21.1,6.4-35.3,6.4c-11.6,0-22.5-3.6-32.7-10.9c-10.3-7.3-17.1-16.5-20.7-27.8h180l0.4-11.6 c0-29.6-6-55.7-18-78.2S1692.6,428.8,1672.2,416.8z M1558.3,503.2c2.1-12.1,7.5-21.8,16.2-29.1s18.7-10.9,30-10.9 s21.2,3.6,29.8,10.9c8.6,7.2,13.9,16.9,16,29.1H1558.3z" transform="translate(0)"/>
|
||||
<path d="M1895.3,411.7c-11,5.6-20.3,13.7-28,24.4h-0.1v-28h-84.5v247.3h84.5V536.3c0-22.6,4.7-38.1,14.2-46.5 c9.5-8.5,22.7-12.7,39.6-12.7c6.2,0,13.5,1,21.8,3.1l10.7-72c-5.9-3.3-14.5-4.9-25.8-4.9C1917.1,403.3,1906.3,406.1,1895.3,411.7z" transform="translate(0)"/>
|
||||
<rect x="1985" y="277.4" width="84.5" height="377.8" transform="translate(0)"/>
|
||||
<path d="M2313.2,416.8c-20.5-12-43-18-67.6-18c-24.9,0-47.6,5.9-68,17.6s-36.5,27.7-48.2,48c-11.7,20.3-17.6,42.7-17.6,67.3 c0.3,25.2,6.2,47.8,17.8,68c11.5,20.2,28,36,49.3,47.6c21.3,11.5,45.9,17.3,73.8,17.3c48.6,0,86.8-14.7,114.7-44l-52.5-48.9 c-8.6,8.3-17.6,14.6-26.7,19c-9.3,4.3-21.1,6.4-35.3,6.4c-11.6,0-22.5-3.6-32.7-10.9c-10.3-7.3-17.1-16.5-20.7-27.8h180l0.4-11.6 c0-29.6-6-55.7-18-78.2S2333.6,428.8,2313.2,416.8z M2199.3,503.2c2.1-12.1,7.5-21.8,16.2-29.1s18.7-10.9,30-10.9 s21.2,3.6,29.8,10.9c8.6,7.2,13.9,16.9,16,29.1H2199.3z" transform="translate(0)"/>
|
||||
<path d="M2583.6,507.7c-13.8-4.4-30.6-8.1-50.5-11.1c-15.1-2.7-26.1-5.2-32.9-7.6c-6.8-2.4-10.2-6.1-10.2-11.1s2.3-8.7,6.7-10.9 c4.4-2.2,11.5-3.3,21.3-3.3c11.6,0,24.3,2.4,38.1,7.2c13.9,4.8,26.2,11,36.9,18.4l32.4-58.2c-11.3-7.4-26.2-14.7-44.9-21.8 c-18.7-7.1-39.6-10.7-62.7-10.7c-33.7,0-60.2,7.6-79.3,22.7c-19.1,15.1-28.7,36.1-28.7,63.1c0,19,4.8,33.9,14.4,44.7 c9.6,10.8,21,18.5,34,22.9c13.1,4.5,28.9,8.3,47.6,11.6c14.6,2.7,25.1,5.3,31.6,7.8s9.8,6.5,9.8,11.8c0,10.4-9.7,15.6-29.3,15.6 c-13.7,0-28.5-2.3-44.7-6.9c-16.1-4.6-29.2-11.3-39.3-20.2l-33.3,60c9.2,7.4,24.6,14.7,46.2,22c21.7,7.3,45.2,10.9,70.7,10.9 c34.7,0,62.9-7.4,84.5-22.4c21.7-15,32.5-37.3,32.5-66.9c0-19.3-5-34.2-15.1-44.9S2597.4,512.1,2583.6,507.7z" transform="translate(0)"/>
|
||||
<path d="M2883.4,575.3c0-19.3-5-34.2-15.1-44.9s-22-18.3-35.8-22.7c-13.8-4.4-30.6-8.1-50.5-11.1c-15.1-2.7-26.1-5.2-32.9-7.6 c-6.8-2.4-10.2-6.1-10.2-11.1s2.3-8.7,6.7-10.9c4.4-2.2,11.5-3.3,21.3-3.3c11.6,0,24.3,2.4,38.1,7.2c13.9,4.8,26.2,11,36.9,18.4 l32.4-58.2c-11.3-7.4-26.2-14.7-44.9-21.8c-18.7-7.1-39.6-10.7-62.7-10.7c-33.7,0-60.2,7.6-79.3,22.7 c-19.1,15.1-28.7,36.1-28.7,63.1c0,19,4.8,33.9,14.4,44.7c9.6,10.8,21,18.5,34,22.9c13.1,4.5,28.9,8.3,47.6,11.6 c14.6,2.7,25.1,5.3,31.6,7.8s9.8,6.5,9.8,11.8c0,10.4-9.7,15.6-29.3,15.6c-13.7,0-28.5-2.3-44.7-6.9c-16.1-4.6-29.2-11.3-39.3-20.2 l-33.3,60c9.2,7.4,24.6,14.7,46.2,22c21.7,7.3,45.2,10.9,70.7,10.9c34.7,0,62.9-7.4,84.5-22.4 C2872.6,627.2,2883.4,604.9,2883.4,575.3z" transform="translate(0)"/>
|
||||
<rect x="2460.7" y="738.7" width="59.6" height="17.2" transform="translate(0)"/>
|
||||
<path d="M2596.5,706.4c-5.7,0-11,1-15.8,3s-9,5-12.5,8.9v-9.4h-19.4v93.6h19.4v-52c0-8.6,2.1-15.3,6.3-20c4.2-4.7,9.5-7.1,15.9-7.1 c7.8,0,13.4,2.3,16.8,6.7c3.4,4.5,5.1,11.3,5.1,20.5v52h19.4v-56.8c0-12.8-3.2-22.6-9.5-29.3 C2615.8,709.8,2607.3,706.4,2596.5,706.4z" transform="translate(0)"/>
|
||||
<path d="M2733.8,717.7c-3.6-3.4-7.9-6.1-13.1-8.2s-10.6-3.1-16.2-3.1c-8.7,0-16.5,2.1-23.5,6.3s-12.5,10-16.5,17.3 c-4,7.3-6,15.4-6,24.4c0,8.9,2,17.1,6,24.3c4,7.3,9.5,13,16.5,17.2s14.9,6.3,23.5,6.3c5.6,0,11-1,16.2-3.1 c5.1-2.1,9.5-4.8,13.1-8.2v24.4c0,8.5-2.5,14.8-7.6,18.7c-5,3.9-11,5.9-18,5.9c-6.7,0-12.4-1.6-17.3-4.7c-4.8-3.1-7.6-7.7-8.3-13.8 h-19.4c0.6,7.7,2.9,14.2,7.1,19.5s9.6,9.3,16.2,12c6.6,2.7,13.8,4,21.7,4c12.8,0,23.5-3.4,32-10.1c8.6-6.7,12.8-17.1,12.8-31.1 V708.9h-19.2V717.7z M2732.2,770.1c-2.5,4.7-6,8.3-10.4,11.2c-4.4,2.7-9.4,4-14.9,4c-5.7,0-10.8-1.4-15.2-4.3s-7.8-6.7-10.2-11.4 c-2.3-4.8-3.5-9.8-3.5-15.2c0-5.5,1.1-10.6,3.5-15.3s5.8-8.5,10.2-11.3s9.5-4.2,15.2-4.2c5.5,0,10.5,1.4,14.9,4s7.9,6.3,10.4,11 s3.8,10,3.8,15.8S2734.7,765.4,2732.2,770.1z" transform="translate(0)"/>
|
||||
<polygon points="2867.9,708.9 2846.5,708.9 2820.9,741.9 2795.5,708.9 2773.1,708.9 2809.1,755 2771.5,802.5 2792.9,802.5 2820.1,767.9 2847.2,802.6 2869.6,802.6 2832,754.4 " transform="translate(0)"/>
|
||||
<path d="M757.6,293.7c-20-10.8-42.6-16.2-67.8-16.2H600c-8.5,39.2-21.1,76.4-37.6,111.3c-9.9,20.8-21.1,40.6-33.6,59.4v207.2h88.9 V521.5h72c25.2,0,47.8-5.4,67.8-16.2s35.7-25.6,47.1-44.2c11.4-18.7,17.1-39.1,17.1-61.3c0.1-22.7-5.6-43.3-17-61.9 C793.3,319.2,777.6,304.5,757.6,293.7z M716.6,434.3c-9.3,8.9-21.6,13.3-36.7,13.3l-62.2,0.4v-92.5l62.2-0.4 c15.1,0,27.3,4.4,36.7,13.3c9.4,8.9,14,19.9,14,32.9C730.6,414.5,726,425.4,716.6,434.3z" transform="translate(0)"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
||||
|
||||
|
||||
|
||||
<p>
|
||||
Please sign in.
|
||||
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="form-floating form-stacked-top">
|
||||
<input type="text" name="login" id="inputUsername" placeholder="Username" class="form-control" autocorrect="off" autocapitalize="none" required autofocus>
|
||||
<label for="inputUsername">Username</label>
|
||||
</div>
|
||||
<div class="form-floating form-stacked-bottom">
|
||||
<input type="password" name="password" id="inputPassword" placeholder="Password" class="form-control" required>
|
||||
<label for="inputPassword">Password</label>
|
||||
</div>
|
||||
<div class="d-grid mt-3">
|
||||
<button class="btn btn-lg btn-primary" type="submit">Sign in</button>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</form>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
46
llm.py
Normal file
46
llm.py
Normal file
@@ -0,0 +1,46 @@
|
||||
import os
|
||||
import logging
|
||||
|
||||
from openai import OpenAI
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
|
||||
class LLMClient:
|
||||
def __init__(self):
|
||||
llama_url = os.getenv("LLAMA_SERVER_URL")
|
||||
if llama_url:
|
||||
self.client = OpenAI(base_url=llama_url, api_key="not-needed")
|
||||
self.model = os.getenv("LLAMA_MODEL_NAME", "llama-3.1-8b-instruct")
|
||||
self.PROVIDER = "llama_server"
|
||||
logging.info("Using llama_server as LLM backend")
|
||||
else:
|
||||
self.client = OpenAI()
|
||||
self.model = "gpt-4o-mini"
|
||||
self.PROVIDER = "openai"
|
||||
logging.info("Using OpenAI as LLM backend")
|
||||
|
||||
def chat(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: str,
|
||||
):
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt,
|
||||
},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
client = LLMClient()
|
||||
print(client.chat(prompt="Hello!", system_prompt="You are a helpful assistant."))
|
||||
@@ -5,23 +5,17 @@ import os
|
||||
import sqlite3
|
||||
import time
|
||||
|
||||
import ollama
|
||||
from dotenv import load_dotenv
|
||||
|
||||
import chromadb
|
||||
from chunker import Chunker
|
||||
from cleaner import pdf_to_image, summarize_pdf_image
|
||||
from utils.chunker import Chunker
|
||||
from utils.cleaner import pdf_to_image, summarize_pdf_image
|
||||
from llm import LLMClient
|
||||
from query import QueryGenerator
|
||||
from request import PaperlessNGXService
|
||||
from scripts.query import QueryGenerator
|
||||
from utils.request import PaperlessNGXService
|
||||
|
||||
_dotenv_loaded = load_dotenv()
|
||||
|
||||
# Configure ollama client with URL from environment or default to localhost
|
||||
ollama_client = ollama.Client(
|
||||
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=10.0
|
||||
)
|
||||
|
||||
client = chromadb.PersistentClient(path=os.getenv("CHROMADB_PATH", ""))
|
||||
simba_docs = client.get_or_create_collection(name="simba_docs2")
|
||||
feline_vet_lookup = client.get_or_create_collection(name="feline_vet_lookup")
|
||||
72
migrations/models/1_20260131214411_None.py
Normal file
72
migrations/models/1_20260131214411_None.py
Normal file
@@ -0,0 +1,72 @@
|
||||
from tortoise import BaseDBAsyncClient
|
||||
|
||||
RUN_IN_TRANSACTION = True
|
||||
|
||||
|
||||
async def upgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
CREATE TABLE IF NOT EXISTS "users" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"username" VARCHAR(255) NOT NULL,
|
||||
"password" BYTEA,
|
||||
"email" VARCHAR(100) NOT NULL UNIQUE,
|
||||
"oidc_subject" VARCHAR(255) UNIQUE,
|
||||
"auth_provider" VARCHAR(50) NOT NULL DEFAULT 'local',
|
||||
"ldap_groups" JSONB NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS "idx_users_oidc_su_5aec5a" ON "users" ("oidc_subject");
|
||||
CREATE TABLE IF NOT EXISTS "conversations" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"name" VARCHAR(255) NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"user_id" UUID REFERENCES "users" ("id") ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS "conversation_messages" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"text" TEXT NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"speaker" VARCHAR(10) NOT NULL,
|
||||
"conversation_id" UUID NOT NULL REFERENCES "conversations" ("id") ON DELETE CASCADE
|
||||
);
|
||||
COMMENT ON COLUMN "conversation_messages"."speaker" IS 'USER: user\nSIMBA: simba';
|
||||
CREATE TABLE IF NOT EXISTS "aerich" (
|
||||
"id" SERIAL NOT NULL PRIMARY KEY,
|
||||
"version" VARCHAR(255) NOT NULL,
|
||||
"app" VARCHAR(100) NOT NULL,
|
||||
"content" JSONB NOT NULL
|
||||
);"""
|
||||
|
||||
|
||||
async def downgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
"""
|
||||
|
||||
|
||||
MODELS_STATE = (
|
||||
"eJztmm1v4jgQx78Kyquu1KtatnRX1emkQOkttwuceNinXhWZ2ICviZ1NnG1R1e9+tkmIkz"
|
||||
"gUKFDY401bxh5s/zzO/Mfpo+FSiJzgpEbJT+QHgGFKjMvSo0GAi/gf2vbjkgE8L2kVBgYG"
|
||||
"jnSwlZ6yBQwC5gOb8cYhcALETRAFto+9aDASOo4wUpt3xGSUmEKCf4TIYnSE2Bj5vOHmlp"
|
||||
"sxgegBBfFH784aYuTA1LwxFGNLu8UmnrT1+42ra9lTDDewbOqELkl6exM2pmTWPQwxPBE+"
|
||||
"om2ECPIBQ1BZhphltOzYNJ0xNzA/RLOpwsQA0RCEjoBh/D4MiS0YlORI4sf5H8YSeDhqgR"
|
||||
"YTJlg8Pk1XlaxZWg0xVO2D2Tl6e/FGrpIGbOTLRknEeJKOgIGpq+SagJS/cyhrY+DrUcb9"
|
||||
"MzD5RFfBGBsSjkkMxSBjQKtRM1zwYDmIjNiYfyxXKnMwfjY7kiTvJVFSHtfTqG9FTeVpm0"
|
||||
"CaILR9JJZsAZYHecVbGHaRHmbaM4MURq4n8R87CpivAbaJM4kOwRy+vUaz3u2Zzb/FStwg"
|
||||
"+OFIRGavLlrK0jrJWI8uMlsx+5LSl0bvQ0l8LH1vt+rZ2J/16303xJxAyKhF6L0FoHJeY2"
|
||||
"sMJrWxoQdX3Ni052FjX3Vjo8kr+xog31ougyguL0gj0dy2uImrJw2Reod32pwhYOThXVMf"
|
||||
"4RH5iCYSYYPPAxBblywi0dGPvmZXoSXWZBY+uJ+pETUo+Or4mhCbZk+zWzOv6oZkOAD23T"
|
||||
"3woVUA00VBAEYoyAOtRp7XHzvImUkzPUtVwDWn37ibT5UitpIVLVOFUYpevsktu1kLIHzd"
|
||||
"MBpbjDSHzjMqWIG4mBi21I08iOK9FsUMPWhSfo9b9Sjj/vsiiuel8vrXXiqLx9L3qGl+fZ"
|
||||
"PK5J/arT/j7opUrn1qVw8K+VcUUnmFHHgI3OnEgCgg6yR0c1IgtbuK+ysfHaPfrXcuSyKj"
|
||||
"/0O6jWbVvCwF2B0AY7EtTlWZZ6cLFJlnp4U1pmjKHCA10Sz3mNe4rvOZv6cS1s5ceL1Qym"
|
||||
"bvz3aW4rOaVhMuy2rbTSo5WTNopFtcSxRrNXG0D9ps/7WZ2MdlLy1Vn33RaFu4uPRAENxT"
|
||||
"XxOZVUyAP9HDVL0yMAcTNq1/drWk18GrCr2qyi2OrNpomZ1veskb91fjtvqtVzczdJELsL"
|
||||
"NMlM4c1hOiz5/4dQbo2eliomee6snJHoqhbQXh4F9kayqHYpJZv5WAZoN0uzw3cuC5lh9b"
|
||||
"nk9/Ylgk2vVAc47be4oaDrWB84I0lOZaWSRMK8VRWskFqQOBZ418GnqaO7y/uu2WHmnGLQ"
|
||||
"O0T/gqbyC22XHJwQG73Rjem9vNpHix8vkXCdk7g8wzVXzB4SLhf3KRcHjV9kts7OwmP1cQ"
|
||||
"PvcaJPd/Jet5F7LLYnS770BM5GN7bGhq56jleF71DJI+O1M+N0jBdby2ehaYM8EQ7fyrim"
|
||||
"j5Juq38tn5u/P3by/O3/MuciYzy7s5D4NGq/dMtSwOgvaKq1jrKS6HWjmRzvxoLCOYp933"
|
||||
"E+BGajk+IkNEk96LJbLi8lryeGO3DmuTx0tk2/Wnl6f/AHvgrXs="
|
||||
)
|
||||
25
mkdocs.yml
Normal file
25
mkdocs.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
site_name: SimbaRAG Documentation
|
||||
site_description: Documentation for SimbaRAG - RAG-powered conversational AI
|
||||
|
||||
theme:
|
||||
name: material
|
||||
features:
|
||||
- content.code.copy
|
||||
- navigation.sections
|
||||
- navigation.expand
|
||||
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
- pymdownx.superfences
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- tables
|
||||
- toc:
|
||||
permalink: true
|
||||
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Architecture:
|
||||
- Authentication: authentication.md
|
||||
@@ -9,7 +9,6 @@ dependencies = [
|
||||
"python-dotenv>=1.0.0",
|
||||
"flask>=3.1.2",
|
||||
"httpx>=0.28.1",
|
||||
"ollama>=0.6.0",
|
||||
"openai>=2.0.1",
|
||||
"pydantic>=2.11.9",
|
||||
"pillow>=10.0.0",
|
||||
@@ -34,8 +33,8 @@ dependencies = [
|
||||
"langchain-chroma>=1.0.0",
|
||||
"langchain-community>=0.4.1",
|
||||
"jq>=1.10.0",
|
||||
"langchain-ollama>=1.0.1",
|
||||
"tavily-python>=0.7.17",
|
||||
"ynab>=1.3.0",
|
||||
]
|
||||
|
||||
[tool.aerich]
|
||||
@@ -35,12 +35,14 @@ class ConversationService {
|
||||
async sendQuery(
|
||||
query: string,
|
||||
conversation_id: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<QueryResponse> {
|
||||
const response = await userService.fetchWithRefreshToken(
|
||||
`${this.conversationBaseUrl}/query`,
|
||||
{
|
||||
method: "POST",
|
||||
body: JSON.stringify({ query, conversation_id }),
|
||||
signal,
|
||||
},
|
||||
);
|
||||
|
||||
|
Before Width: | Height: | Size: 5.8 KiB After Width: | Height: | Size: 5.8 KiB |
|
Before Width: | Height: | Size: 163 B After Width: | Height: | Size: 163 B |
@@ -43,12 +43,26 @@ export const ChatScreen = ({ setAuthenticated }: ChatScreenProps) => {
|
||||
const [isLoading, setIsLoading] = useState<boolean>(false);
|
||||
|
||||
const messagesEndRef = useRef<HTMLDivElement>(null);
|
||||
const isMountedRef = useRef<boolean>(true);
|
||||
const abortControllerRef = useRef<AbortController | null>(null);
|
||||
const simbaAnswers = ["meow.", "hiss...", "purrrrrr", "yowOWROWWowowr"];
|
||||
|
||||
const scrollToBottom = () => {
|
||||
messagesEndRef.current?.scrollIntoView({ behavior: "smooth" });
|
||||
};
|
||||
|
||||
// Cleanup effect to handle component unmounting
|
||||
useEffect(() => {
|
||||
isMountedRef.current = true;
|
||||
return () => {
|
||||
isMountedRef.current = false;
|
||||
// Abort any pending requests when component unmounts
|
||||
if (abortControllerRef.current) {
|
||||
abortControllerRef.current.abort();
|
||||
}
|
||||
};
|
||||
}, []);
|
||||
|
||||
const handleSelectConversation = (conversation: Conversation) => {
|
||||
setShowConversations(false);
|
||||
setSelectedConversation(conversation);
|
||||
@@ -156,10 +170,15 @@ export const ChatScreen = ({ setAuthenticated }: ChatScreenProps) => {
|
||||
return;
|
||||
}
|
||||
|
||||
// Create a new AbortController for this request
|
||||
const abortController = new AbortController();
|
||||
abortControllerRef.current = abortController;
|
||||
|
||||
try {
|
||||
const result = await conversationService.sendQuery(
|
||||
query,
|
||||
selectedConversation.id,
|
||||
abortController.signal,
|
||||
);
|
||||
setQuestionsAnswers(
|
||||
questionsAnswers.concat([{ question: query, answer: result.response }]),
|
||||
@@ -168,14 +187,24 @@ export const ChatScreen = ({ setAuthenticated }: ChatScreenProps) => {
|
||||
currMessages.concat([{ text: result.response, speaker: "simba" }]),
|
||||
);
|
||||
} catch (error) {
|
||||
// Ignore abort errors (these are intentional cancellations)
|
||||
if (error instanceof Error && error.name === "AbortError") {
|
||||
console.log("Request was aborted");
|
||||
} else {
|
||||
console.error("Failed to send query:", error);
|
||||
// If session expired, redirect to login
|
||||
if (error instanceof Error && error.message.includes("Session expired")) {
|
||||
setAuthenticated(false);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
// Only update loading state if component is still mounted
|
||||
if (isMountedRef.current) {
|
||||
setIsLoading(false);
|
||||
}
|
||||
// Clear the abort controller reference
|
||||
abortControllerRef.current = null;
|
||||
}
|
||||
};
|
||||
|
||||
const handleQueryChange = (event: React.ChangeEvent<HTMLTextAreaElement>) => {
|
||||
|
Before Width: | Height: | Size: 3.4 MiB After Width: | Height: | Size: 3.4 MiB |
|
Before Width: | Height: | Size: 2.1 MiB After Width: | Height: | Size: 2.1 MiB |
0
scripts/__init__.py
Normal file
0
scripts/__init__.py
Normal file
@@ -4,9 +4,14 @@ import sqlite3
|
||||
|
||||
import httpx
|
||||
from dotenv import load_dotenv
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from image_process import describe_simba_image
|
||||
from request import PaperlessNGXService
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from utils.image_process import describe_simba_image
|
||||
from utils.request import PaperlessNGXService
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
@@ -1,18 +1,11 @@
|
||||
import json
|
||||
import os
|
||||
from typing import Literal
|
||||
import datetime
|
||||
from ollama import Client
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# Configure ollama client with URL from environment or default to localhost
|
||||
ollama_client = Client(
|
||||
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=10.0
|
||||
)
|
||||
|
||||
# This uses inferred filters — which means using LLM to create the metadata filters
|
||||
|
||||
|
||||
79
scripts/user_message_stats.py
Normal file
79
scripts/user_message_stats.py
Normal file
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Script to show how many messages each user has written
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from tortoise import Tortoise
|
||||
from blueprints.users.models import User
|
||||
from blueprints.conversation.models import Speaker
|
||||
import os
|
||||
|
||||
|
||||
async def get_user_message_stats():
|
||||
"""Get message count statistics per user"""
|
||||
|
||||
# Initialize database connection
|
||||
database_url = os.getenv("DATABASE_URL", "sqlite://raggr.db")
|
||||
await Tortoise.init(
|
||||
db_url=database_url,
|
||||
modules={
|
||||
"models": [
|
||||
"blueprints.users.models",
|
||||
"blueprints.conversation.models",
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
print("\n📊 User Message Statistics\n")
|
||||
print(
|
||||
f"{'Username':<20} {'Total Messages':<15} {'User Messages':<15} {'Conversations':<15}"
|
||||
)
|
||||
print("=" * 70)
|
||||
|
||||
# Get all users
|
||||
users = await User.all()
|
||||
|
||||
total_users = 0
|
||||
total_messages = 0
|
||||
|
||||
for user in users:
|
||||
# Get all conversations for this user
|
||||
conversations = await user.conversations.all()
|
||||
|
||||
if not conversations:
|
||||
continue
|
||||
|
||||
total_users += 1
|
||||
|
||||
# Count messages across all conversations
|
||||
user_message_count = 0
|
||||
total_message_count = 0
|
||||
|
||||
for conversation in conversations:
|
||||
messages = await conversation.messages.all()
|
||||
total_message_count += len(messages)
|
||||
|
||||
# Count only user messages (not assistant responses)
|
||||
user_messages = [msg for msg in messages if msg.speaker == Speaker.USER]
|
||||
user_message_count += len(user_messages)
|
||||
|
||||
total_messages += user_message_count
|
||||
|
||||
print(
|
||||
f"{user.username:<20} {total_message_count:<15} {user_message_count:<15} {len(conversations):<15}"
|
||||
)
|
||||
|
||||
print("=" * 70)
|
||||
print("\n📈 Summary:")
|
||||
print(f" Total active users: {total_users}")
|
||||
print(f" Total user messages: {total_messages}")
|
||||
print(
|
||||
f" Average messages per user: {total_messages / total_users if total_users > 0 else 0:.1f}\n"
|
||||
)
|
||||
|
||||
await Tortoise.close_connections()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(get_user_message_stats())
|
||||
@@ -1,16 +0,0 @@
|
||||
.git
|
||||
.gitignore
|
||||
README.md
|
||||
.env
|
||||
.DS_Store
|
||||
chromadb/
|
||||
chroma_db/
|
||||
raggr-frontend/node_modules/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
.pytest_cache/
|
||||
@@ -1 +0,0 @@
|
||||
3.13
|
||||
@@ -1,78 +0,0 @@
|
||||
import os
|
||||
from typing import cast
|
||||
|
||||
from langchain.agents import create_agent
|
||||
from langchain.chat_models import BaseChatModel
|
||||
from langchain.tools import tool
|
||||
from langchain_ollama import ChatOllama
|
||||
from langchain_openai import ChatOpenAI
|
||||
from tavily import AsyncTavilyClient
|
||||
|
||||
from blueprints.rag.logic import query_vector_store
|
||||
|
||||
openai_gpt_5_mini = ChatOpenAI(model="gpt-5-mini")
|
||||
ollama_deepseek = ChatOllama(model="llama3.1:8b", base_url=os.getenv("OLLAMA_URL"))
|
||||
model_with_fallback = cast(
|
||||
BaseChatModel, ollama_deepseek.with_fallbacks([openai_gpt_5_mini])
|
||||
)
|
||||
client = AsyncTavilyClient(os.getenv("TAVILY_KEY"), "")
|
||||
|
||||
|
||||
@tool
|
||||
async def web_search(query: str) -> str:
|
||||
"""Search the web for current information using Tavily.
|
||||
|
||||
Use this tool when you need to:
|
||||
- Find current information not in the knowledge base
|
||||
- Look up recent events, news, or updates
|
||||
- Verify facts or get additional context
|
||||
- Search for information outside of Simba's documents
|
||||
|
||||
Args:
|
||||
query: The search query to look up on the web
|
||||
|
||||
Returns:
|
||||
Search results from the web with titles, content, and source URLs
|
||||
"""
|
||||
response = await client.search(query=query, search_depth="basic")
|
||||
results = response.get("results", [])
|
||||
|
||||
if not results:
|
||||
return "No results found for the query."
|
||||
|
||||
formatted = "\n\n".join(
|
||||
[
|
||||
f"**{result['title']}**\n{result['content']}\nSource: {result['url']}"
|
||||
for result in results[:5]
|
||||
]
|
||||
)
|
||||
return formatted
|
||||
|
||||
|
||||
@tool(response_format="content_and_artifact")
|
||||
async def simba_search(query: str):
|
||||
"""Search through Simba's medical records, veterinary documents, and personal information.
|
||||
|
||||
Use this tool whenever the user asks questions about:
|
||||
- Simba's health history, medical records, or veterinary visits
|
||||
- Medications, treatments, or diagnoses
|
||||
- Weight, diet, or physical characteristics over time
|
||||
- Veterinary recommendations or advice
|
||||
- Ryan's (the owner's) information related to Simba
|
||||
- Any factual information that would be found in documents
|
||||
|
||||
Args:
|
||||
query: The user's question or information need about Simba
|
||||
|
||||
Returns:
|
||||
Relevant information from Simba's documents
|
||||
"""
|
||||
print(f"[SIMBA SEARCH] Tool called with query: {query}")
|
||||
serialized, docs = await query_vector_store(query=query)
|
||||
print(f"[SIMBA SEARCH] Found {len(docs)} documents")
|
||||
print(f"[SIMBA SEARCH] Serialized result length: {len(serialized)}")
|
||||
print(f"[SIMBA SEARCH] First 200 chars: {serialized[:200]}")
|
||||
return serialized, docs
|
||||
|
||||
|
||||
main_agent = create_agent(model=model_with_fallback, tools=[simba_search, web_search])
|
||||
@@ -1,73 +0,0 @@
|
||||
import os
|
||||
|
||||
from ollama import Client
|
||||
from openai import OpenAI
|
||||
|
||||
import logging
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
TRY_OLLAMA = os.getenv("TRY_OLLAMA", False)
|
||||
|
||||
|
||||
class LLMClient:
|
||||
def __init__(self):
|
||||
try:
|
||||
self.ollama_client = Client(
|
||||
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=1.0
|
||||
)
|
||||
self.ollama_client.chat(
|
||||
model="gemma3:4b", messages=[{"role": "system", "content": "test"}]
|
||||
)
|
||||
self.PROVIDER = "ollama"
|
||||
logging.info("Using Ollama as LLM backend")
|
||||
except Exception as e:
|
||||
print(e)
|
||||
self.openai_client = OpenAI()
|
||||
self.PROVIDER = "openai"
|
||||
logging.info("Using OpenAI as LLM backend")
|
||||
|
||||
def chat(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: str,
|
||||
):
|
||||
# Instituting a fallback if my gaming PC is not on
|
||||
if self.PROVIDER == "ollama":
|
||||
try:
|
||||
response = self.ollama_client.chat(
|
||||
model="gemma3:4b",
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt,
|
||||
},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
)
|
||||
output = response.message.content
|
||||
return output
|
||||
except Exception as e:
|
||||
logging.error(f"Could not connect to OLLAMA: {str(e)}")
|
||||
|
||||
response = self.openai_client.responses.create(
|
||||
model="gpt-4o-mini",
|
||||
input=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt,
|
||||
},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
)
|
||||
output = response.output_text
|
||||
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
client = Client()
|
||||
client.chat(model="gemma3:4b", messages=[{"role": "system", "promp": "hack"}])
|
||||
@@ -1,71 +0,0 @@
|
||||
from tortoise import BaseDBAsyncClient
|
||||
|
||||
RUN_IN_TRANSACTION = True
|
||||
|
||||
|
||||
async def upgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
CREATE TABLE IF NOT EXISTS "users" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"username" VARCHAR(255) NOT NULL,
|
||||
"password" BYTEA,
|
||||
"email" VARCHAR(100) NOT NULL UNIQUE,
|
||||
"oidc_subject" VARCHAR(255) UNIQUE,
|
||||
"auth_provider" VARCHAR(50) NOT NULL DEFAULT 'local',
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS "idx_users_oidc_su_5aec5a" ON "users" ("oidc_subject");
|
||||
CREATE TABLE IF NOT EXISTS "conversations" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"name" VARCHAR(255) NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"user_id" UUID REFERENCES "users" ("id") ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS "conversation_messages" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"text" TEXT NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"speaker" VARCHAR(10) NOT NULL,
|
||||
"conversation_id" UUID NOT NULL REFERENCES "conversations" ("id") ON DELETE CASCADE
|
||||
);
|
||||
COMMENT ON COLUMN "conversation_messages"."speaker" IS 'USER: user\nSIMBA: simba';
|
||||
CREATE TABLE IF NOT EXISTS "aerich" (
|
||||
"id" SERIAL NOT NULL PRIMARY KEY,
|
||||
"version" VARCHAR(255) NOT NULL,
|
||||
"app" VARCHAR(100) NOT NULL,
|
||||
"content" JSONB NOT NULL
|
||||
);"""
|
||||
|
||||
|
||||
async def downgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
"""
|
||||
|
||||
|
||||
MODELS_STATE = (
|
||||
"eJztmmtP4zgUhv9KlE+MxCLoUGaEViulpex0Z9qO2nR3LjuK3MRtvSROJnYGKsR/X9u5J0"
|
||||
"56AUqL+gXosU9sPz7OeY/Lveq4FrTJSdvFv6BPAEUuVi+VexUDB7I/pO3Higo8L23lBgom"
|
||||
"tnAwMz1FC5gQ6gOTssYpsAlkJgsS00deNBgObJsbXZN1RHiWmgKMfgbQoO4M0jn0WcP3H8"
|
||||
"yMsAXvIIk/ejfGFEHbys0bWXxsYTfowhO28bh7dS168uEmhunagYPT3t6Czl2cdA8CZJ1w"
|
||||
"H942gxj6gEIrsww+y2jZsSmcMTNQP4DJVK3UYMEpCGwOQ/19GmCTM1DESPzH+R/qGngYao"
|
||||
"4WYcpZ3D+Eq0rXLKwqH6r9QRsevb14I1bpEjrzRaMgoj4IR0BB6Cq4piDF7xLK9hz4cpRx"
|
||||
"/wJMNtFNMMaGlGMaQzHIGNBm1FQH3Bk2xDM6Zx8bzWYNxr+1oSDJegmULovrMOr7UVMjbO"
|
||||
"NIU4SmD/mSDUDLIK9YC0UOlMPMexaQWpHrSfzHjgJma7AG2F5Eh6CGr97tdUa61vvMV+IQ"
|
||||
"8tMWiDS9w1sawrooWI8uCluRPET5p6t/UPhH5dug3ynGftJP/6byOYGAugZ2bw1gZc5rbI"
|
||||
"3B5DY28KwNNzbvedjYF93YaPKZfSXQN9bLIBmXR6SRaG5b3MTNkwZPvdMbac7gMMrwrl0f"
|
||||
"ohn+CBcCYZfNA2BTliwi0TGOHrOr0FJrOgsf3CZqJBsUbHVsTZCG2VMbtbWrjioYToB5cw"
|
||||
"t8y6iA6UBCwAySMtBW5Hn9cQjtRJrJWWYFXC984m6+VarYClZuw80wytErNzkNp2gBmK3b"
|
||||
"isbmI9XQWaKCMxBXE8NGdiMPonivRTGFd5KUrzOrHGXcf19EcV0q73zRc1k8lr5HPe3Lm1"
|
||||
"wm/zTo/xl3z0jl9qdB66CQX6OQKitk4kFwIxMDvIDs4MApSYHc7mbcX/joqONRZ3ip8Iz+"
|
||||
"Lx51ey3tUiHImQB1tS3OVZlnpysUmWenlTUmbyocoGyiWe81L3F9ynf+nkpYs3Dh9UgpW7"
|
||||
"w/21mKSzWtJFzW1bbPqeREzSCRbnEtUa3V+NE+aLP912Z8H9e9tMz67ItG28LFpQcIuXV9"
|
||||
"SWS2EAb+Qg4z61WAOVnQsP7Z1ZJeBq/F9WpWbjFkrW5fG36VS964fzZuW1/1jlagCx2A7H"
|
||||
"WiNHF4mhBdfuKfMkDPTlcTPXWqpyR7XGSZBgkm/0FTUjlUkyz6bQS0GKTb5fksB55p+bnh"
|
||||
"+e4vZFWJdjnQkuP23qKq7ZrAfkQaynNtrhKmzeoobZa1+aG4fZ3F7eHrn1exscntcqlIWX"
|
||||
"Y1X/pfh6e5n99lgbTde3kN+sicq5J6Lmo5rqvoQNpnZ0q6Lq64IpZWdBxzIRiinX9RYSe+"
|
||||
"HfmtcXb+7vz924vz96yLmElieVfzMuj29SUVHD8I0muXav2RcTnUb6mcY0djHREXdt9PgM"
|
||||
"9SX7ARKcSS9P7XaNCvvE6NXQogx5gt8LuFTHqs2IjQH7uJtYYiX3X9dz/Fr3kKuZk/oCW7"
|
||||
"eN3mZeHD/9BpOYI="
|
||||
)
|
||||
12
users.py
12
users.py
@@ -1,12 +0,0 @@
|
||||
import sqlite3
|
||||
|
||||
|
||||
class User:
|
||||
def __init__(self, email: str, password_hash: str):
|
||||
self.email = email
|
||||
self.is_authenticated
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
connection = sqlite3.connect("users.db")
|
||||
c = connection.cursor()
|
||||
0
utils/__init__.py
Normal file
0
utils/__init__.py
Normal file
@@ -3,7 +3,6 @@ from math import ceil
|
||||
import re
|
||||
from typing import Union
|
||||
from uuid import UUID, uuid4
|
||||
from ollama import Client
|
||||
from chromadb.utils.embedding_functions.openai_embedding_function import (
|
||||
OpenAIEmbeddingFunction,
|
||||
)
|
||||
@@ -13,10 +12,6 @@ from llm import LLMClient
|
||||
|
||||
load_dotenv()
|
||||
|
||||
ollama_client = Client(
|
||||
host=os.getenv("OLLAMA_HOST", "http://localhost:11434"), timeout=1.0
|
||||
)
|
||||
|
||||
|
||||
def remove_headers_footers(text, header_patterns=None, footer_patterns=None):
|
||||
if header_patterns is None:
|
||||
@@ -8,7 +8,7 @@ import ollama
|
||||
from PIL import Image
|
||||
import fitz
|
||||
|
||||
from request import PaperlessNGXService
|
||||
from .request import PaperlessNGXService
|
||||
|
||||
load_dotenv()
|
||||
|
||||
342
utils/ynab_service.py
Normal file
342
utils/ynab_service.py
Normal file
@@ -0,0 +1,342 @@
|
||||
"""YNAB API service for querying budget data."""
|
||||
|
||||
import os
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Optional
|
||||
|
||||
from dotenv import load_dotenv
|
||||
import ynab
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class YNABService:
|
||||
"""Service for interacting with YNAB API."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize YNAB API client."""
|
||||
self.access_token = os.getenv("YNAB_ACCESS_TOKEN", "")
|
||||
self.budget_id = os.getenv("YNAB_BUDGET_ID", "")
|
||||
|
||||
if not self.access_token:
|
||||
raise ValueError("YNAB_ACCESS_TOKEN environment variable is required")
|
||||
|
||||
# Configure API client
|
||||
configuration = ynab.Configuration(access_token=self.access_token)
|
||||
self.api_client = ynab.ApiClient(configuration)
|
||||
|
||||
# Initialize API endpoints
|
||||
self.budgets_api = ynab.BudgetsApi(self.api_client)
|
||||
self.transactions_api = ynab.TransactionsApi(self.api_client)
|
||||
self.months_api = ynab.MonthsApi(self.api_client)
|
||||
self.categories_api = ynab.CategoriesApi(self.api_client)
|
||||
|
||||
# Get budget ID if not provided
|
||||
if not self.budget_id:
|
||||
budgets_response = self.budgets_api.get_budgets()
|
||||
if budgets_response.data and budgets_response.data.budgets:
|
||||
self.budget_id = budgets_response.data.budgets[0].id
|
||||
else:
|
||||
raise ValueError("No YNAB budgets found")
|
||||
|
||||
def get_budget_summary(self) -> dict[str, Any]:
|
||||
"""Get overall budget summary and health status.
|
||||
|
||||
Returns:
|
||||
Dictionary containing budget summary with to-be-budgeted amount,
|
||||
total budgeted, total activity, and overall budget health.
|
||||
"""
|
||||
budget_response = self.budgets_api.get_budget_by_id(self.budget_id)
|
||||
budget_data = budget_response.data.budget
|
||||
|
||||
# Calculate totals from categories
|
||||
to_be_budgeted = (
|
||||
budget_data.months[0].to_be_budgeted / 1000 if budget_data.months else 0
|
||||
)
|
||||
|
||||
total_budgeted = 0
|
||||
total_activity = 0
|
||||
total_available = 0
|
||||
|
||||
for category_group in budget_data.category_groups or []:
|
||||
if category_group.deleted or category_group.hidden:
|
||||
continue
|
||||
for category in category_group.categories or []:
|
||||
if category.deleted or category.hidden:
|
||||
continue
|
||||
total_budgeted += category.budgeted / 1000
|
||||
total_activity += category.activity / 1000
|
||||
total_available += category.balance / 1000
|
||||
|
||||
return {
|
||||
"budget_name": budget_data.name,
|
||||
"to_be_budgeted": round(to_be_budgeted, 2),
|
||||
"total_budgeted": round(total_budgeted, 2),
|
||||
"total_activity": round(total_activity, 2),
|
||||
"total_available": round(total_available, 2),
|
||||
"currency_format": budget_data.currency_format.iso_code
|
||||
if budget_data.currency_format
|
||||
else "USD",
|
||||
"summary": f"Budget '{budget_data.name}' has ${abs(to_be_budgeted):.2f} {'to be budgeted' if to_be_budgeted > 0 else 'overbudgeted'}. "
|
||||
f"Total budgeted: ${total_budgeted:.2f}, Total spent: ${abs(total_activity):.2f}, "
|
||||
f"Total available: ${total_available:.2f}.",
|
||||
}
|
||||
|
||||
def get_transactions(
|
||||
self,
|
||||
start_date: Optional[str] = None,
|
||||
end_date: Optional[str] = None,
|
||||
category_name: Optional[str] = None,
|
||||
payee_name: Optional[str] = None,
|
||||
limit: int = 50,
|
||||
) -> dict[str, Any]:
|
||||
"""Get transactions filtered by date range, category, or payee.
|
||||
|
||||
Args:
|
||||
start_date: Start date in YYYY-MM-DD format (defaults to 30 days ago)
|
||||
end_date: End date in YYYY-MM-DD format (defaults to today)
|
||||
category_name: Filter by category name (case-insensitive partial match)
|
||||
payee_name: Filter by payee name (case-insensitive partial match)
|
||||
limit: Maximum number of transactions to return (default 50)
|
||||
|
||||
Returns:
|
||||
Dictionary containing matching transactions and summary statistics.
|
||||
"""
|
||||
# Set default date range if not provided
|
||||
if not start_date:
|
||||
start_date = (datetime.now() - timedelta(days=30)).strftime("%Y-%m-%d")
|
||||
if not end_date:
|
||||
end_date = datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
# Get transactions
|
||||
transactions_response = self.transactions_api.get_transactions(
|
||||
self.budget_id, since_date=start_date
|
||||
)
|
||||
|
||||
transactions = transactions_response.data.transactions or []
|
||||
|
||||
# Filter by date range, category, and payee
|
||||
filtered_transactions = []
|
||||
total_amount = 0
|
||||
|
||||
for txn in transactions:
|
||||
# Skip if deleted or before start date or after end date
|
||||
if txn.deleted:
|
||||
continue
|
||||
txn_date = str(txn.date)
|
||||
if txn_date < start_date or txn_date > end_date:
|
||||
continue
|
||||
|
||||
# Filter by category if specified
|
||||
if category_name and txn.category_name:
|
||||
if category_name.lower() not in txn.category_name.lower():
|
||||
continue
|
||||
|
||||
# Filter by payee if specified
|
||||
if payee_name and txn.payee_name:
|
||||
if payee_name.lower() not in txn.payee_name.lower():
|
||||
continue
|
||||
|
||||
amount = txn.amount / 1000 # Convert milliunits to dollars
|
||||
filtered_transactions.append(
|
||||
{
|
||||
"date": txn_date,
|
||||
"payee": txn.payee_name,
|
||||
"category": txn.category_name,
|
||||
"memo": txn.memo,
|
||||
"amount": round(amount, 2),
|
||||
"approved": txn.approved,
|
||||
}
|
||||
)
|
||||
total_amount += amount
|
||||
|
||||
# Sort by date (most recent first) and limit
|
||||
filtered_transactions.sort(key=lambda x: x["date"], reverse=True)
|
||||
filtered_transactions = filtered_transactions[:limit]
|
||||
|
||||
return {
|
||||
"transactions": filtered_transactions,
|
||||
"count": len(filtered_transactions),
|
||||
"total_amount": round(total_amount, 2),
|
||||
"start_date": start_date,
|
||||
"end_date": end_date,
|
||||
"filters": {"category": category_name, "payee": payee_name},
|
||||
}
|
||||
|
||||
def get_category_spending(self, month: Optional[str] = None) -> dict[str, Any]:
|
||||
"""Get spending breakdown by category for a specific month.
|
||||
|
||||
Args:
|
||||
month: Month in YYYY-MM format (defaults to current month)
|
||||
|
||||
Returns:
|
||||
Dictionary containing spending by category and summary.
|
||||
"""
|
||||
if not month:
|
||||
month = datetime.now().strftime("%Y-%m-01")
|
||||
else:
|
||||
# Ensure format is YYYY-MM-01
|
||||
if len(month) == 7: # YYYY-MM
|
||||
month = f"{month}-01"
|
||||
|
||||
# Get budget month
|
||||
month_response = self.months_api.get_budget_month(self.budget_id, month)
|
||||
|
||||
month_data = month_response.data.month
|
||||
|
||||
categories_spending = []
|
||||
total_budgeted = 0
|
||||
total_activity = 0
|
||||
total_available = 0
|
||||
overspent_categories = []
|
||||
|
||||
for category in month_data.categories or []:
|
||||
if category.deleted or category.hidden:
|
||||
continue
|
||||
|
||||
budgeted = category.budgeted / 1000
|
||||
activity = category.activity / 1000
|
||||
available = category.balance / 1000
|
||||
|
||||
total_budgeted += budgeted
|
||||
total_activity += activity
|
||||
total_available += available
|
||||
|
||||
# Track overspent categories
|
||||
if available < 0:
|
||||
overspent_categories.append(
|
||||
{
|
||||
"name": category.name,
|
||||
"budgeted": round(budgeted, 2),
|
||||
"spent": round(abs(activity), 2),
|
||||
"overspent_by": round(abs(available), 2),
|
||||
}
|
||||
)
|
||||
|
||||
# Only include categories with activity
|
||||
if activity != 0:
|
||||
categories_spending.append(
|
||||
{
|
||||
"category": category.name,
|
||||
"budgeted": round(budgeted, 2),
|
||||
"activity": round(activity, 2),
|
||||
"available": round(available, 2),
|
||||
"goal_type": category.goal_type
|
||||
if hasattr(category, "goal_type")
|
||||
else None,
|
||||
}
|
||||
)
|
||||
|
||||
# Sort by absolute activity (highest spending first)
|
||||
categories_spending.sort(key=lambda x: abs(x["activity"]), reverse=True)
|
||||
|
||||
return {
|
||||
"month": month[:7], # Return YYYY-MM format
|
||||
"categories": categories_spending,
|
||||
"total_budgeted": round(total_budgeted, 2),
|
||||
"total_spent": round(abs(total_activity), 2),
|
||||
"total_available": round(total_available, 2),
|
||||
"overspent_categories": overspent_categories,
|
||||
"to_be_budgeted": round(month_data.to_be_budgeted / 1000, 2)
|
||||
if month_data.to_be_budgeted
|
||||
else 0,
|
||||
}
|
||||
|
||||
def get_spending_insights(self, months_back: int = 3) -> dict[str, Any]:
|
||||
"""Generate insights about spending patterns and budget health.
|
||||
|
||||
Args:
|
||||
months_back: Number of months to analyze (default 3)
|
||||
|
||||
Returns:
|
||||
Dictionary containing spending insights, trends, and recommendations.
|
||||
"""
|
||||
current_month = datetime.now()
|
||||
monthly_data = []
|
||||
|
||||
# Collect data for the last N months
|
||||
for i in range(months_back):
|
||||
month_date = current_month - timedelta(days=30 * i)
|
||||
month_str = month_date.strftime("%Y-%m-01")
|
||||
|
||||
try:
|
||||
month_spending = self.get_category_spending(month_str)
|
||||
monthly_data.append(
|
||||
{
|
||||
"month": month_str[:7],
|
||||
"total_spent": month_spending["total_spent"],
|
||||
"total_budgeted": month_spending["total_budgeted"],
|
||||
"overspent_categories": month_spending["overspent_categories"],
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
# Skip months that don't have data yet
|
||||
continue
|
||||
|
||||
if not monthly_data:
|
||||
return {"error": "No spending data available for analysis"}
|
||||
|
||||
# Calculate average spending
|
||||
avg_spending = sum(m["total_spent"] for m in monthly_data) / len(monthly_data)
|
||||
current_spending = monthly_data[0]["total_spent"] if monthly_data else 0
|
||||
|
||||
# Identify spending trend
|
||||
if len(monthly_data) >= 2:
|
||||
recent_avg = sum(m["total_spent"] for m in monthly_data[:2]) / 2
|
||||
older_avg = sum(m["total_spent"] for m in monthly_data[1:]) / (
|
||||
len(monthly_data) - 1
|
||||
)
|
||||
trend = (
|
||||
"increasing"
|
||||
if recent_avg > older_avg * 1.1
|
||||
else "decreasing"
|
||||
if recent_avg < older_avg * 0.9
|
||||
else "stable"
|
||||
)
|
||||
else:
|
||||
trend = "insufficient data"
|
||||
|
||||
# Find frequently overspent categories
|
||||
overspent_frequency = {}
|
||||
for month in monthly_data:
|
||||
for cat in month["overspent_categories"]:
|
||||
cat_name = cat["name"]
|
||||
if cat_name not in overspent_frequency:
|
||||
overspent_frequency[cat_name] = 0
|
||||
overspent_frequency[cat_name] += 1
|
||||
|
||||
frequently_overspent = [
|
||||
{"category": cat, "months_overspent": count}
|
||||
for cat, count in overspent_frequency.items()
|
||||
if count > 1
|
||||
]
|
||||
frequently_overspent.sort(key=lambda x: x["months_overspent"], reverse=True)
|
||||
|
||||
# Generate recommendations
|
||||
recommendations = []
|
||||
if current_spending > avg_spending * 1.2:
|
||||
recommendations.append(
|
||||
f"Current month spending (${current_spending:.2f}) is significantly higher than your {months_back}-month average (${avg_spending:.2f})"
|
||||
)
|
||||
|
||||
if frequently_overspent:
|
||||
top_overspent = frequently_overspent[0]
|
||||
recommendations.append(
|
||||
f"'{top_overspent['category']}' has been overspent in {top_overspent['months_overspent']} of the last {months_back} months"
|
||||
)
|
||||
|
||||
if trend == "increasing":
|
||||
recommendations.append(
|
||||
"Your spending trend is increasing. Consider reviewing your budget allocations."
|
||||
)
|
||||
|
||||
return {
|
||||
"analysis_period": f"Last {months_back} months",
|
||||
"average_monthly_spending": round(avg_spending, 2),
|
||||
"current_month_spending": round(current_spending, 2),
|
||||
"spending_trend": trend,
|
||||
"frequently_overspent_categories": frequently_overspent,
|
||||
"recommendations": recommendations,
|
||||
"monthly_breakdown": monthly_data,
|
||||
}
|
||||
18
services/raggr/uv.lock → uv.lock
generated
18
services/raggr/uv.lock → uv.lock
generated
@@ -2551,6 +2551,7 @@ dependencies = [
|
||||
{ name = "tomlkit" },
|
||||
{ name = "tortoise-orm" },
|
||||
{ name = "tortoise-orm-stubs" },
|
||||
{ name = "ynab" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
@@ -2586,6 +2587,7 @@ requires-dist = [
|
||||
{ name = "tomlkit", specifier = ">=0.13.3" },
|
||||
{ name = "tortoise-orm", specifier = ">=0.25.1" },
|
||||
{ name = "tortoise-orm-stubs", specifier = ">=1.0.2" },
|
||||
{ name = "ynab", specifier = ">=1.3.0" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3385,6 +3387,22 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/73/ae/b48f95715333080afb75a4504487cbe142cae1268afc482d06692d605ae6/yarl-1.22.0-py3-none-any.whl", hash = "sha256:1380560bdba02b6b6c90de54133c81c9f2a453dee9912fe58c1dcced1edb7cff", size = 46814, upload-time = "2025-10-06T14:12:53.872Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ynab"
|
||||
version = "1.9.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "certifi" },
|
||||
{ name = "pydantic" },
|
||||
{ name = "python-dateutil" },
|
||||
{ name = "typing-extensions" },
|
||||
{ name = "urllib3" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/9a/3e/36599ae876db3e1d32e393ab0934547df75bab70373c14ca5805246f99bc/ynab-1.9.0.tar.gz", hash = "sha256:fa50bdff641b3a273661e9f6e8a210f5ad98991a998dc09dec0a8122d734d1c6", size = 64898, upload-time = "2025-10-06T19:14:32.707Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b2/9c/0ccd11bcdf7522fcb2823fcd7ffbb48e3164d72caaf3f920c7b068347175/ynab-1.9.0-py3-none-any.whl", hash = "sha256:72ac0219605b4280149684ecd0fec3bd75d938772d65cdeea9b3e66a1b2f470d", size = 208674, upload-time = "2025-10-06T19:14:31.719Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zipp"
|
||||
version = "3.23.0"
|
||||
Reference in New Issue
Block a user