Compare commits
10 Commits
6eb3775e0f
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
6ae36b51a0 | ||
|
|
f0f72cce36 | ||
|
|
32020a6c60 | ||
|
|
713a058c4f | ||
|
|
12f7d9ead1 | ||
|
|
ad39904dda | ||
|
|
1fd2e860b2 | ||
|
|
7cfad5baba | ||
|
|
f68a79bdb7 | ||
|
|
52153cdf1e |
16
.env.example
16
.env.example
@@ -14,9 +14,10 @@ JWT_SECRET_KEY=your-secret-key-here
|
||||
PAPERLESS_TOKEN=your-paperless-token
|
||||
BASE_URL=192.168.1.5:8000
|
||||
|
||||
# Ollama Configuration
|
||||
OLLAMA_URL=http://192.168.1.14:11434
|
||||
OLLAMA_HOST=http://192.168.1.14:11434
|
||||
# llama-server Configuration (OpenAI-compatible API)
|
||||
# If set, uses llama-server as the primary LLM backend with OpenAI as fallback
|
||||
LLAMA_SERVER_URL=http://192.168.1.213:8080/v1
|
||||
LLAMA_MODEL_NAME=llama-3.1-8b-instruct
|
||||
|
||||
# ChromaDB Configuration
|
||||
# For Docker: This is automatically set to /app/data/chromadb
|
||||
@@ -26,6 +27,9 @@ CHROMADB_PATH=./data/chromadb
|
||||
# OpenAI Configuration
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# Tavily Configuration (for web search)
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
|
||||
# Immich Configuration
|
||||
IMMICH_URL=http://192.168.1.5:2283
|
||||
IMMICH_API_KEY=your-immich-api-key
|
||||
@@ -44,3 +48,9 @@ OIDC_USE_DISCOVERY=true
|
||||
# OIDC_TOKEN_ENDPOINT=https://auth.example.com/api/oidc/token
|
||||
# OIDC_USERINFO_ENDPOINT=https://auth.example.com/api/oidc/userinfo
|
||||
# OIDC_JWKS_URI=https://auth.example.com/api/oidc/jwks
|
||||
|
||||
# YNAB Configuration
|
||||
# Get your Personal Access Token from https://app.ynab.com/settings/developer
|
||||
YNAB_ACCESS_TOKEN=your-ynab-personal-access-token
|
||||
# Optional: Specify a budget ID, or leave empty to use the default/first budget
|
||||
YNAB_BUDGET_ID=
|
||||
|
||||
6
.pre-commit-config.yaml
Normal file
6
.pre-commit-config.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
repos:
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
rev: v0.8.2
|
||||
hooks:
|
||||
- id: ruff # Linter
|
||||
- id: ruff-format # Formatter
|
||||
109
CLAUDE.md
Normal file
109
CLAUDE.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
SimbaRAG is a RAG (Retrieval-Augmented Generation) conversational AI system for querying information about Simba (a cat). It ingests documents from Paperless-NGX, stores embeddings in ChromaDB, and uses LLMs (Ollama or OpenAI) to answer questions.
|
||||
|
||||
## Commands
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Start dev environment with hot reload
|
||||
docker compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# View logs
|
||||
docker compose -f docker-compose.dev.yml logs -f raggr
|
||||
```
|
||||
|
||||
### Database Migrations (Aerich/Tortoise ORM)
|
||||
|
||||
```bash
|
||||
# Generate migration (must run in Docker with DB access)
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name describe_change
|
||||
|
||||
# Apply migrations (auto-runs on startup, manual if needed)
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich upgrade
|
||||
|
||||
# View migration history
|
||||
docker compose exec raggr aerich history
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
yarn install
|
||||
yarn build # Production build
|
||||
yarn dev # Dev server (rarely needed, backend serves frontend)
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
docker compose build raggr
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Docker Compose │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ raggr (port 8080) │ postgres (port 5432) │
|
||||
│ ├── Quart backend │ PostgreSQL 16 │
|
||||
│ ├── React frontend (served) │ │
|
||||
│ └── ChromaDB (volume) │ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Backend** (root directory):
|
||||
- `app.py` - Quart application entry, serves API and static frontend
|
||||
- `main.py` - RAG logic, document indexing, LLM interaction, LangChain agent
|
||||
- `llm.py` - LLM client with Ollama primary, OpenAI fallback
|
||||
- `aerich_config.py` - Database migration configuration
|
||||
- `blueprints/` - API routes organized as Quart blueprints
|
||||
- `users/` - OIDC auth, JWT tokens, RBAC with LDAP groups
|
||||
- `conversation/` - Chat conversations and message history
|
||||
- `rag/` - Document indexing endpoints (admin-only)
|
||||
- `config/` - Configuration modules
|
||||
- `oidc_config.py` - OIDC authentication configuration
|
||||
- `utils/` - Reusable utilities
|
||||
- `chunker.py` - Document chunking for embeddings
|
||||
- `cleaner.py` - PDF cleaning and summarization
|
||||
- `image_process.py` - Image description with LLM
|
||||
- `request.py` - Paperless-NGX API client
|
||||
- `scripts/` - Administrative and utility scripts
|
||||
- `add_user.py` - Create users manually
|
||||
- `user_message_stats.py` - User message statistics
|
||||
- `manage_vectorstore.py` - Vector store management CLI
|
||||
- `inspect_vector_store.py` - Inspect ChromaDB contents
|
||||
- `query.py` - Query generation utilities
|
||||
- `migrations/` - Database migration files
|
||||
|
||||
**Frontend** (`raggr-frontend/`):
|
||||
- React 19 with Rsbuild bundler
|
||||
- Tailwind CSS for styling
|
||||
- Built to `dist/`, served by backend at `/`
|
||||
|
||||
**Auth Flow**: LLDAP → Authelia (OIDC) → Backend JWT → Frontend localStorage
|
||||
|
||||
## Key Patterns
|
||||
|
||||
- All endpoints are async (`async def`)
|
||||
- Use `@jwt_refresh_token_required` for authenticated endpoints
|
||||
- Use `@admin_required` for admin-only endpoints (checks `lldap_admin` group)
|
||||
- Tortoise ORM models in `blueprints/*/models.py`
|
||||
- Frontend API services in `raggr-frontend/src/api/`
|
||||
|
||||
## Environment Variables
|
||||
|
||||
See `.env.example`. Key ones:
|
||||
- `DATABASE_URL` - PostgreSQL connection
|
||||
- `OIDC_*` - Authelia OIDC configuration
|
||||
- `OLLAMA_URL` - Local LLM server
|
||||
- `OPENAI_API_KEY` - Fallback LLM
|
||||
- `PAPERLESS_TOKEN` / `BASE_URL` - Document source
|
||||
110
DEV-README.md
110
DEV-README.md
@@ -1,110 +0,0 @@
|
||||
# Development Environment Setup
|
||||
|
||||
This guide explains how to run the application in development mode with hot reload enabled.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Development Mode (Hot Reload)
|
||||
|
||||
```bash
|
||||
# Start all services in development mode
|
||||
docker-compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# Or run in detached mode
|
||||
docker-compose -f docker-compose.dev.yml up -d --build
|
||||
```
|
||||
|
||||
### Production Mode
|
||||
|
||||
```bash
|
||||
# Start production services
|
||||
docker-compose up --build
|
||||
```
|
||||
|
||||
## What's Different in Dev Mode?
|
||||
|
||||
### Backend (Quart/Flask)
|
||||
- **Hot Reload**: Python code changes are automatically detected and the server restarts
|
||||
- **Source Mounted**: Your local `services/raggr` directory is mounted as a volume
|
||||
- **Debug Mode**: Flask runs with `debug=True` for better error messages
|
||||
- **Environment**: `FLASK_ENV=development` and `PYTHONUNBUFFERED=1` for immediate log output
|
||||
|
||||
### Frontend (React + rsbuild)
|
||||
- **Auto Rebuild**: Frontend automatically rebuilds when files change
|
||||
- **Watch Mode**: rsbuild runs in watch mode, rebuilding to `dist/` on save
|
||||
- **Source Mounted**: Your local `services/raggr/raggr-frontend` directory is mounted as a volume
|
||||
- **Served by Backend**: Built files are served by the backend, no separate dev server
|
||||
|
||||
## Ports
|
||||
|
||||
- **Application**: 8080 (accessible at `http://localhost:8080` or `http://YOUR_IP:8080`)
|
||||
|
||||
The backend serves both the API and the auto-rebuilt frontend, making it accessible from other machines on your network.
|
||||
|
||||
## Useful Commands
|
||||
|
||||
```bash
|
||||
# View logs
|
||||
docker-compose -f docker-compose.dev.yml logs -f
|
||||
|
||||
# View logs for specific service
|
||||
docker-compose -f docker-compose.dev.yml logs -f raggr-backend
|
||||
docker-compose -f docker-compose.dev.yml logs -f raggr-frontend
|
||||
|
||||
# Rebuild after dependency changes
|
||||
docker-compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# Stop all services
|
||||
docker-compose -f docker-compose.dev.yml down
|
||||
|
||||
# Stop and remove volumes (fresh start)
|
||||
docker-compose -f docker-compose.dev.yml down -v
|
||||
```
|
||||
|
||||
## Making Changes
|
||||
|
||||
### Backend Changes
|
||||
1. Edit any Python file in `services/raggr/`
|
||||
2. Save the file
|
||||
3. The Quart server will automatically restart
|
||||
4. Check logs to confirm reload
|
||||
|
||||
### Frontend Changes
|
||||
1. Edit any file in `services/raggr/raggr-frontend/src/`
|
||||
2. Save the file
|
||||
3. The browser will automatically refresh (Hot Module Replacement)
|
||||
4. No need to rebuild
|
||||
|
||||
### Dependency Changes
|
||||
|
||||
**Backend** (pyproject.toml):
|
||||
```bash
|
||||
# Rebuild the backend service
|
||||
docker-compose -f docker-compose.dev.yml up --build raggr-backend
|
||||
```
|
||||
|
||||
**Frontend** (package.json):
|
||||
```bash
|
||||
# Rebuild the frontend service
|
||||
docker-compose -f docker-compose.dev.yml up --build raggr-frontend
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port Already in Use
|
||||
If you see port binding errors, make sure no other services are running on ports 8080 or 3000.
|
||||
|
||||
### Changes Not Reflected
|
||||
1. Check if the file is properly mounted (check docker-compose.dev.yml volumes)
|
||||
2. Verify the file isn't in an excluded directory (node_modules, __pycache__)
|
||||
3. Check container logs for errors
|
||||
|
||||
### Frontend Not Connecting to Backend
|
||||
Make sure your frontend API calls point to the correct backend URL. If accessing from the same machine, use `http://localhost:8080`. If accessing from another device on the network, use `http://YOUR_IP:8080`.
|
||||
|
||||
## Notes
|
||||
|
||||
- Both services bind to `0.0.0.0` and expose ports, making them accessible on your network
|
||||
- Node modules and Python cache are excluded from volume mounts to use container versions
|
||||
- Database and ChromaDB data persist in Docker volumes across restarts
|
||||
- Access the app from any device on your network using your host machine's IP address
|
||||
@@ -25,6 +25,9 @@ RUN uv pip install --system -e .
|
||||
COPY *.py ./
|
||||
COPY blueprints ./blueprints
|
||||
COPY migrations ./migrations
|
||||
COPY utils ./utils
|
||||
COPY config ./config
|
||||
COPY scripts ./scripts
|
||||
COPY startup.sh ./
|
||||
RUN chmod +x startup.sh
|
||||
|
||||
371
README.md
371
README.md
@@ -1,7 +1,370 @@
|
||||
# simbarag
|
||||
# SimbaRAG 🐱
|
||||
|
||||
**Goal:** Learn how retrieval-augmented generation works and also create a neat little tool to ask about Simba's health.
|
||||
A Retrieval-Augmented Generation (RAG) conversational AI system for querying information about Simba the cat. Built with LangChain, ChromaDB, and modern web technologies.
|
||||
|
||||
**Current objectives:**
|
||||
## Features
|
||||
|
||||
- [ ] Successfully use RAG to ask a question about existing information (e.g. how many teeth has Simba had extracted)
|
||||
- 🤖 **Intelligent Conversations** - LangChain-powered agent with tool use and memory
|
||||
- 📚 **Document Retrieval** - RAG system using ChromaDB vector store
|
||||
- 🔍 **Web Search** - Integrated Tavily API for real-time web searches
|
||||
- 🔐 **OIDC Authentication** - Secure auth via Authelia with LDAP group support
|
||||
- 💬 **Multi-Conversation** - Manage multiple conversation threads per user
|
||||
- 🎨 **Modern UI** - React 19 frontend with Tailwind CSS
|
||||
- 🐳 **Docker Ready** - Containerized deployment with Docker Compose
|
||||
|
||||
## System Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Client Layer"
|
||||
Browser[Web Browser]
|
||||
end
|
||||
|
||||
subgraph "Frontend - React"
|
||||
UI[React UI<br/>Tailwind CSS]
|
||||
Auth[Auth Service]
|
||||
API[API Client]
|
||||
end
|
||||
|
||||
subgraph "Backend - Quart/Python"
|
||||
App[Quart App<br/>app.py]
|
||||
|
||||
subgraph "Blueprints"
|
||||
Users[Users Blueprint<br/>OIDC + JWT]
|
||||
Conv[Conversation Blueprint<br/>Chat Management]
|
||||
RAG[RAG Blueprint<br/>Document Indexing]
|
||||
end
|
||||
|
||||
Agent[LangChain Agent<br/>main.py]
|
||||
LLM[LLM Client<br/>llm.py]
|
||||
end
|
||||
|
||||
subgraph "Tools & Utilities"
|
||||
Search[Simba Search Tool]
|
||||
Web[Web Search Tool<br/>Tavily]
|
||||
end
|
||||
|
||||
subgraph "Data Layer"
|
||||
Postgres[(PostgreSQL<br/>Users & Conversations)]
|
||||
Chroma[(ChromaDB<br/>Vector Store)]
|
||||
end
|
||||
|
||||
subgraph "External Services"
|
||||
Authelia[Authelia<br/>OIDC Provider]
|
||||
LLDAP[LLDAP<br/>User Directory]
|
||||
Ollama[Ollama<br/>Local LLM]
|
||||
OpenAI[OpenAI<br/>Fallback LLM]
|
||||
Paperless[Paperless-NGX<br/>Documents]
|
||||
TavilyAPI[Tavily API<br/>Web Search]
|
||||
end
|
||||
|
||||
Browser --> UI
|
||||
UI --> Auth
|
||||
UI --> API
|
||||
API --> App
|
||||
|
||||
App --> Users
|
||||
App --> Conv
|
||||
App --> RAG
|
||||
|
||||
Conv --> Agent
|
||||
Agent --> Search
|
||||
Agent --> Web
|
||||
Agent --> LLM
|
||||
|
||||
Search --> Chroma
|
||||
Web --> TavilyAPI
|
||||
RAG --> Chroma
|
||||
RAG --> Paperless
|
||||
|
||||
Users --> Postgres
|
||||
Conv --> Postgres
|
||||
|
||||
Users --> Authelia
|
||||
Authelia --> LLDAP
|
||||
|
||||
LLM --> Ollama
|
||||
LLM -.Fallback.-> OpenAI
|
||||
|
||||
style Browser fill:#e1f5ff
|
||||
style UI fill:#fff3cd
|
||||
style App fill:#d4edda
|
||||
style Agent fill:#d4edda
|
||||
style Postgres fill:#f8d7da
|
||||
style Chroma fill:#f8d7da
|
||||
style Ollama fill:#e2e3e5
|
||||
style OpenAI fill:#e2e3e5
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker & Docker Compose
|
||||
- PostgreSQL (or use Docker)
|
||||
- Ollama (optional, for local LLM)
|
||||
- Paperless-NGX instance (for document source)
|
||||
|
||||
### Installation
|
||||
|
||||
1. **Clone the repository**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/yourusername/simbarag.git
|
||||
cd simbarag
|
||||
```
|
||||
|
||||
2. **Configure environment variables**
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your configuration
|
||||
```
|
||||
|
||||
3. **Start the services**
|
||||
|
||||
```bash
|
||||
# Development (local PostgreSQL only)
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# Or full Docker deployment
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. **Access the application**
|
||||
|
||||
Open `http://localhost:8080` in your browser.
|
||||
|
||||
## Development
|
||||
|
||||
### Local Development Setup
|
||||
|
||||
```bash
|
||||
# 1. Start PostgreSQL
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# 2. Set environment variables
|
||||
export DATABASE_URL="postgres://raggr:raggr_dev_password@localhost:5432/raggr"
|
||||
export CHROMADB_PATH="./chromadb"
|
||||
export $(grep -v '^#' .env | xargs)
|
||||
|
||||
# 3. Install dependencies
|
||||
pip install -r requirements.txt
|
||||
cd raggr-frontend && yarn install && yarn build && cd ..
|
||||
|
||||
# 4. Run migrations
|
||||
aerich upgrade
|
||||
|
||||
# 5. Start the server
|
||||
python app.py
|
||||
```
|
||||
|
||||
See [docs/development.md](docs/development.md) for detailed development guide.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
simbarag/
|
||||
├── app.py # Quart application entry point
|
||||
├── main.py # RAG logic & LangChain agent
|
||||
├── llm.py # LLM client with Ollama/OpenAI
|
||||
├── aerich_config.py # Database migration configuration
|
||||
│
|
||||
├── blueprints/ # API route blueprints
|
||||
│ ├── users/ # Authentication & authorization
|
||||
│ ├── conversation/ # Chat conversations
|
||||
│ └── rag/ # Document indexing
|
||||
│
|
||||
├── config/ # Configuration modules
|
||||
│ └── oidc_config.py # OIDC authentication settings
|
||||
│
|
||||
├── utils/ # Reusable utilities
|
||||
│ ├── chunker.py # Document chunking for embeddings
|
||||
│ ├── cleaner.py # PDF cleaning and summarization
|
||||
│ ├── image_process.py # Image description with LLM
|
||||
│ └── request.py # Paperless-NGX API client
|
||||
│
|
||||
├── scripts/ # Administrative scripts
|
||||
│ ├── add_user.py
|
||||
│ ├── user_message_stats.py
|
||||
│ ├── manage_vectorstore.py
|
||||
│ └── inspect_vector_store.py
|
||||
│
|
||||
├── raggr-frontend/ # React frontend
|
||||
│ └── src/
|
||||
│
|
||||
├── migrations/ # Database migrations
|
||||
│
|
||||
├── docs/ # Documentation
|
||||
│ ├── index.md # Documentation hub
|
||||
│ ├── development.md # Development guide
|
||||
│ ├── deployment.md # Deployment & migrations
|
||||
│ ├── VECTORSTORE.md # Vector store management
|
||||
│ ├── MIGRATIONS.md # Migration reference
|
||||
│ └── authentication.md # Authentication setup
|
||||
│
|
||||
├── docker-compose.yml # Production compose
|
||||
├── docker-compose.dev.yml # Development compose
|
||||
├── Dockerfile # Production Dockerfile
|
||||
├── Dockerfile.dev # Development Dockerfile
|
||||
├── CLAUDE.md # AI assistant instructions
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Key Technologies
|
||||
|
||||
### Backend
|
||||
- **Quart** - Async Python web framework
|
||||
- **LangChain** - Agent framework with tool use
|
||||
- **Tortoise ORM** - Async ORM for PostgreSQL
|
||||
- **Aerich** - Database migration tool
|
||||
- **ChromaDB** - Vector database for embeddings
|
||||
- **OpenAI** - Embeddings & LLM (fallback)
|
||||
- **Ollama** - Local LLM (primary)
|
||||
|
||||
### Frontend
|
||||
- **React 19** - UI framework
|
||||
- **Rsbuild** - Fast bundler
|
||||
- **Tailwind CSS** - Utility-first styling
|
||||
- **Axios** - HTTP client
|
||||
|
||||
### Authentication
|
||||
- **Authelia** - OIDC provider
|
||||
- **LLDAP** - Lightweight LDAP server
|
||||
- **JWT** - Token-based auth
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Authentication
|
||||
- `GET /api/user/oidc/login` - Initiate OIDC login
|
||||
- `GET /api/user/oidc/callback` - OIDC callback handler
|
||||
- `POST /api/user/refresh` - Refresh JWT token
|
||||
|
||||
### Conversations
|
||||
- `POST /api/conversation/` - Create conversation
|
||||
- `GET /api/conversation/` - List conversations
|
||||
- `GET /api/conversation/<id>` - Get conversation with messages
|
||||
- `POST /api/conversation/query` - Send message and get response
|
||||
|
||||
### RAG (Admin Only)
|
||||
- `GET /api/rag/stats` - Vector store statistics
|
||||
- `POST /api/rag/index` - Index new documents
|
||||
- `POST /api/rag/reindex` - Clear and reindex all
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `DATABASE_URL` | PostgreSQL connection string | `postgres://...` |
|
||||
| `CHROMADB_PATH` | ChromaDB storage path | `./chromadb` |
|
||||
| `OLLAMA_URL` | Ollama server URL | `http://localhost:11434` |
|
||||
| `OPENAI_API_KEY` | OpenAI API key | - |
|
||||
| `PAPERLESS_TOKEN` | Paperless-NGX API token | - |
|
||||
| `BASE_URL` | Paperless-NGX base URL | - |
|
||||
| `OIDC_ISSUER` | OIDC provider URL | - |
|
||||
| `OIDC_CLIENT_ID` | OIDC client ID | - |
|
||||
| `OIDC_CLIENT_SECRET` | OIDC client secret | - |
|
||||
| `JWT_SECRET_KEY` | JWT signing key | - |
|
||||
| `TAVILY_KEY` | Tavily web search API key | - |
|
||||
|
||||
See `.env.example` for full list.
|
||||
|
||||
## Scripts
|
||||
|
||||
### User Management
|
||||
```bash
|
||||
# Add a new user
|
||||
python scripts/add_user.py
|
||||
|
||||
# View message statistics
|
||||
python scripts/user_message_stats.py
|
||||
```
|
||||
|
||||
### Vector Store Management
|
||||
```bash
|
||||
# Show vector store statistics
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index new documents from Paperless
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Clear and reindex everything
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
|
||||
# Inspect vector store contents
|
||||
python scripts/inspect_vector_store.py
|
||||
```
|
||||
|
||||
See [docs/vectorstore.md](docs/vectorstore.md) for details.
|
||||
|
||||
## Database Migrations
|
||||
|
||||
```bash
|
||||
# Generate a new migration
|
||||
aerich migrate --name "describe_your_changes"
|
||||
|
||||
# Apply pending migrations
|
||||
aerich upgrade
|
||||
|
||||
# View migration history
|
||||
aerich history
|
||||
|
||||
# Rollback last migration
|
||||
aerich downgrade
|
||||
```
|
||||
|
||||
See [docs/deployment.md](docs/deployment.md) for detailed migration workflows.
|
||||
|
||||
## LangChain Agent
|
||||
|
||||
The conversational agent has access to two tools:
|
||||
|
||||
1. **simba_search** - Query the vector store for Simba's documents
|
||||
- Used for: Medical records, veterinary history, factual information
|
||||
|
||||
2. **web_search** - Search the web via Tavily API
|
||||
- Used for: Recent events, external knowledge, general questions
|
||||
|
||||
The agent automatically selects the appropriate tool based on the user's query.
|
||||
|
||||
## Authentication Flow
|
||||
|
||||
```
|
||||
User → Authelia (OIDC) → Backend (JWT) → Frontend (localStorage)
|
||||
↓
|
||||
LLDAP
|
||||
```
|
||||
|
||||
1. User clicks "Login"
|
||||
2. Frontend redirects to Authelia
|
||||
3. User authenticates via Authelia (backed by LLDAP)
|
||||
4. Authelia redirects back with authorization code
|
||||
5. Backend exchanges code for OIDC tokens
|
||||
6. Backend issues JWT tokens
|
||||
7. Frontend stores tokens in localStorage
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Make your changes
|
||||
4. Run tests and linting
|
||||
5. Submit a pull request
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Development Guide](docs/development.md) - Setup and development workflow
|
||||
- [Deployment Guide](docs/deployment.md) - Deployment and migrations
|
||||
- [Vector Store Guide](docs/vectorstore.md) - Managing the vector database
|
||||
- [Authentication Guide](docs/authentication.md) - OIDC and LDAP setup
|
||||
|
||||
## License
|
||||
|
||||
[Your License Here]
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- Built for Simba, the most important cat in the world 🐱
|
||||
- Powered by LangChain, ChromaDB, and the open-source community
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
# Database configuration with environment variable support
|
||||
# Use DATABASE_PATH for relative paths or DATABASE_URL for full connection strings
|
||||
@@ -1,5 +1,6 @@
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from quart import Quart, jsonify, render_template, request, send_from_directory
|
||||
from quart_jwt_extended import JWTManager, get_jwt_identity, jwt_refresh_token_required
|
||||
from tortoise.contrib.quart import register_tortoise
|
||||
@@ -11,6 +12,9 @@ import blueprints.users
|
||||
import blueprints.users.models
|
||||
from main import consult_simba_oracle
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
app = Quart(
|
||||
__name__,
|
||||
static_folder="raggr-frontend/dist/static",
|
||||
@@ -52,7 +52,47 @@ async def query():
|
||||
messages_payload = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are a helpful cat assistant named Simba that understands veterinary terms. When there are questions to you specifically, they are referring to Simba the cat. Answer the user in as if you were a cat named Simba. Don't act too catlike. Be assertive.\n\nIMPORTANT: When users ask factual questions about Simba's health, medical history, veterinary visits, medications, weight, or any information that would be in documents, you MUST use the simba_search tool to retrieve accurate information before answering. Do not rely on general knowledge - always search the documents for factual questions.",
|
||||
"content": """You are a helpful cat assistant named Simba that understands veterinary terms. When there are questions to you specifically, they are referring to Simba the cat. Answer the user in as if you were a cat named Simba. Don't act too catlike. Be assertive.
|
||||
|
||||
SIMBA FACTS (as of January 2026):
|
||||
- Name: Simba
|
||||
- Species: Feline (Domestic Short Hair / American Short Hair)
|
||||
- Sex: Male, Neutered
|
||||
- Date of Birth: August 8, 2016 (approximately 9 years 5 months old)
|
||||
- Color: Orange
|
||||
- Current Weight: 16 lbs (as of 1/8/2026)
|
||||
- Owner: Ryan Chen
|
||||
- Location: Long Island City, NY
|
||||
- Veterinarian: Court Square Animal Hospital
|
||||
|
||||
Medical Conditions:
|
||||
- Hypertrophic Cardiomyopathy (HCM): Diagnosed 12/11/2025. Concentric left ventricular hypertrophy with no left atrial dilation. Grade II-III/VI systolic heart murmur. No cardiac medications currently needed. Must avoid Domitor, acepromazine, and ketamine during anesthesia.
|
||||
- Dental Issues: Prior extraction of teeth 307 and 407 due to resorption. Tooth 107 extracted on 1/8/2026. Early resorption lesions present on teeth 207, 309, and 409.
|
||||
|
||||
Recent Medical Events:
|
||||
- 1/8/2026: Dental cleaning and tooth 107 extraction. Prescribed Onsior for 3 days. Oravet sealant applied.
|
||||
- 12/11/2025: Echocardiogram confirming HCM diagnosis. Pre-op bloodwork was normal.
|
||||
- 12/1/2025: Visited for decreased appetite/nausea. Received subcutaneous fluids and Cerenia.
|
||||
|
||||
Diet & Lifestyle:
|
||||
- Diet: Hill's I/D wet and dry food
|
||||
- Supplements: Plaque Off
|
||||
- Indoor only cat, only pet in the household
|
||||
|
||||
Upcoming Appointments:
|
||||
- Rabies Vaccine: Due 2/19/2026
|
||||
- Routine Examination: Due 6/1/2026
|
||||
- FVRCP-3yr Vaccine: Due 10/2/2026
|
||||
|
||||
IMPORTANT: When users ask factual questions about Simba's health, medical history, veterinary visits, medications, weight, or any information that would be in documents, you MUST use the simba_search tool to retrieve accurate information before answering. Do not rely on general knowledge - always search the documents for factual questions.
|
||||
|
||||
BUDGET & FINANCE (YNAB Integration):
|
||||
You have access to Ryan's budget data through YNAB (You Need A Budget). When users ask about financial matters, use the appropriate YNAB tools:
|
||||
- Use ynab_budget_summary for overall budget health and status questions
|
||||
- Use ynab_search_transactions to find specific purchases or spending at particular stores
|
||||
- Use ynab_category_spending to analyze spending by category for a month
|
||||
- Use ynab_insights to provide spending trends, patterns, and recommendations
|
||||
Always use these tools when asked about budgets, spending, transactions, or financial health.""",
|
||||
}
|
||||
]
|
||||
|
||||
295
blueprints/conversation/agents.py
Normal file
295
blueprints/conversation/agents.py
Normal file
@@ -0,0 +1,295 @@
|
||||
import os
|
||||
from typing import cast
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from langchain.agents import create_agent
|
||||
from langchain.chat_models import BaseChatModel
|
||||
from langchain.tools import tool
|
||||
from langchain_openai import ChatOpenAI
|
||||
from tavily import AsyncTavilyClient
|
||||
|
||||
from blueprints.rag.logic import query_vector_store
|
||||
from utils.ynab_service import YNABService
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
# Configure LLM with llama-server or OpenAI fallback
|
||||
llama_url = os.getenv("LLAMA_SERVER_URL")
|
||||
if llama_url:
|
||||
llama_chat = ChatOpenAI(
|
||||
base_url=llama_url,
|
||||
api_key="not-needed",
|
||||
model=os.getenv("LLAMA_MODEL_NAME", "llama-3.1-8b-instruct"),
|
||||
)
|
||||
else:
|
||||
llama_chat = None
|
||||
|
||||
openai_fallback = ChatOpenAI(model="gpt-5-mini")
|
||||
model_with_fallback = cast(
|
||||
BaseChatModel,
|
||||
llama_chat.with_fallbacks([openai_fallback]) if llama_chat else openai_fallback,
|
||||
)
|
||||
client = AsyncTavilyClient(api_key=os.getenv("TAVILY_API_KEY", ""))
|
||||
|
||||
# Initialize YNAB service (will only work if YNAB_ACCESS_TOKEN is set)
|
||||
try:
|
||||
ynab_service = YNABService()
|
||||
ynab_enabled = True
|
||||
except (ValueError, Exception) as e:
|
||||
print(f"YNAB service not initialized: {e}")
|
||||
ynab_enabled = False
|
||||
|
||||
|
||||
@tool
|
||||
async def web_search(query: str) -> str:
|
||||
"""Search the web for current information using Tavily.
|
||||
|
||||
Use this tool when you need to:
|
||||
- Find current information not in the knowledge base
|
||||
- Look up recent events, news, or updates
|
||||
- Verify facts or get additional context
|
||||
- Search for information outside of Simba's documents
|
||||
|
||||
Args:
|
||||
query: The search query to look up on the web
|
||||
|
||||
Returns:
|
||||
Search results from the web with titles, content, and source URLs
|
||||
"""
|
||||
response = await client.search(query=query, search_depth="basic")
|
||||
results = response.get("results", [])
|
||||
|
||||
if not results:
|
||||
return "No results found for the query."
|
||||
|
||||
formatted = "\n\n".join(
|
||||
[
|
||||
f"**{result['title']}**\n{result['content']}\nSource: {result['url']}"
|
||||
for result in results[:5]
|
||||
]
|
||||
)
|
||||
return formatted
|
||||
|
||||
|
||||
@tool(response_format="content_and_artifact")
|
||||
async def simba_search(query: str):
|
||||
"""Search through Simba's medical records, veterinary documents, and personal information.
|
||||
|
||||
Use this tool whenever the user asks questions about:
|
||||
- Simba's health history, medical records, or veterinary visits
|
||||
- Medications, treatments, or diagnoses
|
||||
- Weight, diet, or physical characteristics over time
|
||||
- Veterinary recommendations or advice
|
||||
- Ryan's (the owner's) information related to Simba
|
||||
- Any factual information that would be found in documents
|
||||
|
||||
Args:
|
||||
query: The user's question or information need about Simba
|
||||
|
||||
Returns:
|
||||
Relevant information from Simba's documents
|
||||
"""
|
||||
print(f"[SIMBA SEARCH] Tool called with query: {query}")
|
||||
serialized, docs = await query_vector_store(query=query)
|
||||
print(f"[SIMBA SEARCH] Found {len(docs)} documents")
|
||||
print(f"[SIMBA SEARCH] Serialized result length: {len(serialized)}")
|
||||
print(f"[SIMBA SEARCH] First 200 chars: {serialized[:200]}")
|
||||
return serialized, docs
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_budget_summary() -> str:
|
||||
"""Get overall budget summary and health status from YNAB.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Overall budget health or status
|
||||
- How much money is to be budgeted
|
||||
- Total budget amounts or spending
|
||||
- General budget overview questions
|
||||
|
||||
Returns:
|
||||
Summary of budget health, to-be-budgeted amount, total budgeted,
|
||||
total activity, and available amounts.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
summary = ynab_service.get_budget_summary()
|
||||
return summary["summary"]
|
||||
except Exception as e:
|
||||
return f"Error fetching budget summary: {str(e)}"
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_search_transactions(
|
||||
start_date: str = "",
|
||||
end_date: str = "",
|
||||
category_name: str = "",
|
||||
payee_name: str = "",
|
||||
) -> str:
|
||||
"""Search YNAB transactions by date range, category, or payee.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Specific transactions or purchases
|
||||
- Spending at a particular store or payee
|
||||
- Transactions in a specific category
|
||||
- What was spent during a time period
|
||||
|
||||
Args:
|
||||
start_date: Start date in YYYY-MM-DD format (optional, defaults to 30 days ago)
|
||||
end_date: End date in YYYY-MM-DD format (optional, defaults to today)
|
||||
category_name: Filter by category name (optional, partial match)
|
||||
payee_name: Filter by payee/store name (optional, partial match)
|
||||
|
||||
Returns:
|
||||
List of matching transactions with dates, amounts, categories, and payees.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
result = ynab_service.get_transactions(
|
||||
start_date=start_date or None,
|
||||
end_date=end_date or None,
|
||||
category_name=category_name or None,
|
||||
payee_name=payee_name or None,
|
||||
)
|
||||
|
||||
if result["count"] == 0:
|
||||
return "No transactions found matching the specified criteria."
|
||||
|
||||
# Format transactions for readability
|
||||
txn_list = []
|
||||
for txn in result["transactions"][:10]: # Limit to 10 for readability
|
||||
txn_list.append(
|
||||
f"- {txn['date']}: {txn['payee']} - ${abs(txn['amount']):.2f} ({txn['category'] or 'Uncategorized'})"
|
||||
)
|
||||
|
||||
return (
|
||||
f"Found {result['count']} transactions from {result['start_date']} to {result['end_date']}. "
|
||||
f"Total: ${abs(result['total_amount']):.2f}\n\n"
|
||||
+ "\n".join(txn_list)
|
||||
+ (
|
||||
f"\n\n(Showing first 10 of {result['count']} transactions)"
|
||||
if result["count"] > 10
|
||||
else ""
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
return f"Error searching transactions: {str(e)}"
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_category_spending(month: str = "") -> str:
|
||||
"""Get spending breakdown by category for a specific month.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Spending by category
|
||||
- What categories were overspent
|
||||
- Monthly spending breakdown
|
||||
- Budget vs actual spending for a month
|
||||
|
||||
Args:
|
||||
month: Month in YYYY-MM format (optional, defaults to current month)
|
||||
|
||||
Returns:
|
||||
Spending breakdown by category with budgeted, spent, and available amounts.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
result = ynab_service.get_category_spending(month=month or None)
|
||||
|
||||
summary = (
|
||||
f"Budget spending for {result['month']}:\n"
|
||||
f"Total budgeted: ${result['total_budgeted']:.2f}\n"
|
||||
f"Total spent: ${result['total_spent']:.2f}\n"
|
||||
f"Total available: ${result['total_available']:.2f}\n"
|
||||
)
|
||||
|
||||
if result["overspent_categories"]:
|
||||
summary += (
|
||||
f"\nOverspent categories ({len(result['overspent_categories'])}):\n"
|
||||
)
|
||||
for cat in result["overspent_categories"][:5]:
|
||||
summary += f"- {cat['name']}: Budgeted ${cat['budgeted']:.2f}, Spent ${cat['spent']:.2f}, Over by ${cat['overspent_by']:.2f}\n"
|
||||
|
||||
# Add top spending categories
|
||||
summary += "\nTop spending categories:\n"
|
||||
for cat in result["categories"][:10]:
|
||||
if cat["activity"] < 0: # Only show spending (negative activity)
|
||||
summary += f"- {cat['category']}: ${abs(cat['activity']):.2f} (budgeted: ${cat['budgeted']:.2f}, available: ${cat['available']:.2f})\n"
|
||||
|
||||
return summary
|
||||
except Exception as e:
|
||||
return f"Error fetching category spending: {str(e)}"
|
||||
|
||||
|
||||
@tool
|
||||
def ynab_insights(months_back: int = 3) -> str:
|
||||
"""Generate insights about spending patterns and budget health over time.
|
||||
|
||||
Use this tool when the user asks about:
|
||||
- Spending trends or patterns
|
||||
- Budget recommendations
|
||||
- Which categories are frequently overspent
|
||||
- How current spending compares to past months
|
||||
- Overall budget health analysis
|
||||
|
||||
Args:
|
||||
months_back: Number of months to analyze (default 3, max 6)
|
||||
|
||||
Returns:
|
||||
Insights about spending trends, frequently overspent categories,
|
||||
and personalized recommendations.
|
||||
"""
|
||||
if not ynab_enabled:
|
||||
return "YNAB integration is not configured. Please set YNAB_ACCESS_TOKEN environment variable."
|
||||
|
||||
try:
|
||||
# Limit to reasonable range
|
||||
months_back = min(max(1, months_back), 6)
|
||||
result = ynab_service.get_spending_insights(months_back=months_back)
|
||||
|
||||
if "error" in result:
|
||||
return result["error"]
|
||||
|
||||
summary = (
|
||||
f"Spending insights for the last {months_back} months:\n\n"
|
||||
f"Average monthly spending: ${result['average_monthly_spending']:.2f}\n"
|
||||
f"Current month spending: ${result['current_month_spending']:.2f}\n"
|
||||
f"Spending trend: {result['spending_trend']}\n"
|
||||
)
|
||||
|
||||
if result["frequently_overspent_categories"]:
|
||||
summary += "\nFrequently overspent categories:\n"
|
||||
for cat in result["frequently_overspent_categories"][:5]:
|
||||
summary += f"- {cat['category']}: overspent in {cat['months_overspent']} of {months_back} months\n"
|
||||
|
||||
if result["recommendations"]:
|
||||
summary += "\nRecommendations:\n"
|
||||
for rec in result["recommendations"]:
|
||||
summary += f"- {rec}\n"
|
||||
|
||||
return summary
|
||||
except Exception as e:
|
||||
return f"Error generating insights: {str(e)}"
|
||||
|
||||
|
||||
# Create tools list based on what's available
|
||||
tools = [simba_search, web_search]
|
||||
if ynab_enabled:
|
||||
tools.extend(
|
||||
[
|
||||
ynab_budget_summary,
|
||||
ynab_search_transactions,
|
||||
ynab_category_spending,
|
||||
ynab_insights,
|
||||
]
|
||||
)
|
||||
|
||||
# Llama 3.1 supports native function calling via OpenAI-compatible API
|
||||
main_agent = create_agent(model=model_with_fallback, tools=tools)
|
||||
@@ -2,6 +2,7 @@ from quart import Blueprint, jsonify
|
||||
from quart_jwt_extended import jwt_refresh_token_required
|
||||
|
||||
from .logic import get_vector_store_stats, index_documents, vector_store
|
||||
from blueprints.users.decorators import admin_required
|
||||
|
||||
rag_blueprint = Blueprint("rag_api", __name__, url_prefix="/api/rag")
|
||||
|
||||
@@ -15,9 +16,9 @@ async def get_stats():
|
||||
|
||||
|
||||
@rag_blueprint.post("/index")
|
||||
@jwt_refresh_token_required
|
||||
@admin_required
|
||||
async def trigger_index():
|
||||
"""Trigger indexing of documents from Paperless-NGX."""
|
||||
"""Trigger indexing of documents from Paperless-NGX. Admin only."""
|
||||
try:
|
||||
await index_documents()
|
||||
stats = get_vector_store_stats()
|
||||
@@ -27,9 +28,9 @@ async def trigger_index():
|
||||
|
||||
|
||||
@rag_blueprint.post("/reindex")
|
||||
@jwt_refresh_token_required
|
||||
@admin_required
|
||||
async def trigger_reindex():
|
||||
"""Clear and reindex all documents."""
|
||||
"""Clear and reindex all documents. Admin only."""
|
||||
try:
|
||||
# Clear existing documents
|
||||
collection = vector_store._collection
|
||||
@@ -1,8 +1,12 @@
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
from dotenv import load_dotenv
|
||||
import httpx
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class PaperlessNGXService:
|
||||
def __init__(self):
|
||||
@@ -1,6 +1,7 @@
|
||||
import datetime
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from langchain_chroma import Chroma
|
||||
from langchain_core.documents import Document
|
||||
from langchain_openai import OpenAIEmbeddings
|
||||
@@ -8,6 +9,9 @@ from langchain_text_splitters import RecursiveCharacterTextSplitter
|
||||
|
||||
from .fetchers import PaperlessNGXService
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
|
||||
|
||||
vector_store = Chroma(
|
||||
@@ -7,7 +7,7 @@ from quart_jwt_extended import (
|
||||
)
|
||||
from .models import User
|
||||
from .oidc_service import OIDCUserService
|
||||
from oidc_config import oidc_config
|
||||
from config.oidc_config import oidc_config
|
||||
import secrets
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
@@ -60,7 +60,7 @@ async def oidc_login():
|
||||
"client_id": oidc_config.client_id,
|
||||
"response_type": "code",
|
||||
"redirect_uri": oidc_config.redirect_uri,
|
||||
"scope": "openid email profile",
|
||||
"scope": "openid email profile groups",
|
||||
"state": state,
|
||||
"code_challenge": code_challenge,
|
||||
"code_challenge_method": "S256",
|
||||
@@ -115,7 +115,9 @@ async def oidc_callback():
|
||||
token_response = await client.post(token_endpoint, data=token_data)
|
||||
|
||||
if token_response.status_code != 200:
|
||||
return jsonify({"error": f"Failed to exchange code for token: {token_response.text}"}), 400
|
||||
return jsonify(
|
||||
{"error": f"Failed to exchange code for token: {token_response.text}"}
|
||||
), 400
|
||||
|
||||
tokens = token_response.json()
|
||||
|
||||
@@ -141,7 +143,13 @@ async def oidc_callback():
|
||||
return jsonify(
|
||||
access_token=access_token,
|
||||
refresh_token=refresh_token,
|
||||
user={"id": str(user.id), "username": user.username, "email": user.email},
|
||||
user={
|
||||
"id": str(user.id),
|
||||
"username": user.username,
|
||||
"email": user.email,
|
||||
"groups": user.ldap_groups,
|
||||
"is_admin": user.is_admin(),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
26
blueprints/users/decorators.py
Normal file
26
blueprints/users/decorators.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""
|
||||
Authentication decorators for role-based access control.
|
||||
"""
|
||||
|
||||
from functools import wraps
|
||||
from quart import jsonify
|
||||
from quart_jwt_extended import jwt_refresh_token_required, get_jwt_identity
|
||||
from .models import User
|
||||
|
||||
|
||||
def admin_required(fn):
|
||||
"""
|
||||
Decorator that requires the user to be an admin (member of lldap_admin group).
|
||||
Must be used on async route handlers.
|
||||
"""
|
||||
|
||||
@wraps(fn)
|
||||
@jwt_refresh_token_required
|
||||
async def wrapper(*args, **kwargs):
|
||||
user_id = get_jwt_identity()
|
||||
user = await User.get_or_none(id=user_id)
|
||||
if not user or not user.is_admin():
|
||||
return jsonify({"error": "Admin access required"}), 403
|
||||
return await fn(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
@@ -12,8 +12,13 @@ class User(Model):
|
||||
email = fields.CharField(max_length=100, unique=True)
|
||||
|
||||
# OIDC fields
|
||||
oidc_subject = fields.CharField(max_length=255, unique=True, null=True, index=True) # "sub" claim from OIDC
|
||||
auth_provider = fields.CharField(max_length=50, default="local") # "local" or "oidc"
|
||||
oidc_subject = fields.CharField(
|
||||
max_length=255, unique=True, null=True, index=True
|
||||
) # "sub" claim from OIDC
|
||||
auth_provider = fields.CharField(
|
||||
max_length=50, default="local"
|
||||
) # "local" or "oidc"
|
||||
ldap_groups = fields.JSONField(default=[]) # LDAP groups from OIDC claims
|
||||
|
||||
created_at = fields.DatetimeField(auto_now_add=True)
|
||||
updated_at = fields.DatetimeField(auto_now=True)
|
||||
@@ -21,6 +26,14 @@ class User(Model):
|
||||
class Meta:
|
||||
table = "users"
|
||||
|
||||
def has_group(self, group: str) -> bool:
|
||||
"""Check if user belongs to a specific LDAP group."""
|
||||
return group in (self.ldap_groups or [])
|
||||
|
||||
def is_admin(self) -> bool:
|
||||
"""Check if user is an admin (member of lldap_admin group)."""
|
||||
return self.has_group("lldap_admin")
|
||||
|
||||
def set_password(self, plain_password: str):
|
||||
self.password = bcrypt.hashpw(
|
||||
plain_password.encode("utf-8"),
|
||||
@@ -1,6 +1,7 @@
|
||||
"""
|
||||
OIDC User Management Service
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional
|
||||
from uuid import uuid4
|
||||
from .models import User
|
||||
@@ -31,10 +32,10 @@ class OIDCUserService:
|
||||
# Update user info from latest claims (optional)
|
||||
user.email = claims.get("email", user.email)
|
||||
user.username = (
|
||||
claims.get("preferred_username")
|
||||
or claims.get("name")
|
||||
or user.username
|
||||
claims.get("preferred_username") or claims.get("name") or user.username
|
||||
)
|
||||
# Update LDAP groups from claims
|
||||
user.ldap_groups = claims.get("groups", [])
|
||||
await user.save()
|
||||
return user
|
||||
|
||||
@@ -47,6 +48,7 @@ class OIDCUserService:
|
||||
user.oidc_subject = oidc_subject
|
||||
user.auth_provider = "oidc"
|
||||
user.password = None # Clear password
|
||||
user.ldap_groups = claims.get("groups", [])
|
||||
await user.save()
|
||||
return user
|
||||
|
||||
@@ -58,14 +60,17 @@ class OIDCUserService:
|
||||
or f"user_{oidc_subject[:8]}"
|
||||
)
|
||||
|
||||
# Extract LDAP groups from claims
|
||||
groups = claims.get("groups", [])
|
||||
|
||||
user = await User.create(
|
||||
id=uuid4(),
|
||||
username=username,
|
||||
email=email
|
||||
or f"{oidc_subject}@oidc.local", # Fallback if no email claim
|
||||
email=email or f"{oidc_subject}@oidc.local", # Fallback if no email claim
|
||||
oidc_subject=oidc_subject,
|
||||
auth_provider="oidc",
|
||||
password=None,
|
||||
ldap_groups=groups,
|
||||
)
|
||||
|
||||
return user
|
||||
@@ -1,13 +0,0 @@
|
||||
import os
|
||||
|
||||
from llm import LLMClient
|
||||
|
||||
USE_OPENAI = os.getenv("OLLAMA_URL")
|
||||
|
||||
|
||||
class Classifier:
|
||||
def __init__(self):
|
||||
self.llm_client = LLMClient()
|
||||
|
||||
def classify_query_by_action(self, query):
|
||||
_prompt = "Classify the query into one of the following options: "
|
||||
0
config/__init__.py
Normal file
0
config/__init__.py
Normal file
@@ -1,11 +1,16 @@
|
||||
"""
|
||||
OIDC Configuration for Authelia Integration
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Dict, Any
|
||||
from authlib.jose import jwt
|
||||
from authlib.jose.errors import JoseError
|
||||
import httpx
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class OIDCConfig:
|
||||
@@ -15,52 +15,56 @@ services:
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
raggr:
|
||||
build:
|
||||
context: ./services/raggr
|
||||
dockerfile: Dockerfile.dev
|
||||
image: torrtle/simbarag:dev
|
||||
ports:
|
||||
- "8080:8080"
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
- PAPERLESS_TOKEN=${PAPERLESS_TOKEN}
|
||||
- BASE_URL=${BASE_URL}
|
||||
- OLLAMA_URL=${OLLAMA_URL:-http://localhost:11434}
|
||||
- CHROMADB_PATH=/app/data/chromadb
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
|
||||
- OIDC_ISSUER=${OIDC_ISSUER}
|
||||
- OIDC_CLIENT_ID=${OIDC_CLIENT_ID}
|
||||
- OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET}
|
||||
- OIDC_REDIRECT_URI=${OIDC_REDIRECT_URI}
|
||||
- OIDC_USE_DISCOVERY=${OIDC_USE_DISCOVERY:-true}
|
||||
- DATABASE_URL=postgres://raggr:raggr_dev_password@postgres:5432/raggr
|
||||
- FLASK_ENV=development
|
||||
- PYTHONUNBUFFERED=1
|
||||
- NODE_ENV=development
|
||||
- TAVILY_KEY=${TAVILIY_KEY}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- chromadb_data:/app/data/chromadb
|
||||
develop:
|
||||
watch:
|
||||
# Sync+restart on any file change under services/raggr
|
||||
- action: sync+restart
|
||||
path: ./services/raggr
|
||||
target: /app
|
||||
ignore:
|
||||
- __pycache__/
|
||||
- "*.pyc"
|
||||
- "*.pyo"
|
||||
- "*.pyd"
|
||||
- .git/
|
||||
- chromadb/
|
||||
- node_modules/
|
||||
- raggr-frontend/dist/
|
||||
# raggr service disabled - run locally for development
|
||||
# raggr:
|
||||
# build:
|
||||
# context: .
|
||||
# dockerfile: Dockerfile.dev
|
||||
# image: torrtle/simbarag:dev
|
||||
# ports:
|
||||
# - "8080:8080"
|
||||
# env_file:
|
||||
# - .env
|
||||
# environment:
|
||||
# - PAPERLESS_TOKEN=${PAPERLESS_TOKEN}
|
||||
# - BASE_URL=${BASE_URL}
|
||||
# - OLLAMA_URL=${OLLAMA_URL:-http://localhost:11434}
|
||||
# - CHROMADB_PATH=/app/data/chromadb
|
||||
# - OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
# - JWT_SECRET_KEY=${JWT_SECRET_KEY}
|
||||
# - OIDC_ISSUER=${OIDC_ISSUER}
|
||||
# - OIDC_CLIENT_ID=${OIDC_CLIENT_ID}
|
||||
# - OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET}
|
||||
# - OIDC_REDIRECT_URI=${OIDC_REDIRECT_URI}
|
||||
# - OIDC_USE_DISCOVERY=${OIDC_USE_DISCOVERY:-true}
|
||||
# - DATABASE_URL=postgres://raggr:raggr_dev_password@postgres:5432/raggr
|
||||
# - FLASK_ENV=development
|
||||
# - PYTHONUNBUFFERED=1
|
||||
# - NODE_ENV=development
|
||||
# - TAVILY_KEY=${TAVILIY_KEY}
|
||||
# depends_on:
|
||||
# postgres:
|
||||
# condition: service_healthy
|
||||
# volumes:
|
||||
# - chromadb_data:/app/data/chromadb
|
||||
# - ./migrations:/app/migrations # Bind mount for migrations (bidirectional)
|
||||
# develop:
|
||||
# watch:
|
||||
# # Sync+restart on any file change in root directory
|
||||
# - action: sync+restart
|
||||
# path: .
|
||||
# target: /app
|
||||
# ignore:
|
||||
# - __pycache__/
|
||||
# - "*.pyc"
|
||||
# - "*.pyo"
|
||||
# - "*.pyd"
|
||||
# - .git/
|
||||
# - chromadb/
|
||||
# - node_modules/
|
||||
# - raggr-frontend/dist/
|
||||
# - docs/
|
||||
# - .venv/
|
||||
|
||||
volumes:
|
||||
chromadb_data:
|
||||
|
||||
@@ -3,6 +3,8 @@ version: "3.8"
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
ports:
|
||||
- "5432:5432"
|
||||
environment:
|
||||
- POSTGRES_USER=${POSTGRES_USER:-raggr}
|
||||
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
|
||||
@@ -18,10 +20,11 @@ services:
|
||||
|
||||
raggr:
|
||||
build:
|
||||
context: ./services/raggr
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
image: torrtle/simbarag:latest
|
||||
network_mode: host
|
||||
ports:
|
||||
- "8080:8080"
|
||||
environment:
|
||||
- PAPERLESS_TOKEN=${PAPERLESS_TOKEN}
|
||||
- BASE_URL=${BASE_URL}
|
||||
@@ -35,6 +38,7 @@ services:
|
||||
- OIDC_REDIRECT_URI=${OIDC_REDIRECT_URI}
|
||||
- OIDC_USE_DISCOVERY=${OIDC_USE_DISCOVERY:-true}
|
||||
- DATABASE_URL=${DATABASE_URL:-postgres://raggr:changeme@postgres:5432/raggr}
|
||||
- TAVILY_KEY=${TAVILIY_KEY}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
|
||||
53
docs/TASKS.md
Normal file
53
docs/TASKS.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Tasks & Feature Requests
|
||||
|
||||
## Feature Requests
|
||||
|
||||
### YNAB Integration (Admin-Only)
|
||||
- **Description**: Integration with YNAB (You Need A Budget) API to enable financial data queries and insights
|
||||
- **Requirements**:
|
||||
- Admin-guarded endpoint (requires `lldap_admin` group)
|
||||
- YNAB API token configuration in environment variables
|
||||
- Sync budget data, transactions, and categories
|
||||
- Store YNAB data for RAG queries
|
||||
- **Endpoints**:
|
||||
- `POST /api/admin/ynab/sync` - Trigger YNAB data sync
|
||||
- `GET /api/admin/ynab/status` - Check sync status and last update
|
||||
- `GET /api/admin/ynab/budgets` - List available budgets
|
||||
- **Implementation Notes**:
|
||||
- Use YNAB API v1 (https://api.youneedabudget.com/v1)
|
||||
- Consider rate limiting (200 requests per hour)
|
||||
- Store transaction data in PostgreSQL with appropriate indexing
|
||||
- Index transaction descriptions and categories in ChromaDB for RAG queries
|
||||
|
||||
### Money Insights
|
||||
- **Description**: AI-powered financial insights and analysis based on YNAB data
|
||||
- **Features**:
|
||||
- Spending pattern analysis
|
||||
- Budget vs. actual comparisons
|
||||
- Category-based spending trends
|
||||
- Anomaly detection (unusual transactions)
|
||||
- Natural language queries like "How much did I spend on groceries last month?"
|
||||
- Month-over-month and year-over-year comparisons
|
||||
- **Implementation Notes**:
|
||||
- Leverage existing LangChain agent architecture
|
||||
- Add custom tools for financial calculations
|
||||
- Use LLM to generate insights and summaries
|
||||
- Create visualizations or data exports for frontend display
|
||||
|
||||
## Backlog
|
||||
|
||||
- [ ] YNAB API client module
|
||||
- [ ] YNAB data models (Budget, Transaction, Category, Account)
|
||||
- [ ] Database schema for financial data
|
||||
- [ ] YNAB sync background job/scheduler
|
||||
- [ ] Financial insights LangChain tools
|
||||
- [ ] Admin UI for YNAB configuration
|
||||
- [ ] Frontend components for money insights display
|
||||
|
||||
## Technical Debt
|
||||
|
||||
_To be added_
|
||||
|
||||
## Bugs
|
||||
|
||||
_To be added_
|
||||
@@ -13,21 +13,21 @@ The vector store location is controlled by the `CHROMADB_PATH` environment varia
|
||||
|
||||
### CLI (Command Line)
|
||||
|
||||
Use the `manage_vectorstore.py` script for vector store operations:
|
||||
Use the `scripts/manage_vectorstore.py` script for vector store operations:
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
python manage_vectorstore.py stats
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index documents from Paperless-NGX (incremental)
|
||||
python manage_vectorstore.py index
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Clear and reindex all documents
|
||||
python manage_vectorstore.py reindex
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
|
||||
# List documents
|
||||
python manage_vectorstore.py list 10
|
||||
python manage_vectorstore.py list 20 --show-content
|
||||
python scripts/manage_vectorstore.py list 10
|
||||
python scripts/manage_vectorstore.py list 20 --show-content
|
||||
```
|
||||
|
||||
### Docker
|
||||
@@ -36,10 +36,10 @@ Run commands inside the Docker container:
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
docker compose -f docker-compose.dev.yml exec -T raggr python manage_vectorstore.py stats
|
||||
docker compose exec raggr python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Reindex all documents
|
||||
docker compose -f docker-compose.dev.yml exec -T raggr python manage_vectorstore.py reindex
|
||||
docker compose exec raggr python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
@@ -65,7 +65,7 @@ The following authenticated endpoints are available:
|
||||
This indicates a corrupted index. Solution:
|
||||
|
||||
```bash
|
||||
python manage_vectorstore.py reindex
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
### Empty results
|
||||
@@ -73,20 +73,20 @@ python manage_vectorstore.py reindex
|
||||
Check if documents are indexed:
|
||||
|
||||
```bash
|
||||
python manage_vectorstore.py stats
|
||||
python scripts/manage_vectorstore.py stats
|
||||
```
|
||||
|
||||
If count is 0, run:
|
||||
|
||||
```bash
|
||||
python manage_vectorstore.py index
|
||||
python scripts/manage_vectorstore.py index
|
||||
```
|
||||
|
||||
### Different results in Docker vs local
|
||||
|
||||
Docker and local environments use separate ChromaDB instances. To sync:
|
||||
|
||||
1. Index inside Docker: `docker compose exec -T raggr python manage_vectorstore.py reindex`
|
||||
1. Index inside Docker: `docker compose exec raggr python scripts/manage_vectorstore.py reindex`
|
||||
2. Or mount the same volume for both environments
|
||||
|
||||
## Production Considerations
|
||||
274
docs/authentication.md
Normal file
274
docs/authentication.md
Normal file
@@ -0,0 +1,274 @@
|
||||
# Authentication Architecture
|
||||
|
||||
This document describes the authentication stack for SimbaRAG: LLDAP → Authelia → OAuth2/OIDC.
|
||||
|
||||
## Overview
|
||||
|
||||
```
|
||||
┌─────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────┐
|
||||
│ LLDAP │────▶│ Authelia │────▶│ OAuth2/OIDC │────▶│ SimbaRAG │
|
||||
│ (Users) │ │ (IdP) │ │ (Flow) │ │ (App) │
|
||||
└─────────┘ └──────────┘ └──────────────┘ └──────────┘
|
||||
```
|
||||
|
||||
| Component | Role |
|
||||
|-----------|------|
|
||||
| **LLDAP** | Lightweight LDAP server storing users and groups |
|
||||
| **Authelia** | Identity provider that authenticates against LLDAP and issues OIDC tokens |
|
||||
| **SimbaRAG** | Relying party that consumes OIDC tokens and manages sessions |
|
||||
|
||||
## OIDC Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `OIDC_ISSUER` | Authelia server URL | Required |
|
||||
| `OIDC_CLIENT_ID` | Client ID registered in Authelia | Required |
|
||||
| `OIDC_CLIENT_SECRET` | Client secret for token exchange | Required |
|
||||
| `OIDC_REDIRECT_URI` | Callback URL after authentication | Required |
|
||||
| `OIDC_USE_DISCOVERY` | Enable automatic discovery | `true` |
|
||||
| `JWT_SECRET_KEY` | Secret for signing backend JWTs | Required |
|
||||
|
||||
### Discovery
|
||||
|
||||
When `OIDC_USE_DISCOVERY=true`, the application fetches endpoints from:
|
||||
|
||||
```
|
||||
{OIDC_ISSUER}/.well-known/openid-configuration
|
||||
```
|
||||
|
||||
This provides:
|
||||
|
||||
- Authorization endpoint
|
||||
- Token endpoint
|
||||
- JWKS URI for signature verification
|
||||
- Supported scopes and claims
|
||||
|
||||
## Authentication Flow
|
||||
|
||||
### 1. Login Initiation
|
||||
|
||||
```
|
||||
GET /api/user/oidc/login
|
||||
```
|
||||
|
||||
1. Generate PKCE code verifier and challenge (S256)
|
||||
2. Generate CSRF state token
|
||||
3. Store state in session storage
|
||||
4. Return authorization URL for frontend redirect
|
||||
|
||||
### 2. Authorization
|
||||
|
||||
User is redirected to Authelia where they:
|
||||
|
||||
1. Enter LDAP credentials
|
||||
2. Complete MFA if configured
|
||||
3. Consent to requested scopes
|
||||
|
||||
### 3. Callback
|
||||
|
||||
```
|
||||
GET /api/user/oidc/callback?code=...&state=...
|
||||
```
|
||||
|
||||
1. Validate state matches stored value (CSRF protection)
|
||||
2. Exchange authorization code for tokens using PKCE verifier
|
||||
3. Verify ID token signature using JWKS
|
||||
4. Validate claims (issuer, audience, expiration)
|
||||
5. Create or update user in database
|
||||
6. Issue backend JWT tokens (access + refresh)
|
||||
|
||||
### 4. Token Refresh
|
||||
|
||||
```
|
||||
POST /api/user/refresh
|
||||
Authorization: Bearer <refresh_token>
|
||||
```
|
||||
|
||||
Issues a new access token without re-authentication.
|
||||
|
||||
## User Model
|
||||
|
||||
```python
|
||||
class User(Model):
|
||||
id = UUIDField(primary_key=True)
|
||||
username = CharField(max_length=255)
|
||||
password = BinaryField(null=True) # Nullable for OIDC-only users
|
||||
email = CharField(max_length=100, unique=True)
|
||||
|
||||
# OIDC fields
|
||||
oidc_subject = CharField(max_length=255, unique=True, null=True)
|
||||
auth_provider = CharField(max_length=50, default="local") # "local" or "oidc"
|
||||
ldap_groups = JSONField(default=[]) # LDAP groups from OIDC claims
|
||||
|
||||
created_at = DatetimeField(auto_now_add=True)
|
||||
updated_at = DatetimeField(auto_now=True)
|
||||
|
||||
def has_group(self, group: str) -> bool:
|
||||
"""Check if user belongs to a specific LDAP group."""
|
||||
return group in (self.ldap_groups or [])
|
||||
|
||||
def is_admin(self) -> bool:
|
||||
"""Check if user is an admin (member of lldap_admin group)."""
|
||||
return self.has_group("lldap_admin")
|
||||
```
|
||||
|
||||
### User Provisioning
|
||||
|
||||
The `OIDCUserService` handles automatic user creation:
|
||||
|
||||
1. Extract claims from ID token (`sub`, `email`, `preferred_username`)
|
||||
2. Check if user exists by `oidc_subject`
|
||||
3. If not, check by email for migration from local auth
|
||||
4. Create new user or update existing
|
||||
|
||||
## JWT Tokens
|
||||
|
||||
Backend issues its own JWTs after OIDC authentication:
|
||||
|
||||
| Token Type | Purpose | Typical Lifetime |
|
||||
|------------|---------|------------------|
|
||||
| Access Token | API authorization | 15 minutes |
|
||||
| Refresh Token | Obtain new access tokens | 7 days |
|
||||
|
||||
### Claims
|
||||
|
||||
```json
|
||||
{
|
||||
"identity": "<user-uuid>",
|
||||
"type": "access|refresh",
|
||||
"exp": 1234567890,
|
||||
"iat": 1234567890
|
||||
}
|
||||
```
|
||||
|
||||
## Protected Endpoints
|
||||
|
||||
All API endpoints use the `@jwt_refresh_token_required` decorator for basic authentication:
|
||||
|
||||
```python
|
||||
@blueprint.route("/example")
|
||||
@jwt_refresh_token_required
|
||||
async def protected_endpoint():
|
||||
user_id = get_jwt_identity()
|
||||
# ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Role-Based Access Control (RBAC)
|
||||
|
||||
RBAC is implemented using LDAP groups passed through Authelia as OIDC claims. Users in the `lldap_admin` group have admin privileges.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ LLDAP │
|
||||
│ Groups: lldap_admin, lldap_user │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Authelia │
|
||||
│ Scope: groups → Claim: groups = ["lldap_admin"] │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ SimbaRAG │
|
||||
│ 1. Extract groups from ID token │
|
||||
│ 2. Store in User.ldap_groups │
|
||||
│ 3. Check membership with @admin_required decorator │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Authelia Configuration
|
||||
|
||||
Ensure Authelia is configured to pass the `groups` claim:
|
||||
|
||||
```yaml
|
||||
identity_providers:
|
||||
oidc:
|
||||
clients:
|
||||
- client_id: simbarag
|
||||
scopes:
|
||||
- openid
|
||||
- profile
|
||||
- email
|
||||
- groups # Required for RBAC
|
||||
```
|
||||
|
||||
### Admin-Only Endpoints
|
||||
|
||||
The `@admin_required` decorator protects privileged endpoints:
|
||||
|
||||
```python
|
||||
from blueprints.users.decorators import admin_required
|
||||
|
||||
@blueprint.post("/admin-action")
|
||||
@admin_required
|
||||
async def admin_only_endpoint():
|
||||
# Only users in lldap_admin group can access
|
||||
...
|
||||
```
|
||||
|
||||
**Protected endpoints:**
|
||||
|
||||
| Endpoint | Access | Description |
|
||||
|----------|--------|-------------|
|
||||
| `POST /api/rag/index` | Admin | Trigger document indexing |
|
||||
| `POST /api/rag/reindex` | Admin | Clear and reindex all documents |
|
||||
| `GET /api/rag/stats` | All users | View vector store statistics |
|
||||
|
||||
### User Response
|
||||
|
||||
The OIDC callback returns group information:
|
||||
|
||||
```json
|
||||
{
|
||||
"access_token": "...",
|
||||
"refresh_token": "...",
|
||||
"user": {
|
||||
"id": "uuid",
|
||||
"username": "john",
|
||||
"email": "john@example.com",
|
||||
"groups": ["lldap_admin", "lldap_user"],
|
||||
"is_admin": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Current Gaps
|
||||
|
||||
| Issue | Risk | Mitigation |
|
||||
|-------|------|------------|
|
||||
| In-memory session storage | State lost on restart, not scalable | Use Redis for production |
|
||||
| No token revocation | Tokens valid until expiry | Implement blacklist or short expiry |
|
||||
| No audit logging | Cannot track auth events | Add event logging |
|
||||
| Single JWT secret | Compromise affects all tokens | Rotate secrets, use asymmetric keys |
|
||||
|
||||
### Recommendations
|
||||
|
||||
1. **Use Redis** for OIDC state storage in production
|
||||
2. **Implement logout** with token blacklisting
|
||||
3. **Add audit logging** for authentication events
|
||||
4. **Rotate JWT secrets** regularly
|
||||
5. **Use short-lived access tokens** (15 min) with refresh
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `services/raggr/oidc_config.py` | OIDC client configuration and discovery |
|
||||
| `services/raggr/blueprints/users/models.py` | User model definition with group helpers |
|
||||
| `services/raggr/blueprints/users/oidc_service.py` | User provisioning from OIDC claims |
|
||||
| `services/raggr/blueprints/users/__init__.py` | Auth endpoints and flow |
|
||||
| `services/raggr/blueprints/users/decorators.py` | Auth decorators (`@admin_required`) |
|
||||
188
docs/deployment.md
Normal file
188
docs/deployment.md
Normal file
@@ -0,0 +1,188 @@
|
||||
# Deployment & Migrations Guide
|
||||
|
||||
This document covers database migrations and deployment workflows for SimbaRAG.
|
||||
|
||||
## Migration Workflow
|
||||
|
||||
Migrations are managed by [Aerich](https://github.com/tortoise/aerich), the migration tool for Tortoise ORM.
|
||||
|
||||
### Key Principles
|
||||
|
||||
1. **Generate migrations in Docker** - Aerich needs database access to detect schema changes
|
||||
2. **Migrations auto-apply on startup** - Both `startup.sh` and `startup-dev.sh` run `aerich upgrade`
|
||||
3. **Commit migrations to git** - Migration files must be in the repo for production deploys
|
||||
|
||||
### Generating a New Migration
|
||||
|
||||
#### Development (Recommended)
|
||||
|
||||
With `docker-compose.dev.yml`, your local `services/raggr` directory is synced to the container. Migrations generated inside the container appear on your host automatically.
|
||||
|
||||
```bash
|
||||
# 1. Start the dev environment
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# 2. Generate migration (runs inside container, syncs to host)
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name describe_your_change
|
||||
|
||||
# 3. Verify migration was created
|
||||
ls services/raggr/migrations/models/
|
||||
|
||||
# 4. Commit the migration
|
||||
git add services/raggr/migrations/
|
||||
git commit -m "Add migration: describe_your_change"
|
||||
```
|
||||
|
||||
#### Production Container
|
||||
|
||||
For production, migration files are baked into the image. You must generate migrations in dev first.
|
||||
|
||||
```bash
|
||||
# If you need to generate a migration from production (not recommended):
|
||||
docker compose exec raggr aerich migrate --name describe_your_change
|
||||
|
||||
# Copy the file out of the container
|
||||
docker cp $(docker compose ps -q raggr):/app/migrations/models/ ./services/raggr/migrations/
|
||||
```
|
||||
|
||||
### Applying Migrations
|
||||
|
||||
Migrations apply automatically on container start via the startup scripts.
|
||||
|
||||
**Manual application (if needed):**
|
||||
|
||||
```bash
|
||||
# Dev
|
||||
docker compose -f docker-compose.dev.yml exec raggr aerich upgrade
|
||||
|
||||
# Production
|
||||
docker compose exec raggr aerich upgrade
|
||||
```
|
||||
|
||||
### Checking Migration Status
|
||||
|
||||
```bash
|
||||
# View applied migrations
|
||||
docker compose exec raggr aerich history
|
||||
|
||||
# View pending migrations
|
||||
docker compose exec raggr aerich heads
|
||||
```
|
||||
|
||||
### Rolling Back
|
||||
|
||||
```bash
|
||||
# Downgrade one migration
|
||||
docker compose exec raggr aerich downgrade
|
||||
|
||||
# Downgrade to specific version
|
||||
docker compose exec raggr aerich downgrade -v 1
|
||||
```
|
||||
|
||||
## Deployment Workflows
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Start with watch mode (auto-restarts on file changes)
|
||||
docker compose -f docker-compose.dev.yml up
|
||||
|
||||
# Or with docker compose watch (requires Docker Compose v2.22+)
|
||||
docker compose -f docker-compose.dev.yml watch
|
||||
```
|
||||
|
||||
The dev environment:
|
||||
- Syncs `services/raggr/` to `/app` in the container
|
||||
- Rebuilds frontend on changes
|
||||
- Auto-applies migrations on startup
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
# Build and deploy
|
||||
docker compose build raggr
|
||||
docker compose up -d
|
||||
|
||||
# View logs
|
||||
docker compose logs -f raggr
|
||||
|
||||
# Verify migrations applied
|
||||
docker compose exec raggr aerich history
|
||||
```
|
||||
|
||||
### Fresh Deploy (New Database)
|
||||
|
||||
On first deploy with an empty database, `startup-dev.sh` runs `aerich init-db` instead of `aerich upgrade`. This creates all tables from the current models.
|
||||
|
||||
For production (`startup.sh`), ensure the database exists and run:
|
||||
|
||||
```bash
|
||||
# If aerich table doesn't exist yet
|
||||
docker compose exec raggr aerich init-db
|
||||
|
||||
# Or if migrating from existing schema
|
||||
docker compose exec raggr aerich upgrade
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No migrations found" on startup
|
||||
|
||||
The `migrations/models/` directory is empty or not copied into the image.
|
||||
|
||||
**Fix:** Ensure migrations are committed and the Dockerfile copies them:
|
||||
```dockerfile
|
||||
COPY migrations ./migrations
|
||||
```
|
||||
|
||||
### Migration fails with "relation already exists"
|
||||
|
||||
The database has tables but aerich doesn't know about them (fresh aerich setup on existing DB).
|
||||
|
||||
**Fix:** Fake the initial migration:
|
||||
```bash
|
||||
# Mark initial migration as applied without running it
|
||||
docker compose exec raggr aerich upgrade --fake
|
||||
```
|
||||
|
||||
### Model changes not detected
|
||||
|
||||
Aerich compares models against the last migration's state. If state is out of sync:
|
||||
|
||||
```bash
|
||||
# Regenerate migration state (dangerous - review carefully)
|
||||
docker compose exec raggr aerich migrate --name fix_state
|
||||
```
|
||||
|
||||
### Database connection errors
|
||||
|
||||
Ensure PostgreSQL is healthy before running migrations:
|
||||
|
||||
```bash
|
||||
# Check postgres status
|
||||
docker compose ps postgres
|
||||
|
||||
# Wait for postgres then run migrations
|
||||
docker compose exec raggr bash -c "sleep 5 && aerich upgrade"
|
||||
```
|
||||
|
||||
## File Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `pyproject.toml` | Aerich config (`[tool.aerich]` section) |
|
||||
| `migrations/models/` | Migration files |
|
||||
| `startup.sh` | Production startup (runs `aerich upgrade`) |
|
||||
| `startup-dev.sh` | Dev startup (runs `aerich upgrade` or `init-db`) |
|
||||
| `app.py` | Contains `TORTOISE_CONFIG` |
|
||||
| `aerich_config.py` | Aerich initialization configuration |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Generate migration | `docker compose -f docker-compose.dev.yml exec raggr aerich migrate --name name` |
|
||||
| Apply migrations | `docker compose exec raggr aerich upgrade` |
|
||||
| View history | `docker compose exec raggr aerich history` |
|
||||
| Rollback | `docker compose exec raggr aerich downgrade` |
|
||||
| Fresh init | `docker compose exec raggr aerich init-db` |
|
||||
258
docs/development.md
Normal file
258
docs/development.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# Development Guide
|
||||
|
||||
This guide explains how to run SimbaRAG in development mode.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Option 1: Local Development (Recommended)
|
||||
|
||||
Run PostgreSQL in Docker and the application locally for faster iteration:
|
||||
|
||||
```bash
|
||||
# 1. Start PostgreSQL
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
|
||||
# 2. Set environment variables
|
||||
export DATABASE_URL="postgres://raggr:raggr_dev_password@localhost:5432/raggr"
|
||||
export CHROMADB_PATH="./chromadb"
|
||||
export $(grep -v '^#' .env | xargs) # Load other vars from .env
|
||||
|
||||
# 3. Install dependencies (first time)
|
||||
pip install -r requirements.txt
|
||||
cd raggr-frontend && yarn install && yarn build && cd ..
|
||||
|
||||
# 4. Run migrations
|
||||
aerich upgrade
|
||||
|
||||
# 5. Start the server
|
||||
python app.py
|
||||
```
|
||||
|
||||
The application will be available at `http://localhost:8080`.
|
||||
|
||||
### Option 2: Full Docker Development
|
||||
|
||||
Run everything in Docker with hot reload (slower, but matches production):
|
||||
|
||||
```bash
|
||||
# Uncomment the raggr service in docker-compose.dev.yml first!
|
||||
|
||||
# Start all services
|
||||
docker compose -f docker-compose.dev.yml up --build
|
||||
|
||||
# View logs
|
||||
docker compose -f docker-compose.dev.yml logs -f raggr
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
raggr/
|
||||
├── app.py # Quart application entry point
|
||||
├── main.py # RAG logic and LangChain agent
|
||||
├── llm.py # LLM client (Ollama + OpenAI fallback)
|
||||
├── aerich_config.py # Database migration configuration
|
||||
│
|
||||
├── blueprints/ # API route blueprints
|
||||
│ ├── users/ # Authentication (OIDC, JWT, RBAC)
|
||||
│ ├── conversation/ # Chat conversations and messages
|
||||
│ └── rag/ # Document indexing (admin only)
|
||||
│
|
||||
├── config/ # Configuration modules
|
||||
│ └── oidc_config.py # OIDC authentication settings
|
||||
│
|
||||
├── utils/ # Reusable utilities
|
||||
│ ├── chunker.py # Document chunking for embeddings
|
||||
│ ├── cleaner.py # PDF cleaning and summarization
|
||||
│ ├── image_process.py # Image description with LLM
|
||||
│ └── request.py # Paperless-NGX API client
|
||||
│
|
||||
├── scripts/ # Administrative scripts
|
||||
│ ├── add_user.py # Create users manually
|
||||
│ ├── user_message_stats.py # User message statistics
|
||||
│ ├── manage_vectorstore.py # Vector store management
|
||||
│ ├── inspect_vector_store.py # Inspect ChromaDB contents
|
||||
│ └── query.py # Query generation utilities
|
||||
│
|
||||
├── raggr-frontend/ # React frontend
|
||||
│ └── src/ # Frontend source code
|
||||
│
|
||||
├── migrations/ # Database migrations
|
||||
└── docs/ # Documentation
|
||||
```
|
||||
|
||||
## Making Changes
|
||||
|
||||
### Backend Changes
|
||||
|
||||
**Local development:**
|
||||
1. Edit Python files
|
||||
2. Save
|
||||
3. Restart `python app.py` (or use a tool like `watchdog` for auto-reload)
|
||||
|
||||
**Docker development:**
|
||||
1. Edit Python files
|
||||
2. Files are synced via Docker watch mode
|
||||
3. Container automatically restarts
|
||||
|
||||
### Frontend Changes
|
||||
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
|
||||
# Development mode with hot reload
|
||||
yarn dev
|
||||
|
||||
# Production build (for testing)
|
||||
yarn build
|
||||
```
|
||||
|
||||
The backend serves built files from `raggr-frontend/dist/`.
|
||||
|
||||
### Database Model Changes
|
||||
|
||||
When you modify Tortoise ORM models:
|
||||
|
||||
```bash
|
||||
# Generate migration
|
||||
aerich migrate --name "describe_your_change"
|
||||
|
||||
# Apply migration
|
||||
aerich upgrade
|
||||
|
||||
# View history
|
||||
aerich history
|
||||
```
|
||||
|
||||
See [deployment.md](deployment.md) for detailed migration workflows.
|
||||
|
||||
### Adding Dependencies
|
||||
|
||||
**Backend:**
|
||||
```bash
|
||||
# Add to requirements.txt or use uv
|
||||
pip install package-name
|
||||
pip freeze > requirements.txt
|
||||
```
|
||||
|
||||
**Frontend:**
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
yarn add package-name
|
||||
```
|
||||
|
||||
## Useful Commands
|
||||
|
||||
### Database
|
||||
|
||||
```bash
|
||||
# Connect to PostgreSQL
|
||||
docker compose -f docker-compose.dev.yml exec postgres psql -U raggr -d raggr
|
||||
|
||||
# Reset database
|
||||
docker compose -f docker-compose.dev.yml down -v
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
aerich init-db
|
||||
```
|
||||
|
||||
### Vector Store
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index new documents from Paperless
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Clear and reindex everything
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
See [vectorstore.md](vectorstore.md) for details.
|
||||
|
||||
### Scripts
|
||||
|
||||
```bash
|
||||
# Add a new user
|
||||
python scripts/add_user.py
|
||||
|
||||
# View message statistics
|
||||
python scripts/user_message_stats.py
|
||||
|
||||
# Inspect vector store contents
|
||||
python scripts/inspect_vector_store.py
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Copy `.env.example` to `.env` and configure:
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `DATABASE_URL` | PostgreSQL connection | `postgres://user:pass@localhost:5432/db` |
|
||||
| `CHROMADB_PATH` | ChromaDB storage path | `./chromadb` |
|
||||
| `OLLAMA_URL` | Ollama server URL | `http://localhost:11434` |
|
||||
| `OPENAI_API_KEY` | OpenAI API key (fallback LLM) | `sk-...` |
|
||||
| `PAPERLESS_TOKEN` | Paperless-NGX API token | `...` |
|
||||
| `BASE_URL` | Paperless-NGX URL | `https://paperless.example.com` |
|
||||
| `OIDC_ISSUER` | OIDC provider URL | `https://auth.example.com` |
|
||||
| `OIDC_CLIENT_ID` | OIDC client ID | `simbarag` |
|
||||
| `OIDC_CLIENT_SECRET` | OIDC client secret | `...` |
|
||||
| `JWT_SECRET_KEY` | JWT signing key | `random-secret` |
|
||||
| `TAVILY_KEY` | Tavily web search API key | `tvly-...` |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
```bash
|
||||
# Find and kill process on port 8080
|
||||
lsof -ti:8080 | xargs kill -9
|
||||
|
||||
# Or change the port in app.py
|
||||
```
|
||||
|
||||
### Database Connection Errors
|
||||
|
||||
```bash
|
||||
# Check if PostgreSQL is running
|
||||
docker compose -f docker-compose.dev.yml ps postgres
|
||||
|
||||
# View PostgreSQL logs
|
||||
docker compose -f docker-compose.dev.yml logs postgres
|
||||
```
|
||||
|
||||
### Frontend Not Building
|
||||
|
||||
```bash
|
||||
cd raggr-frontend
|
||||
rm -rf node_modules dist
|
||||
yarn install
|
||||
yarn build
|
||||
```
|
||||
|
||||
### ChromaDB Errors
|
||||
|
||||
```bash
|
||||
# Clear and recreate ChromaDB
|
||||
rm -rf chromadb/
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
### Import Errors After Reorganization
|
||||
|
||||
Ensure you're in the project root directory when running scripts, or use:
|
||||
|
||||
```bash
|
||||
# Add project root to Python path
|
||||
export PYTHONPATH="${PYTHONPATH}:$(pwd)"
|
||||
python scripts/your_script.py
|
||||
```
|
||||
|
||||
## Hot Tips
|
||||
|
||||
- Use `python -m pdb app.py` for debugging
|
||||
- Enable Quart debug mode in `app.py`: `app.run(debug=True)`
|
||||
- Check API logs: They appear in the terminal running `python app.py`
|
||||
- Frontend logs: Open browser DevTools console
|
||||
- Use `docker compose -f docker-compose.dev.yml down -v` for a clean slate
|
||||
203
docs/index.md
Normal file
203
docs/index.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# SimbaRAG Documentation
|
||||
|
||||
Welcome to the SimbaRAG documentation! This guide will help you understand, develop, and deploy the SimbaRAG conversational AI system.
|
||||
|
||||
## Getting Started
|
||||
|
||||
New to SimbaRAG? Start here:
|
||||
|
||||
1. Read the main [README](../README.md) for project overview and architecture
|
||||
2. Follow the [Development Guide](development.md) to set up your environment
|
||||
3. Learn about [Authentication](authentication.md) setup with OIDC and LDAP
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
### Core Guides
|
||||
|
||||
- **[Development Guide](development.md)** - Local development setup, project structure, and workflows
|
||||
- **[Deployment Guide](deployment.md)** - Database migrations, deployment workflows, and troubleshooting
|
||||
- **[Vector Store Guide](VECTORSTORE.md)** - Managing ChromaDB, indexing documents, and RAG operations
|
||||
- **[Migrations Guide](MIGRATIONS.md)** - Database migration reference
|
||||
- **[Authentication Guide](authentication.md)** - OIDC, Authelia, LLDAP configuration and user management
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| Task | Documentation |
|
||||
|------|---------------|
|
||||
| Set up local dev environment | [Development Guide → Quick Start](development.md#quick-start) |
|
||||
| Run database migrations | [Deployment Guide → Migration Workflow](deployment.md#migration-workflow) |
|
||||
| Index documents | [Vector Store Guide → Management Commands](VECTORSTORE.md#management-commands) |
|
||||
| Configure authentication | [Authentication Guide](authentication.md) |
|
||||
| Run administrative scripts | [Development Guide → Scripts](development.md#scripts) |
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Start local development
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
export DATABASE_URL="postgres://raggr:raggr_dev_password@localhost:5432/raggr"
|
||||
export CHROMADB_PATH="./chromadb"
|
||||
python app.py
|
||||
```
|
||||
|
||||
### Database Migrations
|
||||
|
||||
```bash
|
||||
# Generate migration
|
||||
aerich migrate --name "your_change"
|
||||
|
||||
# Apply migrations
|
||||
aerich upgrade
|
||||
|
||||
# View history
|
||||
aerich history
|
||||
```
|
||||
|
||||
### Vector Store Management
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
python scripts/manage_vectorstore.py stats
|
||||
|
||||
# Index new documents
|
||||
python scripts/manage_vectorstore.py index
|
||||
|
||||
# Reindex everything
|
||||
python scripts/manage_vectorstore.py reindex
|
||||
```
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
SimbaRAG is built with:
|
||||
|
||||
- **Backend**: Quart (async Python), LangChain, Tortoise ORM
|
||||
- **Frontend**: React 19, Rsbuild, Tailwind CSS
|
||||
- **Database**: PostgreSQL (users, conversations)
|
||||
- **Vector Store**: ChromaDB (document embeddings)
|
||||
- **LLM**: Ollama (primary), OpenAI (fallback)
|
||||
- **Auth**: Authelia (OIDC), LLDAP (user directory)
|
||||
|
||||
See the [README](../README.md#system-architecture) for detailed architecture diagram.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
simbarag/
|
||||
├── app.py # Quart app entry point
|
||||
├── main.py # RAG & LangChain agent
|
||||
├── llm.py # LLM client
|
||||
├── blueprints/ # API routes
|
||||
├── config/ # Configuration
|
||||
├── utils/ # Utilities
|
||||
├── scripts/ # Admin scripts
|
||||
├── raggr-frontend/ # React UI
|
||||
├── migrations/ # Database migrations
|
||||
├── docs/ # This documentation
|
||||
├── docker-compose.yml # Production Docker setup
|
||||
└── docker-compose.dev.yml # Development Docker setup
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
### RAG (Retrieval-Augmented Generation)
|
||||
|
||||
SimbaRAG uses RAG to answer questions about Simba:
|
||||
|
||||
1. Documents are fetched from Paperless-NGX
|
||||
2. Documents are chunked and embedded using OpenAI
|
||||
3. Embeddings are stored in ChromaDB
|
||||
4. User queries are embedded and matched against the store
|
||||
5. Relevant chunks are passed to the LLM for context
|
||||
6. LLM generates an answer using retrieved context
|
||||
|
||||
### LangChain Agent
|
||||
|
||||
The conversational agent has two tools:
|
||||
|
||||
- **simba_search**: Queries the vector store for Simba's documents
|
||||
- **web_search**: Searches the web via Tavily API
|
||||
|
||||
The agent automatically selects tools based on the query.
|
||||
|
||||
### Authentication Flow
|
||||
|
||||
1. User initiates OIDC login via Authelia
|
||||
2. Authelia authenticates against LLDAP
|
||||
3. Backend receives OIDC tokens and issues JWT
|
||||
4. Frontend stores JWT in localStorage
|
||||
5. Subsequent requests use JWT for authorization
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Key environment variables (see `.env.example` for complete list):
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| `DATABASE_URL` | PostgreSQL connection |
|
||||
| `CHROMADB_PATH` | Vector store location |
|
||||
| `OLLAMA_URL` | Local LLM server |
|
||||
| `OPENAI_API_KEY` | OpenAI for embeddings/fallback |
|
||||
| `PAPERLESS_TOKEN` | Document source API |
|
||||
| `OIDC_*` | Authentication configuration |
|
||||
| `TAVILY_KEY` | Web search API |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Authentication
|
||||
- `GET /api/user/oidc/login` - Start OIDC flow
|
||||
- `GET /api/user/oidc/callback` - OIDC callback
|
||||
- `POST /api/user/refresh` - Refresh JWT
|
||||
|
||||
### Conversations
|
||||
- `POST /api/conversation/` - Create conversation
|
||||
- `GET /api/conversation/` - List conversations
|
||||
- `POST /api/conversation/query` - Chat message
|
||||
|
||||
### RAG (Admin Only)
|
||||
- `GET /api/rag/stats` - Vector store stats
|
||||
- `POST /api/rag/index` - Index documents
|
||||
- `POST /api/rag/reindex` - Reindex all
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| Port already in use | Check if services are running: `lsof -ti:8080` |
|
||||
| Database connection error | Ensure PostgreSQL is running: `docker compose ps` |
|
||||
| ChromaDB errors | Clear and reindex: `python scripts/manage_vectorstore.py reindex` |
|
||||
| Import errors | Check you're in `services/raggr/` directory |
|
||||
| Frontend not building | `cd raggr-frontend && yarn install && yarn build` |
|
||||
|
||||
See individual guides for detailed troubleshooting.
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Read the [Development Guide](development.md)
|
||||
2. Set up your local environment
|
||||
3. Make changes and test locally
|
||||
4. Generate migrations if needed
|
||||
5. Submit a pull request
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [LangChain Documentation](https://python.langchain.com/)
|
||||
- [ChromaDB Documentation](https://docs.trychroma.com/)
|
||||
- [Quart Documentation](https://quart.palletsprojects.com/)
|
||||
- [Tortoise ORM Documentation](https://tortoise.github.io/)
|
||||
- [Authelia Documentation](https://www.authelia.com/)
|
||||
|
||||
## Need Help?
|
||||
|
||||
- Check the relevant guide in this documentation
|
||||
- Review troubleshooting sections
|
||||
- Check application logs: `docker compose logs -f`
|
||||
- Inspect database: `docker compose exec postgres psql -U raggr`
|
||||
|
||||
---
|
||||
|
||||
**Documentation Version**: 1.0
|
||||
**Last Updated**: January 2026
|
||||
0
docs/ynab_integration/specification.md
Normal file
0
docs/ynab_integration/specification.md
Normal file
81
index.html
Normal file
81
index.html
Normal file
@@ -0,0 +1,81 @@
|
||||
<!doctype html>
|
||||
|
||||
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||||
<meta name="author" content="Paperless-ngx project and contributors">
|
||||
<meta name="robots" content="noindex,nofollow">
|
||||
|
||||
<title>
|
||||
|
||||
Paperless-ngx sign in
|
||||
|
||||
</title>
|
||||
<link href="/static/bootstrap.min.css" rel="stylesheet">
|
||||
<link href="/static/base.css" rel="stylesheet">
|
||||
</head>
|
||||
|
||||
<body class="text-center">
|
||||
<div class="position-absolute top-50 start-50 translate-middle">
|
||||
<form class="form-accounts" id="form-account" method="post">
|
||||
<input type="hidden" name="csrfmiddlewaretoken" value="KLQ3mMraTFHfK9sMmc6DJcNIS6YixeHnSJiT3A12LYB49HeEXOpx5RnY9V6uPSrD">
|
||||
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 2897.4 896.6" width='300' class='logo mb-4'>
|
||||
<path class="leaf" d="M140,713.7c-3.4-16.4-10.3-49.1-11.2-49.1c-145.7-87.1-128.4-238-80.2-324.2C59,449,251.2,524,139.1,656.8 c-0.9,1.7,5.2,22.4,10.3,41.4c22.4-37.9,56-83.6,54.3-87.9C65.9,273.9,496.9,248.1,586.6,39.4c40.5,201.8-20.7,513.9-367.2,593.2 c-1.7,0.9-62.9,108.6-65.5,109.5c0-1.7-25.9-0.9-22.4-9.5C133.1,727.4,136.6,720.6,140,713.7L140,713.7z M135.7,632.6 c44-50.9-7.8-137.9-38.8-166.4C149.5,556.7,146,609.3,135.7,632.6L135.7,632.6z" transform="translate(0)" style="fill:#17541f"/>
|
||||
<g class="text" style="fill:#000">
|
||||
<path d="M1022.3,428.7c-17.8-19.9-42.7-29.8-74.7-29.8c-22.3,0-42.4,5.7-60.5,17.3c-18.1,11.6-32.3,27.5-42.5,47.8 s-15.3,42.9-15.3,67.8c0,24.9,5.1,47.5,15.3,67.8c10.3,20.3,24.4,36.2,42.5,47.8c18.1,11.5,38.3,17.3,60.5,17.3 c32,0,56.9-9.9,74.7-29.8v20.4v0.2h84.5V408.3h-84.5V428.7z M1010.5,575c-10.2,11.7-23.6,17.6-40.2,17.6s-29.9-5.9-40-17.6 s-15.1-26.1-15.1-43.3c0-17.1,5-31.6,15.1-43.3s23.4-17.6,40-17.6c16.6,0,30,5.9,40.2,17.6s15.3,26.1,15.3,43.3 S1020.7,563.3,1010.5,575z" transform="translate(0)"/>
|
||||
<path d="M1381,416.1c-18.1-11.5-38.3-17.3-60.5-17.4c-32,0-56.9,9.9-74.7,29.8v-20.4h-84.5v390.7h84.5v-164 c17.8,19.9,42.7,29.8,74.7,29.8c22.3,0,42.4-5.7,60.5-17.3s32.3-27.5,42.5-47.8c10.2-20.3,15.3-42.9,15.3-67.8s-5.1-47.5-15.3-67.8 C1413.2,443.6,1399.1,427.7,1381,416.1z M1337.9,575c-10.1,11.7-23.4,17.6-40,17.6s-29.9-5.9-40-17.6s-15.1-26.1-15.1-43.3 c0-17.1,5-31.6,15.1-43.3s23.4-17.6,40-17.6s29.9,5.9,40,17.6s15.1,26.1,15.1,43.3S1347.9,563.3,1337.9,575z" transform="translate(0)"/>
|
||||
<path d="M1672.2,416.8c-20.5-12-43-18-67.6-18c-24.9,0-47.6,5.9-68,17.6c-20.4,11.7-36.5,27.7-48.2,48s-17.6,42.7-17.6,67.3 c0.3,25.2,6.2,47.8,17.8,68c11.5,20.2,28,36,49.3,47.6c21.3,11.5,45.9,17.3,73.8,17.3c48.6,0,86.8-14.7,114.7-44l-52.5-48.9 c-8.6,8.3-17.6,14.6-26.7,19c-9.3,4.3-21.1,6.4-35.3,6.4c-11.6,0-22.5-3.6-32.7-10.9c-10.3-7.3-17.1-16.5-20.7-27.8h180l0.4-11.6 c0-29.6-6-55.7-18-78.2S1692.6,428.8,1672.2,416.8z M1558.3,503.2c2.1-12.1,7.5-21.8,16.2-29.1s18.7-10.9,30-10.9 s21.2,3.6,29.8,10.9c8.6,7.2,13.9,16.9,16,29.1H1558.3z" transform="translate(0)"/>
|
||||
<path d="M1895.3,411.7c-11,5.6-20.3,13.7-28,24.4h-0.1v-28h-84.5v247.3h84.5V536.3c0-22.6,4.7-38.1,14.2-46.5 c9.5-8.5,22.7-12.7,39.6-12.7c6.2,0,13.5,1,21.8,3.1l10.7-72c-5.9-3.3-14.5-4.9-25.8-4.9C1917.1,403.3,1906.3,406.1,1895.3,411.7z" transform="translate(0)"/>
|
||||
<rect x="1985" y="277.4" width="84.5" height="377.8" transform="translate(0)"/>
|
||||
<path d="M2313.2,416.8c-20.5-12-43-18-67.6-18c-24.9,0-47.6,5.9-68,17.6s-36.5,27.7-48.2,48c-11.7,20.3-17.6,42.7-17.6,67.3 c0.3,25.2,6.2,47.8,17.8,68c11.5,20.2,28,36,49.3,47.6c21.3,11.5,45.9,17.3,73.8,17.3c48.6,0,86.8-14.7,114.7-44l-52.5-48.9 c-8.6,8.3-17.6,14.6-26.7,19c-9.3,4.3-21.1,6.4-35.3,6.4c-11.6,0-22.5-3.6-32.7-10.9c-10.3-7.3-17.1-16.5-20.7-27.8h180l0.4-11.6 c0-29.6-6-55.7-18-78.2S2333.6,428.8,2313.2,416.8z M2199.3,503.2c2.1-12.1,7.5-21.8,16.2-29.1s18.7-10.9,30-10.9 s21.2,3.6,29.8,10.9c8.6,7.2,13.9,16.9,16,29.1H2199.3z" transform="translate(0)"/>
|
||||
<path d="M2583.6,507.7c-13.8-4.4-30.6-8.1-50.5-11.1c-15.1-2.7-26.1-5.2-32.9-7.6c-6.8-2.4-10.2-6.1-10.2-11.1s2.3-8.7,6.7-10.9 c4.4-2.2,11.5-3.3,21.3-3.3c11.6,0,24.3,2.4,38.1,7.2c13.9,4.8,26.2,11,36.9,18.4l32.4-58.2c-11.3-7.4-26.2-14.7-44.9-21.8 c-18.7-7.1-39.6-10.7-62.7-10.7c-33.7,0-60.2,7.6-79.3,22.7c-19.1,15.1-28.7,36.1-28.7,63.1c0,19,4.8,33.9,14.4,44.7 c9.6,10.8,21,18.5,34,22.9c13.1,4.5,28.9,8.3,47.6,11.6c14.6,2.7,25.1,5.3,31.6,7.8s9.8,6.5,9.8,11.8c0,10.4-9.7,15.6-29.3,15.6 c-13.7,0-28.5-2.3-44.7-6.9c-16.1-4.6-29.2-11.3-39.3-20.2l-33.3,60c9.2,7.4,24.6,14.7,46.2,22c21.7,7.3,45.2,10.9,70.7,10.9 c34.7,0,62.9-7.4,84.5-22.4c21.7-15,32.5-37.3,32.5-66.9c0-19.3-5-34.2-15.1-44.9S2597.4,512.1,2583.6,507.7z" transform="translate(0)"/>
|
||||
<path d="M2883.4,575.3c0-19.3-5-34.2-15.1-44.9s-22-18.3-35.8-22.7c-13.8-4.4-30.6-8.1-50.5-11.1c-15.1-2.7-26.1-5.2-32.9-7.6 c-6.8-2.4-10.2-6.1-10.2-11.1s2.3-8.7,6.7-10.9c4.4-2.2,11.5-3.3,21.3-3.3c11.6,0,24.3,2.4,38.1,7.2c13.9,4.8,26.2,11,36.9,18.4 l32.4-58.2c-11.3-7.4-26.2-14.7-44.9-21.8c-18.7-7.1-39.6-10.7-62.7-10.7c-33.7,0-60.2,7.6-79.3,22.7 c-19.1,15.1-28.7,36.1-28.7,63.1c0,19,4.8,33.9,14.4,44.7c9.6,10.8,21,18.5,34,22.9c13.1,4.5,28.9,8.3,47.6,11.6 c14.6,2.7,25.1,5.3,31.6,7.8s9.8,6.5,9.8,11.8c0,10.4-9.7,15.6-29.3,15.6c-13.7,0-28.5-2.3-44.7-6.9c-16.1-4.6-29.2-11.3-39.3-20.2 l-33.3,60c9.2,7.4,24.6,14.7,46.2,22c21.7,7.3,45.2,10.9,70.7,10.9c34.7,0,62.9-7.4,84.5-22.4 C2872.6,627.2,2883.4,604.9,2883.4,575.3z" transform="translate(0)"/>
|
||||
<rect x="2460.7" y="738.7" width="59.6" height="17.2" transform="translate(0)"/>
|
||||
<path d="M2596.5,706.4c-5.7,0-11,1-15.8,3s-9,5-12.5,8.9v-9.4h-19.4v93.6h19.4v-52c0-8.6,2.1-15.3,6.3-20c4.2-4.7,9.5-7.1,15.9-7.1 c7.8,0,13.4,2.3,16.8,6.7c3.4,4.5,5.1,11.3,5.1,20.5v52h19.4v-56.8c0-12.8-3.2-22.6-9.5-29.3 C2615.8,709.8,2607.3,706.4,2596.5,706.4z" transform="translate(0)"/>
|
||||
<path d="M2733.8,717.7c-3.6-3.4-7.9-6.1-13.1-8.2s-10.6-3.1-16.2-3.1c-8.7,0-16.5,2.1-23.5,6.3s-12.5,10-16.5,17.3 c-4,7.3-6,15.4-6,24.4c0,8.9,2,17.1,6,24.3c4,7.3,9.5,13,16.5,17.2s14.9,6.3,23.5,6.3c5.6,0,11-1,16.2-3.1 c5.1-2.1,9.5-4.8,13.1-8.2v24.4c0,8.5-2.5,14.8-7.6,18.7c-5,3.9-11,5.9-18,5.9c-6.7,0-12.4-1.6-17.3-4.7c-4.8-3.1-7.6-7.7-8.3-13.8 h-19.4c0.6,7.7,2.9,14.2,7.1,19.5s9.6,9.3,16.2,12c6.6,2.7,13.8,4,21.7,4c12.8,0,23.5-3.4,32-10.1c8.6-6.7,12.8-17.1,12.8-31.1 V708.9h-19.2V717.7z M2732.2,770.1c-2.5,4.7-6,8.3-10.4,11.2c-4.4,2.7-9.4,4-14.9,4c-5.7,0-10.8-1.4-15.2-4.3s-7.8-6.7-10.2-11.4 c-2.3-4.8-3.5-9.8-3.5-15.2c0-5.5,1.1-10.6,3.5-15.3s5.8-8.5,10.2-11.3s9.5-4.2,15.2-4.2c5.5,0,10.5,1.4,14.9,4s7.9,6.3,10.4,11 s3.8,10,3.8,15.8S2734.7,765.4,2732.2,770.1z" transform="translate(0)"/>
|
||||
<polygon points="2867.9,708.9 2846.5,708.9 2820.9,741.9 2795.5,708.9 2773.1,708.9 2809.1,755 2771.5,802.5 2792.9,802.5 2820.1,767.9 2847.2,802.6 2869.6,802.6 2832,754.4 " transform="translate(0)"/>
|
||||
<path d="M757.6,293.7c-20-10.8-42.6-16.2-67.8-16.2H600c-8.5,39.2-21.1,76.4-37.6,111.3c-9.9,20.8-21.1,40.6-33.6,59.4v207.2h88.9 V521.5h72c25.2,0,47.8-5.4,67.8-16.2s35.7-25.6,47.1-44.2c11.4-18.7,17.1-39.1,17.1-61.3c0.1-22.7-5.6-43.3-17-61.9 C793.3,319.2,777.6,304.5,757.6,293.7z M716.6,434.3c-9.3,8.9-21.6,13.3-36.7,13.3l-62.2,0.4v-92.5l62.2-0.4 c15.1,0,27.3,4.4,36.7,13.3c9.4,8.9,14,19.9,14,32.9C730.6,414.5,726,425.4,716.6,434.3z" transform="translate(0)"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
||||
|
||||
|
||||
|
||||
<p>
|
||||
Please sign in.
|
||||
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="form-floating form-stacked-top">
|
||||
<input type="text" name="login" id="inputUsername" placeholder="Username" class="form-control" autocorrect="off" autocapitalize="none" required autofocus>
|
||||
<label for="inputUsername">Username</label>
|
||||
</div>
|
||||
<div class="form-floating form-stacked-bottom">
|
||||
<input type="password" name="password" id="inputPassword" placeholder="Password" class="form-control" required>
|
||||
<label for="inputPassword">Password</label>
|
||||
</div>
|
||||
<div class="d-grid mt-3">
|
||||
<button class="btn btn-lg btn-primary" type="submit">Sign in</button>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
</form>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
46
llm.py
Normal file
46
llm.py
Normal file
@@ -0,0 +1,46 @@
|
||||
import os
|
||||
import logging
|
||||
|
||||
from openai import OpenAI
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
|
||||
class LLMClient:
|
||||
def __init__(self):
|
||||
llama_url = os.getenv("LLAMA_SERVER_URL")
|
||||
if llama_url:
|
||||
self.client = OpenAI(base_url=llama_url, api_key="not-needed")
|
||||
self.model = os.getenv("LLAMA_MODEL_NAME", "llama-3.1-8b-instruct")
|
||||
self.PROVIDER = "llama_server"
|
||||
logging.info("Using llama_server as LLM backend")
|
||||
else:
|
||||
self.client = OpenAI()
|
||||
self.model = "gpt-4o-mini"
|
||||
self.PROVIDER = "openai"
|
||||
logging.info("Using OpenAI as LLM backend")
|
||||
|
||||
def chat(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: str,
|
||||
):
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt,
|
||||
},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
client = LLMClient()
|
||||
print(client.chat(prompt="Hello!", system_prompt="You are a helpful assistant."))
|
||||
@@ -5,23 +5,17 @@ import os
|
||||
import sqlite3
|
||||
import time
|
||||
|
||||
import ollama
|
||||
from dotenv import load_dotenv
|
||||
|
||||
import chromadb
|
||||
from chunker import Chunker
|
||||
from cleaner import pdf_to_image, summarize_pdf_image
|
||||
from utils.chunker import Chunker
|
||||
from utils.cleaner import pdf_to_image, summarize_pdf_image
|
||||
from llm import LLMClient
|
||||
from query import QueryGenerator
|
||||
from request import PaperlessNGXService
|
||||
from scripts.query import QueryGenerator
|
||||
from utils.request import PaperlessNGXService
|
||||
|
||||
_dotenv_loaded = load_dotenv()
|
||||
|
||||
# Configure ollama client with URL from environment or default to localhost
|
||||
ollama_client = ollama.Client(
|
||||
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=10.0
|
||||
)
|
||||
|
||||
client = chromadb.PersistentClient(path=os.getenv("CHROMADB_PATH", ""))
|
||||
simba_docs = client.get_or_create_collection(name="simba_docs2")
|
||||
feline_vet_lookup = client.get_or_create_collection(name="feline_vet_lookup")
|
||||
72
migrations/models/1_20260131214411_None.py
Normal file
72
migrations/models/1_20260131214411_None.py
Normal file
@@ -0,0 +1,72 @@
|
||||
from tortoise import BaseDBAsyncClient
|
||||
|
||||
RUN_IN_TRANSACTION = True
|
||||
|
||||
|
||||
async def upgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
CREATE TABLE IF NOT EXISTS "users" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"username" VARCHAR(255) NOT NULL,
|
||||
"password" BYTEA,
|
||||
"email" VARCHAR(100) NOT NULL UNIQUE,
|
||||
"oidc_subject" VARCHAR(255) UNIQUE,
|
||||
"auth_provider" VARCHAR(50) NOT NULL DEFAULT 'local',
|
||||
"ldap_groups" JSONB NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS "idx_users_oidc_su_5aec5a" ON "users" ("oidc_subject");
|
||||
CREATE TABLE IF NOT EXISTS "conversations" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"name" VARCHAR(255) NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"user_id" UUID REFERENCES "users" ("id") ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS "conversation_messages" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"text" TEXT NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"speaker" VARCHAR(10) NOT NULL,
|
||||
"conversation_id" UUID NOT NULL REFERENCES "conversations" ("id") ON DELETE CASCADE
|
||||
);
|
||||
COMMENT ON COLUMN "conversation_messages"."speaker" IS 'USER: user\nSIMBA: simba';
|
||||
CREATE TABLE IF NOT EXISTS "aerich" (
|
||||
"id" SERIAL NOT NULL PRIMARY KEY,
|
||||
"version" VARCHAR(255) NOT NULL,
|
||||
"app" VARCHAR(100) NOT NULL,
|
||||
"content" JSONB NOT NULL
|
||||
);"""
|
||||
|
||||
|
||||
async def downgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
"""
|
||||
|
||||
|
||||
MODELS_STATE = (
|
||||
"eJztmm1v4jgQx78Kyquu1KtatnRX1emkQOkttwuceNinXhWZ2ICviZ1NnG1R1e9+tkmIkz"
|
||||
"gUKFDY401bxh5s/zzO/Mfpo+FSiJzgpEbJT+QHgGFKjMvSo0GAi/gf2vbjkgE8L2kVBgYG"
|
||||
"jnSwlZ6yBQwC5gOb8cYhcALETRAFto+9aDASOo4wUpt3xGSUmEKCf4TIYnSE2Bj5vOHmlp"
|
||||
"sxgegBBfFH784aYuTA1LwxFGNLu8UmnrT1+42ra9lTDDewbOqELkl6exM2pmTWPQwxPBE+"
|
||||
"om2ECPIBQ1BZhphltOzYNJ0xNzA/RLOpwsQA0RCEjoBh/D4MiS0YlORI4sf5H8YSeDhqgR"
|
||||
"YTJlg8Pk1XlaxZWg0xVO2D2Tl6e/FGrpIGbOTLRknEeJKOgIGpq+SagJS/cyhrY+DrUcb9"
|
||||
"MzD5RFfBGBsSjkkMxSBjQKtRM1zwYDmIjNiYfyxXKnMwfjY7kiTvJVFSHtfTqG9FTeVpm0"
|
||||
"CaILR9JJZsAZYHecVbGHaRHmbaM4MURq4n8R87CpivAbaJM4kOwRy+vUaz3u2Zzb/FStwg"
|
||||
"+OFIRGavLlrK0jrJWI8uMlsx+5LSl0bvQ0l8LH1vt+rZ2J/16303xJxAyKhF6L0FoHJeY2"
|
||||
"sMJrWxoQdX3Ni052FjX3Vjo8kr+xog31ougyguL0gj0dy2uImrJw2Reod32pwhYOThXVMf"
|
||||
"4RH5iCYSYYPPAxBblywi0dGPvmZXoSXWZBY+uJ+pETUo+Or4mhCbZk+zWzOv6oZkOAD23T"
|
||||
"3woVUA00VBAEYoyAOtRp7XHzvImUkzPUtVwDWn37ibT5UitpIVLVOFUYpevsktu1kLIHzd"
|
||||
"MBpbjDSHzjMqWIG4mBi21I08iOK9FsUMPWhSfo9b9Sjj/vsiiuel8vrXXiqLx9L3qGl+fZ"
|
||||
"PK5J/arT/j7opUrn1qVw8K+VcUUnmFHHgI3OnEgCgg6yR0c1IgtbuK+ysfHaPfrXcuSyKj"
|
||||
"/0O6jWbVvCwF2B0AY7EtTlWZZ6cLFJlnp4U1pmjKHCA10Sz3mNe4rvOZv6cS1s5ceL1Qym"
|
||||
"bvz3aW4rOaVhMuy2rbTSo5WTNopFtcSxRrNXG0D9ps/7WZ2MdlLy1Vn33RaFu4uPRAENxT"
|
||||
"XxOZVUyAP9HDVL0yMAcTNq1/drWk18GrCr2qyi2OrNpomZ1veskb91fjtvqtVzczdJELsL"
|
||||
"NMlM4c1hOiz5/4dQbo2eliomee6snJHoqhbQXh4F9kayqHYpJZv5WAZoN0uzw3cuC5lh9b"
|
||||
"nk9/Ylgk2vVAc47be4oaDrWB84I0lOZaWSRMK8VRWskFqQOBZ418GnqaO7y/uu2WHmnGLQ"
|
||||
"O0T/gqbyC22XHJwQG73Rjem9vNpHix8vkXCdk7g8wzVXzB4SLhf3KRcHjV9kts7OwmP1cQ"
|
||||
"PvcaJPd/Jet5F7LLYnS770BM5GN7bGhq56jleF71DJI+O1M+N0jBdby2ehaYM8EQ7fyrim"
|
||||
"j5Juq38tn5u/P3by/O3/MuciYzy7s5D4NGq/dMtSwOgvaKq1jrKS6HWjmRzvxoLCOYp933"
|
||||
"E+BGajk+IkNEk96LJbLi8lryeGO3DmuTx0tk2/Wnl6f/AHvgrXs="
|
||||
)
|
||||
25
mkdocs.yml
Normal file
25
mkdocs.yml
Normal file
@@ -0,0 +1,25 @@
|
||||
site_name: SimbaRAG Documentation
|
||||
site_description: Documentation for SimbaRAG - RAG-powered conversational AI
|
||||
|
||||
theme:
|
||||
name: material
|
||||
features:
|
||||
- content.code.copy
|
||||
- navigation.sections
|
||||
- navigation.expand
|
||||
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
- pymdownx.superfences
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- tables
|
||||
- toc:
|
||||
permalink: true
|
||||
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Architecture:
|
||||
- Authentication: authentication.md
|
||||
@@ -9,7 +9,6 @@ dependencies = [
|
||||
"python-dotenv>=1.0.0",
|
||||
"flask>=3.1.2",
|
||||
"httpx>=0.28.1",
|
||||
"ollama>=0.6.0",
|
||||
"openai>=2.0.1",
|
||||
"pydantic>=2.11.9",
|
||||
"pillow>=10.0.0",
|
||||
@@ -34,8 +33,8 @@ dependencies = [
|
||||
"langchain-chroma>=1.0.0",
|
||||
"langchain-community>=0.4.1",
|
||||
"jq>=1.10.0",
|
||||
"langchain-ollama>=1.0.1",
|
||||
"tavily-python>=0.7.17",
|
||||
"ynab>=1.3.0",
|
||||
]
|
||||
|
||||
[tool.aerich]
|
||||
@@ -35,12 +35,14 @@ class ConversationService {
|
||||
async sendQuery(
|
||||
query: string,
|
||||
conversation_id: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<QueryResponse> {
|
||||
const response = await userService.fetchWithRefreshToken(
|
||||
`${this.conversationBaseUrl}/query`,
|
||||
{
|
||||
method: "POST",
|
||||
body: JSON.stringify({ query, conversation_id }),
|
||||
signal,
|
||||
},
|
||||
);
|
||||
|
||||
|
Before Width: | Height: | Size: 5.8 KiB After Width: | Height: | Size: 5.8 KiB |
|
Before Width: | Height: | Size: 163 B After Width: | Height: | Size: 163 B |
@@ -43,12 +43,26 @@ export const ChatScreen = ({ setAuthenticated }: ChatScreenProps) => {
|
||||
const [isLoading, setIsLoading] = useState<boolean>(false);
|
||||
|
||||
const messagesEndRef = useRef<HTMLDivElement>(null);
|
||||
const isMountedRef = useRef<boolean>(true);
|
||||
const abortControllerRef = useRef<AbortController | null>(null);
|
||||
const simbaAnswers = ["meow.", "hiss...", "purrrrrr", "yowOWROWWowowr"];
|
||||
|
||||
const scrollToBottom = () => {
|
||||
messagesEndRef.current?.scrollIntoView({ behavior: "smooth" });
|
||||
};
|
||||
|
||||
// Cleanup effect to handle component unmounting
|
||||
useEffect(() => {
|
||||
isMountedRef.current = true;
|
||||
return () => {
|
||||
isMountedRef.current = false;
|
||||
// Abort any pending requests when component unmounts
|
||||
if (abortControllerRef.current) {
|
||||
abortControllerRef.current.abort();
|
||||
}
|
||||
};
|
||||
}, []);
|
||||
|
||||
const handleSelectConversation = (conversation: Conversation) => {
|
||||
setShowConversations(false);
|
||||
setSelectedConversation(conversation);
|
||||
@@ -156,10 +170,15 @@ export const ChatScreen = ({ setAuthenticated }: ChatScreenProps) => {
|
||||
return;
|
||||
}
|
||||
|
||||
// Create a new AbortController for this request
|
||||
const abortController = new AbortController();
|
||||
abortControllerRef.current = abortController;
|
||||
|
||||
try {
|
||||
const result = await conversationService.sendQuery(
|
||||
query,
|
||||
selectedConversation.id,
|
||||
abortController.signal,
|
||||
);
|
||||
setQuestionsAnswers(
|
||||
questionsAnswers.concat([{ question: query, answer: result.response }]),
|
||||
@@ -168,13 +187,23 @@ export const ChatScreen = ({ setAuthenticated }: ChatScreenProps) => {
|
||||
currMessages.concat([{ text: result.response, speaker: "simba" }]),
|
||||
);
|
||||
} catch (error) {
|
||||
console.error("Failed to send query:", error);
|
||||
// If session expired, redirect to login
|
||||
if (error instanceof Error && error.message.includes("Session expired")) {
|
||||
setAuthenticated(false);
|
||||
// Ignore abort errors (these are intentional cancellations)
|
||||
if (error instanceof Error && error.name === "AbortError") {
|
||||
console.log("Request was aborted");
|
||||
} else {
|
||||
console.error("Failed to send query:", error);
|
||||
// If session expired, redirect to login
|
||||
if (error instanceof Error && error.message.includes("Session expired")) {
|
||||
setAuthenticated(false);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
setIsLoading(false);
|
||||
// Only update loading state if component is still mounted
|
||||
if (isMountedRef.current) {
|
||||
setIsLoading(false);
|
||||
}
|
||||
// Clear the abort controller reference
|
||||
abortControllerRef.current = null;
|
||||
}
|
||||
};
|
||||
|
||||
|
Before Width: | Height: | Size: 3.4 MiB After Width: | Height: | Size: 3.4 MiB |
|
Before Width: | Height: | Size: 2.1 MiB After Width: | Height: | Size: 2.1 MiB |
0
scripts/__init__.py
Normal file
0
scripts/__init__.py
Normal file
@@ -4,9 +4,14 @@ import sqlite3
|
||||
|
||||
import httpx
|
||||
from dotenv import load_dotenv
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from image_process import describe_simba_image
|
||||
from request import PaperlessNGXService
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from utils.image_process import describe_simba_image
|
||||
from utils.request import PaperlessNGXService
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
@@ -1,18 +1,11 @@
|
||||
import json
|
||||
import os
|
||||
from typing import Literal
|
||||
import datetime
|
||||
from ollama import Client
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# Configure ollama client with URL from environment or default to localhost
|
||||
ollama_client = Client(
|
||||
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=10.0
|
||||
)
|
||||
|
||||
# This uses inferred filters — which means using LLM to create the metadata filters
|
||||
|
||||
|
||||
79
scripts/user_message_stats.py
Normal file
79
scripts/user_message_stats.py
Normal file
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Script to show how many messages each user has written
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from tortoise import Tortoise
|
||||
from blueprints.users.models import User
|
||||
from blueprints.conversation.models import Speaker
|
||||
import os
|
||||
|
||||
|
||||
async def get_user_message_stats():
|
||||
"""Get message count statistics per user"""
|
||||
|
||||
# Initialize database connection
|
||||
database_url = os.getenv("DATABASE_URL", "sqlite://raggr.db")
|
||||
await Tortoise.init(
|
||||
db_url=database_url,
|
||||
modules={
|
||||
"models": [
|
||||
"blueprints.users.models",
|
||||
"blueprints.conversation.models",
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
print("\n📊 User Message Statistics\n")
|
||||
print(
|
||||
f"{'Username':<20} {'Total Messages':<15} {'User Messages':<15} {'Conversations':<15}"
|
||||
)
|
||||
print("=" * 70)
|
||||
|
||||
# Get all users
|
||||
users = await User.all()
|
||||
|
||||
total_users = 0
|
||||
total_messages = 0
|
||||
|
||||
for user in users:
|
||||
# Get all conversations for this user
|
||||
conversations = await user.conversations.all()
|
||||
|
||||
if not conversations:
|
||||
continue
|
||||
|
||||
total_users += 1
|
||||
|
||||
# Count messages across all conversations
|
||||
user_message_count = 0
|
||||
total_message_count = 0
|
||||
|
||||
for conversation in conversations:
|
||||
messages = await conversation.messages.all()
|
||||
total_message_count += len(messages)
|
||||
|
||||
# Count only user messages (not assistant responses)
|
||||
user_messages = [msg for msg in messages if msg.speaker == Speaker.USER]
|
||||
user_message_count += len(user_messages)
|
||||
|
||||
total_messages += user_message_count
|
||||
|
||||
print(
|
||||
f"{user.username:<20} {total_message_count:<15} {user_message_count:<15} {len(conversations):<15}"
|
||||
)
|
||||
|
||||
print("=" * 70)
|
||||
print("\n📈 Summary:")
|
||||
print(f" Total active users: {total_users}")
|
||||
print(f" Total user messages: {total_messages}")
|
||||
print(
|
||||
f" Average messages per user: {total_messages / total_users if total_users > 0 else 0:.1f}\n"
|
||||
)
|
||||
|
||||
await Tortoise.close_connections()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(get_user_message_stats())
|
||||
@@ -1,16 +0,0 @@
|
||||
.git
|
||||
.gitignore
|
||||
README.md
|
||||
.env
|
||||
.DS_Store
|
||||
chromadb/
|
||||
chroma_db/
|
||||
raggr-frontend/node_modules/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
.pytest_cache/
|
||||
@@ -1 +0,0 @@
|
||||
3.13
|
||||
@@ -1,78 +0,0 @@
|
||||
import os
|
||||
from typing import cast
|
||||
|
||||
from langchain.agents import create_agent
|
||||
from langchain.chat_models import BaseChatModel
|
||||
from langchain.tools import tool
|
||||
from langchain_ollama import ChatOllama
|
||||
from langchain_openai import ChatOpenAI
|
||||
from tavily import AsyncTavilyClient
|
||||
|
||||
from blueprints.rag.logic import query_vector_store
|
||||
|
||||
openai_gpt_5_mini = ChatOpenAI(model="gpt-5-mini")
|
||||
ollama_deepseek = ChatOllama(model="llama3.1:8b", base_url=os.getenv("OLLAMA_URL"))
|
||||
model_with_fallback = cast(
|
||||
BaseChatModel, ollama_deepseek.with_fallbacks([openai_gpt_5_mini])
|
||||
)
|
||||
client = AsyncTavilyClient(os.getenv("TAVILY_KEY"), "")
|
||||
|
||||
|
||||
@tool
|
||||
async def web_search(query: str) -> str:
|
||||
"""Search the web for current information using Tavily.
|
||||
|
||||
Use this tool when you need to:
|
||||
- Find current information not in the knowledge base
|
||||
- Look up recent events, news, or updates
|
||||
- Verify facts or get additional context
|
||||
- Search for information outside of Simba's documents
|
||||
|
||||
Args:
|
||||
query: The search query to look up on the web
|
||||
|
||||
Returns:
|
||||
Search results from the web with titles, content, and source URLs
|
||||
"""
|
||||
response = await client.search(query=query, search_depth="basic")
|
||||
results = response.get("results", [])
|
||||
|
||||
if not results:
|
||||
return "No results found for the query."
|
||||
|
||||
formatted = "\n\n".join(
|
||||
[
|
||||
f"**{result['title']}**\n{result['content']}\nSource: {result['url']}"
|
||||
for result in results[:5]
|
||||
]
|
||||
)
|
||||
return formatted
|
||||
|
||||
|
||||
@tool(response_format="content_and_artifact")
|
||||
async def simba_search(query: str):
|
||||
"""Search through Simba's medical records, veterinary documents, and personal information.
|
||||
|
||||
Use this tool whenever the user asks questions about:
|
||||
- Simba's health history, medical records, or veterinary visits
|
||||
- Medications, treatments, or diagnoses
|
||||
- Weight, diet, or physical characteristics over time
|
||||
- Veterinary recommendations or advice
|
||||
- Ryan's (the owner's) information related to Simba
|
||||
- Any factual information that would be found in documents
|
||||
|
||||
Args:
|
||||
query: The user's question or information need about Simba
|
||||
|
||||
Returns:
|
||||
Relevant information from Simba's documents
|
||||
"""
|
||||
print(f"[SIMBA SEARCH] Tool called with query: {query}")
|
||||
serialized, docs = await query_vector_store(query=query)
|
||||
print(f"[SIMBA SEARCH] Found {len(docs)} documents")
|
||||
print(f"[SIMBA SEARCH] Serialized result length: {len(serialized)}")
|
||||
print(f"[SIMBA SEARCH] First 200 chars: {serialized[:200]}")
|
||||
return serialized, docs
|
||||
|
||||
|
||||
main_agent = create_agent(model=model_with_fallback, tools=[simba_search, web_search])
|
||||
@@ -1,73 +0,0 @@
|
||||
import os
|
||||
|
||||
from ollama import Client
|
||||
from openai import OpenAI
|
||||
|
||||
import logging
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
TRY_OLLAMA = os.getenv("TRY_OLLAMA", False)
|
||||
|
||||
|
||||
class LLMClient:
|
||||
def __init__(self):
|
||||
try:
|
||||
self.ollama_client = Client(
|
||||
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=1.0
|
||||
)
|
||||
self.ollama_client.chat(
|
||||
model="gemma3:4b", messages=[{"role": "system", "content": "test"}]
|
||||
)
|
||||
self.PROVIDER = "ollama"
|
||||
logging.info("Using Ollama as LLM backend")
|
||||
except Exception as e:
|
||||
print(e)
|
||||
self.openai_client = OpenAI()
|
||||
self.PROVIDER = "openai"
|
||||
logging.info("Using OpenAI as LLM backend")
|
||||
|
||||
def chat(
|
||||
self,
|
||||
prompt: str,
|
||||
system_prompt: str,
|
||||
):
|
||||
# Instituting a fallback if my gaming PC is not on
|
||||
if self.PROVIDER == "ollama":
|
||||
try:
|
||||
response = self.ollama_client.chat(
|
||||
model="gemma3:4b",
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt,
|
||||
},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
)
|
||||
output = response.message.content
|
||||
return output
|
||||
except Exception as e:
|
||||
logging.error(f"Could not connect to OLLAMA: {str(e)}")
|
||||
|
||||
response = self.openai_client.responses.create(
|
||||
model="gpt-4o-mini",
|
||||
input=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt,
|
||||
},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
)
|
||||
output = response.output_text
|
||||
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
client = Client()
|
||||
client.chat(model="gemma3:4b", messages=[{"role": "system", "promp": "hack"}])
|
||||
@@ -1,71 +0,0 @@
|
||||
from tortoise import BaseDBAsyncClient
|
||||
|
||||
RUN_IN_TRANSACTION = True
|
||||
|
||||
|
||||
async def upgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
CREATE TABLE IF NOT EXISTS "users" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"username" VARCHAR(255) NOT NULL,
|
||||
"password" BYTEA,
|
||||
"email" VARCHAR(100) NOT NULL UNIQUE,
|
||||
"oidc_subject" VARCHAR(255) UNIQUE,
|
||||
"auth_provider" VARCHAR(50) NOT NULL DEFAULT 'local',
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS "idx_users_oidc_su_5aec5a" ON "users" ("oidc_subject");
|
||||
CREATE TABLE IF NOT EXISTS "conversations" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"name" VARCHAR(255) NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"user_id" UUID REFERENCES "users" ("id") ON DELETE CASCADE
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS "conversation_messages" (
|
||||
"id" UUID NOT NULL PRIMARY KEY,
|
||||
"text" TEXT NOT NULL,
|
||||
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"speaker" VARCHAR(10) NOT NULL,
|
||||
"conversation_id" UUID NOT NULL REFERENCES "conversations" ("id") ON DELETE CASCADE
|
||||
);
|
||||
COMMENT ON COLUMN "conversation_messages"."speaker" IS 'USER: user\nSIMBA: simba';
|
||||
CREATE TABLE IF NOT EXISTS "aerich" (
|
||||
"id" SERIAL NOT NULL PRIMARY KEY,
|
||||
"version" VARCHAR(255) NOT NULL,
|
||||
"app" VARCHAR(100) NOT NULL,
|
||||
"content" JSONB NOT NULL
|
||||
);"""
|
||||
|
||||
|
||||
async def downgrade(db: BaseDBAsyncClient) -> str:
|
||||
return """
|
||||
"""
|
||||
|
||||
|
||||
MODELS_STATE = (
|
||||
"eJztmmtP4zgUhv9KlE+MxCLoUGaEViulpex0Z9qO2nR3LjuK3MRtvSROJnYGKsR/X9u5J0"
|
||||
"56AUqL+gXosU9sPz7OeY/Lveq4FrTJSdvFv6BPAEUuVi+VexUDB7I/pO3Higo8L23lBgom"
|
||||
"tnAwMz1FC5gQ6gOTssYpsAlkJgsS00deNBgObJsbXZN1RHiWmgKMfgbQoO4M0jn0WcP3H8"
|
||||
"yMsAXvIIk/ejfGFEHbys0bWXxsYTfowhO28bh7dS168uEmhunagYPT3t6Czl2cdA8CZJ1w"
|
||||
"H942gxj6gEIrsww+y2jZsSmcMTNQP4DJVK3UYMEpCGwOQ/19GmCTM1DESPzH+R/qGngYao"
|
||||
"4WYcpZ3D+Eq0rXLKwqH6r9QRsevb14I1bpEjrzRaMgoj4IR0BB6Cq4piDF7xLK9hz4cpRx"
|
||||
"/wJMNtFNMMaGlGMaQzHIGNBm1FQH3Bk2xDM6Zx8bzWYNxr+1oSDJegmULovrMOr7UVMjbO"
|
||||
"NIU4SmD/mSDUDLIK9YC0UOlMPMexaQWpHrSfzHjgJma7AG2F5Eh6CGr97tdUa61vvMV+IQ"
|
||||
"8tMWiDS9w1sawrooWI8uCluRPET5p6t/UPhH5dug3ynGftJP/6byOYGAugZ2bw1gZc5rbI"
|
||||
"3B5DY28KwNNzbvedjYF93YaPKZfSXQN9bLIBmXR6SRaG5b3MTNkwZPvdMbac7gMMrwrl0f"
|
||||
"ohn+CBcCYZfNA2BTliwi0TGOHrOr0FJrOgsf3CZqJBsUbHVsTZCG2VMbtbWrjioYToB5cw"
|
||||
"t8y6iA6UBCwAySMtBW5Hn9cQjtRJrJWWYFXC984m6+VarYClZuw80wytErNzkNp2gBmK3b"
|
||||
"isbmI9XQWaKCMxBXE8NGdiMPonivRTGFd5KUrzOrHGXcf19EcV0q73zRc1k8lr5HPe3Lm1"
|
||||
"wm/zTo/xl3z0jl9qdB66CQX6OQKitk4kFwIxMDvIDs4MApSYHc7mbcX/joqONRZ3ip8Iz+"
|
||||
"Lx51ey3tUiHImQB1tS3OVZlnpysUmWenlTUmbyocoGyiWe81L3F9ynf+nkpYs3Dh9UgpW7"
|
||||
"w/21mKSzWtJFzW1bbPqeREzSCRbnEtUa3V+NE+aLP912Z8H9e9tMz67ItG28LFpQcIuXV9"
|
||||
"SWS2EAb+Qg4z61WAOVnQsP7Z1ZJeBq/F9WpWbjFkrW5fG36VS964fzZuW1/1jlagCx2A7H"
|
||||
"WiNHF4mhBdfuKfMkDPTlcTPXWqpyR7XGSZBgkm/0FTUjlUkyz6bQS0GKTb5fksB55p+bnh"
|
||||
"+e4vZFWJdjnQkuP23qKq7ZrAfkQaynNtrhKmzeoobZa1+aG4fZ3F7eHrn1exscntcqlIWX"
|
||||
"Y1X/pfh6e5n99lgbTde3kN+sicq5J6Lmo5rqvoQNpnZ0q6Lq64IpZWdBxzIRiinX9RYSe+"
|
||||
"HfmtcXb+7vz924vz96yLmElieVfzMuj29SUVHD8I0muXav2RcTnUb6mcY0djHREXdt9PgM"
|
||||
"9SX7ARKcSS9P7XaNCvvE6NXQogx5gt8LuFTHqs2IjQH7uJtYYiX3X9dz/Fr3kKuZk/oCW7"
|
||||
"eN3mZeHD/9BpOYI="
|
||||
)
|
||||
12
users.py
12
users.py
@@ -1,12 +0,0 @@
|
||||
import sqlite3
|
||||
|
||||
|
||||
class User:
|
||||
def __init__(self, email: str, password_hash: str):
|
||||
self.email = email
|
||||
self.is_authenticated
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
connection = sqlite3.connect("users.db")
|
||||
c = connection.cursor()
|
||||
0
utils/__init__.py
Normal file
0
utils/__init__.py
Normal file
@@ -3,7 +3,6 @@ from math import ceil
|
||||
import re
|
||||
from typing import Union
|
||||
from uuid import UUID, uuid4
|
||||
from ollama import Client
|
||||
from chromadb.utils.embedding_functions.openai_embedding_function import (
|
||||
OpenAIEmbeddingFunction,
|
||||
)
|
||||
@@ -13,10 +12,6 @@ from llm import LLMClient
|
||||
|
||||
load_dotenv()
|
||||
|
||||
ollama_client = Client(
|
||||
host=os.getenv("OLLAMA_HOST", "http://localhost:11434"), timeout=1.0
|
||||
)
|
||||
|
||||
|
||||
def remove_headers_footers(text, header_patterns=None, footer_patterns=None):
|
||||
if header_patterns is None:
|
||||
@@ -8,7 +8,7 @@ import ollama
|
||||
from PIL import Image
|
||||
import fitz
|
||||
|
||||
from request import PaperlessNGXService
|
||||
from .request import PaperlessNGXService
|
||||
|
||||
load_dotenv()
|
||||
|
||||
342
utils/ynab_service.py
Normal file
342
utils/ynab_service.py
Normal file
@@ -0,0 +1,342 @@
|
||||
"""YNAB API service for querying budget data."""
|
||||
|
||||
import os
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Optional
|
||||
|
||||
from dotenv import load_dotenv
|
||||
import ynab
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class YNABService:
|
||||
"""Service for interacting with YNAB API."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize YNAB API client."""
|
||||
self.access_token = os.getenv("YNAB_ACCESS_TOKEN", "")
|
||||
self.budget_id = os.getenv("YNAB_BUDGET_ID", "")
|
||||
|
||||
if not self.access_token:
|
||||
raise ValueError("YNAB_ACCESS_TOKEN environment variable is required")
|
||||
|
||||
# Configure API client
|
||||
configuration = ynab.Configuration(access_token=self.access_token)
|
||||
self.api_client = ynab.ApiClient(configuration)
|
||||
|
||||
# Initialize API endpoints
|
||||
self.budgets_api = ynab.BudgetsApi(self.api_client)
|
||||
self.transactions_api = ynab.TransactionsApi(self.api_client)
|
||||
self.months_api = ynab.MonthsApi(self.api_client)
|
||||
self.categories_api = ynab.CategoriesApi(self.api_client)
|
||||
|
||||
# Get budget ID if not provided
|
||||
if not self.budget_id:
|
||||
budgets_response = self.budgets_api.get_budgets()
|
||||
if budgets_response.data and budgets_response.data.budgets:
|
||||
self.budget_id = budgets_response.data.budgets[0].id
|
||||
else:
|
||||
raise ValueError("No YNAB budgets found")
|
||||
|
||||
def get_budget_summary(self) -> dict[str, Any]:
|
||||
"""Get overall budget summary and health status.
|
||||
|
||||
Returns:
|
||||
Dictionary containing budget summary with to-be-budgeted amount,
|
||||
total budgeted, total activity, and overall budget health.
|
||||
"""
|
||||
budget_response = self.budgets_api.get_budget_by_id(self.budget_id)
|
||||
budget_data = budget_response.data.budget
|
||||
|
||||
# Calculate totals from categories
|
||||
to_be_budgeted = (
|
||||
budget_data.months[0].to_be_budgeted / 1000 if budget_data.months else 0
|
||||
)
|
||||
|
||||
total_budgeted = 0
|
||||
total_activity = 0
|
||||
total_available = 0
|
||||
|
||||
for category_group in budget_data.category_groups or []:
|
||||
if category_group.deleted or category_group.hidden:
|
||||
continue
|
||||
for category in category_group.categories or []:
|
||||
if category.deleted or category.hidden:
|
||||
continue
|
||||
total_budgeted += category.budgeted / 1000
|
||||
total_activity += category.activity / 1000
|
||||
total_available += category.balance / 1000
|
||||
|
||||
return {
|
||||
"budget_name": budget_data.name,
|
||||
"to_be_budgeted": round(to_be_budgeted, 2),
|
||||
"total_budgeted": round(total_budgeted, 2),
|
||||
"total_activity": round(total_activity, 2),
|
||||
"total_available": round(total_available, 2),
|
||||
"currency_format": budget_data.currency_format.iso_code
|
||||
if budget_data.currency_format
|
||||
else "USD",
|
||||
"summary": f"Budget '{budget_data.name}' has ${abs(to_be_budgeted):.2f} {'to be budgeted' if to_be_budgeted > 0 else 'overbudgeted'}. "
|
||||
f"Total budgeted: ${total_budgeted:.2f}, Total spent: ${abs(total_activity):.2f}, "
|
||||
f"Total available: ${total_available:.2f}.",
|
||||
}
|
||||
|
||||
def get_transactions(
|
||||
self,
|
||||
start_date: Optional[str] = None,
|
||||
end_date: Optional[str] = None,
|
||||
category_name: Optional[str] = None,
|
||||
payee_name: Optional[str] = None,
|
||||
limit: int = 50,
|
||||
) -> dict[str, Any]:
|
||||
"""Get transactions filtered by date range, category, or payee.
|
||||
|
||||
Args:
|
||||
start_date: Start date in YYYY-MM-DD format (defaults to 30 days ago)
|
||||
end_date: End date in YYYY-MM-DD format (defaults to today)
|
||||
category_name: Filter by category name (case-insensitive partial match)
|
||||
payee_name: Filter by payee name (case-insensitive partial match)
|
||||
limit: Maximum number of transactions to return (default 50)
|
||||
|
||||
Returns:
|
||||
Dictionary containing matching transactions and summary statistics.
|
||||
"""
|
||||
# Set default date range if not provided
|
||||
if not start_date:
|
||||
start_date = (datetime.now() - timedelta(days=30)).strftime("%Y-%m-%d")
|
||||
if not end_date:
|
||||
end_date = datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
# Get transactions
|
||||
transactions_response = self.transactions_api.get_transactions(
|
||||
self.budget_id, since_date=start_date
|
||||
)
|
||||
|
||||
transactions = transactions_response.data.transactions or []
|
||||
|
||||
# Filter by date range, category, and payee
|
||||
filtered_transactions = []
|
||||
total_amount = 0
|
||||
|
||||
for txn in transactions:
|
||||
# Skip if deleted or before start date or after end date
|
||||
if txn.deleted:
|
||||
continue
|
||||
txn_date = str(txn.date)
|
||||
if txn_date < start_date or txn_date > end_date:
|
||||
continue
|
||||
|
||||
# Filter by category if specified
|
||||
if category_name and txn.category_name:
|
||||
if category_name.lower() not in txn.category_name.lower():
|
||||
continue
|
||||
|
||||
# Filter by payee if specified
|
||||
if payee_name and txn.payee_name:
|
||||
if payee_name.lower() not in txn.payee_name.lower():
|
||||
continue
|
||||
|
||||
amount = txn.amount / 1000 # Convert milliunits to dollars
|
||||
filtered_transactions.append(
|
||||
{
|
||||
"date": txn_date,
|
||||
"payee": txn.payee_name,
|
||||
"category": txn.category_name,
|
||||
"memo": txn.memo,
|
||||
"amount": round(amount, 2),
|
||||
"approved": txn.approved,
|
||||
}
|
||||
)
|
||||
total_amount += amount
|
||||
|
||||
# Sort by date (most recent first) and limit
|
||||
filtered_transactions.sort(key=lambda x: x["date"], reverse=True)
|
||||
filtered_transactions = filtered_transactions[:limit]
|
||||
|
||||
return {
|
||||
"transactions": filtered_transactions,
|
||||
"count": len(filtered_transactions),
|
||||
"total_amount": round(total_amount, 2),
|
||||
"start_date": start_date,
|
||||
"end_date": end_date,
|
||||
"filters": {"category": category_name, "payee": payee_name},
|
||||
}
|
||||
|
||||
def get_category_spending(self, month: Optional[str] = None) -> dict[str, Any]:
|
||||
"""Get spending breakdown by category for a specific month.
|
||||
|
||||
Args:
|
||||
month: Month in YYYY-MM format (defaults to current month)
|
||||
|
||||
Returns:
|
||||
Dictionary containing spending by category and summary.
|
||||
"""
|
||||
if not month:
|
||||
month = datetime.now().strftime("%Y-%m-01")
|
||||
else:
|
||||
# Ensure format is YYYY-MM-01
|
||||
if len(month) == 7: # YYYY-MM
|
||||
month = f"{month}-01"
|
||||
|
||||
# Get budget month
|
||||
month_response = self.months_api.get_budget_month(self.budget_id, month)
|
||||
|
||||
month_data = month_response.data.month
|
||||
|
||||
categories_spending = []
|
||||
total_budgeted = 0
|
||||
total_activity = 0
|
||||
total_available = 0
|
||||
overspent_categories = []
|
||||
|
||||
for category in month_data.categories or []:
|
||||
if category.deleted or category.hidden:
|
||||
continue
|
||||
|
||||
budgeted = category.budgeted / 1000
|
||||
activity = category.activity / 1000
|
||||
available = category.balance / 1000
|
||||
|
||||
total_budgeted += budgeted
|
||||
total_activity += activity
|
||||
total_available += available
|
||||
|
||||
# Track overspent categories
|
||||
if available < 0:
|
||||
overspent_categories.append(
|
||||
{
|
||||
"name": category.name,
|
||||
"budgeted": round(budgeted, 2),
|
||||
"spent": round(abs(activity), 2),
|
||||
"overspent_by": round(abs(available), 2),
|
||||
}
|
||||
)
|
||||
|
||||
# Only include categories with activity
|
||||
if activity != 0:
|
||||
categories_spending.append(
|
||||
{
|
||||
"category": category.name,
|
||||
"budgeted": round(budgeted, 2),
|
||||
"activity": round(activity, 2),
|
||||
"available": round(available, 2),
|
||||
"goal_type": category.goal_type
|
||||
if hasattr(category, "goal_type")
|
||||
else None,
|
||||
}
|
||||
)
|
||||
|
||||
# Sort by absolute activity (highest spending first)
|
||||
categories_spending.sort(key=lambda x: abs(x["activity"]), reverse=True)
|
||||
|
||||
return {
|
||||
"month": month[:7], # Return YYYY-MM format
|
||||
"categories": categories_spending,
|
||||
"total_budgeted": round(total_budgeted, 2),
|
||||
"total_spent": round(abs(total_activity), 2),
|
||||
"total_available": round(total_available, 2),
|
||||
"overspent_categories": overspent_categories,
|
||||
"to_be_budgeted": round(month_data.to_be_budgeted / 1000, 2)
|
||||
if month_data.to_be_budgeted
|
||||
else 0,
|
||||
}
|
||||
|
||||
def get_spending_insights(self, months_back: int = 3) -> dict[str, Any]:
|
||||
"""Generate insights about spending patterns and budget health.
|
||||
|
||||
Args:
|
||||
months_back: Number of months to analyze (default 3)
|
||||
|
||||
Returns:
|
||||
Dictionary containing spending insights, trends, and recommendations.
|
||||
"""
|
||||
current_month = datetime.now()
|
||||
monthly_data = []
|
||||
|
||||
# Collect data for the last N months
|
||||
for i in range(months_back):
|
||||
month_date = current_month - timedelta(days=30 * i)
|
||||
month_str = month_date.strftime("%Y-%m-01")
|
||||
|
||||
try:
|
||||
month_spending = self.get_category_spending(month_str)
|
||||
monthly_data.append(
|
||||
{
|
||||
"month": month_str[:7],
|
||||
"total_spent": month_spending["total_spent"],
|
||||
"total_budgeted": month_spending["total_budgeted"],
|
||||
"overspent_categories": month_spending["overspent_categories"],
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
# Skip months that don't have data yet
|
||||
continue
|
||||
|
||||
if not monthly_data:
|
||||
return {"error": "No spending data available for analysis"}
|
||||
|
||||
# Calculate average spending
|
||||
avg_spending = sum(m["total_spent"] for m in monthly_data) / len(monthly_data)
|
||||
current_spending = monthly_data[0]["total_spent"] if monthly_data else 0
|
||||
|
||||
# Identify spending trend
|
||||
if len(monthly_data) >= 2:
|
||||
recent_avg = sum(m["total_spent"] for m in monthly_data[:2]) / 2
|
||||
older_avg = sum(m["total_spent"] for m in monthly_data[1:]) / (
|
||||
len(monthly_data) - 1
|
||||
)
|
||||
trend = (
|
||||
"increasing"
|
||||
if recent_avg > older_avg * 1.1
|
||||
else "decreasing"
|
||||
if recent_avg < older_avg * 0.9
|
||||
else "stable"
|
||||
)
|
||||
else:
|
||||
trend = "insufficient data"
|
||||
|
||||
# Find frequently overspent categories
|
||||
overspent_frequency = {}
|
||||
for month in monthly_data:
|
||||
for cat in month["overspent_categories"]:
|
||||
cat_name = cat["name"]
|
||||
if cat_name not in overspent_frequency:
|
||||
overspent_frequency[cat_name] = 0
|
||||
overspent_frequency[cat_name] += 1
|
||||
|
||||
frequently_overspent = [
|
||||
{"category": cat, "months_overspent": count}
|
||||
for cat, count in overspent_frequency.items()
|
||||
if count > 1
|
||||
]
|
||||
frequently_overspent.sort(key=lambda x: x["months_overspent"], reverse=True)
|
||||
|
||||
# Generate recommendations
|
||||
recommendations = []
|
||||
if current_spending > avg_spending * 1.2:
|
||||
recommendations.append(
|
||||
f"Current month spending (${current_spending:.2f}) is significantly higher than your {months_back}-month average (${avg_spending:.2f})"
|
||||
)
|
||||
|
||||
if frequently_overspent:
|
||||
top_overspent = frequently_overspent[0]
|
||||
recommendations.append(
|
||||
f"'{top_overspent['category']}' has been overspent in {top_overspent['months_overspent']} of the last {months_back} months"
|
||||
)
|
||||
|
||||
if trend == "increasing":
|
||||
recommendations.append(
|
||||
"Your spending trend is increasing. Consider reviewing your budget allocations."
|
||||
)
|
||||
|
||||
return {
|
||||
"analysis_period": f"Last {months_back} months",
|
||||
"average_monthly_spending": round(avg_spending, 2),
|
||||
"current_month_spending": round(current_spending, 2),
|
||||
"spending_trend": trend,
|
||||
"frequently_overspent_categories": frequently_overspent,
|
||||
"recommendations": recommendations,
|
||||
"monthly_breakdown": monthly_data,
|
||||
}
|
||||
18
services/raggr/uv.lock → uv.lock
generated
18
services/raggr/uv.lock → uv.lock
generated
@@ -2551,6 +2551,7 @@ dependencies = [
|
||||
{ name = "tomlkit" },
|
||||
{ name = "tortoise-orm" },
|
||||
{ name = "tortoise-orm-stubs" },
|
||||
{ name = "ynab" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
@@ -2586,6 +2587,7 @@ requires-dist = [
|
||||
{ name = "tomlkit", specifier = ">=0.13.3" },
|
||||
{ name = "tortoise-orm", specifier = ">=0.25.1" },
|
||||
{ name = "tortoise-orm-stubs", specifier = ">=1.0.2" },
|
||||
{ name = "ynab", specifier = ">=1.3.0" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3385,6 +3387,22 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/73/ae/b48f95715333080afb75a4504487cbe142cae1268afc482d06692d605ae6/yarl-1.22.0-py3-none-any.whl", hash = "sha256:1380560bdba02b6b6c90de54133c81c9f2a453dee9912fe58c1dcced1edb7cff", size = 46814, upload-time = "2025-10-06T14:12:53.872Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ynab"
|
||||
version = "1.9.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "certifi" },
|
||||
{ name = "pydantic" },
|
||||
{ name = "python-dateutil" },
|
||||
{ name = "typing-extensions" },
|
||||
{ name = "urllib3" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/9a/3e/36599ae876db3e1d32e393ab0934547df75bab70373c14ca5805246f99bc/ynab-1.9.0.tar.gz", hash = "sha256:fa50bdff641b3a273661e9f6e8a210f5ad98991a998dc09dec0a8122d734d1c6", size = 64898, upload-time = "2025-10-06T19:14:32.707Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b2/9c/0ccd11bcdf7522fcb2823fcd7ffbb48e3164d72caaf3f920c7b068347175/ynab-1.9.0-py3-none-any.whl", hash = "sha256:72ac0219605b4280149684ecd0fec3bd75d938772d65cdeea9b3e66a1b2f470d", size = 208674, upload-time = "2025-10-06T19:14:31.719Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zipp"
|
||||
version = "3.23.0"
|
||||
Reference in New Issue
Block a user