72 Commits

Author SHA1 Message Date
ryan
0895668ddd Merge pull request 'rc/langchain-migration' (#11) from rc/langchain-migration into main
Reviewed-on: #11
2026-01-11 09:22:40 -05:00
Ryan Chen
07512409f1 Adding loading indicator 2026-01-11 09:22:28 -05:00
Ryan Chen
12eb110313 linter 2026-01-11 09:12:37 -05:00
ryan
1a026f76a1 Merge pull request 'okok' (#10) from rc/01012025-retitling into main
Reviewed-on: #10
2026-01-01 22:00:32 -05:00
Ryan Chen
da3a464897 okok 2026-01-01 22:00:12 -05:00
Ryan Chen
913875188a oidc 2025-12-25 07:36:26 -08:00
Ryan Chen
f5e2d68cd2 Making UI changes 2025-12-24 17:12:56 -08:00
Ryan Chen
70799ffb7d refactor 2025-11-10 15:51:13 -05:00
Ryan Chen
7f1d4fbdda asdf 2025-10-29 22:17:45 -04:00
Ryan Chen
5ebdd60ea0 Making better 2025-10-29 21:28:23 -04:00
ryan
289045e7d0 Merge pull request 'mobile-responsive-layout' (#9) from mobile-responsive-layout into main
Reviewed-on: #9
2025-10-29 21:15:14 -04:00
ryan
ceea83cb54 Merge branch 'main' into mobile-responsive-layout 2025-10-29 21:15:10 -04:00
Ryan Chen
1b60aab97c sdf 2025-10-29 21:14:52 -04:00
ryan
210bfc1476 Merge pull request 'query classification' (#8) from async-reindexing into main
Reviewed-on: #8
2025-10-29 21:13:42 -04:00
Ryan Chen
454fb1b52c Add authentication validation on login screen load
- Add validateToken() method to userService to check if refresh token is valid
- Automatically redirect to chat if user already has valid session
- Show 'Checking authentication...' loading state during validation
- Prevents unnecessary login if user is already authenticated
- Improves UX by skipping login screen when not needed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 12:24:10 -04:00
Ryan Chen
c3f2501585 Clear text input immediately upon message submission
- Clear input field right after user sends message (before API call)
- Add validation to prevent submitting empty/whitespace-only messages
- Improve UX by allowing user to type next message while waiting for response
- Works for both simba mode and normal mode

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 12:22:32 -04:00
Ryan Chen
1da21fabee Add auto-scroll to bottom for new messages
- Automatically scroll to latest message when new messages arrive
- Uses smooth scrolling behavior for better UX
- Triggers on message array changes
- Improves chat experience by keeping conversation in view

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 12:12:05 -04:00
Ryan Chen
dd5690ee53 Add submit on Enter for chat textarea
- Press Enter to submit message
- Press Shift+Enter to insert new line
- Add helpful placeholder text explaining keyboard shortcuts
- Improve chat UX with standard messaging behavior

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 12:07:47 -04:00
Ryan Chen
5e7ac28b6f Update add_user.py to use configurable database path
- Use DATABASE_PATH and DATABASE_URL environment variables
- Consistent with app.py and aerich_config.py configuration
- Add environment variable documentation to help text
- Default remains database/raggr.db for backward compatibility

Usage:
  DATABASE_PATH=dev.db python add_user.py list

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 12:02:56 -04:00
Ryan Chen
29f8894e4a Add configurable database path via environment variables
- Add DATABASE_PATH environment variable support in app.py and aerich_config.py
- DATABASE_PATH: For simple relative/absolute paths (default: database/raggr.db)
- DATABASE_URL: For full connection strings (overrides DATABASE_PATH if set)
- Create .env.example with all configuration options documented
- Maintains backward compatibility with default database location

Usage:
  # Use default path
  python app.py

  # Use custom path for development
  DATABASE_PATH=dev.db python app.py

  # Use full connection string
  DATABASE_URL=sqlite://custom/path.db python app.py

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 12:01:16 -04:00
Ryan Chen
19d1df2f68 Improve mobile responsiveness with Tailwind breakpoints
- Replace fixed-width containers (min-w-xl max-w-xl) with responsive classes
- Mobile: full width with padding, Tablet: 90% max 768px, Desktop: max 1024px
- Make ChatScreen header stack vertically on mobile, horizontal on desktop
- Add touch-friendly button sizes (min 44x44px tap targets)
- Optimize textarea and form inputs for mobile keyboards
- Add text wrapping (break-words) to message bubbles to prevent overflow
- Apply responsive text sizing (text-sm sm:text-base) throughout
- Improve ConversationList with touch-friendly hit areas
- Add responsive padding/spacing across all components

All components now use standard Tailwind breakpoints:
- sm: 640px+ (tablet)
- md: 768px+ (larger tablet)
- lg: 1024px+ (desktop)
- xl: 1280px+ (large desktop)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 11:57:54 -04:00
Ryan Chen
e577cb335b query classification 2025-10-26 17:29:00 -04:00
Ryan Chen
591788dfa4 reindex pls 2025-10-26 11:06:32 -04:00
Ryan Chen
561b5bddce reindex pls 2025-10-26 11:04:33 -04:00
Ryan Chen
ddd455a4c6 reindex pls 2025-10-26 11:02:51 -04:00
ryan
07424e77e0 Merge pull request 'favicon' (#7) from update-favicon-and-title into main
Reviewed-on: #7
2025-10-26 10:49:27 -04:00
Ryan Chen
a56f752917 favicon 2025-10-26 10:48:59 -04:00
Ryan Chen
e8264e80ce Changing DB thing 2025-10-26 09:36:33 -04:00
ryan
04350045d3 Merge pull request 'Adding support for conversations and multiple threads' (#6) from conversation-uplift into main
Reviewed-on: #6
2025-10-26 09:25:52 -04:00
Ryan Chen
f16e13fccc big uplift 2025-10-26 09:25:17 -04:00
ryan
245db92524 Merge pull request 'enabling login btw users' (#5) from quart-login into main
Reviewed-on: #5
2025-10-25 09:34:08 -04:00
Ryan Chen
29ac724d50 enabling login btw users 2025-10-25 09:30:54 -04:00
Ryan Chen
7161c09a4e do not fully delete lol 2025-10-24 08:47:59 -04:00
Ryan Chen
68d73b62e8 Instituting LLM fallback to OpenAI if gaming PC is not on 2025-10-24 08:44:08 -04:00
Ryan Chen
6b616137d3 oops 2025-10-23 22:45:14 -04:00
Ryan Chen
841b6ebd4f i commit 2025-10-23 22:39:26 -04:00
Ryan Chen
45a5e92aee Added conversation history (#4)
Reviewed-on: #4
Co-authored-by: Ryan Chen <ryan@torrtle.co>
Co-committed-by: Ryan Chen <ryan@torrtle.co>
2025-10-23 22:29:12 -04:00
Ryan Chen
8479898cc4 Logging 2025-10-16 22:43:14 -04:00
Ryan Chen
acaf681927 Metadata filtering 2025-10-16 22:36:21 -04:00
Ryan Chen
2bbe33fedc Starting attempt #2 at metadata filtering 2025-10-14 22:13:01 -04:00
Ryan Chen
b872750444 Only use OpenAI for embedding 2025-10-14 20:06:32 -04:00
Ryan Chen
376baccadb message-style frontend 2025-10-10 23:28:41 -04:00
Ryan Chen
c978b1a255 Reducing startup time/cost 2025-10-08 23:21:22 -04:00
Ryan Chen
51b9932389 fixing loal llm 2025-10-08 22:52:49 -04:00
Ryan Chen
ebf39480b6 urf 2025-10-08 22:46:16 -04:00
Ryan Chen
e4a04331cb add some more debugging 2025-10-08 21:17:45 -04:00
Ryan Chen
166ffb4c09 i only ship bugs 2025-10-08 21:13:15 -04:00
Ryan Chen
64e286e623 oops 2025-10-08 21:07:33 -04:00
Ryan Chen
c6c14729dd interseting 2025-10-08 21:03:42 -04:00
Ryan Chen
910097d13b data 2025-10-05 20:31:46 -04:00
Ryan Chen
0bb3e3172b adding image processing pipeline immich -> paperless 2025-10-04 08:54:10 -04:00
Ryan Chen
24b30bc8a3 Adding Simba mode 2025-10-03 20:25:57 -04:00
Ryan Chen
3ffc95a1b0 Switch to OpenAI embeddings for ChromaDB
Replace Ollama embedding function with OpenAI's text-embedding-3-small
model for improved embedding quality and consistency.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 21:05:17 -04:00
Ryan Chen
c5091dc07a Configure Docker for Linux host networking and add startup reindex
- Switch to host network mode for direct access to Ollama on host
- Update OLLAMA_URL to use localhost:11434
- Add startup.sh script to trigger reindex before app starts
- Update Dockerfile to execute startup script

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 21:02:55 -04:00
Ryan Chen
c140758560 asfd 2025-10-02 20:57:19 -04:00
Ryan Chen
ab3a0eb442 Reorganize Dockerfile to copy application code before frontend build
Move Python application code copy before frontend build step to improve
Dockerfile organization and ensure all app code is available earlier.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 20:48:52 -04:00
Ryan Chen
c619d78922 Adding axios 2025-10-02 20:46:10 -04:00
Ryan Chen
c20ae0a4b9 Add missing @tailwindcss/postcss dependency to frontend
Fix Docker build failure by adding @tailwindcss/postcss package
required by postcss.config.mjs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 20:44:49 -04:00
Ryan Chen
26cc01b58b Add frontend build step to Dockerfile
Install Node.js and Yarn, then build the raggr-frontend during Docker image build process.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 20:42:01 -04:00
Ryan Chen
746b60e070 Switch to using torrtle/simbarag:latest Docker image
Replace local build with pre-built image from Docker Hub

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 20:39:36 -04:00
Ryan Chen
577c9144ac Switch Dockerfile to use uv for dependency management
- Install uv via official installer script
- Replace pip with uv pip install --system
- Add uv to PATH for container usage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 20:36:45 -04:00
Ryan Chen
2b2891bd79 Fix and add missing dependencies to pyproject.toml
- Fix dotenv package name to python-dotenv
- Add pillow for image processing
- Add pymupdf for PDF handling

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 20:34:59 -04:00
Ryan Chen
03b033e9a4 Configure ollama to use external host instead of docker service
- Update all ollama clients to use configurable OLLAMA_URL environment variable
- Remove ollama service from docker-compose.yml to use external ollama instance
- Configure docker-compose to connect to host ollama via 172.17.0.1:11434 (Linux) or host.docker.internal (macOS/Windows)
- Add cross-platform compatibility with extra_hosts mapping
- Update embedding function fallback URL for consistency

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-02 20:29:48 -04:00
Ryan Chen
a640ae5fed Docker stuff 2025-10-02 20:21:48 -04:00
Ryan Chen
99c98b7e42 yeet 2025-10-02 19:21:24 -04:00
ryan
a69f7864f3 Merge pull request 'yeat' (#3) from rc/9-metadata-date-filtering into main
Reviewed-on: #3
2025-08-07 17:43:59 -04:00
Ryan Chen
679cfb08e4 yeat 2025-08-07 17:43:24 -04:00
ryan
fc504d3e9c Merge pull request 'Adding some funny stuff' (#2) from data-preprocessing into main
Reviewed-on: #2

implements #1
2025-07-30 20:30:34 -04:00
Ryan Chen
c7152d3f32 Moving chromadb to env var 2025-07-30 20:27:03 -04:00
Ryan Chen
0a88a03c90 Expanded context window, CLI'd the app, and added preprocessing 2025-07-30 19:58:29 -04:00
Ryan Chen
b43ef63449 Adding some funny stuff 2025-07-29 22:59:40 -04:00
ryan
b698109183 Merge pull request 'Adding more embeddings' (#1) from better-embeddings into main
Reviewed-on: #1
2025-07-26 19:55:31 -04:00
80 changed files with 13761 additions and 134 deletions

46
.env.example Normal file
View File

@@ -0,0 +1,46 @@
# Database Configuration
# PostgreSQL is recommended (required for OIDC features)
DATABASE_URL=postgres://raggr:changeme@postgres:5432/raggr
# PostgreSQL credentials (if using docker-compose postgres service)
POSTGRES_USER=raggr
POSTGRES_PASSWORD=changeme
POSTGRES_DB=raggr
# JWT Configuration
JWT_SECRET_KEY=your-secret-key-here
# Paperless Configuration
PAPERLESS_TOKEN=your-paperless-token
BASE_URL=192.168.1.5:8000
# Ollama Configuration
OLLAMA_URL=http://192.168.1.14:11434
OLLAMA_HOST=http://192.168.1.14:11434
# ChromaDB Configuration
# For Docker: This is automatically set to /app/data/chromadb
# For local development: Set to a local directory path
CHROMADB_PATH=./data/chromadb
# OpenAI Configuration
OPENAI_API_KEY=your-openai-api-key
# Immich Configuration
IMMICH_URL=http://192.168.1.5:2283
IMMICH_API_KEY=your-immich-api-key
SEARCH_QUERY=simba cat
DOWNLOAD_DIR=./simba_photos
# OIDC Configuration (Authelia)
OIDC_ISSUER=https://auth.example.com
OIDC_CLIENT_ID=simbarag
OIDC_CLIENT_SECRET=your-client-secret-here
OIDC_REDIRECT_URI=http://localhost:8080/
OIDC_USE_DISCOVERY=true
# Optional: Manual OIDC endpoints (if discovery is disabled)
# OIDC_AUTHORIZATION_ENDPOINT=https://auth.example.com/api/oidc/authorization
# OIDC_TOKEN_ENDPOINT=https://auth.example.com/api/oidc/token
# OIDC_USERINFO_ENDPOINT=https://auth.example.com/api/oidc/userinfo
# OIDC_JWKS_URI=https://auth.example.com/api/oidc/jwks

9
.gitignore vendored
View File

@@ -9,5 +9,12 @@ wheels/
# Virtual environments
.venv
# Environment files
.env
# Database files
chromadb/
chromadb_openai/
chroma_db/
database/
*.db

110
DEV-README.md Normal file
View File

@@ -0,0 +1,110 @@
# Development Environment Setup
This guide explains how to run the application in development mode with hot reload enabled.
## Quick Start
### Development Mode (Hot Reload)
```bash
# Start all services in development mode
docker-compose -f docker-compose.dev.yml up --build
# Or run in detached mode
docker-compose -f docker-compose.dev.yml up -d --build
```
### Production Mode
```bash
# Start production services
docker-compose up --build
```
## What's Different in Dev Mode?
### Backend (Quart/Flask)
- **Hot Reload**: Python code changes are automatically detected and the server restarts
- **Source Mounted**: Your local `services/raggr` directory is mounted as a volume
- **Debug Mode**: Flask runs with `debug=True` for better error messages
- **Environment**: `FLASK_ENV=development` and `PYTHONUNBUFFERED=1` for immediate log output
### Frontend (React + rsbuild)
- **Auto Rebuild**: Frontend automatically rebuilds when files change
- **Watch Mode**: rsbuild runs in watch mode, rebuilding to `dist/` on save
- **Source Mounted**: Your local `services/raggr/raggr-frontend` directory is mounted as a volume
- **Served by Backend**: Built files are served by the backend, no separate dev server
## Ports
- **Application**: 8080 (accessible at `http://localhost:8080` or `http://YOUR_IP:8080`)
The backend serves both the API and the auto-rebuilt frontend, making it accessible from other machines on your network.
## Useful Commands
```bash
# View logs
docker-compose -f docker-compose.dev.yml logs -f
# View logs for specific service
docker-compose -f docker-compose.dev.yml logs -f raggr-backend
docker-compose -f docker-compose.dev.yml logs -f raggr-frontend
# Rebuild after dependency changes
docker-compose -f docker-compose.dev.yml up --build
# Stop all services
docker-compose -f docker-compose.dev.yml down
# Stop and remove volumes (fresh start)
docker-compose -f docker-compose.dev.yml down -v
```
## Making Changes
### Backend Changes
1. Edit any Python file in `services/raggr/`
2. Save the file
3. The Quart server will automatically restart
4. Check logs to confirm reload
### Frontend Changes
1. Edit any file in `services/raggr/raggr-frontend/src/`
2. Save the file
3. The browser will automatically refresh (Hot Module Replacement)
4. No need to rebuild
### Dependency Changes
**Backend** (pyproject.toml):
```bash
# Rebuild the backend service
docker-compose -f docker-compose.dev.yml up --build raggr-backend
```
**Frontend** (package.json):
```bash
# Rebuild the frontend service
docker-compose -f docker-compose.dev.yml up --build raggr-frontend
```
## Troubleshooting
### Port Already in Use
If you see port binding errors, make sure no other services are running on ports 8080 or 3000.
### Changes Not Reflected
1. Check if the file is properly mounted (check docker-compose.dev.yml volumes)
2. Verify the file isn't in an excluded directory (node_modules, __pycache__)
3. Check container logs for errors
### Frontend Not Connecting to Backend
Make sure your frontend API calls point to the correct backend URL. If accessing from the same machine, use `http://localhost:8080`. If accessing from another device on the network, use `http://YOUR_IP:8080`.
## Notes
- Both services bind to `0.0.0.0` and expose ports, making them accessible on your network
- Node modules and Python cache are excluded from volume mounts to use container versions
- Database and ChromaDB data persist in Docker volumes across restarts
- Access the app from any device on your network using your host machine's IP address

13
classifier.py Normal file
View File

@@ -0,0 +1,13 @@
import os
from llm import LLMClient
USE_OPENAI = os.getenv("OLLAMA_URL")
class Classifier:
def __init__(self):
self.llm_client = LLMClient()
def classify_query_by_action(self, query):
_prompt = "Classify the query into one of the following options: "

66
docker-compose.dev.yml Normal file
View File

@@ -0,0 +1,66 @@
services:
postgres:
image: postgres:16-alpine
environment:
- POSTGRES_USER=raggr
- POSTGRES_PASSWORD=raggr_dev_password
- POSTGRES_DB=raggr
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U raggr"]
interval: 5s
timeout: 5s
retries: 5
raggr:
build:
context: ./services/raggr
dockerfile: Dockerfile.dev
image: torrtle/simbarag:dev
ports:
- "8080:8080"
env_file:
- .env
environment:
- PAPERLESS_TOKEN=${PAPERLESS_TOKEN}
- BASE_URL=${BASE_URL}
- OLLAMA_URL=${OLLAMA_URL:-http://localhost:11434}
- CHROMADB_PATH=/app/data/chromadb
- OPENAI_API_KEY=${OPENAI_API_KEY}
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
- OIDC_ISSUER=${OIDC_ISSUER}
- OIDC_CLIENT_ID=${OIDC_CLIENT_ID}
- OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET}
- OIDC_REDIRECT_URI=${OIDC_REDIRECT_URI}
- OIDC_USE_DISCOVERY=${OIDC_USE_DISCOVERY:-true}
- DATABASE_URL=postgres://raggr:raggr_dev_password@postgres:5432/raggr
- FLASK_ENV=development
- PYTHONUNBUFFERED=1
- NODE_ENV=development
depends_on:
postgres:
condition: service_healthy
volumes:
- chromadb_data:/app/data/chromadb
develop:
watch:
# Sync+restart on any file change under services/raggr
- action: sync+restart
path: ./services/raggr
target: /app
ignore:
- __pycache__/
- "*.pyc"
- "*.pyo"
- "*.pyd"
- .git/
- chromadb/
- node_modules/
- raggr-frontend/dist/
volumes:
chromadb_data:
postgres_data:

47
docker-compose.yml Normal file
View File

@@ -0,0 +1,47 @@
version: "3.8"
services:
postgres:
image: postgres:16-alpine
environment:
- POSTGRES_USER=${POSTGRES_USER:-raggr}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
- POSTGRES_DB=${POSTGRES_DB:-raggr}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-raggr}"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
raggr:
build:
context: ./services/raggr
dockerfile: Dockerfile
image: torrtle/simbarag:latest
network_mode: host
environment:
- PAPERLESS_TOKEN=${PAPERLESS_TOKEN}
- BASE_URL=${BASE_URL}
- OLLAMA_URL=${OLLAMA_URL:-http://localhost:11434}
- CHROMADB_PATH=/app/data/chromadb
- OPENAI_API_KEY=${OPENAI_API_KEY}
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
- OIDC_ISSUER=${OIDC_ISSUER}
- OIDC_CLIENT_ID=${OIDC_CLIENT_ID}
- OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET}
- OIDC_REDIRECT_URI=${OIDC_REDIRECT_URI}
- OIDC_USE_DISCOVERY=${OIDC_USE_DISCOVERY:-true}
- DATABASE_URL=${DATABASE_URL:-postgres://raggr:changeme@postgres:5432/raggr}
depends_on:
postgres:
condition: service_healthy
volumes:
- chromadb_data:/app/data/chromadb
restart: unless-stopped
volumes:
chromadb_data:
postgres_data:

102
main.py
View File

@@ -1,102 +0,0 @@
import ollama
import os
from uuid import uuid4, UUID
from request import PaperlessNGXService
from math import ceil
import chromadb
from chromadb.utils.embedding_functions.ollama_embedding_function import (
OllamaEmbeddingFunction,
)
from dotenv import load_dotenv
client = chromadb.EphemeralClient()
collection = client.create_collection(name="docs")
load_dotenv()
class Chunk:
def __init__(
self,
text: str,
size: int,
document_id: UUID,
chunk_id: int,
embedding,
):
self.text = text
self.size = size
self.document_id = document_id
self.chunk_id = chunk_id
self.embedding = embedding
class Chunker:
def __init__(self) -> None:
self.embedding_fx = OllamaEmbeddingFunction(
url=os.getenv("OLLAMA_URL", ""),
model_name="mxbai-embed-large",
)
pass
def chunk_document(self, document: str, chunk_size: int = 300) -> list[Chunk]:
doc_uuid = uuid4()
chunks = []
num_chunks = ceil(len(document) / chunk_size)
document_length = len(document)
for i in range(num_chunks):
curr_pos = i * num_chunks
to_pos = (
curr_pos + num_chunks
if curr_pos + num_chunks < document_length
else document_length
)
text_chunk = document[curr_pos:to_pos]
embedding = self.embedding_fx([text_chunk])
collection.add(
ids=[str(doc_uuid) + ":" + str(i)],
documents=[text_chunk],
embeddings=embedding,
)
return chunks
embedding_fx = OllamaEmbeddingFunction(
url=os.getenv("OLLAMA_URL", ""),
model_name="mxbai-embed-large",
)
# Step 1: Get the text
ppngx = PaperlessNGXService()
docs = ppngx.get_data()
texts = [doc["content"] for doc in docs]
# Step 2: Create chunks
chunker = Chunker()
print(f"chunking {len(texts)} documents")
for text in texts:
chunker.chunk_document(document=text)
# Ask
input = "How many teeth has Simba had removed? Who is his current vet?"
embeddings = embedding_fx(input=[input])
results = collection.query(query_texts=[input], query_embeddings=embeddings)
print(results)
# Generate
output = ollama.generate(
model="gemma3n:e4b",
prompt=f"Using this data: {results}. Respond to this prompt: {input}",
)
print(output["response"])

View File

@@ -1,7 +0,0 @@
[project]
name = "raggr"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = []

View File

@@ -1,24 +0,0 @@
import os
import httpx
from dotenv import load_dotenv
load_dotenv()
class PaperlessNGXService:
def __init__(self):
self.base_url = os.getenv("BASE_URL")
self.token = os.getenv("PAPERLESS_TOKEN")
self.url = f"http://{os.getenv("BASE_URL")}/api/documents/?query=simba"
self.headers = {"Authorization": f"Token {os.getenv("PAPERLESS_TOKEN")}"}
def get_data(self):
print(f"Getting data from: {self.url}")
r = httpx.get(self.url, headers=self.headers)
return r.json()["results"]
if __name__ == "__main__":
pp = PaperlessNGXService()
print(pp.get_data()[0].keys())

View File

@@ -0,0 +1,16 @@
.git
.gitignore
README.md
.env
.DS_Store
chromadb/
chroma_db/
raggr-frontend/node_modules/
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
.venv/
venv/
.pytest_cache/

View File

@@ -0,0 +1 @@
3.13

48
services/raggr/Dockerfile Normal file
View File

@@ -0,0 +1,48 @@
FROM python:3.13-slim
WORKDIR /app
# Install system dependencies, Node.js, Yarn, and uv
RUN apt-get update && apt-get install -y \
build-essential \
curl \
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs \
&& npm install -g yarn \
&& rm -rf /var/lib/apt/lists/* \
&& curl -LsSf https://astral.sh/uv/install.sh | sh
# Add uv to PATH
ENV PATH="/root/.local/bin:$PATH"
# Copy dependency files
COPY pyproject.toml ./
# Install Python dependencies using uv
RUN uv pip install --system -e .
# Copy application code
COPY *.py ./
COPY blueprints ./blueprints
COPY migrations ./migrations
COPY startup.sh ./
RUN chmod +x startup.sh
# Copy frontend code and build
COPY raggr-frontend ./raggr-frontend
WORKDIR /app/raggr-frontend
RUN yarn install && yarn build
WORKDIR /app
# Create ChromaDB and database directories
RUN mkdir -p /app/chromadb /app/database
# Expose port
EXPOSE 8080
# Set environment variables
ENV PYTHONPATH=/app
ENV CHROMADB_PATH=/app/chromadb
# Run the startup script
CMD ["./startup.sh"]

View File

@@ -0,0 +1,53 @@
FROM python:3.13-slim
WORKDIR /app
# Install system dependencies, Node.js, uv, and yarn
RUN apt-get update && apt-get install -y \
build-essential \
curl \
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs \
&& npm install -g yarn \
&& rm -rf /var/lib/apt/lists/* \
&& curl -LsSf https://astral.sh/uv/install.sh | sh
# Add uv to PATH
ENV PATH="/root/.local/bin:$PATH"
# Copy dependency files
COPY pyproject.toml ./
# Install Python dependencies using uv
RUN uv pip install --system -e .
# Copy frontend package files and install dependencies
COPY raggr-frontend/package.json raggr-frontend/yarn.lock* raggr-frontend/
WORKDIR /app/raggr-frontend
RUN yarn install
# Copy application source code
WORKDIR /app
COPY . .
# Build frontend
WORKDIR /app/raggr-frontend
RUN yarn build
# Create ChromaDB and database directories
WORKDIR /app
RUN mkdir -p /app/chromadb /app/database
# Make startup script executable
RUN chmod +x /app/startup-dev.sh
# Set environment variables
ENV PYTHONPATH=/app
ENV CHROMADB_PATH=/app/chromadb
ENV PYTHONUNBUFFERED=1
# Expose port
EXPOSE 8080
# Default command
CMD ["/app/startup-dev.sh"]

View File

@@ -0,0 +1,54 @@
# Database Migrations with Aerich
## Initial Setup (Run Once)
1. Install dependencies:
```bash
uv pip install -e .
```
2. Initialize Aerich:
```bash
aerich init-db
```
This will:
- Create a `migrations/` directory
- Generate the initial migration based on your models
- Create all tables in the database
## When You Add/Change Models
1. Generate a new migration:
```bash
aerich migrate --name "describe_your_changes"
```
Example:
```bash
aerich migrate --name "add_user_profile_model"
```
2. Apply the migration:
```bash
aerich upgrade
```
## Common Commands
- `aerich init-db` - Initialize database (first time only)
- `aerich migrate --name "description"` - Generate new migration
- `aerich upgrade` - Apply pending migrations
- `aerich downgrade` - Rollback last migration
- `aerich history` - Show migration history
- `aerich heads` - Show current migration heads
## Docker Setup
In Docker, migrations run automatically on container startup via the startup script.
## Notes
- Migration files are stored in `migrations/models/`
- Always commit migration files to version control
- Don't modify migration files manually after they're created

View File

@@ -0,0 +1,97 @@
# Vector Store Management
This document describes how to manage the ChromaDB vector store used for RAG (Retrieval-Augmented Generation).
## Configuration
The vector store location is controlled by the `CHROMADB_PATH` environment variable:
- **Development (local)**: Set in `.env` to a local path (e.g., `/path/to/chromadb`)
- **Docker**: Automatically set to `/app/data/chromadb` and persisted via Docker volume
## Management Commands
### CLI (Command Line)
Use the `manage_vectorstore.py` script for vector store operations:
```bash
# Show statistics
python manage_vectorstore.py stats
# Index documents from Paperless-NGX (incremental)
python manage_vectorstore.py index
# Clear and reindex all documents
python manage_vectorstore.py reindex
# List documents
python manage_vectorstore.py list 10
python manage_vectorstore.py list 20 --show-content
```
### Docker
Run commands inside the Docker container:
```bash
# Show statistics
docker compose -f docker-compose.dev.yml exec -T raggr python manage_vectorstore.py stats
# Reindex all documents
docker compose -f docker-compose.dev.yml exec -T raggr python manage_vectorstore.py reindex
```
### API Endpoints
The following authenticated endpoints are available:
- `GET /api/rag/stats` - Get vector store statistics
- `POST /api/rag/index` - Trigger indexing of new documents
- `POST /api/rag/reindex` - Clear and reindex all documents
## How It Works
1. **Document Fetching**: Documents are fetched from Paperless-NGX via the API
2. **Chunking**: Documents are split into chunks of ~1000 characters with 200 character overlap
3. **Embedding**: Chunks are embedded using OpenAI's `text-embedding-3-large` model
4. **Storage**: Embeddings are stored in ChromaDB with metadata (filename, document type, date)
5. **Retrieval**: User queries are embedded and similar chunks are retrieved for RAG
## Troubleshooting
### "Error creating hnsw segment reader"
This indicates a corrupted index. Solution:
```bash
python manage_vectorstore.py reindex
```
### Empty results
Check if documents are indexed:
```bash
python manage_vectorstore.py stats
```
If count is 0, run:
```bash
python manage_vectorstore.py index
```
### Different results in Docker vs local
Docker and local environments use separate ChromaDB instances. To sync:
1. Index inside Docker: `docker compose exec -T raggr python manage_vectorstore.py reindex`
2. Or mount the same volume for both environments
## Production Considerations
1. **Volume Persistence**: Use Docker volumes or persistent storage for ChromaDB
2. **Backup**: Regularly backup the ChromaDB data directory
3. **Reindexing**: Schedule periodic reindexing to keep data fresh
4. **Monitoring**: Monitor the `/api/rag/stats` endpoint for document counts

146
services/raggr/add_user.py Normal file
View File

@@ -0,0 +1,146 @@
# GENERATED BY CLAUDE
import os
import sys
import uuid
import asyncio
from tortoise import Tortoise
from blueprints.users.models import User
from dotenv import load_dotenv
load_dotenv()
# Database configuration with environment variable support
DATABASE_PATH = os.getenv("DATABASE_PATH", "database/raggr.db")
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite://{DATABASE_PATH}")
print(DATABASE_URL)
async def add_user(username: str, email: str, password: str):
"""Add a new user to the database"""
await Tortoise.init(
db_url=DATABASE_URL,
modules={
"models": [
"blueprints.users.models",
"blueprints.conversation.models",
]
},
)
try:
# Check if user already exists
existing_user = await User.filter(email=email).first()
if existing_user:
print(f"Error: User with email '{email}' already exists!")
return False
existing_username = await User.filter(username=username).first()
if existing_username:
print(f"Error: Username '{username}' is already taken!")
return False
# Create new user
user = User(
id=uuid.uuid4(),
username=username,
email=email,
)
user.set_password(password)
await user.save()
print("✓ User created successfully!")
print(f" Username: {username}")
print(f" Email: {email}")
print(f" ID: {user.id}")
return True
except Exception as e:
print(f"Error creating user: {e}")
return False
finally:
await Tortoise.close_connections()
async def list_users():
"""List all users in the database"""
await Tortoise.init(
db_url=DATABASE_URL,
modules={
"models": [
"blueprints.users.models",
"blueprints.conversation.models",
]
},
)
try:
users = await User.all()
if not users:
print("No users found in database.")
return
print(f"\nFound {len(users)} user(s):")
print("-" * 60)
for user in users:
print(f"Username: {user.username}")
print(f"Email: {user.email}")
print(f"ID: {user.id}")
print(f"Created: {user.created_at}")
print("-" * 60)
except Exception as e:
print(f"Error listing users: {e}")
finally:
await Tortoise.close_connections()
def print_usage():
"""Print usage instructions"""
print("Usage:")
print(" python add_user.py add <username> <email> <password>")
print(" python add_user.py list")
print("\nExamples:")
print(" python add_user.py add ryan ryan@example.com mypassword123")
print(" python add_user.py list")
print("\nEnvironment Variables:")
print(" DATABASE_PATH - Path to database file (default: database/raggr.db)")
print(" DATABASE_URL - Full database URL (overrides DATABASE_PATH)")
print("\n Example with custom database:")
print(" DATABASE_PATH=dev.db python add_user.py list")
async def main():
if len(sys.argv) < 2:
print_usage()
sys.exit(1)
command = sys.argv[1].lower()
if command == "add":
if len(sys.argv) != 5:
print("Error: Missing arguments for 'add' command")
print_usage()
sys.exit(1)
username = sys.argv[2]
email = sys.argv[3]
password = sys.argv[4]
success = await add_user(username, email, password)
sys.exit(0 if success else 1)
elif command == "list":
await list_users()
sys.exit(0)
else:
print(f"Error: Unknown command '{command}'")
print_usage()
sys.exit(1)
if __name__ == "__main__":
asyncio.run(main())

View File

@@ -0,0 +1,20 @@
import os
# Database configuration with environment variable support
# Use DATABASE_PATH for relative paths or DATABASE_URL for full connection strings
DATABASE_PATH = os.getenv("DATABASE_PATH", "database/raggr.db")
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite://{DATABASE_PATH}")
TORTOISE_ORM = {
"connections": {"default": DATABASE_URL},
"apps": {
"models": {
"models": [
"blueprints.conversation.models",
"blueprints.users.models",
"aerich.models",
],
"default_connection": "default",
},
},
}

146
services/raggr/app.py Normal file
View File

@@ -0,0 +1,146 @@
import os
from quart import Quart, jsonify, render_template, request, send_from_directory
from quart_jwt_extended import JWTManager, get_jwt_identity, jwt_refresh_token_required
from tortoise.contrib.quart import register_tortoise
import blueprints.conversation
import blueprints.conversation.logic
import blueprints.rag
import blueprints.users
import blueprints.users.models
from main import consult_simba_oracle
app = Quart(
__name__,
static_folder="raggr-frontend/dist/static",
template_folder="raggr-frontend/dist",
)
app.config["JWT_SECRET_KEY"] = os.getenv("JWT_SECRET_KEY", "SECRET_KEY")
jwt = JWTManager(app)
# Register blueprints
app.register_blueprint(blueprints.users.user_blueprint)
app.register_blueprint(blueprints.conversation.conversation_blueprint)
app.register_blueprint(blueprints.rag.rag_blueprint)
# Database configuration with environment variable support
DATABASE_URL = os.getenv(
"DATABASE_URL", "postgres://raggr:raggr_dev_password@localhost:5432/raggr"
)
TORTOISE_CONFIG = {
"connections": {"default": DATABASE_URL},
"apps": {
"models": {
"models": [
"blueprints.conversation.models",
"blueprints.users.models",
"aerich.models",
]
},
},
}
# Initialize Tortoise ORM
register_tortoise(
app,
config=TORTOISE_CONFIG,
generate_schemas=False, # Disabled - using Aerich for migrations
)
# Serve React static files
@app.route("/static/<path:filename>")
async def static_files(filename):
return await send_from_directory(app.static_folder, filename)
# Serve the React app for all routes (catch-all)
@app.route("/", defaults={"path": ""})
@app.route("/<path:path>")
async def serve_react_app(path):
if path and os.path.exists(os.path.join(app.template_folder, path)):
return await send_from_directory(app.template_folder, path)
return await render_template("index.html")
@app.route("/api/query", methods=["POST"])
@jwt_refresh_token_required
async def query():
current_user_uuid = get_jwt_identity()
user = await blueprints.users.models.User.get(id=current_user_uuid)
data = await request.get_json()
query = data.get("query")
conversation_id = data.get("conversation_id")
conversation = await blueprints.conversation.logic.get_conversation_by_id(
conversation_id
)
await conversation.fetch_related("messages")
await blueprints.conversation.logic.add_message_to_conversation(
conversation=conversation,
message=query,
speaker="user",
user=user,
)
transcript = await blueprints.conversation.logic.get_conversation_transcript(
user=user, conversation=conversation
)
response = consult_simba_oracle(input=query, transcript=transcript)
await blueprints.conversation.logic.add_message_to_conversation(
conversation=conversation,
message=response,
speaker="simba",
user=user,
)
return jsonify({"response": response})
@app.route("/api/messages", methods=["GET"])
@jwt_refresh_token_required
async def get_messages():
current_user_uuid = get_jwt_identity()
user = await blueprints.users.models.User.get(id=current_user_uuid)
conversation = await blueprints.conversation.logic.get_conversation_for_user(
user=user
)
# Prefetch related messages
await conversation.fetch_related("messages")
# Manually serialize the conversation with messages
messages = []
for msg in conversation.messages:
messages.append(
{
"id": str(msg.id),
"text": msg.text,
"speaker": msg.speaker.value,
"created_at": msg.created_at.isoformat(),
}
)
name = conversation.name
if len(messages) > 8:
name = await blueprints.conversation.logic.rename_conversation(
user=user,
conversation=conversation,
)
return jsonify(
{
"id": str(conversation.id),
"name": name,
"messages": messages,
"created_at": conversation.created_at.isoformat(),
"updated_at": conversation.updated_at.isoformat(),
}
)
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8080, debug=True)

View File

@@ -0,0 +1 @@
# Blueprints package

View File

@@ -0,0 +1,134 @@
import datetime
from quart import Blueprint, jsonify, request
from quart_jwt_extended import (
get_jwt_identity,
jwt_refresh_token_required,
)
import blueprints.users.models
from .agents import main_agent
from .logic import (
add_message_to_conversation,
get_conversation_by_id,
get_conversation_transcript,
rename_conversation,
)
from .models import (
Conversation,
PydConversation,
PydListConversation,
)
conversation_blueprint = Blueprint(
"conversation_api", __name__, url_prefix="/api/conversation"
)
@conversation_blueprint.post("/query")
@jwt_refresh_token_required
async def query():
current_user_uuid = get_jwt_identity()
user = await blueprints.users.models.User.get(id=current_user_uuid)
data = await request.get_json()
query = data.get("query")
conversation_id = data.get("conversation_id")
conversation = await get_conversation_by_id(conversation_id)
await conversation.fetch_related("messages")
await add_message_to_conversation(
conversation=conversation,
message=query,
speaker="user",
user=user,
)
transcript = await get_conversation_transcript(user=user, conversation=conversation)
transcript_prompt = f"Here is the message transcript thus far {transcript}."
prompt = f"""Answer the user in as if you were a cat named Simba. Don't act too catlike. Be assertive.
{transcript_prompt if len(transcript) > 0 else ""}
Respond to this prompt: {query}"""
payload = {
"messages": [
{
"role": "system",
"content": "You are a helpful cat assistant named Simba that understands veterinary terms. When there are questions to you specifically, they are referring to Simba the cat. Answer the user in as if you were a cat named Simba. Don't act too catlike. Be assertive.\n\nIMPORTANT: When users ask factual questions about Simba's health, medical history, veterinary visits, medications, weight, or any information that would be in documents, you MUST use the simba_search tool to retrieve accurate information before answering. Do not rely on general knowledge - always search the documents for factual questions.",
},
{"role": "user", "content": prompt},
]
}
response = await main_agent.ainvoke(payload)
message = response.get("messages", [])[-1].content
await add_message_to_conversation(
conversation=conversation,
message=message,
speaker="simba",
user=user,
)
return jsonify({"response": message})
@conversation_blueprint.route("/<conversation_id>")
@jwt_refresh_token_required
async def get_conversation(conversation_id: str):
conversation = await Conversation.get(id=conversation_id)
current_user_uuid = get_jwt_identity()
user = await blueprints.users.models.User.get(id=current_user_uuid)
await conversation.fetch_related("messages")
# Manually serialize the conversation with messages
messages = []
for msg in conversation.messages:
messages.append(
{
"id": str(msg.id),
"text": msg.text,
"speaker": msg.speaker.value,
"created_at": msg.created_at.isoformat(),
}
)
name = conversation.name
if len(messages) > 8 and "datetime" in name.lower():
name = await rename_conversation(
user=user,
conversation=conversation,
)
print(name)
return jsonify(
{
"id": str(conversation.id),
"name": name,
"messages": messages,
"created_at": conversation.created_at.isoformat(),
"updated_at": conversation.updated_at.isoformat(),
}
)
@conversation_blueprint.post("/")
@jwt_refresh_token_required
async def create_conversation():
user_uuid = get_jwt_identity()
user = await blueprints.users.models.User.get(id=user_uuid)
conversation = await Conversation.create(
name=f"{user.username} {datetime.datetime.now().timestamp}",
user=user,
)
serialized_conversation = await PydConversation.from_tortoise_orm(conversation)
return jsonify(serialized_conversation.model_dump())
@conversation_blueprint.get("/")
@jwt_refresh_token_required
async def get_all_conversations():
user_uuid = get_jwt_identity()
user = await blueprints.users.models.User.get(id=user_uuid)
conversations = Conversation.filter(user=user)
serialized_conversations = await PydListConversation.from_queryset(conversations)
return jsonify(serialized_conversations.model_dump())

View File

@@ -0,0 +1,36 @@
from langchain.agents import create_agent
from langchain.tools import tool
from langchain_openai import ChatOpenAI
from blueprints.rag.logic import query_vector_store
openai_gpt_5_mini = ChatOpenAI(model="gpt-5-mini")
@tool(response_format="content_and_artifact")
async def simba_search(query: str):
"""Search through Simba's medical records, veterinary documents, and personal information.
Use this tool whenever the user asks questions about:
- Simba's health history, medical records, or veterinary visits
- Medications, treatments, or diagnoses
- Weight, diet, or physical characteristics over time
- Veterinary recommendations or advice
- Ryan's (the owner's) information related to Simba
- Any factual information that would be found in documents
Args:
query: The user's question or information need about Simba
Returns:
Relevant information from Simba's documents
"""
print(f"[SIMBA SEARCH] Tool called with query: {query}")
serialized, docs = await query_vector_store(query=query)
print(f"[SIMBA SEARCH] Found {len(docs)} documents")
print(f"[SIMBA SEARCH] Serialized result length: {len(serialized)}")
print(f"[SIMBA SEARCH] First 200 chars: {serialized[:200]}")
return serialized, docs
main_agent = create_agent(model=openai_gpt_5_mini, tools=[simba_search])

View File

@@ -0,0 +1,80 @@
import tortoise.exceptions
from langchain_openai import ChatOpenAI
import blueprints.users.models
from .models import Conversation, ConversationMessage, RenameConversationOutputSchema
async def create_conversation(name: str = "") -> Conversation:
conversation = await Conversation.create(name=name)
return conversation
async def add_message_to_conversation(
conversation: Conversation,
message: str,
speaker: str,
user: blueprints.users.models.User,
) -> ConversationMessage:
print(conversation, message, speaker)
message = await ConversationMessage.create(
text=message,
speaker=speaker,
conversation=conversation,
)
return message
async def get_the_only_conversation() -> Conversation:
try:
conversation = await Conversation.all().first()
if conversation is None:
conversation = await Conversation.create(name="simba_chat")
except Exception as _e:
conversation = await Conversation.create(name="simba_chat")
return conversation
async def get_conversation_for_user(user: blueprints.users.models.User) -> Conversation:
try:
return await Conversation.get(user=user)
except tortoise.exceptions.DoesNotExist:
await Conversation.get_or_create(name=f"{user.username}'s chat", user=user)
return await Conversation.get(user=user)
async def get_conversation_by_id(id: str) -> Conversation:
return await Conversation.get(id=id)
async def get_conversation_transcript(
user: blueprints.users.models.User, conversation: Conversation
) -> str:
messages = []
for message in conversation.messages:
messages.append(f"{message.speaker} at {message.created_at}: {message.text}")
return "\n".join(messages)
async def rename_conversation(
user: blueprints.users.models.User,
conversation: Conversation,
) -> str:
messages: str = await get_conversation_transcript(
user=user, conversation=conversation
)
llm = ChatOpenAI(model="gpt-4o-mini")
structured_llm = llm.with_structured_output(RenameConversationOutputSchema)
prompt = f"Summarize the following conversation into a sassy one-liner title:\n\n{messages}"
response = structured_llm.invoke(prompt)
new_name: str = response.get("title", "")
conversation.name = new_name
await conversation.save()
return new_name

View File

@@ -0,0 +1,61 @@
import enum
from dataclasses import dataclass
from tortoise import fields
from tortoise.contrib.pydantic import (
pydantic_model_creator,
pydantic_queryset_creator,
)
from tortoise.models import Model
@dataclass
class RenameConversationOutputSchema:
title: str
justification: str
class Speaker(enum.Enum):
USER = "user"
SIMBA = "simba"
class Conversation(Model):
id = fields.UUIDField(primary_key=True)
name = fields.CharField(max_length=255)
created_at = fields.DatetimeField(auto_now_add=True)
updated_at = fields.DatetimeField(auto_now=True)
user: fields.ForeignKeyRelation = fields.ForeignKeyField(
"models.User", related_name="conversations", null=True
)
class Meta:
table = "conversations"
class ConversationMessage(Model):
id = fields.UUIDField(primary_key=True)
text = fields.TextField()
conversation = fields.ForeignKeyField(
"models.Conversation", related_name="messages"
)
created_at = fields.DatetimeField(auto_now_add=True)
speaker = fields.CharEnumField(enum_type=Speaker, max_length=10)
class Meta:
table = "conversation_messages"
PydConversationMessage = pydantic_model_creator(ConversationMessage)
PydConversation = pydantic_model_creator(
Conversation, name="Conversation", allow_cycles=True, exclude=("user",)
)
PydConversationWithMessages = pydantic_model_creator(
Conversation,
name="ConversationWithMessages",
allow_cycles=True,
exclude=("user",),
include=("messages",),
)
PydListConversation = pydantic_queryset_creator(Conversation)
PydListConversationMessage = pydantic_queryset_creator(ConversationMessage)

View File

@@ -0,0 +1,46 @@
from quart import Blueprint, jsonify
from quart_jwt_extended import jwt_refresh_token_required
from .logic import get_vector_store_stats, index_documents, vector_store
rag_blueprint = Blueprint("rag_api", __name__, url_prefix="/api/rag")
@rag_blueprint.get("/stats")
@jwt_refresh_token_required
async def get_stats():
"""Get vector store statistics."""
stats = get_vector_store_stats()
return jsonify(stats)
@rag_blueprint.post("/index")
@jwt_refresh_token_required
async def trigger_index():
"""Trigger indexing of documents from Paperless-NGX."""
try:
await index_documents()
stats = get_vector_store_stats()
return jsonify({"status": "success", "stats": stats})
except Exception as e:
return jsonify({"status": "error", "message": str(e)}), 500
@rag_blueprint.post("/reindex")
@jwt_refresh_token_required
async def trigger_reindex():
"""Clear and reindex all documents."""
try:
# Clear existing documents
collection = vector_store._collection
all_docs = collection.get()
if all_docs["ids"]:
collection.delete(ids=all_docs["ids"])
# Reindex
await index_documents()
stats = get_vector_store_stats()
return jsonify({"status": "success", "stats": stats})
except Exception as e:
return jsonify({"status": "error", "message": str(e)}), 500

View File

@@ -0,0 +1,75 @@
import os
import tempfile
import httpx
class PaperlessNGXService:
def __init__(self):
self.base_url = os.getenv("BASE_URL")
self.token = os.getenv("PAPERLESS_TOKEN")
self.url = f"http://{os.getenv('BASE_URL')}/api/documents/?tags__id=8"
self.headers = {"Authorization": f"Token {os.getenv('PAPERLESS_TOKEN')}"}
def get_data(self):
print(f"Getting data from: {self.url}")
r = httpx.get(self.url, headers=self.headers)
results = r.json()["results"]
nextLink = r.json().get("next")
while nextLink:
r = httpx.get(nextLink, headers=self.headers)
results += r.json()["results"]
nextLink = r.json().get("next")
return results
def get_doc_by_id(self, doc_id: int):
url = f"http://{os.getenv('BASE_URL')}/api/documents/{doc_id}/"
r = httpx.get(url, headers=self.headers)
return r.json()
def download_pdf_from_id(self, id: int) -> str:
download_url = f"http://{os.getenv('BASE_URL')}/api/documents/{id}/download/"
response = httpx.get(
download_url, headers=self.headers, follow_redirects=True, timeout=30
)
response.raise_for_status()
# Use a temporary file for the downloaded PDF
temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=".pdf")
temp_file.write(response.content)
temp_file.close()
temp_pdf_path = temp_file.name
pdf_to_process = temp_pdf_path
return pdf_to_process
def upload_cleaned_content(self, document_id, data):
PUTS_URL = f"http://{os.getenv('BASE_URL')}/api/documents/{document_id}/"
r = httpx.put(PUTS_URL, headers=self.headers, data=data)
r.raise_for_status()
def upload_description(self, description_filepath, file, title, exif_date: str):
POST_URL = f"http://{os.getenv('BASE_URL')}/api/documents/post_document/"
files = {"document": ("description_filepath", file, "application/txt")}
data = {
"title": title,
"create": exif_date,
"document_type": 3,
"tags": [7],
}
r = httpx.post(POST_URL, headers=self.headers, data=data, files=files)
r.raise_for_status()
def get_tags(self):
GET_URL = f"http://{os.getenv('BASE_URL')}/api/tags/"
r = httpx.get(GET_URL, headers=self.headers)
data = r.json()
return {tag["id"]: tag["name"] for tag in data["results"]}
def get_doctypes(self):
GET_URL = f"http://{os.getenv('BASE_URL')}/api/document_types/"
r = httpx.get(GET_URL, headers=self.headers)
data = r.json()
return {doctype["id"]: doctype["name"] for doctype in data["results"]}

View File

@@ -0,0 +1,101 @@
import datetime
import os
from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from .fetchers import PaperlessNGXService
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vector_store = Chroma(
collection_name="simba_docs",
embedding_function=embeddings,
persist_directory=os.getenv("CHROMADB_PATH", ""),
)
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, # chunk size (characters)
chunk_overlap=200, # chunk overlap (characters)
add_start_index=True, # track index in original document
)
def date_to_epoch(date_str: str) -> float:
split_date = date_str.split("-")
date = datetime.datetime(
int(split_date[0]),
int(split_date[1]),
int(split_date[2]),
0,
0,
0,
)
return date.timestamp()
async def fetch_documents_from_paperless_ngx() -> list[Document]:
ppngx = PaperlessNGXService()
data = ppngx.get_data()
doctypes = ppngx.get_doctypes()
documents = []
for doc in data:
metadata = {
"created_date": date_to_epoch(doc["created_date"]),
"filename": doc["original_file_name"],
"document_type": doctypes.get(doc["document_type"], ""),
}
documents.append(Document(page_content=doc["content"], metadata=metadata))
return documents
async def index_documents():
documents = await fetch_documents_from_paperless_ngx()
splits = text_splitter.split_documents(documents)
await vector_store.aadd_documents(documents=splits)
async def query_vector_store(query: str):
retrieved_docs = vector_store.similarity_search(query, k=2)
serialized = "\n\n".join(
(f"Source: {doc.metadata}\nContent: {doc.page_content}")
for doc in retrieved_docs
)
return serialized, retrieved_docs
def get_vector_store_stats():
"""Get statistics about the vector store."""
collection = vector_store._collection
count = collection.count()
return {
"total_documents": count,
"collection_name": collection.name,
}
def list_all_documents(limit: int = 10):
"""List documents in the vector store with their metadata."""
collection = vector_store._collection
results = collection.get(limit=limit, include=["metadatas", "documents"])
documents = []
for i, doc_id in enumerate(results["ids"]):
documents.append(
{
"id": doc_id,
"metadata": results["metadatas"][i]
if results.get("metadatas")
else None,
"content_preview": results["documents"][i][:200]
if results.get("documents")
else None,
}
)
return documents

View File

View File

@@ -0,0 +1,180 @@
from quart import Blueprint, jsonify, request
from quart_jwt_extended import (
create_access_token,
create_refresh_token,
jwt_refresh_token_required,
get_jwt_identity,
)
from .models import User
from .oidc_service import OIDCUserService
from oidc_config import oidc_config
import secrets
import httpx
from urllib.parse import urlencode
import hashlib
import base64
user_blueprint = Blueprint("user_api", __name__, url_prefix="/api/user")
# In-memory storage for OIDC state/PKCE (production: use Redis or database)
# Format: {state: {"pkce_verifier": str, "redirect_after_login": str}}
_oidc_sessions = {}
@user_blueprint.route("/oidc/login", methods=["GET"])
async def oidc_login():
"""
Initiate OIDC login flow
Generates PKCE parameters and redirects to Authelia
"""
if not oidc_config.validate_config():
return jsonify({"error": "OIDC not configured"}), 500
try:
# Generate PKCE parameters
code_verifier = secrets.token_urlsafe(64)
# For PKCE, we need code_challenge = BASE64URL(SHA256(code_verifier))
code_challenge = (
base64.urlsafe_b64encode(hashlib.sha256(code_verifier.encode()).digest())
.decode()
.rstrip("=")
)
# Generate state for CSRF protection
state = secrets.token_urlsafe(32)
# Store PKCE verifier and state for callback validation
_oidc_sessions[state] = {
"pkce_verifier": code_verifier,
"redirect_after_login": request.args.get("redirect", "/"),
}
# Get authorization endpoint from discovery
discovery = await oidc_config.get_discovery_document()
auth_endpoint = discovery.get("authorization_endpoint")
# Build authorization URL
params = {
"client_id": oidc_config.client_id,
"response_type": "code",
"redirect_uri": oidc_config.redirect_uri,
"scope": "openid email profile",
"state": state,
"code_challenge": code_challenge,
"code_challenge_method": "S256",
}
auth_url = f"{auth_endpoint}?{urlencode(params)}"
return jsonify({"auth_url": auth_url})
except Exception as e:
return jsonify({"error": f"OIDC login failed: {str(e)}"}), 500
@user_blueprint.route("/oidc/callback", methods=["GET"])
async def oidc_callback():
"""
Handle OIDC callback from Authelia
Exchanges authorization code for tokens, verifies ID token, and creates/updates user
"""
# Get authorization code and state from callback
code = request.args.get("code")
state = request.args.get("state")
error = request.args.get("error")
if error:
return jsonify({"error": f"OIDC error: {error}"}), 400
if not code or not state:
return jsonify({"error": "Missing code or state"}), 400
# Validate state and retrieve PKCE verifier
session = _oidc_sessions.pop(state, None)
if not session:
return jsonify({"error": "Invalid or expired state"}), 400
pkce_verifier = session["pkce_verifier"]
# Exchange authorization code for tokens
discovery = await oidc_config.get_discovery_document()
token_endpoint = discovery.get("token_endpoint")
token_data = {
"grant_type": "authorization_code",
"code": code,
"redirect_uri": oidc_config.redirect_uri,
"client_id": oidc_config.client_id,
"client_secret": oidc_config.client_secret,
"code_verifier": pkce_verifier,
}
# Use client_secret_post method (credentials in POST body)
async with httpx.AsyncClient() as client:
token_response = await client.post(token_endpoint, data=token_data)
if token_response.status_code != 200:
return jsonify({"error": f"Failed to exchange code for token: {token_response.text}"}), 400
tokens = token_response.json()
id_token = tokens.get("id_token")
if not id_token:
return jsonify({"error": "No ID token received"}), 400
# Verify ID token
try:
claims = await oidc_config.verify_id_token(id_token)
except Exception as e:
return jsonify({"error": f"ID token verification failed: {str(e)}"}), 400
# Get or create user from OIDC claims
user = await OIDCUserService.get_or_create_user_from_oidc(claims)
# Issue backend JWT tokens
access_token = create_access_token(identity=str(user.id))
refresh_token = create_refresh_token(identity=str(user.id))
# Return tokens to frontend
# Frontend will handle storing these and redirecting
return jsonify(
access_token=access_token,
refresh_token=refresh_token,
user={"id": str(user.id), "username": user.username, "email": user.email},
)
@user_blueprint.route("/refresh", methods=["POST"])
@jwt_refresh_token_required
async def refresh():
"""Refresh access token (unchanged from original)"""
user_id = get_jwt_identity()
new_token = create_access_token(identity=user_id)
return jsonify(access_token=new_token)
# Legacy username/password login - kept for backward compatibility during migration
@user_blueprint.route("/login", methods=["POST"])
async def login():
"""
Legacy username/password login
This can be removed after full OIDC migration is complete
"""
data = await request.get_json()
username = data.get("username")
password = data.get("password")
user = await User.filter(username=username).first()
if not user or not user.verify_password(password):
return jsonify({"msg": "Invalid credentials"}), 401
access_token = create_access_token(identity=str(user.id))
refresh_token = create_refresh_token(identity=str(user.id))
return jsonify(
access_token=access_token,
refresh_token=refresh_token,
user={"id": str(user.id), "username": user.username},
)

View File

@@ -0,0 +1,33 @@
from tortoise.models import Model
from tortoise import fields
import bcrypt
class User(Model):
id = fields.UUIDField(primary_key=True)
username = fields.CharField(max_length=255)
password = fields.BinaryField(null=True) # Hashed - nullable for OIDC users
email = fields.CharField(max_length=100, unique=True)
# OIDC fields
oidc_subject = fields.CharField(max_length=255, unique=True, null=True, index=True) # "sub" claim from OIDC
auth_provider = fields.CharField(max_length=50, default="local") # "local" or "oidc"
created_at = fields.DatetimeField(auto_now_add=True)
updated_at = fields.DatetimeField(auto_now=True)
class Meta:
table = "users"
def set_password(self, plain_password: str):
self.password = bcrypt.hashpw(
plain_password.encode("utf-8"),
bcrypt.gensalt(),
)
def verify_password(self, plain_password: str):
if not self.password:
return False
return bcrypt.checkpw(plain_password.encode("utf-8"), self.password)

View File

@@ -0,0 +1,76 @@
"""
OIDC User Management Service
"""
from typing import Dict, Any, Optional
from uuid import uuid4
from .models import User
class OIDCUserService:
"""Service for managing OIDC user authentication and provisioning"""
@staticmethod
async def get_or_create_user_from_oidc(claims: Dict[str, Any]) -> User:
"""
Get existing user by OIDC subject, or create new user from OIDC claims
Args:
claims: Decoded OIDC ID token claims
Returns:
User object (existing or newly created)
"""
oidc_subject = claims.get("sub")
if not oidc_subject:
raise ValueError("No 'sub' claim in ID token")
# Try to find existing user by OIDC subject
user = await User.filter(oidc_subject=oidc_subject).first()
if user:
# Update user info from latest claims (optional)
user.email = claims.get("email", user.email)
user.username = (
claims.get("preferred_username")
or claims.get("name")
or user.username
)
await user.save()
return user
# Check if user exists by email (migration case)
email = claims.get("email")
if email:
user = await User.filter(email=email, auth_provider="local").first()
if user:
# Migrate existing local user to OIDC
user.oidc_subject = oidc_subject
user.auth_provider = "oidc"
user.password = None # Clear password
await user.save()
return user
# Create new user from OIDC claims
username = (
claims.get("preferred_username")
or claims.get("name")
or claims.get("email", "").split("@")[0]
or f"user_{oidc_subject[:8]}"
)
user = await User.create(
id=uuid4(),
username=username,
email=email
or f"{oidc_subject}@oidc.local", # Fallback if no email claim
oidc_subject=oidc_subject,
auth_provider="oidc",
password=None,
)
return user
@staticmethod
async def find_user_by_oidc_subject(oidc_subject: str) -> Optional[User]:
"""Find user by OIDC subject ID"""
return await User.filter(oidc_subject=oidc_subject).first()

142
services/raggr/chunker.py Normal file
View File

@@ -0,0 +1,142 @@
import os
from math import ceil
import re
from typing import Union
from uuid import UUID, uuid4
from ollama import Client
from chromadb.utils.embedding_functions.openai_embedding_function import (
OpenAIEmbeddingFunction,
)
from dotenv import load_dotenv
from llm import LLMClient
load_dotenv()
ollama_client = Client(
host=os.getenv("OLLAMA_HOST", "http://localhost:11434"), timeout=1.0
)
def remove_headers_footers(text, header_patterns=None, footer_patterns=None):
if header_patterns is None:
header_patterns = [r"^.*Header.*$"]
if footer_patterns is None:
footer_patterns = [r"^.*Footer.*$"]
for pattern in header_patterns + footer_patterns:
text = re.sub(pattern, "", text, flags=re.MULTILINE)
return text.strip()
def remove_special_characters(text, special_chars=None):
if special_chars is None:
special_chars = r"[^A-Za-z0-9\s\.,;:\'\"\?\!\-]"
text = re.sub(special_chars, "", text)
return text.strip()
def remove_repeated_substrings(text, pattern=r"\.{2,}"):
text = re.sub(pattern, ".", text)
return text.strip()
def remove_extra_spaces(text):
text = re.sub(r"\n\s*\n", "\n\n", text)
text = re.sub(r"\s+", " ", text)
return text.strip()
def preprocess_text(text):
# Remove headers and footers
text = remove_headers_footers(text)
# Remove special characters
text = remove_special_characters(text)
# Remove repeated substrings like dots
text = remove_repeated_substrings(text)
# Remove extra spaces between lines and within lines
text = remove_extra_spaces(text)
# Additional cleaning steps can be added here
return text.strip()
class Chunk:
def __init__(
self,
text: str,
size: int,
document_id: UUID,
chunk_id: int,
embedding,
):
self.text = text
self.size = size
self.document_id = document_id
self.chunk_id = chunk_id
self.embedding = embedding
class Chunker:
def __init__(self, collection) -> None:
self.collection = collection
self.llm_client = LLMClient()
def embedding_fx(self, inputs):
openai_embedding_fx = OpenAIEmbeddingFunction(
api_key=os.getenv("OPENAI_API_KEY"),
model_name="text-embedding-3-small",
)
return openai_embedding_fx(inputs)
def chunk_document(
self,
document: str,
chunk_size: int = 1000,
metadata: dict[str, Union[str, float]] = {},
) -> list[Chunk]:
doc_uuid = uuid4()
chunk_size = min(chunk_size, len(document)) or 1
chunks = []
num_chunks = ceil(len(document) / chunk_size)
document_length = len(document)
for i in range(num_chunks):
curr_pos = i * num_chunks
to_pos = (
curr_pos + chunk_size
if curr_pos + chunk_size < document_length
else document_length
)
text_chunk = self.clean_document(document[curr_pos:to_pos])
embedding = self.embedding_fx([text_chunk])
self.collection.add(
ids=[str(doc_uuid) + ":" + str(i)],
documents=[text_chunk],
embeddings=embedding,
metadatas=[metadata],
)
return chunks
def clean_document(self, document: str) -> str:
"""This function will remove information that is noise or already known.
Example: We already know all the things in here are Simba-related, so we don't need things like
"Sumamry of simba's visit"
"""
document = document.replace("\\n", "")
document = document.strip()
return preprocess_text(document)

165
services/raggr/cleaner.py Normal file
View File

@@ -0,0 +1,165 @@
import os
import sys
import tempfile
import argparse
from dotenv import load_dotenv
import ollama
from PIL import Image
import fitz
from request import PaperlessNGXService
load_dotenv()
# Configure ollama client with URL from environment or default to localhost
ollama_client = ollama.Client(host=os.getenv("OLLAMA_URL", "http://localhost:11434"))
parser = argparse.ArgumentParser(description="use llm to clean documents")
parser.add_argument("document_id", type=str, help="questions about simba's health")
def pdf_to_image(filepath: str, dpi=300) -> list[str]:
"""Returns the filepaths to the created images"""
image_temp_files = []
try:
pdf_document = fitz.open(filepath)
print(f"\nConverting '{os.path.basename(filepath)}' to temporary images...")
for page_num in range(len(pdf_document)):
page = pdf_document.load_page(page_num)
zoom = dpi / 72
mat = fitz.Matrix(zoom, zoom)
pix = page.get_pixmap(matrix=mat)
# Create a temporary file for the image. delete=False is crucial.
with tempfile.NamedTemporaryFile(
delete=False,
suffix=".png",
prefix=f"pdf_page_{page_num + 1}_",
) as temp_image_file:
temp_image_path = temp_image_file.name
# Save the pixel data to the temporary file
pix.save(temp_image_path)
image_temp_files.append(temp_image_path)
print(
f" -> Saved page {page_num + 1} to temporary file: '{temp_image_path}'"
)
print("\nConversion successful! ✨")
return image_temp_files
except Exception as e:
print(f"An error occurred during PDF conversion: {e}", file=sys.stderr)
# Clean up any image files that were created before the error
for path in image_temp_files:
os.remove(path)
return []
def merge_images_vertically_to_tempfile(image_paths):
"""
Merges a list of images vertically and saves the result to a temporary file.
Args:
image_paths (list): A list of strings, where each string is the
filepath to an image.
Returns:
str: The filepath of the temporary merged image file.
"""
if not image_paths:
print("Error: The list of image paths is empty.")
return None
# Open all images and check for consistency
try:
images = [Image.open(path) for path in image_paths]
except FileNotFoundError as e:
print(f"Error: Could not find image file: {e}")
return None
widths, heights = zip(*(img.size for img in images))
max_width = max(widths)
# All images must have the same width
if not all(width == max_width for width in widths):
print("Warning: Images have different widths. They will be resized.")
resized_images = []
for img in images:
if img.size[0] != max_width:
img = img.resize(
(max_width, int(img.size[1] * (max_width / img.size[0])))
)
resized_images.append(img)
images = resized_images
heights = [img.size[1] for img in images]
# Calculate the total height of the merged image
total_height = sum(heights)
# Create a new blank image with the combined dimensions
merged_image = Image.new("RGB", (max_width, total_height))
# Paste each image onto the new blank image
y_offset = 0
for img in images:
merged_image.paste(img, (0, y_offset))
y_offset += img.height
# Create a temporary file and save the image
temp_file = tempfile.NamedTemporaryFile(suffix=".png", delete=False)
temp_path = temp_file.name
merged_image.save(temp_path)
temp_file.close()
print(f"Successfully merged {len(images)} images into temporary file: {temp_path}")
return temp_path
OCR_PROMPT = """
You job is to extract text from the images I provide you. Extract every bit of the text in the image. Don't say anything just do your job. Text should be same as in the images. If there are multiple images, categorize the transcriptions by page.
Things to avoid:
- Don't miss anything to extract from the images
Things to include:
- Include everything, even anything inside [], (), {} or anything.
- Include any repetitive things like "..." or anything
- If you think there is any mistake in image just include it too
Someone will kill the innocent kittens if you don't extract the text exactly. So, make sure you extract every bit of the text. Only output the extracted text.
"""
def summarize_pdf_image(filepaths: list[str]):
res = ollama_client.chat(
model="gemma3:4b",
messages=[
{
"role": "user",
"content": OCR_PROMPT,
"images": filepaths,
}
],
)
return res["message"]["content"]
if __name__ == "__main__":
args = parser.parse_args()
ppngx = PaperlessNGXService()
if args.document_id:
doc_id = args.document_id
file = ppngx.get_doc_by_id(doc_id=doc_id)
pdf_path = ppngx.download_pdf_from_id(doc_id)
print(pdf_path)
image_paths = pdf_to_image(filepath=pdf_path)
summary = summarize_pdf_image(filepaths=image_paths)
print(summary)
file["content"] = summary
print(file)
ppngx.upload_cleaned_content(doc_id, file)

View File

@@ -0,0 +1,83 @@
from ollama import Client
import argparse
import os
import logging
from PIL import Image, ExifTags
from pillow_heif import register_heif_opener
from pydantic import BaseModel
from dotenv import load_dotenv
load_dotenv()
register_heif_opener()
logging.basicConfig(level=logging.INFO)
parser = argparse.ArgumentParser(
prog="SimbaImageProcessor",
description="What the program does",
epilog="Text at the bottom of help",
)
parser.add_argument("filepath")
client = Client(host=os.getenv("OLLAMA_HOST", "http://localhost:11434"))
class SimbaImageDescription(BaseModel):
image_date: str
description: str
def describe_simba_image(input):
logging.info("Opening image of Simba ...")
if "heic" in input.lower() or "heif" in input.lower():
new_filepath = input.split(".")[0] + ".jpg"
img = Image.open(input)
img.save(new_filepath, "JPEG")
logging.info("Extracting EXIF...")
exif = {
ExifTags.TAGS[k]: v for k, v in img.getexif().items() if k in ExifTags.TAGS
}
img = Image.open(new_filepath)
input = new_filepath
else:
img = Image.open(input)
logging.info("Extracting EXIF...")
exif = {
ExifTags.TAGS[k]: v for k, v in img.getexif().items() if k in ExifTags.TAGS
}
if "MakerNote" in exif:
exif.pop("MakerNote")
logging.info(exif)
prompt = f"Simba is an orange cat belonging to Ryan Chen. In 2025, they lived in New York. In 2024, they lived in California. Analyze the following image and tell me what Simba seems to be doing. Be extremely descriptive about Simba, things in the background, and the setting of the image. I will also include the EXIF data of the image, please use it to help you determine information about Simba. EXIF: {exif}. Put the notes in the description field and the date in the image_date field."
logging.info("Sending info to Ollama ...")
response = client.chat(
model="gemma3:4b",
messages=[
{
"role": "system",
"content": "you are a very shrewd and descriptive note taker. all of your responses will be formatted like notes in bullet points. be very descriptive. do not leave a single thing out.",
},
{"role": "user", "content": prompt, "images": [input]},
],
format=SimbaImageDescription.model_json_schema(),
)
result = SimbaImageDescription.model_validate_json(response["message"]["content"])
return result
if __name__ == "__main__":
args = parser.parse_args()
if args.filepath:
logging.info
describe_simba_image(input=args.filepath)

View File

@@ -0,0 +1,113 @@
import logging
import os
import sqlite3
import httpx
from dotenv import load_dotenv
from image_process import describe_simba_image
from request import PaperlessNGXService
logging.basicConfig(level=logging.INFO)
load_dotenv()
# Configuration from environment variables
IMMICH_URL = os.getenv("IMMICH_URL", "http://localhost:2283")
API_KEY = os.getenv("IMMICH_API_KEY")
PERSON_NAME = os.getenv("PERSON_NAME", "Simba") # Name of the tagged person/pet
DOWNLOAD_DIR = os.getenv("DOWNLOAD_DIR", "./simba_photos")
# Set up headers
headers = {"x-api-key": API_KEY, "Content-Type": "application/json"}
VISITED = {}
if __name__ == "__main__":
conn = sqlite3.connect("./database/visited.db")
c = conn.cursor()
c.execute("select immich_id from visited")
rows = c.fetchall()
for row in rows:
VISITED.add(row[0])
ppngx = PaperlessNGXService()
people_url = f"{IMMICH_URL}/api/search/person?name=Simba"
people = httpx.get(people_url, headers=headers).json()
simba_id = people[0]["id"]
ids = {}
asset_search = f"{IMMICH_URL}/api/search/smart"
request_body = {"query": "orange cat"}
results = httpx.post(asset_search, headers=headers, json=request_body)
assets = results.json()["assets"]
for asset in assets["items"]:
if asset["type"] == "IMAGE" and asset["id"] not in VISITED:
ids[asset["id"]] = asset.get("originalFileName")
nextPage = assets.get("nextPage")
# while nextPage != None:
# logging.info(f"next page: {nextPage}")
# request_body["page"] = nextPage
# results = httpx.post(asset_search, headers=headers, json=request_body)
# assets = results.json()["assets"]
# for asset in assets["items"]:
# if asset["type"] == "IMAGE":
# ids.add(asset['id'])
# nextPage = assets.get("nextPage")
asset_search = f"{IMMICH_URL}/api/search/smart"
request_body = {"query": "simba"}
results = httpx.post(asset_search, headers=headers, json=request_body)
for asset in results.json()["assets"]["items"]:
if asset["type"] == "IMAGE":
ids[asset["id"]] = asset.get("originalFileName")
for immich_asset_id, immich_filename in ids.items():
try:
response = httpx.get(
f"{IMMICH_URL}/api/assets/{immich_asset_id}/original", headers=headers
)
path = os.path.join("/Users/ryanchen/Programs/raggr", immich_filename)
file = open(path, "wb+")
for chunk in response.iter_bytes(chunk_size=8192):
file.write(chunk)
logging.info("Processing image ...")
description = describe_simba_image(path)
image_description = description.description
image_date = description.image_date
description_filepath = os.path.join(
"/Users/ryanchen/Programs/raggr", "SIMBA_DESCRIBE_001.txt"
)
file = open(description_filepath, "w+")
file.write(image_description)
file.close()
file = open(description_filepath, "rb")
ppngx.upload_description(
description_filepath=description_filepath,
file=file,
title="SIMBA_DESCRIBE_001.txt",
exif_date=image_date,
)
file.close()
c.execute("INSERT INTO visited (immich_id) values (?)", (immich_asset_id,))
conn.commit()
logging.info("Processing complete. Deleting file.")
os.remove(file.name)
except Exception as e:
logging.info(f"something went wrong for {immich_filename}")
logging.info(e)
conn.close()

View File

@@ -0,0 +1,92 @@
#!/usr/bin/env python3
"""CLI tool to inspect the vector store contents."""
import argparse
import os
from dotenv import load_dotenv
from blueprints.rag.logic import (
get_vector_store_stats,
index_documents,
list_all_documents,
)
# Load .env from the root directory
root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "../.."))
env_path = os.path.join(root_dir, ".env")
load_dotenv(env_path)
def print_stats():
"""Print vector store statistics."""
stats = get_vector_store_stats()
print("=== Vector Store Statistics ===")
print(f"Collection Name: {stats['collection_name']}")
print(f"Total Documents: {stats['total_documents']}")
print()
def print_documents(limit: int = 10, show_content: bool = False):
"""Print documents in the vector store."""
docs = list_all_documents(limit=limit)
print(f"=== Documents (showing {len(docs)} of {limit} requested) ===\n")
for i, doc in enumerate(docs, 1):
print(f"Document {i}:")
print(f" ID: {doc['id']}")
print(f" Metadata: {doc['metadata']}")
if show_content:
print(f" Content Preview: {doc['content_preview']}")
print()
async def run_index():
"""Run the indexing process."""
print("Starting indexing process...")
await index_documents()
print("Indexing complete!")
print_stats()
def main():
import asyncio
parser = argparse.ArgumentParser(description="Inspect the vector store contents")
parser.add_argument(
"--stats", action="store_true", help="Show vector store statistics"
)
parser.add_argument(
"--list", type=int, metavar="N", help="List N documents from the vector store"
)
parser.add_argument(
"--show-content",
action="store_true",
help="Show content preview when listing documents",
)
parser.add_argument(
"--index",
action="store_true",
help="Index documents from Paperless-NGX into the vector store",
)
args = parser.parse_args()
# Handle indexing first if requested
if args.index:
asyncio.run(run_index())
return
# If no arguments provided, show stats by default
if not any([args.stats, args.list]):
args.stats = True
if args.stats:
print_stats()
if args.list:
print_documents(limit=args.list, show_content=args.show_content)
if __name__ == "__main__":
main()

73
services/raggr/llm.py Normal file
View File

@@ -0,0 +1,73 @@
import os
from ollama import Client
from openai import OpenAI
import logging
from dotenv import load_dotenv
load_dotenv()
logging.basicConfig(level=logging.INFO)
TRY_OLLAMA = os.getenv("TRY_OLLAMA", False)
class LLMClient:
def __init__(self):
try:
self.ollama_client = Client(
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=1.0
)
self.ollama_client.chat(
model="gemma3:4b", messages=[{"role": "system", "content": "test"}]
)
self.PROVIDER = "ollama"
logging.info("Using Ollama as LLM backend")
except Exception as e:
print(e)
self.openai_client = OpenAI()
self.PROVIDER = "openai"
logging.info("Using OpenAI as LLM backend")
def chat(
self,
prompt: str,
system_prompt: str,
):
# Instituting a fallback if my gaming PC is not on
if self.PROVIDER == "ollama":
try:
response = self.ollama_client.chat(
model="gemma3:4b",
messages=[
{
"role": "system",
"content": system_prompt,
},
{"role": "user", "content": prompt},
],
)
output = response.message.content
return output
except Exception as e:
logging.error(f"Could not connect to OLLAMA: {str(e)}")
response = self.openai_client.responses.create(
model="gpt-4o-mini",
input=[
{
"role": "system",
"content": system_prompt,
},
{"role": "user", "content": prompt},
],
)
output = response.output_text
return output
if __name__ == "__main__":
client = Client()
client.chat(model="gemma3:4b", messages=[{"role": "system", "promp": "hack"}])

280
services/raggr/main.py Normal file
View File

@@ -0,0 +1,280 @@
import argparse
import datetime
import logging
import os
import sqlite3
import time
import ollama
from dotenv import load_dotenv
import chromadb
from chunker import Chunker
from cleaner import pdf_to_image, summarize_pdf_image
from llm import LLMClient
from query import QueryGenerator
from request import PaperlessNGXService
_dotenv_loaded = load_dotenv()
# Configure ollama client with URL from environment or default to localhost
ollama_client = ollama.Client(
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=10.0
)
client = chromadb.PersistentClient(path=os.getenv("CHROMADB_PATH", ""))
simba_docs = client.get_or_create_collection(name="simba_docs2")
feline_vet_lookup = client.get_or_create_collection(name="feline_vet_lookup")
parser = argparse.ArgumentParser(
description="An LLM tool to query information about Simba <3"
)
parser.add_argument("query", type=str, help="questions about simba's health")
parser.add_argument(
"--reindex", action="store_true", help="re-index the simba documents"
)
parser.add_argument("--classify", action="store_true", help="test classification")
parser.add_argument("--index", help="index a file")
ppngx = PaperlessNGXService()
llm_client = LLMClient()
def index_using_pdf_llm(doctypes):
logging.info("reindex data...")
files = ppngx.get_data()
for file in files:
document_id: int = file["id"]
pdf_path = ppngx.download_pdf_from_id(id=document_id)
image_paths = pdf_to_image(filepath=pdf_path)
logging.info(f"summarizing {file}")
generated_summary = summarize_pdf_image(filepaths=image_paths)
file["content"] = generated_summary
chunk_data(files, simba_docs, doctypes=doctypes)
def date_to_epoch(date_str: str) -> float:
split_date = date_str.split("-")
date = datetime.datetime(
int(split_date[0]),
int(split_date[1]),
int(split_date[2]),
0,
0,
0,
)
return date.timestamp()
def chunk_data(docs, collection, doctypes):
# Step 2: Create chunks
chunker = Chunker(collection)
logging.info(f"chunking {len(docs)} documents")
texts: list[str] = [doc["content"] for doc in docs]
with sqlite3.connect("database/visited.db") as conn:
to_insert = []
c = conn.cursor()
for index, text in enumerate(texts):
metadata = {
"created_date": date_to_epoch(docs[index]["created_date"]),
"filename": docs[index]["original_file_name"],
"document_type": doctypes.get(docs[index]["document_type"], ""),
}
if doctypes:
metadata["type"] = doctypes.get(docs[index]["document_type"])
chunker.chunk_document(
document=text,
metadata=metadata,
)
to_insert.append((docs[index]["id"],))
c.executemany(
"INSERT INTO indexed_documents (paperless_id) values (?)", to_insert
)
conn.commit()
def chunk_text(texts: list[str], collection):
chunker = Chunker(collection)
for index, text in enumerate(texts):
metadata = {}
chunker.chunk_document(
document=text,
metadata=metadata,
)
def classify_query(query: str, transcript: str) -> bool:
logging.info("Starting query generation")
qg_start = time.time()
qg = QueryGenerator()
query_type = qg.get_query_type(input=query, transcript=transcript)
logging.info(query_type)
qg_end = time.time()
logging.info(f"Query generation took {qg_end - qg_start:.2f} seconds")
return query_type == "Simba"
def consult_oracle(
input: str,
collection,
transcript: str = "",
):
chunker = Chunker(collection)
start_time = time.time()
# Ask
logging.info("Starting query generation")
qg_start = time.time()
qg = QueryGenerator()
doctype_query = qg.get_doctype_query(input=input)
# metadata_filter = qg.get_query(input)
metadata_filter = {**doctype_query}
logging.info(metadata_filter)
qg_end = time.time()
logging.info(f"Query generation took {qg_end - qg_start:.2f} seconds")
logging.info("Starting embedding generation")
embedding_start = time.time()
embeddings = chunker.embedding_fx(inputs=[input])
embedding_end = time.time()
logging.info(
f"Embedding generation took {embedding_end - embedding_start:.2f} seconds"
)
logging.info("Starting collection query")
query_start = time.time()
results = collection.query(
query_texts=[input],
query_embeddings=embeddings,
where=metadata_filter,
)
query_end = time.time()
logging.info(f"Collection query took {query_end - query_start:.2f} seconds")
# Generate
logging.info("Starting LLM generation")
llm_start = time.time()
system_prompt = "You are a helpful assistant that understands veterinary terms."
transcript_prompt = f"Here is the message transcript thus far {transcript}."
prompt = f"""Using the following data, help answer the user's query by providing as many details as possible.
Using this data: {results}. {transcript_prompt if len(transcript) > 0 else ""}
Respond to this prompt: {input}"""
output = llm_client.chat(prompt=prompt, system_prompt=system_prompt)
llm_end = time.time()
logging.info(f"LLM generation took {llm_end - llm_start:.2f} seconds")
total_time = time.time() - start_time
logging.info(f"Total consult_oracle execution took {total_time:.2f} seconds")
return output
def llm_chat(input: str, transcript: str = "") -> str:
system_prompt = "You are a helpful assistant that understands veterinary terms."
transcript_prompt = f"Here is the message transcript thus far {transcript}."
prompt = f"""Answer the user in as if you were a cat named Simba. Don't act too catlike. Be assertive.
{transcript_prompt if len(transcript) > 0 else ""}
Respond to this prompt: {input}"""
output = llm_client.chat(prompt=prompt, system_prompt=system_prompt)
return output
def paperless_workflow(input):
# Step 1: Get the text
ppngx = PaperlessNGXService()
docs = ppngx.get_data()
chunk_data(docs, collection=simba_docs)
consult_oracle(input, simba_docs)
def consult_simba_oracle(input: str, transcript: str = ""):
is_simba_related = classify_query(query=input, transcript=transcript)
if is_simba_related:
logging.info("Query is related to simba")
return consult_oracle(
input=input,
collection=simba_docs,
transcript=transcript,
)
logging.info("Query is NOT related to simba")
return llm_chat(input=input, transcript=transcript)
def filter_indexed_files(docs):
with sqlite3.connect("database/visited.db") as conn:
c = conn.cursor()
c.execute(
"CREATE TABLE IF NOT EXISTS indexed_documents (id INTEGER PRIMARY KEY AUTOINCREMENT, paperless_id INTEGER)"
)
c.execute("SELECT paperless_id FROM indexed_documents")
rows = c.fetchall()
conn.commit()
visited = {row[0] for row in rows}
return [doc for doc in docs if doc["id"] not in visited]
def reindex():
with sqlite3.connect("database/visited.db") as conn:
c = conn.cursor()
c.execute("DELETE FROM indexed_documents")
conn.commit()
# Delete all documents from the collection
all_docs = simba_docs.get()
if all_docs["ids"]:
simba_docs.delete(ids=all_docs["ids"])
logging.info("Fetching documents from Paperless-NGX")
ppngx = PaperlessNGXService()
docs = ppngx.get_data()
docs = filter_indexed_files(docs)
logging.info(f"Fetched {len(docs)} documents")
# Delete all chromadb data
ids = simba_docs.get(ids=None, limit=None, offset=0)
all_ids = ids["ids"]
if len(all_ids) > 0:
simba_docs.delete(ids=all_ids)
# Chunk documents
logging.info("Chunking documents now ...")
doctype_lookup = ppngx.get_doctypes()
chunk_data(docs, collection=simba_docs, doctypes=doctype_lookup)
logging.info("Done chunking documents")
if __name__ == "__main__":
args = parser.parse_args()
if args.reindex:
reindex()
if args.classify:
consult_simba_oracle(input="yohohoho testing")
consult_simba_oracle(input="write an email")
consult_simba_oracle(input="how much does simba weigh")
if args.query:
logging.info("Consulting oracle ...")
print(
consult_oracle(
input=args.query,
collection=simba_docs,
)
)
else:
logging.info("please provide a query")

View File

@@ -0,0 +1,121 @@
#!/usr/bin/env python3
"""Management script for vector store operations."""
import argparse
import asyncio
import sys
from blueprints.rag.logic import (
get_vector_store_stats,
index_documents,
list_all_documents,
vector_store,
)
def stats():
"""Show vector store statistics."""
stats = get_vector_store_stats()
print("=== Vector Store Statistics ===")
print(f"Collection: {stats['collection_name']}")
print(f"Total Documents: {stats['total_documents']}")
async def index():
"""Index documents from Paperless-NGX."""
print("Starting indexing process...")
print("Fetching documents from Paperless-NGX...")
await index_documents()
print("✓ Indexing complete!")
stats()
async def reindex():
"""Clear and reindex all documents."""
print("Clearing existing documents...")
collection = vector_store._collection
all_docs = collection.get()
if all_docs["ids"]:
print(f"Deleting {len(all_docs['ids'])} existing documents...")
collection.delete(ids=all_docs["ids"])
print("✓ Cleared")
else:
print("Collection is already empty")
await index()
def list_docs(limit: int = 10, show_content: bool = False):
"""List documents in the vector store."""
docs = list_all_documents(limit=limit)
print(f"\n=== Documents (showing {len(docs)}) ===\n")
for i, doc in enumerate(docs, 1):
print(f"Document {i}:")
print(f" ID: {doc['id']}")
print(f" Metadata: {doc['metadata']}")
if show_content:
print(f" Content: {doc['content_preview']}")
print()
def main():
parser = argparse.ArgumentParser(
description="Manage vector store for RAG system",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
%(prog)s stats # Show vector store statistics
%(prog)s index # Index new documents from Paperless-NGX
%(prog)s reindex # Clear and reindex all documents
%(prog)s list 10 # List first 10 documents
%(prog)s list 20 --show-content # List 20 documents with content preview
""",
)
subparsers = parser.add_subparsers(dest="command", help="Command to execute")
# Stats command
subparsers.add_parser("stats", help="Show vector store statistics")
# Index command
subparsers.add_parser("index", help="Index documents from Paperless-NGX")
# Reindex command
subparsers.add_parser("reindex", help="Clear and reindex all documents")
# List command
list_parser = subparsers.add_parser("list", help="List documents in vector store")
list_parser.add_argument(
"limit", type=int, default=10, nargs="?", help="Number of documents to list"
)
list_parser.add_argument(
"--show-content", action="store_true", help="Show content preview"
)
args = parser.parse_args()
if not args.command:
parser.print_help()
sys.exit(1)
try:
if args.command == "stats":
stats()
elif args.command == "index":
asyncio.run(index())
elif args.command == "reindex":
asyncio.run(reindex())
elif args.command == "list":
list_docs(limit=args.limit, show_content=args.show_content)
except KeyboardInterrupt:
print("\n\nOperation cancelled by user")
sys.exit(1)
except Exception as e:
print(f"\n❌ Error: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,71 @@
from tortoise import BaseDBAsyncClient
RUN_IN_TRANSACTION = True
async def upgrade(db: BaseDBAsyncClient) -> str:
return """
CREATE TABLE IF NOT EXISTS "users" (
"id" UUID NOT NULL PRIMARY KEY,
"username" VARCHAR(255) NOT NULL,
"password" BYTEA,
"email" VARCHAR(100) NOT NULL UNIQUE,
"oidc_subject" VARCHAR(255) UNIQUE,
"auth_provider" VARCHAR(50) NOT NULL DEFAULT 'local',
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS "idx_users_oidc_su_5aec5a" ON "users" ("oidc_subject");
CREATE TABLE IF NOT EXISTS "conversations" (
"id" UUID NOT NULL PRIMARY KEY,
"name" VARCHAR(255) NOT NULL,
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
"user_id" UUID REFERENCES "users" ("id") ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS "conversation_messages" (
"id" UUID NOT NULL PRIMARY KEY,
"text" TEXT NOT NULL,
"created_at" TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
"speaker" VARCHAR(10) NOT NULL,
"conversation_id" UUID NOT NULL REFERENCES "conversations" ("id") ON DELETE CASCADE
);
COMMENT ON COLUMN "conversation_messages"."speaker" IS 'USER: user\nSIMBA: simba';
CREATE TABLE IF NOT EXISTS "aerich" (
"id" SERIAL NOT NULL PRIMARY KEY,
"version" VARCHAR(255) NOT NULL,
"app" VARCHAR(100) NOT NULL,
"content" JSONB NOT NULL
);"""
async def downgrade(db: BaseDBAsyncClient) -> str:
return """
"""
MODELS_STATE = (
"eJztmmtP4zgUhv9KlE+MxCLoUGaEViulpex0Z9qO2nR3LjuK3MRtvSROJnYGKsR/X9u5J0"
"56AUqL+gXosU9sPz7OeY/Lveq4FrTJSdvFv6BPAEUuVi+VexUDB7I/pO3Higo8L23lBgom"
"tnAwMz1FC5gQ6gOTssYpsAlkJgsS00deNBgObJsbXZN1RHiWmgKMfgbQoO4M0jn0WcP3H8"
"yMsAXvIIk/ejfGFEHbys0bWXxsYTfowhO28bh7dS168uEmhunagYPT3t6Czl2cdA8CZJ1w"
"H942gxj6gEIrsww+y2jZsSmcMTNQP4DJVK3UYMEpCGwOQ/19GmCTM1DESPzH+R/qGngYao"
"4WYcpZ3D+Eq0rXLKwqH6r9QRsevb14I1bpEjrzRaMgoj4IR0BB6Cq4piDF7xLK9hz4cpRx"
"/wJMNtFNMMaGlGMaQzHIGNBm1FQH3Bk2xDM6Zx8bzWYNxr+1oSDJegmULovrMOr7UVMjbO"
"NIU4SmD/mSDUDLIK9YC0UOlMPMexaQWpHrSfzHjgJma7AG2F5Eh6CGr97tdUa61vvMV+IQ"
"8tMWiDS9w1sawrooWI8uCluRPET5p6t/UPhH5dug3ynGftJP/6byOYGAugZ2bw1gZc5rbI"
"3B5DY28KwNNzbvedjYF93YaPKZfSXQN9bLIBmXR6SRaG5b3MTNkwZPvdMbac7gMMrwrl0f"
"ohn+CBcCYZfNA2BTliwi0TGOHrOr0FJrOgsf3CZqJBsUbHVsTZCG2VMbtbWrjioYToB5cw"
"t8y6iA6UBCwAySMtBW5Hn9cQjtRJrJWWYFXC984m6+VarYClZuw80wytErNzkNp2gBmK3b"
"isbmI9XQWaKCMxBXE8NGdiMPonivRTGFd5KUrzOrHGXcf19EcV0q73zRc1k8lr5HPe3Lm1"
"wm/zTo/xl3z0jl9qdB66CQX6OQKitk4kFwIxMDvIDs4MApSYHc7mbcX/joqONRZ3ip8Iz+"
"Lx51ey3tUiHImQB1tS3OVZlnpysUmWenlTUmbyocoGyiWe81L3F9ynf+nkpYs3Dh9UgpW7"
"w/21mKSzWtJFzW1bbPqeREzSCRbnEtUa3V+NE+aLP912Z8H9e9tMz67ItG28LFpQcIuXV9"
"SWS2EAb+Qg4z61WAOVnQsP7Z1ZJeBq/F9WpWbjFkrW5fG36VS964fzZuW1/1jlagCx2A7H"
"WiNHF4mhBdfuKfMkDPTlcTPXWqpyR7XGSZBgkm/0FTUjlUkyz6bQS0GKTb5fksB55p+bnh"
"+e4vZFWJdjnQkuP23qKq7ZrAfkQaynNtrhKmzeoobZa1+aG4fZ3F7eHrn1exscntcqlIWX"
"Y1X/pfh6e5n99lgbTde3kN+sicq5J6Lmo5rqvoQNpnZ0q6Lq64IpZWdBxzIRiinX9RYSe+"
"HfmtcXb+7vz924vz96yLmElieVfzMuj29SUVHD8I0muXav2RcTnUb6mcY0djHREXdt9PgM"
"9SX7ARKcSS9P7XaNCvvE6NXQogx5gt8LuFTHqs2IjQH7uJtYYiX3X9dz/Fr3kKuZk/oCW7"
"eN3mZeHD/9BpOYI="
)

View File

@@ -0,0 +1,113 @@
"""
OIDC Configuration for Authelia Integration
"""
import os
from typing import Dict, Any
from authlib.jose import jwt
from authlib.jose.errors import JoseError
import httpx
class OIDCConfig:
"""OIDC Configuration Manager"""
def __init__(self):
# Load from environment variables
self.issuer = os.getenv("OIDC_ISSUER") # e.g., https://auth.example.com
self.client_id = os.getenv("OIDC_CLIENT_ID")
self.client_secret = os.getenv("OIDC_CLIENT_SECRET")
self.redirect_uri = os.getenv(
"OIDC_REDIRECT_URI", "http://localhost:8080/api/user/oidc/callback"
)
# OIDC endpoints (can use discovery or manual config)
self.use_discovery = os.getenv("OIDC_USE_DISCOVERY", "true").lower() == "true"
# Manual endpoint configuration (fallback if discovery fails)
self.authorization_endpoint = os.getenv("OIDC_AUTHORIZATION_ENDPOINT")
self.token_endpoint = os.getenv("OIDC_TOKEN_ENDPOINT")
self.userinfo_endpoint = os.getenv("OIDC_USERINFO_ENDPOINT")
self.jwks_uri = os.getenv("OIDC_JWKS_URI")
# Cached discovery document and JWKS
self._discovery_doc: Dict[str, Any] | None = None
self._jwks: Dict[str, Any] | None = None
def validate_config(self) -> bool:
"""Validate that required configuration is present"""
if not self.issuer or not self.client_id or not self.client_secret:
return False
return True
async def get_discovery_document(self) -> Dict[str, Any]:
"""Fetch OIDC discovery document from .well-known endpoint"""
if self._discovery_doc:
return self._discovery_doc
if not self.use_discovery:
# Return manual configuration
return {
"issuer": self.issuer,
"authorization_endpoint": self.authorization_endpoint,
"token_endpoint": self.token_endpoint,
"userinfo_endpoint": self.userinfo_endpoint,
"jwks_uri": self.jwks_uri,
}
discovery_url = f"{self.issuer.rstrip('/')}/.well-known/openid-configuration"
async with httpx.AsyncClient() as client:
response = await client.get(discovery_url)
response.raise_for_status()
self._discovery_doc = response.json()
return self._discovery_doc
async def get_jwks(self) -> Dict[str, Any]:
"""Fetch JSON Web Key Set for token verification"""
if self._jwks:
return self._jwks
discovery = await self.get_discovery_document()
jwks_uri = discovery.get("jwks_uri")
if not jwks_uri:
raise ValueError("No jwks_uri found in discovery document")
async with httpx.AsyncClient() as client:
response = await client.get(jwks_uri)
response.raise_for_status()
self._jwks = response.json()
return self._jwks
async def verify_id_token(self, id_token: str) -> Dict[str, Any]:
"""
Verify and decode ID token from OIDC provider
Returns the decoded claims if valid
Raises exception if invalid
"""
jwks = await self.get_jwks()
try:
# Verify token signature and claims
claims = jwt.decode(
id_token,
jwks,
claims_options={
"iss": {"essential": True, "value": self.issuer},
"aud": {"essential": True, "value": self.client_id},
"exp": {"essential": True},
},
)
# Additional validation
claims.validate()
return claims
except JoseError as e:
raise ValueError(f"Invalid ID token: {str(e)}")
# Global instance
oidc_config = OIDCConfig()

View File

@@ -0,0 +1,24 @@
from bs4 import BeautifulSoup
import chromadb
import httpx
client = chromadb.PersistentClient(path="/Users/ryanchen/Programs/raggr/chromadb")
# Scrape
BASE_URL = "https://www.vet.cornell.edu"
LIST_URL = "/departments-centers-and-institutes/cornell-feline-health-center/health-information/feline-health-topics"
QUERY_URL = BASE_URL + LIST_URL
r = httpx.get(QUERY_URL)
soup = BeautifulSoup(r.text)
container = soup.find("div", class_="field-body")
a_s = container.find_all("a", href=True)
new_texts = []
for link in a_s:
endpoint = link["href"]
query_url = BASE_URL + endpoint
r2 = httpx.get(query_url)
article_soup = BeautifulSoup(r2.text)

View File

@@ -0,0 +1,42 @@
[project]
name = "raggr"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"chromadb>=1.1.0",
"python-dotenv>=1.0.0",
"flask>=3.1.2",
"httpx>=0.28.1",
"ollama>=0.6.0",
"openai>=2.0.1",
"pydantic>=2.11.9",
"pillow>=10.0.0",
"pymupdf>=1.24.0",
"black>=25.9.0",
"pillow-heif>=1.1.1",
"flask-jwt-extended>=4.7.1",
"bcrypt>=5.0.0",
"pony>=0.7.19",
"flask-login>=0.6.3",
"quart>=0.20.0",
"tortoise-orm>=0.25.1",
"quart-jwt-extended>=0.1.0",
"pre-commit>=4.3.0",
"tortoise-orm-stubs>=1.0.2",
"aerich>=0.8.0",
"tomlkit>=0.13.3",
"authlib>=1.3.0",
"asyncpg>=0.30.0",
"langchain-openai>=1.1.6",
"langchain>=1.2.0",
"langchain-chroma>=1.0.0",
"langchain-community>=0.4.1",
"jq>=1.10.0",
]
[tool.aerich]
tortoise_orm = "app.TORTOISE_CONFIG"
location = "./migrations"
src_folder = "./."

251
services/raggr/query.py Normal file
View File

@@ -0,0 +1,251 @@
import json
import os
from typing import Literal
import datetime
from ollama import Client
from openai import OpenAI
from pydantic import BaseModel, Field
# Configure ollama client with URL from environment or default to localhost
ollama_client = Client(
host=os.getenv("OLLAMA_URL", "http://localhost:11434"), timeout=10.0
)
# This uses inferred filters — which means using LLM to create the metadata filters
class FilterOperation(BaseModel):
op: Literal["$gt", "$gte", "$eq", "$ne", "$lt", "$lte", "$in", "$nin"]
value: str | list[str]
class FilterQuery(BaseModel):
field_name: Literal["created_date, tags"]
op: FilterOperation
class AndQuery(BaseModel):
op: Literal["$and", "$or"]
subqueries: list[FilterQuery]
class GeneratedQuery(BaseModel):
fields: list[str]
extracted_metadata_fields: str
class Time(BaseModel):
time: int
DOCTYPE_OPTIONS = [
"Bill",
"Image Description",
"Insurance",
"Medical Record",
"Documentation",
"Letter",
]
QUERY_TYPE_OPTIONS = [
"Simba",
"Other",
]
class DocumentType(BaseModel):
type: list[str] = Field(description="type of document", enum=DOCTYPE_OPTIONS)
class QueryType(BaseModel):
type: str = Field(desciption="type of query", enum=QUERY_TYPE_OPTIONS)
PROMPT = """
You are an information specialist that processes user queries. The current year is 2025. The user queries are all about
a cat, Simba, and its records. The types of records are listed below. Using the query, extract the
the date range the user is trying to query. You should return it as a JSON. The date tag is created_date. Return the date in epoch time.
If the created_date cannot be ascertained, set it to epoch time start.
You have several operators at your disposal:
- $gt: greater than
- $gte: greater than or equal
- $eq: equal
- $ne: not equal
- $lt: less than
- $lte: less than or equal to
- $in: in
- $nin: not in
Logical operators:
- $and, $or
### Example 1
Query: "Who is Simba's current vet?"
Metadata fields: "{"created_date"}"
Extracted metadata fields: {"created_date: {"$gt": "2025-01-01"}}
### Example 2
Query: "How many teeth has Simba had removed?"
Metadata fields: {}
Extracted metadata fields: {}
### Example 3
Query: "How many times has Simba been to the vet this year?"
Metadata fields: {"created_date"}
Extracted metadata fields: {"created_date": {"gt": "2025-01-01"}}
document_types:
- aftercare
- bill
- insurance claim
- medical records
Only return the extracted metadata fields. Make sure the extracted metadata fields are valid JSON
"""
DOCTYPE_PROMPT = f"""You are an information specialist that processes user queries. A query can have two tags attached from the following options. Based on the query, determine which of the following options is most appropriate: {",".join(DOCTYPE_OPTIONS)}
### Example 1
Query: "Who is Simba's current vet?"
Tags: ["Bill", "Medical Record"]
### Example 2
Query: "Who does Simba know?"
Tags: ["Letter", "Documentation"]
"""
QUERY_TYPE_PROMPT = f"""You are an information specialist that processes user queries.
A query can have one tag attached from the following options. Based on the query and the transcript which is listed below, determine
which of the following options is most appropriate: {",".join(QUERY_TYPE_OPTIONS)}
### Example 1
Query: "Who is Simba's current vet?"
Tags: ["Simba"]
### Example 2
Query: "What is the capital of Tokyo?"
Tags: ["Other"]
### Example 3
Query: "Can you help me write an email?"
Tags: ["Other"]
TRANSCRIPT:
"""
class QueryGenerator:
def __init__(self) -> None:
pass
def date_to_epoch(self, date_str: str) -> float:
split_date = date_str.split("-")
date = datetime.datetime(
int(split_date[0]),
int(split_date[1]),
int(split_date[2]),
0,
0,
0,
)
return date.timestamp()
def get_doctype_query(self, input: str):
client = OpenAI()
response = client.chat.completions.create(
messages=[
{
"role": "system",
"content": "You are an information specialist that is really good at deciding what tags a query should have",
},
{"role": "user", "content": DOCTYPE_PROMPT + " " + input},
],
model="gpt-4o",
response_format={
"type": "json_schema",
"json_schema": {
"name": "document_type",
"schema": DocumentType.model_json_schema(),
},
},
)
response_json_str = response.choices[0].message.content
type_data = json.loads(response_json_str)
metadata_query = {"document_type": {"$in": type_data["type"]}}
return metadata_query
def get_query_type(self, input: str, transcript: str):
client = OpenAI()
response = client.chat.completions.create(
messages=[
{
"role": "system",
"content": "You are an information specialist that is really good at deciding what tags a query should have",
},
{
"role": "user",
"content": f"{QUERY_TYPE_PROMPT}\nTRANSCRIPT:\n{transcript}\nQUERY:{input}",
},
],
model="gpt-4o",
response_format={
"type": "json_schema",
"json_schema": {
"name": "query_type",
"schema": QueryType.model_json_schema(),
},
},
)
response_json_str = response.choices[0].message.content
type_data = json.loads(response_json_str)
return type_data["type"]
def get_query(self, input: str):
client = OpenAI()
response = client.responses.parse(
model="gpt-4o",
input=[
{"role": "system", "content": PROMPT},
{"role": "user", "content": input},
],
text_format=GeneratedQuery,
)
print(response.output)
query = json.loads(response.output_parsed.extracted_metadata_fields)
# response: ChatResponse = ollama_client.chat(
# model="gemma3n:e4b",
# messages=[
# {"role": "system", "content": PROMPT},
# {"role": "user", "content": input},
# ],
# format=GeneratedQuery.model_json_schema(),
# )
# query = json.loads(
# json.loads(response["message"]["content"])["extracted_metadata_fields"]
# )
# date_key = list(query["created_date"].keys())[0]
# query["created_date"][date_key] = self.date_to_epoch(
# query["created_date"][date_key]
# )
# if "$" not in date_key:
# query["created_date"]["$" + date_key] = query["created_date"][date_key]
return query
if __name__ == "__main__":
qg = QueryGenerator()
print(qg.get_doctype_query("How heavy is Simba?"))

View File

@@ -0,0 +1,9 @@
.git
.gitignore
README.md
.DS_Store
node_modules
dist
.cache
coverage
*.log

View File

@@ -0,0 +1,17 @@
# Local
.DS_Store
*.local
*.log*
# Dist
node_modules
dist/
.yarn
# Profile
.rspack-profile-*/
# IDE
.vscode/*
!.vscode/extensions.json
.idea

View File

@@ -0,0 +1 @@
nodeLinker: node-modules

View File

@@ -0,0 +1,18 @@
FROM node:20-slim
WORKDIR /app
# Copy package files
COPY package.json yarn.lock* ./
# Install dependencies
RUN yarn install
# Copy application source code
COPY . .
# Expose rsbuild dev server port (default 3000)
EXPOSE 3000
# Default command
CMD ["sh", "-c", "yarn build && yarn watch:build"]

View File

@@ -0,0 +1,36 @@
# Rsbuild project
## Setup
Install the dependencies:
```bash
pnpm install
```
## Get started
Start the dev server, and the app will be available at [http://localhost:3000](http://localhost:3000).
```bash
pnpm dev
```
Build the app for production:
```bash
pnpm build
```
Preview the production build locally:
```bash
pnpm preview
```
## Learn more
To learn more about Rsbuild, check out the following resources:
- [Rsbuild documentation](https://rsbuild.rs) - explore Rsbuild features and APIs.
- [Rsbuild GitHub repository](https://github.com/web-infra-dev/rsbuild) - your feedback and contributions are welcome!

View File

@@ -0,0 +1,63 @@
# Token Refresh Implementation
## Overview
The API services now automatically handle token refresh when access tokens expire. This provides a seamless user experience without requiring manual re-authentication.
## How It Works
### 1. **userService.ts**
The `userService` now includes:
- **`refreshToken()`**: Automatically gets the refresh token from localStorage, calls the `/api/user/refresh` endpoint, and updates the access token
- **`fetchWithAuth()`**: A wrapper around `fetch()` that:
1. Automatically adds the Authorization header with the access token
2. Detects 401 (Unauthorized) responses
3. Automatically refreshes the token using the refresh token
4. Retries the original request with the new access token
5. Throws an error if refresh fails (e.g., refresh token expired)
### 2. **conversationService.ts**
Now uses `userService.fetchWithAuth()` for all API calls:
- `sendQuery()` - No longer needs token parameter
- `getMessages()` - No longer needs token parameter
### 3. **Components Updated**
**ChatScreen.tsx**:
- Removed manual token handling
- Now simply calls `conversationService.sendQuery(query)` and `conversationService.getMessages()`
## Benefits
**Automatic token refresh** - Users stay logged in longer
**Transparent retry logic** - Failed requests due to expired tokens are automatically retried
**Cleaner code** - Components don't need to manage tokens
**Better UX** - No interruptions when access token expires
**Centralized auth logic** - All auth handling in one place
## Error Handling
- If refresh token is missing or invalid, the error is thrown
- Components can catch these errors and redirect to login
- LocalStorage is automatically cleared when refresh fails
## Usage Example
```typescript
// Old way (manual token management)
const token = localStorage.getItem("access_token");
const result = await conversationService.sendQuery(query, token);
// New way (automatic token refresh)
const result = await conversationService.sendQuery(query);
```
## Token Storage
- **Access Token**: `localStorage.getItem("access_token")`
- **Refresh Token**: `localStorage.getItem("refresh_token")`
Both are automatically managed by the services.

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,42 @@
{
"name": "raggr-frontend",
"version": "1.0.0",
"private": true,
"type": "module",
"scripts": {
"build": "rsbuild build",
"dev": "rsbuild dev --open",
"preview": "rsbuild preview",
"watch": "npm-watch build",
"watch:build": "rsbuild build --watch"
},
"dependencies": {
"axios": "^1.12.2",
"marked": "^16.3.0",
"npm-watch": "^0.13.0",
"react": "^19.1.1",
"react-dom": "^19.1.1",
"react-markdown": "^10.1.0",
"watch": "^1.0.2"
},
"devDependencies": {
"@biomejs/biome": "2.3.10",
"@rsbuild/core": "^1.5.6",
"@rsbuild/plugin-react": "^1.4.0",
"@tailwindcss/postcss": "^4.0.0",
"@types/react": "^19.1.13",
"@types/react-dom": "^19.1.9",
"typescript": "^5.9.2"
},
"watch": {
"build": {
"patterns": [
"src"
],
"extensions": "ts,tsx,css,js,jsx",
"delay": 1000,
"quiet": false,
"inherit": true
}
}
}

View File

@@ -0,0 +1,5 @@
export default {
plugins: {
"@tailwindcss/postcss": {},
},
};

View File

@@ -0,0 +1,10 @@
import { defineConfig } from '@rsbuild/core';
import { pluginReact } from '@rsbuild/plugin-react';
export default defineConfig({
plugins: [pluginReact()],
html: {
title: 'Raggr',
favicon: './src/assets/favicon.svg',
},
});

View File

@@ -0,0 +1,7 @@
@import "tailwindcss";
body {
margin: 0;
font-family: Inter, Avenir, Helvetica, Arial, sans-serif;
background-color: #F9F5EB;
}

View File

@@ -0,0 +1,72 @@
import { useState, useEffect } from "react";
import "./App.css";
import { AuthProvider } from "./contexts/AuthContext";
import { ChatScreen } from "./components/ChatScreen";
import { LoginScreen } from "./components/LoginScreen";
import { conversationService } from "./api/conversationService";
const AppContainer = () => {
const [isAuthenticated, setAuthenticated] = useState<boolean>(false);
const [isChecking, setIsChecking] = useState<boolean>(true);
useEffect(() => {
const checkAuth = async () => {
const accessToken = localStorage.getItem("access_token");
const refreshToken = localStorage.getItem("refresh_token");
// No tokens at all, not authenticated
if (!accessToken && !refreshToken) {
setIsChecking(false);
setAuthenticated(false);
return;
}
// Try to verify token by making a request
try {
await conversationService.getAllConversations();
// If successful, user is authenticated
setAuthenticated(true);
} catch (error) {
// Token is invalid or expired
console.error("Authentication check failed:", error);
localStorage.removeItem("access_token");
localStorage.removeItem("refresh_token");
setAuthenticated(false);
} finally {
setIsChecking(false);
}
};
checkAuth();
}, []);
// Show loading state while checking authentication
if (isChecking) {
return (
<div className="h-screen flex items-center justify-center bg-white/85">
<div className="text-xl">Loading...</div>
</div>
);
}
return (
<>
{isAuthenticated ? (
<ChatScreen setAuthenticated={setAuthenticated} />
) : (
<LoginScreen setAuthenticated={setAuthenticated} />
)}
</>
);
};
const App = () => {
return (
<AuthProvider>
<AppContainer />
</AuthProvider>
);
};
export default App;

View File

@@ -0,0 +1,115 @@
import { userService } from "./userService";
interface Message {
id: string;
text: string;
speaker: "user" | "simba";
created_at: string;
}
interface Conversation {
id: string;
name: string;
messages?: Message[];
created_at: string;
updated_at: string;
user_id?: string;
}
interface QueryRequest {
query: string;
}
interface QueryResponse {
response: string;
}
interface CreateConversationRequest {
user_id: string;
}
class ConversationService {
private baseUrl = "/api";
private conversationBaseUrl = "/api/conversation";
async sendQuery(
query: string,
conversation_id: string,
): Promise<QueryResponse> {
const response = await userService.fetchWithRefreshToken(
`${this.conversationBaseUrl}/query`,
{
method: "POST",
body: JSON.stringify({ query, conversation_id }),
},
);
if (!response.ok) {
throw new Error("Failed to send query");
}
return await response.json();
}
async getMessages(): Promise<Conversation> {
const response = await userService.fetchWithRefreshToken(
`${this.baseUrl}/messages`,
{
method: "GET",
},
);
if (!response.ok) {
throw new Error("Failed to fetch messages");
}
return await response.json();
}
async getConversation(conversationId: string): Promise<Conversation> {
const response = await userService.fetchWithRefreshToken(
`${this.conversationBaseUrl}/${conversationId}`,
{
method: "GET",
},
);
if (!response.ok) {
throw new Error("Failed to fetch conversation");
}
return await response.json();
}
async createConversation(): Promise<Conversation> {
const response = await userService.fetchWithRefreshToken(
`${this.conversationBaseUrl}/`,
{
method: "POST",
},
);
if (!response.ok) {
throw new Error("Failed to create conversation");
}
return await response.json();
}
async getAllConversations(): Promise<Conversation[]> {
const response = await userService.fetchWithRefreshToken(
`${this.conversationBaseUrl}/`,
{
method: "GET",
},
);
if (!response.ok) {
throw new Error("Failed to fetch conversations");
}
return await response.json();
}
}
export const conversationService = new ConversationService();

View File

@@ -0,0 +1,94 @@
/**
* OIDC Authentication Service
* Handles OAuth 2.0 Authorization Code flow with PKCE
*/
interface OIDCLoginResponse {
auth_url: string;
}
interface OIDCCallbackResponse {
access_token: string;
refresh_token: string;
user: {
id: string;
username: string;
email: string;
};
}
class OIDCService {
private baseUrl = "/api/user/oidc";
/**
* Initiate OIDC login flow
* Returns authorization URL to redirect user to
*/
async initiateLogin(redirectAfterLogin: string = "/"): Promise<string> {
const response = await fetch(
`${this.baseUrl}/login?redirect=${encodeURIComponent(redirectAfterLogin)}`,
{
method: "GET",
headers: { "Content-Type": "application/json" },
}
);
if (!response.ok) {
throw new Error("Failed to initiate OIDC login");
}
const data: OIDCLoginResponse = await response.json();
return data.auth_url;
}
/**
* Handle OIDC callback
* Exchanges authorization code for tokens
*/
async handleCallback(
code: string,
state: string
): Promise<OIDCCallbackResponse> {
const response = await fetch(
`${this.baseUrl}/callback?code=${encodeURIComponent(code)}&state=${encodeURIComponent(state)}`,
{
method: "GET",
headers: { "Content-Type": "application/json" },
}
);
if (!response.ok) {
throw new Error("OIDC callback failed");
}
return await response.json();
}
/**
* Extract OIDC callback parameters from URL
*/
getCallbackParamsFromURL(): { code: string; state: string } | null {
const params = new URLSearchParams(window.location.search);
const code = params.get("code");
const state = params.get("state");
if (code && state) {
return { code, state };
}
return null;
}
/**
* Clear callback parameters from URL without reload
*/
clearCallbackParams(): void {
const url = new URL(window.location.href);
url.searchParams.delete("code");
url.searchParams.delete("state");
url.searchParams.delete("error");
window.history.replaceState({}, "", url.toString());
}
}
export const oidcService = new OIDCService();

View File

@@ -0,0 +1,139 @@
interface LoginResponse {
access_token: string;
refresh_token: string;
user: {
id: string;
username: string;
email?: string;
};
}
interface RefreshResponse {
access_token: string;
}
class UserService {
private baseUrl = "/api/user";
async login(username: string, password: string): Promise<LoginResponse> {
const response = await fetch(`${this.baseUrl}/login`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ username, password }),
});
if (!response.ok) {
throw new Error("Invalid credentials");
}
return await response.json();
}
async refreshToken(): Promise<string> {
const refreshToken = localStorage.getItem("refresh_token");
if (!refreshToken) {
throw new Error("No refresh token available");
}
const response = await fetch(`${this.baseUrl}/refresh`, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${refreshToken}`,
},
});
if (!response.ok) {
// Refresh token is invalid or expired, clear storage
localStorage.removeItem("access_token");
localStorage.removeItem("refresh_token");
throw new Error("Failed to refresh token");
}
const data: RefreshResponse = await response.json();
localStorage.setItem("access_token", data.access_token);
return data.access_token;
}
async validateToken(): Promise<boolean> {
const refreshToken = localStorage.getItem("refresh_token");
if (!refreshToken) {
return false;
}
try {
await this.refreshToken();
return true;
} catch (error) {
return false;
}
}
async fetchWithAuth(
url: string,
options: RequestInit = {},
): Promise<Response> {
const accessToken = localStorage.getItem("access_token");
// Add authorization header
const headers = {
"Content-Type": "application/json",
...(options.headers || {}),
...(accessToken && { Authorization: `Bearer ${accessToken}` }),
};
let response = await fetch(url, { ...options, headers });
// If unauthorized, try refreshing the token
if (response.status === 401) {
try {
const newAccessToken = await this.refreshToken();
// Retry the request with new token
headers.Authorization = `Bearer ${newAccessToken}`;
response = await fetch(url, { ...options, headers });
} catch (error) {
// Refresh failed, redirect to login or throw error
throw new Error("Session expired. Please log in again.");
}
}
return response;
}
async fetchWithRefreshToken(
url: string,
options: RequestInit = {},
): Promise<Response> {
const refreshToken = localStorage.getItem("refresh_token");
// Add authorization header
const headers = {
"Content-Type": "application/json",
...(options.headers || {}),
...(refreshToken && { Authorization: `Bearer ${refreshToken}` }),
};
let response = await fetch(url, { ...options, headers });
// If unauthorized, try refreshing the token
if (response.status === 401) {
try {
const newAccessToken = await this.refreshToken();
// Retry the request with new token
headers.Authorization = `Bearer ${newAccessToken}`;
response = await fetch(url, { ...options, headers });
} catch (error) {
// Refresh failed, redirect to login or throw error
throw new Error("Session expired. Please log in again.");
}
}
return response;
}
}
export const userService = new UserService();

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.8 KiB

View File

@@ -0,0 +1,3 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
<text y="80" font-size="80" font-family="system-ui, -apple-system, sans-serif">🐱</text>
</svg>

After

Width:  |  Height:  |  Size: 163 B

View File

@@ -0,0 +1,31 @@
import ReactMarkdown from "react-markdown";
type AnswerBubbleProps = {
text: string;
loading?: boolean;
};
export const AnswerBubble = ({ text, loading }: AnswerBubbleProps) => {
return (
<div className="rounded-md bg-orange-100 p-3 sm:p-4 w-2/3">
{loading ? (
<div className="flex flex-col w-full animate-pulse gap-2">
<div className="flex flex-row gap-2 w-full">
<div className="bg-gray-400 w-1/2 p-3 rounded-lg" />
<div className="bg-gray-400 w-1/2 p-3 rounded-lg" />
</div>
<div className="flex flex-row gap-2 w-full">
<div className="bg-gray-400 w-1/3 p-3 rounded-lg" />
<div className="bg-gray-400 w-2/3 p-3 rounded-lg" />
</div>
</div>
) : (
<div className=" flex flex-col break-words overflow-wrap-anywhere text-sm sm:text-base [&>*]:break-words">
<ReactMarkdown>
{"🐈: " + text}
</ReactMarkdown>
</div>
)}
</div>
);
};

View File

@@ -0,0 +1,310 @@
import { useEffect, useState, useRef } from "react";
import { conversationService } from "../api/conversationService";
import { QuestionBubble } from "./QuestionBubble";
import { AnswerBubble } from "./AnswerBubble";
import { MessageInput } from "./MessageInput";
import { ConversationList } from "./ConversationList";
import catIcon from "../assets/cat.png";
type Message = {
text: string;
speaker: "simba" | "user";
};
type QuestionAnswer = {
question: string;
answer: string;
};
type Conversation = {
title: string;
id: string;
};
type ChatScreenProps = {
setAuthenticated: (isAuth: boolean) => void;
};
export const ChatScreen = ({ setAuthenticated }: ChatScreenProps) => {
const [query, setQuery] = useState<string>("");
const [answer, setAnswer] = useState<string>("");
const [simbaMode, setSimbaMode] = useState<boolean>(false);
const [questionsAnswers, setQuestionsAnswers] = useState<QuestionAnswer[]>(
[],
);
const [messages, setMessages] = useState<Message[]>([]);
const [conversations, setConversations] = useState<Conversation[]>([
{ title: "simba meow meow", id: "uuid" },
]);
const [showConversations, setShowConversations] = useState<boolean>(false);
const [selectedConversation, setSelectedConversation] =
useState<Conversation | null>(null);
const [sidebarCollapsed, setSidebarCollapsed] = useState<boolean>(false);
const [isLoading, setIsLoading] = useState<boolean>(false);
const messagesEndRef = useRef<HTMLDivElement>(null);
const simbaAnswers = ["meow.", "hiss...", "purrrrrr", "yowOWROWWowowr"];
const scrollToBottom = () => {
messagesEndRef.current?.scrollIntoView({ behavior: "smooth" });
};
const handleSelectConversation = (conversation: Conversation) => {
setShowConversations(false);
setSelectedConversation(conversation);
const loadMessages = async () => {
try {
const fetchedConversation = await conversationService.getConversation(
conversation.id,
);
setMessages(
fetchedConversation.messages.map((message) => ({
text: message.text,
speaker: message.speaker,
})),
);
} catch (error) {
console.error("Failed to load messages:", error);
}
};
loadMessages();
};
const loadConversations = async () => {
try {
const fetchedConversations =
await conversationService.getAllConversations();
const parsedConversations = fetchedConversations.map((conversation) => ({
id: conversation.id,
title: conversation.name,
}));
setConversations(parsedConversations);
setSelectedConversation(parsedConversations[0]);
console.log(parsedConversations);
console.log("JELLYFISH@");
} catch (error) {
console.error("Failed to load messages:", error);
}
};
const handleCreateNewConversation = async () => {
const newConversation = await conversationService.createConversation();
await loadConversations();
setSelectedConversation({
title: newConversation.name,
id: newConversation.id,
});
};
useEffect(() => {
loadConversations();
}, []);
useEffect(() => {
scrollToBottom();
}, [messages]);
useEffect(() => {
const loadMessages = async () => {
console.log(selectedConversation);
console.log("JELLYFISH");
if (selectedConversation == null) return;
try {
const conversation = await conversationService.getConversation(
selectedConversation.id,
);
// Update the conversation title in case it changed
setSelectedConversation({
id: conversation.id,
title: conversation.name,
});
setMessages(
conversation.messages.map((message) => ({
text: message.text,
speaker: message.speaker,
})),
);
} catch (error) {
console.error("Failed to load messages:", error);
}
};
loadMessages();
}, [selectedConversation?.id]);
const handleQuestionSubmit = async () => {
if (!query.trim() || isLoading) return; // Don't submit empty messages or while loading
const currMessages = messages.concat([{ text: query, speaker: "user" }]);
setMessages(currMessages);
setQuery(""); // Clear input immediately after submission
setIsLoading(true);
if (simbaMode) {
console.log("simba mode activated");
const randomIndex = Math.floor(Math.random() * simbaAnswers.length);
const randomElement = simbaAnswers[randomIndex];
setAnswer(randomElement);
setQuestionsAnswers(
questionsAnswers.concat([
{
question: query,
answer: randomElement,
},
]),
);
setIsLoading(false);
return;
}
try {
const result = await conversationService.sendQuery(
query,
selectedConversation.id,
);
setQuestionsAnswers(
questionsAnswers.concat([{ question: query, answer: result.response }]),
);
setMessages(
currMessages.concat([{ text: result.response, speaker: "simba" }]),
);
} catch (error) {
console.error("Failed to send query:", error);
// If session expired, redirect to login
if (error instanceof Error && error.message.includes("Session expired")) {
setAuthenticated(false);
}
} finally {
setIsLoading(false);
}
};
const handleQueryChange = (event: React.ChangeEvent<HTMLTextAreaElement>) => {
setQuery(event.target.value);
};
const handleKeyDown = (event: React.KeyboardEvent<HTMLTextAreaElement>) => {
// Submit on Enter, but allow Shift+Enter for new line
if (event.key === "Enter" && !event.shiftKey) {
event.preventDefault();
handleQuestionSubmit();
}
};
return (
<div className="h-screen flex flex-row bg-[#F9F5EB]">
{/* Sidebar - Expanded */}
<aside
className={`hidden md:flex md:flex-col bg-[#F9F5EB] border-r border-gray-200 p-4 overflow-y-auto transition-all duration-300 ${sidebarCollapsed ? "w-20" : "w-64"}`}
>
{!sidebarCollapsed ? (
<div className="bg-[#F9F5EB]">
<div className="flex flex-row items-center gap-2 mb-6">
<img
src={catIcon}
alt="Simba"
className="cursor-pointer hover:opacity-80"
onClick={() => setSidebarCollapsed(true)}
/>
<h2 className="text-3xl bg-[#F9F5EB] font-semibold">asksimba!</h2>
</div>
<ConversationList
conversations={conversations}
onCreateNewConversation={handleCreateNewConversation}
onSelectConversation={handleSelectConversation}
/>
<div className="mt-auto pt-4">
<button
className="w-full p-2 border border-red-400 bg-red-200 hover:bg-red-400 cursor-pointer rounded-md text-sm"
onClick={() => setAuthenticated(false)}
>
logout
</button>
</div>
</div>
) : (
<div className="flex flex-col items-center gap-4">
<img
src={catIcon}
alt="Simba"
className="cursor-pointer hover:opacity-80"
onClick={() => setSidebarCollapsed(false)}
/>
</div>
)}
</aside>
{/* Main chat area */}
<div className="flex-1 flex flex-col h-screen overflow-hidden">
{/* Mobile header */}
<header className="md:hidden flex flex-row justify-between items-center gap-3 p-4 border-b border-gray-200 bg-white">
<div className="flex flex-row items-center gap-2">
<img src={catIcon} alt="Simba" className="w-10 h-10" />
<h1 className="text-xl">asksimba!</h1>
</div>
<div className="flex flex-row gap-2">
<button
className="p-2 border border-green-400 bg-green-200 hover:bg-green-400 cursor-pointer rounded-md text-sm"
onClick={() => setShowConversations(!showConversations)}
>
{showConversations ? "hide" : "show"}
</button>
<button
className="p-2 border border-red-400 bg-red-200 hover:bg-red-400 cursor-pointer rounded-md text-sm"
onClick={() => setAuthenticated(false)}
>
logout
</button>
</div>
</header>
{/* Messages area */}
{selectedConversation && (
<div className="sticky top-0 mx-auto w-full">
<div className="bg-[#F9F5EB] text-black px-6 w-full py-3">
<h2 className="text-lg font-semibold">
{selectedConversation.title || "Untitled Conversation"}
</h2>
</div>
</div>
)}
<div className="flex-1 overflow-y-auto relative px-4 py-6">
{/* Floating conversation name */}
<div className="max-w-2xl mx-auto flex flex-col gap-4">
{showConversations && (
<div className="md:hidden">
<ConversationList
conversations={conversations}
onCreateNewConversation={handleCreateNewConversation}
onSelectConversation={handleSelectConversation}
/>
</div>
)}
{messages.map((msg, index) => {
if (msg.speaker === "simba") {
return <AnswerBubble key={index} text={msg.text} />;
}
return <QuestionBubble key={index} text={msg.text} />;
})}
{isLoading && <AnswerBubble text="" loading={true} />}
<div ref={messagesEndRef} />
</div>
</div>
{/* Input area */}
<footer className="p-4 bg-[#F9F5EB]">
<div className="max-w-2xl mx-auto">
<MessageInput
query={query}
handleQueryChange={handleQueryChange}
handleKeyDown={handleKeyDown}
handleQuestionSubmit={handleQuestionSubmit}
setSimbaMode={setSimbaMode}
isLoading={isLoading}
/>
</div>
</footer>
</div>
</div>
);
};

View File

@@ -0,0 +1,69 @@
import { useState, useEffect } from "react";
import { conversationService } from "../api/conversationService";
type Conversation = {
title: string;
id: string;
};
type ConversationProps = {
conversations: Conversation[];
onSelectConversation: (conversation: Conversation) => void;
onCreateNewConversation: () => void;
};
export const ConversationList = ({
conversations,
onSelectConversation,
onCreateNewConversation,
}: ConversationProps) => {
const [conservations, setConversations] = useState(conversations);
useEffect(() => {
const loadConversations = async () => {
try {
let fetchedConversations =
await conversationService.getAllConversations();
if (conversations.length == 0) {
await conversationService.createConversation();
fetchedConversations =
await conversationService.getAllConversations();
}
setConversations(
fetchedConversations.map((conversation) => ({
id: conversation.id,
title: conversation.name,
})),
);
} catch (error) {
console.error("Failed to load messages:", error);
}
};
loadConversations();
}, []);
return (
<div className="bg-indigo-300 rounded-md p-3 sm:p-4 flex flex-col gap-1">
{conservations.map((conversation) => {
return (
<div
key={conversation.id}
className="border-blue-400 bg-indigo-300 hover:bg-indigo-200 cursor-pointer rounded-md p-3 min-h-[44px] flex items-center"
onClick={() => onSelectConversation(conversation)}
>
<p className="text-sm sm:text-base truncate w-full">
{conversation.title}
</p>
</div>
);
})}
<div
className="border-blue-400 bg-indigo-300 hover:bg-indigo-200 cursor-pointer rounded-md p-3 min-h-[44px] flex items-center"
onClick={() => onCreateNewConversation()}
>
<p className="text-sm sm:text-base"> + Start a new thread</p>
</div>
</div>
);
};

View File

@@ -0,0 +1,24 @@
type Conversation = {
title: string;
id: string;
};
type ConversationMenuProps = {
conversations: Conversation[];
};
export const ConversationMenu = ({ conversations }: ConversationMenuProps) => {
return (
<div className="absolute bg-white w-md rounded-md shadow-xl m-4 p-4">
<p className="py-2 px-4 rounded-md w-full text-xl font-bold">askSimba!</p>
{conversations.map((conversation) => (
<p
key={conversation.id}
className="py-2 px-4 rounded-md hover:bg-stone-200 w-full text-xl font-bold cursor-pointer"
>
{conversation.title}
</p>
))}
</div>
);
};

View File

@@ -0,0 +1,130 @@
import { useState, useEffect } from "react";
import { userService } from "../api/userService";
import { oidcService } from "../api/oidcService";
type LoginScreenProps = {
setAuthenticated: (isAuth: boolean) => void;
};
export const LoginScreen = ({ setAuthenticated }: LoginScreenProps) => {
const [error, setError] = useState<string>("");
const [isChecking, setIsChecking] = useState<boolean>(true);
const [isLoggingIn, setIsLoggingIn] = useState<boolean>(false);
useEffect(() => {
const initAuth = async () => {
// First, check for OIDC callback parameters
const callbackParams = oidcService.getCallbackParamsFromURL();
if (callbackParams) {
// Handle OIDC callback
try {
setIsLoggingIn(true);
const result = await oidcService.handleCallback(
callbackParams.code,
callbackParams.state
);
// Store tokens
localStorage.setItem("access_token", result.access_token);
localStorage.setItem("refresh_token", result.refresh_token);
// Clear URL parameters
oidcService.clearCallbackParams();
setAuthenticated(true);
setIsChecking(false);
return;
} catch (err) {
console.error("OIDC callback error:", err);
setError("Login failed. Please try again.");
oidcService.clearCallbackParams();
setIsLoggingIn(false);
setIsChecking(false);
return;
}
}
// Check if user is already authenticated
const isValid = await userService.validateToken();
if (isValid) {
setAuthenticated(true);
}
setIsChecking(false);
};
initAuth();
}, [setAuthenticated]);
const handleOIDCLogin = async () => {
try {
setIsLoggingIn(true);
setError("");
// Get authorization URL from backend
const authUrl = await oidcService.initiateLogin();
// Redirect to Authelia
window.location.href = authUrl;
} catch (err) {
setError("Failed to initiate login. Please try again.");
console.error("OIDC login error:", err);
setIsLoggingIn(false);
}
};
// Show loading state while checking authentication or processing callback
if (isChecking || isLoggingIn) {
return (
<div className="h-screen bg-opacity-20">
<div className="bg-white/85 h-screen flex items-center justify-center">
<div className="text-center">
<p className="text-lg sm:text-xl">
{isLoggingIn ? "Logging in..." : "Checking authentication..."}
</p>
</div>
</div>
</div>
);
}
return (
<div className="h-screen bg-opacity-20">
<div className="bg-white/85 h-screen">
<div className="flex flex-row justify-center py-4">
<div className="flex flex-col gap-4 w-full px-4 sm:w-11/12 sm:max-w-2xl lg:max-w-4xl sm:px-0">
<div className="flex flex-col gap-4">
<div className="flex flex-grow justify-center w-full bg-amber-400 p-2">
<h1 className="text-base sm:text-xl font-bold text-center">
I AM LOOKING FOR A DESIGNER. THIS APP WILL REMAIN UGLY UNTIL A
DESIGNER COMES.
</h1>
</div>
<header className="flex flex-row justify-center gap-2 grow sticky top-0 z-10 bg-white">
<h1 className="text-2xl sm:text-3xl">ask simba!</h1>
</header>
{error && (
<div className="text-red-600 font-semibold text-sm sm:text-base bg-red-50 p-3 rounded-md">
{error}
</div>
)}
<div className="text-center text-sm sm:text-base text-gray-600 py-2">
Click below to login with Authelia
</div>
</div>
<button
className="p-3 sm:p-4 min-h-[44px] border border-blue-400 bg-blue-200 hover:bg-blue-400 cursor-pointer rounded-md flex-grow text-sm sm:text-base font-semibold"
onClick={handleOIDCLogin}
disabled={isLoggingIn}
>
{isLoggingIn ? "Redirecting..." : "Login with Authelia"}
</button>
</div>
</div>
</div>
</div>
);
};

View File

@@ -0,0 +1,56 @@
import { useEffect, useState, useRef } from "react";
type MessageInputProps = {
handleQueryChange: (event: React.ChangeEvent<HTMLTextAreaElement>) => void;
handleKeyDown: (event: React.ChangeEvent<HTMLTextAreaElement>) => void;
handleQuestionSubmit: () => void;
setSimbaMode: (sdf: boolean) => void;
query: string;
isLoading: boolean;
};
export const MessageInput = ({
query,
handleKeyDown,
handleQueryChange,
handleQuestionSubmit,
setSimbaMode,
isLoading,
}: MessageInputProps) => {
return (
<div className="flex flex-col gap-4 sticky bottom-0 bg-[#3D763A] p-6 rounded-xl">
<div className="flex flex-row justify-between grow">
<textarea
className="p-3 sm:p-4 border border-blue-200 rounded-md grow bg-[#F9F5EB] min-h-[44px] resize-y"
onChange={handleQueryChange}
onKeyDown={handleKeyDown}
value={query}
rows={2}
placeholder="Type your message... (Press Enter to send, Shift+Enter for new line)"
/>
</div>
<div className="flex flex-row justify-between gap-2 grow">
<button
className={`p-3 sm:p-4 min-h-[44px] border border-blue-400 rounded-md flex-grow text-sm sm:text-base ${
isLoading
? "bg-gray-400 cursor-not-allowed opacity-50"
: "bg-[#EDA541] hover:bg-blue-400 cursor-pointer"
}`}
onClick={() => handleQuestionSubmit()}
type="submit"
disabled={isLoading}
>
{isLoading ? "Sending..." : "Submit"}
</button>
</div>
<div className="flex flex-row justify-center gap-2 grow items-center">
<input
type="checkbox"
onChange={(event) => setSimbaMode(event.target.checked)}
className="w-5 h-5 cursor-pointer"
/>
<p className="text-sm sm:text-base">simba mode?</p>
</div>
</div>
);
};

View File

@@ -0,0 +1,11 @@
type QuestionBubbleProps = {
text: string;
};
export const QuestionBubble = ({ text }: QuestionBubbleProps) => {
return (
<div className="w-2/3 rounded-md bg-stone-200 p-3 sm:p-4 break-words overflow-wrap-anywhere text-sm sm:text-base ml-auto">
🤦: {text}
</div>
);
};

View File

@@ -0,0 +1,56 @@
import { createContext, useContext, useState, ReactNode } from "react";
import { userService } from "../api/userService";
interface AuthContextType {
token: string | null;
login: (username: string, password: string) => Promise<any>;
logout: () => void;
isAuthenticated: () => boolean;
}
const AuthContext = createContext<AuthContextType | undefined>(undefined);
interface AuthProviderProps {
children: ReactNode;
}
export const AuthProvider = ({ children }: AuthProviderProps) => {
const [token, setToken] = useState(localStorage.getItem("access_token"));
const login = async (username: string, password: string) => {
try {
const data = await userService.login(username, password);
setToken(data.access_token);
localStorage.setItem("access_token", data.access_token);
localStorage.setItem("refresh_token", data.refresh_token);
return data;
} catch (error) {
console.error("Login failed:", error);
throw error;
}
};
const logout = () => {
setToken(null);
localStorage.removeItem("access_token");
localStorage.removeItem("refresh_token");
};
const isAuthenticated = () => {
return token !== null && token !== undefined && token !== "";
};
return (
<AuthContext.Provider value={{ token, login, logout, isAuthenticated }}>
{children}
</AuthContext.Provider>
);
};
export const useAuth = () => {
const context = useContext(AuthContext);
if (context === undefined) {
throw new Error("useAuth must be used within an AuthProvider");
}
return context;
};

View File

@@ -0,0 +1,11 @@
/// <reference types="@rsbuild/core/types" />
/**
* Imports the SVG file as a React component.
* @requires [@rsbuild/plugin-svgr](https://npmjs.com/package/@rsbuild/plugin-svgr)
*/
declare module '*.svg?react' {
import type React from 'react';
const ReactComponent: React.FunctionComponent<React.SVGProps<SVGSVGElement>>;
export default ReactComponent;
}

View File

@@ -0,0 +1,13 @@
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
const rootEl = document.getElementById('root');
if (rootEl) {
const root = ReactDOM.createRoot(rootEl);
root.render(
<React.StrictMode>
<App />
</React.StrictMode>,
);
}

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 MiB

View File

@@ -0,0 +1,25 @@
{
"compilerOptions": {
"lib": ["DOM", "ES2020"],
"jsx": "react-jsx",
"target": "ES2020",
"noEmit": true,
"skipLibCheck": true,
"useDefineForClassFields": true,
/* modules */
"module": "ESNext",
"moduleDetection": "force",
"moduleResolution": "bundler",
"verbatimModuleSyntax": true,
"resolveJsonModule": true,
"allowImportingTsExtensions": true,
"noUncheckedSideEffectImports": true,
/* type checking */
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true
},
"include": ["src"]
}

File diff suppressed because it is too large Load Diff

86
services/raggr/request.py Normal file
View File

@@ -0,0 +1,86 @@
import os
import tempfile
import httpx
import logging
from dotenv import load_dotenv
load_dotenv()
logging.basicConfig(level=logging.INFO)
class PaperlessNGXService:
def __init__(self):
self.base_url = os.getenv("BASE_URL")
self.token = os.getenv("PAPERLESS_TOKEN")
self.url = f"http://{os.getenv('BASE_URL')}/api/documents/?tags__id=8"
self.headers = {"Authorization": f"Token {os.getenv('PAPERLESS_TOKEN')}"}
def get_data(self):
print(f"Getting data from: {self.url}")
r = httpx.get(self.url, headers=self.headers)
results = r.json()["results"]
nextLink = r.json().get("next")
while nextLink:
r = httpx.get(nextLink, headers=self.headers)
results += r.json()["results"]
nextLink = r.json().get("next")
return results
def get_doc_by_id(self, doc_id: int):
url = f"http://{os.getenv('BASE_URL')}/api/documents/{doc_id}/"
r = httpx.get(url, headers=self.headers)
return r.json()
def download_pdf_from_id(self, id: int) -> str:
download_url = f"http://{os.getenv('BASE_URL')}/api/documents/{id}/download/"
response = httpx.get(
download_url, headers=self.headers, follow_redirects=True, timeout=30
)
response.raise_for_status()
# Use a temporary file for the downloaded PDF
temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=".pdf")
temp_file.write(response.content)
temp_file.close()
temp_pdf_path = temp_file.name
pdf_to_process = temp_pdf_path
return pdf_to_process
def upload_cleaned_content(self, document_id, data):
PUTS_URL = f"http://{os.getenv('BASE_URL')}/api/documents/{document_id}/"
r = httpx.put(PUTS_URL, headers=self.headers, data=data)
r.raise_for_status()
def upload_description(self, description_filepath, file, title, exif_date: str):
POST_URL = f"http://{os.getenv('BASE_URL')}/api/documents/post_document/"
files = {"document": ("description_filepath", file, "application/txt")}
data = {
"title": title,
"create": exif_date,
"document_type": 3,
"tags": [7],
}
r = httpx.post(POST_URL, headers=self.headers, data=data, files=files)
r.raise_for_status()
def get_tags(self):
GET_URL = f"http://{os.getenv('BASE_URL')}/api/tags/"
r = httpx.get(GET_URL, headers=self.headers)
data = r.json()
return {tag["id"]: tag["name"] for tag in data["results"]}
def get_doctypes(self):
GET_URL = f"http://{os.getenv('BASE_URL')}/api/document_types/"
r = httpx.get(GET_URL, headers=self.headers)
data = r.json()
return {doctype["id"]: doctype["name"] for doctype in data["results"]}
if __name__ == "__main__":
pp = PaperlessNGXService()
pp.get_data()

25
services/raggr/startup-dev.sh Executable file
View File

@@ -0,0 +1,25 @@
#!/bin/bash
set -e
echo "Initializing directories..."
mkdir -p /app/data/chromadb
echo "Rebuilding frontend..."
cd /app/raggr-frontend
yarn build
cd /app
echo "Setting up database..."
# Give PostgreSQL a moment to be ready (healthcheck in docker-compose handles this)
sleep 3
if ls migrations/models/0_*.py 1> /dev/null 2>&1; then
echo "Running database migrations..."
aerich upgrade
else
echo "No migrations found, initializing database..."
aerich init-db
fi
echo "Starting Flask application in debug mode..."
python app.py

10
services/raggr/startup.sh Normal file
View File

@@ -0,0 +1,10 @@
#!/bin/bash
echo "Running database migrations..."
aerich upgrade
echo "Starting reindex process..."
python main.py "" --reindex
echo "Starting Flask application..."
python app.py

View File

@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""Test the query_vector_store function."""
import asyncio
import os
from dotenv import load_dotenv
from blueprints.rag.logic import query_vector_store
# Load .env from the root directory
root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "../.."))
env_path = os.path.join(root_dir, ".env")
load_dotenv(env_path)
async def test_query(query: str):
"""Test a query against the vector store."""
print(f"Query: {query}\n")
result, docs = await query_vector_store(query)
print(f"Found {len(docs)} documents\n")
print("Serialized result:")
print(result)
print("\n" + "=" * 80 + "\n")
async def main():
queries = [
"What is Simba's weight?",
"What medications is Simba taking?",
"Tell me about Simba's recent vet visits",
]
for query in queries:
await test_query(query)
if __name__ == "__main__":
asyncio.run(main())

3404
services/raggr/uv.lock generated Normal file

File diff suppressed because it is too large Load Diff

12
users.py Normal file
View File

@@ -0,0 +1,12 @@
import sqlite3
class User:
def __init__(self, email: str, password_hash: str):
self.email = email
self.is_authenticated
if __name__ == "__main__":
connection = sqlite3.connect("users.db")
c = connection.cursor()