92 lines
4.6 KiB
Markdown
92 lines
4.6 KiB
Markdown
# SimbaRAG Email Integration
|
|
|
|
## What This Is
|
|
|
|
A personal RAG (Retrieval-Augmented Generation) conversational AI system that answers questions about your life through document search, budget tracking, meal planning, and now email inbox analytics. It ingests documents from Paperless-NGX, YNAB transactions, Mealie recipes, and (new) IMAP email to provide intelligent, context-aware responses.
|
|
|
|
## Core Value
|
|
|
|
Personal information retrieval through natural conversation - ask about any aspect of your documented life (papers, finances, meals, emails) and get accurate, context-aware answers drawn from your own data sources.
|
|
|
|
## Requirements
|
|
|
|
### Validated
|
|
|
|
- ✓ OIDC authentication via Authelia with PKCE flow — existing
|
|
- ✓ RBAC using LDAP groups (lldap_admin for admin privileges) — existing
|
|
- ✓ Multi-user conversations with persistent message history — existing
|
|
- ✓ RAG document search from Paperless-NGX documents — existing
|
|
- ✓ Multi-agent LangChain orchestration with tool calling — existing
|
|
- ✓ YNAB budget integration (budget summary, transactions, spending insights) — existing
|
|
- ✓ Mealie meal planning integration (shopping lists, meal plans, recipes) — existing
|
|
- ✓ Tavily web search for real-time information — existing
|
|
- ✓ Streaming SSE chat responses for real-time feedback — existing
|
|
- ✓ Vector embeddings in ChromaDB for similarity search — existing
|
|
- ✓ JWT session management with refresh tokens — existing
|
|
- ✓ Local LLM support via llama-server with OpenAI fallback — existing
|
|
|
|
### Active
|
|
|
|
- [ ] IMAP email ingestion for inbox analytics
|
|
- [ ] Multi-account email support (multiple IMAP connections)
|
|
- [ ] Admin-only email access (configuration and queries)
|
|
- [ ] Scheduled email sync (configurable interval)
|
|
- [ ] Auto-purge emails older than 30 days from vector index
|
|
- [ ] Index email metadata: subject, body text, sender information
|
|
- [ ] Read-only email analysis (no modification/deletion of emails)
|
|
- [ ] Email-aware LangChain tools (who's emailing, what subjects, subscription patterns)
|
|
|
|
### Out of Scope
|
|
|
|
- Email actions (mark read/unread, delete, archive) — read-only analytics only
|
|
- SMTP sending capabilities — inbox ingestion only
|
|
- Email attachment indexing — too complex for v1, focus on text content
|
|
- Real-time email sync — scheduled sync sufficient, reduces server load
|
|
- POP3 support — IMAP provides better state management
|
|
- Non-admin email access — privacy-sensitive feature, admin-only
|
|
|
|
## Context
|
|
|
|
**Existing Architecture:**
|
|
- Python/Quart async backend with React frontend
|
|
- Tortoise ORM with PostgreSQL for relational data
|
|
- ChromaDB for vector embeddings (persistent storage)
|
|
- Blueprint-based API organization with `/api/rag`, `/api/conversation`, `/api/user`
|
|
- LangChain agent with `@tool` decorated functions for extended capabilities
|
|
- Existing integrations: Paperless-NGX (documents), YNAB (finance), Mealie (meals), Tavily (web)
|
|
|
|
**Email Use Cases:**
|
|
- "What emails did I get this week?"
|
|
- "Who has been emailing me most frequently?"
|
|
- "Show me subscription emails I should unsubscribe from"
|
|
- "What topics am I being emailed about?"
|
|
- Inbox pattern recognition and analytics through natural language
|
|
|
|
**Privacy Considerations:**
|
|
- Email is highly personal - admin-only access prevents exposure to other users
|
|
- 30-day retention window limits data exposure and storage growth
|
|
- Self-hosted deployment keeps email content on user's infrastructure
|
|
|
|
## Constraints
|
|
|
|
- **Tech Stack**: Python/Quart backend — must use existing framework and patterns
|
|
- **Storage**: ChromaDB vector store — email embeddings live alongside documents
|
|
- **Authentication**: LDAP group-based RBAC — email features gated to `lldap_admin` group
|
|
- **Deployment**: Docker Compose self-hosted — no cloud email storage or processing
|
|
- **Retention**: 30-day sliding window — automatic purge of older emails from index
|
|
- **Performance**: Scheduled sync only — avoid real-time polling overhead on mail servers
|
|
|
|
## Key Decisions
|
|
|
|
| Decision | Rationale | Outcome |
|
|
|----------|-----------|---------|
|
|
| IMAP only (no SMTP) | User wants inbox analytics, not sending capabilities | — Pending |
|
|
| Admin-only access | Email is privacy-sensitive, limit to trusted admins | — Pending |
|
|
| 30-day retention | Balance utility with privacy/storage concerns | — Pending |
|
|
| Scheduled sync | Reduces server load vs real-time polling | — Pending |
|
|
| No attachment indexing | Complexity vs value, focus on text content first | — Pending |
|
|
| ChromaDB for emails | Reuse existing vector store, no new infrastructure | — Pending |
|
|
|
|
---
|
|
*Last updated: 2026-02-04 after initialization*
|