simbarag/.planning/ROADMAP.md

Roadmap: SimbaRAG Email Integration

Overview

Add IMAP email ingestion to SimbaRAG's existing document/finance/meal analytics capabilities. Admin users can configure email accounts, system syncs and embeds emails into ChromaDB on a schedule, automatically purges emails older than 30 days, and provides LangChain tools for inbox analytics through natural conversation.

Phases

Phase Numbering:

• Integer phases (1, 2, 3): Planned milestone work
• Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED)

Decimal phases appear between their surrounding integers in numeric order.

• Phase 1: Foundation - Database models and IMAP utilities
• Phase 2: Account Management - Admin UI for configuring email accounts
• Phase 3: Email Ingestion - Sync engine, embeddings, retention cleanup
• Phase 4: Query Tools - LangChain tools for email analytics

Phase Details

Phase 1: Foundation

Goal: Core infrastructure for email ingestion is in place Depends on: Nothing (first phase) Requirements: None (foundational infrastructure) Success Criteria (what must be TRUE):

1. Database tables exist for email accounts, sync status, and email metadata
2. IMAP connection utility can authenticate and list folders from test server
3. Email body parser extracts text from both plain text and HTML formats
4. Encryption utility securely stores and retrieves IMAP credentials Plans: TBD

Plans:

• 01-01: TBD

Phase 2: Account Management

Goal: Admin users can configure and manage IMAP email accounts Depends on: Phase 1 Requirements: ACCT-01, ACCT-02, ACCT-03, ACCT-04, ACCT-05, ACCT-06, ACCT-07 Success Criteria (what must be TRUE):

1. Admin can add new IMAP account with host, port, username, password, folder selection
2. Admin can test IMAP connection and see success/failure before saving
3. Admin can view list of configured accounts with masked credentials
4. Admin can edit existing account configuration and delete accounts
5. Only users in lldap_admin group can access email account endpoints Plans: TBD

Plans:

• 02-01: TBD

Phase 3: Email Ingestion

Goal: System automatically syncs emails, creates embeddings, and purges old content Depends on: Phase 2 Requirements: SYNC-01, SYNC-02, SYNC-03, SYNC-04, SYNC-05, SYNC-06, SYNC-07, SYNC-08, SYNC-09, RETN-01, RETN-02, RETN-03, RETN-04, RETN-05 Success Criteria (what must be TRUE):

1. System connects to configured IMAP accounts and fetches messages from selected folders
2. System parses email metadata (subject, sender, date) and extracts body text from plain/HTML
3. System generates embeddings and stores emails in ChromaDB with metadata
4. System performs scheduled sync at configurable intervals (default hourly)
5. System tracks last sync timestamp and performs incremental sync (only new emails)
6. System automatically purges emails older than retention period (default 30 days)
7. Admin can view sync logs showing success/failure, counts, and errors Plans: TBD

Plans:

• 03-01: TBD

Phase 4: Query Tools

Goal: Admin users can query email content through conversational interface Depends on: Phase 3 Requirements: QUERY-01, QUERY-02, QUERY-03, QUERY-04, QUERY-05, QUERY-06 Success Criteria (what must be TRUE):

1. LangChain agent has tool to search emails by content, sender, or date range
2. Agent can identify most frequent senders in a timeframe
3. Agent can analyze subject lines and identify common topics
4. Agent can detect subscription/newsletter patterns (recurring senders, unsubscribe links)
5. Agent can answer time-based queries ("emails this week", "emails in January")
6. Only admin users can query email content via conversation interface Plans: TBD

Plans:

• 04-01: TBD

Progress

Execution Order: Phases execute in numeric order: 1 → 2 → 3 → 4

Phase	Plans Complete	Status	Completed
1. Foundation	0/1	Not started	-
2. Account Management	0/1	Not started	-
3. Email Ingestion	0/1	Not started	-
4. Query Tools	0/1	Not started	-