simbarag/docs/desktop-client.md

Desktop Client (exploration)

Status: proposal, no app code yet. Branch explore/desktop-client.

Goal: a persistent, native-feeling macOS app where Simba works alongside the user rather than a chat box in a browser tab. "Cowork-like" means:

• visible tool activity and progress while the agent works
• side panels for artifacts: budget reports, recipes, notes, documents
• drag files into the agent
• desktop notifications (scheduled messages, cron jobs, finished runs)
• a menu-bar icon and a global-hotkey quick-ask window

Decisions (2026-10-12)

Question	Decision	Consequence
Reachable off-LAN?	Yes, publicly via Caddy	Loading the remote origin works anywhere; offline.html only covers "no network / server down"
Who uses it?	Just the admin (Ryan)	Every desktop-only endpoint is @admin_required; no multi-user notification fan-out, no distribution story
Act on the Mac (local tools)?	Yes	Adds a local tool bridge (below). Tauri is still the pick: the bridge lives in the Rust core and can spawn local MCP servers as child processes
Artifact storage	Own artifacts table	Artifacts can be listed, reopened, edited and written back
Stop sending the refresh token on every request?	Yes, for web too	Becomes phase 0 (see "Auth fix")
Apple Developer account?	No; a local dev build is fine	No notarization or updater. make desktop builds an ad-hoc-signed .app locally

What exists today

Area	Current state	Desktop implication
Frontend	React 19 + Rsbuild + Tailwind, BrowserRouter, all API calls use relative /api/...	Reusable as-is if the shell loads it from the server origin; a bundled copy needs an API base URL and CORS
PWA	manifest.json + sw.js (static cache, network-only /api)	"Install app" already works in Chrome / Safari "Add to Dock"
Streaming	POST /api/conversation/stream-query → SSE over fetch with events thinking, thinking_delta, tool_start {tool}, tool_end {tool}, content, response, error	Too thin for Cowork: tool events have no run id, no args, no output, no structured payload
Run lifecycle	The agent runs inside the HTTP response generator. A client disconnect cancels it (CancelledError, partial answer persisted)	You can't close the window and let Simba keep working
Tool output	Tools return strings; output is persisted as speaker="tool" text (capped at 4000 chars) and the UI only shows the first line. simba_search already uses response_format="content_and_artifact"	Natural hook for structured artifacts
Tool labels	TOOL_MESSAGES map in useChat.ts is hand-maintained and stale (no Mealie, SimbaKit, Homebox, BookStack, Beli…)	Labels should come from the server
Uploads	POST /api/conversation/upload-image: images only, 10 MB, stored in S3, attached as an image block	Need generic file upload (PDF, text, markdown, CSV)
Auth	Backend is the confidential OIDC client. /oidc/login returns an Authelia URL; Authelia redirects to OIDC_REDIRECT_URI (the web root); the SPA posts code+state to /oidc/callback and stores JWTs in localStorage	In-webview login works unchanged if the shell loads the real origin. A system-browser login needs a handoff (see below)
Auth quirks	_oidc_sessions is an in-process dict (a login that spans a blue/green flip fails with "Invalid or expired state"); redirect_after_login is stored but never used; every API call sends the 30-day refresh token as the bearer (@jwt_refresh_token_required everywhere)	Desktop must keep that token in the Keychain, not plain storage
Notifications	Scheduled messages and cron jobs deliver to iMessage / email only (admin-only APIs)	No in-app notification channel exists
CORS	None configured	Only matters for a bundled frontend

Shell options

A. PWA ("Install app" / "Add to Dock")

• Reuse: 100%. Already shipped.
• Size: 0.
• macOS integration: own window and Dock icon; Web Push and the Badging API work for installed PWAs. No menu-bar icon, no global hotkey, no quick-ask panel, no Keychain, file access only through the picker/drop.
• OIDC: unchanged.
• Verdict: a free baseline and the fallback. It can't deliver the menu-bar or hotkey parts, which are most of what makes it feel like a desktop app.

B. Tauri v2 (recommended)

• Reuse: high. Either load the deployed frontend from the server origin (no fork), or bundle raggr-frontend/dist (needs an API base URL plus CORS).
• Size: ~5–15 MB .app, low idle memory (system WKWebView). Rust and cargo are already installed on this machine.
• macOS integration: first-party plugins for tray/menu-bar (core TrayIcon), global-shortcut, notification, deep-link, single-instance, window-state, autostart, updater, positioner (anchor the quick-ask panel under the tray icon), keychain storage, and window file drag-drop events with real paths. ActivationPolicy::Accessory gives a menu-bar-only mode.
• Costs: Safari/WebKit engine (the PWA already targets Safari, so low risk). Service workers don't run in WKWebView here (the sw.js register call already fails soft). Native code is Rust, but the native layer stays small: tray, hotkey, notifications, keychain, deep link.
• Security: remote content can reach Tauri IPC only through an explicit remote.urls capability. Grant the remote origin only notification and window commands; register shortcut and tray from Rust, not JS.

C. Electron

• Reuse: same as Tauri.
• Size: ~100 MB+ download, ~200 MB+ on disk, 150–300 MB RAM idle.
• macOS integration: everything Tauri has, more mature (menubar package, globalShortcut, Notification, safeStorage), plus Node in the main process. That would matter if the client ran local tools or MCP servers on the user's machine (see open questions).
• Verdict: the better choice only if we want heavy client-side tool execution in JS/Node. For a thin client to a self-hosted agent, the size and memory cost buys nothing. Local tools are wanted, but the bridge design below runs local MCP servers as child processes (Node is already installed), so Electron's in-process Node isn't needed.

Decision matrix

	PWA	Tauri v2	Electron
Reuse raggr-frontend	✅ as-is	✅ remote or bundled	✅ remote or bundled
Bundle / RAM	none	~10 MB / low	~100 MB+ / high
Menu bar + quick-ask	❌	✅	✅
Global hotkey	❌	✅	✅
Notifications	Web Push (VAPID)	✅ native	✅ native
File drag-drop w/ paths	browser File only	✅	✅
Secure token storage	❌ localStorage	✅ Keychain	✅ safeStorage
Auto-update	free (server)	updater plugin, or free with a remote UI	electron-updater
Local tool execution	❌	Rust	Node (easiest)

Recommended architecture

A Tauri v2 shell that loads the deployed SimbaRAG origin, plus a tiny bundled "offline/connect" page. Cowork features are built in raggr-frontend behind feature detection, so the web app gets them too.

┌──────────────────────── Simba.app (Tauri v2) ─────────────────────────┐
│ Rust core (small)                                                      │
│  • tray icon + menu (Ask…, New chat, recent convos, Quit)              │
│  • global shortcut ⌥Space → quick-ask panel (positioner, always-on-top)│
│  • notifications (plugin-notification)                                 │
│  • keychain token store, deep-link `simba://`, single-instance         │
│  • window-state, autostart                                             │
│                                                                        │
│  Main window  ──loads──▶  https://<simba-host>/        (full UI)       │
│  Quick panel  ──loads──▶  https://<simba-host>/quick   (compact route) │
│  Fallback     ──bundled─▶ offline.html (server unreachable / retry)    │
└───────────────────────────────┬────────────────────────────────────────┘
                                │ HTTPS: REST + SSE (same origin, no CORS)
┌───────────────────────────────▼────────────────────────────────────────┐
│ Caddy → raggr-blue / raggr-green (Quart)                               │
│  /api/conversation/*  (+ structured tool/artifact events)              │
│  /api/runs/*          (detached agent runs, resumable SSE)  [phase 3]  │
│  /api/files           (generic upload)                                 │
│  /api/notifications/stream (per-user SSE feed)                         │
└────────────────────────────────────────────────────────────────────────┘

Why load the remote origin rather than bundle:

• One codebase, one deploy. make redeploy updates web, PWA and desktop together. No version skew between the desktop UI and the SSE protocol, and no desktop release for UI changes.
• No CORS, no API-base refactor, and the current OIDC flow works unchanged inside the webview (redirect back to the web root on the same origin).
• The agent needs the server anyway, so offline-first buys little. The bundled offline.html covers "server down / not on VPN".

The frontend detects the shell with window.__TAURI_INTERNALS__ (or a ?shell=desktop query param) and turns on the native bits: notifications, drag-drop paths, and the "open in quick panel" route. Everything else renders the same in a browser.

Cowork UI (in raggr-frontend, shared by web and desktop)

• Three panes: conversations | chat | artifact panel (collapsible, a tab per artifact). On mobile/web the artifact panel becomes a sheet.
• Activity timeline in place of single-line ToolBubble pills: one row per tool call with a server-provided label, args summary, a spinner, duration, and an expandable result. Clicking a row with an artifact opens it in the panel.
• Artifact renderers: budget (YNAB summary/category table + chart), transactions (table), recipe (Mealie: ingredients/steps, "add to meal plan"), note (Obsidian markdown, "open in Obsidian" via obsidian:// deep link), document (Paperless hit: title, snippet, link to Paperless), photo (Immich), device (SimbaKit status/weight chart).
• Drop zone over the whole window: images go to the existing vision path; PDFs and text go to the new upload endpoint (below).
• Quick-ask route (/quick): input + streamed answer only; "Open in Simba" hands the conversation to the main window.

Local tool bridge (Simba acts on the Mac)

The backend can't reach the Mac (it's behind NAT), so the desktop app dials out and the backend calls tools back over that connection.

Mac                                              Server
┌─ Simba.app Rust core ────────────┐   wss    ┌─ Quart ────────────────────────┐
│ bridge client ───────────────────┼─────────▶│ /api/desktop/bridge (websocket)│
│   │ JSON-RPC (MCP messages)      │          │   │                            │
│   ▼                              │          │   ▼                            │
│ local MCP servers (child procs)  │          │ MCP ClientSession over the ws  │
│  • filesystem (scoped folders)   │          │  → langchain-mcp-adapters tools│
│  • later: screenshot, clipboard, │          │  → mcp_tools.select_tools(     │
│    open app/URL, Shortcuts       │          │      prefix="mac", admin-only) │
│ confirm dialog for write tools   │          │  → injected per run only while │
└──────────────────────────────────┘          │    the bridge is connected     │
                                              └────────────────────────────────┘

• The protocol is MCP, so the backend reuses the existing pattern in utils/mcp_tools.py (allowlist, prefix_ rename, admin re-check). Only the transport is new: an MCP ClientSession over the websocket's read/write streams instead of stdio.
• The bridge runs in Rust, not in the webview. The webview loads remote content, so giving page JS file access would turn any XSS into local file access. The Rust core owns the socket, the child processes, and the allowlist of exposed folders (picked in a native settings window).
• Writes need consent: tools tagged as writes (save file, move, run a Shortcut) show a native confirm dialog before running. Reads inside the scoped folders are allowed without asking.
• Availability: the agent is compiled once at startup, so mac_* tools are added per run, e.g. via get_agent_for(user) returning a variant with bridge tools when a bridge session is connected. If the bridge drops mid-call, the tool returns "Mac is offline" rather than hanging.
• Auth: the websocket authenticates with the same JWT, so this depends on the auth fix landing first. A blue/green flip drops the socket and the app reconnects to the new color.
• Open spike question: do the mcp Python SDK and langchain-mcp-adapters accept a custom transport cleanly? Fallback: a small hand-rolled JSON-RPC tool-call protocol with tools defined server-side.

Auth fix (web + desktop)

Today the frontend sends the refresh token as the bearer on every call, and 20 routes use @jwt_refresh_token_required (conversation ×9, beli ×6, users ×2 incl. /refresh, rag ×1, app ×1, plus admin_required in blueprints/users/decorators.py).

Change:

1. Backend: every route except /api/user/refresh uses @jwt_required (access token, 1h). The same goes for the inner decorator of admin_required.
2. Frontend: collapse fetchWithRefreshToken and fetchWithAuth into one function that sends the access token and, on 401, refreshes once and retries. validateToken keeps using the refresh token.
3. Desktop: the refresh token lives in the Keychain. The webview keeps only the access token (memory or localStorage).

This is blue/green-safe in both directions. An old frontend talking to the new backend sends the refresh token, gets a 401, calls /refresh, and retries with an access token, so it works. A new frontend talking to the old backend sends an access token where a refresh token is required, gets a 401, and the retry with a fresh access token also fails. That window lasts only as long as the deploy flip. To avoid it, deploy the backend change first: for one release, accept either token type on the protected routes (a small custom decorator), then flip the frontend.

SSE streams are authenticated once at request start, so a stream running past the 1h access-token expiry is fine.

Also in scope: move _oidc_sessions into Postgres (or signed, stateless state) so a login spanning a deploy doesn't fail, and either honour or delete the unused redirect_after_login.

Backend changes

Ordered by how much the client depends on them. All are additive, so blue/green stays safe: the old frontend ignores the new fields.

1. Richer SSE events (blueprints/conversation/__init__.py):

   • tool_start {run_id, tool, label, input}: input comes from event["data"]["input"], trimmed/redacted; label comes from a server-side map that replaces TOOL_MESSAGES.
   • tool_end {run_id, tool, ms, summary, artifact?}: run_id lets parallel tool calls pair up correctly.
   • optional event: id lines so a client can resume (needed for phase 3).

2. Structured artifacts. Move high-value tools to response_format="content_and_artifact" (as simba_search already does): the LLM still sees compact text, and the UI gets {kind, title, data}. Persist them in an artifacts table: id, user_id, conversation_id, message_id (nullable FK to the tool row), kind, title, data JSON, source_ref (e.g. Mealie slug, Obsidian path, Paperless id), version, created_at, updated_at. Generate the migration with make migrate-new (never hand-written); a new table is backward-compatible. Endpoints: GET /api/artifacts?conversation_id= and GET/PATCH /api/artifacts/<id>. PATCH writes back to the source integration where one exists (edit a recipe → Mealie, edit a note → Obsidian) and bumps version. GET /api/conversation/<id> includes artifact ids on tool messages.

3. Generic file upload POST /api/conversation/upload (supersedes upload-image, which stays for compatibility): accept images, PDF, md/txt, CSV. Store in S3 under conversations/<id>/, extract text (reuse utils/cleaner.py for PDFs), and attach it as a document block capped to the context budget (32k slots). Possible extra: a "file it in Paperless" action that pushes the PDF into Paperless so it gets indexed. Revisit MAX_CONTENT_LENGTH (10 MB).

4. Notifications feed. A notifications table (user, kind, title, body, link, created_at, read_at) written by the scheduler (process_due_messages), cron runs, and completed detached runs. Add GET /api/notifications/stream (per-user SSE with Last-Event-ID replay from the table, so a blue/green flip only causes a reconnect) and POST /api/notifications/<id>/read. Also consider a desktop value on MessageChannel so a scheduled message can target the app instead of iMessage.

5. Detached agent runs (phase 3): POST /api/runs starts the agent in a background task and returns run_id; GET /api/runs/<id>/events streams with replay. This decouples work from the window. Constraint: an in-process run dies on a blue/green stop. Either persist events (Postgres) and mark runs interrupted on startup, or have deploy.sh drain active runs before stopping the old color.

6. Native auth handoff (phase 3, optional): sign in through the system browser (password manager, passkeys, existing Authelia session) without changing Authelia config:

   1. The app generates a PKCE verifier and opens https://<simba-host>/api/user/oidc/login?client=desktop&challenge=… in the default browser.
   2. The normal web OIDC round-trip runs against the existing redirect URI.
   3. The backend sees client=desktop on the stored session and, instead of returning JWTs to the page, issues a one-time 60-second handoff code and redirects to simba://auth?code=….
   4. The app exchanges code + verifier at POST /api/user/desktop/token for JWTs and stores them in the Keychain.

   This also requires moving _oidc_sessions into Postgres or signed state so it survives blue/green. The alternative, registering a loopback or custom-scheme redirect URI in Authelia, depends on Authelia's redirect matching rules (exact match, ports?). That needs checking, and the client secret would have to stay on the server anyway.

7. Local tool bridge: /api/desktop/bridge websocket, an MCP-over-websocket client, and per-run injection of mac_* tools (see above). Admin-only.

8. CORS: only needed if we ever bundle the frontend (quart-cors allowing tauri://localhost / http://tauri.localhost). Not needed with the recommendation.

Phased plan

Phase	Scope	Backend	Exit criteria
0a. Auth fix (own PR)	access token on every call, single fetch helper, OIDC state in Postgres	auth fix (+ migration if state goes in a table)	Web works through a blue/green flip; the refresh token is only ever sent to /refresh
0b. Spike (½ day)	desktop/ Tauri v2 app: one window loading <simba-host>, tray icon with Show/Quit, ⌥Space toggles the window, offline.html fallback	none	Log in through Authelia inside the webview, stream an answer, survive a make redeploy
1. Native shell	quick-ask panel (/quick route), notifications, window-state, single-instance, autostart, refresh token in the Keychain	none	Hotkey → ask → answer without touching the main window
2. Cowork UI	activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone	items 1–3 (artifacts table)	Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised
3. Local tools	Rust bridge client, filesystem MCP scoped to chosen folders, write-confirm dialog, folder picker	item 7	"Summarise the PDFs in ~/Downloads/taxes" works; a write prompts first; quitting the app removes the tools
4. Background agent	detached runs, notifications feed, desktop channel for scheduled messages, system-browser auth handoff	items 4–6	Close the window mid-run → notification when done → reopen shows the full timeline

0b status: scaffolded in desktop/. It has a bundled connect page that probes /api/health and hands off to $SIMBA_URL, a menu-bar icon (Show / New Chat / Reconnect / Quit), ⌥Space to toggle, close-to-hide, single-instance, and Dock-click reopen. The remote origin gets no IPC. desktop/src-tauri/rust-toolchain.toml pins Rust 1.90, the minimum Tauri 2.12 accepts, so the global toolchain is untouched. Run it with make desktop-dev SIMBA_URL=… or make desktop. Verified 2026-10-12: Authelia login inside the webview works, and the conversation list loads.

Phases 1–3 are independent once 0a/0b land. Phase 2 also improves the web UI. There's no distribution phase: make desktop runs tauri build and copies the ad-hoc-signed .app to /Applications. A locally built app isn't quarantined, so Gatekeeper won't complain. Because the UI loads from the server, rebuilds are only needed when the Rust shell changes.

Open questions

1. Keychain and ad-hoc signing: the ad-hoc signature changes on every rebuild, so macOS may ask again for Keychain access after each make desktop. Acceptable, or use a self-signed local code-signing certificate to keep the identity stable?
2. Local tool scope for v1: filesystem (read + confirmed writes) only, or also screenshot, clipboard, "open URL/app", and running Shortcuts?
3. MCP over websocket: does langchain-mcp-adapters take a custom session or transport cleanly? Settle this in a short spike before phase 3; otherwise fall back to a hand-rolled tool-call protocol.
4. Engine risk: does anything in the UI rely on Chromium-only behaviour? The spike should check markdown rendering, streaming fetch, and the image picker in WKWebView.
5. Artifact write-back conflicts: if a recipe or note changed in Mealie/Obsidian after the artifact was captured, should we refuse, show a diff, or let the last write win?