simbarag/docs/desktop-client.md

# Desktop Client (exploration)

Status: **proposal**, no app code yet. Branch `explore/desktop-client`.

Goal: a persistent, native-feeling macOS app where Simba works *alongside* the
user rather than a chat box in a browser tab. "Cowork-like" means:

- visible tool activity and progress while the agent works
- side panels for artifacts: budget reports, recipes, notes, documents
- drag files into the agent
- desktop notifications (scheduled messages, cron jobs, finished runs)
- a menu-bar icon and a global hotkey to summon the window

## Decisions (2026-10-12)

| Question | Decision | Consequence |
|---|---|---|
| Reachable off-LAN? | Yes, publicly via Caddy | Loading the remote origin works anywhere; `offline.html` only covers "no network / server down" |
| Who uses it? | Just the admin (Ryan) | Every desktop-only endpoint is `@admin_required`; no multi-user notification fan-out, no distribution story |
| Act on the Mac (local tools)? | **Yes** | Adds a *local tool bridge* (below). Tauri is still the pick: the bridge lives in the Rust core and can spawn local MCP servers as child processes |
| Artifact storage | Own `artifacts` table | Artifacts can be listed, reopened, edited and written back |
| Stop sending the refresh token on every request? | Yes, for web too | Becomes phase 0 (see "Auth fix") |
| Apple Developer account? | No; a local dev build is fine | No notarization or updater. `make desktop` builds an ad-hoc-signed `.app` locally |

## What exists today

| Area | Current state | Desktop implication |
|---|---|---|
| Frontend | React 19 + Rsbuild + Tailwind, `BrowserRouter`, all API calls use relative `/api/...` | Reusable as-is if the shell loads it from the server origin; a bundled copy needs an API base URL and CORS |
| PWA | `manifest.json` + `sw.js` (static cache, network-only `/api`) | "Install app" already works in Chrome / Safari "Add to Dock" |
| Streaming | `POST /api/conversation/stream-query` → SSE over `fetch` with events `thinking`, `thinking_delta`, `tool_start {tool}`, `tool_end {tool}`, `content`, `response`, `error` | Too thin for Cowork: tool events have no run id, no args, no output, no structured payload |
| Run lifecycle | The agent runs *inside* the HTTP response generator. A client disconnect cancels it (`CancelledError`, partial answer persisted) | You can't close the window and let Simba keep working |
| Tool output | Tools return strings; output is persisted as `speaker="tool"` text (capped at 4000 chars) and the UI only shows the first line. `simba_search` already uses `response_format="content_and_artifact"` | Natural hook for structured artifacts |
| Tool labels | `TOOL_MESSAGES` map in `useChat.ts` is hand-maintained and stale (no Mealie, SimbaKit, Homebox, BookStack, Beli…) | Labels should come from the server |
| Uploads | `POST /api/conversation/upload-image`: images only, 10 MB, stored in S3, attached as an image block | Need generic file upload (PDF, text, markdown, CSV) |
| Auth | Backend is the confidential OIDC client. `/oidc/login` returns an Authelia URL; Authelia redirects to `OIDC_REDIRECT_URI` (the web root); the SPA posts `code`+`state` to `/oidc/callback` and stores JWTs in `localStorage` | In-webview login works unchanged if the shell loads the real origin. A system-browser login needs a handoff (see below) |
| Auth quirks | `_oidc_sessions` is an in-process dict (a login that spans a blue/green flip fails with "Invalid or expired state"); `redirect_after_login` is stored but never used; every API call sends the **30-day refresh token** as the bearer (`@jwt_refresh_token_required` everywhere) | Desktop must keep that token in the Keychain, not plain storage |
| Notifications | Scheduled messages and cron jobs deliver to iMessage / email only (admin-only APIs) | No in-app notification channel exists |
| CORS | None configured | Only matters for a bundled frontend |

## Shell options

### A. PWA ("Install app" / "Add to Dock")

- **Reuse:** 100%. Already shipped.
- **Size:** 0.
- **macOS integration:** own window and Dock icon; Web Push and the Badging API
  work for installed PWAs. **No** menu-bar icon, **no** global hotkey, no
  quick-ask panel, no Keychain, file access only through the picker/drop.
- **OIDC:** unchanged.
- **Verdict:** a free baseline and the fallback. It can't deliver the
  menu-bar or hotkey parts, which are most of what makes it feel like a
  desktop app.

### B. Tauri v2 (recommended)

- **Reuse:** high. Either load the deployed frontend from the server origin
  (no fork), or bundle `raggr-frontend/dist` (needs an API base URL plus CORS).
- **Size:** ~5–15 MB `.app`, low idle memory (system WKWebView). Rust and
  cargo are already installed on this machine.
- **macOS integration:** first-party plugins for tray/menu-bar (core
  `TrayIcon`), `global-shortcut`, `notification`, `deep-link`,
  `single-instance`, `window-state`, `autostart`, `updater`, `positioner`
  (anchor the quick-ask panel under the tray icon), keychain storage, and
  window file drag-drop events with real paths. `ActivationPolicy::Accessory`
  gives a menu-bar-only mode.
- **Costs:** Safari/WebKit engine (the PWA already targets Safari, so low
  risk). Service workers don't run in WKWebView here (the `sw.js` register
  call already fails soft). Native code is Rust, but the native layer stays
  small: tray, hotkey, notifications, keychain, deep link.
- **Security:** remote content can reach Tauri IPC only through an explicit
  `remote.urls` capability. Grant the remote origin only notification and
  window commands; register shortcut and tray from Rust, not JS.

### C. Electron

- **Reuse:** same as Tauri.
- **Size:** ~100 MB+ download, ~200 MB+ on disk, 150–300 MB RAM idle.
- **macOS integration:** everything Tauri has, more mature (`menubar`
  package, `globalShortcut`, `Notification`, `safeStorage`), plus Node in the
  main process. That would matter if the client ran local tools or MCP
  servers on the user's machine (see open questions).
- **Verdict:** the better choice only if we want heavy client-side tool
  execution in JS/Node. For a thin client to a self-hosted agent, the size
  and memory cost buys nothing. Local tools are wanted, but the bridge design
  below runs local MCP servers as child processes (Node is already installed),
  so Electron's in-process Node isn't needed.

### Decision matrix

| | PWA | Tauri v2 | Electron |
|---|---|---|---|
| Reuse raggr-frontend | ✅ as-is | ✅ remote or bundled | ✅ remote or bundled |
| Bundle / RAM | none | ~10 MB / low | ~100 MB+ / high |
| Menu bar + quick-ask | ❌ | ✅ | ✅ |
| Global hotkey | ❌ | ✅ | ✅ |
| Notifications | Web Push (VAPID) | ✅ native | ✅ native |
| File drag-drop w/ paths | browser `File` only | ✅ | ✅ |
| Secure token storage | ❌ localStorage | ✅ Keychain | ✅ safeStorage |
| Auto-update | free (server) | `updater` plugin, or free with a remote UI | `electron-updater` |
| Local tool execution | ❌ | Rust | Node (easiest) |

## Recommended architecture

**A Tauri v2 shell that loads the deployed SimbaRAG origin, plus a tiny bundled
"offline/connect" page. Cowork features are built in `raggr-frontend` behind
feature detection, so the web app gets them too.**

```
┌──────────────────────── Simba.app (Tauri v2) ─────────────────────────┐
│ Rust core (small)                                                      │
│  • tray icon + menu (Ask…, New chat, recent convos, Quit)              │
│  • global shortcut ⌥Space → show/hide main window                      │
│  • notifications (plugin-notification)                                 │
│  • keychain token store, deep-link `simba://`, single-instance         │
│  • window-state, autostart                                             │
│                                                                        │
│  Main window  ──loads──▶  https://<simba-host>/        (full UI)       │
│  Fallback     ──bundled─▶ offline.html (server unreachable / retry)    │
└───────────────────────────────┬────────────────────────────────────────┘
                                │ HTTPS: REST + SSE (same origin, no CORS)
┌───────────────────────────────▼────────────────────────────────────────┐
│ Caddy → raggr-blue / raggr-green (Quart)                               │
│  /api/conversation/*  (+ structured tool/artifact events)              │
│  /api/runs/*          (detached agent runs, resumable SSE)  [phase 3]  │
│  /api/files           (generic upload)                                 │
│  /api/notifications/stream (per-user SSE feed)                         │
└────────────────────────────────────────────────────────────────────────┘
```

Why load the remote origin rather than bundle:

- **One codebase, one deploy.** `make redeploy` updates web, PWA and desktop
  together. No version skew between the desktop UI and the SSE protocol, and
  no desktop release for UI changes.
- **No CORS, no API-base refactor, and the current OIDC flow works unchanged**
  inside the webview (redirect back to the web root on the same origin).
- The agent needs the server anyway, so offline-first buys little. The bundled
  `offline.html` covers "server down / not on VPN".

The frontend detects the shell with `window.__TAURI_INTERNALS__` (or a
`?shell=desktop` query param) and turns on the native bits: notifications
and drag-drop paths. Everything else renders
the same in a browser.

### Cowork UI (in raggr-frontend, shared by web and desktop)

- **Three panes:** conversations | chat | **artifact panel** (collapsible, a
  tab per artifact). On mobile/web the artifact panel becomes a sheet.
- **Activity timeline** in place of single-line `ToolBubble` pills: one row per
  tool call with a server-provided label, args summary, a spinner, duration,
  and an expandable result. Clicking a row with an artifact opens it in the
  panel.
- **Artifact renderers:** `budget` (YNAB summary/category table + chart),
  `transactions` (table), `recipe` (Mealie: ingredients/steps, "add to meal
  plan"), `note` (Obsidian markdown, "open in Obsidian" via
  `obsidian://` deep link), `document` (Paperless hit: title, snippet, link to
  Paperless), `photo` (Immich), `device` (SimbaKit status/weight chart).
- **Drop zone** over the whole window: images go to the existing vision path;
  PDFs and text go to the new upload endpoint (below).

### Local tool bridge (Simba acts on the Mac)

The backend can't reach the Mac (it's behind NAT), so the desktop app dials
out and the backend calls tools back over that connection.

```
Mac                                              Server
┌─ Simba.app Rust core ────────────┐   wss    ┌─ Quart ────────────────────────┐
│ bridge client ───────────────────┼─────────▶│ /api/desktop/bridge (websocket)│
│   │ JSON-RPC (MCP messages)      │          │   │                            │
│   ▼                              │          │   ▼                            │
│ local MCP servers (child procs)  │          │ MCP ClientSession over the ws  │
│  • filesystem (scoped folders)   │          │  → langchain-mcp-adapters tools│
│  • later: screenshot, clipboard, │          │  → mcp_tools.select_tools(     │
│    open app/URL, Shortcuts       │          │      prefix="mac", admin-only) │
│ confirm dialog for write tools   │          │  → injected per run only while │
└──────────────────────────────────┘          │    the bridge is connected     │
                                              └────────────────────────────────┘
```

- **The protocol is MCP**, so the backend reuses the existing pattern in
  `utils/mcp_tools.py` (allowlist, `prefix_` rename, admin re-check). Only the
  transport is new: an MCP `ClientSession` over the websocket's read/write
  streams instead of stdio.
- **The bridge runs in Rust, not in the webview.** The webview loads remote
  content, so giving page JS file access would turn any XSS into local file
  access. The Rust core owns the socket, the child processes, and the
  allowlist of exposed folders (picked in a native settings window).
- **Writes need consent:** tools tagged as writes (save file, move, run a
  Shortcut) show a native confirm dialog before running. Reads inside the
  scoped folders are allowed without asking.
- **Availability:** the agent is compiled once at startup, so `mac_*` tools are
  added per run, e.g. via `get_agent_for(user)` returning a variant with
  bridge tools when a bridge session is connected. If the bridge drops
  mid-call, the tool returns "Mac is offline" rather than hanging.
- **Auth:** the websocket authenticates with the same JWT, so this depends on
  the auth fix landing first. A blue/green flip drops the socket and the app
  reconnects to the new color.
- **Open spike question:** do the `mcp` Python SDK and `langchain-mcp-adapters`
  accept a custom transport cleanly? Fallback: a small hand-rolled JSON-RPC
  tool-call protocol with tools defined server-side.

## Auth fix (web + desktop)

Today the frontend sends the **refresh** token as the bearer on every call,
and 20 routes use `@jwt_refresh_token_required` (conversation ×9, beli ×6,
users ×2 incl. `/refresh`, rag ×1, app ×1, plus `admin_required` in
`blueprints/users/decorators.py`).

Change:

1. Backend: every route except `/api/user/refresh` uses `@jwt_required`
   (access token, 1h). The same goes for the inner decorator of
   `admin_required`.
2. Frontend: collapse `fetchWithRefreshToken` and `fetchWithAuth` into one
   function that sends the access token and, on 401, refreshes once and
   retries. `validateToken` keeps using the refresh token.
3. Desktop: the refresh token lives in the Keychain. The webview keeps only
   the access token (memory or `localStorage`).

This is blue/green-safe in both directions. An *old* frontend talking to the
new backend sends the refresh token, gets a 401, calls `/refresh`, and retries
with an access token, so it works. A *new* frontend talking to the old backend
sends an access token where a refresh token is required, gets a 401, and the
retry with a fresh access token also fails. That window lasts only as long as
the deploy flip. To avoid it, deploy the backend change first: for one release,
accept either token type on the protected routes (a small custom decorator),
then flip the frontend.

SSE streams are authenticated once at request start, so a stream running past
the 1h access-token expiry is fine.

Also in scope: move `_oidc_sessions` into Postgres (or signed, stateless
`state`) so a login spanning a deploy doesn't fail, and either honour or
delete the unused `redirect_after_login`.

## Backend changes

Ordered by how much the client depends on them. All are additive, so
blue/green stays safe: the old frontend ignores the new fields.

1. **Richer SSE events** (`blueprints/conversation/__init__.py`):
   - `tool_start {run_id, tool, label, input}`: `input` comes from
     `event["data"]["input"]`, trimmed/redacted; `label` comes from a
     server-side map that replaces `TOOL_MESSAGES`.
   - `tool_end {run_id, tool, ms, summary, artifact?}`: `run_id` lets parallel
     tool calls pair up correctly.
   - optional `event: id` lines so a client can resume (needed for phase 3).
2. **Structured artifacts.** Move high-value tools to
   `response_format="content_and_artifact"` (as `simba_search` already does): the
   LLM still sees compact text, and the UI gets
   `{kind, title, data}`. Persist them in an **`artifacts` table**:
   `id, user_id, conversation_id, message_id (nullable FK to the tool row),
   kind, title, data JSON, source_ref (e.g. Mealie slug, Obsidian path,
   Paperless id), version, created_at, updated_at`. Generate the migration with
   `make migrate-new` (never hand-written); a new table is backward-compatible.
   Endpoints: `GET /api/artifacts?conversation_id=` and
   `GET/PATCH /api/artifacts/<id>`. `PATCH` writes back to the source
   integration where one exists (edit a recipe → Mealie, edit a note →
   Obsidian) and bumps `version`. `GET /api/conversation/<id>` includes artifact
   ids on tool messages.
3. **Generic file upload** `POST /api/conversation/upload` (supersedes
   `upload-image`, which stays for compatibility): accept images, PDF, md/txt,
   CSV. Store in S3 under `conversations/<id>/`, extract text (reuse
   `utils/cleaner.py` for PDFs), and attach it as a document block capped to the
   context budget (32k slots). Possible extra: a "file it in Paperless" action
   that pushes the PDF into Paperless so it gets indexed. Revisit
   `MAX_CONTENT_LENGTH` (10 MB).
4. **Notifications feed.** A `notifications` table (user, kind, title, body,
   link, created_at, read_at) written by the scheduler (`process_due_messages`),
   cron runs, and completed detached runs. Add
   `GET /api/notifications/stream` (per-user SSE with `Last-Event-ID` replay
   from the table, so a blue/green flip only causes a reconnect) and
   `POST /api/notifications/<id>/read`. Also consider a `desktop` value on
   `MessageChannel` so a scheduled message can target the app instead of
   iMessage.
5. **Detached agent runs** (phase 3): `POST /api/runs` starts the agent in a
   background task and returns `run_id`; `GET /api/runs/<id>/events` streams
   with replay. This decouples work from the window. Constraint: an in-process
   run dies on a blue/green stop. Either persist events (Postgres) and mark
   runs `interrupted` on startup, or have `deploy.sh` drain active runs before
   stopping the old color.
6. **Native auth handoff** (phase 3, optional): sign in through the system
   browser (password manager, passkeys, existing Authelia session) without
   changing Authelia config:
   1. The app generates a PKCE verifier and opens
      `https://<simba-host>/api/user/oidc/login?client=desktop&challenge=…` in
      the default browser.
   2. The normal web OIDC round-trip runs against the existing redirect URI.
   3. The backend sees `client=desktop` on the stored session and, instead of
      returning JWTs to the page, issues a one-time 60-second handoff code and
      redirects to `simba://auth?code=…`.
   4. The app exchanges `code + verifier` at `POST /api/user/desktop/token` for
      JWTs and stores them in the Keychain.

   This also requires moving `_oidc_sessions` into Postgres or signed state so
   it survives blue/green. The alternative, registering a loopback or
   custom-scheme redirect URI in Authelia, depends on Authelia's redirect
   matching rules (exact match, ports?). That needs checking, and the client
   secret would have to stay on the server anyway.
7. **Local tool bridge:** `/api/desktop/bridge` websocket, an MCP-over-websocket
   client, and per-run injection of `mac_*` tools (see above). Admin-only.
8. **CORS:** only needed if we ever bundle the frontend (`quart-cors`
   allowing `tauri://localhost` / `http://tauri.localhost`). Not needed
   with the recommendation.

## Phased plan

| Phase | Scope | Backend | Exit criteria |
|---|---|---|---|
| **0a. Auth fix** (own PR) | access token on every call, single fetch helper, OIDC state in Postgres | auth fix (+ migration if state goes in a table) | Web works through a blue/green flip; the refresh token is only ever sent to `/refresh` |
| **0b. Spike** (½ day) | `desktop/` Tauri v2 app: one window loading `<simba-host>`, tray icon with Show/Quit, ⌥Space toggles the window, `offline.html` fallback | none | Log in through Authelia inside the webview, stream an answer, survive a `make redeploy` |
| **1. Native shell** | notifications, window-state, autostart, refresh token in the Keychain (no quick-ask window: dropped 2026-10-12) | none | A scheduled-message notification arrives; the login survives an app restart |
| **2. Cowork UI** ([plan](cowork-ui-plan.md)) | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised |
| **3. Local tools** | Rust bridge client, filesystem MCP scoped to chosen folders, write-confirm dialog, folder picker | item 7 | "Summarise the PDFs in ~/Downloads/taxes" works; a write prompts first; quitting the app removes the tools |
| **4. Background agent** | detached runs, notifications feed, desktop channel for scheduled messages, system-browser auth handoff | items 4–6 | Close the window mid-run → notification when done → reopen shows the full timeline |

**0b status:** scaffolded in `desktop/`. It has a bundled connect page that
probes `/api/health` and hands off to `$SIMBA_URL`, a menu-bar icon
(Show / New Chat / Reconnect / Quit), ⌥Space to toggle, close-to-hide,
single-instance, and Dock-click reopen. The remote origin gets no IPC.
`desktop/src-tauri/rust-toolchain.toml` pins Rust 1.90, the minimum
Tauri 2.12 accepts, so the global toolchain is untouched. Run it with
`make desktop-dev SIMBA_URL=…` or `make desktop`.
Verified 2026-10-12: Authelia login inside the webview works, and the
conversation list loads.

Phases 1–3 are independent once 0a/0b land. Phase 2 also improves the web
UI. There's no distribution phase: `make desktop` runs `tauri build` and
copies the ad-hoc-signed `.app` to `/Applications`. A locally built app isn't
quarantined, so Gatekeeper won't complain. Because the UI loads from the
server, rebuilds are only needed when the Rust shell changes.

## Open questions

1. **Keychain and ad-hoc signing:** the ad-hoc signature changes on every
   rebuild, so macOS may ask again for Keychain access after each
   `make desktop`. Acceptable, or use a self-signed local code-signing
   certificate to keep the identity stable?
2. **Local tool scope for v1:** filesystem (read + confirmed writes) only, or
   also screenshot, clipboard, "open URL/app", and running Shortcuts?
3. **MCP over websocket:** does `langchain-mcp-adapters` take a custom session
   or transport cleanly? Settle this in a short spike before phase 3;
   otherwise fall back to a hand-rolled tool-call protocol.
4. **Engine risk:** does anything in the UI rely on Chromium-only behaviour?
   The spike should check markdown rendering, streaming `fetch`, and the
   image picker in WKWebView.
5. **Artifact write-back conflicts:** if a recipe or note changed in
   Mealie/Obsidian after the artifact was captured, should we refuse, show a
   diff, or let the last write win?