Merge origin/main into feat/ynab-mirror

Resolve the Makefile .PHONY conflict (desktop + ynab targets) and regenerate the YNAB migration with aerich as 17 on top of main's 16_add_oidc_states, replacing the colliding 16_add_ynab_mirror. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 18:06:47 -04:00
parent d1c3b7693e 1a73e26f23
commit 10f3f1bcab
39 changed files with 6854 additions and 359 deletions
@@ -0,0 +1,229 @@
+# Cowork UI (phase 2) plan
+
+Status: **proposal, awaiting approval.** Parent doc: [desktop-client.md](desktop-client.md).
+
+Goal: make the agent's work visible and useful. Tool calls appear as a live
+activity timeline. Results that are more than text (budget breakdowns, recipes,
+notes, documents) open in a side panel. Files can be dropped into the
+conversation. This is all built in `raggr-frontend` and the Quart backend, so
+the web app gets it too; the Tauri shell needs only a one-line drag-drop
+setting.
+
+## Where we are
+
+- `stream-query` SSE sends `tool_start {tool}` and `tool_end {tool}` and nothing
+  else: no id, inputs, duration or result. The UI shows a one-line pill whose
+  label comes from a stale `TOOL_MESSAGES` map in `useChat.ts`.
+- Tools return formatted strings. `on_tool_end` saves them as `speaker="tool"`
+  messages, and `GET /conversation/<id>` returns only their first line
+  (`[tool result]`).
+- Only `simba_search` uses `response_format="content_and_artifact"`, and its
+  artifact is a list of LangChain `Document`s that nothing reads.
+- Auth: the backend accepting access tokens is merged (#57). The frontend
+  half (`fix/access-token-auth`) is waiting on a deploy.
+- **#55 (YNAB mirror) is open.** It replaces every YNAB tool (new
+  `blueprints/ynab/tools.py`: `ynab_budget_summary`, `ynab_account_balances`,
+  `ynab_search_transactions`, `ynab_spending_report`, `ynab_category_budget`,
+  `ynab_recent_changes`). ⚠️ Its migration is numbered `16_…`, which clashes
+  with #57's `16_…_add_oidc_states` already on main, so it must be
+  regenerated before #55 merges.
+
+## Contract
+
+Every task codes against this. It lands on main with this doc before any
+worker starts, so all workers share it.
+
+### SSE events (additive; old clients ignore new fields)
+
+```ts
+// tool_start
+{ type: "tool_start", run_id: string, tool: string, label: string,
+  input: Record<string, string | number | boolean> }   // values truncated to 200 chars, `config` dropped
+// tool_end
+{ type: "tool_end", run_id: string, tool: string, label: string,
+  status: "ok" | "error", ms: number,
+  artifact: ArtifactRef | null }
+// unchanged: thinking, thinking_delta, content, response, error, [DONE]
+```
+
+- `label` comes from a server-side registry
+  (`blueprints/conversation/tool_labels.py`): `{tool_name: "Checking budget…"}`,
+  with the fallback `"Using <tool>…"`. MCP tools match by prefix (`homebox_`,
+  `bookstack_`). `TOOL_MESSAGES` in `useChat.ts` is deleted.
+- `status="error"` comes from `on_tool_error`. Most tools catch exceptions and
+  return an error string, so they report `ok`; that's acceptable.
+- Parallel tool calls are paired by `run_id`.
+
+### Artifacts
+
+A tool opts in by returning `(text, artifact)` with
+`@tool(response_format="content_and_artifact")`. The LLM sees only `text`. The
+artifact is a plain JSON-serialisable dict:
+
+```ts
+type ToolArtifact = {            // what a tool returns
+  kind: ArtifactKind;
+  title: string;                 // "October spending by category"
+  data: object;                  // per-kind shape below; ≤ 256 KB serialised or it is dropped
+  source_ref?: { system: "mealie" | "obsidian" | "paperless" | "ynab" | "web";
+                 id?: string; path?: string; url?: string };
+};
+type ArtifactRef = { id: string; kind: ArtifactKind; title: string };   // in tool_end + conversation payload
+type Artifact = ArtifactRef & { data: object; source_ref: object | null; tool: string;
+                                conversation_id: string; message_id: string | null; created_at: string };
+```
+
+V1 kinds and their `data` shapes:
+
+| kind | produced by | data |
+|---|---|---|
+| `mealie.recipe` | `mealie_get_recipe` | `{slug, name, description, servings, prep_time, cook_time, total_time, ingredients: string[], instructions: {title?, text}[], tags: string[], image_url?}` |
+| `mealie.meal_plan` | `mealie_meal_plans`, `mealie_todays_meals` | `{start_date, end_date, days: {date, meals: {entry_type, name, recipe_slug?, note?}[]}[]}` |
+| `mealie.shopping_lists` | `mealie_shopping_lists` | `{lists: {name, items: {name, quantity, checked}[]}[]}` |
+| `obsidian.note` | `obsidian_read_note`, `journal_get_today` | `{path, title, markdown (≤ 50k chars), tags: string[], frontmatter: object}` |
+| `search.documents` | `simba_search`, `obsidian_search_notes` | `{query, source, results: {title, snippet (≤ 400), filename?, path?, document_type?, date?}[]}` |
+| `ynab.summary` | `ynab_budget_summary` | `{month, ready_to_assign, income, assigned, spent, age_of_money?, net_worth, overspent: {name, available}[]}` |
+| `ynab.transactions` | `ynab_search_transactions` | `{start_date, end_date, count, total, transactions: {date, payee, category, account, memo?, amount}[]}` (≤ 500 rows) |
+| `ynab.spending_report` | `ynab_spending_report` | `{start_date, end_date, group_by, rows: {key, amount, count?}[], total}` |
+| `ynab.category_budget` | `ynab_category_budget` | `{month, groups: {name, categories: {name, assigned, activity, available, goal_target?}[]}[]}` |
+
+YNAB kinds are built against #55's tools, so they wait for #55.
+`source_ref.url` is filled where the backend knows it (Mealie base URL plus the
+recipe path). Obsidian gets `obsidian://open?vault=<OBSIDIAN_VAULT_NAME>&file=<path>`
+only if that env var is set; it's added to `.env`, `.env.example` and
+`docker-compose.yml`.
+
+### Storage and API
+
+- New Tortoise model `Artifact` (`blueprints/artifacts/models.py`, table
+  `artifacts`): `id UUID, user FK, conversation FK (cascade), message FK →
+  conversation_messages (nullable, set null), tool, kind, title, data JSON,
+  source_ref JSON null, version int default 1, created_at, updated_at`.
+  Migration generated with `make migrate-new` / aerich, adds a table only.
+- Created in `stream_query`'s `on_tool_end` when `output.artifact` is a dict
+  with a `kind`, right after the tool message is saved and linked to it.
+  Non-web channels (iMessage, WhatsApp, `/query`) don't create artifacts.
+- `GET /api/artifacts?conversation_id=…` returns `ArtifactRef[]` plus
+  `{tool, message_id, created_at}`, without `data`.
+  `GET /api/artifacts/<id>` returns the full `Artifact`. Both require an access
+  token and are scoped to the owner (404 otherwise).
+- `GET /api/conversation/<id>`: tool messages gain
+  `tool: string, label: string, artifact: ArtifactRef | null`. `text` keeps
+  the header line, for older clients.
+- Write-back (`PATCH`) is **out of scope** for phase 2.
+
+### Uploads
+
+- `POST /api/conversation/upload` (multipart `file`, `conversation_id`):
+  - images go through the existing `process_image` and return
+    `{kind: "image", image_key}`
+  - `application/pdf`, `text/plain`, `text/markdown`, `text/csv` (by MIME type
+    or extension) are stored at `conversations/<id>/files/<uuid>.<ext>` and
+    return `{kind: "file", file_key, name, chars}`. PDF text is extracted with
+    PyMuPDF (`fitz`, already a dependency).
+  - max 10 MB; anything else gets a 415
+  - `upload-image` stays as is
+- `stream-query` accepts an optional `file_key`. The extracted text (capped at
+  12k chars, with a `[truncated]` marker) is appended to the user turn as
+  `<attached file "name">…</attached>`. The saved user message gets
+  `📎 name` appended. Saved user messages get no new column.
+
+## Tasks
+
+Each task has its own branch off main (after the contract lands), its own herdr
+worktree and pane, and its own Claude session started from a brief.
+
+| # | Branch / pane | Scope | Owns (files) | Depends on |
+|---|---|---|---|---|
+| **A** | `feat/cowork-stream` | SSE v2 events + label registry; `Artifact` model, migration, API, creation in `on_tool_end`; conversation payload fields | `blueprints/conversation/__init__.py`, new `tool_labels.py`, new `blueprints/artifacts/`, `config/db.py`, `app.py` (register), a migration | contract on main. **Migration last, after rebasing** (see sequencing) |
+| **B** | `feat/cowork-tool-artifacts` | `content_and_artifact` for the Mealie, Obsidian and search tools; then a second commit for the YNAB tools once #55 merges | `blueprints/conversation/agents.py`, `blueprints/ynab/tools.py` (2nd commit), tests | contract; #55 for the YNAB part |
+| **C** | `feat/cowork-timeline` | Activity timeline replaces `ToolBubble`: grouped "Simba worked · N steps · 4.2 s" block per turn, rows with icon, label, input summary, spinner/✓/✗, duration and an "Open" chip that calls an `onOpenArtifact(ref)` prop; restored from conversation payload on reload | `useChat.ts`, new `ActivityTimeline.tsx`, `ToolBubble.tsx` (delete), `conversationService.ts` types, `ChatScreen.tsx` (message list only) | frontend auth PR merged |
+| **D** | `feat/cowork-artifact-panel` | Right-side artifact panel: tabs for opened artifacts, `artifactService.ts`, a renderer per kind (CSS bars, no chart library), "Open in Mealie/Obsidian", a sheet below `lg`; auto-opens the newest artifact of a turn when the window is ≥ 1280 px wide, otherwise just the chip | new `components/artifacts/*`, `api/artifactService.ts`, `ChatScreen.tsx` (layout + wiring `onOpenArtifact`) | C merged first (wires its prop) |
+| **E** | `feat/cowork-file-drop` | Upload endpoint + `file_key` in `stream-query`; drop overlay over the whole window, attachment chip in `MessageInput`; Tauri `disable_drag_drop_handler()` so HTML5 drops reach the webview | `blueprints/conversation/__init__.py` (new route + small `stream_query` hunk), new `utils/file_extract.py`, `MessageInput.tsx`, `ChatScreen.tsx` (overlay), `useChat.ts` (pending file), `desktop/src-tauri/src/lib.rs` | rebase after A and C (shared files) |
+
+### Parallelism and sequencing
+
+```
+now ──► contract PR (this doc) merged
+        frontend-auth PR merged   (after your deploy)
+            │
+            ├── A  feat/cowork-stream ─────────────┐ (merges 1st)
+            ├── B  feat/cowork-tool-artifacts ─────┤ (Mealie/Obsidian/search: merges anytime; YNAB commit after #55)
+            ├── C  feat/cowork-timeline ───────────┤ (merges after A)
+            ├── D  feat/cowork-artifact-panel ─────┤ (merges after C)
+            └── E  feat/cowork-file-drop ──────────┘ (merges last)
+```
+
+- **All five start in parallel.** A and B are pure backend. C and D build
+  against a mock `/api` server that speaks the contract, so they don't wait for
+  A.
+- **Migration order.** #55's migration needs regenerating (it clashes with
+  #57). A's migration must be generated *after* whatever lands before it. A
+  writes the model and code first, then rebases onto the latest main and runs
+  `make migrate-new` as its final step. If #55 merges while A is in review, A
+  rebases and regenerates.
+- **Shared-file conflicts** (`ChatScreen.tsx`, `useChat.ts`,
+  `conversation/__init__.py`) are resolved by the merge order A → C → D → E. Each
+  later branch rebases onto main before its review.
+- B's Mealie/Obsidian/search commit is safe to merge before A: without A,
+  artifacts ride along in `ToolMessage.artifact` and are ignored.
+
+### Verification per task
+
+- **Backend (A, B, E):** unit tests in `tests/unit/`, run with
+  `TAVILY_API_KEY=dummy .venv/bin/pytest tests/ -q` (create the venv with
+  `uv sync --all-extras`).
+  - A: event shaping becomes pure functions (`tool_start_payload(event)`,
+    `tool_end_payload(event, started_at, artifact_ref)`), which are tested,
+    along with the 256 KB cap and owner scoping.
+  - B: each converted tool, with its service mocked, returns `(str, dict)`
+    matching the kind's shape.
+  - E: extraction and the 415 case.
+- **Migrations:** a throwaway Postgres container + aerich, the same approach
+  the auth fix used.
+- **Frontend (C, D, E):** `yarn build`, plus the mock-server + Playwright recipe
+  (Node `http` mock serving `dist/` and SSE fixtures, Playwright on installed
+  Chrome, `addInitScript` to seed tokens). Screenshots are checked by eye.
+  Fixtures live in `raggr-frontend/test-fixtures/cowork/*.json`, and D owns one
+  per kind. Don't commit the Playwright devDep or a rewritten
+  `package-lock.json`.
+
+## Process
+
+1. Approve this plan. I commit it on `plan/cowork-ui`, open the PR and merge it
+   so the contract is on main.
+2. Wait for the frontend-auth PR to merge after your deploy. A and B can start
+   right away; C, D and E start once it's in, so they use the new fetch helper.
+3. For each task: `herdr worktree create … --base origin/main`, a
+   `.claude-brief.md` with that task's slice of this doc, and
+   `herdr agent start cowork-<x> … -- claude "Read .claude-brief.md…"`.
+   Workers commit locally and never push.
+4. When a worker goes idle with its task done, I review:
+   - **pass 1:** I read the diff against the contract, run the tests and
+     build, and send numbered findings to the worker.
+   - **pass 2:** same again after its fixes.
+   - Then I push the branch and open the PR (`tea pr create`).
+   - I don't merge; you merge, or tell me to.
+5. Before each review, the branch rebases on main per the merge order.
+
+## Decisions I made (say if you disagree)
+
+- **No write-back in phase 2.** Editing a recipe or note in the panel and
+  saving it to Mealie/Obsidian is phase 2.5.
+- **No chart library.** CSS bars and tables are enough for budgets. Revisit if
+  trends need real line charts.
+- **The panel auto-opens on wide windows**, otherwise shows a chip.
+- **One attached file per message** (like images today). Dropped PDFs aren't
+  filed into Paperless yet.
+- **Out of scope for now:** the bridge, notifications, the Keychain, and
+  detached runs. They're phases 3–4 in the parent doc.
+
+## Open questions
+
+1. Is **#55** close to merging? It decides when the YNAB artifacts happen, and
+   its migration needs regenerating against main either way.
+2. What's the **Obsidian vault name** (for `obsidian://` links), or is the
+   link not worth having?
+3. What's the **Mealie recipe URL** format on your instance
+   (`/g/home/r/<slug>` or `/recipe/<slug>`)? Worker B can check against the
+   API otherwise.
@@ -0,0 +1,351 @@
+# Desktop Client (exploration)
+
+Status: **proposal**, no app code yet. Branch `explore/desktop-client`.
+
+Goal: a persistent, native-feeling macOS app where Simba works *alongside* the
+user rather than a chat box in a browser tab. "Cowork-like" means:
+
+- visible tool activity and progress while the agent works
+- side panels for artifacts: budget reports, recipes, notes, documents
+- drag files into the agent
+- desktop notifications (scheduled messages, cron jobs, finished runs)
+- a menu-bar icon and a global hotkey to summon the window
+
+## Decisions (2026-10-12)
+
+| Question | Decision | Consequence |
+|---|---|---|
+| Reachable off-LAN? | Yes, publicly via Caddy | Loading the remote origin works anywhere; `offline.html` only covers "no network / server down" |
+| Who uses it? | Just the admin (Ryan) | Every desktop-only endpoint is `@admin_required`; no multi-user notification fan-out, no distribution story |
+| Act on the Mac (local tools)? | **Yes** | Adds a *local tool bridge* (below). Tauri is still the pick: the bridge lives in the Rust core and can spawn local MCP servers as child processes |
+| Artifact storage | Own `artifacts` table | Artifacts can be listed, reopened, edited and written back |
+| Stop sending the refresh token on every request? | Yes, for web too | Becomes phase 0 (see "Auth fix") |
+| Apple Developer account? | No; a local dev build is fine | No notarization or updater. `make desktop` builds an ad-hoc-signed `.app` locally |
+
+## What exists today
+
+| Area | Current state | Desktop implication |
+|---|---|---|
+| Frontend | React 19 + Rsbuild + Tailwind, `BrowserRouter`, all API calls use relative `/api/...` | Reusable as-is if the shell loads it from the server origin; a bundled copy needs an API base URL and CORS |
+| PWA | `manifest.json` + `sw.js` (static cache, network-only `/api`) | "Install app" already works in Chrome / Safari "Add to Dock" |
+| Streaming | `POST /api/conversation/stream-query` → SSE over `fetch` with events `thinking`, `thinking_delta`, `tool_start {tool}`, `tool_end {tool}`, `content`, `response`, `error` | Too thin for Cowork: tool events have no run id, no args, no output, no structured payload |
+| Run lifecycle | The agent runs *inside* the HTTP response generator. A client disconnect cancels it (`CancelledError`, partial answer persisted) | You can't close the window and let Simba keep working |
+| Tool output | Tools return strings; output is persisted as `speaker="tool"` text (capped at 4000 chars) and the UI only shows the first line. `simba_search` already uses `response_format="content_and_artifact"` | Natural hook for structured artifacts |
+| Tool labels | `TOOL_MESSAGES` map in `useChat.ts` is hand-maintained and stale (no Mealie, SimbaKit, Homebox, BookStack, Beli…) | Labels should come from the server |
+| Uploads | `POST /api/conversation/upload-image`: images only, 10 MB, stored in S3, attached as an image block | Need generic file upload (PDF, text, markdown, CSV) |
+| Auth | Backend is the confidential OIDC client. `/oidc/login` returns an Authelia URL; Authelia redirects to `OIDC_REDIRECT_URI` (the web root); the SPA posts `code`+`state` to `/oidc/callback` and stores JWTs in `localStorage` | In-webview login works unchanged if the shell loads the real origin. A system-browser login needs a handoff (see below) |
+| Auth quirks | `_oidc_sessions` is an in-process dict (a login that spans a blue/green flip fails with "Invalid or expired state"); `redirect_after_login` is stored but never used; every API call sends the **30-day refresh token** as the bearer (`@jwt_refresh_token_required` everywhere) | Desktop must keep that token in the Keychain, not plain storage |
+| Notifications | Scheduled messages and cron jobs deliver to iMessage / email only (admin-only APIs) | No in-app notification channel exists |
+| CORS | None configured | Only matters for a bundled frontend |
+
+## Shell options
+
+### A. PWA ("Install app" / "Add to Dock")
+
+- **Reuse:** 100%. Already shipped.
+- **Size:** 0.
+- **macOS integration:** own window and Dock icon; Web Push and the Badging API
+  work for installed PWAs. **No** menu-bar icon, **no** global hotkey, no
+  quick-ask panel, no Keychain, file access only through the picker/drop.
+- **OIDC:** unchanged.
+- **Verdict:** a free baseline and the fallback. It can't deliver the
+  menu-bar or hotkey parts, which are most of what makes it feel like a
+  desktop app.
+
+### B. Tauri v2 (recommended)
+
+- **Reuse:** high. Either load the deployed frontend from the server origin
+  (no fork), or bundle `raggr-frontend/dist` (needs an API base URL plus CORS).
+- **Size:** ~5–15 MB `.app`, low idle memory (system WKWebView). Rust and
+  cargo are already installed on this machine.
+- **macOS integration:** first-party plugins for tray/menu-bar (core
+  `TrayIcon`), `global-shortcut`, `notification`, `deep-link`,
+  `single-instance`, `window-state`, `autostart`, `updater`, `positioner`
+  (anchor the quick-ask panel under the tray icon), keychain storage, and
+  window file drag-drop events with real paths. `ActivationPolicy::Accessory`
+  gives a menu-bar-only mode.
+- **Costs:** Safari/WebKit engine (the PWA already targets Safari, so low
+  risk). Service workers don't run in WKWebView here (the `sw.js` register
+  call already fails soft). Native code is Rust, but the native layer stays
+  small: tray, hotkey, notifications, keychain, deep link.
+- **Security:** remote content can reach Tauri IPC only through an explicit
+  `remote.urls` capability. Grant the remote origin only notification and
+  window commands; register shortcut and tray from Rust, not JS.
+
+### C. Electron
+
+- **Reuse:** same as Tauri.
+- **Size:** ~100 MB+ download, ~200 MB+ on disk, 150–300 MB RAM idle.
+- **macOS integration:** everything Tauri has, more mature (`menubar`
+  package, `globalShortcut`, `Notification`, `safeStorage`), plus Node in the
+  main process. That would matter if the client ran local tools or MCP
+  servers on the user's machine (see open questions).
+- **Verdict:** the better choice only if we want heavy client-side tool
+  execution in JS/Node. For a thin client to a self-hosted agent, the size
+  and memory cost buys nothing. Local tools are wanted, but the bridge design
+  below runs local MCP servers as child processes (Node is already installed),
+  so Electron's in-process Node isn't needed.
+
+### Decision matrix
+
+| | PWA | Tauri v2 | Electron |
+|---|---|---|---|
+| Reuse raggr-frontend | ✅ as-is | ✅ remote or bundled | ✅ remote or bundled |
+| Bundle / RAM | none | ~10 MB / low | ~100 MB+ / high |
+| Menu bar + quick-ask | ❌ | ✅ | ✅ |
+| Global hotkey | ❌ | ✅ | ✅ |
+| Notifications | Web Push (VAPID) | ✅ native | ✅ native |
+| File drag-drop w/ paths | browser `File` only | ✅ | ✅ |
+| Secure token storage | ❌ localStorage | ✅ Keychain | ✅ safeStorage |
+| Auto-update | free (server) | `updater` plugin, or free with a remote UI | `electron-updater` |
+| Local tool execution | ❌ | Rust | Node (easiest) |
+
+## Recommended architecture
+
+**A Tauri v2 shell that loads the deployed SimbaRAG origin, plus a tiny bundled
+"offline/connect" page. Cowork features are built in `raggr-frontend` behind
+feature detection, so the web app gets them too.**
+
+```
+┌──────────────────────── Simba.app (Tauri v2) ─────────────────────────┐
+│ Rust core (small)                                                      │
+│  • tray icon + menu (Ask…, New chat, recent convos, Quit)              │
+│  • global shortcut ⌥Space → show/hide main window                      │
+│  • notifications (plugin-notification)                                 │
+│  • keychain token store, deep-link `simba://`, single-instance         │
+│  • window-state, autostart                                             │
+│                                                                        │
+│  Main window  ──loads──▶  https://<simba-host>/        (full UI)       │
+│  Fallback     ──bundled─▶ offline.html (server unreachable / retry)    │
+└───────────────────────────────┬────────────────────────────────────────┘
+                                │ HTTPS: REST + SSE (same origin, no CORS)
+┌───────────────────────────────▼────────────────────────────────────────┐
+│ Caddy → raggr-blue / raggr-green (Quart)                               │
+│  /api/conversation/*  (+ structured tool/artifact events)              │
+│  /api/runs/*          (detached agent runs, resumable SSE)  [phase 3]  │
+│  /api/files           (generic upload)                                 │
+│  /api/notifications/stream (per-user SSE feed)                         │
+└────────────────────────────────────────────────────────────────────────┘
+```
+
+Why load the remote origin rather than bundle:
+
+- **One codebase, one deploy.** `make redeploy` updates web, PWA and desktop
+  together. No version skew between the desktop UI and the SSE protocol, and
+  no desktop release for UI changes.
+- **No CORS, no API-base refactor, and the current OIDC flow works unchanged**
+  inside the webview (redirect back to the web root on the same origin).
+- The agent needs the server anyway, so offline-first buys little. The bundled
+  `offline.html` covers "server down / not on VPN".
+
+The frontend detects the shell with `window.__TAURI_INTERNALS__` (or a
+`?shell=desktop` query param) and turns on the native bits: notifications
+and drag-drop paths. Everything else renders
+the same in a browser.
+
+### Cowork UI (in raggr-frontend, shared by web and desktop)
+
+- **Three panes:** conversations | chat | **artifact panel** (collapsible, a
+  tab per artifact). On mobile/web the artifact panel becomes a sheet.
+- **Activity timeline** in place of single-line `ToolBubble` pills: one row per
+  tool call with a server-provided label, args summary, a spinner, duration,
+  and an expandable result. Clicking a row with an artifact opens it in the
+  panel.
+- **Artifact renderers:** `budget` (YNAB summary/category table + chart),
+  `transactions` (table), `recipe` (Mealie: ingredients/steps, "add to meal
+  plan"), `note` (Obsidian markdown, "open in Obsidian" via
+  `obsidian://` deep link), `document` (Paperless hit: title, snippet, link to
+  Paperless), `photo` (Immich), `device` (SimbaKit status/weight chart).
+- **Drop zone** over the whole window: images go to the existing vision path;
+  PDFs and text go to the new upload endpoint (below).
+
+### Local tool bridge (Simba acts on the Mac)
+
+The backend can't reach the Mac (it's behind NAT), so the desktop app dials
+out and the backend calls tools back over that connection.
+
+```
+Mac                                              Server
+┌─ Simba.app Rust core ────────────┐   wss    ┌─ Quart ────────────────────────┐
+│ bridge client ───────────────────┼─────────▶│ /api/desktop/bridge (websocket)│
+│   │ JSON-RPC (MCP messages)      │          │   │                            │
+│   ▼                              │          │   ▼                            │
+│ local MCP servers (child procs)  │          │ MCP ClientSession over the ws  │
+│  • filesystem (scoped folders)   │          │  → langchain-mcp-adapters tools│
+│  • later: screenshot, clipboard, │          │  → mcp_tools.select_tools(     │
+│    open app/URL, Shortcuts       │          │      prefix="mac", admin-only) │
+│ confirm dialog for write tools   │          │  → injected per run only while │
+└──────────────────────────────────┘          │    the bridge is connected     │
+                                              └────────────────────────────────┘
+```
+
+- **The protocol is MCP**, so the backend reuses the existing pattern in
+  `utils/mcp_tools.py` (allowlist, `prefix_` rename, admin re-check). Only the
+  transport is new: an MCP `ClientSession` over the websocket's read/write
+  streams instead of stdio.
+- **The bridge runs in Rust, not in the webview.** The webview loads remote
+  content, so giving page JS file access would turn any XSS into local file
+  access. The Rust core owns the socket, the child processes, and the
+  allowlist of exposed folders (picked in a native settings window).
+- **Writes need consent:** tools tagged as writes (save file, move, run a
+  Shortcut) show a native confirm dialog before running. Reads inside the
+  scoped folders are allowed without asking.
+- **Availability:** the agent is compiled once at startup, so `mac_*` tools are
+  added per run, e.g. via `get_agent_for(user)` returning a variant with
+  bridge tools when a bridge session is connected. If the bridge drops
+  mid-call, the tool returns "Mac is offline" rather than hanging.
+- **Auth:** the websocket authenticates with the same JWT, so this depends on
+  the auth fix landing first. A blue/green flip drops the socket and the app
+  reconnects to the new color.
+- **Open spike question:** do the `mcp` Python SDK and `langchain-mcp-adapters`
+  accept a custom transport cleanly? Fallback: a small hand-rolled JSON-RPC
+  tool-call protocol with tools defined server-side.
+
+## Auth fix (web + desktop)
+
+Today the frontend sends the **refresh** token as the bearer on every call,
+and 20 routes use `@jwt_refresh_token_required` (conversation ×9, beli ×6,
+users ×2 incl. `/refresh`, rag ×1, app ×1, plus `admin_required` in
+`blueprints/users/decorators.py`).
+
+Change:
+
+1. Backend: every route except `/api/user/refresh` uses `@jwt_required`
+   (access token, 1h). The same goes for the inner decorator of
+   `admin_required`.
+2. Frontend: collapse `fetchWithRefreshToken` and `fetchWithAuth` into one
+   function that sends the access token and, on 401, refreshes once and
+   retries. `validateToken` keeps using the refresh token.
+3. Desktop: the refresh token lives in the Keychain. The webview keeps only
+   the access token (memory or `localStorage`).
+
+This is blue/green-safe in both directions. An *old* frontend talking to the
+new backend sends the refresh token, gets a 401, calls `/refresh`, and retries
+with an access token, so it works. A *new* frontend talking to the old backend
+sends an access token where a refresh token is required, gets a 401, and the
+retry with a fresh access token also fails. That window lasts only as long as
+the deploy flip. To avoid it, deploy the backend change first: for one release,
+accept either token type on the protected routes (a small custom decorator),
+then flip the frontend.
+
+SSE streams are authenticated once at request start, so a stream running past
+the 1h access-token expiry is fine.
+
+Also in scope: move `_oidc_sessions` into Postgres (or signed, stateless
+`state`) so a login spanning a deploy doesn't fail, and either honour or
+delete the unused `redirect_after_login`.
+
+## Backend changes
+
+Ordered by how much the client depends on them. All are additive, so
+blue/green stays safe: the old frontend ignores the new fields.
+
+1. **Richer SSE events** (`blueprints/conversation/__init__.py`):
+   - `tool_start {run_id, tool, label, input}`: `input` comes from
+     `event["data"]["input"]`, trimmed/redacted; `label` comes from a
+     server-side map that replaces `TOOL_MESSAGES`.
+   - `tool_end {run_id, tool, ms, summary, artifact?}`: `run_id` lets parallel
+     tool calls pair up correctly.
+   - optional `event: id` lines so a client can resume (needed for phase 3).
+2. **Structured artifacts.** Move high-value tools to
+   `response_format="content_and_artifact"` (as `simba_search` already does): the
+   LLM still sees compact text, and the UI gets
+   `{kind, title, data}`. Persist them in an **`artifacts` table**:
+   `id, user_id, conversation_id, message_id (nullable FK to the tool row),
+   kind, title, data JSON, source_ref (e.g. Mealie slug, Obsidian path,
+   Paperless id), version, created_at, updated_at`. Generate the migration with
+   `make migrate-new` (never hand-written); a new table is backward-compatible.
+   Endpoints: `GET /api/artifacts?conversation_id=` and
+   `GET/PATCH /api/artifacts/<id>`. `PATCH` writes back to the source
+   integration where one exists (edit a recipe → Mealie, edit a note →
+   Obsidian) and bumps `version`. `GET /api/conversation/<id>` includes artifact
+   ids on tool messages.
+3. **Generic file upload** `POST /api/conversation/upload` (supersedes
+   `upload-image`, which stays for compatibility): accept images, PDF, md/txt,
+   CSV. Store in S3 under `conversations/<id>/`, extract text (reuse
+   `utils/cleaner.py` for PDFs), and attach it as a document block capped to the
+   context budget (32k slots). Possible extra: a "file it in Paperless" action
+   that pushes the PDF into Paperless so it gets indexed. Revisit
+   `MAX_CONTENT_LENGTH` (10 MB).
+4. **Notifications feed.** A `notifications` table (user, kind, title, body,
+   link, created_at, read_at) written by the scheduler (`process_due_messages`),
+   cron runs, and completed detached runs. Add
+   `GET /api/notifications/stream` (per-user SSE with `Last-Event-ID` replay
+   from the table, so a blue/green flip only causes a reconnect) and
+   `POST /api/notifications/<id>/read`. Also consider a `desktop` value on
+   `MessageChannel` so a scheduled message can target the app instead of
+   iMessage.
+5. **Detached agent runs** (phase 3): `POST /api/runs` starts the agent in a
+   background task and returns `run_id`; `GET /api/runs/<id>/events` streams
+   with replay. This decouples work from the window. Constraint: an in-process
+   run dies on a blue/green stop. Either persist events (Postgres) and mark
+   runs `interrupted` on startup, or have `deploy.sh` drain active runs before
+   stopping the old color.
+6. **Native auth handoff** (phase 3, optional): sign in through the system
+   browser (password manager, passkeys, existing Authelia session) without
+   changing Authelia config:
+   1. The app generates a PKCE verifier and opens
+      `https://<simba-host>/api/user/oidc/login?client=desktop&challenge=…` in
+      the default browser.
+   2. The normal web OIDC round-trip runs against the existing redirect URI.
+   3. The backend sees `client=desktop` on the stored session and, instead of
+      returning JWTs to the page, issues a one-time 60-second handoff code and
+      redirects to `simba://auth?code=…`.
+   4. The app exchanges `code + verifier` at `POST /api/user/desktop/token` for
+      JWTs and stores them in the Keychain.
+
+   This also requires moving `_oidc_sessions` into Postgres or signed state so
+   it survives blue/green. The alternative, registering a loopback or
+   custom-scheme redirect URI in Authelia, depends on Authelia's redirect
+   matching rules (exact match, ports?). That needs checking, and the client
+   secret would have to stay on the server anyway.
+7. **Local tool bridge:** `/api/desktop/bridge` websocket, an MCP-over-websocket
+   client, and per-run injection of `mac_*` tools (see above). Admin-only.
+8. **CORS:** only needed if we ever bundle the frontend (`quart-cors`
+   allowing `tauri://localhost` / `http://tauri.localhost`). Not needed
+   with the recommendation.
+
+## Phased plan
+
+| Phase | Scope | Backend | Exit criteria |
+|---|---|---|---|
+| **0a. Auth fix** (own PR) | access token on every call, single fetch helper, OIDC state in Postgres | auth fix (+ migration if state goes in a table) | Web works through a blue/green flip; the refresh token is only ever sent to `/refresh` |
+| **0b. Spike** (½ day) | `desktop/` Tauri v2 app: one window loading `<simba-host>`, tray icon with Show/Quit, ⌥Space toggles the window, `offline.html` fallback | none | Log in through Authelia inside the webview, stream an answer, survive a `make redeploy` |
+| **1. Native shell** | notifications, window-state, autostart, refresh token in the Keychain (no quick-ask window: dropped 2026-10-12) | none | A scheduled-message notification arrives; the login survives an app restart |
+| **2. Cowork UI** ([plan](cowork-ui-plan.md)) | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised |
+| **3. Local tools** | Rust bridge client, filesystem MCP scoped to chosen folders, write-confirm dialog, folder picker | item 7 | "Summarise the PDFs in ~/Downloads/taxes" works; a write prompts first; quitting the app removes the tools |
+| **4. Background agent** | detached runs, notifications feed, desktop channel for scheduled messages, system-browser auth handoff | items 4–6 | Close the window mid-run → notification when done → reopen shows the full timeline |
+
+**0b status:** scaffolded in `desktop/`. It has a bundled connect page that
+probes `/api/health` and hands off to `$SIMBA_URL`, a menu-bar icon
+(Show / New Chat / Reconnect / Quit), ⌥Space to toggle, close-to-hide,
+single-instance, and Dock-click reopen. The remote origin gets no IPC.
+`desktop/src-tauri/rust-toolchain.toml` pins Rust 1.90, the minimum
+Tauri 2.12 accepts, so the global toolchain is untouched. Run it with
+`make desktop-dev SIMBA_URL=…` or `make desktop`.
+Verified 2026-10-12: Authelia login inside the webview works, and the
+conversation list loads.
+
+Phases 1–3 are independent once 0a/0b land. Phase 2 also improves the web
+UI. There's no distribution phase: `make desktop` runs `tauri build` and
+copies the ad-hoc-signed `.app` to `/Applications`. A locally built app isn't
+quarantined, so Gatekeeper won't complain. Because the UI loads from the
+server, rebuilds are only needed when the Rust shell changes.
+
+## Open questions
+
+1. **Keychain and ad-hoc signing:** the ad-hoc signature changes on every
+   rebuild, so macOS may ask again for Keychain access after each
+   `make desktop`. Acceptable, or use a self-signed local code-signing
+   certificate to keep the identity stable?
+2. **Local tool scope for v1:** filesystem (read + confirmed writes) only, or
+   also screenshot, clipboard, "open URL/app", and running Shortcuts?
+3. **MCP over websocket:** does `langchain-mcp-adapters` take a custom session
+   or transport cleanly? Settle this in a short spike before phase 3;
+   otherwise fall back to a hand-rolled tool-call protocol.
+4. **Engine risk:** does anything in the UI rely on Chromium-only behaviour?
+   The spike should check markdown rendering, streaming `fetch`, and the
+   image picker in WKWebView.
+5. **Artifact write-back conflicts:** if a recipe or note changed in
+   Mealie/Obsidian after the artifact was captured, should we refuse, show a
+   diff, or let the last write win?