docs/cowork-ui-plan.md

@@ -0,0 +1,229 @@
+# Cowork UI (phase 2) plan
+
+Status: **proposal, awaiting approval.** Parent doc: [desktop-client.md](desktop-client.md).
+
+Goal: make the agent's work visible and useful. Tool calls appear as a live
+activity timeline. Results that are more than text (budget breakdowns, recipes,
+notes, documents) open in a side panel. Files can be dropped into the
+conversation. This is all built in `raggr-frontend` and the Quart backend, so
+the web app gets it too; the Tauri shell needs only a one-line drag-drop
+setting.
+
+## Where we are
+
+- `stream-query` SSE sends `tool_start {tool}` and `tool_end {tool}` and nothing
+  else: no id, inputs, duration or result. The UI shows a one-line pill whose
+  label comes from a stale `TOOL_MESSAGES` map in `useChat.ts`.
+- Tools return formatted strings. `on_tool_end` saves them as `speaker="tool"`
+  messages, and `GET /conversation/<id>` returns only their first line
+  (`[tool result]`).
+- Only `simba_search` uses `response_format="content_and_artifact"`, and its
+  artifact is a list of LangChain `Document`s that nothing reads.
+- Auth: the backend accepting access tokens is merged (#57). The frontend
+  half (`fix/access-token-auth`) is waiting on a deploy.
+- **#55 (YNAB mirror) is open.** It replaces every YNAB tool (new
+  `blueprints/ynab/tools.py`: `ynab_budget_summary`, `ynab_account_balances`,
+  `ynab_search_transactions`, `ynab_spending_report`, `ynab_category_budget`,
+  `ynab_recent_changes`). ⚠️ Its migration is numbered `16_…`, which clashes
+  with #57's `16_…_add_oidc_states` already on main, so it must be
+  regenerated before #55 merges.
+
+## Contract
+
+Every task codes against this. It lands on main with this doc before any
+worker starts, so all workers share it.
+
+### SSE events (additive; old clients ignore new fields)
+
+```ts
+// tool_start
+{ type: "tool_start", run_id: string, tool: string, label: string,
+  input: Record<string, string | number | boolean> }   // values truncated to 200 chars, `config` dropped
+// tool_end
+{ type: "tool_end", run_id: string, tool: string, label: string,
+  status: "ok" | "error", ms: number,
+  artifact: ArtifactRef | null }
+// unchanged: thinking, thinking_delta, content, response, error, [DONE]
+```
+
+- `label` comes from a server-side registry
+  (`blueprints/conversation/tool_labels.py`): `{tool_name: "Checking budget…"}`,
+  with the fallback `"Using <tool>…"`. MCP tools match by prefix (`homebox_`,
+  `bookstack_`). `TOOL_MESSAGES` in `useChat.ts` is deleted.
+- `status="error"` comes from `on_tool_error`. Most tools catch exceptions and
+  return an error string, so they report `ok`; that's acceptable.
+- Parallel tool calls are paired by `run_id`.
+
+### Artifacts
+
+A tool opts in by returning `(text, artifact)` with
+`@tool(response_format="content_and_artifact")`. The LLM sees only `text`. The
+artifact is a plain JSON-serialisable dict:
+
+```ts
+type ToolArtifact = {            // what a tool returns
+  kind: ArtifactKind;
+  title: string;                 // "October spending by category"
+  data: object;                  // per-kind shape below; ≤ 256 KB serialised or it is dropped
+  source_ref?: { system: "mealie" | "obsidian" | "paperless" | "ynab" | "web";
+                 id?: string; path?: string; url?: string };
+};
+type ArtifactRef = { id: string; kind: ArtifactKind; title: string };   // in tool_end + conversation payload
+type Artifact = ArtifactRef & { data: object; source_ref: object | null; tool: string;
+                                conversation_id: string; message_id: string | null; created_at: string };
+```
+
+V1 kinds and their `data` shapes:
+
+| kind | produced by | data |
+|---|---|---|
+| `mealie.recipe` | `mealie_get_recipe` | `{slug, name, description, servings, prep_time, cook_time, total_time, ingredients: string[], instructions: {title?, text}[], tags: string[], image_url?}` |
+| `mealie.meal_plan` | `mealie_meal_plans`, `mealie_todays_meals` | `{start_date, end_date, days: {date, meals: {entry_type, name, recipe_slug?, note?}[]}[]}` |
+| `mealie.shopping_lists` | `mealie_shopping_lists` | `{lists: {name, items: {name, quantity, checked}[]}[]}` |
+| `obsidian.note` | `obsidian_read_note`, `journal_get_today` | `{path, title, markdown (≤ 50k chars), tags: string[], frontmatter: object}` |
+| `search.documents` | `simba_search`, `obsidian_search_notes` | `{query, source, results: {title, snippet (≤ 400), filename?, path?, document_type?, date?}[]}` |
+| `ynab.summary` | `ynab_budget_summary` | `{month, ready_to_assign, income, assigned, spent, age_of_money?, net_worth, overspent: {name, available}[]}` |
+| `ynab.transactions` | `ynab_search_transactions` | `{start_date, end_date, count, total, transactions: {date, payee, category, account, memo?, amount}[]}` (≤ 500 rows) |
+| `ynab.spending_report` | `ynab_spending_report` | `{start_date, end_date, group_by, rows: {key, amount, count?}[], total}` |
+| `ynab.category_budget` | `ynab_category_budget` | `{month, groups: {name, categories: {name, assigned, activity, available, goal_target?}[]}[]}` |
+
+YNAB kinds are built against #55's tools, so they wait for #55.
+`source_ref.url` is filled where the backend knows it (Mealie base URL plus the
+recipe path). Obsidian gets `obsidian://open?vault=<OBSIDIAN_VAULT_NAME>&file=<path>`
+only if that env var is set; it's added to `.env`, `.env.example` and
+`docker-compose.yml`.
+
+### Storage and API
+
+- New Tortoise model `Artifact` (`blueprints/artifacts/models.py`, table
+  `artifacts`): `id UUID, user FK, conversation FK (cascade), message FK →
+  conversation_messages (nullable, set null), tool, kind, title, data JSON,
+  source_ref JSON null, version int default 1, created_at, updated_at`.
+  Migration generated with `make migrate-new` / aerich, adds a table only.
+- Created in `stream_query`'s `on_tool_end` when `output.artifact` is a dict
+  with a `kind`, right after the tool message is saved and linked to it.
+  Non-web channels (iMessage, WhatsApp, `/query`) don't create artifacts.
+- `GET /api/artifacts?conversation_id=…` returns `ArtifactRef[]` plus
+  `{tool, message_id, created_at}`, without `data`.
+  `GET /api/artifacts/<id>` returns the full `Artifact`. Both require an access
+  token and are scoped to the owner (404 otherwise).
+- `GET /api/conversation/<id>`: tool messages gain
+  `tool: string, label: string, artifact: ArtifactRef | null`. `text` keeps
+  the header line, for older clients.
+- Write-back (`PATCH`) is **out of scope** for phase 2.
+
+### Uploads
+
+- `POST /api/conversation/upload` (multipart `file`, `conversation_id`):
+  - images go through the existing `process_image` and return
+    `{kind: "image", image_key}`
+  - `application/pdf`, `text/plain`, `text/markdown`, `text/csv` (by MIME type
+    or extension) are stored at `conversations/<id>/files/<uuid>.<ext>` and
+    return `{kind: "file", file_key, name, chars}`. PDF text is extracted with
+    PyMuPDF (`fitz`, already a dependency).
+  - max 10 MB; anything else gets a 415
+  - `upload-image` stays as is
+- `stream-query` accepts an optional `file_key`. The extracted text (capped at
+  12k chars, with a `[truncated]` marker) is appended to the user turn as
+  `<attached file "name">…</attached>`. The saved user message gets
+  `📎 name` appended. Saved user messages get no new column.
+
+## Tasks
+
+Each task has its own branch off main (after the contract lands), its own herdr
+worktree and pane, and its own Claude session started from a brief.
+
+| # | Branch / pane | Scope | Owns (files) | Depends on |
+|---|---|---|---|---|
+| **A** | `feat/cowork-stream` | SSE v2 events + label registry; `Artifact` model, migration, API, creation in `on_tool_end`; conversation payload fields | `blueprints/conversation/__init__.py`, new `tool_labels.py`, new `blueprints/artifacts/`, `config/db.py`, `app.py` (register), a migration | contract on main. **Migration last, after rebasing** (see sequencing) |
+| **B** | `feat/cowork-tool-artifacts` | `content_and_artifact` for the Mealie, Obsidian and search tools; then a second commit for the YNAB tools once #55 merges | `blueprints/conversation/agents.py`, `blueprints/ynab/tools.py` (2nd commit), tests | contract; #55 for the YNAB part |
+| **C** | `feat/cowork-timeline` | Activity timeline replaces `ToolBubble`: grouped "Simba worked · N steps · 4.2 s" block per turn, rows with icon, label, input summary, spinner/✓/✗, duration and an "Open" chip that calls an `onOpenArtifact(ref)` prop; restored from conversation payload on reload | `useChat.ts`, new `ActivityTimeline.tsx`, `ToolBubble.tsx` (delete), `conversationService.ts` types, `ChatScreen.tsx` (message list only) | frontend auth PR merged |
+| **D** | `feat/cowork-artifact-panel` | Right-side artifact panel: tabs for opened artifacts, `artifactService.ts`, a renderer per kind (CSS bars, no chart library), "Open in Mealie/Obsidian", a sheet below `lg`; auto-opens the newest artifact of a turn when the window is ≥ 1280 px wide, otherwise just the chip | new `components/artifacts/*`, `api/artifactService.ts`, `ChatScreen.tsx` (layout + wiring `onOpenArtifact`) | C merged first (wires its prop) |
+| **E** | `feat/cowork-file-drop` | Upload endpoint + `file_key` in `stream-query`; drop overlay over the whole window, attachment chip in `MessageInput`; Tauri `disable_drag_drop_handler()` so HTML5 drops reach the webview | `blueprints/conversation/__init__.py` (new route + small `stream_query` hunk), new `utils/file_extract.py`, `MessageInput.tsx`, `ChatScreen.tsx` (overlay), `useChat.ts` (pending file), `desktop/src-tauri/src/lib.rs` | rebase after A and C (shared files) |
+
+### Parallelism and sequencing
+
+```
+now ──► contract PR (this doc) merged
+        frontend-auth PR merged   (after your deploy)
+            │
+            ├── A  feat/cowork-stream ─────────────┐ (merges 1st)
+            ├── B  feat/cowork-tool-artifacts ─────┤ (Mealie/Obsidian/search: merges anytime; YNAB commit after #55)
+            ├── C  feat/cowork-timeline ───────────┤ (merges after A)
+            ├── D  feat/cowork-artifact-panel ─────┤ (merges after C)
+            └── E  feat/cowork-file-drop ──────────┘ (merges last)
+```
+
+- **All five start in parallel.** A and B are pure backend. C and D build
+  against a mock `/api` server that speaks the contract, so they don't wait for
+  A.
+- **Migration order.** #55's migration needs regenerating (it clashes with
+  #57). A's migration must be generated *after* whatever lands before it. A
+  writes the model and code first, then rebases onto the latest main and runs
+  `make migrate-new` as its final step. If #55 merges while A is in review, A
+  rebases and regenerates.
+- **Shared-file conflicts** (`ChatScreen.tsx`, `useChat.ts`,
+  `conversation/__init__.py`) are resolved by the merge order A → C → D → E. Each
+  later branch rebases onto main before its review.
+- B's Mealie/Obsidian/search commit is safe to merge before A: without A,
+  artifacts ride along in `ToolMessage.artifact` and are ignored.
+
+### Verification per task
+
+- **Backend (A, B, E):** unit tests in `tests/unit/`, run with
+  `TAVILY_API_KEY=dummy .venv/bin/pytest tests/ -q` (create the venv with
+  `uv sync --all-extras`).
+  - A: event shaping becomes pure functions (`tool_start_payload(event)`,
+    `tool_end_payload(event, started_at, artifact_ref)`), which are tested,
+    along with the 256 KB cap and owner scoping.
+  - B: each converted tool, with its service mocked, returns `(str, dict)`
+    matching the kind's shape.
+  - E: extraction and the 415 case.
+- **Migrations:** a throwaway Postgres container + aerich, the same approach
+  the auth fix used.
+- **Frontend (C, D, E):** `yarn build`, plus the mock-server + Playwright recipe
+  (Node `http` mock serving `dist/` and SSE fixtures, Playwright on installed
+  Chrome, `addInitScript` to seed tokens). Screenshots are checked by eye.
+  Fixtures live in `raggr-frontend/test-fixtures/cowork/*.json`, and D owns one
+  per kind. Don't commit the Playwright devDep or a rewritten
+  `package-lock.json`.
+
+## Process
+
+1. Approve this plan. I commit it on `plan/cowork-ui`, open the PR and merge it
+   so the contract is on main.
+2. Wait for the frontend-auth PR to merge after your deploy. A and B can start
+   right away; C, D and E start once it's in, so they use the new fetch helper.
+3. For each task: `herdr worktree create … --base origin/main`, a
+   `.claude-brief.md` with that task's slice of this doc, and
+   `herdr agent start cowork-<x> … -- claude "Read .claude-brief.md…"`.
+   Workers commit locally and never push.
+4. When a worker goes idle with its task done, I review:
+   - **pass 1:** I read the diff against the contract, run the tests and
+     build, and send numbered findings to the worker.
+   - **pass 2:** same again after its fixes.
+   - Then I push the branch and open the PR (`tea pr create`).
+   - I don't merge; you merge, or tell me to.
+5. Before each review, the branch rebases on main per the merge order.
+
+## Decisions I made (say if you disagree)
+
+- **No write-back in phase 2.** Editing a recipe or note in the panel and
+  saving it to Mealie/Obsidian is phase 2.5.
+- **No chart library.** CSS bars and tables are enough for budgets. Revisit if
+  trends need real line charts.
+- **The panel auto-opens on wide windows**, otherwise shows a chip.
+- **One attached file per message** (like images today). Dropped PDFs aren't
+  filed into Paperless yet.
+- **Out of scope for now:** the bridge, notifications, the Keychain, and
+  detached runs. They're phases 3–4 in the parent doc.
+
+## Open questions
+
+1. Is **#55** close to merging? It decides when the YNAB artifacts happen, and
+   its migration needs regenerating against main either way.
+2. What's the **Obsidian vault name** (for `obsidian://` links), or is the
+   link not worth having?
+3. What's the **Mealie recipe URL** format on your instance
+   (`/g/home/r/<slug>` or `/recipe/<slug>`)? Worker B can check against the
+   API otherwise.

docs/desktop-client.md

@@ -9,7 +9,7 @@ user rather than a chat box in a browser tab. "Cowork-like" means:
 - side panels for artifacts: budget reports, recipes, notes, documents
 - drag files into the agent
 - desktop notifications (scheduled messages, cron jobs, finished runs)
-- a menu-bar icon and a global-hotkey quick-ask window
+- a menu-bar icon and a global hotkey to summon the window
 
 ## Decisions (2026-10-12)
 
@@ -110,13 +110,12 @@ feature detection, so the web app gets them too.**
 ┌──────────────────────── Simba.app (Tauri v2) ─────────────────────────┐
 │ Rust core (small)                                                      │
 │  • tray icon + menu (Ask…, New chat, recent convos, Quit)              │
-│  • global shortcut ⌥Space → quick-ask panel (positioner, always-on-top)│
+│  • global shortcut ⌥Space → show/hide main window                      │
 │  • notifications (plugin-notification)                                 │
 │  • keychain token store, deep-link `simba://`, single-instance         │
 │  • window-state, autostart                                             │
 │                                                                        │
 │  Main window  ──loads──▶  https://<simba-host>/        (full UI)       │
-│  Quick panel  ──loads──▶  https://<simba-host>/quick   (compact route) │
 │  Fallback     ──bundled─▶ offline.html (server unreachable / retry)    │
 └───────────────────────────────┬────────────────────────────────────────┘
                                 │ HTTPS: REST + SSE (same origin, no CORS)
@@ -140,8 +139,8 @@ Why load the remote origin rather than bundle:
   `offline.html` covers "server down / not on VPN".
 
 The frontend detects the shell with `window.__TAURI_INTERNALS__` (or a
-`?shell=desktop` query param) and turns on the native bits: notifications,
-drag-drop paths, and the "open in quick panel" route. Everything else renders
+`?shell=desktop` query param) and turns on the native bits: notifications
+and drag-drop paths. Everything else renders
 the same in a browser.
 
 ### Cowork UI (in raggr-frontend, shared by web and desktop)
@@ -159,8 +158,6 @@ the same in a browser.
   Paperless), `photo` (Immich), `device` (SimbaKit status/weight chart).
 - **Drop zone** over the whole window: images go to the existing vision path;
   PDFs and text go to the new upload endpoint (below).
-- **Quick-ask route** (`/quick`): input + streamed answer only; "Open in Simba"
-  hands the conversation to the main window.
 
 ### Local tool bridge (Simba acts on the Mac)
 
@@ -314,8 +311,8 @@ blue/green stays safe: the old frontend ignores the new fields.
 |---|---|---|---|
 | **0a. Auth fix** (own PR) | access token on every call, single fetch helper, OIDC state in Postgres | auth fix (+ migration if state goes in a table) | Web works through a blue/green flip; the refresh token is only ever sent to `/refresh` |
 | **0b. Spike** (½ day) | `desktop/` Tauri v2 app: one window loading `<simba-host>`, tray icon with Show/Quit, ⌥Space toggles the window, `offline.html` fallback | none | Log in through Authelia inside the webview, stream an answer, survive a `make redeploy` |
-| **1. Native shell** | quick-ask panel (`/quick` route), notifications, window-state, single-instance, autostart, refresh token in the Keychain | none | Hotkey → ask → answer without touching the main window |
-| **2. Cowork UI** | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised |
+| **1. Native shell** | notifications, window-state, autostart, refresh token in the Keychain (no quick-ask window: dropped 2026-10-12) | none | A scheduled-message notification arrives; the login survives an app restart |
+| **2. Cowork UI** ([plan](cowork-ui-plan.md)) | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised |
 | **3. Local tools** | Rust bridge client, filesystem MCP scoped to chosen folders, write-confirm dialog, folder picker | item 7 | "Summarise the PDFs in ~/Downloads/taxes" works; a write prompts first; quitting the app removes the tools |
 | **4. Background agent** | detached runs, notifications feed, desktop channel for scheduled messages, system-browser auth handoff | items 4–6 | Close the window mid-run → notification when done → reopen shows the full timeline |
 

mkdocs.yml

@@ -24,3 +24,4 @@ nav:
   - Architecture:
       - Authentication: authentication.md
       - Desktop Client (proposal): desktop-client.md
+      - Cowork UI plan: cowork-ui-plan.md