From abf941569626f345ba7d5c84f8acd37a3c7bf933 Mon Sep 17 00:00:00 2001 From: Ryan Chen Date: Mon, 12 Oct 2026 17:47:43 -0400 Subject: [PATCH] Add Cowork UI (phase 2) plan and drop quick-ask window Defines the shared contract (SSE v2 tool events, artifact kinds and storage, artifacts API, uploads), five parallel tasks with file ownership, merge order and verification, and removes the quick-ask window from the desktop plan. Co-Authored-By: Claude Opus 5.5 --- docs/cowork-ui-plan.md | 229 +++++++++++++++++++++++++++++++++++++++++ docs/desktop-client.md | 15 ++- mkdocs.yml | 1 + 3 files changed, 236 insertions(+), 9 deletions(-) create mode 100644 docs/cowork-ui-plan.md diff --git a/docs/cowork-ui-plan.md b/docs/cowork-ui-plan.md new file mode 100644 index 0000000..d82bfb9 --- /dev/null +++ b/docs/cowork-ui-plan.md @@ -0,0 +1,229 @@ +# Cowork UI (phase 2) plan + +Status: **proposal, awaiting approval.** Parent doc: [desktop-client.md](desktop-client.md). + +Goal: make the agent's work visible and useful. Tool calls appear as a live +activity timeline. Results that are more than text (budget breakdowns, recipes, +notes, documents) open in a side panel. Files can be dropped into the +conversation. This is all built in `raggr-frontend` and the Quart backend, so +the web app gets it too; the Tauri shell needs only a one-line drag-drop +setting. + +## Where we are + +- `stream-query` SSE sends `tool_start {tool}` and `tool_end {tool}` and nothing + else: no id, inputs, duration or result. The UI shows a one-line pill whose + label comes from a stale `TOOL_MESSAGES` map in `useChat.ts`. +- Tools return formatted strings. `on_tool_end` saves them as `speaker="tool"` + messages, and `GET /conversation/` returns only their first line + (`[tool result]`). +- Only `simba_search` uses `response_format="content_and_artifact"`, and its + artifact is a list of LangChain `Document`s that nothing reads. +- Auth: the backend accepting access tokens is merged (#57). The frontend + half (`fix/access-token-auth`) is waiting on a deploy. +- **#55 (YNAB mirror) is open.** It replaces every YNAB tool (new + `blueprints/ynab/tools.py`: `ynab_budget_summary`, `ynab_account_balances`, + `ynab_search_transactions`, `ynab_spending_report`, `ynab_category_budget`, + `ynab_recent_changes`). ⚠️ Its migration is numbered `16_…`, which clashes + with #57's `16_…_add_oidc_states` already on main, so it must be + regenerated before #55 merges. + +## Contract + +Every task codes against this. It lands on main with this doc before any +worker starts, so all workers share it. + +### SSE events (additive; old clients ignore new fields) + +```ts +// tool_start +{ type: "tool_start", run_id: string, tool: string, label: string, + input: Record } // values truncated to 200 chars, `config` dropped +// tool_end +{ type: "tool_end", run_id: string, tool: string, label: string, + status: "ok" | "error", ms: number, + artifact: ArtifactRef | null } +// unchanged: thinking, thinking_delta, content, response, error, [DONE] +``` + +- `label` comes from a server-side registry + (`blueprints/conversation/tool_labels.py`): `{tool_name: "Checking budget…"}`, + with the fallback `"Using …"`. MCP tools match by prefix (`homebox_`, + `bookstack_`). `TOOL_MESSAGES` in `useChat.ts` is deleted. +- `status="error"` comes from `on_tool_error`. Most tools catch exceptions and + return an error string, so they report `ok`; that's acceptable. +- Parallel tool calls are paired by `run_id`. + +### Artifacts + +A tool opts in by returning `(text, artifact)` with +`@tool(response_format="content_and_artifact")`. The LLM sees only `text`. The +artifact is a plain JSON-serialisable dict: + +```ts +type ToolArtifact = { // what a tool returns + kind: ArtifactKind; + title: string; // "October spending by category" + data: object; // per-kind shape below; ≤ 256 KB serialised or it is dropped + source_ref?: { system: "mealie" | "obsidian" | "paperless" | "ynab" | "web"; + id?: string; path?: string; url?: string }; +}; +type ArtifactRef = { id: string; kind: ArtifactKind; title: string }; // in tool_end + conversation payload +type Artifact = ArtifactRef & { data: object; source_ref: object | null; tool: string; + conversation_id: string; message_id: string | null; created_at: string }; +``` + +V1 kinds and their `data` shapes: + +| kind | produced by | data | +|---|---|---| +| `mealie.recipe` | `mealie_get_recipe` | `{slug, name, description, servings, prep_time, cook_time, total_time, ingredients: string[], instructions: {title?, text}[], tags: string[], image_url?}` | +| `mealie.meal_plan` | `mealie_meal_plans`, `mealie_todays_meals` | `{start_date, end_date, days: {date, meals: {entry_type, name, recipe_slug?, note?}[]}[]}` | +| `mealie.shopping_lists` | `mealie_shopping_lists` | `{lists: {name, items: {name, quantity, checked}[]}[]}` | +| `obsidian.note` | `obsidian_read_note`, `journal_get_today` | `{path, title, markdown (≤ 50k chars), tags: string[], frontmatter: object}` | +| `search.documents` | `simba_search`, `obsidian_search_notes` | `{query, source, results: {title, snippet (≤ 400), filename?, path?, document_type?, date?}[]}` | +| `ynab.summary` | `ynab_budget_summary` | `{month, ready_to_assign, income, assigned, spent, age_of_money?, net_worth, overspent: {name, available}[]}` | +| `ynab.transactions` | `ynab_search_transactions` | `{start_date, end_date, count, total, transactions: {date, payee, category, account, memo?, amount}[]}` (≤ 500 rows) | +| `ynab.spending_report` | `ynab_spending_report` | `{start_date, end_date, group_by, rows: {key, amount, count?}[], total}` | +| `ynab.category_budget` | `ynab_category_budget` | `{month, groups: {name, categories: {name, assigned, activity, available, goal_target?}[]}[]}` | + +YNAB kinds are built against #55's tools, so they wait for #55. +`source_ref.url` is filled where the backend knows it (Mealie base URL plus the +recipe path). Obsidian gets `obsidian://open?vault=&file=` +only if that env var is set; it's added to `.env`, `.env.example` and +`docker-compose.yml`. + +### Storage and API + +- New Tortoise model `Artifact` (`blueprints/artifacts/models.py`, table + `artifacts`): `id UUID, user FK, conversation FK (cascade), message FK → + conversation_messages (nullable, set null), tool, kind, title, data JSON, + source_ref JSON null, version int default 1, created_at, updated_at`. + Migration generated with `make migrate-new` / aerich, adds a table only. +- Created in `stream_query`'s `on_tool_end` when `output.artifact` is a dict + with a `kind`, right after the tool message is saved and linked to it. + Non-web channels (iMessage, WhatsApp, `/query`) don't create artifacts. +- `GET /api/artifacts?conversation_id=…` returns `ArtifactRef[]` plus + `{tool, message_id, created_at}`, without `data`. + `GET /api/artifacts/` returns the full `Artifact`. Both require an access + token and are scoped to the owner (404 otherwise). +- `GET /api/conversation/`: tool messages gain + `tool: string, label: string, artifact: ArtifactRef | null`. `text` keeps + the header line, for older clients. +- Write-back (`PATCH`) is **out of scope** for phase 2. + +### Uploads + +- `POST /api/conversation/upload` (multipart `file`, `conversation_id`): + - images go through the existing `process_image` and return + `{kind: "image", image_key}` + - `application/pdf`, `text/plain`, `text/markdown`, `text/csv` (by MIME type + or extension) are stored at `conversations//files/.` and + return `{kind: "file", file_key, name, chars}`. PDF text is extracted with + PyMuPDF (`fitz`, already a dependency). + - max 10 MB; anything else gets a 415 + - `upload-image` stays as is +- `stream-query` accepts an optional `file_key`. The extracted text (capped at + 12k chars, with a `[truncated]` marker) is appended to the user turn as + `…`. The saved user message gets + `📎 name` appended. Saved user messages get no new column. + +## Tasks + +Each task has its own branch off main (after the contract lands), its own herdr +worktree and pane, and its own Claude session started from a brief. + +| # | Branch / pane | Scope | Owns (files) | Depends on | +|---|---|---|---|---| +| **A** | `feat/cowork-stream` | SSE v2 events + label registry; `Artifact` model, migration, API, creation in `on_tool_end`; conversation payload fields | `blueprints/conversation/__init__.py`, new `tool_labels.py`, new `blueprints/artifacts/`, `config/db.py`, `app.py` (register), a migration | contract on main. **Migration last, after rebasing** (see sequencing) | +| **B** | `feat/cowork-tool-artifacts` | `content_and_artifact` for the Mealie, Obsidian and search tools; then a second commit for the YNAB tools once #55 merges | `blueprints/conversation/agents.py`, `blueprints/ynab/tools.py` (2nd commit), tests | contract; #55 for the YNAB part | +| **C** | `feat/cowork-timeline` | Activity timeline replaces `ToolBubble`: grouped "Simba worked · N steps · 4.2 s" block per turn, rows with icon, label, input summary, spinner/✓/✗, duration and an "Open" chip that calls an `onOpenArtifact(ref)` prop; restored from conversation payload on reload | `useChat.ts`, new `ActivityTimeline.tsx`, `ToolBubble.tsx` (delete), `conversationService.ts` types, `ChatScreen.tsx` (message list only) | frontend auth PR merged | +| **D** | `feat/cowork-artifact-panel` | Right-side artifact panel: tabs for opened artifacts, `artifactService.ts`, a renderer per kind (CSS bars, no chart library), "Open in Mealie/Obsidian", a sheet below `lg`; auto-opens the newest artifact of a turn when the window is ≥ 1280 px wide, otherwise just the chip | new `components/artifacts/*`, `api/artifactService.ts`, `ChatScreen.tsx` (layout + wiring `onOpenArtifact`) | C merged first (wires its prop) | +| **E** | `feat/cowork-file-drop` | Upload endpoint + `file_key` in `stream-query`; drop overlay over the whole window, attachment chip in `MessageInput`; Tauri `disable_drag_drop_handler()` so HTML5 drops reach the webview | `blueprints/conversation/__init__.py` (new route + small `stream_query` hunk), new `utils/file_extract.py`, `MessageInput.tsx`, `ChatScreen.tsx` (overlay), `useChat.ts` (pending file), `desktop/src-tauri/src/lib.rs` | rebase after A and C (shared files) | + +### Parallelism and sequencing + +``` +now ──► contract PR (this doc) merged + frontend-auth PR merged (after your deploy) + │ + ├── A feat/cowork-stream ─────────────┐ (merges 1st) + ├── B feat/cowork-tool-artifacts ─────┤ (Mealie/Obsidian/search: merges anytime; YNAB commit after #55) + ├── C feat/cowork-timeline ───────────┤ (merges after A) + ├── D feat/cowork-artifact-panel ─────┤ (merges after C) + └── E feat/cowork-file-drop ──────────┘ (merges last) +``` + +- **All five start in parallel.** A and B are pure backend. C and D build + against a mock `/api` server that speaks the contract, so they don't wait for + A. +- **Migration order.** #55's migration needs regenerating (it clashes with + #57). A's migration must be generated *after* whatever lands before it. A + writes the model and code first, then rebases onto the latest main and runs + `make migrate-new` as its final step. If #55 merges while A is in review, A + rebases and regenerates. +- **Shared-file conflicts** (`ChatScreen.tsx`, `useChat.ts`, + `conversation/__init__.py`) are resolved by the merge order A → C → D → E. Each + later branch rebases onto main before its review. +- B's Mealie/Obsidian/search commit is safe to merge before A: without A, + artifacts ride along in `ToolMessage.artifact` and are ignored. + +### Verification per task + +- **Backend (A, B, E):** unit tests in `tests/unit/`, run with + `TAVILY_API_KEY=dummy .venv/bin/pytest tests/ -q` (create the venv with + `uv sync --all-extras`). + - A: event shaping becomes pure functions (`tool_start_payload(event)`, + `tool_end_payload(event, started_at, artifact_ref)`), which are tested, + along with the 256 KB cap and owner scoping. + - B: each converted tool, with its service mocked, returns `(str, dict)` + matching the kind's shape. + - E: extraction and the 415 case. +- **Migrations:** a throwaway Postgres container + aerich, the same approach + the auth fix used. +- **Frontend (C, D, E):** `yarn build`, plus the mock-server + Playwright recipe + (Node `http` mock serving `dist/` and SSE fixtures, Playwright on installed + Chrome, `addInitScript` to seed tokens). Screenshots are checked by eye. + Fixtures live in `raggr-frontend/test-fixtures/cowork/*.json`, and D owns one + per kind. Don't commit the Playwright devDep or a rewritten + `package-lock.json`. + +## Process + +1. Approve this plan. I commit it on `plan/cowork-ui`, open the PR and merge it + so the contract is on main. +2. Wait for the frontend-auth PR to merge after your deploy. A and B can start + right away; C, D and E start once it's in, so they use the new fetch helper. +3. For each task: `herdr worktree create … --base origin/main`, a + `.claude-brief.md` with that task's slice of this doc, and + `herdr agent start cowork- … -- claude "Read .claude-brief.md…"`. + Workers commit locally and never push. +4. When a worker goes idle with its task done, I review: + - **pass 1:** I read the diff against the contract, run the tests and + build, and send numbered findings to the worker. + - **pass 2:** same again after its fixes. + - Then I push the branch and open the PR (`tea pr create`). + - I don't merge; you merge, or tell me to. +5. Before each review, the branch rebases on main per the merge order. + +## Decisions I made (say if you disagree) + +- **No write-back in phase 2.** Editing a recipe or note in the panel and + saving it to Mealie/Obsidian is phase 2.5. +- **No chart library.** CSS bars and tables are enough for budgets. Revisit if + trends need real line charts. +- **The panel auto-opens on wide windows**, otherwise shows a chip. +- **One attached file per message** (like images today). Dropped PDFs aren't + filed into Paperless yet. +- **Out of scope for now:** the bridge, notifications, the Keychain, and + detached runs. They're phases 3–4 in the parent doc. + +## Open questions + +1. Is **#55** close to merging? It decides when the YNAB artifacts happen, and + its migration needs regenerating against main either way. +2. What's the **Obsidian vault name** (for `obsidian://` links), or is the + link not worth having? +3. What's the **Mealie recipe URL** format on your instance + (`/g/home/r/` or `/recipe/`)? Worker B can check against the + API otherwise. diff --git a/docs/desktop-client.md b/docs/desktop-client.md index 8542072..39bbd7c 100644 --- a/docs/desktop-client.md +++ b/docs/desktop-client.md @@ -9,7 +9,7 @@ user rather than a chat box in a browser tab. "Cowork-like" means: - side panels for artifacts: budget reports, recipes, notes, documents - drag files into the agent - desktop notifications (scheduled messages, cron jobs, finished runs) -- a menu-bar icon and a global-hotkey quick-ask window +- a menu-bar icon and a global hotkey to summon the window ## Decisions (2026-10-12) @@ -110,13 +110,12 @@ feature detection, so the web app gets them too.** ┌──────────────────────── Simba.app (Tauri v2) ─────────────────────────┐ │ Rust core (small) │ │ • tray icon + menu (Ask…, New chat, recent convos, Quit) │ -│ • global shortcut ⌥Space → quick-ask panel (positioner, always-on-top)│ +│ • global shortcut ⌥Space → show/hide main window │ │ • notifications (plugin-notification) │ │ • keychain token store, deep-link `simba://`, single-instance │ │ • window-state, autostart │ │ │ │ Main window ──loads──▶ https:/// (full UI) │ -│ Quick panel ──loads──▶ https:///quick (compact route) │ │ Fallback ──bundled─▶ offline.html (server unreachable / retry) │ └───────────────────────────────┬────────────────────────────────────────┘ │ HTTPS: REST + SSE (same origin, no CORS) @@ -140,8 +139,8 @@ Why load the remote origin rather than bundle: `offline.html` covers "server down / not on VPN". The frontend detects the shell with `window.__TAURI_INTERNALS__` (or a -`?shell=desktop` query param) and turns on the native bits: notifications, -drag-drop paths, and the "open in quick panel" route. Everything else renders +`?shell=desktop` query param) and turns on the native bits: notifications +and drag-drop paths. Everything else renders the same in a browser. ### Cowork UI (in raggr-frontend, shared by web and desktop) @@ -159,8 +158,6 @@ the same in a browser. Paperless), `photo` (Immich), `device` (SimbaKit status/weight chart). - **Drop zone** over the whole window: images go to the existing vision path; PDFs and text go to the new upload endpoint (below). -- **Quick-ask route** (`/quick`): input + streamed answer only; "Open in Simba" - hands the conversation to the main window. ### Local tool bridge (Simba acts on the Mac) @@ -314,8 +311,8 @@ blue/green stays safe: the old frontend ignores the new fields. |---|---|---|---| | **0a. Auth fix** (own PR) | access token on every call, single fetch helper, OIDC state in Postgres | auth fix (+ migration if state goes in a table) | Web works through a blue/green flip; the refresh token is only ever sent to `/refresh` | | **0b. Spike** (½ day) | `desktop/` Tauri v2 app: one window loading ``, tray icon with Show/Quit, ⌥Space toggles the window, `offline.html` fallback | none | Log in through Authelia inside the webview, stream an answer, survive a `make redeploy` | -| **1. Native shell** | quick-ask panel (`/quick` route), notifications, window-state, single-instance, autostart, refresh token in the Keychain | none | Hotkey → ask → answer without touching the main window | -| **2. Cowork UI** | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised | +| **1. Native shell** | notifications, window-state, autostart, refresh token in the Keychain (no quick-ask window: dropped 2026-10-12) | none | A scheduled-message notification arrives; the login survives an app restart | +| **2. Cowork UI** ([plan](cowork-ui-plan.md)) | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised | | **3. Local tools** | Rust bridge client, filesystem MCP scoped to chosen folders, write-confirm dialog, folder picker | item 7 | "Summarise the PDFs in ~/Downloads/taxes" works; a write prompts first; quitting the app removes the tools | | **4. Background agent** | detached runs, notifications feed, desktop channel for scheduled messages, system-browser auth handoff | items 4–6 | Close the window mid-run → notification when done → reopen shows the full timeline | diff --git a/mkdocs.yml b/mkdocs.yml index e6656ee..006c1e4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -24,3 +24,4 @@ nav: - Architecture: - Authentication: authentication.md - Desktop Client (proposal): desktop-client.md + - Cowork UI plan: cowork-ui-plan.md -- 2.52.0