simbarag/docs/cowork-ui-plan.md

Cowork UI (phase 2) plan

Status: proposal, awaiting approval. Parent doc: desktop-client.md.

Goal: make the agent's work visible and useful. Tool calls appear as a live activity timeline. Results that are more than text (budget breakdowns, recipes, notes, documents) open in a side panel. Files can be dropped into the conversation. This is all built in raggr-frontend and the Quart backend, so the web app gets it too; the Tauri shell needs only a one-line drag-drop setting.

Where we are

• stream-query SSE sends tool_start {tool} and tool_end {tool} and nothing else: no id, inputs, duration or result. The UI shows a one-line pill whose label comes from a stale TOOL_MESSAGES map in useChat.ts.
• Tools return formatted strings. on_tool_end saves them as speaker="tool" messages, and GET /conversation/<id> returns only their first line ([tool result]).
• Only simba_search uses response_format="content_and_artifact", and its artifact is a list of LangChain Documents that nothing reads.
• Auth: the backend accepting access tokens is merged (#57). The frontend half (fix/access-token-auth) is waiting on a deploy.
• #55 (YNAB mirror) is open. It replaces every YNAB tool (new blueprints/ynab/tools.py: ynab_budget_summary, ynab_account_balances, ynab_search_transactions, ynab_spending_report, ynab_category_budget, ynab_recent_changes). ⚠️ Its migration is numbered 16_…, which clashes with #57's 16_…_add_oidc_states already on main, so it must be regenerated before #55 merges.

Contract

Every task codes against this. It lands on main with this doc before any worker starts, so all workers share it.

SSE events (additive; old clients ignore new fields)

// tool_start
{ type: "tool_start", run_id: string, tool: string, label: string,
  input: Record<string, string | number | boolean> }   // values truncated to 200 chars, `config` dropped
// tool_end
{ type: "tool_end", run_id: string, tool: string, label: string,
  status: "ok" | "error", ms: number,
  artifact: ArtifactRef | null }
// unchanged: thinking, thinking_delta, content, response, error, [DONE]

• label comes from a server-side registry (blueprints/conversation/tool_labels.py): {tool_name: "Checking budget…"}, with the fallback "Using <tool>…". MCP tools match by prefix (homebox_, bookstack_). TOOL_MESSAGES in useChat.ts is deleted.
• status="error" comes from on_tool_error. Most tools catch exceptions and return an error string, so they report ok; that's acceptable.
• Parallel tool calls are paired by run_id.

Artifacts

A tool opts in by returning (text, artifact) with @tool(response_format="content_and_artifact"). The LLM sees only text. The artifact is a plain JSON-serialisable dict:

type ToolArtifact = {            // what a tool returns
  kind: ArtifactKind;
  title: string;                 // "October spending by category"
  data: object;                  // per-kind shape below; ≤ 256 KB serialised or it is dropped
  source_ref?: { system: "mealie" | "obsidian" | "paperless" | "ynab" | "web";
                 id?: string; path?: string; url?: string };
};
type ArtifactRef = { id: string; kind: ArtifactKind; title: string };   // in tool_end + conversation payload
type Artifact = ArtifactRef & { data: object; source_ref: object | null; tool: string;
                                conversation_id: string; message_id: string | null; created_at: string };

V1 kinds and their data shapes:

kind	produced by	data
mealie.recipe	mealie_get_recipe	{slug, name, description, servings, prep_time, cook_time, total_time, ingredients: string[], instructions: {title?, text}[], tags: string[], image_url?}
mealie.meal_plan	mealie_meal_plans, mealie_todays_meals	{start_date, end_date, days: {date, meals: {entry_type, name, recipe_slug?, note?}[]}[]}
mealie.shopping_lists	mealie_shopping_lists	{lists: {name, items: {name, quantity, checked}[]}[]}
obsidian.note	obsidian_read_note, journal_get_today	{path, title, markdown (≤ 50k chars), tags: string[], frontmatter: object}
search.documents	simba_search, obsidian_search_notes	{query, source, results: {title, snippet (≤ 400), filename?, path?, document_type?, date?}[]}
ynab.summary	ynab_budget_summary	{month, ready_to_assign, income, assigned, spent, age_of_money?, net_worth, overspent: {name, available}[]}
ynab.transactions	ynab_search_transactions	{start_date, end_date, count, total, transactions: {date, payee, category, account, memo?, amount}[]} (≤ 500 rows)
ynab.spending_report	ynab_spending_report	{start_date, end_date, group_by, rows: {key, amount, count?}[], total}
ynab.category_budget	ynab_category_budget	{month, groups: {name, categories: {name, assigned, activity, available, goal_target?}[]}[]}

YNAB kinds are built against #55's tools, so they wait for #55. source_ref.url is filled where the backend knows it (Mealie base URL plus the recipe path). Obsidian gets obsidian://open?vault=<OBSIDIAN_VAULT_NAME>&file=<path> only if that env var is set; it's added to .env, .env.example and docker-compose.yml.

Storage and API

• New Tortoise model Artifact (blueprints/artifacts/models.py, table artifacts): id UUID, user FK, conversation FK (cascade), message FK → conversation_messages (nullable, set null), tool, kind, title, data JSON, source_ref JSON null, version int default 1, created_at, updated_at. Migration generated with make migrate-new / aerich, adds a table only.
• Created in stream_query's on_tool_end when output.artifact is a dict with a kind, right after the tool message is saved and linked to it. Non-web channels (iMessage, WhatsApp, /query) don't create artifacts.
• GET /api/artifacts?conversation_id=… returns ArtifactRef[] plus {tool, message_id, created_at}, without data. GET /api/artifacts/<id> returns the full Artifact. Both require an access token and are scoped to the owner (404 otherwise).
• GET /api/conversation/<id>: tool messages gain tool: string, label: string, artifact: ArtifactRef | null. text keeps the header line, for older clients.
• Write-back (PATCH) is out of scope for phase 2.

Uploads

• POST /api/conversation/upload (multipart file, conversation_id):
  • images go through the existing process_image and return {kind: "image", image_key}
  • application/pdf, text/plain, text/markdown, text/csv (by MIME type or extension) are stored at conversations/<id>/files/<uuid>.<ext> and return {kind: "file", file_key, name, chars}. PDF text is extracted with PyMuPDF (fitz, already a dependency).
  • max 10 MB; anything else gets a 415
  • upload-image stays as is
• stream-query accepts an optional file_key. The extracted text (capped at 12k chars, with a [truncated] marker) is appended to the user turn as <attached file "name">…</attached>. The saved user message gets 📎 name appended. Saved user messages get no new column.

Tasks

Each task has its own branch off main (after the contract lands), its own herdr worktree and pane, and its own Claude session started from a brief.

#	Branch / pane	Scope	Owns (files)	Depends on
A	feat/cowork-stream	SSE v2 events + label registry; Artifact model, migration, API, creation in on_tool_end; conversation payload fields	blueprints/conversation/__init__.py, new tool_labels.py, new blueprints/artifacts/, config/db.py, app.py (register), a migration	contract on main. Migration last, after rebasing (see sequencing)
B	feat/cowork-tool-artifacts	content_and_artifact for the Mealie, Obsidian and search tools; then a second commit for the YNAB tools once #55 merges	blueprints/conversation/agents.py, blueprints/ynab/tools.py (2nd commit), tests	contract; #55 for the YNAB part
C	feat/cowork-timeline	Activity timeline replaces ToolBubble: grouped "Simba worked · N steps · 4.2 s" block per turn, rows with icon, label, input summary, spinner/✓/✗, duration and an "Open" chip that calls an onOpenArtifact(ref) prop; restored from conversation payload on reload	useChat.ts, new ActivityTimeline.tsx, ToolBubble.tsx (delete), conversationService.ts types, ChatScreen.tsx (message list only)	frontend auth PR merged
D	feat/cowork-artifact-panel	Right-side artifact panel: tabs for opened artifacts, artifactService.ts, a renderer per kind (CSS bars, no chart library), "Open in Mealie/Obsidian", a sheet below lg; auto-opens the newest artifact of a turn when the window is ≥ 1280 px wide, otherwise just the chip	new components/artifacts/*, api/artifactService.ts, ChatScreen.tsx (layout + wiring onOpenArtifact)	C merged first (wires its prop)
E	feat/cowork-file-drop	Upload endpoint + file_key in stream-query; drop overlay over the whole window, attachment chip in MessageInput; Tauri disable_drag_drop_handler() so HTML5 drops reach the webview	blueprints/conversation/__init__.py (new route + small stream_query hunk), new utils/file_extract.py, MessageInput.tsx, ChatScreen.tsx (overlay), useChat.ts (pending file), desktop/src-tauri/src/lib.rs	rebase after A and C (shared files)

Parallelism and sequencing

now ──► contract PR (this doc) merged
        frontend-auth PR merged   (after your deploy)
            │
            ├── A  feat/cowork-stream ─────────────┐ (merges 1st)
            ├── B  feat/cowork-tool-artifacts ─────┤ (Mealie/Obsidian/search: merges anytime; YNAB commit after #55)
            ├── C  feat/cowork-timeline ───────────┤ (merges after A)
            ├── D  feat/cowork-artifact-panel ─────┤ (merges after C)
            └── E  feat/cowork-file-drop ──────────┘ (merges last)

• All five start in parallel. A and B are pure backend. C and D build against a mock /api server that speaks the contract, so they don't wait for A.
• Migration order. #55's migration needs regenerating (it clashes with #57). A's migration must be generated after whatever lands before it. A writes the model and code first, then rebases onto the latest main and runs make migrate-new as its final step. If #55 merges while A is in review, A rebases and regenerates.
• Shared-file conflicts (ChatScreen.tsx, useChat.ts, conversation/__init__.py) are resolved by the merge order A → C → D → E. Each later branch rebases onto main before its review.
• B's Mealie/Obsidian/search commit is safe to merge before A: without A, artifacts ride along in ToolMessage.artifact and are ignored.

Verification per task

• Backend (A, B, E): unit tests in tests/unit/, run with TAVILY_API_KEY=dummy .venv/bin/pytest tests/ -q (create the venv with uv sync --all-extras).
  • A: event shaping becomes pure functions (tool_start_payload(event), tool_end_payload(event, started_at, artifact_ref)), which are tested, along with the 256 KB cap and owner scoping.
  • B: each converted tool, with its service mocked, returns (str, dict) matching the kind's shape.
  • E: extraction and the 415 case.
• Migrations: a throwaway Postgres container + aerich, the same approach the auth fix used.
• Frontend (C, D, E): yarn build, plus the mock-server + Playwright recipe (Node http mock serving dist/ and SSE fixtures, Playwright on installed Chrome, addInitScript to seed tokens). Screenshots are checked by eye. Fixtures live in raggr-frontend/test-fixtures/cowork/*.json, and D owns one per kind. Don't commit the Playwright devDep or a rewritten package-lock.json.

Process

1. Approve this plan. I commit it on plan/cowork-ui, open the PR and merge it so the contract is on main.
2. Wait for the frontend-auth PR to merge after your deploy. A and B can start right away; C, D and E start once it's in, so they use the new fetch helper.
3. For each task: herdr worktree create … --base origin/main, a .claude-brief.md with that task's slice of this doc, and herdr agent start cowork-<x> … -- claude "Read .claude-brief.md…". Workers commit locally and never push.
4. When a worker goes idle with its task done, I review:
   • pass 1: I read the diff against the contract, run the tests and build, and send numbered findings to the worker.
   • pass 2: same again after its fixes.
   • Then I push the branch and open the PR (tea pr create).
   • I don't merge; you merge, or tell me to.
5. Before each review, the branch rebases on main per the merge order.

Decisions I made (say if you disagree)

• No write-back in phase 2. Editing a recipe or note in the panel and saving it to Mealie/Obsidian is phase 2.5.
• No chart library. CSS bars and tables are enough for budgets. Revisit if trends need real line charts.
• The panel auto-opens on wide windows, otherwise shows a chip.
• One attached file per message (like images today). Dropped PDFs aren't filed into Paperless yet.
• Out of scope for now: the bridge, notifications, the Keychain, and detached runs. They're phases 3–4 in the parent doc.

Open questions

1. Is #55 close to merging? It decides when the YNAB artifacts happen, and its migration needs regenerating against main either way.
2. What's the Obsidian vault name (for obsidian:// links), or is the link not worth having?
3. What's the Mealie recipe URL format on your instance (/g/home/r/<slug> or /recipe/<slug>)? Worker B can check against the API otherwise.