diff --git a/docs/cowork-ui-plan.md b/docs/cowork-ui-plan.md new file mode 100644 index 0000000..d82bfb9 --- /dev/null +++ b/docs/cowork-ui-plan.md @@ -0,0 +1,229 @@ +# Cowork UI (phase 2) plan + +Status: **proposal, awaiting approval.** Parent doc: [desktop-client.md](desktop-client.md). + +Goal: make the agent's work visible and useful. Tool calls appear as a live +activity timeline. Results that are more than text (budget breakdowns, recipes, +notes, documents) open in a side panel. Files can be dropped into the +conversation. This is all built in `raggr-frontend` and the Quart backend, so +the web app gets it too; the Tauri shell needs only a one-line drag-drop +setting. + +## Where we are + +- `stream-query` SSE sends `tool_start {tool}` and `tool_end {tool}` and nothing + else: no id, inputs, duration or result. The UI shows a one-line pill whose + label comes from a stale `TOOL_MESSAGES` map in `useChat.ts`. +- Tools return formatted strings. `on_tool_end` saves them as `speaker="tool"` + messages, and `GET /conversation/` returns only their first line + (`[tool result]`). +- Only `simba_search` uses `response_format="content_and_artifact"`, and its + artifact is a list of LangChain `Document`s that nothing reads. +- Auth: the backend accepting access tokens is merged (#57). The frontend + half (`fix/access-token-auth`) is waiting on a deploy. +- **#55 (YNAB mirror) is open.** It replaces every YNAB tool (new + `blueprints/ynab/tools.py`: `ynab_budget_summary`, `ynab_account_balances`, + `ynab_search_transactions`, `ynab_spending_report`, `ynab_category_budget`, + `ynab_recent_changes`). ⚠️ Its migration is numbered `16_…`, which clashes + with #57's `16_…_add_oidc_states` already on main, so it must be + regenerated before #55 merges. + +## Contract + +Every task codes against this. It lands on main with this doc before any +worker starts, so all workers share it. + +### SSE events (additive; old clients ignore new fields) + +```ts +// tool_start +{ type: "tool_start", run_id: string, tool: string, label: string, + input: Record } // values truncated to 200 chars, `config` dropped +// tool_end +{ type: "tool_end", run_id: string, tool: string, label: string, + status: "ok" | "error", ms: number, + artifact: ArtifactRef | null } +// unchanged: thinking, thinking_delta, content, response, error, [DONE] +``` + +- `label` comes from a server-side registry + (`blueprints/conversation/tool_labels.py`): `{tool_name: "Checking budget…"}`, + with the fallback `"Using …"`. MCP tools match by prefix (`homebox_`, + `bookstack_`). `TOOL_MESSAGES` in `useChat.ts` is deleted. +- `status="error"` comes from `on_tool_error`. Most tools catch exceptions and + return an error string, so they report `ok`; that's acceptable. +- Parallel tool calls are paired by `run_id`. + +### Artifacts + +A tool opts in by returning `(text, artifact)` with +`@tool(response_format="content_and_artifact")`. The LLM sees only `text`. The +artifact is a plain JSON-serialisable dict: + +```ts +type ToolArtifact = { // what a tool returns + kind: ArtifactKind; + title: string; // "October spending by category" + data: object; // per-kind shape below; ≤ 256 KB serialised or it is dropped + source_ref?: { system: "mealie" | "obsidian" | "paperless" | "ynab" | "web"; + id?: string; path?: string; url?: string }; +}; +type ArtifactRef = { id: string; kind: ArtifactKind; title: string }; // in tool_end + conversation payload +type Artifact = ArtifactRef & { data: object; source_ref: object | null; tool: string; + conversation_id: string; message_id: string | null; created_at: string }; +``` + +V1 kinds and their `data` shapes: + +| kind | produced by | data | +|---|---|---| +| `mealie.recipe` | `mealie_get_recipe` | `{slug, name, description, servings, prep_time, cook_time, total_time, ingredients: string[], instructions: {title?, text}[], tags: string[], image_url?}` | +| `mealie.meal_plan` | `mealie_meal_plans`, `mealie_todays_meals` | `{start_date, end_date, days: {date, meals: {entry_type, name, recipe_slug?, note?}[]}[]}` | +| `mealie.shopping_lists` | `mealie_shopping_lists` | `{lists: {name, items: {name, quantity, checked}[]}[]}` | +| `obsidian.note` | `obsidian_read_note`, `journal_get_today` | `{path, title, markdown (≤ 50k chars), tags: string[], frontmatter: object}` | +| `search.documents` | `simba_search`, `obsidian_search_notes` | `{query, source, results: {title, snippet (≤ 400), filename?, path?, document_type?, date?}[]}` | +| `ynab.summary` | `ynab_budget_summary` | `{month, ready_to_assign, income, assigned, spent, age_of_money?, net_worth, overspent: {name, available}[]}` | +| `ynab.transactions` | `ynab_search_transactions` | `{start_date, end_date, count, total, transactions: {date, payee, category, account, memo?, amount}[]}` (≤ 500 rows) | +| `ynab.spending_report` | `ynab_spending_report` | `{start_date, end_date, group_by, rows: {key, amount, count?}[], total}` | +| `ynab.category_budget` | `ynab_category_budget` | `{month, groups: {name, categories: {name, assigned, activity, available, goal_target?}[]}[]}` | + +YNAB kinds are built against #55's tools, so they wait for #55. +`source_ref.url` is filled where the backend knows it (Mealie base URL plus the +recipe path). Obsidian gets `obsidian://open?vault=&file=` +only if that env var is set; it's added to `.env`, `.env.example` and +`docker-compose.yml`. + +### Storage and API + +- New Tortoise model `Artifact` (`blueprints/artifacts/models.py`, table + `artifacts`): `id UUID, user FK, conversation FK (cascade), message FK → + conversation_messages (nullable, set null), tool, kind, title, data JSON, + source_ref JSON null, version int default 1, created_at, updated_at`. + Migration generated with `make migrate-new` / aerich, adds a table only. +- Created in `stream_query`'s `on_tool_end` when `output.artifact` is a dict + with a `kind`, right after the tool message is saved and linked to it. + Non-web channels (iMessage, WhatsApp, `/query`) don't create artifacts. +- `GET /api/artifacts?conversation_id=…` returns `ArtifactRef[]` plus + `{tool, message_id, created_at}`, without `data`. + `GET /api/artifacts/` returns the full `Artifact`. Both require an access + token and are scoped to the owner (404 otherwise). +- `GET /api/conversation/`: tool messages gain + `tool: string, label: string, artifact: ArtifactRef | null`. `text` keeps + the header line, for older clients. +- Write-back (`PATCH`) is **out of scope** for phase 2. + +### Uploads + +- `POST /api/conversation/upload` (multipart `file`, `conversation_id`): + - images go through the existing `process_image` and return + `{kind: "image", image_key}` + - `application/pdf`, `text/plain`, `text/markdown`, `text/csv` (by MIME type + or extension) are stored at `conversations//files/.` and + return `{kind: "file", file_key, name, chars}`. PDF text is extracted with + PyMuPDF (`fitz`, already a dependency). + - max 10 MB; anything else gets a 415 + - `upload-image` stays as is +- `stream-query` accepts an optional `file_key`. The extracted text (capped at + 12k chars, with a `[truncated]` marker) is appended to the user turn as + `…`. The saved user message gets + `📎 name` appended. Saved user messages get no new column. + +## Tasks + +Each task has its own branch off main (after the contract lands), its own herdr +worktree and pane, and its own Claude session started from a brief. + +| # | Branch / pane | Scope | Owns (files) | Depends on | +|---|---|---|---|---| +| **A** | `feat/cowork-stream` | SSE v2 events + label registry; `Artifact` model, migration, API, creation in `on_tool_end`; conversation payload fields | `blueprints/conversation/__init__.py`, new `tool_labels.py`, new `blueprints/artifacts/`, `config/db.py`, `app.py` (register), a migration | contract on main. **Migration last, after rebasing** (see sequencing) | +| **B** | `feat/cowork-tool-artifacts` | `content_and_artifact` for the Mealie, Obsidian and search tools; then a second commit for the YNAB tools once #55 merges | `blueprints/conversation/agents.py`, `blueprints/ynab/tools.py` (2nd commit), tests | contract; #55 for the YNAB part | +| **C** | `feat/cowork-timeline` | Activity timeline replaces `ToolBubble`: grouped "Simba worked · N steps · 4.2 s" block per turn, rows with icon, label, input summary, spinner/✓/✗, duration and an "Open" chip that calls an `onOpenArtifact(ref)` prop; restored from conversation payload on reload | `useChat.ts`, new `ActivityTimeline.tsx`, `ToolBubble.tsx` (delete), `conversationService.ts` types, `ChatScreen.tsx` (message list only) | frontend auth PR merged | +| **D** | `feat/cowork-artifact-panel` | Right-side artifact panel: tabs for opened artifacts, `artifactService.ts`, a renderer per kind (CSS bars, no chart library), "Open in Mealie/Obsidian", a sheet below `lg`; auto-opens the newest artifact of a turn when the window is ≥ 1280 px wide, otherwise just the chip | new `components/artifacts/*`, `api/artifactService.ts`, `ChatScreen.tsx` (layout + wiring `onOpenArtifact`) | C merged first (wires its prop) | +| **E** | `feat/cowork-file-drop` | Upload endpoint + `file_key` in `stream-query`; drop overlay over the whole window, attachment chip in `MessageInput`; Tauri `disable_drag_drop_handler()` so HTML5 drops reach the webview | `blueprints/conversation/__init__.py` (new route + small `stream_query` hunk), new `utils/file_extract.py`, `MessageInput.tsx`, `ChatScreen.tsx` (overlay), `useChat.ts` (pending file), `desktop/src-tauri/src/lib.rs` | rebase after A and C (shared files) | + +### Parallelism and sequencing + +``` +now ──► contract PR (this doc) merged + frontend-auth PR merged (after your deploy) + │ + ├── A feat/cowork-stream ─────────────┐ (merges 1st) + ├── B feat/cowork-tool-artifacts ─────┤ (Mealie/Obsidian/search: merges anytime; YNAB commit after #55) + ├── C feat/cowork-timeline ───────────┤ (merges after A) + ├── D feat/cowork-artifact-panel ─────┤ (merges after C) + └── E feat/cowork-file-drop ──────────┘ (merges last) +``` + +- **All five start in parallel.** A and B are pure backend. C and D build + against a mock `/api` server that speaks the contract, so they don't wait for + A. +- **Migration order.** #55's migration needs regenerating (it clashes with + #57). A's migration must be generated *after* whatever lands before it. A + writes the model and code first, then rebases onto the latest main and runs + `make migrate-new` as its final step. If #55 merges while A is in review, A + rebases and regenerates. +- **Shared-file conflicts** (`ChatScreen.tsx`, `useChat.ts`, + `conversation/__init__.py`) are resolved by the merge order A → C → D → E. Each + later branch rebases onto main before its review. +- B's Mealie/Obsidian/search commit is safe to merge before A: without A, + artifacts ride along in `ToolMessage.artifact` and are ignored. + +### Verification per task + +- **Backend (A, B, E):** unit tests in `tests/unit/`, run with + `TAVILY_API_KEY=dummy .venv/bin/pytest tests/ -q` (create the venv with + `uv sync --all-extras`). + - A: event shaping becomes pure functions (`tool_start_payload(event)`, + `tool_end_payload(event, started_at, artifact_ref)`), which are tested, + along with the 256 KB cap and owner scoping. + - B: each converted tool, with its service mocked, returns `(str, dict)` + matching the kind's shape. + - E: extraction and the 415 case. +- **Migrations:** a throwaway Postgres container + aerich, the same approach + the auth fix used. +- **Frontend (C, D, E):** `yarn build`, plus the mock-server + Playwright recipe + (Node `http` mock serving `dist/` and SSE fixtures, Playwright on installed + Chrome, `addInitScript` to seed tokens). Screenshots are checked by eye. + Fixtures live in `raggr-frontend/test-fixtures/cowork/*.json`, and D owns one + per kind. Don't commit the Playwright devDep or a rewritten + `package-lock.json`. + +## Process + +1. Approve this plan. I commit it on `plan/cowork-ui`, open the PR and merge it + so the contract is on main. +2. Wait for the frontend-auth PR to merge after your deploy. A and B can start + right away; C, D and E start once it's in, so they use the new fetch helper. +3. For each task: `herdr worktree create … --base origin/main`, a + `.claude-brief.md` with that task's slice of this doc, and + `herdr agent start cowork- … -- claude "Read .claude-brief.md…"`. + Workers commit locally and never push. +4. When a worker goes idle with its task done, I review: + - **pass 1:** I read the diff against the contract, run the tests and + build, and send numbered findings to the worker. + - **pass 2:** same again after its fixes. + - Then I push the branch and open the PR (`tea pr create`). + - I don't merge; you merge, or tell me to. +5. Before each review, the branch rebases on main per the merge order. + +## Decisions I made (say if you disagree) + +- **No write-back in phase 2.** Editing a recipe or note in the panel and + saving it to Mealie/Obsidian is phase 2.5. +- **No chart library.** CSS bars and tables are enough for budgets. Revisit if + trends need real line charts. +- **The panel auto-opens on wide windows**, otherwise shows a chip. +- **One attached file per message** (like images today). Dropped PDFs aren't + filed into Paperless yet. +- **Out of scope for now:** the bridge, notifications, the Keychain, and + detached runs. They're phases 3–4 in the parent doc. + +## Open questions + +1. Is **#55** close to merging? It decides when the YNAB artifacts happen, and + its migration needs regenerating against main either way. +2. What's the **Obsidian vault name** (for `obsidian://` links), or is the + link not worth having? +3. What's the **Mealie recipe URL** format on your instance + (`/g/home/r/` or `/recipe/`)? Worker B can check against the + API otherwise. diff --git a/docs/desktop-client.md b/docs/desktop-client.md index 8542072..39bbd7c 100644 --- a/docs/desktop-client.md +++ b/docs/desktop-client.md @@ -9,7 +9,7 @@ user rather than a chat box in a browser tab. "Cowork-like" means: - side panels for artifacts: budget reports, recipes, notes, documents - drag files into the agent - desktop notifications (scheduled messages, cron jobs, finished runs) -- a menu-bar icon and a global-hotkey quick-ask window +- a menu-bar icon and a global hotkey to summon the window ## Decisions (2026-10-12) @@ -110,13 +110,12 @@ feature detection, so the web app gets them too.** ┌──────────────────────── Simba.app (Tauri v2) ─────────────────────────┐ │ Rust core (small) │ │ • tray icon + menu (Ask…, New chat, recent convos, Quit) │ -│ • global shortcut ⌥Space → quick-ask panel (positioner, always-on-top)│ +│ • global shortcut ⌥Space → show/hide main window │ │ • notifications (plugin-notification) │ │ • keychain token store, deep-link `simba://`, single-instance │ │ • window-state, autostart │ │ │ │ Main window ──loads──▶ https:/// (full UI) │ -│ Quick panel ──loads──▶ https:///quick (compact route) │ │ Fallback ──bundled─▶ offline.html (server unreachable / retry) │ └───────────────────────────────┬────────────────────────────────────────┘ │ HTTPS: REST + SSE (same origin, no CORS) @@ -140,8 +139,8 @@ Why load the remote origin rather than bundle: `offline.html` covers "server down / not on VPN". The frontend detects the shell with `window.__TAURI_INTERNALS__` (or a -`?shell=desktop` query param) and turns on the native bits: notifications, -drag-drop paths, and the "open in quick panel" route. Everything else renders +`?shell=desktop` query param) and turns on the native bits: notifications +and drag-drop paths. Everything else renders the same in a browser. ### Cowork UI (in raggr-frontend, shared by web and desktop) @@ -159,8 +158,6 @@ the same in a browser. Paperless), `photo` (Immich), `device` (SimbaKit status/weight chart). - **Drop zone** over the whole window: images go to the existing vision path; PDFs and text go to the new upload endpoint (below). -- **Quick-ask route** (`/quick`): input + streamed answer only; "Open in Simba" - hands the conversation to the main window. ### Local tool bridge (Simba acts on the Mac) @@ -314,8 +311,8 @@ blue/green stays safe: the old frontend ignores the new fields. |---|---|---|---| | **0a. Auth fix** (own PR) | access token on every call, single fetch helper, OIDC state in Postgres | auth fix (+ migration if state goes in a table) | Web works through a blue/green flip; the refresh token is only ever sent to `/refresh` | | **0b. Spike** (½ day) | `desktop/` Tauri v2 app: one window loading ``, tray icon with Show/Quit, ⌥Space toggles the window, `offline.html` fallback | none | Log in through Authelia inside the webview, stream an answer, survive a `make redeploy` | -| **1. Native shell** | quick-ask panel (`/quick` route), notifications, window-state, single-instance, autostart, refresh token in the Keychain | none | Hotkey → ask → answer without touching the main window | -| **2. Cowork UI** | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised | +| **1. Native shell** | notifications, window-state, autostart, refresh token in the Keychain (no quick-ask window: dropped 2026-10-12) | none | A scheduled-message notification arrives; the login survives an app restart | +| **2. Cowork UI** ([plan](cowork-ui-plan.md)) | activity timeline, artifact panel + renderers (budget, recipe, note, document first), whole-window drop zone | items 1–3 (`artifacts` table) | Ask "how's the grocery budget" → YNAB artifact opens in the side panel; drop a PDF → summarised | | **3. Local tools** | Rust bridge client, filesystem MCP scoped to chosen folders, write-confirm dialog, folder picker | item 7 | "Summarise the PDFs in ~/Downloads/taxes" works; a write prompts first; quitting the app removes the tools | | **4. Background agent** | detached runs, notifications feed, desktop channel for scheduled messages, system-browser auth handoff | items 4–6 | Close the window mid-run → notification when done → reopen shows the full timeline | diff --git a/mkdocs.yml b/mkdocs.yml index e6656ee..006c1e4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -24,3 +24,4 @@ nav: - Architecture: - Authentication: authentication.md - Desktop Client (proposal): desktop-client.md + - Cowork UI plan: cowork-ui-plan.md