ryan/simbarag

Agent improvements: image passthrough, context fixes, thinking indicator, observability #50

Open

ryan wants to merge 5 commits from agent-improvements into main

Author	SHA1	Message	Date
Ryan Chen	d52c934b9a	Add structured timing log to stream_query for slow-request diagnosis Every SSE request emits a JSON summary line to the logger on completion, cancellation, or error: {"event":"stream_request","conv":"...","user":"ryan", "total_ms":45210,"thinking_ms":38100,"ttft_ms":38200, "tokens":112,"tools":[{"tool":"web_search","ms":820}], "model":"llama","cancelled":false} Fields: - total_ms end-to-end wall time - thinking_ms time from first LLM call to first content token (reasoning) - ttft_ms time to first content token from request start - tokens approximate content token count (stream deltas) - tools each tool invoked with wall-clock duration - model LLM name from LangChain metadata (catches fallback to OpenAI) - cancelled true when client disconnected mid-stream Tool timers use the LangChain run_id to correctly pair start/end even when the same tool is called multiple times in one turn. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-09-12 15:43:02 -04:00
Ryan Chen	d28dfb9435	Don't cancel stream on navigation away Aborting the fetch on component unmount sent a disconnect to the server, which cancelled the in-flight generation. Now the stream runs to completion in the background (isMountedRef guards prevent stale state updates), the response is persisted, and it's there when the user returns to the conversation. stopGeneration() still aborts immediately on explicit user cancel. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-09-12 15:43:02 -04:00
Ryan Chen	b530974438	Show a thinking indicator while the model reasons qwen3.8-27b is a reasoning model: it can think for hundreds of tokens before any visible content streams. The SSE stream only forwarded content deltas, so the UI showed dead air for the whole think and long answers looked hung (users cancel ~1 min in). The stream now emits a throttled {"type": "thinking"} heartbeat while the current model turn has produced no content yet, and the frontend renders a pulsing "Thinking..." pill until content or a tool call arrives. The heartbeat also keeps bytes flowing on the SSE connection during long thinks, so idle-timeout disconnects in the proxy chain can't kill the stream. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-09-12 15:43:02 -04:00
Ryan Chen	395c97958f	Fix history roles and persist tool results for cross-turn context Two fixes for the agent losing track of conversation context: - History replay compared msg.speaker (a Speaker enum) to the string "user", which is always false, so every prior message — including the user's — was sent to the LLM with role "assistant". Compare against Speaker.USER instead. Affected web, iMessage, WhatsApp, and email. - Tool outputs (web searches, RAG chunks) were discarded after each turn, so follow-up questions about fetched content had nothing to reference. Persist them as speaker="tool" messages (capped at 4000 chars) and replay them in history as assistant context. The conversation GET endpoint serves only their header line so the UI tool pill stays compact. Also bumps the replayed history window from 10 to 20 messages since tool messages now share it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-09-12 15:43:02 -04:00
Ryan Chen	0389ec45c8	Pass images directly to the agent as multimodal content Image messages previously ran two sequential LLM generations: a full non-streamed vision call to describe the image, then a second agent call with the description pasted in as text. The agent's llama-server model is already vision-capable, so attach the image to the user message as an OpenAI-style image_url content part and let the single agent call see it. Roughly halves image-turn latency and streams from the start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-09-12 15:43:02 -04:00