Agent improvements: image passthrough, context fixes, thinking indicator, observability #50

2026-09-12T15:43:19-04:00

ryan commented

2026-09-12 15:43:19 -04:00

Consolidates the work from this session into a single PR off main.

Changes

Image passthrough — images are now sent directly to the agent as multimodal content instead of running a separate describe-then-answer LLM call. Roughly halves image-turn latency; streaming starts immediately.

Context fixes — two bugs that caused the model to lose track of conversation context:

msg.speaker (a Speaker enum) was compared to the string "user" — always false — so every prior message was replayed as role: assistant. The model saw a transcript where it apparently said everything itself. Fixed across web, iMessage, WhatsApp, and email.
Tool outputs (web searches, RAG chunks) were discarded after each turn. Now persisted as speaker="tool" messages (capped at 4000 chars) and replayed in history. History window bumped 10 → 20 to accommodate.

Thinking indicator — qwen3.8-27b reasons silently for tens of seconds before producing content. The stream now emits a heartbeat every second during the thinking phase; the frontend shows a pulsing "Thinking..." pill so the UI doesn't look hung. The heartbeat also keeps the SSE connection alive through proxy idle-timeouts.

No cancel on nav-away — navigating to a different conversation no longer aborts the in-flight stream. Generation completes in the background, response is persisted, and it's there when you return. The stop button still cancels immediately.

Structured timing logs — every stream-query request emits a JSON summary on completion:

{"event":"stream_request","conv":"...","user":"ryan","total_ms":45210,
 "thinking_ms":38100,"ttft_ms":38200,"tokens":112,
 "tools":[{"tool":"web_search","ms":820}],"model":"llama","cancelled":false}

Grep: journalctl -u raggr-blue | grep stream_request

74/74 tests pass.

🤖 Generated with Claude Code

Consolidates the work from this session into a single PR off main. ## Changes **Image passthrough** — images are now sent directly to the agent as multimodal content instead of running a separate describe-then-answer LLM call. Roughly halves image-turn latency; streaming starts immediately. **Context fixes** — two bugs that caused the model to lose track of conversation context: - `msg.speaker` (a `Speaker` enum) was compared to the string `"user"` — always false — so every prior message was replayed as `role: assistant`. The model saw a transcript where it apparently said everything itself. Fixed across web, iMessage, WhatsApp, and email. - Tool outputs (web searches, RAG chunks) were discarded after each turn. Now persisted as `speaker="tool"` messages (capped at 4000 chars) and replayed in history. History window bumped 10 → 20 to accommodate. **Thinking indicator** — qwen3.8-27b reasons silently for tens of seconds before producing content. The stream now emits a heartbeat every second during the thinking phase; the frontend shows a pulsing "Thinking..." pill so the UI doesn't look hung. The heartbeat also keeps the SSE connection alive through proxy idle-timeouts. **No cancel on nav-away** — navigating to a different conversation no longer aborts the in-flight stream. Generation completes in the background, response is persisted, and it's there when you return. The stop button still cancels immediately. **Structured timing logs** — every `stream-query` request emits a JSON summary on completion: ```json {"event":"stream_request","conv":"...","user":"ryan","total_ms":45210, "thinking_ms":38100,"ttft_ms":38200,"tokens":112, "tools":[{"tool":"web_search","ms":820}],"model":"llama","cancelled":false} ``` Grep: `journalctl -u raggr-blue | grep stream_request` 74/74 tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ryan added 5 commits 2026-09-12 15:43:20 -04:00

Pass images directly to the agent as multimodal content 0389ec45c8

Image messages previously ran two sequential LLM generations: a full
non-streamed vision call to describe the image, then a second agent call
with the description pasted in as text. The agent's llama-server model is
already vision-capable, so attach the image to the user message as an
OpenAI-style image_url content part and let the single agent call see it.
Roughly halves image-turn latency and streams from the start.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fix history roles and persist tool results for cross-turn context 395c97958f

Two fixes for the agent losing track of conversation context:

- History replay compared msg.speaker (a Speaker enum) to the string
  "user", which is always false, so every prior message — including the
  user's — was sent to the LLM with role "assistant". Compare against
  Speaker.USER instead. Affected web, iMessage, WhatsApp, and email.

- Tool outputs (web searches, RAG chunks) were discarded after each
  turn, so follow-up questions about fetched content had nothing to
  reference. Persist them as speaker="tool" messages (capped at 4000
  chars) and replay them in history as assistant context. The
  conversation GET endpoint serves only their header line so the UI
  tool pill stays compact.

Also bumps the replayed history window from 10 to 20 messages since
tool messages now share it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Show a thinking indicator while the model reasons b530974438

qwen3.8-27b is a reasoning model: it can think for hundreds of tokens
before any visible content streams. The SSE stream only forwarded
content deltas, so the UI showed dead air for the whole think and long
answers looked hung (users cancel ~1 min in).

The stream now emits a throttled {"type": "thinking"} heartbeat while
the current model turn has produced no content yet, and the frontend
renders a pulsing "Thinking..." pill until content or a tool call
arrives. The heartbeat also keeps bytes flowing on the SSE connection
during long thinks, so idle-timeout disconnects in the proxy chain
can't kill the stream.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Don't cancel stream on navigation away d28dfb9435

Aborting the fetch on component unmount sent a disconnect to the server,
which cancelled the in-flight generation. Now the stream runs to
completion in the background (isMountedRef guards prevent stale state
updates), the response is persisted, and it's there when the user
returns to the conversation. stopGeneration() still aborts immediately
on explicit user cancel.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add structured timing log to stream_query for slow-request diagnosis d52c934b9a

Every SSE request emits a JSON summary line to the logger on completion,
cancellation, or error:

  {"event":"stream_request","conv":"...","user":"ryan",
   "total_ms":45210,"thinking_ms":38100,"ttft_ms":38200,
   "tokens":112,"tools":[{"tool":"web_search","ms":820}],
   "model":"llama","cancelled":false}

Fields:
- total_ms    end-to-end wall time
- thinking_ms time from first LLM call to first content token (reasoning)
- ttft_ms     time to first content token from request start
- tokens      approximate content token count (stream deltas)
- tools       each tool invoked with wall-clock duration
- model       LLM name from LangChain metadata (catches fallback to OpenAI)
- cancelled   true when client disconnected mid-stream

Tool timers use the LangChain run_id to correctly pair start/end even
when the same tool is called multiple times in one turn.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Checking for merge conflicts…

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin agent-improvements:agent-improvements

git checkout agent-improvements

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ryan/simbarag#50