Match llama-server speed and context handling for images and multi-turn chat #48

2026-09-12T15:14:49-04:00

ryan commented

2026-09-12 15:14:49 -04:00

Investigation into why asksimba was much slower and worse with context than the llama-server UI turned up three causes, fixed in two commits:

Commit 1 — images: one LLM call instead of two. Image messages ran a full non-streamed vision call to describe the image, then a second agent call with the description pasted in. The agent model (qwen3.8-27b on llama-server) is already vision-capable, so the image is now attached to the user message as multimodal content and the single agent call sees it directly. Roughly halves image-turn latency; tokens stream from the start.

Commit 2 — context fixes:

History replay compared msg.speaker (a Speaker enum) to the string "user" — always false — so every prior message, including the user's, was replayed with role assistant. The model saw a transcript where it apparently said everything itself. Fixed in web, iMessage, WhatsApp, and email paths.
Tool outputs (web searches, RAG chunks) were discarded after each turn; follow-ups had nothing to reference. They're now persisted as speaker="tool" messages (capped at 4000 chars), replayed in history, and served to the UI as just their header line so the tool pill stays compact. No DB migration needed — speaker is a plain varchar and the frontend already knows the tool speaker.
History window bumped 10 → 20 messages since tool messages share it.

Measured context budget for reference: the agent's fixed overhead (system prompt + 26 tool schemas) is ~5.8k tokens of the 32k/slot window, hence the 4000-char cap on persisted tool results.

74/74 tests pass.

🤖 Generated with Claude Code

Investigation into why asksimba was much slower and worse with context than the llama-server UI turned up three causes, fixed in two commits: **Commit 1 — images: one LLM call instead of two.** Image messages ran a full non-streamed vision call to describe the image, then a second agent call with the description pasted in. The agent model (qwen3.8-27b on llama-server) is already vision-capable, so the image is now attached to the user message as multimodal content and the single agent call sees it directly. Roughly halves image-turn latency; tokens stream from the start. **Commit 2 — context fixes:** - History replay compared `msg.speaker` (a `Speaker` enum) to the string `"user"` — always false — so every prior message, including the user's, was replayed with role `assistant`. The model saw a transcript where it apparently said everything itself. Fixed in web, iMessage, WhatsApp, and email paths. - Tool outputs (web searches, RAG chunks) were discarded after each turn; follow-ups had nothing to reference. They're now persisted as `speaker="tool"` messages (capped at 4000 chars), replayed in history, and served to the UI as just their header line so the tool pill stays compact. No DB migration needed — `speaker` is a plain varchar and the frontend already knows the `tool` speaker. - History window bumped 10 → 20 messages since tool messages share it. Measured context budget for reference: the agent's fixed overhead (system prompt + 26 tool schemas) is ~5.8k tokens of the 32k/slot window, hence the 4000-char cap on persisted tool results. 74/74 tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ryan added 2 commits 2026-09-12 15:14:49 -04:00

Pass images directly to the agent as multimodal content 7527ef3adf

Image messages previously ran two sequential LLM generations: a full
non-streamed vision call to describe the image, then a second agent call
with the description pasted in as text. The agent's llama-server model is
already vision-capable, so attach the image to the user message as an
OpenAI-style image_url content part and let the single agent call see it.
Roughly halves image-turn latency and streams from the start.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fix history roles and persist tool results for cross-turn context ca37c4a780

Two fixes for the agent losing track of conversation context:

- History replay compared msg.speaker (a Speaker enum) to the string
  "user", which is always false, so every prior message — including the
  user's — was sent to the LLM with role "assistant". Compare against
  Speaker.USER instead. Affected web, iMessage, WhatsApp, and email.

- Tool outputs (web searches, RAG chunks) were discarded after each
  turn, so follow-up questions about fetched content had nothing to
  reference. Persist them as speaker="tool" messages (capped at 4000
  chars) and replay them in history as assistant context. The
  conversation GET endpoint serves only their header line so the UI
  tool pill stays compact.

Also bumps the replayed history window from 10 to 20 messages since
tool messages now share it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ryan merged commit a591cfa343 into main

2026-09-12 15:15:25 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ryan/simbarag#48