Answer concisely by default and drop prompt text the tool schemas repeat #82

2026-10-13T03:40:40-04:00

ryan commented

2026-10-13 03:40:40 -04:00

The local model (Qwen3.8-27B on one RTX 3090) is decode-bound, so output length drives latency. The system prompt asked for "thorough, detailed answers" and to "think things through carefully", which lengthened both the answers and the reasoning.

Style: lead with the answer; short paragraphs or a compact list; expand only when asked or when the task needs it (plans, comparisons, requested explanations). Accuracy guidance replaces the deliberation nudge: reasoning tokens cost the same decode time as answer tokens.
Integrations: the per-tool lists that repeated the bound tool docstrings collapse into one paragraph that keeps only routing and behaviour rules the docstrings don't have (search before creating, edit in place, journal note path, when to save memories).
Persona, Simba facts, the mandatory simba_search rule and citation rules are unchanged. The prompt stays static text, so #77's prompt cache still hits.

Size: 1,446 → 897 tokens (cl100k), −38%.

Review: implemented by a Sonnet worker in a herdr pane, then two review rounds. Round 1 removed the "think things through" nudge and em-dashes, and added a guard test against the nudge returning. Round 2 was clean.

544 unit tests, Ruff.

🤖 Generated with Claude Code

The local model (Qwen3.8-27B on one RTX 3090) is decode-bound, so output length drives latency. The system prompt asked for "thorough, detailed answers" and to "think things through carefully", which lengthened both the answers and the reasoning. - **Style:** lead with the answer; short paragraphs or a compact list; expand only when asked or when the task needs it (plans, comparisons, requested explanations). Accuracy guidance replaces the deliberation nudge: reasoning tokens cost the same decode time as answer tokens. - **Integrations:** the per-tool lists that repeated the bound tool docstrings collapse into one paragraph that keeps only routing and behaviour rules the docstrings don't have (search before creating, edit in place, journal note path, when to save memories). - Persona, Simba facts, the mandatory `simba_search` rule and citation rules are unchanged. The prompt stays static text, so #77's prompt cache still hits. **Size:** 1,446 → 897 tokens (cl100k), −38%. **Review:** implemented by a Sonnet worker in a herdr pane, then two review rounds. Round 1 removed the "think things through" nudge and em-dashes, and added a guard test against the nudge returning. Round 2 was clean. 544 unit tests, Ruff. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ryan added 1 commit 2026-10-13 03:40:40 -04:00

Answer concisely by default and drop prompt text the tool schemas repeat

CI / backend (pull_request) Successful in 41s

Details

CI / frontend (pull_request) Failing after 7s

Details

CI / backend (push) Successful in 42s

Details

CI / frontend (push) Failing after 7s

Details

ef3a438048

The local model is decode-bound, so output length drives latency. The system
prompt asked for "thorough, detailed answers" and to "think things through",
which lengthened both answers and reasoning. It now leads with the answer and
expands only on request. Per-integration tool lists that duplicated the bound
tool docstrings collapse to one paragraph of routing and duplicate-avoidance
rules. The prompt drops from about 1,446 to 897 tokens and stays static, so
the llama-server prompt cache still hits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ryan merged commit 0b1600e524 into main

2026-10-13 03:40:43 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ryan/simbarag#82