retoor
afc88b8087
- New ask_user core tool: mid-turn user question with option picking, refusing cleanly in autonomous mode, for background workers, and without a tty; system prompt points at it instead of prose questions. - Structural loop control for prompt-JSON CLI backends (opencode, grok, claude, codex, gemini): shared cli_json_turn repairs a missing JSON envelope exactly once, then accepts prose; prompt contract hardened against narrate-without-calling. - research/agentic-tool-loop-design: 74 sources plus 2 conclusions backing the loop-control decisions; INDEX updated to 6 subjects.
56 lines
3.8 KiB
Markdown
56 lines
3.8 KiB
Markdown
# Small Model Tool Reliability
|
|
|
|
## Metadata
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Institute | Molodetz Institute — Research & Retrieval Division |
|
|
| Subject | agentic-tool-loop-design |
|
|
| Document type | Source Dossier |
|
|
| Tier | 1 |
|
|
| Source title | small-model-tool-reliability |
|
|
| Source URL | https://github.com/elsung/small-model-tool-reliability |
|
|
| Source publisher / author | elsung (GitHub) |
|
|
| Source type | repository |
|
|
| Source date (published) | Unknown — not stated |
|
|
| Date researched | 2026-10-11 |
|
|
| Researched by | tai (agent), model: Muse Spark, on behalf of retoor |
|
|
| Research method | both |
|
|
| Confidence | High |
|
|
| Status | Active |
|
|
|
|
## Summary
|
|
|
|
A field report plus methodology survey on making small local models (Gemma-4-12B / Qwen3-14B class) reliable at tool calling. Its measured headline: on a real multi-tool write phase, a Gemma-4-12B-class model narrated completion instead of emitting the tool call about 25% of the time; `tool_choice: required` did not fix it because the serving parser treats it as an ignorable hint; grammar-constrained decoding fixed it completely (25% to 0% narration over ~45 runs) by making narration unreachable at the token-sampling level.
|
|
|
|
## Key findings
|
|
|
|
- Measured baseline: 4/16 trials narrated with no file written (~25% flake) on a real agent CLI with the real prompt; narration text alongside a real call is common and benign, narration *instead of* the call is the failure.
|
|
- `tool_choice: required` is not portable: enforced only where the serving parser backs it with a grammar (Qwen3-style native parsers), ignored where it does not (Gemma-4 on the tested build); enforcement lives in the parser, not the API field, and must be verified per model and per build, adversarially.
|
|
- Grammar-constrained decoding masks illegal tokens before sampling, so prose narration is not a reachable state; measured 20/20 clean runs after, 45/45 across three phases, with an adversarial "tell me a joke, do NOT write" check still producing valid JSON.
|
|
- Grammar design rule: force only the structural skeleton and keep the body permissive, to avoid the "constraint tax" where over-tight formatting degrades small-model task quality.
|
|
- Grammar sampling works even where the native templated-tool path is broken, since it is applied by the sampler independently of the chat-template/tool-parsing pipeline.
|
|
- Reliability tracks the model family's serving tool-parser maturity, not its size; per-request `max_tokens` caps are required, not optional, with open-ended grammars.
|
|
- Recommendation ladder: grammar-constrain single-output phases (skipping the fragile tool loop entirely) and route genuine multi-tool loops to a model whose parser enforces `tool_choice: required`.
|
|
|
|
## Pros / strengths
|
|
|
|
- Real measurements on a real agent loop with adversarial verification, including negative results (`tool_choice` failing) reported honestly.
|
|
- Precisely names the mechanism (parser maturity, sampler-level masking) instead of hand-waving about "small models are bad at tools."
|
|
|
|
## Cons / limitations / caveats
|
|
|
|
- Single-author field report on mid-2026 llama.cpp builds; parser behavior is version-specific and moves fast.
|
|
- Grammar-constrained decoding requires sampler access the caller may not have (API users, CLI wrappers), which bounds its applicability.
|
|
|
|
## Relevance to tai
|
|
|
|
- This is the exact failure tai users see on its prompt-JSON CLI backends (opencode free models saying "Ik kijk even..." with no tool call): weak-model narration-instead-of-call, with no portable `tool_choice` lever.
|
|
- Tai cannot use the report's primary fix (no sampler access through the opencode/grok/claude/codex/gemini CLIs), so the conclusion routes around it: a structural JSON-contract repair retry plus routing genuine multi-tool turns toward backends with native tool-call support, per the report's own ladder.
|
|
|
|
## Changelog
|
|
|
|
| Date | Change |
|
|
|---|---|
|
|
| 2026-10-11 | Initial dossier created. |
|