Files

T

retoor afc88b8087 ask_user tool, prompt-JSON CLI repair loop, tool-loop research archive

- New ask_user core tool: mid-turn user question with option picking,
  refusing cleanly in autonomous mode, for background workers, and
  without a tty; system prompt points at it instead of prose questions.
- Structural loop control for prompt-JSON CLI backends (opencode,
  grok, claude, codex, gemini): shared cli_json_turn repairs a
  missing JSON envelope exactly once, then accepts prose; prompt
  contract hardened against narrate-without-calling.
- research/agentic-tool-loop-design: 74 sources plus 2 conclusions
  backing the loop-control decisions; INDEX updated to 6 subjects.

2026-10-11 02:54:17 +02:00

3.8 KiB

Raw Blame History

Small Model Tool Reliability

Metadata

Field	Value
Institute	Molodetz Institute — Research & Retrieval Division
Subject	agentic-tool-loop-design
Document type	Source Dossier
Tier	1
Source title	small-model-tool-reliability
Source URL	https://github.com/elsung/small-model-tool-reliability
Source publisher / author	elsung (GitHub)
Source type	repository
Source date (published)	Unknown — not stated
Date researched	2026-10-11
Researched by	tai (agent), model: Muse Spark, on behalf of retoor
Research method	both
Confidence	High
Status	Active

Summary

A field report plus methodology survey on making small local models (Gemma-4-12B / Qwen3-14B class) reliable at tool calling. Its measured headline: on a real multi-tool write phase, a Gemma-4-12B-class model narrated completion instead of emitting the tool call about 25% of the time; tool_choice: required did not fix it because the serving parser treats it as an ignorable hint; grammar-constrained decoding fixed it completely (25% to 0% narration over ~45 runs) by making narration unreachable at the token-sampling level.

Key findings

Measured baseline: 4/16 trials narrated with no file written (~25% flake) on a real agent CLI with the real prompt; narration text alongside a real call is common and benign, narration instead of the call is the failure.
tool_choice: required is not portable: enforced only where the serving parser backs it with a grammar (Qwen3-style native parsers), ignored where it does not (Gemma-4 on the tested build); enforcement lives in the parser, not the API field, and must be verified per model and per build, adversarially.
Grammar-constrained decoding masks illegal tokens before sampling, so prose narration is not a reachable state; measured 20/20 clean runs after, 45/45 across three phases, with an adversarial "tell me a joke, do NOT write" check still producing valid JSON.
Grammar design rule: force only the structural skeleton and keep the body permissive, to avoid the "constraint tax" where over-tight formatting degrades small-model task quality.
Grammar sampling works even where the native templated-tool path is broken, since it is applied by the sampler independently of the chat-template/tool-parsing pipeline.
Reliability tracks the model family's serving tool-parser maturity, not its size; per-request max_tokens caps are required, not optional, with open-ended grammars.
Recommendation ladder: grammar-constrain single-output phases (skipping the fragile tool loop entirely) and route genuine multi-tool loops to a model whose parser enforces tool_choice: required.

Pros / strengths

Real measurements on a real agent loop with adversarial verification, including negative results (tool_choice failing) reported honestly.
Precisely names the mechanism (parser maturity, sampler-level masking) instead of hand-waving about "small models are bad at tools."

Cons / limitations / caveats

Single-author field report on mid-2026 llama.cpp builds; parser behavior is version-specific and moves fast.
Grammar-constrained decoding requires sampler access the caller may not have (API users, CLI wrappers), which bounds its applicability.

Relevance to tai

This is the exact failure tai users see on its prompt-JSON CLI backends (opencode free models saying "Ik kijk even..." with no tool call): weak-model narration-instead-of-call, with no portable tool_choice lever.
Tai cannot use the report's primary fix (no sampler access through the opencode/grok/claude/codex/gemini CLIs), so the conclusion routes around it: a structural JSON-contract repair retry plus routing genuine multi-tool turns toward backends with native tool-call support, per the report's own ladder.

Changelog

Date	Change
2026-10-11	Initial dossier created.

3.8 KiB Raw Blame History