# Small Model Tool Reliability

## Metadata

| Field | Value |
|---|---|
| Institute | Molodetz Institute — Research & Retrieval Division |
| Subject | agentic-tool-loop-design |
| Document type | Source Dossier |
| Tier | 1 |
| Source title | small-model-tool-reliability |
| Source URL | https://github.com/elsung/small-model-tool-reliability |
| Source publisher / author | elsung (GitHub) |
| Source type | repository |
| Source date (published) | Unknown — not stated |
| Date researched | 2026-10-11 |
| Researched by | tai (agent), model: Muse Spark, on behalf of retoor |
| Research method | both |
| Confidence | High |
| Status | Active |

## Summary

A field report plus methodology survey on making small local models (Gemma-4-12B / Qwen3-14B class) reliable at tool calling. Its measured headline: on a real multi-tool write phase, a Gemma-4-12B-class model narrated completion instead of emitting the tool call about 25% of the time; `tool_choice: required` did not fix it because the serving parser treats it as an ignorable hint; grammar-constrained decoding fixed it completely (25% to 0% narration over ~45 runs) by making narration unreachable at the token-sampling level.

## Key findings

- Measured baseline: 4/16 trials narrated with no file written (~25% flake) on a real agent CLI with the real prompt; narration text alongside a real call is common and benign, narration *instead of* the call is the failure.
- `tool_choice: required` is not portable: enforced only where the serving parser backs it with a grammar (Qwen3-style native parsers), ignored where it does not (Gemma-4 on the tested build); enforcement lives in the parser, not the API field, and must be verified per model and per build, adversarially.
- Grammar-constrained decoding masks illegal tokens before sampling, so prose narration is not a reachable state; measured 20/20 clean runs after, 45/45 across three phases, with an adversarial "tell me a joke, do NOT write" check still producing valid JSON.
- Grammar design rule: force only the structural skeleton and keep the body permissive, to avoid the "constraint tax" where over-tight formatting degrades small-model task quality.
- Grammar sampling works even where the native templated-tool path is broken, since it is applied by the sampler independently of the chat-template/tool-parsing pipeline.
- Reliability tracks the model family's serving tool-parser maturity, not its size; per-request `max_tokens` caps are required, not optional, with open-ended grammars.
- Recommendation ladder: grammar-constrain single-output phases (skipping the fragile tool loop entirely) and route genuine multi-tool loops to a model whose parser enforces `tool_choice: required`.

## Pros / strengths

- Real measurements on a real agent loop with adversarial verification, including negative results (`tool_choice` failing) reported honestly.
- Precisely names the mechanism (parser maturity, sampler-level masking) instead of hand-waving about "small models are bad at tools."

## Cons / limitations / caveats

- Single-author field report on mid-2026 llama.cpp builds; parser behavior is version-specific and moves fast.
- Grammar-constrained decoding requires sampler access the caller may not have (API users, CLI wrappers), which bounds its applicability.

## Relevance to tai

- This is the exact failure tai users see on its prompt-JSON CLI backends (opencode free models saying "Ik kijk even..." with no tool call): weak-model narration-instead-of-call, with no portable `tool_choice` lever.
- Tai cannot use the report's primary fix (no sampler access through the opencode/grok/claude/codex/gemini CLIs), so the conclusion routes around it: a structural JSON-contract repair retry plus routing genuine multi-tool turns toward backends with native tool-call support, per the report's own ladder.

## Changelog

| Date | Change |
|---|---|
| 2026-10-11 | Initial dossier created. |