Render markdown in gist and project sidebar titles; add content visibility and GIF picker
DevPlace CI / test (push) Failing after 1h5m29s

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 17:59:21 +01:00
co-authored by Claude Sonnet 5.5
parent bc11475f29
commit 05d6633204
154 changed files with 6394 additions and 289 deletions
@@ -36,6 +36,17 @@ Callers may send an optional `X-App-Reference` header to tag gateway calls by ap
`apps.py` is the single specification of every internal gateway app: the `GATEWAY_APPS` registry (`GatewayApp(reference, name)`, title `DevPlace <name>`), the `APP_REFERENCE_*` constants callers import, `validate_app_reference` and `attribution_headers(app_reference)`. Every upstream request (chat, embeddings, images, vision sub-call, fallback retries) carries, for ALL providers and not only OpenRouter: `X-OpenRouter-Title` (OpenRouter's current app-title header, https://openrouter.ai/docs/app-attribution), the legacy `X-Title` with the same value, and `HTTP-Referer` = `DEVPLACE_SITE_URL` when set. The title is per app (`DevPlace Devii`, `DevPlace News`, ...), so a provider dashboard splits spend by sub-app; unknown or `default` references fall back to `DevPlace`. OpenRouter documents no sub-app field beyond the title, so the title is the carrier. Apps: Devii, News, Bots, AI Correction, AI Modifier, Quiz Grading, SEO Metadata, DeepSearch, AI Usage Analyzer, Issue Enhance, Issue Planning, DB API, Awards, Image Generation, Community Discovery, Community Sync. A new internal caller adds one registry row and sends `X-App-Reference`. `gateway_complete(..., app_reference=)` carries the label for correction-style callers.
## Tool-call emulation (`tool_emulation.py`, per-model `tools_mode`)
A model route (`gateway_models.tools_mode`, `native` default or `emulated`) declares whether the model supports tool calling. **The admin's word is the truth: there is no probing and no auto-detection**, and an `emulated` flag is never second-guessed. The flag rides the route overlay as `gateway_tools_mode`, so it is decided per attempt and a fallback chain may freely mix native and emulated models. The client keeps speaking plain OpenAI: it sends `tools` and gets `tool_calls` back, whatever the model underneath.
- **Request seam.** `gateway._chat_upstream_body` calls `emulate_request(payload)` first, BEFORE the Responses/Meta transpile, so the Meta path (`chat_to_responses`, Muse aliasing) simply receives a tool-less Chat payload. As a side effect the Meta reserved tool-name gate never fires, because `tools[]` is gone; `_response_tool_aliases` returns `{}` for emulated routes. `emulate_request` strips `tools`/`tool_choice`/`parallel_tool_calls`/`functions`/`function_call`, appends a Hermes-format catalog (`<tools>` JSON lines plus the `<tool_call>{"name","arguments"}</tool_call>` contract) to the system message (creating one when absent), maps `tool_choice` (`none` drops the catalog, `required`/named add a MUST directive, `parallel_tool_calls: false` adds a one-call rule), rewrites history (assistant `tool_calls` become `<tool_call>` blocks, `role: tool`/`function` messages become one merged user message of `<tool_response>` blocks matched back to names by call id), and adds `<tool_response>` as a stop sequence so the model cannot invent a result. It never mutates the caller's payload.
- **Response seams.** Non-streaming: `convert_completion` parses blocks out of `message.content` into `tool_calls` (generated `call_` ids, `finish_reason: "tool_calls"`, leftover text kept, `content: null` when empty), applied on both the chat and the Responses non-stream branches. Streaming: `_emulate_tool_stream` wraps the finished SSE iterator, so ledger/TTFT accounting stays inside the inner generator. `ToolCallStream` relays text live with a partial-`<tool_call>` holdback, buffers once a call opens, and emits each call whole in ONE `tool_calls` delta (never incremental arguments: that is where vLLM's Hermes parser kept breaking). Usage chunks and non-content deltas pass through.
- **Parsing is tolerant, validation strict.** Accepts code fences, an array of calls in one block, `parameters` for `arguments`, string-encoded arguments, and an unterminated final block. A call is valid only if the name is declared and the schema's `required` keys are present. An invalid block is NEVER dropped silently: it stays in `content` as text and is logged (`emulated tools: ...`). Known gap: there is no repair retry; an invalid call is returned as text.
- **Hard caps (400, no upstream call, no ledger row).** `MAX_TOOLS` 64, `MAX_CATALOG_CHARS` 60000, only `function` tools, `MAX_CALLS_PER_TURN` 8 (extra calls dropped with a warning). The catalog costs prompt tokens on every call, billed like any input.
- **Surfaces.** `ModelRouteIn.tools_mode`, the admin model form (**Tool calling** select), the `tools: emulated` badge, `POST /admin/gateway/models` (`tools_mode`, documented in `docs_api`) and the Devii `gateway_model_set` tool.
- **Not covered:** constrained/guided decoding (provider-specific), and the format is Hermes JSON only. Tests: `tests/unit/services/openai_gateway/tool_emulation.py` (pure) and the `test_emulated_tools_*` tests in `gateway.py` (through the real Meta Responses path, streaming and non-streaming).
## Per-worker runtime
**Serves in every worker.** Config/enabled come from `site_settings` (via the service's `get_config()`/`is_enabled()`), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (`GatewayRuntime`: `httpx.AsyncClient` pool + `asyncio.Semaphore` sized to `gateway_instances`, plus counters and a `VisionCache`) is created lazily on first request and rebuilt when `instances`/`timeout`/cache size change. `gateway_instances` is the **scaling knob** (concurrency per worker); process scaling is uvicorn workers.
+91 -5
View File
@@ -30,6 +30,15 @@ from devplacepy.services.openai_gateway.responses import (
responses_object_to_chat_completion,
)
from devplacepy.services.openai_gateway.system_message import apply_system_directives
from devplacepy.services.openai_gateway.tool_emulation import (
ToolCallStream,
ToolEmulationError,
convert_completion,
declared_tools,
emulate_request,
emulation_enabled,
parallel_allowed,
)
from devplacepy.services.openai_gateway.thinking import (
apply_thinking,
client_thinking_enabled,
@@ -151,6 +160,8 @@ def _chat_upstream_body(
) -> dict:
# The handler reasons about a Chat payload throughout (routing, vision,
# precheck, fallback); only the bytes on the wire change per protocol.
if emulation_enabled(cfg):
payload = emulate_request(payload)
if protocol != "responses":
return payload
if muse_code.is_meta_upstream(cfg.get("gateway_upstream_url")):
@@ -183,13 +194,69 @@ def _response_tool_aliases(payload: dict, protocol: str, cfg: dict) -> dict:
# Rebuilds the exact map _chat_upstream_body applied on the wire for
# this attempt (same payload, same deterministic rule), so response
# names restore without threading state through the fallback path.
if protocol != "responses":
if protocol != "responses" or emulation_enabled(cfg):
return {}
if not muse_code.is_meta_upstream(cfg.get("gateway_upstream_url")):
return {}
return muse_code.tool_aliases_for(payload)
def _tool_emulation_rejection(error: ToolEmulationError) -> JSONResponse:
return JSONResponse(
status_code=400,
content={"error": {"message": str(error), "type": "invalid_request_error"}},
)
def _emulated_tools(payload: dict, cfg: dict):
# Same rebuild-from-the-attempt's-own-payload rule as _response_tool_aliases:
# the declared tools come from the Chat payload of the attempt that was
# actually sent, so a fallback crossing into or out of an emulated model
# needs no threaded state.
if not emulation_enabled(cfg):
return None
declared = declared_tools(payload)
if not declared:
return None
return declared, parallel_allowed(payload)
def _emulate_completion(data: dict, emulated, log) -> dict:
if emulated is None:
return data
declared, allow_parallel = emulated
data, warnings = convert_completion(data, declared, allow_parallel)
for warning in warnings[:3]:
log(f"emulated tools: {warning}")
return data
async def _emulate_tool_stream(inner, emulated, log):
declared, allow_parallel = emulated
translator = ToolCallStream(declared, allow_parallel)
try:
async for line in inner:
body = line[len("data:") :].strip() if line.startswith("data:") else ""
if body == "[DONE]":
for out in translator.finish():
yield f"data: {json.dumps(out)}\n\n"
yield line
continue
try:
chunk = json.loads(body)
except ValueError:
yield line
continue
if not isinstance(chunk, dict) or chunk.get("error"):
yield line
continue
for out in translator.feed(chunk):
yield f"data: {json.dumps(out)}\n\n"
finally:
for warning in translator.warnings[:3]:
log(f"emulated tools: {warning}")
def _apply_served_model(headers: dict, resp) -> dict:
# Everything else in `headers` (cost, tokens, latency, context) is our
# own measurement and must never be overwritten by an upstream that
@@ -633,7 +700,10 @@ class GatewayRuntime:
break
send_start = time.monotonic()
send_body = _chat_upstream_body(payload, body, cfg, protocol, log)
try:
send_body = _chat_upstream_body(payload, body, cfg, protocol, log)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
resp, exc, timing = await self._send(
client,
sem,
@@ -683,9 +753,12 @@ class GatewayRuntime:
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
send_start = time.monotonic()
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log
)
try:
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log
)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
resp, exc, timing = await self._send(
client,
sem,
@@ -843,6 +916,9 @@ class GatewayRuntime:
send_start,
log,
)
emulated = _emulated_tools(payload, cfg)
if emulated is not None:
body_iter = _emulate_tool_stream(body_iter, emulated, log)
return StreamingResponse(
body_iter,
media_type="text/event-stream",
@@ -878,6 +954,9 @@ class GatewayRuntime:
converted = muse_code.restore_completion_names(
converted, _response_tool_aliases(payload, protocol, cfg)
)
converted = _emulate_completion(
converted, _emulated_tools(payload, cfg), log
)
if convert_warnings:
log(
"responses->chat conversion warnings: "
@@ -892,6 +971,13 @@ class GatewayRuntime:
)
resp_headers = finalize(200, True, None, data.get("usage"))
log(f"chat POST -> 200 ({timing['upstream_latency_ms']:.0f}ms)")
emulated = _emulated_tools(payload, cfg)
if emulated is not None:
return Response(
content=json.dumps(_emulate_completion(data, emulated, log)).encode("utf-8"),
media_type="application/json",
headers=resp_headers,
)
return Response(
content=resp.content,
media_type="application/json",
@@ -16,6 +16,10 @@ from devplacepy.database import (
sync_local_cache,
)
from devplacepy.services.openai_gateway import muse_code, opencode_zen
from devplacepy.services.openai_gateway.tool_emulation import (
TOOLS_MODE_NATIVE,
TOOLS_MODES,
)
logger = logging.getLogger(__name__)
@@ -205,6 +209,8 @@ MODEL_HEALTH_COLUMNS = (
"health_checked_at",
)
MODEL_TOOLS_COLUMN = "tools_mode"
HEALTH_OK = "ok"
HEALTH_BAD = "bad"
@@ -241,6 +247,7 @@ def ensure_tables() -> None:
"is_active INTEGER DEFAULT 1, health_status TEXT DEFAULT '', "
"health_code TEXT DEFAULT '', health_message TEXT DEFAULT '', "
"health_checked_at TEXT DEFAULT '', "
"tools_mode TEXT DEFAULT 'native', "
"created_at TEXT, updated_at TEXT)"
)
models_table = get_table(MODELS_TABLE)
@@ -252,6 +259,8 @@ def ensure_tables() -> None:
for column in MODEL_HEALTH_COLUMNS:
if not models_table.has_column(column):
models_table.create_column_by_example(column, "")
if not models_table.has_column(MODEL_TOOLS_COLUMN):
models_table.create_column_by_example(MODEL_TOOLS_COLUMN, TOOLS_MODE_NATIVE)
try:
db.query(
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_providers_name ON "
@@ -340,8 +349,17 @@ class ModelRouteIn(BaseModel):
off_peak_end_minute: Optional[int] = Field(default=None, ge=0, le=1439)
off_peak_discount_pct: float = Field(default=0.0, ge=0, le=100)
fallback_model: str = Field(default="", max_length=128)
tools_mode: str = Field(default=TOOLS_MODE_NATIVE, max_length=16)
is_active: bool = True
@field_validator("tools_mode")
@classmethod
def _clean_tools_mode(cls, value: str) -> str:
value = (value or TOOLS_MODE_NATIVE).strip().lower()
if value not in TOOLS_MODES:
raise ValueError(f"tools_mode must be one of: {', '.join(TOOLS_MODES)}")
return value
@field_validator("source_model", "target_model")
@classmethod
def _clean_model(cls, value: str) -> str:
@@ -407,6 +425,7 @@ class ModelRoute:
health_code: str
health_message: str
health_checked_at: str
tools_mode: str
def _opt_float(row: dict, key: str) -> Optional[float]:
@@ -448,6 +467,7 @@ def _route_from_row(row: dict) -> ModelRoute:
health_code=str(row.get("health_code") or ""),
health_message=str(row.get("health_message") or ""),
health_checked_at=str(row.get("health_checked_at") or ""),
tools_mode=str(row.get("tools_mode") or TOOLS_MODE_NATIVE),
)
@@ -579,6 +599,7 @@ class ModelStore:
"off_peak_end_minute": payload.off_peak_end_minute,
"off_peak_discount_pct": payload.off_peak_discount_pct,
"fallback_model": payload.fallback_model,
"tools_mode": payload.tools_mode,
"is_active": 1 if payload.is_active else 0,
"updated_at": _now(),
}
@@ -789,6 +810,7 @@ def chat_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[dic
"gateway_off_peak_end_minute": route.off_peak_end_minute,
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
"gateway_provider_name": route.provider or "default",
"gateway_tools_mode": route.tools_mode,
}
if route.provider:
_provider_overlay(
@@ -0,0 +1,444 @@
# retoor <retoor@molodetz.nl>
import copy
import json
import re
import uuid
from typing import Optional
TOOLS_MODE_NATIVE = "native"
TOOLS_MODE_EMULATED = "emulated"
TOOLS_MODES = (TOOLS_MODE_NATIVE, TOOLS_MODE_EMULATED)
MAX_TOOLS = 64
MAX_CATALOG_CHARS = 60_000
MAX_CALLS_PER_TURN = 8
MAX_STOP_SEQUENCES = 4
CALL_OPEN = "<tool_call>"
CALL_CLOSE = "</tool_call>"
RESPONSE_OPEN = "<tool_response>"
RESPONSE_CLOSE = "</tool_response>"
_CALL_BLOCK = re.compile(
re.escape(CALL_OPEN) + r"(.*?)(?:" + re.escape(CALL_CLOSE) + r"|\Z)", re.DOTALL
)
_FENCE = re.compile(r"^```[a-zA-Z0-9_-]*\s*\n?(.*?)\n?```$", re.DOTALL)
REMOVED_PAYLOAD_KEYS = (
"tools",
"tool_choice",
"parallel_tool_calls",
"functions",
"function_call",
)
class ToolEmulationError(ValueError):
pass
def emulation_enabled(cfg: dict) -> bool:
return (cfg.get("gateway_tools_mode") or TOOLS_MODE_NATIVE) == TOOLS_MODE_EMULATED
def _function_specs(payload: dict) -> list[dict]:
tools = payload.get("tools")
if not tools:
return []
if not isinstance(tools, list):
raise ToolEmulationError("tools must be a list")
if len(tools) > MAX_TOOLS:
raise ToolEmulationError(
f"too many tools for an emulated-tools model: {len(tools)} > {MAX_TOOLS}"
)
specs = []
for tool in tools:
if not isinstance(tool, dict) or tool.get("type") != "function":
raise ToolEmulationError(
"an emulated-tools model supports only tools of type 'function'"
)
function = tool.get("function")
if not isinstance(function, dict) or not str(function.get("name") or "").strip():
raise ToolEmulationError("every tool needs function.name")
specs.append(function)
return specs
def declared_tools(payload: dict) -> dict[str, dict]:
return {str(spec["name"]): spec for spec in _function_specs(payload)}
def parallel_allowed(payload: dict) -> bool:
return payload.get("parallel_tool_calls") is not False
def _choice_directive(payload: dict) -> tuple[bool, str]:
choice = payload.get("tool_choice")
if choice == "none":
return False, ""
if choice == "required":
return True, "You MUST call at least one tool in this reply."
if isinstance(choice, dict):
name = str((choice.get("function") or {}).get("name") or "").strip()
if name:
return True, f'You MUST call the tool "{name}" in this reply.'
return True, ""
def _catalog_text(specs: list[dict]) -> str:
lines = [json.dumps({"type": "function", "function": spec}, ensure_ascii=False) for spec in specs]
catalog = "\n".join(lines)
if len(catalog) > MAX_CATALOG_CHARS:
raise ToolEmulationError(
f"tool catalog too large for an emulated-tools model: "
f"{len(catalog)} > {MAX_CATALOG_CHARS} characters"
)
return catalog
def build_instruction(payload: dict, specs: list[dict]) -> str:
include, directive = _choice_directive(payload)
if not include or not specs:
return ""
parts = [
"You can call tools. The available tools are listed below inside "
"<tools></tools>, one JSON object per line.",
"<tools>\n" + _catalog_text(specs) + "\n</tools>",
"To call a tool, reply with one block per call, in exactly this format:\n"
f"{CALL_OPEN}\n"
'{"name": "<tool name>", "arguments": {<arguments object>}}\n'
f"{CALL_CLOSE}",
"Rules: use only the tool names listed above. The arguments must be a "
"valid JSON object that matches the tool's parameter schema. After the "
"call blocks, stop: the results arrive in the next message as "
f"{RESPONSE_OPEN} blocks. Never write {RESPONSE_OPEN} yourself. When no "
"tool is needed, answer normally without any call block.",
]
if not parallel_allowed(payload):
parts.append("Emit at most one tool call per reply.")
if directive:
parts.append(directive)
return "\n\n".join(parts)
def _text_of(content) -> str:
if content is None:
return ""
if isinstance(content, str):
return content
if isinstance(content, list):
return "\n".join(
str(part.get("text", ""))
for part in content
if isinstance(part, dict) and part.get("type") == "text"
)
return str(content)
def _arguments_object(raw):
if isinstance(raw, dict):
return raw
if isinstance(raw, str):
try:
parsed = json.loads(raw) if raw.strip() else {}
except ValueError:
return raw
return parsed
return {}
def _call_block(name: str, arguments) -> str:
body = json.dumps({"name": name, "arguments": arguments}, ensure_ascii=False)
return f"{CALL_OPEN}\n{body}\n{CALL_CLOSE}"
def _response_block(call_id: str, name: str, content) -> str:
body = json.dumps(
{"tool_call_id": call_id, "name": name, "content": _text_of(content)},
ensure_ascii=False,
)
return f"{RESPONSE_OPEN}\n{body}\n{RESPONSE_CLOSE}"
def _rewrite_history(messages: list) -> list:
names: dict[str, str] = {}
rewritten: list = []
for message in messages:
if not isinstance(message, dict):
rewritten.append(message)
continue
role = message.get("role")
if role == "assistant" and message.get("tool_calls"):
blocks = []
for call in message["tool_calls"]:
function = (call or {}).get("function") or {}
name = str(function.get("name") or "")
call_id = str((call or {}).get("id") or "")
if call_id:
names[call_id] = name
blocks.append(_call_block(name, _arguments_object(function.get("arguments"))))
text = _text_of(message.get("content")).strip()
converted = {k: v for k, v in message.items() if k != "tool_calls"}
converted["content"] = "\n".join(([text] if text else []) + blocks)
rewritten.append(converted)
elif role in ("tool", "function"):
call_id = str(message.get("tool_call_id") or "")
name = str(message.get("name") or names.get(call_id) or "")
block = _response_block(call_id, name, message.get("content"))
previous = rewritten[-1] if rewritten else None
if (
isinstance(previous, dict)
and previous.get("_emulated_tool_results")
):
previous["content"] += "\n" + block
else:
rewritten.append(
{"role": "user", "content": block, "_emulated_tool_results": True}
)
else:
rewritten.append(message)
for message in rewritten:
if isinstance(message, dict):
message.pop("_emulated_tool_results", None)
return rewritten
def _with_system_instruction(messages: list, instruction: str) -> list:
if not instruction:
return messages
if messages and isinstance(messages[0], dict) and messages[0].get("role") == "system":
first = dict(messages[0])
content = first.get("content")
if isinstance(content, list):
first["content"] = list(content) + [{"type": "text", "text": instruction}]
else:
existing = _text_of(content)
first["content"] = (existing + "\n\n" if existing else "") + instruction
return [first] + messages[1:]
return [{"role": "system", "content": instruction}] + messages
def _with_stop(payload: dict) -> None:
stop = payload.get("stop")
sequences = [stop] if isinstance(stop, str) else list(stop or [])
if RESPONSE_OPEN in sequences or len(sequences) >= MAX_STOP_SEQUENCES:
return
payload["stop"] = sequences + [RESPONSE_OPEN]
def emulate_request(payload: dict) -> dict:
specs = _function_specs(payload)
history = payload.get("messages") or []
has_tool_history = any(
isinstance(m, dict) and (m.get("role") in ("tool", "function") or m.get("tool_calls"))
for m in history
)
if not specs and not has_tool_history:
return payload
converted = {k: copy.copy(v) for k, v in payload.items() if k not in REMOVED_PAYLOAD_KEYS}
messages = _rewrite_history(list(history))
converted["messages"] = _with_system_instruction(messages, build_instruction(payload, specs))
if specs:
_with_stop(converted)
return converted
def _strip_fence(text: str) -> str:
stripped = text.strip()
match = _FENCE.match(stripped)
return match.group(1).strip() if match else stripped
def _call_objects(raw: str) -> list:
try:
parsed = json.loads(_strip_fence(raw))
except ValueError:
return []
items = parsed if isinstance(parsed, list) else [parsed]
return [item for item in items if isinstance(item, dict)]
def _normalize_call(item: dict, declared: dict[str, dict]) -> tuple[Optional[dict], Optional[str]]:
function = item.get("function") if isinstance(item.get("function"), dict) else item
name = str(function.get("name") or "").strip()
if name not in declared:
return None, f"unknown tool {name!r}"
arguments = function.get("arguments", function.get("parameters", {}))
if isinstance(arguments, str):
try:
arguments = json.loads(arguments) if arguments.strip() else {}
except ValueError:
return None, f"arguments of {name!r} are not valid JSON"
if not isinstance(arguments, dict):
return None, f"arguments of {name!r} are not an object"
schema = declared[name].get("parameters")
required = schema.get("required") if isinstance(schema, dict) else None
missing = [key for key in (required or []) if key not in arguments]
if missing:
return None, f"call to {name!r} misses required arguments: {', '.join(missing)}"
return (
{
"id": "call_" + uuid.uuid4().hex[:24],
"type": "function",
"function": {
"name": name,
"arguments": json.dumps(arguments, ensure_ascii=False),
},
},
None,
)
def parse_tool_calls(
text: str, declared: dict[str, dict], allow_parallel: bool = True
) -> tuple[str, list[dict], list[str]]:
if CALL_OPEN not in (text or ""):
return text or "", [], []
calls: list[dict] = []
warnings: list[str] = []
kept: list[str] = []
cursor = 0
for match in _CALL_BLOCK.finditer(text):
kept.append(text[cursor : match.start()])
cursor = match.end()
items = _call_objects(match.group(1))
if not items:
warnings.append("a tool_call block did not contain valid JSON")
kept.append(match.group(0))
continue
for item in items:
call, problem = _normalize_call(item, declared)
if call is None:
warnings.append(problem or "invalid tool call")
kept.append(match.group(0))
continue
calls.append(call)
kept.append(text[cursor:])
if not allow_parallel and len(calls) > 1:
warnings.append(f"parallel calls disabled: dropped {len(calls) - 1} extra call(s)")
calls = calls[:1]
if len(calls) > MAX_CALLS_PER_TURN:
warnings.append(f"dropped {len(calls) - MAX_CALLS_PER_TURN} call(s) over the per-turn cap")
calls = calls[:MAX_CALLS_PER_TURN]
return "".join(kept).strip(), calls, warnings
def convert_completion(
completion: dict, declared: dict[str, dict], allow_parallel: bool = True
) -> tuple[dict, list[str]]:
warnings: list[str] = []
for choice in completion.get("choices") or []:
message = choice.get("message") if isinstance(choice, dict) else None
if not isinstance(message, dict) or message.get("tool_calls"):
continue
text = message.get("content")
if not isinstance(text, str):
continue
remaining, calls, found = parse_tool_calls(text, declared, allow_parallel)
warnings.extend(found)
if calls:
message["tool_calls"] = calls
message["content"] = remaining or None
choice["finish_reason"] = "tool_calls"
return completion, warnings
def _holdback(text: str) -> int:
longest = min(len(CALL_OPEN) - 1, len(text))
for size in range(longest, 0, -1):
if CALL_OPEN.startswith(text[-size:]):
return size
return 0
class ToolCallStream:
def __init__(self, declared: dict[str, dict], allow_parallel: bool = True):
self.declared = declared
self.allow_parallel = allow_parallel
self.warnings: list[str] = []
self._base: dict = {}
self._pending = ""
self._call_text = ""
self._in_call = False
self._flushed = False
self._saw_calls = False
def _chunk(self, delta: dict, finish_reason=None) -> dict:
chunk = {k: v for k, v in self._base.items() if k not in ("choices", "usage")}
chunk["choices"] = [{"index": 0, "delta": delta, "finish_reason": finish_reason}]
return chunk
def _content_chunk(self, text: str) -> list[dict]:
return [self._chunk({"content": text})] if text else []
def _consume(self, text: str) -> list[dict]:
if self._in_call:
self._call_text += text
return []
self._pending += text
index = self._pending.find(CALL_OPEN)
if index >= 0:
out = self._content_chunk(self._pending[:index])
self._call_text = self._pending[index:]
self._pending = ""
self._in_call = True
return out
keep = _holdback(self._pending)
emit = self._pending[: len(self._pending) - keep]
self._pending = self._pending[len(self._pending) - keep :]
return self._content_chunk(emit)
def flush(self) -> list[dict]:
if self._flushed:
return []
self._flushed = True
if not self._in_call:
text, self._pending = self._pending, ""
return self._content_chunk(text)
remaining, calls, found = parse_tool_calls(
self._call_text, self.declared, self.allow_parallel
)
self.warnings.extend(found)
out = self._content_chunk(remaining)
if calls:
self._saw_calls = True
out.append(
self._chunk(
{
"tool_calls": [
{"index": i, **call} for i, call in enumerate(calls)
]
}
)
)
return out
def feed(self, chunk: dict) -> list[dict]:
choices = chunk.get("choices") or []
if not choices:
return [chunk]
self._base = chunk
choice = choices[0]
delta = choice.get("delta") or {}
out: list[dict] = []
text = delta.get("content")
passthrough = {k: v for k, v in delta.items() if k != "content"}
if isinstance(text, str) and text:
out.extend(self._consume(text))
finish = choice.get("finish_reason")
if passthrough:
out.append(self._chunk(passthrough))
if finish is not None:
out.extend(self.flush())
out.append(
self._chunk(
{}, "tool_calls" if self._saw_calls and finish == "stop" else finish
)
)
if chunk.get("usage"):
out.append({**{k: v for k, v in chunk.items() if k != "choices"}, "choices": []})
return out
def finish(self) -> list[dict]:
return self.flush()