Render markdown in gist and project sidebar titles; add content visibility and GIF picker
DevPlace CI / test (push) Failing after 1h5m29s
DevPlace CI / test (push) Failing after 1h5m29s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -36,6 +36,17 @@ Callers may send an optional `X-App-Reference` header to tag gateway calls by ap
|
||||
|
||||
`apps.py` is the single specification of every internal gateway app: the `GATEWAY_APPS` registry (`GatewayApp(reference, name)`, title `DevPlace <name>`), the `APP_REFERENCE_*` constants callers import, `validate_app_reference` and `attribution_headers(app_reference)`. Every upstream request (chat, embeddings, images, vision sub-call, fallback retries) carries, for ALL providers and not only OpenRouter: `X-OpenRouter-Title` (OpenRouter's current app-title header, https://openrouter.ai/docs/app-attribution), the legacy `X-Title` with the same value, and `HTTP-Referer` = `DEVPLACE_SITE_URL` when set. The title is per app (`DevPlace Devii`, `DevPlace News`, ...), so a provider dashboard splits spend by sub-app; unknown or `default` references fall back to `DevPlace`. OpenRouter documents no sub-app field beyond the title, so the title is the carrier. Apps: Devii, News, Bots, AI Correction, AI Modifier, Quiz Grading, SEO Metadata, DeepSearch, AI Usage Analyzer, Issue Enhance, Issue Planning, DB API, Awards, Image Generation, Community Discovery, Community Sync. A new internal caller adds one registry row and sends `X-App-Reference`. `gateway_complete(..., app_reference=)` carries the label for correction-style callers.
|
||||
|
||||
## Tool-call emulation (`tool_emulation.py`, per-model `tools_mode`)
|
||||
|
||||
A model route (`gateway_models.tools_mode`, `native` default or `emulated`) declares whether the model supports tool calling. **The admin's word is the truth: there is no probing and no auto-detection**, and an `emulated` flag is never second-guessed. The flag rides the route overlay as `gateway_tools_mode`, so it is decided per attempt and a fallback chain may freely mix native and emulated models. The client keeps speaking plain OpenAI: it sends `tools` and gets `tool_calls` back, whatever the model underneath.
|
||||
|
||||
- **Request seam.** `gateway._chat_upstream_body` calls `emulate_request(payload)` first, BEFORE the Responses/Meta transpile, so the Meta path (`chat_to_responses`, Muse aliasing) simply receives a tool-less Chat payload. As a side effect the Meta reserved tool-name gate never fires, because `tools[]` is gone; `_response_tool_aliases` returns `{}` for emulated routes. `emulate_request` strips `tools`/`tool_choice`/`parallel_tool_calls`/`functions`/`function_call`, appends a Hermes-format catalog (`<tools>` JSON lines plus the `<tool_call>{"name","arguments"}</tool_call>` contract) to the system message (creating one when absent), maps `tool_choice` (`none` drops the catalog, `required`/named add a MUST directive, `parallel_tool_calls: false` adds a one-call rule), rewrites history (assistant `tool_calls` become `<tool_call>` blocks, `role: tool`/`function` messages become one merged user message of `<tool_response>` blocks matched back to names by call id), and adds `<tool_response>` as a stop sequence so the model cannot invent a result. It never mutates the caller's payload.
|
||||
- **Response seams.** Non-streaming: `convert_completion` parses blocks out of `message.content` into `tool_calls` (generated `call_` ids, `finish_reason: "tool_calls"`, leftover text kept, `content: null` when empty), applied on both the chat and the Responses non-stream branches. Streaming: `_emulate_tool_stream` wraps the finished SSE iterator, so ledger/TTFT accounting stays inside the inner generator. `ToolCallStream` relays text live with a partial-`<tool_call>` holdback, buffers once a call opens, and emits each call whole in ONE `tool_calls` delta (never incremental arguments: that is where vLLM's Hermes parser kept breaking). Usage chunks and non-content deltas pass through.
|
||||
- **Parsing is tolerant, validation strict.** Accepts code fences, an array of calls in one block, `parameters` for `arguments`, string-encoded arguments, and an unterminated final block. A call is valid only if the name is declared and the schema's `required` keys are present. An invalid block is NEVER dropped silently: it stays in `content` as text and is logged (`emulated tools: ...`). Known gap: there is no repair retry; an invalid call is returned as text.
|
||||
- **Hard caps (400, no upstream call, no ledger row).** `MAX_TOOLS` 64, `MAX_CATALOG_CHARS` 60000, only `function` tools, `MAX_CALLS_PER_TURN` 8 (extra calls dropped with a warning). The catalog costs prompt tokens on every call, billed like any input.
|
||||
- **Surfaces.** `ModelRouteIn.tools_mode`, the admin model form (**Tool calling** select), the `tools: emulated` badge, `POST /admin/gateway/models` (`tools_mode`, documented in `docs_api`) and the Devii `gateway_model_set` tool.
|
||||
- **Not covered:** constrained/guided decoding (provider-specific), and the format is Hermes JSON only. Tests: `tests/unit/services/openai_gateway/tool_emulation.py` (pure) and the `test_emulated_tools_*` tests in `gateway.py` (through the real Meta Responses path, streaming and non-streaming).
|
||||
|
||||
## Per-worker runtime
|
||||
|
||||
**Serves in every worker.** Config/enabled come from `site_settings` (via the service's `get_config()`/`is_enabled()`), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (`GatewayRuntime`: `httpx.AsyncClient` pool + `asyncio.Semaphore` sized to `gateway_instances`, plus counters and a `VisionCache`) is created lazily on first request and rebuilt when `instances`/`timeout`/cache size change. `gateway_instances` is the **scaling knob** (concurrency per worker); process scaling is uvicorn workers.
|
||||
|
||||
@@ -30,6 +30,15 @@ from devplacepy.services.openai_gateway.responses import (
|
||||
responses_object_to_chat_completion,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.system_message import apply_system_directives
|
||||
from devplacepy.services.openai_gateway.tool_emulation import (
|
||||
ToolCallStream,
|
||||
ToolEmulationError,
|
||||
convert_completion,
|
||||
declared_tools,
|
||||
emulate_request,
|
||||
emulation_enabled,
|
||||
parallel_allowed,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.thinking import (
|
||||
apply_thinking,
|
||||
client_thinking_enabled,
|
||||
@@ -151,6 +160,8 @@ def _chat_upstream_body(
|
||||
) -> dict:
|
||||
# The handler reasons about a Chat payload throughout (routing, vision,
|
||||
# precheck, fallback); only the bytes on the wire change per protocol.
|
||||
if emulation_enabled(cfg):
|
||||
payload = emulate_request(payload)
|
||||
if protocol != "responses":
|
||||
return payload
|
||||
if muse_code.is_meta_upstream(cfg.get("gateway_upstream_url")):
|
||||
@@ -183,13 +194,69 @@ def _response_tool_aliases(payload: dict, protocol: str, cfg: dict) -> dict:
|
||||
# Rebuilds the exact map _chat_upstream_body applied on the wire for
|
||||
# this attempt (same payload, same deterministic rule), so response
|
||||
# names restore without threading state through the fallback path.
|
||||
if protocol != "responses":
|
||||
if protocol != "responses" or emulation_enabled(cfg):
|
||||
return {}
|
||||
if not muse_code.is_meta_upstream(cfg.get("gateway_upstream_url")):
|
||||
return {}
|
||||
return muse_code.tool_aliases_for(payload)
|
||||
|
||||
|
||||
def _tool_emulation_rejection(error: ToolEmulationError) -> JSONResponse:
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={"error": {"message": str(error), "type": "invalid_request_error"}},
|
||||
)
|
||||
|
||||
|
||||
def _emulated_tools(payload: dict, cfg: dict):
|
||||
# Same rebuild-from-the-attempt's-own-payload rule as _response_tool_aliases:
|
||||
# the declared tools come from the Chat payload of the attempt that was
|
||||
# actually sent, so a fallback crossing into or out of an emulated model
|
||||
# needs no threaded state.
|
||||
if not emulation_enabled(cfg):
|
||||
return None
|
||||
declared = declared_tools(payload)
|
||||
if not declared:
|
||||
return None
|
||||
return declared, parallel_allowed(payload)
|
||||
|
||||
|
||||
def _emulate_completion(data: dict, emulated, log) -> dict:
|
||||
if emulated is None:
|
||||
return data
|
||||
declared, allow_parallel = emulated
|
||||
data, warnings = convert_completion(data, declared, allow_parallel)
|
||||
for warning in warnings[:3]:
|
||||
log(f"emulated tools: {warning}")
|
||||
return data
|
||||
|
||||
|
||||
async def _emulate_tool_stream(inner, emulated, log):
|
||||
declared, allow_parallel = emulated
|
||||
translator = ToolCallStream(declared, allow_parallel)
|
||||
try:
|
||||
async for line in inner:
|
||||
body = line[len("data:") :].strip() if line.startswith("data:") else ""
|
||||
if body == "[DONE]":
|
||||
for out in translator.finish():
|
||||
yield f"data: {json.dumps(out)}\n\n"
|
||||
yield line
|
||||
continue
|
||||
try:
|
||||
chunk = json.loads(body)
|
||||
except ValueError:
|
||||
yield line
|
||||
continue
|
||||
if not isinstance(chunk, dict) or chunk.get("error"):
|
||||
yield line
|
||||
continue
|
||||
for out in translator.feed(chunk):
|
||||
yield f"data: {json.dumps(out)}\n\n"
|
||||
finally:
|
||||
for warning in translator.warnings[:3]:
|
||||
log(f"emulated tools: {warning}")
|
||||
|
||||
|
||||
def _apply_served_model(headers: dict, resp) -> dict:
|
||||
# Everything else in `headers` (cost, tokens, latency, context) is our
|
||||
# own measurement and must never be overwritten by an upstream that
|
||||
@@ -633,7 +700,10 @@ class GatewayRuntime:
|
||||
break
|
||||
|
||||
send_start = time.monotonic()
|
||||
send_body = _chat_upstream_body(payload, body, cfg, protocol, log)
|
||||
try:
|
||||
send_body = _chat_upstream_body(payload, body, cfg, protocol, log)
|
||||
except ToolEmulationError as tool_error:
|
||||
return _tool_emulation_rejection(tool_error)
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
@@ -683,9 +753,12 @@ class GatewayRuntime:
|
||||
payload["model"] = model
|
||||
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
|
||||
send_start = time.monotonic()
|
||||
send_body = _chat_upstream_body(
|
||||
payload, body, cfg, protocol, log
|
||||
)
|
||||
try:
|
||||
send_body = _chat_upstream_body(
|
||||
payload, body, cfg, protocol, log
|
||||
)
|
||||
except ToolEmulationError as tool_error:
|
||||
return _tool_emulation_rejection(tool_error)
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
@@ -843,6 +916,9 @@ class GatewayRuntime:
|
||||
send_start,
|
||||
log,
|
||||
)
|
||||
emulated = _emulated_tools(payload, cfg)
|
||||
if emulated is not None:
|
||||
body_iter = _emulate_tool_stream(body_iter, emulated, log)
|
||||
return StreamingResponse(
|
||||
body_iter,
|
||||
media_type="text/event-stream",
|
||||
@@ -878,6 +954,9 @@ class GatewayRuntime:
|
||||
converted = muse_code.restore_completion_names(
|
||||
converted, _response_tool_aliases(payload, protocol, cfg)
|
||||
)
|
||||
converted = _emulate_completion(
|
||||
converted, _emulated_tools(payload, cfg), log
|
||||
)
|
||||
if convert_warnings:
|
||||
log(
|
||||
"responses->chat conversion warnings: "
|
||||
@@ -892,6 +971,13 @@ class GatewayRuntime:
|
||||
)
|
||||
resp_headers = finalize(200, True, None, data.get("usage"))
|
||||
log(f"chat POST -> 200 ({timing['upstream_latency_ms']:.0f}ms)")
|
||||
emulated = _emulated_tools(payload, cfg)
|
||||
if emulated is not None:
|
||||
return Response(
|
||||
content=json.dumps(_emulate_completion(data, emulated, log)).encode("utf-8"),
|
||||
media_type="application/json",
|
||||
headers=resp_headers,
|
||||
)
|
||||
return Response(
|
||||
content=resp.content,
|
||||
media_type="application/json",
|
||||
|
||||
@@ -16,6 +16,10 @@ from devplacepy.database import (
|
||||
sync_local_cache,
|
||||
)
|
||||
from devplacepy.services.openai_gateway import muse_code, opencode_zen
|
||||
from devplacepy.services.openai_gateway.tool_emulation import (
|
||||
TOOLS_MODE_NATIVE,
|
||||
TOOLS_MODES,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -205,6 +209,8 @@ MODEL_HEALTH_COLUMNS = (
|
||||
"health_checked_at",
|
||||
)
|
||||
|
||||
MODEL_TOOLS_COLUMN = "tools_mode"
|
||||
|
||||
HEALTH_OK = "ok"
|
||||
HEALTH_BAD = "bad"
|
||||
|
||||
@@ -241,6 +247,7 @@ def ensure_tables() -> None:
|
||||
"is_active INTEGER DEFAULT 1, health_status TEXT DEFAULT '', "
|
||||
"health_code TEXT DEFAULT '', health_message TEXT DEFAULT '', "
|
||||
"health_checked_at TEXT DEFAULT '', "
|
||||
"tools_mode TEXT DEFAULT 'native', "
|
||||
"created_at TEXT, updated_at TEXT)"
|
||||
)
|
||||
models_table = get_table(MODELS_TABLE)
|
||||
@@ -252,6 +259,8 @@ def ensure_tables() -> None:
|
||||
for column in MODEL_HEALTH_COLUMNS:
|
||||
if not models_table.has_column(column):
|
||||
models_table.create_column_by_example(column, "")
|
||||
if not models_table.has_column(MODEL_TOOLS_COLUMN):
|
||||
models_table.create_column_by_example(MODEL_TOOLS_COLUMN, TOOLS_MODE_NATIVE)
|
||||
try:
|
||||
db.query(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_providers_name ON "
|
||||
@@ -340,8 +349,17 @@ class ModelRouteIn(BaseModel):
|
||||
off_peak_end_minute: Optional[int] = Field(default=None, ge=0, le=1439)
|
||||
off_peak_discount_pct: float = Field(default=0.0, ge=0, le=100)
|
||||
fallback_model: str = Field(default="", max_length=128)
|
||||
tools_mode: str = Field(default=TOOLS_MODE_NATIVE, max_length=16)
|
||||
is_active: bool = True
|
||||
|
||||
@field_validator("tools_mode")
|
||||
@classmethod
|
||||
def _clean_tools_mode(cls, value: str) -> str:
|
||||
value = (value or TOOLS_MODE_NATIVE).strip().lower()
|
||||
if value not in TOOLS_MODES:
|
||||
raise ValueError(f"tools_mode must be one of: {', '.join(TOOLS_MODES)}")
|
||||
return value
|
||||
|
||||
@field_validator("source_model", "target_model")
|
||||
@classmethod
|
||||
def _clean_model(cls, value: str) -> str:
|
||||
@@ -407,6 +425,7 @@ class ModelRoute:
|
||||
health_code: str
|
||||
health_message: str
|
||||
health_checked_at: str
|
||||
tools_mode: str
|
||||
|
||||
|
||||
def _opt_float(row: dict, key: str) -> Optional[float]:
|
||||
@@ -448,6 +467,7 @@ def _route_from_row(row: dict) -> ModelRoute:
|
||||
health_code=str(row.get("health_code") or ""),
|
||||
health_message=str(row.get("health_message") or ""),
|
||||
health_checked_at=str(row.get("health_checked_at") or ""),
|
||||
tools_mode=str(row.get("tools_mode") or TOOLS_MODE_NATIVE),
|
||||
)
|
||||
|
||||
|
||||
@@ -579,6 +599,7 @@ class ModelStore:
|
||||
"off_peak_end_minute": payload.off_peak_end_minute,
|
||||
"off_peak_discount_pct": payload.off_peak_discount_pct,
|
||||
"fallback_model": payload.fallback_model,
|
||||
"tools_mode": payload.tools_mode,
|
||||
"is_active": 1 if payload.is_active else 0,
|
||||
"updated_at": _now(),
|
||||
}
|
||||
@@ -789,6 +810,7 @@ def chat_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[dic
|
||||
"gateway_off_peak_end_minute": route.off_peak_end_minute,
|
||||
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
|
||||
"gateway_provider_name": route.provider or "default",
|
||||
"gateway_tools_mode": route.tools_mode,
|
||||
}
|
||||
if route.provider:
|
||||
_provider_overlay(
|
||||
|
||||
@@ -0,0 +1,444 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import copy
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from typing import Optional
|
||||
|
||||
TOOLS_MODE_NATIVE = "native"
|
||||
TOOLS_MODE_EMULATED = "emulated"
|
||||
TOOLS_MODES = (TOOLS_MODE_NATIVE, TOOLS_MODE_EMULATED)
|
||||
|
||||
MAX_TOOLS = 64
|
||||
MAX_CATALOG_CHARS = 60_000
|
||||
MAX_CALLS_PER_TURN = 8
|
||||
MAX_STOP_SEQUENCES = 4
|
||||
|
||||
CALL_OPEN = "<tool_call>"
|
||||
CALL_CLOSE = "</tool_call>"
|
||||
RESPONSE_OPEN = "<tool_response>"
|
||||
RESPONSE_CLOSE = "</tool_response>"
|
||||
|
||||
_CALL_BLOCK = re.compile(
|
||||
re.escape(CALL_OPEN) + r"(.*?)(?:" + re.escape(CALL_CLOSE) + r"|\Z)", re.DOTALL
|
||||
)
|
||||
_FENCE = re.compile(r"^```[a-zA-Z0-9_-]*\s*\n?(.*?)\n?```$", re.DOTALL)
|
||||
|
||||
REMOVED_PAYLOAD_KEYS = (
|
||||
"tools",
|
||||
"tool_choice",
|
||||
"parallel_tool_calls",
|
||||
"functions",
|
||||
"function_call",
|
||||
)
|
||||
|
||||
|
||||
class ToolEmulationError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def emulation_enabled(cfg: dict) -> bool:
|
||||
return (cfg.get("gateway_tools_mode") or TOOLS_MODE_NATIVE) == TOOLS_MODE_EMULATED
|
||||
|
||||
|
||||
def _function_specs(payload: dict) -> list[dict]:
|
||||
tools = payload.get("tools")
|
||||
if not tools:
|
||||
return []
|
||||
if not isinstance(tools, list):
|
||||
raise ToolEmulationError("tools must be a list")
|
||||
if len(tools) > MAX_TOOLS:
|
||||
raise ToolEmulationError(
|
||||
f"too many tools for an emulated-tools model: {len(tools)} > {MAX_TOOLS}"
|
||||
)
|
||||
specs = []
|
||||
for tool in tools:
|
||||
if not isinstance(tool, dict) or tool.get("type") != "function":
|
||||
raise ToolEmulationError(
|
||||
"an emulated-tools model supports only tools of type 'function'"
|
||||
)
|
||||
function = tool.get("function")
|
||||
if not isinstance(function, dict) or not str(function.get("name") or "").strip():
|
||||
raise ToolEmulationError("every tool needs function.name")
|
||||
specs.append(function)
|
||||
return specs
|
||||
|
||||
|
||||
def declared_tools(payload: dict) -> dict[str, dict]:
|
||||
return {str(spec["name"]): spec for spec in _function_specs(payload)}
|
||||
|
||||
|
||||
def parallel_allowed(payload: dict) -> bool:
|
||||
return payload.get("parallel_tool_calls") is not False
|
||||
|
||||
|
||||
def _choice_directive(payload: dict) -> tuple[bool, str]:
|
||||
choice = payload.get("tool_choice")
|
||||
if choice == "none":
|
||||
return False, ""
|
||||
if choice == "required":
|
||||
return True, "You MUST call at least one tool in this reply."
|
||||
if isinstance(choice, dict):
|
||||
name = str((choice.get("function") or {}).get("name") or "").strip()
|
||||
if name:
|
||||
return True, f'You MUST call the tool "{name}" in this reply.'
|
||||
return True, ""
|
||||
|
||||
|
||||
def _catalog_text(specs: list[dict]) -> str:
|
||||
lines = [json.dumps({"type": "function", "function": spec}, ensure_ascii=False) for spec in specs]
|
||||
catalog = "\n".join(lines)
|
||||
if len(catalog) > MAX_CATALOG_CHARS:
|
||||
raise ToolEmulationError(
|
||||
f"tool catalog too large for an emulated-tools model: "
|
||||
f"{len(catalog)} > {MAX_CATALOG_CHARS} characters"
|
||||
)
|
||||
return catalog
|
||||
|
||||
|
||||
def build_instruction(payload: dict, specs: list[dict]) -> str:
|
||||
include, directive = _choice_directive(payload)
|
||||
if not include or not specs:
|
||||
return ""
|
||||
parts = [
|
||||
"You can call tools. The available tools are listed below inside "
|
||||
"<tools></tools>, one JSON object per line.",
|
||||
"<tools>\n" + _catalog_text(specs) + "\n</tools>",
|
||||
"To call a tool, reply with one block per call, in exactly this format:\n"
|
||||
f"{CALL_OPEN}\n"
|
||||
'{"name": "<tool name>", "arguments": {<arguments object>}}\n'
|
||||
f"{CALL_CLOSE}",
|
||||
"Rules: use only the tool names listed above. The arguments must be a "
|
||||
"valid JSON object that matches the tool's parameter schema. After the "
|
||||
"call blocks, stop: the results arrive in the next message as "
|
||||
f"{RESPONSE_OPEN} blocks. Never write {RESPONSE_OPEN} yourself. When no "
|
||||
"tool is needed, answer normally without any call block.",
|
||||
]
|
||||
if not parallel_allowed(payload):
|
||||
parts.append("Emit at most one tool call per reply.")
|
||||
if directive:
|
||||
parts.append(directive)
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def _text_of(content) -> str:
|
||||
if content is None:
|
||||
return ""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
return "\n".join(
|
||||
str(part.get("text", ""))
|
||||
for part in content
|
||||
if isinstance(part, dict) and part.get("type") == "text"
|
||||
)
|
||||
return str(content)
|
||||
|
||||
|
||||
def _arguments_object(raw):
|
||||
if isinstance(raw, dict):
|
||||
return raw
|
||||
if isinstance(raw, str):
|
||||
try:
|
||||
parsed = json.loads(raw) if raw.strip() else {}
|
||||
except ValueError:
|
||||
return raw
|
||||
return parsed
|
||||
return {}
|
||||
|
||||
|
||||
def _call_block(name: str, arguments) -> str:
|
||||
body = json.dumps({"name": name, "arguments": arguments}, ensure_ascii=False)
|
||||
return f"{CALL_OPEN}\n{body}\n{CALL_CLOSE}"
|
||||
|
||||
|
||||
def _response_block(call_id: str, name: str, content) -> str:
|
||||
body = json.dumps(
|
||||
{"tool_call_id": call_id, "name": name, "content": _text_of(content)},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
return f"{RESPONSE_OPEN}\n{body}\n{RESPONSE_CLOSE}"
|
||||
|
||||
|
||||
def _rewrite_history(messages: list) -> list:
|
||||
names: dict[str, str] = {}
|
||||
rewritten: list = []
|
||||
for message in messages:
|
||||
if not isinstance(message, dict):
|
||||
rewritten.append(message)
|
||||
continue
|
||||
role = message.get("role")
|
||||
if role == "assistant" and message.get("tool_calls"):
|
||||
blocks = []
|
||||
for call in message["tool_calls"]:
|
||||
function = (call or {}).get("function") or {}
|
||||
name = str(function.get("name") or "")
|
||||
call_id = str((call or {}).get("id") or "")
|
||||
if call_id:
|
||||
names[call_id] = name
|
||||
blocks.append(_call_block(name, _arguments_object(function.get("arguments"))))
|
||||
text = _text_of(message.get("content")).strip()
|
||||
converted = {k: v for k, v in message.items() if k != "tool_calls"}
|
||||
converted["content"] = "\n".join(([text] if text else []) + blocks)
|
||||
rewritten.append(converted)
|
||||
elif role in ("tool", "function"):
|
||||
call_id = str(message.get("tool_call_id") or "")
|
||||
name = str(message.get("name") or names.get(call_id) or "")
|
||||
block = _response_block(call_id, name, message.get("content"))
|
||||
previous = rewritten[-1] if rewritten else None
|
||||
if (
|
||||
isinstance(previous, dict)
|
||||
and previous.get("_emulated_tool_results")
|
||||
):
|
||||
previous["content"] += "\n" + block
|
||||
else:
|
||||
rewritten.append(
|
||||
{"role": "user", "content": block, "_emulated_tool_results": True}
|
||||
)
|
||||
else:
|
||||
rewritten.append(message)
|
||||
for message in rewritten:
|
||||
if isinstance(message, dict):
|
||||
message.pop("_emulated_tool_results", None)
|
||||
return rewritten
|
||||
|
||||
|
||||
def _with_system_instruction(messages: list, instruction: str) -> list:
|
||||
if not instruction:
|
||||
return messages
|
||||
if messages and isinstance(messages[0], dict) and messages[0].get("role") == "system":
|
||||
first = dict(messages[0])
|
||||
content = first.get("content")
|
||||
if isinstance(content, list):
|
||||
first["content"] = list(content) + [{"type": "text", "text": instruction}]
|
||||
else:
|
||||
existing = _text_of(content)
|
||||
first["content"] = (existing + "\n\n" if existing else "") + instruction
|
||||
return [first] + messages[1:]
|
||||
return [{"role": "system", "content": instruction}] + messages
|
||||
|
||||
|
||||
def _with_stop(payload: dict) -> None:
|
||||
stop = payload.get("stop")
|
||||
sequences = [stop] if isinstance(stop, str) else list(stop or [])
|
||||
if RESPONSE_OPEN in sequences or len(sequences) >= MAX_STOP_SEQUENCES:
|
||||
return
|
||||
payload["stop"] = sequences + [RESPONSE_OPEN]
|
||||
|
||||
|
||||
def emulate_request(payload: dict) -> dict:
|
||||
specs = _function_specs(payload)
|
||||
history = payload.get("messages") or []
|
||||
has_tool_history = any(
|
||||
isinstance(m, dict) and (m.get("role") in ("tool", "function") or m.get("tool_calls"))
|
||||
for m in history
|
||||
)
|
||||
if not specs and not has_tool_history:
|
||||
return payload
|
||||
converted = {k: copy.copy(v) for k, v in payload.items() if k not in REMOVED_PAYLOAD_KEYS}
|
||||
messages = _rewrite_history(list(history))
|
||||
converted["messages"] = _with_system_instruction(messages, build_instruction(payload, specs))
|
||||
if specs:
|
||||
_with_stop(converted)
|
||||
return converted
|
||||
|
||||
|
||||
def _strip_fence(text: str) -> str:
|
||||
stripped = text.strip()
|
||||
match = _FENCE.match(stripped)
|
||||
return match.group(1).strip() if match else stripped
|
||||
|
||||
|
||||
def _call_objects(raw: str) -> list:
|
||||
try:
|
||||
parsed = json.loads(_strip_fence(raw))
|
||||
except ValueError:
|
||||
return []
|
||||
items = parsed if isinstance(parsed, list) else [parsed]
|
||||
return [item for item in items if isinstance(item, dict)]
|
||||
|
||||
|
||||
def _normalize_call(item: dict, declared: dict[str, dict]) -> tuple[Optional[dict], Optional[str]]:
|
||||
function = item.get("function") if isinstance(item.get("function"), dict) else item
|
||||
name = str(function.get("name") or "").strip()
|
||||
if name not in declared:
|
||||
return None, f"unknown tool {name!r}"
|
||||
arguments = function.get("arguments", function.get("parameters", {}))
|
||||
if isinstance(arguments, str):
|
||||
try:
|
||||
arguments = json.loads(arguments) if arguments.strip() else {}
|
||||
except ValueError:
|
||||
return None, f"arguments of {name!r} are not valid JSON"
|
||||
if not isinstance(arguments, dict):
|
||||
return None, f"arguments of {name!r} are not an object"
|
||||
schema = declared[name].get("parameters")
|
||||
required = schema.get("required") if isinstance(schema, dict) else None
|
||||
missing = [key for key in (required or []) if key not in arguments]
|
||||
if missing:
|
||||
return None, f"call to {name!r} misses required arguments: {', '.join(missing)}"
|
||||
return (
|
||||
{
|
||||
"id": "call_" + uuid.uuid4().hex[:24],
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": name,
|
||||
"arguments": json.dumps(arguments, ensure_ascii=False),
|
||||
},
|
||||
},
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def parse_tool_calls(
|
||||
text: str, declared: dict[str, dict], allow_parallel: bool = True
|
||||
) -> tuple[str, list[dict], list[str]]:
|
||||
if CALL_OPEN not in (text or ""):
|
||||
return text or "", [], []
|
||||
calls: list[dict] = []
|
||||
warnings: list[str] = []
|
||||
kept: list[str] = []
|
||||
cursor = 0
|
||||
for match in _CALL_BLOCK.finditer(text):
|
||||
kept.append(text[cursor : match.start()])
|
||||
cursor = match.end()
|
||||
items = _call_objects(match.group(1))
|
||||
if not items:
|
||||
warnings.append("a tool_call block did not contain valid JSON")
|
||||
kept.append(match.group(0))
|
||||
continue
|
||||
for item in items:
|
||||
call, problem = _normalize_call(item, declared)
|
||||
if call is None:
|
||||
warnings.append(problem or "invalid tool call")
|
||||
kept.append(match.group(0))
|
||||
continue
|
||||
calls.append(call)
|
||||
kept.append(text[cursor:])
|
||||
if not allow_parallel and len(calls) > 1:
|
||||
warnings.append(f"parallel calls disabled: dropped {len(calls) - 1} extra call(s)")
|
||||
calls = calls[:1]
|
||||
if len(calls) > MAX_CALLS_PER_TURN:
|
||||
warnings.append(f"dropped {len(calls) - MAX_CALLS_PER_TURN} call(s) over the per-turn cap")
|
||||
calls = calls[:MAX_CALLS_PER_TURN]
|
||||
return "".join(kept).strip(), calls, warnings
|
||||
|
||||
|
||||
def convert_completion(
|
||||
completion: dict, declared: dict[str, dict], allow_parallel: bool = True
|
||||
) -> tuple[dict, list[str]]:
|
||||
warnings: list[str] = []
|
||||
for choice in completion.get("choices") or []:
|
||||
message = choice.get("message") if isinstance(choice, dict) else None
|
||||
if not isinstance(message, dict) or message.get("tool_calls"):
|
||||
continue
|
||||
text = message.get("content")
|
||||
if not isinstance(text, str):
|
||||
continue
|
||||
remaining, calls, found = parse_tool_calls(text, declared, allow_parallel)
|
||||
warnings.extend(found)
|
||||
if calls:
|
||||
message["tool_calls"] = calls
|
||||
message["content"] = remaining or None
|
||||
choice["finish_reason"] = "tool_calls"
|
||||
return completion, warnings
|
||||
|
||||
|
||||
def _holdback(text: str) -> int:
|
||||
longest = min(len(CALL_OPEN) - 1, len(text))
|
||||
for size in range(longest, 0, -1):
|
||||
if CALL_OPEN.startswith(text[-size:]):
|
||||
return size
|
||||
return 0
|
||||
|
||||
|
||||
class ToolCallStream:
|
||||
def __init__(self, declared: dict[str, dict], allow_parallel: bool = True):
|
||||
self.declared = declared
|
||||
self.allow_parallel = allow_parallel
|
||||
self.warnings: list[str] = []
|
||||
self._base: dict = {}
|
||||
self._pending = ""
|
||||
self._call_text = ""
|
||||
self._in_call = False
|
||||
self._flushed = False
|
||||
self._saw_calls = False
|
||||
|
||||
def _chunk(self, delta: dict, finish_reason=None) -> dict:
|
||||
chunk = {k: v for k, v in self._base.items() if k not in ("choices", "usage")}
|
||||
chunk["choices"] = [{"index": 0, "delta": delta, "finish_reason": finish_reason}]
|
||||
return chunk
|
||||
|
||||
def _content_chunk(self, text: str) -> list[dict]:
|
||||
return [self._chunk({"content": text})] if text else []
|
||||
|
||||
def _consume(self, text: str) -> list[dict]:
|
||||
if self._in_call:
|
||||
self._call_text += text
|
||||
return []
|
||||
self._pending += text
|
||||
index = self._pending.find(CALL_OPEN)
|
||||
if index >= 0:
|
||||
out = self._content_chunk(self._pending[:index])
|
||||
self._call_text = self._pending[index:]
|
||||
self._pending = ""
|
||||
self._in_call = True
|
||||
return out
|
||||
keep = _holdback(self._pending)
|
||||
emit = self._pending[: len(self._pending) - keep]
|
||||
self._pending = self._pending[len(self._pending) - keep :]
|
||||
return self._content_chunk(emit)
|
||||
|
||||
def flush(self) -> list[dict]:
|
||||
if self._flushed:
|
||||
return []
|
||||
self._flushed = True
|
||||
if not self._in_call:
|
||||
text, self._pending = self._pending, ""
|
||||
return self._content_chunk(text)
|
||||
remaining, calls, found = parse_tool_calls(
|
||||
self._call_text, self.declared, self.allow_parallel
|
||||
)
|
||||
self.warnings.extend(found)
|
||||
out = self._content_chunk(remaining)
|
||||
if calls:
|
||||
self._saw_calls = True
|
||||
out.append(
|
||||
self._chunk(
|
||||
{
|
||||
"tool_calls": [
|
||||
{"index": i, **call} for i, call in enumerate(calls)
|
||||
]
|
||||
}
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
def feed(self, chunk: dict) -> list[dict]:
|
||||
choices = chunk.get("choices") or []
|
||||
if not choices:
|
||||
return [chunk]
|
||||
self._base = chunk
|
||||
choice = choices[0]
|
||||
delta = choice.get("delta") or {}
|
||||
out: list[dict] = []
|
||||
text = delta.get("content")
|
||||
passthrough = {k: v for k, v in delta.items() if k != "content"}
|
||||
if isinstance(text, str) and text:
|
||||
out.extend(self._consume(text))
|
||||
finish = choice.get("finish_reason")
|
||||
if passthrough:
|
||||
out.append(self._chunk(passthrough))
|
||||
if finish is not None:
|
||||
out.extend(self.flush())
|
||||
out.append(
|
||||
self._chunk(
|
||||
{}, "tool_calls" if self._saw_calls and finish == "stop" else finish
|
||||
)
|
||||
)
|
||||
if chunk.get("usage"):
|
||||
out.append({**{k: v for k, v in chunk.items() if k != "choices"}, "choices": []})
|
||||
return out
|
||||
|
||||
def finish(self) -> list[dict]:
|
||||
return self.flush()
|
||||
Reference in New Issue
Block a user