Keep composers usable, group actions and notifications, and drop the tool-count cap
DevPlace CI / test (push) Failing after 3h10m39s

Comment and post layouts stay full width on small screens. Secondary actions live in the More menu. Notifications can be grouped and throttled, and Enter can submit when that profile setting is on. Emulated-tool models accept large catalogs, with reserved names aliased on the wire and the catalog cached per owner. A dropped service-worker navigation is retried once. The test server waits 120s to finish importing before it is stopped.
This commit is contained in:
2026-11-12 15:29:27 +01:00
parent dbdd0d29dd
commit 8a48e3ef85
103 changed files with 2680 additions and 378 deletions
+67 -7
View File
@@ -43,7 +43,10 @@ A model route (`gateway_models.tools_mode`, `native` default or `emulated`) decl
- **Request seam.** `gateway._chat_upstream_body` calls `emulate_request(payload)` first, BEFORE the Responses/Meta transpile, so the Meta path (`chat_to_responses`, Muse aliasing) simply receives a tool-less Chat payload. As a side effect the Meta reserved tool-name gate never fires, because `tools[]` is gone; `_response_tool_aliases` returns `{}` for emulated routes. `emulate_request` strips `tools`/`tool_choice`/`parallel_tool_calls`/`functions`/`function_call`, appends a Hermes-format catalog (`<tools>` JSON lines plus the `<tool_call>{"name","arguments"}</tool_call>` contract) to the system message (creating one when absent), maps `tool_choice` (`none` drops the catalog, `required`/named add a MUST directive, `parallel_tool_calls: false` adds a one-call rule), rewrites history (assistant `tool_calls` become `<tool_call>` blocks, `role: tool`/`function` messages become one merged user message of `<tool_response>` blocks matched back to names by call id), and adds `<tool_response>` as a stop sequence so the model cannot invent a result. It never mutates the caller's payload.
- **Response seams.** Non-streaming: `convert_completion` parses blocks out of `message.content` into `tool_calls` (generated `call_` ids, `finish_reason: "tool_calls"`, leftover text kept, `content: null` when empty), applied on both the chat and the Responses non-stream branches. Streaming: `_emulate_tool_stream` wraps the finished SSE iterator, so ledger/TTFT accounting stays inside the inner generator. `ToolCallStream` relays text live with a partial-`<tool_call>` holdback, buffers once a call opens, and emits each call whole in ONE `tool_calls` delta (never incremental arguments: that is where vLLM's Hermes parser kept breaking). Usage chunks and non-content deltas pass through.
- **Parsing is tolerant, validation strict.** Accepts code fences, an array of calls in one block, `parameters` for `arguments`, string-encoded arguments, and an unterminated final block. A call is valid only if the name is declared and the schema's `required` keys are present. An invalid block is NEVER dropped silently: it stays in `content` as text and is logged (`emulated tools: ...`). Known gap: there is no repair retry; an invalid call is returned as text.
- **Hard caps (400, no upstream call, no ledger row).** `MAX_TOOLS` 64, `MAX_CATALOG_CHARS` 60000, only `function` tools, `MAX_CALLS_PER_TURN` 8 (extra calls dropped with a warning). The catalog costs prompt tokens on every call, billed like any input.
- **No count/size caps (deliberate).** Any number of `function` tools is accepted and the full catalog is always injected - Devii alone declares hundreds of tools, so a count cap would make large catalogs unservable everywhere. Servability is bounded downstream: the context-window precheck routes an oversized request to a fallback model before dispatch, and the upstream's own context limit is relayed when nothing can serve it. What still 400s (no upstream call, no ledger row) is a malformed declaration: non-list `tools`, a non-`function` tool type, or a nameless function. `MAX_CALLS_PER_TURN` 8 (extra calls dropped with a warning) is unchanged. The catalog costs prompt tokens on every call, billed like any input.
- **Per-owner catalog cache (`tool_emulation.CatalogCache`, owned by `GatewayRuntime`).** Building the catalog (spec validation + instruction assembly, milliseconds at hundreds of tools) runs once per distinct `(owner, tools, tool_choice, parallel_tool_calls)` fingerprint and is reused across turns, retries, and fallback attempts; the response seams (`_emulated_tools`) resolve declarations from the same entry instead of re-validating. Entries are namespaced by owner (`(owner_kind, owner_id)`, so one user's tool set never serves another's) and bounded (`MAX_ENTRIES` 64, `MAX_CHARS` 2M, oldest-first eviction) - a cache key is content-derived, so no invalidation is needed.
- **Injection cadence: once per request, every request.** The catalog is injected exactly once per upstream attempt (one `<tools>` block; history converts without repeated blocks; the stop sequence is added only when absent), but it is re-sent on every turn, retry, and fallback attempt, because the upstream is stateless and each call needs the catalog in context. Fallback attempts never stack injections: every attempt rebuilds from the pristine client payload (`emulate_request` never mutates its input), and the per-owner cache only skips the rebuild, never changes the bytes.
- **Upstream prefix caching.** A prefix-caching upstream (DeepSeek context caching, tracked as `prompt_cache_hit_tokens`) can serve the repeated catalog from cache, but only while the bytes before it are stable. The catalog is appended after the client's system content, so anything volatile there re-bills the catalog too. Devii's system prompt is stable within the hour (its trailing `# CURRENT TIME` block is rounded to the hour, see the Devii file), so turns inside one hour share one prefix and the catalog hits; the hour rollover breaks it. Watch the cache-hit token split in `/admin/ai-usage` to confirm.
- **Surfaces.** `ModelRouteIn.tools_mode`, the admin model form (**Tool calling** select), the `tools: emulated` badge, `POST /admin/gateway/models` (`tools_mode`, documented in `docs_api`) and the Devii `gateway_model_set` tool.
- **Not covered:** constrained/guided decoding (provider-specific), and the format is Hermes JSON only. Tests: `tests/unit/services/openai_gateway/tool_emulation.py` (pure) and the `test_emulated_tools_*` tests in `gateway.py` (through the real Meta Responses path, streaming and non-streaming).
@@ -71,6 +74,10 @@ A dedicated per-IP throttle protects `/openai/v1/*` against unauthenticated prob
All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble` and `gateway_thinking` (default off), timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
## Vision model (current: `google/gemma-4-31b-it`)
Image content is described by the vision model before forwarding (augmentation defaults on via `gateway_vision_enabled`). The live value is the `gateway_vision_model` site setting (database, editable in admin UI, hot-reloads); the code fallback for fresh databases is `VISION_MODEL_DEFAULT` in `config.py` with its context window in `MODEL_CONTEXT_MAP_DEFAULT`. Keep the two in sync when switching models: the DB row wins at runtime, the code default wins on a fresh install. History: `gemma-3-12b-it` default, `gemma-3-27b-it` via DB, `gemma-4-31b-it` since 08/11/2026 (cheaper small variant `gemma-4-26b-a4b-it` also verified on OpenRouter and rejected in favour of quality).
## System-message composition (date awareness + operator preamble)
`system_message.py` (`apply_system_directives`) runs in `handle_chat` only (NOT `handle_embeddings`/`handle_passthrough`), after vision augmentation and before the upstream payload is assembled, so it reaches every chat consumer (news, bots, Devii guests, per-user direct API calls). It composes ONE system message in a fixed, cache-friendly order: **operator preamble (`gateway_system_preamble`) -> date line (`Current date: DD/MM/YYYY`) -> the client's original system content**.
@@ -337,15 +344,68 @@ The `muse` headers are defense in depth, NOT the attestation fix:
probes proved the key works from any client/TLS stack for plain calls
and the gate only fires on reserved `tools[].name` values. The real
fix is automatic tool-name aliasing (`muse_code.tool_aliases_for` /
`alias_chat_tools`, wired in `_chat_upstream_body` for Meta Responses
attempts only): any declared/history/tool_choice name exactly matching
`alias_chat_tools`, wired in `_chat_upstream_body` for Meta attempts on
BOTH wire protocols): any declared/history/tool_choice name exactly matching
`MUSE_RESERVED_TOOL_NAMES` (20 names, each measured denying while
near-misses pass) is rewritten to `ext_<name>` (growing underscores
on collision) and restored on the way back
(`restore_completion_names` / `restore_chunk_names`), so the model
(`restore_completion_names` / `restore_chunk_names`, with the alias map
threaded through `_stream_chat_response` for streamed chat), so the model
sees one consistent world and the client never sees the alias. The
map is rebuilt deterministically from the same payload at both ends,
so fallbacks crossing providers need no state. Clean requests pass
through untouched (same object, no log); aliased ones log
map is rebuilt deterministically from the same payload at both ends
(`_response_tool_aliases`, protocol-independent), so fallbacks crossing
providers need no state. Clean requests pass through untouched (same
object, no log, verbatim relay); aliased ones log
`Meta reserved tool names aliased: bash->ext_bash, ...`. If 403s
return, re-probe: Meta may have added a builtin.
Incident 2026-11-10 (tai `devplace/molodetz` 403s, Devii unaffected):
the gate scans NAMES, not counts. tai's 63-tool catalog shares 7 names
with the denylist (`shell`, `read_file`, `write_file`, `edit_file`,
`web_fetch`, `web_search`, `search`); Devii's 400+ domain tools share
0, so 400 tool calls never tripped it. Three rounds, 2026-11-10: (1) the aliasing
originally ran only for `responses`-protocol attempts while tai/detz
speak `chat_completions`, so colliding names reached Meta verbatim -
fixed by hoisting the aliasing above the protocol branch in
`_chat_upstream_body` and restoring names on the chat return paths
(non-streaming re-encode, streamed chunk restore); (2) the denylist was
missing `shell`, which tai/detz send on EVERY turn (first entry of
`CORE_TOOLS`, core loads always) - added (21 names), corroborated by
Pi's `pi-meta-oauth` denylist, live via `--reload` 18:26; (3) REFUTED
18:33: requests with all 5 known triggers aliased (`shell->ext_shell`
on the wire) plus stamped `muse` profile STILL 403. The 21-name gate
theory is therefore insufficient. Remaining fork: another ungated name
in the payload (candidates never probed by anyone: `web_fetch`,
`fork`, `poll`, `ask_user`, `remember`, `recall`, `load_skill`, ...),
the `ext_` scheme itself recognized, or the key fully gated for
non-genuine clients. Discriminator: two authenticated direct calls
against the route, one tool-less and one with a single clean tool -
see the research conclusion for the exact probes. If the tool-less call
also 403s, stop guessing names and put a pay-as-you-go key on the
route. Outcome 19:05, after a clean service restart: single-name live
probes proved `web_fetch` denied alone (added as 22nd name) while
`fork`, `poll`, `ask_user`, `remember`, `recall`, `load_skill`,
`ext_shell` each 200 alone; the full 63-tool tai catalog then 200d in
one call, and the journal shows `web_fetch->ext_web_fetch` aliasing on
real traffic with 9 upstream 200s and 0 403s since. The gate is names,
the denylist at 22 covers the whole static catalog, no key change
needed. Count caps measured live 2026-11-10 on the Meta route
(native `tools_mode`): 70, 100, 200 and 400 tools per call all HTTP
200 (63 real catalog names plus synthetic clean names) - there is no
count gate on this path. The only count cap in the system is the
gateway's own emulated-tools path (which had a 64-tool cap until
it was removed from the working tree the same day - `test_large_catalogs_are_served_with_no_cap`
pins 424 tools served, unrelated
to Meta). The companion
`openrouter/... 401 No cookie auth credentials found` in the same
report is unrelated: tai calling OpenRouter with no API key
configured, an operator setup gap, not a gateway behavior. Tests:
`tests/unit/services/openai_gateway/gateway.py`
(`test_meta_tool_names_aliased_on_wire_and_restored` /
`test_meta_tool_names_restored_on_stream_chunks` for Responses,
`test_meta_tool_names_aliased_on_chat_completions_wire` /
`test_meta_tool_names_restored_on_chat_completions_stream` for chat,
the chat tests pin `shell`+`read_file` end to end)
and `tests/unit/services/openai_gateway/muse_code.py` (denylist
membership + alias/round-trip unit tests). Full external-research
record: `research/muse-attestation-gate/` in the tai project.
+2 -1
View File
@@ -13,7 +13,7 @@ SYSTEM_PREAMBLE_DEFAULT = ""
THINKING_DEFAULT = False
VISION_URL_DEFAULT = "https://openrouter.ai/api/v1/chat/completions"
VISION_MODEL_DEFAULT = "google/gemma-3-12b-it"
VISION_MODEL_DEFAULT = "google/gemma-4-31b-it"
VISION_CACHE_SIZE_DEFAULT = 256
EMBED_URL_DEFAULT = "https://openrouter.ai/api/v1/embeddings"
@@ -50,4 +50,5 @@ MODEL_CONTEXT_MAP_DEFAULT = {
"deepseek-chat": 1_048_576,
"deepseek-reasoner": 1_048_576,
"google/gemma-3-12b-it": 8192,
"google/gemma-4-31b-it": 262144,
}
+47 -17
View File
@@ -31,13 +31,13 @@ from devplacepy.services.openai_gateway.responses import (
)
from devplacepy.services.openai_gateway.system_message import apply_system_directives
from devplacepy.services.openai_gateway.tool_emulation import (
CatalogCache,
ToolCallStream,
ToolEmulationError,
cached_emulated_tools,
convert_completion,
declared_tools,
emulate_request,
emulation_enabled,
parallel_allowed,
)
from devplacepy.services.openai_gateway.thinking import (
apply_thinking,
@@ -156,14 +156,22 @@ def _upstream_auth_guidance(resp, content: dict) -> Optional[str]:
def _chat_upstream_body(
payload: dict, client_body: dict, cfg: dict, protocol: str, log
payload: dict,
client_body: dict,
cfg: dict,
protocol: str,
log,
catalog_cache=None,
owner: tuple = ("anonymous", "anonymous"),
) -> dict:
# The handler reasons about a Chat payload throughout (routing, vision,
# precheck, fallback); only the bytes on the wire change per protocol.
# Meta aliasing applies to both wire protocols: api.meta.ai scans
# tools[] names on chat_completions exactly like on responses, so a
# chat_dialect client (tai, detz, any generic OpenAI caller) would 403
# without this. The restore side mirrors per protocol below.
if emulation_enabled(cfg):
payload = emulate_request(payload)
if protocol != "responses":
return payload
payload = emulate_request(payload, catalog_cache, owner)
if muse_code.is_meta_upstream(cfg.get("gateway_upstream_url")):
aliases = muse_code.tool_aliases_for(payload)
if aliases:
@@ -173,6 +181,8 @@ def _chat_upstream_body(
for alias, original in sorted(aliases.items())
)
log(f"Meta reserved tool names aliased: {renamed}")
if protocol != "responses":
return payload
send_body, warnings = chat_to_responses(payload)
if warnings:
shown = "; ".join(warnings[:3])
@@ -194,7 +204,9 @@ def _response_tool_aliases(payload: dict, protocol: str, cfg: dict) -> dict:
# Rebuilds the exact map _chat_upstream_body applied on the wire for
# this attempt (same payload, same deterministic rule), so response
# names restore without threading state through the fallback path.
if protocol != "responses" or emulation_enabled(cfg):
# Protocol-independent: aliasing now happens on the wire for both
# chat_completions and responses (see _chat_upstream_body).
if emulation_enabled(cfg):
return {}
if not muse_code.is_meta_upstream(cfg.get("gateway_upstream_url")):
return {}
@@ -208,17 +220,14 @@ def _tool_emulation_rejection(error: ToolEmulationError) -> JSONResponse:
)
def _emulated_tools(payload: dict, cfg: dict):
def _emulated_tools(payload: dict, cfg: dict, catalog_cache=None, owner=None):
# Same rebuild-from-the-attempt's-own-payload rule as _response_tool_aliases:
# the declared tools come from the Chat payload of the attempt that was
# actually sent, so a fallback crossing into or out of an emulated model
# needs no threaded state.
if not emulation_enabled(cfg):
return None
declared = declared_tools(payload)
if not declared:
return None
return declared, parallel_allowed(payload)
return cached_emulated_tools(payload, catalog_cache, owner)
def _emulate_completion(data: dict, emulated, log) -> dict:
@@ -361,6 +370,7 @@ class GatewayRuntime:
self._vision_breaker = CircuitBreaker(
config.CIRCUIT_THRESHOLD_DEFAULT, config.CIRCUIT_COOLDOWN_SECONDS_DEFAULT
)
self._catalog_cache = CatalogCache()
self.requests = 0
self.errors = 0
self.in_flight = 0
@@ -701,7 +711,9 @@ class GatewayRuntime:
send_start = time.monotonic()
try:
send_body = _chat_upstream_body(payload, body, cfg, protocol, log)
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log, self._catalog_cache, owner
)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
resp, exc, timing = await self._send(
@@ -755,7 +767,7 @@ class GatewayRuntime:
send_start = time.monotonic()
try:
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log
payload, body, cfg, protocol, log, self._catalog_cache, owner
)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
@@ -915,8 +927,11 @@ class GatewayRuntime:
handle_start,
send_start,
log,
tool_aliases=_response_tool_aliases(
payload, protocol, cfg
),
)
emulated = _emulated_tools(payload, cfg)
emulated = _emulated_tools(payload, cfg, self._catalog_cache, owner)
if emulated is not None:
body_iter = _emulate_tool_stream(body_iter, emulated, log)
return StreamingResponse(
@@ -955,7 +970,7 @@ class GatewayRuntime:
converted, _response_tool_aliases(payload, protocol, cfg)
)
converted = _emulate_completion(
converted, _emulated_tools(payload, cfg), log
converted, _emulated_tools(payload, cfg, self._catalog_cache, owner), log
)
if convert_warnings:
log(
@@ -969,15 +984,26 @@ class GatewayRuntime:
media_type="application/json",
headers=resp_headers,
)
aliases = _response_tool_aliases(payload, protocol, cfg)
if aliases:
# chat_completions upstreams echo the aliased wire names back;
# the client must see the names it declared.
data = muse_code.restore_completion_names(data, aliases)
resp_headers = finalize(200, True, None, data.get("usage"))
log(f"chat POST -> 200 ({timing['upstream_latency_ms']:.0f}ms)")
emulated = _emulated_tools(payload, cfg)
emulated = _emulated_tools(payload, cfg, self._catalog_cache, owner)
if emulated is not None:
return Response(
content=json.dumps(_emulate_completion(data, emulated, log)).encode("utf-8"),
media_type="application/json",
headers=resp_headers,
)
if aliases:
return Response(
content=json.dumps(data).encode("utf-8"),
media_type="application/json",
headers=resp_headers,
)
return Response(
content=resp.content,
media_type="application/json",
@@ -1106,7 +1132,9 @@ class GatewayRuntime:
handle_start: float,
send_start: float,
log,
tool_aliases: Optional[dict] = None,
):
tool_aliases = tool_aliases or {}
first_at: Optional[float] = None
last_at: Optional[float] = None
content_chunks = 0
@@ -1132,6 +1160,8 @@ class GatewayRuntime:
chunk = json.loads(data_str)
except ValueError:
continue
if tool_aliases:
chunk = muse_code.restore_chunk_names(chunk, tool_aliases)
choices = chunk.get("choices") or []
delta = (choices[0].get("delta") if choices else None) or {}
if delta.get("content") or delta.get("reasoning") or delta.get("reasoning_content"):
@@ -41,8 +41,12 @@ def _muse_user_agent() -> str:
# Every one of these trips api.meta.ai's client gate with 403
# client_attestation_denied when declared as a tools[] function name
# (measured 2026-10-30, all 20 deny, near-misses pass). Text, history,
# params, and descriptions are not scanned - only exact tools[].name.
# (measured 2026-10-30, all 20 deny, near-misses pass; "shell" and
# "web_fetch" added 2026-11-10 after live single-name probes proved
# each denied alone - "shell" also independently verified by Pi's
# pi-meta-oauth denylist, which names it among its 9 blocking names
# while ours covered the other 8). Text, history, params, and
# descriptions are not scanned - only exact tools[].name.
MUSE_RESERVED_TOOL_NAMES = frozenset(
{
"workflow",
@@ -64,6 +68,8 @@ MUSE_RESERVED_TOOL_NAMES = frozenset(
"update_goal",
"report_progress",
"read_skill",
"shell",
"web_fetch",
"write_todos",
}
)
@@ -1,17 +1,24 @@
# retoor <retoor@molodetz.nl>
import copy
import hashlib
import json
import re
import uuid
from collections import OrderedDict
from typing import Optional
TOOLS_MODE_NATIVE = "native"
TOOLS_MODE_EMULATED = "emulated"
TOOLS_MODES = (TOOLS_MODE_NATIVE, TOOLS_MODE_EMULATED)
MAX_TOOLS = 64
MAX_CATALOG_CHARS = 60_000
# There is deliberately no cap on the tool count or the catalog size: a
# client may declare any number of tools and the full catalog is always
# injected. Servability is bounded downstream, not here - the gateway's
# context-window precheck routes an oversized request to a fallback model
# before dispatch, and the upstream's own context limit is relayed when
# nothing can serve it. A count/size rejection here would make large
# catalogs (Devii alone declares hundreds of tools) unservable everywhere.
MAX_CALLS_PER_TURN = 8
MAX_STOP_SEQUENCES = 4
@@ -48,10 +55,6 @@ def _function_specs(payload: dict) -> list[dict]:
return []
if not isinstance(tools, list):
raise ToolEmulationError("tools must be a list")
if len(tools) > MAX_TOOLS:
raise ToolEmulationError(
f"too many tools for an emulated-tools model: {len(tools)} > {MAX_TOOLS}"
)
specs = []
for tool in tools:
if not isinstance(tool, dict) or tool.get("type") != "function":
@@ -88,13 +91,7 @@ def _choice_directive(payload: dict) -> tuple[bool, str]:
def _catalog_text(specs: list[dict]) -> str:
lines = [json.dumps({"type": "function", "function": spec}, ensure_ascii=False) for spec in specs]
catalog = "\n".join(lines)
if len(catalog) > MAX_CATALOG_CHARS:
raise ToolEmulationError(
f"tool catalog too large for an emulated-tools model: "
f"{len(catalog)} > {MAX_CATALOG_CHARS} characters"
)
return catalog
return "\n".join(lines)
def build_instruction(payload: dict, specs: list[dict]) -> str:
@@ -227,7 +224,91 @@ def _with_stop(payload: dict) -> None:
payload["stop"] = sequences + [RESPONSE_OPEN]
def emulate_request(payload: dict) -> dict:
class CatalogCache:
MAX_ENTRIES = 64
MAX_CHARS = 2_000_000
def __init__(self) -> None:
self._entries: OrderedDict[tuple, tuple[str, dict, bool]] = OrderedDict()
self._chars = 0
@staticmethod
def fingerprint(payload: dict) -> Optional[str]:
try:
canonical = json.dumps(
{
"tools": payload.get("tools"),
"tool_choice": payload.get("tool_choice"),
"parallel_tool_calls": payload.get("parallel_tool_calls"),
},
sort_keys=True,
ensure_ascii=False,
separators=(",", ":"),
default=str,
)
except (TypeError, ValueError):
return None
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
def lookup(
self, payload: dict, owner: Optional[tuple] = None
) -> Optional[tuple[str, dict, bool]]:
digest = self.fingerprint(payload)
if digest is None:
return None
key = ((owner or (None, None)), digest)
try:
found = self._entries.pop(key)
except KeyError:
return None
self._entries[key] = found
return found
def store(
self,
payload: dict,
owner: Optional[tuple],
instruction: str,
declared: dict,
allow_parallel: bool,
) -> None:
digest = self.fingerprint(payload)
if digest is None:
return
key = ((owner or (None, None)), digest)
previous = self._entries.pop(key, None)
if previous is not None:
self._chars -= len(previous[0])
self._entries[key] = (instruction, declared, allow_parallel)
self._chars += len(instruction)
while len(self._entries) > self.MAX_ENTRIES or self._chars > self.MAX_CHARS:
_, dropped = self._entries.popitem(last=False)
self._chars -= len(dropped[0])
def _cached_parts(
payload: dict,
specs: list[dict],
catalog_cache: Optional[CatalogCache],
owner: Optional[tuple],
) -> tuple[str, dict, bool]:
allow_parallel = parallel_allowed(payload)
if catalog_cache is not None:
found = catalog_cache.lookup(payload, owner)
if found is not None:
return found
declared = {str(spec["name"]): spec for spec in specs}
instruction = build_instruction(payload, specs)
if catalog_cache is not None:
catalog_cache.store(payload, owner, instruction, declared, allow_parallel)
return instruction, declared, allow_parallel
def emulate_request(
payload: dict,
catalog_cache: Optional[CatalogCache] = None,
owner: Optional[tuple] = None,
) -> dict:
specs = _function_specs(payload)
history = payload.get("messages") or []
has_tool_history = any(
@@ -236,14 +317,33 @@ def emulate_request(payload: dict) -> dict:
)
if not specs and not has_tool_history:
return payload
instruction, _, _ = _cached_parts(payload, specs, catalog_cache, owner)
converted = {k: copy.copy(v) for k, v in payload.items() if k not in REMOVED_PAYLOAD_KEYS}
messages = _rewrite_history(list(history))
converted["messages"] = _with_system_instruction(messages, build_instruction(payload, specs))
converted["messages"] = _with_system_instruction(messages, instruction)
if specs:
_with_stop(converted)
return converted
def cached_emulated_tools(
payload: dict,
catalog_cache: Optional[CatalogCache] = None,
owner: Optional[tuple] = None,
) -> Optional[tuple[dict, bool]]:
if catalog_cache is not None:
found = catalog_cache.lookup(payload, owner)
if found is not None:
_, declared, allow_parallel = found
if not declared:
return None
return declared, allow_parallel
declared = declared_tools(payload)
if not declared:
return None
return declared, parallel_allowed(payload)
def _strip_fence(text: str) -> str:
stripped = text.strip()
match = _FENCE.match(stripped)