Gateway: auto-resolve request shape per provider and model

An upstream that rejects a key the gateway adds (e.g. thinking) now gets
retried with other shapes on a parsable 400/422; the working method is
stored per (provider, target model) and applied first-try from then on.
Fallback attempts are re-shaped per attempt. Admin list/reset, Devii
tools, audit event and ledger resolved_method included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-13 11:34:26 +01:00
co-authored by Claude Sonnet 5.5
parent 3f6e83c753
commit 151476125f
22 changed files with 1597 additions and 129 deletions
@@ -106,6 +106,8 @@ A structured/deterministic internal caller (one that expects a strict, machine-p
- OpenRouter: `reasoning: {"effort": "none"}` or `"high"`
- Ollama (`:11434` or a host containing `ollama`): `think: false` or `true`
`thinking_dialect` can also be `none` (no field written); it is only ever chosen by the auto-resolve ladder when an upstream rejects the thinking key (see "Auto-resolve of request shape").
Do **not** send `reasoning_effort: "none"` to DeepSeek (400: unknown variant). Conflicting client fields (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs.enable_thinking`) are stripped and translated to the dialect above.
**Default is off.** `gateway_thinking` (Prompt group, bool, default `False`) is the operator override for the silent-client case. A client still wins: `think` / `thinking.type` / `reasoning.effort` / `reasoning.enabled` / `reasoning_effort` / `enable_thinking` on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.
@@ -116,6 +118,20 @@ Vision describe-image calls always disable thinking (they are not a reasoning jo
`thinking.upstream_capabilities(url, dialect="")` derives per-upstream protocol flags (`supports_stream_options`, `supports_stream_usage`, `supports_thinking_field`) from the same effective dialect. Ollama's OpenAI-compatibility layer ignores `stream_options` (and rejects the request outright on some versions), so `handle_chat`'s streaming branch only sends `stream_options: {"include_usage": true}` upstream when `capabilities.supports_stream_options` is true; otherwise a client that asked for `include_usage` gets a logged notice and its stream relayed with no usage chunk. DeepSeek/OpenRouter (and any unrecognised URL) report full support, so this is a no-op for them.
## Auto-resolve of request shape (`shape.py`, `learned.py`)
An upstream with a strict schema can reject a key the gateway adds on its own (observed 2026-11-13: route `vaids` answered `400 {"issues":[{"code":"unrecognized_keys","keys":["thinking"]}]}` because `thinking_dialect` falls back to `deepseek` for any unrecognised URL). A 400 is `bad_request`, never retried and never falls back, so the gateway now LEARNS, per (provider, target model), which shape the upstream accepts and applies it from then on.
- **One funnel.** `ShapeMethod.apply(payload, cfg, protocol, stream, log)` is the only place stream_options, the thinking dialect and droppable keys are written; chat, every fallback attempt and the vision describe call all go through it, each from the PRISTINE client payload with that attempt's own cfg (fallback attempts used to inherit the primary's already-shaped payload). `ShapeMethod.strip_wire` removes `drop_keys` (and `reasoning` for a Responses `none` method) from the final wire body in `_chat_upstream_body`. A method is a frozen `ShapeMethod(thinking_dialect, drop_keys)`; `thinking_dialect` is `native` (today's behaviour: the URL-sniffed or `gateway_thinking_dialect` dialect, the default and first ladder entry), `none` (write no thinking field at all, new in `thinking.py`), or an explicit `deepseek`/`openrouter`/`ollama`.
- **Trigger.** Only a 400/422 whose body names rejected keys (`RejectedKeys.parse`: zod `issues[].keys` with `unrecognized_keys`, OpenAI `Unrecognized request argument supplied`/`Unknown parameter` + `param`, serde `unknown field`, "Extra inputs are not permitted"). An unparsable 400 stays terminal and loud, returned verbatim, never retried blind.
- **Ladder and allowlist (static ceilings, no autotuning).** `ShapeResolver.next_method` picks the next untried method: a rejected thinking key (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs`) skips every dialect that emits it (`CHAT_LADDER` order: native, none, deepseek, openrouter, ollama; Responses protocol: native, none, rejecting `reasoning`); a rejected key in `DROPPABLE_KEYS` (`stream_options`, `parallel_tool_calls`, `user`, `seed`, `logprobs`, `top_logprobs`, `n`, `presence_penalty`, `frequency_penalty`) is added to `drop_keys`. `messages`, `model`, `tools` and any unknown key are NEVER dropped. Hard cap `MAX_RESOLVE_ATTEMPTS = 4` retries per route attempt.
- **Retry placement.** `GatewayRuntime._chat_attempt` (shared by the primary and every fallback attempt) sends, and on a parsable shape 400 re-sends through `ShapeRetrier`; the retry goes through `_send` so the breaker and retries apply, and streaming status is known before any byte is relayed so a retried stream is safe. On success `learned_store.remember` is called (never on a guess) and one `auto-resolved <provider>/<model>: <method>` line is logged. On exhaustion the ORIGINAL 400 response is returned unchanged, nothing is stored, and `shape_stats.exhausted` is incremented. The vision describe call uses the same `ShapeRetrier`, keyed `vision:<host>` + vision model.
- **Persistence.** Table `gateway_learned_methods` (`learned.py`, `ensure_table()` from `init_db`): `provider` (`default` for the base upstream), `target_model`, `method` (JSON), `learned_at`, unique index on (provider, target_model), cap `MAX_LEARNED_ROWS` 1000 (oldest by `learned_at` replaced). Cross-worker cache under the `gateway_learned` cache-version name; the upsert is one `ON CONFLICT DO UPDATE` so two workers learning the same route never collide. It is derived data in the garbage-collection class (NOT in `SOFT_DELETE_TABLES`): forget = delete row. Editing a provider's `base_url` or `client_profile`, or deleting it, forgets all its learned methods (`ProviderStore.set`/`remove`).
- **Self-healing floor.** If a request using a LEARNED method itself gets a shape 400, the record is forgotten, the ladder restarts once from native, and `shape_stats.healed` counts it.
- **Ledger.** One row per client call as before; `resolved_method` (text, `gateway_usage_ledger`) holds the non-native method label that served the call, empty for native.
- **Operator surface.** `/admin/gateway` models grid shows a `shape: <method>` badge and a "Reset learned" form (`POST /admin/gateway/learned/reset-page`); JSON `GET /admin/gateway/learned` and `POST /admin/gateway/learned/reset` (audited `gateway.learned.reset`, category `ai`) back the Devii tools `gateway_learned_list` / `gateway_learned_reset` (the reset is in `CONFIRM_REQUIRED` with a declared `confirm`).
- **Not covered:** value-level rejections (`unknown variant`), and keys outside the closed allowlists. Tests: `tests/unit/services/openai_gateway/{shape,learned}.py`, the `test_chat_auto_resolve*`/`test_chat_*learned*`/`test_fallback_attempt_is_shaped*` tests in `gateway.py`, `tests/api/admin/gateway/learned.py`, and the grid badge in `tests/e2e/admin/gateway.py`.
## Real upstream streaming (`GatewayRuntime._stream_chat_response`)
`stream: true` is forwarded to the upstream verbatim (`payload["stream"] = stream`, no more forced `false`) and the connection is opened with `client.send(request, stream=True)` so the response body is read incrementally via `resp.aiter_lines()` instead of buffered whole. Every retry/circuit-breaker mechanic in `_send`/`retry_send` is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; `retry_send` now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns `200` does the code path diverge into `_stream_chat_response`.
@@ -238,6 +254,7 @@ Any `gateway_models` route (chat, embed, or image) can optionally name `fallback
- **One ledger row per client call, not one per attempt.** The primary failure never writes a `gateway_usage_ledger` row by itself (only `_send`'s own internal retries and the circuit breaker counters observe it); `finalize()`/`self._ledger.record(...)` still runs exactly once, after the fallback decision, with `model` = whichever model ultimately answered and `requested_model` = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in `/admin/ai-usage` as "requested X, served Y" rather than as two calls.
- **Streaming falls back before any byte reaches the client.** `client.send(request, stream=True)` returns as soon as the response headers arrive, so `resp.status_code` is already known before the SSE body is ever touched - `handle_chat` decides whether to fall back at that same point, before it ever opens `_stream_chat_response` to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
- **Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (`self._breaker`) is one instance per `GatewayRuntime`, shared across every route.** A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see `circuit_open` too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
- **A 400 is terminal UNLESS its body names rejected keys** the auto-resolve ladder can fix (see "Auto-resolve of request shape"): that case is retried on the SAME route with another shape before any fallback is considered.
- **Deliberately NOT covered:** `gateway_max_retries`/`gateway_retry_backoff_ms`, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.
**Tests:** `tests/unit/services/openai_gateway/routing.py` (`resolve_fallback` self-reference/kind/active-flag guards, chain skip/cycle/hop-cap/entry tests), `tests/unit/services/openai_gateway/gateway.py` (`test_chat_falls_back_when_the_primary_model_fails`, `test_chat_returns_the_fallback_failure_when_both_models_fail`, `test_chat_without_a_fallback_never_makes_a_second_call`, `test_chat_walks_fallback_chain_past_failures`, `test_chat_preroutes_around_unhealthy_models`, `test_chat_tries_original_when_chain_is_exhausted`), and `tests/api/admin/gateway/models.py` (`test_model_route_fallback_round_trip`, `test_model_route_fallback_must_reference_an_existing_route`, `test_model_route_fallback_must_be_the_same_kind`, `test_model_route_fallback_rejects_self_reference`).
+149 -93
View File
@@ -4,7 +4,8 @@ import asyncio
import json
import logging
import time
from typing import Optional
from dataclasses import dataclass
from typing import Any, Optional
import httpx
from fastapi.responses import JSONResponse, Response, StreamingResponse
@@ -39,8 +40,13 @@ from devplacepy.services.openai_gateway.tool_emulation import (
emulate_request,
emulation_enabled,
)
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import (
NATIVE_METHOD,
ShapeMethod,
ShapeRetrier,
)
from devplacepy.services.openai_gateway.thinking import (
apply_thinking,
client_thinking_enabled,
upstream_capabilities,
)
@@ -60,6 +66,15 @@ from devplacepy.services.openai_gateway.vision import VisionAugmenter, VisionCac
logger = logging.getLogger(__name__)
async def _close_quietly(resp) -> None:
if resp is None:
return
try:
await resp.aclose()
except Exception: # noqa: BLE001 - discarding an already-failed response
pass
def _call_failed(resp, exc, timing: dict) -> bool:
if timing.get("circuit_open") or exc is not None:
return True
@@ -163,6 +178,7 @@ def _chat_upstream_body(
log,
catalog_cache=None,
owner: tuple = ("anonymous", "anonymous"),
method: ShapeMethod = NATIVE_METHOD,
) -> dict:
# The handler reasons about a Chat payload throughout (routing, vision,
# precheck, fallback); only the bytes on the wire change per protocol.
@@ -182,13 +198,13 @@ def _chat_upstream_body(
)
log(f"Meta reserved tool names aliased: {renamed}")
if protocol != "responses":
return payload
return method.strip_wire(payload, protocol)
send_body, warnings = chat_to_responses(payload)
if warnings:
shown = "; ".join(warnings[:3])
more = f" (+{len(warnings) - 3} more)" if len(warnings) > 3 else ""
log(f"chat->responses transpile warnings: {shown}{more}")
if "reasoning" not in send_body:
if "reasoning" not in send_body and not method.suppresses_reasoning(protocol):
explicit = client_thinking_enabled(client_body)
enabled = (
explicit
@@ -197,7 +213,7 @@ def _chat_upstream_body(
)
if not enabled:
send_body["reasoning"] = {"effort": "minimal"}
return send_body
return method.strip_wire(send_body, protocol)
def _response_tool_aliases(payload: dict, protocol: str, cfg: dict) -> dict:
@@ -350,6 +366,20 @@ def _connect_tracer(holder: dict):
return trace
@dataclass
class ChatAttempt:
resp: Any
exc: Any
timing: dict
shaped: dict
send_start: float
method: ShapeMethod
@property
def resolved_label(self) -> Optional[str]:
return None if self.method.is_native else self.method.label
class GatewayRuntime:
def __init__(self):
self._client: Optional[httpx.AsyncClient] = None
@@ -515,6 +545,102 @@ class GatewayRuntime:
timing["retry_succeeded"] = attempts > 1
return resp, None, timing
async def _shaped_send(
self,
client,
sem,
pristine: dict,
body: dict,
cfg: dict,
protocol: str,
model: str,
headers: dict,
stream: bool,
owner: tuple,
log,
method: ShapeMethod,
) -> ChatAttempt:
shaped = method.apply(pristine, cfg, protocol, stream, log)
shaped["model"] = model
send_body = _chat_upstream_body(
shaped, body, cfg, protocol, log, self._catalog_cache, owner, method
)
send_start = time.monotonic()
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_upstream_url"],
headers,
cfg,
log,
json_body=send_body,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
# The streamed response body hasn't been read yet; the shared error
# handling needs resp.text/.content, which requires an explicit
# read for a stream=True response (a no-op if already buffered).
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
# error body must never crash the request; fall through with
# whatever text/content httpx managed to buffer (possibly empty).
log(f"failed to read non-200 stream error body: {e}")
return ChatAttempt(resp, exc, timing, shaped, send_start, method)
async def _chat_attempt(
self,
client,
sem,
pristine: dict,
body: dict,
cfg: dict,
protocol: str,
model: str,
headers: dict,
stream: bool,
owner: tuple,
log,
) -> ChatAttempt:
native_dialect = upstream_capabilities(
cfg.get("gateway_upstream_url", ""),
cfg.get("gateway_thinking_dialect", "auto"),
).dialect
retrier = ShapeRetrier.for_route(
learned_store,
cfg.get("gateway_provider_name") or "default",
model,
protocol,
native_dialect,
log,
)
first = await self._shaped_send(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log, retrier.method,
)
current = first
while current.resp is not None and retrier.eligible(
current.resp.status_code
):
method = retrier.next_after(current.resp.status_code, current.resp.text)
if method is None:
break
if current is not first:
await _close_quietly(current.resp)
current = await self._shaped_send(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log, method,
)
if current is first:
return first
if current.resp is not None and current.resp.status_code < 400:
retrier.succeeded()
return current
retrier.exhausted()
await _close_quietly(current.resp)
return first
async def handle_chat(
self,
body: dict,
@@ -620,46 +746,10 @@ class GatewayRuntime:
stream = bool(body.get("stream"))
include_usage = bool((body.get("stream_options") or {}).get("include_usage"))
payload = dict(body)
payload["model"] = model
payload["messages"] = messages
payload["stream"] = stream
if stream and protocol == "responses":
# Responses streams carry usage on response.completed natively;
# there is no include_usage to request (stream_options only
# supports include_obfuscation there). The translator captures it
# and the client's own include_usage still controls the relay.
payload.pop("stream_options", None)
if stream and protocol != "responses":
capabilities = upstream_capabilities(
cfg.get("gateway_upstream_url", ""),
cfg.get("gateway_thinking_dialect", "auto"),
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
)
if capabilities.supports_stream_options:
stream_options = dict(body.get("stream_options") or {})
stream_options["include_usage"] = True
payload["stream_options"] = stream_options
else:
payload.pop("stream_options", None)
if include_usage:
log(
"stream_options.include_usage requested but the upstream "
f"({capabilities.dialect}) does not support it; relaying "
"without a usage chunk"
)
else:
payload.pop("stream_options", None)
if protocol != "responses":
apply_thinking(
payload,
cfg.get("gateway_upstream_url", ""),
default_enabled=bool(cfg.get("gateway_thinking", False)),
dialect=cfg.get("gateway_thinking_dialect", "auto"),
)
# For responses upstreams the reasoning effort is set at transpile
# time instead (see _chat_upstream_body): chat-dialect thinking
# fields have no meaning on that wire.
pristine = dict(body)
pristine["model"] = model
pristine["messages"] = messages
pristine["stream"] = stream
headers = {
"Content-Type": "application/json",
@@ -678,7 +768,7 @@ class GatewayRuntime:
)
context_window = _route_context_window(cfg, model)
if context_window:
estimated_tokens = _estimate_request_tokens(payload)
estimated_tokens = _estimate_request_tokens(pristine)
if estimated_tokens > context_window:
for precheck_route in resolve_fallback_chain(chain_start, "chat"):
precheck_overlay = chat_overlay(
@@ -704,39 +794,19 @@ class GatewayRuntime:
or "chat_completions"
)
model = cfg["gateway_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
break
send_start = time.monotonic()
try:
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log, self._catalog_cache, owner
attempt = await self._chat_attempt(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log,
)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_upstream_url"],
headers,
cfg,
log,
json_body=send_body,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
# The streamed response body hasn't been read yet; the shared error
# handling below needs resp.text/.content, which requires an explicit
# read for a stream=True response (a no-op if already buffered).
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
# error body must never crash the request; fall through with
# whatever text/content httpx managed to buffer (possibly empty).
log(f"failed to read non-200 stream error body: {e}")
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
payload = attempt.shaped
send_start = attempt.send_start
requested_model = requested
body_error = _embedded_failure_message(resp, stream, protocol)
@@ -761,32 +831,17 @@ class GatewayRuntime:
cfg.get("gateway_upstream_protocol") or "chat_completions"
)
model = cfg["gateway_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
send_start = time.monotonic()
try:
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log, self._catalog_cache, owner
attempt = await self._chat_attempt(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log,
)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_upstream_url"],
headers,
cfg,
log,
json_body=send_body,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001
log(f"failed to read fallback non-200 stream error body: {e}")
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
payload = attempt.shaped
send_start = attempt.send_start
body_error = _embedded_failure_message(resp, stream, protocol)
pricing = pricing_from_cfg(cfg)
@@ -803,6 +858,7 @@ class GatewayRuntime:
"app_reference": app_reference,
"provider": cfg.get("gateway_provider_name") or "default",
"fallback_used_route": fallback_used_route,
"resolved_method": attempt.resolved_label,
**params,
**timing,
}
@@ -0,0 +1,165 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import Optional
from pydantic import BaseModel, Field, field_validator
from devplacepy.database import (
bump_cache_version,
db,
get_table,
sync_local_cache,
)
from devplacepy.services.openai_gateway.shape import ShapeMethod
logger = logging.getLogger(__name__)
TABLE = "gateway_learned_methods"
CACHE_NAME = "gateway_learned"
MAX_LEARNED_ROWS = 1000
DEFAULT_PROVIDER = "default"
_LEARNED_CACHE: dict = {}
class LearnedResetIn(BaseModel):
provider: str = Field(default="", max_length=120)
target_model: str = Field(min_length=1, max_length=200)
@field_validator("provider", "target_model")
@classmethod
def _strip(cls, value: str) -> str:
return value.strip()
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _provider_key(provider: Optional[str]) -> str:
return (provider or DEFAULT_PROVIDER).strip().lower() or DEFAULT_PROVIDER
def ensure_table() -> None:
db.query(
"CREATE TABLE IF NOT EXISTS "
+ TABLE
+ " (id INTEGER PRIMARY KEY, provider TEXT, target_model TEXT, "
"method TEXT, learned_at TEXT)"
)
try:
db.query(
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_learned_route ON "
+ TABLE
+ " (provider, target_model)"
)
db.query(
"CREATE INDEX IF NOT EXISTS idx_gateway_learned_at ON "
+ TABLE
+ " (learned_at)"
)
except Exception as exc:
logger.warning("gateway learned index creation failed: %s", exc)
def _load() -> dict:
sync_local_cache(CACHE_NAME, _LEARNED_CACHE)
if "methods" not in _LEARNED_CACHE:
methods: dict = {}
try:
if TABLE in db.tables:
for row in get_table(TABLE).all():
method = ShapeMethod.from_json(row.get("method"))
if method is None:
continue
methods[
(_provider_key(row.get("provider")), str(row["target_model"]))
] = (method, str(row.get("learned_at") or ""))
except Exception as exc:
logger.warning("gateway learned load failed: %s", exc)
_LEARNED_CACHE["methods"] = methods
return _LEARNED_CACHE["methods"]
def _invalidate() -> None:
bump_cache_version(CACHE_NAME)
_LEARNED_CACHE.clear()
class LearnedMethodStore:
def get(self, provider: Optional[str], model: str) -> Optional[ShapeMethod]:
entry = _load().get((_provider_key(provider), str(model)))
return entry[0] if entry else None
def list(self) -> list[dict]:
rows = [
{
"provider": provider,
"target_model": model,
"method": method.label,
"thinking_dialect": method.thinking_dialect,
"drop_keys": list(method.drop_keys),
"learned_at": learned_at,
}
for (provider, model), (method, learned_at) in _load().items()
]
return sorted(rows, key=lambda r: (r["provider"], r["target_model"]))
def remember(
self, provider: Optional[str], model: str, method: ShapeMethod
) -> None:
if method.is_native or not model:
return
try:
ensure_table()
with db:
db.query(
"INSERT INTO " + TABLE + " (provider, target_model, method, learned_at) "
"VALUES (:provider, :model, :method, :now) "
"ON CONFLICT(provider, target_model) DO UPDATE SET "
"method = excluded.method, learned_at = excluded.learned_at",
provider=_provider_key(provider),
model=str(model),
method=method.to_json(),
now=_now(),
)
db.query(
"DELETE FROM " + TABLE + " WHERE id IN (SELECT id FROM "
+ TABLE
+ " ORDER BY learned_at ASC, id ASC LIMIT "
"MAX((SELECT COUNT(*) FROM " + TABLE + ") - :cap, 0))",
cap=MAX_LEARNED_ROWS,
)
except Exception as exc:
logger.warning("gateway learned remember failed: %s", exc)
return
_invalidate()
def forget(self, provider: Optional[str], model: str) -> bool:
if TABLE not in db.tables:
return False
removed = int(
get_table(TABLE).delete(
provider=_provider_key(provider), target_model=str(model)
)
)
if removed:
_invalidate()
return bool(removed)
def forget_provider(self, provider: str) -> int:
if TABLE not in db.tables:
return 0
table = get_table(TABLE)
removed = table.count(provider=_provider_key(provider))
if removed:
table.delete(provider=_provider_key(provider))
_invalidate()
return removed
learned_store = LearnedMethodStore()
@@ -16,6 +16,7 @@ from devplacepy.database import (
sync_local_cache,
)
from devplacepy.services.openai_gateway import muse_code, opencode_zen
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.tool_emulation import (
TOOLS_MODE_NATIVE,
TOOLS_MODES,
@@ -531,11 +532,17 @@ class ProviderStore:
"protocol": payload.protocol,
"updated_at": _now(),
}
endpoint_changed = bool(existing) and (
str(existing.get("base_url") or "") != payload.base_url
or str(existing.get("client_profile") or "") != payload.client_profile
)
if existing:
table.update({**record, "id": existing["id"]}, ["id"])
else:
record["created_at"] = _now()
table.insert(record)
if endpoint_changed:
learned_store.forget_provider(payload.name)
bump_cache_version(CACHE_NAME)
_ROUTING_CACHE.clear()
return self.get(payload.name) or record
@@ -546,6 +553,7 @@ class ProviderStore:
return False
removed = int(get_table(PROVIDERS_TABLE).delete(name=name))
if removed:
learned_store.forget_provider(name)
bump_cache_version(CACHE_NAME)
_ROUTING_CACHE.clear()
return bool(removed)
+388
View File
@@ -0,0 +1,388 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import json
import logging
import re
from dataclasses import dataclass, field
from typing import Any, Callable, Optional
from devplacepy.services.openai_gateway.thinking import (
CLIENT_THINKING_KEYS,
apply_thinking,
upstream_capabilities,
)
logger = logging.getLogger(__name__)
PROTOCOL_RESPONSES = "responses"
DIALECT_NATIVE = "native"
DIALECT_NONE = "none"
DIALECT_DEEPSEEK = "deepseek"
DIALECT_OPENROUTER = "openrouter"
DIALECT_OLLAMA = "ollama"
CHAT_LADDER = (
DIALECT_NATIVE,
DIALECT_NONE,
DIALECT_DEEPSEEK,
DIALECT_OPENROUTER,
DIALECT_OLLAMA,
)
RESPONSES_LADDER = (DIALECT_NATIVE, DIALECT_NONE)
DIALECTS = frozenset(CHAT_LADDER)
DROPPABLE_KEYS = frozenset(
{
"stream_options",
"parallel_tool_calls",
"user",
"seed",
"logprobs",
"top_logprobs",
"n",
"presence_penalty",
"frequency_penalty",
}
)
THINKING_KEYS = frozenset(CLIENT_THINKING_KEYS) | {"chat_template_kwargs"}
DIALECT_EMITS: dict = {
DIALECT_DEEPSEEK: frozenset({"thinking"}),
DIALECT_OPENROUTER: frozenset({"reasoning"}),
DIALECT_OLLAMA: frozenset({"think", "reasoning_effort"}),
DIALECT_NONE: frozenset(),
}
RESPONSES_EMITS = frozenset({"reasoning"})
MAX_RESOLVE_ATTEMPTS = 4
SHAPE_STATUSES = (400, 422)
MAX_REJECTED_KEYS = 16
MAX_KEY_LENGTH = 64
_KEY_PATTERNS = (
re.compile(r"Unrecognized key\(s\) in object:\s*((?:'[^']+'(?:,\s*)?)+)", re.I),
re.compile(r"Unrecognized request arguments? supplied:\s*([\w.,\s]+)", re.I),
re.compile(r"Unknown (?:parameter|field|argument)s?:?\s*[`'\"]?([\w.]+)", re.I),
re.compile(r"unknown field\s*[`'\"]([\w.]+)[`'\"]", re.I),
re.compile(r"Extra inputs are not permitted.{0,40}?['\"`]([\w.]+)['\"`]", re.I),
)
_QUOTED = re.compile(r"'([^']+)'")
_BARE_SPLIT = re.compile(r"[,\s]+")
@dataclass(frozen=True)
class ShapeMethod:
thinking_dialect: str = DIALECT_NATIVE
drop_keys: tuple = ()
@property
def is_native(self) -> bool:
return self.thinking_dialect == DIALECT_NATIVE and not self.drop_keys
@property
def label(self) -> str:
return self.thinking_dialect + "".join(f" -{key}" for key in self.drop_keys)
def dialect_for(self, cfg: dict) -> str:
if self.thinking_dialect == DIALECT_NATIVE:
return str(cfg.get("gateway_thinking_dialect", "auto") or "auto")
return self.thinking_dialect
def to_json(self) -> str:
return json.dumps(
{
"thinking_dialect": self.thinking_dialect,
"drop_keys": list(self.drop_keys),
}
)
@classmethod
def from_json(cls, text: Any) -> Optional["ShapeMethod"]:
try:
data = json.loads(text)
except (TypeError, ValueError):
return None
if not isinstance(data, dict):
return None
dialect = data.get("thinking_dialect")
drops = data.get("drop_keys")
if dialect not in DIALECTS or not isinstance(drops, list):
return None
if any(key not in DROPPABLE_KEYS for key in drops):
return None
return cls(dialect, tuple(sorted(set(drops))))
def apply(
self,
payload: dict,
cfg: dict,
protocol: str,
stream: bool,
log: Callable[[str], None],
) -> dict:
shaped = dict(payload)
kwargs = shaped.get("chat_template_kwargs")
if isinstance(kwargs, dict):
shaped["chat_template_kwargs"] = dict(kwargs)
if protocol == PROTOCOL_RESPONSES:
shaped.pop("stream_options", None)
else:
self._apply_stream_options(shaped, payload, cfg, stream, log)
apply_thinking(
shaped,
cfg.get("gateway_upstream_url", ""),
default_enabled=bool(cfg.get("gateway_thinking", False)),
dialect=self.dialect_for(cfg),
)
for key in self.drop_keys:
shaped.pop(key, None)
return shaped
def _apply_stream_options(
self,
shaped: dict,
payload: dict,
cfg: dict,
stream: bool,
log: Callable[[str], None],
) -> None:
include_usage = bool((payload.get("stream_options") or {}).get("include_usage"))
capabilities = upstream_capabilities(
cfg.get("gateway_upstream_url", ""),
cfg.get("gateway_thinking_dialect", "auto"),
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
)
if (
stream
and capabilities.supports_stream_options
and "stream_options" not in self.drop_keys
):
stream_options = dict(payload.get("stream_options") or {})
stream_options["include_usage"] = True
shaped["stream_options"] = stream_options
return
shaped.pop("stream_options", None)
if stream and include_usage:
log(
"stream_options.include_usage requested but the upstream "
f"({capabilities.dialect}) does not support it; relaying "
"without a usage chunk"
)
def suppresses_reasoning(self, protocol: str) -> bool:
return protocol == PROTOCOL_RESPONSES and self.thinking_dialect == DIALECT_NONE
def strip_wire(self, body: dict, protocol: str) -> dict:
doomed = set(self.drop_keys)
if self.suppresses_reasoning(protocol):
doomed.add("reasoning")
if not doomed.intersection(body):
return body
return {key: value for key, value in body.items() if key not in doomed}
NATIVE_METHOD = ShapeMethod()
class RejectedKeys:
@staticmethod
def parse(body_text: Any) -> tuple:
text = body_text if isinstance(body_text, str) else ""
if not text:
return ()
found: list = []
RejectedKeys._from_json(text, found)
RejectedKeys._from_patterns(text, found)
cleaned: list = []
for key in found:
top = str(key).strip().strip("`'\"").split(".")[0].split("[")[0]
if top and len(top) <= MAX_KEY_LENGTH and top not in cleaned:
cleaned.append(top)
return tuple(cleaned[:MAX_REJECTED_KEYS])
@staticmethod
def _from_json(text: str, found: list) -> None:
try:
data = json.loads(text)
except ValueError:
return
if not isinstance(data, dict):
return
issues = data.get("issues")
if isinstance(issues, list):
for issue in issues:
if isinstance(issue, dict) and issue.get("code") == "unrecognized_keys":
keys = issue.get("keys")
if isinstance(keys, list):
found.extend(str(key) for key in keys)
error = data.get("error")
if isinstance(error, dict):
param = error.get("param")
code = str(error.get("code") or "")
if isinstance(param, str) and param and code in (
"unknown_parameter",
"unrecognized_request_argument",
):
found.append(param)
@staticmethod
def _from_patterns(text: str, found: list) -> None:
for pattern in _KEY_PATTERNS:
for match in pattern.finditer(text):
chunk = match.group(1)
quoted = _QUOTED.findall(chunk)
if quoted:
found.extend(quoted)
else:
found.extend(
part for part in _BARE_SPLIT.split(chunk.strip()) if part
)
class ShapeResolver:
def __init__(self, native_dialect: str, protocol: str) -> None:
self.native_dialect = native_dialect
self.responses = protocol == PROTOCOL_RESPONSES
def emits(self, dialect: str) -> frozenset:
if self.responses:
return frozenset() if dialect == DIALECT_NONE else RESPONSES_EMITS
if dialect == DIALECT_NATIVE:
return DIALECT_EMITS.get(self.native_dialect, frozenset())
return DIALECT_EMITS.get(dialect, frozenset())
def next_method(
self, tried: set, current: ShapeMethod, rejected: tuple
) -> Optional[ShapeMethod]:
keys = set(rejected)
drops = set(current.drop_keys) | (keys & DROPPABLE_KEYS)
if self.responses:
thinking_hit = keys & RESPONSES_EMITS
else:
thinking_hit = keys & THINKING_KEYS
if not thinking_hit and drops == set(current.drop_keys):
return None
sorted_drops = tuple(sorted(drops))
if not thinking_hit:
candidate = ShapeMethod(current.thinking_dialect, sorted_drops)
return None if candidate in tried else candidate
ladder = RESPONSES_LADDER if self.responses else CHAT_LADDER
for dialect in ladder:
if self.emits(dialect) & keys:
continue
candidate = ShapeMethod(dialect, sorted_drops)
if candidate not in tried:
return candidate
return None
@dataclass
class ShapeStats:
resolved: int = 0
exhausted: int = 0
healed: int = 0
def snapshot(self) -> dict:
return {
"resolved": self.resolved,
"exhausted": self.exhausted,
"healed": self.healed,
}
shape_stats = ShapeStats()
@dataclass
class ShapeRetrier:
store: Any
provider: str
model: str
resolver: ShapeResolver
log: Callable[[str], None]
method: ShapeMethod = field(default=NATIVE_METHOD)
from_store: bool = False
healed: bool = False
attempts: int = 0
tried: set = field(default_factory=set)
@classmethod
def for_route(
cls,
store: Any,
provider: str,
model: str,
protocol: str,
native_dialect: str,
log: Callable[[str], None],
) -> "ShapeRetrier":
learned = store.get(provider, model)
retrier = cls(
store=store,
provider=provider,
model=model,
resolver=ShapeResolver(native_dialect, protocol),
log=log,
method=learned or NATIVE_METHOD,
from_store=learned is not None,
)
retrier.tried.add(retrier.method)
return retrier
@staticmethod
def eligible(status: int) -> bool:
return status in SHAPE_STATUSES
def next_after(self, status: int, body_text: Any) -> Optional[ShapeMethod]:
if not self.eligible(status) or self.attempts >= MAX_RESOLVE_ATTEMPTS:
return None
rejected = RejectedKeys.parse(body_text)
if not rejected:
return None
if self.from_store and not self.healed:
self.store.forget(self.provider, self.model)
self.healed = True
self.from_store = False
shape_stats.healed += 1
self.log(
f"learned shape '{self.method.label}' for {self.provider}/"
f"{self.model} was rejected ({', '.join(rejected)}); "
"forgotten, restarting the ladder"
)
self.tried = {self.method}
if NATIVE_METHOD not in self.tried:
return self._advance(NATIVE_METHOD)
candidate = self.resolver.next_method(self.tried, self.method, rejected)
if candidate is None:
return None
return self._advance(candidate)
def _advance(self, candidate: ShapeMethod) -> ShapeMethod:
self.attempts += 1
self.tried.add(candidate)
self.method = candidate
self.log(
f"upstream rejected the request shape for {self.provider}/"
f"{self.model}, trying '{candidate.label}'"
)
return candidate
def succeeded(self) -> None:
if self.attempts == 0 or self.method.is_native:
return
self.store.remember(self.provider, self.model, self.method)
shape_stats.resolved += 1
self.log(
f"auto-resolved {self.provider}/{self.model}: {self.method.label}"
)
def exhausted(self) -> None:
if self.attempts == 0:
return
shape_stats.exhausted += 1
self.log(
f"could not auto-resolve the request shape for {self.provider}/"
f"{self.model} after {self.attempts} attempt(s)"
)
@property
def resolved_label(self) -> Optional[str]:
return None if self.method.is_native else self.method.label
@@ -144,6 +144,8 @@ def _clear_thinking_fields(payload: dict) -> None:
def write_thinking(payload: dict, dialect: str, enabled: bool) -> None:
_clear_thinking_fields(payload)
if dialect == "none":
return
if dialect == "openrouter":
payload["reasoning"] = {"effort": "high" if enabled else "none"}
return
@@ -587,6 +587,11 @@ class GatewayUsageLedger:
if raw.get("fallback_used_route")
else None
),
"resolved_method": (
str(raw["resolved_method"])[:128]
if raw.get("resolved_method")
else None
),
"status_code": int(raw.get("status_code") or 0),
"success": 1 if raw.get("success") else 0,
"error_category": raw.get("error_category"),
+69 -32
View File
@@ -6,6 +6,7 @@ import logging
import time
from collections import OrderedDict
from typing import Any, Optional
from urllib.parse import urlparse
import httpx
@@ -16,7 +17,9 @@ from devplacepy.services.openai_gateway.reliability import (
_is_retryable_status,
_retry_after_seconds,
)
from devplacepy.services.openai_gateway.thinking import apply_thinking
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import ShapeRetrier
from devplacepy.services.openai_gateway.thinking import upstream_capabilities
from devplacepy.services.openai_gateway.usage import classify_error
logger = logging.getLogger(__name__)
@@ -151,37 +154,9 @@ class VisionAugmenter:
if row:
self.cost_usd += float(row.get("cost_usd") or 0.0)
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
if not self.vision_key:
return "[vision unavailable: vision API key not configured]"
if self.breaker is not None and not self.breaker.allow():
self._record((0.0), 503, False, "circuit_open", None)
return "[vision unavailable: circuit breaker open]"
payload = {
"model": self.vision_model,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": VISION_INSTRUCTION},
image_block,
],
}
],
"temperature": 0.2,
"stream": False,
}
apply_thinking(payload, self.vision_url, default_enabled=False, dialect=self.vision_dialect)
headers = {
"Authorization": f"Bearer {self.vision_key}",
"Content-Type": "application/json",
}
if self.referer:
headers["HTTP-Referer"] = self.referer
if self.title:
headers[TITLE_HEADER] = self.title
headers[LEGACY_TITLE_HEADER] = self.title
start = time.monotonic()
async def _post_with_retries(
self, client: httpx.AsyncClient, payload: dict, headers: dict
) -> tuple:
attempt = 0
resp: Optional[httpx.Response] = None
exc: Optional[Exception] = None
@@ -210,6 +185,68 @@ class VisionAugmenter:
except Exception: # noqa: BLE001
pass
await _backoff(self.retry_backoff_ms, attempt, retry_after)
return resp, exc
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
if not self.vision_key:
return "[vision unavailable: vision API key not configured]"
if self.breaker is not None and not self.breaker.allow():
self._record((0.0), 503, False, "circuit_open", None)
return "[vision unavailable: circuit breaker open]"
payload = {
"model": self.vision_model,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": VISION_INSTRUCTION},
image_block,
],
}
],
"temperature": 0.2,
"stream": False,
}
shape_cfg = {
"gateway_upstream_url": self.vision_url,
"gateway_thinking_dialect": self.vision_dialect or "auto",
"gateway_thinking": False,
}
retrier = ShapeRetrier.for_route(
learned_store,
"vision:" + (urlparse(self.vision_url).hostname or "default"),
self.vision_model,
"chat_completions",
upstream_capabilities(self.vision_url, self.vision_dialect).dialect,
logger.info,
)
headers = {
"Authorization": f"Bearer {self.vision_key}",
"Content-Type": "application/json",
}
if self.referer:
headers["HTTP-Referer"] = self.referer
if self.title:
headers[TITLE_HEADER] = self.title
headers[LEGACY_TITLE_HEADER] = self.title
start = time.monotonic()
shaped = retrier.method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
resp, exc = await self._post_with_retries(client, shaped, headers)
first = (resp, exc)
resolving = False
while resp is not None and retrier.eligible(resp.status_code):
method = retrier.next_after(resp.status_code, resp.text)
if method is None:
break
resolving = True
shaped = method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
resp, exc = await self._post_with_retries(client, shaped, headers)
if resolving:
if resp is not None and resp.status_code < 400:
retrier.succeeded()
else:
retrier.exhausted()
resp, exc = first
if self.breaker is not None:
if exc is not None or (resp is not None and resp.status_code >= 500):
self.breaker.record_failure()