Gateway: auto-resolve request shape per provider and model
An upstream that rejects a key the gateway adds (e.g. thinking) now gets retried with other shapes on a parsable 400/422; the working method is stored per (provider, target model) and applied first-try from then on. Fallback attempts are re-shaped per attempt. Admin list/reset, Devii tools, audit event and ledger resolved_method included. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -106,6 +106,8 @@ A structured/deterministic internal caller (one that expects a strict, machine-p
|
||||
- OpenRouter: `reasoning: {"effort": "none"}` or `"high"`
|
||||
- Ollama (`:11434` or a host containing `ollama`): `think: false` or `true`
|
||||
|
||||
`thinking_dialect` can also be `none` (no field written); it is only ever chosen by the auto-resolve ladder when an upstream rejects the thinking key (see "Auto-resolve of request shape").
|
||||
|
||||
Do **not** send `reasoning_effort: "none"` to DeepSeek (400: unknown variant). Conflicting client fields (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs.enable_thinking`) are stripped and translated to the dialect above.
|
||||
|
||||
**Default is off.** `gateway_thinking` (Prompt group, bool, default `False`) is the operator override for the silent-client case. A client still wins: `think` / `thinking.type` / `reasoning.effort` / `reasoning.enabled` / `reasoning_effort` / `enable_thinking` on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.
|
||||
@@ -116,6 +118,20 @@ Vision describe-image calls always disable thinking (they are not a reasoning jo
|
||||
|
||||
`thinking.upstream_capabilities(url, dialect="")` derives per-upstream protocol flags (`supports_stream_options`, `supports_stream_usage`, `supports_thinking_field`) from the same effective dialect. Ollama's OpenAI-compatibility layer ignores `stream_options` (and rejects the request outright on some versions), so `handle_chat`'s streaming branch only sends `stream_options: {"include_usage": true}` upstream when `capabilities.supports_stream_options` is true; otherwise a client that asked for `include_usage` gets a logged notice and its stream relayed with no usage chunk. DeepSeek/OpenRouter (and any unrecognised URL) report full support, so this is a no-op for them.
|
||||
|
||||
## Auto-resolve of request shape (`shape.py`, `learned.py`)
|
||||
|
||||
An upstream with a strict schema can reject a key the gateway adds on its own (observed 2026-11-13: route `vaids` answered `400 {"issues":[{"code":"unrecognized_keys","keys":["thinking"]}]}` because `thinking_dialect` falls back to `deepseek` for any unrecognised URL). A 400 is `bad_request`, never retried and never falls back, so the gateway now LEARNS, per (provider, target model), which shape the upstream accepts and applies it from then on.
|
||||
|
||||
- **One funnel.** `ShapeMethod.apply(payload, cfg, protocol, stream, log)` is the only place stream_options, the thinking dialect and droppable keys are written; chat, every fallback attempt and the vision describe call all go through it, each from the PRISTINE client payload with that attempt's own cfg (fallback attempts used to inherit the primary's already-shaped payload). `ShapeMethod.strip_wire` removes `drop_keys` (and `reasoning` for a Responses `none` method) from the final wire body in `_chat_upstream_body`. A method is a frozen `ShapeMethod(thinking_dialect, drop_keys)`; `thinking_dialect` is `native` (today's behaviour: the URL-sniffed or `gateway_thinking_dialect` dialect, the default and first ladder entry), `none` (write no thinking field at all, new in `thinking.py`), or an explicit `deepseek`/`openrouter`/`ollama`.
|
||||
- **Trigger.** Only a 400/422 whose body names rejected keys (`RejectedKeys.parse`: zod `issues[].keys` with `unrecognized_keys`, OpenAI `Unrecognized request argument supplied`/`Unknown parameter` + `param`, serde `unknown field`, "Extra inputs are not permitted"). An unparsable 400 stays terminal and loud, returned verbatim, never retried blind.
|
||||
- **Ladder and allowlist (static ceilings, no autotuning).** `ShapeResolver.next_method` picks the next untried method: a rejected thinking key (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs`) skips every dialect that emits it (`CHAT_LADDER` order: native, none, deepseek, openrouter, ollama; Responses protocol: native, none, rejecting `reasoning`); a rejected key in `DROPPABLE_KEYS` (`stream_options`, `parallel_tool_calls`, `user`, `seed`, `logprobs`, `top_logprobs`, `n`, `presence_penalty`, `frequency_penalty`) is added to `drop_keys`. `messages`, `model`, `tools` and any unknown key are NEVER dropped. Hard cap `MAX_RESOLVE_ATTEMPTS = 4` retries per route attempt.
|
||||
- **Retry placement.** `GatewayRuntime._chat_attempt` (shared by the primary and every fallback attempt) sends, and on a parsable shape 400 re-sends through `ShapeRetrier`; the retry goes through `_send` so the breaker and retries apply, and streaming status is known before any byte is relayed so a retried stream is safe. On success `learned_store.remember` is called (never on a guess) and one `auto-resolved <provider>/<model>: <method>` line is logged. On exhaustion the ORIGINAL 400 response is returned unchanged, nothing is stored, and `shape_stats.exhausted` is incremented. The vision describe call uses the same `ShapeRetrier`, keyed `vision:<host>` + vision model.
|
||||
- **Persistence.** Table `gateway_learned_methods` (`learned.py`, `ensure_table()` from `init_db`): `provider` (`default` for the base upstream), `target_model`, `method` (JSON), `learned_at`, unique index on (provider, target_model), cap `MAX_LEARNED_ROWS` 1000 (oldest by `learned_at` replaced). Cross-worker cache under the `gateway_learned` cache-version name; the upsert is one `ON CONFLICT DO UPDATE` so two workers learning the same route never collide. It is derived data in the garbage-collection class (NOT in `SOFT_DELETE_TABLES`): forget = delete row. Editing a provider's `base_url` or `client_profile`, or deleting it, forgets all its learned methods (`ProviderStore.set`/`remove`).
|
||||
- **Self-healing floor.** If a request using a LEARNED method itself gets a shape 400, the record is forgotten, the ladder restarts once from native, and `shape_stats.healed` counts it.
|
||||
- **Ledger.** One row per client call as before; `resolved_method` (text, `gateway_usage_ledger`) holds the non-native method label that served the call, empty for native.
|
||||
- **Operator surface.** `/admin/gateway` models grid shows a `shape: <method>` badge and a "Reset learned" form (`POST /admin/gateway/learned/reset-page`); JSON `GET /admin/gateway/learned` and `POST /admin/gateway/learned/reset` (audited `gateway.learned.reset`, category `ai`) back the Devii tools `gateway_learned_list` / `gateway_learned_reset` (the reset is in `CONFIRM_REQUIRED` with a declared `confirm`).
|
||||
- **Not covered:** value-level rejections (`unknown variant`), and keys outside the closed allowlists. Tests: `tests/unit/services/openai_gateway/{shape,learned}.py`, the `test_chat_auto_resolve*`/`test_chat_*learned*`/`test_fallback_attempt_is_shaped*` tests in `gateway.py`, `tests/api/admin/gateway/learned.py`, and the grid badge in `tests/e2e/admin/gateway.py`.
|
||||
|
||||
## Real upstream streaming (`GatewayRuntime._stream_chat_response`)
|
||||
|
||||
`stream: true` is forwarded to the upstream verbatim (`payload["stream"] = stream`, no more forced `false`) and the connection is opened with `client.send(request, stream=True)` so the response body is read incrementally via `resp.aiter_lines()` instead of buffered whole. Every retry/circuit-breaker mechanic in `_send`/`retry_send` is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; `retry_send` now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns `200` does the code path diverge into `_stream_chat_response`.
|
||||
@@ -238,6 +254,7 @@ Any `gateway_models` route (chat, embed, or image) can optionally name `fallback
|
||||
- **One ledger row per client call, not one per attempt.** The primary failure never writes a `gateway_usage_ledger` row by itself (only `_send`'s own internal retries and the circuit breaker counters observe it); `finalize()`/`self._ledger.record(...)` still runs exactly once, after the fallback decision, with `model` = whichever model ultimately answered and `requested_model` = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in `/admin/ai-usage` as "requested X, served Y" rather than as two calls.
|
||||
- **Streaming falls back before any byte reaches the client.** `client.send(request, stream=True)` returns as soon as the response headers arrive, so `resp.status_code` is already known before the SSE body is ever touched - `handle_chat` decides whether to fall back at that same point, before it ever opens `_stream_chat_response` to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
|
||||
- **Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (`self._breaker`) is one instance per `GatewayRuntime`, shared across every route.** A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see `circuit_open` too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
|
||||
- **A 400 is terminal UNLESS its body names rejected keys** the auto-resolve ladder can fix (see "Auto-resolve of request shape"): that case is retried on the SAME route with another shape before any fallback is considered.
|
||||
- **Deliberately NOT covered:** `gateway_max_retries`/`gateway_retry_backoff_ms`, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.
|
||||
|
||||
**Tests:** `tests/unit/services/openai_gateway/routing.py` (`resolve_fallback` self-reference/kind/active-flag guards, chain skip/cycle/hop-cap/entry tests), `tests/unit/services/openai_gateway/gateway.py` (`test_chat_falls_back_when_the_primary_model_fails`, `test_chat_returns_the_fallback_failure_when_both_models_fail`, `test_chat_without_a_fallback_never_makes_a_second_call`, `test_chat_walks_fallback_chain_past_failures`, `test_chat_preroutes_around_unhealthy_models`, `test_chat_tries_original_when_chain_is_exhausted`), and `tests/api/admin/gateway/models.py` (`test_model_route_fallback_round_trip`, `test_model_route_fallback_must_reference_an_existing_route`, `test_model_route_fallback_must_be_the_same_kind`, `test_model_route_fallback_rejects_self_reference`).
|
||||
|
||||
@@ -4,7 +4,8 @@ import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Optional
|
||||
|
||||
import httpx
|
||||
from fastapi.responses import JSONResponse, Response, StreamingResponse
|
||||
@@ -39,8 +40,13 @@ from devplacepy.services.openai_gateway.tool_emulation import (
|
||||
emulate_request,
|
||||
emulation_enabled,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import (
|
||||
NATIVE_METHOD,
|
||||
ShapeMethod,
|
||||
ShapeRetrier,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.thinking import (
|
||||
apply_thinking,
|
||||
client_thinking_enabled,
|
||||
upstream_capabilities,
|
||||
)
|
||||
@@ -60,6 +66,15 @@ from devplacepy.services.openai_gateway.vision import VisionAugmenter, VisionCac
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def _close_quietly(resp) -> None:
|
||||
if resp is None:
|
||||
return
|
||||
try:
|
||||
await resp.aclose()
|
||||
except Exception: # noqa: BLE001 - discarding an already-failed response
|
||||
pass
|
||||
|
||||
|
||||
def _call_failed(resp, exc, timing: dict) -> bool:
|
||||
if timing.get("circuit_open") or exc is not None:
|
||||
return True
|
||||
@@ -163,6 +178,7 @@ def _chat_upstream_body(
|
||||
log,
|
||||
catalog_cache=None,
|
||||
owner: tuple = ("anonymous", "anonymous"),
|
||||
method: ShapeMethod = NATIVE_METHOD,
|
||||
) -> dict:
|
||||
# The handler reasons about a Chat payload throughout (routing, vision,
|
||||
# precheck, fallback); only the bytes on the wire change per protocol.
|
||||
@@ -182,13 +198,13 @@ def _chat_upstream_body(
|
||||
)
|
||||
log(f"Meta reserved tool names aliased: {renamed}")
|
||||
if protocol != "responses":
|
||||
return payload
|
||||
return method.strip_wire(payload, protocol)
|
||||
send_body, warnings = chat_to_responses(payload)
|
||||
if warnings:
|
||||
shown = "; ".join(warnings[:3])
|
||||
more = f" (+{len(warnings) - 3} more)" if len(warnings) > 3 else ""
|
||||
log(f"chat->responses transpile warnings: {shown}{more}")
|
||||
if "reasoning" not in send_body:
|
||||
if "reasoning" not in send_body and not method.suppresses_reasoning(protocol):
|
||||
explicit = client_thinking_enabled(client_body)
|
||||
enabled = (
|
||||
explicit
|
||||
@@ -197,7 +213,7 @@ def _chat_upstream_body(
|
||||
)
|
||||
if not enabled:
|
||||
send_body["reasoning"] = {"effort": "minimal"}
|
||||
return send_body
|
||||
return method.strip_wire(send_body, protocol)
|
||||
|
||||
|
||||
def _response_tool_aliases(payload: dict, protocol: str, cfg: dict) -> dict:
|
||||
@@ -350,6 +366,20 @@ def _connect_tracer(holder: dict):
|
||||
return trace
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatAttempt:
|
||||
resp: Any
|
||||
exc: Any
|
||||
timing: dict
|
||||
shaped: dict
|
||||
send_start: float
|
||||
method: ShapeMethod
|
||||
|
||||
@property
|
||||
def resolved_label(self) -> Optional[str]:
|
||||
return None if self.method.is_native else self.method.label
|
||||
|
||||
|
||||
class GatewayRuntime:
|
||||
def __init__(self):
|
||||
self._client: Optional[httpx.AsyncClient] = None
|
||||
@@ -515,6 +545,102 @@ class GatewayRuntime:
|
||||
timing["retry_succeeded"] = attempts > 1
|
||||
return resp, None, timing
|
||||
|
||||
async def _shaped_send(
|
||||
self,
|
||||
client,
|
||||
sem,
|
||||
pristine: dict,
|
||||
body: dict,
|
||||
cfg: dict,
|
||||
protocol: str,
|
||||
model: str,
|
||||
headers: dict,
|
||||
stream: bool,
|
||||
owner: tuple,
|
||||
log,
|
||||
method: ShapeMethod,
|
||||
) -> ChatAttempt:
|
||||
shaped = method.apply(pristine, cfg, protocol, stream, log)
|
||||
shaped["model"] = model
|
||||
send_body = _chat_upstream_body(
|
||||
shaped, body, cfg, protocol, log, self._catalog_cache, owner, method
|
||||
)
|
||||
send_start = time.monotonic()
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
"POST",
|
||||
cfg["gateway_upstream_url"],
|
||||
headers,
|
||||
cfg,
|
||||
log,
|
||||
json_body=send_body,
|
||||
stream=stream,
|
||||
)
|
||||
if resp is not None and stream and resp.status_code != 200:
|
||||
# The streamed response body hasn't been read yet; the shared error
|
||||
# handling needs resp.text/.content, which requires an explicit
|
||||
# read for a stream=True response (a no-op if already buffered).
|
||||
try:
|
||||
await resp.aread()
|
||||
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
|
||||
# error body must never crash the request; fall through with
|
||||
# whatever text/content httpx managed to buffer (possibly empty).
|
||||
log(f"failed to read non-200 stream error body: {e}")
|
||||
return ChatAttempt(resp, exc, timing, shaped, send_start, method)
|
||||
|
||||
async def _chat_attempt(
|
||||
self,
|
||||
client,
|
||||
sem,
|
||||
pristine: dict,
|
||||
body: dict,
|
||||
cfg: dict,
|
||||
protocol: str,
|
||||
model: str,
|
||||
headers: dict,
|
||||
stream: bool,
|
||||
owner: tuple,
|
||||
log,
|
||||
) -> ChatAttempt:
|
||||
native_dialect = upstream_capabilities(
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
cfg.get("gateway_thinking_dialect", "auto"),
|
||||
).dialect
|
||||
retrier = ShapeRetrier.for_route(
|
||||
learned_store,
|
||||
cfg.get("gateway_provider_name") or "default",
|
||||
model,
|
||||
protocol,
|
||||
native_dialect,
|
||||
log,
|
||||
)
|
||||
first = await self._shaped_send(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log, retrier.method,
|
||||
)
|
||||
current = first
|
||||
while current.resp is not None and retrier.eligible(
|
||||
current.resp.status_code
|
||||
):
|
||||
method = retrier.next_after(current.resp.status_code, current.resp.text)
|
||||
if method is None:
|
||||
break
|
||||
if current is not first:
|
||||
await _close_quietly(current.resp)
|
||||
current = await self._shaped_send(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log, method,
|
||||
)
|
||||
if current is first:
|
||||
return first
|
||||
if current.resp is not None and current.resp.status_code < 400:
|
||||
retrier.succeeded()
|
||||
return current
|
||||
retrier.exhausted()
|
||||
await _close_quietly(current.resp)
|
||||
return first
|
||||
|
||||
async def handle_chat(
|
||||
self,
|
||||
body: dict,
|
||||
@@ -620,46 +746,10 @@ class GatewayRuntime:
|
||||
|
||||
stream = bool(body.get("stream"))
|
||||
include_usage = bool((body.get("stream_options") or {}).get("include_usage"))
|
||||
payload = dict(body)
|
||||
payload["model"] = model
|
||||
payload["messages"] = messages
|
||||
payload["stream"] = stream
|
||||
if stream and protocol == "responses":
|
||||
# Responses streams carry usage on response.completed natively;
|
||||
# there is no include_usage to request (stream_options only
|
||||
# supports include_obfuscation there). The translator captures it
|
||||
# and the client's own include_usage still controls the relay.
|
||||
payload.pop("stream_options", None)
|
||||
if stream and protocol != "responses":
|
||||
capabilities = upstream_capabilities(
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
cfg.get("gateway_thinking_dialect", "auto"),
|
||||
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
|
||||
)
|
||||
if capabilities.supports_stream_options:
|
||||
stream_options = dict(body.get("stream_options") or {})
|
||||
stream_options["include_usage"] = True
|
||||
payload["stream_options"] = stream_options
|
||||
else:
|
||||
payload.pop("stream_options", None)
|
||||
if include_usage:
|
||||
log(
|
||||
"stream_options.include_usage requested but the upstream "
|
||||
f"({capabilities.dialect}) does not support it; relaying "
|
||||
"without a usage chunk"
|
||||
)
|
||||
else:
|
||||
payload.pop("stream_options", None)
|
||||
if protocol != "responses":
|
||||
apply_thinking(
|
||||
payload,
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
default_enabled=bool(cfg.get("gateway_thinking", False)),
|
||||
dialect=cfg.get("gateway_thinking_dialect", "auto"),
|
||||
)
|
||||
# For responses upstreams the reasoning effort is set at transpile
|
||||
# time instead (see _chat_upstream_body): chat-dialect thinking
|
||||
# fields have no meaning on that wire.
|
||||
pristine = dict(body)
|
||||
pristine["model"] = model
|
||||
pristine["messages"] = messages
|
||||
pristine["stream"] = stream
|
||||
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
@@ -678,7 +768,7 @@ class GatewayRuntime:
|
||||
)
|
||||
context_window = _route_context_window(cfg, model)
|
||||
if context_window:
|
||||
estimated_tokens = _estimate_request_tokens(payload)
|
||||
estimated_tokens = _estimate_request_tokens(pristine)
|
||||
if estimated_tokens > context_window:
|
||||
for precheck_route in resolve_fallback_chain(chain_start, "chat"):
|
||||
precheck_overlay = chat_overlay(
|
||||
@@ -704,39 +794,19 @@ class GatewayRuntime:
|
||||
or "chat_completions"
|
||||
)
|
||||
model = cfg["gateway_model"]
|
||||
payload = dict(payload)
|
||||
payload["model"] = model
|
||||
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
|
||||
break
|
||||
|
||||
send_start = time.monotonic()
|
||||
try:
|
||||
send_body = _chat_upstream_body(
|
||||
payload, body, cfg, protocol, log, self._catalog_cache, owner
|
||||
attempt = await self._chat_attempt(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log,
|
||||
)
|
||||
except ToolEmulationError as tool_error:
|
||||
return _tool_emulation_rejection(tool_error)
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
"POST",
|
||||
cfg["gateway_upstream_url"],
|
||||
headers,
|
||||
cfg,
|
||||
log,
|
||||
json_body=send_body,
|
||||
stream=stream,
|
||||
)
|
||||
if resp is not None and stream and resp.status_code != 200:
|
||||
# The streamed response body hasn't been read yet; the shared error
|
||||
# handling below needs resp.text/.content, which requires an explicit
|
||||
# read for a stream=True response (a no-op if already buffered).
|
||||
try:
|
||||
await resp.aread()
|
||||
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
|
||||
# error body must never crash the request; fall through with
|
||||
# whatever text/content httpx managed to buffer (possibly empty).
|
||||
log(f"failed to read non-200 stream error body: {e}")
|
||||
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
|
||||
payload = attempt.shaped
|
||||
send_start = attempt.send_start
|
||||
|
||||
requested_model = requested
|
||||
body_error = _embedded_failure_message(resp, stream, protocol)
|
||||
@@ -761,32 +831,17 @@ class GatewayRuntime:
|
||||
cfg.get("gateway_upstream_protocol") or "chat_completions"
|
||||
)
|
||||
model = cfg["gateway_model"]
|
||||
payload = dict(payload)
|
||||
payload["model"] = model
|
||||
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
|
||||
send_start = time.monotonic()
|
||||
try:
|
||||
send_body = _chat_upstream_body(
|
||||
payload, body, cfg, protocol, log, self._catalog_cache, owner
|
||||
attempt = await self._chat_attempt(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log,
|
||||
)
|
||||
except ToolEmulationError as tool_error:
|
||||
return _tool_emulation_rejection(tool_error)
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
"POST",
|
||||
cfg["gateway_upstream_url"],
|
||||
headers,
|
||||
cfg,
|
||||
log,
|
||||
json_body=send_body,
|
||||
stream=stream,
|
||||
)
|
||||
if resp is not None and stream and resp.status_code != 200:
|
||||
try:
|
||||
await resp.aread()
|
||||
except httpx.HTTPError as e: # noqa: BLE001
|
||||
log(f"failed to read fallback non-200 stream error body: {e}")
|
||||
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
|
||||
payload = attempt.shaped
|
||||
send_start = attempt.send_start
|
||||
body_error = _embedded_failure_message(resp, stream, protocol)
|
||||
|
||||
pricing = pricing_from_cfg(cfg)
|
||||
@@ -803,6 +858,7 @@ class GatewayRuntime:
|
||||
"app_reference": app_reference,
|
||||
"provider": cfg.get("gateway_provider_name") or "default",
|
||||
"fallback_used_route": fallback_used_route,
|
||||
"resolved_method": attempt.resolved_label,
|
||||
**params,
|
||||
**timing,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from devplacepy.database import (
|
||||
bump_cache_version,
|
||||
db,
|
||||
get_table,
|
||||
sync_local_cache,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.shape import ShapeMethod
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TABLE = "gateway_learned_methods"
|
||||
CACHE_NAME = "gateway_learned"
|
||||
MAX_LEARNED_ROWS = 1000
|
||||
DEFAULT_PROVIDER = "default"
|
||||
|
||||
_LEARNED_CACHE: dict = {}
|
||||
|
||||
|
||||
class LearnedResetIn(BaseModel):
|
||||
provider: str = Field(default="", max_length=120)
|
||||
target_model: str = Field(min_length=1, max_length=200)
|
||||
|
||||
@field_validator("provider", "target_model")
|
||||
@classmethod
|
||||
def _strip(cls, value: str) -> str:
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _provider_key(provider: Optional[str]) -> str:
|
||||
return (provider or DEFAULT_PROVIDER).strip().lower() or DEFAULT_PROVIDER
|
||||
|
||||
|
||||
def ensure_table() -> None:
|
||||
db.query(
|
||||
"CREATE TABLE IF NOT EXISTS "
|
||||
+ TABLE
|
||||
+ " (id INTEGER PRIMARY KEY, provider TEXT, target_model TEXT, "
|
||||
"method TEXT, learned_at TEXT)"
|
||||
)
|
||||
try:
|
||||
db.query(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_learned_route ON "
|
||||
+ TABLE
|
||||
+ " (provider, target_model)"
|
||||
)
|
||||
db.query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_gateway_learned_at ON "
|
||||
+ TABLE
|
||||
+ " (learned_at)"
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("gateway learned index creation failed: %s", exc)
|
||||
|
||||
|
||||
def _load() -> dict:
|
||||
sync_local_cache(CACHE_NAME, _LEARNED_CACHE)
|
||||
if "methods" not in _LEARNED_CACHE:
|
||||
methods: dict = {}
|
||||
try:
|
||||
if TABLE in db.tables:
|
||||
for row in get_table(TABLE).all():
|
||||
method = ShapeMethod.from_json(row.get("method"))
|
||||
if method is None:
|
||||
continue
|
||||
methods[
|
||||
(_provider_key(row.get("provider")), str(row["target_model"]))
|
||||
] = (method, str(row.get("learned_at") or ""))
|
||||
except Exception as exc:
|
||||
logger.warning("gateway learned load failed: %s", exc)
|
||||
_LEARNED_CACHE["methods"] = methods
|
||||
return _LEARNED_CACHE["methods"]
|
||||
|
||||
|
||||
def _invalidate() -> None:
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_LEARNED_CACHE.clear()
|
||||
|
||||
|
||||
class LearnedMethodStore:
|
||||
def get(self, provider: Optional[str], model: str) -> Optional[ShapeMethod]:
|
||||
entry = _load().get((_provider_key(provider), str(model)))
|
||||
return entry[0] if entry else None
|
||||
|
||||
def list(self) -> list[dict]:
|
||||
rows = [
|
||||
{
|
||||
"provider": provider,
|
||||
"target_model": model,
|
||||
"method": method.label,
|
||||
"thinking_dialect": method.thinking_dialect,
|
||||
"drop_keys": list(method.drop_keys),
|
||||
"learned_at": learned_at,
|
||||
}
|
||||
for (provider, model), (method, learned_at) in _load().items()
|
||||
]
|
||||
return sorted(rows, key=lambda r: (r["provider"], r["target_model"]))
|
||||
|
||||
def remember(
|
||||
self, provider: Optional[str], model: str, method: ShapeMethod
|
||||
) -> None:
|
||||
if method.is_native or not model:
|
||||
return
|
||||
try:
|
||||
ensure_table()
|
||||
with db:
|
||||
db.query(
|
||||
"INSERT INTO " + TABLE + " (provider, target_model, method, learned_at) "
|
||||
"VALUES (:provider, :model, :method, :now) "
|
||||
"ON CONFLICT(provider, target_model) DO UPDATE SET "
|
||||
"method = excluded.method, learned_at = excluded.learned_at",
|
||||
provider=_provider_key(provider),
|
||||
model=str(model),
|
||||
method=method.to_json(),
|
||||
now=_now(),
|
||||
)
|
||||
db.query(
|
||||
"DELETE FROM " + TABLE + " WHERE id IN (SELECT id FROM "
|
||||
+ TABLE
|
||||
+ " ORDER BY learned_at ASC, id ASC LIMIT "
|
||||
"MAX((SELECT COUNT(*) FROM " + TABLE + ") - :cap, 0))",
|
||||
cap=MAX_LEARNED_ROWS,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("gateway learned remember failed: %s", exc)
|
||||
return
|
||||
_invalidate()
|
||||
|
||||
def forget(self, provider: Optional[str], model: str) -> bool:
|
||||
if TABLE not in db.tables:
|
||||
return False
|
||||
removed = int(
|
||||
get_table(TABLE).delete(
|
||||
provider=_provider_key(provider), target_model=str(model)
|
||||
)
|
||||
)
|
||||
if removed:
|
||||
_invalidate()
|
||||
return bool(removed)
|
||||
|
||||
def forget_provider(self, provider: str) -> int:
|
||||
if TABLE not in db.tables:
|
||||
return 0
|
||||
table = get_table(TABLE)
|
||||
removed = table.count(provider=_provider_key(provider))
|
||||
if removed:
|
||||
table.delete(provider=_provider_key(provider))
|
||||
_invalidate()
|
||||
return removed
|
||||
|
||||
|
||||
learned_store = LearnedMethodStore()
|
||||
@@ -16,6 +16,7 @@ from devplacepy.database import (
|
||||
sync_local_cache,
|
||||
)
|
||||
from devplacepy.services.openai_gateway import muse_code, opencode_zen
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.tool_emulation import (
|
||||
TOOLS_MODE_NATIVE,
|
||||
TOOLS_MODES,
|
||||
@@ -531,11 +532,17 @@ class ProviderStore:
|
||||
"protocol": payload.protocol,
|
||||
"updated_at": _now(),
|
||||
}
|
||||
endpoint_changed = bool(existing) and (
|
||||
str(existing.get("base_url") or "") != payload.base_url
|
||||
or str(existing.get("client_profile") or "") != payload.client_profile
|
||||
)
|
||||
if existing:
|
||||
table.update({**record, "id": existing["id"]}, ["id"])
|
||||
else:
|
||||
record["created_at"] = _now()
|
||||
table.insert(record)
|
||||
if endpoint_changed:
|
||||
learned_store.forget_provider(payload.name)
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_ROUTING_CACHE.clear()
|
||||
return self.get(payload.name) or record
|
||||
@@ -546,6 +553,7 @@ class ProviderStore:
|
||||
return False
|
||||
removed = int(get_table(PROVIDERS_TABLE).delete(name=name))
|
||||
if removed:
|
||||
learned_store.forget_provider(name)
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_ROUTING_CACHE.clear()
|
||||
return bool(removed)
|
||||
|
||||
@@ -0,0 +1,388 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
from devplacepy.services.openai_gateway.thinking import (
|
||||
CLIENT_THINKING_KEYS,
|
||||
apply_thinking,
|
||||
upstream_capabilities,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PROTOCOL_RESPONSES = "responses"
|
||||
DIALECT_NATIVE = "native"
|
||||
DIALECT_NONE = "none"
|
||||
DIALECT_DEEPSEEK = "deepseek"
|
||||
DIALECT_OPENROUTER = "openrouter"
|
||||
DIALECT_OLLAMA = "ollama"
|
||||
CHAT_LADDER = (
|
||||
DIALECT_NATIVE,
|
||||
DIALECT_NONE,
|
||||
DIALECT_DEEPSEEK,
|
||||
DIALECT_OPENROUTER,
|
||||
DIALECT_OLLAMA,
|
||||
)
|
||||
RESPONSES_LADDER = (DIALECT_NATIVE, DIALECT_NONE)
|
||||
DIALECTS = frozenset(CHAT_LADDER)
|
||||
DROPPABLE_KEYS = frozenset(
|
||||
{
|
||||
"stream_options",
|
||||
"parallel_tool_calls",
|
||||
"user",
|
||||
"seed",
|
||||
"logprobs",
|
||||
"top_logprobs",
|
||||
"n",
|
||||
"presence_penalty",
|
||||
"frequency_penalty",
|
||||
}
|
||||
)
|
||||
THINKING_KEYS = frozenset(CLIENT_THINKING_KEYS) | {"chat_template_kwargs"}
|
||||
DIALECT_EMITS: dict = {
|
||||
DIALECT_DEEPSEEK: frozenset({"thinking"}),
|
||||
DIALECT_OPENROUTER: frozenset({"reasoning"}),
|
||||
DIALECT_OLLAMA: frozenset({"think", "reasoning_effort"}),
|
||||
DIALECT_NONE: frozenset(),
|
||||
}
|
||||
RESPONSES_EMITS = frozenset({"reasoning"})
|
||||
MAX_RESOLVE_ATTEMPTS = 4
|
||||
SHAPE_STATUSES = (400, 422)
|
||||
MAX_REJECTED_KEYS = 16
|
||||
MAX_KEY_LENGTH = 64
|
||||
|
||||
_KEY_PATTERNS = (
|
||||
re.compile(r"Unrecognized key\(s\) in object:\s*((?:'[^']+'(?:,\s*)?)+)", re.I),
|
||||
re.compile(r"Unrecognized request arguments? supplied:\s*([\w.,\s]+)", re.I),
|
||||
re.compile(r"Unknown (?:parameter|field|argument)s?:?\s*[`'\"]?([\w.]+)", re.I),
|
||||
re.compile(r"unknown field\s*[`'\"]([\w.]+)[`'\"]", re.I),
|
||||
re.compile(r"Extra inputs are not permitted.{0,40}?['\"`]([\w.]+)['\"`]", re.I),
|
||||
)
|
||||
_QUOTED = re.compile(r"'([^']+)'")
|
||||
_BARE_SPLIT = re.compile(r"[,\s]+")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ShapeMethod:
|
||||
thinking_dialect: str = DIALECT_NATIVE
|
||||
drop_keys: tuple = ()
|
||||
|
||||
@property
|
||||
def is_native(self) -> bool:
|
||||
return self.thinking_dialect == DIALECT_NATIVE and not self.drop_keys
|
||||
|
||||
@property
|
||||
def label(self) -> str:
|
||||
return self.thinking_dialect + "".join(f" -{key}" for key in self.drop_keys)
|
||||
|
||||
def dialect_for(self, cfg: dict) -> str:
|
||||
if self.thinking_dialect == DIALECT_NATIVE:
|
||||
return str(cfg.get("gateway_thinking_dialect", "auto") or "auto")
|
||||
return self.thinking_dialect
|
||||
|
||||
def to_json(self) -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"thinking_dialect": self.thinking_dialect,
|
||||
"drop_keys": list(self.drop_keys),
|
||||
}
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_json(cls, text: Any) -> Optional["ShapeMethod"]:
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
dialect = data.get("thinking_dialect")
|
||||
drops = data.get("drop_keys")
|
||||
if dialect not in DIALECTS or not isinstance(drops, list):
|
||||
return None
|
||||
if any(key not in DROPPABLE_KEYS for key in drops):
|
||||
return None
|
||||
return cls(dialect, tuple(sorted(set(drops))))
|
||||
|
||||
def apply(
|
||||
self,
|
||||
payload: dict,
|
||||
cfg: dict,
|
||||
protocol: str,
|
||||
stream: bool,
|
||||
log: Callable[[str], None],
|
||||
) -> dict:
|
||||
shaped = dict(payload)
|
||||
kwargs = shaped.get("chat_template_kwargs")
|
||||
if isinstance(kwargs, dict):
|
||||
shaped["chat_template_kwargs"] = dict(kwargs)
|
||||
if protocol == PROTOCOL_RESPONSES:
|
||||
shaped.pop("stream_options", None)
|
||||
else:
|
||||
self._apply_stream_options(shaped, payload, cfg, stream, log)
|
||||
apply_thinking(
|
||||
shaped,
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
default_enabled=bool(cfg.get("gateway_thinking", False)),
|
||||
dialect=self.dialect_for(cfg),
|
||||
)
|
||||
for key in self.drop_keys:
|
||||
shaped.pop(key, None)
|
||||
return shaped
|
||||
|
||||
def _apply_stream_options(
|
||||
self,
|
||||
shaped: dict,
|
||||
payload: dict,
|
||||
cfg: dict,
|
||||
stream: bool,
|
||||
log: Callable[[str], None],
|
||||
) -> None:
|
||||
include_usage = bool((payload.get("stream_options") or {}).get("include_usage"))
|
||||
capabilities = upstream_capabilities(
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
cfg.get("gateway_thinking_dialect", "auto"),
|
||||
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
|
||||
)
|
||||
if (
|
||||
stream
|
||||
and capabilities.supports_stream_options
|
||||
and "stream_options" not in self.drop_keys
|
||||
):
|
||||
stream_options = dict(payload.get("stream_options") or {})
|
||||
stream_options["include_usage"] = True
|
||||
shaped["stream_options"] = stream_options
|
||||
return
|
||||
shaped.pop("stream_options", None)
|
||||
if stream and include_usage:
|
||||
log(
|
||||
"stream_options.include_usage requested but the upstream "
|
||||
f"({capabilities.dialect}) does not support it; relaying "
|
||||
"without a usage chunk"
|
||||
)
|
||||
|
||||
def suppresses_reasoning(self, protocol: str) -> bool:
|
||||
return protocol == PROTOCOL_RESPONSES and self.thinking_dialect == DIALECT_NONE
|
||||
|
||||
def strip_wire(self, body: dict, protocol: str) -> dict:
|
||||
doomed = set(self.drop_keys)
|
||||
if self.suppresses_reasoning(protocol):
|
||||
doomed.add("reasoning")
|
||||
if not doomed.intersection(body):
|
||||
return body
|
||||
return {key: value for key, value in body.items() if key not in doomed}
|
||||
|
||||
|
||||
NATIVE_METHOD = ShapeMethod()
|
||||
|
||||
|
||||
class RejectedKeys:
|
||||
@staticmethod
|
||||
def parse(body_text: Any) -> tuple:
|
||||
text = body_text if isinstance(body_text, str) else ""
|
||||
if not text:
|
||||
return ()
|
||||
found: list = []
|
||||
RejectedKeys._from_json(text, found)
|
||||
RejectedKeys._from_patterns(text, found)
|
||||
cleaned: list = []
|
||||
for key in found:
|
||||
top = str(key).strip().strip("`'\"").split(".")[0].split("[")[0]
|
||||
if top and len(top) <= MAX_KEY_LENGTH and top not in cleaned:
|
||||
cleaned.append(top)
|
||||
return tuple(cleaned[:MAX_REJECTED_KEYS])
|
||||
|
||||
@staticmethod
|
||||
def _from_json(text: str, found: list) -> None:
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except ValueError:
|
||||
return
|
||||
if not isinstance(data, dict):
|
||||
return
|
||||
issues = data.get("issues")
|
||||
if isinstance(issues, list):
|
||||
for issue in issues:
|
||||
if isinstance(issue, dict) and issue.get("code") == "unrecognized_keys":
|
||||
keys = issue.get("keys")
|
||||
if isinstance(keys, list):
|
||||
found.extend(str(key) for key in keys)
|
||||
error = data.get("error")
|
||||
if isinstance(error, dict):
|
||||
param = error.get("param")
|
||||
code = str(error.get("code") or "")
|
||||
if isinstance(param, str) and param and code in (
|
||||
"unknown_parameter",
|
||||
"unrecognized_request_argument",
|
||||
):
|
||||
found.append(param)
|
||||
|
||||
@staticmethod
|
||||
def _from_patterns(text: str, found: list) -> None:
|
||||
for pattern in _KEY_PATTERNS:
|
||||
for match in pattern.finditer(text):
|
||||
chunk = match.group(1)
|
||||
quoted = _QUOTED.findall(chunk)
|
||||
if quoted:
|
||||
found.extend(quoted)
|
||||
else:
|
||||
found.extend(
|
||||
part for part in _BARE_SPLIT.split(chunk.strip()) if part
|
||||
)
|
||||
|
||||
|
||||
class ShapeResolver:
|
||||
def __init__(self, native_dialect: str, protocol: str) -> None:
|
||||
self.native_dialect = native_dialect
|
||||
self.responses = protocol == PROTOCOL_RESPONSES
|
||||
|
||||
def emits(self, dialect: str) -> frozenset:
|
||||
if self.responses:
|
||||
return frozenset() if dialect == DIALECT_NONE else RESPONSES_EMITS
|
||||
if dialect == DIALECT_NATIVE:
|
||||
return DIALECT_EMITS.get(self.native_dialect, frozenset())
|
||||
return DIALECT_EMITS.get(dialect, frozenset())
|
||||
|
||||
def next_method(
|
||||
self, tried: set, current: ShapeMethod, rejected: tuple
|
||||
) -> Optional[ShapeMethod]:
|
||||
keys = set(rejected)
|
||||
drops = set(current.drop_keys) | (keys & DROPPABLE_KEYS)
|
||||
if self.responses:
|
||||
thinking_hit = keys & RESPONSES_EMITS
|
||||
else:
|
||||
thinking_hit = keys & THINKING_KEYS
|
||||
if not thinking_hit and drops == set(current.drop_keys):
|
||||
return None
|
||||
sorted_drops = tuple(sorted(drops))
|
||||
if not thinking_hit:
|
||||
candidate = ShapeMethod(current.thinking_dialect, sorted_drops)
|
||||
return None if candidate in tried else candidate
|
||||
ladder = RESPONSES_LADDER if self.responses else CHAT_LADDER
|
||||
for dialect in ladder:
|
||||
if self.emits(dialect) & keys:
|
||||
continue
|
||||
candidate = ShapeMethod(dialect, sorted_drops)
|
||||
if candidate not in tried:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ShapeStats:
|
||||
resolved: int = 0
|
||||
exhausted: int = 0
|
||||
healed: int = 0
|
||||
|
||||
def snapshot(self) -> dict:
|
||||
return {
|
||||
"resolved": self.resolved,
|
||||
"exhausted": self.exhausted,
|
||||
"healed": self.healed,
|
||||
}
|
||||
|
||||
|
||||
shape_stats = ShapeStats()
|
||||
|
||||
|
||||
@dataclass
|
||||
class ShapeRetrier:
|
||||
store: Any
|
||||
provider: str
|
||||
model: str
|
||||
resolver: ShapeResolver
|
||||
log: Callable[[str], None]
|
||||
method: ShapeMethod = field(default=NATIVE_METHOD)
|
||||
from_store: bool = False
|
||||
healed: bool = False
|
||||
attempts: int = 0
|
||||
tried: set = field(default_factory=set)
|
||||
|
||||
@classmethod
|
||||
def for_route(
|
||||
cls,
|
||||
store: Any,
|
||||
provider: str,
|
||||
model: str,
|
||||
protocol: str,
|
||||
native_dialect: str,
|
||||
log: Callable[[str], None],
|
||||
) -> "ShapeRetrier":
|
||||
learned = store.get(provider, model)
|
||||
retrier = cls(
|
||||
store=store,
|
||||
provider=provider,
|
||||
model=model,
|
||||
resolver=ShapeResolver(native_dialect, protocol),
|
||||
log=log,
|
||||
method=learned or NATIVE_METHOD,
|
||||
from_store=learned is not None,
|
||||
)
|
||||
retrier.tried.add(retrier.method)
|
||||
return retrier
|
||||
|
||||
@staticmethod
|
||||
def eligible(status: int) -> bool:
|
||||
return status in SHAPE_STATUSES
|
||||
|
||||
def next_after(self, status: int, body_text: Any) -> Optional[ShapeMethod]:
|
||||
if not self.eligible(status) or self.attempts >= MAX_RESOLVE_ATTEMPTS:
|
||||
return None
|
||||
rejected = RejectedKeys.parse(body_text)
|
||||
if not rejected:
|
||||
return None
|
||||
if self.from_store and not self.healed:
|
||||
self.store.forget(self.provider, self.model)
|
||||
self.healed = True
|
||||
self.from_store = False
|
||||
shape_stats.healed += 1
|
||||
self.log(
|
||||
f"learned shape '{self.method.label}' for {self.provider}/"
|
||||
f"{self.model} was rejected ({', '.join(rejected)}); "
|
||||
"forgotten, restarting the ladder"
|
||||
)
|
||||
self.tried = {self.method}
|
||||
if NATIVE_METHOD not in self.tried:
|
||||
return self._advance(NATIVE_METHOD)
|
||||
candidate = self.resolver.next_method(self.tried, self.method, rejected)
|
||||
if candidate is None:
|
||||
return None
|
||||
return self._advance(candidate)
|
||||
|
||||
def _advance(self, candidate: ShapeMethod) -> ShapeMethod:
|
||||
self.attempts += 1
|
||||
self.tried.add(candidate)
|
||||
self.method = candidate
|
||||
self.log(
|
||||
f"upstream rejected the request shape for {self.provider}/"
|
||||
f"{self.model}, trying '{candidate.label}'"
|
||||
)
|
||||
return candidate
|
||||
|
||||
def succeeded(self) -> None:
|
||||
if self.attempts == 0 or self.method.is_native:
|
||||
return
|
||||
self.store.remember(self.provider, self.model, self.method)
|
||||
shape_stats.resolved += 1
|
||||
self.log(
|
||||
f"auto-resolved {self.provider}/{self.model}: {self.method.label}"
|
||||
)
|
||||
|
||||
def exhausted(self) -> None:
|
||||
if self.attempts == 0:
|
||||
return
|
||||
shape_stats.exhausted += 1
|
||||
self.log(
|
||||
f"could not auto-resolve the request shape for {self.provider}/"
|
||||
f"{self.model} after {self.attempts} attempt(s)"
|
||||
)
|
||||
|
||||
@property
|
||||
def resolved_label(self) -> Optional[str]:
|
||||
return None if self.method.is_native else self.method.label
|
||||
@@ -144,6 +144,8 @@ def _clear_thinking_fields(payload: dict) -> None:
|
||||
|
||||
def write_thinking(payload: dict, dialect: str, enabled: bool) -> None:
|
||||
_clear_thinking_fields(payload)
|
||||
if dialect == "none":
|
||||
return
|
||||
if dialect == "openrouter":
|
||||
payload["reasoning"] = {"effort": "high" if enabled else "none"}
|
||||
return
|
||||
|
||||
@@ -587,6 +587,11 @@ class GatewayUsageLedger:
|
||||
if raw.get("fallback_used_route")
|
||||
else None
|
||||
),
|
||||
"resolved_method": (
|
||||
str(raw["resolved_method"])[:128]
|
||||
if raw.get("resolved_method")
|
||||
else None
|
||||
),
|
||||
"status_code": int(raw.get("status_code") or 0),
|
||||
"success": 1 if raw.get("success") else 0,
|
||||
"error_category": raw.get("error_category"),
|
||||
|
||||
@@ -6,6 +6,7 @@ import logging
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from typing import Any, Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
@@ -16,7 +17,9 @@ from devplacepy.services.openai_gateway.reliability import (
|
||||
_is_retryable_status,
|
||||
_retry_after_seconds,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.thinking import apply_thinking
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import ShapeRetrier
|
||||
from devplacepy.services.openai_gateway.thinking import upstream_capabilities
|
||||
from devplacepy.services.openai_gateway.usage import classify_error
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -151,37 +154,9 @@ class VisionAugmenter:
|
||||
if row:
|
||||
self.cost_usd += float(row.get("cost_usd") or 0.0)
|
||||
|
||||
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
|
||||
if not self.vision_key:
|
||||
return "[vision unavailable: vision API key not configured]"
|
||||
if self.breaker is not None and not self.breaker.allow():
|
||||
self._record((0.0), 503, False, "circuit_open", None)
|
||||
return "[vision unavailable: circuit breaker open]"
|
||||
payload = {
|
||||
"model": self.vision_model,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": VISION_INSTRUCTION},
|
||||
image_block,
|
||||
],
|
||||
}
|
||||
],
|
||||
"temperature": 0.2,
|
||||
"stream": False,
|
||||
}
|
||||
apply_thinking(payload, self.vision_url, default_enabled=False, dialect=self.vision_dialect)
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.vision_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
if self.referer:
|
||||
headers["HTTP-Referer"] = self.referer
|
||||
if self.title:
|
||||
headers[TITLE_HEADER] = self.title
|
||||
headers[LEGACY_TITLE_HEADER] = self.title
|
||||
start = time.monotonic()
|
||||
async def _post_with_retries(
|
||||
self, client: httpx.AsyncClient, payload: dict, headers: dict
|
||||
) -> tuple:
|
||||
attempt = 0
|
||||
resp: Optional[httpx.Response] = None
|
||||
exc: Optional[Exception] = None
|
||||
@@ -210,6 +185,68 @@ class VisionAugmenter:
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
await _backoff(self.retry_backoff_ms, attempt, retry_after)
|
||||
return resp, exc
|
||||
|
||||
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
|
||||
if not self.vision_key:
|
||||
return "[vision unavailable: vision API key not configured]"
|
||||
if self.breaker is not None and not self.breaker.allow():
|
||||
self._record((0.0), 503, False, "circuit_open", None)
|
||||
return "[vision unavailable: circuit breaker open]"
|
||||
payload = {
|
||||
"model": self.vision_model,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": VISION_INSTRUCTION},
|
||||
image_block,
|
||||
],
|
||||
}
|
||||
],
|
||||
"temperature": 0.2,
|
||||
"stream": False,
|
||||
}
|
||||
shape_cfg = {
|
||||
"gateway_upstream_url": self.vision_url,
|
||||
"gateway_thinking_dialect": self.vision_dialect or "auto",
|
||||
"gateway_thinking": False,
|
||||
}
|
||||
retrier = ShapeRetrier.for_route(
|
||||
learned_store,
|
||||
"vision:" + (urlparse(self.vision_url).hostname or "default"),
|
||||
self.vision_model,
|
||||
"chat_completions",
|
||||
upstream_capabilities(self.vision_url, self.vision_dialect).dialect,
|
||||
logger.info,
|
||||
)
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.vision_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
if self.referer:
|
||||
headers["HTTP-Referer"] = self.referer
|
||||
if self.title:
|
||||
headers[TITLE_HEADER] = self.title
|
||||
headers[LEGACY_TITLE_HEADER] = self.title
|
||||
start = time.monotonic()
|
||||
shaped = retrier.method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
|
||||
resp, exc = await self._post_with_retries(client, shaped, headers)
|
||||
first = (resp, exc)
|
||||
resolving = False
|
||||
while resp is not None and retrier.eligible(resp.status_code):
|
||||
method = retrier.next_after(resp.status_code, resp.text)
|
||||
if method is None:
|
||||
break
|
||||
resolving = True
|
||||
shaped = method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
|
||||
resp, exc = await self._post_with_retries(client, shaped, headers)
|
||||
if resolving:
|
||||
if resp is not None and resp.status_code < 400:
|
||||
retrier.succeeded()
|
||||
else:
|
||||
retrier.exhausted()
|
||||
resp, exc = first
|
||||
if self.breaker is not None:
|
||||
if exc is not None or (resp is not None and resp.status_code >= 500):
|
||||
self.breaker.record_failure()
|
||||
|
||||
Reference in New Issue
Block a user