Gateway: auto-resolve request shape per provider and model

An upstream that rejects a key the gateway adds (e.g. thinking) now gets
retried with other shapes on a parsable 400/422; the working method is
stored per (provider, target model) and applied first-try from then on.
Fallback attempts are re-shaped per attempt. Admin list/reset, Devii
tools, audit event and ledger resolved_method included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-13 11:34:26 +01:00
co-authored by Claude Sonnet 5.5
parent 3f6e83c753
commit 151476125f
22 changed files with 1597 additions and 129 deletions
+1 -1
View File
@@ -533,7 +533,7 @@ and its full configuration are documented automatically - including future servi
- **`ServiceManager`** - singleton: `register`, `describe_all`, `set_enabled`, `send_command`, `save_config`, `supervise`, `shutdown_all`
- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one against a free, local, deterministic quality model (aquality - no billed LLM call, no API key), optionally reformats articles that clear the publish grade threshold into clean Markdown with a generative AI model (off by default, since the default grading model cannot generate text), and auto-rotates the best articles to Featured and the landing page. Any AI spend it does incur is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages)
- **`BotsService`** - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics and a live screenshot monitor at `/admin/bots` (opt-in; install the `bots` extra)
- **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU `DD/MM/YYYY` format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (`gateway_system_preamble`) can be prepended ahead of the client's system message on every call. **Thinking is disabled by default** on every chat and vision call (`thinking.type=disabled` on DeepSeek, `reasoning.effort=none` on OpenRouter, `think=false` on Ollama) so the fast path is the default; a client may re-enable it per request, and an administrator may flip `gateway_thinking` on. **Provider and model routing** (admin **Gateway** page, `/admin/gateway`) maps any number of requested model names onto named upstream providers and target models, each with its own pricing economy (input, output and cache-hit/cache-miss rates per million tokens) and an optional vision model that describes image content before forwarding, so text and vision models are merged transparently. Unmapped requests fall through to the default upstream unchanged, so existing clients are unaffected. Each route may also name a **fallback model** - another already-configured model of the same kind, picked from a select box of the public model names (never a free-text provider model id) - that the gateway retries once, automatically, whenever the primary model fails after its own retries are exhausted, so a struggling model degrades to a working one instead of failing the caller
- **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU `DD/MM/YYYY` format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (`gateway_system_preamble`) can be prepended ahead of the client's system message on every call. **Thinking is disabled by default** on every chat and vision call (`thinking.type=disabled` on DeepSeek, `reasoning.effort=none` on OpenRouter, `think=false` on Ollama) so the fast path is the default; a client may re-enable it per request, and an administrator may flip `gateway_thinking` on. **Provider and model routing** (admin **Gateway** page, `/admin/gateway`) maps any number of requested model names onto named upstream providers and target models, each with its own pricing economy (input, output and cache-hit/cache-miss rates per million tokens) and an optional vision model that describes image content before forwarding, so text and vision models are merged transparently. Unmapped requests fall through to the default upstream unchanged, so existing clients are unaffected. Each route may also name a **fallback model** - another already-configured model of the same kind, picked from a select box of the public model names (never a free-text provider model id) - that the gateway retries once, automatically, whenever the primary model fails after its own retries are exhausted, so a struggling model degrades to a working one instead of failing the caller. When an upstream rejects a request key with a parsable unrecognized-key 400 (for example a strict-schema provider refusing the `thinking` field), the gateway automatically tries other request shapes, remembers the one that works per provider and target model, and applies it from then on; administrators see it as a `shape:` badge on the model grid with a Reset learned action
- **`JobService` / `ZipService` / `ForkService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` builds project zip archives in a subprocess, `ForkService` copies a project into a new project owned by the forking user
- **`ContainerService`** - the admin container manager (`services/containers/`): a reconciling supervisor for container instances, all running one shared prebuilt image
- **`AcceptanceService`** - grants every policy agreement (Terms of Service, Privacy Policy, third-party AI processing, activity recording, container credentials) to every account that has not declined it, so an instance kept production-identical for extended manual testing never interrupts with an acceptance dialog. Administrator-only, **off by default**, with a separate switch per agreement type and a dry run that reports what it would do without writing. An account that withdrew a consent is never granted it again, with no further action: the consent ledger itself is the decline register. It is not appropriate on a real production host
+5
View File
@@ -685,6 +685,7 @@ def init_db():
("inter_token_ms", 0.0),
("provider", ""),
("fallback_used_route", ""),
("resolved_method", ""),
):
if not gateway_usage_ledger.has_column(column):
gateway_usage_ledger.create_column_by_example(column, example)
@@ -950,6 +951,10 @@ def init_db():
gateway_routing.ensure_tables()
from devplacepy.services.openai_gateway import learned as gateway_learned
gateway_learned.ensure_table()
from devplacepy.services.openai_gateway import quota as gateway_quota
gateway_quota.ensure_tables()
+43
View File
@@ -796,6 +796,49 @@ four ways to sign requests.
field("app_reference", "json", "string", False, "typosaurus", "App label (the X-App-Reference header). Blank = every app."),
],
),
endpoint(
id="admin-gateway-learned",
method="GET",
path="/admin/gateway/learned",
title="List learned gateway request shapes",
summary=(
"Every request shape the gateway learned automatically after an upstream rejected the "
"default one with an unrecognized-key 400, per provider and target model, plus the "
"resolved/exhausted/healed counters."
),
auth="admin",
sample_response={
"learned": [
{
"provider": "vaids",
"target_model": "deepseek-v4-1-flash-1017-uncensored",
"method": "none",
"thinking_dialect": "none",
"drop_keys": [],
"learned_at": "2026-11-13T00:55:00+00:00",
}
],
"count": 1,
"stats": {"resolved": 1, "exhausted": 0, "healed": 0},
},
),
endpoint(
id="admin-gateway-learned-reset",
method="POST",
path="/admin/gateway/learned/reset",
title="Reset a learned gateway request shape",
summary=(
"Forget the learned method for one provider and target model so the gateway starts from "
"the default shape again. Use provider 'default' for the base upstream."
),
auth="admin",
destructive=True,
params=[
field("provider", "json", "string", False, "vaids", "Provider name, or default."),
field("target_model", "json", "string", True, "deepseek-v4-1-flash-1017-uncensored", "Upstream target model."),
],
sample_response={"ok": True},
),
endpoint(
id="admin-gateway-quota-rule-delete",
method="DELETE",
+75 -1
View File
@@ -24,6 +24,12 @@ from devplacepy.services.openai_gateway import (
quota,
routing,
)
from devplacepy.services.openai_gateway.learned import (
DEFAULT_PROVIDER,
LearnedResetIn,
learned_store,
)
from devplacepy.services.openai_gateway.shape import shape_stats
from devplacepy.templating import templates
from devplacepy.utils import not_found, require_admin
@@ -151,6 +157,36 @@ def _error_response(
)
def _models_with_learned() -> list[dict]:
models = routing.model_store.list()
for model in models:
method = learned_store.get(
model["provider"] or DEFAULT_PROVIDER, model["target_model"]
)
model["learned_shape"] = method.label if method is not None else ""
model["learned_provider"] = model["provider"] or DEFAULT_PROVIDER
return models
def _reset_learned(request: Request, admin: dict, payload: LearnedResetIn) -> bool:
provider = payload.provider or DEFAULT_PROVIDER
existed = learned_store.forget(provider, payload.target_model)
if existed:
audit.record(
request,
"gateway.learned.reset",
user=admin,
target_type="gateway_learned_method",
target_uid=f"{provider}/{payload.target_model}",
target_label=f"{provider}/{payload.target_model}",
summary=(
f"admin {admin['username']} reset the learned request shape "
f"for {provider}/{payload.target_model}"
),
)
return existed
# --- Index page (tabs: models / providers / quota) --------------------------
@@ -177,7 +213,7 @@ async def gateway_config_page(
"admin_section": "gateway",
"tab": tab,
"providers": routing.provider_store.list(),
"models": routing.model_store.list(),
"models": _models_with_learned(),
"quota_rules": quota_rules,
"default_provider": _default_provider_summary(),
"quota_defaults": _quota_defaults_summary(),
@@ -388,6 +424,44 @@ async def list_models(request: Request):
)
@router.get("/gateway/learned")
async def list_learned(request: Request):
require_admin(request)
rows = learned_store.list()
return JSONResponse(
{"learned": rows, "count": len(rows), "stats": shape_stats.snapshot()}
)
@router.post("/gateway/learned/reset")
async def reset_learned_json(request: Request):
admin = require_admin(request)
body = await _payload(request)
try:
payload = LearnedResetIn(**body)
except ValidationError as exc:
return _validation_error(exc)
if not _reset_learned(request, admin, payload):
return JSONResponse(
{"ok": False, "error": "No learned method for that route"},
status_code=404,
)
return JSONResponse({"ok": True})
@router.post("/gateway/learned/reset-page")
async def reset_learned_page(request: Request):
admin = require_admin(request)
body = await _payload(request)
try:
payload = LearnedResetIn(**body)
except ValidationError as exc:
return _validation_error(exc)
if not _reset_learned(request, admin, payload):
raise not_found("No learned method for that route")
return RedirectResponse(url=f"{INDEX_URL}?tab=models", status_code=302)
@router.get("/gateway/provider-models")
async def provider_models(request: Request, provider: str = ""):
require_admin(request)
@@ -208,6 +208,39 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
confirm(),
),
),
Action(
name="gateway_learned_list",
method="GET",
path="/admin/gateway/learned",
summary="List the request shapes the gateway learned automatically per provider and target model (admin only)",
description=(
"Returns JSON: every learned method (provider, target_model, the method label, its "
"thinking dialect and dropped keys, when it was learned) plus the resolve/exhausted/"
"healed counters. A method is learned when an upstream rejected the default request "
"shape with a parsable unrecognized-key 400 and a retry with another shape succeeded."
),
handler="http",
requires_admin=True,
read_only=True,
),
Action(
name="gateway_learned_reset",
method="POST",
path="/admin/gateway/learned/reset",
summary="Forget a learned request shape so the gateway re-learns it (admin only, confirmation required)",
description=(
"Deletes the learned method for one provider and target model; the next request starts "
"from the default shape again and re-resolves if the upstream still rejects it. Use "
"provider 'default' for the base upstream. 404 when nothing is learned for that route."
),
handler="http",
requires_admin=True,
params=(
body("provider", "Provider name, or 'default' for the base upstream."),
body("target_model", "The upstream (target) model name the method was learned for."),
confirm(),
),
),
Action(
name="gateway_quota_rule_delete",
method="DELETE",
@@ -83,6 +83,7 @@ CONFIRM_REQUIRED = {
"gateway_model_delete",
"gateway_quota_rule_delete",
"gateway_quota_reset",
"gateway_learned_reset",
"email_account_delete",
"email_delete_message",
"game_prestige",
@@ -106,6 +106,8 @@ A structured/deterministic internal caller (one that expects a strict, machine-p
- OpenRouter: `reasoning: {"effort": "none"}` or `"high"`
- Ollama (`:11434` or a host containing `ollama`): `think: false` or `true`
`thinking_dialect` can also be `none` (no field written); it is only ever chosen by the auto-resolve ladder when an upstream rejects the thinking key (see "Auto-resolve of request shape").
Do **not** send `reasoning_effort: "none"` to DeepSeek (400: unknown variant). Conflicting client fields (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs.enable_thinking`) are stripped and translated to the dialect above.
**Default is off.** `gateway_thinking` (Prompt group, bool, default `False`) is the operator override for the silent-client case. A client still wins: `think` / `thinking.type` / `reasoning.effort` / `reasoning.enabled` / `reasoning_effort` / `enable_thinking` on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.
@@ -116,6 +118,20 @@ Vision describe-image calls always disable thinking (they are not a reasoning jo
`thinking.upstream_capabilities(url, dialect="")` derives per-upstream protocol flags (`supports_stream_options`, `supports_stream_usage`, `supports_thinking_field`) from the same effective dialect. Ollama's OpenAI-compatibility layer ignores `stream_options` (and rejects the request outright on some versions), so `handle_chat`'s streaming branch only sends `stream_options: {"include_usage": true}` upstream when `capabilities.supports_stream_options` is true; otherwise a client that asked for `include_usage` gets a logged notice and its stream relayed with no usage chunk. DeepSeek/OpenRouter (and any unrecognised URL) report full support, so this is a no-op for them.
## Auto-resolve of request shape (`shape.py`, `learned.py`)
An upstream with a strict schema can reject a key the gateway adds on its own (observed 2026-11-13: route `vaids` answered `400 {"issues":[{"code":"unrecognized_keys","keys":["thinking"]}]}` because `thinking_dialect` falls back to `deepseek` for any unrecognised URL). A 400 is `bad_request`, never retried and never falls back, so the gateway now LEARNS, per (provider, target model), which shape the upstream accepts and applies it from then on.
- **One funnel.** `ShapeMethod.apply(payload, cfg, protocol, stream, log)` is the only place stream_options, the thinking dialect and droppable keys are written; chat, every fallback attempt and the vision describe call all go through it, each from the PRISTINE client payload with that attempt's own cfg (fallback attempts used to inherit the primary's already-shaped payload). `ShapeMethod.strip_wire` removes `drop_keys` (and `reasoning` for a Responses `none` method) from the final wire body in `_chat_upstream_body`. A method is a frozen `ShapeMethod(thinking_dialect, drop_keys)`; `thinking_dialect` is `native` (today's behaviour: the URL-sniffed or `gateway_thinking_dialect` dialect, the default and first ladder entry), `none` (write no thinking field at all, new in `thinking.py`), or an explicit `deepseek`/`openrouter`/`ollama`.
- **Trigger.** Only a 400/422 whose body names rejected keys (`RejectedKeys.parse`: zod `issues[].keys` with `unrecognized_keys`, OpenAI `Unrecognized request argument supplied`/`Unknown parameter` + `param`, serde `unknown field`, "Extra inputs are not permitted"). An unparsable 400 stays terminal and loud, returned verbatim, never retried blind.
- **Ladder and allowlist (static ceilings, no autotuning).** `ShapeResolver.next_method` picks the next untried method: a rejected thinking key (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs`) skips every dialect that emits it (`CHAT_LADDER` order: native, none, deepseek, openrouter, ollama; Responses protocol: native, none, rejecting `reasoning`); a rejected key in `DROPPABLE_KEYS` (`stream_options`, `parallel_tool_calls`, `user`, `seed`, `logprobs`, `top_logprobs`, `n`, `presence_penalty`, `frequency_penalty`) is added to `drop_keys`. `messages`, `model`, `tools` and any unknown key are NEVER dropped. Hard cap `MAX_RESOLVE_ATTEMPTS = 4` retries per route attempt.
- **Retry placement.** `GatewayRuntime._chat_attempt` (shared by the primary and every fallback attempt) sends, and on a parsable shape 400 re-sends through `ShapeRetrier`; the retry goes through `_send` so the breaker and retries apply, and streaming status is known before any byte is relayed so a retried stream is safe. On success `learned_store.remember` is called (never on a guess) and one `auto-resolved <provider>/<model>: <method>` line is logged. On exhaustion the ORIGINAL 400 response is returned unchanged, nothing is stored, and `shape_stats.exhausted` is incremented. The vision describe call uses the same `ShapeRetrier`, keyed `vision:<host>` + vision model.
- **Persistence.** Table `gateway_learned_methods` (`learned.py`, `ensure_table()` from `init_db`): `provider` (`default` for the base upstream), `target_model`, `method` (JSON), `learned_at`, unique index on (provider, target_model), cap `MAX_LEARNED_ROWS` 1000 (oldest by `learned_at` replaced). Cross-worker cache under the `gateway_learned` cache-version name; the upsert is one `ON CONFLICT DO UPDATE` so two workers learning the same route never collide. It is derived data in the garbage-collection class (NOT in `SOFT_DELETE_TABLES`): forget = delete row. Editing a provider's `base_url` or `client_profile`, or deleting it, forgets all its learned methods (`ProviderStore.set`/`remove`).
- **Self-healing floor.** If a request using a LEARNED method itself gets a shape 400, the record is forgotten, the ladder restarts once from native, and `shape_stats.healed` counts it.
- **Ledger.** One row per client call as before; `resolved_method` (text, `gateway_usage_ledger`) holds the non-native method label that served the call, empty for native.
- **Operator surface.** `/admin/gateway` models grid shows a `shape: <method>` badge and a "Reset learned" form (`POST /admin/gateway/learned/reset-page`); JSON `GET /admin/gateway/learned` and `POST /admin/gateway/learned/reset` (audited `gateway.learned.reset`, category `ai`) back the Devii tools `gateway_learned_list` / `gateway_learned_reset` (the reset is in `CONFIRM_REQUIRED` with a declared `confirm`).
- **Not covered:** value-level rejections (`unknown variant`), and keys outside the closed allowlists. Tests: `tests/unit/services/openai_gateway/{shape,learned}.py`, the `test_chat_auto_resolve*`/`test_chat_*learned*`/`test_fallback_attempt_is_shaped*` tests in `gateway.py`, `tests/api/admin/gateway/learned.py`, and the grid badge in `tests/e2e/admin/gateway.py`.
## Real upstream streaming (`GatewayRuntime._stream_chat_response`)
`stream: true` is forwarded to the upstream verbatim (`payload["stream"] = stream`, no more forced `false`) and the connection is opened with `client.send(request, stream=True)` so the response body is read incrementally via `resp.aiter_lines()` instead of buffered whole. Every retry/circuit-breaker mechanic in `_send`/`retry_send` is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; `retry_send` now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns `200` does the code path diverge into `_stream_chat_response`.
@@ -238,6 +254,7 @@ Any `gateway_models` route (chat, embed, or image) can optionally name `fallback
- **One ledger row per client call, not one per attempt.** The primary failure never writes a `gateway_usage_ledger` row by itself (only `_send`'s own internal retries and the circuit breaker counters observe it); `finalize()`/`self._ledger.record(...)` still runs exactly once, after the fallback decision, with `model` = whichever model ultimately answered and `requested_model` = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in `/admin/ai-usage` as "requested X, served Y" rather than as two calls.
- **Streaming falls back before any byte reaches the client.** `client.send(request, stream=True)` returns as soon as the response headers arrive, so `resp.status_code` is already known before the SSE body is ever touched - `handle_chat` decides whether to fall back at that same point, before it ever opens `_stream_chat_response` to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
- **Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (`self._breaker`) is one instance per `GatewayRuntime`, shared across every route.** A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see `circuit_open` too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
- **A 400 is terminal UNLESS its body names rejected keys** the auto-resolve ladder can fix (see "Auto-resolve of request shape"): that case is retried on the SAME route with another shape before any fallback is considered.
- **Deliberately NOT covered:** `gateway_max_retries`/`gateway_retry_backoff_ms`, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.
**Tests:** `tests/unit/services/openai_gateway/routing.py` (`resolve_fallback` self-reference/kind/active-flag guards, chain skip/cycle/hop-cap/entry tests), `tests/unit/services/openai_gateway/gateway.py` (`test_chat_falls_back_when_the_primary_model_fails`, `test_chat_returns_the_fallback_failure_when_both_models_fail`, `test_chat_without_a_fallback_never_makes_a_second_call`, `test_chat_walks_fallback_chain_past_failures`, `test_chat_preroutes_around_unhealthy_models`, `test_chat_tries_original_when_chain_is_exhausted`), and `tests/api/admin/gateway/models.py` (`test_model_route_fallback_round_trip`, `test_model_route_fallback_must_reference_an_existing_route`, `test_model_route_fallback_must_be_the_same_kind`, `test_model_route_fallback_rejects_self_reference`).
+149 -93
View File
@@ -4,7 +4,8 @@ import asyncio
import json
import logging
import time
from typing import Optional
from dataclasses import dataclass
from typing import Any, Optional
import httpx
from fastapi.responses import JSONResponse, Response, StreamingResponse
@@ -39,8 +40,13 @@ from devplacepy.services.openai_gateway.tool_emulation import (
emulate_request,
emulation_enabled,
)
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import (
NATIVE_METHOD,
ShapeMethod,
ShapeRetrier,
)
from devplacepy.services.openai_gateway.thinking import (
apply_thinking,
client_thinking_enabled,
upstream_capabilities,
)
@@ -60,6 +66,15 @@ from devplacepy.services.openai_gateway.vision import VisionAugmenter, VisionCac
logger = logging.getLogger(__name__)
async def _close_quietly(resp) -> None:
if resp is None:
return
try:
await resp.aclose()
except Exception: # noqa: BLE001 - discarding an already-failed response
pass
def _call_failed(resp, exc, timing: dict) -> bool:
if timing.get("circuit_open") or exc is not None:
return True
@@ -163,6 +178,7 @@ def _chat_upstream_body(
log,
catalog_cache=None,
owner: tuple = ("anonymous", "anonymous"),
method: ShapeMethod = NATIVE_METHOD,
) -> dict:
# The handler reasons about a Chat payload throughout (routing, vision,
# precheck, fallback); only the bytes on the wire change per protocol.
@@ -182,13 +198,13 @@ def _chat_upstream_body(
)
log(f"Meta reserved tool names aliased: {renamed}")
if protocol != "responses":
return payload
return method.strip_wire(payload, protocol)
send_body, warnings = chat_to_responses(payload)
if warnings:
shown = "; ".join(warnings[:3])
more = f" (+{len(warnings) - 3} more)" if len(warnings) > 3 else ""
log(f"chat->responses transpile warnings: {shown}{more}")
if "reasoning" not in send_body:
if "reasoning" not in send_body and not method.suppresses_reasoning(protocol):
explicit = client_thinking_enabled(client_body)
enabled = (
explicit
@@ -197,7 +213,7 @@ def _chat_upstream_body(
)
if not enabled:
send_body["reasoning"] = {"effort": "minimal"}
return send_body
return method.strip_wire(send_body, protocol)
def _response_tool_aliases(payload: dict, protocol: str, cfg: dict) -> dict:
@@ -350,6 +366,20 @@ def _connect_tracer(holder: dict):
return trace
@dataclass
class ChatAttempt:
resp: Any
exc: Any
timing: dict
shaped: dict
send_start: float
method: ShapeMethod
@property
def resolved_label(self) -> Optional[str]:
return None if self.method.is_native else self.method.label
class GatewayRuntime:
def __init__(self):
self._client: Optional[httpx.AsyncClient] = None
@@ -515,6 +545,102 @@ class GatewayRuntime:
timing["retry_succeeded"] = attempts > 1
return resp, None, timing
async def _shaped_send(
self,
client,
sem,
pristine: dict,
body: dict,
cfg: dict,
protocol: str,
model: str,
headers: dict,
stream: bool,
owner: tuple,
log,
method: ShapeMethod,
) -> ChatAttempt:
shaped = method.apply(pristine, cfg, protocol, stream, log)
shaped["model"] = model
send_body = _chat_upstream_body(
shaped, body, cfg, protocol, log, self._catalog_cache, owner, method
)
send_start = time.monotonic()
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_upstream_url"],
headers,
cfg,
log,
json_body=send_body,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
# The streamed response body hasn't been read yet; the shared error
# handling needs resp.text/.content, which requires an explicit
# read for a stream=True response (a no-op if already buffered).
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
# error body must never crash the request; fall through with
# whatever text/content httpx managed to buffer (possibly empty).
log(f"failed to read non-200 stream error body: {e}")
return ChatAttempt(resp, exc, timing, shaped, send_start, method)
async def _chat_attempt(
self,
client,
sem,
pristine: dict,
body: dict,
cfg: dict,
protocol: str,
model: str,
headers: dict,
stream: bool,
owner: tuple,
log,
) -> ChatAttempt:
native_dialect = upstream_capabilities(
cfg.get("gateway_upstream_url", ""),
cfg.get("gateway_thinking_dialect", "auto"),
).dialect
retrier = ShapeRetrier.for_route(
learned_store,
cfg.get("gateway_provider_name") or "default",
model,
protocol,
native_dialect,
log,
)
first = await self._shaped_send(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log, retrier.method,
)
current = first
while current.resp is not None and retrier.eligible(
current.resp.status_code
):
method = retrier.next_after(current.resp.status_code, current.resp.text)
if method is None:
break
if current is not first:
await _close_quietly(current.resp)
current = await self._shaped_send(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log, method,
)
if current is first:
return first
if current.resp is not None and current.resp.status_code < 400:
retrier.succeeded()
return current
retrier.exhausted()
await _close_quietly(current.resp)
return first
async def handle_chat(
self,
body: dict,
@@ -620,46 +746,10 @@ class GatewayRuntime:
stream = bool(body.get("stream"))
include_usage = bool((body.get("stream_options") or {}).get("include_usage"))
payload = dict(body)
payload["model"] = model
payload["messages"] = messages
payload["stream"] = stream
if stream and protocol == "responses":
# Responses streams carry usage on response.completed natively;
# there is no include_usage to request (stream_options only
# supports include_obfuscation there). The translator captures it
# and the client's own include_usage still controls the relay.
payload.pop("stream_options", None)
if stream and protocol != "responses":
capabilities = upstream_capabilities(
cfg.get("gateway_upstream_url", ""),
cfg.get("gateway_thinking_dialect", "auto"),
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
)
if capabilities.supports_stream_options:
stream_options = dict(body.get("stream_options") or {})
stream_options["include_usage"] = True
payload["stream_options"] = stream_options
else:
payload.pop("stream_options", None)
if include_usage:
log(
"stream_options.include_usage requested but the upstream "
f"({capabilities.dialect}) does not support it; relaying "
"without a usage chunk"
)
else:
payload.pop("stream_options", None)
if protocol != "responses":
apply_thinking(
payload,
cfg.get("gateway_upstream_url", ""),
default_enabled=bool(cfg.get("gateway_thinking", False)),
dialect=cfg.get("gateway_thinking_dialect", "auto"),
)
# For responses upstreams the reasoning effort is set at transpile
# time instead (see _chat_upstream_body): chat-dialect thinking
# fields have no meaning on that wire.
pristine = dict(body)
pristine["model"] = model
pristine["messages"] = messages
pristine["stream"] = stream
headers = {
"Content-Type": "application/json",
@@ -678,7 +768,7 @@ class GatewayRuntime:
)
context_window = _route_context_window(cfg, model)
if context_window:
estimated_tokens = _estimate_request_tokens(payload)
estimated_tokens = _estimate_request_tokens(pristine)
if estimated_tokens > context_window:
for precheck_route in resolve_fallback_chain(chain_start, "chat"):
precheck_overlay = chat_overlay(
@@ -704,39 +794,19 @@ class GatewayRuntime:
or "chat_completions"
)
model = cfg["gateway_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
break
send_start = time.monotonic()
try:
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log, self._catalog_cache, owner
attempt = await self._chat_attempt(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log,
)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_upstream_url"],
headers,
cfg,
log,
json_body=send_body,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
# The streamed response body hasn't been read yet; the shared error
# handling below needs resp.text/.content, which requires an explicit
# read for a stream=True response (a no-op if already buffered).
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
# error body must never crash the request; fall through with
# whatever text/content httpx managed to buffer (possibly empty).
log(f"failed to read non-200 stream error body: {e}")
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
payload = attempt.shaped
send_start = attempt.send_start
requested_model = requested
body_error = _embedded_failure_message(resp, stream, protocol)
@@ -761,32 +831,17 @@ class GatewayRuntime:
cfg.get("gateway_upstream_protocol") or "chat_completions"
)
model = cfg["gateway_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
send_start = time.monotonic()
try:
send_body = _chat_upstream_body(
payload, body, cfg, protocol, log, self._catalog_cache, owner
attempt = await self._chat_attempt(
client, sem, pristine, body, cfg, protocol, model, headers,
stream, owner, log,
)
except ToolEmulationError as tool_error:
return _tool_emulation_rejection(tool_error)
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_upstream_url"],
headers,
cfg,
log,
json_body=send_body,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001
log(f"failed to read fallback non-200 stream error body: {e}")
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
payload = attempt.shaped
send_start = attempt.send_start
body_error = _embedded_failure_message(resp, stream, protocol)
pricing = pricing_from_cfg(cfg)
@@ -803,6 +858,7 @@ class GatewayRuntime:
"app_reference": app_reference,
"provider": cfg.get("gateway_provider_name") or "default",
"fallback_used_route": fallback_used_route,
"resolved_method": attempt.resolved_label,
**params,
**timing,
}
@@ -0,0 +1,165 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import Optional
from pydantic import BaseModel, Field, field_validator
from devplacepy.database import (
bump_cache_version,
db,
get_table,
sync_local_cache,
)
from devplacepy.services.openai_gateway.shape import ShapeMethod
logger = logging.getLogger(__name__)
TABLE = "gateway_learned_methods"
CACHE_NAME = "gateway_learned"
MAX_LEARNED_ROWS = 1000
DEFAULT_PROVIDER = "default"
_LEARNED_CACHE: dict = {}
class LearnedResetIn(BaseModel):
provider: str = Field(default="", max_length=120)
target_model: str = Field(min_length=1, max_length=200)
@field_validator("provider", "target_model")
@classmethod
def _strip(cls, value: str) -> str:
return value.strip()
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _provider_key(provider: Optional[str]) -> str:
return (provider or DEFAULT_PROVIDER).strip().lower() or DEFAULT_PROVIDER
def ensure_table() -> None:
db.query(
"CREATE TABLE IF NOT EXISTS "
+ TABLE
+ " (id INTEGER PRIMARY KEY, provider TEXT, target_model TEXT, "
"method TEXT, learned_at TEXT)"
)
try:
db.query(
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_learned_route ON "
+ TABLE
+ " (provider, target_model)"
)
db.query(
"CREATE INDEX IF NOT EXISTS idx_gateway_learned_at ON "
+ TABLE
+ " (learned_at)"
)
except Exception as exc:
logger.warning("gateway learned index creation failed: %s", exc)
def _load() -> dict:
sync_local_cache(CACHE_NAME, _LEARNED_CACHE)
if "methods" not in _LEARNED_CACHE:
methods: dict = {}
try:
if TABLE in db.tables:
for row in get_table(TABLE).all():
method = ShapeMethod.from_json(row.get("method"))
if method is None:
continue
methods[
(_provider_key(row.get("provider")), str(row["target_model"]))
] = (method, str(row.get("learned_at") or ""))
except Exception as exc:
logger.warning("gateway learned load failed: %s", exc)
_LEARNED_CACHE["methods"] = methods
return _LEARNED_CACHE["methods"]
def _invalidate() -> None:
bump_cache_version(CACHE_NAME)
_LEARNED_CACHE.clear()
class LearnedMethodStore:
def get(self, provider: Optional[str], model: str) -> Optional[ShapeMethod]:
entry = _load().get((_provider_key(provider), str(model)))
return entry[0] if entry else None
def list(self) -> list[dict]:
rows = [
{
"provider": provider,
"target_model": model,
"method": method.label,
"thinking_dialect": method.thinking_dialect,
"drop_keys": list(method.drop_keys),
"learned_at": learned_at,
}
for (provider, model), (method, learned_at) in _load().items()
]
return sorted(rows, key=lambda r: (r["provider"], r["target_model"]))
def remember(
self, provider: Optional[str], model: str, method: ShapeMethod
) -> None:
if method.is_native or not model:
return
try:
ensure_table()
with db:
db.query(
"INSERT INTO " + TABLE + " (provider, target_model, method, learned_at) "
"VALUES (:provider, :model, :method, :now) "
"ON CONFLICT(provider, target_model) DO UPDATE SET "
"method = excluded.method, learned_at = excluded.learned_at",
provider=_provider_key(provider),
model=str(model),
method=method.to_json(),
now=_now(),
)
db.query(
"DELETE FROM " + TABLE + " WHERE id IN (SELECT id FROM "
+ TABLE
+ " ORDER BY learned_at ASC, id ASC LIMIT "
"MAX((SELECT COUNT(*) FROM " + TABLE + ") - :cap, 0))",
cap=MAX_LEARNED_ROWS,
)
except Exception as exc:
logger.warning("gateway learned remember failed: %s", exc)
return
_invalidate()
def forget(self, provider: Optional[str], model: str) -> bool:
if TABLE not in db.tables:
return False
removed = int(
get_table(TABLE).delete(
provider=_provider_key(provider), target_model=str(model)
)
)
if removed:
_invalidate()
return bool(removed)
def forget_provider(self, provider: str) -> int:
if TABLE not in db.tables:
return 0
table = get_table(TABLE)
removed = table.count(provider=_provider_key(provider))
if removed:
table.delete(provider=_provider_key(provider))
_invalidate()
return removed
learned_store = LearnedMethodStore()
@@ -16,6 +16,7 @@ from devplacepy.database import (
sync_local_cache,
)
from devplacepy.services.openai_gateway import muse_code, opencode_zen
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.tool_emulation import (
TOOLS_MODE_NATIVE,
TOOLS_MODES,
@@ -531,11 +532,17 @@ class ProviderStore:
"protocol": payload.protocol,
"updated_at": _now(),
}
endpoint_changed = bool(existing) and (
str(existing.get("base_url") or "") != payload.base_url
or str(existing.get("client_profile") or "") != payload.client_profile
)
if existing:
table.update({**record, "id": existing["id"]}, ["id"])
else:
record["created_at"] = _now()
table.insert(record)
if endpoint_changed:
learned_store.forget_provider(payload.name)
bump_cache_version(CACHE_NAME)
_ROUTING_CACHE.clear()
return self.get(payload.name) or record
@@ -546,6 +553,7 @@ class ProviderStore:
return False
removed = int(get_table(PROVIDERS_TABLE).delete(name=name))
if removed:
learned_store.forget_provider(name)
bump_cache_version(CACHE_NAME)
_ROUTING_CACHE.clear()
return bool(removed)
+388
View File
@@ -0,0 +1,388 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import json
import logging
import re
from dataclasses import dataclass, field
from typing import Any, Callable, Optional
from devplacepy.services.openai_gateway.thinking import (
CLIENT_THINKING_KEYS,
apply_thinking,
upstream_capabilities,
)
logger = logging.getLogger(__name__)
PROTOCOL_RESPONSES = "responses"
DIALECT_NATIVE = "native"
DIALECT_NONE = "none"
DIALECT_DEEPSEEK = "deepseek"
DIALECT_OPENROUTER = "openrouter"
DIALECT_OLLAMA = "ollama"
CHAT_LADDER = (
DIALECT_NATIVE,
DIALECT_NONE,
DIALECT_DEEPSEEK,
DIALECT_OPENROUTER,
DIALECT_OLLAMA,
)
RESPONSES_LADDER = (DIALECT_NATIVE, DIALECT_NONE)
DIALECTS = frozenset(CHAT_LADDER)
DROPPABLE_KEYS = frozenset(
{
"stream_options",
"parallel_tool_calls",
"user",
"seed",
"logprobs",
"top_logprobs",
"n",
"presence_penalty",
"frequency_penalty",
}
)
THINKING_KEYS = frozenset(CLIENT_THINKING_KEYS) | {"chat_template_kwargs"}
DIALECT_EMITS: dict = {
DIALECT_DEEPSEEK: frozenset({"thinking"}),
DIALECT_OPENROUTER: frozenset({"reasoning"}),
DIALECT_OLLAMA: frozenset({"think", "reasoning_effort"}),
DIALECT_NONE: frozenset(),
}
RESPONSES_EMITS = frozenset({"reasoning"})
MAX_RESOLVE_ATTEMPTS = 4
SHAPE_STATUSES = (400, 422)
MAX_REJECTED_KEYS = 16
MAX_KEY_LENGTH = 64
_KEY_PATTERNS = (
re.compile(r"Unrecognized key\(s\) in object:\s*((?:'[^']+'(?:,\s*)?)+)", re.I),
re.compile(r"Unrecognized request arguments? supplied:\s*([\w.,\s]+)", re.I),
re.compile(r"Unknown (?:parameter|field|argument)s?:?\s*[`'\"]?([\w.]+)", re.I),
re.compile(r"unknown field\s*[`'\"]([\w.]+)[`'\"]", re.I),
re.compile(r"Extra inputs are not permitted.{0,40}?['\"`]([\w.]+)['\"`]", re.I),
)
_QUOTED = re.compile(r"'([^']+)'")
_BARE_SPLIT = re.compile(r"[,\s]+")
@dataclass(frozen=True)
class ShapeMethod:
thinking_dialect: str = DIALECT_NATIVE
drop_keys: tuple = ()
@property
def is_native(self) -> bool:
return self.thinking_dialect == DIALECT_NATIVE and not self.drop_keys
@property
def label(self) -> str:
return self.thinking_dialect + "".join(f" -{key}" for key in self.drop_keys)
def dialect_for(self, cfg: dict) -> str:
if self.thinking_dialect == DIALECT_NATIVE:
return str(cfg.get("gateway_thinking_dialect", "auto") or "auto")
return self.thinking_dialect
def to_json(self) -> str:
return json.dumps(
{
"thinking_dialect": self.thinking_dialect,
"drop_keys": list(self.drop_keys),
}
)
@classmethod
def from_json(cls, text: Any) -> Optional["ShapeMethod"]:
try:
data = json.loads(text)
except (TypeError, ValueError):
return None
if not isinstance(data, dict):
return None
dialect = data.get("thinking_dialect")
drops = data.get("drop_keys")
if dialect not in DIALECTS or not isinstance(drops, list):
return None
if any(key not in DROPPABLE_KEYS for key in drops):
return None
return cls(dialect, tuple(sorted(set(drops))))
def apply(
self,
payload: dict,
cfg: dict,
protocol: str,
stream: bool,
log: Callable[[str], None],
) -> dict:
shaped = dict(payload)
kwargs = shaped.get("chat_template_kwargs")
if isinstance(kwargs, dict):
shaped["chat_template_kwargs"] = dict(kwargs)
if protocol == PROTOCOL_RESPONSES:
shaped.pop("stream_options", None)
else:
self._apply_stream_options(shaped, payload, cfg, stream, log)
apply_thinking(
shaped,
cfg.get("gateway_upstream_url", ""),
default_enabled=bool(cfg.get("gateway_thinking", False)),
dialect=self.dialect_for(cfg),
)
for key in self.drop_keys:
shaped.pop(key, None)
return shaped
def _apply_stream_options(
self,
shaped: dict,
payload: dict,
cfg: dict,
stream: bool,
log: Callable[[str], None],
) -> None:
include_usage = bool((payload.get("stream_options") or {}).get("include_usage"))
capabilities = upstream_capabilities(
cfg.get("gateway_upstream_url", ""),
cfg.get("gateway_thinking_dialect", "auto"),
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
)
if (
stream
and capabilities.supports_stream_options
and "stream_options" not in self.drop_keys
):
stream_options = dict(payload.get("stream_options") or {})
stream_options["include_usage"] = True
shaped["stream_options"] = stream_options
return
shaped.pop("stream_options", None)
if stream and include_usage:
log(
"stream_options.include_usage requested but the upstream "
f"({capabilities.dialect}) does not support it; relaying "
"without a usage chunk"
)
def suppresses_reasoning(self, protocol: str) -> bool:
return protocol == PROTOCOL_RESPONSES and self.thinking_dialect == DIALECT_NONE
def strip_wire(self, body: dict, protocol: str) -> dict:
doomed = set(self.drop_keys)
if self.suppresses_reasoning(protocol):
doomed.add("reasoning")
if not doomed.intersection(body):
return body
return {key: value for key, value in body.items() if key not in doomed}
NATIVE_METHOD = ShapeMethod()
class RejectedKeys:
@staticmethod
def parse(body_text: Any) -> tuple:
text = body_text if isinstance(body_text, str) else ""
if not text:
return ()
found: list = []
RejectedKeys._from_json(text, found)
RejectedKeys._from_patterns(text, found)
cleaned: list = []
for key in found:
top = str(key).strip().strip("`'\"").split(".")[0].split("[")[0]
if top and len(top) <= MAX_KEY_LENGTH and top not in cleaned:
cleaned.append(top)
return tuple(cleaned[:MAX_REJECTED_KEYS])
@staticmethod
def _from_json(text: str, found: list) -> None:
try:
data = json.loads(text)
except ValueError:
return
if not isinstance(data, dict):
return
issues = data.get("issues")
if isinstance(issues, list):
for issue in issues:
if isinstance(issue, dict) and issue.get("code") == "unrecognized_keys":
keys = issue.get("keys")
if isinstance(keys, list):
found.extend(str(key) for key in keys)
error = data.get("error")
if isinstance(error, dict):
param = error.get("param")
code = str(error.get("code") or "")
if isinstance(param, str) and param and code in (
"unknown_parameter",
"unrecognized_request_argument",
):
found.append(param)
@staticmethod
def _from_patterns(text: str, found: list) -> None:
for pattern in _KEY_PATTERNS:
for match in pattern.finditer(text):
chunk = match.group(1)
quoted = _QUOTED.findall(chunk)
if quoted:
found.extend(quoted)
else:
found.extend(
part for part in _BARE_SPLIT.split(chunk.strip()) if part
)
class ShapeResolver:
def __init__(self, native_dialect: str, protocol: str) -> None:
self.native_dialect = native_dialect
self.responses = protocol == PROTOCOL_RESPONSES
def emits(self, dialect: str) -> frozenset:
if self.responses:
return frozenset() if dialect == DIALECT_NONE else RESPONSES_EMITS
if dialect == DIALECT_NATIVE:
return DIALECT_EMITS.get(self.native_dialect, frozenset())
return DIALECT_EMITS.get(dialect, frozenset())
def next_method(
self, tried: set, current: ShapeMethod, rejected: tuple
) -> Optional[ShapeMethod]:
keys = set(rejected)
drops = set(current.drop_keys) | (keys & DROPPABLE_KEYS)
if self.responses:
thinking_hit = keys & RESPONSES_EMITS
else:
thinking_hit = keys & THINKING_KEYS
if not thinking_hit and drops == set(current.drop_keys):
return None
sorted_drops = tuple(sorted(drops))
if not thinking_hit:
candidate = ShapeMethod(current.thinking_dialect, sorted_drops)
return None if candidate in tried else candidate
ladder = RESPONSES_LADDER if self.responses else CHAT_LADDER
for dialect in ladder:
if self.emits(dialect) & keys:
continue
candidate = ShapeMethod(dialect, sorted_drops)
if candidate not in tried:
return candidate
return None
@dataclass
class ShapeStats:
resolved: int = 0
exhausted: int = 0
healed: int = 0
def snapshot(self) -> dict:
return {
"resolved": self.resolved,
"exhausted": self.exhausted,
"healed": self.healed,
}
shape_stats = ShapeStats()
@dataclass
class ShapeRetrier:
store: Any
provider: str
model: str
resolver: ShapeResolver
log: Callable[[str], None]
method: ShapeMethod = field(default=NATIVE_METHOD)
from_store: bool = False
healed: bool = False
attempts: int = 0
tried: set = field(default_factory=set)
@classmethod
def for_route(
cls,
store: Any,
provider: str,
model: str,
protocol: str,
native_dialect: str,
log: Callable[[str], None],
) -> "ShapeRetrier":
learned = store.get(provider, model)
retrier = cls(
store=store,
provider=provider,
model=model,
resolver=ShapeResolver(native_dialect, protocol),
log=log,
method=learned or NATIVE_METHOD,
from_store=learned is not None,
)
retrier.tried.add(retrier.method)
return retrier
@staticmethod
def eligible(status: int) -> bool:
return status in SHAPE_STATUSES
def next_after(self, status: int, body_text: Any) -> Optional[ShapeMethod]:
if not self.eligible(status) or self.attempts >= MAX_RESOLVE_ATTEMPTS:
return None
rejected = RejectedKeys.parse(body_text)
if not rejected:
return None
if self.from_store and not self.healed:
self.store.forget(self.provider, self.model)
self.healed = True
self.from_store = False
shape_stats.healed += 1
self.log(
f"learned shape '{self.method.label}' for {self.provider}/"
f"{self.model} was rejected ({', '.join(rejected)}); "
"forgotten, restarting the ladder"
)
self.tried = {self.method}
if NATIVE_METHOD not in self.tried:
return self._advance(NATIVE_METHOD)
candidate = self.resolver.next_method(self.tried, self.method, rejected)
if candidate is None:
return None
return self._advance(candidate)
def _advance(self, candidate: ShapeMethod) -> ShapeMethod:
self.attempts += 1
self.tried.add(candidate)
self.method = candidate
self.log(
f"upstream rejected the request shape for {self.provider}/"
f"{self.model}, trying '{candidate.label}'"
)
return candidate
def succeeded(self) -> None:
if self.attempts == 0 or self.method.is_native:
return
self.store.remember(self.provider, self.model, self.method)
shape_stats.resolved += 1
self.log(
f"auto-resolved {self.provider}/{self.model}: {self.method.label}"
)
def exhausted(self) -> None:
if self.attempts == 0:
return
shape_stats.exhausted += 1
self.log(
f"could not auto-resolve the request shape for {self.provider}/"
f"{self.model} after {self.attempts} attempt(s)"
)
@property
def resolved_label(self) -> Optional[str]:
return None if self.method.is_native else self.method.label
@@ -144,6 +144,8 @@ def _clear_thinking_fields(payload: dict) -> None:
def write_thinking(payload: dict, dialect: str, enabled: bool) -> None:
_clear_thinking_fields(payload)
if dialect == "none":
return
if dialect == "openrouter":
payload["reasoning"] = {"effort": "high" if enabled else "none"}
return
@@ -587,6 +587,11 @@ class GatewayUsageLedger:
if raw.get("fallback_used_route")
else None
),
"resolved_method": (
str(raw["resolved_method"])[:128]
if raw.get("resolved_method")
else None
),
"status_code": int(raw.get("status_code") or 0),
"success": 1 if raw.get("success") else 0,
"error_category": raw.get("error_category"),
+69 -32
View File
@@ -6,6 +6,7 @@ import logging
import time
from collections import OrderedDict
from typing import Any, Optional
from urllib.parse import urlparse
import httpx
@@ -16,7 +17,9 @@ from devplacepy.services.openai_gateway.reliability import (
_is_retryable_status,
_retry_after_seconds,
)
from devplacepy.services.openai_gateway.thinking import apply_thinking
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import ShapeRetrier
from devplacepy.services.openai_gateway.thinking import upstream_capabilities
from devplacepy.services.openai_gateway.usage import classify_error
logger = logging.getLogger(__name__)
@@ -151,37 +154,9 @@ class VisionAugmenter:
if row:
self.cost_usd += float(row.get("cost_usd") or 0.0)
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
if not self.vision_key:
return "[vision unavailable: vision API key not configured]"
if self.breaker is not None and not self.breaker.allow():
self._record((0.0), 503, False, "circuit_open", None)
return "[vision unavailable: circuit breaker open]"
payload = {
"model": self.vision_model,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": VISION_INSTRUCTION},
image_block,
],
}
],
"temperature": 0.2,
"stream": False,
}
apply_thinking(payload, self.vision_url, default_enabled=False, dialect=self.vision_dialect)
headers = {
"Authorization": f"Bearer {self.vision_key}",
"Content-Type": "application/json",
}
if self.referer:
headers["HTTP-Referer"] = self.referer
if self.title:
headers[TITLE_HEADER] = self.title
headers[LEGACY_TITLE_HEADER] = self.title
start = time.monotonic()
async def _post_with_retries(
self, client: httpx.AsyncClient, payload: dict, headers: dict
) -> tuple:
attempt = 0
resp: Optional[httpx.Response] = None
exc: Optional[Exception] = None
@@ -210,6 +185,68 @@ class VisionAugmenter:
except Exception: # noqa: BLE001
pass
await _backoff(self.retry_backoff_ms, attempt, retry_after)
return resp, exc
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
if not self.vision_key:
return "[vision unavailable: vision API key not configured]"
if self.breaker is not None and not self.breaker.allow():
self._record((0.0), 503, False, "circuit_open", None)
return "[vision unavailable: circuit breaker open]"
payload = {
"model": self.vision_model,
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": VISION_INSTRUCTION},
image_block,
],
}
],
"temperature": 0.2,
"stream": False,
}
shape_cfg = {
"gateway_upstream_url": self.vision_url,
"gateway_thinking_dialect": self.vision_dialect or "auto",
"gateway_thinking": False,
}
retrier = ShapeRetrier.for_route(
learned_store,
"vision:" + (urlparse(self.vision_url).hostname or "default"),
self.vision_model,
"chat_completions",
upstream_capabilities(self.vision_url, self.vision_dialect).dialect,
logger.info,
)
headers = {
"Authorization": f"Bearer {self.vision_key}",
"Content-Type": "application/json",
}
if self.referer:
headers["HTTP-Referer"] = self.referer
if self.title:
headers[TITLE_HEADER] = self.title
headers[LEGACY_TITLE_HEADER] = self.title
start = time.monotonic()
shaped = retrier.method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
resp, exc = await self._post_with_retries(client, shaped, headers)
first = (resp, exc)
resolving = False
while resp is not None and retrier.eligible(resp.status_code):
method = retrier.next_after(resp.status_code, resp.text)
if method is None:
break
resolving = True
shaped = method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
resp, exc = await self._post_with_retries(client, shaped, headers)
if resolving:
if resp is not None and resp.status_code < 400:
retrier.succeeded()
else:
retrier.exhausted()
resp, exc = first
if self.breaker is not None:
if exc is not None or (resp is not None and resp.status_code >= 500):
self.breaker.record_failure()
+9 -1
View File
@@ -56,7 +56,15 @@
{% if m.kind == 'image' and m.price_input_per_m %}<span class="gw-badge" title="Flat price per generated image">${{ m.price_input_per_m }}/img</span>{% endif %}
{% if m.fallback_model %}<span class="gw-badge" title="Falls back to {{ m.fallback_model }} on failure">fallback: {{ m.fallback_model }}</span>{% endif %}
{% if m.tools_mode == 'emulated' %}<span class="gw-badge" title="This model does not support tool calling; tools are emulated through the prompt">tools: emulated</span>{% endif %}
{% if not m.context_tier_threshold_tokens and m.off_peak_start_minute is none and not (m.kind == 'image' and m.price_input_per_m) and not m.fallback_model and m.tools_mode != 'emulated' %}<span class="gw-muted">-</span>{% endif %}
{% if m.learned_shape %}
<span class="gw-badge" title="Request shape learned automatically after the upstream rejected the default">shape: {{ m.learned_shape }}</span>
<form method="post" action="/admin/gateway/learned/reset-page" class="gw-inline-form">
<input type="hidden" name="provider" value="{{ m.learned_provider }}">
<input type="hidden" name="target_model" value="{{ m.target_model }}">
<button type="submit" class="admin-btn admin-btn-sm" data-confirm="Forget the learned request shape for &quot;{{ m.target_model }}&quot;?">Reset learned</button>
</form>
{% endif %}
{% if not m.context_tier_threshold_tokens and m.off_peak_start_minute is none and not (m.kind == 'image' and m.price_input_per_m) and not m.fallback_model and m.tools_mode != 'emulated' and not m.learned_shape %}<span class="gw-muted">-</span>{% endif %}
</td>
<td>
{% if m.health_status == 'bad' %}
+1
View File
@@ -119,6 +119,7 @@ Every state-changing action in DevPlace records one append-only row through `dev
| `ai.gateway.call` | `services/openai_gateway/gateway.py`, `services/openai_gateway/usage.py` |
| `ai.gateway_auth_throttle.tripped` | `services/openai_gateway/service.py` |
| `ai.quota.exceeded` | `routers/devii.py`, `services/openai_gateway/service.py` |
| `gateway.learned.reset` | `routers/admin/gateway_configs.py` |
| `gateway.model.delete` | `routers/admin/gateway_configs.py` |
| `gateway.model.update` | `routers/admin/gateway_configs.py` |
| `gateway.provider.delete` | `routers/admin/gateway_configs.py` |
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "devplacepy"
version = "1.1.22"
version = "1.1.23"
description = "DevPlace - The Developer Social Network"
requires-python = ">=3.12"
dependencies = [
+77
View File
@@ -0,0 +1,77 @@
# retoor <retoor@molodetz.nl>
import time
import requests
from devplacepy.database import refresh_snapshot
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import ShapeMethod
from tests.api.admin.gateway.index import (
JSON_gateway,
admin_session,
member_key,
_unique_gateway,
)
from tests.conftest import BASE_URL
LIST_URL = f"{BASE_URL}/admin/gateway/learned"
RESET_URL = f"{BASE_URL}/admin/gateway/learned/reset"
SYNC_SECONDS = 1.5
def _seed(provider, model, method):
learned_store.remember(provider, model, method)
time.sleep(SYNC_SECONDS)
def test_learned_endpoints_require_admin(seeded_db):
assert requests.get(LIST_URL, headers=JSON_gateway, allow_redirects=False).status_code == 401
assert (
requests.post(RESET_URL, json={}, headers=JSON_gateway, allow_redirects=False).status_code
== 401
)
key = member_key()
member = {**JSON_gateway, "X-API-KEY": key}
assert requests.get(LIST_URL, headers=member, allow_redirects=False).status_code == 403
assert (
requests.post(RESET_URL, json={}, headers=member, allow_redirects=False).status_code == 403
)
def test_admin_lists_a_learned_method(seeded_db):
provider = _unique_gateway("lprov")
_seed(provider, "vendor/learned", ShapeMethod("none", ("stream_options",)))
try:
payload = admin_session(seeded_db).get(LIST_URL).json()
row = next(r for r in payload["learned"] if r["provider"] == provider)
assert row["target_model"] == "vendor/learned"
assert row["thinking_dialect"] == "none"
assert row["drop_keys"] == ["stream_options"]
assert set(payload["stats"]) == {"resolved", "exhausted", "healed"}
assert payload["count"] >= 1
finally:
learned_store.forget(provider, "vendor/learned")
def test_admin_resets_a_learned_method_and_it_is_audited(seeded_db):
provider = _unique_gateway("lprov")
_seed(provider, "vendor/reset", ShapeMethod("none"))
admin = admin_session(seeded_db)
response = admin.post(RESET_URL, json={"provider": provider, "target_model": "vendor/reset"})
assert response.status_code == 200
assert response.json()["ok"] is True
refresh_snapshot()
entries = admin.get(
f"{BASE_URL}/admin/audit-log",
headers=JSON_gateway,
params={"event_key": "gateway.learned.reset"},
).json()["entries"]
assert any(e.get("target_type") == "gateway_learned_method" for e in entries)
again = admin.post(RESET_URL, json={"provider": provider, "target_model": "vendor/reset"})
assert again.status_code == 404
def test_reset_validates_the_target_model(seeded_db):
response = admin_session(seeded_db).post(RESET_URL, json={"provider": "x"})
assert response.status_code in (400, 422)
+47
View File
@@ -299,3 +299,50 @@ def test_image_test_opens_result_modal(page, seeded_db):
httpd.shutdown()
requests.delete(f"{BASE_URL}/admin/gateway/models/{source}", headers=auth)
requests.delete(f"{BASE_URL}/admin/gateway/providers/{provider}", headers=auth)
def test_model_grid_shows_a_learned_shape_badge_and_reset(page, seeded_db):
import time
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import ShapeMethod
user = seeded_db["alice"]
_promote_to_admin(user["username"])
key = _admin_api_key(user["username"])
auth = {"X-API-KEY": key}
provider = f"e2elearn{int(time.time() * 1000)}"
source = f"{provider}-src"
try:
requests.post(
f"{BASE_URL}/admin/gateway/providers",
json={"name": provider, "base_url": "http://127.0.0.1:9/v1/chat/completions"},
headers=auth,
)
requests.post(
f"{BASE_URL}/admin/gateway/models",
json={
"source_model": source,
"provider": provider,
"target_model": "vendor/learned",
"kind": "chat",
},
headers=auth,
)
learned_store.remember(provider, "vendor/learned", ShapeMethod("none"))
time.sleep(1.5)
login_user(page, user)
page.goto(f"{BASE_URL}/admin/gateway?tab=models", wait_until="domcontentloaded")
row = page.locator("table.admin-table tr", has_text=source)
row.wait_for(state="visible")
assert "shape: none" in (row.inner_text() or "")
row.get_by_role("button", name="Reset learned", exact=True).click()
page.locator(".dialog-overlay.visible .dialog-confirm").click()
page.wait_for_url(f"{BASE_URL}/admin/gateway?tab=models", wait_until="domcontentloaded")
row = page.locator("table.admin-table tr", has_text=source)
row.wait_for(state="visible")
assert "shape: none" not in (row.inner_text() or "")
finally:
learned_store.forget(provider, "vendor/learned")
requests.delete(f"{BASE_URL}/admin/gateway/models/{source}", headers=auth)
requests.delete(f"{BASE_URL}/admin/gateway/providers/{provider}", headers=auth)
@@ -3441,3 +3441,240 @@ def test_native_route_keeps_forwarding_tools(local_db, monkeypatch):
assert "<tools>" not in sent["instructions"]
finally:
_teardown_responses_route_openai_gateway()
VAIDS_REJECTION_openai_gateway = {
"details": {"_errors": ["Unrecognized key(s) in object: 'thinking'"]},
"error": "Invalid request parameters",
"issues": [
{
"code": "unrecognized_keys",
"keys": ["thinking"],
"path": [],
"message": "Unrecognized key(s) in object: 'thinking'",
}
],
}
def _rejection_openai_gateway(key):
return {
"details": {"_errors": [f"Unrecognized key(s) in object: '{key}'"]},
"error": "Invalid request parameters",
"issues": [{"code": "unrecognized_keys", "keys": [key], "path": []}],
}
class FakeStrictClient_openai_gateway:
rejected = ("thinking",)
def __init__(self, *a, **k):
self.calls = []
def build_request(self, method, url, headers=None, json=None, content=None):
return FakeRequest(method, url, json)
async def send(self, request, stream=False):
body = request.json_body or {}
self.calls.append((request.url, dict(body)))
for key in self.rejected:
if key in body:
return FakeResp_openai_gateway(
status=400, payload=_rejection_openai_gateway(key)
)
return FakeResp_openai_gateway(
payload={
"id": "x",
"model": body.get("model"),
"choices": [{"message": {"content": "hi there"}}],
}
)
async def aclose(self):
pass
class FakeStubbornClient_openai_gateway(FakeStrictClient_openai_gateway):
async def send(self, request, stream=False):
self.calls.append((request.url, dict(request.json_body or {})))
return FakeResp_openai_gateway(
status=400, payload=_rejection_openai_gateway("thinking")
)
class FakeReasoningStrictClient_openai_gateway(FakeStrictClient_openai_gateway):
rejected = ("reasoning",)
def _shape_cfg_openai_gateway():
svc = GatewayService()
cfg = svc.effective_config()
cfg["gateway_vision_enabled"] = False
cfg["gateway_max_retries"] = 0
return svc, cfg
def _chat_openai_gateway(rt, cfg, owner, stream=False):
body = {"messages": [{"role": "user", "content": "hi"}]}
if stream:
body["stream"] = True
return run_async(rt.handle_chat(body, cfg, ("guest", owner), "test", "default"))
def test_chat_auto_resolves_a_rejected_thinking_key_and_remembers_it(local_db, monkeypatch):
from devplacepy.services.openai_gateway.learned import learned_store
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeStrictClient_openai_gateway)
svc, cfg = _shape_cfg_openai_gateway()
model = cfg["gateway_model"]
learned_store.forget("default", model)
try:
rt = svc.runtime()
response = _chat_openai_gateway(rt, cfg, "shape_resolve_first")
assert response.status_code == 200
assert len(rt._client.calls) == 2
assert "thinking" in rt._client.calls[0][1]
assert "thinking" not in rt._client.calls[1][1]
learned = learned_store.get("default", model)
assert learned is not None and learned.thinking_dialect == "none"
row = get_table("gateway_usage_ledger").find_one(owner_id="shape_resolve_first")
assert row["resolved_method"] == "none"
assert row["success"] == 1
second = _chat_openai_gateway(rt, cfg, "shape_resolve_second")
assert second.status_code == 200
assert len(rt._client.calls) == 3
assert "thinking" not in rt._client.calls[2][1]
finally:
learned_store.forget("default", model)
def test_chat_auto_resolve_works_for_streams(local_db, monkeypatch):
from devplacepy.services.openai_gateway.learned import learned_store
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeStrictClient_openai_gateway)
svc, cfg = _shape_cfg_openai_gateway()
model = cfg["gateway_model"]
learned_store.forget("default", model)
try:
rt = svc.runtime()
response = _chat_openai_gateway(rt, cfg, "shape_resolve_stream", stream=True)
async def drain():
out = []
async for chunk in response.body_iterator:
out.append(chunk if isinstance(chunk, str) else chunk.decode())
return "".join(out)
assert "[DONE]" in run_async(drain())
assert len(rt._client.calls) == 2
assert learned_store.get("default", model) is not None
finally:
learned_store.forget("default", model)
def test_chat_genuine_bad_request_is_not_auto_resolved(local_db, monkeypatch):
from devplacepy.services.openai_gateway.learned import learned_store
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeBadRequestClient_openai_gateway)
svc, cfg = _shape_cfg_openai_gateway()
model = cfg["gateway_model"]
learned_store.forget("default", model)
rt = svc.runtime()
response = _chat_openai_gateway(rt, cfg, "shape_genuine_bad")
assert response.status_code == 400
assert len(rt._client.calls) == 1
assert learned_store.get("default", model) is None
def test_chat_exhausted_resolution_returns_the_original_error_and_stores_nothing(
local_db, monkeypatch
):
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import MAX_RESOLVE_ATTEMPTS
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeStubbornClient_openai_gateway)
svc, cfg = _shape_cfg_openai_gateway()
model = cfg["gateway_model"]
learned_store.forget("default", model)
rt = svc.runtime()
response = _chat_openai_gateway(rt, cfg, "shape_exhausted")
assert response.status_code == 400
assert "thinking" in response.body.decode()
assert len(rt._client.calls) <= 1 + MAX_RESOLVE_ATTEMPTS
assert learned_store.get("default", model) is None
def test_chat_rejected_learned_method_is_forgotten_and_ladder_restarts(local_db, monkeypatch):
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import ShapeMethod
monkeypatch.setattr(
gwmod.httpx, "AsyncClient", FakeReasoningStrictClient_openai_gateway
)
svc, cfg = _shape_cfg_openai_gateway()
model = cfg["gateway_model"]
learned_store.remember("default", model, ShapeMethod("openrouter"))
try:
rt = svc.runtime()
response = _chat_openai_gateway(rt, cfg, "shape_heal")
assert response.status_code == 200
assert "reasoning" in rt._client.calls[0][1]
assert "thinking" in rt._client.calls[1][1]
assert learned_store.get("default", model) is None
finally:
learned_store.forget("default", model)
def test_fallback_attempt_is_shaped_with_its_own_learned_method(local_db, monkeypatch):
from devplacepy.services.openai_gateway import routing
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import ShapeMethod
class FakeFallbackShapeClient(FakeStrictClient_openai_gateway):
rejected = ()
async def send(self, request, stream=False):
body = request.json_body or {}
self.calls.append((request.url, dict(body)))
if body.get("model") == "shape-primary-target":
return FakeResp_openai_gateway(status=503, payload={"error": {"message": "down"}})
return FakeResp_openai_gateway(
payload={
"id": "x",
"model": body.get("model"),
"choices": [{"message": {"content": "ok"}}],
}
)
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeFallbackShapeClient)
routing.model_store.set(
routing.ModelRouteIn(source_model="shape-backup", target_model="shape-backup-target")
)
routing.model_store.set(
routing.ModelRouteIn(
source_model="shape-primary",
target_model="shape-primary-target",
fallback_model="shape-backup",
)
)
learned_store.remember("default", "shape-backup-target", ShapeMethod("none"))
try:
svc, cfg = _shape_cfg_openai_gateway()
rt = svc.runtime()
response = run_async(
rt.handle_chat(
{"model": "shape-primary", "messages": [{"role": "user", "content": "hi"}]},
cfg,
("guest", "shape_fallback"),
"test",
"default",
)
)
assert response.status_code == 200
assert "thinking" in rt._client.calls[0][1]
assert rt._client.calls[-1][1]["model"] == "shape-backup-target"
assert "thinking" not in rt._client.calls[-1][1]
finally:
learned_store.forget("default", "shape-backup-target")
routing.model_store.remove("shape-primary")
routing.model_store.remove("shape-backup")
@@ -0,0 +1,90 @@
# retoor <retoor@molodetz.nl>
from devplacepy.services.openai_gateway import learned
from devplacepy.services.openai_gateway.learned import learned_store
from devplacepy.services.openai_gateway.shape import NATIVE_METHOD, ShapeMethod
def test_remember_get_list_and_forget(local_db):
learned_store.forget("lt-prov", "lt-model")
method = ShapeMethod("none", ("stream_options",))
learned_store.remember("LT-Prov", "lt-model", method)
try:
assert learned_store.get("lt-prov", "lt-model") == method
rows = [r for r in learned_store.list() if r["target_model"] == "lt-model"]
assert rows[0]["provider"] == "lt-prov"
assert rows[0]["drop_keys"] == ["stream_options"]
assert learned_store.forget("lt-prov", "lt-model") is True
assert learned_store.get("lt-prov", "lt-model") is None
assert learned_store.forget("lt-prov", "lt-model") is False
finally:
learned_store.forget("lt-prov", "lt-model")
def test_blank_provider_maps_to_default_and_native_is_never_stored(local_db):
learned_store.forget("default", "lt-default-model")
learned_store.remember("", "lt-default-model", NATIVE_METHOD)
assert learned_store.get("default", "lt-default-model") is None
learned_store.remember("", "lt-default-model", ShapeMethod("none"))
try:
assert learned_store.get("default", "lt-default-model") == ShapeMethod("none")
finally:
learned_store.forget("default", "lt-default-model")
def test_remember_upserts_on_the_unique_route(local_db):
learned_store.forget("lt-up", "lt-m")
try:
learned_store.remember("lt-up", "lt-m", ShapeMethod("none"))
learned_store.remember("lt-up", "lt-m", ShapeMethod("openrouter"))
assert learned_store.get("lt-up", "lt-m") == ShapeMethod("openrouter")
assert len([r for r in learned_store.list() if r["provider"] == "lt-up"]) == 1
finally:
learned_store.forget("lt-up", "lt-m")
def test_forget_provider_clears_only_that_provider(local_db):
learned_store.remember("lt-fp", "a", ShapeMethod("none"))
learned_store.remember("lt-fp", "b", ShapeMethod("none"))
learned_store.remember("lt-other", "a", ShapeMethod("none"))
try:
assert learned_store.forget_provider("lt-fp") == 2
assert learned_store.get("lt-fp", "a") is None
assert learned_store.get("lt-other", "a") is not None
finally:
learned_store.forget_provider("lt-fp")
learned_store.forget_provider("lt-other")
def test_row_cap_replaces_the_oldest(local_db, monkeypatch):
monkeypatch.setattr(learned, "MAX_LEARNED_ROWS", 2)
keys = [("lt-cap", f"m{i}") for i in range(3)]
try:
for provider, model in keys:
learned_store.remember(provider, model, ShapeMethod("none"))
present = [k for k in keys if learned_store.get(*k) is not None]
assert len(present) <= 2
assert learned_store.get(*keys[2]) is not None
finally:
learned_store.forget_provider("lt-cap")
def test_provider_endpoint_edit_forgets_learned_methods(local_db):
from devplacepy.services.openai_gateway import routing
routing.provider_store.set(
routing.ProviderIn(name="lt-edit", base_url="https://a.example/v1/chat/completions")
)
learned_store.remember("lt-edit", "m", ShapeMethod("none"))
try:
routing.provider_store.set(
routing.ProviderIn(name="lt-edit", base_url="https://a.example/v1/chat/completions")
)
assert learned_store.get("lt-edit", "m") is not None
routing.provider_store.set(
routing.ProviderIn(name="lt-edit", base_url="https://b.example/v1/chat/completions")
)
assert learned_store.get("lt-edit", "m") is None
finally:
routing.provider_store.remove("lt-edit")
learned_store.forget_provider("lt-edit")
+174
View File
@@ -0,0 +1,174 @@
# retoor <retoor@molodetz.nl>
import json
from devplacepy.services.openai_gateway.shape import (
MAX_RESOLVE_ATTEMPTS,
NATIVE_METHOD,
RejectedKeys,
ShapeMethod,
ShapeResolver,
ShapeRetrier,
)
VAIDS_BODY = json.dumps(
{
"details": {"_errors": ["Unrecognized key(s) in object: 'thinking'"]},
"error": "Invalid request parameters",
"issues": [
{
"code": "unrecognized_keys",
"keys": ["thinking"],
"path": [],
"message": "Unrecognized key(s) in object: 'thinking'",
}
],
}
)
class MemoryStore_openai_gateway:
def __init__(self):
self.rows = {}
self.forgotten = []
def get(self, provider, model):
return self.rows.get((provider, model))
def remember(self, provider, model, method):
self.rows[(provider, model)] = method
def forget(self, provider, model):
self.forgotten.append((provider, model))
return self.rows.pop((provider, model), None) is not None
def test_parse_real_vaids_body():
assert RejectedKeys.parse(VAIDS_BODY) == ("thinking",)
def test_parse_openai_style_bodies():
assert RejectedKeys.parse(
'{"error":{"message":"Unrecognized request argument supplied: foo","param":null}}'
) == ("foo",)
assert RejectedKeys.parse(
'{"error":{"message":"Unknown parameter: \'stream_options\'.",'
'"param":"stream_options","code":"unknown_parameter"}}'
) == ("stream_options",)
def test_parse_serde_style_and_plain_text():
assert RejectedKeys.parse("unknown field `thinking`, expected one of `model`") == (
"thinking",
)
def test_parse_returns_empty_for_a_genuine_bad_request():
assert RejectedKeys.parse('{"error":{"message":"invalid request: bad param"}}') == ()
assert RejectedKeys.parse("") == ()
assert RejectedKeys.parse(None) == ()
def test_resolver_prefers_none_when_thinking_is_rejected():
resolver = ShapeResolver("deepseek", "chat_completions")
method = resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("thinking",))
assert method == ShapeMethod("none")
def test_resolver_skips_dialects_that_emit_the_rejected_key():
resolver = ShapeResolver("deepseek", "chat_completions")
tried = {NATIVE_METHOD, ShapeMethod("none")}
method = resolver.next_method(tried, ShapeMethod("none"), ("reasoning",))
assert method == ShapeMethod("deepseek")
def test_resolver_adds_a_droppable_key():
resolver = ShapeResolver("deepseek", "chat_completions")
method = resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("stream_options",))
assert method == ShapeMethod("native", ("stream_options",))
def test_resolver_refuses_unknown_and_protected_keys():
resolver = ShapeResolver("deepseek", "chat_completions")
for key in ("messages", "model", "tools", "something_else"):
assert resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, (key,)) is None
def test_resolver_responses_ladder_is_reasoning_only():
resolver = ShapeResolver("deepseek", "responses")
method = resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("reasoning",))
assert method == ShapeMethod("none")
assert resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("thinking",)) is None
def test_method_json_round_trip_and_validation():
method = ShapeMethod("none", ("stream_options",))
assert ShapeMethod.from_json(method.to_json()) == method
assert ShapeMethod.from_json('{"thinking_dialect":"bogus","drop_keys":[]}') is None
assert ShapeMethod.from_json('{"thinking_dialect":"none","drop_keys":["tools"]}') is None
assert ShapeMethod.from_json("not json") is None
def test_apply_none_clears_thinking_and_native_writes_the_dialect():
cfg = {"gateway_upstream_url": "https://strict.example/v1/chat/completions"}
payload = {"model": "m", "messages": [], "thinking": {"type": "enabled"}}
native = NATIVE_METHOD.apply(payload, cfg, "chat_completions", False, lambda m: None)
assert native["thinking"] == {"type": "enabled"}
none = ShapeMethod("none").apply(payload, cfg, "chat_completions", False, lambda m: None)
assert "thinking" not in none and "reasoning" not in none
assert payload["thinking"] == {"type": "enabled"}
def test_apply_drops_stream_options_when_learned():
cfg = {"gateway_upstream_url": "https://strict.example/v1/chat/completions"}
payload = {"model": "m", "messages": [], "stream": True}
kept = NATIVE_METHOD.apply(payload, cfg, "chat_completions", True, lambda m: None)
assert kept["stream_options"] == {"include_usage": True}
dropped = ShapeMethod("native", ("stream_options",)).apply(
payload, cfg, "chat_completions", True, lambda m: None
)
assert "stream_options" not in dropped
def test_strip_wire_removes_reasoning_for_responses_none():
body = {"model": "m", "reasoning": {"effort": "minimal"}, "input": []}
stripped = ShapeMethod("none").strip_wire(body, "responses")
assert "reasoning" not in stripped and body["reasoning"]
assert NATIVE_METHOD.strip_wire(body, "responses") is body
def test_retrier_success_remembers_and_native_success_does_not():
store = MemoryStore_openai_gateway()
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
retrier.succeeded()
assert store.rows == {}
method = retrier.next_after(400, VAIDS_BODY)
assert method == ShapeMethod("none")
retrier.succeeded()
assert store.rows[("p", "m")] == ShapeMethod("none")
def test_retrier_ignores_non_shape_statuses_and_unparsable_bodies():
store = MemoryStore_openai_gateway()
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
assert retrier.next_after(500, VAIDS_BODY) is None
assert retrier.next_after(400, '{"error":"nope"}') is None
def test_retrier_attempts_are_capped():
store = MemoryStore_openai_gateway()
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
retrier.attempts = MAX_RESOLVE_ATTEMPTS
assert retrier.next_after(400, VAIDS_BODY) is None
def test_retrier_forgets_a_rejected_learned_method_once():
store = MemoryStore_openai_gateway()
store.rows[("p", "m")] = ShapeMethod("openrouter")
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
assert retrier.method == ShapeMethod("openrouter")
method = retrier.next_after(
400, '{"issues":[{"code":"unrecognized_keys","keys":["reasoning"]}]}'
)
assert method == NATIVE_METHOD
assert store.forgotten == [("p", "m")]