forked from retoor/devplacepy
Gateway: auto-resolve request shape per provider and model
An upstream that rejects a key the gateway adds (e.g. thinking) now gets retried with other shapes on a parsable 400/422; the working method is stored per (provider, target model) and applied first-try from then on. Fallback attempts are re-shaped per attempt. Admin list/reset, Devii tools, audit event and ledger resolved_method included. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -533,7 +533,7 @@ and its full configuration are documented automatically - including future servi
|
||||
- **`ServiceManager`** - singleton: `register`, `describe_all`, `set_enabled`, `send_command`, `save_config`, `supervise`, `shutdown_all`
|
||||
- **`NewsService`** - a fully automatic, zero-maintenance news pipeline: fetches news from `news.app.molodetz.nl/api`, cleans each article, fetches and perceptually compares the images to reject placeholders and detect uniqueness, grades each one against a free, local, deterministic quality model (aquality - no billed LLM call, no API key), optionally reformats articles that clear the publish grade threshold into clean Markdown with a generative AI model (off by default, since the default grading model cannot generate text), and auto-rotates the best articles to Featured and the landing page. Any AI spend it does incur is metered from the gateway response headers and reported on the admin Services page (calls, tokens, total cost, and per-call averages)
|
||||
- **`BotsService`** - Playwright fleet of AI personas that browse and interact with a DevPlace instance, with live cost/usage metrics and a live screenshot monitor at `/admin/bots` (opt-in; install the `bots` extra)
|
||||
- **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU `DD/MM/YYYY` format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (`gateway_system_preamble`) can be prepended ahead of the client's system message on every call. **Thinking is disabled by default** on every chat and vision call (`thinking.type=disabled` on DeepSeek, `reasoning.effort=none` on OpenRouter, `think=false` on Ollama) so the fast path is the default; a client may re-enable it per request, and an administrator may flip `gateway_thinking` on. **Provider and model routing** (admin **Gateway** page, `/admin/gateway`) maps any number of requested model names onto named upstream providers and target models, each with its own pricing economy (input, output and cache-hit/cache-miss rates per million tokens) and an optional vision model that describes image content before forwarding, so text and vision models are merged transparently. Unmapped requests fall through to the default upstream unchanged, so existing clients are unaffected. Each route may also name a **fallback model** - another already-configured model of the same kind, picked from a select box of the public model names (never a free-text provider model id) - that the gateway retries once, automatically, whenever the primary model fails after its own retries are exhausted, so a struggling model degrades to a working one instead of failing the caller
|
||||
- **`GatewayService`** - OpenAI-compatible LLM gateway at `/openai/v1/*`, forwarding to DeepSeek (default) with optional vision augmentation; the single point of truth for AI that every other service routes through (enabled by default). Every chat request is made date-aware by injecting the current date in EU `DD/MM/YYYY` format into the system message when it contains no date already (date only, never time, to keep upstream prompt caching effective), and an admin-configurable system preamble (`gateway_system_preamble`) can be prepended ahead of the client's system message on every call. **Thinking is disabled by default** on every chat and vision call (`thinking.type=disabled` on DeepSeek, `reasoning.effort=none` on OpenRouter, `think=false` on Ollama) so the fast path is the default; a client may re-enable it per request, and an administrator may flip `gateway_thinking` on. **Provider and model routing** (admin **Gateway** page, `/admin/gateway`) maps any number of requested model names onto named upstream providers and target models, each with its own pricing economy (input, output and cache-hit/cache-miss rates per million tokens) and an optional vision model that describes image content before forwarding, so text and vision models are merged transparently. Unmapped requests fall through to the default upstream unchanged, so existing clients are unaffected. Each route may also name a **fallback model** - another already-configured model of the same kind, picked from a select box of the public model names (never a free-text provider model id) - that the gateway retries once, automatically, whenever the primary model fails after its own retries are exhausted, so a struggling model degrades to a working one instead of failing the caller. When an upstream rejects a request key with a parsable unrecognized-key 400 (for example a strict-schema provider refusing the `thinking` field), the gateway automatically tries other request shapes, remembers the one that works per provider and target model, and applies it from then on; administrators see it as a `shape:` badge on the model grid with a Reset learned action
|
||||
- **`JobService` / `ZipService` / `ForkService`** - generic async job framework (`services/jobs/`) for heavy, blocking work run off the request path; `ZipService` builds project zip archives in a subprocess, `ForkService` copies a project into a new project owned by the forking user
|
||||
- **`ContainerService`** - the admin container manager (`services/containers/`): a reconciling supervisor for container instances, all running one shared prebuilt image
|
||||
- **`AcceptanceService`** - grants every policy agreement (Terms of Service, Privacy Policy, third-party AI processing, activity recording, container credentials) to every account that has not declined it, so an instance kept production-identical for extended manual testing never interrupts with an acceptance dialog. Administrator-only, **off by default**, with a separate switch per agreement type and a dry run that reports what it would do without writing. An account that withdrew a consent is never granted it again, with no further action: the consent ledger itself is the decline register. It is not appropriate on a real production host
|
||||
|
||||
@@ -685,6 +685,7 @@ def init_db():
|
||||
("inter_token_ms", 0.0),
|
||||
("provider", ""),
|
||||
("fallback_used_route", ""),
|
||||
("resolved_method", ""),
|
||||
):
|
||||
if not gateway_usage_ledger.has_column(column):
|
||||
gateway_usage_ledger.create_column_by_example(column, example)
|
||||
@@ -950,6 +951,10 @@ def init_db():
|
||||
|
||||
gateway_routing.ensure_tables()
|
||||
|
||||
from devplacepy.services.openai_gateway import learned as gateway_learned
|
||||
|
||||
gateway_learned.ensure_table()
|
||||
|
||||
from devplacepy.services.openai_gateway import quota as gateway_quota
|
||||
|
||||
gateway_quota.ensure_tables()
|
||||
|
||||
@@ -796,6 +796,49 @@ four ways to sign requests.
|
||||
field("app_reference", "json", "string", False, "typosaurus", "App label (the X-App-Reference header). Blank = every app."),
|
||||
],
|
||||
),
|
||||
endpoint(
|
||||
id="admin-gateway-learned",
|
||||
method="GET",
|
||||
path="/admin/gateway/learned",
|
||||
title="List learned gateway request shapes",
|
||||
summary=(
|
||||
"Every request shape the gateway learned automatically after an upstream rejected the "
|
||||
"default one with an unrecognized-key 400, per provider and target model, plus the "
|
||||
"resolved/exhausted/healed counters."
|
||||
),
|
||||
auth="admin",
|
||||
sample_response={
|
||||
"learned": [
|
||||
{
|
||||
"provider": "vaids",
|
||||
"target_model": "deepseek-v4-1-flash-1017-uncensored",
|
||||
"method": "none",
|
||||
"thinking_dialect": "none",
|
||||
"drop_keys": [],
|
||||
"learned_at": "2026-11-13T00:55:00+00:00",
|
||||
}
|
||||
],
|
||||
"count": 1,
|
||||
"stats": {"resolved": 1, "exhausted": 0, "healed": 0},
|
||||
},
|
||||
),
|
||||
endpoint(
|
||||
id="admin-gateway-learned-reset",
|
||||
method="POST",
|
||||
path="/admin/gateway/learned/reset",
|
||||
title="Reset a learned gateway request shape",
|
||||
summary=(
|
||||
"Forget the learned method for one provider and target model so the gateway starts from "
|
||||
"the default shape again. Use provider 'default' for the base upstream."
|
||||
),
|
||||
auth="admin",
|
||||
destructive=True,
|
||||
params=[
|
||||
field("provider", "json", "string", False, "vaids", "Provider name, or default."),
|
||||
field("target_model", "json", "string", True, "deepseek-v4-1-flash-1017-uncensored", "Upstream target model."),
|
||||
],
|
||||
sample_response={"ok": True},
|
||||
),
|
||||
endpoint(
|
||||
id="admin-gateway-quota-rule-delete",
|
||||
method="DELETE",
|
||||
|
||||
@@ -24,6 +24,12 @@ from devplacepy.services.openai_gateway import (
|
||||
quota,
|
||||
routing,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.learned import (
|
||||
DEFAULT_PROVIDER,
|
||||
LearnedResetIn,
|
||||
learned_store,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.shape import shape_stats
|
||||
from devplacepy.templating import templates
|
||||
from devplacepy.utils import not_found, require_admin
|
||||
|
||||
@@ -151,6 +157,36 @@ def _error_response(
|
||||
)
|
||||
|
||||
|
||||
def _models_with_learned() -> list[dict]:
|
||||
models = routing.model_store.list()
|
||||
for model in models:
|
||||
method = learned_store.get(
|
||||
model["provider"] or DEFAULT_PROVIDER, model["target_model"]
|
||||
)
|
||||
model["learned_shape"] = method.label if method is not None else ""
|
||||
model["learned_provider"] = model["provider"] or DEFAULT_PROVIDER
|
||||
return models
|
||||
|
||||
|
||||
def _reset_learned(request: Request, admin: dict, payload: LearnedResetIn) -> bool:
|
||||
provider = payload.provider or DEFAULT_PROVIDER
|
||||
existed = learned_store.forget(provider, payload.target_model)
|
||||
if existed:
|
||||
audit.record(
|
||||
request,
|
||||
"gateway.learned.reset",
|
||||
user=admin,
|
||||
target_type="gateway_learned_method",
|
||||
target_uid=f"{provider}/{payload.target_model}",
|
||||
target_label=f"{provider}/{payload.target_model}",
|
||||
summary=(
|
||||
f"admin {admin['username']} reset the learned request shape "
|
||||
f"for {provider}/{payload.target_model}"
|
||||
),
|
||||
)
|
||||
return existed
|
||||
|
||||
|
||||
# --- Index page (tabs: models / providers / quota) --------------------------
|
||||
|
||||
|
||||
@@ -177,7 +213,7 @@ async def gateway_config_page(
|
||||
"admin_section": "gateway",
|
||||
"tab": tab,
|
||||
"providers": routing.provider_store.list(),
|
||||
"models": routing.model_store.list(),
|
||||
"models": _models_with_learned(),
|
||||
"quota_rules": quota_rules,
|
||||
"default_provider": _default_provider_summary(),
|
||||
"quota_defaults": _quota_defaults_summary(),
|
||||
@@ -388,6 +424,44 @@ async def list_models(request: Request):
|
||||
)
|
||||
|
||||
|
||||
@router.get("/gateway/learned")
|
||||
async def list_learned(request: Request):
|
||||
require_admin(request)
|
||||
rows = learned_store.list()
|
||||
return JSONResponse(
|
||||
{"learned": rows, "count": len(rows), "stats": shape_stats.snapshot()}
|
||||
)
|
||||
|
||||
|
||||
@router.post("/gateway/learned/reset")
|
||||
async def reset_learned_json(request: Request):
|
||||
admin = require_admin(request)
|
||||
body = await _payload(request)
|
||||
try:
|
||||
payload = LearnedResetIn(**body)
|
||||
except ValidationError as exc:
|
||||
return _validation_error(exc)
|
||||
if not _reset_learned(request, admin, payload):
|
||||
return JSONResponse(
|
||||
{"ok": False, "error": "No learned method for that route"},
|
||||
status_code=404,
|
||||
)
|
||||
return JSONResponse({"ok": True})
|
||||
|
||||
|
||||
@router.post("/gateway/learned/reset-page")
|
||||
async def reset_learned_page(request: Request):
|
||||
admin = require_admin(request)
|
||||
body = await _payload(request)
|
||||
try:
|
||||
payload = LearnedResetIn(**body)
|
||||
except ValidationError as exc:
|
||||
return _validation_error(exc)
|
||||
if not _reset_learned(request, admin, payload):
|
||||
raise not_found("No learned method for that route")
|
||||
return RedirectResponse(url=f"{INDEX_URL}?tab=models", status_code=302)
|
||||
|
||||
|
||||
@router.get("/gateway/provider-models")
|
||||
async def provider_models(request: Request, provider: str = ""):
|
||||
require_admin(request)
|
||||
|
||||
@@ -208,6 +208,39 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
|
||||
confirm(),
|
||||
),
|
||||
),
|
||||
Action(
|
||||
name="gateway_learned_list",
|
||||
method="GET",
|
||||
path="/admin/gateway/learned",
|
||||
summary="List the request shapes the gateway learned automatically per provider and target model (admin only)",
|
||||
description=(
|
||||
"Returns JSON: every learned method (provider, target_model, the method label, its "
|
||||
"thinking dialect and dropped keys, when it was learned) plus the resolve/exhausted/"
|
||||
"healed counters. A method is learned when an upstream rejected the default request "
|
||||
"shape with a parsable unrecognized-key 400 and a retry with another shape succeeded."
|
||||
),
|
||||
handler="http",
|
||||
requires_admin=True,
|
||||
read_only=True,
|
||||
),
|
||||
Action(
|
||||
name="gateway_learned_reset",
|
||||
method="POST",
|
||||
path="/admin/gateway/learned/reset",
|
||||
summary="Forget a learned request shape so the gateway re-learns it (admin only, confirmation required)",
|
||||
description=(
|
||||
"Deletes the learned method for one provider and target model; the next request starts "
|
||||
"from the default shape again and re-resolves if the upstream still rejects it. Use "
|
||||
"provider 'default' for the base upstream. 404 when nothing is learned for that route."
|
||||
),
|
||||
handler="http",
|
||||
requires_admin=True,
|
||||
params=(
|
||||
body("provider", "Provider name, or 'default' for the base upstream."),
|
||||
body("target_model", "The upstream (target) model name the method was learned for."),
|
||||
confirm(),
|
||||
),
|
||||
),
|
||||
Action(
|
||||
name="gateway_quota_rule_delete",
|
||||
method="DELETE",
|
||||
|
||||
@@ -83,6 +83,7 @@ CONFIRM_REQUIRED = {
|
||||
"gateway_model_delete",
|
||||
"gateway_quota_rule_delete",
|
||||
"gateway_quota_reset",
|
||||
"gateway_learned_reset",
|
||||
"email_account_delete",
|
||||
"email_delete_message",
|
||||
"game_prestige",
|
||||
|
||||
@@ -106,6 +106,8 @@ A structured/deterministic internal caller (one that expects a strict, machine-p
|
||||
- OpenRouter: `reasoning: {"effort": "none"}` or `"high"`
|
||||
- Ollama (`:11434` or a host containing `ollama`): `think: false` or `true`
|
||||
|
||||
`thinking_dialect` can also be `none` (no field written); it is only ever chosen by the auto-resolve ladder when an upstream rejects the thinking key (see "Auto-resolve of request shape").
|
||||
|
||||
Do **not** send `reasoning_effort: "none"` to DeepSeek (400: unknown variant). Conflicting client fields (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs.enable_thinking`) are stripped and translated to the dialect above.
|
||||
|
||||
**Default is off.** `gateway_thinking` (Prompt group, bool, default `False`) is the operator override for the silent-client case. A client still wins: `think` / `thinking.type` / `reasoning.effort` / `reasoning.enabled` / `reasoning_effort` / `enable_thinking` on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.
|
||||
@@ -116,6 +118,20 @@ Vision describe-image calls always disable thinking (they are not a reasoning jo
|
||||
|
||||
`thinking.upstream_capabilities(url, dialect="")` derives per-upstream protocol flags (`supports_stream_options`, `supports_stream_usage`, `supports_thinking_field`) from the same effective dialect. Ollama's OpenAI-compatibility layer ignores `stream_options` (and rejects the request outright on some versions), so `handle_chat`'s streaming branch only sends `stream_options: {"include_usage": true}` upstream when `capabilities.supports_stream_options` is true; otherwise a client that asked for `include_usage` gets a logged notice and its stream relayed with no usage chunk. DeepSeek/OpenRouter (and any unrecognised URL) report full support, so this is a no-op for them.
|
||||
|
||||
## Auto-resolve of request shape (`shape.py`, `learned.py`)
|
||||
|
||||
An upstream with a strict schema can reject a key the gateway adds on its own (observed 2026-11-13: route `vaids` answered `400 {"issues":[{"code":"unrecognized_keys","keys":["thinking"]}]}` because `thinking_dialect` falls back to `deepseek` for any unrecognised URL). A 400 is `bad_request`, never retried and never falls back, so the gateway now LEARNS, per (provider, target model), which shape the upstream accepts and applies it from then on.
|
||||
|
||||
- **One funnel.** `ShapeMethod.apply(payload, cfg, protocol, stream, log)` is the only place stream_options, the thinking dialect and droppable keys are written; chat, every fallback attempt and the vision describe call all go through it, each from the PRISTINE client payload with that attempt's own cfg (fallback attempts used to inherit the primary's already-shaped payload). `ShapeMethod.strip_wire` removes `drop_keys` (and `reasoning` for a Responses `none` method) from the final wire body in `_chat_upstream_body`. A method is a frozen `ShapeMethod(thinking_dialect, drop_keys)`; `thinking_dialect` is `native` (today's behaviour: the URL-sniffed or `gateway_thinking_dialect` dialect, the default and first ladder entry), `none` (write no thinking field at all, new in `thinking.py`), or an explicit `deepseek`/`openrouter`/`ollama`.
|
||||
- **Trigger.** Only a 400/422 whose body names rejected keys (`RejectedKeys.parse`: zod `issues[].keys` with `unrecognized_keys`, OpenAI `Unrecognized request argument supplied`/`Unknown parameter` + `param`, serde `unknown field`, "Extra inputs are not permitted"). An unparsable 400 stays terminal and loud, returned verbatim, never retried blind.
|
||||
- **Ladder and allowlist (static ceilings, no autotuning).** `ShapeResolver.next_method` picks the next untried method: a rejected thinking key (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs`) skips every dialect that emits it (`CHAT_LADDER` order: native, none, deepseek, openrouter, ollama; Responses protocol: native, none, rejecting `reasoning`); a rejected key in `DROPPABLE_KEYS` (`stream_options`, `parallel_tool_calls`, `user`, `seed`, `logprobs`, `top_logprobs`, `n`, `presence_penalty`, `frequency_penalty`) is added to `drop_keys`. `messages`, `model`, `tools` and any unknown key are NEVER dropped. Hard cap `MAX_RESOLVE_ATTEMPTS = 4` retries per route attempt.
|
||||
- **Retry placement.** `GatewayRuntime._chat_attempt` (shared by the primary and every fallback attempt) sends, and on a parsable shape 400 re-sends through `ShapeRetrier`; the retry goes through `_send` so the breaker and retries apply, and streaming status is known before any byte is relayed so a retried stream is safe. On success `learned_store.remember` is called (never on a guess) and one `auto-resolved <provider>/<model>: <method>` line is logged. On exhaustion the ORIGINAL 400 response is returned unchanged, nothing is stored, and `shape_stats.exhausted` is incremented. The vision describe call uses the same `ShapeRetrier`, keyed `vision:<host>` + vision model.
|
||||
- **Persistence.** Table `gateway_learned_methods` (`learned.py`, `ensure_table()` from `init_db`): `provider` (`default` for the base upstream), `target_model`, `method` (JSON), `learned_at`, unique index on (provider, target_model), cap `MAX_LEARNED_ROWS` 1000 (oldest by `learned_at` replaced). Cross-worker cache under the `gateway_learned` cache-version name; the upsert is one `ON CONFLICT DO UPDATE` so two workers learning the same route never collide. It is derived data in the garbage-collection class (NOT in `SOFT_DELETE_TABLES`): forget = delete row. Editing a provider's `base_url` or `client_profile`, or deleting it, forgets all its learned methods (`ProviderStore.set`/`remove`).
|
||||
- **Self-healing floor.** If a request using a LEARNED method itself gets a shape 400, the record is forgotten, the ladder restarts once from native, and `shape_stats.healed` counts it.
|
||||
- **Ledger.** One row per client call as before; `resolved_method` (text, `gateway_usage_ledger`) holds the non-native method label that served the call, empty for native.
|
||||
- **Operator surface.** `/admin/gateway` models grid shows a `shape: <method>` badge and a "Reset learned" form (`POST /admin/gateway/learned/reset-page`); JSON `GET /admin/gateway/learned` and `POST /admin/gateway/learned/reset` (audited `gateway.learned.reset`, category `ai`) back the Devii tools `gateway_learned_list` / `gateway_learned_reset` (the reset is in `CONFIRM_REQUIRED` with a declared `confirm`).
|
||||
- **Not covered:** value-level rejections (`unknown variant`), and keys outside the closed allowlists. Tests: `tests/unit/services/openai_gateway/{shape,learned}.py`, the `test_chat_auto_resolve*`/`test_chat_*learned*`/`test_fallback_attempt_is_shaped*` tests in `gateway.py`, `tests/api/admin/gateway/learned.py`, and the grid badge in `tests/e2e/admin/gateway.py`.
|
||||
|
||||
## Real upstream streaming (`GatewayRuntime._stream_chat_response`)
|
||||
|
||||
`stream: true` is forwarded to the upstream verbatim (`payload["stream"] = stream`, no more forced `false`) and the connection is opened with `client.send(request, stream=True)` so the response body is read incrementally via `resp.aiter_lines()` instead of buffered whole. Every retry/circuit-breaker mechanic in `_send`/`retry_send` is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; `retry_send` now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns `200` does the code path diverge into `_stream_chat_response`.
|
||||
@@ -238,6 +254,7 @@ Any `gateway_models` route (chat, embed, or image) can optionally name `fallback
|
||||
- **One ledger row per client call, not one per attempt.** The primary failure never writes a `gateway_usage_ledger` row by itself (only `_send`'s own internal retries and the circuit breaker counters observe it); `finalize()`/`self._ledger.record(...)` still runs exactly once, after the fallback decision, with `model` = whichever model ultimately answered and `requested_model` = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in `/admin/ai-usage` as "requested X, served Y" rather than as two calls.
|
||||
- **Streaming falls back before any byte reaches the client.** `client.send(request, stream=True)` returns as soon as the response headers arrive, so `resp.status_code` is already known before the SSE body is ever touched - `handle_chat` decides whether to fall back at that same point, before it ever opens `_stream_chat_response` to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
|
||||
- **Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (`self._breaker`) is one instance per `GatewayRuntime`, shared across every route.** A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see `circuit_open` too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
|
||||
- **A 400 is terminal UNLESS its body names rejected keys** the auto-resolve ladder can fix (see "Auto-resolve of request shape"): that case is retried on the SAME route with another shape before any fallback is considered.
|
||||
- **Deliberately NOT covered:** `gateway_max_retries`/`gateway_retry_backoff_ms`, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.
|
||||
|
||||
**Tests:** `tests/unit/services/openai_gateway/routing.py` (`resolve_fallback` self-reference/kind/active-flag guards, chain skip/cycle/hop-cap/entry tests), `tests/unit/services/openai_gateway/gateway.py` (`test_chat_falls_back_when_the_primary_model_fails`, `test_chat_returns_the_fallback_failure_when_both_models_fail`, `test_chat_without_a_fallback_never_makes_a_second_call`, `test_chat_walks_fallback_chain_past_failures`, `test_chat_preroutes_around_unhealthy_models`, `test_chat_tries_original_when_chain_is_exhausted`), and `tests/api/admin/gateway/models.py` (`test_model_route_fallback_round_trip`, `test_model_route_fallback_must_reference_an_existing_route`, `test_model_route_fallback_must_be_the_same_kind`, `test_model_route_fallback_rejects_self_reference`).
|
||||
|
||||
@@ -4,7 +4,8 @@ import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Optional
|
||||
|
||||
import httpx
|
||||
from fastapi.responses import JSONResponse, Response, StreamingResponse
|
||||
@@ -39,8 +40,13 @@ from devplacepy.services.openai_gateway.tool_emulation import (
|
||||
emulate_request,
|
||||
emulation_enabled,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import (
|
||||
NATIVE_METHOD,
|
||||
ShapeMethod,
|
||||
ShapeRetrier,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.thinking import (
|
||||
apply_thinking,
|
||||
client_thinking_enabled,
|
||||
upstream_capabilities,
|
||||
)
|
||||
@@ -60,6 +66,15 @@ from devplacepy.services.openai_gateway.vision import VisionAugmenter, VisionCac
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def _close_quietly(resp) -> None:
|
||||
if resp is None:
|
||||
return
|
||||
try:
|
||||
await resp.aclose()
|
||||
except Exception: # noqa: BLE001 - discarding an already-failed response
|
||||
pass
|
||||
|
||||
|
||||
def _call_failed(resp, exc, timing: dict) -> bool:
|
||||
if timing.get("circuit_open") or exc is not None:
|
||||
return True
|
||||
@@ -163,6 +178,7 @@ def _chat_upstream_body(
|
||||
log,
|
||||
catalog_cache=None,
|
||||
owner: tuple = ("anonymous", "anonymous"),
|
||||
method: ShapeMethod = NATIVE_METHOD,
|
||||
) -> dict:
|
||||
# The handler reasons about a Chat payload throughout (routing, vision,
|
||||
# precheck, fallback); only the bytes on the wire change per protocol.
|
||||
@@ -182,13 +198,13 @@ def _chat_upstream_body(
|
||||
)
|
||||
log(f"Meta reserved tool names aliased: {renamed}")
|
||||
if protocol != "responses":
|
||||
return payload
|
||||
return method.strip_wire(payload, protocol)
|
||||
send_body, warnings = chat_to_responses(payload)
|
||||
if warnings:
|
||||
shown = "; ".join(warnings[:3])
|
||||
more = f" (+{len(warnings) - 3} more)" if len(warnings) > 3 else ""
|
||||
log(f"chat->responses transpile warnings: {shown}{more}")
|
||||
if "reasoning" not in send_body:
|
||||
if "reasoning" not in send_body and not method.suppresses_reasoning(protocol):
|
||||
explicit = client_thinking_enabled(client_body)
|
||||
enabled = (
|
||||
explicit
|
||||
@@ -197,7 +213,7 @@ def _chat_upstream_body(
|
||||
)
|
||||
if not enabled:
|
||||
send_body["reasoning"] = {"effort": "minimal"}
|
||||
return send_body
|
||||
return method.strip_wire(send_body, protocol)
|
||||
|
||||
|
||||
def _response_tool_aliases(payload: dict, protocol: str, cfg: dict) -> dict:
|
||||
@@ -350,6 +366,20 @@ def _connect_tracer(holder: dict):
|
||||
return trace
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatAttempt:
|
||||
resp: Any
|
||||
exc: Any
|
||||
timing: dict
|
||||
shaped: dict
|
||||
send_start: float
|
||||
method: ShapeMethod
|
||||
|
||||
@property
|
||||
def resolved_label(self) -> Optional[str]:
|
||||
return None if self.method.is_native else self.method.label
|
||||
|
||||
|
||||
class GatewayRuntime:
|
||||
def __init__(self):
|
||||
self._client: Optional[httpx.AsyncClient] = None
|
||||
@@ -515,6 +545,102 @@ class GatewayRuntime:
|
||||
timing["retry_succeeded"] = attempts > 1
|
||||
return resp, None, timing
|
||||
|
||||
async def _shaped_send(
|
||||
self,
|
||||
client,
|
||||
sem,
|
||||
pristine: dict,
|
||||
body: dict,
|
||||
cfg: dict,
|
||||
protocol: str,
|
||||
model: str,
|
||||
headers: dict,
|
||||
stream: bool,
|
||||
owner: tuple,
|
||||
log,
|
||||
method: ShapeMethod,
|
||||
) -> ChatAttempt:
|
||||
shaped = method.apply(pristine, cfg, protocol, stream, log)
|
||||
shaped["model"] = model
|
||||
send_body = _chat_upstream_body(
|
||||
shaped, body, cfg, protocol, log, self._catalog_cache, owner, method
|
||||
)
|
||||
send_start = time.monotonic()
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
"POST",
|
||||
cfg["gateway_upstream_url"],
|
||||
headers,
|
||||
cfg,
|
||||
log,
|
||||
json_body=send_body,
|
||||
stream=stream,
|
||||
)
|
||||
if resp is not None and stream and resp.status_code != 200:
|
||||
# The streamed response body hasn't been read yet; the shared error
|
||||
# handling needs resp.text/.content, which requires an explicit
|
||||
# read for a stream=True response (a no-op if already buffered).
|
||||
try:
|
||||
await resp.aread()
|
||||
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
|
||||
# error body must never crash the request; fall through with
|
||||
# whatever text/content httpx managed to buffer (possibly empty).
|
||||
log(f"failed to read non-200 stream error body: {e}")
|
||||
return ChatAttempt(resp, exc, timing, shaped, send_start, method)
|
||||
|
||||
async def _chat_attempt(
|
||||
self,
|
||||
client,
|
||||
sem,
|
||||
pristine: dict,
|
||||
body: dict,
|
||||
cfg: dict,
|
||||
protocol: str,
|
||||
model: str,
|
||||
headers: dict,
|
||||
stream: bool,
|
||||
owner: tuple,
|
||||
log,
|
||||
) -> ChatAttempt:
|
||||
native_dialect = upstream_capabilities(
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
cfg.get("gateway_thinking_dialect", "auto"),
|
||||
).dialect
|
||||
retrier = ShapeRetrier.for_route(
|
||||
learned_store,
|
||||
cfg.get("gateway_provider_name") or "default",
|
||||
model,
|
||||
protocol,
|
||||
native_dialect,
|
||||
log,
|
||||
)
|
||||
first = await self._shaped_send(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log, retrier.method,
|
||||
)
|
||||
current = first
|
||||
while current.resp is not None and retrier.eligible(
|
||||
current.resp.status_code
|
||||
):
|
||||
method = retrier.next_after(current.resp.status_code, current.resp.text)
|
||||
if method is None:
|
||||
break
|
||||
if current is not first:
|
||||
await _close_quietly(current.resp)
|
||||
current = await self._shaped_send(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log, method,
|
||||
)
|
||||
if current is first:
|
||||
return first
|
||||
if current.resp is not None and current.resp.status_code < 400:
|
||||
retrier.succeeded()
|
||||
return current
|
||||
retrier.exhausted()
|
||||
await _close_quietly(current.resp)
|
||||
return first
|
||||
|
||||
async def handle_chat(
|
||||
self,
|
||||
body: dict,
|
||||
@@ -620,46 +746,10 @@ class GatewayRuntime:
|
||||
|
||||
stream = bool(body.get("stream"))
|
||||
include_usage = bool((body.get("stream_options") or {}).get("include_usage"))
|
||||
payload = dict(body)
|
||||
payload["model"] = model
|
||||
payload["messages"] = messages
|
||||
payload["stream"] = stream
|
||||
if stream and protocol == "responses":
|
||||
# Responses streams carry usage on response.completed natively;
|
||||
# there is no include_usage to request (stream_options only
|
||||
# supports include_obfuscation there). The translator captures it
|
||||
# and the client's own include_usage still controls the relay.
|
||||
payload.pop("stream_options", None)
|
||||
if stream and protocol != "responses":
|
||||
capabilities = upstream_capabilities(
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
cfg.get("gateway_thinking_dialect", "auto"),
|
||||
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
|
||||
)
|
||||
if capabilities.supports_stream_options:
|
||||
stream_options = dict(body.get("stream_options") or {})
|
||||
stream_options["include_usage"] = True
|
||||
payload["stream_options"] = stream_options
|
||||
else:
|
||||
payload.pop("stream_options", None)
|
||||
if include_usage:
|
||||
log(
|
||||
"stream_options.include_usage requested but the upstream "
|
||||
f"({capabilities.dialect}) does not support it; relaying "
|
||||
"without a usage chunk"
|
||||
)
|
||||
else:
|
||||
payload.pop("stream_options", None)
|
||||
if protocol != "responses":
|
||||
apply_thinking(
|
||||
payload,
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
default_enabled=bool(cfg.get("gateway_thinking", False)),
|
||||
dialect=cfg.get("gateway_thinking_dialect", "auto"),
|
||||
)
|
||||
# For responses upstreams the reasoning effort is set at transpile
|
||||
# time instead (see _chat_upstream_body): chat-dialect thinking
|
||||
# fields have no meaning on that wire.
|
||||
pristine = dict(body)
|
||||
pristine["model"] = model
|
||||
pristine["messages"] = messages
|
||||
pristine["stream"] = stream
|
||||
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
@@ -678,7 +768,7 @@ class GatewayRuntime:
|
||||
)
|
||||
context_window = _route_context_window(cfg, model)
|
||||
if context_window:
|
||||
estimated_tokens = _estimate_request_tokens(payload)
|
||||
estimated_tokens = _estimate_request_tokens(pristine)
|
||||
if estimated_tokens > context_window:
|
||||
for precheck_route in resolve_fallback_chain(chain_start, "chat"):
|
||||
precheck_overlay = chat_overlay(
|
||||
@@ -704,39 +794,19 @@ class GatewayRuntime:
|
||||
or "chat_completions"
|
||||
)
|
||||
model = cfg["gateway_model"]
|
||||
payload = dict(payload)
|
||||
payload["model"] = model
|
||||
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
|
||||
break
|
||||
|
||||
send_start = time.monotonic()
|
||||
try:
|
||||
send_body = _chat_upstream_body(
|
||||
payload, body, cfg, protocol, log, self._catalog_cache, owner
|
||||
attempt = await self._chat_attempt(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log,
|
||||
)
|
||||
except ToolEmulationError as tool_error:
|
||||
return _tool_emulation_rejection(tool_error)
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
"POST",
|
||||
cfg["gateway_upstream_url"],
|
||||
headers,
|
||||
cfg,
|
||||
log,
|
||||
json_body=send_body,
|
||||
stream=stream,
|
||||
)
|
||||
if resp is not None and stream and resp.status_code != 200:
|
||||
# The streamed response body hasn't been read yet; the shared error
|
||||
# handling below needs resp.text/.content, which requires an explicit
|
||||
# read for a stream=True response (a no-op if already buffered).
|
||||
try:
|
||||
await resp.aread()
|
||||
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
|
||||
# error body must never crash the request; fall through with
|
||||
# whatever text/content httpx managed to buffer (possibly empty).
|
||||
log(f"failed to read non-200 stream error body: {e}")
|
||||
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
|
||||
payload = attempt.shaped
|
||||
send_start = attempt.send_start
|
||||
|
||||
requested_model = requested
|
||||
body_error = _embedded_failure_message(resp, stream, protocol)
|
||||
@@ -761,32 +831,17 @@ class GatewayRuntime:
|
||||
cfg.get("gateway_upstream_protocol") or "chat_completions"
|
||||
)
|
||||
model = cfg["gateway_model"]
|
||||
payload = dict(payload)
|
||||
payload["model"] = model
|
||||
headers = _fallback_headers(cfg, "gateway_api_key", app_reference)
|
||||
send_start = time.monotonic()
|
||||
try:
|
||||
send_body = _chat_upstream_body(
|
||||
payload, body, cfg, protocol, log, self._catalog_cache, owner
|
||||
attempt = await self._chat_attempt(
|
||||
client, sem, pristine, body, cfg, protocol, model, headers,
|
||||
stream, owner, log,
|
||||
)
|
||||
except ToolEmulationError as tool_error:
|
||||
return _tool_emulation_rejection(tool_error)
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
"POST",
|
||||
cfg["gateway_upstream_url"],
|
||||
headers,
|
||||
cfg,
|
||||
log,
|
||||
json_body=send_body,
|
||||
stream=stream,
|
||||
)
|
||||
if resp is not None and stream and resp.status_code != 200:
|
||||
try:
|
||||
await resp.aread()
|
||||
except httpx.HTTPError as e: # noqa: BLE001
|
||||
log(f"failed to read fallback non-200 stream error body: {e}")
|
||||
resp, exc, timing = attempt.resp, attempt.exc, attempt.timing
|
||||
payload = attempt.shaped
|
||||
send_start = attempt.send_start
|
||||
body_error = _embedded_failure_message(resp, stream, protocol)
|
||||
|
||||
pricing = pricing_from_cfg(cfg)
|
||||
@@ -803,6 +858,7 @@ class GatewayRuntime:
|
||||
"app_reference": app_reference,
|
||||
"provider": cfg.get("gateway_provider_name") or "default",
|
||||
"fallback_used_route": fallback_used_route,
|
||||
"resolved_method": attempt.resolved_label,
|
||||
**params,
|
||||
**timing,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from devplacepy.database import (
|
||||
bump_cache_version,
|
||||
db,
|
||||
get_table,
|
||||
sync_local_cache,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.shape import ShapeMethod
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TABLE = "gateway_learned_methods"
|
||||
CACHE_NAME = "gateway_learned"
|
||||
MAX_LEARNED_ROWS = 1000
|
||||
DEFAULT_PROVIDER = "default"
|
||||
|
||||
_LEARNED_CACHE: dict = {}
|
||||
|
||||
|
||||
class LearnedResetIn(BaseModel):
|
||||
provider: str = Field(default="", max_length=120)
|
||||
target_model: str = Field(min_length=1, max_length=200)
|
||||
|
||||
@field_validator("provider", "target_model")
|
||||
@classmethod
|
||||
def _strip(cls, value: str) -> str:
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _provider_key(provider: Optional[str]) -> str:
|
||||
return (provider or DEFAULT_PROVIDER).strip().lower() or DEFAULT_PROVIDER
|
||||
|
||||
|
||||
def ensure_table() -> None:
|
||||
db.query(
|
||||
"CREATE TABLE IF NOT EXISTS "
|
||||
+ TABLE
|
||||
+ " (id INTEGER PRIMARY KEY, provider TEXT, target_model TEXT, "
|
||||
"method TEXT, learned_at TEXT)"
|
||||
)
|
||||
try:
|
||||
db.query(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_learned_route ON "
|
||||
+ TABLE
|
||||
+ " (provider, target_model)"
|
||||
)
|
||||
db.query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_gateway_learned_at ON "
|
||||
+ TABLE
|
||||
+ " (learned_at)"
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("gateway learned index creation failed: %s", exc)
|
||||
|
||||
|
||||
def _load() -> dict:
|
||||
sync_local_cache(CACHE_NAME, _LEARNED_CACHE)
|
||||
if "methods" not in _LEARNED_CACHE:
|
||||
methods: dict = {}
|
||||
try:
|
||||
if TABLE in db.tables:
|
||||
for row in get_table(TABLE).all():
|
||||
method = ShapeMethod.from_json(row.get("method"))
|
||||
if method is None:
|
||||
continue
|
||||
methods[
|
||||
(_provider_key(row.get("provider")), str(row["target_model"]))
|
||||
] = (method, str(row.get("learned_at") or ""))
|
||||
except Exception as exc:
|
||||
logger.warning("gateway learned load failed: %s", exc)
|
||||
_LEARNED_CACHE["methods"] = methods
|
||||
return _LEARNED_CACHE["methods"]
|
||||
|
||||
|
||||
def _invalidate() -> None:
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_LEARNED_CACHE.clear()
|
||||
|
||||
|
||||
class LearnedMethodStore:
|
||||
def get(self, provider: Optional[str], model: str) -> Optional[ShapeMethod]:
|
||||
entry = _load().get((_provider_key(provider), str(model)))
|
||||
return entry[0] if entry else None
|
||||
|
||||
def list(self) -> list[dict]:
|
||||
rows = [
|
||||
{
|
||||
"provider": provider,
|
||||
"target_model": model,
|
||||
"method": method.label,
|
||||
"thinking_dialect": method.thinking_dialect,
|
||||
"drop_keys": list(method.drop_keys),
|
||||
"learned_at": learned_at,
|
||||
}
|
||||
for (provider, model), (method, learned_at) in _load().items()
|
||||
]
|
||||
return sorted(rows, key=lambda r: (r["provider"], r["target_model"]))
|
||||
|
||||
def remember(
|
||||
self, provider: Optional[str], model: str, method: ShapeMethod
|
||||
) -> None:
|
||||
if method.is_native or not model:
|
||||
return
|
||||
try:
|
||||
ensure_table()
|
||||
with db:
|
||||
db.query(
|
||||
"INSERT INTO " + TABLE + " (provider, target_model, method, learned_at) "
|
||||
"VALUES (:provider, :model, :method, :now) "
|
||||
"ON CONFLICT(provider, target_model) DO UPDATE SET "
|
||||
"method = excluded.method, learned_at = excluded.learned_at",
|
||||
provider=_provider_key(provider),
|
||||
model=str(model),
|
||||
method=method.to_json(),
|
||||
now=_now(),
|
||||
)
|
||||
db.query(
|
||||
"DELETE FROM " + TABLE + " WHERE id IN (SELECT id FROM "
|
||||
+ TABLE
|
||||
+ " ORDER BY learned_at ASC, id ASC LIMIT "
|
||||
"MAX((SELECT COUNT(*) FROM " + TABLE + ") - :cap, 0))",
|
||||
cap=MAX_LEARNED_ROWS,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("gateway learned remember failed: %s", exc)
|
||||
return
|
||||
_invalidate()
|
||||
|
||||
def forget(self, provider: Optional[str], model: str) -> bool:
|
||||
if TABLE not in db.tables:
|
||||
return False
|
||||
removed = int(
|
||||
get_table(TABLE).delete(
|
||||
provider=_provider_key(provider), target_model=str(model)
|
||||
)
|
||||
)
|
||||
if removed:
|
||||
_invalidate()
|
||||
return bool(removed)
|
||||
|
||||
def forget_provider(self, provider: str) -> int:
|
||||
if TABLE not in db.tables:
|
||||
return 0
|
||||
table = get_table(TABLE)
|
||||
removed = table.count(provider=_provider_key(provider))
|
||||
if removed:
|
||||
table.delete(provider=_provider_key(provider))
|
||||
_invalidate()
|
||||
return removed
|
||||
|
||||
|
||||
learned_store = LearnedMethodStore()
|
||||
@@ -16,6 +16,7 @@ from devplacepy.database import (
|
||||
sync_local_cache,
|
||||
)
|
||||
from devplacepy.services.openai_gateway import muse_code, opencode_zen
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.tool_emulation import (
|
||||
TOOLS_MODE_NATIVE,
|
||||
TOOLS_MODES,
|
||||
@@ -531,11 +532,17 @@ class ProviderStore:
|
||||
"protocol": payload.protocol,
|
||||
"updated_at": _now(),
|
||||
}
|
||||
endpoint_changed = bool(existing) and (
|
||||
str(existing.get("base_url") or "") != payload.base_url
|
||||
or str(existing.get("client_profile") or "") != payload.client_profile
|
||||
)
|
||||
if existing:
|
||||
table.update({**record, "id": existing["id"]}, ["id"])
|
||||
else:
|
||||
record["created_at"] = _now()
|
||||
table.insert(record)
|
||||
if endpoint_changed:
|
||||
learned_store.forget_provider(payload.name)
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_ROUTING_CACHE.clear()
|
||||
return self.get(payload.name) or record
|
||||
@@ -546,6 +553,7 @@ class ProviderStore:
|
||||
return False
|
||||
removed = int(get_table(PROVIDERS_TABLE).delete(name=name))
|
||||
if removed:
|
||||
learned_store.forget_provider(name)
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_ROUTING_CACHE.clear()
|
||||
return bool(removed)
|
||||
|
||||
@@ -0,0 +1,388 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
from devplacepy.services.openai_gateway.thinking import (
|
||||
CLIENT_THINKING_KEYS,
|
||||
apply_thinking,
|
||||
upstream_capabilities,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PROTOCOL_RESPONSES = "responses"
|
||||
DIALECT_NATIVE = "native"
|
||||
DIALECT_NONE = "none"
|
||||
DIALECT_DEEPSEEK = "deepseek"
|
||||
DIALECT_OPENROUTER = "openrouter"
|
||||
DIALECT_OLLAMA = "ollama"
|
||||
CHAT_LADDER = (
|
||||
DIALECT_NATIVE,
|
||||
DIALECT_NONE,
|
||||
DIALECT_DEEPSEEK,
|
||||
DIALECT_OPENROUTER,
|
||||
DIALECT_OLLAMA,
|
||||
)
|
||||
RESPONSES_LADDER = (DIALECT_NATIVE, DIALECT_NONE)
|
||||
DIALECTS = frozenset(CHAT_LADDER)
|
||||
DROPPABLE_KEYS = frozenset(
|
||||
{
|
||||
"stream_options",
|
||||
"parallel_tool_calls",
|
||||
"user",
|
||||
"seed",
|
||||
"logprobs",
|
||||
"top_logprobs",
|
||||
"n",
|
||||
"presence_penalty",
|
||||
"frequency_penalty",
|
||||
}
|
||||
)
|
||||
THINKING_KEYS = frozenset(CLIENT_THINKING_KEYS) | {"chat_template_kwargs"}
|
||||
DIALECT_EMITS: dict = {
|
||||
DIALECT_DEEPSEEK: frozenset({"thinking"}),
|
||||
DIALECT_OPENROUTER: frozenset({"reasoning"}),
|
||||
DIALECT_OLLAMA: frozenset({"think", "reasoning_effort"}),
|
||||
DIALECT_NONE: frozenset(),
|
||||
}
|
||||
RESPONSES_EMITS = frozenset({"reasoning"})
|
||||
MAX_RESOLVE_ATTEMPTS = 4
|
||||
SHAPE_STATUSES = (400, 422)
|
||||
MAX_REJECTED_KEYS = 16
|
||||
MAX_KEY_LENGTH = 64
|
||||
|
||||
_KEY_PATTERNS = (
|
||||
re.compile(r"Unrecognized key\(s\) in object:\s*((?:'[^']+'(?:,\s*)?)+)", re.I),
|
||||
re.compile(r"Unrecognized request arguments? supplied:\s*([\w.,\s]+)", re.I),
|
||||
re.compile(r"Unknown (?:parameter|field|argument)s?:?\s*[`'\"]?([\w.]+)", re.I),
|
||||
re.compile(r"unknown field\s*[`'\"]([\w.]+)[`'\"]", re.I),
|
||||
re.compile(r"Extra inputs are not permitted.{0,40}?['\"`]([\w.]+)['\"`]", re.I),
|
||||
)
|
||||
_QUOTED = re.compile(r"'([^']+)'")
|
||||
_BARE_SPLIT = re.compile(r"[,\s]+")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ShapeMethod:
|
||||
thinking_dialect: str = DIALECT_NATIVE
|
||||
drop_keys: tuple = ()
|
||||
|
||||
@property
|
||||
def is_native(self) -> bool:
|
||||
return self.thinking_dialect == DIALECT_NATIVE and not self.drop_keys
|
||||
|
||||
@property
|
||||
def label(self) -> str:
|
||||
return self.thinking_dialect + "".join(f" -{key}" for key in self.drop_keys)
|
||||
|
||||
def dialect_for(self, cfg: dict) -> str:
|
||||
if self.thinking_dialect == DIALECT_NATIVE:
|
||||
return str(cfg.get("gateway_thinking_dialect", "auto") or "auto")
|
||||
return self.thinking_dialect
|
||||
|
||||
def to_json(self) -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"thinking_dialect": self.thinking_dialect,
|
||||
"drop_keys": list(self.drop_keys),
|
||||
}
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_json(cls, text: Any) -> Optional["ShapeMethod"]:
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
dialect = data.get("thinking_dialect")
|
||||
drops = data.get("drop_keys")
|
||||
if dialect not in DIALECTS or not isinstance(drops, list):
|
||||
return None
|
||||
if any(key not in DROPPABLE_KEYS for key in drops):
|
||||
return None
|
||||
return cls(dialect, tuple(sorted(set(drops))))
|
||||
|
||||
def apply(
|
||||
self,
|
||||
payload: dict,
|
||||
cfg: dict,
|
||||
protocol: str,
|
||||
stream: bool,
|
||||
log: Callable[[str], None],
|
||||
) -> dict:
|
||||
shaped = dict(payload)
|
||||
kwargs = shaped.get("chat_template_kwargs")
|
||||
if isinstance(kwargs, dict):
|
||||
shaped["chat_template_kwargs"] = dict(kwargs)
|
||||
if protocol == PROTOCOL_RESPONSES:
|
||||
shaped.pop("stream_options", None)
|
||||
else:
|
||||
self._apply_stream_options(shaped, payload, cfg, stream, log)
|
||||
apply_thinking(
|
||||
shaped,
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
default_enabled=bool(cfg.get("gateway_thinking", False)),
|
||||
dialect=self.dialect_for(cfg),
|
||||
)
|
||||
for key in self.drop_keys:
|
||||
shaped.pop(key, None)
|
||||
return shaped
|
||||
|
||||
def _apply_stream_options(
|
||||
self,
|
||||
shaped: dict,
|
||||
payload: dict,
|
||||
cfg: dict,
|
||||
stream: bool,
|
||||
log: Callable[[str], None],
|
||||
) -> None:
|
||||
include_usage = bool((payload.get("stream_options") or {}).get("include_usage"))
|
||||
capabilities = upstream_capabilities(
|
||||
cfg.get("gateway_upstream_url", ""),
|
||||
cfg.get("gateway_thinking_dialect", "auto"),
|
||||
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
|
||||
)
|
||||
if (
|
||||
stream
|
||||
and capabilities.supports_stream_options
|
||||
and "stream_options" not in self.drop_keys
|
||||
):
|
||||
stream_options = dict(payload.get("stream_options") or {})
|
||||
stream_options["include_usage"] = True
|
||||
shaped["stream_options"] = stream_options
|
||||
return
|
||||
shaped.pop("stream_options", None)
|
||||
if stream and include_usage:
|
||||
log(
|
||||
"stream_options.include_usage requested but the upstream "
|
||||
f"({capabilities.dialect}) does not support it; relaying "
|
||||
"without a usage chunk"
|
||||
)
|
||||
|
||||
def suppresses_reasoning(self, protocol: str) -> bool:
|
||||
return protocol == PROTOCOL_RESPONSES and self.thinking_dialect == DIALECT_NONE
|
||||
|
||||
def strip_wire(self, body: dict, protocol: str) -> dict:
|
||||
doomed = set(self.drop_keys)
|
||||
if self.suppresses_reasoning(protocol):
|
||||
doomed.add("reasoning")
|
||||
if not doomed.intersection(body):
|
||||
return body
|
||||
return {key: value for key, value in body.items() if key not in doomed}
|
||||
|
||||
|
||||
NATIVE_METHOD = ShapeMethod()
|
||||
|
||||
|
||||
class RejectedKeys:
|
||||
@staticmethod
|
||||
def parse(body_text: Any) -> tuple:
|
||||
text = body_text if isinstance(body_text, str) else ""
|
||||
if not text:
|
||||
return ()
|
||||
found: list = []
|
||||
RejectedKeys._from_json(text, found)
|
||||
RejectedKeys._from_patterns(text, found)
|
||||
cleaned: list = []
|
||||
for key in found:
|
||||
top = str(key).strip().strip("`'\"").split(".")[0].split("[")[0]
|
||||
if top and len(top) <= MAX_KEY_LENGTH and top not in cleaned:
|
||||
cleaned.append(top)
|
||||
return tuple(cleaned[:MAX_REJECTED_KEYS])
|
||||
|
||||
@staticmethod
|
||||
def _from_json(text: str, found: list) -> None:
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except ValueError:
|
||||
return
|
||||
if not isinstance(data, dict):
|
||||
return
|
||||
issues = data.get("issues")
|
||||
if isinstance(issues, list):
|
||||
for issue in issues:
|
||||
if isinstance(issue, dict) and issue.get("code") == "unrecognized_keys":
|
||||
keys = issue.get("keys")
|
||||
if isinstance(keys, list):
|
||||
found.extend(str(key) for key in keys)
|
||||
error = data.get("error")
|
||||
if isinstance(error, dict):
|
||||
param = error.get("param")
|
||||
code = str(error.get("code") or "")
|
||||
if isinstance(param, str) and param and code in (
|
||||
"unknown_parameter",
|
||||
"unrecognized_request_argument",
|
||||
):
|
||||
found.append(param)
|
||||
|
||||
@staticmethod
|
||||
def _from_patterns(text: str, found: list) -> None:
|
||||
for pattern in _KEY_PATTERNS:
|
||||
for match in pattern.finditer(text):
|
||||
chunk = match.group(1)
|
||||
quoted = _QUOTED.findall(chunk)
|
||||
if quoted:
|
||||
found.extend(quoted)
|
||||
else:
|
||||
found.extend(
|
||||
part for part in _BARE_SPLIT.split(chunk.strip()) if part
|
||||
)
|
||||
|
||||
|
||||
class ShapeResolver:
|
||||
def __init__(self, native_dialect: str, protocol: str) -> None:
|
||||
self.native_dialect = native_dialect
|
||||
self.responses = protocol == PROTOCOL_RESPONSES
|
||||
|
||||
def emits(self, dialect: str) -> frozenset:
|
||||
if self.responses:
|
||||
return frozenset() if dialect == DIALECT_NONE else RESPONSES_EMITS
|
||||
if dialect == DIALECT_NATIVE:
|
||||
return DIALECT_EMITS.get(self.native_dialect, frozenset())
|
||||
return DIALECT_EMITS.get(dialect, frozenset())
|
||||
|
||||
def next_method(
|
||||
self, tried: set, current: ShapeMethod, rejected: tuple
|
||||
) -> Optional[ShapeMethod]:
|
||||
keys = set(rejected)
|
||||
drops = set(current.drop_keys) | (keys & DROPPABLE_KEYS)
|
||||
if self.responses:
|
||||
thinking_hit = keys & RESPONSES_EMITS
|
||||
else:
|
||||
thinking_hit = keys & THINKING_KEYS
|
||||
if not thinking_hit and drops == set(current.drop_keys):
|
||||
return None
|
||||
sorted_drops = tuple(sorted(drops))
|
||||
if not thinking_hit:
|
||||
candidate = ShapeMethod(current.thinking_dialect, sorted_drops)
|
||||
return None if candidate in tried else candidate
|
||||
ladder = RESPONSES_LADDER if self.responses else CHAT_LADDER
|
||||
for dialect in ladder:
|
||||
if self.emits(dialect) & keys:
|
||||
continue
|
||||
candidate = ShapeMethod(dialect, sorted_drops)
|
||||
if candidate not in tried:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ShapeStats:
|
||||
resolved: int = 0
|
||||
exhausted: int = 0
|
||||
healed: int = 0
|
||||
|
||||
def snapshot(self) -> dict:
|
||||
return {
|
||||
"resolved": self.resolved,
|
||||
"exhausted": self.exhausted,
|
||||
"healed": self.healed,
|
||||
}
|
||||
|
||||
|
||||
shape_stats = ShapeStats()
|
||||
|
||||
|
||||
@dataclass
|
||||
class ShapeRetrier:
|
||||
store: Any
|
||||
provider: str
|
||||
model: str
|
||||
resolver: ShapeResolver
|
||||
log: Callable[[str], None]
|
||||
method: ShapeMethod = field(default=NATIVE_METHOD)
|
||||
from_store: bool = False
|
||||
healed: bool = False
|
||||
attempts: int = 0
|
||||
tried: set = field(default_factory=set)
|
||||
|
||||
@classmethod
|
||||
def for_route(
|
||||
cls,
|
||||
store: Any,
|
||||
provider: str,
|
||||
model: str,
|
||||
protocol: str,
|
||||
native_dialect: str,
|
||||
log: Callable[[str], None],
|
||||
) -> "ShapeRetrier":
|
||||
learned = store.get(provider, model)
|
||||
retrier = cls(
|
||||
store=store,
|
||||
provider=provider,
|
||||
model=model,
|
||||
resolver=ShapeResolver(native_dialect, protocol),
|
||||
log=log,
|
||||
method=learned or NATIVE_METHOD,
|
||||
from_store=learned is not None,
|
||||
)
|
||||
retrier.tried.add(retrier.method)
|
||||
return retrier
|
||||
|
||||
@staticmethod
|
||||
def eligible(status: int) -> bool:
|
||||
return status in SHAPE_STATUSES
|
||||
|
||||
def next_after(self, status: int, body_text: Any) -> Optional[ShapeMethod]:
|
||||
if not self.eligible(status) or self.attempts >= MAX_RESOLVE_ATTEMPTS:
|
||||
return None
|
||||
rejected = RejectedKeys.parse(body_text)
|
||||
if not rejected:
|
||||
return None
|
||||
if self.from_store and not self.healed:
|
||||
self.store.forget(self.provider, self.model)
|
||||
self.healed = True
|
||||
self.from_store = False
|
||||
shape_stats.healed += 1
|
||||
self.log(
|
||||
f"learned shape '{self.method.label}' for {self.provider}/"
|
||||
f"{self.model} was rejected ({', '.join(rejected)}); "
|
||||
"forgotten, restarting the ladder"
|
||||
)
|
||||
self.tried = {self.method}
|
||||
if NATIVE_METHOD not in self.tried:
|
||||
return self._advance(NATIVE_METHOD)
|
||||
candidate = self.resolver.next_method(self.tried, self.method, rejected)
|
||||
if candidate is None:
|
||||
return None
|
||||
return self._advance(candidate)
|
||||
|
||||
def _advance(self, candidate: ShapeMethod) -> ShapeMethod:
|
||||
self.attempts += 1
|
||||
self.tried.add(candidate)
|
||||
self.method = candidate
|
||||
self.log(
|
||||
f"upstream rejected the request shape for {self.provider}/"
|
||||
f"{self.model}, trying '{candidate.label}'"
|
||||
)
|
||||
return candidate
|
||||
|
||||
def succeeded(self) -> None:
|
||||
if self.attempts == 0 or self.method.is_native:
|
||||
return
|
||||
self.store.remember(self.provider, self.model, self.method)
|
||||
shape_stats.resolved += 1
|
||||
self.log(
|
||||
f"auto-resolved {self.provider}/{self.model}: {self.method.label}"
|
||||
)
|
||||
|
||||
def exhausted(self) -> None:
|
||||
if self.attempts == 0:
|
||||
return
|
||||
shape_stats.exhausted += 1
|
||||
self.log(
|
||||
f"could not auto-resolve the request shape for {self.provider}/"
|
||||
f"{self.model} after {self.attempts} attempt(s)"
|
||||
)
|
||||
|
||||
@property
|
||||
def resolved_label(self) -> Optional[str]:
|
||||
return None if self.method.is_native else self.method.label
|
||||
@@ -144,6 +144,8 @@ def _clear_thinking_fields(payload: dict) -> None:
|
||||
|
||||
def write_thinking(payload: dict, dialect: str, enabled: bool) -> None:
|
||||
_clear_thinking_fields(payload)
|
||||
if dialect == "none":
|
||||
return
|
||||
if dialect == "openrouter":
|
||||
payload["reasoning"] = {"effort": "high" if enabled else "none"}
|
||||
return
|
||||
|
||||
@@ -587,6 +587,11 @@ class GatewayUsageLedger:
|
||||
if raw.get("fallback_used_route")
|
||||
else None
|
||||
),
|
||||
"resolved_method": (
|
||||
str(raw["resolved_method"])[:128]
|
||||
if raw.get("resolved_method")
|
||||
else None
|
||||
),
|
||||
"status_code": int(raw.get("status_code") or 0),
|
||||
"success": 1 if raw.get("success") else 0,
|
||||
"error_category": raw.get("error_category"),
|
||||
|
||||
@@ -6,6 +6,7 @@ import logging
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from typing import Any, Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
@@ -16,7 +17,9 @@ from devplacepy.services.openai_gateway.reliability import (
|
||||
_is_retryable_status,
|
||||
_retry_after_seconds,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.thinking import apply_thinking
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import ShapeRetrier
|
||||
from devplacepy.services.openai_gateway.thinking import upstream_capabilities
|
||||
from devplacepy.services.openai_gateway.usage import classify_error
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -151,37 +154,9 @@ class VisionAugmenter:
|
||||
if row:
|
||||
self.cost_usd += float(row.get("cost_usd") or 0.0)
|
||||
|
||||
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
|
||||
if not self.vision_key:
|
||||
return "[vision unavailable: vision API key not configured]"
|
||||
if self.breaker is not None and not self.breaker.allow():
|
||||
self._record((0.0), 503, False, "circuit_open", None)
|
||||
return "[vision unavailable: circuit breaker open]"
|
||||
payload = {
|
||||
"model": self.vision_model,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": VISION_INSTRUCTION},
|
||||
image_block,
|
||||
],
|
||||
}
|
||||
],
|
||||
"temperature": 0.2,
|
||||
"stream": False,
|
||||
}
|
||||
apply_thinking(payload, self.vision_url, default_enabled=False, dialect=self.vision_dialect)
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.vision_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
if self.referer:
|
||||
headers["HTTP-Referer"] = self.referer
|
||||
if self.title:
|
||||
headers[TITLE_HEADER] = self.title
|
||||
headers[LEGACY_TITLE_HEADER] = self.title
|
||||
start = time.monotonic()
|
||||
async def _post_with_retries(
|
||||
self, client: httpx.AsyncClient, payload: dict, headers: dict
|
||||
) -> tuple:
|
||||
attempt = 0
|
||||
resp: Optional[httpx.Response] = None
|
||||
exc: Optional[Exception] = None
|
||||
@@ -210,6 +185,68 @@ class VisionAugmenter:
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
await _backoff(self.retry_backoff_ms, attempt, retry_after)
|
||||
return resp, exc
|
||||
|
||||
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
|
||||
if not self.vision_key:
|
||||
return "[vision unavailable: vision API key not configured]"
|
||||
if self.breaker is not None and not self.breaker.allow():
|
||||
self._record((0.0), 503, False, "circuit_open", None)
|
||||
return "[vision unavailable: circuit breaker open]"
|
||||
payload = {
|
||||
"model": self.vision_model,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": VISION_INSTRUCTION},
|
||||
image_block,
|
||||
],
|
||||
}
|
||||
],
|
||||
"temperature": 0.2,
|
||||
"stream": False,
|
||||
}
|
||||
shape_cfg = {
|
||||
"gateway_upstream_url": self.vision_url,
|
||||
"gateway_thinking_dialect": self.vision_dialect or "auto",
|
||||
"gateway_thinking": False,
|
||||
}
|
||||
retrier = ShapeRetrier.for_route(
|
||||
learned_store,
|
||||
"vision:" + (urlparse(self.vision_url).hostname or "default"),
|
||||
self.vision_model,
|
||||
"chat_completions",
|
||||
upstream_capabilities(self.vision_url, self.vision_dialect).dialect,
|
||||
logger.info,
|
||||
)
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.vision_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
if self.referer:
|
||||
headers["HTTP-Referer"] = self.referer
|
||||
if self.title:
|
||||
headers[TITLE_HEADER] = self.title
|
||||
headers[LEGACY_TITLE_HEADER] = self.title
|
||||
start = time.monotonic()
|
||||
shaped = retrier.method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
|
||||
resp, exc = await self._post_with_retries(client, shaped, headers)
|
||||
first = (resp, exc)
|
||||
resolving = False
|
||||
while resp is not None and retrier.eligible(resp.status_code):
|
||||
method = retrier.next_after(resp.status_code, resp.text)
|
||||
if method is None:
|
||||
break
|
||||
resolving = True
|
||||
shaped = method.apply(payload, shape_cfg, "chat_completions", False, logger.info)
|
||||
resp, exc = await self._post_with_retries(client, shaped, headers)
|
||||
if resolving:
|
||||
if resp is not None and resp.status_code < 400:
|
||||
retrier.succeeded()
|
||||
else:
|
||||
retrier.exhausted()
|
||||
resp, exc = first
|
||||
if self.breaker is not None:
|
||||
if exc is not None or (resp is not None and resp.status_code >= 500):
|
||||
self.breaker.record_failure()
|
||||
|
||||
@@ -56,7 +56,15 @@
|
||||
{% if m.kind == 'image' and m.price_input_per_m %}<span class="gw-badge" title="Flat price per generated image">${{ m.price_input_per_m }}/img</span>{% endif %}
|
||||
{% if m.fallback_model %}<span class="gw-badge" title="Falls back to {{ m.fallback_model }} on failure">fallback: {{ m.fallback_model }}</span>{% endif %}
|
||||
{% if m.tools_mode == 'emulated' %}<span class="gw-badge" title="This model does not support tool calling; tools are emulated through the prompt">tools: emulated</span>{% endif %}
|
||||
{% if not m.context_tier_threshold_tokens and m.off_peak_start_minute is none and not (m.kind == 'image' and m.price_input_per_m) and not m.fallback_model and m.tools_mode != 'emulated' %}<span class="gw-muted">-</span>{% endif %}
|
||||
{% if m.learned_shape %}
|
||||
<span class="gw-badge" title="Request shape learned automatically after the upstream rejected the default">shape: {{ m.learned_shape }}</span>
|
||||
<form method="post" action="/admin/gateway/learned/reset-page" class="gw-inline-form">
|
||||
<input type="hidden" name="provider" value="{{ m.learned_provider }}">
|
||||
<input type="hidden" name="target_model" value="{{ m.target_model }}">
|
||||
<button type="submit" class="admin-btn admin-btn-sm" data-confirm="Forget the learned request shape for "{{ m.target_model }}"?">Reset learned</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
{% if not m.context_tier_threshold_tokens and m.off_peak_start_minute is none and not (m.kind == 'image' and m.price_input_per_m) and not m.fallback_model and m.tools_mode != 'emulated' and not m.learned_shape %}<span class="gw-muted">-</span>{% endif %}
|
||||
</td>
|
||||
<td>
|
||||
{% if m.health_status == 'bad' %}
|
||||
|
||||
@@ -119,6 +119,7 @@ Every state-changing action in DevPlace records one append-only row through `dev
|
||||
| `ai.gateway.call` | `services/openai_gateway/gateway.py`, `services/openai_gateway/usage.py` |
|
||||
| `ai.gateway_auth_throttle.tripped` | `services/openai_gateway/service.py` |
|
||||
| `ai.quota.exceeded` | `routers/devii.py`, `services/openai_gateway/service.py` |
|
||||
| `gateway.learned.reset` | `routers/admin/gateway_configs.py` |
|
||||
| `gateway.model.delete` | `routers/admin/gateway_configs.py` |
|
||||
| `gateway.model.update` | `routers/admin/gateway_configs.py` |
|
||||
| `gateway.provider.delete` | `routers/admin/gateway_configs.py` |
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "devplacepy"
|
||||
version = "1.1.22"
|
||||
version = "1.1.23"
|
||||
description = "DevPlace - The Developer Social Network"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import time
|
||||
|
||||
import requests
|
||||
|
||||
from devplacepy.database import refresh_snapshot
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import ShapeMethod
|
||||
from tests.api.admin.gateway.index import (
|
||||
JSON_gateway,
|
||||
admin_session,
|
||||
member_key,
|
||||
_unique_gateway,
|
||||
)
|
||||
from tests.conftest import BASE_URL
|
||||
|
||||
LIST_URL = f"{BASE_URL}/admin/gateway/learned"
|
||||
RESET_URL = f"{BASE_URL}/admin/gateway/learned/reset"
|
||||
SYNC_SECONDS = 1.5
|
||||
|
||||
|
||||
def _seed(provider, model, method):
|
||||
learned_store.remember(provider, model, method)
|
||||
time.sleep(SYNC_SECONDS)
|
||||
|
||||
|
||||
def test_learned_endpoints_require_admin(seeded_db):
|
||||
assert requests.get(LIST_URL, headers=JSON_gateway, allow_redirects=False).status_code == 401
|
||||
assert (
|
||||
requests.post(RESET_URL, json={}, headers=JSON_gateway, allow_redirects=False).status_code
|
||||
== 401
|
||||
)
|
||||
key = member_key()
|
||||
member = {**JSON_gateway, "X-API-KEY": key}
|
||||
assert requests.get(LIST_URL, headers=member, allow_redirects=False).status_code == 403
|
||||
assert (
|
||||
requests.post(RESET_URL, json={}, headers=member, allow_redirects=False).status_code == 403
|
||||
)
|
||||
|
||||
|
||||
def test_admin_lists_a_learned_method(seeded_db):
|
||||
provider = _unique_gateway("lprov")
|
||||
_seed(provider, "vendor/learned", ShapeMethod("none", ("stream_options",)))
|
||||
try:
|
||||
payload = admin_session(seeded_db).get(LIST_URL).json()
|
||||
row = next(r for r in payload["learned"] if r["provider"] == provider)
|
||||
assert row["target_model"] == "vendor/learned"
|
||||
assert row["thinking_dialect"] == "none"
|
||||
assert row["drop_keys"] == ["stream_options"]
|
||||
assert set(payload["stats"]) == {"resolved", "exhausted", "healed"}
|
||||
assert payload["count"] >= 1
|
||||
finally:
|
||||
learned_store.forget(provider, "vendor/learned")
|
||||
|
||||
|
||||
def test_admin_resets_a_learned_method_and_it_is_audited(seeded_db):
|
||||
provider = _unique_gateway("lprov")
|
||||
_seed(provider, "vendor/reset", ShapeMethod("none"))
|
||||
admin = admin_session(seeded_db)
|
||||
response = admin.post(RESET_URL, json={"provider": provider, "target_model": "vendor/reset"})
|
||||
assert response.status_code == 200
|
||||
assert response.json()["ok"] is True
|
||||
refresh_snapshot()
|
||||
entries = admin.get(
|
||||
f"{BASE_URL}/admin/audit-log",
|
||||
headers=JSON_gateway,
|
||||
params={"event_key": "gateway.learned.reset"},
|
||||
).json()["entries"]
|
||||
assert any(e.get("target_type") == "gateway_learned_method" for e in entries)
|
||||
again = admin.post(RESET_URL, json={"provider": provider, "target_model": "vendor/reset"})
|
||||
assert again.status_code == 404
|
||||
|
||||
|
||||
def test_reset_validates_the_target_model(seeded_db):
|
||||
response = admin_session(seeded_db).post(RESET_URL, json={"provider": "x"})
|
||||
assert response.status_code in (400, 422)
|
||||
@@ -299,3 +299,50 @@ def test_image_test_opens_result_modal(page, seeded_db):
|
||||
httpd.shutdown()
|
||||
requests.delete(f"{BASE_URL}/admin/gateway/models/{source}", headers=auth)
|
||||
requests.delete(f"{BASE_URL}/admin/gateway/providers/{provider}", headers=auth)
|
||||
|
||||
|
||||
def test_model_grid_shows_a_learned_shape_badge_and_reset(page, seeded_db):
|
||||
import time
|
||||
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import ShapeMethod
|
||||
|
||||
user = seeded_db["alice"]
|
||||
_promote_to_admin(user["username"])
|
||||
key = _admin_api_key(user["username"])
|
||||
auth = {"X-API-KEY": key}
|
||||
provider = f"e2elearn{int(time.time() * 1000)}"
|
||||
source = f"{provider}-src"
|
||||
try:
|
||||
requests.post(
|
||||
f"{BASE_URL}/admin/gateway/providers",
|
||||
json={"name": provider, "base_url": "http://127.0.0.1:9/v1/chat/completions"},
|
||||
headers=auth,
|
||||
)
|
||||
requests.post(
|
||||
f"{BASE_URL}/admin/gateway/models",
|
||||
json={
|
||||
"source_model": source,
|
||||
"provider": provider,
|
||||
"target_model": "vendor/learned",
|
||||
"kind": "chat",
|
||||
},
|
||||
headers=auth,
|
||||
)
|
||||
learned_store.remember(provider, "vendor/learned", ShapeMethod("none"))
|
||||
time.sleep(1.5)
|
||||
login_user(page, user)
|
||||
page.goto(f"{BASE_URL}/admin/gateway?tab=models", wait_until="domcontentloaded")
|
||||
row = page.locator("table.admin-table tr", has_text=source)
|
||||
row.wait_for(state="visible")
|
||||
assert "shape: none" in (row.inner_text() or "")
|
||||
row.get_by_role("button", name="Reset learned", exact=True).click()
|
||||
page.locator(".dialog-overlay.visible .dialog-confirm").click()
|
||||
page.wait_for_url(f"{BASE_URL}/admin/gateway?tab=models", wait_until="domcontentloaded")
|
||||
row = page.locator("table.admin-table tr", has_text=source)
|
||||
row.wait_for(state="visible")
|
||||
assert "shape: none" not in (row.inner_text() or "")
|
||||
finally:
|
||||
learned_store.forget(provider, "vendor/learned")
|
||||
requests.delete(f"{BASE_URL}/admin/gateway/models/{source}", headers=auth)
|
||||
requests.delete(f"{BASE_URL}/admin/gateway/providers/{provider}", headers=auth)
|
||||
|
||||
@@ -3441,3 +3441,240 @@ def test_native_route_keeps_forwarding_tools(local_db, monkeypatch):
|
||||
assert "<tools>" not in sent["instructions"]
|
||||
finally:
|
||||
_teardown_responses_route_openai_gateway()
|
||||
|
||||
|
||||
VAIDS_REJECTION_openai_gateway = {
|
||||
"details": {"_errors": ["Unrecognized key(s) in object: 'thinking'"]},
|
||||
"error": "Invalid request parameters",
|
||||
"issues": [
|
||||
{
|
||||
"code": "unrecognized_keys",
|
||||
"keys": ["thinking"],
|
||||
"path": [],
|
||||
"message": "Unrecognized key(s) in object: 'thinking'",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _rejection_openai_gateway(key):
|
||||
return {
|
||||
"details": {"_errors": [f"Unrecognized key(s) in object: '{key}'"]},
|
||||
"error": "Invalid request parameters",
|
||||
"issues": [{"code": "unrecognized_keys", "keys": [key], "path": []}],
|
||||
}
|
||||
|
||||
|
||||
class FakeStrictClient_openai_gateway:
|
||||
rejected = ("thinking",)
|
||||
|
||||
def __init__(self, *a, **k):
|
||||
self.calls = []
|
||||
|
||||
def build_request(self, method, url, headers=None, json=None, content=None):
|
||||
return FakeRequest(method, url, json)
|
||||
|
||||
async def send(self, request, stream=False):
|
||||
body = request.json_body or {}
|
||||
self.calls.append((request.url, dict(body)))
|
||||
for key in self.rejected:
|
||||
if key in body:
|
||||
return FakeResp_openai_gateway(
|
||||
status=400, payload=_rejection_openai_gateway(key)
|
||||
)
|
||||
return FakeResp_openai_gateway(
|
||||
payload={
|
||||
"id": "x",
|
||||
"model": body.get("model"),
|
||||
"choices": [{"message": {"content": "hi there"}}],
|
||||
}
|
||||
)
|
||||
|
||||
async def aclose(self):
|
||||
pass
|
||||
|
||||
|
||||
class FakeStubbornClient_openai_gateway(FakeStrictClient_openai_gateway):
|
||||
async def send(self, request, stream=False):
|
||||
self.calls.append((request.url, dict(request.json_body or {})))
|
||||
return FakeResp_openai_gateway(
|
||||
status=400, payload=_rejection_openai_gateway("thinking")
|
||||
)
|
||||
|
||||
|
||||
class FakeReasoningStrictClient_openai_gateway(FakeStrictClient_openai_gateway):
|
||||
rejected = ("reasoning",)
|
||||
|
||||
|
||||
def _shape_cfg_openai_gateway():
|
||||
svc = GatewayService()
|
||||
cfg = svc.effective_config()
|
||||
cfg["gateway_vision_enabled"] = False
|
||||
cfg["gateway_max_retries"] = 0
|
||||
return svc, cfg
|
||||
|
||||
|
||||
def _chat_openai_gateway(rt, cfg, owner, stream=False):
|
||||
body = {"messages": [{"role": "user", "content": "hi"}]}
|
||||
if stream:
|
||||
body["stream"] = True
|
||||
return run_async(rt.handle_chat(body, cfg, ("guest", owner), "test", "default"))
|
||||
|
||||
|
||||
def test_chat_auto_resolves_a_rejected_thinking_key_and_remembers_it(local_db, monkeypatch):
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
|
||||
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeStrictClient_openai_gateway)
|
||||
svc, cfg = _shape_cfg_openai_gateway()
|
||||
model = cfg["gateway_model"]
|
||||
learned_store.forget("default", model)
|
||||
try:
|
||||
rt = svc.runtime()
|
||||
response = _chat_openai_gateway(rt, cfg, "shape_resolve_first")
|
||||
assert response.status_code == 200
|
||||
assert len(rt._client.calls) == 2
|
||||
assert "thinking" in rt._client.calls[0][1]
|
||||
assert "thinking" not in rt._client.calls[1][1]
|
||||
learned = learned_store.get("default", model)
|
||||
assert learned is not None and learned.thinking_dialect == "none"
|
||||
row = get_table("gateway_usage_ledger").find_one(owner_id="shape_resolve_first")
|
||||
assert row["resolved_method"] == "none"
|
||||
assert row["success"] == 1
|
||||
second = _chat_openai_gateway(rt, cfg, "shape_resolve_second")
|
||||
assert second.status_code == 200
|
||||
assert len(rt._client.calls) == 3
|
||||
assert "thinking" not in rt._client.calls[2][1]
|
||||
finally:
|
||||
learned_store.forget("default", model)
|
||||
|
||||
|
||||
def test_chat_auto_resolve_works_for_streams(local_db, monkeypatch):
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
|
||||
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeStrictClient_openai_gateway)
|
||||
svc, cfg = _shape_cfg_openai_gateway()
|
||||
model = cfg["gateway_model"]
|
||||
learned_store.forget("default", model)
|
||||
try:
|
||||
rt = svc.runtime()
|
||||
response = _chat_openai_gateway(rt, cfg, "shape_resolve_stream", stream=True)
|
||||
|
||||
async def drain():
|
||||
out = []
|
||||
async for chunk in response.body_iterator:
|
||||
out.append(chunk if isinstance(chunk, str) else chunk.decode())
|
||||
return "".join(out)
|
||||
|
||||
assert "[DONE]" in run_async(drain())
|
||||
assert len(rt._client.calls) == 2
|
||||
assert learned_store.get("default", model) is not None
|
||||
finally:
|
||||
learned_store.forget("default", model)
|
||||
|
||||
|
||||
def test_chat_genuine_bad_request_is_not_auto_resolved(local_db, monkeypatch):
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
|
||||
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeBadRequestClient_openai_gateway)
|
||||
svc, cfg = _shape_cfg_openai_gateway()
|
||||
model = cfg["gateway_model"]
|
||||
learned_store.forget("default", model)
|
||||
rt = svc.runtime()
|
||||
response = _chat_openai_gateway(rt, cfg, "shape_genuine_bad")
|
||||
assert response.status_code == 400
|
||||
assert len(rt._client.calls) == 1
|
||||
assert learned_store.get("default", model) is None
|
||||
|
||||
|
||||
def test_chat_exhausted_resolution_returns_the_original_error_and_stores_nothing(
|
||||
local_db, monkeypatch
|
||||
):
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import MAX_RESOLVE_ATTEMPTS
|
||||
|
||||
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeStubbornClient_openai_gateway)
|
||||
svc, cfg = _shape_cfg_openai_gateway()
|
||||
model = cfg["gateway_model"]
|
||||
learned_store.forget("default", model)
|
||||
rt = svc.runtime()
|
||||
response = _chat_openai_gateway(rt, cfg, "shape_exhausted")
|
||||
assert response.status_code == 400
|
||||
assert "thinking" in response.body.decode()
|
||||
assert len(rt._client.calls) <= 1 + MAX_RESOLVE_ATTEMPTS
|
||||
assert learned_store.get("default", model) is None
|
||||
|
||||
|
||||
def test_chat_rejected_learned_method_is_forgotten_and_ladder_restarts(local_db, monkeypatch):
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import ShapeMethod
|
||||
|
||||
monkeypatch.setattr(
|
||||
gwmod.httpx, "AsyncClient", FakeReasoningStrictClient_openai_gateway
|
||||
)
|
||||
svc, cfg = _shape_cfg_openai_gateway()
|
||||
model = cfg["gateway_model"]
|
||||
learned_store.remember("default", model, ShapeMethod("openrouter"))
|
||||
try:
|
||||
rt = svc.runtime()
|
||||
response = _chat_openai_gateway(rt, cfg, "shape_heal")
|
||||
assert response.status_code == 200
|
||||
assert "reasoning" in rt._client.calls[0][1]
|
||||
assert "thinking" in rt._client.calls[1][1]
|
||||
assert learned_store.get("default", model) is None
|
||||
finally:
|
||||
learned_store.forget("default", model)
|
||||
|
||||
|
||||
def test_fallback_attempt_is_shaped_with_its_own_learned_method(local_db, monkeypatch):
|
||||
from devplacepy.services.openai_gateway import routing
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import ShapeMethod
|
||||
|
||||
class FakeFallbackShapeClient(FakeStrictClient_openai_gateway):
|
||||
rejected = ()
|
||||
|
||||
async def send(self, request, stream=False):
|
||||
body = request.json_body or {}
|
||||
self.calls.append((request.url, dict(body)))
|
||||
if body.get("model") == "shape-primary-target":
|
||||
return FakeResp_openai_gateway(status=503, payload={"error": {"message": "down"}})
|
||||
return FakeResp_openai_gateway(
|
||||
payload={
|
||||
"id": "x",
|
||||
"model": body.get("model"),
|
||||
"choices": [{"message": {"content": "ok"}}],
|
||||
}
|
||||
)
|
||||
|
||||
monkeypatch.setattr(gwmod.httpx, "AsyncClient", FakeFallbackShapeClient)
|
||||
routing.model_store.set(
|
||||
routing.ModelRouteIn(source_model="shape-backup", target_model="shape-backup-target")
|
||||
)
|
||||
routing.model_store.set(
|
||||
routing.ModelRouteIn(
|
||||
source_model="shape-primary",
|
||||
target_model="shape-primary-target",
|
||||
fallback_model="shape-backup",
|
||||
)
|
||||
)
|
||||
learned_store.remember("default", "shape-backup-target", ShapeMethod("none"))
|
||||
try:
|
||||
svc, cfg = _shape_cfg_openai_gateway()
|
||||
rt = svc.runtime()
|
||||
response = run_async(
|
||||
rt.handle_chat(
|
||||
{"model": "shape-primary", "messages": [{"role": "user", "content": "hi"}]},
|
||||
cfg,
|
||||
("guest", "shape_fallback"),
|
||||
"test",
|
||||
"default",
|
||||
)
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert "thinking" in rt._client.calls[0][1]
|
||||
assert rt._client.calls[-1][1]["model"] == "shape-backup-target"
|
||||
assert "thinking" not in rt._client.calls[-1][1]
|
||||
finally:
|
||||
learned_store.forget("default", "shape-backup-target")
|
||||
routing.model_store.remove("shape-primary")
|
||||
routing.model_store.remove("shape-backup")
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from devplacepy.services.openai_gateway import learned
|
||||
from devplacepy.services.openai_gateway.learned import learned_store
|
||||
from devplacepy.services.openai_gateway.shape import NATIVE_METHOD, ShapeMethod
|
||||
|
||||
|
||||
def test_remember_get_list_and_forget(local_db):
|
||||
learned_store.forget("lt-prov", "lt-model")
|
||||
method = ShapeMethod("none", ("stream_options",))
|
||||
learned_store.remember("LT-Prov", "lt-model", method)
|
||||
try:
|
||||
assert learned_store.get("lt-prov", "lt-model") == method
|
||||
rows = [r for r in learned_store.list() if r["target_model"] == "lt-model"]
|
||||
assert rows[0]["provider"] == "lt-prov"
|
||||
assert rows[0]["drop_keys"] == ["stream_options"]
|
||||
assert learned_store.forget("lt-prov", "lt-model") is True
|
||||
assert learned_store.get("lt-prov", "lt-model") is None
|
||||
assert learned_store.forget("lt-prov", "lt-model") is False
|
||||
finally:
|
||||
learned_store.forget("lt-prov", "lt-model")
|
||||
|
||||
|
||||
def test_blank_provider_maps_to_default_and_native_is_never_stored(local_db):
|
||||
learned_store.forget("default", "lt-default-model")
|
||||
learned_store.remember("", "lt-default-model", NATIVE_METHOD)
|
||||
assert learned_store.get("default", "lt-default-model") is None
|
||||
learned_store.remember("", "lt-default-model", ShapeMethod("none"))
|
||||
try:
|
||||
assert learned_store.get("default", "lt-default-model") == ShapeMethod("none")
|
||||
finally:
|
||||
learned_store.forget("default", "lt-default-model")
|
||||
|
||||
|
||||
def test_remember_upserts_on_the_unique_route(local_db):
|
||||
learned_store.forget("lt-up", "lt-m")
|
||||
try:
|
||||
learned_store.remember("lt-up", "lt-m", ShapeMethod("none"))
|
||||
learned_store.remember("lt-up", "lt-m", ShapeMethod("openrouter"))
|
||||
assert learned_store.get("lt-up", "lt-m") == ShapeMethod("openrouter")
|
||||
assert len([r for r in learned_store.list() if r["provider"] == "lt-up"]) == 1
|
||||
finally:
|
||||
learned_store.forget("lt-up", "lt-m")
|
||||
|
||||
|
||||
def test_forget_provider_clears_only_that_provider(local_db):
|
||||
learned_store.remember("lt-fp", "a", ShapeMethod("none"))
|
||||
learned_store.remember("lt-fp", "b", ShapeMethod("none"))
|
||||
learned_store.remember("lt-other", "a", ShapeMethod("none"))
|
||||
try:
|
||||
assert learned_store.forget_provider("lt-fp") == 2
|
||||
assert learned_store.get("lt-fp", "a") is None
|
||||
assert learned_store.get("lt-other", "a") is not None
|
||||
finally:
|
||||
learned_store.forget_provider("lt-fp")
|
||||
learned_store.forget_provider("lt-other")
|
||||
|
||||
|
||||
def test_row_cap_replaces_the_oldest(local_db, monkeypatch):
|
||||
monkeypatch.setattr(learned, "MAX_LEARNED_ROWS", 2)
|
||||
keys = [("lt-cap", f"m{i}") for i in range(3)]
|
||||
try:
|
||||
for provider, model in keys:
|
||||
learned_store.remember(provider, model, ShapeMethod("none"))
|
||||
present = [k for k in keys if learned_store.get(*k) is not None]
|
||||
assert len(present) <= 2
|
||||
assert learned_store.get(*keys[2]) is not None
|
||||
finally:
|
||||
learned_store.forget_provider("lt-cap")
|
||||
|
||||
|
||||
def test_provider_endpoint_edit_forgets_learned_methods(local_db):
|
||||
from devplacepy.services.openai_gateway import routing
|
||||
|
||||
routing.provider_store.set(
|
||||
routing.ProviderIn(name="lt-edit", base_url="https://a.example/v1/chat/completions")
|
||||
)
|
||||
learned_store.remember("lt-edit", "m", ShapeMethod("none"))
|
||||
try:
|
||||
routing.provider_store.set(
|
||||
routing.ProviderIn(name="lt-edit", base_url="https://a.example/v1/chat/completions")
|
||||
)
|
||||
assert learned_store.get("lt-edit", "m") is not None
|
||||
routing.provider_store.set(
|
||||
routing.ProviderIn(name="lt-edit", base_url="https://b.example/v1/chat/completions")
|
||||
)
|
||||
assert learned_store.get("lt-edit", "m") is None
|
||||
finally:
|
||||
routing.provider_store.remove("lt-edit")
|
||||
learned_store.forget_provider("lt-edit")
|
||||
@@ -0,0 +1,174 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import json
|
||||
|
||||
from devplacepy.services.openai_gateway.shape import (
|
||||
MAX_RESOLVE_ATTEMPTS,
|
||||
NATIVE_METHOD,
|
||||
RejectedKeys,
|
||||
ShapeMethod,
|
||||
ShapeResolver,
|
||||
ShapeRetrier,
|
||||
)
|
||||
|
||||
VAIDS_BODY = json.dumps(
|
||||
{
|
||||
"details": {"_errors": ["Unrecognized key(s) in object: 'thinking'"]},
|
||||
"error": "Invalid request parameters",
|
||||
"issues": [
|
||||
{
|
||||
"code": "unrecognized_keys",
|
||||
"keys": ["thinking"],
|
||||
"path": [],
|
||||
"message": "Unrecognized key(s) in object: 'thinking'",
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class MemoryStore_openai_gateway:
|
||||
def __init__(self):
|
||||
self.rows = {}
|
||||
self.forgotten = []
|
||||
|
||||
def get(self, provider, model):
|
||||
return self.rows.get((provider, model))
|
||||
|
||||
def remember(self, provider, model, method):
|
||||
self.rows[(provider, model)] = method
|
||||
|
||||
def forget(self, provider, model):
|
||||
self.forgotten.append((provider, model))
|
||||
return self.rows.pop((provider, model), None) is not None
|
||||
|
||||
|
||||
def test_parse_real_vaids_body():
|
||||
assert RejectedKeys.parse(VAIDS_BODY) == ("thinking",)
|
||||
|
||||
|
||||
def test_parse_openai_style_bodies():
|
||||
assert RejectedKeys.parse(
|
||||
'{"error":{"message":"Unrecognized request argument supplied: foo","param":null}}'
|
||||
) == ("foo",)
|
||||
assert RejectedKeys.parse(
|
||||
'{"error":{"message":"Unknown parameter: \'stream_options\'.",'
|
||||
'"param":"stream_options","code":"unknown_parameter"}}'
|
||||
) == ("stream_options",)
|
||||
|
||||
|
||||
def test_parse_serde_style_and_plain_text():
|
||||
assert RejectedKeys.parse("unknown field `thinking`, expected one of `model`") == (
|
||||
"thinking",
|
||||
)
|
||||
|
||||
|
||||
def test_parse_returns_empty_for_a_genuine_bad_request():
|
||||
assert RejectedKeys.parse('{"error":{"message":"invalid request: bad param"}}') == ()
|
||||
assert RejectedKeys.parse("") == ()
|
||||
assert RejectedKeys.parse(None) == ()
|
||||
|
||||
|
||||
def test_resolver_prefers_none_when_thinking_is_rejected():
|
||||
resolver = ShapeResolver("deepseek", "chat_completions")
|
||||
method = resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("thinking",))
|
||||
assert method == ShapeMethod("none")
|
||||
|
||||
|
||||
def test_resolver_skips_dialects_that_emit_the_rejected_key():
|
||||
resolver = ShapeResolver("deepseek", "chat_completions")
|
||||
tried = {NATIVE_METHOD, ShapeMethod("none")}
|
||||
method = resolver.next_method(tried, ShapeMethod("none"), ("reasoning",))
|
||||
assert method == ShapeMethod("deepseek")
|
||||
|
||||
|
||||
def test_resolver_adds_a_droppable_key():
|
||||
resolver = ShapeResolver("deepseek", "chat_completions")
|
||||
method = resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("stream_options",))
|
||||
assert method == ShapeMethod("native", ("stream_options",))
|
||||
|
||||
|
||||
def test_resolver_refuses_unknown_and_protected_keys():
|
||||
resolver = ShapeResolver("deepseek", "chat_completions")
|
||||
for key in ("messages", "model", "tools", "something_else"):
|
||||
assert resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, (key,)) is None
|
||||
|
||||
|
||||
def test_resolver_responses_ladder_is_reasoning_only():
|
||||
resolver = ShapeResolver("deepseek", "responses")
|
||||
method = resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("reasoning",))
|
||||
assert method == ShapeMethod("none")
|
||||
assert resolver.next_method({NATIVE_METHOD}, NATIVE_METHOD, ("thinking",)) is None
|
||||
|
||||
|
||||
def test_method_json_round_trip_and_validation():
|
||||
method = ShapeMethod("none", ("stream_options",))
|
||||
assert ShapeMethod.from_json(method.to_json()) == method
|
||||
assert ShapeMethod.from_json('{"thinking_dialect":"bogus","drop_keys":[]}') is None
|
||||
assert ShapeMethod.from_json('{"thinking_dialect":"none","drop_keys":["tools"]}') is None
|
||||
assert ShapeMethod.from_json("not json") is None
|
||||
|
||||
|
||||
def test_apply_none_clears_thinking_and_native_writes_the_dialect():
|
||||
cfg = {"gateway_upstream_url": "https://strict.example/v1/chat/completions"}
|
||||
payload = {"model": "m", "messages": [], "thinking": {"type": "enabled"}}
|
||||
native = NATIVE_METHOD.apply(payload, cfg, "chat_completions", False, lambda m: None)
|
||||
assert native["thinking"] == {"type": "enabled"}
|
||||
none = ShapeMethod("none").apply(payload, cfg, "chat_completions", False, lambda m: None)
|
||||
assert "thinking" not in none and "reasoning" not in none
|
||||
assert payload["thinking"] == {"type": "enabled"}
|
||||
|
||||
|
||||
def test_apply_drops_stream_options_when_learned():
|
||||
cfg = {"gateway_upstream_url": "https://strict.example/v1/chat/completions"}
|
||||
payload = {"model": "m", "messages": [], "stream": True}
|
||||
kept = NATIVE_METHOD.apply(payload, cfg, "chat_completions", True, lambda m: None)
|
||||
assert kept["stream_options"] == {"include_usage": True}
|
||||
dropped = ShapeMethod("native", ("stream_options",)).apply(
|
||||
payload, cfg, "chat_completions", True, lambda m: None
|
||||
)
|
||||
assert "stream_options" not in dropped
|
||||
|
||||
|
||||
def test_strip_wire_removes_reasoning_for_responses_none():
|
||||
body = {"model": "m", "reasoning": {"effort": "minimal"}, "input": []}
|
||||
stripped = ShapeMethod("none").strip_wire(body, "responses")
|
||||
assert "reasoning" not in stripped and body["reasoning"]
|
||||
assert NATIVE_METHOD.strip_wire(body, "responses") is body
|
||||
|
||||
|
||||
def test_retrier_success_remembers_and_native_success_does_not():
|
||||
store = MemoryStore_openai_gateway()
|
||||
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
|
||||
retrier.succeeded()
|
||||
assert store.rows == {}
|
||||
method = retrier.next_after(400, VAIDS_BODY)
|
||||
assert method == ShapeMethod("none")
|
||||
retrier.succeeded()
|
||||
assert store.rows[("p", "m")] == ShapeMethod("none")
|
||||
|
||||
|
||||
def test_retrier_ignores_non_shape_statuses_and_unparsable_bodies():
|
||||
store = MemoryStore_openai_gateway()
|
||||
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
|
||||
assert retrier.next_after(500, VAIDS_BODY) is None
|
||||
assert retrier.next_after(400, '{"error":"nope"}') is None
|
||||
|
||||
|
||||
def test_retrier_attempts_are_capped():
|
||||
store = MemoryStore_openai_gateway()
|
||||
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
|
||||
retrier.attempts = MAX_RESOLVE_ATTEMPTS
|
||||
assert retrier.next_after(400, VAIDS_BODY) is None
|
||||
|
||||
|
||||
def test_retrier_forgets_a_rejected_learned_method_once():
|
||||
store = MemoryStore_openai_gateway()
|
||||
store.rows[("p", "m")] = ShapeMethod("openrouter")
|
||||
retrier = ShapeRetrier.for_route(store, "p", "m", "chat_completions", "deepseek", lambda m: None)
|
||||
assert retrier.method == ShapeMethod("openrouter")
|
||||
method = retrier.next_after(
|
||||
400, '{"issues":[{"code":"unrecognized_keys","keys":["reasoning"]}]}'
|
||||
)
|
||||
assert method == NATIVE_METHOD
|
||||
assert store.forgotten == [("p", "m")]
|
||||
Reference in New Issue
Block a user