Add OpenCode Zen support, model health/stats dashboard, and gateway fallback fixes

AI gateway:
- Add a generic, admin-selectable `client_profile` field on gateway_providers
  (e.g. "opencode") so a provider needing special request headers (OpenCode
  Zen's client-identity spoofing) is configured like any other provider, not
  hardcoded by name.
- Track per-(provider, model) reliability/speed/latency health in memory,
  seeded from the existing gateway_usage_ledger at startup - purely
  observational, never influences routing.
- New Stats tab on /admin/gateway: request volume, latency, per-model
  breakdowns, and reliability weight, charted with a vendored Chart.js and
  devplace's own theme tokens.
- Record which model a failed request actually fell back to
  (fallback_used_route), surfaced in the Recent Failures table.
- Stop excluding context_length errors from fallback, and skip a primary
  attempt outright when its known context window is already too small for
  the estimated request size, going straight to the fallback.
- gateway_usage_ledger's provider/fallback_used_route columns and indexes
  are ensured centrally in database/schema.py's init_db(), the single point
  of truth for this table's schema.
- Non-OpenAI upstream routing and client-model passthrough; trust only the
  upstream's own X-Gateway-Model header for served-model attribution.

Devii agent:
- Fix a real lockup: plan/verify tools could be individually disabled via
  the admin tool toggles while still being required by the protocol gate,
  permanently bricking any task that needed tools. They can no longer be
  disabled, and the gate now also checks the tool is actually offered.
- Fix compaction being silently calibrated for a 1M-token model while
  running a much smaller one: context budget is now percentage-based and
  the summarizer's own request is sized to fit the real model.
- Give a specific, actionable retry message when plan()'s own arguments get
  cut off by the output limit, and tighten its schema to discourage
  overlong plans.

Other:
- Backup service: offload completed backups to a remote Hetzner Storage Box.
- Container manager: fix orphan blob leaks from sync races, add a two-phase
  plan/execute `system prune` CLI command.
- Admin gateway UI: replace the JS-rendered model/provider tables with
  server-rendered forms and pages.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DhmEkvutuwtzFVcLbTrhdo
This commit is contained in:
2026-09-09 07:38:24 +02:00
co-authored by Claude Sonnet 5
parent b475e7d6ed
commit 569f1dcc64
67 changed files with 6151 additions and 814 deletions
+30 -5
View File
@@ -117,7 +117,9 @@ The gateway records one row per upstream call (chat, vision, passthrough) and su
**The two AI quotas are separate systems and the reset surfaces must say so.** `/admin/ai-usage`'s *Reset all quotas* clears the Devii `devii_usage_ledger` AND now also stamps a global gateway watermark, because a caller hitting `429 AI gateway daily quota exceeded` had no reset at all before and the button looked global. *Reset guest quotas* stays Devii-only (guest gateway calls ride the shared internal key, so there is no per-guest gateway scope to clear).
**CRUD.** Admin JSON at `/admin/gateway/quota-rules` (`routers/admin/gateway_configs.py`, list returns each rule's live `spent_24h_usd` plus the Layer A defaults for context), audited `gateway.quota_rule.update`/`gateway.quota_rule.delete` (category `ai`, both already in `events.md`), rendered in the **Quota rules** section of `/admin/gateway` (`GatewayAdmin.js`, mirrors the providers/models CRUD tables). Devii tools `gateway_quota_rules`/`gateway_quota_rule_set`/`gateway_quota_rule_delete` (`requires_admin=True`, delete is `CONFIRM_REQUIRED`) proxy the same endpoints via `handler="http"`, same as the provider/model tools. Reset is `POST /admin/gateway/quota-resets` (same file, `_payload`/`ValidationError` shape as the rule CRUD), audited `gateway.quota.reset` (category `ai`), surfaced as a per-rule **Reset spend** button in the Quota rules table (`GatewayAdmin.js`), and exposed as the Devii tool `gateway_quota_reset` (`requires_admin=True`, in `CONFIRM_REQUIRED` with a declared `confirm` param, like the other quota-lifting admin resets). CLI: `devplace gateway quota list|set|delete|reset`.
**CRUD.** Admin JSON at `/admin/gateway/quota-rules` (`routers/admin/gateway_configs.py`, list returns each rule's live `spent_24h_usd` plus the Layer A defaults for context), audited `gateway.quota_rule.update`/`gateway.quota_rule.delete` (category `ai`, both already in `events.md`). Devii tools `gateway_quota_rules`/`gateway_quota_rule_set`/`gateway_quota_rule_delete` (`requires_admin=True`, delete is `CONFIRM_REQUIRED`) proxy the same JSON endpoints via `handler="http"`, same as the provider/model tools. Reset is `POST /admin/gateway/quota-resets` (same file, `_payload`/`ValidationError` shape as the rule CRUD), audited `gateway.quota.reset` (category `ai`), and exposed as the Devii tool `gateway_quota_reset` (`requires_admin=True`, in `CONFIRM_REQUIRED` with a declared `confirm` param, like the other quota-lifting admin resets). CLI: `devplace gateway quota list|set|delete|reset`.
The human-facing **Quota rules** tab of `/admin/gateway` is a third, independent backend-rendered surface over the same `quota_rule_store` (see "Admin UI (backend-rendered, no JS)" above): `GET/POST /admin/gateway/quota-rules/new`, `GET/POST /admin/gateway/quota-rules/{uid}/edit`, `POST /admin/gateway/quota-rules/{uid}/delete`, and a per-row `POST /admin/gateway/quota-rules/{uid}/reset` button (`templates/admin_gateway_quota_form.html` for add/edit; the reset/delete actions are plain `<form>` buttons with `data-confirm` on the index page, no fetch call). These do not replace the JSON endpoints above - Devii and the CLI still talk to those directly.
## Image generation
@@ -152,21 +154,44 @@ Layered ON TOP of the single-provider service config above, which stays THE impl
**No matching route = None overlay = model fallback.** When no route matches, the overlay returns `None` and the handler's model selection logic falls back to the default configured model (`gateway_model` for chat, `gateway_embed_model` for embeddings, `gateway_image_model` for images). The unknown model name is discarded, not forwarded upstream. This means an unknown or misspelled model name never causes a 4xx from the upstream - it is gracefully downgraded to the default. The `force_model` guard and the built-in `molodetz`/`molodetz~embed`/`molodetz-img-small` aliases are still respected before the route check: `force_model` or empty/named-alias model -> default directly; known model -> route resolution; unknown model with no route -> default fallback with a log message.
**CRUD.** Admin JSON at `/admin/gateway/{providers,models}` (`routers/admin/gateway_configs.py`, `require_admin`, Pydantic `ProviderIn`/`ModelRouteIn` validation, accepts both JSON from `static/js/GatewayAdmin.js` and form from Devii), audited under `gateway.provider.*`/`gateway.model.*` (category `ai`), included in the admin package with the page at `/admin/gateway` (`templates/admin_gateway.html`, sidebar link, `admin_section="gateway"`).
**CRUD.** Admin JSON at `/admin/gateway/{providers,models}` (`routers/admin/gateway_configs.py`, `require_admin`, Pydantic `ProviderIn`/`ModelRouteIn` validation via the shared `_payload()` JSON-or-form reader, `_save_provider`/`_save_model`/`_check_fallback` helpers), audited under `gateway.provider.*`/`gateway.model.*` (category `ai`). This JSON surface is for Devii and programmatic callers only - see "Admin UI (backend-rendered, no JS)" below for the human-facing page, which is a completely separate set of routes over the same stores.
**Devii tools:** `gateway_providers`/`gateway_models` (read) and `gateway_provider_set`/`gateway_provider_delete`/`gateway_model_set`/`gateway_model_delete` (admin; the two deletes are `CONFIRM_REQUIRED`).
## Admin UI (backend-rendered, no JS)
`/admin/gateway` (`templates/admin_gateway.html`, sidebar link, `admin_section="gateway"`) is a plain server-rendered page - no `GatewayAdmin.js` (deleted), no page controller driving CRUD. It is a three-tab index (`?tab=models|providers|quota`, plain `<a>` links, `.admin-tabs`/`.admin-tab`) listing each resource in an `.admin-table`; every add/edit is its **own dedicated page**, never an inline form under the table:
- Providers: `GET/POST /admin/gateway/providers/new`, `GET/POST /admin/gateway/providers/{name}/edit`, `POST /admin/gateway/providers/{name}/delete` (`templates/admin_gateway_provider_form.html`). `name` is the primary key and is rendered `readonly` on the edit page - renaming would silently orphan the old row in `gateway_providers` rather than updating it (`ProviderStore.set` keys on `find_one(name=...)`), so renaming is not offered; delete and re-add instead.
- Model routes: `GET/POST /admin/gateway/models/new`, `GET/POST /admin/gateway/models/{source_model:path}/edit` (path converter, since a source model name is not guaranteed to be slash-free), `POST /admin/gateway/models/{source_model:path}/delete` (`templates/admin_gateway_model_form.html`). `source_model` is likewise `readonly` on edit for the same orphan-row reason. Every field is always rendered (chat pricing, embed/vision pricing, tiered pricing, off-peak window) grouped into `<fieldset>`s by applicability rather than JS-toggled by the selected `kind` - the hint text under each fieldset states which kind it applies to. The fallback-model `<select>` is grouped into one `<optgroup>` per kind (pure HTML, no live filtering); the server still authoritatively rejects a cross-kind fallback via `_check_fallback`. The **target model** field is the one genuinely dynamic control on this page - see "Live target-model lookup" below.
- Quota rules: `GET/POST /admin/gateway/quota-rules/new`, `GET/POST /admin/gateway/quota-rules/{uid}/edit`, `POST /admin/gateway/quota-rules/{uid}/delete`, `POST /admin/gateway/quota-rules/{uid}/reset` (`templates/admin_gateway_quota_form.html`). Unlike providers/models, a quota rule's `uid` is never user-editable, so there is no orphan-row risk and no field needs to be readonly.
### Live target-model lookup (the one dynamic field on this page)
The model-route form's **target model** field is a deliberate, narrow exception to "backend-rendered, no JS": picking a provider can turn it from a free-text input into a `<select>` populated with that provider's own real model list, but the browser never talks to the provider directly.
- **Server does the URL resolution and the outbound call.** `routing._models_url_from_base(base_url)` derives the OpenAI-style listing endpoint from a provider's chat-completions `base_url` (swaps `/chat/completions` -> `/models`, exactly like the existing `_embed_url_from_base`/`_image_url_from_base` derivations), and `routing.fetch_provider_models(provider_name)` (async) resolves the named provider's (or, for a blank name, the **default** provider's - `_default_provider_credentials()` reads `gateway_upstream_url`/`gateway_api_key` off the service config the same way `_default_provider_summary()` does) `base_url`/`api_key`, calls the derived URL through `stealth.stealth_async_client` (never a bare `httpx.AsyncClient` - see the root `CLAUDE.md` outbound-HTTP rule) with an `authorization: Bearer <api_key>` header, and parses the OpenAI `{"data": [{"id": ...}, ...]}` shape into a plain list of id strings. It returns `None` - never raises - for every failure mode: unknown provider, blank/unresolvable base URL, a non-200 response, an unparseable body, an empty list, or any network exception (timeout, connection refused, DNS failure). `None` is the single "can't do this" signal the caller (and the frontend) all key off.
- **`GET /admin/gateway/provider-models?provider=<name-or-blank>`** (`routers/admin/gateway_configs.py::provider_models`, admin-gated, JSON-only, no page) is the one endpoint the browser is allowed to call: `require_admin` then `fetch_provider_models(provider)`, returning `{"provider", "models": [...]}` on success or a `404 {"error": {...}}` (via `json_error`) when the result was `None`. This is the ONLY thing `static/js/GatewayModelForm.js` fetches - it never reaches a third-party host, satisfying "we do not let the JavaScript make a connection to a remote site."
- **`static/js/GatewayModelForm.js`** (loaded only by `admin_gateway_model_form.html`'s `{% block extra_js %}`, not global) is the one page-specific controller left in this subsystem. The template always renders BOTH `#gw-model-target` (a `required` text input, the no-JS/JS-still-loading baseline and the true fallback) and `#gw-model-target-select` (a `required` but initially `hidden disabled` `<select>` with the same `name="target_model"`) - exactly one of the pair is ever `disabled` at a time (a `disabled` field is excluded from form submission, so there is never a duplicate `target_model` value), and the controller toggles `.hidden`/`.disabled` on both together. On the provider `<select>`'s `change` event, and once on page load ONLY when a provider is already selected (an edit page for a named-provider route - a brand-new form or a "default"/blank provider selection never auto-probes on load, since that would fire a real outbound network probe against the default gateway just from opening the page), it calls `Http.getJson("/admin/gateway/provider-models?provider=...")`; a 200 swaps to the select (populated from the response, the field's current value prepended as an extra option if the provider's own list does not contain it, so an existing customized/legacy target is never silently discarded) and a thrown error (404 or any other failure) swaps back to the free-text input. The `<select>`'s own HTML `required` attribute is what makes "not allowed to be empty" hold even with JS disabled-then-re-enabled - no JS-side validation was added.
- **Tests:** `tests/unit/services/openai_gateway/routing.py` (`_models_url_from_base`, `fetch_provider_models` success/unknown-provider/non-200/malformed-payload/empty-list/network-error/no-base-url, all via a fake `stealth_async_client`), `tests/api/admin/gateway/provider_models.py` (the endpoint over a real local `http.server` fake upstream, mirroring the pattern already used by `tests/api/containers.py`'s ingress test), and `tests/e2e/admin/gateway.py` (a real browser swapping the field between a listing-capable and a blind provider, and a full submit through the dynamically-populated select).
**Validation errors re-render the same form, not a silent redirect.** Each POST handler builds the pydantic model itself (`ModelRouteIn`/`ProviderIn`/`QuotaRuleIn`) rather than going through the generic `Depends(json_or_form(...))` + `RequestValidationError` path used by simpler admin forms (e.g. `/admin/settings`) - that generic handler redirects non-auth pages back to the referer with the message dropped, which would silently swallow a genuine validation failure (a bad URL, a cross-kind fallback) on a page whose whole point is surfacing exactly that feedback. `_error_response()` re-renders the form template with a `.gw-error` banner and every submitted value preserved, at `400`.
**Time-of-day fields are converted at the router, not in the browser.** `off_peak_start`/`off_peak_end` are plain `<input type="time">` fields (`HH:MM` strings); `_time_to_minutes`/`_minutes_to_time` in `gateway_configs.py` convert to/from the stored `off_peak_start_minute`/`off_peak_end_minute` integers - this used to be a client-side JS computation (`GatewayAdmin.js` `timeToMinutes`/`minutesToTime`), now server-side only.
**The JSON API above is untouched and still the only surface Devii/`devplace gateway quota` talk to** - the backend-rendered pages are a second, independent set of routes over the same `provider_store`/`model_store`/`quota_rule_store`, sharing the save/delete/fallback-check logic via `_save_provider`/`_save_model`/`_check_fallback`/`_save_quota_rule` helper functions so the two surfaces can never drift on what counts as valid.
When adding a routed value, overlay it as the matching `gateway_*` cfg key so the runtime needs no new branch.
**Tests:** `tests/unit/services/openai_gateway/routing.py` (overlay/economy/kind isolation), `tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream` (end-to-end through `handle_chat`), and `tests/api/admin/gateway/` (admin CRUD + validation + role gating).
## Automatic model fallback (`fallback_model`, one hop, per route)
Any `gateway_models` route (chat, embed, or image) can optionally name `fallback_model`: another already-configured `source_model` of the SAME kind, tried once, automatically, when the primary route fails. This is admin-configured on the `/admin/gateway` model form as a select box populated from every OTHER route of the current kind (the public `source_model` names such as `molodetz`/`molodetz-pro`, never the internal `target_model` a provider actually sees) - never a free-text field, so a fallback can only ever point at a model the gateway already knows how to serve.
Any `gateway_models` route (chat, embed, or image) can optionally name `fallback_model`: another already-configured `source_model` of the SAME kind, tried once, automatically, when the primary route fails. This is admin-configured on the `/admin/gateway` model form as a select box grouped into one `<optgroup>` per kind over every OTHER route (the public `source_model` names such as `molodetz`/`molodetz-pro`, never the internal `target_model` a provider actually sees) - never a free-text field, so a fallback can only ever point at a model the gateway already knows how to serve.
- **Trigger.** `_call_failed(resp, exc, timing)` in `gateway.py` treats a call as failed when the circuit breaker rejected it, the upstream connection raised, or the upstream answered with any status `>= 400`. This runs AFTER `_send`'s own `gateway_max_retries` retries against the primary model are exhausted - a fallback is the next escalation once retrying the SAME model has already given up, not a replacement for that retry loop.
- **One hop, never a chain.** `routing.resolve_fallback(source_model, kind)` resolves the primary route, reads its `fallback_model`, and resolves THAT model's own route (must exist, be active, and share the kind) - it does NOT recurse into the fallback's own `fallback_model`, so there is no possibility of a cycle or an unbounded retry chain. A route that fails even after redirecting to its fallback returns the fallback attempt's own failure to the caller.
- **Self-reference is rejected at write time.** `ModelRouteIn._check_fallback_is_not_self` (pydantic `model_validator`) refuses `fallback_model == source_model`; the admin route handler (`routers/admin/gateway_configs.py::save_model`) additionally rejects a `fallback_model` that does not resolve to an existing, same-kind route (`routing.model_store.get(...)`) before writing the row, since a pydantic validator alone cannot see other rows.
- **Self-reference is rejected at write time.** `ModelRouteIn._check_fallback_is_not_self` (pydantic `model_validator`) refuses `fallback_model == source_model`; `routers/admin/gateway_configs.py::_check_fallback` (shared by the JSON `save_model_json` route and both backend-rendered model-route pages) additionally rejects a `fallback_model` that does not resolve to an existing, same-kind route (`routing.model_store.get(...)`) before writing the row, since a pydantic validator alone cannot see other rows.
- **Rebuilt like a fresh routed call, not retried in place.** On failure, `handle_chat`/`handle_embeddings`/`handle_images` recompute the overlay from `base_cfg` (the pre-primary-overlay config) via `chat_overlay(fallback_route.source_model, base_cfg)`/`embed_overlay`/`image_overlay`, so the fallback gets its OWN provider, URL, key, and pricing - never the primary route's. `pricing`/`context_map` for the ledger are (re)computed AFTER the fallback decision so the recorded cost always reflects whichever model actually served the request.
- **One ledger row per client call, not one per attempt.** The primary failure never writes a `gateway_usage_ledger` row by itself (only `_send`'s own internal retries and the circuit breaker counters observe it); `finalize()`/`self._ledger.record(...)` still runs exactly once, after the fallback decision, with `model` = whichever model ultimately answered and `requested_model` = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in `/admin/ai-usage` as "requested X, served Y" rather than as two calls.
- **Streaming falls back before any byte reaches the client.** `client.send(request, stream=True)` returns as soon as the response headers arrive, so `resp.status_code` is already known before the SSE body is ever touched - `handle_chat` decides whether to fall back at that same point, before it ever opens `_stream_chat_response` to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
@@ -182,7 +207,7 @@ Real providers sometimes charge more than a flat per-1M rate for one component:
- **Per-route fields** (each optional/zero by default = feature off, so an unconfigured route is byte-identical to before): `context_tier_threshold_tokens` (0 disables tiering; when the request's input token count exceeds it, `price_*_per_m_tier2` rates apply instead of the tier-1 rates above) and the four nullable `price_cache_hit_per_m_tier2`/`price_cache_miss_per_m_tier2`/`price_output_per_m_tier2`/`price_input_per_m_tier2` (a component left `None` keeps its tier-1 rate even above threshold - so a provider that only re-prices input above a size threshold, keeping output flat, needs just one tier2 field set), plus `off_peak_start_minute`/`off_peak_end_minute` (nullable, UTC minutes-since-midnight, both-or-neither enforced by `ModelRouteIn`'s model validator; a window where start > end wraps past midnight, e.g. DeepSeek's old V3/R1-era 16:30-00:30 UTC window) and `off_peak_discount_pct` (0-100, multiplies whichever tier rate is active). The same fields double for vision (`price_input/output_per_m_tier2`) and embed (`price_input_per_m_tier2`) exactly like their tier-1 counterparts already do.
- **Overlay + computation.** `chat_overlay`/`embed_overlay` propagate all of these into the per-request cfg dict under `gateway_*` keys exactly like the existing price fields (section above); `usage.pricing_from_cfg` reads them generically (defaulting to `None`/`0`/disabled when absent), so **Layer A (the single global flat Pricing config fields) never gains these dimensions** - only a `gateway_models` route can enable them, preserving the "no matching route = byte-identical legacy behavior" guarantee. `usage.compute_cost` selects tier1 vs tier2 per rate component (`_tiered_rate`) based on whether `norm["prompt"]` (billable input tokens) exceeds the threshold, then applies the off-peak discount (`_effective_rate`/`_off_peak_active`, UTC wraparound-aware) to whichever rate was selected. **The response header format and `compute_cost`'s return shape (`total, input_cost, output_cost, native`) are unchanged** - this is purely an internal rate-selection step before the existing input/output split math runs; a native upstream `cost` (OpenRouter) still overrides the modeled total exactly as before.
- **Migration.** New `gateway_models` columns are added via `has_column`/`create_column_by_example` in `routing.ensure_tables()` (the `CREATE TABLE IF NOT EXISTS` DDL string alone would never reach a pre-existing table - see the `database/CLAUDE.md` column-ensure idiom).
- **Admin UI.** `/admin/gateway`'s model-route form has a "Tiered / off-peak pricing (optional)" subsection; off-peak start/end render as `<input type="time">` (converted to/from UTC minutes-of-day by `GatewayAdmin.js`), and the routes table shows `tiered`/`off-peak` badges when a route has either dimension configured.
- **Admin UI.** `/admin/gateway`'s model-route add/edit page has a "Tiered pricing" and an "Off-peak discount" fieldset; off-peak start/end render as `<input type="time">` (converted to/from UTC minutes-of-day server-side by `_time_to_minutes`/`_minutes_to_time` in `routers/admin/gateway_configs.py`), and the models tab table shows `tiered`/`off-peak` badges when a route has either dimension configured.
- **DeepSeek's real pricing is already the tier-1 shape, not a new dimension.** DeepSeek's actual API (verified against `api-docs.deepseek.com/quick_start/pricing`) bills three flat per-1M rates - cache-hit input, cache-miss input, output - with no current context-length tier or off-peak window for the V4 models; that shape was already fully modeled by the pre-existing `chat_cache_hit_per_m`/`chat_cache_miss_per_m`/`chat_output_per_m` fields before this section's tier2/off-peak fields existed. `routing.seed_default_deepseek_routes()` (called once from `database.migrate_ai_gateway_settings()` at the end of `init_db()`) idempotently inserts four ready-made routes - `deepseek-v4-flash` (`$0.0028`/`$0.14`/`$0.28` per 1M, 1M context), `deepseek-v4-pro` (`$0.003625`/`$0.435`/`$0.87` per 1M, 1M context), and the two public `molodetz` aliases `molodetz` -> `deepseek-v4-flash` (flash rates) and `molodetz-pro` -> `deepseek-v4-pro` (pro rates) - only when that `source_model` row does not already exist, so a caller or Devii can request any name explicitly and get correctly-priced, decoupled from whatever the single global `gateway_model` default happens to be set to (switching that global setting between the two real models does NOT retroactively fix the flat Pricing config fields - the seeded routes are the model-agnostic, always-correct way to reference a specific priced model). The `molodetz`/`molodetz-pro` aliases are the public model names; `GET /v1/models` is served locally from these `source_model` rows (not proxied upstream), so it publishes exactly the models the gateway accepts. Neither seeded route sets the tier2/off-peak fields (DeepSeek does not use them today); an admin can add them later on the same row if DeepSeek (or any other provider routed here) introduces such pricing.
## Third-party AI consent gate (one place, two owner classes)
+255 -16
View File
@@ -9,6 +9,7 @@ from typing import Optional
import httpx
from fastapi.responses import JSONResponse, Response, StreamingResponse
from devplacepy import config as devplace_config
from devplacepy import stealth
from devplacepy.services.background import background
from devplacepy.services.openai_gateway import config
@@ -27,6 +28,7 @@ from devplacepy.services.openai_gateway.thinking import (
from devplacepy.services.openai_gateway.usage import (
GatewayUsageLedger,
classify_error,
embedded_error_message,
extract_image_usage,
extract_params,
parse_context_map,
@@ -45,6 +47,50 @@ def _call_failed(resp, exc, timing: dict) -> bool:
return resp is not None and resp.status_code >= 400
# A malformed/unsupported request (bad JSON schema, an unsupported param)
# fails identically against a fallback model - retrying it there only doubles
# latency and upstream call count for a guaranteed repeat failure.
# context_length is deliberately NOT in this set: a fallback model commonly
# has a different (often larger) context window than the primary, so it can
# genuinely recover a request the primary couldn't fit - see
# _fits_context_window, which additionally skips a doomed primary attempt
# outright when its own declared window is already known to be too small.
# Every other failure class (timeout, connection error, circuit-open, auth,
# rate limit, 5xx, an embedded 200-with-error body) still falls back too,
# since a different provider/model plausibly can recover.
_NO_FALLBACK_CATEGORIES = {"bad_request"}
def _should_fallback(resp, exc, timing: dict, body_error: Optional[str] = None) -> bool:
if not (_call_failed(resp, exc, timing) or body_error):
return False
if exc is not None or timing.get("circuit_open") or body_error:
return True
if resp is not None:
try:
message = resp.text
except Exception: # noqa: BLE001 - a decode failure is not a reason to skip fallback
message = ""
if classify_error(resp.status_code, None, message) in _NO_FALLBACK_CATEGORIES:
return False
return True
def _embedded_failure_message(resp, stream: bool) -> Optional[str]:
# A 200 with the failure embedded in the JSON body (OpenRouter and the
# providers it fronts do this - see usage.embedded_error_message) is
# invisible to _call_failed's status-code check. Only safe to peek for a
# non-streaming response: a streaming 200's body has not been read yet
# here and must stay unread for real incremental relay.
if stream or resp is None or resp.status_code != 200:
return None
try:
data = resp.json()
except (ValueError, httpx.HTTPError):
return None
return embedded_error_message(data)
def _apply_served_model(headers: dict, resp) -> dict:
# Everything else in `headers` (cost, tokens, latency, context) is our
# own measurement and must never be overwritten by an upstream that
@@ -58,13 +104,58 @@ def _apply_served_model(headers: dict, resp) -> dict:
return headers
def _attribution_headers() -> dict:
# OpenRouter's optional HTTP-Referer/X-Title (openrouter.ai/docs/api-reference/overview)
# attribute calls to DevPlace on their rankings; harmless extra headers for
# any other upstream, so sent unconditionally rather than dialect-gated.
headers: dict = {"X-Title": "DevPlace"}
if devplace_config.SITE_URL:
headers["HTTP-Referer"] = devplace_config.SITE_URL
return headers
def _extra_provider_headers(cfg: dict) -> dict:
# Set by chat_overlay/embed_overlay/image_overlay for a provider that
# needs special request headers (e.g. OpenCode Zen's client-identity
# spoofing) - a no-op dict for every provider that doesn't.
return cfg.get("gateway_extra_request_headers") or {}
def _fallback_headers(cfg: dict, key_field: str) -> dict:
headers = {"Content-Type": "application/json"}
headers = {
"Content-Type": "application/json",
**_attribution_headers(),
**_extra_provider_headers(cfg),
}
if cfg.get(key_field):
headers["Authorization"] = f"Bearer {cfg[key_field]}"
return headers
REQUEST_TOKEN_ESTIMATE_CHARS_PER_TOKEN = 4
REQUEST_TOKEN_ESTIMATE_DEFAULT_MAX_TOKENS = 512
REQUEST_TOKEN_ESTIMATE_SAFETY_MARGIN = 256
def _estimate_request_tokens(payload: dict) -> int:
# A rough chars/4 heuristic, not a real tokenizer - deliberately
# conservative (rounds up via the safety margin) since it only ever
# decides whether to skip a doomed attempt in favor of the fallback,
# never whether to dispatch at all.
body_tokens = len(json.dumps(payload)) // REQUEST_TOKEN_ESTIMATE_CHARS_PER_TOKEN
max_tokens = (
payload.get("max_tokens")
or payload.get("max_completion_tokens")
or REQUEST_TOKEN_ESTIMATE_DEFAULT_MAX_TOKENS
)
return body_tokens + int(max_tokens) + REQUEST_TOKEN_ESTIMATE_SAFETY_MARGIN
def _route_context_window(cfg: dict, model: str) -> Optional[int]:
window = parse_context_map(cfg.get("gateway_model_context_map")).get(model)
return int(window) if window else None
def _notify_gateway_status(is_open: bool) -> None:
from devplacepy.database import get_table
from devplacepy.utils import create_notification
@@ -106,6 +197,14 @@ class GatewayRuntime:
self._breaker = CircuitBreaker(
config.CIRCUIT_THRESHOLD_DEFAULT, config.CIRCUIT_COOLDOWN_SECONDS_DEFAULT
)
# Vision calls default to a different upstream (OpenRouter) than chat
# (DeepSeek) and are not routed through _send, so they get their own
# breaker instance - sharing self._breaker would let a struggling
# vision provider suppress unrelated chat traffic to a healthy
# provider, which is a new coupling this change must not introduce.
self._vision_breaker = CircuitBreaker(
config.CIRCUIT_THRESHOLD_DEFAULT, config.CIRCUIT_COOLDOWN_SECONDS_DEFAULT
)
self.requests = 0
self.errors = 0
self.in_flight = 0
@@ -128,8 +227,17 @@ class GatewayRuntime:
limits = httpx.Limits(
max_connections=instances, max_keepalive_connections=instances
)
# A single blanket timeout makes a dead-on-arrival upstream (DNS
# blackhole, firewalled port, refused connection behind a broken
# LB) take the FULL configured timeout - minutes, since
# gateway_timeout floors at 5 minutes - just to fail the connect
# phase, once per retry attempt. Connect should fail fast; only
# read/write/pool (waiting on a slow-but-alive generation) need
# the long budget.
connect_timeout = min(10.0, float(timeout))
self._client = stealth.stealth_async_client(
timeout=float(timeout), limits=limits
timeout=httpx.Timeout(float(timeout), connect=connect_timeout),
limits=limits,
)
self._sem = asyncio.Semaphore(instances)
self._instances = instances
@@ -143,6 +251,9 @@ class GatewayRuntime:
self._breaker.configure(
cfg["gateway_circuit_threshold"], cfg["gateway_circuit_cooldown_seconds"]
)
self._vision_breaker.configure(
cfg["gateway_circuit_threshold"], cfg["gateway_circuit_cooldown_seconds"]
)
return self._client, self._sem
async def aclose(self) -> None:
@@ -219,7 +330,12 @@ class GatewayRuntime:
log(f"{method} {url} connection failed after {attempts} attempt(s): {exc}")
return None, exc, timing
self.last_status = resp.status_code
if resp.status_code >= 500:
if resp.status_code >= 500 or resp.status_code == 429:
# A 429 that survived retry_send's own retries means the upstream
# is still rate-limiting us after gateway_max_retries attempts -
# score it like a 5xx so sustained rate-limit exhaustion opens the
# breaker instead of paying the full retry/backoff cost on every
# single request indefinitely.
self.errors += 1
was_open = self._breaker.is_open
self._breaker.record_failure()
@@ -251,6 +367,7 @@ class GatewayRuntime:
vision_cost = 0.0
if cfg["gateway_vision_enabled"]:
attribution = _attribution_headers()
augmenter = VisionAugmenter(
cfg["gateway_vision_url"],
cfg["gateway_vision_model"],
@@ -262,6 +379,13 @@ class GatewayRuntime:
context_map=context_map,
app_reference=app_reference,
vision_dialect=cfg.get("gateway_thinking_dialect", "auto"),
breaker=self._vision_breaker,
max_retries=cfg.get("gateway_max_retries", config.MAX_RETRIES_DEFAULT),
retry_backoff_ms=cfg.get(
"gateway_retry_backoff_ms", config.RETRY_BACKOFF_MS_DEFAULT
),
referer=attribution.get("HTTP-Referer", ""),
title=attribution.get("X-Title", ""),
)
messages = await augmenter.augment_messages(client, messages)
self.vision_calls += augmenter.calls
@@ -301,6 +425,7 @@ class GatewayRuntime:
capabilities = upstream_capabilities(
cfg.get("gateway_upstream_url", ""),
cfg.get("gateway_thinking_dialect", "auto"),
ollama_stream_usage=bool(cfg.get("gateway_ollama_stream_usage", False)),
)
if capabilities.supports_stream_options:
stream_options = dict(body.get("stream_options") or {})
@@ -323,7 +448,11 @@ class GatewayRuntime:
dialect=cfg.get("gateway_thinking_dialect", "auto"),
)
headers = {"Content-Type": "application/json"}
headers = {
"Content-Type": "application/json",
**_attribution_headers(),
**_extra_provider_headers(cfg),
}
if cfg["gateway_api_key"]:
headers["Authorization"] = f"Bearer {cfg['gateway_api_key']}"
else:
@@ -331,6 +460,35 @@ class GatewayRuntime:
"No upstream API key configured (gateway_api_key / DEEPSEEK_API_KEY / OPENROUTER_API_KEY); upstream will likely reject the request"
)
fallback_used_route = None
context_window = _route_context_window(cfg, model)
if context_window:
estimated_tokens = _estimate_request_tokens(payload)
if estimated_tokens > context_window:
precheck_route = resolve_fallback(fallback_key, "chat")
precheck_overlay = (
chat_overlay(precheck_route.source_model, base_cfg)
if precheck_route is not None
else None
)
if precheck_overlay:
fallback_window = _route_context_window(
precheck_overlay, precheck_overlay["gateway_model"]
)
if not fallback_window or estimated_tokens <= fallback_window:
log(
f"model {model!r} context window ({context_window} tokens) is "
f"smaller than the estimated request size (~{estimated_tokens} "
f"tokens); skipping straight to fallback "
f"{precheck_route.source_model!r} -> {precheck_overlay['gateway_model']!r}"
)
fallback_used_route = precheck_route.source_model
cfg = {**base_cfg, **precheck_overlay}
model = cfg["gateway_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_api_key")
send_start = time.monotonic()
resp, exc, timing = await self._send(
client,
@@ -347,10 +505,18 @@ class GatewayRuntime:
# The streamed response body hasn't been read yet; the shared error
# handling below needs resp.text/.content, which requires an explicit
# read for a stream=True response (a no-op if already buffered).
await resp.aread()
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001 - a corrupt/truncated
# error body must never crash the request; fall through with
# whatever text/content httpx managed to buffer (possibly empty).
log(f"failed to read non-200 stream error body: {e}")
requested_model = requested
if _call_failed(resp, exc, timing):
body_error = _embedded_failure_message(resp, stream)
if fallback_used_route is None and _should_fallback(resp, exc, timing, body_error):
if body_error:
log(f"model {requested!r} returned 200 with an embedded error: {body_error}")
fallback_route = resolve_fallback(fallback_key, "chat")
fallback_overlay = (
chat_overlay(fallback_route.source_model, base_cfg)
@@ -362,6 +528,7 @@ class GatewayRuntime:
f"model {requested!r} failed, falling back to "
f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_model']!r}"
)
fallback_used_route = fallback_route.source_model
cfg = {**base_cfg, **fallback_overlay}
model = cfg["gateway_model"]
payload = dict(payload)
@@ -380,7 +547,11 @@ class GatewayRuntime:
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
await resp.aread()
try:
await resp.aread()
except httpx.HTTPError as e: # noqa: BLE001
log(f"failed to read fallback non-200 stream error body: {e}")
body_error = _embedded_failure_message(resp, stream)
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
@@ -394,6 +565,8 @@ class GatewayRuntime:
"model": model,
"user_agent": user_agent,
"app_reference": app_reference,
"provider": cfg.get("gateway_provider_name") or "default",
"fallback_used_route": fallback_used_route,
**params,
**timing,
}
@@ -459,6 +632,18 @@ class GatewayRuntime:
content={"error": {"message": resp.text, "type": "upstream_error"}},
headers=resp_headers,
)
if body_error:
# 200 OK with the failure embedded in the body (see
# _embedded_failure_message) - the fallback, if any, already ran
# and also failed. Report it as the upstream error it is instead
# of billing/recording it as a successful call.
resp_headers = finalize(502, False, "upstream_error")
log(f"chat upstream POST -> 200 with embedded error: {body_error}")
return JSONResponse(
status_code=502,
content={"error": {"message": body_error, "type": "upstream_error"}},
headers=resp_headers,
)
if stream:
# Real upstream streaming: forward SSE chunks to the client as they
# arrive (so TTFT/inter-token latency are genuine), and finalize the
@@ -490,7 +675,7 @@ class GatewayRuntime:
)
try:
data = resp.json()
except ValueError:
except (ValueError, httpx.HTTPError):
self.errors += 1
resp_headers = finalize(502, False, "gateway")
log("chat upstream returned 200 but body was not valid JSON")
@@ -550,8 +735,21 @@ class GatewayRuntime:
continue
choices = chunk.get("choices") or []
delta = (choices[0].get("delta") if choices else None) or {}
if delta.get("content") or delta.get("reasoning_content"):
if delta.get("content") or delta.get("reasoning") or delta.get("reasoning_content"):
content_chunks += 1
# OpenRouter (and providers it fronts) can report a mid-stream
# failure as an SSE chunk carrying an `error` key or
# finish_reason "error" instead of dropping the connection -
# see openrouter.ai/docs/api-reference/errors. The bytes are
# already committed to the client at this point so they must
# still be relayed, but the ledger must not record this as a
# clean success.
chunk_error = chunk.get("error")
if not chunk_error and choices and choices[0].get("finish_reason") == "error":
chunk_error = choices[0].get("error") or "upstream reported finish_reason=error"
if chunk_error:
success = False
error_category = "upstream_error"
usage = chunk.get("usage")
if usage:
usage_captured = usage
@@ -664,7 +862,11 @@ class GatewayRuntime:
payload = dict(body)
payload["model"] = model
headers = {"Content-Type": "application/json"}
headers = {
"Content-Type": "application/json",
**_attribution_headers(),
**_extra_provider_headers(cfg),
}
if cfg["gateway_embed_key"]:
headers["Authorization"] = f"Bearer {cfg['gateway_embed_key']}"
else:
@@ -684,7 +886,11 @@ class GatewayRuntime:
)
requested_model = requested
if _call_failed(resp, exc, timing):
fallback_used_route = None
body_error = _embedded_failure_message(resp, False)
if _should_fallback(resp, exc, timing, body_error):
if body_error:
log(f"embed model {requested!r} returned 200 with an embedded error: {body_error}")
fallback_route = resolve_fallback(fallback_key, "embed")
fallback_overlay = (
embed_overlay(fallback_route.source_model, base_cfg)
@@ -696,6 +902,7 @@ class GatewayRuntime:
f"embed model {requested!r} failed, falling back to "
f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_embed_model']!r}"
)
fallback_used_route = fallback_route.source_model
cfg = {**base_cfg, **fallback_overlay}
model = cfg["gateway_embed_model"]
payload = dict(payload)
@@ -711,6 +918,7 @@ class GatewayRuntime:
log,
json_body=payload,
)
body_error = _embedded_failure_message(resp, False)
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
@@ -724,6 +932,8 @@ class GatewayRuntime:
"model": model,
"user_agent": user_agent,
"app_reference": app_reference,
"provider": cfg.get("gateway_provider_name") or "default",
"fallback_used_route": fallback_used_route,
**params,
**timing,
}
@@ -789,9 +999,17 @@ class GatewayRuntime:
content={"error": {"message": resp.text, "type": "upstream_error"}},
headers=resp_headers,
)
if body_error:
resp_headers = finalize(502, False, "upstream_error")
log(f"embed upstream POST -> 200 with embedded error: {body_error}")
return JSONResponse(
status_code=502,
content={"error": {"message": body_error, "type": "upstream_error"}},
headers=resp_headers,
)
try:
data = resp.json()
except ValueError:
except (ValueError, httpx.HTTPError):
self.errors += 1
resp_headers = finalize(502, False, "gateway")
log("embed upstream returned 200 but body was not valid JSON")
@@ -874,7 +1092,11 @@ class GatewayRuntime:
payload = dict(body)
payload["model"] = model
headers = {"Content-Type": "application/json"}
headers = {
"Content-Type": "application/json",
**_attribution_headers(),
**_extra_provider_headers(cfg),
}
if cfg["gateway_image_key"]:
headers["Authorization"] = f"Bearer {cfg['gateway_image_key']}"
else:
@@ -894,7 +1116,11 @@ class GatewayRuntime:
)
requested_model = requested
if _call_failed(resp, exc, timing):
fallback_used_route = None
body_error = _embedded_failure_message(resp, False)
if _should_fallback(resp, exc, timing, body_error):
if body_error:
log(f"image model {requested!r} returned 200 with an embedded error: {body_error}")
fallback_route = resolve_fallback(fallback_key, "image")
fallback_overlay = (
image_overlay(fallback_route.source_model, base_cfg)
@@ -906,6 +1132,7 @@ class GatewayRuntime:
f"image model {requested!r} failed, falling back to "
f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_image_model']!r}"
)
fallback_used_route = fallback_route.source_model
cfg = {**base_cfg, **fallback_overlay}
model = cfg["gateway_image_model"]
payload = dict(payload)
@@ -921,6 +1148,7 @@ class GatewayRuntime:
log,
json_body=payload,
)
body_error = _embedded_failure_message(resp, False)
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
@@ -934,6 +1162,8 @@ class GatewayRuntime:
"model": model,
"user_agent": user_agent,
"app_reference": app_reference,
"provider": cfg.get("gateway_provider_name") or "default",
"fallback_used_route": fallback_used_route,
**params,
**timing,
}
@@ -994,9 +1224,17 @@ class GatewayRuntime:
content={"error": {"message": resp.text, "type": "upstream_error"}},
headers=resp_headers,
)
if body_error:
resp_headers = finalize(502, False, "upstream_error")
log(f"image upstream POST -> 200 with embedded error: {body_error}")
return JSONResponse(
status_code=502,
content={"error": {"message": body_error, "type": "upstream_error"}},
headers=resp_headers,
)
try:
data = resp.json()
except ValueError:
except (ValueError, httpx.HTTPError):
self.errors += 1
resp_headers = finalize(502, False, "gateway")
log("image upstream returned 200 but body was not valid JSON")
@@ -1113,7 +1351,7 @@ class GatewayRuntime:
):
try:
usage = resp.json().get("usage")
except ValueError:
except (ValueError, httpx.HTTPError):
usage = None
resp_headers = finalize(
resp.status_code,
@@ -1146,4 +1384,5 @@ class GatewayRuntime:
"last_latency_ms": self.last_latency_ms,
"pool": self._instances,
"circuit_open": self._breaker.is_open,
"vision_circuit_open": self._vision_breaker.is_open,
}
@@ -0,0 +1,210 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import logging
import time
from dataclasses import dataclass, field
from typing import Optional
logger = logging.getLogger(__name__)
NEUTRAL_REWARD = 0.5
WEIGHT_PRIOR_ATTEMPTS = 2.0
PERFORMANCE_REFERENCE_TOKENS_PER_SECOND = 25.0
PERFORMANCE_LATENCY_REFERENCE_MS = 8000.0
CIRCUIT_BREAKER_FAILURE_THRESHOLD = 3
CIRCUIT_BREAKER_COOLDOWN_SECONDS = 300.0
SEED_HISTORY_DAYS = 7
def speed_reward(tokens_per_second: Optional[float]) -> float:
if not tokens_per_second or tokens_per_second <= 0:
return NEUTRAL_REWARD
return tokens_per_second / (tokens_per_second + PERFORMANCE_REFERENCE_TOKENS_PER_SECOND)
def latency_reward(latency_ms: Optional[float]) -> float:
if not latency_ms or latency_ms <= 0:
return 1.0
return 1.0 / (1.0 + latency_ms / PERFORMANCE_LATENCY_REFERENCE_MS)
def outcome_reward(
latency_ms: Optional[float] = None, tokens_per_second: Optional[float] = None
) -> float:
return speed_reward(tokens_per_second) * latency_reward(latency_ms)
@dataclass
class ModelHealth:
consecutive_failures: int = 0
disabled_until: float = 0.0
times_chosen: int = 0
last_used_at: float = 0.0
success_count: int = 0
failure_count: int = 0
total_reward: float = 0.0
total_latency_ms: float = 0.0
total_tokens_per_second: float = 0.0
def is_circuit_open(self) -> bool:
return self.disabled_until > time.time()
def weight(self) -> float:
attempts = self.success_count + self.failure_count
return (self.total_reward + NEUTRAL_REWARD * WEIGHT_PRIOR_ATTEMPTS) / (
attempts + WEIGHT_PRIOR_ATTEMPTS
)
def avg_latency_ms(self) -> Optional[float]:
return (self.total_latency_ms / self.success_count) if self.success_count else None
def avg_tokens_per_second(self) -> Optional[float]:
return (
(self.total_tokens_per_second / self.success_count)
if self.success_count
else None
)
def snapshot(self) -> dict:
return {
"times_chosen": self.times_chosen,
"consecutive_failures": self.consecutive_failures,
"circuit_open": self.is_circuit_open(),
"last_used_at": self.last_used_at,
"success_count": self.success_count,
"failure_count": self.failure_count,
"weight": round(self.weight(), 4),
"avg_latency_ms": self.avg_latency_ms(),
"avg_tokens_per_second": self.avg_tokens_per_second(),
}
Key = tuple[str, str]
_health: dict[Key, ModelHealth] = {}
def _health_for(provider: str, model: str) -> ModelHealth:
key = (provider or "default", model or "")
health = _health.get(key)
if health is None:
health = ModelHealth()
_health[key] = health
return health
def record_outcome(
provider: str,
model: str,
success: bool,
latency_ms: Optional[float] = None,
tokens_per_second: Optional[float] = None,
) -> None:
"""Track per-(provider, model) reliability/speed/latency, purely for
display on the stats page. Never gates or influences dispatch - routing
stays exactly one target + one fallback hop, unchanged."""
if not model:
return
health = _health_for(provider, model)
health.times_chosen += 1
health.last_used_at = time.time()
if success:
reward = (
NEUTRAL_REWARD
if latency_ms is None and tokens_per_second is None
else outcome_reward(latency_ms, tokens_per_second)
)
health.success_count += 1
health.consecutive_failures = 0
health.disabled_until = 0.0
health.total_reward += reward
if latency_ms is not None:
health.total_latency_ms += latency_ms
if tokens_per_second is not None:
health.total_tokens_per_second += tokens_per_second
else:
health.failure_count += 1
health.consecutive_failures += 1
if health.consecutive_failures >= CIRCUIT_BREAKER_FAILURE_THRESHOLD:
health.disabled_until = time.time() + CIRCUIT_BREAKER_COOLDOWN_SECONDS
def apply_history(
provider: str, model: str, success_count: int, failure_count: int, total_reward: float
) -> None:
health = _health_for(provider, model)
health.success_count += success_count
health.failure_count += failure_count
health.total_reward += total_reward
def snapshot_all() -> dict:
return {
f"{provider}:{model}": health.snapshot()
for (provider, model), health in _health.items()
}
def snapshot_for(provider: str, model: str) -> Optional[dict]:
key = (provider or "default", model or "")
health = _health.get(key)
return health.snapshot() if health is not None else None
def reset() -> None:
"""Test-only: clear all in-memory health state."""
_health.clear()
def seed_from_ledger(days: int = SEED_HISTORY_DAYS) -> int:
"""Fold `days` of existing gateway_usage_ledger rows into the in-memory
health state at startup, so a restart doesn't forget every model's track
record. Reuses data already being recorded - no new table."""
from datetime import datetime, timedelta, timezone
from devplacepy.database import db
from devplacepy.services.openai_gateway.usage import GATEWAY_LEDGER
# database/schema.py's init_db() is the single point of truth that
# guarantees the provider column and its index exist before any request
# is ever served - no defensive existence check needed here.
cutoff = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat(
timespec="microseconds"
)
try:
rows = db.query(
f"SELECT provider, model, success, total_latency_ms, tokens_per_second "
f"FROM {GATEWAY_LEDGER} WHERE created_at >= :cutoff AND model != ''",
cutoff=cutoff,
)
grouped: dict[Key, dict] = {}
for row in rows:
provider = str(row.get("provider") or "default")
model = str(row.get("model") or "")
if not model:
continue
key = (provider, model)
agg = grouped.setdefault(
key, {"success_count": 0, "failure_count": 0, "total_reward": 0.0}
)
if row.get("success"):
agg["success_count"] += 1
agg["total_reward"] += outcome_reward(
row.get("total_latency_ms"), row.get("tokens_per_second")
)
else:
agg["failure_count"] += 1
for (provider, model), agg in grouped.items():
apply_history(
provider,
model,
agg["success_count"],
agg["failure_count"],
agg["total_reward"],
)
return len(grouped)
except Exception as exc: # noqa: BLE001 - startup seeding must never crash boot
logger.warning("model health seed from ledger failed: %s", exc)
return 0
@@ -0,0 +1,368 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import time
from datetime import datetime, timedelta, timezone
from typing import Optional
from devplacepy.database import db
from devplacepy.services.openai_gateway import model_health
from devplacepy.services.openai_gateway.usage import GATEWAY_LEDGER
RANGE_SECONDS = {
"1h": 3600,
"12h": 12 * 3600,
"24h": 24 * 3600,
"48h": 48 * 3600,
"7d": 7 * 86400,
"14d": 14 * 86400,
"30d": 30 * 86400,
"60d": 60 * 86400,
"180d": 180 * 86400,
}
LATENCY_BUCKET_EDGES_MS = (200, 500, 1000, 2000, 5000, 15000)
LATENCY_BUCKET_LABELS = ("<200ms", "200-500ms", "500ms-1s", "1-2s", "2-5s", "5-15s", "15s+")
TPS_BUCKET_EDGES = (5, 15, 30, 60, 120)
TPS_BUCKET_LABELS = ("<5", "5-15", "15-30", "30-60", "60-120", "120+")
RECENT_FAILURES_LIMIT = 50
def _cutoff_iso(range_seconds: int) -> str:
return (datetime.now(timezone.utc) - timedelta(seconds=range_seconds)).isoformat(
timespec="microseconds"
)
def _bucket_seconds(range_seconds: int) -> int:
return max(60, range_seconds // 120)
def _histogram(values: list[float], edges: tuple, labels: tuple) -> list[dict]:
counts = [0] * len(labels)
for value in values:
placed = False
for index, edge in enumerate(edges):
if value < edge:
counts[index] += 1
placed = True
break
if not placed:
counts[-1] += 1
return [{"label": label, "count": count} for label, count in zip(labels, counts)]
def _ledger_rows(cutoff: str) -> list[dict]:
# database/schema.py's init_db() is the single point of truth that
# guarantees every gateway_usage_ledger column (including provider and
# fallback_used_route) and its indexes exist before any request is ever
# served - no defensive existence checks needed here.
return list(
db.query(
f"SELECT created_at, model, provider, endpoint, backend, success, "
f"status_code, total_latency_ms, tokens_per_second, prompt_tokens, "
f"completion_tokens, error_category, stream_requested, fallback_used_route "
f"FROM {GATEWAY_LEDGER} WHERE created_at >= :cutoff",
cutoff=cutoff,
)
)
def _bump(counter: dict, key, amount: int = 1) -> None:
counter[key] = counter.get(key, 0) + amount
def _resolve_range(range_key: str) -> int:
if range_key not in RANGE_SECONDS:
raise ValueError(f"Unknown range: {range_key}")
return RANGE_SECONDS[range_key]
def compute_summary(range_key: str) -> dict:
range_seconds = _resolve_range(range_key)
cutoff = _cutoff_iso(range_seconds)
rows = _ledger_rows(cutoff)
bucket_seconds = _bucket_seconds(range_seconds)
totals = {"success": 0, "error": 0}
per_model: dict[tuple, dict] = {}
per_endpoint: dict[str, dict] = {}
per_provider: dict[str, dict] = {}
status_codes: dict[int, int] = {}
streaming_split = {"streamed": 0, "non_streamed": 0}
failure_reasons: dict[str, int] = {}
hourly: dict[int, int] = {}
timeseries: dict[int, dict] = {}
latency_values: list[float] = []
tps_values: list[float] = []
recent_failures: list[dict] = []
for row in rows:
success = bool(row.get("success"))
totals["success" if success else "error"] += 1
if not success:
recent_failures.append(
{
"created_at": row.get("created_at"),
"model": str(row.get("model") or ""),
"provider": str(row.get("provider") or "default"),
"endpoint": row.get("endpoint"),
"status_code": row.get("status_code"),
"reason": row.get("error_category"),
"fallback_used_route": row.get("fallback_used_route"),
}
)
model = str(row.get("model") or "")
provider = str(row.get("provider") or "default")
key = (model, provider)
bucket = per_model.setdefault(
key,
{
"model": model,
"provider": provider,
"total_requests": 0,
"success_requests": 0,
"_latency_sum": 0.0,
"_latency_n": 0,
"_tps_sum": 0.0,
"_tps_n": 0,
"prompt_tokens": 0,
"completion_tokens": 0,
},
)
bucket["total_requests"] += 1
bucket["prompt_tokens"] += int(row.get("prompt_tokens") or 0)
bucket["completion_tokens"] += int(row.get("completion_tokens") or 0)
latency = row.get("total_latency_ms") or 0.0
tps = row.get("tokens_per_second") or 0.0
if success:
bucket["success_requests"] += 1
if latency:
bucket["_latency_sum"] += latency
bucket["_latency_n"] += 1
latency_values.append(latency)
if tps and tps > 0:
bucket["_tps_sum"] += tps
bucket["_tps_n"] += 1
tps_values.append(tps)
endpoint_bucket = per_endpoint.setdefault(
str(row.get("endpoint") or ""), {"total_requests": 0, "success_requests": 0}
)
endpoint_bucket["total_requests"] += 1
endpoint_bucket["success_requests"] += int(success)
provider_bucket = per_provider.setdefault(
provider, {"total_requests": 0, "success_requests": 0}
)
provider_bucket["total_requests"] += 1
provider_bucket["success_requests"] += int(success)
if row.get("status_code"):
_bump(status_codes, int(row["status_code"]))
streaming_split["streamed" if row.get("stream_requested") else "non_streamed"] += 1
if not success and row.get("error_category"):
_bump(failure_reasons, str(row["error_category"]))
created_at = str(row.get("created_at") or "")
if len(created_at) >= 13:
_bump(hourly, int(created_at[11:13]))
epoch = datetime.fromisoformat(created_at).timestamp()
slot = int(epoch // bucket_seconds) * bucket_seconds
ts_bucket = timeseries.setdefault(slot, {"requests": 0, "success": 0})
ts_bucket["requests"] += 1
ts_bucket["success"] += int(success)
per_model_out = []
for (model, provider), bucket in per_model.items():
health = model_health.snapshot_for(provider, model)
per_model_out.append(
{
"model": bucket["model"],
"provider": bucket["provider"],
"total_requests": bucket["total_requests"],
"success_requests": bucket["success_requests"],
"avg_latency_ms": (
round(bucket["_latency_sum"] / bucket["_latency_n"], 2)
if bucket["_latency_n"]
else None
),
"avg_tokens_per_second": (
round(bucket["_tps_sum"] / bucket["_tps_n"], 2)
if bucket["_tps_n"]
else None
),
"prompt_tokens": bucket["prompt_tokens"],
"completion_tokens": bucket["completion_tokens"],
"health": health,
}
)
per_model_out.sort(key=lambda r: r["total_requests"], reverse=True)
return {
"range": range_key,
"range_seconds": range_seconds,
"bucket_seconds": bucket_seconds,
"generated_at": time.time(),
"totals": totals,
"per_model": per_model_out,
"per_endpoint": [
{"endpoint": endpoint, **bucket} for endpoint, bucket in per_endpoint.items()
],
"per_provider": [
{"provider": provider, **bucket} for provider, bucket in per_provider.items()
],
"timeseries": [
{"bucket_start": slot, **bucket}
for slot, bucket in sorted(timeseries.items())
],
"status_codes": [
{"status_code": code, "count": count} for code, count in sorted(status_codes.items())
],
"streaming_split": streaming_split,
"failure_reasons": sorted(
({"reason": reason, "count": count} for reason, count in failure_reasons.items()),
key=lambda r: r["count"],
reverse=True,
)[:10],
"hourly_distribution": [
{"hour": hour, "count": count} for hour, count in sorted(hourly.items())
],
"recent_failures": sorted(
recent_failures, key=lambda r: r["created_at"] or "", reverse=True
)[:RECENT_FAILURES_LIMIT],
"latency_histogram": _histogram(
latency_values, LATENCY_BUCKET_EDGES_MS, LATENCY_BUCKET_LABELS
),
"tokens_per_second_histogram": _histogram(
tps_values, TPS_BUCKET_EDGES, TPS_BUCKET_LABELS
),
}
def compute_model_detail(provider: str, model: str, range_key: str) -> dict:
range_seconds = _resolve_range(range_key)
cutoff = _cutoff_iso(range_seconds)
bucket_seconds = _bucket_seconds(range_seconds)
rows = [
row
for row in _ledger_rows(cutoff)
if str(row.get("model") or "") == model
and str(row.get("provider") or "default") == provider
]
total_requests = len(rows)
success_rows = [row for row in rows if row.get("success")]
success_requests = len(success_rows)
latencies = [row["total_latency_ms"] for row in success_rows if row.get("total_latency_ms")]
tps_values = [
row["tokens_per_second"] for row in success_rows if (row.get("tokens_per_second") or 0) > 0
]
status_codes: dict[int, int] = {}
per_endpoint: dict[str, dict] = {}
timeseries: dict[int, dict] = {}
recent_failures = []
for row in rows:
if row.get("status_code"):
_bump(status_codes, int(row["status_code"]))
endpoint_bucket = per_endpoint.setdefault(
str(row.get("endpoint") or ""), {"total_requests": 0, "success_requests": 0}
)
endpoint_bucket["total_requests"] += 1
endpoint_bucket["success_requests"] += int(bool(row.get("success")))
created_at = str(row.get("created_at") or "")
if len(created_at) >= 13:
epoch = datetime.fromisoformat(created_at).timestamp()
slot = int(epoch // bucket_seconds) * bucket_seconds
ts_bucket = timeseries.setdefault(slot, {"requests": 0, "success": 0})
ts_bucket["requests"] += 1
ts_bucket["success"] += int(bool(row.get("success")))
if not row.get("success"):
recent_failures.append(
{
"created_at": row.get("created_at"),
"endpoint": row.get("endpoint"),
"status_code": row.get("status_code"),
"reason": row.get("error_category"),
"fallback_used_route": row.get("fallback_used_route"),
}
)
recent_failures.sort(key=lambda r: r["created_at"] or "", reverse=True)
return {
"model": model,
"provider": provider,
"range": range_key,
"range_seconds": range_seconds,
"bucket_seconds": bucket_seconds,
"generated_at": time.time(),
"health": model_health.snapshot_for(provider, model),
"summary": {
"total_requests": total_requests,
"success_requests": success_requests,
"avg_latency_ms": round(sum(latencies) / len(latencies), 2) if latencies else None,
"min_latency_ms": min(latencies) if latencies else None,
"max_latency_ms": max(latencies) if latencies else None,
"avg_tokens_per_second": (
round(sum(tps_values) / len(tps_values), 2) if tps_values else None
),
"prompt_tokens": sum(int(row.get("prompt_tokens") or 0) for row in rows),
"completion_tokens": sum(int(row.get("completion_tokens") or 0) for row in rows),
},
"timeseries": [
{"bucket_start": slot, **bucket} for slot, bucket in sorted(timeseries.items())
],
"status_codes": [
{"status_code": code, "count": count} for code, count in sorted(status_codes.items())
],
"per_endpoint": [
{"endpoint": endpoint, **bucket} for endpoint, bucket in per_endpoint.items()
],
"recent_failures": recent_failures[:RECENT_FAILURES_LIMIT],
"latency_histogram": _histogram(latencies, LATENCY_BUCKET_EDGES_MS, LATENCY_BUCKET_LABELS),
"tokens_per_second_histogram": _histogram(tps_values, TPS_BUCKET_EDGES, TPS_BUCKET_LABELS),
}
def list_known_models() -> list[dict]:
"""Every (provider, model) pair with either a configured route or recent
traffic, enriched with health - the merge that lets a freshly-configured,
never-called route still show up alongside historically active ones."""
from devplacepy.services.openai_gateway.routing import model_store
seen: dict[tuple, dict] = {}
for route in model_store.list():
if route.get("kind") != "chat" or not route.get("is_active", True):
continue
provider = str(route.get("provider") or "default")
model = str(route.get("target_model") or "")
if not model:
continue
key = (provider, model)
seen[key] = {
"provider": provider,
"model": model,
"source_model": route.get("source_model"),
"context_window": int(route.get("context_window") or 0),
}
for key_str in model_health.snapshot_all().keys():
provider, _, model = key_str.partition(":")
key = (provider, model)
if key not in seen:
seen[key] = {
"provider": provider,
"model": model,
"source_model": None,
"context_window": 0,
}
out = []
for (provider, model), entry in seen.items():
entry["health"] = model_health.snapshot_for(provider, model)
out.append(entry)
out.sort(key=lambda r: (r["provider"], r["model"]))
return out
@@ -0,0 +1,42 @@
# retoor <retoor@molodetz.nl>
from __future__ import annotations
import os
import secrets
OPENCODE_CLIENT_NAME = os.environ.get("OPENCODE_CLIENT_NAME", "cli")
OPENCODE_CLIENT_USER_AGENT = os.environ.get(
"OPENCODE_CLIENT_USER_AGENT",
"opencode/1.18.29 ai-sdk/provider-utils/4.0.23 runtime/bun/1.3.15",
)
_OPCODE_ID_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
_OPCODE_ID_LENGTH = 26
def _opencode_id() -> str:
return "".join(secrets.choice(_OPCODE_ID_ALPHABET) for _ in range(_OPCODE_ID_LENGTH))
_opencode_project_id = "global"
_opencode_session_id = "ses_" + _opencode_id()
def opencode_impersonation_headers() -> dict:
"""Reproduce the exact headers the real opencode CLI sends.
OpenCode Zen's free tier only serves the official opencode client: no
User-Agent -> 429 FreeUsageLimitError, a CLI User-Agent without a session
id -> 400 MissingSessionID, the full set below -> 200. The session id is
stable for the process lifetime; the request id is fresh every call -
both generated the same way the real client does (26 random base-62
characters, ses_/msg_ prefixed).
"""
return {
"User-Agent": OPENCODE_CLIENT_USER_AGENT,
"x-opencode-client": OPENCODE_CLIENT_NAME,
"x-opencode-project": _opencode_project_id,
"x-opencode-session": _opencode_session_id,
"x-opencode-request": "msg_" + _opencode_id(),
}
+9 -5
View File
@@ -410,10 +410,14 @@ def spent_24h(
clauses.append("app_reference = :app_reference")
params["app_reference"] = app_reference
where = " AND ".join(clauses)
rows = list(
db.query(
f"SELECT COALESCE(SUM(cost_usd), 0) AS spent FROM {GATEWAY_LEDGER} WHERE {where}",
**params,
try:
rows = list(
db.query(
f"SELECT COALESCE(SUM(cost_usd), 0) AS spent FROM {GATEWAY_LEDGER} WHERE {where}",
**params,
)
)
)
except Exception as exc:
logger.warning("gateway quota spend query failed: %s", exc)
return 0.0
return float(rows[0].get("spent") or 0.0) if rows else 0.0
@@ -5,6 +5,7 @@ from __future__ import annotations
import asyncio
import logging
import time
from datetime import datetime, timezone
from typing import Awaitable, Callable, Optional
import httpx
@@ -58,12 +59,53 @@ class CircuitBreaker:
self.opened_at = time.monotonic()
async def _backoff(backoff_ms: int, attempt: int) -> None:
MAX_RETRY_AFTER_SECONDS = 30.0
def _retry_after_seconds(resp: httpx.Response) -> Optional[float]:
# DeepSeek documents no Retry-After; OpenRouter documents a standard
# HTTP Retry-After (seconds, sometimes an HTTP-date) on 429/503 and
# instructs clients to honor it - see openrouter.ai/docs/api-reference/errors.
raw = resp.headers.get("retry-after")
if not raw:
return None
try:
seconds = float(raw)
except ValueError:
try:
from email.utils import parsedate_to_datetime
when = parsedate_to_datetime(raw)
if when.tzinfo is None:
return None
seconds = (when - datetime.now(timezone.utc)).total_seconds()
except (TypeError, ValueError, IndexError):
return None
if seconds < 0:
return 0.0
return min(seconds, MAX_RETRY_AFTER_SECONDS)
async def _backoff(
backoff_ms: int, attempt: int, retry_after: Optional[float] = None
) -> None:
if retry_after is not None:
if retry_after > 0:
await asyncio.sleep(retry_after)
return
delay = max(0, backoff_ms) * attempt / 1000.0
if delay > 0:
await asyncio.sleep(delay)
def _is_retryable_status(status_code: int) -> bool:
# 5xx is a transient upstream fault. 429 is a rate limit - both DeepSeek
# and OpenRouter use it and OpenRouter explicitly documents retrying it
# (with Retry-After honored); a single retry after backoff is safe since
# gateway_max_retries already bounds the total attempts.
return status_code >= 500 or status_code == 429
async def retry_send(
do_call: Callable[[], Awaitable[httpx.Response]],
sem: asyncio.Semaphore,
@@ -95,13 +137,17 @@ async def retry_send(
)
await _backoff(backoff_ms, attempts)
continue
if resp.status_code >= 500 and attempts <= max_retries:
log(f"upstream {resp.status_code}, retrying ({attempts}/{max_retries})")
if _is_retryable_status(resp.status_code) and attempts <= max_retries:
retry_after = _retry_after_seconds(resp)
log(
f"upstream {resp.status_code}, retrying ({attempts}/{max_retries})"
+ (f" after Retry-After={retry_after:.1f}s" if retry_after else "")
)
try:
await resp.aclose()
except Exception: # noqa: BLE001 - releasing the connection must never block a retry
pass
await _backoff(backoff_ms, attempts)
await _backoff(backoff_ms, attempts, retry_after)
continue
return resp, None, attempts, queue_wait_ms
return None, last_exc, attempts, queue_wait_ms
+109 -1
View File
@@ -15,6 +15,7 @@ from devplacepy.database import (
get_table,
sync_local_cache,
)
from devplacepy.services.openai_gateway import opencode_zen
logger = logging.getLogger(__name__)
@@ -23,6 +24,18 @@ MODELS_TABLE = "gateway_models"
CACHE_NAME = "gateway_routing"
KINDS = ("chat", "embed", "image")
# Admin-selectable on a gateway_providers row's `client_profile` field. A
# generic provider (the default, empty profile) sends no extra headers -
# this exists only for upstreams that reject requests unless they see a
# specific client's headers, e.g. OpenCode Zen's free tier only serves the
# real opencode CLI (spoofed via opencode_zen.opencode_impersonation_headers)
# and returns 429/400 to anything else. Adding a new profile is one entry
# here plus one option in the provider form - no other special-casing.
CLIENT_PROFILES: dict = {
"": None,
"opencode": opencode_zen.opencode_impersonation_headers,
}
_ROUTING_CACHE: dict = {}
@@ -56,6 +69,63 @@ def _image_url_from_base(base_url: str) -> str:
return base_url
def _models_url_from_base(base_url: str) -> str:
base_url = (base_url or "").strip()
if not base_url:
return ""
if base_url.endswith("/chat/completions"):
return base_url[: -len("/chat/completions")] + "/models"
return base_url.rstrip("/") + "/models"
PROVIDER_MODELS_PROBE_TIMEOUT = 8.0
def _default_provider_credentials() -> tuple[str, str]:
from devplacepy.services.manager import service_manager
svc = service_manager.get_service("openai")
cfg = svc.get_config() if svc is not None else {}
return cfg.get("gateway_upstream_url", ""), cfg.get("gateway_api_key", "")
async def fetch_provider_models(
provider_name: str, timeout: float = PROVIDER_MODELS_PROBE_TIMEOUT
) -> Optional[list[str]]:
provider_name = (provider_name or "").strip()
if provider_name:
provider = provider_store.get(provider_name)
if provider is None:
return None
base_url, api_key = provider.get("base_url", ""), provider.get("api_key", "")
else:
base_url, api_key = _default_provider_credentials()
models_url = _models_url_from_base(base_url)
if not models_url:
return None
from devplacepy import stealth
headers = {"authorization": f"Bearer {api_key}"} if api_key else {}
try:
async with stealth.stealth_async_client(timeout=timeout, headers=headers) as client:
response = await client.get(models_url)
except Exception as exc:
logger.info("gateway provider model listing failed for %s: %s", models_url, exc)
return None
if response.status_code != 200:
return None
try:
payload = response.json()
except Exception:
return None
rows = payload.get("data") if isinstance(payload, dict) else None
if not isinstance(rows, list):
return None
ids = [str(row["id"]) for row in rows if isinstance(row, dict) and row.get("id")]
return ids or None
MODEL_TIER2_COLUMNS = (
"context_tier_threshold_tokens",
"price_cache_hit_per_m_tier2",
@@ -73,8 +143,12 @@ def ensure_tables() -> None:
"CREATE TABLE IF NOT EXISTS "
+ PROVIDERS_TABLE
+ " (id INTEGER PRIMARY KEY, name TEXT, base_url TEXT, api_key TEXT, "
"is_active INTEGER DEFAULT 1, created_at TEXT, updated_at TEXT)"
"is_active INTEGER DEFAULT 1, client_profile TEXT DEFAULT '', "
"created_at TEXT, updated_at TEXT)"
)
providers_table = get_table(PROVIDERS_TABLE)
if not providers_table.has_column("client_profile"):
providers_table.create_column_by_example("client_profile", "")
db.query(
"CREATE TABLE IF NOT EXISTS "
+ MODELS_TABLE
@@ -121,6 +195,7 @@ class ProviderIn(BaseModel):
base_url: str = Field(default="", max_length=500)
api_key: str = Field(default="", max_length=400)
is_active: bool = True
client_profile: str = Field(default="", max_length=32)
@field_validator("name")
@classmethod
@@ -140,6 +215,16 @@ class ProviderIn(BaseModel):
raise ValueError("Base URL must be a http(s) URL")
return value
@field_validator("client_profile")
@classmethod
def _clean_client_profile(cls, value: str) -> str:
value = (value or "").strip().lower()
if value not in CLIENT_PROFILES:
raise ValueError(
f"client_profile must be one of: {', '.join(sorted(CLIENT_PROFILES)) or '(none)'}"
)
return value
class ModelRouteIn(BaseModel):
source_model: str = Field(min_length=1, max_length=128)
@@ -280,6 +365,7 @@ def _load() -> dict:
"base_url": str(row.get("base_url") or ""),
"api_key": str(row.get("api_key") or ""),
"is_active": _as_bool(row.get("is_active", 1)),
"client_profile": str(row.get("client_profile") or ""),
}
if MODELS_TABLE in db.tables:
for row in get_table(MODELS_TABLE).all():
@@ -317,6 +403,7 @@ class ProviderStore:
"base_url": payload.base_url,
"api_key": payload.api_key,
"is_active": 1 if payload.is_active else 0,
"client_profile": payload.client_profile,
"updated_at": _now(),
}
if existing:
@@ -500,6 +587,15 @@ def seed_default_deepseek_routes() -> None:
_ROUTING_CACHE.clear()
def _header_hook_for(provider_name: str):
if not provider_name:
return None
provider = provider_store.get(provider_name)
if provider is None:
return None
return CLIENT_PROFILES.get(provider.get("client_profile") or "")
def _provider_overlay(name: str, base_key: str, url_key: str, overlay: dict) -> None:
provider = provider_store.get(name)
if provider is None:
@@ -527,11 +623,15 @@ def chat_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[dic
"gateway_off_peak_start_minute": route.off_peak_start_minute,
"gateway_off_peak_end_minute": route.off_peak_end_minute,
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
"gateway_provider_name": route.provider or "default",
}
if route.provider:
_provider_overlay(
route.provider, "gateway_api_key", "gateway_upstream_url", overlay
)
header_hook = _header_hook_for(route.provider)
if header_hook is not None:
overlay["gateway_extra_request_headers"] = header_hook()
if route.context_window:
from devplacepy.services.openai_gateway.usage import parse_context_map
@@ -570,6 +670,7 @@ def embed_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[di
"gateway_off_peak_start_minute": route.off_peak_start_minute,
"gateway_off_peak_end_minute": route.off_peak_end_minute,
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
"gateway_provider_name": route.provider or "default",
}
provider = provider_store.get(route.provider) if route.provider else None
if provider:
@@ -577,6 +678,9 @@ def embed_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[di
overlay["gateway_embed_url"] = _embed_url_from_base(provider["base_url"])
if provider.get("api_key"):
overlay["gateway_embed_key"] = provider["api_key"]
header_hook = _header_hook_for(route.provider)
if header_hook is not None:
overlay["gateway_extra_request_headers"] = header_hook()
return overlay
@@ -591,6 +695,7 @@ def image_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[di
"gateway_off_peak_start_minute": route.off_peak_start_minute,
"gateway_off_peak_end_minute": route.off_peak_end_minute,
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
"gateway_provider_name": route.provider or "default",
}
provider = provider_store.get(route.provider) if route.provider else None
if provider:
@@ -598,6 +703,9 @@ def image_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[di
overlay["gateway_image_url"] = _image_url_from_base(provider["base_url"])
if provider.get("api_key"):
overlay["gateway_image_key"] = provider["api_key"]
header_hook = _header_hook_for(route.provider)
if header_hook is not None:
overlay["gateway_extra_request_headers"] = header_hook()
return overlay
+39 -15
View File
@@ -163,6 +163,18 @@ class GatewayService(BaseService):
"proxy - so thinking/stream_options are emitted in the right shape.",
group="Prompt",
),
ConfigField(
"gateway_ollama_stream_usage",
"Ollama streamed usage",
type="bool",
default=False,
help="Send stream_options.include_usage to an Ollama upstream so a "
"streamed call's tokens/cost are metered. Current Ollama docs list this "
"as supported, but older server builds reject or ignore it - off by "
"default; enable it once the deployed Ollama version is confirmed to "
"support it. Only applies when the upstream dialect resolves to Ollama.",
group="Prompt",
),
ConfigField(
"gateway_vision_enabled",
"Vision augmentation",
@@ -671,24 +683,36 @@ class GatewayService(BaseService):
)
if subpath == "models" and request.method == "GET":
return self._models_response()
if self.consent_denied(owner):
self.log(
f"Rejected {owner[0]}:{owner[1]} app={app_reference}: "
f"third-party AI consent not granted"
)
self._audit_consent_denied(owner, app_reference)
raise HTTPException(status_code=403, detail=CONSENT_REQUIRED_MESSAGE)
limit, scope, rule = quota.resolve(owner[0], owner[1], app_reference, cfg)
if limit > 0:
spent = quota.spent_24h(*scope)
if spent >= limit:
try:
if self.consent_denied(owner):
self.log(
f"Rejected {owner[0]}:{owner[1]} app={app_reference}: "
f"quota exceeded (${spent:.4f}/${limit:.4f}"
f"{' rule ' + rule.uid if rule else ' default'})"
f"third-party AI consent not granted"
)
self._audit_quota_exceeded(owner[0], owner[1], app_reference, spent, limit, rule)
raise HTTPException(status_code=429, detail="AI gateway daily quota exceeded")
self._audit_consent_denied(owner, app_reference)
raise HTTPException(status_code=403, detail=CONSENT_REQUIRED_MESSAGE)
limit, scope, rule = quota.resolve(owner[0], owner[1], app_reference, cfg)
if limit > 0:
spent = quota.spent_24h(*scope)
if spent >= limit:
self.log(
f"Rejected {owner[0]}:{owner[1]} app={app_reference}: "
f"quota exceeded (${spent:.4f}/${limit:.4f}"
f"{' rule ' + rule.uid if rule else ' default'})"
)
self._audit_quota_exceeded(
owner[0], owner[1], app_reference, spent, limit, rule
)
raise HTTPException(status_code=429, detail="AI gateway daily quota exceeded")
except HTTPException:
raise
except Exception as exc:
self.log(
f"Pre-flight check failed for {owner[0]}:{owner[1]} app={app_reference}: {exc}"
)
raise HTTPException(
status_code=503, detail="AI gateway temporarily unavailable"
) from exc
if subpath == "chat/completions" and request.method == "POST":
try:
body = await request.json()
+15 -3
View File
@@ -43,14 +43,18 @@ class UpstreamCapabilities:
def upstream_capabilities(
url: str, dialect: str = ""
url: str, dialect: str = "", ollama_stream_usage: bool = False
) -> UpstreamCapabilities:
# Current Ollama docs list stream_options.include_usage as supported, but
# real deployed server versions vary and some reject it outright (see
# github.com/ollama/ollama issues #15288/#15293/#14820) - default off,
# gateway_ollama_stream_usage lets an operator opt in for a known-good build.
effective = _resolve_dialect(url, dialect)
if effective == "ollama":
return UpstreamCapabilities(
dialect=effective,
supports_stream_options=False,
supports_stream_usage=False,
supports_stream_options=bool(ollama_stream_usage),
supports_stream_usage=bool(ollama_stream_usage),
supports_thinking_field=True,
)
return UpstreamCapabilities(
@@ -144,7 +148,15 @@ def write_thinking(payload: dict, dialect: str, enabled: bool) -> None:
payload["reasoning"] = {"effort": "high" if enabled else "none"}
return
if dialect == "ollama":
# Ollama's native /api/chat honors a boolean `think`, but its
# OpenAI-compatible /v1/chat/completions layer does not (confirmed
# against github.com/ollama/ollama issues #15288/#15293/#14820,
# Sep 2026): that endpoint maps `reasoning_effort`/`reasoning.effort`
# onto its internal think flag instead. Send both - `think` is
# harmless if ignored, `reasoning_effort` is the field the
# OpenAI-compat endpoint actually honors.
payload["think"] = bool(enabled)
payload["reasoning_effort"] = "high" if enabled else "none"
return
payload["thinking"] = {"type": "enabled" if enabled else "disabled"}
@@ -350,6 +350,8 @@ def classify_error(
return "rate_limit"
if status_code in (401, 403):
return "auth"
if status_code == 402:
return "insufficient_balance"
if status_code == 404:
return "model_not_found"
if status_code in (400, 422):
@@ -362,6 +364,32 @@ def classify_error(
return "gateway"
def embedded_error_message(data: Any) -> Optional[str]:
"""Detect a provider-side failure reported with HTTP 200.
OpenRouter (and providers it fronts) can answer 200 OK with the failure
embedded in the JSON body instead of a non-2xx status - documented at
openrouter.ai/docs/api-reference/errors: "the HTTP response status is
matched with the error.code for validation/credits issues, otherwise
returns 200 OK with the error embedded in the response body." A caller
that only checks status_code would silently record this as a success
with no usage and never trigger a retry or model fallback.
"""
if not isinstance(data, dict):
return None
error = data.get("error")
if not error:
return None
if "choices" in data or "data" in data:
# A well-formed choices/data payload alongside a per-item `error`
# field (e.g. a single failed choice) is not a whole-response failure.
return None
if isinstance(error, dict):
message = error.get("message") or error.get("code") or "unknown error"
return str(message)[:500]
return str(error)[:500]
def audit_actor_for(owner_kind: str, owner_id: str) -> tuple[str, Optional[str], str]:
actor_kind = (
"guest"
@@ -545,6 +573,12 @@ class GatewayUsageLedger:
"endpoint": raw.get("endpoint") or "",
"requested_model": raw.get("requested_model") or "",
"model": raw.get("model") or "",
"provider": (raw.get("provider") or "default")[:60],
"fallback_used_route": (
str(raw["fallback_used_route"])[:128]
if raw.get("fallback_used_route")
else None
),
"status_code": int(raw.get("status_code") or 0),
"success": 1 if raw.get("success") else 0,
"error_category": raw.get("error_category"),
@@ -581,11 +615,29 @@ class GatewayUsageLedger:
}
get_table(GATEWAY_LEDGER).insert(row)
self._audit(raw, norm, cost_usd)
self._track_health(row)
return row
except Exception as exc:
logger.warning("gateway usage record failed: %s", exc)
return None
def _track_health(self, row: dict) -> None:
# Observational only - never gates or influences dispatch. Isolated in
# its own try/except so a health-tracking bug can never turn an
# already-committed ledger write into a failed record() call.
try:
from devplacepy.services.openai_gateway import model_health
model_health.record_outcome(
row["provider"],
row["model"],
bool(row["success"]),
latency_ms=row.get("upstream_latency_ms") or None,
tokens_per_second=row.get("tokens_per_second") or None,
)
except Exception as exc: # noqa: BLE001
logger.debug("model health tracking failed: %s", exc)
def record_external(
self,
*,
+57 -7
View File
@@ -10,6 +10,11 @@ from typing import Any, Optional
import httpx
from devplacepy.services.openai_gateway.config import VISION_INSTRUCTION
from devplacepy.services.openai_gateway.reliability import (
_backoff,
_is_retryable_status,
_retry_after_seconds,
)
from devplacepy.services.openai_gateway.thinking import apply_thinking
from devplacepy.services.openai_gateway.usage import classify_error
@@ -97,6 +102,9 @@ class VisionAugmenter:
pricing=None,
context_map=None,
app_reference: str = "default",
breaker=None,
max_retries: int = 2,
retry_backoff_ms: int = 250,
):
self.vision_url = vision_url
self.vision_model = vision_model
@@ -110,6 +118,9 @@ class VisionAugmenter:
self.pricing = pricing
self.context_map = context_map or {}
self.app_reference = app_reference
self.breaker = breaker
self.max_retries = max(0, int(max_retries or 0))
self.retry_backoff_ms = max(0, int(retry_backoff_ms or 0))
self.calls = 0
self.cost_usd = 0.0
@@ -142,6 +153,9 @@ class VisionAugmenter:
async def _describe_one(self, client: httpx.AsyncClient, image_block: dict) -> str:
if not self.vision_key:
return "[vision unavailable: vision API key not configured]"
if self.breaker is not None and not self.breaker.allow():
self._record((0.0), 503, False, "circuit_open", None)
return "[vision unavailable: circuit breaker open]"
payload = {
"model": self.vision_model,
"messages": [
@@ -166,18 +180,49 @@ class VisionAugmenter:
if self.title:
headers["X-Title"] = self.title
start = time.monotonic()
try:
resp = await client.post(self.vision_url, json=payload, headers=headers)
except httpx.RequestError as e:
logger.warning("vision connection failed: %s", e)
attempt = 0
resp: Optional[httpx.Response] = None
exc: Optional[Exception] = None
while True:
attempt += 1
try:
resp = await client.post(self.vision_url, json=payload, headers=headers)
exc = None
except httpx.RequestError as e:
resp = None
exc = e
if exc is None and not _is_retryable_status(resp.status_code):
break
if attempt > self.max_retries:
break
retry_after = _retry_after_seconds(resp) if resp is not None else None
logger.info(
"vision call retrying (%d/%d): %s",
attempt,
self.max_retries,
exc or f"HTTP {resp.status_code}",
)
if resp is not None:
try:
await resp.aclose()
except Exception: # noqa: BLE001
pass
await _backoff(self.retry_backoff_ms, attempt, retry_after)
if self.breaker is not None:
if exc is not None or (resp is not None and resp.status_code >= 500):
self.breaker.record_failure()
else:
self.breaker.record_success()
if exc is not None:
logger.warning("vision connection failed: %s", exc)
self._record(
(time.monotonic() - start) * 1000,
502,
False,
classify_error(0, e),
classify_error(0, exc),
None,
)
return f"[vision call failed: {e}]"
return f"[vision call failed: {exc}]"
latency_ms = (time.monotonic() - start) * 1000
if resp.status_code != 200:
logger.warning("vision %s: %s", resp.status_code, resp.text[:200])
@@ -191,10 +236,15 @@ class VisionAugmenter:
return f"[vision failed: HTTP {resp.status_code}]"
try:
data = resp.json()
error = data.get("error") if isinstance(data, dict) else None
if error and "choices" not in data:
message = error.get("message") if isinstance(error, dict) else str(error)
self._record(latency_ms, 502, False, "upstream_error", None)
return f"[vision failed: {message}]"
text = data["choices"][0]["message"].get("content") or ""
self._record(latency_ms, 200, True, None, data.get("usage"))
return text.strip() or "[vision returned empty response]"
except (KeyError, IndexError, ValueError) as e:
except (KeyError, IndexError, ValueError, httpx.HTTPError) as e:
self._record(latency_ms, 200, False, "gateway", None)
return f"[vision parse error: {e}]"