Add admin-unlimited workspaces, AI gateway model fallback, and real streaming/thinking control

Admin-unlimited Dev Workspaces: an admin-owned workspace is now exempt from the
max-workspace-count limit, the max-tunnel-count limit, and the whole
idle-stop/idle-warn/retention-delete lifecycle. Resolved once in
quota.resolve() as Limits.unlimited (owner uid checked against
get_admin_uids()), consumed at the three enforcement points
(provision.ensure, provision.publish_tunnel,
WorkspaceService._advance_lifecycle). Also hardens
get_admin_uids()/get_primary_admin_uid() against a partially-schemaed users
table (uid/role column guard), which a fresh test/init_db() path could hit.

AI gateway per-model automatic fallback: any gateway_models route
(chat/embed/image) can now name a fallback_model, picked on /admin/gateway
from a select box of other configured public model names of the same kind
only (never an internal upstream model id). When a route fails after its own
retries are exhausted, the gateway retries once, automatically, against the
fallback's own provider/pricing/key, before any bytes reach the client
(including for a streaming response). One hop only, no chains or cycles;
self-reference and cross-kind fallbacks are rejected at write time.

AI gateway real upstream streaming and thinking-default control: stream:true
is now forwarded to the upstream and relayed to the client as real SSE
chunks (measured TTFT/inter-token latency) instead of a simulated split
response, and every chat/vision call explicitly disables model "thinking" by
default (admin-overridable via gateway_thinking), with per-dialect handling
for DeepSeek, OpenRouter, and Ollama.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TjdKTWgWpW2SMNW8SFqxz5
This commit is contained in:
2026-09-03 08:47:57 +02:00
co-authored by Claude Sonnet 5
parent 8ae3f628c7
commit a693a6f4d8
33 changed files with 1310 additions and 110 deletions
+46 -4
View File
@@ -4,13 +4,15 @@ This file documents the AI gateway subsystem (devplacepy/services/openai_gateway
`GatewayService` (`services/openai_gateway/`) is an OpenAI-compatible LLM gateway (the ported `openai5.py`), mounted at `/openai/v1/*` (`routers/openai_gateway.py`, prefix `/openai`). It is a **service that serves an HTTP endpoint** rather than a loop.
- The router is thin: it calls `service_manager.get_service("openai").handle(request, subpath)`. `POST /v1/chat/completions` runs the full gateway (vision augment -> model override -> forward -> optional SSE re-emit via `_fake_stream`); `POST /v1/embeddings` forwards to the configured embeddings upstream (`handle_embeddings`, no vision/streaming); `GET /v1/models` is answered locally from the `gateway_models` chat routes (`_models_response`, publishing the public `molodetz`/`molodetz-pro` names), NOT proxied upstream; `/v1/{path:path}` is a transparent passthrough to the upstream base. **The gateway always forwards NON-streaming upstream** (`payload["stream"] = False`), then re-emits SSE itself when the client asked for `stream`; because `stream_options` is only valid alongside `stream=true`, `handle_chat` strips `stream_options` from the upstream payload (otherwise DeepSeek rejects it with `stream_options should be set along with stream = true`) and, when the client requested `stream_options.include_usage`, `_fake_stream` appends a final `choices: []` usage chunk built from the upstream `usage` before `[DONE]`. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
- The router is thin: it calls `service_manager.get_service("openai").handle(request, subpath)`. `POST /v1/chat/completions` runs the full gateway (vision augment -> model override -> thinking default -> forward, real upstream streaming when the client asked for it); `POST /v1/embeddings` forwards to the configured embeddings upstream (`handle_embeddings`, no vision/streaming); `GET /v1/models` is answered locally from the `gateway_models` chat routes (`_models_response`, publishing the public `molodetz`/`molodetz-pro` names), NOT proxied upstream; `/v1/{path:path}` is a transparent passthrough to the upstream base. **The gateway forwards `stream: true` to the upstream verbatim and relays its SSE chunks as they arrive** (`GatewayRuntime._stream_chat_response`, `client.send(request, stream=True)` + `resp.aiter_lines()`) - so TTFT and inter-token latency are real, measured timings, not simulated. When streaming, the gateway ALWAYS additionally sends `stream_options: {"include_usage": true}` upstream regardless of what the client asked (so the ledger always gets a real `usage` object off the final chunk); the client's own `include_usage` only controls whether that usage-only chunk is relayed downstream to it - the gateway still swallows-and-records it either way. **Load-bearing header trade-off: a streaming response carries no `X-Gateway-Cost-USD`/token headers** (only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference`) - HTTP headers must be sent before the body, and cost/tokens aren't known until the stream ends, so the usage ledger row (and the client's own final `usage` chunk, if it asked for one) are the only ways to learn a streamed call's cost; every non-streaming response is completely unaffected and keeps the full header set exactly as before. The ledger row for a streamed call is written from inside the generator's `finally` once the stream ends (or is interrupted - a caught mid-stream exception or a client disconnect, i.e. `GeneratorExit`, both still write a row with `success=False` and an `error_category`, never silently drop the accounting); a client hangup never leaves an unclosed upstream connection (`resp.aclose()` always runs). **Thinking is off by default** (`thinking.py` `apply_thinking`): DeepSeek V4 enables thinking unless `thinking.type=disabled` is sent, so the gateway always writes an explicit disable unless the client (or `gateway_thinking`) asked for thinking. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
- `main.py` registers it and exempts `/openai` from the rate-limit middleware and the maintenance gate. No new dependency (`httpx` already required).
- `GatewayService` is `default_enabled=True` so internal callers work out of the box.
## Per-call cost/token response headers
Every gateway response (chat, embeddings, passthrough; success and error) carries `X-Gateway-*` headers describing that single call: `Model`, `Backend`, `Prompt-Tokens`, `Completion-Tokens`, `Total-Tokens`, `Cache-Hit-Tokens`, `Cache-Miss-Tokens`, `Reasoning-Tokens`, `Cost-USD`/`Input-Cost-USD`/`Output-Cost-USD` (dollars), `Cost-Native` (1 if the upstream returned a native cost), `Tokens-Per-Second`, `Upstream-Latency-Ms`, and `Context-Window`/`Context-Utilization` when the model's window is known, plus the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`. `GatewayUsageLedger.record(...)` RETURNS the computed ledger row (or `None` on failure); each handler's `finalize` closure maps it through `usage.usage_response_headers(row)` and attaches it to the returned `Response` (`resp_headers`). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads `X-Gateway-Cost-USD`/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root `CLAUDE.md`).
Every NON-STREAMING gateway response (chat, embeddings, images, passthrough; success and error) carries `X-Gateway-*` headers describing that single call: `Model`, `Backend`, `Prompt-Tokens`, `Completion-Tokens`, `Total-Tokens`, `Cache-Hit-Tokens`, `Cache-Miss-Tokens`, `Reasoning-Tokens`, `Cost-USD`/`Input-Cost-USD`/`Output-Cost-USD` (dollars), `Cost-Native` (1 if the upstream returned a native cost), `Tokens-Per-Second`, `Upstream-Latency-Ms`, and `Context-Window`/`Context-Utilization` when the model's window is known, plus the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`. `GatewayUsageLedger.record(...)` RETURNS the computed ledger row (or `None` on failure); each handler's `finalize` closure maps it through `usage.usage_response_headers(row)` and attaches it to the returned `Response` (`resp_headers`). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads `X-Gateway-Cost-USD`/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root `CLAUDE.md`); every internal consumer sends `stream: false` (or omits it), so this path is completely unaffected by the streaming trade-off below.
**A STREAMING chat response (`stream: true`) is the one exception and carries only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference`** - HTTP headers are sent before the body, and cost/tokens for a streamed call are only known once the stream ends, so they cannot be headers on that same response. The call is still fully metered: the ledger row (including `ttft_ms`/`inter_token_ms`) is written server-side once the stream completes (see "Real upstream streaming" below), and a client that requests `stream_options.include_usage` still gets the upstream's real `usage` object on the final SSE chunk, exactly as the OpenAI streaming API itself works - it is simply not summarized into response headers.
## App-reference header (`X-App-Reference`)
@@ -33,7 +35,7 @@ Callers may send an optional `X-App-Reference` header to tag gateway calls by ap
## Config fields
All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble`, timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble` and `gateway_thinking` (default off), timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
## System-message composition (date awareness + operator preamble)
@@ -46,6 +48,31 @@ Two behaviors:
Embeddings/passthrough are untouched by this composition step.
## Thinking default (fast path)
`thinking.py` (`apply_thinking`) runs in `handle_chat` after `stream_options` is stripped, and in the vision describe call. DeepSeek V4 **enables thinking by default**; leaving the payload alone is the slow path. The gateway therefore always writes an explicit thinking field in the upstream dialect:
- DeepSeek (`*deepseek.com*` and any unrecognised OpenAI-compat URL, because that is the production default): `thinking: {"type": "disabled"}` or `"enabled"`
- OpenRouter: `reasoning: {"effort": "none"}` or `"high"`
- Ollama (`:11434` or a host containing `ollama`): `think: false` or `true`
Do **not** send `reasoning_effort: "none"` to DeepSeek (400: unknown variant). Conflicting client fields (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs.enable_thinking`) are stripped and translated to the dialect above.
**Default is off.** `gateway_thinking` (Prompt group, bool, default `False`) is the operator override for the silent-client case. A client still wins: `think` / `thinking.type` / `reasoning.effort` / `reasoning.enabled` / `reasoning_effort` / `enable_thinking` on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.
Vision describe-image calls always disable thinking (they are not a reasoning job).
## Real upstream streaming (`GatewayRuntime._stream_chat_response`)
`stream: true` is forwarded to the upstream verbatim (`payload["stream"] = stream`, no more forced `false`) and the connection is opened with `client.send(request, stream=True)` so the response body is read incrementally via `resp.aiter_lines()` instead of buffered whole. Every retry/circuit-breaker mechanic in `_send`/`retry_send` is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; `retry_send` now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns `200` does the code path diverge into `_stream_chat_response`.
- **The client always gets real chunks, in real time**, each parsed `data: {...}` line re-serialized and yielded as its own SSE event - no more splitting a complete response into fake 50-character deltas.
- **TTFT and inter-token latency are measured, not simulated.** `ttft_ms` = time from just before the upstream POST to the first SSE line received; `inter_token_ms` = the average gap between content-bearing chunks (`delta.content`/`delta.reasoning_content`) after the first one, `None` when fewer than two content chunks arrived. Both ride the ledger row (`gateway_usage_ledger.ttft_ms`/`.inter_token_ms`, nullable REAL columns dataset auto-adds on the first insert carrying them - no explicit `init_db()` ensure-block needed, same as every other column on this insert-only table) and surface in `/admin/ai-usage`'s `latency.ttft_ms`/`latency.inter_token_ms` (`analytics.py`, same `_pset` percentile shape as every other latency dimension).
- **Usage is always requested from the upstream when streaming**, regardless of what the client asked (`stream_options.include_usage` is forced `true` in the upstream payload), so the ledger always gets a real `usage` object off the final chunk. The client's OWN `stream_options.include_usage` only controls whether that usage-only chunk is relayed downstream to it; the gateway swallows-and-records it either way.
- **The ledger row is written once, from inside the generator's `finally`**, after the stream ends (`saw_done`), is interrupted by an upstream error, or the client disconnects (`GeneratorExit` - re-raised after cleanup, never swallowed, and never yielded-from during close: Python forbids yielding while handling `GeneratorExit`). All three paths close the upstream response (`resp.aclose()`) so a dropped client never leaks the connection, and all three still write a ledger row (`success=False`/`error_category="client_disconnected"` or a classified error on interruption) so accounting is never silently skipped.
- **No `X-Gateway-Cost-USD`/token headers on the streaming response** - see "Per-call cost/token response headers" above; this is the one deliberate exception to "every response carries the headers."
- Non-streaming chat, embeddings, and images are completely unchanged - this only touches the `stream: true` branch of `handle_chat`.
## Usage, cost, latency, and reliability tracking
The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:
@@ -53,7 +80,7 @@ The gateway records one row per upstream call (chat, vision, passthrough) and su
- **`usage.py`** - `GatewayUsageLedger` writes to `gateway_usage_ledger` (one row per call, success and failure) and samples `gateway_concurrency_samples` each 30s tick. `normalize_usage` handles BOTH upstream shapes: DeepSeek (`prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`, `completion_tokens_details.reasoning_tokens`) and OpenRouter (`prompt_tokens_details.cached_tokens`, native `cost`, `cost_details`). `compute_cost` PREFERS the upstream native `cost` (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share `input_cost / (input_cost + output_cost)` from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). `record()` is wrapped in try/except so a tracking failure never breaks the proxy.
- **`gateway.py`** - `_send` is the gated/timed/retried path: a `CircuitBreaker` (`reliability.py`) short-circuits when open; `retry_send` retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the `queue_wait_ms`; an httpx per-request `trace` extension captures `connect_ms` (0 on pooled reuse). `handle_chat`/`handle_passthrough` record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from `VisionAugmenter._describe_one`.
- **Owner attribution** - `GatewayService.resolve_owner` maps the caller to `(owner_kind, owner_id)`: `internal:devii` (internal key), `key:access` (static key), `user:<uid>`/`admin:<uid>` (session or API key), else `anonymous`.
- **`analytics.py`** - `build_analytics(hours, top_n, pricing)` pulls the bounded window once (clamped to `MAX_WINDOW_HOURS=168`) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via `reliability.percentile`), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. `summary_metrics()` runs cheap SQL aggregates for the live service panel. TTFT/inter-token are returned `null` (gateway forwards non-streaming).
- **`analytics.py`** - `build_analytics(hours, top_n, pricing)` pulls the bounded window once (clamped to `MAX_WINDOW_HOURS=168`) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via `reliability.percentile`), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. `summary_metrics()` runs cheap SQL aggregates for the live service panel. TTFT/inter-token (`latency.ttft_ms`/`latency.inter_token_ms`, the same `_pset` percentile-set shape as every other latency dimension) are real, measured only for calls that requested streaming - `count: 0` in a window means no streaming traffic occurred, not that the metric is unavailable.
- **Admin** - `/admin/ai-usage` (HTML, `templates/ai_usage.html` + `static/js/AiUsageMonitor.js`) and `/admin/ai-usage/data?hours=&top_n=` (JSON) in `routers/admin/` package, admin-gated. Schema `GatewayUsageOut`. Devii action `ai_usage` and docs `admin-ai-usage` expose the same endpoint. The `ai_usage` and `site_analytics` Devii catalog tools are both `requires_admin=True`.
- **Per-user usage** - `analytics.build_user_usage(owner_id, hours=24, pricing)` filters the ledger by `owner_id` (a uid matches both `user:` and `admin:` rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at `GET /admin/users/{uid}/ai-usage` (schema `UserAiUsageOut`, docs `admin-user-ai-usage`) and rendered on the profile page by `static/js/UserAiUsage.js` (the `[data-ai-usage]` card, gated `{% if is_admin(user) %}` in `templates/profile.html`). The 24h window is a sound projection basis (unlike a startup burst), so `cost_24h * 30` is honest.
- **Profile visibility split** - `routers/profile/` package `_ai_quota(uid, include_cost=False)` builds the profile quota bar. **The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal:** `spent_usd`/`used_pct`/`requests`/`last_used` come from `analytics.user_spend_24h(uid)`, the SAME `gateway_usage_ledger` source (summed by `owner_id`, matching both `user:` and `admin:` rows) that feeds the stat grid's `cost.window_usd`. So the server-rendered `spent_usd` is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. `turns` is still the `devii_usage_ledger` `turns_24h` count (Devii conversation turns) and is labeled **"Devii turns"** in the template precisely because it is a *different metric* from the grid's total **API requests** (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why `0 Devii turns` can sit beside `18 API requests`). The `limit`/`turns` still come from the Devii service (`daily_limit_for`/`turns_24h`); an **unlimited** limit (`<= 0`, the admin default) sets `unlimited=True` so the template renders "no cap" / "No daily spend cap" instead of a misleading `/ $0.00`. **This is display-only: the Devii daily-cap ENFORCEMENT and `/devii/usage` still read the Devii-scoped `devii_usage_ledger.spent_24h`** (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: **admin** sees the full gateway stats card *plus* a server-rendered quota bar (with spend/limit/Devii turns); the **owner** (non-admin, `is_self`) sees a quota-**only** card showing just the percentage; everyone else sees nothing AI-related. `_ai_quota` is fail-safe (returns `None` on any error or when the Devii service is unregistered) and the result rides on `ProfileOut.ai_quota`. **Dollar figures (`spent_usd`/`limit_usd`) are included only when `include_cost=viewer_is_admin`** - never for a non-admin owner. This matters because the route serves the same context as JSON via `respond(..., model=ProfileOut)` and `ProfileOut.ai_quota` is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with `Accept: application/json`; the server-side omission is the real control. Non-admins (HTML and JSON) get only `used_pct`/`turns`/`requests`.
@@ -120,6 +147,21 @@ When adding a routed value, overlay it as the matching `gateway_*` cfg key so th
**Tests:** `tests/unit/services/openai_gateway/routing.py` (overlay/economy/kind isolation), `tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream` (end-to-end through `handle_chat`), and `tests/api/admin/gateway/` (admin CRUD + validation + role gating).
## Automatic model fallback (`fallback_model`, one hop, per route)
Any `gateway_models` route (chat, embed, or image) can optionally name `fallback_model`: another already-configured `source_model` of the SAME kind, tried once, automatically, when the primary route fails. This is admin-configured on the `/admin/gateway` model form as a select box populated from every OTHER route of the current kind (the public `source_model` names such as `molodetz`/`molodetz-pro`, never the internal `target_model` a provider actually sees) - never a free-text field, so a fallback can only ever point at a model the gateway already knows how to serve.
- **Trigger.** `_call_failed(resp, exc, timing)` in `gateway.py` treats a call as failed when the circuit breaker rejected it, the upstream connection raised, or the upstream answered with any status `>= 400`. This runs AFTER `_send`'s own `gateway_max_retries` retries against the primary model are exhausted - a fallback is the next escalation once retrying the SAME model has already given up, not a replacement for that retry loop.
- **One hop, never a chain.** `routing.resolve_fallback(source_model, kind)` resolves the primary route, reads its `fallback_model`, and resolves THAT model's own route (must exist, be active, and share the kind) - it does NOT recurse into the fallback's own `fallback_model`, so there is no possibility of a cycle or an unbounded retry chain. A route that fails even after redirecting to its fallback returns the fallback attempt's own failure to the caller.
- **Self-reference is rejected at write time.** `ModelRouteIn._check_fallback_is_not_self` (pydantic `model_validator`) refuses `fallback_model == source_model`; the admin route handler (`routers/admin/gateway_configs.py::save_model`) additionally rejects a `fallback_model` that does not resolve to an existing, same-kind route (`routing.model_store.get(...)`) before writing the row, since a pydantic validator alone cannot see other rows.
- **Rebuilt like a fresh routed call, not retried in place.** On failure, `handle_chat`/`handle_embeddings`/`handle_images` recompute the overlay from `base_cfg` (the pre-primary-overlay config) via `chat_overlay(fallback_route.source_model, base_cfg)`/`embed_overlay`/`image_overlay`, so the fallback gets its OWN provider, URL, key, and pricing - never the primary route's. `pricing`/`context_map` for the ledger are (re)computed AFTER the fallback decision so the recorded cost always reflects whichever model actually served the request.
- **One ledger row per client call, not one per attempt.** The primary failure never writes a `gateway_usage_ledger` row by itself (only `_send`'s own internal retries and the circuit breaker counters observe it); `finalize()`/`self._ledger.record(...)` still runs exactly once, after the fallback decision, with `model` = whichever model ultimately answered and `requested_model` = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in `/admin/ai-usage` as "requested X, served Y" rather than as two calls.
- **Streaming falls back before any byte reaches the client.** `client.send(request, stream=True)` returns as soon as the response headers arrive, so `resp.status_code` is already known before the SSE body is ever touched - `handle_chat` decides whether to fall back at that same point, before it ever opens `_stream_chat_response` to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
- **Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (`self._breaker`) is one instance per `GatewayRuntime`, shared across every route.** A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see `circuit_open` too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
- **Deliberately NOT covered:** `gateway_max_retries`/`gateway_retry_backoff_ms`, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.
**Tests:** `tests/unit/services/openai_gateway/routing.py` (`resolve_fallback` self-reference/kind/active-flag guards), `tests/unit/services/openai_gateway/gateway.py` (`test_chat_falls_back_when_the_primary_model_fails`, `test_chat_returns_the_fallback_failure_when_both_models_fail`, `test_chat_without_a_fallback_never_makes_a_second_call`), and `tests/api/admin/gateway/models.py` (`test_model_route_fallback_round_trip`, `test_model_route_fallback_must_reference_an_existing_route`, `test_model_route_fallback_must_be_the_same_kind`, `test_model_route_fallback_rejects_self_reference`).
## Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)
Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME `gateway_models` route row, so it stays fully provider-agnostic and opt-in per route.
@@ -135,7 +135,7 @@ def empty_payload(hours: int = 48) -> dict:
"cost": {},
"behavior": {},
"hourly": [],
"notes": {"ttft": "not available: gateway forwards non-streaming upstream"},
"notes": {"ttft": "measured only for calls that requested streaming (stream: true); zero count means no streaming calls in this window"},
}
@@ -309,8 +309,8 @@ def build_analytics(
"upstream_availability_pct": round(success / requests * 100, 2)
if requests
else 0.0,
"ttft_ms": None,
"inter_token_ms": None,
"ttft_ms": _pset(_positive(rows, "ttft_ms")),
"inter_token_ms": _pset(_positive(rows, "inter_token_ms")),
}
errors = {
@@ -377,7 +377,7 @@ def build_analytics(
"cost": cost,
"behavior": behavior,
"hourly": hourly,
"notes": {"ttft": "not available: gateway forwards non-streaming upstream"},
"notes": {"ttft": "measured only for calls that requested streaming (stream: true); zero count means no streaming calls in this window"},
}
@@ -7,6 +7,7 @@ TIMEOUT_MIN = 300
INSTANCES_DEFAULT = 4
SYSTEM_PREAMBLE_DEFAULT = ""
THINKING_DEFAULT = False
VISION_URL_DEFAULT = "https://openrouter.ai/api/v1/chat/completions"
VISION_MODEL_DEFAULT = "google/gemma-3-12b-it"
+275 -74
View File
@@ -4,7 +4,6 @@ import asyncio
import json
import logging
import time
import uuid
from typing import Optional
import httpx
@@ -18,8 +17,10 @@ from devplacepy.services.openai_gateway.routing import (
chat_overlay,
embed_overlay,
image_overlay,
resolve_fallback,
)
from devplacepy.services.openai_gateway.system_message import apply_system_directives
from devplacepy.services.openai_gateway.thinking import apply_thinking
from devplacepy.services.openai_gateway.usage import (
GatewayUsageLedger,
classify_error,
@@ -34,6 +35,19 @@ from devplacepy.services.openai_gateway.vision import VisionAugmenter, VisionCac
logger = logging.getLogger(__name__)
def _call_failed(resp, exc, timing: dict) -> bool:
if timing.get("circuit_open") or exc is not None:
return True
return resp is not None and resp.status_code >= 400
def _fallback_headers(cfg: dict, key_field: str) -> dict:
headers = {"Content-Type": "application/json"}
if cfg.get(key_field):
headers["Authorization"] = f"Bearer {cfg[key_field]}"
return headers
def _notify_gateway_status(is_open: bool) -> None:
from devplacepy.database import get_table
from devplacepy.utils import create_notification
@@ -47,65 +61,6 @@ def _notify_gateway_status(is_open: bool) -> None:
create_notification(admin["uid"], "system", message, "gateway", "/admin/services/openai")
def _fake_stream(data: dict, model: str, include_usage: bool = False):
chunk_id = data.get("id", f"chatcmpl-{uuid.uuid4().hex[:12]}")
created = data.get("created", int(time.time()))
out_model = data.get("model", model)
try:
msg = data["choices"][0]["message"]
except (KeyError, IndexError):
msg = {"content": ""}
tool_calls = msg.get("tool_calls")
content = msg.get("content") or ""
reasoning_content = msg.get("reasoning_content") or ""
def _chunk(delta: dict, finish: Optional[str] = None) -> str:
return (
"data: "
+ json.dumps(
{
"id": chunk_id,
"object": "chat.completion.chunk",
"created": created,
"model": out_model,
"choices": [{"index": 0, "delta": delta, "finish_reason": finish}],
}
)
+ "\n\n"
)
async def gen():
yield _chunk({"role": "assistant"})
if reasoning_content:
for i in range(0, len(reasoning_content), 50):
yield _chunk({"reasoning_content": reasoning_content[i : i + 50]})
if tool_calls:
yield _chunk({"tool_calls": tool_calls})
elif content:
for i in range(0, len(content), 50):
yield _chunk({"content": content[i : i + 50]})
yield _chunk({}, finish="tool_calls" if tool_calls else "stop")
if include_usage and data.get("usage"):
yield (
"data: "
+ json.dumps(
{
"id": chunk_id,
"object": "chat.completion.chunk",
"created": created,
"model": out_model,
"choices": [],
"usage": data["usage"],
}
)
+ "\n\n"
)
yield "data: [DONE]\n\n"
return gen()
def _connect_tracer(holder: dict):
started: dict = {}
@@ -180,7 +135,17 @@ class GatewayRuntime:
self._instances = 0
async def _send(
self, client, sem, method, url, headers, cfg, log, json_body=None, content=None
self,
client,
sem,
method,
url,
headers,
cfg,
log,
json_body=None,
content=None,
stream=False,
):
timing = {
"queue_wait_ms": 0.0,
@@ -209,7 +174,7 @@ class GatewayRuntime:
method, url, headers=headers, json=json_body, content=content
)
request.extensions["trace"] = _connect_tracer(connect_holder)
return await client.send(request)
return await client.send(request, stream=stream)
send_start = time.monotonic()
resp, exc, attempts, queue_wait_ms = await retry_send(
@@ -256,6 +221,7 @@ class GatewayRuntime:
):
log = log or (lambda message: None)
overlay = chat_overlay(body.get("model"), cfg)
base_cfg = cfg
if overlay:
cfg = {**cfg, **overlay}
log(f"routed model {body.get('model')!r} -> {cfg['gateway_model']!r}")
@@ -299,8 +265,22 @@ class GatewayRuntime:
payload = dict(body)
payload["model"] = model
payload["messages"] = messages
payload["stream"] = False
payload.pop("stream_options", None)
payload["stream"] = stream
if stream:
# Always ask the upstream for usage on its final chunk, regardless of
# whether the client itself requested it, so the ledger always has real
# cost/token numbers for a streamed call; `include_usage` (the client's
# own ask) only controls whether that chunk is relayed to the client.
stream_options = dict(body.get("stream_options") or {})
stream_options["include_usage"] = True
payload["stream_options"] = stream_options
else:
payload.pop("stream_options", None)
apply_thinking(
payload,
cfg.get("gateway_upstream_url", ""),
default_enabled=bool(cfg.get("gateway_thinking", False)),
)
headers = {"Content-Type": "application/json"}
if cfg["gateway_api_key"]:
@@ -310,6 +290,7 @@ class GatewayRuntime:
"No upstream API key configured (gateway_api_key / DEEPSEEK_API_KEY / OPENROUTER_API_KEY); upstream will likely reject the request"
)
send_start = time.monotonic()
resp, exc, timing = await self._send(
client,
sem,
@@ -319,13 +300,56 @@ class GatewayRuntime:
cfg,
log,
json_body=payload,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
# The streamed response body hasn't been read yet; the shared error
# handling below needs resp.text/.content, which requires an explicit
# read for a stream=True response (a no-op if already buffered).
await resp.aread()
requested_model = requested
if _call_failed(resp, exc, timing):
fallback_route = resolve_fallback(requested, "chat")
fallback_overlay = (
chat_overlay(fallback_route.source_model, base_cfg)
if fallback_route is not None
else None
)
if fallback_overlay:
log(
f"model {requested!r} failed, falling back to "
f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_model']!r}"
)
cfg = {**base_cfg, **fallback_overlay}
model = cfg["gateway_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_api_key")
send_start = time.monotonic()
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_upstream_url"],
headers,
cfg,
log,
json_body=payload,
stream=stream,
)
if resp is not None and stream and resp.status_code != 200:
await resp.aread()
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
base = {
"owner_kind": owner[0],
"owner_id": owner[1],
"backend": "chat",
"endpoint": "chat/completions",
"requested_model": requested_model,
"model": model,
"user_agent": user_agent,
"app_reference": app_reference,
@@ -394,6 +418,32 @@ class GatewayRuntime:
content={"error": {"message": resp.text, "type": "upstream_error"}},
headers=resp_headers,
)
if stream:
# Real upstream streaming: forward SSE chunks to the client as they
# arrive (so TTFT/inter-token latency are genuine), and finalize the
# ledger row from inside the generator once the stream ends - cost and
# token counts are not known yet at this point, so unlike the
# non-streaming response below, this response carries no
# X-Gateway-Cost-USD/token headers (HTTP headers must precede the body).
log(f"chat POST -> 200 stream ({timing['upstream_latency_ms']:.0f}ms to first byte)")
return StreamingResponse(
self._stream_chat_response(
resp,
base,
pricing,
context_map,
include_usage,
handle_start,
send_start,
log,
),
media_type="text/event-stream",
headers={
"X-Gateway-Model": model,
"X-Gateway-Backend": "chat",
"X-App-Reference": app_reference or "default",
},
)
try:
data = resp.json()
except ValueError:
@@ -412,24 +462,112 @@ class GatewayRuntime:
)
resp_headers = finalize(200, True, None, data.get("usage"))
log(f"chat POST -> 200 ({timing['upstream_latency_ms']:.0f}ms)")
if stream:
return StreamingResponse(
_fake_stream(data, model, include_usage),
media_type="text/event-stream",
headers=resp_headers,
)
return Response(
content=resp.content,
media_type="application/json",
headers=resp_headers,
)
async def _stream_chat_response(
self,
resp: httpx.Response,
base: dict,
pricing,
context_map: dict,
include_usage: bool,
handle_start: float,
send_start: float,
log,
):
first_at: Optional[float] = None
last_at: Optional[float] = None
content_chunks = 0
usage_captured: Optional[dict] = None
saw_done = False
success = True
error_category: Optional[str] = None
try:
async for raw_line in resp.aiter_lines():
line = raw_line.strip()
if not line or not line.startswith("data:"):
continue
now = time.monotonic()
if first_at is None:
first_at = now
last_at = now
data_str = line[len("data:") :].strip()
if data_str == "[DONE]":
saw_done = True
yield "data: [DONE]\n\n"
break
try:
chunk = json.loads(data_str)
except ValueError:
continue
choices = chunk.get("choices") or []
delta = (choices[0].get("delta") if choices else None) or {}
if delta.get("content") or delta.get("reasoning_content"):
content_chunks += 1
usage = chunk.get("usage")
if usage:
usage_captured = usage
if not include_usage:
# The client did not ask for the usage chunk; swallow it,
# we still captured it above for the ledger.
continue
yield f"data: {json.dumps(chunk)}\n\n"
except GeneratorExit:
success = False
error_category = "client_disconnected"
raise
except Exception as exc: # noqa: BLE001 - an upstream stream can drop mid-flight; never crash the response
success = False
error_category = classify_error(0, exc)
log(f"chat stream interrupted: {exc}")
finally:
try:
await resp.aclose()
except Exception: # noqa: BLE001 - releasing the connection must never mask the real outcome
pass
ttft_ms = (
round((first_at - send_start) * 1000, 3)
if first_at is not None
else None
)
inter_token_ms = (
round((last_at - first_at) * 1000 / (content_chunks - 1), 3)
if first_at is not None and last_at is not None and content_chunks > 1
else None
)
base["total_latency_ms"] = round(
(time.monotonic() - handle_start) * 1000, 3
)
base["gateway_overhead_ms"] = round(
max(
base["total_latency_ms"]
- base.get("upstream_latency_ms", 0.0)
- base.get("queue_wait_ms", 0.0),
0.0,
),
3,
)
base["status_code"] = 200
base["success"] = success
base["error_category"] = error_category
base["usage"] = usage_captured
base["ttft_ms"] = ttft_ms
base["inter_token_ms"] = inter_token_ms
self._ledger.record(base, pricing, context_map)
if not saw_done:
yield "data: [DONE]\n\n"
async def handle_embeddings(
self, body: dict, cfg: dict, owner: tuple, user_agent: str, app_reference: str, log=None
):
log = log or (lambda message: None)
vision_cost = 0.0
overlay = embed_overlay(body.get("model"), cfg)
base_cfg = cfg
if overlay:
cfg = {**cfg, **overlay}
log(f"routed embed model {body.get('model')!r} -> {cfg['gateway_embed_model']!r}")
@@ -463,8 +601,6 @@ class GatewayRuntime:
},
)
client, sem = self._ensure(cfg)
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
params = extract_params(body)
handle_start = time.monotonic()
@@ -499,11 +635,44 @@ class GatewayRuntime:
json_body=payload,
)
requested_model = requested
if _call_failed(resp, exc, timing):
fallback_route = resolve_fallback(requested, "embed")
fallback_overlay = (
embed_overlay(fallback_route.source_model, base_cfg)
if fallback_route is not None
else None
)
if fallback_overlay:
log(
f"embed model {requested!r} failed, falling back to "
f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_embed_model']!r}"
)
cfg = {**base_cfg, **fallback_overlay}
model = cfg["gateway_embed_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_embed_key")
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_embed_url"],
headers,
cfg,
log,
json_body=payload,
)
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
base = {
"owner_kind": owner[0],
"owner_id": owner[1],
"backend": "embed",
"endpoint": "embeddings",
"requested_model": requested_model,
"model": model,
"user_agent": user_agent,
"app_reference": app_reference,
@@ -598,6 +767,7 @@ class GatewayRuntime:
):
log = log or (lambda message: None)
overlay = image_overlay(body.get("model"), cfg)
base_cfg = cfg
if overlay:
cfg = {**cfg, **overlay}
log(
@@ -633,8 +803,6 @@ class GatewayRuntime:
},
)
client, sem = self._ensure(cfg)
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
params = extract_params(body)
handle_start = time.monotonic()
@@ -673,11 +841,44 @@ class GatewayRuntime:
json_body=payload,
)
requested_model = requested
if _call_failed(resp, exc, timing):
fallback_route = resolve_fallback(requested, "image")
fallback_overlay = (
image_overlay(fallback_route.source_model, base_cfg)
if fallback_route is not None
else None
)
if fallback_overlay:
log(
f"image model {requested!r} failed, falling back to "
f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_image_model']!r}"
)
cfg = {**base_cfg, **fallback_overlay}
model = cfg["gateway_image_model"]
payload = dict(payload)
payload["model"] = model
headers = _fallback_headers(cfg, "gateway_image_key")
resp, exc, timing = await self._send(
client,
sem,
"POST",
cfg["gateway_image_url"],
headers,
cfg,
log,
json_body=payload,
)
pricing = pricing_from_cfg(cfg)
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
base = {
"owner_kind": owner[0],
"owner_id": owner[1],
"backend": "image",
"endpoint": "images/generations",
"requested_model": requested_model,
"model": model,
"user_agent": user_agent,
"app_reference": app_reference,
@@ -97,6 +97,10 @@ async def retry_send(
continue
if resp.status_code >= 500 and attempts <= max_retries:
log(f"upstream {resp.status_code}, retrying ({attempts}/{max_retries})")
try:
await resp.aclose()
except Exception: # noqa: BLE001 - releasing the connection must never block a retry
pass
await _backoff(backoff_ms, attempts)
continue
return resp, None, attempts, queue_wait_ms
+26 -2
View File
@@ -87,13 +87,15 @@ def ensure_tables() -> None:
"price_cache_hit_per_m_tier2 REAL, price_cache_miss_per_m_tier2 REAL, "
"price_output_per_m_tier2 REAL, price_input_per_m_tier2 REAL, "
"off_peak_start_minute INTEGER, off_peak_end_minute INTEGER, "
"off_peak_discount_pct REAL DEFAULT 0, "
"off_peak_discount_pct REAL DEFAULT 0, fallback_model TEXT DEFAULT '', "
"is_active INTEGER DEFAULT 1, created_at TEXT, updated_at TEXT)"
)
models_table = get_table(MODELS_TABLE)
for column in MODEL_TIER2_COLUMNS:
if not models_table.has_column(column):
models_table.create_column_by_example(column, 0.0)
if not models_table.has_column("fallback_model"):
models_table.create_column_by_example("fallback_model", "")
try:
db.query(
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_providers_name ON "
@@ -159,6 +161,7 @@ class ModelRouteIn(BaseModel):
off_peak_start_minute: Optional[int] = Field(default=None, ge=0, le=1439)
off_peak_end_minute: Optional[int] = Field(default=None, ge=0, le=1439)
off_peak_discount_pct: float = Field(default=0.0, ge=0, le=100)
fallback_model: str = Field(default="", max_length=128)
is_active: bool = True
@field_validator("source_model", "target_model")
@@ -169,7 +172,7 @@ class ModelRouteIn(BaseModel):
raise ValueError("Model name is required")
return value
@field_validator("provider", "vision_provider", "vision_model")
@field_validator("provider", "vision_provider", "vision_model", "fallback_model")
@classmethod
def _strip(cls, value: str) -> str:
return (value or "").strip()
@@ -192,6 +195,12 @@ class ModelRouteIn(BaseModel):
)
return self
@model_validator(mode="after")
def _check_fallback_is_not_self(self) -> "ModelRouteIn":
if self.fallback_model and self.fallback_model == self.source_model:
raise ValueError("A model cannot fall back to itself")
return self
@dataclass(frozen=True)
class ModelRoute:
@@ -214,6 +223,7 @@ class ModelRoute:
off_peak_start_minute: Optional[int]
off_peak_end_minute: Optional[int]
off_peak_discount_pct: float
fallback_model: str
is_active: bool
@@ -250,6 +260,7 @@ def _route_from_row(row: dict) -> ModelRoute:
off_peak_start_minute=_opt_int(row, "off_peak_start_minute"),
off_peak_end_minute=_opt_int(row, "off_peak_end_minute"),
off_peak_discount_pct=float(row.get("off_peak_discount_pct") or 0.0),
fallback_model=str(row.get("fallback_model") or ""),
is_active=_as_bool(row.get("is_active", 1)),
)
@@ -375,6 +386,7 @@ class ModelStore:
"off_peak_start_minute": payload.off_peak_start_minute,
"off_peak_end_minute": payload.off_peak_end_minute,
"off_peak_discount_pct": payload.off_peak_discount_pct,
"fallback_model": payload.fallback_model,
"is_active": 1 if payload.is_active else 0,
"updated_at": _now(),
}
@@ -402,6 +414,18 @@ class ModelStore:
provider_store = ProviderStore()
model_store = ModelStore()
def resolve_fallback(source_model: Optional[str], kind: str) -> Optional[ModelRoute]:
route = model_store.resolve(source_model, kind)
if route is None or not route.fallback_model:
return None
if route.fallback_model == route.source_model:
return None
fallback = model_store.resolve(route.fallback_model, kind)
if fallback is None or fallback.source_model == route.source_model:
return None
return fallback
DEEPSEEK_DEFAULT_ROUTES = (
{
"source_model": "deepseek-v4-flash",
@@ -120,6 +120,18 @@ class GatewayService(BaseService):
"content, all in one system message). Leave blank to disable.",
group="Prompt",
),
ConfigField(
"gateway_thinking",
"Enable thinking by default",
type="bool",
default=config.THINKING_DEFAULT,
help="Off by default: the gateway disables model thinking on every chat "
"call (DeepSeek thinking.type=disabled, OpenRouter reasoning.effort=none, "
"Ollama think=false) unless the client explicitly enables it with think, "
"thinking, or reasoning. Fastest path. On: thinking is enabled unless the "
"client disables it.",
group="Prompt",
),
ConfigField(
"gateway_vision_enabled",
"Vision augmentation",
@@ -0,0 +1,125 @@
# retoor <retoor@molodetz.nl>
from typing import Any, Optional
CLIENT_THINKING_KEYS = (
"think",
"thinking",
"reasoning",
"reasoning_effort",
"enable_thinking",
)
FALSE_VALUES = {False, 0, "0", "false", "no", "off", "none", "disabled"}
TRUE_VALUES = {True, 1, "1", "true", "yes", "on", "enabled"}
EFFORT_VALUES = {"low", "medium", "high", "max", "xhigh"}
def thinking_dialect(url: str) -> str:
text = (url or "").lower()
if "deepseek.com" in text:
return "deepseek"
if "openrouter.ai" in text:
return "openrouter"
if "11434" in text or "ollama" in text:
return "ollama"
return "deepseek"
def _as_text(value: Any) -> str:
if isinstance(value, str):
return value.strip().lower()
return ""
def _is_false(value: Any) -> bool:
if isinstance(value, str):
return value.strip().lower() in FALSE_VALUES
return value in FALSE_VALUES
def _is_true(value: Any) -> bool:
if isinstance(value, str):
text = value.strip().lower()
return text in TRUE_VALUES or text in EFFORT_VALUES
return value in TRUE_VALUES
def client_thinking_enabled(body: Any) -> Optional[bool]:
if not isinstance(body, dict):
return None
thinking = body.get("thinking")
if isinstance(thinking, dict) and "type" in thinking:
kind = _as_text(thinking.get("type"))
if kind in ("disabled", "none", "off"):
return False
if kind in ("enabled", "on"):
return True
if "think" in body:
value = body.get("think")
if _is_false(value):
return False
if _is_true(value):
return True
return bool(value)
reasoning = body.get("reasoning")
if isinstance(reasoning, dict):
if "enabled" in reasoning:
return _flag_enabled(reasoning.get("enabled"))
effort = _as_text(reasoning.get("effort"))
if effort == "none":
return False
if effort in EFFORT_VALUES:
return True
if "reasoning_effort" in body:
effort = _as_text(body.get("reasoning_effort"))
if effort == "none":
return False
if effort in EFFORT_VALUES:
return True
if "enable_thinking" in body:
return _flag_enabled(body.get("enable_thinking"))
kwargs = body.get("chat_template_kwargs")
if isinstance(kwargs, dict) and "enable_thinking" in kwargs:
return _flag_enabled(kwargs.get("enable_thinking"))
return None
def _flag_enabled(value: Any) -> bool:
if _is_false(value):
return False
if _is_true(value):
return True
return bool(value)
def _clear_thinking_fields(payload: dict) -> None:
for key in CLIENT_THINKING_KEYS:
payload.pop(key, None)
kwargs = payload.get("chat_template_kwargs")
if isinstance(kwargs, dict):
kwargs.pop("enable_thinking", None)
if not kwargs:
payload.pop("chat_template_kwargs", None)
def write_thinking(payload: dict, dialect: str, enabled: bool) -> None:
_clear_thinking_fields(payload)
if dialect == "openrouter":
payload["reasoning"] = {"effort": "high" if enabled else "none"}
return
if dialect == "ollama":
payload["think"] = bool(enabled)
return
payload["thinking"] = {"type": "enabled" if enabled else "disabled"}
def apply_thinking(
payload: dict,
upstream_url: str,
default_enabled: bool = False,
) -> dict:
specified = client_thinking_enabled(payload)
enabled = default_enabled if specified is None else specified
write_thinking(payload, thinking_dialect(upstream_url), enabled)
return payload
@@ -533,6 +533,8 @@ class GatewayUsageLedger:
"circuit_open": 1 if raw.get("circuit_open") else 0,
"user_agent": (raw.get("user_agent") or "")[:300],
"app_reference": (raw.get("app_reference") or "default")[:30],
"ttft_ms": raw.get("ttft_ms"),
"inter_token_ms": raw.get("inter_token_ms"),
}
get_table(GATEWAY_LEDGER).insert(row)
self._audit(raw, norm, cost_usd)
@@ -595,6 +597,8 @@ class GatewayUsageLedger:
"circuit_open": 0,
"user_agent": "",
"app_reference": app_reference or "default",
"ttft_ms": None,
"inter_token_ms": None,
}
get_table(GATEWAY_LEDGER).insert(row)
self._audit_external(row)
@@ -10,6 +10,7 @@ from typing import Any, Optional
import httpx
from devplacepy.services.openai_gateway.config import VISION_INSTRUCTION
from devplacepy.services.openai_gateway.thinking import apply_thinking
from devplacepy.services.openai_gateway.usage import classify_error
logger = logging.getLogger(__name__)
@@ -153,6 +154,7 @@ class VisionAugmenter:
"temperature": 0.2,
"stream": False,
}
apply_thinking(payload, self.vision_url, default_enabled=False)
headers = {
"Authorization": f"Bearer {self.vision_key}",
"Content-Type": "application/json",