Files
devplacepy/devplacepy/services/openai_gateway/CLAUDE.md
T
retoorandClaude Sonnet 5.5 151476125f Gateway: auto-resolve request shape per provider and model
An upstream that rejects a key the gateway adds (e.g. thinking) now gets
retried with other shapes on a parsable 400/422; the working method is
stored per (provider, target model) and applied first-try from then on.
Fallback attempts are re-shaped per attempt. Admin list/reset, Devii
tools, audit event and ledger resolved_method included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-13 11:34:26 +01:00

94 KiB

This file documents the AI gateway subsystem (devplacepy/services/openai_gateway/) - the single point of truth for AI calls, usage metering, and provider/model routing. Claude Code loads it automatically whenever a file under this directory is read or edited.

Overview

GatewayService (services/openai_gateway/) is an OpenAI-compatible LLM gateway (the ported openai5.py), mounted at /openai/v1/* (routers/openai_gateway.py, prefix /openai). It is a service that serves an HTTP endpoint rather than a loop.

  • The router is thin: it calls service_manager.get_service("openai").handle(request, subpath). POST /v1/chat/completions runs the full gateway (vision augment -> model override -> thinking default -> forward, real upstream streaming when the client asked for it); POST /v1/embeddings forwards to the configured embeddings upstream (handle_embeddings, no vision/streaming); GET /v1/models is answered locally from the gateway_models chat routes (_models_response, publishing the public molodetz/molodetz-pro names), NOT proxied upstream; /v1/{path:path} is a transparent passthrough to the upstream base. The gateway forwards stream: true to the upstream verbatim and relays its SSE chunks as they arrive (GatewayRuntime._stream_chat_response, client.send(request, stream=True) + resp.aiter_lines()) - so TTFT and inter-token latency are real, measured timings, not simulated. When streaming, the gateway ALWAYS additionally sends stream_options: {"include_usage": true} upstream regardless of what the client asked (so the ledger always gets a real usage object off the final chunk); the client's own include_usage only controls whether that usage-only chunk is relayed downstream to it - the gateway still swallows-and-records it either way. Load-bearing header trade-off: a streaming response carries no X-Gateway-Cost-USD/token headers (only X-Gateway-Model/X-Gateway-Backend/X-App-Reference) - HTTP headers must be sent before the body, and cost/tokens aren't known until the stream ends, so the usage ledger row (and the client's own final usage chunk, if it asked for one) are the only ways to learn a streamed call's cost; every non-streaming response is completely unaffected and keeps the full header set exactly as before. The ledger row for a streamed call is written from inside the generator's finally once the stream ends (or is interrupted - a caught mid-stream exception or a client disconnect, i.e. GeneratorExit, both still write a row with success=False and an error_category, never silently drop the accounting); a client hangup never leaves an unclosed upstream connection (resp.aclose() always runs). Thinking is off by default (thinking.py apply_thinking): DeepSeek V4 enables thinking unless thinking.type=disabled is sent, so the gateway always writes an explicit disable unless the client (or gateway_thinking) asked for thinking. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
  • main.py registers it and exempts /openai from the rate-limit middleware and the maintenance gate. No new dependency (httpx already required).
  • GatewayService is default_enabled=True so internal callers work out of the box.

Per-call cost/token response headers

Every NON-STREAMING gateway response (chat, embeddings, images, passthrough; success and error) carries X-Gateway-* headers describing that single call: Model, Backend, Prompt-Tokens, Completion-Tokens, Total-Tokens, Cache-Hit-Tokens, Cache-Miss-Tokens, Reasoning-Tokens, Cost-USD/Input-Cost-USD/Output-Cost-USD (dollars), Cost-Native (1 if the upstream returned a native cost), Tokens-Per-Second, Upstream-Latency-Ms, and Context-Window/Context-Utilization when the model's window is known, plus the timing headers X-Gateway-Upstream-Latency-Ms/X-Gateway-Total-Latency-Ms. GatewayUsageLedger.record(...) RETURNS the computed ledger row (or None on failure); each handler's finalize closure maps it through usage.usage_response_headers(row) and attaches it to the returned Response (resp_headers). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads X-Gateway-Cost-USD/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root CLAUDE.md); every internal consumer sends stream: false (or omits it), so this path is completely unaffected by the streaming trade-off below.

A STREAMING chat response (stream: true) is the one exception and carries only X-Gateway-Model/X-Gateway-Backend/X-App-Reference - HTTP headers are sent before the body, and cost/tokens for a streamed call are only known once the stream ends, so they cannot be headers on that same response. The call is still fully metered: the ledger row (including ttft_ms/inter_token_ms) is written server-side once the stream completes (see "Real upstream streaming" below), and a client that requests stream_options.include_usage still gets the upstream's real usage object on the final SSE chunk, exactly as the OpenAI streaming API itself works - it is simply not summarized into response headers.

Upstream header collision policy (usage.upstream_reported_model, gateway._apply_served_model)

An upstream this gateway forwards to may itself be a DevPlace-style gateway (or any provider that happens to emit the same X-Gateway-* header names on its own response), so its raw response headers can collide with the ones we are about to build. Exactly one upstream header is ever trusted and relayed: X-Gateway-Model. Every other X-Gateway-*/X-App-Reference header is always our own measurement, computed as if the upstream had sent nothing at all - an upstream cannot be an authority on our own cost table, our own tokenizer counts, our own socket timings, or our own caller attribution.

  • Why only the model name. It is the one field an upstream can legitimately know better than we do: it may have resolved an alias, redirected to a fallback, or served a different pinned version than what we requested. Cost/tokens/latency/context/app-reference are, by construction, things WE measured about OUR call (our pricing table, our tokenizer expectations, our socket, our caller) - an upstream's own figures for those describe its own accounting, not ours, and blending them in would corrupt gateway_usage_ledger's per-model cost/token rollups and the quota math that reads these exact header names back out (usage.parse_usage_headers, consumed by services/correction.py/services/ai_modifier.py).
  • usage.upstream_reported_model(headers) extracts that one value defensively: it treats the upstream response as untrusted input and never raises. It looks up X-Gateway-Model case-insensitively (a direct .get, then a manual case-insensitive scan for a plain non-case-insensitive mapping), and rejects anything that is not a non-empty string, exceeds 200 characters, or contains a control character (\r/\n/\x00, a header-injection guard) - any of those makes it return None, i.e. "upstream said nothing usable."
  • gateway._apply_served_model(headers, resp) is the one place this is applied: called at the tail of every response-header build (chat non-streaming and streaming, embeddings, images, passthrough), it overwrites headers["X-Gateway-Model"] with the upstream's claim ONLY when one was found, and leaves every other header untouched. It never touches the ledger row itself (GatewayUsageLedger.record's model field, and therefore context_utilization/pricing lookups, stay keyed on the model WE resolved and billed against) - the override is display-only, informing the client which model actually answered without altering what we measured or charged for.
  • Tests: tests/unit/services/openai_gateway/usage.py (upstream_reported_model - case-insensitivity, missing/oversized/control-character rejection, non-dict/None inputs) and tests/unit/services/openai_gateway/gateway.py (test_upstream_x_gateway_model_header_is_forwarded_* - chat/streaming/embeddings/images/passthrough all forward an upstream-reported model while every cost/token header stays ours; test_upstream_x_gateway_model_header_absent_keeps_our_own_model - no upstream header is a no-op).

App-reference header (X-App-Reference)

Callers may send an optional X-App-Reference header to tag gateway calls by application. The value is validated and stored in the app_reference column of gateway_usage_ledger, surfaced in analytics and admin reporting.

  • Validation (service._validate_app_reference): trimmed whitespace, then matched against ^[a-zA-Z0-9_.-]{1,30}$. Any value failing validation (empty, >30 chars, contains spaces or @// etc.) silently falls back to "default".
  • Header name: X-App-Reference.
  • All internal callers (news, bots, Devii, correction, jobs, deepsearch, dbapi, gitea) should pass a devplace-<component>-v-<major>-<minor>-<patch> value, e.g. devplace-devii-v-1-0-0, devplace-news-v-1-0-0.
  • The column is indexed (CREATE INDEX IF NOT EXISTS) for fast per-app queries.

Provider attribution headers (apps.py)

apps.py is the single specification of every internal gateway app: the GATEWAY_APPS registry (GatewayApp(reference, name), title DevPlace <name>), the APP_REFERENCE_* constants callers import, validate_app_reference and attribution_headers(app_reference). Every upstream request (chat, embeddings, images, vision sub-call, fallback retries) carries, for ALL providers and not only OpenRouter: X-OpenRouter-Title (OpenRouter's current app-title header, https://openrouter.ai/docs/app-attribution), the legacy X-Title with the same value, and HTTP-Referer = DEVPLACE_SITE_URL when set. The title is per app (DevPlace Devii, DevPlace News, ...), so a provider dashboard splits spend by sub-app; unknown or default references fall back to DevPlace. OpenRouter documents no sub-app field beyond the title, so the title is the carrier. Apps: Devii, News, Bots, AI Correction, AI Modifier, Quiz Grading, SEO Metadata, DeepSearch, AI Usage Analyzer, Issue Enhance, Issue Planning, DB API, Awards, Image Generation, Community Discovery, Community Sync. A new internal caller adds one registry row and sends X-App-Reference. gateway_complete(..., app_reference=) carries the label for correction-style callers.

Tool-call emulation (tool_emulation.py, per-model tools_mode)

A model route (gateway_models.tools_mode, native default or emulated) declares whether the model supports tool calling. The admin's word is the truth: there is no probing and no auto-detection, and an emulated flag is never second-guessed. The flag rides the route overlay as gateway_tools_mode, so it is decided per attempt and a fallback chain may freely mix native and emulated models. The client keeps speaking plain OpenAI: it sends tools and gets tool_calls back, whatever the model underneath.

  • Request seam. gateway._chat_upstream_body calls emulate_request(payload) first, BEFORE the Responses/Meta transpile, so the Meta path (chat_to_responses, Muse aliasing) simply receives a tool-less Chat payload. As a side effect the Meta reserved tool-name gate never fires, because tools[] is gone; _response_tool_aliases returns {} for emulated routes. emulate_request strips tools/tool_choice/parallel_tool_calls/functions/function_call, appends a Hermes-format catalog (<tools> JSON lines plus the <tool_call>{"name","arguments"}</tool_call> contract) to the system message (creating one when absent), maps tool_choice (none drops the catalog, required/named add a MUST directive, parallel_tool_calls: false adds a one-call rule), rewrites history (assistant tool_calls become <tool_call> blocks, role: tool/function messages become one merged user message of <tool_response> blocks matched back to names by call id), and adds <tool_response> as a stop sequence so the model cannot invent a result. It never mutates the caller's payload.
  • Response seams. Non-streaming: convert_completion parses blocks out of message.content into tool_calls (generated call_ ids, finish_reason: "tool_calls", leftover text kept, content: null when empty), applied on both the chat and the Responses non-stream branches. Streaming: _emulate_tool_stream wraps the finished SSE iterator, so ledger/TTFT accounting stays inside the inner generator. ToolCallStream relays text live with a partial-<tool_call> holdback, buffers once a call opens, and emits each call whole in ONE tool_calls delta (never incremental arguments: that is where vLLM's Hermes parser kept breaking). Usage chunks and non-content deltas pass through.
  • Parsing is tolerant, validation strict. Accepts code fences, an array of calls in one block, parameters for arguments, string-encoded arguments, and an unterminated final block. A call is valid only if the name is declared and the schema's required keys are present. An invalid block is NEVER dropped silently: it stays in content as text and is logged (emulated tools: ...). Known gap: there is no repair retry; an invalid call is returned as text.
  • No count/size caps (deliberate). Any number of function tools is accepted and the full catalog is always injected - Devii alone declares hundreds of tools, so a count cap would make large catalogs unservable everywhere. Servability is bounded downstream: the context-window precheck routes an oversized request to a fallback model before dispatch, and the upstream's own context limit is relayed when nothing can serve it. What still 400s (no upstream call, no ledger row) is a malformed declaration: non-list tools, a non-function tool type, or a nameless function. MAX_CALLS_PER_TURN 8 (extra calls dropped with a warning) is unchanged. The catalog costs prompt tokens on every call, billed like any input.
  • Per-owner catalog cache (tool_emulation.CatalogCache, owned by GatewayRuntime). Building the catalog (spec validation + instruction assembly, milliseconds at hundreds of tools) runs once per distinct (owner, tools, tool_choice, parallel_tool_calls) fingerprint and is reused across turns, retries, and fallback attempts; the response seams (_emulated_tools) resolve declarations from the same entry instead of re-validating. Entries are namespaced by owner ((owner_kind, owner_id), so one user's tool set never serves another's) and bounded (MAX_ENTRIES 64, MAX_CHARS 2M, oldest-first eviction) - a cache key is content-derived, so no invalidation is needed.
  • Injection cadence: once per request, every request. The catalog is injected exactly once per upstream attempt (one <tools> block; history converts without repeated blocks; the stop sequence is added only when absent), but it is re-sent on every turn, retry, and fallback attempt, because the upstream is stateless and each call needs the catalog in context. Fallback attempts never stack injections: every attempt rebuilds from the pristine client payload (emulate_request never mutates its input), and the per-owner cache only skips the rebuild, never changes the bytes.
  • Upstream prefix caching. A prefix-caching upstream (DeepSeek context caching, tracked as prompt_cache_hit_tokens) can serve the repeated catalog from cache, but only while the bytes before it are stable. The catalog is appended after the client's system content, so anything volatile there re-bills the catalog too. Devii's system prompt is stable within the hour (its trailing # CURRENT TIME block is rounded to the hour, see the Devii file), so turns inside one hour share one prefix and the catalog hits; the hour rollover breaks it. Watch the cache-hit token split in /admin/ai-usage to confirm.
  • Surfaces. ModelRouteIn.tools_mode, the admin model form (Tool calling select), the tools: emulated badge, POST /admin/gateway/models (tools_mode, documented in docs_api) and the Devii gateway_model_set tool.
  • Not covered: constrained/guided decoding (provider-specific), and the format is Hermes JSON only. Tests: tests/unit/services/openai_gateway/tool_emulation.py (pure) and the test_emulated_tools_* tests in gateway.py (through the real Meta Responses path, streaming and non-streaming).

Per-worker runtime

Serves in every worker. Config/enabled come from site_settings (via the service's get_config()/is_enabled()), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (GatewayRuntime: httpx.AsyncClient pool + asyncio.Semaphore sized to gateway_instances, plus counters and a VisionCache) is created lazily on first request and rebuilt when instances/timeout/cache size change. gateway_instances is the scaling knob (concurrency per worker); process scaling is uvicorn workers.

Auth

authorize() reuses get_current_user (cookie/X-API-KEY/Bearer/Basic). Allowed if gateway_require_auth is off, the presented X-API-KEY/Bearer equals the static gateway_access_key or the auto-generated gateway_internal_key, or the resolved user is an admin (gateway_allow_admins) or any user (gateway_allow_users).

Per-user attribution: the gateway also accepts a real user's api_key (Bearer / X-API-KEY / session), gated by gateway_allow_users (default on); resolve_owner() records (owner_kind, owner_id) per call, so usage is attributed and limitable per user. With gateway_allow_users off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own api_key (set as the session's ai_key in build_settings(..., owner_kind="user")), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via build_user_usage() / GET /admin/users/{uid}/ai-usage. Guests keep the internal key.

Failed-auth throttle (services/openai_gateway/auth_throttle.py)

A dedicated per-IP throttle protects /openai/v1/* against unauthenticated probing WITHOUT IP-allowlisting the endpoint (which would break legitimate external API consumers using gateway_access_key/gateway_allow_users). It is scoped ONLY to failed authentication attempts, never to general traffic - the existing global rate limiter already exempts /openai entirely (main.py), and this throttle does not change that exemption.

  • Config (gateway_auth_throttle_enabled default on, gateway_auth_throttle_max_failures default 10, gateway_auth_throttle_window_seconds default 60, all group "Access"). An admin can retune or disable the throttle with no code change, same as every other gateway setting.
  • Counter. auth_throttle.py is a small in-process sliding-window store (dict[ip] -> list[float] timestamps), the same shape as the global rate limiter's _rate_limit_store in middleware/state.py, but a separate, dedicated store local to the gateway rather than reusing that one (the two throttles key on different events - all mutating traffic vs. failed gateway auth - and /openai is deliberately exempt from the general limiter). record_failure(ip, window) appends a timestamp and returns the running in-window count; is_throttled(ip, threshold, window) checks without recording. Per-worker, like every other in-process cache in this codebase - not divided by worker count, so the effective aggregate threshold across N workers is up to N x max_failures; this is an accepted tradeoff (it still closes the probe, just at a slightly higher aggregate bound) rather than importing state.WEB_WORKERS into a leaf service module.
  • GatewayService.authorize() records a failure on every deny path (a cheap static-key/internal-key match still short-circuits first and is never counted), so a wrong or garbage key counts exactly like presenting nothing. The fast 429 short-circuit fires only when the request carries no credential at all (no X-API-KEY, no Authorization header of any scheme, no session cookie) AND the IP is already over threshold - this is the "skip the rest of authorize()" defense against wasted CPU on a blind flood. A request presenting ANY credential - even one that turns out invalid - always runs the full check, so it can never be blocked by another caller's failures sharing its IP (a NAT/proxy caller with its own valid key is never punished for a neighbor's failed probes); only a request presenting nothing is fast-denied once its IP has tripped.
  • Audit. Tripping the throttle (the exact request whose failure brings the window count to the configured threshold) records ai.gateway_auth_throttle.tripped (category ai, result="denied", metadata ip/failed_attempts/threshold/window_seconds) via audit.record(request, ...). Every subsequent fast-denied request in the same trip is silent (no audit spam under an actual flood) - the trip itself is the visible signal for admins.
  • Internal traffic cannot trip this by construction. Every internal consumer (Devii guests, AI correction/modifier, quiz grading, SEO metadata generation, news, bots) authenticates with database.internal_gateway_key(), which is the exact same value authorize() compares against as gateway_internal_key - that comparison is the FIRST check in authorize(), before the throttle is even consulted, so internal self-dial calls always succeed there and never reach the failure-recording branch.

Config fields

All config is config_fields (upstream url/model/key, force-model, the Prompt group's gateway_system_preamble and gateway_thinking (default off), timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (gateway_embed_enabled/_url/_model/_key), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via collect_metrics.

Vision model (current: google/gemma-4-31b-it)

Image content is described by the vision model before forwarding (augmentation defaults on via gateway_vision_enabled). The live value is the gateway_vision_model site setting (database, editable in admin UI, hot-reloads); the code fallback for fresh databases is VISION_MODEL_DEFAULT in config.py with its context window in MODEL_CONTEXT_MAP_DEFAULT. Keep the two in sync when switching models: the DB row wins at runtime, the code default wins on a fresh install. History: gemma-3-12b-it default, gemma-3-27b-it via DB, gemma-4-31b-it since 08/11/2026 (cheaper small variant gemma-4-26b-a4b-it also verified on OpenRouter and rejected in favour of quality).

System-message composition (date awareness + operator preamble)

system_message.py (apply_system_directives) runs in handle_chat only (NOT handle_embeddings/handle_passthrough), after vision augmentation and before the upstream payload is assembled, so it reaches every chat consumer (news, bots, Devii guests, per-user direct API calls). It composes ONE system message in a fixed, cache-friendly order: operator preamble (gateway_system_preamble) -> date line (Current date: DD/MM/YYYY) -> the client's original system content.

Two behaviors:

  • Date awareness (EU, no time). The current date is injected only when the composed text (preamble + client system content) contains NO date in any common format. Detection lives in contains_date via the _DATE_PATTERNS regex set: YYYY-MM-DD (ISO), D/M/YYYY and DD/MM/YYYY (also matches MM/DD/YYYY), DD-MM-YYYY, DD.MM.YYYY, and textual months (14 June 2026, June 14, 2026, 14 Jun 2026, with abbreviations and optional ordinal/comma). The date is datetime.now().strftime("%d/%m/%Y") - date only, never time: omitting the time keeps the system message byte-stable for the whole day so upstream prompt/token caching stays effective.
  • Operator preamble. gateway_system_preamble (Prompt group, type="text" textarea, default empty) is prepended ahead of the client's system content on every chat call so all calls carry operator preferences. When the client sent NO system message, the composed content (preamble and/or date) becomes a brand-new leading system message; when the client DID send one, its content is replaced in place (preamble + date + original text). A blank/whitespace-only preamble with an already-dated request is a no-op (no empty system message is created). The date detection runs over the preamble too, so an operator preamble that already states a date suppresses the injected date line.

Embeddings/passthrough are untouched by this composition step.

Per-call operator preamble bypass (bypass_preamble, internal-only)

A structured/deterministic internal caller (one that expects a strict, machine-parsed output shape - grading JSON, a corrected text field, generated SEO metadata) can ask the gateway to skip gateway_system_preamble for that one call, since an operator preamble written for conversational tone can conflict with a task that demands exact output discipline. The date line and the client's own system content are unaffected either way.

  • Signal. The chat request body carries "bypass_preamble": true. GatewayRuntime.handle_chat pops it off body before it is ever copied into the upstream payload (so it never leaks to the upstream provider) and only honors it when the caller also proved internal credentials for that same request (see below); otherwise it is silently discarded, exactly like an ordinary, unsupported field.
  • Internal-only gate. GatewayService.internal_bypass_allowed(request, cfg) requires the request to carry the X-Gateway-Internal-Key header equal to the configured gateway_internal_key setting. This is a SEPARATE header from Authorization/X-API-KEY on purpose: a caller that authenticates with a real user's own api_key (correction, the @ai modifier, quiz free-text grading - all per-user-attributed on purpose, for correct spend attribution) still proves it is DevPlace's own server code, not the user's browser or a malicious client holding that same api_key, by additionally presenting the internal key on this header. A caller that authenticates AS the internal key directly (SEO metadata generation, via database.internal_gateway_key()) can present the same header too - one mechanism covers both shapes of internal caller. GatewayService.handle() computes this boolean once per request and passes it into handle_chat as bypass_allowed; a request with no valid internal credentials gets bypass_allowed=False and the body field is ignored even if present - it can never come from an external API consumer or a public-facing client call.
  • Callers that set it. The four structured/deterministic internal callers all route through the single shared services/correction.py::gateway_complete(..., bypass_preamble=True) helper, which sets both the body field and the X-Gateway-Internal-Key header (via database.internal_gateway_key()) in one place: services/correction.py::correct_text (AI content correction), services/ai_modifier.py::modify_text (the @ai inline modifier), services/quiz/grading.py::grade_free_text (quiz free-text grading), and services/jobs/seo_meta_service.py's generation call (SEO metadata). Genuinely conversational internal callers are deliberately left unchanged and keep receiving the operator preamble: Devii chat sessions and the docs chat do not call gateway_complete at all, so they are unaffected by construction.
  • Other internal self-dial callers found but NOT updated (each builds its own request instead of using gateway_complete, and each is a candidate for the same treatment in a follow-up): services/gitea/enhance.py::enhance_ticket and services/gitea/planning.py::generate_plan (both structured, single-shot ticket text generation), services/dbapi/nl2sql.py::design_query (NL-to-SQL, strictly structured), services/deepsearch/llm.py/services/jobs/deepsearch/enhance.py (planner/synthesis calls), and services/jobs/isslop/agent/llm.py (per-file classification). None of these were in scope for this change.

Thinking default (fast path)

thinking.py (apply_thinking) runs in handle_chat after stream_options is stripped, and in the vision describe call. DeepSeek V4 enables thinking by default; leaving the payload alone is the slow path. The gateway therefore always writes an explicit thinking field in the upstream dialect:

  • DeepSeek (*deepseek.com* and any unrecognised OpenAI-compat URL, because that is the production default): thinking: {"type": "disabled"} or "enabled"
  • OpenRouter: reasoning: {"effort": "none"} or "high"
  • Ollama (:11434 or a host containing ollama): think: false or true

thinking_dialect can also be none (no field written); it is only ever chosen by the auto-resolve ladder when an upstream rejects the thinking key (see "Auto-resolve of request shape").

Do not send reasoning_effort: "none" to DeepSeek (400: unknown variant). Conflicting client fields (think, thinking, reasoning, reasoning_effort, enable_thinking, chat_template_kwargs.enable_thinking) are stripped and translated to the dialect above.

Default is off. gateway_thinking (Prompt group, bool, default False) is the operator override for the silent-client case. A client still wins: think / thinking.type / reasoning.effort / reasoning.enabled / reasoning_effort / enable_thinking on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.

Vision describe-image calls always disable thinking (they are not a reasoning job).

Dialect can be overridden when the URL does not identify the provider. gateway_thinking_dialect (Prompt group, select, default auto, options auto/deepseek/openrouter/ollama) forces thinking.py::thinking_dialect's URL-sniffing result - needed for a reverse-proxied Ollama whose URL carries no :11434/ollama token. apply_thinking(payload, url, default_enabled, dialect=...) and the vision augmenter (VisionAugmenter.vision_dialect, passed from cfg["gateway_thinking_dialect"]) both accept it; "auto"/blank falls through to the existing URL-based detection unchanged.

thinking.upstream_capabilities(url, dialect="") derives per-upstream protocol flags (supports_stream_options, supports_stream_usage, supports_thinking_field) from the same effective dialect. Ollama's OpenAI-compatibility layer ignores stream_options (and rejects the request outright on some versions), so handle_chat's streaming branch only sends stream_options: {"include_usage": true} upstream when capabilities.supports_stream_options is true; otherwise a client that asked for include_usage gets a logged notice and its stream relayed with no usage chunk. DeepSeek/OpenRouter (and any unrecognised URL) report full support, so this is a no-op for them.

Auto-resolve of request shape (shape.py, learned.py)

An upstream with a strict schema can reject a key the gateway adds on its own (observed 2026-11-13: route vaids answered 400 {"issues":[{"code":"unrecognized_keys","keys":["thinking"]}]} because thinking_dialect falls back to deepseek for any unrecognised URL). A 400 is bad_request, never retried and never falls back, so the gateway now LEARNS, per (provider, target model), which shape the upstream accepts and applies it from then on.

  • One funnel. ShapeMethod.apply(payload, cfg, protocol, stream, log) is the only place stream_options, the thinking dialect and droppable keys are written; chat, every fallback attempt and the vision describe call all go through it, each from the PRISTINE client payload with that attempt's own cfg (fallback attempts used to inherit the primary's already-shaped payload). ShapeMethod.strip_wire removes drop_keys (and reasoning for a Responses none method) from the final wire body in _chat_upstream_body. A method is a frozen ShapeMethod(thinking_dialect, drop_keys); thinking_dialect is native (today's behaviour: the URL-sniffed or gateway_thinking_dialect dialect, the default and first ladder entry), none (write no thinking field at all, new in thinking.py), or an explicit deepseek/openrouter/ollama.
  • Trigger. Only a 400/422 whose body names rejected keys (RejectedKeys.parse: zod issues[].keys with unrecognized_keys, OpenAI Unrecognized request argument supplied/Unknown parameter + param, serde unknown field, "Extra inputs are not permitted"). An unparsable 400 stays terminal and loud, returned verbatim, never retried blind.
  • Ladder and allowlist (static ceilings, no autotuning). ShapeResolver.next_method picks the next untried method: a rejected thinking key (think, thinking, reasoning, reasoning_effort, enable_thinking, chat_template_kwargs) skips every dialect that emits it (CHAT_LADDER order: native, none, deepseek, openrouter, ollama; Responses protocol: native, none, rejecting reasoning); a rejected key in DROPPABLE_KEYS (stream_options, parallel_tool_calls, user, seed, logprobs, top_logprobs, n, presence_penalty, frequency_penalty) is added to drop_keys. messages, model, tools and any unknown key are NEVER dropped. Hard cap MAX_RESOLVE_ATTEMPTS = 4 retries per route attempt.
  • Retry placement. GatewayRuntime._chat_attempt (shared by the primary and every fallback attempt) sends, and on a parsable shape 400 re-sends through ShapeRetrier; the retry goes through _send so the breaker and retries apply, and streaming status is known before any byte is relayed so a retried stream is safe. On success learned_store.remember is called (never on a guess) and one auto-resolved <provider>/<model>: <method> line is logged. On exhaustion the ORIGINAL 400 response is returned unchanged, nothing is stored, and shape_stats.exhausted is incremented. The vision describe call uses the same ShapeRetrier, keyed vision:<host> + vision model.
  • Persistence. Table gateway_learned_methods (learned.py, ensure_table() from init_db): provider (default for the base upstream), target_model, method (JSON), learned_at, unique index on (provider, target_model), cap MAX_LEARNED_ROWS 1000 (oldest by learned_at replaced). Cross-worker cache under the gateway_learned cache-version name; the upsert is one ON CONFLICT DO UPDATE so two workers learning the same route never collide. It is derived data in the garbage-collection class (NOT in SOFT_DELETE_TABLES): forget = delete row. Editing a provider's base_url or client_profile, or deleting it, forgets all its learned methods (ProviderStore.set/remove).
  • Self-healing floor. If a request using a LEARNED method itself gets a shape 400, the record is forgotten, the ladder restarts once from native, and shape_stats.healed counts it.
  • Ledger. One row per client call as before; resolved_method (text, gateway_usage_ledger) holds the non-native method label that served the call, empty for native.
  • Operator surface. /admin/gateway models grid shows a shape: <method> badge and a "Reset learned" form (POST /admin/gateway/learned/reset-page); JSON GET /admin/gateway/learned and POST /admin/gateway/learned/reset (audited gateway.learned.reset, category ai) back the Devii tools gateway_learned_list / gateway_learned_reset (the reset is in CONFIRM_REQUIRED with a declared confirm).
  • Not covered: value-level rejections (unknown variant), and keys outside the closed allowlists. Tests: tests/unit/services/openai_gateway/{shape,learned}.py, the test_chat_auto_resolve*/test_chat_*learned*/test_fallback_attempt_is_shaped* tests in gateway.py, tests/api/admin/gateway/learned.py, and the grid badge in tests/e2e/admin/gateway.py.

Real upstream streaming (GatewayRuntime._stream_chat_response)

stream: true is forwarded to the upstream verbatim (payload["stream"] = stream, no more forced false) and the connection is opened with client.send(request, stream=True) so the response body is read incrementally via resp.aiter_lines() instead of buffered whole. Every retry/circuit-breaker mechanic in _send/retry_send is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; retry_send now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns 200 does the code path diverge into _stream_chat_response.

  • The client always gets real chunks, in real time, each parsed data: {...} line re-serialized and yielded as its own SSE event - no more splitting a complete response into fake 50-character deltas.
  • TTFT and inter-token latency are measured, not simulated. ttft_ms = time from just before the upstream POST to the first SSE line received; inter_token_ms = the average gap between content-bearing chunks (delta.content/delta.reasoning_content) after the first one, None when fewer than two content chunks arrived. Both ride the ledger row (gateway_usage_ledger.ttft_ms/.inter_token_ms, nullable REAL columns dataset auto-adds on the first insert carrying them - no explicit init_db() ensure-block needed, same as every other column on this insert-only table) and surface in /admin/ai-usage's latency.ttft_ms/latency.inter_token_ms (analytics.py, same _pset percentile shape as every other latency dimension).
  • Usage is always requested from the upstream when streaming, regardless of what the client asked (stream_options.include_usage is forced true in the upstream payload), so the ledger always gets a real usage object off the final chunk. The client's OWN stream_options.include_usage only controls whether that usage-only chunk is relayed downstream to it; the gateway swallows-and-records it either way.
  • The ledger row is written once, from inside the generator's finally, after the stream ends (saw_done), is interrupted by an upstream error, or the client disconnects (GeneratorExit - re-raised after cleanup, never swallowed, and never yielded-from during close: Python forbids yielding while handling GeneratorExit). All three paths close the upstream response (resp.aclose()) so a dropped client never leaks the connection, and all three still write a ledger row (success=False/error_category="client_disconnected" or a classified error on interruption) so accounting is never silently skipped.
  • No X-Gateway-Cost-USD/token headers on the streaming response - see "Per-call cost/token response headers" above; this is the one deliberate exception to "every response carries the headers."
  • Non-streaming chat, embeddings, and images are completely unchanged - this only touches the stream: true branch of handle_chat.

Usage, cost, latency, and reliability tracking

The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:

  • usage.py - GatewayUsageLedger writes to gateway_usage_ledger (one row per call, success and failure) and samples gateway_concurrency_samples each 30s tick. normalize_usage handles BOTH upstream shapes: DeepSeek (prompt_cache_hit_tokens/prompt_cache_miss_tokens, completion_tokens_details.reasoning_tokens) and OpenRouter (prompt_tokens_details.cached_tokens, native cost, cost_details). compute_cost PREFERS the upstream native cost (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share input_cost / (input_cost + output_cost) from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). record() is wrapped in try/except so a tracking failure never breaks the proxy.
  • gateway.py - _send is the gated/timed/retried path: a CircuitBreaker (reliability.py) short-circuits when open; retry_send retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the queue_wait_ms; an httpx per-request trace extension captures connect_ms (0 on pooled reuse). handle_chat/handle_passthrough record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from VisionAugmenter._describe_one.
  • Owner attribution - GatewayService.resolve_owner maps the caller to (owner_kind, owner_id): internal:devii (internal key), key:access (static key), user:<uid>/admin:<uid> (session or API key), else anonymous.
  • analytics.py - build_analytics(hours, top_n, pricing) pulls the bounded window once (clamped to MAX_WINDOW_HOURS=168) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via reliability.percentile), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. summary_metrics() runs cheap SQL aggregates for the live service panel. TTFT/inter-token (latency.ttft_ms/latency.inter_token_ms, the same _pset percentile-set shape as every other latency dimension) are real, measured only for calls that requested streaming - count: 0 in a window means no streaming traffic occurred, not that the metric is unavailable.
  • Admin - /admin/ai-usage (HTML, templates/ai_usage.html + static/js/AiUsageMonitor.js) and /admin/ai-usage/data?hours=&top_n= (JSON) in routers/admin/ package, admin-gated. Schema GatewayUsageOut. Devii action ai_usage and docs admin-ai-usage expose the same endpoint. The ai_usage and site_analytics Devii catalog tools are both requires_admin=True.
  • Per-user usage - analytics.build_user_usage(owner_id, hours=24, pricing) filters the ledger by owner_id (a uid matches both user: and admin: rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at GET /admin/users/{uid}/ai-usage (schema UserAiUsageOut, docs admin-user-ai-usage) and rendered on the profile page by static/js/UserAiUsage.js (the [data-ai-usage] card, gated {% if is_admin(user) %} in templates/profile.html). The 24h window is a sound projection basis (unlike a startup burst), so cost_24h * 30 is honest.
  • Profile visibility split - routers/profile/ package _ai_quota(uid, include_cost=False) builds the profile quota bar. The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal: spent_usd/used_pct/requests/last_used come from analytics.user_spend_24h(uid), the SAME gateway_usage_ledger source (summed by owner_id, matching both user: and admin: rows) that feeds the stat grid's cost.window_usd. So the server-rendered spent_usd is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. turns is still the devii_usage_ledger turns_24h count (Devii conversation turns) and is labeled "Devii turns" in the template precisely because it is a different metric from the grid's total API requests (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why 0 Devii turns can sit beside 18 API requests). The limit/turns still come from the Devii service (daily_limit_for/turns_24h); an unlimited limit (<= 0, the admin default) sets unlimited=True so the template renders "no cap" / "No daily spend cap" instead of a misleading / $0.00. This is display-only: the Devii daily-cap ENFORCEMENT and /devii/usage still read the Devii-scoped devii_usage_ledger.spent_24h (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: admin sees the full gateway stats card plus a server-rendered quota bar (with spend/limit/Devii turns); the owner (non-admin, is_self) sees a quota-only card showing just the percentage; everyone else sees nothing AI-related. _ai_quota is fail-safe (returns None on any error or when the Devii service is unregistered) and the result rides on ProfileOut.ai_quota. Dollar figures (spent_usd/limit_usd) are included only when include_cost=viewer_is_admin - never for a non-admin owner. This matters because the route serves the same context as JSON via respond(..., model=ProfileOut) and ProfileOut.ai_quota is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with Accept: application/json; the server-side omission is the real control. Non-admins (HTML and JSON) get only used_pct/turns/requests.
  • Retention - run_once prunes both tables past gateway_usage_retention_hours (default 720 = 30 days).

Financial data is admin-only everywhere. Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, ai_correction/ai_modifier usage displays, and the Devii cost tools.

Quota rules (quota.py, admin /admin/gateway "Quota rules" section)

This caps /openai/v1/* itself, independent of Devii's own daily cap. Devii's devii_user_daily_usd/devii_guest_daily_usd/devii_admin_daily_usd (documented in devplacepy/services/devii/CLAUDE.md) only gate turns that go through Devii. A caller hitting the gateway directly with their own api_key bypasses that entirely - quota.py is the root-level enforcement that closes this, checked in GatewayService.handle() right after owner/app_reference resolution and before every billed dispatch (chat, embeddings, images, passthrough; GET /v1/models is exempt, it makes no upstream call).

Two layers, same shape as provider/model routing above. Layer A is five flat config_fields on GatewayService (gateway_default_user_daily_usd $1.00, gateway_default_admin_daily_usd $0/unlimited, gateway_default_guest_daily_usd $0.05, gateway_default_internal_daily_usd $0/unlimited, gateway_default_key_daily_usd $0/unlimited - group Quota) applied per specific caller (owner_id) when no rule matches; internal/key default unlimited so shipping this never starts blocking DevPlace's own news/bots/Devii-guest/correction traffic on the internal key. Layer B is the gateway_quota_rules table (ensure_tables(), called from init_db() alongside routing.ensure_tables(); hard CRUD, not in SOFT_DELETE_TABLES, cross-worker cache-invalidated under the "gateway_quota" name): each row scopes by any combination of owner_kind (internal/key/user/admin/anonymous - DevPlace's only "roles" here), a specific owner_id, and app_reference (the X-App-Reference label), each nullable = wildcard; a QuotaRuleIn Pydantic validator rejects a rule with all three blank (that belongs in Layer A). quota.resolve(owner_kind, owner_id, app_reference, cfg) gathers every active rule whose non-null dimensions all equal the request, picks the one with the most non-null dimensions (ties broken toward the smaller limit, unlimited 0 never wins a tie against a finite cap), and returns (limit_usd, scope, rule) where scope is the exact (owner_kind, owner_id, app_reference) triple - each possibly None - that spend must be summed over. Layer A is internally just the maximally-specific implicit scope (owner_kind, owner_id, None), so one code path (quota.spent_24h(*scope), a plain SUM(cost_usd) over gateway_usage_ledger filtered by whichever scope dimensions are non-null) serves both layers.

A wildcard dimension means a shared pool, by design. A rule scoped only by app_reference caps that app's combined spend across every caller using it; a rule scoped only by owner_kind caps that whole role's combined spend. Pin owner_id to get a true per-caller cap (the Layer A default's own behavior). anonymous/internal/key owner_ids are already fixed constants ("anonymous"/"devii"/"access", from resolve_owner()), not per-caller identities, so any cap on those kinds is inherently pooled - there is no per-guest identity at this layer (unlike Devii's own guest-cookie-scoped ledger).

No lock, no hold, bounded overshoot by design - this is deliberate, not an oversight. Cost is only known after the upstream call returns, so a true atomic pre-authorization would need a reserve-then-reconcile ("hold") mechanism, and a bug in releasing a hold is exactly the kind of thing that gets a caller stuck forever. Instead this mirrors Devii's own already-shipped mechanism exactly: read the 24h sum, compare, raise HTTPException(429, ...) if already at/over - a single SELECT and a conditional raise, nothing held, nothing to leak, structurally impossible to deadlock. The tradeoff is a small, bounded overshoot (at most a few concurrent in-flight calls' worth of cost past the cap before the next request sees the updated sum and blocks) - acceptable and industry-standard for a cost whose exact size isn't known until the call finishes, and it is the property actually being enforced: once tripped, every subsequent separate request stays blocked until the 24h window rolls off or an admin adjusts the rule.

429 body never carries a dollar figure, admin or not ({"detail": "AI gateway daily quota exceeded"}) - mirrors Devii's own over-limit WS message, which likewise never states a number. The admin-only services log line and the ai.quota.exceeded audit row (GatewayService._audit_quota_exceeded, reusing usage.audit_actor_for) do carry the spend/limit/matched-rule-uid, since those are admin-only surfaces.

Resetting the counted spend (gateway_quota_resets). A cap is only lifted by time otherwise, so there is a reset that clears what has been counted without deleting any ledger row - gateway_usage_ledger is the cost-analytics source for /admin/ai-usage, so a reset must never truncate it. quota.reset(QuotaResetIn, created_by=) upserts one watermark row into gateway_quota_resets (same ensure_tables()/"gateway_quota" cache-version/hard-CRUD shape as the rules table, same two indexes) scoped by the SAME three nullable dimensions as a rule, and quota.spent_24h sums from max(24h cutoff, reset_watermark(scope)). A reset row applies to a queried scope when each of its non-null dimensions equals that scope's - so an all-null reset clears everyone, while a reset scoped to one app deliberately does NOT clear a broader per-user-all-apps scope (clearing a narrower window can only over-credit). Spend recorded after the reset counts again immediately against the same limit. QuotaScopeIn is the shared base holding the three dimensions and their validators; QuotaRuleIn and QuotaResetIn both extend it, so scope parsing exists once.

The two AI quotas are separate systems and the reset surfaces must say so. /admin/ai-usage's Reset all quotas clears the Devii devii_usage_ledger AND now also stamps a global gateway watermark, because a caller hitting 429 AI gateway daily quota exceeded had no reset at all before and the button looked global. Reset guest quotas stays Devii-only (guest gateway calls ride the shared internal key, so there is no per-guest gateway scope to clear).

CRUD. Admin JSON at /admin/gateway/quota-rules (routers/admin/gateway_configs.py, list returns each rule's live spent_24h_usd plus the Layer A defaults for context), audited gateway.quota_rule.update/gateway.quota_rule.delete (category ai, both already in events.md). Devii tools gateway_quota_rules/gateway_quota_rule_set/gateway_quota_rule_delete (requires_admin=True, delete is CONFIRM_REQUIRED) proxy the same JSON endpoints via handler="http", same as the provider/model tools. Reset is POST /admin/gateway/quota-resets (same file, _payload/ValidationError shape as the rule CRUD), audited gateway.quota.reset (category ai), and exposed as the Devii tool gateway_quota_reset (requires_admin=True, in CONFIRM_REQUIRED with a declared confirm param, like the other quota-lifting admin resets). CLI: devplace gateway quota list|set|delete|reset.

The human-facing Quota rules tab of /admin/gateway is a third, independent backend-rendered surface over the same quota_rule_store (see "Admin UI (backend-rendered, no JS)" above): GET/POST /admin/gateway/quota-rules/new, GET/POST /admin/gateway/quota-rules/{uid}/edit, POST /admin/gateway/quota-rules/{uid}/delete, and a per-row POST /admin/gateway/quota-rules/{uid}/reset button (templates/admin_gateway_quota_form.html for add/edit; the reset/delete actions are plain <form> buttons with data-confirm on the index page, no fetch call). These do not replace the JSON endpoints above - Devii and the CLI still talk to those directly.

Image generation

POST /openai/v1/images/generations exposes an OpenAI-compatible image-generation endpoint. Clients send the generic model molodetz-img-small (config.INTERNAL_IMAGE_MODEL), which handle_images remaps to gateway_image_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty or molodetz-img). It defaults to OpenRouter's black-forest-labs/flux-1.1-pro at https://openrouter.ai/api/v1/images/generations (config.IMAGE_*_DEFAULT, $0.04 per image fallback). handle_images mirrors handle_embeddings: build the payload, forward via _send, and record one ledger row. The config fields are the Images group (gateway_image_enabled default on, gateway_image_url, gateway_image_model, gateway_image_key) plus the Pricing-group gateway_image_price_per_call. effective_config() falls the image key back to gateway_api_key then OPENROUTER_API_KEY. Usage is recorded with backend="image"; usage.compute_cost adds an image branch (flat per-call, native OpenRouter cost still preferred via extract_image_usage). routing.image_overlay resolves per-route provider/url/key and uses price_input_per_m as the per-image price. routing.seed_default_image_routes() (from migrate_ai_gateway_settings) idempotently seeds molodetz-img-small -> Flux on the openrouter provider when OPENROUTER_API_KEY is set. When gateway_image_enabled is off the endpoint returns 503 with no ledger row.

Embeddings

POST /openai/v1/embeddings exposes an OpenAI-compatible text-embeddings model. Clients send the generic model molodetz~embed (config.INTERNAL_EMBED_MODEL), which handle_embeddings remaps to gateway_embed_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty). It defaults to OpenRouter's qwen/qwen3-embedding-8b at https://openrouter.ai/api/v1/embeddings (config.EMBED_*_DEFAULT, $0.01 per 1M input tokens). handle_embeddings mirrors handle_chat but is simpler: no vision augmentation and no streaming - build the payload, forward via _send, and record one ledger row through the same finalize(...) closure. The config fields are the Embeddings group (gateway_embed_enabled default on, gateway_embed_url, gateway_embed_model, gateway_embed_key) plus the Pricing-group gateway_embed_price_input_per_m. effective_config() falls the embed key back to gateway_vision_key then OPENROUTER_API_KEY (NOT gateway_api_key: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with backend="embed"; usage.compute_cost adds an embed branch (input-only, completion always 0, native OpenRouter cost still preferred) and Pricing gained embed_input_per_m. analytics.py groups by backend generically, so embed rows roll up automatically; caching_savings counts only non-native chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When gateway_embed_enabled is off the endpoint returns 503 with no ledger row. When the resolved gateway_embed_key is still blank after the effective_config()/route-overlay merge (all of gateway_embed_key, gateway_vision_key, and OPENROUTER_API_KEY are unset, or a per-route provider was configured with no api_key of its own), handle_embeddings returns the same 503 shape ({"error": {"message": "Embeddings are not configured", "type": "embeddings_not_configured"}}) with an ai.gateway.call audit row (result="denied", summary="no embeddings key configured") instead of forwarding an unauthenticated request upstream - mirrors the gateway_embed_enabled disabled-check immediately above it, no ledger row either. routing.embed_overlay only overlays gateway_embed_key when the matched route's provider actually has a non-blank api_key, so a misconfigured route can never downgrade an already-good fallback key to empty.

Single point of truth for AI

The gateway is the only place that holds real provider URLs/models/keys. Every other AI consumer (news, bots, Devii guests) points at it by default and never touches a provider key:

  • Defaults live in config.py: INTERNAL_GATEWAY_URL (http://localhost:{DEVPLACE_PORT, default 10500}/openai/v1/chat/completions; the whole base can also be overridden with DEVPLACE_INTERNAL_BASE_URL) and INTERNAL_MODEL (molodetz). news_ai_url, bot_api_url, devii_ai_url default to INTERNAL_GATEWAY_URL; their model defaults to molodetz.
  • Each consumer's key falls back to database.internal_gateway_key() (reads the gateway_internal_key setting) when its own key field/env is unset - the provider-key fallbacks (DEEPSEEK_API_KEY/OPENROUTER_API_KEY) were removed from news and bots.
  • gateway_force_model (default on) and a molodetz/empty alias in handle_chat make the upstream always receive gateway_model, so molodetz is a stable generic alias. gateway_allow_client_model (Upstream group, bool, default off) lets a client-requested model name through even while gateway_force_model is on - useful for an upstream that serves many models by name with no single stable alias (a self-hosted Ollama). The molodetz alias is always remapped to gateway_model regardless of this flag, so the generic name keeps working for internal callers; only a genuinely named model (llama3.2, qwen3, ...) bypasses the force.
  • database.migrate_ai_gateway_settings() (called at the end of init_db(), under the startup init_lock): generates gateway_internal_key (uuid4) if missing; migrates DEEPSEEK_API_KEY/OPENROUTER_API_KEY env into gateway_api_key/gateway_vision_key when the db value is empty; and rewrites any consumer AI URL still equal to the old openai.app.molodetz.nl default to the gateway, plus bot_model deepseek-chat -> molodetz (only uncustomized values).
  • The gateway's gateway_api_key/gateway_vision_key/gateway_internal_key fields are non-secret so the admin services page shows the value actually in use, editable.
  • Bot LLM calls are synchronous urllib but already run via asyncio.to_thread (bot/bot.py), so the local round-trip never blocks the event loop.
  • Real provider keys/URLs/models live ONLY in the gateway. The gateway is default_enabled=True.

Provider and model routing (routing.py, admin /admin/gateway)

Layered ON TOP of the single-provider service config above, which stays THE implicit default provider (env migration, internal key, molodetz/molodetz~embed, vision, embeddings, ledger all unchanged).

Storage. Two dataset tables, ensured in init_db via routing.ensure_tables, cross-worker cache-invalidated under the "gateway_routing" cache-version name (module-level provider_store/model_store over a shared _ROUTING_CACHE; writes bump_cache_version and clear the cache). They are admin config, NOT in SOFT_DELETE_TABLES - hard CRUD, mirroring site_settings:

  • gateway_providers - named upstreams: name + base_url (chat-completions URL) + api_key + is_active; the embeddings URL is derived by swapping /chat/completions -> /embeddings.
  • gateway_models - source->target routes: source_model (unique, what clients request) -> provider (blank = default) + target_model + kind (chat|embed|image) + optional vision_provider/vision_model for the text+vision merge (when set, image content is described by that vision model before forwarding) + context_window + its own economy (price_cache_hit_per_m/price_cache_miss_per_m/price_output_per_m/price_input_per_m, USD per 1M tokens for chat/embed/vision; for image routes price_input_per_m is a flat USD per image, plus the tiered/off-peak fields below) + is_active.

Resolution is a per-request overlay, not a fork. At request time routing.chat_overlay(requested, cfg) / routing.embed_overlay(requested, cfg) / routing.image_overlay(requested, cfg) resolve an active route by the requested model name and return a per-request OVERLAY dict of gateway_* cfg keys (gateway_force_model+gateway_model=target, gateway_upstream_url/gateway_api_key from the provider, the price keys, an augmented gateway_model_context_map, and vision overlay keys); handle_chat/handle_embeddings merge it onto the base cfg (cfg = {**cfg, **overlay}) BEFORE everything else, so the existing model-selection / pricing_from_cfg / parse_context_map / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. _ensure (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.

No matching route = None overlay = model fallback. When no route matches, the overlay returns None and the handler's model selection logic falls back to the default configured model (gateway_model for chat, gateway_embed_model for embeddings, gateway_image_model for images). The unknown model name is discarded, not forwarded upstream. This means an unknown or misspelled model name never causes a 4xx from the upstream - it is gracefully downgraded to the default. The force_model guard and the built-in molodetz/molodetz~embed/molodetz-img-small aliases are still respected before the route check: force_model or empty/named-alias model -> default directly; known model -> route resolution; unknown model with no route -> default fallback with a log message.

CRUD. Admin JSON at /admin/gateway/{providers,models} (routers/admin/gateway_configs.py, require_admin, Pydantic ProviderIn/ModelRouteIn validation via the shared _payload() JSON-or-form reader, _save_provider/_save_model/_check_fallback helpers), audited under gateway.provider.*/gateway.model.* (category ai). This JSON surface is for Devii and programmatic callers only - see "Admin UI (backend-rendered, no JS)" below for the human-facing page, which is a completely separate set of routes over the same stores.

Devii tools: gateway_providers/gateway_models (read) and gateway_provider_set/gateway_provider_delete/gateway_model_set/gateway_model_delete (admin; the two deletes are CONFIRM_REQUIRED).

Admin UI (backend-rendered, no JS)

/admin/gateway (templates/admin_gateway.html, sidebar link, admin_section="gateway") is a plain server-rendered page - no GatewayAdmin.js (deleted), no page controller driving CRUD. It is a three-tab index (?tab=models|providers|quota, plain <a> links, .admin-tabs/.admin-tab) listing each resource in an .admin-table; every add/edit is its own dedicated page, never an inline form under the table:

  • Providers: GET/POST /admin/gateway/providers/new, GET/POST /admin/gateway/providers/{name}/edit, POST /admin/gateway/providers/{name}/delete (templates/admin_gateway_provider_form.html). name is the primary key and is rendered readonly on the edit page - renaming would silently orphan the old row in gateway_providers rather than updating it (ProviderStore.set keys on find_one(name=...)), so renaming is not offered; delete and re-add instead.
  • Model routes: GET/POST /admin/gateway/models/new, GET/POST /admin/gateway/models/{source_model:path}/edit (path converter, since a source model name is not guaranteed to be slash-free), POST /admin/gateway/models/{source_model:path}/delete (templates/admin_gateway_model_form.html). source_model is likewise readonly on edit for the same orphan-row reason. Every field is always rendered (chat pricing, embed/vision pricing, tiered pricing, off-peak window) grouped into <fieldset>s by applicability rather than JS-toggled by the selected kind - the hint text under each fieldset states which kind it applies to. The fallback-model <select> is grouped into one <optgroup> per kind (pure HTML, no live filtering); the server still authoritatively rejects a cross-kind fallback via _check_fallback. The target model field is the one genuinely dynamic control on this page - see "Live target-model lookup" below.
  • Quota rules: GET/POST /admin/gateway/quota-rules/new, GET/POST /admin/gateway/quota-rules/{uid}/edit, POST /admin/gateway/quota-rules/{uid}/delete, POST /admin/gateway/quota-rules/{uid}/reset (templates/admin_gateway_quota_form.html). Unlike providers/models, a quota rule's uid is never user-editable, so there is no orphan-row risk and no field needs to be readonly.

Live target-model lookup (the one dynamic field on this page)

The model-route form's target model field is a deliberate, narrow exception to "backend-rendered, no JS": picking a provider can turn it from a free-text input into a <select> populated with that provider's own real model list, but the browser never talks to the provider directly.

  • Server does the URL resolution and the outbound call. routing._models_url_from_base(base_url) derives the OpenAI-style listing endpoint from a provider's chat-completions base_url (swaps /chat/completions -> /models, exactly like the existing _embed_url_from_base/_image_url_from_base derivations), and routing.fetch_provider_models(provider_name) (async) resolves the named provider's (or, for a blank name, the default provider's - _default_provider_credentials() reads gateway_upstream_url/gateway_api_key off the service config the same way _default_provider_summary() does) base_url/api_key, calls the derived URL through stealth.stealth_async_client (never a bare httpx.AsyncClient - see the root CLAUDE.md outbound-HTTP rule) with an authorization: Bearer <api_key> header, and parses the OpenAI {"data": [{"id": ...}, ...]} shape into a plain list of id strings. It returns None - never raises - for every failure mode: unknown provider, blank/unresolvable base URL, a non-200 response, an unparseable body, an empty list, or any network exception (timeout, connection refused, DNS failure). None is the single "can't do this" signal the caller (and the frontend) all key off.
  • GET /admin/gateway/provider-models?provider=<name-or-blank> (routers/admin/gateway_configs.py::provider_models, admin-gated, JSON-only, no page) is the one endpoint the browser is allowed to call: require_admin then fetch_provider_models(provider), returning {"provider", "models": [...]} on success or a 404 {"error": {...}} (via json_error) when the result was None. This is the ONLY thing static/js/GatewayModelForm.js fetches - it never reaches a third-party host, satisfying "we do not let the JavaScript make a connection to a remote site."
  • static/js/GatewayModelForm.js (loaded only by admin_gateway_model_form.html's {% block extra_js %}, not global) is the one page-specific controller left in this subsystem. The template always renders BOTH #gw-model-target (a required text input, the no-JS/JS-still-loading baseline and the true fallback) and #gw-model-target-select (a required but initially hidden disabled <select> with the same name="target_model") - exactly one of the pair is ever disabled at a time (a disabled field is excluded from form submission, so there is never a duplicate target_model value), and the controller toggles .hidden/.disabled on both together. On the provider <select>'s change event, and once on page load ONLY when a provider is already selected (an edit page for a named-provider route - a brand-new form or a "default"/blank provider selection never auto-probes on load, since that would fire a real outbound network probe against the default gateway just from opening the page), it calls Http.getJson("/admin/gateway/provider-models?provider=..."); a 200 swaps to the select (populated from the response, the field's current value prepended as an extra option if the provider's own list does not contain it, so an existing customized/legacy target is never silently discarded) and a thrown error (404 or any other failure) swaps back to the free-text input. The <select>'s own HTML required attribute is what makes "not allowed to be empty" hold even with JS disabled-then-re-enabled - no JS-side validation was added.
  • Tests: tests/unit/services/openai_gateway/routing.py (_models_url_from_base, fetch_provider_models success/unknown-provider/non-200/malformed-payload/empty-list/network-error/no-base-url, all via a fake stealth_async_client), tests/api/admin/gateway/provider_models.py (the endpoint over a real local http.server fake upstream, mirroring the pattern already used by tests/api/containers.py's ingress test), and tests/e2e/admin/gateway.py (a real browser swapping the field between a listing-capable and a blind provider, and a full submit through the dynamically-populated select).

Validation errors re-render the same form, not a silent redirect. Each POST handler builds the pydantic model itself (ModelRouteIn/ProviderIn/QuotaRuleIn) rather than going through the generic Depends(json_or_form(...)) + RequestValidationError path used by simpler admin forms (e.g. /admin/settings) - that generic handler redirects non-auth pages back to the referer with the message dropped, which would silently swallow a genuine validation failure (a bad URL, a cross-kind fallback) on a page whose whole point is surfacing exactly that feedback. _error_response() re-renders the form template with a .gw-error banner and every submitted value preserved, at 400.

Time-of-day fields are converted at the router, not in the browser. off_peak_start/off_peak_end are plain <input type="time"> fields (HH:MM strings); _time_to_minutes/_minutes_to_time in gateway_configs.py convert to/from the stored off_peak_start_minute/off_peak_end_minute integers - this used to be a client-side JS computation (GatewayAdmin.js timeToMinutes/minutesToTime), now server-side only.

The JSON API above is untouched and still the only surface Devii/devplace gateway quota talk to - the backend-rendered pages are a second, independent set of routes over the same provider_store/model_store/quota_rule_store, sharing the save/delete/fallback-check logic via _save_provider/_save_model/_check_fallback/_save_quota_rule helper functions so the two surfaces can never drift on what counts as valid.

When adding a routed value, overlay it as the matching gateway_* cfg key so the runtime needs no new branch.

Tests: tests/unit/services/openai_gateway/routing.py (overlay/economy/kind isolation), tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream (end-to-end through handle_chat), and tests/api/admin/gateway/ (admin CRUD + validation + role gating).

Automatic model fallback (fallback_model, chained, per route)

Any gateway_models route (chat, embed, or image) can optionally name fallback_model: another already-configured source_model of the SAME kind, tried automatically, when the primary route fails - and when THAT attempt fails, its own fallback is tried next, walking the whole line. This is admin-configured on the /admin/gateway model form as a select box grouped into one <optgroup> per kind over every OTHER route (the public source_model names such as molodetz/molodetz-pro, never the internal target_model a provider actually sees) - never a free-text field, so a fallback can only ever point at a model the gateway already knows how to serve.

  • Trigger. _call_failed(resp, exc, timing) in gateway.py treats a call as failed when the circuit breaker rejected it, the upstream connection raised, or the upstream answered with any status >= 400. This runs AFTER _send's own gateway_max_retries retries against the primary model are exhausted - a fallback is the next escalation once retrying the SAME model has already given up, not a replacement for that retry loop. Only bad_request never falls back; a 400 naming the model (Model is unavailable, model not found, unknown model, no such model, model_not_found) classifies as model_not_found and DOES fall back, since trying a different model is the correct recovery.
  • A guarded chain, not one hop. routing.resolve_fallback_chain(source_model, kind) walks fallback_model links collecting usable routes: inactive and unhealthy links are skipped transparently (their own fallback is followed instead), the source itself is never re-added, a visited set makes cycles impossible, and MAX_FALLBACK_HOPS (5) caps the line. handle_chat/handle_embeddings/handle_images attempt each member in turn until one succeeds (an attempted_routes set additionally skips anything already tried, so the context precheck's jump never causes a repeat attempt). A route that fails down the whole line returns the last attempt's own failure to the caller. The old one-hop routing.resolve_fallback stays as a tested primitive but is no longer on the request path.
  • Direct requests preroute around unhealthy models. routing.resolve_entry_route(source_model, kind) returns the requested route when healthy, else the first usable chain member - so a request naming a bad model starts on its fallback WITHOUT attempting the bad model first (logged as is unhealthy, starting on). When the line is empty the original route is still tried: health is advisory, and the model may have recovered since its last test. Only manual tests write health - a request-time failure never marks a route, so a blip can never hide a model.
  • Self-reference is rejected at write time. ModelRouteIn._check_fallback_is_not_self (pydantic model_validator) refuses fallback_model == source_model; routers/admin/gateway_configs.py::_check_fallback (shared by the JSON save_model_json route and both backend-rendered model-route pages) additionally rejects a fallback_model that does not resolve to an existing, same-kind route (routing.model_store.get(...)) before writing the row, since a pydantic validator alone cannot see other rows.
  • Rebuilt like a fresh routed call, not retried in place. On failure, handle_chat/handle_embeddings/handle_images recompute the overlay from base_cfg (the pre-primary-overlay config) via chat_overlay(fallback_route.source_model, base_cfg)/embed_overlay/image_overlay, so the fallback gets its OWN provider, URL, key, and pricing - never the primary route's. pricing/context_map for the ledger are (re)computed AFTER the fallback decision so the recorded cost always reflects whichever model actually served the request.
  • One ledger row per client call, not one per attempt. The primary failure never writes a gateway_usage_ledger row by itself (only _send's own internal retries and the circuit breaker counters observe it); finalize()/self._ledger.record(...) still runs exactly once, after the fallback decision, with model = whichever model ultimately answered and requested_model = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in /admin/ai-usage as "requested X, served Y" rather than as two calls.
  • Streaming falls back before any byte reaches the client. client.send(request, stream=True) returns as soon as the response headers arrive, so resp.status_code is already known before the SSE body is ever touched - handle_chat decides whether to fall back at that same point, before it ever opens _stream_chat_response to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
  • Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (self._breaker) is one instance per GatewayRuntime, shared across every route. A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see circuit_open too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
  • A 400 is terminal UNLESS its body names rejected keys the auto-resolve ladder can fix (see "Auto-resolve of request shape"): that case is retried on the SAME route with another shape before any fallback is considered.
  • Deliberately NOT covered: gateway_max_retries/gateway_retry_backoff_ms, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.

Tests: tests/unit/services/openai_gateway/routing.py (resolve_fallback self-reference/kind/active-flag guards, chain skip/cycle/hop-cap/entry tests), tests/unit/services/openai_gateway/gateway.py (test_chat_falls_back_when_the_primary_model_fails, test_chat_returns_the_fallback_failure_when_both_models_fail, test_chat_without_a_fallback_never_makes_a_second_call, test_chat_walks_fallback_chain_past_failures, test_chat_preroutes_around_unhealthy_models, test_chat_tries_original_when_chain_is_exhausted), and tests/api/admin/gateway/models.py (test_model_route_fallback_round_trip, test_model_route_fallback_must_reference_an_existing_route, test_model_route_fallback_must_be_the_same_kind, test_model_route_fallback_rejects_self_reference).

Route health tests (health.py, Test buttons, unhealthy hiding)

Every model route carries health_status ("" untested, "ok", "bad"), health_code, health_message, health_checked_at - plain columns on gateway_models (same ensure idiom as tier2/off-peak), written ONLY by manual tests. The models tab has a per-row Test button plus Test-all (sequential, one real call per route); both are plain POST forms redirecting back to the grid. health.probe_route(source) resolves the route and sends one minimal non-stream call straight at its upstream (chat: tiny completion, transpiled for Responses upstreams with reasoning minimal; embed: one vector; image: the fixed a hedgehog and a bear prompt), bypassing fallback and the ledger, 30s cap - then record_health persists ok/bad and invalidates the routing cache, and the router audits gateway.model.test / gateway.models.test_all. A bad route is hidden from GET /v1/models, tinted in the grid with an unhealthy badge plus small code: message, and skipped transparently by the fallback chain above. A passing image test redirects with test_image=<url> and the grid pre-opens the standard modal showing the picture (URLs over ~2000 chars are not passed through). Cost honesty: chat/embed probes are sub-cent; an image test spends one generation - the modal shows what was paid for.

Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)

Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME gateway_models route row, so it stays fully provider-agnostic and opt-in per route.

  • Per-route fields (each optional/zero by default = feature off, so an unconfigured route is byte-identical to before): context_tier_threshold_tokens (0 disables tiering; when the request's input token count exceeds it, price_*_per_m_tier2 rates apply instead of the tier-1 rates above) and the four nullable price_cache_hit_per_m_tier2/price_cache_miss_per_m_tier2/price_output_per_m_tier2/price_input_per_m_tier2 (a component left None keeps its tier-1 rate even above threshold - so a provider that only re-prices input above a size threshold, keeping output flat, needs just one tier2 field set), plus off_peak_start_minute/off_peak_end_minute (nullable, UTC minutes-since-midnight, both-or-neither enforced by ModelRouteIn's model validator; a window where start > end wraps past midnight, e.g. DeepSeek's old V3/R1-era 16:30-00:30 UTC window) and off_peak_discount_pct (0-100, multiplies whichever tier rate is active). The same fields double for vision (price_input/output_per_m_tier2) and embed (price_input_per_m_tier2) exactly like their tier-1 counterparts already do.
  • Overlay + computation. chat_overlay/embed_overlay propagate all of these into the per-request cfg dict under gateway_* keys exactly like the existing price fields (section above); usage.pricing_from_cfg reads them generically (defaulting to None/0/disabled when absent), so Layer A (the single global flat Pricing config fields) never gains these dimensions - only a gateway_models route can enable them, preserving the "no matching route = byte-identical legacy behavior" guarantee. usage.compute_cost selects tier1 vs tier2 per rate component (_tiered_rate) based on whether norm["prompt"] (billable input tokens) exceeds the threshold, then applies the off-peak discount (_effective_rate/_off_peak_active, UTC wraparound-aware) to whichever rate was selected. The response header format and compute_cost's return shape (total, input_cost, output_cost, native) are unchanged - this is purely an internal rate-selection step before the existing input/output split math runs; a native upstream cost (OpenRouter) still overrides the modeled total exactly as before.
  • Migration. New gateway_models columns are added via has_column/create_column_by_example in routing.ensure_tables() (the CREATE TABLE IF NOT EXISTS DDL string alone would never reach a pre-existing table - see the database/CLAUDE.md column-ensure idiom).
  • Admin UI. /admin/gateway's model-route add/edit page has a "Tiered pricing" and an "Off-peak discount" fieldset; off-peak start/end render as <input type="time"> (converted to/from UTC minutes-of-day server-side by _time_to_minutes/_minutes_to_time in routers/admin/gateway_configs.py), and the models tab table shows tiered/off-peak badges when a route has either dimension configured.
  • DeepSeek's real pricing is already the tier-1 shape, not a new dimension. DeepSeek's actual API (verified against api-docs.deepseek.com/quick_start/pricing) bills three flat per-1M rates - cache-hit input, cache-miss input, output - with no current context-length tier or off-peak window for the V4 models; that shape was already fully modeled by the pre-existing chat_cache_hit_per_m/chat_cache_miss_per_m/chat_output_per_m fields before this section's tier2/off-peak fields existed. routing.seed_default_deepseek_routes() (called once from database.migrate_ai_gateway_settings() at the end of init_db()) idempotently inserts four ready-made routes - deepseek-v4-flash ($0.0028/$0.14/$0.28 per 1M, 1M context), deepseek-v4-pro ($0.003625/$0.435/$0.87 per 1M, 1M context), and the two public molodetz aliases molodetz -> deepseek-v4-flash (flash rates) and molodetz-pro -> deepseek-v4-pro (pro rates) - only when that source_model row does not already exist, so a caller or Devii can request any name explicitly and get correctly-priced, decoupled from whatever the single global gateway_model default happens to be set to (switching that global setting between the two real models does NOT retroactively fix the flat Pricing config fields - the seeded routes are the model-agnostic, always-correct way to reference a specific priced model). The molodetz/molodetz-pro aliases are the public model names; GET /v1/models is served locally from these source_model rows (not proxied upstream), so it publishes exactly the models the gateway accepts. Neither seeded route sets the tier2/off-peak fields (DeepSeek does not use them today); an admin can add them later on the same row if DeepSeek (or any other provider routed here) introduces such pricing.

GatewayService.handle() calls consent_denied(owner) immediately after resolve_owner, before the quota check. This is the only consent gate in the platform; no consumer implements one of its own.

The split it enforces is the whole point:

  • owner kind user / admin -> the call carries that user's own content (Devii, AI correction, the AI modifier, a member calling /openai/v1/* with their own key). It requires a granted ai_third_party consent and answers 403 with CONSENT_REQUIRED_MESSAGE otherwise, plus an ai.consent.denied audit row.
  • owner kind internal / key / anonymous -> platform processing (news import, the bot fleet, SEO metadata, issue enhancement). Never gated: it is not the user's content.

ai_third_party is never granted at signup. The existing ai_correction_enabled / ai_modifier_enabled user columns survive unchanged as preferences subordinate to consent: withdrawing consent turns those features off regardless of the flag, so ai_modifier_enabled's default of 1 is harmless. No existing preference was flipped and no consumer changed - the gate was simply added above them. USER_CONTENT_OWNER_KINDS is the single tuple defining the split; widen it only if a new owner kind genuinely carries a user's own content.

Responses-protocol providers

A provider row (or the default upstream) can speak the Responses API instead of Chat Completions. ProviderIn.protocol is chat_completions (default, forwarded verbatim) or responses (transpiled both ways). chat_overlay carries the provider choice as gateway_upstream_protocol; gateway_upstream_protocol is also a service config field for the default upstream. A responses provider's base URL ends in /responses (e.g. https://api.meta.ai/v1/responses); the models probe tries /models then /muse-code/models next to it.

responses.py is the transpiler (stdlib only, no gateway imports): chat_to_responses / responses_to_chat for requests, responses_object_to_chat_completion / chat_completion_to_responses_object for objects, ResponsesToChatStream / ChatToResponsesStream for SSE. Every function returns (converted, warnings); lossy mappings warn, never drop silently. handle_chat keeps reasoning about a Chat payload through routing/vision/precheck/fallback and transpiles at the send boundary (_chat_upstream_body), so fallbacks can cross protocols per attempt. Responses upstreams skip apply_thinking; reasoning effort comes from the client (reasoning_effort, reasoning.effort, thinking flags) or defaults to {"effort": "minimal"} when thinking is off - "none" is listed in OpenAI's shared ReasoningEffort enum but the Meta upstream rejects it (measured: only minimal/low/medium/high/xhigh/max accepted), so minimal is the portable disabled value. Usage token-details ride both object conversions (input_tokens_details.cached_tokens <-> prompt_tokens_details.cached_tokens, output_tokens_details.reasoning_tokens <-> completion_tokens_details.reasoning_tokens) so cache/reasoning cost splits survive the round trip. stream_options.include_usage is never sent upstream (usage arrives on response.completed); the translator emits the spec shape

  • terminal chunk with finish_reason, then a usage-only chunk with empty choices - and the client's flag still controls whether that usage chunk is relayed. Only the Chat side of each conversion pair is live in production (gateway.py calls chat_to_responses, responses_object_to_chat_completion, ResponsesToChatStream plus the error helpers); the Responses-request side is tested but unused, since passthrough stays raw. Non-200 {error} envelopes (message/type/code/param) and 200 failed/incomplete Response objects are detected and relayed as Chat-style errors. Passthrough stays raw (no transpiling) and strips a trailing /responses base segment so /openai/v1/responses routes without doubling the path.

A Meta provider row either uses a pay-as-you-go key or sets client_profile="muse": Meta's subscription applies only to the auto-connected Muse Code CLI key ("for use with Muse Code only", dev.meta.ai/docs/muse-code/subscriptions), and since the 2026-10-29 rollout api.meta.ai answers any other client with 403 client_attestation_denied. The muse profile (muse_code.muse_impersonation_headers, same seam as the opencode profile) stamps the captured Muse Code 1.4.14 headers (User-Agent: muse-build/..., X-Client-Id: tbh:exec, the four X-Tbh-*/X-Meta-Ai-Gateway-Session-Id UUID-7 session headers sharing one process-stable id, X-Tbh-Agent-Role: main, X-Tbh-Contributor-Mode: 1) on chat/embed/images forwards AND on the /muse-code/models probe; env overrides are MUSE_CLIENT_VERSION, MUSE_CLIENT_BUILD, MUSE_CLIENT_SURFACE, MUSE_CLIENT_USER_AGENT, MUSE_CONTRIBUTOR_MODE. The default contributor mode 1 matches the verified-200 probe but opts gateway content into Meta training - flip it to 0 (a legitimate headless-client state) if that is unwanted. The applied profile is visible in the services log: the routed-model line gains (client profile 'muse') when a profile's headers are stamped, so a 403 with no such line means the row still says generic. Auth-class upstream rejections log their remediation via _upstream_auth_guidance (chat/embed/images alike, relay body never rewritten): residual attestation denials point at the denylist below or a pay-as-you-go key, 401s point at the route's provider key. The 403 still classifies as auth, so a configured fallback_model is tried exactly like any other auth failure.

The muse headers are defense in depth, NOT the attestation fix: probes proved the key works from any client/TLS stack for plain calls and the gate only fires on reserved tools[].name values. The real fix is automatic tool-name aliasing (muse_code.tool_aliases_for / alias_chat_tools, wired in _chat_upstream_body for Meta attempts on BOTH wire protocols): any declared/history/tool_choice name exactly matching MUSE_RESERVED_TOOL_NAMES (20 names, each measured denying while near-misses pass) is rewritten to ext_<name> (growing underscores on collision) and restored on the way back (restore_completion_names / restore_chunk_names, with the alias map threaded through _stream_chat_response for streamed chat), so the model sees one consistent world and the client never sees the alias. The map is rebuilt deterministically from the same payload at both ends (_response_tool_aliases, protocol-independent), so fallbacks crossing providers need no state. Clean requests pass through untouched (same object, no log, verbatim relay); aliased ones log Meta reserved tool names aliased: bash->ext_bash, .... If 403s return, re-probe: Meta may have added a builtin.

Incident 2026-11-10 (tai devplace/molodetz 403s, Devii unaffected): the gate scans NAMES, not counts. tai's 63-tool catalog shares 7 names with the denylist (shell, read_file, write_file, edit_file, web_fetch, web_search, search); Devii's 400+ domain tools share 0, so 400 tool calls never tripped it. Three rounds, 2026-11-10: (1) the aliasing originally ran only for responses-protocol attempts while tai/detz speak chat_completions, so colliding names reached Meta verbatim - fixed by hoisting the aliasing above the protocol branch in _chat_upstream_body and restoring names on the chat return paths (non-streaming re-encode, streamed chunk restore); (2) the denylist was missing shell, which tai/detz send on EVERY turn (first entry of CORE_TOOLS, core loads always) - added (21 names), corroborated by Pi's pi-meta-oauth denylist, live via --reload 18:26; (3) REFUTED 18:33: requests with all 5 known triggers aliased (shell->ext_shell on the wire) plus stamped muse profile STILL 403. The 21-name gate theory is therefore insufficient. Remaining fork: another ungated name in the payload (candidates never probed by anyone: web_fetch, fork, poll, ask_user, remember, recall, load_skill, ...), the ext_ scheme itself recognized, or the key fully gated for non-genuine clients. Discriminator: two authenticated direct calls against the route, one tool-less and one with a single clean tool - see the research conclusion for the exact probes. If the tool-less call also 403s, stop guessing names and put a pay-as-you-go key on the route. Outcome 19:05, after a clean service restart: single-name live probes proved web_fetch denied alone (added as 22nd name) while fork, poll, ask_user, remember, recall, load_skill, ext_shell each 200 alone; the full 63-tool tai catalog then 200d in one call, and the journal shows web_fetch->ext_web_fetch aliasing on real traffic with 9 upstream 200s and 0 403s since. The gate is names, the denylist at 22 covers the whole static catalog, no key change needed. Count caps measured live 2026-11-10 on the Meta route (native tools_mode): 70, 100, 200 and 400 tools per call all HTTP 200 (63 real catalog names plus synthetic clean names) - there is no count gate on this path. The only count cap in the system is the gateway's own emulated-tools path (which had a 64-tool cap until it was removed from the working tree the same day - test_large_catalogs_are_served_with_no_cap pins 424 tools served, unrelated to Meta). The companion openrouter/... 401 No cookie auth credentials found in the same report is unrelated: tai calling OpenRouter with no API key configured, an operator setup gap, not a gateway behavior. Tests: tests/unit/services/openai_gateway/gateway.py (test_meta_tool_names_aliased_on_wire_and_restored / test_meta_tool_names_restored_on_stream_chunks for Responses, test_meta_tool_names_aliased_on_chat_completions_wire / test_meta_tool_names_restored_on_chat_completions_stream for chat, the chat tests pin shell+read_file end to end) and tests/unit/services/openai_gateway/muse_code.py (denylist membership + alias/round-trip unit tests). Full external-research record: research/muse-attestation-gate/ in the tai project.