devplacepy/devplacepy/services/openai_gateway/CLAUDE.md

This file documents the AI gateway subsystem (devplacepy/services/openai_gateway/) - the single point of truth for AI calls, usage metering, and provider/model routing. Claude Code loads it automatically whenever a file under this directory is read or edited.

Overview

GatewayService (services/openai_gateway/) is an OpenAI-compatible LLM gateway (the ported openai5.py), mounted at /openai/v1/* (routers/openai_gateway.py, prefix /openai). It is a service that serves an HTTP endpoint rather than a loop.

• The router is thin: it calls service_manager.get_service("openai").handle(request, subpath). POST /v1/chat/completions runs the full gateway (vision augment -> model override -> thinking default -> forward, real upstream streaming when the client asked for it); POST /v1/embeddings forwards to the configured embeddings upstream (handle_embeddings, no vision/streaming); GET /v1/models is answered locally from the gateway_models chat routes (_models_response, publishing the public molodetz/molodetz-pro names), NOT proxied upstream; /v1/{path:path} is a transparent passthrough to the upstream base. The gateway forwards stream: true to the upstream verbatim and relays its SSE chunks as they arrive (GatewayRuntime._stream_chat_response, client.send(request, stream=True) + resp.aiter_lines()) - so TTFT and inter-token latency are real, measured timings, not simulated. When streaming, the gateway ALWAYS additionally sends stream_options: {"include_usage": true} upstream regardless of what the client asked (so the ledger always gets a real usage object off the final chunk); the client's own include_usage only controls whether that usage-only chunk is relayed downstream to it - the gateway still swallows-and-records it either way. Load-bearing header trade-off: a streaming response carries no X-Gateway-Cost-USD/token headers (only X-Gateway-Model/X-Gateway-Backend/X-App-Reference) - HTTP headers must be sent before the body, and cost/tokens aren't known until the stream ends, so the usage ledger row (and the client's own final usage chunk, if it asked for one) are the only ways to learn a streamed call's cost; every non-streaming response is completely unaffected and keeps the full header set exactly as before. The ledger row for a streamed call is written from inside the generator's finally once the stream ends (or is interrupted - a caught mid-stream exception or a client disconnect, i.e. GeneratorExit, both still write a row with success=False and an error_category, never silently drop the accounting); a client hangup never leaves an unclosed upstream connection (resp.aclose() always runs). Thinking is off by default (thinking.py apply_thinking): DeepSeek V4 enables thinking unless thinking.type=disabled is sent, so the gateway always writes an explicit disable unless the client (or gateway_thinking) asked for thinking. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
• main.py registers it and exempts /openai from the rate-limit middleware and the maintenance gate. No new dependency (httpx already required).
• GatewayService is default_enabled=True so internal callers work out of the box.

Per-call cost/token response headers

Every NON-STREAMING gateway response (chat, embeddings, images, passthrough; success and error) carries X-Gateway-* headers describing that single call: Model, Backend, Prompt-Tokens, Completion-Tokens, Total-Tokens, Cache-Hit-Tokens, Cache-Miss-Tokens, Reasoning-Tokens, Cost-USD/Input-Cost-USD/Output-Cost-USD (dollars), Cost-Native (1 if the upstream returned a native cost), Tokens-Per-Second, Upstream-Latency-Ms, and Context-Window/Context-Utilization when the model's window is known, plus the timing headers X-Gateway-Upstream-Latency-Ms/X-Gateway-Total-Latency-Ms. GatewayUsageLedger.record(...) RETURNS the computed ledger row (or None on failure); each handler's finalize closure maps it through usage.usage_response_headers(row) and attaches it to the returned Response (resp_headers). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads X-Gateway-Cost-USD/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root CLAUDE.md); every internal consumer sends stream: false (or omits it), so this path is completely unaffected by the streaming trade-off below.

A STREAMING chat response (stream: true) is the one exception and carries only X-Gateway-Model/X-Gateway-Backend/X-App-Reference - HTTP headers are sent before the body, and cost/tokens for a streamed call are only known once the stream ends, so they cannot be headers on that same response. The call is still fully metered: the ledger row (including ttft_ms/inter_token_ms) is written server-side once the stream completes (see "Real upstream streaming" below), and a client that requests stream_options.include_usage still gets the upstream's real usage object on the final SSE chunk, exactly as the OpenAI streaming API itself works - it is simply not summarized into response headers.

App-reference header (X-App-Reference)

Callers may send an optional X-App-Reference header to tag gateway calls by application. The value is validated and stored in the app_reference column of gateway_usage_ledger, surfaced in analytics and admin reporting.

• Validation (service._validate_app_reference): trimmed whitespace, then matched against ^[a-zA-Z0-9_.-]{1,30}$. Any value failing validation (empty, >30 chars, contains spaces or @// etc.) silently falls back to "default".
• Header name: X-App-Reference.
• All internal callers (news, bots, Devii, correction, jobs, deepsearch, dbapi, gitea) should pass a devplace-<component>-v-<major>-<minor>-<patch> value, e.g. devplace-devii-v-1-0-0, devplace-news-v-1-0-0.
• The column is indexed (CREATE INDEX IF NOT EXISTS) for fast per-app queries.

Per-worker runtime

Serves in every worker. Config/enabled come from site_settings (via the service's get_config()/is_enabled()), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (GatewayRuntime: httpx.AsyncClient pool + asyncio.Semaphore sized to gateway_instances, plus counters and a VisionCache) is created lazily on first request and rebuilt when instances/timeout/cache size change. gateway_instances is the scaling knob (concurrency per worker); process scaling is uvicorn workers.

Auth

authorize() reuses get_current_user (cookie/X-API-KEY/Bearer/Basic). Allowed if gateway_require_auth is off, the presented X-API-KEY/Bearer equals the static gateway_access_key or the auto-generated gateway_internal_key, or the resolved user is an admin (gateway_allow_admins) or any user (gateway_allow_users).

Per-user attribution: the gateway also accepts a real user's api_key (Bearer / X-API-KEY / session), gated by gateway_allow_users (default on); resolve_owner() records (owner_kind, owner_id) per call, so usage is attributed and limitable per user. With gateway_allow_users off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own api_key (set as the session's ai_key in build_settings(..., owner_kind="user")), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via build_user_usage() / GET /admin/users/{uid}/ai-usage. Guests keep the internal key.

Config fields

All config is config_fields (upstream url/model/key, force-model, the Prompt group's gateway_system_preamble and gateway_thinking (default off), timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (gateway_embed_enabled/_url/_model/_key), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via collect_metrics.

System-message composition (date awareness + operator preamble)

system_message.py (apply_system_directives) runs in handle_chat only (NOT handle_embeddings/handle_passthrough), after vision augmentation and before the upstream payload is assembled, so it reaches every chat consumer (news, bots, Devii guests, per-user direct API calls). It composes ONE system message in a fixed, cache-friendly order: operator preamble (gateway_system_preamble) -> date line (Current date: DD/MM/YYYY) -> the client's original system content.

Two behaviors:

• Date awareness (EU, no time). The current date is injected only when the composed text (preamble + client system content) contains NO date in any common format. Detection lives in contains_date via the _DATE_PATTERNS regex set: YYYY-MM-DD (ISO), D/M/YYYY and DD/MM/YYYY (also matches MM/DD/YYYY), DD-MM-YYYY, DD.MM.YYYY, and textual months (14 June 2026, June 14, 2026, 14 Jun 2026, with abbreviations and optional ordinal/comma). The date is datetime.now().strftime("%d/%m/%Y") - date only, never time: omitting the time keeps the system message byte-stable for the whole day so upstream prompt/token caching stays effective.
• Operator preamble. gateway_system_preamble (Prompt group, type="text" textarea, default empty) is prepended ahead of the client's system content on every chat call so all calls carry operator preferences. When the client sent NO system message, the composed content (preamble and/or date) becomes a brand-new leading system message; when the client DID send one, its content is replaced in place (preamble + date + original text). A blank/whitespace-only preamble with an already-dated request is a no-op (no empty system message is created). The date detection runs over the preamble too, so an operator preamble that already states a date suppresses the injected date line.

Embeddings/passthrough are untouched by this composition step.

Thinking default (fast path)

thinking.py (apply_thinking) runs in handle_chat after stream_options is stripped, and in the vision describe call. DeepSeek V4 enables thinking by default; leaving the payload alone is the slow path. The gateway therefore always writes an explicit thinking field in the upstream dialect:

• DeepSeek (*deepseek.com* and any unrecognised OpenAI-compat URL, because that is the production default): thinking: {"type": "disabled"} or "enabled"
• OpenRouter: reasoning: {"effort": "none"} or "high"
• Ollama (:11434 or a host containing ollama): think: false or true

Do not send reasoning_effort: "none" to DeepSeek (400: unknown variant). Conflicting client fields (think, thinking, reasoning, reasoning_effort, enable_thinking, chat_template_kwargs.enable_thinking) are stripped and translated to the dialect above.

Default is off. gateway_thinking (Prompt group, bool, default False) is the operator override for the silent-client case. A client still wins: think / thinking.type / reasoning.effort / reasoning.enabled / reasoning_effort / enable_thinking on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.

Vision describe-image calls always disable thinking (they are not a reasoning job).

Dialect can be overridden when the URL does not identify the provider. gateway_thinking_dialect (Prompt group, select, default auto, options auto/deepseek/openrouter/ollama) forces thinking.py::thinking_dialect's URL-sniffing result - needed for a reverse-proxied Ollama whose URL carries no :11434/ollama token. apply_thinking(payload, url, default_enabled, dialect=...) and the vision augmenter (VisionAugmenter.vision_dialect, passed from cfg["gateway_thinking_dialect"]) both accept it; "auto"/blank falls through to the existing URL-based detection unchanged.

thinking.upstream_capabilities(url, dialect="") derives per-upstream protocol flags (supports_stream_options, supports_stream_usage, supports_thinking_field) from the same effective dialect. Ollama's OpenAI-compatibility layer ignores stream_options (and rejects the request outright on some versions), so handle_chat's streaming branch only sends stream_options: {"include_usage": true} upstream when capabilities.supports_stream_options is true; otherwise a client that asked for include_usage gets a logged notice and its stream relayed with no usage chunk. DeepSeek/OpenRouter (and any unrecognised URL) report full support, so this is a no-op for them.

Real upstream streaming (GatewayRuntime._stream_chat_response)

stream: true is forwarded to the upstream verbatim (payload["stream"] = stream, no more forced false) and the connection is opened with client.send(request, stream=True) so the response body is read incrementally via resp.aiter_lines() instead of buffered whole. Every retry/circuit-breaker mechanic in _send/retry_send is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; retry_send now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns 200 does the code path diverge into _stream_chat_response.

• The client always gets real chunks, in real time, each parsed data: {...} line re-serialized and yielded as its own SSE event - no more splitting a complete response into fake 50-character deltas.
• TTFT and inter-token latency are measured, not simulated. ttft_ms = time from just before the upstream POST to the first SSE line received; inter_token_ms = the average gap between content-bearing chunks (delta.content/delta.reasoning_content) after the first one, None when fewer than two content chunks arrived. Both ride the ledger row (gateway_usage_ledger.ttft_ms/.inter_token_ms, nullable REAL columns dataset auto-adds on the first insert carrying them - no explicit init_db() ensure-block needed, same as every other column on this insert-only table) and surface in /admin/ai-usage's latency.ttft_ms/latency.inter_token_ms (analytics.py, same _pset percentile shape as every other latency dimension).
• Usage is always requested from the upstream when streaming, regardless of what the client asked (stream_options.include_usage is forced true in the upstream payload), so the ledger always gets a real usage object off the final chunk. The client's OWN stream_options.include_usage only controls whether that usage-only chunk is relayed downstream to it; the gateway swallows-and-records it either way.
• The ledger row is written once, from inside the generator's finally, after the stream ends (saw_done), is interrupted by an upstream error, or the client disconnects (GeneratorExit - re-raised after cleanup, never swallowed, and never yielded-from during close: Python forbids yielding while handling GeneratorExit). All three paths close the upstream response (resp.aclose()) so a dropped client never leaks the connection, and all three still write a ledger row (success=False/error_category="client_disconnected" or a classified error on interruption) so accounting is never silently skipped.
• No X-Gateway-Cost-USD/token headers on the streaming response - see "Per-call cost/token response headers" above; this is the one deliberate exception to "every response carries the headers."
• Non-streaming chat, embeddings, and images are completely unchanged - this only touches the stream: true branch of handle_chat.

Usage, cost, latency, and reliability tracking

The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:

• usage.py - GatewayUsageLedger writes to gateway_usage_ledger (one row per call, success and failure) and samples gateway_concurrency_samples each 30s tick. normalize_usage handles BOTH upstream shapes: DeepSeek (prompt_cache_hit_tokens/prompt_cache_miss_tokens, completion_tokens_details.reasoning_tokens) and OpenRouter (prompt_tokens_details.cached_tokens, native cost, cost_details). compute_cost PREFERS the upstream native cost (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share input_cost / (input_cost + output_cost) from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). record() is wrapped in try/except so a tracking failure never breaks the proxy.
• gateway.py - _send is the gated/timed/retried path: a CircuitBreaker (reliability.py) short-circuits when open; retry_send retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the queue_wait_ms; an httpx per-request trace extension captures connect_ms (0 on pooled reuse). handle_chat/handle_passthrough record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from VisionAugmenter._describe_one.
• Owner attribution - GatewayService.resolve_owner maps the caller to (owner_kind, owner_id): internal:devii (internal key), key:access (static key), user:<uid>/admin:<uid> (session or API key), else anonymous.
• analytics.py - build_analytics(hours, top_n, pricing) pulls the bounded window once (clamped to MAX_WINDOW_HOURS=168) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via reliability.percentile), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. summary_metrics() runs cheap SQL aggregates for the live service panel. TTFT/inter-token (latency.ttft_ms/latency.inter_token_ms, the same _pset percentile-set shape as every other latency dimension) are real, measured only for calls that requested streaming - count: 0 in a window means no streaming traffic occurred, not that the metric is unavailable.
• Admin - /admin/ai-usage (HTML, templates/ai_usage.html + static/js/AiUsageMonitor.js) and /admin/ai-usage/data?hours=&top_n= (JSON) in routers/admin/ package, admin-gated. Schema GatewayUsageOut. Devii action ai_usage and docs admin-ai-usage expose the same endpoint. The ai_usage and site_analytics Devii catalog tools are both requires_admin=True.
• Per-user usage - analytics.build_user_usage(owner_id, hours=24, pricing) filters the ledger by owner_id (a uid matches both user: and admin: rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at GET /admin/users/{uid}/ai-usage (schema UserAiUsageOut, docs admin-user-ai-usage) and rendered on the profile page by static/js/UserAiUsage.js (the [data-ai-usage] card, gated {% if is_admin(user) %} in templates/profile.html). The 24h window is a sound projection basis (unlike a startup burst), so cost_24h * 30 is honest.
• Profile visibility split - routers/profile/ package _ai_quota(uid, include_cost=False) builds the profile quota bar. The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal: spent_usd/used_pct/requests/last_used come from analytics.user_spend_24h(uid), the SAME gateway_usage_ledger source (summed by owner_id, matching both user: and admin: rows) that feeds the stat grid's cost.window_usd. So the server-rendered spent_usd is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. turns is still the devii_usage_ledger turns_24h count (Devii conversation turns) and is labeled "Devii turns" in the template precisely because it is a different metric from the grid's total API requests (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why 0 Devii turns can sit beside 18 API requests). The limit/turns still come from the Devii service (daily_limit_for/turns_24h); an unlimited limit (<= 0, the admin default) sets unlimited=True so the template renders "no cap" / "No daily spend cap" instead of a misleading / $0.00. This is display-only: the Devii daily-cap ENFORCEMENT and /devii/usage still read the Devii-scoped devii_usage_ledger.spent_24h (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: admin sees the full gateway stats card plus a server-rendered quota bar (with spend/limit/Devii turns); the owner (non-admin, is_self) sees a quota-only card showing just the percentage; everyone else sees nothing AI-related. _ai_quota is fail-safe (returns None on any error or when the Devii service is unregistered) and the result rides on ProfileOut.ai_quota. Dollar figures (spent_usd/limit_usd) are included only when include_cost=viewer_is_admin - never for a non-admin owner. This matters because the route serves the same context as JSON via respond(..., model=ProfileOut) and ProfileOut.ai_quota is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with Accept: application/json; the server-side omission is the real control. Non-admins (HTML and JSON) get only used_pct/turns/requests.
• Retention - run_once prunes both tables past gateway_usage_retention_hours (default 720 = 30 days).

Financial data is admin-only everywhere. Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, ai_correction/ai_modifier usage displays, and the Devii cost tools.

Quota rules (quota.py, admin /admin/gateway "Quota rules" section)

This caps /openai/v1/* itself, independent of Devii's own daily cap. Devii's devii_user_daily_usd/devii_guest_daily_usd/devii_admin_daily_usd (documented in devplacepy/services/devii/CLAUDE.md) only gate turns that go through Devii. A caller hitting the gateway directly with their own api_key bypasses that entirely - quota.py is the root-level enforcement that closes this, checked in GatewayService.handle() right after owner/app_reference resolution and before every billed dispatch (chat, embeddings, images, passthrough; GET /v1/models is exempt, it makes no upstream call).

Two layers, same shape as provider/model routing above. Layer A is five flat config_fields on GatewayService (gateway_default_user_daily_usd $1.00, gateway_default_admin_daily_usd $0/unlimited, gateway_default_guest_daily_usd $0.05, gateway_default_internal_daily_usd $0/unlimited, gateway_default_key_daily_usd $0/unlimited - group Quota) applied per specific caller (owner_id) when no rule matches; internal/key default unlimited so shipping this never starts blocking DevPlace's own news/bots/Devii-guest/correction traffic on the internal key. Layer B is the gateway_quota_rules table (ensure_tables(), called from init_db() alongside routing.ensure_tables(); hard CRUD, not in SOFT_DELETE_TABLES, cross-worker cache-invalidated under the "gateway_quota" name): each row scopes by any combination of owner_kind (internal/key/user/admin/anonymous - DevPlace's only "roles" here), a specific owner_id, and app_reference (the X-App-Reference label), each nullable = wildcard; a QuotaRuleIn Pydantic validator rejects a rule with all three blank (that belongs in Layer A). quota.resolve(owner_kind, owner_id, app_reference, cfg) gathers every active rule whose non-null dimensions all equal the request, picks the one with the most non-null dimensions (ties broken toward the smaller limit, unlimited 0 never wins a tie against a finite cap), and returns (limit_usd, scope, rule) where scope is the exact (owner_kind, owner_id, app_reference) triple - each possibly None - that spend must be summed over. Layer A is internally just the maximally-specific implicit scope (owner_kind, owner_id, None), so one code path (quota.spent_24h(*scope), a plain SUM(cost_usd) over gateway_usage_ledger filtered by whichever scope dimensions are non-null) serves both layers.

A wildcard dimension means a shared pool, by design. A rule scoped only by app_reference caps that app's combined spend across every caller using it; a rule scoped only by owner_kind caps that whole role's combined spend. Pin owner_id to get a true per-caller cap (the Layer A default's own behavior). anonymous/internal/key owner_ids are already fixed constants ("anonymous"/"devii"/"access", from resolve_owner()), not per-caller identities, so any cap on those kinds is inherently pooled - there is no per-guest identity at this layer (unlike Devii's own guest-cookie-scoped ledger).

No lock, no hold, bounded overshoot by design - this is deliberate, not an oversight. Cost is only known after the upstream call returns, so a true atomic pre-authorization would need a reserve-then-reconcile ("hold") mechanism, and a bug in releasing a hold is exactly the kind of thing that gets a caller stuck forever. Instead this mirrors Devii's own already-shipped mechanism exactly: read the 24h sum, compare, raise HTTPException(429, ...) if already at/over - a single SELECT and a conditional raise, nothing held, nothing to leak, structurally impossible to deadlock. The tradeoff is a small, bounded overshoot (at most a few concurrent in-flight calls' worth of cost past the cap before the next request sees the updated sum and blocks) - acceptable and industry-standard for a cost whose exact size isn't known until the call finishes, and it is the property actually being enforced: once tripped, every subsequent separate request stays blocked until the 24h window rolls off or an admin adjusts the rule.

429 body never carries a dollar figure, admin or not ({"detail": "AI gateway daily quota exceeded"}) - mirrors Devii's own over-limit WS message, which likewise never states a number. The admin-only services log line and the ai.quota.exceeded audit row (GatewayService._audit_quota_exceeded, reusing usage.audit_actor_for) do carry the spend/limit/matched-rule-uid, since those are admin-only surfaces.

Resetting the counted spend (gateway_quota_resets). A cap is only lifted by time otherwise, so there is a reset that clears what has been counted without deleting any ledger row - gateway_usage_ledger is the cost-analytics source for /admin/ai-usage, so a reset must never truncate it. quota.reset(QuotaResetIn, created_by=) upserts one watermark row into gateway_quota_resets (same ensure_tables()/"gateway_quota" cache-version/hard-CRUD shape as the rules table, same two indexes) scoped by the SAME three nullable dimensions as a rule, and quota.spent_24h sums from max(24h cutoff, reset_watermark(scope)). A reset row applies to a queried scope when each of its non-null dimensions equals that scope's - so an all-null reset clears everyone, while a reset scoped to one app deliberately does NOT clear a broader per-user-all-apps scope (clearing a narrower window can only over-credit). Spend recorded after the reset counts again immediately against the same limit. QuotaScopeIn is the shared base holding the three dimensions and their validators; QuotaRuleIn and QuotaResetIn both extend it, so scope parsing exists once.

The two AI quotas are separate systems and the reset surfaces must say so. /admin/ai-usage's Reset all quotas clears the Devii devii_usage_ledger AND now also stamps a global gateway watermark, because a caller hitting 429 AI gateway daily quota exceeded had no reset at all before and the button looked global. Reset guest quotas stays Devii-only (guest gateway calls ride the shared internal key, so there is no per-guest gateway scope to clear).

CRUD. Admin JSON at /admin/gateway/quota-rules (routers/admin/gateway_configs.py, list returns each rule's live spent_24h_usd plus the Layer A defaults for context), audited gateway.quota_rule.update/gateway.quota_rule.delete (category ai, both already in events.md), rendered in the Quota rules section of /admin/gateway (GatewayAdmin.js, mirrors the providers/models CRUD tables). Devii tools gateway_quota_rules/gateway_quota_rule_set/gateway_quota_rule_delete (requires_admin=True, delete is CONFIRM_REQUIRED) proxy the same endpoints via handler="http", same as the provider/model tools. Reset is POST /admin/gateway/quota-resets (same file, _payload/ValidationError shape as the rule CRUD), audited gateway.quota.reset (category ai), surfaced as a per-rule Reset spend button in the Quota rules table (GatewayAdmin.js), and exposed as the Devii tool gateway_quota_reset (requires_admin=True, in CONFIRM_REQUIRED with a declared confirm param, like the other quota-lifting admin resets). CLI: devplace gateway quota list|set|delete|reset.

Image generation

POST /openai/v1/images/generations exposes an OpenAI-compatible image-generation endpoint. Clients send the generic model molodetz-img-small (config.INTERNAL_IMAGE_MODEL), which handle_images remaps to gateway_image_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty or molodetz-img). It defaults to OpenRouter's black-forest-labs/flux-1.1-pro at https://openrouter.ai/api/v1/images/generations (config.IMAGE_*_DEFAULT, $0.04 per image fallback). handle_images mirrors handle_embeddings: build the payload, forward via _send, and record one ledger row. The config fields are the Images group (gateway_image_enabled default on, gateway_image_url, gateway_image_model, gateway_image_key) plus the Pricing-group gateway_image_price_per_call. effective_config() falls the image key back to gateway_api_key then OPENROUTER_API_KEY. Usage is recorded with backend="image"; usage.compute_cost adds an image branch (flat per-call, native OpenRouter cost still preferred via extract_image_usage). routing.image_overlay resolves per-route provider/url/key and uses price_input_per_m as the per-image price. routing.seed_default_image_routes() (from migrate_ai_gateway_settings) idempotently seeds molodetz-img-small -> Flux on the openrouter provider when OPENROUTER_API_KEY is set. When gateway_image_enabled is off the endpoint returns 503 with no ledger row.

Embeddings

POST /openai/v1/embeddings exposes an OpenAI-compatible text-embeddings model. Clients send the generic model molodetz~embed (config.INTERNAL_EMBED_MODEL), which handle_embeddings remaps to gateway_embed_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty). It defaults to OpenRouter's qwen/qwen3-embedding-8b at https://openrouter.ai/api/v1/embeddings (config.EMBED_*_DEFAULT, $0.01 per 1M input tokens). handle_embeddings mirrors handle_chat but is simpler: no vision augmentation and no streaming - build the payload, forward via _send, and record one ledger row through the same finalize(...) closure. The config fields are the Embeddings group (gateway_embed_enabled default on, gateway_embed_url, gateway_embed_model, gateway_embed_key) plus the Pricing-group gateway_embed_price_input_per_m. effective_config() falls the embed key back to gateway_vision_key then OPENROUTER_API_KEY (NOT gateway_api_key: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with backend="embed"; usage.compute_cost adds an embed branch (input-only, completion always 0, native OpenRouter cost still preferred) and Pricing gained embed_input_per_m. analytics.py groups by backend generically, so embed rows roll up automatically; caching_savings counts only non-native chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When gateway_embed_enabled is off the endpoint returns 503 with no ledger row.

Single point of truth for AI

The gateway is the only place that holds real provider URLs/models/keys. Every other AI consumer (news, bots, Devii guests) points at it by default and never touches a provider key:

• Defaults live in config.py: INTERNAL_GATEWAY_URL (http://localhost:{DEVPLACE_PORT, default 10500}/openai/v1/chat/completions; the whole base can also be overridden with DEVPLACE_INTERNAL_BASE_URL) and INTERNAL_MODEL (molodetz). news_ai_url, bot_api_url, devii_ai_url default to INTERNAL_GATEWAY_URL; their model defaults to molodetz.
• Each consumer's key falls back to database.internal_gateway_key() (reads the gateway_internal_key setting) when its own key field/env is unset - the provider-key fallbacks (DEEPSEEK_API_KEY/OPENROUTER_API_KEY) were removed from news and bots.
• gateway_force_model (default on) and a molodetz/empty alias in handle_chat make the upstream always receive gateway_model, so molodetz is a stable generic alias. gateway_allow_client_model (Upstream group, bool, default off) lets a client-requested model name through even while gateway_force_model is on - useful for an upstream that serves many models by name with no single stable alias (a self-hosted Ollama). The molodetz alias is always remapped to gateway_model regardless of this flag, so the generic name keeps working for internal callers; only a genuinely named model (llama3.2, qwen3, ...) bypasses the force.
• database.migrate_ai_gateway_settings() (called at the end of init_db(), under the startup init_lock): generates gateway_internal_key (uuid4) if missing; migrates DEEPSEEK_API_KEY/OPENROUTER_API_KEY env into gateway_api_key/gateway_vision_key when the db value is empty; and rewrites any consumer AI URL still equal to the old openai.app.molodetz.nl default to the gateway, plus bot_model deepseek-chat -> molodetz (only uncustomized values).
• The gateway's gateway_api_key/gateway_vision_key/gateway_internal_key fields are non-secret so the admin services page shows the value actually in use, editable.
• Bot LLM calls are synchronous urllib but already run via asyncio.to_thread (bot/bot.py), so the local round-trip never blocks the event loop.
• Real provider keys/URLs/models live ONLY in the gateway. The gateway is default_enabled=True.

Provider and model routing (routing.py, admin /admin/gateway)

Layered ON TOP of the single-provider service config above, which stays THE implicit default provider (env migration, internal key, molodetz/molodetz~embed, vision, embeddings, ledger all unchanged).

Storage. Two dataset tables, ensured in init_db via routing.ensure_tables, cross-worker cache-invalidated under the "gateway_routing" cache-version name (module-level provider_store/model_store over a shared _ROUTING_CACHE; writes bump_cache_version and clear the cache). They are admin config, NOT in SOFT_DELETE_TABLES - hard CRUD, mirroring site_settings:

• gateway_providers - named upstreams: name + base_url (chat-completions URL) + api_key + is_active; the embeddings URL is derived by swapping /chat/completions -> /embeddings.
• gateway_models - source->target routes: source_model (unique, what clients request) -> provider (blank = default) + target_model + kind (chat|embed|image) + optional vision_provider/vision_model for the text+vision merge (when set, image content is described by that vision model before forwarding) + context_window + its own economy (price_cache_hit_per_m/price_cache_miss_per_m/price_output_per_m/price_input_per_m, USD per 1M tokens for chat/embed/vision; for image routes price_input_per_m is a flat USD per image, plus the tiered/off-peak fields below) + is_active.

Resolution is a per-request overlay, not a fork. At request time routing.chat_overlay(requested, cfg) / routing.embed_overlay(requested, cfg) / routing.image_overlay(requested, cfg) resolve an active route by the requested model name and return a per-request OVERLAY dict of gateway_* cfg keys (gateway_force_model+gateway_model=target, gateway_upstream_url/gateway_api_key from the provider, the price keys, an augmented gateway_model_context_map, and vision overlay keys); handle_chat/handle_embeddings merge it onto the base cfg (cfg = {**cfg, **overlay}) BEFORE everything else, so the existing model-selection / pricing_from_cfg / parse_context_map / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. _ensure (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.

No matching route = None overlay = model fallback. When no route matches, the overlay returns None and the handler's model selection logic falls back to the default configured model (gateway_model for chat, gateway_embed_model for embeddings, gateway_image_model for images). The unknown model name is discarded, not forwarded upstream. This means an unknown or misspelled model name never causes a 4xx from the upstream - it is gracefully downgraded to the default. The force_model guard and the built-in molodetz/molodetz~embed/molodetz-img-small aliases are still respected before the route check: force_model or empty/named-alias model -> default directly; known model -> route resolution; unknown model with no route -> default fallback with a log message.

CRUD. Admin JSON at /admin/gateway/{providers,models} (routers/admin/gateway_configs.py, require_admin, Pydantic ProviderIn/ModelRouteIn validation, accepts both JSON from static/js/GatewayAdmin.js and form from Devii), audited under gateway.provider.*/gateway.model.* (category ai), included in the admin package with the page at /admin/gateway (templates/admin_gateway.html, sidebar link, admin_section="gateway").

Devii tools: gateway_providers/gateway_models (read) and gateway_provider_set/gateway_provider_delete/gateway_model_set/gateway_model_delete (admin; the two deletes are CONFIRM_REQUIRED).

When adding a routed value, overlay it as the matching gateway_* cfg key so the runtime needs no new branch.

Tests: tests/unit/services/openai_gateway/routing.py (overlay/economy/kind isolation), tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream (end-to-end through handle_chat), and tests/api/admin/gateway/ (admin CRUD + validation + role gating).

Automatic model fallback (fallback_model, one hop, per route)

Any gateway_models route (chat, embed, or image) can optionally name fallback_model: another already-configured source_model of the SAME kind, tried once, automatically, when the primary route fails. This is admin-configured on the /admin/gateway model form as a select box populated from every OTHER route of the current kind (the public source_model names such as molodetz/molodetz-pro, never the internal target_model a provider actually sees) - never a free-text field, so a fallback can only ever point at a model the gateway already knows how to serve.

• Trigger. _call_failed(resp, exc, timing) in gateway.py treats a call as failed when the circuit breaker rejected it, the upstream connection raised, or the upstream answered with any status >= 400. This runs AFTER _send's own gateway_max_retries retries against the primary model are exhausted - a fallback is the next escalation once retrying the SAME model has already given up, not a replacement for that retry loop.
• One hop, never a chain. routing.resolve_fallback(source_model, kind) resolves the primary route, reads its fallback_model, and resolves THAT model's own route (must exist, be active, and share the kind) - it does NOT recurse into the fallback's own fallback_model, so there is no possibility of a cycle or an unbounded retry chain. A route that fails even after redirecting to its fallback returns the fallback attempt's own failure to the caller.
• Self-reference is rejected at write time. ModelRouteIn._check_fallback_is_not_self (pydantic model_validator) refuses fallback_model == source_model; the admin route handler (routers/admin/gateway_configs.py::save_model) additionally rejects a fallback_model that does not resolve to an existing, same-kind route (routing.model_store.get(...)) before writing the row, since a pydantic validator alone cannot see other rows.
• Rebuilt like a fresh routed call, not retried in place. On failure, handle_chat/handle_embeddings/handle_images recompute the overlay from base_cfg (the pre-primary-overlay config) via chat_overlay(fallback_route.source_model, base_cfg)/embed_overlay/image_overlay, so the fallback gets its OWN provider, URL, key, and pricing - never the primary route's. pricing/context_map for the ledger are (re)computed AFTER the fallback decision so the recorded cost always reflects whichever model actually served the request.
• One ledger row per client call, not one per attempt. The primary failure never writes a gateway_usage_ledger row by itself (only _send's own internal retries and the circuit breaker counters observe it); finalize()/self._ledger.record(...) still runs exactly once, after the fallback decision, with model = whichever model ultimately answered and requested_model = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in /admin/ai-usage as "requested X, served Y" rather than as two calls.
• Streaming falls back before any byte reaches the client. client.send(request, stream=True) returns as soon as the response headers arrive, so resp.status_code is already known before the SSE body is ever touched - handle_chat decides whether to fall back at that same point, before it ever opens _stream_chat_response to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
• Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (self._breaker) is one instance per GatewayRuntime, shared across every route. A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see circuit_open too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
• Deliberately NOT covered: gateway_max_retries/gateway_retry_backoff_ms, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.

Tests: tests/unit/services/openai_gateway/routing.py (resolve_fallback self-reference/kind/active-flag guards), tests/unit/services/openai_gateway/gateway.py (test_chat_falls_back_when_the_primary_model_fails, test_chat_returns_the_fallback_failure_when_both_models_fail, test_chat_without_a_fallback_never_makes_a_second_call), and tests/api/admin/gateway/models.py (test_model_route_fallback_round_trip, test_model_route_fallback_must_reference_an_existing_route, test_model_route_fallback_must_be_the_same_kind, test_model_route_fallback_rejects_self_reference).

Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)

Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME gateway_models route row, so it stays fully provider-agnostic and opt-in per route.

• Per-route fields (each optional/zero by default = feature off, so an unconfigured route is byte-identical to before): context_tier_threshold_tokens (0 disables tiering; when the request's input token count exceeds it, price_*_per_m_tier2 rates apply instead of the tier-1 rates above) and the four nullable price_cache_hit_per_m_tier2/price_cache_miss_per_m_tier2/price_output_per_m_tier2/price_input_per_m_tier2 (a component left None keeps its tier-1 rate even above threshold - so a provider that only re-prices input above a size threshold, keeping output flat, needs just one tier2 field set), plus off_peak_start_minute/off_peak_end_minute (nullable, UTC minutes-since-midnight, both-or-neither enforced by ModelRouteIn's model validator; a window where start > end wraps past midnight, e.g. DeepSeek's old V3/R1-era 16:30-00:30 UTC window) and off_peak_discount_pct (0-100, multiplies whichever tier rate is active). The same fields double for vision (price_input/output_per_m_tier2) and embed (price_input_per_m_tier2) exactly like their tier-1 counterparts already do.
• Overlay + computation. chat_overlay/embed_overlay propagate all of these into the per-request cfg dict under gateway_* keys exactly like the existing price fields (section above); usage.pricing_from_cfg reads them generically (defaulting to None/0/disabled when absent), so Layer A (the single global flat Pricing config fields) never gains these dimensions - only a gateway_models route can enable them, preserving the "no matching route = byte-identical legacy behavior" guarantee. usage.compute_cost selects tier1 vs tier2 per rate component (_tiered_rate) based on whether norm["prompt"] (billable input tokens) exceeds the threshold, then applies the off-peak discount (_effective_rate/_off_peak_active, UTC wraparound-aware) to whichever rate was selected. The response header format and compute_cost's return shape (total, input_cost, output_cost, native) are unchanged - this is purely an internal rate-selection step before the existing input/output split math runs; a native upstream cost (OpenRouter) still overrides the modeled total exactly as before.
• Migration. New gateway_models columns are added via has_column/create_column_by_example in routing.ensure_tables() (the CREATE TABLE IF NOT EXISTS DDL string alone would never reach a pre-existing table - see the database/CLAUDE.md column-ensure idiom).
• Admin UI. /admin/gateway's model-route form has a "Tiered / off-peak pricing (optional)" subsection; off-peak start/end render as <input type="time"> (converted to/from UTC minutes-of-day by GatewayAdmin.js), and the routes table shows tiered/off-peak badges when a route has either dimension configured.
• DeepSeek's real pricing is already the tier-1 shape, not a new dimension. DeepSeek's actual API (verified against api-docs.deepseek.com/quick_start/pricing) bills three flat per-1M rates - cache-hit input, cache-miss input, output - with no current context-length tier or off-peak window for the V4 models; that shape was already fully modeled by the pre-existing chat_cache_hit_per_m/chat_cache_miss_per_m/chat_output_per_m fields before this section's tier2/off-peak fields existed. routing.seed_default_deepseek_routes() (called once from database.migrate_ai_gateway_settings() at the end of init_db()) idempotently inserts four ready-made routes - deepseek-v4-flash ($0.0028/$0.14/$0.28 per 1M, 1M context), deepseek-v4-pro ($0.003625/$0.435/$0.87 per 1M, 1M context), and the two public molodetz aliases molodetz -> deepseek-v4-flash (flash rates) and molodetz-pro -> deepseek-v4-pro (pro rates) - only when that source_model row does not already exist, so a caller or Devii can request any name explicitly and get correctly-priced, decoupled from whatever the single global gateway_model default happens to be set to (switching that global setting between the two real models does NOT retroactively fix the flat Pricing config fields - the seeded routes are the model-agnostic, always-correct way to reference a specific priced model). The molodetz/molodetz-pro aliases are the public model names; GET /v1/models is served locally from these source_model rows (not proxied upstream), so it publishes exactly the models the gateway accepts. Neither seeded route sets the tier2/off-peak fields (DeepSeek does not use them today); an admin can add them later on the same row if DeepSeek (or any other provider routed here) introduces such pricing.

Third-party AI consent gate (one place, two owner classes)

GatewayService.handle() calls consent_denied(owner) immediately after resolve_owner, before the quota check. This is the only consent gate in the platform; no consumer implements one of its own.

The split it enforces is the whole point:

• owner kind user / admin -> the call carries that user's own content (Devii, AI correction, the AI modifier, a member calling /openai/v1/* with their own key). It requires a granted ai_third_party consent and answers 403 with CONSENT_REQUIRED_MESSAGE otherwise, plus an ai.consent.denied audit row.
• owner kind internal / key / anonymous -> platform processing (news import, the bot fleet, SEO metadata, issue enhancement). Never gated: it is not the user's content.

ai_third_party is never granted at signup. The existing ai_correction_enabled / ai_modifier_enabled user columns survive unchanged as preferences subordinate to consent: withdrawing consent turns those features off regardless of the flag, so ai_modifier_enabled's default of 1 is harmless. No existing preference was flipped and no consumer changed - the gate was simply added above them. USER_CONTENT_OWNER_KINDS is the single tuple defining the split; widen it only if a new owner kind genuinely carries a user's own content.