devplacepy/CLAUDE.md at 8d5d5f90beb46bbe19fd1b73efdbe996c446b60a

This file documents the AI gateway subsystem (devplacepy/services/openai_gateway/) - the single point of truth for AI calls, usage metering, and provider/model routing. Claude Code loads it automatically whenever a file under this directory is read or edited.

Overview

GatewayService (services/openai_gateway/) is an OpenAI-compatible LLM gateway (the ported openai5.py), mounted at /openai/v1/* (routers/openai_gateway.py, prefix /openai). It is a service that serves an HTTP endpoint rather than a loop.

The router is thin: it calls service_manager.get_service("openai").handle(request, subpath). POST /v1/chat/completions runs the full gateway (vision augment -> model override -> forward -> optional SSE re-emit via _fake_stream); POST /v1/embeddings forwards to the configured embeddings upstream (handle_embeddings, no vision/streaming); GET /v1/models is answered locally from the gateway_models chat routes (_models_response, publishing the public molodetz/molodetz-pro names), NOT proxied upstream; /v1/{path:path} is a transparent passthrough to the upstream base. The gateway always forwards NON-streaming upstream (payload["stream"] = False), then re-emits SSE itself when the client asked for stream; because stream_options is only valid alongside stream=true, handle_chat strips stream_options from the upstream payload (otherwise DeepSeek rejects it with stream_options should be set along with stream = true) and, when the client requested stream_options.include_usage, _fake_stream appends a final choices: [] usage chunk built from the upstream usage before [DONE]. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
main.py registers it and exempts /openai from the rate-limit middleware and the maintenance gate. No new dependency (httpx already required).
GatewayService is default_enabled=True so internal callers work out of the box.

Per-call cost/token response headers

Every gateway response (chat, embeddings, passthrough; success and error) carries X-Gateway-* headers describing that single call: Model, Backend, Prompt-Tokens, Completion-Tokens, Total-Tokens, Cache-Hit-Tokens, Cache-Miss-Tokens, Reasoning-Tokens, Cost-USD/Input-Cost-USD/Output-Cost-USD (dollars), Cost-Native (1 if the upstream returned a native cost), Tokens-Per-Second, Upstream-Latency-Ms, and Context-Window/Context-Utilization when the model's window is known, plus the timing headers X-Gateway-Upstream-Latency-Ms/X-Gateway-Total-Latency-Ms. GatewayUsageLedger.record(...) RETURNS the computed ledger row (or None on failure); each handler's finalize closure maps it through usage.usage_response_headers(row) and attaches it to the returned Response (resp_headers). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads X-Gateway-Cost-USD/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root CLAUDE.md).

App-reference header (`X-App-Reference`)

Callers may send an optional X-App-Reference header to tag gateway calls by application. The value is validated and stored in the app_reference column of gateway_usage_ledger, surfaced in analytics and admin reporting.

Validation (service._validate_app_reference): trimmed whitespace, then matched against ^[a-zA-Z0-9_.-]{1,30}$. Any value failing validation (empty, >30 chars, contains spaces or @// etc.) silently falls back to "default".
Header name: X-App-Reference.
All internal callers (news, bots, Devii, correction, jobs, deepsearch, dbapi, gitea) should pass a devplace-<component>-v-<major>-<minor>-<patch> value, e.g. devplace-devii-v-1-0-0, devplace-news-v-1-0-0.
The column is indexed (CREATE INDEX IF NOT EXISTS) for fast per-app queries.

Per-worker runtime

Serves in every worker. Config/enabled come from site_settings (via the service's get_config()/is_enabled()), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (GatewayRuntime: httpx.AsyncClient pool + asyncio.Semaphore sized to gateway_instances, plus counters and a VisionCache) is created lazily on first request and rebuilt when instances/timeout/cache size change. gateway_instances is the scaling knob (concurrency per worker); process scaling is uvicorn workers.

Auth

authorize() reuses get_current_user (cookie/X-API-KEY/Bearer/Basic). Allowed if gateway_require_auth is off, the presented X-API-KEY/Bearer equals the static gateway_access_key or the auto-generated gateway_internal_key, or the resolved user is an admin (gateway_allow_admins) or any user (gateway_allow_users).

Per-user attribution: the gateway also accepts a real user's api_key (Bearer / X-API-KEY / session), gated by gateway_allow_users (default on); resolve_owner() records (owner_kind, owner_id) per call, so usage is attributed and limitable per user. With gateway_allow_users off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own api_key (set as the session's ai_key in build_settings(..., owner_kind="user")), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via build_user_usage() / GET /admin/users/{uid}/ai-usage. Guests keep the internal key.

Config fields

All config is config_fields (upstream url/model/key, force-model, the Prompt group's gateway_system_preamble, timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (gateway_embed_enabled/_url/_model/_key), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via collect_metrics.

System-message composition (date awareness + operator preamble)

system_message.py (apply_system_directives) runs in handle_chat only (NOT handle_embeddings/handle_passthrough), after vision augmentation and before the upstream payload is assembled, so it reaches every chat consumer (news, bots, Devii guests, per-user direct API calls). It composes ONE system message in a fixed, cache-friendly order: operator preamble (gateway_system_preamble) -> date line (Current date: DD/MM/YYYY) -> the client's original system content.

Two behaviors:

Date awareness (EU, no time). The current date is injected only when the composed text (preamble + client system content) contains NO date in any common format. Detection lives in contains_date via the _DATE_PATTERNS regex set: YYYY-MM-DD (ISO), D/M/YYYY and DD/MM/YYYY (also matches MM/DD/YYYY), DD-MM-YYYY, DD.MM.YYYY, and textual months (14 June 2026, June 14, 2026, 14 Jun 2026, with abbreviations and optional ordinal/comma). The date is datetime.now().strftime("%d/%m/%Y") - date only, never time: omitting the time keeps the system message byte-stable for the whole day so upstream prompt/token caching stays effective.
Operator preamble. gateway_system_preamble (Prompt group, type="text" textarea, default empty) is prepended ahead of the client's system content on every chat call so all calls carry operator preferences. When the client sent NO system message, the composed content (preamble and/or date) becomes a brand-new leading system message; when the client DID send one, its content is replaced in place (preamble + date + original text). A blank/whitespace-only preamble with an already-dated request is a no-op (no empty system message is created). The date detection runs over the preamble too, so an operator preamble that already states a date suppresses the injected date line.

Embeddings/passthrough are untouched by this composition step.

Usage, cost, latency, and reliability tracking

The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:

usage.py - GatewayUsageLedger writes to gateway_usage_ledger (one row per call, success and failure) and samples gateway_concurrency_samples each 30s tick. normalize_usage handles BOTH upstream shapes: DeepSeek (prompt_cache_hit_tokens/prompt_cache_miss_tokens, completion_tokens_details.reasoning_tokens) and OpenRouter (prompt_tokens_details.cached_tokens, native cost, cost_details). compute_cost PREFERS the upstream native cost (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share input_cost / (input_cost + output_cost) from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). record() is wrapped in try/except so a tracking failure never breaks the proxy.
gateway.py - _send is the gated/timed/retried path: a CircuitBreaker (reliability.py) short-circuits when open; retry_send retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the queue_wait_ms; an httpx per-request trace extension captures connect_ms (0 on pooled reuse). handle_chat/handle_passthrough record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from VisionAugmenter._describe_one.
Owner attribution - GatewayService.resolve_owner maps the caller to (owner_kind, owner_id): internal:devii (internal key), key:access (static key), user:<uid>/admin:<uid> (session or API key), else anonymous.
analytics.py - build_analytics(hours, top_n, pricing) pulls the bounded window once (clamped to MAX_WINDOW_HOURS=168) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via reliability.percentile), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. summary_metrics() runs cheap SQL aggregates for the live service panel. TTFT/inter-token are returned null (gateway forwards non-streaming).
Admin - /admin/ai-usage (HTML, templates/ai_usage.html + static/js/AiUsageMonitor.js) and /admin/ai-usage/data?hours=&top_n= (JSON) in routers/admin/ package, admin-gated. Schema GatewayUsageOut. Devii action ai_usage and docs admin-ai-usage expose the same endpoint. The ai_usage and site_analytics Devii catalog tools are both requires_admin=True.
Per-user usage - analytics.build_user_usage(owner_id, hours=24, pricing) filters the ledger by owner_id (a uid matches both user: and admin: rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at GET /admin/users/{uid}/ai-usage (schema UserAiUsageOut, docs admin-user-ai-usage) and rendered on the profile page by static/js/UserAiUsage.js (the [data-ai-usage] card, gated {% if is_admin(user) %} in templates/profile.html). The 24h window is a sound projection basis (unlike a startup burst), so cost_24h * 30 is honest.
Profile visibility split - routers/profile/ package _ai_quota(uid, include_cost=False) builds the profile quota bar. The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal: spent_usd/used_pct/requests/last_used come from analytics.user_spend_24h(uid), the SAME gateway_usage_ledger source (summed by owner_id, matching both user: and admin: rows) that feeds the stat grid's cost.window_usd. So the server-rendered spent_usd is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. turns is still the devii_usage_ledger turns_24h count (Devii conversation turns) and is labeled "Devii turns" in the template precisely because it is a different metric from the grid's total API requests (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why 0 Devii turns can sit beside 18 API requests). The limit/turns still come from the Devii service (daily_limit_for/turns_24h); an unlimited limit (<= 0, the admin default) sets unlimited=True so the template renders "no cap" / "No daily spend cap" instead of a misleading / $0.00. This is display-only: the Devii daily-cap ENFORCEMENT and /devii/usage still read the Devii-scoped devii_usage_ledger.spent_24h (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: admin sees the full gateway stats card plus a server-rendered quota bar (with spend/limit/Devii turns); the owner (non-admin, is_self) sees a quota-only card showing just the percentage; everyone else sees nothing AI-related. _ai_quota is fail-safe (returns None on any error or when the Devii service is unregistered) and the result rides on ProfileOut.ai_quota. Dollar figures (spent_usd/limit_usd) are included only when include_cost=viewer_is_admin - never for a non-admin owner. This matters because the route serves the same context as JSON via respond(..., model=ProfileOut) and ProfileOut.ai_quota is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with Accept: application/json; the server-side omission is the real control. Non-admins (HTML and JSON) get only used_pct/turns/requests.
Retention - run_once prunes both tables past gateway_usage_retention_hours (default 720 = 30 days).

Financial data is admin-only everywhere. Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, ai_correction/ai_modifier usage displays, and the Devii cost tools.

Quota rules (`quota.py`, admin `/admin/gateway` "Quota rules" section)

This caps /openai/v1/* itself, independent of Devii's own daily cap. Devii's devii_user_daily_usd/devii_guest_daily_usd/devii_admin_daily_usd (documented in devplacepy/services/devii/CLAUDE.md) only gate turns that go through Devii. A caller hitting the gateway directly with their own api_key bypasses that entirely - quota.py is the root-level enforcement that closes this, checked in GatewayService.handle() right after owner/app_reference resolution and before every billed dispatch (chat, embeddings, images, passthrough; GET /v1/models is exempt, it makes no upstream call).

Two layers, same shape as provider/model routing above. Layer A is five flat config_fields on GatewayService (gateway_default_user_daily_usd $1.00, gateway_default_admin_daily_usd $0/unlimited, gateway_default_guest_daily_usd $0.05, gateway_default_internal_daily_usd $0/unlimited, gateway_default_key_daily_usd $0/unlimited - group Quota) applied per specific caller (owner_id) when no rule matches; internal/key default unlimited so shipping this never starts blocking DevPlace's own news/bots/Devii-guest/correction traffic on the internal key. Layer B is the gateway_quota_rules table (ensure_tables(), called from init_db() alongside routing.ensure_tables(); hard CRUD, not in SOFT_DELETE_TABLES, cross-worker cache-invalidated under the "gateway_quota" name): each row scopes by any combination of owner_kind (internal/key/user/admin/anonymous - DevPlace's only "roles" here), a specific owner_id, and app_reference (the X-App-Reference label), each nullable = wildcard; a QuotaRuleIn Pydantic validator rejects a rule with all three blank (that belongs in Layer A). quota.resolve(owner_kind, owner_id, app_reference, cfg) gathers every active rule whose non-null dimensions all equal the request, picks the one with the most non-null dimensions (ties broken toward the smaller limit, unlimited 0 never wins a tie against a finite cap), and returns (limit_usd, scope, rule) where scope is the exact (owner_kind, owner_id, app_reference) triple - each possibly None - that spend must be summed over. Layer A is internally just the maximally-specific implicit scope (owner_kind, owner_id, None), so one code path (quota.spent_24h(*scope), a plain SUM(cost_usd) over gateway_usage_ledger filtered by whichever scope dimensions are non-null) serves both layers.

A wildcard dimension means a shared pool, by design. A rule scoped only by app_reference caps that app's combined spend across every caller using it; a rule scoped only by owner_kind caps that whole role's combined spend. Pin owner_id to get a true per-caller cap (the Layer A default's own behavior). anonymous/internal/key owner_ids are already fixed constants ("anonymous"/"devii"/"access", from resolve_owner()), not per-caller identities, so any cap on those kinds is inherently pooled - there is no per-guest identity at this layer (unlike Devii's own guest-cookie-scoped ledger).

No lock, no hold, bounded overshoot by design - this is deliberate, not an oversight. Cost is only known after the upstream call returns, so a true atomic pre-authorization would need a reserve-then-reconcile ("hold") mechanism, and a bug in releasing a hold is exactly the kind of thing that gets a caller stuck forever. Instead this mirrors Devii's own already-shipped mechanism exactly: read the 24h sum, compare, raise HTTPException(429, ...) if already at/over - a single SELECT and a conditional raise, nothing held, nothing to leak, structurally impossible to deadlock. The tradeoff is a small, bounded overshoot (at most a few concurrent in-flight calls' worth of cost past the cap before the next request sees the updated sum and blocks) - acceptable and industry-standard for a cost whose exact size isn't known until the call finishes, and it is the property actually being enforced: once tripped, every subsequent separate request stays blocked until the 24h window rolls off or an admin adjusts the rule.

429 body never carries a dollar figure, admin or not ({"detail": "AI gateway daily quota exceeded"}) - mirrors Devii's own over-limit WS message, which likewise never states a number. The admin-only services log line and the ai.quota.exceeded audit row (GatewayService._audit_quota_exceeded, reusing usage.audit_actor_for) do carry the spend/limit/matched-rule-uid, since those are admin-only surfaces.

CRUD. Admin JSON at /admin/gateway/quota-rules (routers/admin/gateway_configs.py, list returns each rule's live spent_24h_usd plus the Layer A defaults for context), audited gateway.quota_rule.update/gateway.quota_rule.delete (category ai, both already in events.md), rendered in the Quota rules section of /admin/gateway (GatewayAdmin.js, mirrors the providers/models CRUD tables). Devii tools gateway_quota_rules/gateway_quota_rule_set/gateway_quota_rule_delete (requires_admin=True, delete is CONFIRM_REQUIRED) proxy the same endpoints via handler="http", same as the provider/model tools. CLI: devplace gateway quota list|set|delete.

Image generation

POST /openai/v1/images/generations exposes an OpenAI-compatible image-generation endpoint. Clients send the generic model molodetz-img-small (config.INTERNAL_IMAGE_MODEL), which handle_images remaps to gateway_image_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty or molodetz-img). It defaults to OpenRouter's black-forest-labs/flux-1.1-pro at https://openrouter.ai/api/v1/images/generations (config.IMAGE_*_DEFAULT, $0.04 per image fallback). handle_images mirrors handle_embeddings: build the payload, forward via _send, and record one ledger row. The config fields are the Images group (gateway_image_enabled default on, gateway_image_url, gateway_image_model, gateway_image_key) plus the Pricing-group gateway_image_price_per_call. effective_config() falls the image key back to gateway_api_key then OPENROUTER_API_KEY. Usage is recorded with backend="image"; usage.compute_cost adds an image branch (flat per-call, native OpenRouter cost still preferred via extract_image_usage). routing.image_overlay resolves per-route provider/url/key and uses price_input_per_m as the per-image price. routing.seed_default_image_routes() (from migrate_ai_gateway_settings) idempotently seeds molodetz-img-small -> Flux on the openrouter provider when OPENROUTER_API_KEY is set. When gateway_image_enabled is off the endpoint returns 503 with no ledger row.

Embeddings

POST /openai/v1/embeddings exposes an OpenAI-compatible text-embeddings model. Clients send the generic model molodetz~embed (config.INTERNAL_EMBED_MODEL), which handle_embeddings remaps to gateway_embed_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty). It defaults to OpenRouter's qwen/qwen3-embedding-8b at https://openrouter.ai/api/v1/embeddings (config.EMBED_*_DEFAULT, $0.01 per 1M input tokens). handle_embeddings mirrors handle_chat but is simpler: no vision augmentation and no streaming - build the payload, forward via _send, and record one ledger row through the same finalize(...) closure. The config fields are the Embeddings group (gateway_embed_enabled default on, gateway_embed_url, gateway_embed_model, gateway_embed_key) plus the Pricing-group gateway_embed_price_input_per_m. effective_config() falls the embed key back to gateway_vision_key then OPENROUTER_API_KEY (NOT gateway_api_key: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with backend="embed"; usage.compute_cost adds an embed branch (input-only, completion always 0, native OpenRouter cost still preferred) and Pricing gained embed_input_per_m. analytics.py groups by backend generically, so embed rows roll up automatically; caching_savings counts only non-native chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When gateway_embed_enabled is off the endpoint returns 503 with no ledger row.

Single point of truth for AI

The gateway is the only place that holds real provider URLs/models/keys. Every other AI consumer (news, bots, Devii guests) points at it by default and never touches a provider key:

Defaults live in config.py: INTERNAL_GATEWAY_URL (http://localhost:10500/openai/v1/chat/completions, override with DEVPLACE_INTERNAL_BASE_URL) and INTERNAL_MODEL (molodetz). news_ai_url, bot_api_url, devii_ai_url default to INTERNAL_GATEWAY_URL; their model defaults to molodetz.
Each consumer's key falls back to database.internal_gateway_key() (reads the gateway_internal_key setting) when its own key field/env is unset - the provider-key fallbacks (DEEPSEEK_API_KEY/OPENROUTER_API_KEY) were removed from news and bots.
gateway_force_model (default on) and a molodetz/empty alias in handle_chat make the upstream always receive gateway_model, so molodetz is a stable generic alias.
database.migrate_ai_gateway_settings() (called at the end of init_db(), under the startup init_lock): generates gateway_internal_key (uuid4) if missing; migrates DEEPSEEK_API_KEY/OPENROUTER_API_KEY env into gateway_api_key/gateway_vision_key when the db value is empty; and rewrites any consumer AI URL still equal to the old openai.app.molodetz.nl default to the gateway, plus bot_model deepseek-chat -> molodetz (only uncustomized values).
The gateway's gateway_api_key/gateway_vision_key/gateway_internal_key fields are non-secret so the admin services page shows the value actually in use, editable.
Bot LLM calls are synchronous urllib but already run via asyncio.to_thread (bot/bot.py), so the local round-trip never blocks the event loop.
Real provider keys/URLs/models live ONLY in the gateway. The gateway is default_enabled=True.

Provider and model routing (`routing.py`, admin `/admin/gateway`)

Layered ON TOP of the single-provider service config above, which stays THE implicit default provider (env migration, internal key, molodetz/molodetz~embed, vision, embeddings, ledger all unchanged).

Storage. Two dataset tables, ensured in init_db via routing.ensure_tables, cross-worker cache-invalidated under the "gateway_routing" cache-version name (module-level provider_store/model_store over a shared _ROUTING_CACHE; writes bump_cache_version and clear the cache). They are admin config, NOT in SOFT_DELETE_TABLES - hard CRUD, mirroring site_settings:

gateway_providers - named upstreams: name + base_url (chat-completions URL) + api_key + is_active; the embeddings URL is derived by swapping /chat/completions -> /embeddings.
gateway_models - source->target routes: source_model (unique, what clients request) -> provider (blank = default) + target_model + kind (chat|embed|image) + optional vision_provider/vision_model for the text+vision merge (when set, image content is described by that vision model before forwarding) + context_window + its own economy (price_cache_hit_per_m/price_cache_miss_per_m/price_output_per_m/price_input_per_m, USD per 1M tokens for chat/embed/vision; for image routes price_input_per_m is a flat USD per image, plus the tiered/off-peak fields below) + is_active.

Resolution is a per-request overlay, not a fork. At request time routing.chat_overlay(requested, cfg) / routing.embed_overlay(requested, cfg) / routing.image_overlay(requested, cfg) resolve an active route by the requested model name and return a per-request OVERLAY dict of gateway_* cfg keys (gateway_force_model+gateway_model=target, gateway_upstream_url/gateway_api_key from the provider, the price keys, an augmented gateway_model_context_map, and vision overlay keys); handle_chat/handle_embeddings merge it onto the base cfg (cfg = {**cfg, **overlay}) BEFORE everything else, so the existing model-selection / pricing_from_cfg / parse_context_map / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. _ensure (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.

No matching route = None overlay = model fallback. When no route matches, the overlay returns None and the handler's model selection logic falls back to the default configured model (gateway_model for chat, gateway_embed_model for embeddings, gateway_image_model for images). The unknown model name is discarded, not forwarded upstream. This means an unknown or misspelled model name never causes a 4xx from the upstream - it is gracefully downgraded to the default. The force_model guard and the built-in molodetz/molodetz~embed/molodetz-img-small aliases are still respected before the route check: force_model or empty/named-alias model -> default directly; known model -> route resolution; unknown model with no route -> default fallback with a log message.

CRUD. Admin JSON at /admin/gateway/{providers,models} (routers/admin/gateway_configs.py, require_admin, Pydantic ProviderIn/ModelRouteIn validation, accepts both JSON from static/js/GatewayAdmin.js and form from Devii), audited under gateway.provider.*/gateway.model.* (category ai), included in the admin package with the page at /admin/gateway (templates/admin_gateway.html, sidebar link, admin_section="gateway").

Devii tools: gateway_providers/gateway_models (read) and gateway_provider_set/gateway_provider_delete/gateway_model_set/gateway_model_delete (admin; the two deletes are CONFIRM_REQUIRED).

When adding a routed value, overlay it as the matching gateway_* cfg key so the runtime needs no new branch.

Tests: tests/unit/services/openai_gateway/routing.py (overlay/economy/kind isolation), tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream (end-to-end through handle_chat), and tests/api/admin/gateway/ (admin CRUD + validation + role gating).

Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)

Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME gateway_models route row, so it stays fully provider-agnostic and opt-in per route.

Per-route fields (each optional/zero by default = feature off, so an unconfigured route is byte-identical to before): context_tier_threshold_tokens (0 disables tiering; when the request's input token count exceeds it, price_*_per_m_tier2 rates apply instead of the tier-1 rates above) and the four nullable price_cache_hit_per_m_tier2/price_cache_miss_per_m_tier2/price_output_per_m_tier2/price_input_per_m_tier2 (a component left None keeps its tier-1 rate even above threshold - so a provider that only re-prices input above a size threshold, keeping output flat, needs just one tier2 field set), plus off_peak_start_minute/off_peak_end_minute (nullable, UTC minutes-since-midnight, both-or-neither enforced by ModelRouteIn's model validator; a window where start > end wraps past midnight, e.g. DeepSeek's old V3/R1-era 16:30-00:30 UTC window) and off_peak_discount_pct (0-100, multiplies whichever tier rate is active). The same fields double for vision (price_input/output_per_m_tier2) and embed (price_input_per_m_tier2) exactly like their tier-1 counterparts already do.
Overlay + computation. chat_overlay/embed_overlay propagate all of these into the per-request cfg dict under gateway_* keys exactly like the existing price fields (section above); usage.pricing_from_cfg reads them generically (defaulting to None/0/disabled when absent), so Layer A (the single global flat Pricing config fields) never gains these dimensions - only a gateway_models route can enable them, preserving the "no matching route = byte-identical legacy behavior" guarantee. usage.compute_cost selects tier1 vs tier2 per rate component (_tiered_rate) based on whether norm["prompt"] (billable input tokens) exceeds the threshold, then applies the off-peak discount (_effective_rate/_off_peak_active, UTC wraparound-aware) to whichever rate was selected. The response header format and compute_cost's return shape (total, input_cost, output_cost, native) are unchanged - this is purely an internal rate-selection step before the existing input/output split math runs; a native upstream cost (OpenRouter) still overrides the modeled total exactly as before.
Migration. New gateway_models columns are added via has_column/create_column_by_example in routing.ensure_tables() (the CREATE TABLE IF NOT EXISTS DDL string alone would never reach a pre-existing table - see the database/CLAUDE.md column-ensure idiom).
Admin UI. /admin/gateway's model-route form has a "Tiered / off-peak pricing (optional)" subsection; off-peak start/end render as <input type="time"> (converted to/from UTC minutes-of-day by GatewayAdmin.js), and the routes table shows tiered/off-peak badges when a route has either dimension configured.
DeepSeek's real pricing is already the tier-1 shape, not a new dimension. DeepSeek's actual API (verified against api-docs.deepseek.com/quick_start/pricing) bills three flat per-1M rates - cache-hit input, cache-miss input, output - with no current context-length tier or off-peak window for the V4 models; that shape was already fully modeled by the pre-existing chat_cache_hit_per_m/chat_cache_miss_per_m/chat_output_per_m fields before this section's tier2/off-peak fields existed. routing.seed_default_deepseek_routes() (called once from database.migrate_ai_gateway_settings() at the end of init_db()) idempotently inserts four ready-made routes - deepseek-v4-flash ($0.0028/$0.14/$0.28 per 1M, 1M context), deepseek-v4-pro ($0.003625/$0.435/$0.87 per 1M, 1M context), and the two public molodetz aliases molodetz -> deepseek-v4-flash (flash rates) and molodetz-pro -> deepseek-v4-pro (pro rates) - only when that source_model row does not already exist, so a caller or Devii can request any name explicitly and get correctly-priced, decoupled from whatever the single global gateway_model default happens to be set to (switching that global setting between the two real models does NOT retroactively fix the flat Pricing config fields - the seeded routes are the model-agnostic, always-correct way to reference a specific priced model). The molodetz/molodetz-pro aliases are the public model names; GET /v1/models is served locally from these source_model rows (not proxied upstream), so it publishes exactly the models the gateway accepts. Neither seeded route sets the tier2/off-peak fields (DeepSeek does not use them today); an admin can add them later on the same row if DeepSeek (or any other provider routed here) introduces such pricing.