devplacepy/CLAUDE.md at 0b13729c0481b772676dd64da31182b1d5f656ab

This file documents the AI gateway subsystem (devplacepy/services/openai_gateway/) - the single point of truth for AI calls, usage metering, and provider/model routing. Claude Code loads it automatically whenever a file under this directory is read or edited.

Overview

GatewayService (services/openai_gateway/) is an OpenAI-compatible LLM gateway (the ported openai5.py), mounted at /openai/v1/* (routers/openai_gateway.py, prefix /openai). It is a service that serves an HTTP endpoint rather than a loop.

The router is thin: it calls service_manager.get_service("openai").handle(request, subpath). POST /v1/chat/completions runs the full gateway (vision augment -> model override -> forward -> optional SSE re-emit via _fake_stream); POST /v1/embeddings forwards to the configured embeddings upstream (handle_embeddings, no vision/streaming); /v1/{path:path} is a transparent passthrough to the upstream base. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
main.py registers it and exempts /openai from the rate-limit middleware and the maintenance gate. No new dependency (httpx already required).
GatewayService is default_enabled=True so internal callers work out of the box.

Per-call cost/token response headers

Every gateway response (chat, embeddings, passthrough; success and error) carries X-Gateway-* headers describing that single call: Model, Backend, Prompt-Tokens, Completion-Tokens, Total-Tokens, Cache-Hit-Tokens, Cache-Miss-Tokens, Reasoning-Tokens, Cost-USD/Input-Cost-USD/Output-Cost-USD (dollars), Cost-Native (1 if the upstream returned a native cost), Tokens-Per-Second, Upstream-Latency-Ms, and Context-Window/Context-Utilization when the model's window is known, plus the timing headers X-Gateway-Upstream-Latency-Ms/X-Gateway-Total-Latency-Ms. GatewayUsageLedger.record(...) RETURNS the computed ledger row (or None on failure); each handler's finalize closure maps it through usage.usage_response_headers(row) and attaches it to the returned Response (resp_headers). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads X-Gateway-Cost-USD/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root CLAUDE.md).

Per-worker runtime

Serves in every worker. Config/enabled come from site_settings (via the service's get_config()/is_enabled()), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (GatewayRuntime: httpx.AsyncClient pool + asyncio.Semaphore sized to gateway_instances, plus counters and a VisionCache) is created lazily on first request and rebuilt when instances/timeout/cache size change. gateway_instances is the scaling knob (concurrency per worker); process scaling is uvicorn workers.

Auth

authorize() reuses get_current_user (cookie/X-API-KEY/Bearer/Basic). Allowed if gateway_require_auth is off, the presented X-API-KEY/Bearer equals the static gateway_access_key or the auto-generated gateway_internal_key, or the resolved user is an admin (gateway_allow_admins) or any user (gateway_allow_users).

Per-user attribution: the gateway also accepts a real user's api_key (Bearer / X-API-KEY / session), gated by gateway_allow_users (default on); resolve_owner() records (owner_kind, owner_id) per call, so usage is attributed and limitable per user. With gateway_allow_users off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own api_key (set as the session's ai_key in build_settings(..., owner_kind="user")), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via build_user_usage() / GET /admin/users/{uid}/ai-usage. Guests keep the internal key.

Config fields

All config is config_fields (upstream url/model/key, force-model, the Prompt group's gateway_system_preamble, timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (gateway_embed_enabled/_url/_model/_key), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via collect_metrics.

System-message composition (date awareness + operator preamble)

system_message.py (apply_system_directives) runs in handle_chat only (NOT handle_embeddings/handle_passthrough), after vision augmentation and before the upstream payload is assembled, so it reaches every chat consumer (news, bots, Devii guests, per-user direct API calls). It composes ONE system message in a fixed, cache-friendly order: operator preamble (gateway_system_preamble) -> date line (Current date: DD/MM/YYYY) -> the client's original system content.

Two behaviors:

Date awareness (EU, no time). The current date is injected only when the composed text (preamble + client system content) contains NO date in any common format. Detection lives in contains_date via the _DATE_PATTERNS regex set: YYYY-MM-DD (ISO), D/M/YYYY and DD/MM/YYYY (also matches MM/DD/YYYY), DD-MM-YYYY, DD.MM.YYYY, and textual months (14 June 2026, June 14, 2026, 14 Jun 2026, with abbreviations and optional ordinal/comma). The date is datetime.now().strftime("%d/%m/%Y") - date only, never time: omitting the time keeps the system message byte-stable for the whole day so upstream prompt/token caching stays effective.
Operator preamble. gateway_system_preamble (Prompt group, type="text" textarea, default empty) is prepended ahead of the client's system content on every chat call so all calls carry operator preferences. When the client sent NO system message, the composed content (preamble and/or date) becomes a brand-new leading system message; when the client DID send one, its content is replaced in place (preamble + date + original text). A blank/whitespace-only preamble with an already-dated request is a no-op (no empty system message is created). The date detection runs over the preamble too, so an operator preamble that already states a date suppresses the injected date line.

Embeddings/passthrough are untouched by this composition step.

Usage, cost, latency, and reliability tracking

The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:

usage.py - GatewayUsageLedger writes to gateway_usage_ledger (one row per call, success and failure) and samples gateway_concurrency_samples each 30s tick. normalize_usage handles BOTH upstream shapes: DeepSeek (prompt_cache_hit_tokens/prompt_cache_miss_tokens, completion_tokens_details.reasoning_tokens) and OpenRouter (prompt_tokens_details.cached_tokens, native cost, cost_details). compute_cost PREFERS the upstream native cost (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share input_cost / (input_cost + output_cost) from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). record() is wrapped in try/except so a tracking failure never breaks the proxy.
gateway.py - _send is the gated/timed/retried path: a CircuitBreaker (reliability.py) short-circuits when open; retry_send retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the queue_wait_ms; an httpx per-request trace extension captures connect_ms (0 on pooled reuse). handle_chat/handle_passthrough record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from VisionAugmenter._describe_one.
Owner attribution - GatewayService.resolve_owner maps the caller to (owner_kind, owner_id): internal:devii (internal key), key:access (static key), user:<uid>/admin:<uid> (session or API key), else anonymous.
analytics.py - build_analytics(hours, top_n, pricing) pulls the bounded window once (clamped to MAX_WINDOW_HOURS=168) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via reliability.percentile), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. summary_metrics() runs cheap SQL aggregates for the live service panel. TTFT/inter-token are returned null (gateway forwards non-streaming).
Admin - /admin/ai-usage (HTML, templates/ai_usage.html + static/js/AiUsageMonitor.js) and /admin/ai-usage/data?hours=&top_n= (JSON) in routers/admin/ package, admin-gated. Schema GatewayUsageOut. Devii action ai_usage and docs admin-ai-usage expose the same endpoint. The ai_usage and site_analytics Devii catalog tools are both requires_admin=True.
Per-user usage - analytics.build_user_usage(owner_id, hours=24, pricing) filters the ledger by owner_id (a uid matches both user: and admin: rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at GET /admin/users/{uid}/ai-usage (schema UserAiUsageOut, docs admin-user-ai-usage) and rendered on the profile page by static/js/UserAiUsage.js (the [data-ai-usage] card, gated {% if is_admin(user) %} in templates/profile.html). The 24h window is a sound projection basis (unlike a startup burst), so cost_24h * 30 is honest.
Profile visibility split - routers/profile/ package _ai_quota(uid, include_cost=False) builds the profile quota bar. The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal: spent_usd/used_pct/requests/last_used come from analytics.user_spend_24h(uid), the SAME gateway_usage_ledger source (summed by owner_id, matching both user: and admin: rows) that feeds the stat grid's cost.window_usd. So the server-rendered spent_usd is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. turns is still the devii_usage_ledger turns_24h count (Devii conversation turns) and is labeled "Devii turns" in the template precisely because it is a different metric from the grid's total API requests (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why 0 Devii turns can sit beside 18 API requests). The limit/turns still come from the Devii service (daily_limit_for/turns_24h); an unlimited limit (<= 0, the admin default) sets unlimited=True so the template renders "no cap" / "No daily spend cap" instead of a misleading / $0.00. This is display-only: the Devii daily-cap ENFORCEMENT and /devii/usage still read the Devii-scoped devii_usage_ledger.spent_24h (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: admin sees the full gateway stats card plus a server-rendered quota bar (with spend/limit/Devii turns); the owner (non-admin, is_self) sees a quota-only card showing just the percentage; everyone else sees nothing AI-related. _ai_quota is fail-safe (returns None on any error or when the Devii service is unregistered) and the result rides on ProfileOut.ai_quota. Dollar figures (spent_usd/limit_usd) are included only when include_cost=viewer_is_admin - never for a non-admin owner. This matters because the route serves the same context as JSON via respond(..., model=ProfileOut) and ProfileOut.ai_quota is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with Accept: application/json; the server-side omission is the real control. Non-admins (HTML and JSON) get only used_pct/turns/requests.
Retention - run_once prunes both tables past gateway_usage_retention_hours (default 720 = 30 days).

Financial data is admin-only everywhere. Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, ai_correction/ai_modifier usage displays, and the Devii cost tools.

Image generation

POST /openai/v1/images/generations exposes an OpenAI-compatible image-generation endpoint. Clients send the generic model molodetz-img-small (config.INTERNAL_IMAGE_MODEL), which handle_images remaps to gateway_image_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty or molodetz-img). It defaults to OpenRouter's black-forest-labs/flux-1.1-pro at https://openrouter.ai/api/v1/images/generations (config.IMAGE_*_DEFAULT, $0.04 per image fallback). handle_images mirrors handle_embeddings: build the payload, forward via _send, and record one ledger row. The config fields are the Images group (gateway_image_enabled default on, gateway_image_url, gateway_image_model, gateway_image_key) plus the Pricing-group gateway_image_price_per_call. effective_config() falls the image key back to gateway_api_key then OPENROUTER_API_KEY. Usage is recorded with backend="image"; usage.compute_cost adds an image branch (flat per-call, native OpenRouter cost still preferred via extract_image_usage). routing.image_overlay resolves per-route provider/url/key and uses price_input_per_m as the per-image price. routing.seed_default_image_routes() (from migrate_ai_gateway_settings) idempotently seeds molodetz-img-small -> Flux on the openrouter provider when OPENROUTER_API_KEY is set. When gateway_image_enabled is off the endpoint returns 503 with no ledger row.

Embeddings

POST /openai/v1/embeddings exposes an OpenAI-compatible text-embeddings model. Clients send the generic model molodetz~embed (config.INTERNAL_EMBED_MODEL), which handle_embeddings remaps to gateway_embed_model exactly like chat remaps molodetz -> gateway_model (also remapped when gateway_force_model is on or the model is empty). It defaults to OpenRouter's qwen/qwen3-embedding-8b at https://openrouter.ai/api/v1/embeddings (config.EMBED_*_DEFAULT, $0.01 per 1M input tokens). handle_embeddings mirrors handle_chat but is simpler: no vision augmentation and no streaming - build the payload, forward via _send, and record one ledger row through the same finalize(...) closure. The config fields are the Embeddings group (gateway_embed_enabled default on, gateway_embed_url, gateway_embed_model, gateway_embed_key) plus the Pricing-group gateway_embed_price_input_per_m. effective_config() falls the embed key back to gateway_vision_key then OPENROUTER_API_KEY (NOT gateway_api_key: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with backend="embed"; usage.compute_cost adds an embed branch (input-only, completion always 0, native OpenRouter cost still preferred) and Pricing gained embed_input_per_m. analytics.py groups by backend generically, so embed rows roll up automatically; caching_savings counts only non-native chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When gateway_embed_enabled is off the endpoint returns 503 with no ledger row.

Single point of truth for AI

The gateway is the only place that holds real provider URLs/models/keys. Every other AI consumer (news, bots, Devii guests) points at it by default and never touches a provider key:

Defaults live in config.py: INTERNAL_GATEWAY_URL (http://localhost:10500/openai/v1/chat/completions, override with DEVPLACE_INTERNAL_BASE_URL) and INTERNAL_MODEL (molodetz). news_ai_url, bot_api_url, devii_ai_url default to INTERNAL_GATEWAY_URL; their model defaults to molodetz.
Each consumer's key falls back to database.internal_gateway_key() (reads the gateway_internal_key setting) when its own key field/env is unset - the provider-key fallbacks (DEEPSEEK_API_KEY/OPENROUTER_API_KEY) were removed from news and bots.
gateway_force_model (default on) and a molodetz/empty alias in handle_chat make the upstream always receive gateway_model, so molodetz is a stable generic alias.
database.migrate_ai_gateway_settings() (called at the end of init_db(), under the startup init_lock): generates gateway_internal_key (uuid4) if missing; migrates DEEPSEEK_API_KEY/OPENROUTER_API_KEY env into gateway_api_key/gateway_vision_key when the db value is empty; and rewrites any consumer AI URL still equal to the old openai.app.molodetz.nl default to the gateway, plus bot_model deepseek-chat -> molodetz (only uncustomized values).
The gateway's gateway_api_key/gateway_vision_key/gateway_internal_key fields are non-secret so the admin services page shows the value actually in use, editable.
Bot LLM calls are synchronous urllib but already run via asyncio.to_thread (bot/bot.py), so the local round-trip never blocks the event loop.
Real provider keys/URLs/models live ONLY in the gateway. The gateway is default_enabled=True.

Provider and model routing (`routing.py`, admin `/admin/gateway`)

Layered ON TOP of the single-provider service config above, which stays THE implicit default provider (env migration, internal key, molodetz/molodetz~embed, vision, embeddings, ledger all unchanged).

Storage. Two dataset tables, ensured in init_db via routing.ensure_tables, cross-worker cache-invalidated under the "gateway_routing" cache-version name (module-level provider_store/model_store over a shared _ROUTING_CACHE; writes bump_cache_version and clear the cache). They are admin config, NOT in SOFT_DELETE_TABLES - hard CRUD, mirroring site_settings:

gateway_providers - named upstreams: name + base_url (chat-completions URL) + api_key + is_active; the embeddings URL is derived by swapping /chat/completions -> /embeddings.
gateway_models - source->target routes: source_model (unique, what clients request) -> provider (blank = default) + target_model + kind (chat|embed|image) + optional vision_provider/vision_model for the text+vision merge (when set, image content is described by that vision model before forwarding) + context_window + its own economy (price_cache_hit_per_m/price_cache_miss_per_m/price_output_per_m/price_input_per_m, USD per 1M tokens for chat/embed/vision; for image routes price_input_per_m is a flat USD per image, plus the tiered/off-peak fields below) + is_active.

Resolution is a per-request overlay, not a fork. At request time routing.chat_overlay(requested, cfg) / routing.embed_overlay(requested, cfg) / routing.image_overlay(requested, cfg) resolve an active route by the requested model name and return a per-request OVERLAY dict of gateway_* cfg keys (gateway_force_model+gateway_model=target, gateway_upstream_url/gateway_api_key from the provider, the price keys, an augmented gateway_model_context_map, and vision overlay keys); handle_chat/handle_embeddings merge it onto the base cfg (cfg = {**cfg, **overlay}) BEFORE everything else, so the existing model-selection / pricing_from_cfg / parse_context_map / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. _ensure (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.

No matching route = None overlay = byte-identical legacy behavior - this is the "nobody feels the transformation" guarantee: molodetz/molodetz~embed/molodetz-img-small and every existing caller are byte-identical when no route matches. A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key.

CRUD. Admin JSON at /admin/gateway/{providers,models} (routers/admin/gateway_configs.py, require_admin, Pydantic ProviderIn/ModelRouteIn validation, accepts both JSON from static/js/GatewayAdmin.js and form from Devii), audited under gateway.provider.*/gateway.model.* (category ai), included in the admin package with the page at /admin/gateway (templates/admin_gateway.html, sidebar link, admin_section="gateway").

Devii tools: gateway_providers/gateway_models (read) and gateway_provider_set/gateway_provider_delete/gateway_model_set/gateway_model_delete (admin; the two deletes are CONFIRM_REQUIRED).

When adding a routed value, overlay it as the matching gateway_* cfg key so the runtime needs no new branch.

Tests: tests/unit/services/openai_gateway/routing.py (overlay/economy/kind isolation), tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream (end-to-end through handle_chat), and tests/api/admin/gateway/ (admin CRUD + validation + role gating).

Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)

Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME gateway_models route row, so it stays fully provider-agnostic and opt-in per route.

Per-route fields (each optional/zero by default = feature off, so an unconfigured route is byte-identical to before): context_tier_threshold_tokens (0 disables tiering; when the request's input token count exceeds it, price_*_per_m_tier2 rates apply instead of the tier-1 rates above) and the four nullable price_cache_hit_per_m_tier2/price_cache_miss_per_m_tier2/price_output_per_m_tier2/price_input_per_m_tier2 (a component left None keeps its tier-1 rate even above threshold - so a provider that only re-prices input above a size threshold, keeping output flat, needs just one tier2 field set), plus off_peak_start_minute/off_peak_end_minute (nullable, UTC minutes-since-midnight, both-or-neither enforced by ModelRouteIn's model validator; a window where start > end wraps past midnight, e.g. DeepSeek's old V3/R1-era 16:30-00:30 UTC window) and off_peak_discount_pct (0-100, multiplies whichever tier rate is active). The same fields double for vision (price_input/output_per_m_tier2) and embed (price_input_per_m_tier2) exactly like their tier-1 counterparts already do.
Overlay + computation. chat_overlay/embed_overlay propagate all of these into the per-request cfg dict under gateway_* keys exactly like the existing price fields (section above); usage.pricing_from_cfg reads them generically (defaulting to None/0/disabled when absent), so Layer A (the single global flat Pricing config fields) never gains these dimensions - only a gateway_models route can enable them, preserving the "no matching route = byte-identical legacy behavior" guarantee. usage.compute_cost selects tier1 vs tier2 per rate component (_tiered_rate) based on whether norm["prompt"] (billable input tokens) exceeds the threshold, then applies the off-peak discount (_effective_rate/_off_peak_active, UTC wraparound-aware) to whichever rate was selected. The response header format and compute_cost's return shape (total, input_cost, output_cost, native) are unchanged - this is purely an internal rate-selection step before the existing input/output split math runs; a native upstream cost (OpenRouter) still overrides the modeled total exactly as before.
Migration. New gateway_models columns are added via has_column/create_column_by_example in routing.ensure_tables() (the CREATE TABLE IF NOT EXISTS DDL string alone would never reach a pre-existing table - see the database/CLAUDE.md column-ensure idiom).
Admin UI. /admin/gateway's model-route form has a "Tiered / off-peak pricing (optional)" subsection; off-peak start/end render as <input type="time"> (converted to/from UTC minutes-of-day by GatewayAdmin.js), and the routes table shows tiered/off-peak badges when a route has either dimension configured.
DeepSeek's real pricing is already the tier-1 shape, not a new dimension. DeepSeek's actual API (verified against api-docs.deepseek.com/quick_start/pricing) bills three flat per-1M rates - cache-hit input, cache-miss input, output - with no current context-length tier or off-peak window for the V4 models; that shape was already fully modeled by the pre-existing chat_cache_hit_per_m/chat_cache_miss_per_m/chat_output_per_m fields before this section's tier2/off-peak fields existed. routing.seed_default_deepseek_routes() (called once from database.migrate_ai_gateway_settings() at the end of init_db()) idempotently inserts two ready-made routes - deepseek-v4-flash ($0.0028/$0.14/$0.28 per 1M, 1M context) and deepseek-v4-pro ($0.003625/$0.435/$0.87 per 1M, 1M context) - only when that source_model row does not already exist, so a caller or Devii can request either name explicitly and get correctly-priced, decoupled from whatever the single global gateway_model default happens to be set to (switching that global setting between the two real models does NOT retroactively fix the flat Pricing config fields - the seeded routes are the model-agnostic, always-correct way to reference a specific priced model). Neither seeded route sets the tier2/off-peak fields (DeepSeek does not use them today); an admin can add them later on the same row if DeepSeek (or any other provider routed here) introduces such pricing.