<div class="docs-content" data-render>
# GatewayService (AI)
`name` `openai` - enabled by default - 30s interval (minimum 5s). The gateway is an OpenAI-compatible endpoint at `/openai/v1/*` and the **single source of truth for all AI** on the platform. Every other AI consumer (news grading, the bots, Devii) routes through it, so all spend is attributed, priced, and limitable in one place. Source: `services/openai_gateway/`. See [Reading service data](/docs/services-data.html).
> Audience: administrators and maintainers. This page describes credentials, pricing, and cost.
## What it does
It forwards chat-completions and other `/v1/*` calls to a configured upstream (DeepSeek by default), optionally describing image content with a vision model first so vision-less upstreams still work, records every call to a usage ledger, computes cost, retries transient failures, and trips a circuit breaker on sustained upstream errors. The 30s service loop samples concurrency and prunes old usage rows; the real work happens per request.
## Endpoints
```
POST /openai/v1/chat/completions
POST /openai/v1/embeddings
POST /openai/v1/images/generations
GET|POST|PUT|DELETE|PATCH /openai/v1/{path} passthrough
```
Chat requests are augmented (vision), forwarded, retried, priced, and ledgered. Embeddings forward to the configured embeddings upstream. Image generation forwards to the configured images upstream. Other paths pass through to the upstream.
## Authentication and attribution
`authorize()` accepts a request when any of these hold (and `Require authentication` is on):
- the presented key (sent as `X-API-KEY` or `Authorization: Bearer`) equals the **static access key** (`gateway_access_key`), or
- the presented key equals the auto-generated **internal key** (`gateway_internal_key`), which is how DevPlace's own services authenticate, or
- the request carries a session for an admin (and `Allow admins` is on) or any user (and `Allow users` is on).
`resolve_owner()` then records who made the call: `("internal", "devii")` for the internal key, `("key", "access")` for the access key, `("admin", uid)` or `("user", uid)` for a session user, else `("anonymous", "anonymous")`. This owner tuple is written on every ledger row so usage rolls up per user. Devii operating a signed-in user authenticates with that user's own API key, so their full gateway spend rolls up under their uid.
## Configuration fields
**Upstream:** `gateway_upstream_url` (default DeepSeek chat-completions), `gateway_model` (`deepseek-v4-flash`), `gateway_force_model` (override the client-requested model, default on), `gateway_api_key` (auto-migrated from `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY`), `gateway_timeout` (default and minimum 300s), `gateway_instances` (max concurrent forwards per worker, default 4, 1 to 64; this is the connection pool plus semaphore).
**Prompt:** `gateway_system_preamble` (multiline text, default empty) - operator text prepended ahead of every chat request's system message (see "System-message composition" below).
**Vision:** `gateway_vision_enabled` (default on), `gateway_vision_url` (OpenRouter by default), `gateway_vision_model` (`google/gemma-3-12b-it`), `gateway_vision_key` (auto-migrated from `OPENROUTER_API_KEY`), `gateway_vision_cache_size` (LRU entries, default 256, 0 disables).
**Images:** `gateway_image_enabled` (default on), `gateway_image_url` (OpenRouter by default), `gateway_image_model` (`black-forest-labs/flux-1.1-pro`), `gateway_image_key` (falls back to `gateway_api_key` / `OPENROUTER_API_KEY`).
**Access:** `gateway_require_auth` (default on), `gateway_allow_admins`, `gateway_allow_users` (both default on), `gateway_access_key` (secret, write-only), `gateway_internal_key` (auto-generated on boot; clear and restart to rotate).
**Pricing:** `gateway_price_cache_hit_per_m` (0.0028), `gateway_price_cache_miss_per_m` (0.14), `gateway_price_output_per_m` (0.28), `gateway_vision_price_input_per_m` (0.0), `gateway_vision_price_output_per_m` (0.0), `gateway_embed_price_input_per_m` (0.01), `gateway_image_price_per_call` (0.04). Token rates are USD per 1M tokens; the image price is a flat USD per generated image. Used only when the upstream returns no native cost.
**Reliability:** `gateway_max_retries` (default 2, 0 to 10), `gateway_retry_backoff_ms` (250, linear multiplied by attempt number), `gateway_circuit_threshold` (consecutive failures before opening, default 5, 0 disables), `gateway_circuit_cooldown_seconds` (default 30).
**Tracking:** `gateway_usage_retention_hours` (default 720), `gateway_model_context_map` (JSON mapping model name to context window for utilization tracking).
## Model routing and providers
The single upstream above is the implicit `default` provider. On top of it, an administrator can register additional named **providers** and map any number of requested model names onto them, so one gateway can front many models across many backends. Routing is managed on the **Gateway** page (`/admin/gateway`) and stored in two tables (`gateway_providers`, `gateway_models`); it is admin configuration, not user content.
- A **provider** is a named upstream: `name`, a chat-completions `base_url`, and an `api_key`. The embeddings URL is derived from the base URL by swapping `/chat/completions` for `/embeddings`; the images URL swaps `/chat/completions` for `/images/generations`.
- A **model route** maps a `source_model` (what a client requests) to a `provider` (blank uses the default upstream) and a `target_model` (sent upstream), with a `kind` of `chat`, `embed`, or `image`, an optional `context_window`, an optional vision provider and model, and its own per-model **economy**: `price_cache_hit_per_m`, `price_cache_miss_per_m`, `price_output_per_m`, and `price_input_per_m` (USD per 1M tokens for chat/embed/vision; for image routes the input price is a flat USD per generated image).
- A route can optionally add two further, independent pricing dimensions on top of that flat economy, for providers that charge more than one rate for the same token type: **tiered (context-length) pricing** - `context_tier_threshold_tokens` (0 disables it) plus a tier-2 override for any of the four rates above (`price_cache_hit_per_m_tier2`, `price_cache_miss_per_m_tier2`, `price_output_per_m_tier2`, `price_input_per_m_tier2`); once a request's input tokens exceed the threshold, the tier-2 rate is used for any component that has one set, while a component left blank keeps its tier-1 rate at any size - and **off-peak discounting** - `off_peak_start_minute`/`off_peak_end_minute` (UTC minutes since midnight; both must be set together, and an end before the start wraps past midnight) plus `off_peak_discount_pct`, a percentage knocked off whichever rate (tier-1 or tier-2) is currently active. Both dimensions default to off, so an unconfigured route's pricing is unaffected.
When a request's model matches an active route of the right kind, the gateway forwards to that route's provider and target model and meters the call against that route's economy (tiered/off-peak included, if configured). A route with a vision model turns on the **text-and-vision merge** for that route, so a text-only model can answer about images. When no route matches, the request falls through to the default upstream unchanged, so `molodetz`, `molodetz~embed`, `molodetz-img-small`, and every existing client keep behaving exactly as before - and the service-level pricing fields above never gain the tiered/off-peak dimensions, since only a model route can enable them. Resolution is a per-request overlay onto the service config, so all of the behavior below (authentication, system-message composition, vision augmentation, retries, the circuit breaker, the ledger, and the `X-Gateway-*` headers, whose format is unchanged by tiered/off-peak pricing) applies identically to routed requests.
Routes and providers are also managed conversationally by an administrator through Devii (`gateway_providers`, `gateway_models` to read; `gateway_provider_set`, `gateway_provider_delete`, `gateway_model_set`, `gateway_model_delete` to manage, with deletes confirmation-gated).
DeepSeek's real billing (verified against its published API pricing) is exactly the flat cache-hit/cache-miss/output shape above, with no context-length tier or off-peak window on its current models - two ready-made routes, `deepseek-v4-flash` (`$0.0028` / `$0.14` / `$0.28` per 1M tokens, 1M-token context) and `deepseek-v4-pro` (`$0.003625` / `$0.435` / `$0.87` per 1M tokens, 1M-token context), are seeded automatically (idempotently, never overwriting a customized route) so either model can be requested by name with correct pricing regardless of what the single `gateway_model` default above is set to.
## Cost computation
If the upstream response carries a numeric `cost` field, that value is used verbatim and split proportionally into input and output (the row is flagged `native_cost`). Otherwise cost is computed from the per-1M pricing: for chat, `cache_hit_tokens` and `cache_miss_tokens` are priced separately on input plus `completion_tokens` on output; for vision, input and output use the vision rates. When the requested model matches a model route, that route's per-model prices are used instead of the service-level pricing fields, so each model carries its own economy - including, when configured, the tiered and off-peak dimensions described above (tier selection first, then the off-peak discount, then the same input/output split). Prompt-cache hit rate directly drives cost: at the default rates cache hits are roughly fifty times cheaper than misses.
## Response headers
Every gateway response (chat, embeddings, images, and passthrough; success and error) carries `X-Gateway-*` headers built from the same computed ledger row, so any caller can read the token usage and dollar cost of its own call straight from the response without querying the admin analytics. `GatewayUsageLedger.record()` returns the computed row and each handler maps it through `usage.usage_response_headers()` onto the response:
`X-Gateway-Model`, `X-Gateway-Backend`, `X-Gateway-Prompt-Tokens`, `X-Gateway-Completion-Tokens`, `X-Gateway-Total-Tokens`, `X-Gateway-Cache-Hit-Tokens`, `X-Gateway-Cache-Miss-Tokens`, `X-Gateway-Reasoning-Tokens`, `X-Gateway-Cost-USD`, `X-Gateway-Input-Cost-USD`, `X-Gateway-Output-Cost-USD`, `X-Gateway-Cost-Native` (`1` when the dollar cost is the upstream's own value, `0` when computed from the per-1M pricing), `X-Gateway-Tokens-Per-Second`, `X-Gateway-Upstream-Latency-Ms`, `X-Gateway-Total-Latency-Ms` (full end-to-end gateway time), `X-Gateway-Gateway-Overhead-Ms` (gateway processing minus upstream and queue wait), `X-Gateway-Queue-Wait-Ms` (time waiting on the concurrency semaphore), `X-Gateway-Connect-Ms` (upstream connection establishment), and, when the model's window is known, `X-Gateway-Context-Window` and `X-Gateway-Context-Utilization`. The denied embeddings-disabled path makes no upstream call and returns no usage headers.
The AI content correction worker reads `X-Gateway-Cost-USD` and the token headers off its own correction call to accumulate per-user totals (see [AI content correction](/docs/ai-correction.html)).
## Reliability layer
`retry_send()` retries on connection errors, timeouts, and upstream 5xx, up to `max_retries`, with a linear backoff of `backoff_ms * attempt`. The `CircuitBreaker` counts consecutive failures: at `threshold` it opens and new requests are rejected immediately with `503` and error type `circuit_open`; after `cooldown_seconds` it half-opens to test recovery and closes on the next success. Setting the threshold to 0 disables the breaker.
## System-message composition
Before a chat request is forwarded (and only for `chat/completions`, never embeddings or passthrough), the gateway composes a single system message in the fixed order: operator preamble, then a date line, then the client's original system content.
The operator preamble (`gateway_system_preamble`) is prepended on every chat call so all calls carry your preferences. If the client sent no system message at all, the composed content becomes a new leading system message; a blank or whitespace-only preamble is a no-op.
The current date is injected as `Current date: DD/MM/YYYY` (EU format) only when the composed text contains no date in any common format already (ISO `YYYY-MM-DD`, `D/M/YYYY`, `DD/MM/YYYY`, `MM/DD/YYYY`, `DD-MM-YYYY`, `DD.MM.YYYY`, and textual months such as `14 June 2026`, `June 14, 2026`, `14 Jun 2026`). The date is intentionally date-only with no time: keeping it stable for the whole day preserves upstream prompt and token caching.
## Vision augmentation
Before forwarding a chat request, image blocks are described by the vision model and replaced with a text description, so a text-only upstream still answers questions about images. Descriptions are cached in an LRU keyed by image hash; vision calls are themselves ledgered with `backend="vision"`. If vision is disabled or the key is missing, images become a placeholder note rather than failing the request.
## Data it offers
**Live metrics** (in `describe()`, at `GET /admin/services/data`): runtime counters (`requests`, `errors`, `in_flight`, `peak_in_flight`, `vision_calls`, `embed_calls`, `image_calls`, `last_status`, `last_latency_ms`, `pool`, `circuit_open`) and a 24h summary (`requests`, `success_pct`, `error_pct`, `cost_hour`, `cost_24h`, `tokens_24h`, `avg_latency_ms`, `avg_tps`, `peak_concurrency`, `top_model`, `top_caller`).
**Ledger** `gateway_usage_ledger`: one row per upstream call with owner, backend, requested and actual model, status, success, error category, the full latency breakdown (`upstream_latency_ms`, `gateway_overhead_ms`, `queue_wait_ms`, `connect_ms`, `total_latency_ms`), token counts (prompt, completion, cache hit and miss, reasoning, total), `tokens_per_second`, context window and utilization, the cost fields (`cost_usd`, `input_cost_usd`, `output_cost_usd`, `native_cost`), request shape (`stream_requested`, `temperature`, `top_p`, `max_tokens`, `has_tools`), and reliability flags (`retries_attempted`, `retry_succeeded`, `circuit_open`). Pruned to `gateway_usage_retention_hours`.
**Concurrency samples** `gateway_concurrency_samples`: `in_flight` sampled once per service tick.
**Deep analytics** at `GET /admin/ai-usage/data` (`GatewayUsageOut`): per-hour buckets, token and latency percentiles, error breakdowns, cost (this hour, 24h, projected monthly, caching savings, by model, by caller), and behavior. TTFT and inter-token latency are null because the gateway calls the upstream non-streaming. Per-user usage is at `GET /admin/users/{uid}/ai-usage`. It is all rendered on `/admin/ai-usage` by `AiUsageMonitor.js`.
</div>
View Git Blame Copy Permalink