forked from retoor/devplacepy
Update
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
This file documents the AI gateway subsystem (devplacepy/services/openai_gateway/) - the single point of truth for AI calls, usage metering, and provider/model routing. Claude Code loads it automatically whenever a file under this directory is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`GatewayService` (`services/openai_gateway/`) is an OpenAI-compatible LLM gateway (the ported `openai5.py`), mounted at `/openai/v1/*` (`routers/openai_gateway.py`, prefix `/openai`). It is a **service that serves an HTTP endpoint** rather than a loop.
|
||||
|
||||
- The router is thin: it calls `service_manager.get_service("openai").handle(request, subpath)`. `POST /v1/chat/completions` runs the full gateway (vision augment -> model override -> forward -> optional SSE re-emit via `_fake_stream`); `POST /v1/embeddings` forwards to the configured embeddings upstream (`handle_embeddings`, no vision/streaming); `/v1/{path:path}` is a transparent passthrough to the upstream base. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
|
||||
- `main.py` registers it and exempts `/openai` from the rate-limit middleware and the maintenance gate. No new dependency (`httpx` already required).
|
||||
- `GatewayService` is `default_enabled=True` so internal callers work out of the box.
|
||||
|
||||
## Per-call cost/token response headers
|
||||
|
||||
Every gateway response (chat, embeddings, passthrough; success and error) carries `X-Gateway-*` headers describing that single call: `Model`, `Backend`, `Prompt-Tokens`, `Completion-Tokens`, `Total-Tokens`, `Cache-Hit-Tokens`, `Cache-Miss-Tokens`, `Reasoning-Tokens`, `Cost-USD`/`Input-Cost-USD`/`Output-Cost-USD` (dollars), `Cost-Native` (1 if the upstream returned a native cost), `Tokens-Per-Second`, `Upstream-Latency-Ms`, and `Context-Window`/`Context-Utilization` when the model's window is known, plus the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`. `GatewayUsageLedger.record(...)` RETURNS the computed ledger row (or `None` on failure); each handler's `finalize` closure maps it through `usage.usage_response_headers(row)` and attaches it to the returned `Response` (`resp_headers`). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads `X-Gateway-Cost-USD`/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root `CLAUDE.md`).
|
||||
|
||||
## Per-worker runtime
|
||||
|
||||
**Serves in every worker.** Config/enabled come from `site_settings` (via the service's `get_config()`/`is_enabled()`), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (`GatewayRuntime`: `httpx.AsyncClient` pool + `asyncio.Semaphore` sized to `gateway_instances`, plus counters and a `VisionCache`) is created lazily on first request and rebuilt when `instances`/`timeout`/cache size change. `gateway_instances` is the **scaling knob** (concurrency per worker); process scaling is uvicorn workers.
|
||||
|
||||
## Auth
|
||||
|
||||
`authorize()` reuses `get_current_user` (cookie/X-API-KEY/Bearer/Basic). Allowed if `gateway_require_auth` is off, the presented X-API-KEY/Bearer equals the static `gateway_access_key` **or the auto-generated `gateway_internal_key`**, or the resolved user is an admin (`gateway_allow_admins`) or any user (`gateway_allow_users`).
|
||||
|
||||
**Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default **on**); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. With `gateway_allow_users` off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key.
|
||||
|
||||
## Config fields
|
||||
|
||||
All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble`, timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
|
||||
|
||||
## System-message composition (date awareness + operator preamble)
|
||||
|
||||
`system_message.py` (`apply_system_directives`) runs in `handle_chat` only (NOT `handle_embeddings`/`handle_passthrough`), after vision augmentation and before the upstream payload is assembled, so it reaches every chat consumer (news, bots, Devii guests, per-user direct API calls). It composes ONE system message in a fixed, cache-friendly order: **operator preamble (`gateway_system_preamble`) -> date line (`Current date: DD/MM/YYYY`) -> the client's original system content**.
|
||||
|
||||
Two behaviors:
|
||||
|
||||
- **Date awareness (EU, no time).** The current date is injected only when the composed text (preamble + client system content) contains NO date in any common format. Detection lives in `contains_date` via the `_DATE_PATTERNS` regex set: `YYYY-MM-DD` (ISO), `D/M/YYYY` and `DD/MM/YYYY` (also matches `MM/DD/YYYY`), `DD-MM-YYYY`, `DD.MM.YYYY`, and textual months (`14 June 2026`, `June 14, 2026`, `14 Jun 2026`, with abbreviations and optional ordinal/comma). The date is `datetime.now().strftime("%d/%m/%Y")` - **date only, never time**: omitting the time keeps the system message byte-stable for the whole day so upstream prompt/token caching stays effective.
|
||||
- **Operator preamble.** `gateway_system_preamble` (Prompt group, `type="text"` textarea, default empty) is prepended ahead of the client's system content on every chat call so all calls carry operator preferences. When the client sent NO system message, the composed content (preamble and/or date) becomes a brand-new leading system message; when the client DID send one, its content is replaced in place (preamble + date + original text). A blank/whitespace-only preamble with an already-dated request is a no-op (no empty system message is created). The date detection runs over the preamble too, so an operator preamble that already states a date suppresses the injected date line.
|
||||
|
||||
Embeddings/passthrough are untouched by this composition step.
|
||||
|
||||
## Usage, cost, latency, and reliability tracking
|
||||
|
||||
The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:
|
||||
|
||||
- **`usage.py`** - `GatewayUsageLedger` writes to `gateway_usage_ledger` (one row per call, success and failure) and samples `gateway_concurrency_samples` each 30s tick. `normalize_usage` handles BOTH upstream shapes: DeepSeek (`prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`, `completion_tokens_details.reasoning_tokens`) and OpenRouter (`prompt_tokens_details.cached_tokens`, native `cost`, `cost_details`). `compute_cost` PREFERS the upstream native `cost` (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share `input_cost / (input_cost + output_cost)` from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). `record()` is wrapped in try/except so a tracking failure never breaks the proxy.
|
||||
- **`gateway.py`** - `_send` is the gated/timed/retried path: a `CircuitBreaker` (`reliability.py`) short-circuits when open; `retry_send` retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the `queue_wait_ms`; an httpx per-request `trace` extension captures `connect_ms` (0 on pooled reuse). `handle_chat`/`handle_passthrough` record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from `VisionAugmenter._describe_one`.
|
||||
- **Owner attribution** - `GatewayService.resolve_owner` maps the caller to `(owner_kind, owner_id)`: `internal:devii` (internal key), `key:access` (static key), `user:<uid>`/`admin:<uid>` (session or API key), else `anonymous`.
|
||||
- **`analytics.py`** - `build_analytics(hours, top_n, pricing)` pulls the bounded window once (clamped to `MAX_WINDOW_HOURS=168`) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via `reliability.percentile`), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. `summary_metrics()` runs cheap SQL aggregates for the live service panel. TTFT/inter-token are returned `null` (gateway forwards non-streaming).
|
||||
- **Admin** - `/admin/ai-usage` (HTML, `templates/ai_usage.html` + `static/js/AiUsageMonitor.js`) and `/admin/ai-usage/data?hours=&top_n=` (JSON) in `routers/admin/` package, admin-gated. Schema `GatewayUsageOut`. Devii action `ai_usage` and docs `admin-ai-usage` expose the same endpoint. The `ai_usage` and `site_analytics` Devii catalog tools are both `requires_admin=True`.
|
||||
- **Per-user usage** - `analytics.build_user_usage(owner_id, hours=24, pricing)` filters the ledger by `owner_id` (a uid matches both `user:` and `admin:` rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at `GET /admin/users/{uid}/ai-usage` (schema `UserAiUsageOut`, docs `admin-user-ai-usage`) and rendered on the profile page by `static/js/UserAiUsage.js` (the `[data-ai-usage]` card, gated `{% if is_admin(user) %}` in `templates/profile.html`). The 24h window is a sound projection basis (unlike a startup burst), so `cost_24h * 30` is honest.
|
||||
- **Profile visibility split** - `routers/profile/` package `_ai_quota(uid, include_cost=False)` builds the profile quota bar. **The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal:** `spent_usd`/`used_pct`/`requests`/`last_used` come from `analytics.user_spend_24h(uid)`, the SAME `gateway_usage_ledger` source (summed by `owner_id`, matching both `user:` and `admin:` rows) that feeds the stat grid's `cost.window_usd`. So the server-rendered `spent_usd` is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. `turns` is still the `devii_usage_ledger` `turns_24h` count (Devii conversation turns) and is labeled **"Devii turns"** in the template precisely because it is a *different metric* from the grid's total **API requests** (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why `0 Devii turns` can sit beside `18 API requests`). The `limit`/`turns` still come from the Devii service (`daily_limit_for`/`turns_24h`); an **unlimited** limit (`<= 0`, the admin default) sets `unlimited=True` so the template renders "no cap" / "No daily spend cap" instead of a misleading `/ $0.00`. **This is display-only: the Devii daily-cap ENFORCEMENT and `/devii/usage` still read the Devii-scoped `devii_usage_ledger.spent_24h`** (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: **admin** sees the full gateway stats card *plus* a server-rendered quota bar (with spend/limit/Devii turns); the **owner** (non-admin, `is_self`) sees a quota-**only** card showing just the percentage; everyone else sees nothing AI-related. `_ai_quota` is fail-safe (returns `None` on any error or when the Devii service is unregistered) and the result rides on `ProfileOut.ai_quota`. **Dollar figures (`spent_usd`/`limit_usd`) are included only when `include_cost=viewer_is_admin`** - never for a non-admin owner. This matters because the route serves the same context as JSON via `respond(..., model=ProfileOut)` and `ProfileOut.ai_quota` is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with `Accept: application/json`; the server-side omission is the real control. Non-admins (HTML and JSON) get only `used_pct`/`turns`/`requests`.
|
||||
- **Retention** - `run_once` prunes both tables past `gateway_usage_retention_hours` (default 720 = 30 days).
|
||||
|
||||
**Financial data is admin-only everywhere.** Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, `ai_correction`/`ai_modifier` usage displays, and the Devii cost tools.
|
||||
|
||||
## Embeddings
|
||||
|
||||
`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row.
|
||||
|
||||
## Single point of truth for AI
|
||||
|
||||
The gateway is the only place that holds real provider URLs/models/keys. Every other AI consumer (news, bots, Devii guests) points at it by default and never touches a provider key:
|
||||
|
||||
- Defaults live in `config.py`: `INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override with `DEVPLACE_INTERNAL_BASE_URL`) and `INTERNAL_MODEL` (`molodetz`). `news_ai_url`, `bot_api_url`, `devii_ai_url` default to `INTERNAL_GATEWAY_URL`; their model defaults to `molodetz`.
|
||||
- Each consumer's key falls back to `database.internal_gateway_key()` (reads the `gateway_internal_key` setting) when its own key field/env is unset - the provider-key fallbacks (`DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY`) were removed from news and bots.
|
||||
- `gateway_force_model` (default on) and a `molodetz`/empty alias in `handle_chat` make the upstream always receive `gateway_model`, so `molodetz` is a stable generic alias.
|
||||
- `database.migrate_ai_gateway_settings()` (called at the end of `init_db()`, under the startup `init_lock`): generates `gateway_internal_key` (uuid4) if missing; migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into `gateway_api_key`/`gateway_vision_key` when the db value is empty; and rewrites any consumer AI URL still equal to the old `openai.app.molodetz.nl` default to the gateway, plus `bot_model` `deepseek-chat` -> `molodetz` (only uncustomized values).
|
||||
- The gateway's `gateway_api_key`/`gateway_vision_key`/`gateway_internal_key` fields are **non-secret** so the admin services page shows the value actually in use, editable.
|
||||
- Bot LLM calls are synchronous `urllib` but already run via `asyncio.to_thread` (`bot/bot.py`), so the local round-trip never blocks the event loop.
|
||||
- Real provider keys/URLs/models live ONLY in the gateway. The gateway is `default_enabled=True`.
|
||||
|
||||
## Provider and model routing (`routing.py`, admin `/admin/gateway`)
|
||||
|
||||
Layered ON TOP of the single-provider service config above, which stays THE implicit `default` provider (env migration, internal key, `molodetz`/`molodetz~embed`, vision, embeddings, ledger all unchanged).
|
||||
|
||||
**Storage.** Two dataset tables, ensured in `init_db` via `routing.ensure_tables`, cross-worker cache-invalidated under the `"gateway_routing"` cache-version name (module-level `provider_store`/`model_store` over a shared `_ROUTING_CACHE`; writes `bump_cache_version` and clear the cache). They are admin config, NOT in `SOFT_DELETE_TABLES` - hard CRUD, mirroring `site_settings`:
|
||||
|
||||
- `gateway_providers` - named upstreams: `name` + `base_url` (chat-completions URL) + `api_key` + `is_active`; the embeddings URL is derived by swapping `/chat/completions` -> `/embeddings`.
|
||||
- `gateway_models` - source->target routes: `source_model` (unique, what clients request) -> `provider` (blank = default) + `target_model` + `kind` (chat|embed) + optional `vision_provider`/`vision_model` for the **text+vision merge** (when set, image content is described by that vision model before forwarding) + `context_window` + its **own economy** (`price_cache_hit_per_m`/`price_cache_miss_per_m`/`price_output_per_m`/`price_input_per_m`, USD per 1M tokens) + `is_active`.
|
||||
|
||||
**Resolution is a per-request overlay, not a fork.** At request time `routing.chat_overlay(requested, cfg)` / `routing.embed_overlay(requested, cfg)` resolve an active route by the requested model name and return a per-request OVERLAY dict of `gateway_*` cfg keys (`gateway_force_model`+`gateway_model`=target, `gateway_upstream_url`/`gateway_api_key` from the provider, the price keys, an augmented `gateway_model_context_map`, and vision overlay keys); `handle_chat`/`handle_embeddings` merge it onto the base cfg (`cfg = {**cfg, **overlay}`) BEFORE everything else, so the existing model-selection / `pricing_from_cfg` / `parse_context_map` / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. `_ensure` (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.
|
||||
|
||||
**No matching route = `None` overlay = byte-identical legacy behavior** - this is the "nobody feels the transformation" guarantee: `molodetz`/`molodetz~embed` and every existing caller are byte-identical when no route matches. A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key.
|
||||
|
||||
**CRUD.** Admin JSON at `/admin/gateway/{providers,models}` (`routers/admin/gateway_configs.py`, `require_admin`, Pydantic `ProviderIn`/`ModelRouteIn` validation, accepts both JSON from `static/js/GatewayAdmin.js` and form from Devii), audited under `gateway.provider.*`/`gateway.model.*` (category `ai`), included in the admin package with the page at `/admin/gateway` (`templates/admin_gateway.html`, sidebar link, `admin_section="gateway"`).
|
||||
|
||||
**Devii tools:** `gateway_providers`/`gateway_models` (read) and `gateway_provider_set`/`gateway_provider_delete`/`gateway_model_set`/`gateway_model_delete` (admin; the two deletes are `CONFIRM_REQUIRED`).
|
||||
|
||||
When adding a routed value, overlay it as the matching `gateway_*` cfg key so the runtime needs no new branch.
|
||||
|
||||
**Tests:** `tests/unit/services/openai_gateway/routing.py` (overlay/economy/kind isolation), `tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream` (end-to-end through `handle_chat`), and `tests/api/admin/gateway/` (admin CRUD + validation + role gating).
|
||||
Reference in New Issue
Block a user