feat: add provider and model routing tables, admin UI, and audit category for gateway
Implement the multi-provider routing system for the OpenAI gateway, including two new database tables (`gateway_providers`, `gateway_models`) ensured at init, a new admin page at `/admin/gateway` with full CRUD for providers and model routes, and a `"gateway"` audit category mapped to `"ai"`. The routing layer sits transparently on top of the existing single-provider default path: unmatched model names fall through unchanged, while matched routes forward to the configured provider with their own pricing economy, vision model, and context window. Cross-worker cache invalidation uses a shared `_ROUTING_CACHE` bumped via `"gateway_routing"` cache version.
This commit is contained in:
@@ -167,6 +167,8 @@ A sibling of AI content correction that reuses its plumbing (`CORRECTABLE_FIELDS
|
||||
|
||||
`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. **System-message composition (`services/openai_gateway/system_message.py`, `handle_chat` only):** before assembling the upstream chat payload (after vision augmentation, so it reaches every chat consumer) the gateway composes ONE system message in the fixed order **operator preamble -> date line -> client system content**. (1) The admin-editable `gateway_system_preamble` config field (Prompt group, multiline text, default empty) is prepended ahead of the client's system content on every chat call; if the client sent no system message it becomes a new leading one (a blank preamble is a no-op). (2) The **current date in EU `DD/MM/YYYY` (date only, never time** - omitting time keeps the message byte-stable per day so upstream prompt caching stays effective) is injected only when the composed text has NO date in any common format, detected by the `contains_date` regex set (ISO `YYYY-MM-DD`, `D/M/YYYY`, `DD-MM-YYYY`, `DD.MM.YYYY`, `MM/DD/YYYY`, and textual months like `14 June 2026`/`June 14, 2026`/`14 Jun 2026`). Embeddings/passthrough are untouched. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway also serves OpenAI-compatible text **embeddings** at `POST /openai/v1/embeddings`: clients send the client model `molodetz~embed`, which `handle_embeddings` remaps to the configured embedding model (default `qwen/qwen3-embedding-8b` on OpenRouter, $0.01/1M input tokens), tracked per call as `backend="embed"` in `gateway_usage_ledger`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming. **Every gateway response also carries per-call `X-Gateway-*` headers** (model, full token breakdown, `Cost-USD`/input/output dollars, native-cost flag, tokens/sec, latency, context window/utilization): `GatewayUsageLedger.record` returns the computed ledger row and each handler's `finalize` maps it through `usage.usage_response_headers` onto the `Response`, so any caller can read its own spend (the AI-correction worker uses this).
|
||||
|
||||
**Provider and model routing (`services/openai_gateway/routing.py`, admin `/admin/gateway`).** Layered ON TOP of the single-provider service config above, which stays THE implicit `default` provider (env migration, internal key, `molodetz`/`molodetz~embed`, vision, embeddings, ledger all unchanged). Two dataset tables (ensured in `init_db` via `routing.ensure_tables`, cross-worker cache-invalidated under the `"gateway_routing"` cache-version name, NOT in `SOFT_DELETE_TABLES` - they are admin config like `site_settings`, hard CRUD): `gateway_providers` (named upstreams: `base_url` chat-completions URL + `api_key`; the embeddings URL is derived by swapping `/chat/completions` -> `/embeddings`) and `gateway_models` (source->target routes, each with `kind` chat|embed, an optional `vision_provider`/`vision_model` for the **text+vision merge**, a `context_window`, and its **own economy**: `price_cache_hit_per_m`/`price_cache_miss_per_m`/`price_output_per_m`/`price_input_per_m` USD per 1M tokens). At request time `routing.chat_overlay(requested, cfg)` / `embed_overlay(...)` resolve an active route by the requested model name and return a per-request OVERLAY dict that `handle_chat`/`handle_embeddings` merge onto the base cfg (`cfg = {**cfg, **overlay}`) BEFORE everything else, so the existing model-selection / `pricing_from_cfg` / `parse_context_map` / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. **No matching route = `None` overlay = byte-identical legacy behavior** (this is the "nobody feels the transformation" guarantee). A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key. CRUD is admin JSON at `/admin/gateway/{providers,models}` (Pydantic `ProviderIn`/`ModelRouteIn` validation, accepts JSON from `GatewayAdmin.js` and form from Devii), audited under `gateway.provider.*`/`gateway.model.*` (category `ai`), surfaced on the admin **Gateway** page, and driven by Devii via `gateway_providers`/`gateway_models` (read) and `gateway_provider_set`/`gateway_provider_delete`/`gateway_model_set`/`gateway_model_delete` (admin; the two deletes are `CONFIRM_REQUIRED`). When adding a routed value, overlay it as the matching `gateway_*` cfg key so the runtime needs no new branch.
|
||||
|
||||
To add a service: extend `BaseService`, override `async def run_once(self) -> None`, register in `main.py`'s startup with `service_manager.register(YourService())`. It auto-appears on the services page.
|
||||
|
||||
### Async job services (`devplacepy/services/jobs/`)
|
||||
|
||||
Reference in New Issue
Block a user