blindxfish/devplacepy
1
0
Fork 0
forked from retoor/devplacepy
Code Pull Requests Activity

Update

Browse Source
This commit is contained in:
retoor
2026-07-09 02:52:54 +02:00
parent 818568c609
commit 48bb6c2ec2
95 changed files with 6115 additions and 267 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devplacepy/services/openai_gateway/CLAUDE.md
+17 -3
View File
@@ -52,6 +52,10 @@ The gateway records one row per upstream call (chat, vision, passthrough) and su
**Financial data is admin-only everywhere.** Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, `ai_correction`/`ai_modifier` usage displays, and the Devii cost tools.
## Image generation
`POST /openai/v1/images/generations` exposes an OpenAI-compatible image-generation endpoint. Clients send the generic model `molodetz-img-small` (`config.INTERNAL_IMAGE_MODEL`), which `handle_images` remaps to `gateway_image_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty or `molodetz-img`). It defaults to OpenRouter's `black-forest-labs/flux-1.1-pro` at `https://openrouter.ai/api/v1/images/generations` (`config.IMAGE_*_DEFAULT`, $0.04 per image fallback). `handle_images` mirrors `handle_embeddings`: build the payload, forward via `_send`, and record one ledger row. The config fields are the **Images** group (`gateway_image_enabled` default on, `gateway_image_url`, `gateway_image_model`, `gateway_image_key`) plus the Pricing-group `gateway_image_price_per_call`. `effective_config()` falls the image key back to `gateway_api_key` then `OPENROUTER_API_KEY`. Usage is recorded with **`backend="image"`**; `usage.compute_cost` adds an `image` branch (flat per-call, native OpenRouter `cost` still preferred via `extract_image_usage`). `routing.image_overlay` resolves per-route provider/url/key and uses `price_input_per_m` as the per-image price. `routing.seed_default_image_routes()` (from `migrate_ai_gateway_settings`) idempotently seeds `molodetz-img-small` -> Flux on the `openrouter` provider when `OPENROUTER_API_KEY` is set. When `gateway_image_enabled` is off the endpoint returns 503 with no ledger row.
## Embeddings
`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row.
@@ -75,11 +79,11 @@ Layered ON TOP of the single-provider service config above, which stays THE impl
**Storage.** Two dataset tables, ensured in `init_db` via `routing.ensure_tables`, cross-worker cache-invalidated under the `"gateway_routing"` cache-version name (module-level `provider_store`/`model_store` over a shared `_ROUTING_CACHE`; writes `bump_cache_version` and clear the cache). They are admin config, NOT in `SOFT_DELETE_TABLES` - hard CRUD, mirroring `site_settings`:
- `gateway_providers` - named upstreams: `name` + `base_url` (chat-completions URL) + `api_key` + `is_active`; the embeddings URL is derived by swapping `/chat/completions` -> `/embeddings`.
- `gateway_models` - source->target routes: `source_model` (unique, what clients request) -> `provider` (blank = default) + `target_model` + `kind` (chat|embed) + optional `vision_provider`/`vision_model` for the **text+vision merge** (when set, image content is described by that vision model before forwarding) + `context_window` + its **own economy** (`price_cache_hit_per_m`/`price_cache_miss_per_m`/`price_output_per_m`/`price_input_per_m`, USD per 1M tokens) + `is_active`.
- `gateway_models` - source->target routes: `source_model` (unique, what clients request) -> `provider` (blank = default) + `target_model` + `kind` (chat|embed|image) + optional `vision_provider`/`vision_model` for the **text+vision merge** (when set, image content is described by that vision model before forwarding) + `context_window` + its **own economy** (`price_cache_hit_per_m`/`price_cache_miss_per_m`/`price_output_per_m`/`price_input_per_m`, USD per 1M tokens for chat/embed/vision; for image routes `price_input_per_m` is a flat USD per image, plus the tiered/off-peak fields below) + `is_active`.
**Resolution is a per-request overlay, not a fork.** At request time `routing.chat_overlay(requested, cfg)` / `routing.embed_overlay(requested, cfg)` resolve an active route by the requested model name and return a per-request OVERLAY dict of `gateway_*` cfg keys (`gateway_force_model`+`gateway_model`=target, `gateway_upstream_url`/`gateway_api_key` from the provider, the price keys, an augmented `gateway_model_context_map`, and vision overlay keys); `handle_chat`/`handle_embeddings` merge it onto the base cfg (`cfg = {**cfg, **overlay}`) BEFORE everything else, so the existing model-selection / `pricing_from_cfg` / `parse_context_map` / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. `_ensure` (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.
**Resolution is a per-request overlay, not a fork.** At request time `routing.chat_overlay(requested, cfg)` / `routing.embed_overlay(requested, cfg)` / `routing.image_overlay(requested, cfg)` resolve an active route by the requested model name and return a per-request OVERLAY dict of `gateway_*` cfg keys (`gateway_force_model`+`gateway_model`=target, `gateway_upstream_url`/`gateway_api_key` from the provider, the price keys, an augmented `gateway_model_context_map`, and vision overlay keys); `handle_chat`/`handle_embeddings` merge it onto the base cfg (`cfg = {**cfg, **overlay}`) BEFORE everything else, so the existing model-selection / `pricing_from_cfg` / `parse_context_map` / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. `_ensure` (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.
**No matching route = `None` overlay = byte-identical legacy behavior** - this is the "nobody feels the transformation" guarantee: `molodetz`/`molodetz~embed` and every existing caller are byte-identical when no route matches. A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key.
**No matching route = `None` overlay = byte-identical legacy behavior** - this is the "nobody feels the transformation" guarantee: `molodetz`/`molodetz~embed`/`molodetz-img-small` and every existing caller are byte-identical when no route matches. A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key.
**CRUD.** Admin JSON at `/admin/gateway/{providers,models}` (`routers/admin/gateway_configs.py`, `require_admin`, Pydantic `ProviderIn`/`ModelRouteIn` validation, accepts both JSON from `static/js/GatewayAdmin.js` and form from Devii), audited under `gateway.provider.*`/`gateway.model.*` (category `ai`), included in the admin package with the page at `/admin/gateway` (`templates/admin_gateway.html`, sidebar link, `admin_section="gateway"`).
@@ -88,3 +92,13 @@ Layered ON TOP of the single-provider service config above, which stays THE impl
When adding a routed value, overlay it as the matching `gateway_*` cfg key so the runtime needs no new branch.
**Tests:** `tests/unit/services/openai_gateway/routing.py` (overlay/economy/kind isolation), `tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream` (end-to-end through `handle_chat`), and `tests/api/admin/gateway/` (admin CRUD + validation + role gating).
## Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)
Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME `gateway_models` route row, so it stays fully provider-agnostic and opt-in per route.
- **Per-route fields** (each optional/zero by default = feature off, so an unconfigured route is byte-identical to before): `context_tier_threshold_tokens` (0 disables tiering; when the request's input token count exceeds it, `price_*_per_m_tier2` rates apply instead of the tier-1 rates above) and the four nullable `price_cache_hit_per_m_tier2`/`price_cache_miss_per_m_tier2`/`price_output_per_m_tier2`/`price_input_per_m_tier2` (a component left `None` keeps its tier-1 rate even above threshold - so a provider that only re-prices input above a size threshold, keeping output flat, needs just one tier2 field set), plus `off_peak_start_minute`/`off_peak_end_minute` (nullable, UTC minutes-since-midnight, both-or-neither enforced by `ModelRouteIn`'s model validator; a window where start > end wraps past midnight, e.g. DeepSeek's old V3/R1-era 16:30-00:30 UTC window) and `off_peak_discount_pct` (0-100, multiplies whichever tier rate is active). The same fields double for vision (`price_input/output_per_m_tier2`) and embed (`price_input_per_m_tier2`) exactly like their tier-1 counterparts already do.
- **Overlay + computation.** `chat_overlay`/`embed_overlay` propagate all of these into the per-request cfg dict under `gateway_*` keys exactly like the existing price fields (section above); `usage.pricing_from_cfg` reads them generically (defaulting to `None`/`0`/disabled when absent), so **Layer A (the single global flat Pricing config fields) never gains these dimensions** - only a `gateway_models` route can enable them, preserving the "no matching route = byte-identical legacy behavior" guarantee. `usage.compute_cost` selects tier1 vs tier2 per rate component (`_tiered_rate`) based on whether `norm["prompt"]` (billable input tokens) exceeds the threshold, then applies the off-peak discount (`_effective_rate`/`_off_peak_active`, UTC wraparound-aware) to whichever rate was selected. **The response header format and `compute_cost`'s return shape (`total, input_cost, output_cost, native`) are unchanged** - this is purely an internal rate-selection step before the existing input/output split math runs; a native upstream `cost` (OpenRouter) still overrides the modeled total exactly as before.
- **Migration.** New `gateway_models` columns are added via `has_column`/`create_column_by_example` in `routing.ensure_tables()` (the `CREATE TABLE IF NOT EXISTS` DDL string alone would never reach a pre-existing table - see the `database/CLAUDE.md` column-ensure idiom).
- **Admin UI.** `/admin/gateway`'s model-route form has a "Tiered / off-peak pricing (optional)" subsection; off-peak start/end render as `<input type="time">` (converted to/from UTC minutes-of-day by `GatewayAdmin.js`), and the routes table shows `tiered`/`off-peak` badges when a route has either dimension configured.
- **DeepSeek's real pricing is already the tier-1 shape, not a new dimension.** DeepSeek's actual API (verified against `api-docs.deepseek.com/quick_start/pricing`) bills three flat per-1M rates - cache-hit input, cache-miss input, output - with no current context-length tier or off-peak window for the V4 models; that shape was already fully modeled by the pre-existing `chat_cache_hit_per_m`/`chat_cache_miss_per_m`/`chat_output_per_m` fields before this section's tier2/off-peak fields existed. `routing.seed_default_deepseek_routes()` (called once from `database.migrate_ai_gateway_settings()` at the end of `init_db()`) idempotently inserts two ready-made routes - `deepseek-v4-flash` (`$0.0028`/`$0.14`/`$0.28` per 1M, 1M context) and `deepseek-v4-pro` (`$0.003625`/`$0.435`/`$0.87` per 1M, 1M context) - only when that `source_model` row does not already exist, so a caller or Devii can request either name explicitly and get correctly-priced, decoupled from whatever the single global `gateway_model` default happens to be set to (switching that global setting between the two real models does NOT retroactively fix the flat Pricing config fields - the seeded routes are the model-agnostic, always-correct way to reference a specific priced model). Neither seeded route sets the tier2/off-peak fields (DeepSeek does not use them today); an admin can add them later on the same row if DeepSeek (or any other provider routed here) introduces such pricing.