forked from retoor/devplacepy
Update
This commit is contained in:
@@ -52,6 +52,10 @@ The gateway records one row per upstream call (chat, vision, passthrough) and su
|
||||
|
||||
**Financial data is admin-only everywhere.** Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, `ai_correction`/`ai_modifier` usage displays, and the Devii cost tools.
|
||||
|
||||
## Image generation
|
||||
|
||||
`POST /openai/v1/images/generations` exposes an OpenAI-compatible image-generation endpoint. Clients send the generic model `molodetz-img-small` (`config.INTERNAL_IMAGE_MODEL`), which `handle_images` remaps to `gateway_image_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty or `molodetz-img`). It defaults to OpenRouter's `black-forest-labs/flux-1.1-pro` at `https://openrouter.ai/api/v1/images/generations` (`config.IMAGE_*_DEFAULT`, $0.04 per image fallback). `handle_images` mirrors `handle_embeddings`: build the payload, forward via `_send`, and record one ledger row. The config fields are the **Images** group (`gateway_image_enabled` default on, `gateway_image_url`, `gateway_image_model`, `gateway_image_key`) plus the Pricing-group `gateway_image_price_per_call`. `effective_config()` falls the image key back to `gateway_api_key` then `OPENROUTER_API_KEY`. Usage is recorded with **`backend="image"`**; `usage.compute_cost` adds an `image` branch (flat per-call, native OpenRouter `cost` still preferred via `extract_image_usage`). `routing.image_overlay` resolves per-route provider/url/key and uses `price_input_per_m` as the per-image price. `routing.seed_default_image_routes()` (from `migrate_ai_gateway_settings`) idempotently seeds `molodetz-img-small` -> Flux on the `openrouter` provider when `OPENROUTER_API_KEY` is set. When `gateway_image_enabled` is off the endpoint returns 503 with no ledger row.
|
||||
|
||||
## Embeddings
|
||||
|
||||
`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row.
|
||||
@@ -75,11 +79,11 @@ Layered ON TOP of the single-provider service config above, which stays THE impl
|
||||
**Storage.** Two dataset tables, ensured in `init_db` via `routing.ensure_tables`, cross-worker cache-invalidated under the `"gateway_routing"` cache-version name (module-level `provider_store`/`model_store` over a shared `_ROUTING_CACHE`; writes `bump_cache_version` and clear the cache). They are admin config, NOT in `SOFT_DELETE_TABLES` - hard CRUD, mirroring `site_settings`:
|
||||
|
||||
- `gateway_providers` - named upstreams: `name` + `base_url` (chat-completions URL) + `api_key` + `is_active`; the embeddings URL is derived by swapping `/chat/completions` -> `/embeddings`.
|
||||
- `gateway_models` - source->target routes: `source_model` (unique, what clients request) -> `provider` (blank = default) + `target_model` + `kind` (chat|embed) + optional `vision_provider`/`vision_model` for the **text+vision merge** (when set, image content is described by that vision model before forwarding) + `context_window` + its **own economy** (`price_cache_hit_per_m`/`price_cache_miss_per_m`/`price_output_per_m`/`price_input_per_m`, USD per 1M tokens) + `is_active`.
|
||||
- `gateway_models` - source->target routes: `source_model` (unique, what clients request) -> `provider` (blank = default) + `target_model` + `kind` (chat|embed|image) + optional `vision_provider`/`vision_model` for the **text+vision merge** (when set, image content is described by that vision model before forwarding) + `context_window` + its **own economy** (`price_cache_hit_per_m`/`price_cache_miss_per_m`/`price_output_per_m`/`price_input_per_m`, USD per 1M tokens for chat/embed/vision; for image routes `price_input_per_m` is a flat USD per image, plus the tiered/off-peak fields below) + `is_active`.
|
||||
|
||||
**Resolution is a per-request overlay, not a fork.** At request time `routing.chat_overlay(requested, cfg)` / `routing.embed_overlay(requested, cfg)` resolve an active route by the requested model name and return a per-request OVERLAY dict of `gateway_*` cfg keys (`gateway_force_model`+`gateway_model`=target, `gateway_upstream_url`/`gateway_api_key` from the provider, the price keys, an augmented `gateway_model_context_map`, and vision overlay keys); `handle_chat`/`handle_embeddings` merge it onto the base cfg (`cfg = {**cfg, **overlay}`) BEFORE everything else, so the existing model-selection / `pricing_from_cfg` / `parse_context_map` / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. `_ensure` (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.
|
||||
**Resolution is a per-request overlay, not a fork.** At request time `routing.chat_overlay(requested, cfg)` / `routing.embed_overlay(requested, cfg)` / `routing.image_overlay(requested, cfg)` resolve an active route by the requested model name and return a per-request OVERLAY dict of `gateway_*` cfg keys (`gateway_force_model`+`gateway_model`=target, `gateway_upstream_url`/`gateway_api_key` from the provider, the price keys, an augmented `gateway_model_context_map`, and vision overlay keys); `handle_chat`/`handle_embeddings` merge it onto the base cfg (`cfg = {**cfg, **overlay}`) BEFORE everything else, so the existing model-selection / `pricing_from_cfg` / `parse_context_map` / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. `_ensure` (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.
|
||||
|
||||
**No matching route = `None` overlay = byte-identical legacy behavior** - this is the "nobody feels the transformation" guarantee: `molodetz`/`molodetz~embed` and every existing caller are byte-identical when no route matches. A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key.
|
||||
**No matching route = `None` overlay = byte-identical legacy behavior** - this is the "nobody feels the transformation" guarantee: `molodetz`/`molodetz~embed`/`molodetz-img-small` and every existing caller are byte-identical when no route matches. A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key.
|
||||
|
||||
**CRUD.** Admin JSON at `/admin/gateway/{providers,models}` (`routers/admin/gateway_configs.py`, `require_admin`, Pydantic `ProviderIn`/`ModelRouteIn` validation, accepts both JSON from `static/js/GatewayAdmin.js` and form from Devii), audited under `gateway.provider.*`/`gateway.model.*` (category `ai`), included in the admin package with the page at `/admin/gateway` (`templates/admin_gateway.html`, sidebar link, `admin_section="gateway"`).
|
||||
|
||||
@@ -88,3 +92,13 @@ Layered ON TOP of the single-provider service config above, which stays THE impl
|
||||
When adding a routed value, overlay it as the matching `gateway_*` cfg key so the runtime needs no new branch.
|
||||
|
||||
**Tests:** `tests/unit/services/openai_gateway/routing.py` (overlay/economy/kind isolation), `tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream` (end-to-end through `handle_chat`), and `tests/api/admin/gateway/` (admin CRUD + validation + role gating).
|
||||
|
||||
## Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)
|
||||
|
||||
Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME `gateway_models` route row, so it stays fully provider-agnostic and opt-in per route.
|
||||
|
||||
- **Per-route fields** (each optional/zero by default = feature off, so an unconfigured route is byte-identical to before): `context_tier_threshold_tokens` (0 disables tiering; when the request's input token count exceeds it, `price_*_per_m_tier2` rates apply instead of the tier-1 rates above) and the four nullable `price_cache_hit_per_m_tier2`/`price_cache_miss_per_m_tier2`/`price_output_per_m_tier2`/`price_input_per_m_tier2` (a component left `None` keeps its tier-1 rate even above threshold - so a provider that only re-prices input above a size threshold, keeping output flat, needs just one tier2 field set), plus `off_peak_start_minute`/`off_peak_end_minute` (nullable, UTC minutes-since-midnight, both-or-neither enforced by `ModelRouteIn`'s model validator; a window where start > end wraps past midnight, e.g. DeepSeek's old V3/R1-era 16:30-00:30 UTC window) and `off_peak_discount_pct` (0-100, multiplies whichever tier rate is active). The same fields double for vision (`price_input/output_per_m_tier2`) and embed (`price_input_per_m_tier2`) exactly like their tier-1 counterparts already do.
|
||||
- **Overlay + computation.** `chat_overlay`/`embed_overlay` propagate all of these into the per-request cfg dict under `gateway_*` keys exactly like the existing price fields (section above); `usage.pricing_from_cfg` reads them generically (defaulting to `None`/`0`/disabled when absent), so **Layer A (the single global flat Pricing config fields) never gains these dimensions** - only a `gateway_models` route can enable them, preserving the "no matching route = byte-identical legacy behavior" guarantee. `usage.compute_cost` selects tier1 vs tier2 per rate component (`_tiered_rate`) based on whether `norm["prompt"]` (billable input tokens) exceeds the threshold, then applies the off-peak discount (`_effective_rate`/`_off_peak_active`, UTC wraparound-aware) to whichever rate was selected. **The response header format and `compute_cost`'s return shape (`total, input_cost, output_cost, native`) are unchanged** - this is purely an internal rate-selection step before the existing input/output split math runs; a native upstream `cost` (OpenRouter) still overrides the modeled total exactly as before.
|
||||
- **Migration.** New `gateway_models` columns are added via `has_column`/`create_column_by_example` in `routing.ensure_tables()` (the `CREATE TABLE IF NOT EXISTS` DDL string alone would never reach a pre-existing table - see the `database/CLAUDE.md` column-ensure idiom).
|
||||
- **Admin UI.** `/admin/gateway`'s model-route form has a "Tiered / off-peak pricing (optional)" subsection; off-peak start/end render as `<input type="time">` (converted to/from UTC minutes-of-day by `GatewayAdmin.js`), and the routes table shows `tiered`/`off-peak` badges when a route has either dimension configured.
|
||||
- **DeepSeek's real pricing is already the tier-1 shape, not a new dimension.** DeepSeek's actual API (verified against `api-docs.deepseek.com/quick_start/pricing`) bills three flat per-1M rates - cache-hit input, cache-miss input, output - with no current context-length tier or off-peak window for the V4 models; that shape was already fully modeled by the pre-existing `chat_cache_hit_per_m`/`chat_cache_miss_per_m`/`chat_output_per_m` fields before this section's tier2/off-peak fields existed. `routing.seed_default_deepseek_routes()` (called once from `database.migrate_ai_gateway_settings()` at the end of `init_db()`) idempotently inserts two ready-made routes - `deepseek-v4-flash` (`$0.0028`/`$0.14`/`$0.28` per 1M, 1M context) and `deepseek-v4-pro` (`$0.003625`/`$0.435`/`$0.87` per 1M, 1M context) - only when that `source_model` row does not already exist, so a caller or Devii can request either name explicitly and get correctly-priced, decoupled from whatever the single global `gateway_model` default happens to be set to (switching that global setting between the two real models does NOT retroactively fix the flat Pricing config fields - the seeded routes are the model-agnostic, always-correct way to reference a specific priced model). Neither seeded route sets the tier2/off-peak fields (DeepSeek does not use them today); an admin can add them later on the same row if DeepSeek (or any other provider routed here) introduces such pricing.
|
||||
|
||||
@@ -15,6 +15,10 @@ VISION_CACHE_SIZE_DEFAULT = 256
|
||||
EMBED_URL_DEFAULT = "https://openrouter.ai/api/v1/embeddings"
|
||||
EMBED_MODEL_DEFAULT = "qwen/qwen3-embedding-8b"
|
||||
|
||||
IMAGE_URL_DEFAULT = "https://openrouter.ai/api/v1/images"
|
||||
IMAGE_MODEL_DEFAULT = "black-forest-labs/flux.2-pro"
|
||||
IMAGE_PRICE_PER_CALL_DEFAULT = 0.04
|
||||
|
||||
VISION_INSTRUCTION = (
|
||||
"Describe this image in detail. Note objects, people, scene, any visible "
|
||||
"text, layout, colors, and anything else that could be relevant for "
|
||||
|
||||
@@ -13,11 +13,16 @@ from fastapi.responses import JSONResponse, Response, StreamingResponse
|
||||
from devplacepy import stealth
|
||||
from devplacepy.services.openai_gateway import config
|
||||
from devplacepy.services.openai_gateway.reliability import CircuitBreaker, retry_send
|
||||
from devplacepy.services.openai_gateway.routing import chat_overlay, embed_overlay
|
||||
from devplacepy.services.openai_gateway.routing import (
|
||||
chat_overlay,
|
||||
embed_overlay,
|
||||
image_overlay,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.system_message import apply_system_directives
|
||||
from devplacepy.services.openai_gateway.usage import (
|
||||
GatewayUsageLedger,
|
||||
classify_error,
|
||||
extract_image_usage,
|
||||
extract_params,
|
||||
parse_context_map,
|
||||
pricing_from_cfg,
|
||||
@@ -106,6 +111,7 @@ class GatewayRuntime:
|
||||
self.peak_in_flight = 0
|
||||
self.vision_calls = 0
|
||||
self.embed_calls = 0
|
||||
self.image_calls = 0
|
||||
self.last_status = 0
|
||||
self.last_latency_ms = 0
|
||||
|
||||
@@ -537,6 +543,171 @@ class GatewayRuntime:
|
||||
log(f"embed POST -> 200 ({timing['upstream_latency_ms']:.0f}ms)")
|
||||
return JSONResponse(content=data, headers=resp_headers)
|
||||
|
||||
async def handle_images(
|
||||
self, body: dict, cfg: dict, owner: tuple, user_agent: str, log=None
|
||||
):
|
||||
log = log or (lambda message: None)
|
||||
overlay = image_overlay(body.get("model"), cfg)
|
||||
if overlay:
|
||||
cfg = {**cfg, **overlay}
|
||||
log(
|
||||
f"routed image model {body.get('model')!r} -> {cfg['gateway_image_model']!r}"
|
||||
)
|
||||
if not cfg["gateway_image_enabled"]:
|
||||
from devplacepy.services.audit import record as audit
|
||||
from devplacepy.services.openai_gateway.usage import audit_actor_for
|
||||
|
||||
actor_kind, actor_uid, actor_role = audit_actor_for(owner[0], owner[1])
|
||||
audit.record_system(
|
||||
"ai.gateway.call",
|
||||
actor_kind=actor_kind,
|
||||
actor_uid=actor_uid,
|
||||
actor_role=actor_role,
|
||||
origin="api",
|
||||
result="denied",
|
||||
summary="image generation disabled",
|
||||
metadata={
|
||||
"backend": "image",
|
||||
"endpoint": "images/generations",
|
||||
"owner_kind": owner[0],
|
||||
"owner_id": owner[1],
|
||||
},
|
||||
)
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"error": {
|
||||
"message": "Image generation is disabled",
|
||||
"type": "images_disabled",
|
||||
}
|
||||
},
|
||||
)
|
||||
client, sem = self._ensure(cfg)
|
||||
pricing = pricing_from_cfg(cfg)
|
||||
context_map = parse_context_map(cfg.get("gateway_model_context_map"))
|
||||
params = extract_params(body)
|
||||
handle_start = time.monotonic()
|
||||
|
||||
requested = body.get("model")
|
||||
if (
|
||||
cfg["gateway_force_model"]
|
||||
or not requested
|
||||
or requested in ("molodetz-img-small", "molodetz-img")
|
||||
):
|
||||
model = cfg["gateway_image_model"]
|
||||
else:
|
||||
model = requested
|
||||
|
||||
payload = dict(body)
|
||||
payload["model"] = model
|
||||
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if cfg["gateway_image_key"]:
|
||||
headers["Authorization"] = f"Bearer {cfg['gateway_image_key']}"
|
||||
else:
|
||||
log(
|
||||
"No upstream image API key configured (gateway_image_key / gateway_api_key / OPENROUTER_API_KEY); upstream will likely reject the request"
|
||||
)
|
||||
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
sem,
|
||||
"POST",
|
||||
cfg["gateway_image_url"],
|
||||
headers,
|
||||
cfg,
|
||||
log,
|
||||
json_body=payload,
|
||||
)
|
||||
|
||||
base = {
|
||||
"owner_kind": owner[0],
|
||||
"owner_id": owner[1],
|
||||
"backend": "image",
|
||||
"endpoint": "images/generations",
|
||||
"model": model,
|
||||
"user_agent": user_agent,
|
||||
**params,
|
||||
**timing,
|
||||
}
|
||||
|
||||
def finalize(status_code, success, category, usage=None):
|
||||
base["total_latency_ms"] = round(
|
||||
(time.monotonic() - handle_start) * 1000, 3
|
||||
)
|
||||
base["gateway_overhead_ms"] = round(
|
||||
max(
|
||||
base["total_latency_ms"]
|
||||
- timing["upstream_latency_ms"]
|
||||
- timing["queue_wait_ms"],
|
||||
0.0,
|
||||
),
|
||||
3,
|
||||
)
|
||||
base["status_code"] = status_code
|
||||
base["success"] = success
|
||||
base["error_category"] = category
|
||||
base["usage"] = usage
|
||||
row = self._ledger.record(base, pricing, context_map)
|
||||
return usage_response_headers(row)
|
||||
|
||||
if timing["circuit_open"]:
|
||||
resp_headers = finalize(503, False, "circuit_open")
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"error": {
|
||||
"message": "Upstream temporarily unavailable",
|
||||
"type": "circuit_open",
|
||||
}
|
||||
},
|
||||
headers=resp_headers,
|
||||
)
|
||||
if exc is not None:
|
||||
resp_headers = finalize(502, False, classify_error(0, exc))
|
||||
return JSONResponse(
|
||||
status_code=502,
|
||||
content={
|
||||
"error": {
|
||||
"message": f"Upstream connection failed: {exc}",
|
||||
"type": "upstream_error",
|
||||
}
|
||||
},
|
||||
headers=resp_headers,
|
||||
)
|
||||
if resp.status_code != 200:
|
||||
resp_headers = finalize(
|
||||
resp.status_code,
|
||||
False,
|
||||
classify_error(resp.status_code, None, resp.text),
|
||||
)
|
||||
log(f"image upstream POST -> {resp.status_code}: {resp.text[:300]}")
|
||||
return JSONResponse(
|
||||
status_code=resp.status_code,
|
||||
content={"error": {"message": resp.text, "type": "upstream_error"}},
|
||||
headers=resp_headers,
|
||||
)
|
||||
try:
|
||||
data = resp.json()
|
||||
except ValueError:
|
||||
self.errors += 1
|
||||
resp_headers = finalize(502, False, "gateway")
|
||||
log("image upstream returned 200 but body was not valid JSON")
|
||||
return JSONResponse(
|
||||
status_code=502,
|
||||
content={
|
||||
"error": {
|
||||
"message": "invalid upstream response",
|
||||
"type": "upstream_error",
|
||||
}
|
||||
},
|
||||
headers=resp_headers,
|
||||
)
|
||||
self.image_calls += 1
|
||||
resp_headers = finalize(200, True, None, extract_image_usage(data))
|
||||
log(f"image POST -> 200 ({timing['upstream_latency_ms']:.0f}ms)")
|
||||
return JSONResponse(content=data, headers=resp_headers)
|
||||
|
||||
async def handle_passthrough(
|
||||
self,
|
||||
method: str,
|
||||
@@ -661,6 +832,7 @@ class GatewayRuntime:
|
||||
"peak_in_flight": self.peak_in_flight,
|
||||
"vision_calls": self.vision_calls,
|
||||
"embed_calls": self.embed_calls,
|
||||
"image_calls": self.image_calls,
|
||||
"last_status": self.last_status,
|
||||
"last_latency_ms": self.last_latency_ms,
|
||||
"pool": self._instances,
|
||||
|
||||
@@ -7,7 +7,7 @@ from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from pydantic import BaseModel, Field, field_validator, model_validator
|
||||
|
||||
from devplacepy.database import (
|
||||
bump_cache_version,
|
||||
@@ -21,7 +21,7 @@ logger = logging.getLogger(__name__)
|
||||
PROVIDERS_TABLE = "gateway_providers"
|
||||
MODELS_TABLE = "gateway_models"
|
||||
CACHE_NAME = "gateway_routing"
|
||||
KINDS = ("chat", "embed")
|
||||
KINDS = ("chat", "embed", "image")
|
||||
|
||||
_ROUTING_CACHE: dict = {}
|
||||
|
||||
@@ -47,6 +47,27 @@ def _embed_url_from_base(base_url: str) -> str:
|
||||
return base_url
|
||||
|
||||
|
||||
def _image_url_from_base(base_url: str) -> str:
|
||||
base_url = (base_url or "").strip()
|
||||
if not base_url:
|
||||
return ""
|
||||
if base_url.endswith("/chat/completions"):
|
||||
return base_url[: -len("/chat/completions")] + "/images"
|
||||
return base_url
|
||||
|
||||
|
||||
MODEL_TIER2_COLUMNS = (
|
||||
"context_tier_threshold_tokens",
|
||||
"price_cache_hit_per_m_tier2",
|
||||
"price_cache_miss_per_m_tier2",
|
||||
"price_output_per_m_tier2",
|
||||
"price_input_per_m_tier2",
|
||||
"off_peak_start_minute",
|
||||
"off_peak_end_minute",
|
||||
"off_peak_discount_pct",
|
||||
)
|
||||
|
||||
|
||||
def ensure_tables() -> None:
|
||||
db.query(
|
||||
"CREATE TABLE IF NOT EXISTS "
|
||||
@@ -62,8 +83,17 @@ def ensure_tables() -> None:
|
||||
"vision_model TEXT, context_window INTEGER DEFAULT 0, "
|
||||
"price_cache_hit_per_m REAL DEFAULT 0, price_cache_miss_per_m REAL DEFAULT 0, "
|
||||
"price_output_per_m REAL DEFAULT 0, price_input_per_m REAL DEFAULT 0, "
|
||||
"context_tier_threshold_tokens INTEGER DEFAULT 0, "
|
||||
"price_cache_hit_per_m_tier2 REAL, price_cache_miss_per_m_tier2 REAL, "
|
||||
"price_output_per_m_tier2 REAL, price_input_per_m_tier2 REAL, "
|
||||
"off_peak_start_minute INTEGER, off_peak_end_minute INTEGER, "
|
||||
"off_peak_discount_pct REAL DEFAULT 0, "
|
||||
"is_active INTEGER DEFAULT 1, created_at TEXT, updated_at TEXT)"
|
||||
)
|
||||
models_table = get_table(MODELS_TABLE)
|
||||
for column in MODEL_TIER2_COLUMNS:
|
||||
if not models_table.has_column(column):
|
||||
models_table.create_column_by_example(column, 0.0)
|
||||
try:
|
||||
db.query(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_providers_name ON "
|
||||
@@ -121,6 +151,14 @@ class ModelRouteIn(BaseModel):
|
||||
price_cache_miss_per_m: float = Field(default=0.0, ge=0)
|
||||
price_output_per_m: float = Field(default=0.0, ge=0)
|
||||
price_input_per_m: float = Field(default=0.0, ge=0)
|
||||
context_tier_threshold_tokens: int = Field(default=0, ge=0, le=100_000_000)
|
||||
price_cache_hit_per_m_tier2: Optional[float] = Field(default=None, ge=0)
|
||||
price_cache_miss_per_m_tier2: Optional[float] = Field(default=None, ge=0)
|
||||
price_output_per_m_tier2: Optional[float] = Field(default=None, ge=0)
|
||||
price_input_per_m_tier2: Optional[float] = Field(default=None, ge=0)
|
||||
off_peak_start_minute: Optional[int] = Field(default=None, ge=0, le=1439)
|
||||
off_peak_end_minute: Optional[int] = Field(default=None, ge=0, le=1439)
|
||||
off_peak_discount_pct: float = Field(default=0.0, ge=0, le=100)
|
||||
is_active: bool = True
|
||||
|
||||
@field_validator("source_model", "target_model")
|
||||
@@ -141,9 +179,19 @@ class ModelRouteIn(BaseModel):
|
||||
def _clean_kind(cls, value: str) -> str:
|
||||
value = (value or "chat").strip().lower()
|
||||
if value not in KINDS:
|
||||
raise ValueError("Kind must be 'chat' or 'embed'")
|
||||
raise ValueError("Kind must be 'chat', 'embed', or 'image'")
|
||||
return value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _check_off_peak_window(self) -> "ModelRouteIn":
|
||||
has_start = self.off_peak_start_minute is not None
|
||||
has_end = self.off_peak_end_minute is not None
|
||||
if has_start != has_end:
|
||||
raise ValueError(
|
||||
"Off-peak start and end minute must both be set, or both left blank"
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelRoute:
|
||||
@@ -158,9 +206,27 @@ class ModelRoute:
|
||||
price_cache_miss_per_m: float
|
||||
price_output_per_m: float
|
||||
price_input_per_m: float
|
||||
context_tier_threshold_tokens: int
|
||||
price_cache_hit_per_m_tier2: Optional[float]
|
||||
price_cache_miss_per_m_tier2: Optional[float]
|
||||
price_output_per_m_tier2: Optional[float]
|
||||
price_input_per_m_tier2: Optional[float]
|
||||
off_peak_start_minute: Optional[int]
|
||||
off_peak_end_minute: Optional[int]
|
||||
off_peak_discount_pct: float
|
||||
is_active: bool
|
||||
|
||||
|
||||
def _opt_float(row: dict, key: str) -> Optional[float]:
|
||||
value = row.get(key)
|
||||
return float(value) if value is not None else None
|
||||
|
||||
|
||||
def _opt_int(row: dict, key: str) -> Optional[int]:
|
||||
value = row.get(key)
|
||||
return int(value) if value is not None else None
|
||||
|
||||
|
||||
def _route_from_row(row: dict) -> ModelRoute:
|
||||
return ModelRoute(
|
||||
source_model=str(row.get("source_model") or ""),
|
||||
@@ -174,6 +240,16 @@ def _route_from_row(row: dict) -> ModelRoute:
|
||||
price_cache_miss_per_m=float(row.get("price_cache_miss_per_m") or 0.0),
|
||||
price_output_per_m=float(row.get("price_output_per_m") or 0.0),
|
||||
price_input_per_m=float(row.get("price_input_per_m") or 0.0),
|
||||
context_tier_threshold_tokens=int(
|
||||
row.get("context_tier_threshold_tokens") or 0
|
||||
),
|
||||
price_cache_hit_per_m_tier2=_opt_float(row, "price_cache_hit_per_m_tier2"),
|
||||
price_cache_miss_per_m_tier2=_opt_float(row, "price_cache_miss_per_m_tier2"),
|
||||
price_output_per_m_tier2=_opt_float(row, "price_output_per_m_tier2"),
|
||||
price_input_per_m_tier2=_opt_float(row, "price_input_per_m_tier2"),
|
||||
off_peak_start_minute=_opt_int(row, "off_peak_start_minute"),
|
||||
off_peak_end_minute=_opt_int(row, "off_peak_end_minute"),
|
||||
off_peak_discount_pct=float(row.get("off_peak_discount_pct") or 0.0),
|
||||
is_active=_as_bool(row.get("is_active", 1)),
|
||||
)
|
||||
|
||||
@@ -291,6 +367,14 @@ class ModelStore:
|
||||
"price_cache_miss_per_m": payload.price_cache_miss_per_m,
|
||||
"price_output_per_m": payload.price_output_per_m,
|
||||
"price_input_per_m": payload.price_input_per_m,
|
||||
"context_tier_threshold_tokens": payload.context_tier_threshold_tokens,
|
||||
"price_cache_hit_per_m_tier2": payload.price_cache_hit_per_m_tier2,
|
||||
"price_cache_miss_per_m_tier2": payload.price_cache_miss_per_m_tier2,
|
||||
"price_output_per_m_tier2": payload.price_output_per_m_tier2,
|
||||
"price_input_per_m_tier2": payload.price_input_per_m_tier2,
|
||||
"off_peak_start_minute": payload.off_peak_start_minute,
|
||||
"off_peak_end_minute": payload.off_peak_end_minute,
|
||||
"off_peak_discount_pct": payload.off_peak_discount_pct,
|
||||
"is_active": 1 if payload.is_active else 0,
|
||||
"updated_at": _now(),
|
||||
}
|
||||
@@ -318,6 +402,63 @@ class ModelStore:
|
||||
provider_store = ProviderStore()
|
||||
model_store = ModelStore()
|
||||
|
||||
DEEPSEEK_DEFAULT_ROUTES = (
|
||||
{
|
||||
"source_model": "deepseek-v4-flash",
|
||||
"target_model": "deepseek-v4-flash",
|
||||
"context_window": 1_048_576,
|
||||
"price_cache_hit_per_m": 0.0028,
|
||||
"price_cache_miss_per_m": 0.14,
|
||||
"price_output_per_m": 0.28,
|
||||
},
|
||||
{
|
||||
"source_model": "deepseek-v4-pro",
|
||||
"target_model": "deepseek-v4-pro",
|
||||
"context_window": 1_048_576,
|
||||
"price_cache_hit_per_m": 0.003625,
|
||||
"price_cache_miss_per_m": 0.435,
|
||||
"price_output_per_m": 0.87,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def seed_default_deepseek_routes() -> None:
|
||||
ensure_tables()
|
||||
table = get_table(MODELS_TABLE)
|
||||
seeded = False
|
||||
for defaults in DEEPSEEK_DEFAULT_ROUTES:
|
||||
if table.find_one(source_model=defaults["source_model"]):
|
||||
continue
|
||||
record = {
|
||||
"source_model": defaults["source_model"],
|
||||
"provider": "",
|
||||
"target_model": defaults["target_model"],
|
||||
"kind": "chat",
|
||||
"vision_provider": "",
|
||||
"vision_model": "",
|
||||
"context_window": defaults["context_window"],
|
||||
"price_cache_hit_per_m": defaults["price_cache_hit_per_m"],
|
||||
"price_cache_miss_per_m": defaults["price_cache_miss_per_m"],
|
||||
"price_output_per_m": defaults["price_output_per_m"],
|
||||
"price_input_per_m": 0.0,
|
||||
"context_tier_threshold_tokens": 0,
|
||||
"price_cache_hit_per_m_tier2": None,
|
||||
"price_cache_miss_per_m_tier2": None,
|
||||
"price_output_per_m_tier2": None,
|
||||
"price_input_per_m_tier2": None,
|
||||
"off_peak_start_minute": None,
|
||||
"off_peak_end_minute": None,
|
||||
"off_peak_discount_pct": 0.0,
|
||||
"is_active": 1,
|
||||
"created_at": _now(),
|
||||
"updated_at": _now(),
|
||||
}
|
||||
table.insert(record)
|
||||
seeded = True
|
||||
if seeded:
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_ROUTING_CACHE.clear()
|
||||
|
||||
|
||||
def _provider_overlay(name: str, base_key: str, url_key: str, overlay: dict) -> None:
|
||||
provider = provider_store.get(name)
|
||||
@@ -339,6 +480,13 @@ def chat_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[dic
|
||||
"gateway_price_cache_hit_per_m": route.price_cache_hit_per_m,
|
||||
"gateway_price_cache_miss_per_m": route.price_cache_miss_per_m,
|
||||
"gateway_price_output_per_m": route.price_output_per_m,
|
||||
"gateway_price_cache_hit_per_m_tier2": route.price_cache_hit_per_m_tier2,
|
||||
"gateway_price_cache_miss_per_m_tier2": route.price_cache_miss_per_m_tier2,
|
||||
"gateway_price_output_per_m_tier2": route.price_output_per_m_tier2,
|
||||
"gateway_context_tier_threshold_tokens": route.context_tier_threshold_tokens,
|
||||
"gateway_off_peak_start_minute": route.off_peak_start_minute,
|
||||
"gateway_off_peak_end_minute": route.off_peak_end_minute,
|
||||
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
|
||||
}
|
||||
if route.provider:
|
||||
_provider_overlay(
|
||||
@@ -355,6 +503,12 @@ def chat_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[dic
|
||||
overlay["gateway_vision_model"] = route.vision_model
|
||||
overlay["gateway_vision_price_input_per_m"] = route.price_input_per_m
|
||||
overlay["gateway_vision_price_output_per_m"] = route.price_output_per_m
|
||||
overlay["gateway_vision_price_input_per_m_tier2"] = (
|
||||
route.price_input_per_m_tier2
|
||||
)
|
||||
overlay["gateway_vision_price_output_per_m_tier2"] = (
|
||||
route.price_output_per_m_tier2
|
||||
)
|
||||
vision_provider = route.vision_provider or route.provider
|
||||
if vision_provider:
|
||||
_provider_overlay(
|
||||
@@ -371,6 +525,11 @@ def embed_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[di
|
||||
"gateway_force_model": True,
|
||||
"gateway_embed_model": route.target_model,
|
||||
"gateway_embed_price_input_per_m": route.price_input_per_m,
|
||||
"gateway_embed_price_input_per_m_tier2": route.price_input_per_m_tier2,
|
||||
"gateway_context_tier_threshold_tokens": route.context_tier_threshold_tokens,
|
||||
"gateway_off_peak_start_minute": route.off_peak_start_minute,
|
||||
"gateway_off_peak_end_minute": route.off_peak_end_minute,
|
||||
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
|
||||
}
|
||||
provider = provider_store.get(route.provider) if route.provider else None
|
||||
if provider:
|
||||
@@ -379,3 +538,129 @@ def embed_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[di
|
||||
if provider.get("api_key"):
|
||||
overlay["gateway_embed_key"] = provider["api_key"]
|
||||
return overlay
|
||||
|
||||
|
||||
def image_overlay(requested_model: Optional[str], base_cfg: dict) -> Optional[dict]:
|
||||
route = model_store.resolve(requested_model, "image")
|
||||
if route is None:
|
||||
return None
|
||||
overlay: dict = {
|
||||
"gateway_force_model": True,
|
||||
"gateway_image_model": route.target_model,
|
||||
"gateway_image_price_per_call": route.price_input_per_m,
|
||||
"gateway_off_peak_start_minute": route.off_peak_start_minute,
|
||||
"gateway_off_peak_end_minute": route.off_peak_end_minute,
|
||||
"gateway_off_peak_discount_pct": route.off_peak_discount_pct,
|
||||
}
|
||||
provider = provider_store.get(route.provider) if route.provider else None
|
||||
if provider:
|
||||
if provider.get("base_url"):
|
||||
overlay["gateway_image_url"] = _image_url_from_base(provider["base_url"])
|
||||
if provider.get("api_key"):
|
||||
overlay["gateway_image_key"] = provider["api_key"]
|
||||
return overlay
|
||||
|
||||
|
||||
OPENROUTER_PROVIDER_NAME = "openrouter"
|
||||
FLUX_TARGET_MODEL = "black-forest-labs/flux.2-pro"
|
||||
IMAGE_SOURCE_MODEL = "molodetz-img-small"
|
||||
RETIRED_IMAGE_MODELS = {
|
||||
"black-forest-labs/flux-1.1-pro": FLUX_TARGET_MODEL,
|
||||
"black-forest-labs/flux-schnell": "black-forest-labs/flux.2-klein-4b",
|
||||
}
|
||||
LEGACY_IMAGE_URL = "https://openrouter.ai/api/v1/images/generations"
|
||||
|
||||
|
||||
def seed_default_image_routes() -> None:
|
||||
import os
|
||||
|
||||
from devplacepy.services.openai_gateway import config as gw_config
|
||||
|
||||
ensure_tables()
|
||||
openrouter_key = os.environ.get("OPENROUTER_API_KEY", "")
|
||||
if openrouter_key:
|
||||
existing_provider = provider_store.get(OPENROUTER_PROVIDER_NAME)
|
||||
if existing_provider is None:
|
||||
provider_store.set(
|
||||
ProviderIn(
|
||||
name=OPENROUTER_PROVIDER_NAME,
|
||||
base_url="https://openrouter.ai/api/v1/chat/completions",
|
||||
api_key=openrouter_key,
|
||||
is_active=True,
|
||||
)
|
||||
)
|
||||
models_table = get_table(MODELS_TABLE)
|
||||
if models_table.find_one(source_model=IMAGE_SOURCE_MODEL):
|
||||
return
|
||||
record = {
|
||||
"source_model": IMAGE_SOURCE_MODEL,
|
||||
"provider": OPENROUTER_PROVIDER_NAME if openrouter_key else "",
|
||||
"target_model": FLUX_TARGET_MODEL,
|
||||
"kind": "image",
|
||||
"vision_provider": "",
|
||||
"vision_model": "",
|
||||
"context_window": 0,
|
||||
"price_cache_hit_per_m": 0.0,
|
||||
"price_cache_miss_per_m": 0.0,
|
||||
"price_output_per_m": 0.0,
|
||||
"price_input_per_m": gw_config.IMAGE_PRICE_PER_CALL_DEFAULT,
|
||||
"context_tier_threshold_tokens": 0,
|
||||
"price_cache_hit_per_m_tier2": None,
|
||||
"price_cache_miss_per_m_tier2": None,
|
||||
"price_output_per_m_tier2": None,
|
||||
"price_input_per_m_tier2": None,
|
||||
"off_peak_start_minute": None,
|
||||
"off_peak_end_minute": None,
|
||||
"off_peak_discount_pct": 0.0,
|
||||
"is_active": 1,
|
||||
"created_at": _now(),
|
||||
"updated_at": _now(),
|
||||
}
|
||||
models_table.insert(record)
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_ROUTING_CACHE.clear()
|
||||
|
||||
|
||||
def migrate_retired_image_gateway() -> None:
|
||||
from devplacepy.database import get_setting, set_setting
|
||||
from devplacepy.services.openai_gateway import config as gw_config
|
||||
|
||||
ensure_tables()
|
||||
models_table = get_table(MODELS_TABLE)
|
||||
now = _now()
|
||||
changed = False
|
||||
for row in models_table.find(kind="image"):
|
||||
target = str(row.get("target_model") or "")
|
||||
replacement = RETIRED_IMAGE_MODELS.get(target)
|
||||
if not replacement:
|
||||
continue
|
||||
models_table.update(
|
||||
{
|
||||
"source_model": row["source_model"],
|
||||
"target_model": replacement,
|
||||
"updated_at": now,
|
||||
},
|
||||
["source_model"],
|
||||
)
|
||||
changed = True
|
||||
image_url = get_setting("gateway_image_url", "")
|
||||
if image_url in (LEGACY_IMAGE_URL, ""):
|
||||
set_setting("gateway_image_url", gw_config.IMAGE_URL_DEFAULT)
|
||||
changed = True
|
||||
elif image_url.endswith("/images/generations"):
|
||||
set_setting(
|
||||
"gateway_image_url",
|
||||
image_url[: -len("/generations")],
|
||||
)
|
||||
changed = True
|
||||
image_model = get_setting("gateway_image_model", "")
|
||||
replacement = RETIRED_IMAGE_MODELS.get(image_model)
|
||||
if replacement:
|
||||
set_setting("gateway_image_model", replacement)
|
||||
changed = True
|
||||
elif not image_model:
|
||||
set_setting("gateway_image_model", gw_config.IMAGE_MODEL_DEFAULT)
|
||||
changed = True
|
||||
if changed:
|
||||
bump_cache_version(CACHE_NAME)
|
||||
_ROUTING_CACHE.clear()
|
||||
|
||||
@@ -172,6 +172,38 @@ class GatewayService(BaseService):
|
||||
help="The key currently in use; falls back to the vision/OPENROUTER key on boot (embeddings default to OpenRouter). Editable.",
|
||||
group="Embeddings",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_image_enabled",
|
||||
"Image generation",
|
||||
type="bool",
|
||||
default=True,
|
||||
help="Expose image generation at /openai/v1/images/generations.",
|
||||
group="Images",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_image_url",
|
||||
"Images URL",
|
||||
type="url",
|
||||
default=config.IMAGE_URL_DEFAULT,
|
||||
help="OpenAI-compatible images/generations endpoint requests are forwarded to.",
|
||||
group="Images",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_image_model",
|
||||
"Images model",
|
||||
type="str",
|
||||
default=config.IMAGE_MODEL_DEFAULT,
|
||||
help="Image model sent upstream. Clients request it as molodetz-img-small.",
|
||||
group="Images",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_image_key",
|
||||
"Images API key",
|
||||
type="str",
|
||||
default="",
|
||||
help="The key currently in use; falls back to gateway_api_key / OPENROUTER_API_KEY on boot. Editable.",
|
||||
group="Images",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_require_auth",
|
||||
"Require authentication",
|
||||
@@ -267,6 +299,15 @@ class GatewayService(BaseService):
|
||||
help="Fallback only; used when the embeddings upstream returns no native cost.",
|
||||
group="Pricing",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_image_price_per_call",
|
||||
"Images price / call ($)",
|
||||
type="float",
|
||||
default=config.IMAGE_PRICE_PER_CALL_DEFAULT,
|
||||
minimum=0,
|
||||
help="Fallback only; used when the image upstream returns no native cost.",
|
||||
group="Pricing",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_rsearch_cost_per_call",
|
||||
"rsearch cost / call ($)",
|
||||
@@ -355,6 +396,17 @@ class GatewayService(BaseService):
|
||||
or cfg["gateway_vision_key"]
|
||||
or os.environ.get("OPENROUTER_API_KEY", "")
|
||||
)
|
||||
cfg["gateway_image_key"] = (
|
||||
cfg["gateway_image_key"]
|
||||
or cfg["gateway_api_key"]
|
||||
or os.environ.get("OPENROUTER_API_KEY", "")
|
||||
)
|
||||
if not cfg.get("gateway_image_url"):
|
||||
upstream = cfg.get("gateway_upstream_url", "")
|
||||
if upstream.endswith("/chat/completions"):
|
||||
cfg["gateway_image_url"] = (
|
||||
upstream[: -len("/chat/completions")] + "/images"
|
||||
)
|
||||
return cfg
|
||||
|
||||
def authorize(self, request: Request) -> bool:
|
||||
@@ -429,6 +481,18 @@ class GatewayService(BaseService):
|
||||
return await runtime.handle_embeddings(
|
||||
body, cfg, owner, user_agent, self.log
|
||||
)
|
||||
if subpath == "images/generations" and request.method == "POST":
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
self.log("Rejected images request: invalid JSON body")
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON body")
|
||||
if not isinstance(body, dict):
|
||||
self.log("Rejected images request: JSON body was not an object")
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON body")
|
||||
return await runtime.handle_images(
|
||||
body, cfg, owner, user_agent, self.log
|
||||
)
|
||||
body = await request.body()
|
||||
content_type = request.headers.get("content-type", "")
|
||||
return await runtime.handle_passthrough(
|
||||
@@ -473,6 +537,7 @@ class GatewayService(BaseService):
|
||||
"peak_in_flight": 0,
|
||||
"vision_calls": 0,
|
||||
"embed_calls": 0,
|
||||
"image_calls": 0,
|
||||
"last_status": 0,
|
||||
"last_latency_ms": 0,
|
||||
"pool": 0,
|
||||
@@ -485,12 +550,14 @@ class GatewayService(BaseService):
|
||||
{"label": "In flight", "value": m["in_flight"]},
|
||||
{"label": "Vision calls", "value": m["vision_calls"]},
|
||||
{"label": "Embed calls", "value": m["embed_calls"]},
|
||||
{"label": "Image calls", "value": m["image_calls"]},
|
||||
{"label": "Last status", "value": m["last_status"] or "-"},
|
||||
{"label": "Last latency", "value": f"{m['last_latency_ms']} ms"},
|
||||
{"label": "Pool size", "value": m["pool"]},
|
||||
{"label": "Circuit", "value": "open" if m["circuit_open"] else "closed"},
|
||||
{"label": "Model", "value": cfg["gateway_model"]},
|
||||
{"label": "Embed model", "value": cfg["gateway_embed_model"]},
|
||||
{"label": "Image model", "value": cfg["gateway_image_model"]},
|
||||
{"label": "Requests 24h", "value": s["requests"]},
|
||||
{"label": "Success 24h", "value": f"{s['success_pct']}%"},
|
||||
{"label": "Error rate 24h", "value": f"{s['error_pct']}%"},
|
||||
|
||||
@@ -36,6 +36,31 @@ class Pricing:
|
||||
vision_input_per_m: float
|
||||
vision_output_per_m: float
|
||||
embed_input_per_m: float
|
||||
image_per_call: float = 0.0
|
||||
chat_cache_hit_per_m_tier2: Optional[float] = None
|
||||
chat_cache_miss_per_m_tier2: Optional[float] = None
|
||||
chat_output_per_m_tier2: Optional[float] = None
|
||||
vision_input_per_m_tier2: Optional[float] = None
|
||||
vision_output_per_m_tier2: Optional[float] = None
|
||||
embed_input_per_m_tier2: Optional[float] = None
|
||||
context_tier_threshold_tokens: int = 0
|
||||
off_peak_start_minute: Optional[int] = None
|
||||
off_peak_end_minute: Optional[int] = None
|
||||
off_peak_discount_pct: float = 0.0
|
||||
|
||||
|
||||
def _cfg_float(cfg: dict, key: str) -> Optional[float]:
|
||||
value = cfg.get(key)
|
||||
if isinstance(value, (int, float)) and not isinstance(value, bool):
|
||||
return float(value)
|
||||
return None
|
||||
|
||||
|
||||
def _cfg_int(cfg: dict, key: str) -> Optional[int]:
|
||||
value = cfg.get(key)
|
||||
if isinstance(value, (int, float)) and not isinstance(value, bool):
|
||||
return int(value)
|
||||
return None
|
||||
|
||||
|
||||
def pricing_from_cfg(cfg: dict) -> Pricing:
|
||||
@@ -71,6 +96,34 @@ def pricing_from_cfg(cfg: dict) -> Pricing:
|
||||
config.EMBED_PRICE_INPUT_PER_M_DEFAULT,
|
||||
)
|
||||
),
|
||||
image_per_call=float(
|
||||
cfg.get(
|
||||
"gateway_image_price_per_call",
|
||||
config.IMAGE_PRICE_PER_CALL_DEFAULT,
|
||||
)
|
||||
),
|
||||
chat_cache_hit_per_m_tier2=_cfg_float(
|
||||
cfg, "gateway_price_cache_hit_per_m_tier2"
|
||||
),
|
||||
chat_cache_miss_per_m_tier2=_cfg_float(
|
||||
cfg, "gateway_price_cache_miss_per_m_tier2"
|
||||
),
|
||||
chat_output_per_m_tier2=_cfg_float(cfg, "gateway_price_output_per_m_tier2"),
|
||||
vision_input_per_m_tier2=_cfg_float(
|
||||
cfg, "gateway_vision_price_input_per_m_tier2"
|
||||
),
|
||||
vision_output_per_m_tier2=_cfg_float(
|
||||
cfg, "gateway_vision_price_output_per_m_tier2"
|
||||
),
|
||||
embed_input_per_m_tier2=_cfg_float(
|
||||
cfg, "gateway_embed_price_input_per_m_tier2"
|
||||
),
|
||||
context_tier_threshold_tokens=int(
|
||||
cfg.get("gateway_context_tier_threshold_tokens", 0) or 0
|
||||
),
|
||||
off_peak_start_minute=_cfg_int(cfg, "gateway_off_peak_start_minute"),
|
||||
off_peak_end_minute=_cfg_int(cfg, "gateway_off_peak_end_minute"),
|
||||
off_peak_discount_pct=float(cfg.get("gateway_off_peak_discount_pct", 0.0) or 0.0),
|
||||
)
|
||||
|
||||
|
||||
@@ -88,6 +141,18 @@ def parse_context_map(raw: Any) -> dict[str, int]:
|
||||
return dict(config.MODEL_CONTEXT_MAP_DEFAULT)
|
||||
|
||||
|
||||
def extract_image_usage(data: Optional[dict]) -> dict:
|
||||
data = data or {}
|
||||
usage = data.get("usage") if isinstance(data.get("usage"), dict) else {}
|
||||
cost = usage.get("cost")
|
||||
if cost is None:
|
||||
cost = data.get("cost")
|
||||
result: dict = {}
|
||||
if isinstance(cost, (int, float)) and not isinstance(cost, bool):
|
||||
result["cost"] = float(cost)
|
||||
return result
|
||||
|
||||
|
||||
def normalize_usage(usage: Optional[dict]) -> dict:
|
||||
usage = usage or {}
|
||||
prompt = int(usage.get("prompt_tokens", 0) or 0)
|
||||
@@ -117,21 +182,108 @@ def normalize_usage(usage: Optional[dict]) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _off_peak_active(pricing: Pricing, now: Optional[datetime] = None) -> bool:
|
||||
if pricing.off_peak_start_minute is None or pricing.off_peak_end_minute is None:
|
||||
return False
|
||||
moment = now or _now()
|
||||
minute_of_day = moment.hour * 60 + moment.minute
|
||||
start, end = pricing.off_peak_start_minute, pricing.off_peak_end_minute
|
||||
if start == end:
|
||||
return True
|
||||
if start < end:
|
||||
return start <= minute_of_day < end
|
||||
return minute_of_day >= start or minute_of_day < end
|
||||
|
||||
|
||||
def _tiered_rate(base: float, tier2: Optional[float], use_tier2: bool) -> float:
|
||||
if use_tier2 and tier2 is not None:
|
||||
return tier2
|
||||
return base
|
||||
|
||||
|
||||
def _effective_rate(
|
||||
base: float,
|
||||
tier2: Optional[float],
|
||||
use_tier2: bool,
|
||||
pricing: Pricing,
|
||||
off_peak: bool,
|
||||
) -> float:
|
||||
rate = _tiered_rate(base, tier2, use_tier2)
|
||||
if off_peak and pricing.off_peak_discount_pct > 0:
|
||||
rate = rate * (1 - min(pricing.off_peak_discount_pct, 100.0) / 100.0)
|
||||
return rate
|
||||
|
||||
|
||||
def compute_cost(
|
||||
usage: dict, norm: dict, pricing: Pricing, backend: str
|
||||
usage: dict,
|
||||
norm: dict,
|
||||
pricing: Pricing,
|
||||
backend: str,
|
||||
now: Optional[datetime] = None,
|
||||
) -> tuple[float, float, float, bool]:
|
||||
threshold = pricing.context_tier_threshold_tokens
|
||||
use_tier2 = bool(threshold > 0 and norm["prompt"] > threshold)
|
||||
off_peak = _off_peak_active(pricing, now)
|
||||
if backend == "vision":
|
||||
input_cost = norm["prompt"] / PER_MILLION * pricing.vision_input_per_m
|
||||
output_cost = norm["completion"] / PER_MILLION * pricing.vision_output_per_m
|
||||
input_rate = _effective_rate(
|
||||
pricing.vision_input_per_m,
|
||||
pricing.vision_input_per_m_tier2,
|
||||
use_tier2,
|
||||
pricing,
|
||||
off_peak,
|
||||
)
|
||||
output_rate = _effective_rate(
|
||||
pricing.vision_output_per_m,
|
||||
pricing.vision_output_per_m_tier2,
|
||||
use_tier2,
|
||||
pricing,
|
||||
off_peak,
|
||||
)
|
||||
input_cost = norm["prompt"] / PER_MILLION * input_rate
|
||||
output_cost = norm["completion"] / PER_MILLION * output_rate
|
||||
elif backend == "embed":
|
||||
input_cost = norm["prompt"] / PER_MILLION * pricing.embed_input_per_m
|
||||
input_rate = _effective_rate(
|
||||
pricing.embed_input_per_m,
|
||||
pricing.embed_input_per_m_tier2,
|
||||
use_tier2,
|
||||
pricing,
|
||||
off_peak,
|
||||
)
|
||||
input_cost = norm["prompt"] / PER_MILLION * input_rate
|
||||
output_cost = 0.0
|
||||
elif backend == "image":
|
||||
rate = pricing.image_per_call
|
||||
if off_peak and pricing.off_peak_discount_pct > 0:
|
||||
rate = rate * (1 - min(pricing.off_peak_discount_pct, 100.0) / 100.0)
|
||||
input_cost = rate
|
||||
output_cost = 0.0
|
||||
else:
|
||||
input_cost = (
|
||||
norm["cache_hit"] / PER_MILLION * pricing.chat_cache_hit_per_m
|
||||
+ norm["cache_miss"] / PER_MILLION * pricing.chat_cache_miss_per_m
|
||||
hit_rate = _effective_rate(
|
||||
pricing.chat_cache_hit_per_m,
|
||||
pricing.chat_cache_hit_per_m_tier2,
|
||||
use_tier2,
|
||||
pricing,
|
||||
off_peak,
|
||||
)
|
||||
output_cost = norm["completion"] / PER_MILLION * pricing.chat_output_per_m
|
||||
miss_rate = _effective_rate(
|
||||
pricing.chat_cache_miss_per_m,
|
||||
pricing.chat_cache_miss_per_m_tier2,
|
||||
use_tier2,
|
||||
pricing,
|
||||
off_peak,
|
||||
)
|
||||
output_rate = _effective_rate(
|
||||
pricing.chat_output_per_m,
|
||||
pricing.chat_output_per_m_tier2,
|
||||
use_tier2,
|
||||
pricing,
|
||||
off_peak,
|
||||
)
|
||||
input_cost = (
|
||||
norm["cache_hit"] / PER_MILLION * hit_rate
|
||||
+ norm["cache_miss"] / PER_MILLION * miss_rate
|
||||
)
|
||||
output_cost = norm["completion"] / PER_MILLION * output_rate
|
||||
native = usage.get("cost") if isinstance(usage, dict) else None
|
||||
if isinstance(native, (int, float)) and not isinstance(native, bool):
|
||||
total = max(0.0, float(native))
|
||||
|
||||
Reference in New Issue
Block a user