forked from retoor/devplacepy
retoorandClaude Sonnet 5
a693a6f4d8
Admin-unlimited Dev Workspaces: an admin-owned workspace is now exempt from the max-workspace-count limit, the max-tunnel-count limit, and the whole idle-stop/idle-warn/retention-delete lifecycle. Resolved once in quota.resolve() as Limits.unlimited (owner uid checked against get_admin_uids()), consumed at the three enforcement points (provision.ensure, provision.publish_tunnel, WorkspaceService._advance_lifecycle). Also hardens get_admin_uids()/get_primary_admin_uid() against a partially-schemaed users table (uid/role column guard), which a fresh test/init_db() path could hit. AI gateway per-model automatic fallback: any gateway_models route (chat/embed/image) can now name a fallback_model, picked on /admin/gateway from a select box of other configured public model names of the same kind only (never an internal upstream model id). When a route fails after its own retries are exhausted, the gateway retries once, automatically, against the fallback's own provider/pricing/key, before any bytes reach the client (including for a streaming response). One hop only, no chains or cycles; self-reference and cross-kind fallbacks are rejected at write time. AI gateway real upstream streaming and thinking-default control: stream:true is now forwarded to the upstream and relayed to the client as real SSE chunks (measured TTFT/inter-token latency) instead of a simulated split response, and every chat/vision call explicitly disables model "thinking" by default (admin-overridable via gateway_thinking), with per-dialect handling for DeepSeek, OpenRouter, and Ollama. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TjdKTWgWpW2SMNW8SFqxz5
296 lines
15 KiB
Python
296 lines
15 KiB
Python
# retoor <retoor@molodetz.nl>
|
|
|
|
from .._shared import endpoint, field
|
|
|
|
GROUP = {
|
|
"slug": "gateway",
|
|
"title": "OpenAI Gateway",
|
|
"intro": """
|
|
# OpenAI Gateway
|
|
|
|
An OpenAI-compatible proxy mounted at `/openai/v1`. It forwards requests to a configured
|
|
upstream using the gateway's own credentials, so no DevPlace key is required - but an
|
|
administrator must enable the `openai` service first. Point any OpenAI-compatible client at
|
|
`{{ base }}/openai/v1`.
|
|
|
|
This gateway is the **single point of truth for AI** on the platform. Every other DevPlace
|
|
service (news, bots, Devii) calls it by default instead of an external provider, sends the
|
|
generic model name `molodetz`, and authenticates with an internal key that is auto-generated
|
|
on first boot. The real provider URLs, models, and keys (DeepSeek, OpenRouter) live only here,
|
|
so an operator switches providers or backends in one place. `DEEPSEEK_API_KEY` and
|
|
`OPENROUTER_API_KEY` are migrated into the editable settings on boot and the value in use is
|
|
shown. Because `Force model` is on by default, the upstream always receives the configured
|
|
model regardless of what a client (or `molodetz`) requests.
|
|
|
|
The gateway also performs **vision** augmentation: when a request includes an image, the gateway
|
|
describes the image with a configured vision model and rewrites it to text, so a vision-less
|
|
upstream still works. The vision model, URL, and key are configured alongside the other gateway
|
|
settings.
|
|
|
|
The gateway additionally serves **text embeddings** at `/openai/v1/embeddings`. Clients request the
|
|
generic model `molodetz~embed`, which the gateway maps to the configured embedding model (OpenRouter's
|
|
Qwen3 8B embedding model by default). Usage and cost are tracked per call exactly like chat and vision.
|
|
|
|
The gateway also serves **image generation** at `/openai/v1/images/generations`. Clients request the
|
|
generic model `molodetz-img-small`, which the gateway maps to the configured image model (OpenRouter's
|
|
Flux 1.1 Pro by default). Cost is tracked per call with a flat per-image price when the upstream
|
|
returns no native cost.
|
|
|
|
## Quick start
|
|
|
|
Copy the command below and paste it into a terminal. If you are signed in the `{{ api_key }}`
|
|
and `{{ app_reference }}` placeholders are already filled in with your own values; otherwise
|
|
replace them with the API key from your [profile](/profile) page and any application identifier.
|
|
|
|
```bash
|
|
curl -X POST "{{ base }}/openai/v1/chat/completions" \
|
|
-H "Authorization: Bearer {{ api_key }}" \
|
|
-H "X-App-Reference: {{ app_reference }}" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "molodetz",
|
|
"messages": [{"role": "user", "content": "Hello, how are you?"}]
|
|
}'
|
|
```
|
|
|
|
The response carries `X-Gateway-*` headers with token counts and dollar cost for the call.
|
|
For streaming, add `"stream": true` to the JSON body.
|
|
|
|
## Model routing and providers
|
|
|
|
On top of the single default upstream above, an administrator can register additional named
|
|
**providers** and map any number of requested **model names** onto them, so one gateway can front
|
|
many models across many backends. A model route binds a source model name (what a client sends) to a
|
|
target provider and upstream model, and carries:
|
|
|
|
- its **own pricing economy** (input, output, and cache-hit / cache-miss prices per million tokens),
|
|
used to compute that call's cost when the upstream returns no native cost;
|
|
- an optional **vision model**, which turns on the image-to-text merge for that route (so a text-only
|
|
model can answer about images);
|
|
- an optional **context window** used for the context-utilization header.
|
|
|
|
Resolution is transparent to clients: when the requested `model` matches an active route, the gateway
|
|
forwards to that route's provider and target model and meters the call against the route's economy.
|
|
When it matches no route, the request falls through to the default upstream unchanged (so `molodetz`,
|
|
`molodetz~embed`, and any existing client keep working exactly as before). Providers and routes are
|
|
managed by administrators on the **Gateway** page (`/admin/gateway`).
|
|
|
|
## Per-call cost and usage headers
|
|
|
|
Every NON-STREAMING gateway response - chat, embeddings, images, and passthrough, on both success
|
|
and error - carries `X-Gateway-*` response headers describing that single call, so a client can read
|
|
its own token usage and dollar cost directly from the response with no extra request. A streaming
|
|
chat response (`"stream": true`) is the one exception: it carries only `X-Gateway-Model`,
|
|
`X-Gateway-Backend`, and `X-App-Reference` - HTTP headers must be sent before the body, and cost/token
|
|
counts for a streamed call are only known once the stream ends, so they cannot be response headers on
|
|
that same response. The call is still fully metered server-side, and a client that sends
|
|
`"stream_options": {"include_usage": true}` still receives the upstream's real `usage` object on the
|
|
final SSE chunk, exactly as the underlying provider's own streaming API works - it is just not
|
|
summarized into headers.
|
|
|
|
| Header | Meaning |
|
|
|--------|---------|
|
|
| `X-Gateway-Model` | Upstream model actually used for the call |
|
|
| `X-Gateway-Backend` | Backend that served it: `chat`, `embed`, `image`, or passthrough |
|
|
| `X-Gateway-Prompt-Tokens` | Input (prompt) tokens |
|
|
| `X-Gateway-Completion-Tokens` | Output (completion) tokens |
|
|
| `X-Gateway-Total-Tokens` | Total tokens (prompt + completion) |
|
|
| `X-Gateway-Cache-Hit-Tokens` | Prompt tokens served from the upstream prompt cache |
|
|
| `X-Gateway-Cache-Miss-Tokens` | Prompt tokens not served from cache |
|
|
| `X-Gateway-Reasoning-Tokens` | Reasoning tokens, when the model reports them |
|
|
| `X-Gateway-Cost-USD` | Total cost of the call in US dollars |
|
|
| `X-Gateway-Input-Cost-USD` | Input portion of the cost in US dollars |
|
|
| `X-Gateway-Output-Cost-USD` | Output portion of the cost in US dollars |
|
|
| `X-Gateway-Cost-Native` | `1` if the dollar cost is the upstream's own reported cost, `0` if computed from the configured per-million pricing |
|
|
| `X-Gateway-Tokens-Per-Second` | Output tokens per second for the call |
|
|
| `X-Gateway-Upstream-Latency-Ms` | Upstream round-trip latency in milliseconds |
|
|
| `X-Gateway-Total-Latency-Ms` | Full end-to-end gateway time for the call in milliseconds |
|
|
| `X-Gateway-Gateway-Overhead-Ms` | Gateway processing time minus the upstream and queue wait, in milliseconds |
|
|
| `X-Gateway-Queue-Wait-Ms` | Time spent waiting on the concurrency semaphore before dispatch, in milliseconds |
|
|
| `X-Gateway-Connect-Ms` | Upstream connection establishment time in milliseconds |
|
|
| `X-Gateway-Context-Window` | The model's context window in tokens, when known |
|
|
| `X-Gateway-Context-Utilization` | Total tokens as a fraction of the context window, when known |
|
|
|
|
Dollar costs use the upstream's native `cost` field when it returns one
|
|
(`X-Gateway-Cost-Native: 1`); otherwise they are computed from the per-million prices of the matched
|
|
model route, falling back to the prices configured on the `openai` service when no route matches. The
|
|
denied paths that make no upstream call (embeddings or image generation disabled) return no usage headers.
|
|
|
|
## Request header `X-App-Reference`
|
|
|
|
Clients **SHOULD** send an `X-App-Reference` header to identify themselves for cost attribution.
|
|
The value is a free-form slug (max 30 characters, letters, digits, `_`, `.`, `-`). When missing or
|
|
invalid, the gateway defaults to `default`. The value is recorded in every usage ledger row and can
|
|
be queried alongside owner-kind and owner-id to attribute spending per application.
|
|
|
|
```
|
|
X-App-Reference: devplace-devii-v-1-0-0
|
|
```
|
|
|
|
Administrators enable and configure this gateway under [Background Services](/docs/services.html)
|
|
(the `openai` service).
|
|
|
|
The gateway is exempt from rate limiting, but every other endpoint follows the shared
|
|
[Conventions & Errors](/docs/conventions.html); see [Authentication](/docs/authentication.html)
|
|
for signing DevPlace's own requests.
|
|
""",
|
|
"endpoints": [
|
|
endpoint(
|
|
id="gateway-chat",
|
|
method="POST",
|
|
path="/openai/v1/chat/completions",
|
|
title="Chat completions",
|
|
summary="OpenAI-compatible chat completion. Supports streaming.",
|
|
auth="user",
|
|
encoding="json",
|
|
params=[
|
|
field(
|
|
"model",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"gpt-4o-mini",
|
|
"Model id. When it matches a configured model route the gateway forwards to that route's provider and upstream model; otherwise it falls back to the configured default upstream model.",
|
|
),
|
|
field(
|
|
"messages",
|
|
"json",
|
|
"string",
|
|
True,
|
|
'[{"role":"user","content":"Hello"}]',
|
|
"Chat messages array.",
|
|
),
|
|
field(
|
|
"stream",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"false",
|
|
"Set true for a streamed SSE response.",
|
|
),
|
|
field(
|
|
"think",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"false",
|
|
"Optional thinking override. Omitted: the gateway disables thinking (fast path). true / high / medium / low enables it; false disables it. DeepSeek-native `thinking.type` and OpenRouter `reasoning.effort` are also accepted.",
|
|
),
|
|
],
|
|
notes=[
|
|
"Returns `503` when the gateway service is not running.",
|
|
"A non-streaming response carries the `X-Gateway-*` token and dollar-cost headers (see Per-call cost and usage headers above). A streaming response (`stream: true`) is forwarded from the upstream in real time and carries only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference` - request `stream_options: {\"include_usage\": true}` to receive the real token usage on the final SSE chunk instead.",
|
|
"If `model` matches a configured model route it is forwarded to that route's provider, upstream model, and per-model pricing (with an optional vision model); otherwise it falls through to the default upstream (see Model routing and providers above).",
|
|
"Thinking is disabled by default. Pass `think: true` (or `thinking: {\"type\": \"enabled\"}`) to turn it on for that call.",
|
|
],
|
|
),
|
|
endpoint(
|
|
id="gateway-embeddings",
|
|
method="POST",
|
|
path="/openai/v1/embeddings",
|
|
title="Embeddings",
|
|
summary="OpenAI-compatible text embeddings. Request model molodetz~embed.",
|
|
auth="user",
|
|
encoding="json",
|
|
params=[
|
|
field(
|
|
"model",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"molodetz~embed",
|
|
"Embedding model id; the gateway maps molodetz~embed to the configured model, or to a matching embed model route's provider and target model.",
|
|
),
|
|
field(
|
|
"input",
|
|
"json",
|
|
"string",
|
|
True,
|
|
'"some text to embed"',
|
|
"String or array of strings to embed.",
|
|
),
|
|
field(
|
|
"dimensions",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"4096",
|
|
"Optional output vector size (Matryoshka, 32-4096).",
|
|
),
|
|
],
|
|
notes=[
|
|
"Returns `503` when the gateway service is not running or embeddings are disabled.",
|
|
"Every response carries the `X-Gateway-*` token and dollar-cost headers (see Per-call cost and usage headers above).",
|
|
"If `model` matches a configured embed model route it is forwarded to that route's provider and upstream model; otherwise it falls back to the configured default embedding model.",
|
|
],
|
|
),
|
|
endpoint(
|
|
id="gateway-images",
|
|
method="POST",
|
|
path="/openai/v1/images/generations",
|
|
title="Image generation",
|
|
summary="OpenAI-compatible image generation. Request model molodetz-img-small.",
|
|
auth="user",
|
|
encoding="json",
|
|
params=[
|
|
field(
|
|
"model",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"molodetz-img-small",
|
|
"Image model id; the gateway maps molodetz-img-small to the configured model, or to a matching image model route's provider and target model.",
|
|
),
|
|
field(
|
|
"prompt",
|
|
"json",
|
|
"string",
|
|
True,
|
|
'"a decorative developer award emblem"',
|
|
"Text prompt describing the image to generate.",
|
|
),
|
|
field(
|
|
"size",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"512x512",
|
|
"Output dimensions (provider-dependent).",
|
|
),
|
|
field(
|
|
"response_format",
|
|
"json",
|
|
"string",
|
|
False,
|
|
"b64_json",
|
|
"Return format: url or b64_json.",
|
|
),
|
|
],
|
|
notes=[
|
|
"Returns `503` when the gateway service is not running or image generation is disabled.",
|
|
"Every response carries the `X-Gateway-*` token and dollar-cost headers (see Per-call cost and usage headers above).",
|
|
"If `model` matches a configured image model route it is forwarded to that route's provider and upstream model; otherwise it falls back to the configured default image model.",
|
|
],
|
|
),
|
|
endpoint(
|
|
id="gateway-passthrough",
|
|
method="POST",
|
|
path="/openai/v1/{path}",
|
|
title="Passthrough",
|
|
summary="Any other /v1 path is forwarded to the upstream as-is.",
|
|
auth="user",
|
|
interactive=False,
|
|
params=[
|
|
field(
|
|
"path",
|
|
"path",
|
|
"string",
|
|
True,
|
|
"models",
|
|
"Upstream API path after /v1/.",
|
|
)
|
|
],
|
|
),
|
|
],
|
|
}
|