Add admin-unlimited workspaces, AI gateway model fallback, and real streaming/thinking control

Admin-unlimited Dev Workspaces: an admin-owned workspace is now exempt from the
max-workspace-count limit, the max-tunnel-count limit, and the whole
idle-stop/idle-warn/retention-delete lifecycle. Resolved once in
quota.resolve() as Limits.unlimited (owner uid checked against
get_admin_uids()), consumed at the three enforcement points
(provision.ensure, provision.publish_tunnel,
WorkspaceService._advance_lifecycle). Also hardens
get_admin_uids()/get_primary_admin_uid() against a partially-schemaed users
table (uid/role column guard), which a fresh test/init_db() path could hit.

AI gateway per-model automatic fallback: any gateway_models route
(chat/embed/image) can now name a fallback_model, picked on /admin/gateway
from a select box of other configured public model names of the same kind
only (never an internal upstream model id). When a route fails after its own
retries are exhausted, the gateway retries once, automatically, against the
fallback's own provider/pricing/key, before any bytes reach the client
(including for a streaming response). One hop only, no chains or cycles;
self-reference and cross-kind fallbacks are rejected at write time.

AI gateway real upstream streaming and thinking-default control: stream:true
is now forwarded to the upstream and relayed to the client as real SSE
chunks (measured TTFT/inter-token latency) instead of a simulated split
response, and every chat/vision call explicitly disables model "thinking" by
default (admin-overridable via gateway_thinking), with per-dialect handling
for DeepSeek, OpenRouter, and Ollama.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TjdKTWgWpW2SMNW8SFqxz5

devplacepy/database/schema.py

@@ -532,6 +532,10 @@ def init_db():
         "idx_user_cust_scope",
         ["owner_kind", "owner_id", "scope", "lang"],
     )
+    gateway_usage_ledger = get_table("gateway_usage_ledger")
+    for column, example in (("ttft_ms", 0.0), ("inter_token_ms", 0.0)):
+        if not gateway_usage_ledger.has_column(column):
+            gateway_usage_ledger.create_column_by_example(column, example)
     _index(db, "gateway_usage_ledger", "idx_gw_usage_time", ["created_at"])
     _index(
         db,

devplacepy/database/users.py

@@ -30,6 +30,9 @@ def get_admin_uids():
         return list(cached)
     if "users" not in db.tables:
         return []
+    users = db["users"]
+    if "uid" not in users.columns or "role" not in users.columns:
+        return []
     rows = db.query("SELECT uid FROM users WHERE role = 'Admin'")
     uids = [row["uid"] for row in rows]
     _admins_cache.set("uids", uids)
@@ -90,6 +93,9 @@ def get_primary_admin_uid():
         return cached or None
     if "users" not in db.tables:
         return None
+    users = db["users"]
+    if "uid" not in users.columns or "role" not in users.columns:
+        return None
     rows = list(
         db.query(
             "SELECT * FROM users WHERE role = 'Admin' "

devplacepy/docs_api/groups/admin.py

@@ -323,7 +323,7 @@ four ways to sign requests.
                     ),
                 ],
                 notes=[
-                    "TTFT and inter-token latency are not reported: the gateway forwards non-streaming to the upstream."
+                    "TTFT and inter-token latency (latency.ttft_ms/latency.inter_token_ms) are measured only for calls that requested streaming (stream: true); a zero count means no streaming calls occurred in the window, not that the metric is unavailable."
                 ],
             ),
             endpoint(
@@ -700,6 +700,7 @@ four ways to sign requests.
                     field("source_model", "json", "string", True, "gpt-4", "Model name callers request."),
                     field("target_model", "json", "string", True, "x-ai/grok-4.3", "Model actually sent upstream."),
                     field("provider", "json", "string", False, "openrouter", "Provider name to route through. Blank = the default provider."),
+                    field("fallback_model", "json", "string", False, "molodetz-pro", "Another already-configured source_model of the same kind, tried once automatically when this route fails after its own retries are exhausted. Blank = no fallback."),
                 ],
             ),
             endpoint(

devplacepy/docs_api/groups/gateway.py

@@ -77,9 +77,16 @@ managed by administrators on the **Gateway** page (`/admin/gateway`).
 
 ## Per-call cost and usage headers
 
-Every gateway response - chat, embeddings, and passthrough, on both success and error - carries
-`X-Gateway-*` response headers describing that single call, so a client can read its own token usage
-and dollar cost directly from the response with no extra request:
+Every NON-STREAMING gateway response - chat, embeddings, images, and passthrough, on both success
+and error - carries `X-Gateway-*` response headers describing that single call, so a client can read
+its own token usage and dollar cost directly from the response with no extra request. A streaming
+chat response (`"stream": true`) is the one exception: it carries only `X-Gateway-Model`,
+`X-Gateway-Backend`, and `X-App-Reference` - HTTP headers must be sent before the body, and cost/token
+counts for a streamed call are only known once the stream ends, so they cannot be response headers on
+that same response. The call is still fully metered server-side, and a client that sends
+`"stream_options": {"include_usage": true}` still receives the upstream's real `usage` object on the
+final SSE chunk, exactly as the underlying provider's own streaming API works - it is just not
+summarized into headers.
 
 | Header | Meaning |
 |--------|---------|
@@ -161,11 +168,20 @@ for signing DevPlace's own requests.
                         "false",
                         "Set true for a streamed SSE response.",
                     ),
+                    field(
+                        "think",
+                        "json",
+                        "string",
+                        False,
+                        "false",
+                        "Optional thinking override. Omitted: the gateway disables thinking (fast path). true / high / medium / low enables it; false disables it. DeepSeek-native `thinking.type` and OpenRouter `reasoning.effort` are also accepted.",
+                    ),
                 ],
                 notes=[
                     "Returns `503` when the gateway service is not running.",
-                    "Every response carries the `X-Gateway-*` token and dollar-cost headers (see Per-call cost and usage headers above), including the streamed SSE response.",
+                    "A non-streaming response carries the `X-Gateway-*` token and dollar-cost headers (see Per-call cost and usage headers above). A streaming response (`stream: true`) is forwarded from the upstream in real time and carries only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference` - request `stream_options: {\"include_usage\": true}` to receive the real token usage on the final SSE chunk instead.",
                     "If `model` matches a configured model route it is forwarded to that route's provider, upstream model, and per-model pricing (with an optional vision model); otherwise it falls through to the default upstream (see Model routing and providers above).",
+                    "Thinking is disabled by default. Pass `think: true` (or `thinking: {\"type\": \"enabled\"}`) to turn it on for that call.",
                 ],
             ),
             endpoint(

devplacepy/routers/admin/gateway_configs.py

@@ -150,6 +150,16 @@ async def save_model(request: Request):
         payload = routing.ModelRouteIn(**body)
     except ValidationError as exc:
         return _validation_error(exc)
+    if payload.fallback_model:
+        fallback_route = routing.model_store.get(payload.fallback_model)
+        if fallback_route is None or fallback_route.kind != payload.kind:
+            return JSONResponse(
+                {
+                    "ok": False,
+                    "error": "Fallback model must be an existing model route of the same kind",
+                },
+                status_code=400,
+            )
     saved = routing.model_store.set(payload)
     audit.record(
         request,

devplacepy/routers/projects/containers/workspace.py

@@ -79,6 +79,7 @@ async def workspace_page(request: Request, slug: str):
         "viewer_can_workspace": True,
         "workspace_count": provision.count_for_owner(user["uid"]),
         "max_workspaces": limits.max_workspaces,
+        "unlimited_workspaces": limits.unlimited,
         "editor_url": (
             f"/projects/{slug}/containers/instances/{instance['uid']}/code/"
             if instance

devplacepy/schemas/containers.py

@@ -171,6 +171,7 @@ class WorkspaceViewOut(_Out):
     idle_stop_minutes: int = 0
     retention_days: int = 0
     max_tunnels: int = 0
+    unlimited: bool = False
     tunnels: list[TunnelOut] = []
     flags: list[WorkspaceFlagOut] = []
     editor: Optional[EditorProfileOut] = None
@@ -183,6 +184,7 @@ class WorkspaceOut(_Out):
     viewer_can_workspace: bool = False
     workspace_count: int = 0
     max_workspaces: int = 0
+    unlimited_workspaces: bool = False
     editor_url: str = ""
     editor_password: str = ""
     editor: Optional[EditorProfileOut] = None

devplacepy/services/containers/CLAUDE.md

@@ -419,7 +419,16 @@ into one atomic `COALESCE` UPDATE (`store.record_activity`). Both proxy planes c
 plane B traverses DevPlace, public traffic is observed directly rather than inferred.
 
 **`workspace/` package.** `quota.py` resolves limits instance -> user rule -> setting -> default
-through ONE resolver (never read a workspace setting at a call site). `flags.py` is the abuse ledger
+through ONE resolver (never read a workspace setting at a call site). **An administrator-owned
+workspace is unlimited**, decided inside `resolve()` itself (`_is_admin_owner`, `owner_uid in
+get_admin_uids()`) and surfaced as `Limits.unlimited`, never by special-casing role at a call site.
+The three enforcement points check it: `provision.ensure` skips the `max_workspaces` count check,
+`provision.publish_tunnel` skips the `max_tunnels` count check, and
+`WorkspaceService._advance_lifecycle` `continue`s past a row entirely (no idle-stop, no idle-warn, no
+retention-delete, no delete-warn) when `limits.unlimited`. Hard docker resource allocation
+(`cpu_millicores`/`memory_mb`, the `--cpus`/`--memory` flags) and the abuse-flag evaluator
+(`_evaluate_flags`) are deliberately untouched - those protect the host itself and stay in force
+for every owner including admins. `flags.py` is the abuse ledger
 and is **idempotent per `(instance_uid, kind)` while a flag is open**, so a sustained condition is one
 row, not one per tick. `naming.py` generates faker labels with collision retry and owns the hostname
 patterns plus `is_tunnel_host`. `tunnels.py` is CRUD with revive-not-duplicate. `provision.py` is the

devplacepy/services/containers/workspace/provision.py

@@ -46,7 +46,11 @@ async def ensure(project: dict, user: dict) -> dict:
     if existing:
         return existing
     limits = quota.resolve(owner_uid)
-    if limits.max_workspaces and count_for_owner(owner_uid) >= limits.max_workspaces:
+    if (
+        not limits.unlimited
+        and limits.max_workspaces
+        and count_for_owner(owner_uid) >= limits.max_workspaces
+    ):
         raise WorkspaceError(
             f"workspace limit reached ({limits.max_workspaces}); "
             "delete one before creating another"
@@ -79,9 +83,11 @@ def publish_tunnel(
     if container_port <= 0 or container_port > 65535:
         raise WorkspaceError("container_port must be between 1 and 65535")
     limits = quota.resolve(instance.get("workspace_owner_uid", ""), instance)
-    if limits.max_tunnels and tunnels.count_for_instance(
-        instance["uid"]
-    ) >= limits.max_tunnels:
+    if (
+        not limits.unlimited
+        and limits.max_tunnels
+        and tunnels.count_for_instance(instance["uid"]) >= limits.max_tunnels
+    ):
         raise WorkspaceError(f"tunnel limit reached ({limits.max_tunnels})")
     row = tunnels.create(instance, label, container_port, owner_uid)
     if not row:
@@ -245,6 +251,7 @@ def view(instance: dict) -> dict:
         "idle_stop_minutes": limits.idle_stop_minutes,
         "retention_days": limits.retention_days,
         "max_tunnels": limits.max_tunnels,
+        "unlimited": limits.unlimited,
         "tunnels": tunnels.list_for_instance(instance["uid"]),
         "flags": flags.list_flags(instance_uid=instance["uid"]),
         "editor": editor.view(owner_uid, instance),

devplacepy/services/containers/workspace/quota.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 from dataclasses import dataclass
 
-from devplacepy.database import get_int_setting, get_table
+from devplacepy.database import get_admin_uids, get_int_setting, get_table
 
 RULES_TABLE = "workspace_quota_rules"
 
@@ -79,6 +79,7 @@ class Limits:
     disk_warn_percent: int
     cpu_millicores: int
     memory_mb: int
+    unlimited: bool = False
 
     def disk_quota_bytes(self) -> int:
         return self.disk_quota_mb * 1024 * 1024
@@ -104,6 +105,10 @@ def _rule_for(owner_kind: str, owner_id: str) -> dict | None:
     return table.find_one(owner_kind=owner_kind, owner_id=owner_id, deleted_at=None)
 
 
+def _is_admin_owner(user_uid: str) -> bool:
+    return bool(user_uid) and user_uid in get_admin_uids()
+
+
 def resolve(user_uid: str = "", instance: dict | None = None) -> Limits:
     rule = _rule_for("user", user_uid) if user_uid else None
     values: dict[str, int] = {}
@@ -120,7 +125,7 @@ def resolve(user_uid: str = "", instance: dict | None = None) -> Limits:
         values[key] = max(0, value)
     if values["idle_warn_minutes"] >= values["idle_stop_minutes"]:
         values["idle_warn_minutes"] = max(1, values["idle_stop_minutes"] - 1)
-    return Limits(**values)
+    return Limits(unlimited=_is_admin_owner(user_uid), **values)
 
 
 def percent_used(used: int, quota: int) -> int:

devplacepy/services/containers/workspace_service.py

@@ -293,6 +293,8 @@ class WorkspaceService(BaseService):
             if row.get("suspended_at"):
                 continue
             limits = quota.resolve(row.get("workspace_owner_uid", ""), row)
+            if limits.unlimited:
+                continue
             owner = row.get("workspace_owner_uid", "")
             idle = _minutes_since(row.get("last_active_at"))
             if idle is None:

devplacepy/services/devii/actions/catalog/gateway.py

@@ -81,7 +81,10 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
             "context_tier_threshold_tokens, and/or apply a percentage discount during a fixed "
             "UTC off-peak window - leave the tier2/off-peak fields unset to keep the flat rates "
             "above at all times. off_peak_start_minute and off_peak_end_minute must be set "
-            "together (both or neither) or the call is rejected."
+            "together (both or neither) or the call is rejected. fallback_model optionally names "
+            "another already-configured source_model of the same kind; when this route fails after "
+            "its own retries are exhausted, the gateway retries once, automatically, against that "
+            "fallback model instead of returning the error to the caller."
         ),
         handler="http",
         requires_admin=True,
@@ -105,6 +108,7 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
             Param(name="off_peak_start_minute", location="body", description="Off-peak window start, UTC minutes since midnight (0-1439). Must be set together with off_peak_end_minute.", required=False, type="integer"),
             Param(name="off_peak_end_minute", location="body", description="Off-peak window end, UTC minutes since midnight (0-1439). A value less than the start wraps past midnight.", required=False, type="integer"),
             Param(name="off_peak_discount_pct", location="body", description="Percentage discount (0-100) applied to the active tier's rates during the off-peak window.", required=False, type="number"),
+            body("fallback_model", "Another already-configured source_model of the same kind, tried once automatically when this route fails after its own retries are exhausted. Blank disables fallback."),
             Param(name="is_active", location="body", description="Whether the route is active ('1' or '0').", required=False, type="boolean"),
         ),
     ),

devplacepy/services/devii/workspace/controller.py

@@ -193,6 +193,7 @@ class WorkspaceController:
             "idle_stop_minutes": limits.idle_stop_minutes,
             "retention_days": limits.retention_days,
             "used_workspaces": provision.count_for_owner(target),
+            "unlimited": limits.unlimited,
         }
 
     def _workspace_quota_set(self, args: dict) -> Any:

devplacepy/services/openai_gateway/CLAUDE.md

@@ -4,13 +4,15 @@ This file documents the AI gateway subsystem (devplacepy/services/openai_gateway
 
 `GatewayService` (`services/openai_gateway/`) is an OpenAI-compatible LLM gateway (the ported `openai5.py`), mounted at `/openai/v1/*` (`routers/openai_gateway.py`, prefix `/openai`). It is a **service that serves an HTTP endpoint** rather than a loop.
 
-- The router is thin: it calls `service_manager.get_service("openai").handle(request, subpath)`. `POST /v1/chat/completions` runs the full gateway (vision augment -> model override -> forward -> optional SSE re-emit via `_fake_stream`); `POST /v1/embeddings` forwards to the configured embeddings upstream (`handle_embeddings`, no vision/streaming); `GET /v1/models` is answered locally from the `gateway_models` chat routes (`_models_response`, publishing the public `molodetz`/`molodetz-pro` names), NOT proxied upstream; `/v1/{path:path}` is a transparent passthrough to the upstream base. **The gateway always forwards NON-streaming upstream** (`payload["stream"] = False`), then re-emits SSE itself when the client asked for `stream`; because `stream_options` is only valid alongside `stream=true`, `handle_chat` strips `stream_options` from the upstream payload (otherwise DeepSeek rejects it with `stream_options should be set along with stream = true`) and, when the client requested `stream_options.include_usage`, `_fake_stream` appends a final `choices: []` usage chunk built from the upstream `usage` before `[DONE]`. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
+- The router is thin: it calls `service_manager.get_service("openai").handle(request, subpath)`. `POST /v1/chat/completions` runs the full gateway (vision augment -> model override -> thinking default -> forward, real upstream streaming when the client asked for it); `POST /v1/embeddings` forwards to the configured embeddings upstream (`handle_embeddings`, no vision/streaming); `GET /v1/models` is answered locally from the `gateway_models` chat routes (`_models_response`, publishing the public `molodetz`/`molodetz-pro` names), NOT proxied upstream; `/v1/{path:path}` is a transparent passthrough to the upstream base. **The gateway forwards `stream: true` to the upstream verbatim and relays its SSE chunks as they arrive** (`GatewayRuntime._stream_chat_response`, `client.send(request, stream=True)` + `resp.aiter_lines()`) - so TTFT and inter-token latency are real, measured timings, not simulated. When streaming, the gateway ALWAYS additionally sends `stream_options: {"include_usage": true}` upstream regardless of what the client asked (so the ledger always gets a real `usage` object off the final chunk); the client's own `include_usage` only controls whether that usage-only chunk is relayed downstream to it - the gateway still swallows-and-records it either way. **Load-bearing header trade-off: a streaming response carries no `X-Gateway-Cost-USD`/token headers** (only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference`) - HTTP headers must be sent before the body, and cost/tokens aren't known until the stream ends, so the usage ledger row (and the client's own final `usage` chunk, if it asked for one) are the only ways to learn a streamed call's cost; every non-streaming response is completely unaffected and keeps the full header set exactly as before. The ledger row for a streamed call is written from inside the generator's `finally` once the stream ends (or is interrupted - a caught mid-stream exception or a client disconnect, i.e. `GeneratorExit`, both still write a row with `success=False` and an `error_category`, never silently drop the accounting); a client hangup never leaves an unclosed upstream connection (`resp.aclose()` always runs). **Thinking is off by default** (`thinking.py` `apply_thinking`): DeepSeek V4 enables thinking unless `thinking.type=disabled` is sent, so the gateway always writes an explicit disable unless the client (or `gateway_thinking`) asked for thinking. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
 - `main.py` registers it and exempts `/openai` from the rate-limit middleware and the maintenance gate. No new dependency (`httpx` already required).
 - `GatewayService` is `default_enabled=True` so internal callers work out of the box.
 
 ## Per-call cost/token response headers
 
-Every gateway response (chat, embeddings, passthrough; success and error) carries `X-Gateway-*` headers describing that single call: `Model`, `Backend`, `Prompt-Tokens`, `Completion-Tokens`, `Total-Tokens`, `Cache-Hit-Tokens`, `Cache-Miss-Tokens`, `Reasoning-Tokens`, `Cost-USD`/`Input-Cost-USD`/`Output-Cost-USD` (dollars), `Cost-Native` (1 if the upstream returned a native cost), `Tokens-Per-Second`, `Upstream-Latency-Ms`, and `Context-Window`/`Context-Utilization` when the model's window is known, plus the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`. `GatewayUsageLedger.record(...)` RETURNS the computed ledger row (or `None` on failure); each handler's `finalize` closure maps it through `usage.usage_response_headers(row)` and attaches it to the returned `Response` (`resp_headers`). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads `X-Gateway-Cost-USD`/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root `CLAUDE.md`).
+Every NON-STREAMING gateway response (chat, embeddings, images, passthrough; success and error) carries `X-Gateway-*` headers describing that single call: `Model`, `Backend`, `Prompt-Tokens`, `Completion-Tokens`, `Total-Tokens`, `Cache-Hit-Tokens`, `Cache-Miss-Tokens`, `Reasoning-Tokens`, `Cost-USD`/`Input-Cost-USD`/`Output-Cost-USD` (dollars), `Cost-Native` (1 if the upstream returned a native cost), `Tokens-Per-Second`, `Upstream-Latency-Ms`, and `Context-Window`/`Context-Utilization` when the model's window is known, plus the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`. `GatewayUsageLedger.record(...)` RETURNS the computed ledger row (or `None` on failure); each handler's `finalize` closure maps it through `usage.usage_response_headers(row)` and attaches it to the returned `Response` (`resp_headers`). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads `X-Gateway-Cost-USD`/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root `CLAUDE.md`); every internal consumer sends `stream: false` (or omits it), so this path is completely unaffected by the streaming trade-off below.
+
+**A STREAMING chat response (`stream: true`) is the one exception and carries only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference`** - HTTP headers are sent before the body, and cost/tokens for a streamed call are only known once the stream ends, so they cannot be headers on that same response. The call is still fully metered: the ledger row (including `ttft_ms`/`inter_token_ms`) is written server-side once the stream completes (see "Real upstream streaming" below), and a client that requests `stream_options.include_usage` still gets the upstream's real `usage` object on the final SSE chunk, exactly as the OpenAI streaming API itself works - it is simply not summarized into response headers.
 
 ## App-reference header (`X-App-Reference`)
 
@@ -33,7 +35,7 @@ Callers may send an optional `X-App-Reference` header to tag gateway calls by ap
 
 ## Config fields
 
-All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble`, timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
+All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble` and `gateway_thinking` (default off), timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
 
 ## System-message composition (date awareness + operator preamble)
 
@@ -46,6 +48,31 @@ Two behaviors:
 
 Embeddings/passthrough are untouched by this composition step.
 
+## Thinking default (fast path)
+
+`thinking.py` (`apply_thinking`) runs in `handle_chat` after `stream_options` is stripped, and in the vision describe call. DeepSeek V4 **enables thinking by default**; leaving the payload alone is the slow path. The gateway therefore always writes an explicit thinking field in the upstream dialect:
+
+- DeepSeek (`*deepseek.com*` and any unrecognised OpenAI-compat URL, because that is the production default): `thinking: {"type": "disabled"}` or `"enabled"`
+- OpenRouter: `reasoning: {"effort": "none"}` or `"high"`
+- Ollama (`:11434` or a host containing `ollama`): `think: false` or `true`
+
+Do **not** send `reasoning_effort: "none"` to DeepSeek (400: unknown variant). Conflicting client fields (`think`, `thinking`, `reasoning`, `reasoning_effort`, `enable_thinking`, `chat_template_kwargs.enable_thinking`) are stripped and translated to the dialect above.
+
+**Default is off.** `gateway_thinking` (Prompt group, bool, default `False`) is the operator override for the silent-client case. A client still wins: `think` / `thinking.type` / `reasoning.effort` / `reasoning.enabled` / `reasoning_effort` / `enable_thinking` on the request body enable or disable thinking for that call. Internal callers (correction, bots, Devii, SEO, quiz grading) send none of those, so they get the fast path.
+
+Vision describe-image calls always disable thinking (they are not a reasoning job).
+
+## Real upstream streaming (`GatewayRuntime._stream_chat_response`)
+
+`stream: true` is forwarded to the upstream verbatim (`payload["stream"] = stream`, no more forced `false`) and the connection is opened with `client.send(request, stream=True)` so the response body is read incrementally via `resp.aiter_lines()` instead of buffered whole. Every retry/circuit-breaker mechanic in `_send`/`retry_send` is shared unchanged with the non-streaming path (a 5xx or connection failure before any bytes are forwarded to the client retries exactly as before; `retry_send` now also closes an unread streamed response before retrying, so a retried streaming attempt never leaks the previous connection). Only once the upstream returns `200` does the code path diverge into `_stream_chat_response`.
+
+- **The client always gets real chunks, in real time**, each parsed `data: {...}` line re-serialized and yielded as its own SSE event - no more splitting a complete response into fake 50-character deltas.
+- **TTFT and inter-token latency are measured, not simulated.** `ttft_ms` = time from just before the upstream POST to the first SSE line received; `inter_token_ms` = the average gap between content-bearing chunks (`delta.content`/`delta.reasoning_content`) after the first one, `None` when fewer than two content chunks arrived. Both ride the ledger row (`gateway_usage_ledger.ttft_ms`/`.inter_token_ms`, nullable REAL columns dataset auto-adds on the first insert carrying them - no explicit `init_db()` ensure-block needed, same as every other column on this insert-only table) and surface in `/admin/ai-usage`'s `latency.ttft_ms`/`latency.inter_token_ms` (`analytics.py`, same `_pset` percentile shape as every other latency dimension).
+- **Usage is always requested from the upstream when streaming**, regardless of what the client asked (`stream_options.include_usage` is forced `true` in the upstream payload), so the ledger always gets a real `usage` object off the final chunk. The client's OWN `stream_options.include_usage` only controls whether that usage-only chunk is relayed downstream to it; the gateway swallows-and-records it either way.
+- **The ledger row is written once, from inside the generator's `finally`**, after the stream ends (`saw_done`), is interrupted by an upstream error, or the client disconnects (`GeneratorExit` - re-raised after cleanup, never swallowed, and never yielded-from during close: Python forbids yielding while handling `GeneratorExit`). All three paths close the upstream response (`resp.aclose()`) so a dropped client never leaks the connection, and all three still write a ledger row (`success=False`/`error_category="client_disconnected"` or a classified error on interruption) so accounting is never silently skipped.
+- **No `X-Gateway-Cost-USD`/token headers on the streaming response** - see "Per-call cost/token response headers" above; this is the one deliberate exception to "every response carries the headers."
+- Non-streaming chat, embeddings, and images are completely unchanged - this only touches the `stream: true` branch of `handle_chat`.
+
 ## Usage, cost, latency, and reliability tracking
 
 The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:
@@ -53,7 +80,7 @@ The gateway records one row per upstream call (chat, vision, passthrough) and su
 - **`usage.py`** - `GatewayUsageLedger` writes to `gateway_usage_ledger` (one row per call, success and failure) and samples `gateway_concurrency_samples` each 30s tick. `normalize_usage` handles BOTH upstream shapes: DeepSeek (`prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`, `completion_tokens_details.reasoning_tokens`) and OpenRouter (`prompt_tokens_details.cached_tokens`, native `cost`, `cost_details`). `compute_cost` PREFERS the upstream native `cost` (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share `input_cost / (input_cost + output_cost)` from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). `record()` is wrapped in try/except so a tracking failure never breaks the proxy.
 - **`gateway.py`** - `_send` is the gated/timed/retried path: a `CircuitBreaker` (`reliability.py`) short-circuits when open; `retry_send` retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the `queue_wait_ms`; an httpx per-request `trace` extension captures `connect_ms` (0 on pooled reuse). `handle_chat`/`handle_passthrough` record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from `VisionAugmenter._describe_one`.
 - **Owner attribution** - `GatewayService.resolve_owner` maps the caller to `(owner_kind, owner_id)`: `internal:devii` (internal key), `key:access` (static key), `user:<uid>`/`admin:<uid>` (session or API key), else `anonymous`.
-- **`analytics.py`** - `build_analytics(hours, top_n, pricing)` pulls the bounded window once (clamped to `MAX_WINDOW_HOURS=168`) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via `reliability.percentile`), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. `summary_metrics()` runs cheap SQL aggregates for the live service panel. TTFT/inter-token are returned `null` (gateway forwards non-streaming).
+- **`analytics.py`** - `build_analytics(hours, top_n, pricing)` pulls the bounded window once (clamped to `MAX_WINDOW_HOURS=168`) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via `reliability.percentile`), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. `summary_metrics()` runs cheap SQL aggregates for the live service panel. TTFT/inter-token (`latency.ttft_ms`/`latency.inter_token_ms`, the same `_pset` percentile-set shape as every other latency dimension) are real, measured only for calls that requested streaming - `count: 0` in a window means no streaming traffic occurred, not that the metric is unavailable.
 - **Admin** - `/admin/ai-usage` (HTML, `templates/ai_usage.html` + `static/js/AiUsageMonitor.js`) and `/admin/ai-usage/data?hours=&top_n=` (JSON) in `routers/admin/` package, admin-gated. Schema `GatewayUsageOut`. Devii action `ai_usage` and docs `admin-ai-usage` expose the same endpoint. The `ai_usage` and `site_analytics` Devii catalog tools are both `requires_admin=True`.
 - **Per-user usage** - `analytics.build_user_usage(owner_id, hours=24, pricing)` filters the ledger by `owner_id` (a uid matches both `user:` and `admin:` rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at `GET /admin/users/{uid}/ai-usage` (schema `UserAiUsageOut`, docs `admin-user-ai-usage`) and rendered on the profile page by `static/js/UserAiUsage.js` (the `[data-ai-usage]` card, gated `{% if is_admin(user) %}` in `templates/profile.html`). The 24h window is a sound projection basis (unlike a startup burst), so `cost_24h * 30` is honest.
 - **Profile visibility split** - `routers/profile/` package `_ai_quota(uid, include_cost=False)` builds the profile quota bar. **The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal:** `spent_usd`/`used_pct`/`requests`/`last_used` come from `analytics.user_spend_24h(uid)`, the SAME `gateway_usage_ledger` source (summed by `owner_id`, matching both `user:` and `admin:` rows) that feeds the stat grid's `cost.window_usd`. So the server-rendered `spent_usd` is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. `turns` is still the `devii_usage_ledger` `turns_24h` count (Devii conversation turns) and is labeled **"Devii turns"** in the template precisely because it is a *different metric* from the grid's total **API requests** (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why `0 Devii turns` can sit beside `18 API requests`). The `limit`/`turns` still come from the Devii service (`daily_limit_for`/`turns_24h`); an **unlimited** limit (`<= 0`, the admin default) sets `unlimited=True` so the template renders "no cap" / "No daily spend cap" instead of a misleading `/ $0.00`. **This is display-only: the Devii daily-cap ENFORCEMENT and `/devii/usage` still read the Devii-scoped `devii_usage_ledger.spent_24h`** (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: **admin** sees the full gateway stats card *plus* a server-rendered quota bar (with spend/limit/Devii turns); the **owner** (non-admin, `is_self`) sees a quota-**only** card showing just the percentage; everyone else sees nothing AI-related. `_ai_quota` is fail-safe (returns `None` on any error or when the Devii service is unregistered) and the result rides on `ProfileOut.ai_quota`. **Dollar figures (`spent_usd`/`limit_usd`) are included only when `include_cost=viewer_is_admin`** - never for a non-admin owner. This matters because the route serves the same context as JSON via `respond(..., model=ProfileOut)` and `ProfileOut.ai_quota` is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with `Accept: application/json`; the server-side omission is the real control. Non-admins (HTML and JSON) get only `used_pct`/`turns`/`requests`.
@@ -120,6 +147,21 @@ When adding a routed value, overlay it as the matching `gateway_*` cfg key so th
 
 **Tests:** `tests/unit/services/openai_gateway/routing.py` (overlay/economy/kind isolation), `tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream` (end-to-end through `handle_chat`), and `tests/api/admin/gateway/` (admin CRUD + validation + role gating).
 
+## Automatic model fallback (`fallback_model`, one hop, per route)
+
+Any `gateway_models` route (chat, embed, or image) can optionally name `fallback_model`: another already-configured `source_model` of the SAME kind, tried once, automatically, when the primary route fails. This is admin-configured on the `/admin/gateway` model form as a select box populated from every OTHER route of the current kind (the public `source_model` names such as `molodetz`/`molodetz-pro`, never the internal `target_model` a provider actually sees) - never a free-text field, so a fallback can only ever point at a model the gateway already knows how to serve.
+
+- **Trigger.** `_call_failed(resp, exc, timing)` in `gateway.py` treats a call as failed when the circuit breaker rejected it, the upstream connection raised, or the upstream answered with any status `>= 400`. This runs AFTER `_send`'s own `gateway_max_retries` retries against the primary model are exhausted - a fallback is the next escalation once retrying the SAME model has already given up, not a replacement for that retry loop.
+- **One hop, never a chain.** `routing.resolve_fallback(source_model, kind)` resolves the primary route, reads its `fallback_model`, and resolves THAT model's own route (must exist, be active, and share the kind) - it does NOT recurse into the fallback's own `fallback_model`, so there is no possibility of a cycle or an unbounded retry chain. A route that fails even after redirecting to its fallback returns the fallback attempt's own failure to the caller.
+- **Self-reference is rejected at write time.** `ModelRouteIn._check_fallback_is_not_self` (pydantic `model_validator`) refuses `fallback_model == source_model`; the admin route handler (`routers/admin/gateway_configs.py::save_model`) additionally rejects a `fallback_model` that does not resolve to an existing, same-kind route (`routing.model_store.get(...)`) before writing the row, since a pydantic validator alone cannot see other rows.
+- **Rebuilt like a fresh routed call, not retried in place.** On failure, `handle_chat`/`handle_embeddings`/`handle_images` recompute the overlay from `base_cfg` (the pre-primary-overlay config) via `chat_overlay(fallback_route.source_model, base_cfg)`/`embed_overlay`/`image_overlay`, so the fallback gets its OWN provider, URL, key, and pricing - never the primary route's. `pricing`/`context_map` for the ledger are (re)computed AFTER the fallback decision so the recorded cost always reflects whichever model actually served the request.
+- **One ledger row per client call, not one per attempt.** The primary failure never writes a `gateway_usage_ledger` row by itself (only `_send`'s own internal retries and the circuit breaker counters observe it); `finalize()`/`self._ledger.record(...)` still runs exactly once, after the fallback decision, with `model` = whichever model ultimately answered and `requested_model` = the ORIGINALLY requested model name (an existing-but-previously-unpopulated ledger column, now populated by all three handlers) - so a fallback shows up in `/admin/ai-usage` as "requested X, served Y" rather than as two calls.
+- **Streaming falls back before any byte reaches the client.** `client.send(request, stream=True)` returns as soon as the response headers arrive, so `resp.status_code` is already known before the SSE body is ever touched - `handle_chat` decides whether to fall back at that same point, before it ever opens `_stream_chat_response` to the caller. A stream is therefore never abandoned mid-flight in favor of a fallback; the caller either gets the fallback's own stream from byte one, or the fallback's own definitive failure.
+- **Known limitation, inherited from the existing provider routing design, not introduced by this feature: the circuit breaker (`self._breaker`) is one instance per `GatewayRuntime`, shared across every route.** A primary attempt that trips the breaker can make the immediately-following fallback attempt itself see `circuit_open` too. This mirrors the pre-existing behavior for the multi-provider routing overlay in general (a struggling route's failures already could open the breaker in front of an unrelated route before fallback existed) and was left as-is rather than redesigning per-provider breakers, which is a materially larger change than "add a fallback model."
+- **Deliberately NOT covered:** `gateway_max_retries`/`gateway_retry_backoff_ms`, the circuit breaker, and hard docker-adjacent settings are untouched - fallback is additive on top of them, not a replacement retry mechanism.
+
+**Tests:** `tests/unit/services/openai_gateway/routing.py` (`resolve_fallback` self-reference/kind/active-flag guards), `tests/unit/services/openai_gateway/gateway.py` (`test_chat_falls_back_when_the_primary_model_fails`, `test_chat_returns_the_fallback_failure_when_both_models_fail`, `test_chat_without_a_fallback_never_makes_a_second_call`), and `tests/api/admin/gateway/models.py` (`test_model_route_fallback_round_trip`, `test_model_route_fallback_must_reference_an_existing_route`, `test_model_route_fallback_must_be_the_same_kind`, `test_model_route_fallback_rejects_self_reference`).
+
 ## Tiered (context-length) and off-peak pricing (model-agnostic variable pricing)
 
 Real providers sometimes charge more than a flat per-1M rate for one component: a rate that jumps once a request crosses a context-length threshold, or a fixed time-of-day discount window. This is layered on top of the cache-hit/cache-miss/output shape (which is already exactly DeepSeek's real billing model - see below), on the SAME `gateway_models` route row, so it stays fully provider-agnostic and opt-in per route.

devplacepy/services/openai_gateway/analytics.py

@@ -135,7 +135,7 @@ def empty_payload(hours: int = 48) -> dict:
         "cost": {},
         "behavior": {},
         "hourly": [],
-        "notes": {"ttft": "not available: gateway forwards non-streaming upstream"},
+        "notes": {"ttft": "measured only for calls that requested streaming (stream: true); zero count means no streaming calls in this window"},
     }
 
 
@@ -309,8 +309,8 @@ def build_analytics(
         "upstream_availability_pct": round(success / requests * 100, 2)
         if requests
         else 0.0,
-        "ttft_ms": None,
-        "inter_token_ms": None,
+        "ttft_ms": _pset(_positive(rows, "ttft_ms")),
+        "inter_token_ms": _pset(_positive(rows, "inter_token_ms")),
     }
 
     errors = {
@@ -377,7 +377,7 @@ def build_analytics(
         "cost": cost,
         "behavior": behavior,
         "hourly": hourly,
-        "notes": {"ttft": "not available: gateway forwards non-streaming upstream"},
+        "notes": {"ttft": "measured only for calls that requested streaming (stream: true); zero count means no streaming calls in this window"},
     }
 
 

devplacepy/services/openai_gateway/config.py

@@ -7,6 +7,7 @@ TIMEOUT_MIN = 300
 INSTANCES_DEFAULT = 4
 
 SYSTEM_PREAMBLE_DEFAULT = ""
+THINKING_DEFAULT = False
 
 VISION_URL_DEFAULT = "https://openrouter.ai/api/v1/chat/completions"
 VISION_MODEL_DEFAULT = "google/gemma-3-12b-it"

devplacepy/services/openai_gateway/gateway.py

@@ -4,7 +4,6 @@ import asyncio
 import json
 import logging
 import time
-import uuid
 from typing import Optional
 
 import httpx
@@ -18,8 +17,10 @@ from devplacepy.services.openai_gateway.routing import (
     chat_overlay,
     embed_overlay,
     image_overlay,
+    resolve_fallback,
 )
 from devplacepy.services.openai_gateway.system_message import apply_system_directives
+from devplacepy.services.openai_gateway.thinking import apply_thinking
 from devplacepy.services.openai_gateway.usage import (
     GatewayUsageLedger,
     classify_error,
@@ -34,6 +35,19 @@ from devplacepy.services.openai_gateway.vision import VisionAugmenter, VisionCac
 logger = logging.getLogger(__name__)
 
 
+def _call_failed(resp, exc, timing: dict) -> bool:
+    if timing.get("circuit_open") or exc is not None:
+        return True
+    return resp is not None and resp.status_code >= 400
+
+
+def _fallback_headers(cfg: dict, key_field: str) -> dict:
+    headers = {"Content-Type": "application/json"}
+    if cfg.get(key_field):
+        headers["Authorization"] = f"Bearer {cfg[key_field]}"
+    return headers
+
+
 def _notify_gateway_status(is_open: bool) -> None:
     from devplacepy.database import get_table
     from devplacepy.utils import create_notification
@@ -47,65 +61,6 @@ def _notify_gateway_status(is_open: bool) -> None:
         create_notification(admin["uid"], "system", message, "gateway", "/admin/services/openai")
 
 
-def _fake_stream(data: dict, model: str, include_usage: bool = False):
-    chunk_id = data.get("id", f"chatcmpl-{uuid.uuid4().hex[:12]}")
-    created = data.get("created", int(time.time()))
-    out_model = data.get("model", model)
-
-    try:
-        msg = data["choices"][0]["message"]
-    except (KeyError, IndexError):
-        msg = {"content": ""}
-    tool_calls = msg.get("tool_calls")
-    content = msg.get("content") or ""
-    reasoning_content = msg.get("reasoning_content") or ""
-
-    def _chunk(delta: dict, finish: Optional[str] = None) -> str:
-        return (
-            "data: "
-            + json.dumps(
-                {
-                    "id": chunk_id,
-                    "object": "chat.completion.chunk",
-                    "created": created,
-                    "model": out_model,
-                    "choices": [{"index": 0, "delta": delta, "finish_reason": finish}],
-                }
-            )
-            + "\n\n"
-        )
-
-    async def gen():
-        yield _chunk({"role": "assistant"})
-        if reasoning_content:
-            for i in range(0, len(reasoning_content), 50):
-                yield _chunk({"reasoning_content": reasoning_content[i : i + 50]})
-        if tool_calls:
-            yield _chunk({"tool_calls": tool_calls})
-        elif content:
-            for i in range(0, len(content), 50):
-                yield _chunk({"content": content[i : i + 50]})
-        yield _chunk({}, finish="tool_calls" if tool_calls else "stop")
-        if include_usage and data.get("usage"):
-            yield (
-                "data: "
-                + json.dumps(
-                    {
-                        "id": chunk_id,
-                        "object": "chat.completion.chunk",
-                        "created": created,
-                        "model": out_model,
-                        "choices": [],
-                        "usage": data["usage"],
-                    }
-                )
-                + "\n\n"
-            )
-        yield "data: [DONE]\n\n"
-
-    return gen()
-
-
 def _connect_tracer(holder: dict):
     started: dict = {}
 
@@ -180,7 +135,17 @@ class GatewayRuntime:
             self._instances = 0
 
     async def _send(
-        self, client, sem, method, url, headers, cfg, log, json_body=None, content=None
+        self,
+        client,
+        sem,
+        method,
+        url,
+        headers,
+        cfg,
+        log,
+        json_body=None,
+        content=None,
+        stream=False,
     ):
         timing = {
             "queue_wait_ms": 0.0,
@@ -209,7 +174,7 @@ class GatewayRuntime:
                     method, url, headers=headers, json=json_body, content=content
                 )
                 request.extensions["trace"] = _connect_tracer(connect_holder)
-                return await client.send(request)
+                return await client.send(request, stream=stream)
 
             send_start = time.monotonic()
             resp, exc, attempts, queue_wait_ms = await retry_send(
@@ -256,6 +221,7 @@ class GatewayRuntime:
     ):
         log = log or (lambda message: None)
         overlay = chat_overlay(body.get("model"), cfg)
+        base_cfg = cfg
         if overlay:
             cfg = {**cfg, **overlay}
             log(f"routed model {body.get('model')!r} -> {cfg['gateway_model']!r}")
@@ -299,8 +265,22 @@ class GatewayRuntime:
         payload = dict(body)
         payload["model"] = model
         payload["messages"] = messages
-        payload["stream"] = False
-        payload.pop("stream_options", None)
+        payload["stream"] = stream
+        if stream:
+            # Always ask the upstream for usage on its final chunk, regardless of
+            # whether the client itself requested it, so the ledger always has real
+            # cost/token numbers for a streamed call; `include_usage` (the client's
+            # own ask) only controls whether that chunk is relayed to the client.
+            stream_options = dict(body.get("stream_options") or {})
+            stream_options["include_usage"] = True
+            payload["stream_options"] = stream_options
+        else:
+            payload.pop("stream_options", None)
+        apply_thinking(
+            payload,
+            cfg.get("gateway_upstream_url", ""),
+            default_enabled=bool(cfg.get("gateway_thinking", False)),
+        )
 
         headers = {"Content-Type": "application/json"}
         if cfg["gateway_api_key"]:
@@ -310,6 +290,7 @@ class GatewayRuntime:
                 "No upstream API key configured (gateway_api_key / DEEPSEEK_API_KEY / OPENROUTER_API_KEY); upstream will likely reject the request"
             )
 
+        send_start = time.monotonic()
         resp, exc, timing = await self._send(
             client,
             sem,
@@ -319,13 +300,56 @@ class GatewayRuntime:
             cfg,
             log,
             json_body=payload,
+            stream=stream,
         )
+        if resp is not None and stream and resp.status_code != 200:
+            # The streamed response body hasn't been read yet; the shared error
+            # handling below needs resp.text/.content, which requires an explicit
+            # read for a stream=True response (a no-op if already buffered).
+            await resp.aread()
+
+        requested_model = requested
+        if _call_failed(resp, exc, timing):
+            fallback_route = resolve_fallback(requested, "chat")
+            fallback_overlay = (
+                chat_overlay(fallback_route.source_model, base_cfg)
+                if fallback_route is not None
+                else None
+            )
+            if fallback_overlay:
+                log(
+                    f"model {requested!r} failed, falling back to "
+                    f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_model']!r}"
+                )
+                cfg = {**base_cfg, **fallback_overlay}
+                model = cfg["gateway_model"]
+                payload = dict(payload)
+                payload["model"] = model
+                headers = _fallback_headers(cfg, "gateway_api_key")
+                send_start = time.monotonic()
+                resp, exc, timing = await self._send(
+                    client,
+                    sem,
+                    "POST",
+                    cfg["gateway_upstream_url"],
+                    headers,
+                    cfg,
+                    log,
+                    json_body=payload,
+                    stream=stream,
+                )
+                if resp is not None and stream and resp.status_code != 200:
+                    await resp.aread()
+
+        pricing = pricing_from_cfg(cfg)
+        context_map = parse_context_map(cfg.get("gateway_model_context_map"))
 
         base = {
             "owner_kind": owner[0],
             "owner_id": owner[1],
             "backend": "chat",
             "endpoint": "chat/completions",
+            "requested_model": requested_model,
             "model": model,
             "user_agent": user_agent,
             "app_reference": app_reference,
@@ -394,6 +418,32 @@ class GatewayRuntime:
                 content={"error": {"message": resp.text, "type": "upstream_error"}},
                 headers=resp_headers,
             )
+        if stream:
+            # Real upstream streaming: forward SSE chunks to the client as they
+            # arrive (so TTFT/inter-token latency are genuine), and finalize the
+            # ledger row from inside the generator once the stream ends - cost and
+            # token counts are not known yet at this point, so unlike the
+            # non-streaming response below, this response carries no
+            # X-Gateway-Cost-USD/token headers (HTTP headers must precede the body).
+            log(f"chat POST -> 200 stream ({timing['upstream_latency_ms']:.0f}ms to first byte)")
+            return StreamingResponse(
+                self._stream_chat_response(
+                    resp,
+                    base,
+                    pricing,
+                    context_map,
+                    include_usage,
+                    handle_start,
+                    send_start,
+                    log,
+                ),
+                media_type="text/event-stream",
+                headers={
+                    "X-Gateway-Model": model,
+                    "X-Gateway-Backend": "chat",
+                    "X-App-Reference": app_reference or "default",
+                },
+            )
         try:
             data = resp.json()
         except ValueError:
@@ -412,24 +462,112 @@ class GatewayRuntime:
             )
         resp_headers = finalize(200, True, None, data.get("usage"))
         log(f"chat POST -> 200 ({timing['upstream_latency_ms']:.0f}ms)")
-        if stream:
-            return StreamingResponse(
-                _fake_stream(data, model, include_usage),
-                media_type="text/event-stream",
-                headers=resp_headers,
-            )
         return Response(
             content=resp.content,
             media_type="application/json",
             headers=resp_headers,
         )
 
+    async def _stream_chat_response(
+        self,
+        resp: httpx.Response,
+        base: dict,
+        pricing,
+        context_map: dict,
+        include_usage: bool,
+        handle_start: float,
+        send_start: float,
+        log,
+    ):
+        first_at: Optional[float] = None
+        last_at: Optional[float] = None
+        content_chunks = 0
+        usage_captured: Optional[dict] = None
+        saw_done = False
+        success = True
+        error_category: Optional[str] = None
+        try:
+            async for raw_line in resp.aiter_lines():
+                line = raw_line.strip()
+                if not line or not line.startswith("data:"):
+                    continue
+                now = time.monotonic()
+                if first_at is None:
+                    first_at = now
+                last_at = now
+                data_str = line[len("data:") :].strip()
+                if data_str == "[DONE]":
+                    saw_done = True
+                    yield "data: [DONE]\n\n"
+                    break
+                try:
+                    chunk = json.loads(data_str)
+                except ValueError:
+                    continue
+                choices = chunk.get("choices") or []
+                delta = (choices[0].get("delta") if choices else None) or {}
+                if delta.get("content") or delta.get("reasoning_content"):
+                    content_chunks += 1
+                usage = chunk.get("usage")
+                if usage:
+                    usage_captured = usage
+                    if not include_usage:
+                        # The client did not ask for the usage chunk; swallow it,
+                        # we still captured it above for the ledger.
+                        continue
+                yield f"data: {json.dumps(chunk)}\n\n"
+        except GeneratorExit:
+            success = False
+            error_category = "client_disconnected"
+            raise
+        except Exception as exc:  # noqa: BLE001 - an upstream stream can drop mid-flight; never crash the response
+            success = False
+            error_category = classify_error(0, exc)
+            log(f"chat stream interrupted: {exc}")
+        finally:
+            try:
+                await resp.aclose()
+            except Exception:  # noqa: BLE001 - releasing the connection must never mask the real outcome
+                pass
+            ttft_ms = (
+                round((first_at - send_start) * 1000, 3)
+                if first_at is not None
+                else None
+            )
+            inter_token_ms = (
+                round((last_at - first_at) * 1000 / (content_chunks - 1), 3)
+                if first_at is not None and last_at is not None and content_chunks > 1
+                else None
+            )
+            base["total_latency_ms"] = round(
+                (time.monotonic() - handle_start) * 1000, 3
+            )
+            base["gateway_overhead_ms"] = round(
+                max(
+                    base["total_latency_ms"]
+                    - base.get("upstream_latency_ms", 0.0)
+                    - base.get("queue_wait_ms", 0.0),
+                    0.0,
+                ),
+                3,
+            )
+            base["status_code"] = 200
+            base["success"] = success
+            base["error_category"] = error_category
+            base["usage"] = usage_captured
+            base["ttft_ms"] = ttft_ms
+            base["inter_token_ms"] = inter_token_ms
+            self._ledger.record(base, pricing, context_map)
+        if not saw_done:
+            yield "data: [DONE]\n\n"
+
     async def handle_embeddings(
         self, body: dict, cfg: dict, owner: tuple, user_agent: str, app_reference: str, log=None
     ):
         log = log or (lambda message: None)
         vision_cost = 0.0
         overlay = embed_overlay(body.get("model"), cfg)
+        base_cfg = cfg
         if overlay:
             cfg = {**cfg, **overlay}
             log(f"routed embed model {body.get('model')!r} -> {cfg['gateway_embed_model']!r}")
@@ -463,8 +601,6 @@ class GatewayRuntime:
                 },
             )
         client, sem = self._ensure(cfg)
-        pricing = pricing_from_cfg(cfg)
-        context_map = parse_context_map(cfg.get("gateway_model_context_map"))
         params = extract_params(body)
         handle_start = time.monotonic()
 
@@ -499,11 +635,44 @@ class GatewayRuntime:
             json_body=payload,
         )
 
+        requested_model = requested
+        if _call_failed(resp, exc, timing):
+            fallback_route = resolve_fallback(requested, "embed")
+            fallback_overlay = (
+                embed_overlay(fallback_route.source_model, base_cfg)
+                if fallback_route is not None
+                else None
+            )
+            if fallback_overlay:
+                log(
+                    f"embed model {requested!r} failed, falling back to "
+                    f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_embed_model']!r}"
+                )
+                cfg = {**base_cfg, **fallback_overlay}
+                model = cfg["gateway_embed_model"]
+                payload = dict(payload)
+                payload["model"] = model
+                headers = _fallback_headers(cfg, "gateway_embed_key")
+                resp, exc, timing = await self._send(
+                    client,
+                    sem,
+                    "POST",
+                    cfg["gateway_embed_url"],
+                    headers,
+                    cfg,
+                    log,
+                    json_body=payload,
+                )
+
+        pricing = pricing_from_cfg(cfg)
+        context_map = parse_context_map(cfg.get("gateway_model_context_map"))
+
         base = {
             "owner_kind": owner[0],
             "owner_id": owner[1],
             "backend": "embed",
             "endpoint": "embeddings",
+            "requested_model": requested_model,
             "model": model,
             "user_agent": user_agent,
             "app_reference": app_reference,
@@ -598,6 +767,7 @@ class GatewayRuntime:
     ):
         log = log or (lambda message: None)
         overlay = image_overlay(body.get("model"), cfg)
+        base_cfg = cfg
         if overlay:
             cfg = {**cfg, **overlay}
             log(
@@ -633,8 +803,6 @@ class GatewayRuntime:
                 },
             )
         client, sem = self._ensure(cfg)
-        pricing = pricing_from_cfg(cfg)
-        context_map = parse_context_map(cfg.get("gateway_model_context_map"))
         params = extract_params(body)
         handle_start = time.monotonic()
 
@@ -673,11 +841,44 @@ class GatewayRuntime:
             json_body=payload,
         )
 
+        requested_model = requested
+        if _call_failed(resp, exc, timing):
+            fallback_route = resolve_fallback(requested, "image")
+            fallback_overlay = (
+                image_overlay(fallback_route.source_model, base_cfg)
+                if fallback_route is not None
+                else None
+            )
+            if fallback_overlay:
+                log(
+                    f"image model {requested!r} failed, falling back to "
+                    f"{fallback_route.source_model!r} -> {fallback_overlay['gateway_image_model']!r}"
+                )
+                cfg = {**base_cfg, **fallback_overlay}
+                model = cfg["gateway_image_model"]
+                payload = dict(payload)
+                payload["model"] = model
+                headers = _fallback_headers(cfg, "gateway_image_key")
+                resp, exc, timing = await self._send(
+                    client,
+                    sem,
+                    "POST",
+                    cfg["gateway_image_url"],
+                    headers,
+                    cfg,
+                    log,
+                    json_body=payload,
+                )
+
+        pricing = pricing_from_cfg(cfg)
+        context_map = parse_context_map(cfg.get("gateway_model_context_map"))
+
         base = {
             "owner_kind": owner[0],
             "owner_id": owner[1],
             "backend": "image",
             "endpoint": "images/generations",
+            "requested_model": requested_model,
             "model": model,
             "user_agent": user_agent,
             "app_reference": app_reference,

devplacepy/services/openai_gateway/reliability.py

@@ -97,6 +97,10 @@ async def retry_send(
             continue
         if resp.status_code >= 500 and attempts <= max_retries:
             log(f"upstream {resp.status_code}, retrying ({attempts}/{max_retries})")
+            try:
+                await resp.aclose()
+            except Exception:  # noqa: BLE001 - releasing the connection must never block a retry
+                pass
             await _backoff(backoff_ms, attempts)
             continue
         return resp, None, attempts, queue_wait_ms

devplacepy/services/openai_gateway/routing.py

@@ -87,13 +87,15 @@ def ensure_tables() -> None:
         "price_cache_hit_per_m_tier2 REAL, price_cache_miss_per_m_tier2 REAL, "
         "price_output_per_m_tier2 REAL, price_input_per_m_tier2 REAL, "
         "off_peak_start_minute INTEGER, off_peak_end_minute INTEGER, "
-        "off_peak_discount_pct REAL DEFAULT 0, "
+        "off_peak_discount_pct REAL DEFAULT 0, fallback_model TEXT DEFAULT '', "
         "is_active INTEGER DEFAULT 1, created_at TEXT, updated_at TEXT)"
     )
     models_table = get_table(MODELS_TABLE)
     for column in MODEL_TIER2_COLUMNS:
         if not models_table.has_column(column):
             models_table.create_column_by_example(column, 0.0)
+    if not models_table.has_column("fallback_model"):
+        models_table.create_column_by_example("fallback_model", "")
     try:
         db.query(
             "CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_providers_name ON "
@@ -159,6 +161,7 @@ class ModelRouteIn(BaseModel):
     off_peak_start_minute: Optional[int] = Field(default=None, ge=0, le=1439)
     off_peak_end_minute: Optional[int] = Field(default=None, ge=0, le=1439)
     off_peak_discount_pct: float = Field(default=0.0, ge=0, le=100)
+    fallback_model: str = Field(default="", max_length=128)
     is_active: bool = True
 
     @field_validator("source_model", "target_model")
@@ -169,7 +172,7 @@ class ModelRouteIn(BaseModel):
             raise ValueError("Model name is required")
         return value
 
-    @field_validator("provider", "vision_provider", "vision_model")
+    @field_validator("provider", "vision_provider", "vision_model", "fallback_model")
     @classmethod
     def _strip(cls, value: str) -> str:
         return (value or "").strip()
@@ -192,6 +195,12 @@ class ModelRouteIn(BaseModel):
             )
         return self
 
+    @model_validator(mode="after")
+    def _check_fallback_is_not_self(self) -> "ModelRouteIn":
+        if self.fallback_model and self.fallback_model == self.source_model:
+            raise ValueError("A model cannot fall back to itself")
+        return self
+
 
 @dataclass(frozen=True)
 class ModelRoute:
@@ -214,6 +223,7 @@ class ModelRoute:
     off_peak_start_minute: Optional[int]
     off_peak_end_minute: Optional[int]
     off_peak_discount_pct: float
+    fallback_model: str
     is_active: bool
 
 
@@ -250,6 +260,7 @@ def _route_from_row(row: dict) -> ModelRoute:
         off_peak_start_minute=_opt_int(row, "off_peak_start_minute"),
         off_peak_end_minute=_opt_int(row, "off_peak_end_minute"),
         off_peak_discount_pct=float(row.get("off_peak_discount_pct") or 0.0),
+        fallback_model=str(row.get("fallback_model") or ""),
         is_active=_as_bool(row.get("is_active", 1)),
     )
 
@@ -375,6 +386,7 @@ class ModelStore:
             "off_peak_start_minute": payload.off_peak_start_minute,
             "off_peak_end_minute": payload.off_peak_end_minute,
             "off_peak_discount_pct": payload.off_peak_discount_pct,
+            "fallback_model": payload.fallback_model,
             "is_active": 1 if payload.is_active else 0,
             "updated_at": _now(),
         }
@@ -402,6 +414,18 @@ class ModelStore:
 provider_store = ProviderStore()
 model_store = ModelStore()
 
+
+def resolve_fallback(source_model: Optional[str], kind: str) -> Optional[ModelRoute]:
+    route = model_store.resolve(source_model, kind)
+    if route is None or not route.fallback_model:
+        return None
+    if route.fallback_model == route.source_model:
+        return None
+    fallback = model_store.resolve(route.fallback_model, kind)
+    if fallback is None or fallback.source_model == route.source_model:
+        return None
+    return fallback
+
 DEEPSEEK_DEFAULT_ROUTES = (
     {
         "source_model": "deepseek-v4-flash",

devplacepy/services/openai_gateway/service.py

@@ -120,6 +120,18 @@ class GatewayService(BaseService):
             "content, all in one system message). Leave blank to disable.",
             group="Prompt",
         ),
+        ConfigField(
+            "gateway_thinking",
+            "Enable thinking by default",
+            type="bool",
+            default=config.THINKING_DEFAULT,
+            help="Off by default: the gateway disables model thinking on every chat "
+            "call (DeepSeek thinking.type=disabled, OpenRouter reasoning.effort=none, "
+            "Ollama think=false) unless the client explicitly enables it with think, "
+            "thinking, or reasoning. Fastest path. On: thinking is enabled unless the "
+            "client disables it.",
+            group="Prompt",
+        ),
         ConfigField(
             "gateway_vision_enabled",
             "Vision augmentation",

devplacepy/services/openai_gateway/thinking.py

@@ -0,0 +1,125 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Any, Optional
+
+CLIENT_THINKING_KEYS = (
+    "think",
+    "thinking",
+    "reasoning",
+    "reasoning_effort",
+    "enable_thinking",
+)
+
+FALSE_VALUES = {False, 0, "0", "false", "no", "off", "none", "disabled"}
+TRUE_VALUES = {True, 1, "1", "true", "yes", "on", "enabled"}
+EFFORT_VALUES = {"low", "medium", "high", "max", "xhigh"}
+
+
+def thinking_dialect(url: str) -> str:
+    text = (url or "").lower()
+    if "deepseek.com" in text:
+        return "deepseek"
+    if "openrouter.ai" in text:
+        return "openrouter"
+    if "11434" in text or "ollama" in text:
+        return "ollama"
+    return "deepseek"
+
+
+def _as_text(value: Any) -> str:
+    if isinstance(value, str):
+        return value.strip().lower()
+    return ""
+
+
+def _is_false(value: Any) -> bool:
+    if isinstance(value, str):
+        return value.strip().lower() in FALSE_VALUES
+    return value in FALSE_VALUES
+
+
+def _is_true(value: Any) -> bool:
+    if isinstance(value, str):
+        text = value.strip().lower()
+        return text in TRUE_VALUES or text in EFFORT_VALUES
+    return value in TRUE_VALUES
+
+
+def client_thinking_enabled(body: Any) -> Optional[bool]:
+    if not isinstance(body, dict):
+        return None
+    thinking = body.get("thinking")
+    if isinstance(thinking, dict) and "type" in thinking:
+        kind = _as_text(thinking.get("type"))
+        if kind in ("disabled", "none", "off"):
+            return False
+        if kind in ("enabled", "on"):
+            return True
+    if "think" in body:
+        value = body.get("think")
+        if _is_false(value):
+            return False
+        if _is_true(value):
+            return True
+        return bool(value)
+    reasoning = body.get("reasoning")
+    if isinstance(reasoning, dict):
+        if "enabled" in reasoning:
+            return _flag_enabled(reasoning.get("enabled"))
+        effort = _as_text(reasoning.get("effort"))
+        if effort == "none":
+            return False
+        if effort in EFFORT_VALUES:
+            return True
+    if "reasoning_effort" in body:
+        effort = _as_text(body.get("reasoning_effort"))
+        if effort == "none":
+            return False
+        if effort in EFFORT_VALUES:
+            return True
+    if "enable_thinking" in body:
+        return _flag_enabled(body.get("enable_thinking"))
+    kwargs = body.get("chat_template_kwargs")
+    if isinstance(kwargs, dict) and "enable_thinking" in kwargs:
+        return _flag_enabled(kwargs.get("enable_thinking"))
+    return None
+
+
+def _flag_enabled(value: Any) -> bool:
+    if _is_false(value):
+        return False
+    if _is_true(value):
+        return True
+    return bool(value)
+
+
+def _clear_thinking_fields(payload: dict) -> None:
+    for key in CLIENT_THINKING_KEYS:
+        payload.pop(key, None)
+    kwargs = payload.get("chat_template_kwargs")
+    if isinstance(kwargs, dict):
+        kwargs.pop("enable_thinking", None)
+        if not kwargs:
+            payload.pop("chat_template_kwargs", None)
+
+
+def write_thinking(payload: dict, dialect: str, enabled: bool) -> None:
+    _clear_thinking_fields(payload)
+    if dialect == "openrouter":
+        payload["reasoning"] = {"effort": "high" if enabled else "none"}
+        return
+    if dialect == "ollama":
+        payload["think"] = bool(enabled)
+        return
+    payload["thinking"] = {"type": "enabled" if enabled else "disabled"}
+
+
+def apply_thinking(
+    payload: dict,
+    upstream_url: str,
+    default_enabled: bool = False,
+) -> dict:
+    specified = client_thinking_enabled(payload)
+    enabled = default_enabled if specified is None else specified
+    write_thinking(payload, thinking_dialect(upstream_url), enabled)
+    return payload

devplacepy/services/openai_gateway/usage.py

@@ -533,6 +533,8 @@ class GatewayUsageLedger:
                 "circuit_open": 1 if raw.get("circuit_open") else 0,
                 "user_agent": (raw.get("user_agent") or "")[:300],
                 "app_reference": (raw.get("app_reference") or "default")[:30],
+                "ttft_ms": raw.get("ttft_ms"),
+                "inter_token_ms": raw.get("inter_token_ms"),
             }
             get_table(GATEWAY_LEDGER).insert(row)
             self._audit(raw, norm, cost_usd)
@@ -595,6 +597,8 @@ class GatewayUsageLedger:
                 "circuit_open": 0,
                 "user_agent": "",
                 "app_reference": app_reference or "default",
+                "ttft_ms": None,
+                "inter_token_ms": None,
             }
             get_table(GATEWAY_LEDGER).insert(row)
             self._audit_external(row)

devplacepy/services/openai_gateway/vision.py

@@ -10,6 +10,7 @@ from typing import Any, Optional
 import httpx
 
 from devplacepy.services.openai_gateway.config import VISION_INSTRUCTION
+from devplacepy.services.openai_gateway.thinking import apply_thinking
 from devplacepy.services.openai_gateway.usage import classify_error
 
 logger = logging.getLogger(__name__)
@@ -153,6 +154,7 @@ class VisionAugmenter:
             "temperature": 0.2,
             "stream": False,
         }
+        apply_thinking(payload, self.vision_url, default_enabled=False)
         headers = {
             "Authorization": f"Bearer {self.vision_key}",
             "Content-Type": "application/json",

devplacepy/static/js/GatewayAdmin.js

@@ -11,6 +11,7 @@ export class GatewayAdmin {
         this.modelForm = root.querySelector("#gw-model-form");
         this.providerSelects = root.querySelectorAll("[data-provider-select]");
         this.providers = [];
+        this.models = [];
         this.quotaRulesBody = root.querySelector("#gw-quota-rules");
         this.quotaForm = root.querySelector("#gw-quota-form");
         this.quotaCancelEdit = root.querySelector("#gw-quota-cancel-edit");
@@ -36,6 +37,24 @@ export class GatewayAdmin {
             const kinds = (field.dataset.kindField || "").split(/\s+/).filter(Boolean);
             field.style.display = kinds.includes(kind) ? "" : "none";
         });
+        this.fillFallbackSelect(this.modelForm.source_model.value);
+    }
+
+    fillFallbackSelect(excludeSource) {
+        const select = this.modelForm.fallback_model;
+        if (!select) return;
+        const kind = this.modelForm.kind.value || "chat";
+        const current = select.value;
+        const options = [`<option value="">(none)</option>`].concat(
+            this.models
+                .filter((m) => m.kind === kind && m.source_model !== excludeSource)
+                .map(
+                    (m) =>
+                        `<option value="${this.attr(m.source_model)}">${this.escape(m.source_model)}</option>`
+                )
+        );
+        select.innerHTML = options.join("");
+        select.value = current;
     }
 
     bind() {
@@ -138,6 +157,8 @@ export class GatewayAdmin {
     async loadModels() {
         const data = await Http.getJson("/admin/gateway/models");
         const models = data.models || [];
+        this.models = models;
+        this.fillFallbackSelect(this.modelForm.source_model.value);
         if (!models.length) {
             this.modelsBody.innerHTML = `<tr><td colspan="7" class="admin-empty">No model routes. Requests fall through to the default upstream.</td></tr>`;
             return 0;
@@ -160,6 +181,9 @@ export class GatewayAdmin {
                 if (m.kind === "image" && m.price_input_per_m) {
                     badges.push(`<span class="gw-badge" title="Flat price per generated image">$${m.price_input_per_m}/img</span>`);
                 }
+                if (m.fallback_model) {
+                    badges.push(`<span class="gw-badge" title="Falls back to ${this.escape(m.fallback_model)} on failure">fallback: ${this.escape(m.fallback_model)}</span>`);
+                }
                 const economy = badges.length ? badges.join(" ") : `<span class="gw-muted">-</span>`;
                 return `<tr>
                     <td>${this.escape(m.source_model)}</td>
@@ -248,6 +272,7 @@ export class GatewayAdmin {
             off_peak_start_minute: this.timeToMinutes(values.off_peak_start),
             off_peak_end_minute: this.timeToMinutes(values.off_peak_end),
             off_peak_discount_pct: parseFloat(values.off_peak_discount_pct) || 0,
+            fallback_model: values.fallback_model || "",
             is_active: values.is_active === "1",
         };
         try {
@@ -330,6 +355,7 @@ export class GatewayAdmin {
         form.off_peak_start.value = this.minutesToTime(model.off_peak_start_minute);
         form.off_peak_end.value = this.minutesToTime(model.off_peak_end_minute);
         form.off_peak_discount_pct.value = model.off_peak_discount_pct || 0;
+        form.fallback_model.value = model.fallback_model || "";
         form.is_active.value = model.is_active ? "1" : "0";
         form.source_model.scrollIntoView({ block: "center" });
     }

devplacepy/templates/admin_gateway.html

@@ -40,7 +40,7 @@
         <div class="gw-section-head">
             <h3>Model routes</h3>
         </div>
-        <p class="gw-section-hint">Each source model maps to a provider and target model with its own economy. Chat uses cache-hit, cache-miss and output prices (USD per 1M tokens); embeddings use the input price; image routes use the input price as a flat USD per image; a vision model adds input and output pricing for the image description merge.</p>
+        <p class="gw-section-hint">Each source model maps to a provider and target model with its own economy. Chat uses cache-hit, cache-miss and output prices (USD per 1M tokens); embeddings use the input price; image routes use the input price as a flat USD per image; a vision model adds input and output pricing for the image description merge. An optional fallback model is retried once, automatically, whenever this route fails after its own retries are exhausted - pick any other already-configured model of the same kind.</p>
         <div class="admin-table-wrap">
             <table class="admin-table">
                 <caption class="sr-only">Model routes</caption>
@@ -60,6 +60,7 @@
             <div class="gw-field" data-kind-field="chat"><label for="gw-model-vision-model">Vision model</label><input type="text" id="gw-model-vision-model" name="vision_model" placeholder="(optional) google/gemma-3-12b-it"></div>
             <div class="gw-field"><label for="gw-model-context">Context window</label><input type="number" id="gw-model-context" name="context_window" min="0" value="0"></div>
             <div class="gw-field"><label for="gw-model-active">Active</label><select id="gw-model-active" name="is_active"><option value="1">Yes</option><option value="0">No</option></select></div>
+            <div class="gw-field"><label for="gw-model-fallback">Fallback model</label><select id="gw-model-fallback" name="fallback_model" title="Tried automatically when this model fails after retries. Only other models of the same kind are offered."><option value="">(none)</option></select></div>
             <div class="gw-field" data-kind-field="chat"><label for="gw-model-price-cache-hit">Price cache-hit / 1M ($)</label><input type="number" id="gw-model-price-cache-hit" name="price_cache_hit_per_m" min="0" step="0.0001" value="0"></div>
             <div class="gw-field" data-kind-field="chat"><label for="gw-model-price-cache-miss">Price cache-miss / 1M ($)</label><input type="number" id="gw-model-price-cache-miss" name="price_cache_miss_per_m" min="0" step="0.0001" value="0"></div>
             <div class="gw-field" data-kind-field="chat"><label for="gw-model-price-output">Price output / 1M ($)</label><input type="number" id="gw-model-price-output" name="price_output_per_m" min="0" step="0.0001" value="0"></div>

devplacepy/templates/docs/services-gateway.html

@@ -34,7 +34,7 @@ Chat requests are augmented (vision), forwarded, retried, priced, and ledgered.
 
 **Upstream:** `gateway_upstream_url` (default DeepSeek chat-completions), `gateway_model` (`deepseek-v4-flash`), `gateway_force_model` (override the client-requested model, default on), `gateway_api_key` (auto-migrated from `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY`), `gateway_timeout` (default and minimum 300s), `gateway_instances` (max concurrent forwards per worker, default 4, 1 to 64; this is the connection pool plus semaphore).
 
-**Prompt:** `gateway_system_preamble` (multiline text, default empty) - operator text prepended ahead of every chat request's system message (see "System-message composition" below).
+**Prompt:** `gateway_system_preamble` (multiline text, default empty) - operator text prepended ahead of every chat request's system message (see "System-message composition" below). `gateway_thinking` (bool, default off) - when off, the gateway disables model thinking on every chat call unless the client explicitly enables it; when on, thinking is enabled unless the client disables it (see "Thinking default" below).
 
 **Vision:** `gateway_vision_enabled` (default on), `gateway_vision_url` (OpenRouter by default), `gateway_vision_model` (`google/gemma-3-12b-it`), `gateway_vision_key` (auto-migrated from `OPENROUTER_API_KEY`), `gateway_vision_cache_size` (LRU entries, default 256, 0 disables).
 
@@ -86,6 +86,16 @@ The operator preamble (`gateway_system_preamble`) is prepended on every chat cal
 
 The current date is injected as `Current date: DD/MM/YYYY` (EU format) only when the composed text contains no date in any common format already (ISO `YYYY-MM-DD`, `D/M/YYYY`, `DD/MM/YYYY`, `MM/DD/YYYY`, `DD-MM-YYYY`, `DD.MM.YYYY`, and textual months such as `14 June 2026`, `June 14, 2026`, `14 Jun 2026`). The date is intentionally date-only with no time: keeping it stable for the whole day preserves upstream prompt and token caching.
 
+## Thinking default
+
+DeepSeek V4 enables thinking unless the request says otherwise, which is slow. The gateway therefore always writes an explicit thinking field before forwarding chat completions:
+
+- DeepSeek and unrecognised OpenAI-compat URLs: `thinking: {"type": "disabled"}` (or `"enabled"`)
+- OpenRouter: `reasoning: {"effort": "none"}` (or `"high"`)
+- Ollama: `think: false` (or `true`)
+
+The default is **off**. A client overrides it for one call with `think`, `thinking.type`, `reasoning.effort`, `reasoning.enabled`, `reasoning_effort`, or `enable_thinking`. An administrator overrides the silent-client default with `gateway_thinking` on the openai service. Vision describe-image calls always disable thinking.
+
 ## Vision augmentation
 
 Before forwarding a chat request, image blocks are described by the vision model and replaced with a text description, so a text-only upstream still answers questions about images. Descriptions are cached in an LRU keyed by image hash; vision calls are themselves ledgered with `backend="vision"`. If vision is disabled or the key is missing, images become a placeholder note rather than failing the request.
@@ -94,9 +104,11 @@ Before forwarding a chat request, image blocks are described by the vision model
 
 **Live metrics** (in `describe()`, at `GET /admin/services/data`): runtime counters (`requests`, `errors`, `in_flight`, `peak_in_flight`, `vision_calls`, `embed_calls`, `image_calls`, `last_status`, `last_latency_ms`, `pool`, `circuit_open`) and a 24h summary (`requests`, `success_pct`, `error_pct`, `cost_hour`, `cost_24h`, `tokens_24h`, `avg_latency_ms`, `avg_tps`, `peak_concurrency`, `top_model`, `top_caller`).
 
-**Ledger** `gateway_usage_ledger`: one row per upstream call with owner, backend, requested and actual model, status, success, error category, the full latency breakdown (`upstream_latency_ms`, `gateway_overhead_ms`, `queue_wait_ms`, `connect_ms`, `total_latency_ms`), token counts (prompt, completion, cache hit and miss, reasoning, total), `tokens_per_second`, context window and utilization, the cost fields (`cost_usd`, `input_cost_usd`, `output_cost_usd`, `native_cost`), request shape (`stream_requested`, `temperature`, `top_p`, `max_tokens`, `has_tools`), and reliability flags (`retries_attempted`, `retry_succeeded`, `circuit_open`). Pruned to `gateway_usage_retention_hours`.
+**Ledger** `gateway_usage_ledger`: one row per upstream call with owner, backend, requested and actual model, status, success, error category, the full latency breakdown (`upstream_latency_ms`, `gateway_overhead_ms`, `queue_wait_ms`, `connect_ms`, `total_latency_ms`, and for a streamed call `ttft_ms`/`inter_token_ms`), token counts (prompt, completion, cache hit and miss, reasoning, total), `tokens_per_second`, context window and utilization, the cost fields (`cost_usd`, `input_cost_usd`, `output_cost_usd`, `native_cost`), request shape (`stream_requested`, `temperature`, `top_p`, `max_tokens`, `has_tools`), and reliability flags (`retries_attempted`, `retry_succeeded`, `circuit_open`). Pruned to `gateway_usage_retention_hours`.
 
 **Concurrency samples** `gateway_concurrency_samples`: `in_flight` sampled once per service tick.
 
-**Deep analytics** at `GET /admin/ai-usage/data` (`GatewayUsageOut`): per-hour buckets, token and latency percentiles, error breakdowns, cost (this hour, 24h, projected monthly, caching savings, by model, by caller), and behavior. TTFT and inter-token latency are null because the gateway calls the upstream non-streaming. Per-user usage is at `GET /admin/users/{uid}/ai-usage`. It is all rendered on `/admin/ai-usage` by `AiUsageMonitor.js`.
+**Real streaming.** `stream: true` is forwarded to the upstream as-is and its SSE chunks are relayed to the client as they arrive, rather than simulated from a buffered full response - so TTFT (time to first chunk) and inter-token latency are genuine measured timings. A streaming response carries only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference` (no cost/token headers, since those aren't known until the stream ends); the call is still fully metered via the usage ledger, written once the stream completes, and via the client's own final `usage` SSE chunk when it requests `stream_options.include_usage`. Non-streaming calls are unaffected and keep the full header set.
+
+**Deep analytics** at `GET /admin/ai-usage/data` (`GatewayUsageOut`): per-hour buckets, token and latency percentiles, error breakdowns, cost (this hour, 24h, projected monthly, caching savings, by model, by caller), and behavior. TTFT and inter-token latency (`latency.ttft_ms`/`latency.inter_token_ms`) are real, measured only for calls that requested streaming - a zero count in a window means no streaming traffic, not that the metric is unavailable. Per-user usage is at `GET /admin/users/{uid}/ai-usage`. It is all rendered on `/admin/ai-usage` by `AiUsageMonitor.js`.
 </div>

devplacepy/templates/workspace.html

@@ -17,7 +17,13 @@
   <div class="card workspace-empty">
     <p>No workspace yet for this project. Opening one starts a container with your
        project files, a browser editor, and a terminal.</p>
-    <p class="workspace-muted">You are using {{ workspace_count }} of {{ max_workspaces }} workspaces.</p>
+    <p class="workspace-muted">
+      {% if unlimited_workspaces %}
+      You are using {{ workspace_count }} workspaces. Administrators are unlimited.
+      {% else %}
+      You are using {{ workspace_count }} of {{ max_workspaces }} workspaces.
+      {% endif %}
+    </p>
     <form method="post" action="/projects/{{ project.slug or project.uid }}/workspace">
       <button type="submit" class="btn btn-primary">Open workspace</button>
     </form>
@@ -53,8 +59,12 @@
       </div>
     </div>
     <p class="workspace-muted">
+      {% if workspace.unlimited %}
+      Administrator workspace: never auto-stopped or removed for being idle.
+      {% else %}
       Stops after {{ workspace.idle_stop_minutes }} minutes idle. Removed after
       {{ workspace.retention_days }} days idle.
+      {% endif %}
       {% if workspace.last_active_at %}
       Last active {{ dt_ago(workspace.last_active_at) }}.
       {% endif %}
@@ -213,7 +223,12 @@
     <h2>Public tunnels</h2>
     <p class="workspace-muted">
       A tunnel publishes a port from inside your container on a public HTTPS address.
-      Anyone with the link can reach it. You may have up to {{ workspace.max_tunnels }}.
+      Anyone with the link can reach it.
+      {% if workspace.unlimited %}
+      Administrators are unlimited.
+      {% else %}
+      You may have up to {{ workspace.max_tunnels }}.
+      {% endif %}
     </p>
     <ul class="workspace-tunnel-list" data-tunnel-list>
       {% for tunnel in workspace.tunnels %}