Trust only the upstream X-Gateway-Model header in the AI gateway
An upstream the gateway forwards to may itself emit X-Gateway-* headers (e.g. another DevPlace-style gateway), which can collide with the ones about to be built for the response. Only X-Gateway-Model is ever trusted from upstream and relayed as-is - it is the one field an upstream can legitimately know better than we do (it may have resolved an alias or served a different pinned version). Every other header (cost, tokens, latency, context, app-reference) is always our own measurement and is never overwritten, since blending in an upstream's own accounting would corrupt the usage ledger's per-model rollups and the quota math built on top of it. usage.upstream_reported_model() extracts that one header defensively (case-insensitive lookup, rejects anything oversized or containing a control character) and gateway._apply_served_model() applies it, display- only, at the tail of every response-header build across chat, streaming, embeddings, images, and passthrough. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BWJy6PrMMt5hwWxQwia2rd
This commit is contained in:
@@ -14,6 +14,15 @@ Every NON-STREAMING gateway response (chat, embeddings, images, passthrough; suc
|
||||
|
||||
**A STREAMING chat response (`stream: true`) is the one exception and carries only `X-Gateway-Model`/`X-Gateway-Backend`/`X-App-Reference`** - HTTP headers are sent before the body, and cost/tokens for a streamed call are only known once the stream ends, so they cannot be headers on that same response. The call is still fully metered: the ledger row (including `ttft_ms`/`inter_token_ms`) is written server-side once the stream completes (see "Real upstream streaming" below), and a client that requests `stream_options.include_usage` still gets the upstream's real `usage` object on the final SSE chunk, exactly as the OpenAI streaming API itself works - it is simply not summarized into response headers.
|
||||
|
||||
### Upstream header collision policy (`usage.upstream_reported_model`, `gateway._apply_served_model`)
|
||||
|
||||
An upstream this gateway forwards to may itself be a DevPlace-style gateway (or any provider that happens to emit the same `X-Gateway-*` header names on its own response), so its raw response headers can collide with the ones we are about to build. **Exactly one upstream header is ever trusted and relayed: `X-Gateway-Model`.** Every other `X-Gateway-*`/`X-App-Reference` header is always our own measurement, computed as if the upstream had sent nothing at all - an upstream cannot be an authority on our own cost table, our own tokenizer counts, our own socket timings, or our own caller attribution.
|
||||
|
||||
- **Why only the model name.** It is the one field an upstream can legitimately know better than we do: it may have resolved an alias, redirected to a fallback, or served a different pinned version than what we requested. Cost/tokens/latency/context/app-reference are, by construction, things WE measured about OUR call (our pricing table, our tokenizer expectations, our socket, our caller) - an upstream's own figures for those describe its own accounting, not ours, and blending them in would corrupt `gateway_usage_ledger`'s per-model cost/token rollups and the quota math that reads these exact header names back out (`usage.parse_usage_headers`, consumed by `services/correction.py`/`services/ai_modifier.py`).
|
||||
- **`usage.upstream_reported_model(headers)`** extracts that one value defensively: it treats the upstream response as untrusted input and never raises. It looks up `X-Gateway-Model` case-insensitively (a direct `.get`, then a manual case-insensitive scan for a plain non-case-insensitive mapping), and rejects anything that is not a non-empty string, exceeds 200 characters, or contains a control character (`\r`/`\n`/`\x00`, a header-injection guard) - any of those makes it return `None`, i.e. "upstream said nothing usable."
|
||||
- **`gateway._apply_served_model(headers, resp)`** is the one place this is applied: called at the tail of every response-header build (chat non-streaming and streaming, embeddings, images, passthrough), it overwrites `headers["X-Gateway-Model"]` with the upstream's claim ONLY when one was found, and leaves every other header untouched. It never touches the ledger row itself (`GatewayUsageLedger.record`'s `model` field, and therefore `context_utilization`/pricing lookups, stay keyed on the model WE resolved and billed against) - the override is display-only, informing the client which model actually answered without altering what we measured or charged for.
|
||||
- **Tests:** `tests/unit/services/openai_gateway/usage.py` (`upstream_reported_model` - case-insensitivity, missing/oversized/control-character rejection, non-dict/`None` inputs) and `tests/unit/services/openai_gateway/gateway.py` (`test_upstream_x_gateway_model_header_is_forwarded_*` - chat/streaming/embeddings/images/passthrough all forward an upstream-reported model while every cost/token header stays ours; `test_upstream_x_gateway_model_header_absent_keeps_our_own_model` - no upstream header is a no-op).
|
||||
|
||||
## App-reference header (`X-App-Reference`)
|
||||
|
||||
Callers may send an optional `X-App-Reference` header to tag gateway calls by application. The value is validated and stored in the `app_reference` column of `gateway_usage_ledger`, surfaced in analytics and admin reporting.
|
||||
|
||||
@@ -31,6 +31,7 @@ from devplacepy.services.openai_gateway.usage import (
|
||||
extract_params,
|
||||
parse_context_map,
|
||||
pricing_from_cfg,
|
||||
upstream_reported_model,
|
||||
usage_response_headers,
|
||||
)
|
||||
from devplacepy.services.openai_gateway.vision import VisionAugmenter, VisionCache
|
||||
@@ -44,6 +45,19 @@ def _call_failed(resp, exc, timing: dict) -> bool:
|
||||
return resp is not None and resp.status_code >= 400
|
||||
|
||||
|
||||
def _apply_served_model(headers: dict, resp) -> dict:
|
||||
# Everything else in `headers` (cost, tokens, latency, context) is our
|
||||
# own measurement and must never be overwritten by an upstream that
|
||||
# happens to echo the same X-Gateway-* header names - only the served
|
||||
# model name is worth trusting from upstream. See usage.upstream_reported_model.
|
||||
if not headers or resp is None:
|
||||
return headers
|
||||
served = upstream_reported_model(getattr(resp, "headers", None))
|
||||
if served:
|
||||
headers["X-Gateway-Model"] = served
|
||||
return headers
|
||||
|
||||
|
||||
def _fallback_headers(cfg: dict, key_field: str) -> dict:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if cfg.get(key_field):
|
||||
@@ -407,7 +421,7 @@ class GatewayRuntime:
|
||||
chat_cost = float((row or {}).get("cost_usd") or 0.0)
|
||||
headers["X-Gateway-Cost-USD"] = f"{chat_cost + vision_cost:.8f}"
|
||||
headers["X-Gateway-Vision-Cost-USD"] = f"{vision_cost:.8f}"
|
||||
return headers
|
||||
return _apply_served_model(headers, resp)
|
||||
|
||||
if timing["circuit_open"]:
|
||||
resp_headers = finalize(503, False, "circuit_open")
|
||||
@@ -465,11 +479,14 @@ class GatewayRuntime:
|
||||
log,
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"X-Gateway-Model": model,
|
||||
"X-Gateway-Backend": "chat",
|
||||
"X-App-Reference": app_reference or "default",
|
||||
},
|
||||
headers=_apply_served_model(
|
||||
{
|
||||
"X-Gateway-Model": model,
|
||||
"X-Gateway-Backend": "chat",
|
||||
"X-App-Reference": app_reference or "default",
|
||||
},
|
||||
resp,
|
||||
),
|
||||
)
|
||||
try:
|
||||
data = resp.json()
|
||||
@@ -734,7 +751,7 @@ class GatewayRuntime:
|
||||
chat_cost = float((row or {}).get("cost_usd") or 0.0)
|
||||
headers["X-Gateway-Cost-USD"] = f"{chat_cost + vision_cost:.8f}"
|
||||
headers["X-Gateway-Vision-Cost-USD"] = f"{vision_cost:.8f}"
|
||||
return headers
|
||||
return _apply_served_model(headers, resp)
|
||||
|
||||
if timing["circuit_open"]:
|
||||
resp_headers = finalize(503, False, "circuit_open")
|
||||
@@ -939,7 +956,7 @@ class GatewayRuntime:
|
||||
base["error_category"] = category
|
||||
base["usage"] = usage
|
||||
row = self._ledger.record(base, pricing, context_map)
|
||||
return usage_response_headers(row)
|
||||
return _apply_served_model(usage_response_headers(row), resp)
|
||||
|
||||
if timing["circuit_open"]:
|
||||
resp_headers = finalize(503, False, "circuit_open")
|
||||
@@ -1064,7 +1081,7 @@ class GatewayRuntime:
|
||||
chat_cost = float((row or {}).get("cost_usd") or 0.0)
|
||||
headers["X-Gateway-Cost-USD"] = f"{chat_cost + vision_cost:.8f}"
|
||||
headers["X-Gateway-Vision-Cost-USD"] = f"{vision_cost:.8f}"
|
||||
return headers
|
||||
return _apply_served_model(headers, resp)
|
||||
|
||||
if timing["circuit_open"]:
|
||||
resp_headers = finalize(503, False, "circuit_open")
|
||||
|
||||
@@ -408,6 +408,49 @@ def usage_response_headers(row: Optional[dict]) -> dict:
|
||||
return headers
|
||||
|
||||
|
||||
UPSTREAM_MODEL_HEADER_MAX_LEN = 200
|
||||
|
||||
|
||||
def upstream_reported_model(headers: Any) -> Optional[str]:
|
||||
"""Extract a served-model name an upstream (which may itself be another
|
||||
DevPlace-style gateway) reported on its own X-Gateway-Model response
|
||||
header, if present and well-formed.
|
||||
|
||||
This is the ONE upstream response header this gateway ever trusts and
|
||||
relays: model name is the single field an upstream can know better than
|
||||
we do (it may have resolved an alias/fallback on its own side). Every
|
||||
other X-Gateway-* figure (cost, tokens, latency, context) stays our own
|
||||
measurement regardless of what an upstream claims - see usage_response_headers.
|
||||
|
||||
An upstream response is untrusted input, so this never raises: a missing,
|
||||
malformed, oversized, or control-character-laced value is treated as
|
||||
absent rather than propagated or allowed to break our own response.
|
||||
"""
|
||||
if not headers:
|
||||
return None
|
||||
value = None
|
||||
try:
|
||||
value = headers.get("X-Gateway-Model")
|
||||
except Exception:
|
||||
value = None
|
||||
if value is None:
|
||||
try:
|
||||
for key, candidate in headers.items():
|
||||
if isinstance(key, str) and key.lower() == "x-gateway-model":
|
||||
value = candidate
|
||||
break
|
||||
except Exception:
|
||||
value = None
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
value = value.strip()
|
||||
if not value or len(value) > UPSTREAM_MODEL_HEADER_MAX_LEN:
|
||||
return None
|
||||
if any(ch in value for ch in ("\r", "\n", "\x00")):
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
def parse_usage_headers(headers) -> Optional[dict]:
|
||||
if not headers or "X-Gateway-Cost-USD" not in headers:
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user