feat(clients): compat: OpenAI-compatible provider registry (39 rows), chat model, services, demo

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-25 15:01:38 +01:00
co-authored by Claude Sonnet 5.5
parent 3db67fcf1f
commit aaddb29d42
23 changed files with 3348 additions and 3 deletions
+243
View File
@@ -0,0 +1,243 @@
# OpenAI-compatible provider registry (compat) - Specification
Owned directory: `src/vendors/compat/`. Namespace `molo::compat`. Depends on
the `src/llm` foundation (`ApiClient`, `Service`, `SseDecoder`, `ApiError`,
`Config`, `ChatModel`). Plan: `plans/18-breadth.md` section 2 (method:
`plans/17-llm-vendor-platforms.md`). Credentials: one env var per provider,
named in the table of section 4, read through `llm::Config`.
## 1. Scope
One client for every provider that serves OpenAI-shaped endpoints. A provider
is ONE ROW in `registry.cpp`; clients, routes, demo page and tests iterate the
table, so adding a provider is one row plus one line in section 4 (enforced
by `tools/check_spec.py`).
Endpoints, all OpenAI operations (section 3 of the checker verifies them
against OpenAI's OpenAPI document): chat completions (also streaming),
embeddings, models list, image generation, audio transcription, speech. A row
only claims an endpoint its own docs list or a probe distinguished; a call
outside the row's claim is refused before any network traffic.
Not here, said plainly: legacy completions, moderations, translations,
image edits and variations, stored completions, the Responses API (no row's
docs were read for it except Cloudflare's, which lists `/responses`; not
wrapped), files, batches, fine-tuning. Providers with their own native API
have their own packages (mistral, cohere, perplexity, huggingface, ...).
Providers whose OpenAI-compatible endpoint could not be verified are in
section 5 and are not rows.
## 2. Module map
| File | Concern |
|---|---|
| `caps.h` | Static ceilings (`CompatCaps`) |
| `registry.h/.cpp` | `ProviderRow`, `Feature`, `Quirk`, `Evidence`, `Registry` and THE table |
| `calls.h/.cpp` | The call table (6 rows, each an OpenAI `operationId`) |
| `provider.h/.cpp` | `Provider` (row + `ApiClient`, base URL resolution and validation), `ProviderService` base |
| `chat_model.h/.cpp` | `CompatChatModel : llm::ChatModel` (generate, stream, wire mapping, quirks) |
| `services.h/.cpp` | `CompatModels`, `CompatEmbeddings`, `CompatImages`, `CompatAudio` |
| `tools/check_spec.py` | Checks the call table against OpenAI's OpenAPI document and the registry against section 4 |
## 3. Contracts and static caps
| Knob | Value |
|---|---|
| messages / tools / stop sequences | 4096 / 128 / 16 |
| max_tokens | 1..4194304 |
| temperature / top_p | 0..2 / (0,1] |
| n (extra) | 1..8; exactly 1 on rows with `kSingleChoice` |
| stream events | 200000 per stream, then the stream is stopped |
| models returned | 5000, then `truncated=true` (loud) |
| embedding inputs / chars per input | 2048 / 1 MiB |
| image n / prompt chars | 1..10 / 32768 |
| audio upload / speech input chars | 64 MiB / 4096 |
| model id chars / base URL chars | 256 / 512 |
- Auth is `Authorization: Bearer <key>` for every row (the shape OpenAI
clients use). Local rows (`local=true`) are "configured" only when their
`base_env` is set; the key is optional and the placeholder `local` is sent
when none is set. Cloudflare needs `CLOUDFLARE_ACCOUNT_ID` (validated
`[A-Za-z0-9_-]{1,64}`) and substitutes it into the base URL.
- `<ID>_BASE_URL` style overrides are validated: `http(s)://`, no spaces,
control characters, `?`, `#` or `..`; trailing `/` removed.
- The model id is always required: ids differ per provider, so the module has
no honest default (`CompatChatModel` takes an optional default at construction).
- Quirks that the docs state as a 400 are rejected loudly before traffic
(`Quirk::kRejectMessageName`, `kRejectLogprobs`, `kSingleChoice`,
`kNoToolsWithFormat`); everything else is passed through untouched.
- Errors: every non-2xx maps to `llm::ApiError` (the shared envelope parser
handles `{error:{...}}`, `{message,type,code}`, `{detail}` and plain text).
A stream without `finish_reason` throws 502 `STREAM_TRUNCATED`.
- Model list shapes: `{"data":[...]}` and a bare array are both accepted.
## 4. Provider table
Source of truth is `registry.cpp`; this table must match it (checker).
Evidence: `docs+probe` = the provider's docs text names the base URL and an
unauthenticated probe on 2026-10-25 answered like an API; `probe-only` = the
docs page was not readable as text or did not name the URL, the probe answered
like an API; `docs-only` = named by docs, not probeable from here.
| id | name | base URL | credential | features | evidence | documentation | documented quirks and probe results |
|---|---|---|---|---|---|---|---|
| `groq` | Groq | `https://api.groq.com/openai/v1` | `GROQ_API_KEY` | chat,models,transcribe | docs+probe | https://console.groq.com/docs/openai | Docs: logprobs, logit_bias, top_logprobs and messages[].name answer 400; n must be 1; temperature 0 becomes 1e-8; transcription formats vtt and srt unsupported. Probe: 401 invalid_api_key on /models and chat. |
| `together` | Together AI | `https://api.together.ai/v1` | `TOGETHER_API_KEY` | chat,embeddings,models,images,transcribe,speech | docs+probe | https://docs.together.ai/docs/openai-api-compatibility | Docs: logit_bias unsupported on most models; service_tier, store, metadata and prediction are accepted and ignored. Probe: 401 on /models and chat; legacy host api.together.xyz answers 401 too. |
| `fireworks` | Fireworks AI | `https://api.fireworks.ai/inference/v1` | `FIREWORKS_API_KEY` | chat,embeddings,models | docs+probe | https://docs.fireworks.ai/tools-sdks/openai-compatibility | Probe: /models 401 with a docs link; unknown model answers 404 NOT_FOUND; empty embeddings body answers 500. |
| `cerebras` | Cerebras | `https://api.cerebras.ai/v1` | `CEREBRAS_API_KEY` | chat,models | docs+probe | https://inference-docs.cerebras.ai/resources/openai | Docs: tools and response_format must not be combined (gpt-oss-120b rejects it); system and developer roles both act at developer level. Probe: /models 403 {detail}, chat 401 wrong_api_key, embeddings path 404. |
| `sambanova` | SambaNova | `https://api.sambanova.ai/v1` | `SAMBANOVA_API_KEY` | chat,embeddings,models | probe-only | https://docs.sambanova.ai/docs/en/features/openai-compatibility | Docs page answers 403 to a plain fetch. Probe: /models 200 public; chat 401 (also accepts x-api-key); embeddings answers 400 'must provide a model' before auth. |
| `deepinfra` | DeepInfra | `https://api.deepinfra.com/v1/openai` | `DEEPINFRA_API_KEY` | chat,embeddings,models,images,transcribe | docs+probe | https://docs.deepinfra.com/chat/overview | Probe: /models 200 public; chat 401 missing API key; images and audio paths answer 401 with a {detail} body. |
| `novita` | Novita AI | `https://api.novita.ai/openai/v1` | `NOVITA_API_KEY` | chat,models | docs+probe | https://novita.ai/docs/guides/llm-api | Probe: /models 200 public; bad key answers 403 INVALID_API_KEY (not 401); legacy base /v3/openai answers the same. |
| `nvidia` | NVIDIA NIM (build.nvidia.com) | `https://integrate.api.nvidia.com/v1` | `NVIDIA_API_KEY` | chat,embeddings,models | docs+probe | https://docs.api.nvidia.com/nim/reference/llm-apis | Probe: /models 200 public; embeddings answers 400 'model field is required'; retired models answer 410 Gone. |
| `cloudflare` | Cloudflare Workers AI | `https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1` | `CLOUDFLARE_API_TOKEN + CLOUDFLARE_ACCOUNT_ID` | chat,embeddings | docs-only | https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/ | Docs list /chat/completions, /embeddings and /responses under the account path. Probe: GET /models answers 405 (no list endpoint here); not probeable further without an account id. |
| `moonshot` | Moonshot (Kimi) | `https://api.moonshot.ai/v1` | `MOONSHOT_API_KEY` | chat,models | docs+probe | https://platform.kimi.ai/docs/api/chat | Probe: 401 incorrect_api_key_error on /models and chat; the China host api.moonshot.cn answers the same (separate keys). |
| `dashscope` | Alibaba DashScope (Qwen, intl) | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `DASHSCOPE_API_KEY` | chat,embeddings | docs+probe | https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope | Probe: 401 on /models and chat; embeddings answers 400 'you must provide a model parameter'; China host dashscope.aliyuncs.com/compatible-mode/v1 answers 401 too. |
| `zhipu` | Zhipu (GLM, China) | `https://open.bigmodel.cn/api/paas/v4` | `ZHIPU_API_KEY` | chat,embeddings,images | docs+probe | https://docs.bigmodel.cn/cn/guide/develop/openai/introduction | Docs name the base URL; the path is /api/paas/v4, not /v1. Probe: 401 code 1001 (missing Authorization) on chat, embeddings and images. |
| `zai` | Z.ai (GLM, international) | `https://api.z.ai/api/paas/v4` | `ZAI_API_KEY` | chat,embeddings,images | docs+probe | https://docs.z.ai/api-reference/introduction | International host of the Zhipu API, same /api/paas/v4 path. Probe: 401 code 1001. |
| `minimax` | MiniMax | `https://api.minimax.io/v1` | `MINIMAX_API_KEY` | chat | probe-only | https://platform.minimax.io/docs/api-reference/text-openai-api | Docs text did not name the URL. Probe: 401 authorized_error on /models and chat; embeddings answers HTTP 200 with a base_resp status_code 1004 envelope (not OpenAI shaped); China host api.minimaxi.com answers 401. |
| `qianfan` | Baidu Qianfan | `https://qianfan.baidubce.com/v2` | `QIANFAN_API_KEY` | chat,embeddings | probe-only | https://cloud.baidu.com/doc/qianfan-api/s/3m9b5lqft | Docs page is rendered client side. Probe: chat and embeddings 401 invalid_iam_token; GET /models 403 AccessDenied (different envelope). |
| `ark` | Volcengine Ark (Doubao) | `https://ark.cn-beijing.volces.com/api/v3` | `ARK_API_KEY` | chat | probe-only | https://www.volcengine.com/docs/82379/1298454 | Docs page is rendered client side. Probe: 401 AuthenticationError on /models, chat and embeddings; path is /api/v3. |
| `siliconflow` | SiliconFlow | `https://api.siliconflow.com/v1` | `SILICONFLOW_API_KEY` | chat,embeddings,models | docs+probe | https://docs.siliconflow.com/en/api-reference/chat-completions/chat-completions | Probe: 401 code 30014 with a {code,data,message} envelope on every path; China host api.siliconflow.cn identical. |
| `nebius` | Nebius Token Factory | `https://api.studio.nebius.com/v1` | `NEBIUS_API_KEY` | chat,embeddings,models | docs+probe | https://docs.tokenfactory.nebius.com/api-reference/introduction | Probe: 401 unauthenticated on /models, chat and embeddings; images/generations 404 on this host; api.tokenfactory.nebius.com/v1 answers the same. |
| `scaleway` | Scaleway Generative APIs | `https://api.scaleway.ai/v1` | `SCALEWAY_API_KEY` | chat,embeddings,models | docs+probe | https://www.scaleway.com/en/docs/generative-apis/api-cli/using-chat-api/ | Probe: 401 UNAUTHORIZED (missing 'Authorization' header) on /models, chat and embeddings. |
| `upstage` | Upstage (Solar) | `https://api.upstage.ai/v1` | `UPSTAGE_API_KEY` | chat,embeddings | docs+probe | https://console.upstage.ai/docs/getting-started | Probe: 401 invalid API key on /models, chat and embeddings. |
| `inception` | Inception (Mercury) | `https://api.inceptionlabs.ai/v1` | `INCEPTION_API_KEY` | chat,models | docs+probe | https://docs.inceptionlabs.ai/get-started/get-started | Probe: /models 200 public; chat answers 400 listing the valid model names before auth. |
| `stepfun` | StepFun | `https://api.stepfun.com/v1` | `STEPFUN_API_KEY` | chat,models | docs+probe | https://platform.stepfun.com/docs/api-reference/chat/chat-completion-create | Probe: 401 invalid_api_key on /models and chat. |
| `hunyuan` | Tencent Hunyuan | `https://api.hunyuan.cloud.tencent.com/v1` | `HUNYUAN_API_KEY` | chat | probe-only | https://cloud.tencent.com/document/product/1729/111007 | Docs page did not name the URL in readable text. Probe: 401 'Incorrect API key provided' on /models and chat. |
| `venice` | Venice AI | `https://api.venice.ai/api/v1` | `VENICE_API_KEY` | chat,models | docs+probe | https://docs.venice.ai/api-reference/endpoint/chat/completions | Probe: /models 200 public; unauthenticated chat answers 402 with an x402 payment envelope (not 401). |
| `vercel` | Vercel AI Gateway | `https://ai-gateway.vercel.sh/v1` | `AI_GATEWAY_API_KEY` | chat,models | docs+probe | https://vercel.com/docs/ai-gateway/openai-compat | Probe: /models 200 public; chat 401 authentication_error 'Missing Authorization header'. |
| `friendli` | FriendliAI Serverless | `https://api.friendli.ai/serverless/v1` | `FRIENDLI_TOKEN` | chat,models | docs+probe | https://friendli.ai/docs/openapi/serverless/chat-completions | Probe: /models 200 public; chat 401 {detail}. |
| `gemini` | Google Gemini (OpenAI compatibility) | `https://generativelanguage.googleapis.com/v1beta/openai` | `GEMINI_API_KEY` | chat | docs+probe | https://ai.google.dev/gemini-api/docs/openai | Probe: chat answers 400 'Missing or invalid Authorization header'; unauthenticated GET /models and embeddings answer 404, so only chat is claimed. |
| `baseten` | Baseten Model APIs | `https://inference.baseten.co/v1` | `BASETEN_API_KEY` | chat | docs+probe | https://docs.baseten.co/development/model-apis/overview | Probe: /models 401 'No Authorization header provided'; chat 403 'please check the api-key you provided'. |
| `featherless` | Featherless AI | `https://api.featherless.ai/v1` | `FEATHERLESS_API_KEY` | chat,models | probe-only | https://featherless.ai/docs/overview | Docs text did not name the URL. Probe: /models 200 public; chat 401 'must be signed in'. |
| `ovh` | OVHcloud AI Endpoints | `https://oai.endpoints.kepler.ai.cloud.ovh.net/v1` | `OVH_AI_ENDPOINTS_ACCESS_TOKEN` | chat,models | probe-only | https://help.ovhcloud.com/csm/en-gb-public-cloud-ai-endpoints-getting-started | Docs text did not name the URL. Probe: /models 200 public; chat on an unknown model answers 404 model_not_found. |
| `parasail` | Parasail | `https://api.parasail.io/v1` | `PARASAIL_API_KEY` | chat,models | probe-only | https://docs.parasail.io/parasail-docs/serverless/serverless | Docs URL answered 404 at probe time. Probe: 401 'No bearer token provided' on /models and chat. |
| `crusoe` | Crusoe Managed Inference | `https://managed-inference-api-proxy.crusoecloud.com/v1` | `CRUSOE_API_KEY` | chat,models | probe-only | https://docs.crusoe.ai/managed-inference | Docs URL answered 404 at probe time. Probe: 401 {errors:[Authentication failed]} on /models and chat. |
| `xiaomi` | Xiaomi MiMo | `https://api.xiaomimimo.com/v1` | `XIAOMI_API_KEY` | chat | probe-only | https://platform.xiaomimimo.com/ | Docs text did not name the URL. Probe: 401 'Invalid API Key' on /models and chat. |
| `chutes` | Chutes | `https://llm.chutes.ai/v1` | `CHUTES_API_KEY` | chat,models | probe-only | https://docs.chutes.ai/ | Docs text did not name the URL. Probe: /models 200 public; chat answered an nginx 429 HTML page. |
| `ollama` | Ollama | `http://localhost:11434/v1` | `OLLAMA_API_KEY + OLLAMA_BASE_URL` | chat,embeddings,models | docs-only | https://raw.githubusercontent.com/ollama/ollama/main/docs/api/openai-compatibility.mdx | Docs name http://localhost:11434/v1. The key is required by OpenAI clients but ignored by the server; the module sends the placeholder 'local' when none is set. |
| `lmstudio` | LM Studio | `http://localhost:1234/v1` | `LMSTUDIO_API_KEY + LMSTUDIO_BASE_URL` | chat,embeddings,models | docs-only | https://lmstudio.ai/docs/developer/openai-compat | Docs name http://localhost:1234/v1. Key ignored unless configured in the app. |
| `vllm` | vLLM server | `http://localhost:8000/v1` | `VLLM_API_KEY + VLLM_BASE_URL` | chat,embeddings,models,transcribe | docs-only | https://docs.vllm.ai/en/latest/serving/online_serving/ | Docs list /v1/chat/completions, /v1/embeddings and /v1/audio/transcriptions; key only when the server was started with --api-key. |
| `llamacpp` | llama.cpp server | `http://localhost:8080/v1` | `LLAMACPP_API_KEY + LLAMACPP_BASE_URL` | chat,embeddings,models | docs-only | https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md | README lists /v1/chat/completions; key only when started with --api-key. Same default port as LocalAI. |
| `localai` | LocalAI | `http://localhost:8080/v1` | `LOCALAI_API_KEY + LOCALAI_BASE_URL` | chat,embeddings,models | docs-only | https://localai.io/docs/overview/ | Docs name localhost:8080; endpoint list beyond chat, embeddings and models was not read. |
## 5. Providers not in the table
Plan 18 names these; they are NOT rows because the OpenAI-compatible endpoint
could not be verified from here (2026-10-25). Each is one row away once
confirmed.
| Provider | What was seen |
|---|---|
| GitHub Models | Docs name `https://models.github.ai/inference/chat/completions` and a catalog at `https://models.github.ai/catalog/models`, but the docs page read did not contain the URL as text, and every unauthenticated request (any path) answered `200 OK` with body `OK`, which proves nothing. The catalog lives outside the inference base path, which the row model cannot express (`models_path` is relative to the base) |
| Lambda | `api.lambda.ai`, `api.lambdalabs.com` and `inference.lambda.ai` did not resolve from here |
| Lepton | `api.lepton.ai` did not resolve (service wound down) |
| Hyperbolic | `/models` answers 401, but chat and image paths answered `404 "This inference endpoint has been decommissioned"` on every probe |
| Kluster | `api.kluster.ai` did not resolve |
| Anyscale | The Public Endpoints API answers a 404 page announcing its end |
| Sarvam | `/v1/models` is public, chat answered 403 `invalid_api_key_error`; Sarvam documents its own subscription-key header, not Bearer, so the Bearer-only row model would be wrong |
| Reka | `api.reka.ai/v1` answers 401 `missing_api_key`; no OpenAI-compatibility statement was found |
| AI21 | `/studio/v1/models` answers an empty public list and chat 400 "bad or missing authentication"; the chat shape was not confirmed as OpenAI's |
| Mistral, Cohere, Perplexity, Hugging Face, OpenRouter, xAI, DeepSeek, OpenAI | Own packages (plan 17 / 18) |
## 6. Demo routes
Page `/ai/compat`, GET `/ai/compat/status`, `/ai/compat/providers` (the
table plus per-row configured state), `/ai/compat/calls`. POST routes (body
carries `"provider":"<id>"`): `chat`, `models`, `embeddings`, `images`,
`transcribe`, `speech`. 422 on bad input or a call the row does not claim,
503 `compat_not_configured` without the row's credential (the message names
the missing env), paid routes (all but `models`) are dry runs showing the
exact request and URL unless the body has `"confirm":true`. Matrix row
`ai-compat` for the page.
## 7. Sources
- OpenAI OpenAPI document `openai/openai-openapi` at commit
`8243cc8fa6f04430fb1f1a6a2cb926fbeb4f7f26` (`openapi.json`, version 2.3.0):
the shape of every call.
- Per-row documentation URL in the table of section 4. Each page was fetched
as plain text on 2026-10-25 and searched for the base URL; "docs+probe"
rows contain it. The documented quirks encoded as `Quirk` flags were read
from: Groq (`console.groq.com/docs/openai`, "Currently Unsupported OpenAI
Features": logprobs, logit_bias, top_logprobs, messages[].name answer 400;
n must be 1) and Cerebras (`inference-docs.cerebras.ai/resources/openai`,
"Unsupported parameter combinations": tools with response_format).
- Together's docs page lists `/v1/chat/completions`, `/v1/embeddings`,
`/v1/images/generations`, `/v1/audio/speech`, `/v1/audio/transcriptions` and
moved its host to `api.together.ai`; Cloudflare's page lists
`/chat/completions`, `/embeddings`, `/responses` under
`/client/v4/accounts/{account_id}/ai/v1`; vLLM's page lists
`/v1/chat/completions`, `/v1/embeddings`, `/v1/audio/transcriptions`.
## 8. Verified against spec
`python3 -I src/vendors/compat/tools/check_spec.py` on 2026-10-25 against
the pinned OpenAI document (2.3.0): 6 call rows OK (method, path, operation
id, request schema, every body field, response schema, every response field
read), coverage OK (19 operations of the touched families are excluded with a
reason in the script), registry OK (39 rows equal the table of section 4 in
ids, base URLs, credential names and documentation URLs). `--local` runs only
the registry check without network. This checks the OpenAI shape, not any
provider's conformance to it.
## 9. Verified live
No provider key exists in this environment. Unauthenticated probes, no key
sent, 2026-10-25 (`MOLO_LIVE_NET=1`, test `compat_live.invalid_key_probe...`,
which uses an obviously invalid key and asserts "never 404"). Result per row
(status of `GET /models` with the invalid key where the row lists models,
otherwise a one-token `POST /chat/completions`): 401 for groq, together,
fireworks, cerebras, deepinfra, dashscope, zhipu, zai, minimax, ark,
siliconflow, nebius, upstage, moonshot, stepfun, hunyuan, vercel, featherless,
parasail, crusoe, xiaomi; 403 for scaleway, baseten, ovh; 400 for gemini
(`Missing or invalid Authorization header`); 200 for novita, nvidia, inception,
venice, friendli (public lists that ignore the key). Three hosts could not be reached by OUR
client: sambanova and chutes (`wolfSSL handshake failed: ASN parsing error,
invalid input`) and qianfan (`handshake failed: received alert fatal error`)
while `curl` reaches all three; see Gaps. Local rows were not probed (no
server). Opt-in live tests: per row with its key env set, list models (when
claimed) and ONE completion of `max_tokens` 8 on `COMPAT_<ID>_MODEL` (upper
case id); a missing key or model prints a loud SKIP line per row. No success
body from any provider was seen.
## 10. Unverified
- Every success response of every provider (only OpenAI's documented examples
are used as parsing fixtures, labelled FIXTURE in the tests).
- The `features` mask of each row beyond what its docs page or a probe showed.
An unauthenticated 401 does not prove a path exists (many providers check
auth first), so only these claims rest on more: embeddings routes answering
400 before auth (sambanova, nvidia, dashscope), a 404 proving absence
(cerebras embeddings, nebius images, gemini models and embeddings), and the
docs pages read for Together and Cloudflare. All other claims are the
provider's well known documented surface, not re-read here: images and audio
for together, deepinfra, zhipu, zai; transcription for groq and vllm.
- `docs-only` rows: Cloudflare (needs an account id), the five local servers.
- Quirks other than the four flags (for example Groq turning temperature 0
into 1e-8, Moonshot and Kimi temperature limits, per-model parameter
support) are listed in notes or not at all, and are not enforced.
- Provider-specific auth beyond Bearer (Baseten documents its own scheme; the
probe showed Bearer is the shape OpenAI clients send, acceptance with a real
key is unproven).
- China-region hosts (moonshot.cn, dashscope.aliyuncs.com, minimaxi.com,
siliconflow.cn) answer like their international twins but are not rows; set
the override env for local rows only. Hosted rows cannot be overridden by
env today (see Gaps).
## 11. Gaps
- `src/clients` TLS: wolfSSL handshake fails against sambanova, chutes and
qianfan hosts where curl succeeds. Owner: `src/clients/transports` (not this
package); the rows stay because the endpoints answer to curl.
- `src/llm`: the shared `MOLO_LIVE_NET` is unset by a test in `tests/src/llm`
(`test_llm.cpp`) so any suite reading it at run time skips in the shared
binary; this suite reads it at static-init time. Other vendors' live tests
still skip when the whole `ai_test` binary runs.
- No env override for hosted base URLs (regional hosts); the `base_env` switch
exists for local rows only. Closing it is one `base_env` value per hosted
row in `registry.cpp`.
- GitHub Models needs a second base URL for its catalog (see section 5).
- Streaming sends no `stream_options.include_usage` by default because some
providers reject it; pass it through `ChatRequest.extra`.
- Responses API, legacy completions, image edits and moderation are not wrapped.
+50
View File
@@ -0,0 +1,50 @@
#include "vendors/compat/calls.h"
#include "llm/error.h"
#include "vendors/compat/registry.h"
namespace molo::compat {
namespace {
const CallSpec kCalls[] = {
{Call::kChat, "POST", "/chat/completions", "createChatCompletion", "CreateChatCompletionRequest",
"model,messages,max_tokens,temperature,top_p,stop,tools,tool_choice,response_format,reasoning_effort,"
"stream,stream_options,n",
"CreateChatCompletionResponse", "id,model,choices,usage", Feature::kChat, "chat"},
{Call::kEmbeddings, "POST", "/embeddings", "createEmbedding", "CreateEmbeddingRequest",
"model,input,dimensions,encoding_format", "CreateEmbeddingResponse", "data,model,usage",
Feature::kEmbeddings, "embeddings"},
{Call::kModels, "GET", "/models", "listModels", "", "", "ListModelsResponse", "data",
Feature::kModels, "models"},
{Call::kImages, "POST", "/images/generations", "createImage", "CreateImageRequest",
"model,prompt,n,size,response_format", "ImagesResponse", "data", Feature::kImages, "images"},
{Call::kTranscribe, "POST", "/audio/transcriptions", "createTranscription", "CreateTranscriptionRequest",
"file,model,language,prompt,response_format,temperature", "CreateTranscriptionResponseJson", "text",
Feature::kTranscribe, "transcribe"},
{Call::kSpeech, "POST", "/audio/speech", "createSpeech", "CreateSpeechRequest",
"model,input,voice,response_format,speed", "", "", Feature::kSpeech, "speech"},
};
constexpr std::size_t kCallCount = sizeof(kCalls) / sizeof(kCalls[0]);
} // namespace
const CallSpec& Calls::spec_of(Call call) {
for (const CallSpec& s : kCalls)
if (s.call == call) return s;
throw llm::ApiError::invalid("compat", "unknown call");
}
clients::Method Calls::method_of(Call call) {
return std::string(spec_of(call).http_method) == "GET" ? clients::Method::kGet : clients::Method::kPost;
}
std::size_t Calls::count() { return kCallCount; }
const CallSpec& Calls::at(std::size_t index) {
if (index >= kCallCount) throw llm::ApiError::invalid("compat", "call index out of range");
return kCalls[index];
}
} // namespace molo::compat
+38
View File
@@ -0,0 +1,38 @@
#pragma once
// The ONE table of REST calls this module makes. The shapes are OpenAI's
// (every row is an OpenAI operation); providers differ only in base URL, auth
// and quirks, which live in registry.cpp. tools/check_spec.py checks every
// row against OpenAI's published OpenAPI document. Code builds every path
// from this table, never from a literal.
#include <cstddef>
#include <string>
#include "clients/request.h"
namespace molo::compat {
enum class Call { kChat, kEmbeddings, kModels, kImages, kTranscribe, kSpeech };
struct CallSpec {
Call call;
const char* http_method; // GET POST
const char* path; // from the provider base URL
const char* operation_id; // OpenAI OpenAPI operationId
const char* request; // OpenAI request schema name ("" for GET)
const char* body_fields; // comma list: request body fields we send
const char* response; // OpenAI response schema name
const char* fields_read; // comma list: response fields we read
unsigned feature; // registry Feature bit a row must claim
const char* feature_name;
};
struct Calls {
static const CallSpec& spec_of(Call call);
static clients::Method method_of(Call call);
static std::size_t count();
static const CallSpec& at(std::size_t index);
};
} // namespace molo::compat
+28
View File
@@ -0,0 +1,28 @@
#pragma once
// Static ceilings for the compat module. Never autotuned.
#include <cstddef>
namespace molo::compat {
struct CompatCaps {
static constexpr std::size_t kMaxMessages = 4096;
static constexpr std::size_t kMaxTools = 128;
static constexpr std::size_t kMaxStopSequences = 16;
static constexpr std::size_t kMaxStreamEvents = 200000;
static constexpr std::size_t kMaxChoices = 8;
static constexpr std::size_t kMaxModelsReturned = 5000;
static constexpr std::size_t kMaxEmbeddingInputs = 2048;
static constexpr std::size_t kMaxEmbeddingInputChars = 1u * 1024u * 1024u;
static constexpr std::size_t kMaxPromptChars = 32768;
static constexpr int kMaxImages = 10;
static constexpr std::size_t kMaxAudioBytes = 64u * 1024u * 1024u;
static constexpr std::size_t kMaxSpeechChars = 4096;
static constexpr std::size_t kMaxModelChars = 256;
static constexpr std::size_t kMaxBaseUrlChars = 512;
static constexpr std::size_t kMaxFilenameChars = 256;
static constexpr int kMaxTokensCeiling = 4194304;
};
} // namespace molo::compat
+375
View File
@@ -0,0 +1,375 @@
#include "vendors/compat/chat_model.h"
#include <map>
#include "security/crypto/Base64.h"
#include "vendors/compat/caps.h"
namespace molo::compat {
using Json = clients::body::Json;
namespace {
[[noreturn]] void bad(const std::string& vendor, const std::string& why) {
throw llm::ApiError::invalid(vendor, why);
}
std::string str_at(const Json& j, const char* key) {
const Json* v = j.is_object() ? j.find(key) : nullptr;
return v && v->is_string() ? v->as_string() : "";
}
int int_at(const Json& j, const char* key) {
const Json* v = j.is_object() ? j.find(key) : nullptr;
return v && v->is_number() ? static_cast<int>(v->as_int()) : 0;
}
const Json* obj_at(const Json& j, const char* key) {
const Json* v = j.is_object() ? j.find(key) : nullptr;
return v && v->is_object() ? v : nullptr;
}
Json image_part(const std::string& vendor, const llm::ImageData& img) {
std::string url = img.url;
if (url.empty()) {
if (img.bytes.empty()) bad(vendor, "image has neither url nor bytes");
if (img.mime.empty()) bad(vendor, "image bytes need a mime type");
url = "data:" + img.mime + ";base64," + security::crypto::Base64::encode(img.bytes);
}
Json u = Json::object();
u.set("url", Json(url));
Json p = Json::object();
p.set("type", Json("image_url"));
p.set("image_url", u);
return p;
}
Json text_part(const std::string& t) {
Json p = Json::object();
p.set("type", Json("text"));
p.set("text", Json(t));
return p;
}
Json tool_call_json(const llm::ToolCall& c) {
Json fn = Json::object();
fn.set("name", Json(c.name));
fn.set("arguments", Json(c.arguments.is_null() ? std::string("{}") : c.arguments.dump()));
Json o = Json::object();
o.set("id", Json(c.id));
o.set("type", Json("function"));
o.set("function", fn);
return o;
}
void append_message(const std::string& vendor, const ProviderRow& row, Json& out, const llm::Message& m) {
if (!m.name.empty() && (row.quirks & Quirk::kRejectMessageName))
bad(vendor, std::string(row.name) + " rejects messages[].name (documented 400)");
auto base = [&](const char* role) {
Json o = Json::object();
o.set("role", Json(role));
if (!m.name.empty()) o.set("name", Json(m.name));
return o;
};
switch (m.role) {
case llm::Role::kSystem: {
Json o = base("system");
o.set("content", Json(m.text()));
out.push(o);
return;
}
case llm::Role::kUser: {
Json o = base("user");
bool has_image = false;
for (const llm::ContentPart& p : m.parts) has_image = has_image || p.image.has_value();
if (!has_image) {
o.set("content", Json(m.text()));
} else {
Json parts = Json::array();
for (const llm::ContentPart& p : m.parts) {
if (!p.text.empty()) parts.push(text_part(p.text));
if (p.image) parts.push(image_part(vendor, *p.image));
}
o.set("content", parts);
}
out.push(o);
return;
}
case llm::Role::kAssistant: {
Json o = base("assistant");
Json calls = Json::array();
bool any = false;
for (const llm::ContentPart& p : m.parts)
if (p.tool_call) {
calls.push(tool_call_json(*p.tool_call));
any = true;
}
std::string text = m.text();
o.set("content", text.empty() && any ? Json() : Json(text));
if (any) o.set("tool_calls", calls);
out.push(o);
return;
}
case llm::Role::kTool: {
bool any = false;
for (const llm::ContentPart& p : m.parts)
if (p.tool_result) {
Json o = Json::object();
o.set("role", Json("tool"));
o.set("tool_call_id", Json(p.tool_result->call_id));
o.set("content", Json(p.tool_result->content));
out.push(o);
any = true;
}
if (!any) bad(vendor, "a tool message needs a tool result part");
return;
}
}
}
llm::Usage usage_of(const Json& u) {
llm::Usage out;
if (!u.is_object()) return out;
out.input_tokens = int_at(u, "prompt_tokens");
out.output_tokens = int_at(u, "completion_tokens");
out.total_tokens = int_at(u, "total_tokens");
if (const Json* d = obj_at(u, "prompt_tokens_details")) out.cached_tokens = int_at(*d, "cached_tokens");
if (const Json* d = obj_at(u, "completion_tokens_details")) out.reasoning_tokens = int_at(*d, "reasoning_tokens");
return out;
}
std::string reasoning_of(const Json& holder) {
std::string r = str_at(holder, "reasoning_content");
return r.empty() ? str_at(holder, "reasoning") : r;
}
Json arguments_of(const std::string& raw) {
if (raw.empty()) return Json::object();
try {
Json j = Json::parse(raw);
if (j.is_object()) return j;
} catch (const std::exception&) {
}
Json badj = Json::object();
badj.set("_invalid_json", Json(raw));
return badj;
}
} // namespace
CompatChatModel::CompatChatModel(Provider provider, std::string default_model)
: ProviderService(std::move(provider)), default_model_(std::move(default_model)) {}
std::shared_ptr<CompatChatModel> CompatChatModel::from_env(const std::string& provider_id,
const std::string& default_model) {
return std::make_shared<CompatChatModel>(Provider::from_env(provider_id), default_model);
}
Json CompatChatModel::to_body(const llm::ChatRequest& r, bool stream) const {
const std::string vendor = client_->config().vendor;
const ProviderRow& row = provider_.row();
provider_.require(Call::kChat);
std::string model = r.model.empty() ? default_model_ : r.model;
if (model.empty()) bad(vendor, "model is required (ids differ per provider)");
if (model.size() > CompatCaps::kMaxModelChars) bad(vendor, "model id too long");
if (r.messages.empty() && r.system.empty()) bad(vendor, "at least one message is required");
if (r.messages.size() > CompatCaps::kMaxMessages) bad(vendor, "too many messages");
if (r.tools.size() > CompatCaps::kMaxTools) bad(vendor, "too many tools");
if (r.stop.size() > CompatCaps::kMaxStopSequences) bad(vendor, "too many stop sequences");
if (r.max_tokens && (*r.max_tokens < 1 || *r.max_tokens > CompatCaps::kMaxTokensCeiling))
bad(vendor, "max_tokens out of range");
if (r.temperature && (*r.temperature < 0.0 || *r.temperature > 2.0)) bad(vendor, "temperature must be 0..2");
if (r.top_p && (*r.top_p <= 0.0 || *r.top_p > 1.0)) bad(vendor, "top_p must be in (0,1]");
const bool has_format = r.json_mode || !r.response_schema.is_null();
if (r.json_mode && !r.response_schema.is_null()) bad(vendor, "json_mode and response_schema are exclusive");
if ((row.quirks & Quirk::kNoToolsWithFormat) && has_format && !r.tools.empty())
bad(vendor, std::string(row.name) + " documents that tools and response_format must not be combined");
Json body = Json::object();
body.set("model", Json(model));
Json messages = Json::array();
if (!r.system.empty()) append_message(vendor, row, messages, llm::Message::system(r.system));
for (const llm::Message& m : r.messages) append_message(vendor, row, messages, m);
body.set("messages", messages);
if (r.max_tokens) body.set("max_tokens", Json(*r.max_tokens));
if (r.temperature) body.set("temperature", Json(*r.temperature));
if (r.top_p) body.set("top_p", Json(*r.top_p));
if (!r.stop.empty()) {
Json s = Json::array();
for (const std::string& x : r.stop) s.push(Json(x));
body.set("stop", s);
}
if (!r.tools.empty()) {
Json tools = Json::array();
for (const llm::ToolSpec& t : r.tools) {
if (t.name.empty()) bad(vendor, "tool name is required");
Json fn = Json::object();
fn.set("name", Json(t.name));
if (!t.description.empty()) fn.set("description", Json(t.description));
fn.set("parameters", t.parameters);
Json o = Json::object();
o.set("type", Json("function"));
o.set("function", fn);
tools.push(o);
}
body.set("tools", tools);
}
if (!r.tool_choice.empty()) {
if (r.tools.empty()) bad(vendor, "tool_choice needs tools");
if (r.tool_choice == "auto" || r.tool_choice == "none" || r.tool_choice == "required") {
body.set("tool_choice", Json(r.tool_choice));
} else {
Json fn = Json::object();
fn.set("name", Json(r.tool_choice));
Json o = Json::object();
o.set("type", Json("function"));
o.set("function", fn);
body.set("tool_choice", o);
}
}
if (r.json_mode) {
Json f = Json::object();
f.set("type", Json("json_object"));
body.set("response_format", f);
} else if (!r.response_schema.is_null()) {
Json js = Json::object();
js.set("name", Json("response"));
js.set("strict", Json(true));
js.set("schema", r.response_schema);
Json f = Json::object();
f.set("type", Json("json_schema"));
f.set("json_schema", js);
body.set("response_format", f);
}
if (!r.reasoning_effort.empty()) body.set("reasoning_effort", Json(r.reasoning_effort));
if (r.extra.is_object()) {
for (const auto& [k, v] : r.extra.as_object()) {
if (k == "model" || k == "messages" || k == "stream") bad(vendor, "extra may not set '" + k + "'");
if ((row.quirks & Quirk::kRejectLogprobs) && (k == "logprobs" || k == "top_logprobs" || k == "logit_bias"))
bad(vendor, std::string(row.name) + " rejects " + k + " (documented 400)");
if ((row.quirks & Quirk::kSingleChoice) && k == "n" && !(v.is_number() && v.as_int() == 1))
bad(vendor, std::string(row.name) + " requires n = 1");
if (k == "n" && v.is_number() && (v.as_int() < 1 || v.as_int() > static_cast<long>(CompatCaps::kMaxChoices)))
bad(vendor, "n out of range");
body.set(k, v);
}
} else if (!r.extra.is_null()) {
bad(vendor, "extra must be an object");
}
if (stream) body.set("stream", Json(true));
return body;
}
llm::ChatResponse CompatChatModel::parse_response(const Json& doc) {
llm::ChatResponse out;
out.id = str_at(doc, "id");
out.model = str_at(doc, "model");
out.message.role = llm::Role::kAssistant;
out.raw = doc;
out.usage = usage_of(doc.is_object() && doc.find("usage") ? *doc.find("usage") : Json());
const Json* choices = doc.is_object() ? doc.find("choices") : nullptr;
if (!choices || !choices->is_array() || choices->as_array().empty())
throw llm::ApiError("compat", 502, "BAD_RESPONSE", "response has no choices");
const Json& c = choices->as_array()[0];
out.finish_reason = str_at(c, "finish_reason");
const Json* msg = obj_at(c, "message");
if (!msg) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "choice has no message");
std::string reasoning = reasoning_of(*msg);
if (!reasoning.empty()) {
llm::ContentPart r;
r.reasoning = reasoning;
out.message.parts.push_back(r);
}
std::string content = str_at(*msg, "content");
const Json* calls = msg->find("tool_calls");
const bool has_calls = calls && calls->is_array() && !calls->as_array().empty();
if (!content.empty() || !has_calls) out.message.parts.push_back(llm::ContentPart::of_text(content));
if (has_calls)
for (const Json& call : calls->as_array()) {
llm::ToolCall tc;
tc.id = str_at(call, "id");
if (const Json* fn = obj_at(call, "function")) {
tc.name = str_at(*fn, "name");
tc.arguments = arguments_of(str_at(*fn, "arguments"));
}
out.message.parts.push_back(llm::ContentPart::of_call(tc));
}
return out;
}
llm::ChatResponse CompatChatModel::generate(const llm::ChatRequest& request) {
Json body = to_body(request, false);
return parse_response(client_->post(Calls::spec_of(Call::kChat).path, body));
}
void CompatChatModel::stream(const llm::ChatRequest& request,
const std::function<bool(const llm::ChatChunk&)>& on_chunk) {
Json body = to_body(request, true);
std::map<int, std::string> ids, names;
std::size_t events = 0;
bool finished = false, stopped = false;
client_->stream(Calls::spec_of(Call::kChat).path, body, [&](const llm::SseEvent& ev) {
if (++events > CompatCaps::kMaxStreamEvents) {
stopped = true;
return false;
}
const Json& d = ev.data;
if (const Json* err = obj_at(d, "error"))
throw llm::ApiError(vendor(), 502, str_at(*err, "type").empty() ? "STREAM_ERROR" : str_at(*err, "type"),
str_at(*err, "message"));
auto emit = [&](const llm::ChatChunk& c) {
if (!on_chunk(c)) {
stopped = true;
return false;
}
return true;
};
const Json* choices = d.is_object() ? d.find("choices") : nullptr;
if (choices && choices->is_array() && !choices->as_array().empty()) {
const Json& c = choices->as_array()[0];
if (const Json* delta = obj_at(c, "delta")) {
std::string text = str_at(*delta, "content"), reasoning = reasoning_of(*delta);
if (!text.empty() || !reasoning.empty()) {
llm::ChatChunk ch;
ch.text_delta = text;
ch.reasoning_delta = reasoning;
if (!emit(ch)) return false;
}
const Json* calls = delta->find("tool_calls");
if (calls && calls->is_array())
for (const Json& f : calls->as_array()) {
int index = int_at(f, "index");
if (!str_at(f, "id").empty()) ids[index] = str_at(f, "id");
const Json* fn = obj_at(f, "function");
if (fn && !str_at(*fn, "name").empty()) names[index] = str_at(*fn, "name");
llm::ChatChunk ch;
ch.tool_call_id = ids[index];
ch.tool_call_name = names[index];
ch.tool_args_delta = fn ? str_at(*fn, "arguments") : "";
if (!emit(ch)) return false;
}
}
std::string fin = str_at(c, "finish_reason");
if (!fin.empty()) {
finished = true;
llm::ChatChunk ch;
ch.finish_reason = fin;
if (const Json* u = obj_at(d, "usage")) ch.usage = usage_of(*u);
if (!emit(ch)) return false;
return true;
}
}
if (const Json* u = obj_at(d, "usage")) {
llm::ChatChunk ch;
ch.usage = usage_of(*u);
if (!emit(ch)) return false;
}
return true;
});
if (!stopped && !finished)
throw llm::ApiError(vendor(), 502, "STREAM_TRUNCATED", "stream ended without a finish_reason");
}
} // namespace molo::compat
+48
View File
@@ -0,0 +1,48 @@
#pragma once
// CompatChatModel: llm::ChatModel over POST {base}/chat/completions of any
// registry row. Mapping notes (each one is a decision):
// - ChatRequest.model is required unless a default model is given: model ids
// differ per provider and there is no honest module-wide default.
// - ChatRequest.system becomes the first system message.
// - Reasoning text of an assistant message is never sent back.
// - json_mode -> response_format {type: json_object}; response_schema ->
// response_format {type: json_schema, strict: true}.
// - reasoning_effort is sent as reasoning_effort; ChatRequest.extra is merged
// except for keys the module owns (model, messages, stream).
// - Row quirks (registry.h Quirk) are rejected before any traffic.
// - Reasoning comes back from reasoning_content or reasoning, whichever the
// provider sends.
// - Tool call arguments that are not valid JSON arrive as
// {"_invalid_json": "<raw text>"}.
// - Streaming sends no stream_options unless ChatRequest.extra carries them;
// usage then appears only when the provider sends it. A stream that ends
// without a finish_reason throws 502 STREAM_TRUNCATED.
#include <memory>
#include "llm/chat_types.h"
#include "vendors/compat/provider.h"
namespace molo::compat {
class CompatChatModel : public ProviderService, public llm::ChatModel {
public:
explicit CompatChatModel(Provider provider, std::string default_model = "");
static std::shared_ptr<CompatChatModel> from_env(const std::string& provider_id,
const std::string& default_model = "");
std::string vendor() const override { return client_->config().vendor; }
llm::ChatResponse generate(const llm::ChatRequest& request) override;
void stream(const llm::ChatRequest& request,
const std::function<bool(const llm::ChatChunk&)>& on_chunk) override;
// The wire body a neutral request maps to (also used by tests and dry runs).
clients::body::Json to_body(const llm::ChatRequest& request, bool stream) const;
static llm::ChatResponse parse_response(const clients::body::Json& doc);
private:
std::string default_model_;
};
} // namespace molo::compat
+110
View File
@@ -0,0 +1,110 @@
#include "vendors/compat/provider.h"
#include <cctype>
#include "clients/transports/http_transport.h"
#include "llm/config.h"
#include "vendors/compat/caps.h"
namespace molo::compat {
namespace {
bool valid_account_id(const std::string& s) {
if (s.empty() || s.size() > 64) return false;
for (char c : s)
if (!(std::isalnum(static_cast<unsigned char>(c)) || c == '-' || c == '_')) return false;
return true;
}
bool valid_base(const std::string& s) {
if (s.empty() || s.size() > CompatCaps::kMaxBaseUrlChars) return false;
if (s.rfind("http://", 0) != 0 && s.rfind("https://", 0) != 0) return false;
for (char c : s)
if (static_cast<unsigned char>(c) <= 0x20 || c == 0x7f || c == '?' || c == '#') return false;
return s.find("..") == std::string::npos;
}
std::string replace_account(std::string base, const std::string& account) {
const std::string token = "{account_id}";
std::size_t at = base.find(token);
if (at != std::string::npos) base.replace(at, token.size(), account);
return base;
}
// The key a row would use: configured env value; local rows get a
// placeholder when their base_env switch is set (servers ignore it).
std::string key_for(const ProviderRow& row, const std::string& key_env_value, bool switch_set) {
if (!row.local) return key_env_value;
if (!switch_set) return "";
return key_env_value.empty() ? "local" : key_env_value;
}
} // namespace
std::string Provider::resolve_base(const ProviderRow& row, const std::string& base_override,
const std::string& account_id) {
std::string base = base_override.empty() ? row.base_url : base_override;
if (base.find("{account_id}") != std::string::npos) {
if (!valid_account_id(account_id))
throw llm::ApiError::invalid("compat:" + std::string(row.id), "account id missing or malformed");
base = replace_account(base, account_id);
}
while (base.size() > 8 && base.back() == '/') base.pop_back();
if (!valid_base(base))
throw llm::ApiError::invalid("compat:" + std::string(row.id), "base URL must be http(s), no spaces, query or fragment");
return base;
}
llm::ApiClientConfig Provider::config_for(const ProviderRow& row, const std::string& base_url,
const std::string& missing_env) {
llm::ApiClientConfig c;
c.vendor = std::string("compat:") + row.id;
c.base_url = base_url;
c.key_env = missing_env.empty() ? row.key_env : missing_env;
c.auth = llm::AuthStyle::kBearer;
return c;
}
Provider Provider::from_env(const std::string& id) {
const ProviderRow& row = Registry::get(id);
std::string base_override = row.base_env[0] ? llm::Config::get(row.base_env) : "";
std::string account = row.extra_env[0] ? llm::Config::get(row.extra_env) : "";
std::string key = key_for(row, llm::Config::get(row.key_env), !base_override.empty());
std::string missing;
if (row.local && base_override.empty()) missing = row.base_env;
else if (row.extra_env[0] && account.empty()) missing = row.extra_env;
if (!missing.empty()) key.clear();
// An unconfigured row still needs a syntactically valid base URL to exist.
std::string base = (row.extra_env[0] && account.empty())
? replace_account(row.base_url, "unset")
: resolve_base(row, base_override, account);
auto client = std::make_shared<llm::ApiClient>(std::make_unique<clients::transports::HttpTransport>(),
config_for(row, base, missing), key);
return Provider(&row, base, client);
}
Provider Provider::with(const std::string& id, std::unique_ptr<clients::Transport> transport,
const std::string& key, const std::string& base_override,
const std::string& account_id) {
const ProviderRow& row = Registry::get(id);
std::string base = resolve_base(row, base_override, account_id);
auto client = std::make_shared<llm::ApiClient>(std::move(transport), config_for(row, base, ""), key);
return Provider(&row, base, client);
}
std::string Provider::missing_env() const {
if (configured()) return "";
return client_->config().key_env;
}
bool Provider::supports(Call call) const { return Registry::has(*row_, Calls::spec_of(call).feature); }
void Provider::require(Call call) const {
if (!supports(call))
throw llm::ApiError::invalid(client_->config().vendor,
std::string(row_->name) + " has no documented " + Calls::spec_of(call).feature_name +
" endpoint in the registry");
}
} // namespace molo::compat
+64
View File
@@ -0,0 +1,64 @@
#pragma once
// Provider: one registry row bound to one ApiClient. The only place a base
// URL is resolved (account id substitution, env override) and validated.
// Services take a Provider and refuse calls the row does not claim.
#include <memory>
#include <string>
#include "llm/api_client.h"
#include "vendors/compat/calls.h"
#include "vendors/compat/registry.h"
namespace molo::compat {
class Provider {
public:
// Real HTTPS transport. Key and overrides come from llm::Config.
static Provider from_env(const std::string& id);
// Same row, any transport (tests). base_override and account_id may be empty.
static Provider with(const std::string& id, std::unique_ptr<clients::Transport> transport,
const std::string& key, const std::string& base_override = "",
const std::string& account_id = "");
// base_url with {account_id} replaced and a trailing slash removed. Throws
// ApiError::invalid for a malformed override or account id.
static std::string resolve_base(const ProviderRow& row, const std::string& base_override,
const std::string& account_id);
const ProviderRow& row() const { return *row_; }
const std::string& base_url() const { return base_url_; }
std::shared_ptr<llm::ApiClient> client() const { return client_; }
bool configured() const { return client_->configured(); }
// Env var a user must set next ("" when configured).
std::string missing_env() const;
// Throws ApiError::invalid when the row does not claim the call's feature.
void require(Call call) const;
bool supports(Call call) const;
private:
Provider(const ProviderRow* row, std::string base_url, std::shared_ptr<llm::ApiClient> client)
: row_(row), base_url_(std::move(base_url)), client_(std::move(client)) {}
static llm::ApiClientConfig config_for(const ProviderRow& row, const std::string& base_url,
const std::string& missing_env);
const ProviderRow* row_;
std::string base_url_;
std::shared_ptr<llm::ApiClient> client_;
};
// Base for the per-endpoint services: holds the Provider, checks features.
class ProviderService : public llm::Service {
public:
explicit ProviderService(Provider provider)
: llm::Service(provider.client()), provider_(std::move(provider)) {}
const Provider& provider() const { return provider_; }
protected:
Provider provider_;
};
} // namespace molo::compat
+239
View File
@@ -0,0 +1,239 @@
#include "vendors/compat/registry.h"
#include "llm/error.h"
namespace molo::compat {
namespace {
constexpr unsigned C = Feature::kChat;
constexpr unsigned E = Feature::kEmbeddings;
constexpr unsigned M = Feature::kModels;
constexpr unsigned I = Feature::kImages;
constexpr unsigned T = Feature::kTranscribe;
constexpr unsigned S = Feature::kSpeech;
constexpr Evidence DP = Evidence::kDocsAndProbe;
constexpr Evidence PO = Evidence::kProbeOnly;
constexpr Evidence DO = Evidence::kDocsOnly;
// Probe date for every "Probe:" note: 2026-10-25, unauthenticated, no key sent.
// Fields: id, name, base_url, key_env, base_env, extra_env, models_path,
// features, quirks, evidence, local, doc_url, note.
const ProviderRow kRows[] = {
{"groq", "Groq", "https://api.groq.com/openai/v1", "GROQ_API_KEY", "", "", "/models",
C | M | T, Quirk::kRejectLogprobs | Quirk::kRejectMessageName | Quirk::kSingleChoice, DP, false,
"https://console.groq.com/docs/openai",
"Docs: logprobs, logit_bias, top_logprobs and messages[].name answer 400; n must be 1; temperature 0 "
"becomes 1e-8; transcription formats vtt and srt unsupported. Probe: 401 invalid_api_key on /models and chat."},
{"together", "Together AI", "https://api.together.ai/v1", "TOGETHER_API_KEY", "", "", "/models",
C | E | M | I | T | S, Quirk::kNone, DP, false,
"https://docs.together.ai/docs/openai-api-compatibility",
"Docs: logit_bias unsupported on most models; service_tier, store, metadata and prediction are accepted "
"and ignored. Probe: 401 on /models and chat; legacy host api.together.xyz answers 401 too."},
{"fireworks", "Fireworks AI", "https://api.fireworks.ai/inference/v1", "FIREWORKS_API_KEY", "", "", "/models",
C | E | M, Quirk::kNone, DP, false,
"https://docs.fireworks.ai/tools-sdks/openai-compatibility",
"Probe: /models 401 with a docs link; unknown model answers 404 NOT_FOUND; empty embeddings body answers 500."},
{"cerebras", "Cerebras", "https://api.cerebras.ai/v1", "CEREBRAS_API_KEY", "", "", "/models",
C | M, Quirk::kNoToolsWithFormat, DP, false,
"https://inference-docs.cerebras.ai/resources/openai",
"Docs: tools and response_format must not be combined (gpt-oss-120b rejects it); system and developer roles "
"both act at developer level. Probe: /models 403 {detail}, chat 401 wrong_api_key, embeddings path 404."},
{"sambanova", "SambaNova", "https://api.sambanova.ai/v1", "SAMBANOVA_API_KEY", "", "", "/models",
C | E | M, Quirk::kNone, PO, false,
"https://docs.sambanova.ai/docs/en/features/openai-compatibility",
"Docs page answers 403 to a plain fetch. Probe: /models 200 public; chat 401 (also accepts x-api-key); "
"embeddings answers 400 'must provide a model' before auth."},
{"deepinfra", "DeepInfra", "https://api.deepinfra.com/v1/openai", "DEEPINFRA_API_KEY", "", "", "/models",
C | E | M | I | T, Quirk::kNone, DP, false,
"https://docs.deepinfra.com/chat/overview",
"Probe: /models 200 public; chat 401 missing API key; images and audio paths answer 401 with a {detail} body."},
{"novita", "Novita AI", "https://api.novita.ai/openai/v1", "NOVITA_API_KEY", "", "", "/models",
C | M, Quirk::kNone, DP, false,
"https://novita.ai/docs/guides/llm-api",
"Probe: /models 200 public; bad key answers 403 INVALID_API_KEY (not 401); legacy base /v3/openai answers the same."},
{"nvidia", "NVIDIA NIM (build.nvidia.com)", "https://integrate.api.nvidia.com/v1", "NVIDIA_API_KEY", "", "", "/models",
C | E | M, Quirk::kNone, DP, false,
"https://docs.api.nvidia.com/nim/reference/llm-apis",
"Probe: /models 200 public; embeddings answers 400 'model field is required'; retired models answer 410 Gone."},
{"cloudflare", "Cloudflare Workers AI", "https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1",
"CLOUDFLARE_API_TOKEN", "", "CLOUDFLARE_ACCOUNT_ID", "", C | E, Quirk::kNone, DO, false,
"https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/",
"Docs list /chat/completions, /embeddings and /responses under the account path. Probe: GET /models answers "
"405 (no list endpoint here); not probeable further without an account id."},
{"moonshot", "Moonshot (Kimi)", "https://api.moonshot.ai/v1", "MOONSHOT_API_KEY", "", "", "/models",
C | M, Quirk::kNone, DP, false,
"https://platform.kimi.ai/docs/api/chat",
"Probe: 401 incorrect_api_key_error on /models and chat; the China host api.moonshot.cn answers the same "
"(separate keys)."},
{"dashscope", "Alibaba DashScope (Qwen, intl)", "https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
"DASHSCOPE_API_KEY", "", "", "", C | E, Quirk::kNone, DP, false,
"https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope",
"Probe: 401 on /models and chat; embeddings answers 400 'you must provide a model parameter'; China host "
"dashscope.aliyuncs.com/compatible-mode/v1 answers 401 too."},
{"zhipu", "Zhipu (GLM, China)", "https://open.bigmodel.cn/api/paas/v4", "ZHIPU_API_KEY", "", "", "",
C | E | I, Quirk::kNone, DP, false,
"https://docs.bigmodel.cn/cn/guide/develop/openai/introduction",
"Docs name the base URL; the path is /api/paas/v4, not /v1. Probe: 401 code 1001 (missing Authorization) "
"on chat, embeddings and images."},
{"zai", "Z.ai (GLM, international)", "https://api.z.ai/api/paas/v4", "ZAI_API_KEY", "", "", "",
C | E | I, Quirk::kNone, DP, false,
"https://docs.z.ai/api-reference/introduction",
"International host of the Zhipu API, same /api/paas/v4 path. Probe: 401 code 1001."},
{"minimax", "MiniMax", "https://api.minimax.io/v1", "MINIMAX_API_KEY", "", "", "",
C, Quirk::kNone, PO, false,
"https://platform.minimax.io/docs/api-reference/text-openai-api",
"Docs text did not name the URL. Probe: 401 authorized_error on /models and chat; embeddings answers HTTP 200 "
"with a base_resp status_code 1004 envelope (not OpenAI shaped); China host api.minimaxi.com answers 401."},
{"qianfan", "Baidu Qianfan", "https://qianfan.baidubce.com/v2", "QIANFAN_API_KEY", "", "", "",
C | E, Quirk::kNone, PO, false,
"https://cloud.baidu.com/doc/qianfan-api/s/3m9b5lqft",
"Docs page is rendered client side. Probe: chat and embeddings 401 invalid_iam_token; GET /models 403 "
"AccessDenied (different envelope)."},
{"ark", "Volcengine Ark (Doubao)", "https://ark.cn-beijing.volces.com/api/v3", "ARK_API_KEY", "", "", "",
C, Quirk::kNone, PO, false,
"https://www.volcengine.com/docs/82379/1298454",
"Docs page is rendered client side. Probe: 401 AuthenticationError on /models, chat and embeddings; path is /api/v3."},
{"siliconflow", "SiliconFlow", "https://api.siliconflow.com/v1", "SILICONFLOW_API_KEY", "", "", "/models",
C | E | M, Quirk::kNone, DP, false,
"https://docs.siliconflow.com/en/api-reference/chat-completions/chat-completions",
"Probe: 401 code 30014 with a {code,data,message} envelope on every path; China host api.siliconflow.cn identical."},
{"nebius", "Nebius Token Factory", "https://api.studio.nebius.com/v1", "NEBIUS_API_KEY", "", "", "/models",
C | E | M, Quirk::kNone, DP, false,
"https://docs.tokenfactory.nebius.com/api-reference/introduction",
"Probe: 401 unauthenticated on /models, chat and embeddings; images/generations 404 on this host; "
"api.tokenfactory.nebius.com/v1 answers the same."},
{"scaleway", "Scaleway Generative APIs", "https://api.scaleway.ai/v1", "SCALEWAY_API_KEY", "", "", "/models",
C | E | M, Quirk::kNone, DP, false,
"https://www.scaleway.com/en/docs/generative-apis/api-cli/using-chat-api/",
"Probe: 401 UNAUTHORIZED (missing 'Authorization' header) on /models, chat and embeddings."},
{"upstage", "Upstage (Solar)", "https://api.upstage.ai/v1", "UPSTAGE_API_KEY", "", "", "",
C | E, Quirk::kNone, DP, false,
"https://console.upstage.ai/docs/getting-started",
"Probe: 401 invalid API key on /models, chat and embeddings."},
{"inception", "Inception (Mercury)", "https://api.inceptionlabs.ai/v1", "INCEPTION_API_KEY", "", "", "/models",
C | M, Quirk::kNone, DP, false,
"https://docs.inceptionlabs.ai/get-started/get-started",
"Probe: /models 200 public; chat answers 400 listing the valid model names before auth."},
{"stepfun", "StepFun", "https://api.stepfun.com/v1", "STEPFUN_API_KEY", "", "", "/models",
C | M, Quirk::kNone, DP, false,
"https://platform.stepfun.com/docs/api-reference/chat/chat-completion-create",
"Probe: 401 invalid_api_key on /models and chat."},
{"hunyuan", "Tencent Hunyuan", "https://api.hunyuan.cloud.tencent.com/v1", "HUNYUAN_API_KEY", "", "", "",
C, Quirk::kNone, PO, false,
"https://cloud.tencent.com/document/product/1729/111007",
"Docs page did not name the URL in readable text. Probe: 401 'Incorrect API key provided' on /models and chat."},
{"venice", "Venice AI", "https://api.venice.ai/api/v1", "VENICE_API_KEY", "", "", "/models",
C | M, Quirk::kNone, DP, false,
"https://docs.venice.ai/api-reference/endpoint/chat/completions",
"Probe: /models 200 public; unauthenticated chat answers 402 with an x402 payment envelope (not 401)."},
{"vercel", "Vercel AI Gateway", "https://ai-gateway.vercel.sh/v1", "AI_GATEWAY_API_KEY", "", "", "/models",
C | M, Quirk::kNone, DP, false,
"https://vercel.com/docs/ai-gateway/openai-compat",
"Probe: /models 200 public; chat 401 authentication_error 'Missing Authorization header'."},
{"friendli", "FriendliAI Serverless", "https://api.friendli.ai/serverless/v1", "FRIENDLI_TOKEN", "", "", "/models",
C | M, Quirk::kNone, DP, false,
"https://friendli.ai/docs/openapi/serverless/chat-completions",
"Probe: /models 200 public; chat 401 {detail}."},
{"gemini", "Google Gemini (OpenAI compatibility)", "https://generativelanguage.googleapis.com/v1beta/openai",
"GEMINI_API_KEY", "", "", "", C, Quirk::kNone, DP, false,
"https://ai.google.dev/gemini-api/docs/openai",
"Probe: chat answers 400 'Missing or invalid Authorization header'; unauthenticated GET /models and "
"embeddings answer 404, so only chat is claimed."},
{"baseten", "Baseten Model APIs", "https://inference.baseten.co/v1", "BASETEN_API_KEY", "", "", "",
C, Quirk::kNone, DP, false,
"https://docs.baseten.co/development/model-apis/overview",
"Probe: /models 401 'No Authorization header provided'; chat 403 'please check the api-key you provided'."},
{"featherless", "Featherless AI", "https://api.featherless.ai/v1", "FEATHERLESS_API_KEY", "", "", "/models",
C | M, Quirk::kNone, PO, false,
"https://featherless.ai/docs/overview",
"Docs text did not name the URL. Probe: /models 200 public; chat 401 'must be signed in'."},
{"ovh", "OVHcloud AI Endpoints", "https://oai.endpoints.kepler.ai.cloud.ovh.net/v1", "OVH_AI_ENDPOINTS_ACCESS_TOKEN",
"", "", "/models", C | M, Quirk::kNone, PO, false,
"https://help.ovhcloud.com/csm/en-gb-public-cloud-ai-endpoints-getting-started",
"Docs text did not name the URL. Probe: /models 200 public; chat on an unknown model answers 404 model_not_found."},
{"parasail", "Parasail", "https://api.parasail.io/v1", "PARASAIL_API_KEY", "", "", "/models",
C | M, Quirk::kNone, PO, false,
"https://docs.parasail.io/parasail-docs/serverless/serverless",
"Docs URL answered 404 at probe time. Probe: 401 'No bearer token provided' on /models and chat."},
{"crusoe", "Crusoe Managed Inference", "https://managed-inference-api-proxy.crusoecloud.com/v1", "CRUSOE_API_KEY",
"", "", "/models", C | M, Quirk::kNone, PO, false,
"https://docs.crusoe.ai/managed-inference",
"Docs URL answered 404 at probe time. Probe: 401 {errors:[Authentication failed]} on /models and chat."},
{"xiaomi", "Xiaomi MiMo", "https://api.xiaomimimo.com/v1", "XIAOMI_API_KEY", "", "", "",
C, Quirk::kNone, PO, false,
"https://platform.xiaomimimo.com/",
"Docs text did not name the URL. Probe: 401 'Invalid API Key' on /models and chat."},
{"chutes", "Chutes", "https://llm.chutes.ai/v1", "CHUTES_API_KEY", "", "", "/models",
C | M, Quirk::kNone, PO, false,
"https://docs.chutes.ai/",
"Docs text did not name the URL. Probe: /models 200 public; chat answered an nginx 429 HTML page."},
// Self-hosted servers: the key is optional, the row is configured when its base_env is set.
{"ollama", "Ollama", "http://localhost:11434/v1", "OLLAMA_API_KEY", "OLLAMA_BASE_URL", "", "/models",
C | E | M, Quirk::kNone, DO, true,
"https://raw.githubusercontent.com/ollama/ollama/main/docs/api/openai-compatibility.mdx",
"Docs name http://localhost:11434/v1. The key is required by OpenAI clients but ignored by the server; "
"the module sends the placeholder 'local' when none is set."},
{"lmstudio", "LM Studio", "http://localhost:1234/v1", "LMSTUDIO_API_KEY", "LMSTUDIO_BASE_URL", "", "/models",
C | E | M, Quirk::kNone, DO, true,
"https://lmstudio.ai/docs/developer/openai-compat",
"Docs name http://localhost:1234/v1. Key ignored unless configured in the app."},
{"vllm", "vLLM server", "http://localhost:8000/v1", "VLLM_API_KEY", "VLLM_BASE_URL", "", "/models",
C | E | M | T, Quirk::kNone, DO, true,
"https://docs.vllm.ai/en/latest/serving/online_serving/",
"Docs list /v1/chat/completions, /v1/embeddings and /v1/audio/transcriptions; key only when the server "
"was started with --api-key."},
{"llamacpp", "llama.cpp server", "http://localhost:8080/v1", "LLAMACPP_API_KEY", "LLAMACPP_BASE_URL", "", "/models",
C | E | M, Quirk::kNone, DO, true,
"https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md",
"README lists /v1/chat/completions; key only when started with --api-key. Same default port as LocalAI."},
{"localai", "LocalAI", "http://localhost:8080/v1", "LOCALAI_API_KEY", "LOCALAI_BASE_URL", "", "/models",
C | E | M, Quirk::kNone, DO, true,
"https://localai.io/docs/overview/",
"Docs name localhost:8080; endpoint list beyond chat, embeddings and models was not read."},
};
constexpr std::size_t kRowCount = sizeof(kRows) / sizeof(kRows[0]);
} // namespace
std::size_t Registry::count() { return kRowCount; }
const ProviderRow& Registry::at(std::size_t index) {
if (index >= kRowCount) throw llm::ApiError::invalid("compat", "registry index out of range");
return kRows[index];
}
const ProviderRow* Registry::find(const std::string& id) {
for (const ProviderRow& row : kRows)
if (id == row.id) return &row;
return nullptr;
}
const ProviderRow& Registry::get(const std::string& id) {
const ProviderRow* row = find(id);
if (!row) throw llm::ApiError::invalid("compat", "unknown provider '" + id.substr(0, 64) + "'");
return *row;
}
const char* Registry::evidence_name(Evidence e) {
switch (e) {
case Evidence::kDocsAndProbe: return "docs+probe";
case Evidence::kProbeOnly: return "probe-only";
case Evidence::kDocsOnly: return "docs-only";
}
return "";
}
std::string Registry::feature_names(const ProviderRow& row) {
struct Name { unsigned bit; const char* name; };
static const Name names[] = {{Feature::kChat, "chat"}, {Feature::kEmbeddings, "embeddings"},
{Feature::kModels, "models"}, {Feature::kImages, "images"},
{Feature::kTranscribe, "transcribe"}, {Feature::kSpeech, "speech"}};
std::string out;
for (const Name& n : names)
if (row.features & n.bit) out += (out.empty() ? "" : ",") + std::string(n.name);
return out;
}
} // namespace molo::compat
+72
View File
@@ -0,0 +1,72 @@
#pragma once
// The ONE provider table of this module. Adding a provider is one row in
// registry.cpp (plus its line in SPEC.md section 4, which tools/check_spec.py
// enforces). Everything else (clients, routes, demo, tests) iterates this
// table.
#include <cstddef>
#include <string>
namespace molo::compat {
// Endpoints a row claims. Claims come from the provider's own docs or a
// probe; SPEC.md says which. A call outside the mask is refused before any
// network traffic.
struct Feature {
static constexpr unsigned kChat = 1u << 0;
static constexpr unsigned kEmbeddings = 1u << 1;
static constexpr unsigned kModels = 1u << 2;
static constexpr unsigned kImages = 1u << 3;
static constexpr unsigned kTranscribe = 1u << 4;
static constexpr unsigned kSpeech = 1u << 5;
};
// Documented behavior that differs from OpenAI and that the module enforces
// loudly (reject) instead of letting the provider answer 400.
struct Quirk {
static constexpr unsigned kNone = 0;
static constexpr unsigned kRejectMessageName = 1u << 0; // messages[].name answers 400
static constexpr unsigned kRejectLogprobs = 1u << 1; // logprobs, top_logprobs, logit_bias answer 400
static constexpr unsigned kSingleChoice = 1u << 2; // n must be 1
static constexpr unsigned kNoToolsWithFormat = 1u << 3; // tools + response_format not combinable
};
// How the row was verified (see SPEC.md section 5).
enum class Evidence {
kDocsAndProbe, // docs text names the base URL; unauthenticated probe answered like an API
kProbeOnly, // docs text did not name it (or not readable); probe answered like an API
kDocsOnly, // docs name it; not probeable (local server, or needs an account id)
};
struct ProviderRow {
const char* id; // slug, [a-z0-9]+, used in routes and ids
const char* name; // display name
const char* base_url; // no trailing slash; "{account_id}" is replaced from extra_env
const char* key_env; // credential env var (via llm::Config); local rows: optional key
const char* base_env; // env var overriding base_url; for local rows also the "configured" switch
const char* extra_env; // second required env var (account id) or ""
const char* models_path; // "" when the row has no list endpoint
unsigned features; // Feature mask
unsigned quirks; // Quirk mask
Evidence evidence;
bool local; // self-hosted server, no vendor credential
const char* doc_url; // documentation page cited for the base URL
const char* note; // documented quirks and probe results, one line
};
class Registry {
public:
static std::size_t count();
static const ProviderRow& at(std::size_t index);
// nullptr when unknown.
static const ProviderRow* find(const std::string& id);
// Throws llm::ApiError::invalid for an unknown id.
static const ProviderRow& get(const std::string& id);
static bool has(const ProviderRow& row, unsigned feature) { return (row.features & feature) != 0; }
static const char* evidence_name(Evidence e);
// Comma list of the feature names a row claims ("chat,embeddings,models").
static std::string feature_names(const ProviderRow& row);
};
} // namespace molo::compat
+199
View File
@@ -0,0 +1,199 @@
#include "vendors/compat/services.h"
#include "clients/body/form.h"
#include "vendors/compat/caps.h"
namespace molo::compat {
using Json = clients::body::Json;
namespace {
[[noreturn]] void bad(const Provider& p, const std::string& why) {
throw llm::ApiError::invalid(p.client()->config().vendor, why);
}
void check_model(const Provider& p, const std::string& model) {
if (model.empty()) bad(p, "model is required (ids differ per provider)");
if (model.size() > CompatCaps::kMaxModelChars) bad(p, "model id too long");
}
std::string str_at(const Json& j, const char* key) {
const Json* v = j.is_object() ? j.find(key) : nullptr;
return v && v->is_string() ? v->as_string() : "";
}
int int_at(const Json& j, const char* key) {
const Json* v = j.is_object() ? j.find(key) : nullptr;
return v && v->is_number() ? static_cast<int>(v->as_int()) : 0;
}
} // namespace
bool ModelList::contains(const std::string& id) const {
for (const ModelEntry& m : models)
if (m.id == id) return true;
return false;
}
ModelList CompatModels::parse(const Json& doc) {
const Json* arr = &doc;
if (doc.is_object()) arr = doc.find("data");
ModelList out;
if (!arr || !arr->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "model list has no data array");
for (const Json& m : arr->as_array()) {
if (out.models.size() >= CompatCaps::kMaxModelsReturned) {
out.truncated = true;
break;
}
ModelEntry e;
e.id = str_at(m, "id");
if (e.id.empty()) e.id = str_at(m, "name");
if (e.id.empty()) continue;
e.owned_by = str_at(m, "owned_by");
out.models.push_back(e);
}
return out;
}
ModelList CompatModels::list() {
provider_.require(Call::kModels);
if (std::string(provider_.row().models_path).empty())
bad(provider_, std::string(provider_.row().name) + " has no list endpoint under its base URL");
return parse(client_->get(provider_.row().models_path));
}
Json CompatEmbeddings::to_body(const std::string& model, const std::vector<std::string>& inputs,
int dimensions) const {
provider_.require(Call::kEmbeddings);
check_model(provider_, model);
if (inputs.empty()) bad(provider_, "at least one input is required");
if (inputs.size() > CompatCaps::kMaxEmbeddingInputs) bad(provider_, "too many inputs");
if (dimensions < 0 || dimensions > 65536) bad(provider_, "dimensions out of range");
Json arr = Json::array();
for (const std::string& s : inputs) {
if (s.empty()) bad(provider_, "input must not be empty");
if (s.size() > CompatCaps::kMaxEmbeddingInputChars) bad(provider_, "input too long");
arr.push(Json(s));
}
Json body = Json::object();
body.set("model", Json(model));
body.set("input", arr);
if (dimensions > 0) body.set("dimensions", Json(dimensions));
return body;
}
EmbeddingResult CompatEmbeddings::parse(const Json& doc) {
EmbeddingResult out;
out.model = str_at(doc, "model");
const Json* data = doc.is_object() ? doc.find("data") : nullptr;
if (!data || !data->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "embeddings response has no data");
for (const Json& d : data->as_array()) {
Embedding e;
const Json* v = d.is_object() ? d.find("embedding") : nullptr;
if (!v || !v->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "embedding is not an array");
for (const Json& x : v->as_array()) e.vector.push_back(x.is_number() ? x.as_number() : 0.0);
out.items.push_back(std::move(e));
}
if (const Json* u = doc.find("usage"); u && u->is_object()) {
out.prompt_tokens = int_at(*u, "prompt_tokens");
out.total_tokens = int_at(*u, "total_tokens");
}
return out;
}
EmbeddingResult CompatEmbeddings::embed(const std::string& model, const std::vector<std::string>& inputs,
int dimensions) {
Json body = to_body(model, inputs, dimensions);
return parse(client_->post(Calls::spec_of(Call::kEmbeddings).path, body));
}
Json CompatImages::to_body(const ImageRequest& r) const {
provider_.require(Call::kImages);
check_model(provider_, r.model);
if (r.prompt.empty() || r.prompt.size() > CompatCaps::kMaxPromptChars) bad(provider_, "prompt must be 1..32768 chars");
if (r.n < 1 || r.n > CompatCaps::kMaxImages) bad(provider_, "n must be 1..10");
if (!r.response_format.empty() && r.response_format != "url" && r.response_format != "b64_json")
bad(provider_, "response_format must be url or b64_json");
if (!r.size.empty()) {
std::size_t x = r.size.find('x');
bool ok = x != std::string::npos && x > 0 && x + 1 < r.size.size() && r.size.size() <= 16;
for (std::size_t i = 0; ok && i < r.size.size(); ++i)
ok = i == x || (r.size[i] >= '0' && r.size[i] <= '9');
if (!ok) bad(provider_, "size must look like 1024x1024");
}
Json body = Json::object();
body.set("model", Json(r.model));
body.set("prompt", Json(r.prompt));
body.set("n", Json(r.n));
if (!r.size.empty()) body.set("size", Json(r.size));
if (!r.response_format.empty()) body.set("response_format", Json(r.response_format));
return body;
}
std::vector<ImageItem> CompatImages::parse(const Json& doc) {
const Json* data = doc.is_object() ? doc.find("data") : nullptr;
if (!data || !data->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "images response has no data");
std::vector<ImageItem> out;
for (const Json& d : data->as_array()) {
if (out.size() >= static_cast<std::size_t>(CompatCaps::kMaxImages)) break;
ImageItem i;
i.url = str_at(d, "url");
i.b64_json = str_at(d, "b64_json");
i.revised_prompt = str_at(d, "revised_prompt");
out.push_back(std::move(i));
}
return out;
}
std::vector<ImageItem> CompatImages::generate(const ImageRequest& request) {
Json body = to_body(request);
return parse(client_->post(Calls::spec_of(Call::kImages).path, body));
}
std::string CompatAudio::transcribe(const TranscribeRequest& r) {
provider_.require(Call::kTranscribe);
check_model(provider_, r.model);
if (r.audio.empty()) bad(provider_, "audio is empty");
if (r.audio.size() > CompatCaps::kMaxAudioBytes) bad(provider_, "audio over the byte cap");
if (r.filename.empty() || r.filename.size() > CompatCaps::kMaxFilenameChars) bad(provider_, "filename must be 1..256 chars");
if (r.filename.find_first_of("\"\r\n/\\") != std::string::npos) bad(provider_, "filename has forbidden characters");
clients::body::MultipartBody form;
form.add_file("file", r.filename, r.mime, r.audio);
form.add_text("model", r.model);
if (!r.language.empty()) form.add_text("language", r.language);
if (!r.prompt.empty()) form.add_text("prompt", r.prompt);
form.add_text("response_format", "json");
Json doc = client_->post_multipart(Calls::spec_of(Call::kTranscribe).path, form);
return str_at(doc, "text");
}
Json CompatAudio::speech_body(const SpeechRequest& r) const {
provider_.require(Call::kSpeech);
check_model(provider_, r.model);
if (r.input.empty() || r.input.size() > CompatCaps::kMaxSpeechChars) bad(provider_, "input must be 1..4096 chars");
if (r.voice.empty()) bad(provider_, "voice is required (names differ per provider)");
static const char* formats[] = {"mp3", "opus", "aac", "flac", "wav", "pcm"};
if (!r.format.empty()) {
bool ok = false;
for (const char* f : formats) ok = ok || r.format == f;
if (!ok) bad(provider_, "format must be mp3, opus, aac, flac, wav or pcm");
}
Json body = Json::object();
body.set("model", Json(r.model));
body.set("input", Json(r.input));
body.set("voice", Json(r.voice));
if (!r.format.empty()) body.set("response_format", Json(r.format));
return body;
}
std::string CompatAudio::speak(const SpeechRequest& r, std::string& content_type) {
Json body = speech_body(r);
std::string text = body.dump();
llm::RawResponse res = client_->request_raw(clients::Method::kPost, Calls::spec_of(Call::kSpeech).path, &text,
"application/json");
content_type = res.content_type;
return res.body;
}
} // namespace molo::compat
+104
View File
@@ -0,0 +1,104 @@
#pragma once
// The non-chat endpoints of a registry row. Each service refuses a call the
// row does not claim (Provider::require) before any network traffic, and
// validates every input against CompatCaps.
#include <optional>
#include <string>
#include <vector>
#include "clients/body/json.h"
#include "vendors/compat/provider.h"
namespace molo::compat {
struct ModelEntry {
std::string id;
std::string owned_by;
};
struct ModelList {
std::vector<ModelEntry> models;
bool truncated = false; // more entries than CompatCaps::kMaxModelsReturned
bool contains(const std::string& id) const;
};
// GET /models. Accepts {"data":[...]} and a bare array (some providers return one).
class CompatModels : public ProviderService {
public:
using ProviderService::ProviderService;
ModelList list();
static ModelList parse(const clients::body::Json& doc);
};
struct Embedding {
std::vector<double> vector;
};
struct EmbeddingResult {
std::string model;
std::vector<Embedding> items;
int prompt_tokens = 0;
int total_tokens = 0;
};
class CompatEmbeddings : public ProviderService {
public:
using ProviderService::ProviderService;
// dimensions: 0 = provider default.
EmbeddingResult embed(const std::string& model, const std::vector<std::string>& inputs, int dimensions = 0);
clients::body::Json to_body(const std::string& model, const std::vector<std::string>& inputs,
int dimensions) const;
static EmbeddingResult parse(const clients::body::Json& doc);
};
struct ImageItem {
std::string url;
std::string b64_json;
std::string revised_prompt;
};
struct ImageRequest {
std::string model;
std::string prompt;
int n = 1;
std::string size; // "1024x1024"; empty = provider default
std::string response_format; // "", "url" or "b64_json"
};
class CompatImages : public ProviderService {
public:
using ProviderService::ProviderService;
// Paid call: the caller decides (the demo keeps it a dry run).
std::vector<ImageItem> generate(const ImageRequest& request);
clients::body::Json to_body(const ImageRequest& request) const;
static std::vector<ImageItem> parse(const clients::body::Json& doc);
};
struct TranscribeRequest {
std::string model;
std::string filename;
std::string mime = "application/octet-stream";
std::string audio; // raw bytes
std::string language;
std::string prompt;
};
struct SpeechRequest {
std::string model;
std::string input;
std::string voice;
std::string format; // "", mp3, opus, aac, flac, wav, pcm
};
class CompatAudio : public ProviderService {
public:
using ProviderService::ProviderService;
std::string transcribe(const TranscribeRequest& request); // returns the text
// Audio bytes; the content type is returned through `content_type`.
std::string speak(const SpeechRequest& request, std::string& content_type);
clients::body::Json speech_body(const SpeechRequest& request) const;
};
} // namespace molo::compat
+239
View File
@@ -0,0 +1,239 @@
#!/usr/bin/env python3
"""Check src/vendors/compat against OpenAI's OpenAPI document and its own SPEC.
Every compat call is an OpenAI operation (providers differ only in base URL,
auth and quirks). Fails (exit 1) when
call a row of calls.cpp has a wrong method, path, operation id, request
schema, request body field or response field (fields are resolved
through $ref, allOf, oneOf and anyOf);
coverage an operation of the spec in the families this module touches
(chat completions, embeddings, models, images, audio) is neither
in the table nor in EXCLUDED;
registry the provider ids / base URLs / credential names of registry.cpp and
the table of SPEC.md section 4 differ in either direction.
The registry check needs no network: pass --local to run only it.
Python 3 stdlib only.
python3 -I src/vendors/compat/tools/check_spec.py [--local] [path/to/openapi.json]
"""
import json
import os
import re
import sys
import urllib.request
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.join(HERE, "..")
SPEC_SHA = "8243cc8fa6f04430fb1f1a6a2cb926fbeb4f7f26" # same pin as src/openai/core
URL = "https://raw.githubusercontent.com/openai/openai-openapi/%s/openapi.json" % SPEC_SHA
FAMILIES = ("/chat/completions", "/embeddings", "/models", "/images", "/audio", "/completions", "/moderations")
EXCLUDED = {
"listChatCompletions": "stored completions are an OpenAI product feature, not portable",
"getChatCompletion": "stored completions",
"updateChatCompletion": "stored completions",
"deleteChatCompletion": "stored completions",
"getChatCompletionMessages": "stored completions",
"createCompletion": "legacy completions; not claimed by any registry row",
"createModeration": "no registry row documents a moderation endpoint",
"createImageEdit": "multipart image editing; no registry row documents it",
"createImageVariation": "multipart image variations; no registry row documents it",
"createTranslation": "audio translation; no registry row documents it",
"createVoiceConsent": "OpenAI custom voices only",
"listVoiceConsents": "OpenAI custom voices only",
"getVoiceConsent": "OpenAI custom voices only",
"updateVoiceConsent": "OpenAI custom voices only",
"deleteVoiceConsent": "OpenAI custom voices only",
"createVoice": "OpenAI custom voices only",
"deleteVoice": "OpenAI custom voices only",
"retrieveModel": "no registry row documents GET /models/{id}; models are listed",
"deleteModel": "OpenAI fine-tune deletion only",
}
STR = re.compile(r'"((?:[^"\\]|\\.)*)"')
def string_runs(block):
"""String literals of a C++ block with adjacent literals concatenated."""
out, last_end, cur = [], None, None
for m in STR.finditer(block):
if last_end is not None and block[last_end:m.start()].strip() == "":
cur += m.group(1)
else:
if cur is not None:
out.append(cur)
cur = m.group(1)
last_end = m.end()
if cur is not None:
out.append(cur)
return out
def calls_rows():
text = open(os.path.join(ROOT, "calls.cpp")).read()
a = text.index("const CallSpec kCalls[]")
b = text.index("};", a)
rows = []
for blk in re.findall(r"\{Call::k\w+.*?Feature::k\w+, \"\w+\"\}", text[a:b], re.S):
rows.append(string_runs(blk))
return rows # method, path, op, request, fields, response, fields_read, feature_name
class Spec:
def __init__(self, doc):
self.doc = doc
def schema(self, name):
return self.doc["components"]["schemas"][name]
def deref(self, node):
while isinstance(node, dict) and "$ref" in node:
node = self.schema(node["$ref"].rsplit("/", 1)[1])
return node
def props(self, node, seen=None):
seen = seen or set()
out = set()
if isinstance(node, dict) and "$ref" in node:
ref = node["$ref"]
if ref in seen:
return out
seen.add(ref)
return self.props(self.deref(node), seen)
if not isinstance(node, dict):
return out
out |= set(node.get("properties", {}))
for key in ("allOf", "oneOf", "anyOf"):
for sub in node.get(key, []):
out |= self.props(sub, seen)
return out
def schema_name_of(self, content, want=""):
ref = content.get("schema", {})
if "$ref" in ref:
return ref["$ref"].rsplit("/", 1)[1]
# a union (anyOf / oneOf): the wanted name may be one member
for key in ("anyOf", "oneOf"):
for sub in ref.get(key, []):
if "$ref" in sub and sub["$ref"].rsplit("/", 1)[1] == want:
return want
return ""
def check_calls(spec, bad):
rows = calls_rows()
declared = set()
for method, path, op, req, fields, resp, read, _feat in rows:
node = spec.doc["paths"].get(path, {}).get(method.lower())
if not node:
bad.append("%s %s: not in the spec" % (method, path))
continue
declared.add(op)
if node["operationId"] != op:
bad.append("%s %s: operationId %s != %s" % (method, path, node["operationId"], op))
body = node.get("requestBody", {}).get("content", {})
if req:
content = body.get("application/json") or body.get("multipart/form-data") or {}
if spec.schema_name_of(content) != req:
bad.append("%s %s: request schema %s != %s" % (method, path, spec.schema_name_of(content), req))
known = spec.props(content.get("schema", {}))
for f in filter(None, fields.split(",")):
if f not in known:
bad.append("%s %s: body field %s not in %s" % (method, path, f, req))
if resp:
ok = node["responses"]["200"]["content"]["application/json"]
if spec.schema_name_of(ok, resp) != resp:
bad.append("%s %s: response schema %s != %s" % (method, path, spec.schema_name_of(ok), resp))
known = spec.props(ok.get("schema", {}))
if not known:
known = spec.props({"$ref": "#/components/schemas/" + resp})
for f in filter(None, read.split(",")):
if f not in known:
bad.append("%s %s: response field %s not in %s" % (method, path, f, resp))
for path, item in spec.doc["paths"].items():
if not path.startswith(FAMILIES):
continue
for m, node in item.items():
if m not in ("get", "post", "delete"):
continue
op = node["operationId"]
if op not in declared and op not in EXCLUDED:
bad.append("spec operation %s %s (%s) is neither in the table nor excluded" % (m.upper(), path, op))
for op in EXCLUDED:
if op in declared:
bad.append("%s is both declared and excluded" % op)
return len(rows)
def registry_rows():
text = open(os.path.join(ROOT, "registry.cpp")).read()
a = text.index("const ProviderRow kRows[]")
b = text.index("constexpr std::size_t kRowCount")
out = []
for blk in re.findall(r'\n \{"[a-z0-9]+",.*?(?=\n \{"|\n //|\n\};)', text[a:b], re.S):
s = string_runs(blk)
out.append({"id": s[0], "name": s[1], "base": s[2], "key": s[3], "base_env": s[4], "extra_env": s[5],
"models": s[6], "doc": s[7]})
return out
def spec_md_rows():
text = open(os.path.join(ROOT, "SPEC.md")).read()
a = text.index("## 4. Provider table")
b = text.index("\n## ", a + 5)
rows = {}
for line in text[a:b].splitlines():
cells = [c.strip().strip("`") for c in line.strip().strip("|").split("|")]
if line.startswith("|") and len(cells) >= 5 and re.fullmatch(r"[a-z0-9]+", cells[0]) and cells[0] != "id":
rows[cells[0]] = cells
return rows
def check_registry(bad):
reg = registry_rows()
md = spec_md_rows()
ids = [r["id"] for r in reg]
if len(ids) != len(set(ids)):
bad.append("registry.cpp has duplicate ids")
for r in reg:
cells = md.get(r["id"])
if not cells:
bad.append("provider %s is in registry.cpp but not in SPEC.md section 4" % r["id"])
continue
line = " ".join(cells)
for what in ("base", "key", "doc"):
if r[what] not in line:
bad.append("provider %s: %s '%s' differs between registry.cpp and SPEC.md" % (r["id"], what, r[what]))
for i in md:
if i not in ids:
bad.append("provider %s is in SPEC.md section 4 but not in registry.cpp" % i)
return len(reg)
def main():
args = [a for a in sys.argv[1:] if not a.startswith("--")]
local = "--local" in sys.argv
bad = []
n_reg = check_registry(bad)
n_calls = 0
if not local:
if args:
doc = json.load(open(args[0]))
else:
req = urllib.request.Request(URL, headers={"User-Agent": "molo-compat-spec-check/1"})
with urllib.request.urlopen(req, timeout=120) as r:
doc = json.loads(r.read().decode("utf-8"))
n_calls = check_calls(Spec(doc), bad)
print("openapi %s (%s)" % (doc["info"]["version"], SPEC_SHA[:12]))
print("registry rows: %d, call rows checked: %d" % (n_reg, n_calls))
if bad:
print("FAIL")
for b in bad:
print(" " + b)
sys.exit(1)
print("OK")
if __name__ == "__main__":
main()