feat(clients): compat: OpenAI-compatible provider registry (39 rows), chat model, services, demo
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Vendored
+243
@@ -0,0 +1,243 @@
|
||||
# OpenAI-compatible provider registry (compat) - Specification
|
||||
|
||||
Owned directory: `src/vendors/compat/`. Namespace `molo::compat`. Depends on
|
||||
the `src/llm` foundation (`ApiClient`, `Service`, `SseDecoder`, `ApiError`,
|
||||
`Config`, `ChatModel`). Plan: `plans/18-breadth.md` section 2 (method:
|
||||
`plans/17-llm-vendor-platforms.md`). Credentials: one env var per provider,
|
||||
named in the table of section 4, read through `llm::Config`.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
One client for every provider that serves OpenAI-shaped endpoints. A provider
|
||||
is ONE ROW in `registry.cpp`; clients, routes, demo page and tests iterate the
|
||||
table, so adding a provider is one row plus one line in section 4 (enforced
|
||||
by `tools/check_spec.py`).
|
||||
|
||||
Endpoints, all OpenAI operations (section 3 of the checker verifies them
|
||||
against OpenAI's OpenAPI document): chat completions (also streaming),
|
||||
embeddings, models list, image generation, audio transcription, speech. A row
|
||||
only claims an endpoint its own docs list or a probe distinguished; a call
|
||||
outside the row's claim is refused before any network traffic.
|
||||
|
||||
Not here, said plainly: legacy completions, moderations, translations,
|
||||
image edits and variations, stored completions, the Responses API (no row's
|
||||
docs were read for it except Cloudflare's, which lists `/responses`; not
|
||||
wrapped), files, batches, fine-tuning. Providers with their own native API
|
||||
have their own packages (mistral, cohere, perplexity, huggingface, ...).
|
||||
Providers whose OpenAI-compatible endpoint could not be verified are in
|
||||
section 5 and are not rows.
|
||||
|
||||
## 2. Module map
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings (`CompatCaps`) |
|
||||
| `registry.h/.cpp` | `ProviderRow`, `Feature`, `Quirk`, `Evidence`, `Registry` and THE table |
|
||||
| `calls.h/.cpp` | The call table (6 rows, each an OpenAI `operationId`) |
|
||||
| `provider.h/.cpp` | `Provider` (row + `ApiClient`, base URL resolution and validation), `ProviderService` base |
|
||||
| `chat_model.h/.cpp` | `CompatChatModel : llm::ChatModel` (generate, stream, wire mapping, quirks) |
|
||||
| `services.h/.cpp` | `CompatModels`, `CompatEmbeddings`, `CompatImages`, `CompatAudio` |
|
||||
| `tools/check_spec.py` | Checks the call table against OpenAI's OpenAPI document and the registry against section 4 |
|
||||
|
||||
## 3. Contracts and static caps
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| messages / tools / stop sequences | 4096 / 128 / 16 |
|
||||
| max_tokens | 1..4194304 |
|
||||
| temperature / top_p | 0..2 / (0,1] |
|
||||
| n (extra) | 1..8; exactly 1 on rows with `kSingleChoice` |
|
||||
| stream events | 200000 per stream, then the stream is stopped |
|
||||
| models returned | 5000, then `truncated=true` (loud) |
|
||||
| embedding inputs / chars per input | 2048 / 1 MiB |
|
||||
| image n / prompt chars | 1..10 / 32768 |
|
||||
| audio upload / speech input chars | 64 MiB / 4096 |
|
||||
| model id chars / base URL chars | 256 / 512 |
|
||||
|
||||
- Auth is `Authorization: Bearer <key>` for every row (the shape OpenAI
|
||||
clients use). Local rows (`local=true`) are "configured" only when their
|
||||
`base_env` is set; the key is optional and the placeholder `local` is sent
|
||||
when none is set. Cloudflare needs `CLOUDFLARE_ACCOUNT_ID` (validated
|
||||
`[A-Za-z0-9_-]{1,64}`) and substitutes it into the base URL.
|
||||
- `<ID>_BASE_URL` style overrides are validated: `http(s)://`, no spaces,
|
||||
control characters, `?`, `#` or `..`; trailing `/` removed.
|
||||
- The model id is always required: ids differ per provider, so the module has
|
||||
no honest default (`CompatChatModel` takes an optional default at construction).
|
||||
- Quirks that the docs state as a 400 are rejected loudly before traffic
|
||||
(`Quirk::kRejectMessageName`, `kRejectLogprobs`, `kSingleChoice`,
|
||||
`kNoToolsWithFormat`); everything else is passed through untouched.
|
||||
- Errors: every non-2xx maps to `llm::ApiError` (the shared envelope parser
|
||||
handles `{error:{...}}`, `{message,type,code}`, `{detail}` and plain text).
|
||||
A stream without `finish_reason` throws 502 `STREAM_TRUNCATED`.
|
||||
- Model list shapes: `{"data":[...]}` and a bare array are both accepted.
|
||||
|
||||
## 4. Provider table
|
||||
|
||||
Source of truth is `registry.cpp`; this table must match it (checker).
|
||||
Evidence: `docs+probe` = the provider's docs text names the base URL and an
|
||||
unauthenticated probe on 2026-10-25 answered like an API; `probe-only` = the
|
||||
docs page was not readable as text or did not name the URL, the probe answered
|
||||
like an API; `docs-only` = named by docs, not probeable from here.
|
||||
|
||||
| id | name | base URL | credential | features | evidence | documentation | documented quirks and probe results |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `groq` | Groq | `https://api.groq.com/openai/v1` | `GROQ_API_KEY` | chat,models,transcribe | docs+probe | https://console.groq.com/docs/openai | Docs: logprobs, logit_bias, top_logprobs and messages[].name answer 400; n must be 1; temperature 0 becomes 1e-8; transcription formats vtt and srt unsupported. Probe: 401 invalid_api_key on /models and chat. |
|
||||
| `together` | Together AI | `https://api.together.ai/v1` | `TOGETHER_API_KEY` | chat,embeddings,models,images,transcribe,speech | docs+probe | https://docs.together.ai/docs/openai-api-compatibility | Docs: logit_bias unsupported on most models; service_tier, store, metadata and prediction are accepted and ignored. Probe: 401 on /models and chat; legacy host api.together.xyz answers 401 too. |
|
||||
| `fireworks` | Fireworks AI | `https://api.fireworks.ai/inference/v1` | `FIREWORKS_API_KEY` | chat,embeddings,models | docs+probe | https://docs.fireworks.ai/tools-sdks/openai-compatibility | Probe: /models 401 with a docs link; unknown model answers 404 NOT_FOUND; empty embeddings body answers 500. |
|
||||
| `cerebras` | Cerebras | `https://api.cerebras.ai/v1` | `CEREBRAS_API_KEY` | chat,models | docs+probe | https://inference-docs.cerebras.ai/resources/openai | Docs: tools and response_format must not be combined (gpt-oss-120b rejects it); system and developer roles both act at developer level. Probe: /models 403 {detail}, chat 401 wrong_api_key, embeddings path 404. |
|
||||
| `sambanova` | SambaNova | `https://api.sambanova.ai/v1` | `SAMBANOVA_API_KEY` | chat,embeddings,models | probe-only | https://docs.sambanova.ai/docs/en/features/openai-compatibility | Docs page answers 403 to a plain fetch. Probe: /models 200 public; chat 401 (also accepts x-api-key); embeddings answers 400 'must provide a model' before auth. |
|
||||
| `deepinfra` | DeepInfra | `https://api.deepinfra.com/v1/openai` | `DEEPINFRA_API_KEY` | chat,embeddings,models,images,transcribe | docs+probe | https://docs.deepinfra.com/chat/overview | Probe: /models 200 public; chat 401 missing API key; images and audio paths answer 401 with a {detail} body. |
|
||||
| `novita` | Novita AI | `https://api.novita.ai/openai/v1` | `NOVITA_API_KEY` | chat,models | docs+probe | https://novita.ai/docs/guides/llm-api | Probe: /models 200 public; bad key answers 403 INVALID_API_KEY (not 401); legacy base /v3/openai answers the same. |
|
||||
| `nvidia` | NVIDIA NIM (build.nvidia.com) | `https://integrate.api.nvidia.com/v1` | `NVIDIA_API_KEY` | chat,embeddings,models | docs+probe | https://docs.api.nvidia.com/nim/reference/llm-apis | Probe: /models 200 public; embeddings answers 400 'model field is required'; retired models answer 410 Gone. |
|
||||
| `cloudflare` | Cloudflare Workers AI | `https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1` | `CLOUDFLARE_API_TOKEN + CLOUDFLARE_ACCOUNT_ID` | chat,embeddings | docs-only | https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/ | Docs list /chat/completions, /embeddings and /responses under the account path. Probe: GET /models answers 405 (no list endpoint here); not probeable further without an account id. |
|
||||
| `moonshot` | Moonshot (Kimi) | `https://api.moonshot.ai/v1` | `MOONSHOT_API_KEY` | chat,models | docs+probe | https://platform.kimi.ai/docs/api/chat | Probe: 401 incorrect_api_key_error on /models and chat; the China host api.moonshot.cn answers the same (separate keys). |
|
||||
| `dashscope` | Alibaba DashScope (Qwen, intl) | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `DASHSCOPE_API_KEY` | chat,embeddings | docs+probe | https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope | Probe: 401 on /models and chat; embeddings answers 400 'you must provide a model parameter'; China host dashscope.aliyuncs.com/compatible-mode/v1 answers 401 too. |
|
||||
| `zhipu` | Zhipu (GLM, China) | `https://open.bigmodel.cn/api/paas/v4` | `ZHIPU_API_KEY` | chat,embeddings,images | docs+probe | https://docs.bigmodel.cn/cn/guide/develop/openai/introduction | Docs name the base URL; the path is /api/paas/v4, not /v1. Probe: 401 code 1001 (missing Authorization) on chat, embeddings and images. |
|
||||
| `zai` | Z.ai (GLM, international) | `https://api.z.ai/api/paas/v4` | `ZAI_API_KEY` | chat,embeddings,images | docs+probe | https://docs.z.ai/api-reference/introduction | International host of the Zhipu API, same /api/paas/v4 path. Probe: 401 code 1001. |
|
||||
| `minimax` | MiniMax | `https://api.minimax.io/v1` | `MINIMAX_API_KEY` | chat | probe-only | https://platform.minimax.io/docs/api-reference/text-openai-api | Docs text did not name the URL. Probe: 401 authorized_error on /models and chat; embeddings answers HTTP 200 with a base_resp status_code 1004 envelope (not OpenAI shaped); China host api.minimaxi.com answers 401. |
|
||||
| `qianfan` | Baidu Qianfan | `https://qianfan.baidubce.com/v2` | `QIANFAN_API_KEY` | chat,embeddings | probe-only | https://cloud.baidu.com/doc/qianfan-api/s/3m9b5lqft | Docs page is rendered client side. Probe: chat and embeddings 401 invalid_iam_token; GET /models 403 AccessDenied (different envelope). |
|
||||
| `ark` | Volcengine Ark (Doubao) | `https://ark.cn-beijing.volces.com/api/v3` | `ARK_API_KEY` | chat | probe-only | https://www.volcengine.com/docs/82379/1298454 | Docs page is rendered client side. Probe: 401 AuthenticationError on /models, chat and embeddings; path is /api/v3. |
|
||||
| `siliconflow` | SiliconFlow | `https://api.siliconflow.com/v1` | `SILICONFLOW_API_KEY` | chat,embeddings,models | docs+probe | https://docs.siliconflow.com/en/api-reference/chat-completions/chat-completions | Probe: 401 code 30014 with a {code,data,message} envelope on every path; China host api.siliconflow.cn identical. |
|
||||
| `nebius` | Nebius Token Factory | `https://api.studio.nebius.com/v1` | `NEBIUS_API_KEY` | chat,embeddings,models | docs+probe | https://docs.tokenfactory.nebius.com/api-reference/introduction | Probe: 401 unauthenticated on /models, chat and embeddings; images/generations 404 on this host; api.tokenfactory.nebius.com/v1 answers the same. |
|
||||
| `scaleway` | Scaleway Generative APIs | `https://api.scaleway.ai/v1` | `SCALEWAY_API_KEY` | chat,embeddings,models | docs+probe | https://www.scaleway.com/en/docs/generative-apis/api-cli/using-chat-api/ | Probe: 401 UNAUTHORIZED (missing 'Authorization' header) on /models, chat and embeddings. |
|
||||
| `upstage` | Upstage (Solar) | `https://api.upstage.ai/v1` | `UPSTAGE_API_KEY` | chat,embeddings | docs+probe | https://console.upstage.ai/docs/getting-started | Probe: 401 invalid API key on /models, chat and embeddings. |
|
||||
| `inception` | Inception (Mercury) | `https://api.inceptionlabs.ai/v1` | `INCEPTION_API_KEY` | chat,models | docs+probe | https://docs.inceptionlabs.ai/get-started/get-started | Probe: /models 200 public; chat answers 400 listing the valid model names before auth. |
|
||||
| `stepfun` | StepFun | `https://api.stepfun.com/v1` | `STEPFUN_API_KEY` | chat,models | docs+probe | https://platform.stepfun.com/docs/api-reference/chat/chat-completion-create | Probe: 401 invalid_api_key on /models and chat. |
|
||||
| `hunyuan` | Tencent Hunyuan | `https://api.hunyuan.cloud.tencent.com/v1` | `HUNYUAN_API_KEY` | chat | probe-only | https://cloud.tencent.com/document/product/1729/111007 | Docs page did not name the URL in readable text. Probe: 401 'Incorrect API key provided' on /models and chat. |
|
||||
| `venice` | Venice AI | `https://api.venice.ai/api/v1` | `VENICE_API_KEY` | chat,models | docs+probe | https://docs.venice.ai/api-reference/endpoint/chat/completions | Probe: /models 200 public; unauthenticated chat answers 402 with an x402 payment envelope (not 401). |
|
||||
| `vercel` | Vercel AI Gateway | `https://ai-gateway.vercel.sh/v1` | `AI_GATEWAY_API_KEY` | chat,models | docs+probe | https://vercel.com/docs/ai-gateway/openai-compat | Probe: /models 200 public; chat 401 authentication_error 'Missing Authorization header'. |
|
||||
| `friendli` | FriendliAI Serverless | `https://api.friendli.ai/serverless/v1` | `FRIENDLI_TOKEN` | chat,models | docs+probe | https://friendli.ai/docs/openapi/serverless/chat-completions | Probe: /models 200 public; chat 401 {detail}. |
|
||||
| `gemini` | Google Gemini (OpenAI compatibility) | `https://generativelanguage.googleapis.com/v1beta/openai` | `GEMINI_API_KEY` | chat | docs+probe | https://ai.google.dev/gemini-api/docs/openai | Probe: chat answers 400 'Missing or invalid Authorization header'; unauthenticated GET /models and embeddings answer 404, so only chat is claimed. |
|
||||
| `baseten` | Baseten Model APIs | `https://inference.baseten.co/v1` | `BASETEN_API_KEY` | chat | docs+probe | https://docs.baseten.co/development/model-apis/overview | Probe: /models 401 'No Authorization header provided'; chat 403 'please check the api-key you provided'. |
|
||||
| `featherless` | Featherless AI | `https://api.featherless.ai/v1` | `FEATHERLESS_API_KEY` | chat,models | probe-only | https://featherless.ai/docs/overview | Docs text did not name the URL. Probe: /models 200 public; chat 401 'must be signed in'. |
|
||||
| `ovh` | OVHcloud AI Endpoints | `https://oai.endpoints.kepler.ai.cloud.ovh.net/v1` | `OVH_AI_ENDPOINTS_ACCESS_TOKEN` | chat,models | probe-only | https://help.ovhcloud.com/csm/en-gb-public-cloud-ai-endpoints-getting-started | Docs text did not name the URL. Probe: /models 200 public; chat on an unknown model answers 404 model_not_found. |
|
||||
| `parasail` | Parasail | `https://api.parasail.io/v1` | `PARASAIL_API_KEY` | chat,models | probe-only | https://docs.parasail.io/parasail-docs/serverless/serverless | Docs URL answered 404 at probe time. Probe: 401 'No bearer token provided' on /models and chat. |
|
||||
| `crusoe` | Crusoe Managed Inference | `https://managed-inference-api-proxy.crusoecloud.com/v1` | `CRUSOE_API_KEY` | chat,models | probe-only | https://docs.crusoe.ai/managed-inference | Docs URL answered 404 at probe time. Probe: 401 {errors:[Authentication failed]} on /models and chat. |
|
||||
| `xiaomi` | Xiaomi MiMo | `https://api.xiaomimimo.com/v1` | `XIAOMI_API_KEY` | chat | probe-only | https://platform.xiaomimimo.com/ | Docs text did not name the URL. Probe: 401 'Invalid API Key' on /models and chat. |
|
||||
| `chutes` | Chutes | `https://llm.chutes.ai/v1` | `CHUTES_API_KEY` | chat,models | probe-only | https://docs.chutes.ai/ | Docs text did not name the URL. Probe: /models 200 public; chat answered an nginx 429 HTML page. |
|
||||
| `ollama` | Ollama | `http://localhost:11434/v1` | `OLLAMA_API_KEY + OLLAMA_BASE_URL` | chat,embeddings,models | docs-only | https://raw.githubusercontent.com/ollama/ollama/main/docs/api/openai-compatibility.mdx | Docs name http://localhost:11434/v1. The key is required by OpenAI clients but ignored by the server; the module sends the placeholder 'local' when none is set. |
|
||||
| `lmstudio` | LM Studio | `http://localhost:1234/v1` | `LMSTUDIO_API_KEY + LMSTUDIO_BASE_URL` | chat,embeddings,models | docs-only | https://lmstudio.ai/docs/developer/openai-compat | Docs name http://localhost:1234/v1. Key ignored unless configured in the app. |
|
||||
| `vllm` | vLLM server | `http://localhost:8000/v1` | `VLLM_API_KEY + VLLM_BASE_URL` | chat,embeddings,models,transcribe | docs-only | https://docs.vllm.ai/en/latest/serving/online_serving/ | Docs list /v1/chat/completions, /v1/embeddings and /v1/audio/transcriptions; key only when the server was started with --api-key. |
|
||||
| `llamacpp` | llama.cpp server | `http://localhost:8080/v1` | `LLAMACPP_API_KEY + LLAMACPP_BASE_URL` | chat,embeddings,models | docs-only | https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md | README lists /v1/chat/completions; key only when started with --api-key. Same default port as LocalAI. |
|
||||
| `localai` | LocalAI | `http://localhost:8080/v1` | `LOCALAI_API_KEY + LOCALAI_BASE_URL` | chat,embeddings,models | docs-only | https://localai.io/docs/overview/ | Docs name localhost:8080; endpoint list beyond chat, embeddings and models was not read. |
|
||||
|
||||
## 5. Providers not in the table
|
||||
|
||||
Plan 18 names these; they are NOT rows because the OpenAI-compatible endpoint
|
||||
could not be verified from here (2026-10-25). Each is one row away once
|
||||
confirmed.
|
||||
|
||||
| Provider | What was seen |
|
||||
|---|---|
|
||||
| GitHub Models | Docs name `https://models.github.ai/inference/chat/completions` and a catalog at `https://models.github.ai/catalog/models`, but the docs page read did not contain the URL as text, and every unauthenticated request (any path) answered `200 OK` with body `OK`, which proves nothing. The catalog lives outside the inference base path, which the row model cannot express (`models_path` is relative to the base) |
|
||||
| Lambda | `api.lambda.ai`, `api.lambdalabs.com` and `inference.lambda.ai` did not resolve from here |
|
||||
| Lepton | `api.lepton.ai` did not resolve (service wound down) |
|
||||
| Hyperbolic | `/models` answers 401, but chat and image paths answered `404 "This inference endpoint has been decommissioned"` on every probe |
|
||||
| Kluster | `api.kluster.ai` did not resolve |
|
||||
| Anyscale | The Public Endpoints API answers a 404 page announcing its end |
|
||||
| Sarvam | `/v1/models` is public, chat answered 403 `invalid_api_key_error`; Sarvam documents its own subscription-key header, not Bearer, so the Bearer-only row model would be wrong |
|
||||
| Reka | `api.reka.ai/v1` answers 401 `missing_api_key`; no OpenAI-compatibility statement was found |
|
||||
| AI21 | `/studio/v1/models` answers an empty public list and chat 400 "bad or missing authentication"; the chat shape was not confirmed as OpenAI's |
|
||||
| Mistral, Cohere, Perplexity, Hugging Face, OpenRouter, xAI, DeepSeek, OpenAI | Own packages (plan 17 / 18) |
|
||||
|
||||
## 6. Demo routes
|
||||
|
||||
Page `/ai/compat`, GET `/ai/compat/status`, `/ai/compat/providers` (the
|
||||
table plus per-row configured state), `/ai/compat/calls`. POST routes (body
|
||||
carries `"provider":"<id>"`): `chat`, `models`, `embeddings`, `images`,
|
||||
`transcribe`, `speech`. 422 on bad input or a call the row does not claim,
|
||||
503 `compat_not_configured` without the row's credential (the message names
|
||||
the missing env), paid routes (all but `models`) are dry runs showing the
|
||||
exact request and URL unless the body has `"confirm":true`. Matrix row
|
||||
`ai-compat` for the page.
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- OpenAI OpenAPI document `openai/openai-openapi` at commit
|
||||
`8243cc8fa6f04430fb1f1a6a2cb926fbeb4f7f26` (`openapi.json`, version 2.3.0):
|
||||
the shape of every call.
|
||||
- Per-row documentation URL in the table of section 4. Each page was fetched
|
||||
as plain text on 2026-10-25 and searched for the base URL; "docs+probe"
|
||||
rows contain it. The documented quirks encoded as `Quirk` flags were read
|
||||
from: Groq (`console.groq.com/docs/openai`, "Currently Unsupported OpenAI
|
||||
Features": logprobs, logit_bias, top_logprobs, messages[].name answer 400;
|
||||
n must be 1) and Cerebras (`inference-docs.cerebras.ai/resources/openai`,
|
||||
"Unsupported parameter combinations": tools with response_format).
|
||||
- Together's docs page lists `/v1/chat/completions`, `/v1/embeddings`,
|
||||
`/v1/images/generations`, `/v1/audio/speech`, `/v1/audio/transcriptions` and
|
||||
moved its host to `api.together.ai`; Cloudflare's page lists
|
||||
`/chat/completions`, `/embeddings`, `/responses` under
|
||||
`/client/v4/accounts/{account_id}/ai/v1`; vLLM's page lists
|
||||
`/v1/chat/completions`, `/v1/embeddings`, `/v1/audio/transcriptions`.
|
||||
|
||||
## 8. Verified against spec
|
||||
|
||||
`python3 -I src/vendors/compat/tools/check_spec.py` on 2026-10-25 against
|
||||
the pinned OpenAI document (2.3.0): 6 call rows OK (method, path, operation
|
||||
id, request schema, every body field, response schema, every response field
|
||||
read), coverage OK (19 operations of the touched families are excluded with a
|
||||
reason in the script), registry OK (39 rows equal the table of section 4 in
|
||||
ids, base URLs, credential names and documentation URLs). `--local` runs only
|
||||
the registry check without network. This checks the OpenAI shape, not any
|
||||
provider's conformance to it.
|
||||
|
||||
## 9. Verified live
|
||||
|
||||
No provider key exists in this environment. Unauthenticated probes, no key
|
||||
sent, 2026-10-25 (`MOLO_LIVE_NET=1`, test `compat_live.invalid_key_probe...`,
|
||||
which uses an obviously invalid key and asserts "never 404"). Result per row
|
||||
(status of `GET /models` with the invalid key where the row lists models,
|
||||
otherwise a one-token `POST /chat/completions`): 401 for groq, together,
|
||||
fireworks, cerebras, deepinfra, dashscope, zhipu, zai, minimax, ark,
|
||||
siliconflow, nebius, upstage, moonshot, stepfun, hunyuan, vercel, featherless,
|
||||
parasail, crusoe, xiaomi; 403 for scaleway, baseten, ovh; 400 for gemini
|
||||
(`Missing or invalid Authorization header`); 200 for novita, nvidia, inception,
|
||||
venice, friendli (public lists that ignore the key). Three hosts could not be reached by OUR
|
||||
client: sambanova and chutes (`wolfSSL handshake failed: ASN parsing error,
|
||||
invalid input`) and qianfan (`handshake failed: received alert fatal error`)
|
||||
while `curl` reaches all three; see Gaps. Local rows were not probed (no
|
||||
server). Opt-in live tests: per row with its key env set, list models (when
|
||||
claimed) and ONE completion of `max_tokens` 8 on `COMPAT_<ID>_MODEL` (upper
|
||||
case id); a missing key or model prints a loud SKIP line per row. No success
|
||||
body from any provider was seen.
|
||||
|
||||
## 10. Unverified
|
||||
|
||||
- Every success response of every provider (only OpenAI's documented examples
|
||||
are used as parsing fixtures, labelled FIXTURE in the tests).
|
||||
- The `features` mask of each row beyond what its docs page or a probe showed.
|
||||
An unauthenticated 401 does not prove a path exists (many providers check
|
||||
auth first), so only these claims rest on more: embeddings routes answering
|
||||
400 before auth (sambanova, nvidia, dashscope), a 404 proving absence
|
||||
(cerebras embeddings, nebius images, gemini models and embeddings), and the
|
||||
docs pages read for Together and Cloudflare. All other claims are the
|
||||
provider's well known documented surface, not re-read here: images and audio
|
||||
for together, deepinfra, zhipu, zai; transcription for groq and vllm.
|
||||
- `docs-only` rows: Cloudflare (needs an account id), the five local servers.
|
||||
- Quirks other than the four flags (for example Groq turning temperature 0
|
||||
into 1e-8, Moonshot and Kimi temperature limits, per-model parameter
|
||||
support) are listed in notes or not at all, and are not enforced.
|
||||
- Provider-specific auth beyond Bearer (Baseten documents its own scheme; the
|
||||
probe showed Bearer is the shape OpenAI clients send, acceptance with a real
|
||||
key is unproven).
|
||||
- China-region hosts (moonshot.cn, dashscope.aliyuncs.com, minimaxi.com,
|
||||
siliconflow.cn) answer like their international twins but are not rows; set
|
||||
the override env for local rows only. Hosted rows cannot be overridden by
|
||||
env today (see Gaps).
|
||||
|
||||
## 11. Gaps
|
||||
|
||||
- `src/clients` TLS: wolfSSL handshake fails against sambanova, chutes and
|
||||
qianfan hosts where curl succeeds. Owner: `src/clients/transports` (not this
|
||||
package); the rows stay because the endpoints answer to curl.
|
||||
- `src/llm`: the shared `MOLO_LIVE_NET` is unset by a test in `tests/src/llm`
|
||||
(`test_llm.cpp`) so any suite reading it at run time skips in the shared
|
||||
binary; this suite reads it at static-init time. Other vendors' live tests
|
||||
still skip when the whole `ai_test` binary runs.
|
||||
- No env override for hosted base URLs (regional hosts); the `base_env` switch
|
||||
exists for local rows only. Closing it is one `base_env` value per hosted
|
||||
row in `registry.cpp`.
|
||||
- GitHub Models needs a second base URL for its catalog (see section 5).
|
||||
- Streaming sends no `stream_options.include_usage` by default because some
|
||||
providers reject it; pass it through `ChatRequest.extra`.
|
||||
- Responses API, legacy completions, image edits and moderation are not wrapped.
|
||||
Vendored
+50
@@ -0,0 +1,50 @@
|
||||
#include "vendors/compat/calls.h"
|
||||
|
||||
#include "llm/error.h"
|
||||
#include "vendors/compat/registry.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
namespace {
|
||||
|
||||
const CallSpec kCalls[] = {
|
||||
{Call::kChat, "POST", "/chat/completions", "createChatCompletion", "CreateChatCompletionRequest",
|
||||
"model,messages,max_tokens,temperature,top_p,stop,tools,tool_choice,response_format,reasoning_effort,"
|
||||
"stream,stream_options,n",
|
||||
"CreateChatCompletionResponse", "id,model,choices,usage", Feature::kChat, "chat"},
|
||||
{Call::kEmbeddings, "POST", "/embeddings", "createEmbedding", "CreateEmbeddingRequest",
|
||||
"model,input,dimensions,encoding_format", "CreateEmbeddingResponse", "data,model,usage",
|
||||
Feature::kEmbeddings, "embeddings"},
|
||||
{Call::kModels, "GET", "/models", "listModels", "", "", "ListModelsResponse", "data",
|
||||
Feature::kModels, "models"},
|
||||
{Call::kImages, "POST", "/images/generations", "createImage", "CreateImageRequest",
|
||||
"model,prompt,n,size,response_format", "ImagesResponse", "data", Feature::kImages, "images"},
|
||||
{Call::kTranscribe, "POST", "/audio/transcriptions", "createTranscription", "CreateTranscriptionRequest",
|
||||
"file,model,language,prompt,response_format,temperature", "CreateTranscriptionResponseJson", "text",
|
||||
Feature::kTranscribe, "transcribe"},
|
||||
{Call::kSpeech, "POST", "/audio/speech", "createSpeech", "CreateSpeechRequest",
|
||||
"model,input,voice,response_format,speed", "", "", Feature::kSpeech, "speech"},
|
||||
};
|
||||
|
||||
constexpr std::size_t kCallCount = sizeof(kCalls) / sizeof(kCalls[0]);
|
||||
|
||||
} // namespace
|
||||
|
||||
const CallSpec& Calls::spec_of(Call call) {
|
||||
for (const CallSpec& s : kCalls)
|
||||
if (s.call == call) return s;
|
||||
throw llm::ApiError::invalid("compat", "unknown call");
|
||||
}
|
||||
|
||||
clients::Method Calls::method_of(Call call) {
|
||||
return std::string(spec_of(call).http_method) == "GET" ? clients::Method::kGet : clients::Method::kPost;
|
||||
}
|
||||
|
||||
std::size_t Calls::count() { return kCallCount; }
|
||||
|
||||
const CallSpec& Calls::at(std::size_t index) {
|
||||
if (index >= kCallCount) throw llm::ApiError::invalid("compat", "call index out of range");
|
||||
return kCalls[index];
|
||||
}
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+38
@@ -0,0 +1,38 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of REST calls this module makes. The shapes are OpenAI's
|
||||
// (every row is an OpenAI operation); providers differ only in base URL, auth
|
||||
// and quirks, which live in registry.cpp. tools/check_spec.py checks every
|
||||
// row against OpenAI's published OpenAPI document. Code builds every path
|
||||
// from this table, never from a literal.
|
||||
|
||||
#include <cstddef>
|
||||
#include <string>
|
||||
|
||||
#include "clients/request.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
enum class Call { kChat, kEmbeddings, kModels, kImages, kTranscribe, kSpeech };
|
||||
|
||||
struct CallSpec {
|
||||
Call call;
|
||||
const char* http_method; // GET POST
|
||||
const char* path; // from the provider base URL
|
||||
const char* operation_id; // OpenAI OpenAPI operationId
|
||||
const char* request; // OpenAI request schema name ("" for GET)
|
||||
const char* body_fields; // comma list: request body fields we send
|
||||
const char* response; // OpenAI response schema name
|
||||
const char* fields_read; // comma list: response fields we read
|
||||
unsigned feature; // registry Feature bit a row must claim
|
||||
const char* feature_name;
|
||||
};
|
||||
|
||||
struct Calls {
|
||||
static const CallSpec& spec_of(Call call);
|
||||
static clients::Method method_of(Call call);
|
||||
static std::size_t count();
|
||||
static const CallSpec& at(std::size_t index);
|
||||
};
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+28
@@ -0,0 +1,28 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for the compat module. Never autotuned.
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
struct CompatCaps {
|
||||
static constexpr std::size_t kMaxMessages = 4096;
|
||||
static constexpr std::size_t kMaxTools = 128;
|
||||
static constexpr std::size_t kMaxStopSequences = 16;
|
||||
static constexpr std::size_t kMaxStreamEvents = 200000;
|
||||
static constexpr std::size_t kMaxChoices = 8;
|
||||
static constexpr std::size_t kMaxModelsReturned = 5000;
|
||||
static constexpr std::size_t kMaxEmbeddingInputs = 2048;
|
||||
static constexpr std::size_t kMaxEmbeddingInputChars = 1u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxPromptChars = 32768;
|
||||
static constexpr int kMaxImages = 10;
|
||||
static constexpr std::size_t kMaxAudioBytes = 64u * 1024u * 1024u;
|
||||
static constexpr std::size_t kMaxSpeechChars = 4096;
|
||||
static constexpr std::size_t kMaxModelChars = 256;
|
||||
static constexpr std::size_t kMaxBaseUrlChars = 512;
|
||||
static constexpr std::size_t kMaxFilenameChars = 256;
|
||||
static constexpr int kMaxTokensCeiling = 4194304;
|
||||
};
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+375
@@ -0,0 +1,375 @@
|
||||
#include "vendors/compat/chat_model.h"
|
||||
|
||||
#include <map>
|
||||
|
||||
#include "security/crypto/Base64.h"
|
||||
#include "vendors/compat/caps.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
using Json = clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const std::string& vendor, const std::string& why) {
|
||||
throw llm::ApiError::invalid(vendor, why);
|
||||
}
|
||||
|
||||
std::string str_at(const Json& j, const char* key) {
|
||||
const Json* v = j.is_object() ? j.find(key) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
}
|
||||
|
||||
int int_at(const Json& j, const char* key) {
|
||||
const Json* v = j.is_object() ? j.find(key) : nullptr;
|
||||
return v && v->is_number() ? static_cast<int>(v->as_int()) : 0;
|
||||
}
|
||||
|
||||
const Json* obj_at(const Json& j, const char* key) {
|
||||
const Json* v = j.is_object() ? j.find(key) : nullptr;
|
||||
return v && v->is_object() ? v : nullptr;
|
||||
}
|
||||
|
||||
Json image_part(const std::string& vendor, const llm::ImageData& img) {
|
||||
std::string url = img.url;
|
||||
if (url.empty()) {
|
||||
if (img.bytes.empty()) bad(vendor, "image has neither url nor bytes");
|
||||
if (img.mime.empty()) bad(vendor, "image bytes need a mime type");
|
||||
url = "data:" + img.mime + ";base64," + security::crypto::Base64::encode(img.bytes);
|
||||
}
|
||||
Json u = Json::object();
|
||||
u.set("url", Json(url));
|
||||
Json p = Json::object();
|
||||
p.set("type", Json("image_url"));
|
||||
p.set("image_url", u);
|
||||
return p;
|
||||
}
|
||||
|
||||
Json text_part(const std::string& t) {
|
||||
Json p = Json::object();
|
||||
p.set("type", Json("text"));
|
||||
p.set("text", Json(t));
|
||||
return p;
|
||||
}
|
||||
|
||||
Json tool_call_json(const llm::ToolCall& c) {
|
||||
Json fn = Json::object();
|
||||
fn.set("name", Json(c.name));
|
||||
fn.set("arguments", Json(c.arguments.is_null() ? std::string("{}") : c.arguments.dump()));
|
||||
Json o = Json::object();
|
||||
o.set("id", Json(c.id));
|
||||
o.set("type", Json("function"));
|
||||
o.set("function", fn);
|
||||
return o;
|
||||
}
|
||||
|
||||
void append_message(const std::string& vendor, const ProviderRow& row, Json& out, const llm::Message& m) {
|
||||
if (!m.name.empty() && (row.quirks & Quirk::kRejectMessageName))
|
||||
bad(vendor, std::string(row.name) + " rejects messages[].name (documented 400)");
|
||||
auto base = [&](const char* role) {
|
||||
Json o = Json::object();
|
||||
o.set("role", Json(role));
|
||||
if (!m.name.empty()) o.set("name", Json(m.name));
|
||||
return o;
|
||||
};
|
||||
switch (m.role) {
|
||||
case llm::Role::kSystem: {
|
||||
Json o = base("system");
|
||||
o.set("content", Json(m.text()));
|
||||
out.push(o);
|
||||
return;
|
||||
}
|
||||
case llm::Role::kUser: {
|
||||
Json o = base("user");
|
||||
bool has_image = false;
|
||||
for (const llm::ContentPart& p : m.parts) has_image = has_image || p.image.has_value();
|
||||
if (!has_image) {
|
||||
o.set("content", Json(m.text()));
|
||||
} else {
|
||||
Json parts = Json::array();
|
||||
for (const llm::ContentPart& p : m.parts) {
|
||||
if (!p.text.empty()) parts.push(text_part(p.text));
|
||||
if (p.image) parts.push(image_part(vendor, *p.image));
|
||||
}
|
||||
o.set("content", parts);
|
||||
}
|
||||
out.push(o);
|
||||
return;
|
||||
}
|
||||
case llm::Role::kAssistant: {
|
||||
Json o = base("assistant");
|
||||
Json calls = Json::array();
|
||||
bool any = false;
|
||||
for (const llm::ContentPart& p : m.parts)
|
||||
if (p.tool_call) {
|
||||
calls.push(tool_call_json(*p.tool_call));
|
||||
any = true;
|
||||
}
|
||||
std::string text = m.text();
|
||||
o.set("content", text.empty() && any ? Json() : Json(text));
|
||||
if (any) o.set("tool_calls", calls);
|
||||
out.push(o);
|
||||
return;
|
||||
}
|
||||
case llm::Role::kTool: {
|
||||
bool any = false;
|
||||
for (const llm::ContentPart& p : m.parts)
|
||||
if (p.tool_result) {
|
||||
Json o = Json::object();
|
||||
o.set("role", Json("tool"));
|
||||
o.set("tool_call_id", Json(p.tool_result->call_id));
|
||||
o.set("content", Json(p.tool_result->content));
|
||||
out.push(o);
|
||||
any = true;
|
||||
}
|
||||
if (!any) bad(vendor, "a tool message needs a tool result part");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
llm::Usage usage_of(const Json& u) {
|
||||
llm::Usage out;
|
||||
if (!u.is_object()) return out;
|
||||
out.input_tokens = int_at(u, "prompt_tokens");
|
||||
out.output_tokens = int_at(u, "completion_tokens");
|
||||
out.total_tokens = int_at(u, "total_tokens");
|
||||
if (const Json* d = obj_at(u, "prompt_tokens_details")) out.cached_tokens = int_at(*d, "cached_tokens");
|
||||
if (const Json* d = obj_at(u, "completion_tokens_details")) out.reasoning_tokens = int_at(*d, "reasoning_tokens");
|
||||
return out;
|
||||
}
|
||||
|
||||
std::string reasoning_of(const Json& holder) {
|
||||
std::string r = str_at(holder, "reasoning_content");
|
||||
return r.empty() ? str_at(holder, "reasoning") : r;
|
||||
}
|
||||
|
||||
Json arguments_of(const std::string& raw) {
|
||||
if (raw.empty()) return Json::object();
|
||||
try {
|
||||
Json j = Json::parse(raw);
|
||||
if (j.is_object()) return j;
|
||||
} catch (const std::exception&) {
|
||||
}
|
||||
Json badj = Json::object();
|
||||
badj.set("_invalid_json", Json(raw));
|
||||
return badj;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
CompatChatModel::CompatChatModel(Provider provider, std::string default_model)
|
||||
: ProviderService(std::move(provider)), default_model_(std::move(default_model)) {}
|
||||
|
||||
std::shared_ptr<CompatChatModel> CompatChatModel::from_env(const std::string& provider_id,
|
||||
const std::string& default_model) {
|
||||
return std::make_shared<CompatChatModel>(Provider::from_env(provider_id), default_model);
|
||||
}
|
||||
|
||||
Json CompatChatModel::to_body(const llm::ChatRequest& r, bool stream) const {
|
||||
const std::string vendor = client_->config().vendor;
|
||||
const ProviderRow& row = provider_.row();
|
||||
provider_.require(Call::kChat);
|
||||
|
||||
std::string model = r.model.empty() ? default_model_ : r.model;
|
||||
if (model.empty()) bad(vendor, "model is required (ids differ per provider)");
|
||||
if (model.size() > CompatCaps::kMaxModelChars) bad(vendor, "model id too long");
|
||||
if (r.messages.empty() && r.system.empty()) bad(vendor, "at least one message is required");
|
||||
if (r.messages.size() > CompatCaps::kMaxMessages) bad(vendor, "too many messages");
|
||||
if (r.tools.size() > CompatCaps::kMaxTools) bad(vendor, "too many tools");
|
||||
if (r.stop.size() > CompatCaps::kMaxStopSequences) bad(vendor, "too many stop sequences");
|
||||
if (r.max_tokens && (*r.max_tokens < 1 || *r.max_tokens > CompatCaps::kMaxTokensCeiling))
|
||||
bad(vendor, "max_tokens out of range");
|
||||
if (r.temperature && (*r.temperature < 0.0 || *r.temperature > 2.0)) bad(vendor, "temperature must be 0..2");
|
||||
if (r.top_p && (*r.top_p <= 0.0 || *r.top_p > 1.0)) bad(vendor, "top_p must be in (0,1]");
|
||||
const bool has_format = r.json_mode || !r.response_schema.is_null();
|
||||
if (r.json_mode && !r.response_schema.is_null()) bad(vendor, "json_mode and response_schema are exclusive");
|
||||
if ((row.quirks & Quirk::kNoToolsWithFormat) && has_format && !r.tools.empty())
|
||||
bad(vendor, std::string(row.name) + " documents that tools and response_format must not be combined");
|
||||
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(model));
|
||||
Json messages = Json::array();
|
||||
if (!r.system.empty()) append_message(vendor, row, messages, llm::Message::system(r.system));
|
||||
for (const llm::Message& m : r.messages) append_message(vendor, row, messages, m);
|
||||
body.set("messages", messages);
|
||||
if (r.max_tokens) body.set("max_tokens", Json(*r.max_tokens));
|
||||
if (r.temperature) body.set("temperature", Json(*r.temperature));
|
||||
if (r.top_p) body.set("top_p", Json(*r.top_p));
|
||||
if (!r.stop.empty()) {
|
||||
Json s = Json::array();
|
||||
for (const std::string& x : r.stop) s.push(Json(x));
|
||||
body.set("stop", s);
|
||||
}
|
||||
if (!r.tools.empty()) {
|
||||
Json tools = Json::array();
|
||||
for (const llm::ToolSpec& t : r.tools) {
|
||||
if (t.name.empty()) bad(vendor, "tool name is required");
|
||||
Json fn = Json::object();
|
||||
fn.set("name", Json(t.name));
|
||||
if (!t.description.empty()) fn.set("description", Json(t.description));
|
||||
fn.set("parameters", t.parameters);
|
||||
Json o = Json::object();
|
||||
o.set("type", Json("function"));
|
||||
o.set("function", fn);
|
||||
tools.push(o);
|
||||
}
|
||||
body.set("tools", tools);
|
||||
}
|
||||
if (!r.tool_choice.empty()) {
|
||||
if (r.tools.empty()) bad(vendor, "tool_choice needs tools");
|
||||
if (r.tool_choice == "auto" || r.tool_choice == "none" || r.tool_choice == "required") {
|
||||
body.set("tool_choice", Json(r.tool_choice));
|
||||
} else {
|
||||
Json fn = Json::object();
|
||||
fn.set("name", Json(r.tool_choice));
|
||||
Json o = Json::object();
|
||||
o.set("type", Json("function"));
|
||||
o.set("function", fn);
|
||||
body.set("tool_choice", o);
|
||||
}
|
||||
}
|
||||
if (r.json_mode) {
|
||||
Json f = Json::object();
|
||||
f.set("type", Json("json_object"));
|
||||
body.set("response_format", f);
|
||||
} else if (!r.response_schema.is_null()) {
|
||||
Json js = Json::object();
|
||||
js.set("name", Json("response"));
|
||||
js.set("strict", Json(true));
|
||||
js.set("schema", r.response_schema);
|
||||
Json f = Json::object();
|
||||
f.set("type", Json("json_schema"));
|
||||
f.set("json_schema", js);
|
||||
body.set("response_format", f);
|
||||
}
|
||||
if (!r.reasoning_effort.empty()) body.set("reasoning_effort", Json(r.reasoning_effort));
|
||||
if (r.extra.is_object()) {
|
||||
for (const auto& [k, v] : r.extra.as_object()) {
|
||||
if (k == "model" || k == "messages" || k == "stream") bad(vendor, "extra may not set '" + k + "'");
|
||||
if ((row.quirks & Quirk::kRejectLogprobs) && (k == "logprobs" || k == "top_logprobs" || k == "logit_bias"))
|
||||
bad(vendor, std::string(row.name) + " rejects " + k + " (documented 400)");
|
||||
if ((row.quirks & Quirk::kSingleChoice) && k == "n" && !(v.is_number() && v.as_int() == 1))
|
||||
bad(vendor, std::string(row.name) + " requires n = 1");
|
||||
if (k == "n" && v.is_number() && (v.as_int() < 1 || v.as_int() > static_cast<long>(CompatCaps::kMaxChoices)))
|
||||
bad(vendor, "n out of range");
|
||||
body.set(k, v);
|
||||
}
|
||||
} else if (!r.extra.is_null()) {
|
||||
bad(vendor, "extra must be an object");
|
||||
}
|
||||
if (stream) body.set("stream", Json(true));
|
||||
return body;
|
||||
}
|
||||
|
||||
llm::ChatResponse CompatChatModel::parse_response(const Json& doc) {
|
||||
llm::ChatResponse out;
|
||||
out.id = str_at(doc, "id");
|
||||
out.model = str_at(doc, "model");
|
||||
out.message.role = llm::Role::kAssistant;
|
||||
out.raw = doc;
|
||||
out.usage = usage_of(doc.is_object() && doc.find("usage") ? *doc.find("usage") : Json());
|
||||
const Json* choices = doc.is_object() ? doc.find("choices") : nullptr;
|
||||
if (!choices || !choices->is_array() || choices->as_array().empty())
|
||||
throw llm::ApiError("compat", 502, "BAD_RESPONSE", "response has no choices");
|
||||
const Json& c = choices->as_array()[0];
|
||||
out.finish_reason = str_at(c, "finish_reason");
|
||||
const Json* msg = obj_at(c, "message");
|
||||
if (!msg) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "choice has no message");
|
||||
std::string reasoning = reasoning_of(*msg);
|
||||
if (!reasoning.empty()) {
|
||||
llm::ContentPart r;
|
||||
r.reasoning = reasoning;
|
||||
out.message.parts.push_back(r);
|
||||
}
|
||||
std::string content = str_at(*msg, "content");
|
||||
const Json* calls = msg->find("tool_calls");
|
||||
const bool has_calls = calls && calls->is_array() && !calls->as_array().empty();
|
||||
if (!content.empty() || !has_calls) out.message.parts.push_back(llm::ContentPart::of_text(content));
|
||||
if (has_calls)
|
||||
for (const Json& call : calls->as_array()) {
|
||||
llm::ToolCall tc;
|
||||
tc.id = str_at(call, "id");
|
||||
if (const Json* fn = obj_at(call, "function")) {
|
||||
tc.name = str_at(*fn, "name");
|
||||
tc.arguments = arguments_of(str_at(*fn, "arguments"));
|
||||
}
|
||||
out.message.parts.push_back(llm::ContentPart::of_call(tc));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
llm::ChatResponse CompatChatModel::generate(const llm::ChatRequest& request) {
|
||||
Json body = to_body(request, false);
|
||||
return parse_response(client_->post(Calls::spec_of(Call::kChat).path, body));
|
||||
}
|
||||
|
||||
void CompatChatModel::stream(const llm::ChatRequest& request,
|
||||
const std::function<bool(const llm::ChatChunk&)>& on_chunk) {
|
||||
Json body = to_body(request, true);
|
||||
std::map<int, std::string> ids, names;
|
||||
std::size_t events = 0;
|
||||
bool finished = false, stopped = false;
|
||||
client_->stream(Calls::spec_of(Call::kChat).path, body, [&](const llm::SseEvent& ev) {
|
||||
if (++events > CompatCaps::kMaxStreamEvents) {
|
||||
stopped = true;
|
||||
return false;
|
||||
}
|
||||
const Json& d = ev.data;
|
||||
if (const Json* err = obj_at(d, "error"))
|
||||
throw llm::ApiError(vendor(), 502, str_at(*err, "type").empty() ? "STREAM_ERROR" : str_at(*err, "type"),
|
||||
str_at(*err, "message"));
|
||||
auto emit = [&](const llm::ChatChunk& c) {
|
||||
if (!on_chunk(c)) {
|
||||
stopped = true;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
const Json* choices = d.is_object() ? d.find("choices") : nullptr;
|
||||
if (choices && choices->is_array() && !choices->as_array().empty()) {
|
||||
const Json& c = choices->as_array()[0];
|
||||
if (const Json* delta = obj_at(c, "delta")) {
|
||||
std::string text = str_at(*delta, "content"), reasoning = reasoning_of(*delta);
|
||||
if (!text.empty() || !reasoning.empty()) {
|
||||
llm::ChatChunk ch;
|
||||
ch.text_delta = text;
|
||||
ch.reasoning_delta = reasoning;
|
||||
if (!emit(ch)) return false;
|
||||
}
|
||||
const Json* calls = delta->find("tool_calls");
|
||||
if (calls && calls->is_array())
|
||||
for (const Json& f : calls->as_array()) {
|
||||
int index = int_at(f, "index");
|
||||
if (!str_at(f, "id").empty()) ids[index] = str_at(f, "id");
|
||||
const Json* fn = obj_at(f, "function");
|
||||
if (fn && !str_at(*fn, "name").empty()) names[index] = str_at(*fn, "name");
|
||||
llm::ChatChunk ch;
|
||||
ch.tool_call_id = ids[index];
|
||||
ch.tool_call_name = names[index];
|
||||
ch.tool_args_delta = fn ? str_at(*fn, "arguments") : "";
|
||||
if (!emit(ch)) return false;
|
||||
}
|
||||
}
|
||||
std::string fin = str_at(c, "finish_reason");
|
||||
if (!fin.empty()) {
|
||||
finished = true;
|
||||
llm::ChatChunk ch;
|
||||
ch.finish_reason = fin;
|
||||
if (const Json* u = obj_at(d, "usage")) ch.usage = usage_of(*u);
|
||||
if (!emit(ch)) return false;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
if (const Json* u = obj_at(d, "usage")) {
|
||||
llm::ChatChunk ch;
|
||||
ch.usage = usage_of(*u);
|
||||
if (!emit(ch)) return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
if (!stopped && !finished)
|
||||
throw llm::ApiError(vendor(), 502, "STREAM_TRUNCATED", "stream ended without a finish_reason");
|
||||
}
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+48
@@ -0,0 +1,48 @@
|
||||
#pragma once
|
||||
|
||||
// CompatChatModel: llm::ChatModel over POST {base}/chat/completions of any
|
||||
// registry row. Mapping notes (each one is a decision):
|
||||
// - ChatRequest.model is required unless a default model is given: model ids
|
||||
// differ per provider and there is no honest module-wide default.
|
||||
// - ChatRequest.system becomes the first system message.
|
||||
// - Reasoning text of an assistant message is never sent back.
|
||||
// - json_mode -> response_format {type: json_object}; response_schema ->
|
||||
// response_format {type: json_schema, strict: true}.
|
||||
// - reasoning_effort is sent as reasoning_effort; ChatRequest.extra is merged
|
||||
// except for keys the module owns (model, messages, stream).
|
||||
// - Row quirks (registry.h Quirk) are rejected before any traffic.
|
||||
// - Reasoning comes back from reasoning_content or reasoning, whichever the
|
||||
// provider sends.
|
||||
// - Tool call arguments that are not valid JSON arrive as
|
||||
// {"_invalid_json": "<raw text>"}.
|
||||
// - Streaming sends no stream_options unless ChatRequest.extra carries them;
|
||||
// usage then appears only when the provider sends it. A stream that ends
|
||||
// without a finish_reason throws 502 STREAM_TRUNCATED.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "llm/chat_types.h"
|
||||
#include "vendors/compat/provider.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
class CompatChatModel : public ProviderService, public llm::ChatModel {
|
||||
public:
|
||||
explicit CompatChatModel(Provider provider, std::string default_model = "");
|
||||
static std::shared_ptr<CompatChatModel> from_env(const std::string& provider_id,
|
||||
const std::string& default_model = "");
|
||||
|
||||
std::string vendor() const override { return client_->config().vendor; }
|
||||
llm::ChatResponse generate(const llm::ChatRequest& request) override;
|
||||
void stream(const llm::ChatRequest& request,
|
||||
const std::function<bool(const llm::ChatChunk&)>& on_chunk) override;
|
||||
|
||||
// The wire body a neutral request maps to (also used by tests and dry runs).
|
||||
clients::body::Json to_body(const llm::ChatRequest& request, bool stream) const;
|
||||
static llm::ChatResponse parse_response(const clients::body::Json& doc);
|
||||
|
||||
private:
|
||||
std::string default_model_;
|
||||
};
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+110
@@ -0,0 +1,110 @@
|
||||
#include "vendors/compat/provider.h"
|
||||
|
||||
#include <cctype>
|
||||
|
||||
#include "clients/transports/http_transport.h"
|
||||
#include "llm/config.h"
|
||||
#include "vendors/compat/caps.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
namespace {
|
||||
|
||||
bool valid_account_id(const std::string& s) {
|
||||
if (s.empty() || s.size() > 64) return false;
|
||||
for (char c : s)
|
||||
if (!(std::isalnum(static_cast<unsigned char>(c)) || c == '-' || c == '_')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool valid_base(const std::string& s) {
|
||||
if (s.empty() || s.size() > CompatCaps::kMaxBaseUrlChars) return false;
|
||||
if (s.rfind("http://", 0) != 0 && s.rfind("https://", 0) != 0) return false;
|
||||
for (char c : s)
|
||||
if (static_cast<unsigned char>(c) <= 0x20 || c == 0x7f || c == '?' || c == '#') return false;
|
||||
return s.find("..") == std::string::npos;
|
||||
}
|
||||
|
||||
std::string replace_account(std::string base, const std::string& account) {
|
||||
const std::string token = "{account_id}";
|
||||
std::size_t at = base.find(token);
|
||||
if (at != std::string::npos) base.replace(at, token.size(), account);
|
||||
return base;
|
||||
}
|
||||
|
||||
// The key a row would use: configured env value; local rows get a
|
||||
// placeholder when their base_env switch is set (servers ignore it).
|
||||
std::string key_for(const ProviderRow& row, const std::string& key_env_value, bool switch_set) {
|
||||
if (!row.local) return key_env_value;
|
||||
if (!switch_set) return "";
|
||||
return key_env_value.empty() ? "local" : key_env_value;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string Provider::resolve_base(const ProviderRow& row, const std::string& base_override,
|
||||
const std::string& account_id) {
|
||||
std::string base = base_override.empty() ? row.base_url : base_override;
|
||||
if (base.find("{account_id}") != std::string::npos) {
|
||||
if (!valid_account_id(account_id))
|
||||
throw llm::ApiError::invalid("compat:" + std::string(row.id), "account id missing or malformed");
|
||||
base = replace_account(base, account_id);
|
||||
}
|
||||
while (base.size() > 8 && base.back() == '/') base.pop_back();
|
||||
if (!valid_base(base))
|
||||
throw llm::ApiError::invalid("compat:" + std::string(row.id), "base URL must be http(s), no spaces, query or fragment");
|
||||
return base;
|
||||
}
|
||||
|
||||
llm::ApiClientConfig Provider::config_for(const ProviderRow& row, const std::string& base_url,
|
||||
const std::string& missing_env) {
|
||||
llm::ApiClientConfig c;
|
||||
c.vendor = std::string("compat:") + row.id;
|
||||
c.base_url = base_url;
|
||||
c.key_env = missing_env.empty() ? row.key_env : missing_env;
|
||||
c.auth = llm::AuthStyle::kBearer;
|
||||
return c;
|
||||
}
|
||||
|
||||
Provider Provider::from_env(const std::string& id) {
|
||||
const ProviderRow& row = Registry::get(id);
|
||||
std::string base_override = row.base_env[0] ? llm::Config::get(row.base_env) : "";
|
||||
std::string account = row.extra_env[0] ? llm::Config::get(row.extra_env) : "";
|
||||
std::string key = key_for(row, llm::Config::get(row.key_env), !base_override.empty());
|
||||
std::string missing;
|
||||
if (row.local && base_override.empty()) missing = row.base_env;
|
||||
else if (row.extra_env[0] && account.empty()) missing = row.extra_env;
|
||||
if (!missing.empty()) key.clear();
|
||||
// An unconfigured row still needs a syntactically valid base URL to exist.
|
||||
std::string base = (row.extra_env[0] && account.empty())
|
||||
? replace_account(row.base_url, "unset")
|
||||
: resolve_base(row, base_override, account);
|
||||
auto client = std::make_shared<llm::ApiClient>(std::make_unique<clients::transports::HttpTransport>(),
|
||||
config_for(row, base, missing), key);
|
||||
return Provider(&row, base, client);
|
||||
}
|
||||
|
||||
Provider Provider::with(const std::string& id, std::unique_ptr<clients::Transport> transport,
|
||||
const std::string& key, const std::string& base_override,
|
||||
const std::string& account_id) {
|
||||
const ProviderRow& row = Registry::get(id);
|
||||
std::string base = resolve_base(row, base_override, account_id);
|
||||
auto client = std::make_shared<llm::ApiClient>(std::move(transport), config_for(row, base, ""), key);
|
||||
return Provider(&row, base, client);
|
||||
}
|
||||
|
||||
std::string Provider::missing_env() const {
|
||||
if (configured()) return "";
|
||||
return client_->config().key_env;
|
||||
}
|
||||
|
||||
bool Provider::supports(Call call) const { return Registry::has(*row_, Calls::spec_of(call).feature); }
|
||||
|
||||
void Provider::require(Call call) const {
|
||||
if (!supports(call))
|
||||
throw llm::ApiError::invalid(client_->config().vendor,
|
||||
std::string(row_->name) + " has no documented " + Calls::spec_of(call).feature_name +
|
||||
" endpoint in the registry");
|
||||
}
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+64
@@ -0,0 +1,64 @@
|
||||
#pragma once
|
||||
|
||||
// Provider: one registry row bound to one ApiClient. The only place a base
|
||||
// URL is resolved (account id substitution, env override) and validated.
|
||||
// Services take a Provider and refuse calls the row does not claim.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "llm/api_client.h"
|
||||
#include "vendors/compat/calls.h"
|
||||
#include "vendors/compat/registry.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
class Provider {
|
||||
public:
|
||||
// Real HTTPS transport. Key and overrides come from llm::Config.
|
||||
static Provider from_env(const std::string& id);
|
||||
// Same row, any transport (tests). base_override and account_id may be empty.
|
||||
static Provider with(const std::string& id, std::unique_ptr<clients::Transport> transport,
|
||||
const std::string& key, const std::string& base_override = "",
|
||||
const std::string& account_id = "");
|
||||
|
||||
// base_url with {account_id} replaced and a trailing slash removed. Throws
|
||||
// ApiError::invalid for a malformed override or account id.
|
||||
static std::string resolve_base(const ProviderRow& row, const std::string& base_override,
|
||||
const std::string& account_id);
|
||||
|
||||
const ProviderRow& row() const { return *row_; }
|
||||
const std::string& base_url() const { return base_url_; }
|
||||
std::shared_ptr<llm::ApiClient> client() const { return client_; }
|
||||
bool configured() const { return client_->configured(); }
|
||||
// Env var a user must set next ("" when configured).
|
||||
std::string missing_env() const;
|
||||
|
||||
// Throws ApiError::invalid when the row does not claim the call's feature.
|
||||
void require(Call call) const;
|
||||
bool supports(Call call) const;
|
||||
|
||||
private:
|
||||
Provider(const ProviderRow* row, std::string base_url, std::shared_ptr<llm::ApiClient> client)
|
||||
: row_(row), base_url_(std::move(base_url)), client_(std::move(client)) {}
|
||||
|
||||
static llm::ApiClientConfig config_for(const ProviderRow& row, const std::string& base_url,
|
||||
const std::string& missing_env);
|
||||
|
||||
const ProviderRow* row_;
|
||||
std::string base_url_;
|
||||
std::shared_ptr<llm::ApiClient> client_;
|
||||
};
|
||||
|
||||
// Base for the per-endpoint services: holds the Provider, checks features.
|
||||
class ProviderService : public llm::Service {
|
||||
public:
|
||||
explicit ProviderService(Provider provider)
|
||||
: llm::Service(provider.client()), provider_(std::move(provider)) {}
|
||||
const Provider& provider() const { return provider_; }
|
||||
|
||||
protected:
|
||||
Provider provider_;
|
||||
};
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+239
@@ -0,0 +1,239 @@
|
||||
#include "vendors/compat/registry.h"
|
||||
|
||||
#include "llm/error.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr unsigned C = Feature::kChat;
|
||||
constexpr unsigned E = Feature::kEmbeddings;
|
||||
constexpr unsigned M = Feature::kModels;
|
||||
constexpr unsigned I = Feature::kImages;
|
||||
constexpr unsigned T = Feature::kTranscribe;
|
||||
constexpr unsigned S = Feature::kSpeech;
|
||||
constexpr Evidence DP = Evidence::kDocsAndProbe;
|
||||
constexpr Evidence PO = Evidence::kProbeOnly;
|
||||
constexpr Evidence DO = Evidence::kDocsOnly;
|
||||
|
||||
// Probe date for every "Probe:" note: 2026-10-25, unauthenticated, no key sent.
|
||||
// Fields: id, name, base_url, key_env, base_env, extra_env, models_path,
|
||||
// features, quirks, evidence, local, doc_url, note.
|
||||
const ProviderRow kRows[] = {
|
||||
{"groq", "Groq", "https://api.groq.com/openai/v1", "GROQ_API_KEY", "", "", "/models",
|
||||
C | M | T, Quirk::kRejectLogprobs | Quirk::kRejectMessageName | Quirk::kSingleChoice, DP, false,
|
||||
"https://console.groq.com/docs/openai",
|
||||
"Docs: logprobs, logit_bias, top_logprobs and messages[].name answer 400; n must be 1; temperature 0 "
|
||||
"becomes 1e-8; transcription formats vtt and srt unsupported. Probe: 401 invalid_api_key on /models and chat."},
|
||||
{"together", "Together AI", "https://api.together.ai/v1", "TOGETHER_API_KEY", "", "", "/models",
|
||||
C | E | M | I | T | S, Quirk::kNone, DP, false,
|
||||
"https://docs.together.ai/docs/openai-api-compatibility",
|
||||
"Docs: logit_bias unsupported on most models; service_tier, store, metadata and prediction are accepted "
|
||||
"and ignored. Probe: 401 on /models and chat; legacy host api.together.xyz answers 401 too."},
|
||||
{"fireworks", "Fireworks AI", "https://api.fireworks.ai/inference/v1", "FIREWORKS_API_KEY", "", "", "/models",
|
||||
C | E | M, Quirk::kNone, DP, false,
|
||||
"https://docs.fireworks.ai/tools-sdks/openai-compatibility",
|
||||
"Probe: /models 401 with a docs link; unknown model answers 404 NOT_FOUND; empty embeddings body answers 500."},
|
||||
{"cerebras", "Cerebras", "https://api.cerebras.ai/v1", "CEREBRAS_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNoToolsWithFormat, DP, false,
|
||||
"https://inference-docs.cerebras.ai/resources/openai",
|
||||
"Docs: tools and response_format must not be combined (gpt-oss-120b rejects it); system and developer roles "
|
||||
"both act at developer level. Probe: /models 403 {detail}, chat 401 wrong_api_key, embeddings path 404."},
|
||||
{"sambanova", "SambaNova", "https://api.sambanova.ai/v1", "SAMBANOVA_API_KEY", "", "", "/models",
|
||||
C | E | M, Quirk::kNone, PO, false,
|
||||
"https://docs.sambanova.ai/docs/en/features/openai-compatibility",
|
||||
"Docs page answers 403 to a plain fetch. Probe: /models 200 public; chat 401 (also accepts x-api-key); "
|
||||
"embeddings answers 400 'must provide a model' before auth."},
|
||||
{"deepinfra", "DeepInfra", "https://api.deepinfra.com/v1/openai", "DEEPINFRA_API_KEY", "", "", "/models",
|
||||
C | E | M | I | T, Quirk::kNone, DP, false,
|
||||
"https://docs.deepinfra.com/chat/overview",
|
||||
"Probe: /models 200 public; chat 401 missing API key; images and audio paths answer 401 with a {detail} body."},
|
||||
{"novita", "Novita AI", "https://api.novita.ai/openai/v1", "NOVITA_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, DP, false,
|
||||
"https://novita.ai/docs/guides/llm-api",
|
||||
"Probe: /models 200 public; bad key answers 403 INVALID_API_KEY (not 401); legacy base /v3/openai answers the same."},
|
||||
{"nvidia", "NVIDIA NIM (build.nvidia.com)", "https://integrate.api.nvidia.com/v1", "NVIDIA_API_KEY", "", "", "/models",
|
||||
C | E | M, Quirk::kNone, DP, false,
|
||||
"https://docs.api.nvidia.com/nim/reference/llm-apis",
|
||||
"Probe: /models 200 public; embeddings answers 400 'model field is required'; retired models answer 410 Gone."},
|
||||
{"cloudflare", "Cloudflare Workers AI", "https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1",
|
||||
"CLOUDFLARE_API_TOKEN", "", "CLOUDFLARE_ACCOUNT_ID", "", C | E, Quirk::kNone, DO, false,
|
||||
"https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/",
|
||||
"Docs list /chat/completions, /embeddings and /responses under the account path. Probe: GET /models answers "
|
||||
"405 (no list endpoint here); not probeable further without an account id."},
|
||||
{"moonshot", "Moonshot (Kimi)", "https://api.moonshot.ai/v1", "MOONSHOT_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, DP, false,
|
||||
"https://platform.kimi.ai/docs/api/chat",
|
||||
"Probe: 401 incorrect_api_key_error on /models and chat; the China host api.moonshot.cn answers the same "
|
||||
"(separate keys)."},
|
||||
{"dashscope", "Alibaba DashScope (Qwen, intl)", "https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
|
||||
"DASHSCOPE_API_KEY", "", "", "", C | E, Quirk::kNone, DP, false,
|
||||
"https://www.alibabacloud.com/help/en/model-studio/compatibility-of-openai-with-dashscope",
|
||||
"Probe: 401 on /models and chat; embeddings answers 400 'you must provide a model parameter'; China host "
|
||||
"dashscope.aliyuncs.com/compatible-mode/v1 answers 401 too."},
|
||||
{"zhipu", "Zhipu (GLM, China)", "https://open.bigmodel.cn/api/paas/v4", "ZHIPU_API_KEY", "", "", "",
|
||||
C | E | I, Quirk::kNone, DP, false,
|
||||
"https://docs.bigmodel.cn/cn/guide/develop/openai/introduction",
|
||||
"Docs name the base URL; the path is /api/paas/v4, not /v1. Probe: 401 code 1001 (missing Authorization) "
|
||||
"on chat, embeddings and images."},
|
||||
{"zai", "Z.ai (GLM, international)", "https://api.z.ai/api/paas/v4", "ZAI_API_KEY", "", "", "",
|
||||
C | E | I, Quirk::kNone, DP, false,
|
||||
"https://docs.z.ai/api-reference/introduction",
|
||||
"International host of the Zhipu API, same /api/paas/v4 path. Probe: 401 code 1001."},
|
||||
{"minimax", "MiniMax", "https://api.minimax.io/v1", "MINIMAX_API_KEY", "", "", "",
|
||||
C, Quirk::kNone, PO, false,
|
||||
"https://platform.minimax.io/docs/api-reference/text-openai-api",
|
||||
"Docs text did not name the URL. Probe: 401 authorized_error on /models and chat; embeddings answers HTTP 200 "
|
||||
"with a base_resp status_code 1004 envelope (not OpenAI shaped); China host api.minimaxi.com answers 401."},
|
||||
{"qianfan", "Baidu Qianfan", "https://qianfan.baidubce.com/v2", "QIANFAN_API_KEY", "", "", "",
|
||||
C | E, Quirk::kNone, PO, false,
|
||||
"https://cloud.baidu.com/doc/qianfan-api/s/3m9b5lqft",
|
||||
"Docs page is rendered client side. Probe: chat and embeddings 401 invalid_iam_token; GET /models 403 "
|
||||
"AccessDenied (different envelope)."},
|
||||
{"ark", "Volcengine Ark (Doubao)", "https://ark.cn-beijing.volces.com/api/v3", "ARK_API_KEY", "", "", "",
|
||||
C, Quirk::kNone, PO, false,
|
||||
"https://www.volcengine.com/docs/82379/1298454",
|
||||
"Docs page is rendered client side. Probe: 401 AuthenticationError on /models, chat and embeddings; path is /api/v3."},
|
||||
{"siliconflow", "SiliconFlow", "https://api.siliconflow.com/v1", "SILICONFLOW_API_KEY", "", "", "/models",
|
||||
C | E | M, Quirk::kNone, DP, false,
|
||||
"https://docs.siliconflow.com/en/api-reference/chat-completions/chat-completions",
|
||||
"Probe: 401 code 30014 with a {code,data,message} envelope on every path; China host api.siliconflow.cn identical."},
|
||||
{"nebius", "Nebius Token Factory", "https://api.studio.nebius.com/v1", "NEBIUS_API_KEY", "", "", "/models",
|
||||
C | E | M, Quirk::kNone, DP, false,
|
||||
"https://docs.tokenfactory.nebius.com/api-reference/introduction",
|
||||
"Probe: 401 unauthenticated on /models, chat and embeddings; images/generations 404 on this host; "
|
||||
"api.tokenfactory.nebius.com/v1 answers the same."},
|
||||
{"scaleway", "Scaleway Generative APIs", "https://api.scaleway.ai/v1", "SCALEWAY_API_KEY", "", "", "/models",
|
||||
C | E | M, Quirk::kNone, DP, false,
|
||||
"https://www.scaleway.com/en/docs/generative-apis/api-cli/using-chat-api/",
|
||||
"Probe: 401 UNAUTHORIZED (missing 'Authorization' header) on /models, chat and embeddings."},
|
||||
{"upstage", "Upstage (Solar)", "https://api.upstage.ai/v1", "UPSTAGE_API_KEY", "", "", "",
|
||||
C | E, Quirk::kNone, DP, false,
|
||||
"https://console.upstage.ai/docs/getting-started",
|
||||
"Probe: 401 invalid API key on /models, chat and embeddings."},
|
||||
{"inception", "Inception (Mercury)", "https://api.inceptionlabs.ai/v1", "INCEPTION_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, DP, false,
|
||||
"https://docs.inceptionlabs.ai/get-started/get-started",
|
||||
"Probe: /models 200 public; chat answers 400 listing the valid model names before auth."},
|
||||
{"stepfun", "StepFun", "https://api.stepfun.com/v1", "STEPFUN_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, DP, false,
|
||||
"https://platform.stepfun.com/docs/api-reference/chat/chat-completion-create",
|
||||
"Probe: 401 invalid_api_key on /models and chat."},
|
||||
{"hunyuan", "Tencent Hunyuan", "https://api.hunyuan.cloud.tencent.com/v1", "HUNYUAN_API_KEY", "", "", "",
|
||||
C, Quirk::kNone, PO, false,
|
||||
"https://cloud.tencent.com/document/product/1729/111007",
|
||||
"Docs page did not name the URL in readable text. Probe: 401 'Incorrect API key provided' on /models and chat."},
|
||||
{"venice", "Venice AI", "https://api.venice.ai/api/v1", "VENICE_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, DP, false,
|
||||
"https://docs.venice.ai/api-reference/endpoint/chat/completions",
|
||||
"Probe: /models 200 public; unauthenticated chat answers 402 with an x402 payment envelope (not 401)."},
|
||||
{"vercel", "Vercel AI Gateway", "https://ai-gateway.vercel.sh/v1", "AI_GATEWAY_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, DP, false,
|
||||
"https://vercel.com/docs/ai-gateway/openai-compat",
|
||||
"Probe: /models 200 public; chat 401 authentication_error 'Missing Authorization header'."},
|
||||
{"friendli", "FriendliAI Serverless", "https://api.friendli.ai/serverless/v1", "FRIENDLI_TOKEN", "", "", "/models",
|
||||
C | M, Quirk::kNone, DP, false,
|
||||
"https://friendli.ai/docs/openapi/serverless/chat-completions",
|
||||
"Probe: /models 200 public; chat 401 {detail}."},
|
||||
{"gemini", "Google Gemini (OpenAI compatibility)", "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
"GEMINI_API_KEY", "", "", "", C, Quirk::kNone, DP, false,
|
||||
"https://ai.google.dev/gemini-api/docs/openai",
|
||||
"Probe: chat answers 400 'Missing or invalid Authorization header'; unauthenticated GET /models and "
|
||||
"embeddings answer 404, so only chat is claimed."},
|
||||
{"baseten", "Baseten Model APIs", "https://inference.baseten.co/v1", "BASETEN_API_KEY", "", "", "",
|
||||
C, Quirk::kNone, DP, false,
|
||||
"https://docs.baseten.co/development/model-apis/overview",
|
||||
"Probe: /models 401 'No Authorization header provided'; chat 403 'please check the api-key you provided'."},
|
||||
{"featherless", "Featherless AI", "https://api.featherless.ai/v1", "FEATHERLESS_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, PO, false,
|
||||
"https://featherless.ai/docs/overview",
|
||||
"Docs text did not name the URL. Probe: /models 200 public; chat 401 'must be signed in'."},
|
||||
{"ovh", "OVHcloud AI Endpoints", "https://oai.endpoints.kepler.ai.cloud.ovh.net/v1", "OVH_AI_ENDPOINTS_ACCESS_TOKEN",
|
||||
"", "", "/models", C | M, Quirk::kNone, PO, false,
|
||||
"https://help.ovhcloud.com/csm/en-gb-public-cloud-ai-endpoints-getting-started",
|
||||
"Docs text did not name the URL. Probe: /models 200 public; chat on an unknown model answers 404 model_not_found."},
|
||||
{"parasail", "Parasail", "https://api.parasail.io/v1", "PARASAIL_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, PO, false,
|
||||
"https://docs.parasail.io/parasail-docs/serverless/serverless",
|
||||
"Docs URL answered 404 at probe time. Probe: 401 'No bearer token provided' on /models and chat."},
|
||||
{"crusoe", "Crusoe Managed Inference", "https://managed-inference-api-proxy.crusoecloud.com/v1", "CRUSOE_API_KEY",
|
||||
"", "", "/models", C | M, Quirk::kNone, PO, false,
|
||||
"https://docs.crusoe.ai/managed-inference",
|
||||
"Docs URL answered 404 at probe time. Probe: 401 {errors:[Authentication failed]} on /models and chat."},
|
||||
{"xiaomi", "Xiaomi MiMo", "https://api.xiaomimimo.com/v1", "XIAOMI_API_KEY", "", "", "",
|
||||
C, Quirk::kNone, PO, false,
|
||||
"https://platform.xiaomimimo.com/",
|
||||
"Docs text did not name the URL. Probe: 401 'Invalid API Key' on /models and chat."},
|
||||
{"chutes", "Chutes", "https://llm.chutes.ai/v1", "CHUTES_API_KEY", "", "", "/models",
|
||||
C | M, Quirk::kNone, PO, false,
|
||||
"https://docs.chutes.ai/",
|
||||
"Docs text did not name the URL. Probe: /models 200 public; chat answered an nginx 429 HTML page."},
|
||||
// Self-hosted servers: the key is optional, the row is configured when its base_env is set.
|
||||
{"ollama", "Ollama", "http://localhost:11434/v1", "OLLAMA_API_KEY", "OLLAMA_BASE_URL", "", "/models",
|
||||
C | E | M, Quirk::kNone, DO, true,
|
||||
"https://raw.githubusercontent.com/ollama/ollama/main/docs/api/openai-compatibility.mdx",
|
||||
"Docs name http://localhost:11434/v1. The key is required by OpenAI clients but ignored by the server; "
|
||||
"the module sends the placeholder 'local' when none is set."},
|
||||
{"lmstudio", "LM Studio", "http://localhost:1234/v1", "LMSTUDIO_API_KEY", "LMSTUDIO_BASE_URL", "", "/models",
|
||||
C | E | M, Quirk::kNone, DO, true,
|
||||
"https://lmstudio.ai/docs/developer/openai-compat",
|
||||
"Docs name http://localhost:1234/v1. Key ignored unless configured in the app."},
|
||||
{"vllm", "vLLM server", "http://localhost:8000/v1", "VLLM_API_KEY", "VLLM_BASE_URL", "", "/models",
|
||||
C | E | M | T, Quirk::kNone, DO, true,
|
||||
"https://docs.vllm.ai/en/latest/serving/online_serving/",
|
||||
"Docs list /v1/chat/completions, /v1/embeddings and /v1/audio/transcriptions; key only when the server "
|
||||
"was started with --api-key."},
|
||||
{"llamacpp", "llama.cpp server", "http://localhost:8080/v1", "LLAMACPP_API_KEY", "LLAMACPP_BASE_URL", "", "/models",
|
||||
C | E | M, Quirk::kNone, DO, true,
|
||||
"https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md",
|
||||
"README lists /v1/chat/completions; key only when started with --api-key. Same default port as LocalAI."},
|
||||
{"localai", "LocalAI", "http://localhost:8080/v1", "LOCALAI_API_KEY", "LOCALAI_BASE_URL", "", "/models",
|
||||
C | E | M, Quirk::kNone, DO, true,
|
||||
"https://localai.io/docs/overview/",
|
||||
"Docs name localhost:8080; endpoint list beyond chat, embeddings and models was not read."},
|
||||
};
|
||||
|
||||
constexpr std::size_t kRowCount = sizeof(kRows) / sizeof(kRows[0]);
|
||||
|
||||
} // namespace
|
||||
|
||||
std::size_t Registry::count() { return kRowCount; }
|
||||
|
||||
const ProviderRow& Registry::at(std::size_t index) {
|
||||
if (index >= kRowCount) throw llm::ApiError::invalid("compat", "registry index out of range");
|
||||
return kRows[index];
|
||||
}
|
||||
|
||||
const ProviderRow* Registry::find(const std::string& id) {
|
||||
for (const ProviderRow& row : kRows)
|
||||
if (id == row.id) return &row;
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
const ProviderRow& Registry::get(const std::string& id) {
|
||||
const ProviderRow* row = find(id);
|
||||
if (!row) throw llm::ApiError::invalid("compat", "unknown provider '" + id.substr(0, 64) + "'");
|
||||
return *row;
|
||||
}
|
||||
|
||||
const char* Registry::evidence_name(Evidence e) {
|
||||
switch (e) {
|
||||
case Evidence::kDocsAndProbe: return "docs+probe";
|
||||
case Evidence::kProbeOnly: return "probe-only";
|
||||
case Evidence::kDocsOnly: return "docs-only";
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
std::string Registry::feature_names(const ProviderRow& row) {
|
||||
struct Name { unsigned bit; const char* name; };
|
||||
static const Name names[] = {{Feature::kChat, "chat"}, {Feature::kEmbeddings, "embeddings"},
|
||||
{Feature::kModels, "models"}, {Feature::kImages, "images"},
|
||||
{Feature::kTranscribe, "transcribe"}, {Feature::kSpeech, "speech"}};
|
||||
std::string out;
|
||||
for (const Name& n : names)
|
||||
if (row.features & n.bit) out += (out.empty() ? "" : ",") + std::string(n.name);
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+72
@@ -0,0 +1,72 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE provider table of this module. Adding a provider is one row in
|
||||
// registry.cpp (plus its line in SPEC.md section 4, which tools/check_spec.py
|
||||
// enforces). Everything else (clients, routes, demo, tests) iterates this
|
||||
// table.
|
||||
|
||||
#include <cstddef>
|
||||
#include <string>
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
// Endpoints a row claims. Claims come from the provider's own docs or a
|
||||
// probe; SPEC.md says which. A call outside the mask is refused before any
|
||||
// network traffic.
|
||||
struct Feature {
|
||||
static constexpr unsigned kChat = 1u << 0;
|
||||
static constexpr unsigned kEmbeddings = 1u << 1;
|
||||
static constexpr unsigned kModels = 1u << 2;
|
||||
static constexpr unsigned kImages = 1u << 3;
|
||||
static constexpr unsigned kTranscribe = 1u << 4;
|
||||
static constexpr unsigned kSpeech = 1u << 5;
|
||||
};
|
||||
|
||||
// Documented behavior that differs from OpenAI and that the module enforces
|
||||
// loudly (reject) instead of letting the provider answer 400.
|
||||
struct Quirk {
|
||||
static constexpr unsigned kNone = 0;
|
||||
static constexpr unsigned kRejectMessageName = 1u << 0; // messages[].name answers 400
|
||||
static constexpr unsigned kRejectLogprobs = 1u << 1; // logprobs, top_logprobs, logit_bias answer 400
|
||||
static constexpr unsigned kSingleChoice = 1u << 2; // n must be 1
|
||||
static constexpr unsigned kNoToolsWithFormat = 1u << 3; // tools + response_format not combinable
|
||||
};
|
||||
|
||||
// How the row was verified (see SPEC.md section 5).
|
||||
enum class Evidence {
|
||||
kDocsAndProbe, // docs text names the base URL; unauthenticated probe answered like an API
|
||||
kProbeOnly, // docs text did not name it (or not readable); probe answered like an API
|
||||
kDocsOnly, // docs name it; not probeable (local server, or needs an account id)
|
||||
};
|
||||
|
||||
struct ProviderRow {
|
||||
const char* id; // slug, [a-z0-9]+, used in routes and ids
|
||||
const char* name; // display name
|
||||
const char* base_url; // no trailing slash; "{account_id}" is replaced from extra_env
|
||||
const char* key_env; // credential env var (via llm::Config); local rows: optional key
|
||||
const char* base_env; // env var overriding base_url; for local rows also the "configured" switch
|
||||
const char* extra_env; // second required env var (account id) or ""
|
||||
const char* models_path; // "" when the row has no list endpoint
|
||||
unsigned features; // Feature mask
|
||||
unsigned quirks; // Quirk mask
|
||||
Evidence evidence;
|
||||
bool local; // self-hosted server, no vendor credential
|
||||
const char* doc_url; // documentation page cited for the base URL
|
||||
const char* note; // documented quirks and probe results, one line
|
||||
};
|
||||
|
||||
class Registry {
|
||||
public:
|
||||
static std::size_t count();
|
||||
static const ProviderRow& at(std::size_t index);
|
||||
// nullptr when unknown.
|
||||
static const ProviderRow* find(const std::string& id);
|
||||
// Throws llm::ApiError::invalid for an unknown id.
|
||||
static const ProviderRow& get(const std::string& id);
|
||||
static bool has(const ProviderRow& row, unsigned feature) { return (row.features & feature) != 0; }
|
||||
static const char* evidence_name(Evidence e);
|
||||
// Comma list of the feature names a row claims ("chat,embeddings,models").
|
||||
static std::string feature_names(const ProviderRow& row);
|
||||
};
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+199
@@ -0,0 +1,199 @@
|
||||
#include "vendors/compat/services.h"
|
||||
|
||||
#include "clients/body/form.h"
|
||||
#include "vendors/compat/caps.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
using Json = clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
[[noreturn]] void bad(const Provider& p, const std::string& why) {
|
||||
throw llm::ApiError::invalid(p.client()->config().vendor, why);
|
||||
}
|
||||
|
||||
void check_model(const Provider& p, const std::string& model) {
|
||||
if (model.empty()) bad(p, "model is required (ids differ per provider)");
|
||||
if (model.size() > CompatCaps::kMaxModelChars) bad(p, "model id too long");
|
||||
}
|
||||
|
||||
std::string str_at(const Json& j, const char* key) {
|
||||
const Json* v = j.is_object() ? j.find(key) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : "";
|
||||
}
|
||||
|
||||
int int_at(const Json& j, const char* key) {
|
||||
const Json* v = j.is_object() ? j.find(key) : nullptr;
|
||||
return v && v->is_number() ? static_cast<int>(v->as_int()) : 0;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
bool ModelList::contains(const std::string& id) const {
|
||||
for (const ModelEntry& m : models)
|
||||
if (m.id == id) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
ModelList CompatModels::parse(const Json& doc) {
|
||||
const Json* arr = &doc;
|
||||
if (doc.is_object()) arr = doc.find("data");
|
||||
ModelList out;
|
||||
if (!arr || !arr->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "model list has no data array");
|
||||
for (const Json& m : arr->as_array()) {
|
||||
if (out.models.size() >= CompatCaps::kMaxModelsReturned) {
|
||||
out.truncated = true;
|
||||
break;
|
||||
}
|
||||
ModelEntry e;
|
||||
e.id = str_at(m, "id");
|
||||
if (e.id.empty()) e.id = str_at(m, "name");
|
||||
if (e.id.empty()) continue;
|
||||
e.owned_by = str_at(m, "owned_by");
|
||||
out.models.push_back(e);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
ModelList CompatModels::list() {
|
||||
provider_.require(Call::kModels);
|
||||
if (std::string(provider_.row().models_path).empty())
|
||||
bad(provider_, std::string(provider_.row().name) + " has no list endpoint under its base URL");
|
||||
return parse(client_->get(provider_.row().models_path));
|
||||
}
|
||||
|
||||
Json CompatEmbeddings::to_body(const std::string& model, const std::vector<std::string>& inputs,
|
||||
int dimensions) const {
|
||||
provider_.require(Call::kEmbeddings);
|
||||
check_model(provider_, model);
|
||||
if (inputs.empty()) bad(provider_, "at least one input is required");
|
||||
if (inputs.size() > CompatCaps::kMaxEmbeddingInputs) bad(provider_, "too many inputs");
|
||||
if (dimensions < 0 || dimensions > 65536) bad(provider_, "dimensions out of range");
|
||||
Json arr = Json::array();
|
||||
for (const std::string& s : inputs) {
|
||||
if (s.empty()) bad(provider_, "input must not be empty");
|
||||
if (s.size() > CompatCaps::kMaxEmbeddingInputChars) bad(provider_, "input too long");
|
||||
arr.push(Json(s));
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(model));
|
||||
body.set("input", arr);
|
||||
if (dimensions > 0) body.set("dimensions", Json(dimensions));
|
||||
return body;
|
||||
}
|
||||
|
||||
EmbeddingResult CompatEmbeddings::parse(const Json& doc) {
|
||||
EmbeddingResult out;
|
||||
out.model = str_at(doc, "model");
|
||||
const Json* data = doc.is_object() ? doc.find("data") : nullptr;
|
||||
if (!data || !data->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "embeddings response has no data");
|
||||
for (const Json& d : data->as_array()) {
|
||||
Embedding e;
|
||||
const Json* v = d.is_object() ? d.find("embedding") : nullptr;
|
||||
if (!v || !v->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "embedding is not an array");
|
||||
for (const Json& x : v->as_array()) e.vector.push_back(x.is_number() ? x.as_number() : 0.0);
|
||||
out.items.push_back(std::move(e));
|
||||
}
|
||||
if (const Json* u = doc.find("usage"); u && u->is_object()) {
|
||||
out.prompt_tokens = int_at(*u, "prompt_tokens");
|
||||
out.total_tokens = int_at(*u, "total_tokens");
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
EmbeddingResult CompatEmbeddings::embed(const std::string& model, const std::vector<std::string>& inputs,
|
||||
int dimensions) {
|
||||
Json body = to_body(model, inputs, dimensions);
|
||||
return parse(client_->post(Calls::spec_of(Call::kEmbeddings).path, body));
|
||||
}
|
||||
|
||||
Json CompatImages::to_body(const ImageRequest& r) const {
|
||||
provider_.require(Call::kImages);
|
||||
check_model(provider_, r.model);
|
||||
if (r.prompt.empty() || r.prompt.size() > CompatCaps::kMaxPromptChars) bad(provider_, "prompt must be 1..32768 chars");
|
||||
if (r.n < 1 || r.n > CompatCaps::kMaxImages) bad(provider_, "n must be 1..10");
|
||||
if (!r.response_format.empty() && r.response_format != "url" && r.response_format != "b64_json")
|
||||
bad(provider_, "response_format must be url or b64_json");
|
||||
if (!r.size.empty()) {
|
||||
std::size_t x = r.size.find('x');
|
||||
bool ok = x != std::string::npos && x > 0 && x + 1 < r.size.size() && r.size.size() <= 16;
|
||||
for (std::size_t i = 0; ok && i < r.size.size(); ++i)
|
||||
ok = i == x || (r.size[i] >= '0' && r.size[i] <= '9');
|
||||
if (!ok) bad(provider_, "size must look like 1024x1024");
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(r.model));
|
||||
body.set("prompt", Json(r.prompt));
|
||||
body.set("n", Json(r.n));
|
||||
if (!r.size.empty()) body.set("size", Json(r.size));
|
||||
if (!r.response_format.empty()) body.set("response_format", Json(r.response_format));
|
||||
return body;
|
||||
}
|
||||
|
||||
std::vector<ImageItem> CompatImages::parse(const Json& doc) {
|
||||
const Json* data = doc.is_object() ? doc.find("data") : nullptr;
|
||||
if (!data || !data->is_array()) throw llm::ApiError("compat", 502, "BAD_RESPONSE", "images response has no data");
|
||||
std::vector<ImageItem> out;
|
||||
for (const Json& d : data->as_array()) {
|
||||
if (out.size() >= static_cast<std::size_t>(CompatCaps::kMaxImages)) break;
|
||||
ImageItem i;
|
||||
i.url = str_at(d, "url");
|
||||
i.b64_json = str_at(d, "b64_json");
|
||||
i.revised_prompt = str_at(d, "revised_prompt");
|
||||
out.push_back(std::move(i));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<ImageItem> CompatImages::generate(const ImageRequest& request) {
|
||||
Json body = to_body(request);
|
||||
return parse(client_->post(Calls::spec_of(Call::kImages).path, body));
|
||||
}
|
||||
|
||||
std::string CompatAudio::transcribe(const TranscribeRequest& r) {
|
||||
provider_.require(Call::kTranscribe);
|
||||
check_model(provider_, r.model);
|
||||
if (r.audio.empty()) bad(provider_, "audio is empty");
|
||||
if (r.audio.size() > CompatCaps::kMaxAudioBytes) bad(provider_, "audio over the byte cap");
|
||||
if (r.filename.empty() || r.filename.size() > CompatCaps::kMaxFilenameChars) bad(provider_, "filename must be 1..256 chars");
|
||||
if (r.filename.find_first_of("\"\r\n/\\") != std::string::npos) bad(provider_, "filename has forbidden characters");
|
||||
clients::body::MultipartBody form;
|
||||
form.add_file("file", r.filename, r.mime, r.audio);
|
||||
form.add_text("model", r.model);
|
||||
if (!r.language.empty()) form.add_text("language", r.language);
|
||||
if (!r.prompt.empty()) form.add_text("prompt", r.prompt);
|
||||
form.add_text("response_format", "json");
|
||||
Json doc = client_->post_multipart(Calls::spec_of(Call::kTranscribe).path, form);
|
||||
return str_at(doc, "text");
|
||||
}
|
||||
|
||||
Json CompatAudio::speech_body(const SpeechRequest& r) const {
|
||||
provider_.require(Call::kSpeech);
|
||||
check_model(provider_, r.model);
|
||||
if (r.input.empty() || r.input.size() > CompatCaps::kMaxSpeechChars) bad(provider_, "input must be 1..4096 chars");
|
||||
if (r.voice.empty()) bad(provider_, "voice is required (names differ per provider)");
|
||||
static const char* formats[] = {"mp3", "opus", "aac", "flac", "wav", "pcm"};
|
||||
if (!r.format.empty()) {
|
||||
bool ok = false;
|
||||
for (const char* f : formats) ok = ok || r.format == f;
|
||||
if (!ok) bad(provider_, "format must be mp3, opus, aac, flac, wav or pcm");
|
||||
}
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(r.model));
|
||||
body.set("input", Json(r.input));
|
||||
body.set("voice", Json(r.voice));
|
||||
if (!r.format.empty()) body.set("response_format", Json(r.format));
|
||||
return body;
|
||||
}
|
||||
|
||||
std::string CompatAudio::speak(const SpeechRequest& r, std::string& content_type) {
|
||||
Json body = speech_body(r);
|
||||
std::string text = body.dump();
|
||||
llm::RawResponse res = client_->request_raw(clients::Method::kPost, Calls::spec_of(Call::kSpeech).path, &text,
|
||||
"application/json");
|
||||
content_type = res.content_type;
|
||||
return res.body;
|
||||
}
|
||||
|
||||
} // namespace molo::compat
|
||||
Vendored
+104
@@ -0,0 +1,104 @@
|
||||
#pragma once
|
||||
|
||||
// The non-chat endpoints of a registry row. Each service refuses a call the
|
||||
// row does not claim (Provider::require) before any network traffic, and
|
||||
// validates every input against CompatCaps.
|
||||
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "vendors/compat/provider.h"
|
||||
|
||||
namespace molo::compat {
|
||||
|
||||
struct ModelEntry {
|
||||
std::string id;
|
||||
std::string owned_by;
|
||||
};
|
||||
|
||||
struct ModelList {
|
||||
std::vector<ModelEntry> models;
|
||||
bool truncated = false; // more entries than CompatCaps::kMaxModelsReturned
|
||||
bool contains(const std::string& id) const;
|
||||
};
|
||||
|
||||
// GET /models. Accepts {"data":[...]} and a bare array (some providers return one).
|
||||
class CompatModels : public ProviderService {
|
||||
public:
|
||||
using ProviderService::ProviderService;
|
||||
ModelList list();
|
||||
static ModelList parse(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct Embedding {
|
||||
std::vector<double> vector;
|
||||
};
|
||||
|
||||
struct EmbeddingResult {
|
||||
std::string model;
|
||||
std::vector<Embedding> items;
|
||||
int prompt_tokens = 0;
|
||||
int total_tokens = 0;
|
||||
};
|
||||
|
||||
class CompatEmbeddings : public ProviderService {
|
||||
public:
|
||||
using ProviderService::ProviderService;
|
||||
// dimensions: 0 = provider default.
|
||||
EmbeddingResult embed(const std::string& model, const std::vector<std::string>& inputs, int dimensions = 0);
|
||||
clients::body::Json to_body(const std::string& model, const std::vector<std::string>& inputs,
|
||||
int dimensions) const;
|
||||
static EmbeddingResult parse(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct ImageItem {
|
||||
std::string url;
|
||||
std::string b64_json;
|
||||
std::string revised_prompt;
|
||||
};
|
||||
|
||||
struct ImageRequest {
|
||||
std::string model;
|
||||
std::string prompt;
|
||||
int n = 1;
|
||||
std::string size; // "1024x1024"; empty = provider default
|
||||
std::string response_format; // "", "url" or "b64_json"
|
||||
};
|
||||
|
||||
class CompatImages : public ProviderService {
|
||||
public:
|
||||
using ProviderService::ProviderService;
|
||||
// Paid call: the caller decides (the demo keeps it a dry run).
|
||||
std::vector<ImageItem> generate(const ImageRequest& request);
|
||||
clients::body::Json to_body(const ImageRequest& request) const;
|
||||
static std::vector<ImageItem> parse(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct TranscribeRequest {
|
||||
std::string model;
|
||||
std::string filename;
|
||||
std::string mime = "application/octet-stream";
|
||||
std::string audio; // raw bytes
|
||||
std::string language;
|
||||
std::string prompt;
|
||||
};
|
||||
|
||||
struct SpeechRequest {
|
||||
std::string model;
|
||||
std::string input;
|
||||
std::string voice;
|
||||
std::string format; // "", mp3, opus, aac, flac, wav, pcm
|
||||
};
|
||||
|
||||
class CompatAudio : public ProviderService {
|
||||
public:
|
||||
using ProviderService::ProviderService;
|
||||
std::string transcribe(const TranscribeRequest& request); // returns the text
|
||||
// Audio bytes; the content type is returned through `content_type`.
|
||||
std::string speak(const SpeechRequest& request, std::string& content_type);
|
||||
clients::body::Json speech_body(const SpeechRequest& request) const;
|
||||
};
|
||||
|
||||
} // namespace molo::compat
|
||||
+239
@@ -0,0 +1,239 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check src/vendors/compat against OpenAI's OpenAPI document and its own SPEC.
|
||||
|
||||
Every compat call is an OpenAI operation (providers differ only in base URL,
|
||||
auth and quirks). Fails (exit 1) when
|
||||
|
||||
call a row of calls.cpp has a wrong method, path, operation id, request
|
||||
schema, request body field or response field (fields are resolved
|
||||
through $ref, allOf, oneOf and anyOf);
|
||||
coverage an operation of the spec in the families this module touches
|
||||
(chat completions, embeddings, models, images, audio) is neither
|
||||
in the table nor in EXCLUDED;
|
||||
registry the provider ids / base URLs / credential names of registry.cpp and
|
||||
the table of SPEC.md section 4 differ in either direction.
|
||||
|
||||
The registry check needs no network: pass --local to run only it.
|
||||
Python 3 stdlib only.
|
||||
|
||||
python3 -I src/vendors/compat/tools/check_spec.py [--local] [path/to/openapi.json]
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ROOT = os.path.join(HERE, "..")
|
||||
SPEC_SHA = "8243cc8fa6f04430fb1f1a6a2cb926fbeb4f7f26" # same pin as src/openai/core
|
||||
URL = "https://raw.githubusercontent.com/openai/openai-openapi/%s/openapi.json" % SPEC_SHA
|
||||
|
||||
FAMILIES = ("/chat/completions", "/embeddings", "/models", "/images", "/audio", "/completions", "/moderations")
|
||||
EXCLUDED = {
|
||||
"listChatCompletions": "stored completions are an OpenAI product feature, not portable",
|
||||
"getChatCompletion": "stored completions",
|
||||
"updateChatCompletion": "stored completions",
|
||||
"deleteChatCompletion": "stored completions",
|
||||
"getChatCompletionMessages": "stored completions",
|
||||
"createCompletion": "legacy completions; not claimed by any registry row",
|
||||
"createModeration": "no registry row documents a moderation endpoint",
|
||||
"createImageEdit": "multipart image editing; no registry row documents it",
|
||||
"createImageVariation": "multipart image variations; no registry row documents it",
|
||||
"createTranslation": "audio translation; no registry row documents it",
|
||||
"createVoiceConsent": "OpenAI custom voices only",
|
||||
"listVoiceConsents": "OpenAI custom voices only",
|
||||
"getVoiceConsent": "OpenAI custom voices only",
|
||||
"updateVoiceConsent": "OpenAI custom voices only",
|
||||
"deleteVoiceConsent": "OpenAI custom voices only",
|
||||
"createVoice": "OpenAI custom voices only",
|
||||
"deleteVoice": "OpenAI custom voices only",
|
||||
"retrieveModel": "no registry row documents GET /models/{id}; models are listed",
|
||||
"deleteModel": "OpenAI fine-tune deletion only",
|
||||
}
|
||||
STR = re.compile(r'"((?:[^"\\]|\\.)*)"')
|
||||
|
||||
|
||||
def string_runs(block):
|
||||
"""String literals of a C++ block with adjacent literals concatenated."""
|
||||
out, last_end, cur = [], None, None
|
||||
for m in STR.finditer(block):
|
||||
if last_end is not None and block[last_end:m.start()].strip() == "":
|
||||
cur += m.group(1)
|
||||
else:
|
||||
if cur is not None:
|
||||
out.append(cur)
|
||||
cur = m.group(1)
|
||||
last_end = m.end()
|
||||
if cur is not None:
|
||||
out.append(cur)
|
||||
return out
|
||||
|
||||
|
||||
def calls_rows():
|
||||
text = open(os.path.join(ROOT, "calls.cpp")).read()
|
||||
a = text.index("const CallSpec kCalls[]")
|
||||
b = text.index("};", a)
|
||||
rows = []
|
||||
for blk in re.findall(r"\{Call::k\w+.*?Feature::k\w+, \"\w+\"\}", text[a:b], re.S):
|
||||
rows.append(string_runs(blk))
|
||||
return rows # method, path, op, request, fields, response, fields_read, feature_name
|
||||
|
||||
|
||||
class Spec:
|
||||
def __init__(self, doc):
|
||||
self.doc = doc
|
||||
|
||||
def schema(self, name):
|
||||
return self.doc["components"]["schemas"][name]
|
||||
|
||||
def deref(self, node):
|
||||
while isinstance(node, dict) and "$ref" in node:
|
||||
node = self.schema(node["$ref"].rsplit("/", 1)[1])
|
||||
return node
|
||||
|
||||
def props(self, node, seen=None):
|
||||
seen = seen or set()
|
||||
out = set()
|
||||
if isinstance(node, dict) and "$ref" in node:
|
||||
ref = node["$ref"]
|
||||
if ref in seen:
|
||||
return out
|
||||
seen.add(ref)
|
||||
return self.props(self.deref(node), seen)
|
||||
if not isinstance(node, dict):
|
||||
return out
|
||||
out |= set(node.get("properties", {}))
|
||||
for key in ("allOf", "oneOf", "anyOf"):
|
||||
for sub in node.get(key, []):
|
||||
out |= self.props(sub, seen)
|
||||
return out
|
||||
|
||||
def schema_name_of(self, content, want=""):
|
||||
ref = content.get("schema", {})
|
||||
if "$ref" in ref:
|
||||
return ref["$ref"].rsplit("/", 1)[1]
|
||||
# a union (anyOf / oneOf): the wanted name may be one member
|
||||
for key in ("anyOf", "oneOf"):
|
||||
for sub in ref.get(key, []):
|
||||
if "$ref" in sub and sub["$ref"].rsplit("/", 1)[1] == want:
|
||||
return want
|
||||
return ""
|
||||
|
||||
|
||||
def check_calls(spec, bad):
|
||||
rows = calls_rows()
|
||||
declared = set()
|
||||
for method, path, op, req, fields, resp, read, _feat in rows:
|
||||
node = spec.doc["paths"].get(path, {}).get(method.lower())
|
||||
if not node:
|
||||
bad.append("%s %s: not in the spec" % (method, path))
|
||||
continue
|
||||
declared.add(op)
|
||||
if node["operationId"] != op:
|
||||
bad.append("%s %s: operationId %s != %s" % (method, path, node["operationId"], op))
|
||||
body = node.get("requestBody", {}).get("content", {})
|
||||
if req:
|
||||
content = body.get("application/json") or body.get("multipart/form-data") or {}
|
||||
if spec.schema_name_of(content) != req:
|
||||
bad.append("%s %s: request schema %s != %s" % (method, path, spec.schema_name_of(content), req))
|
||||
known = spec.props(content.get("schema", {}))
|
||||
for f in filter(None, fields.split(",")):
|
||||
if f not in known:
|
||||
bad.append("%s %s: body field %s not in %s" % (method, path, f, req))
|
||||
if resp:
|
||||
ok = node["responses"]["200"]["content"]["application/json"]
|
||||
if spec.schema_name_of(ok, resp) != resp:
|
||||
bad.append("%s %s: response schema %s != %s" % (method, path, spec.schema_name_of(ok), resp))
|
||||
known = spec.props(ok.get("schema", {}))
|
||||
if not known:
|
||||
known = spec.props({"$ref": "#/components/schemas/" + resp})
|
||||
for f in filter(None, read.split(",")):
|
||||
if f not in known:
|
||||
bad.append("%s %s: response field %s not in %s" % (method, path, f, resp))
|
||||
for path, item in spec.doc["paths"].items():
|
||||
if not path.startswith(FAMILIES):
|
||||
continue
|
||||
for m, node in item.items():
|
||||
if m not in ("get", "post", "delete"):
|
||||
continue
|
||||
op = node["operationId"]
|
||||
if op not in declared and op not in EXCLUDED:
|
||||
bad.append("spec operation %s %s (%s) is neither in the table nor excluded" % (m.upper(), path, op))
|
||||
for op in EXCLUDED:
|
||||
if op in declared:
|
||||
bad.append("%s is both declared and excluded" % op)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def registry_rows():
|
||||
text = open(os.path.join(ROOT, "registry.cpp")).read()
|
||||
a = text.index("const ProviderRow kRows[]")
|
||||
b = text.index("constexpr std::size_t kRowCount")
|
||||
out = []
|
||||
for blk in re.findall(r'\n \{"[a-z0-9]+",.*?(?=\n \{"|\n //|\n\};)', text[a:b], re.S):
|
||||
s = string_runs(blk)
|
||||
out.append({"id": s[0], "name": s[1], "base": s[2], "key": s[3], "base_env": s[4], "extra_env": s[5],
|
||||
"models": s[6], "doc": s[7]})
|
||||
return out
|
||||
|
||||
|
||||
def spec_md_rows():
|
||||
text = open(os.path.join(ROOT, "SPEC.md")).read()
|
||||
a = text.index("## 4. Provider table")
|
||||
b = text.index("\n## ", a + 5)
|
||||
rows = {}
|
||||
for line in text[a:b].splitlines():
|
||||
cells = [c.strip().strip("`") for c in line.strip().strip("|").split("|")]
|
||||
if line.startswith("|") and len(cells) >= 5 and re.fullmatch(r"[a-z0-9]+", cells[0]) and cells[0] != "id":
|
||||
rows[cells[0]] = cells
|
||||
return rows
|
||||
|
||||
|
||||
def check_registry(bad):
|
||||
reg = registry_rows()
|
||||
md = spec_md_rows()
|
||||
ids = [r["id"] for r in reg]
|
||||
if len(ids) != len(set(ids)):
|
||||
bad.append("registry.cpp has duplicate ids")
|
||||
for r in reg:
|
||||
cells = md.get(r["id"])
|
||||
if not cells:
|
||||
bad.append("provider %s is in registry.cpp but not in SPEC.md section 4" % r["id"])
|
||||
continue
|
||||
line = " ".join(cells)
|
||||
for what in ("base", "key", "doc"):
|
||||
if r[what] not in line:
|
||||
bad.append("provider %s: %s '%s' differs between registry.cpp and SPEC.md" % (r["id"], what, r[what]))
|
||||
for i in md:
|
||||
if i not in ids:
|
||||
bad.append("provider %s is in SPEC.md section 4 but not in registry.cpp" % i)
|
||||
return len(reg)
|
||||
|
||||
|
||||
def main():
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("--")]
|
||||
local = "--local" in sys.argv
|
||||
bad = []
|
||||
n_reg = check_registry(bad)
|
||||
n_calls = 0
|
||||
if not local:
|
||||
if args:
|
||||
doc = json.load(open(args[0]))
|
||||
else:
|
||||
req = urllib.request.Request(URL, headers={"User-Agent": "molo-compat-spec-check/1"})
|
||||
with urllib.request.urlopen(req, timeout=120) as r:
|
||||
doc = json.loads(r.read().decode("utf-8"))
|
||||
n_calls = check_calls(Spec(doc), bad)
|
||||
print("openapi %s (%s)" % (doc["info"]["version"], SPEC_SHA[:12]))
|
||||
print("registry rows: %d, call rows checked: %d" % (n_reg, n_calls))
|
||||
if bad:
|
||||
print("FAIL")
|
||||
for b in bad:
|
||||
print(" " + b)
|
||||
sys.exit(1)
|
||||
print("OK")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user