Files
moloapp/plans/17-llm-vendor-platforms.md
T

12 KiB

17 LLM vendor platforms: xAI, Anthropic, Meta, DeepSeek, OpenAI, OpenRouter

Status: implemented and merged, 2026-10-24 (all ten packages; see each package SPEC for what is verified against a published spec, probed live, or unverified). Same method as plan 16 (Google): one module per product area, derived from authoritative machine-readable sources, one call table per module, a checker script, local deterministic tests, opt-in live tests, a demo page. Scope is each vendor's whole public API platform, not just chat. "Meta" means the Meta Model API (docs: dev.meta.ai/docs, base URL https://api.meta.ai/v1) that serves Muse Spark and the other Muse / Meta models. It does NOT mean the Llama API.

Foundation (already in src/llm/, read it first): ApiClient (bearer or header-key auth, default headers, JSON / raw / multipart / SSE streaming, ApiError), Service base, SseDecoder (named events, [DONE]), Config::get (environment, then .env.json.local, then .env.json), neutral chat types and the ChatModel seam (chat_types.h). src/google/ is the reference implementation of this method: read one finished module (for example src/google/runtime: calls.cpp table, tools/check_discovery.py, SPEC layout, demo, tests).

1. Work packages (one worktree each)

Package Dirs Covers
xai core src/xai/core Chat completions, Responses API (stored responses get/delete), tools (function calling, web search, X search, code execution, collections search, remote MCP), structured outputs, reasoning, image generation and editing, video generation, embeddings if public, models, tokenize, deferred completions, batch API, the Anthropic-compatible messages endpoint, XaiChatModel : ChatModel
xai platform src/xai/platform Voice (realtime WebSocket agent API, TTS, STT), collections and documents, files, management API (API keys, billing, usage, team), API-key info; reuse src/google/live WsClient for WebSocket
anthropic core src/anthropic/core Messages (create, streaming with every event type, thinking, tools: client tools and server tools such as web search, web fetch, code execution, computer use, bash, text editor, memory, tool search, MCP connector), vision / PDF / documents / citations / search results, prompt caching, structured outputs, Message Batches, token counting, Models, Files, Skills, beta header handling, AnthropicChatModel : ChatModel
anthropic admin src/anthropic/admin Admin API (organization, users, invites, workspaces and members, API keys), usage and cost reports (messages, Claude Code), and any public managed-agents / sessions / environments REST surface; admin key is a separate credential
meta src/meta Meta Model API (dev.meta.ai): Muse Spark (Responses-style and any chat/messages endpoints the docs list), tool calling, structured output, reasoning, vision input, Muse Wonder, Muse Image, Muse Voice Transcribe, Segment Anything, Muse Spark Open, Muse Glimmer, models, files / uploads if public, usage and pricing fields, the Muse Code agent surface if it has an API, MetaChatModel : ChatModel
openai core src/openai/core Responses API (create, retrieve, delete, cancel, input items, streaming events), Chat Completions (stored completions get/list/update/delete), legacy Completions, tools (function calling, web search, file search, code interpreter, computer use, image generation tool, remote MCP, custom tools), structured outputs, reasoning, Images (generate, edit, variations), Audio (speech, transcription, translation, streaming), Videos (Sora), Embeddings, Moderations, Models, OpenaiChatModel : ChatModel
openai platform src/openai/platform Realtime (WebSocket and client secrets / ephemeral keys, SIP calls; reuse src/google/live WsClient), Files, Uploads (multipart parts), Batch, Fine-tuning (jobs, events, checkpoints, graders), Vector stores (files, file batches, search), Evals (runs, output items), Conversations (items), Containers (files), Assistants / Threads / Runs (legacy beta), ChatKit and agent builder surfaces if public
openai admin src/openai/admin Organization Admin API (admin keys, users, invites, projects, project users, service accounts, API keys, rate limits, certificates, groups / roles if public), Usage and Costs endpoints, Audit logs, Webhooks (events, signature verification), data residency fields; separate admin key
openrouter src/openrouter Chat completions, Responses (beta) and the Anthropic-compatible messages endpoint if public, models (list, per-model endpoints, user models, filters), providers, generation stats, credits, activity, API keys management (create / list / update / delete with the management key), presets, routing parameters (provider preferences, fallbacks, models array, :online, variants like :nitro), plugins (web search, file parser, response healing), embeddings, auth (OAuth PKCE key exchange), BYOK and guardrails if public, OpenrouterChatModel : ChatModel
deepseek src/deepseek Chat completions (chat and reasoner, reasoning_content, tool calls, JSON output), beta: prefix completion and FIM, context caching usage, models, user balance, the Anthropic-compatible endpoint, errors and rate-limit semantics, DeepseekChatModel : ChatModel

2. Rules for every package

  1. Read AGENTS.md, this plan, src/llm/*.h, and one finished Google module with its tests and demo.
  2. Research first, from machine-readable sources, deeply. Order of preference: the vendor's OpenAPI document (many are published, or referenced by the official SDK repos, for example a .stats.yml or openapi.json link); the official SDK source (Python / TypeScript / Go: types, endpoints, headers, beta flags, error classes); the official API reference and guides. Cross-check at least two sources for every path and field. Record everything under ## Sources in the SPEC.
  3. Each package SPEC (src/<vendor>/<part>/SPEC.md): scope, module map (one class per concern), static caps table, contracts, ## Sources, ## Verified against spec (which document, revision, counts), ## Verified live, ## Unverified (every shape inferred rather than read from a spec or seen), ## Gaps. Never invent an endpoint silently; a product with no public API is said so.
  4. Object oriented, one concern per file, namespace molo::<vendor>::<part> (for example molo::anthropic::admin), services derive from llm::Service, hard caps for every count, size, interval, wait and page walk, ids validated with Service::valid_id before any path is built. List calls report truncation loudly (never silent). Destructive calls (delete, revoke, purge, cancel) are explicit methods with no default.
  5. Every call lives in one table constant in the package (HTTP method, path template, spec operation id, request and response schema names). A committed tools/check_spec.py (python3 stdlib only) fetches the vendor spec (or reads the pinned SDK source file you cite) and fails if a declared call or request field is absent, and fails on any spec operation that is neither in the table nor in an explicit exclusion list. Run it and record the result and date in the SPEC.
  6. NO verification with scripted vendor responses. Tests under tests/src/<vendor>/ are deterministic and local: validation, caps, ids, request bodies against the table and schemas, parsing of the documented examples as fixtures taken verbatim from the vendor docs and labelled as such, SSE decoding of a documented stream transcript, local logic with a fake ChatModel, loopback servers for protocols the vendor does not own (WebSocket). Plus opt-in live tests (see 7).
  7. Live checks. No vendor key exists in this environment yet. What you CAN do: unauthenticated probes (the host resolves, TLS works, a documented path answers 401 rather than 404, the documented error body shape). Record them under ## Verified live. Build opt-in live tests gated on MOLO_LIVE_NET=1 plus the vendor key (ANTHROPIC_API_KEY, ANTHROPIC_ADMIN_KEY, XAI_API_KEY, XAI_MANAGEMENT_KEY, DEEPSEEK_API_KEY, META_API_KEY (the docs' own example uses MODEL_API_KEY; accept both), OPENAI_API_KEY, OPENAI_ADMIN_KEY, OPENROUTER_API_KEY, OPENROUTER_MANAGEMENT_KEY, loaded via llm::Config::get), with a loud SKIP otherwise. Live tests are read-only (models list, balance, retrieve) plus at most one smallest-possible completion. Anything that costs real money (media generation, batches, fine-tuning, long contexts, repeated calls) additionally requires MOLO_LIVE_PAID=1. Never create, update or delete vendor resources in a live test. Never print, log or commit a credential. Do not read .env.json.local.
  8. Example demo: fill your stub(s) in example/app/ai/ (one per package). Routes under /ai/<vendor>[/<part>]. GET page and /status answer 2xx with no credential (the matrix crawler hits them); every route that calls the vendor is POST or DELETE and answers 503 not_configured without a credential, 422 on bad input; paid or destructive routes are dry runs unless the body has "confirm":true. Pages are mobile-first (360px): register one template inside the register function plus an optional script example/assets/ai_<vendor>_<part>.js. Add matrix rows with services.add_feature(name, page) for the GET page only. State lives inside the register function; do not edit Services.
  9. Example tests: tests/example/ai/<vendor>/<part>/*_test.cpp (Arrange / Act / Assert, see tests/README.md): GET pages, 405, 422, 503, limit cases. Never script a vendor reply. Use tests/example/support/Assert.hpp (Fixture).
  10. Locust: everything under /ai is excluded (check_locust_coverage.py). Do not touch example/locustfile.py.
  11. Do not edit shared files: Makefile, src/ai.mk, src/google/**, example/example.mk, example/app/App.*, example/app/AiDemos.hpp, example/app/ai/AiDemos.cpp, example/app/Templates.cpp, example/locustfile.py, example/check_locust_coverage.py, tests/example/support/*, any README, example/SPEC.md, src/llm/* (if the foundation blocks you, work around it inside your package and report it), other packages' files. Build globs pick new files up automatically. The src/clients tree is not yours either.
  12. Build and test: make ai-test (all vendor and shared tests) and make -f example/example.mk example-test. Do not run make lint. No em dashes anywhere. Commit subjects at most 100 characters, feat(clients): <package>: <summary>, ending with the Co-Authored-By line from your system reminder. Final report under 250 words: files, routes, test counts, what was verified how, Unverified list, foundation gaps.

3. Integration (after the worktrees finish)

Merge each branch; write the vendor-level SPEC.md overview, READMEs and example/SPEC.md rows; run the whole suite and lint; collect gaps below.

4. Gaps

  • No vendor keys yet: every package is verified against its published spec (or, where the vendor publishes none, its official SDK source plus the rendered reference pages) and unauthenticated probes only. No success response body was seen from any vendor. Provide keys in .env.json.local and run MOLO_LIVE_NET=1 make ai-test (plus MOLO_LIVE_PAID=1 for paid calls) to close this.
  • Anthropic and DeepSeek publish no OpenAPI document; Meta publishes prose docs only. Their checkers compare the call tables with SDK source and rendered pages, which is weaker than a schema check.
  • src/llm gaps (GET streaming, response headers on streams, per-call timeouts, opaque reasoning fields, comment lines in SSE, error code handling) are listed in src/llm/SPEC.md section 4.
  • Anthropic core and admin both wrap user profiles; unify when a key allows checking which package should own them.
  • Meta's Muse Code protocol client never ran against a real muse binary.
  • OpenAI Assistants/Threads answered 404 unauthenticated (announced sunset, not confirmed with a key); GET /v1/videos and GET /v1/moderations answered 404 unexplained.
  • xAI GET /v1/skills is in the OpenAPI document but answered 404 live.