docs(clients): vendor platform integration: llm spec, readmes, example spec, plan gaps, counts
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -31,7 +31,7 @@ make test # whole repo
|
||||
make -f example/example.mk example-test # this part only, standalone
|
||||
```
|
||||
|
||||
Expected: `example suite: 163 passed, 0 failed`. Failure action: reproduce
|
||||
Expected: `example suite: 205 passed, 0 failed`. Failure action: reproduce
|
||||
through the real stack first (failing route-level test), then fix; the
|
||||
reproduction stays in the suite. Conventions: `tests/README.md`.
|
||||
|
||||
|
||||
@@ -35,13 +35,13 @@ is not defined in the Makefile yet.
|
||||
|
||||
## The living example
|
||||
|
||||
A route-level demo of every feature, with 163 integration tests on the real
|
||||
A route-level demo of every feature, with 205 integration tests on the real
|
||||
stack (no mocks). Standalone, no other part needed.
|
||||
|
||||
```sh
|
||||
make -f example/example.mk example # build
|
||||
./build/example/living-server 8080 # run
|
||||
make -f example/example.mk example-test # expect: 163 passed, 0 failed
|
||||
make -f example/example.mk example-test # expect: 205 passed, 0 failed
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:8080/` and click through the matrix. Narrow the
|
||||
@@ -78,7 +78,22 @@ Credentials (environment, then `.env.json.local`, then `.env.json`):
|
||||
`GOOGLE_APPLICATION_CREDENTIALS` (git-ignored `.secrets/`). `make
|
||||
google-token && ./build/google/sa-token` prints an access token.
|
||||
The Agent Platform demo pages are under `/agent/<module>` and, like all
|
||||
`/voice`, `/google` and `/agent` routes, are excluded from locust.
|
||||
`/voice`, `/google`, `/agent` and `/ai` routes, are excluded from locust.
|
||||
|
||||
Other vendors (plan `plans/17-llm-vendor-platforms.md`, shared layer
|
||||
`src/llm/SPEC.md`): pages `/ai/xai`, `/ai/xai/platform`, `/ai/anthropic`,
|
||||
`/ai/anthropic/admin`, `/ai/meta`, `/ai/deepseek`, `/ai/openai`,
|
||||
`/ai/openai/platform`, `/ai/openai/admin`, `/ai/openrouter`. Keys go in
|
||||
`.env.json.local` or the environment: `XAI_API_KEY`, `XAI_MANAGEMENT_KEY`,
|
||||
`ANTHROPIC_API_KEY`, `ANTHROPIC_ADMIN_KEY`, `META_API_KEY` (or
|
||||
`MODEL_API_KEY`), `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, `OPENAI_ADMIN_KEY`,
|
||||
`OPENROUTER_API_KEY`, `OPENROUTER_MANAGEMENT_KEY`. Without a key a page and
|
||||
its status route still load and every calling route answers 503
|
||||
`not_configured`; paid or destructive routes are dry runs unless the body has
|
||||
`"confirm":true`. `MOLO_LIVE_NET=1 make ai-test` adds opt-in live checks;
|
||||
anything that costs money also needs `MOLO_LIVE_PAID=1`. No vendor key
|
||||
exists in this repo's environment yet, so every vendor package is verified
|
||||
against its published spec and unauthenticated probes only.
|
||||
Test conventions (one test file per URL, Arrange/Act/Assert, integration
|
||||
only) are in `tests/README.md`.
|
||||
|
||||
@@ -208,6 +223,10 @@ src/app application, routers, views, forms, validation, templates
|
||||
src/realtime WebSocket, SSE, Socket.IO, streaming, WebDAV, expose tunnels
|
||||
src/supervisor process supervisor, tasks, replication
|
||||
src/clients HTTP/REST client, OpenAPI, test double
|
||||
src/llm shared layer for the vendor platforms below (client, SSE, errors, ChatModel)
|
||||
src/xai src/anthropic src/meta src/deepseek src/openai src/openrouter
|
||||
whole public API platforms of Grok, Claude, Meta Model API (Muse Spark),
|
||||
DeepSeek, OpenAI and OpenRouter, one dir per work package
|
||||
src/google Google Cloud: generic client, OAuth + service accounts, voice, and the
|
||||
Gemini Enterprise Agent Platform (models, runtime, sessions, sandbox,
|
||||
registry, armor, optimize, adk, live), one dir per module
|
||||
|
||||
+11
-1
@@ -14,7 +14,7 @@ Open `http://127.0.0.1:8080/` and click through the matrix; `/stats` has the
|
||||
charts (Chart.js is embedded in the binary, no CDN). Narrow the
|
||||
window to 360px: the top nav collapses into a dropdown menu.
|
||||
|
||||
Run the suite (163 route-level integration tests, real stack, no mocks):
|
||||
Run the suite (205 route-level integration tests, real stack, no mocks):
|
||||
|
||||
```sh
|
||||
make -f example/example.mk example-test
|
||||
@@ -129,6 +129,16 @@ on a missing row.
|
||||
| agent evaluation and optimize | `/agent/optimize` | `tests/example/agent/optimize/page_test.cpp` | `plans/16-agent-platform.md` |
|
||||
| agent development kit | `/agent/adk` | `tests/example/agent/adk/page_test.cpp` | `plans/16-agent-platform.md` |
|
||||
| agent live streaming | `/agent/live` | `tests/example/agent/live/page_test.cpp` | `plans/16-agent-platform.md` |
|
||||
| xai grok chat and tools | `/ai/xai` | `tests/example/ai/xai/core/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| xai grok voice, collections, management | `/ai/xai/platform` | `tests/example/ai/xai/platform/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| anthropic messages and tools | `/ai/anthropic` | `tests/example/ai/anthropic/core/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| anthropic admin and agents | `/ai/anthropic/admin` | `tests/example/ai/anthropic/admin/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| meta model api (muse spark) | `/ai/meta` | `tests/example/ai/meta/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| deepseek | `/ai/deepseek` | `tests/example/ai/deepseek/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| openai responses and media | `/ai/openai` | `tests/example/ai/openai/core/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| openai platform services | `/ai/openai/platform` | `tests/example/ai/openai/platform/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| openai organization admin | `/ai/openai/admin` | `tests/example/ai/openai/admin/` | `plans/17-llm-vendor-platforms.md` |
|
||||
| openrouter | `/ai/openrouter` | `tests/example/ai/openrouter/` | `plans/17-llm-vendor-platforms.md` |
|
||||
|
||||
### Extended features
|
||||
|
||||
|
||||
@@ -66,6 +66,14 @@ owning sibling plan and the file that will close it.
|
||||
routes need an explicit confirmation. No route test scripts a Google
|
||||
reply. All `/agent`, `/voice` and `/google` routes are excluded from
|
||||
locust; `check_locust_coverage.py` fails if one is referenced.
|
||||
18. LLM vendor platforms (`src/<vendor>`, shared layer `src/llm`): ten demo
|
||||
pages at `/ai/<vendor>[/<part>]` for xAI, Anthropic, Meta, DeepSeek,
|
||||
OpenAI and OpenRouter. Same contract as item 17: GET pages and `/status`
|
||||
answer 2xx with no credential, every calling route is POST or DELETE and
|
||||
answers 503 `not_configured` without one and 422 on bad input; paid or
|
||||
destructive routes are dry runs unless the body has `"confirm":true`.
|
||||
No route test scripts a vendor reply. All `/ai` routes are excluded from
|
||||
locust.
|
||||
|
||||
## Caps
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 17 LLM vendor platforms: xAI, Anthropic, Meta, DeepSeek, OpenAI, OpenRouter
|
||||
|
||||
Status: plan, 2026-10-24. Same method as plan 16 (Google): one module per
|
||||
Status: implemented and merged, 2026-10-24 (all ten packages; see each package SPEC for what is verified against a published spec, probed live, or unverified). Same method as plan 16 (Google): one module per
|
||||
product area, derived from authoritative machine-readable sources, one call
|
||||
table per module, a checker script, local deterministic tests, opt-in live
|
||||
tests, a demo page. Scope is each vendor's whole public API platform, not
|
||||
@@ -123,4 +123,21 @@ Merge each branch; write the vendor-level `SPEC.md` overview, READMEs and
|
||||
## 4. Gaps
|
||||
|
||||
- No vendor keys yet: every package is verified against its published spec
|
||||
and unauthenticated probes only until keys are provided.
|
||||
(or, where the vendor publishes none, its official SDK source plus the
|
||||
rendered reference pages) and unauthenticated probes only. No success
|
||||
response body was seen from any vendor. Provide keys in `.env.json.local`
|
||||
and run `MOLO_LIVE_NET=1 make ai-test` (plus `MOLO_LIVE_PAID=1` for paid
|
||||
calls) to close this.
|
||||
- Anthropic and DeepSeek publish no OpenAPI document; Meta publishes prose
|
||||
docs only. Their checkers compare the call tables with SDK source and
|
||||
rendered pages, which is weaker than a schema check.
|
||||
- `src/llm` gaps (GET streaming, response headers on streams, per-call
|
||||
timeouts, opaque reasoning fields, comment lines in SSE, error `code`
|
||||
handling) are listed in `src/llm/SPEC.md` section 4.
|
||||
- Anthropic core and admin both wrap user profiles; unify when a key allows
|
||||
checking which package should own them.
|
||||
- Meta's Muse Code protocol client never ran against a real `muse` binary.
|
||||
- OpenAI Assistants/Threads answered 404 unauthenticated (announced sunset,
|
||||
not confirmed with a key); `GET /v1/videos` and `GET /v1/moderations`
|
||||
answered 404 unexplained.
|
||||
- xAI `GET /v1/skills` is in the OpenAPI document but answered 404 live.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# LLM vendor foundation - Specification
|
||||
|
||||
Owned directory: `src/llm/`. Namespace `molo::llm`. Depends on `molo::clients`
|
||||
only. Used by the vendor modules `src/xai`, `src/anthropic`, `src/meta`,
|
||||
`src/deepseek`, `src/openai`, `src/openrouter` (plan `plans/17-llm-vendor-platforms.md`).
|
||||
|
||||
## 1. Module map (one concern per file)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings, never autotuned |
|
||||
| `error.h/.cpp` | `ApiError` (vendor, status, type, message, request id), `NotConfiguredError`, parser for the common error envelopes |
|
||||
| `config.h/.cpp` | `Config::get`: environment, then `.env.json.local`, then `.env.json`; `MOLO_ENV_FILES=off` for tests |
|
||||
| `sse.h/.cpp` | `SseDecoder`: named events, `[DONE]`, comments, JSON error bodies in place of a stream |
|
||||
| `api_client.h/.cpp` | `ApiClient` (bearer or header-key auth, default headers, JSON / raw / multipart / SSE POST) and the `Service` base |
|
||||
| `chat_types.h/.cpp` | Vendor-neutral messages, tools, request, response, chunk, and the `ChatModel` seam |
|
||||
|
||||
## 2. Static caps (see caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Request body | 32 MiB |
|
||||
| Response body | 64 MiB |
|
||||
| Streamed bytes | 64 MiB |
|
||||
| One SSE event | 4 MiB |
|
||||
| Upload | 512 MiB |
|
||||
| Messages / tools per request | 4096 / 256 |
|
||||
| Page walk | 100 pages |
|
||||
| Config file | 64 KiB |
|
||||
|
||||
## 3. Contracts
|
||||
|
||||
- No key: `configured()` is false and every call throws `NotConfiguredError`
|
||||
(503, `NOT_CONFIGURED`) before any network traffic.
|
||||
- The key travels in a header only, never in a URL, never in an error.
|
||||
- Every failure is an `ApiError`; HTTP errors keep the vendor's own type or
|
||||
code and the request id header when present.
|
||||
- Paths must start with `/` and contain no `..`; ids go through
|
||||
`Service::valid_id` before they are placed in a path.
|
||||
- A JSON body where a stream was expected is the vendor's error document and
|
||||
throws.
|
||||
- Test runs set `MOLO_ENV_FILES=off`; `MOLO_LIVE_NET=1` re-enables config
|
||||
files for opt-in live checks; `MOLO_LIVE_PAID=1` additionally allows calls
|
||||
that cost money.
|
||||
|
||||
## 4. Gaps
|
||||
|
||||
Reported by the vendor modules, owner `src/llm`:
|
||||
|
||||
- `ApiClient::stream` is POST-only: no GET stream (resource event replay),
|
||||
and it exposes no response headers (rate-limit headers on streams).
|
||||
- `ApiClient` cannot send an unauthenticated request, and has no per-call
|
||||
timeout; `post_multipart` only parses JSON replies.
|
||||
- `ApiError` drops the error `code`/`param` when the type is taken from
|
||||
elsewhere, drops `Retry-After`, and `error_from_response` loses the
|
||||
top-level `code` when `error` is a plain string (xAI).
|
||||
- `SseDecoder` drops comment lines, so keep-alives cannot be counted.
|
||||
- `ContentPart`/`ChatChunk` have no field for opaque reasoning payloads
|
||||
(Anthropic redacted thinking, Meta encrypted reasoning, OpenRouter
|
||||
`reasoning_details`); vendor models carry them on the side.
|
||||
- `Url` percent-encodes `[` and `]` in query names.
|
||||
Reference in New Issue
Block a user