From 38687de58f2e5d19874d246ffe29efd3a78c22e9 Mon Sep 17 00:00:00 2001 From: retoor Date: Sat, 24 Oct 2026 08:21:14 +0200 Subject: [PATCH] docs(clients): vendor platform integration: llm spec, readmes, example spec, plan gaps, counts Co-Authored-By: Claude Sonnet 5.5 --- AGENTS.md | 2 +- README.md | 25 +++++++++++-- example/README.md | 12 ++++++- example/SPEC.md | 8 +++++ plans/17-llm-vendor-platforms.md | 21 +++++++++-- src/llm/SPEC.md | 61 ++++++++++++++++++++++++++++++++ 6 files changed, 122 insertions(+), 7 deletions(-) create mode 100644 src/llm/SPEC.md diff --git a/AGENTS.md b/AGENTS.md index 6745b7a..6407afc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ make test # whole repo make -f example/example.mk example-test # this part only, standalone ``` -Expected: `example suite: 163 passed, 0 failed`. Failure action: reproduce +Expected: `example suite: 205 passed, 0 failed`. Failure action: reproduce through the real stack first (failing route-level test), then fix; the reproduction stays in the suite. Conventions: `tests/README.md`. diff --git a/README.md b/README.md index 984575f..58605ac 100644 --- a/README.md +++ b/README.md @@ -35,13 +35,13 @@ is not defined in the Makefile yet. ## The living example -A route-level demo of every feature, with 163 integration tests on the real +A route-level demo of every feature, with 205 integration tests on the real stack (no mocks). Standalone, no other part needed. ```sh make -f example/example.mk example # build ./build/example/living-server 8080 # run -make -f example/example.mk example-test # expect: 163 passed, 0 failed +make -f example/example.mk example-test # expect: 205 passed, 0 failed ``` Open `http://127.0.0.1:8080/` and click through the matrix. Narrow the @@ -78,7 +78,22 @@ Credentials (environment, then `.env.json.local`, then `.env.json`): `GOOGLE_APPLICATION_CREDENTIALS` (git-ignored `.secrets/`). `make google-token && ./build/google/sa-token` prints an access token. The Agent Platform demo pages are under `/agent/` and, like all -`/voice`, `/google` and `/agent` routes, are excluded from locust. +`/voice`, `/google`, `/agent` and `/ai` routes, are excluded from locust. + +Other vendors (plan `plans/17-llm-vendor-platforms.md`, shared layer +`src/llm/SPEC.md`): pages `/ai/xai`, `/ai/xai/platform`, `/ai/anthropic`, +`/ai/anthropic/admin`, `/ai/meta`, `/ai/deepseek`, `/ai/openai`, +`/ai/openai/platform`, `/ai/openai/admin`, `/ai/openrouter`. Keys go in +`.env.json.local` or the environment: `XAI_API_KEY`, `XAI_MANAGEMENT_KEY`, +`ANTHROPIC_API_KEY`, `ANTHROPIC_ADMIN_KEY`, `META_API_KEY` (or +`MODEL_API_KEY`), `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, `OPENAI_ADMIN_KEY`, +`OPENROUTER_API_KEY`, `OPENROUTER_MANAGEMENT_KEY`. Without a key a page and +its status route still load and every calling route answers 503 +`not_configured`; paid or destructive routes are dry runs unless the body has +`"confirm":true`. `MOLO_LIVE_NET=1 make ai-test` adds opt-in live checks; +anything that costs money also needs `MOLO_LIVE_PAID=1`. No vendor key +exists in this repo's environment yet, so every vendor package is verified +against its published spec and unauthenticated probes only. Test conventions (one test file per URL, Arrange/Act/Assert, integration only) are in `tests/README.md`. @@ -208,6 +223,10 @@ src/app application, routers, views, forms, validation, templates src/realtime WebSocket, SSE, Socket.IO, streaming, WebDAV, expose tunnels src/supervisor process supervisor, tasks, replication src/clients HTTP/REST client, OpenAPI, test double +src/llm shared layer for the vendor platforms below (client, SSE, errors, ChatModel) +src/xai src/anthropic src/meta src/deepseek src/openai src/openrouter + whole public API platforms of Grok, Claude, Meta Model API (Muse Spark), + DeepSeek, OpenAI and OpenRouter, one dir per work package src/google Google Cloud: generic client, OAuth + service accounts, voice, and the Gemini Enterprise Agent Platform (models, runtime, sessions, sandbox, registry, armor, optimize, adk, live), one dir per module diff --git a/example/README.md b/example/README.md index 8044769..4b13419 100644 --- a/example/README.md +++ b/example/README.md @@ -14,7 +14,7 @@ Open `http://127.0.0.1:8080/` and click through the matrix; `/stats` has the charts (Chart.js is embedded in the binary, no CDN). Narrow the window to 360px: the top nav collapses into a dropdown menu. -Run the suite (163 route-level integration tests, real stack, no mocks): +Run the suite (205 route-level integration tests, real stack, no mocks): ```sh make -f example/example.mk example-test @@ -129,6 +129,16 @@ on a missing row. | agent evaluation and optimize | `/agent/optimize` | `tests/example/agent/optimize/page_test.cpp` | `plans/16-agent-platform.md` | | agent development kit | `/agent/adk` | `tests/example/agent/adk/page_test.cpp` | `plans/16-agent-platform.md` | | agent live streaming | `/agent/live` | `tests/example/agent/live/page_test.cpp` | `plans/16-agent-platform.md` | +| xai grok chat and tools | `/ai/xai` | `tests/example/ai/xai/core/` | `plans/17-llm-vendor-platforms.md` | +| xai grok voice, collections, management | `/ai/xai/platform` | `tests/example/ai/xai/platform/` | `plans/17-llm-vendor-platforms.md` | +| anthropic messages and tools | `/ai/anthropic` | `tests/example/ai/anthropic/core/` | `plans/17-llm-vendor-platforms.md` | +| anthropic admin and agents | `/ai/anthropic/admin` | `tests/example/ai/anthropic/admin/` | `plans/17-llm-vendor-platforms.md` | +| meta model api (muse spark) | `/ai/meta` | `tests/example/ai/meta/` | `plans/17-llm-vendor-platforms.md` | +| deepseek | `/ai/deepseek` | `tests/example/ai/deepseek/` | `plans/17-llm-vendor-platforms.md` | +| openai responses and media | `/ai/openai` | `tests/example/ai/openai/core/` | `plans/17-llm-vendor-platforms.md` | +| openai platform services | `/ai/openai/platform` | `tests/example/ai/openai/platform/` | `plans/17-llm-vendor-platforms.md` | +| openai organization admin | `/ai/openai/admin` | `tests/example/ai/openai/admin/` | `plans/17-llm-vendor-platforms.md` | +| openrouter | `/ai/openrouter` | `tests/example/ai/openrouter/` | `plans/17-llm-vendor-platforms.md` | ### Extended features diff --git a/example/SPEC.md b/example/SPEC.md index 714ce84..ee4005f 100644 --- a/example/SPEC.md +++ b/example/SPEC.md @@ -66,6 +66,14 @@ owning sibling plan and the file that will close it. routes need an explicit confirmation. No route test scripts a Google reply. All `/agent`, `/voice` and `/google` routes are excluded from locust; `check_locust_coverage.py` fails if one is referenced. +18. LLM vendor platforms (`src/`, shared layer `src/llm`): ten demo + pages at `/ai/[/]` for xAI, Anthropic, Meta, DeepSeek, + OpenAI and OpenRouter. Same contract as item 17: GET pages and `/status` + answer 2xx with no credential, every calling route is POST or DELETE and + answers 503 `not_configured` without one and 422 on bad input; paid or + destructive routes are dry runs unless the body has `"confirm":true`. + No route test scripts a vendor reply. All `/ai` routes are excluded from + locust. ## Caps diff --git a/plans/17-llm-vendor-platforms.md b/plans/17-llm-vendor-platforms.md index 12881a8..af806a2 100644 --- a/plans/17-llm-vendor-platforms.md +++ b/plans/17-llm-vendor-platforms.md @@ -1,6 +1,6 @@ # 17 LLM vendor platforms: xAI, Anthropic, Meta, DeepSeek, OpenAI, OpenRouter -Status: plan, 2026-10-24. Same method as plan 16 (Google): one module per +Status: implemented and merged, 2026-10-24 (all ten packages; see each package SPEC for what is verified against a published spec, probed live, or unverified). Same method as plan 16 (Google): one module per product area, derived from authoritative machine-readable sources, one call table per module, a checker script, local deterministic tests, opt-in live tests, a demo page. Scope is each vendor's whole public API platform, not @@ -123,4 +123,21 @@ Merge each branch; write the vendor-level `SPEC.md` overview, READMEs and ## 4. Gaps - No vendor keys yet: every package is verified against its published spec - and unauthenticated probes only until keys are provided. + (or, where the vendor publishes none, its official SDK source plus the + rendered reference pages) and unauthenticated probes only. No success + response body was seen from any vendor. Provide keys in `.env.json.local` + and run `MOLO_LIVE_NET=1 make ai-test` (plus `MOLO_LIVE_PAID=1` for paid + calls) to close this. +- Anthropic and DeepSeek publish no OpenAPI document; Meta publishes prose + docs only. Their checkers compare the call tables with SDK source and + rendered pages, which is weaker than a schema check. +- `src/llm` gaps (GET streaming, response headers on streams, per-call + timeouts, opaque reasoning fields, comment lines in SSE, error `code` + handling) are listed in `src/llm/SPEC.md` section 4. +- Anthropic core and admin both wrap user profiles; unify when a key allows + checking which package should own them. +- Meta's Muse Code protocol client never ran against a real `muse` binary. +- OpenAI Assistants/Threads answered 404 unauthenticated (announced sunset, + not confirmed with a key); `GET /v1/videos` and `GET /v1/moderations` + answered 404 unexplained. +- xAI `GET /v1/skills` is in the OpenAPI document but answered 404 live. diff --git a/src/llm/SPEC.md b/src/llm/SPEC.md new file mode 100644 index 0000000..98d882d --- /dev/null +++ b/src/llm/SPEC.md @@ -0,0 +1,61 @@ +# LLM vendor foundation - Specification + +Owned directory: `src/llm/`. Namespace `molo::llm`. Depends on `molo::clients` +only. Used by the vendor modules `src/xai`, `src/anthropic`, `src/meta`, +`src/deepseek`, `src/openai`, `src/openrouter` (plan `plans/17-llm-vendor-platforms.md`). + +## 1. Module map (one concern per file) + +| File | Concern | +|---|---| +| `caps.h` | Static ceilings, never autotuned | +| `error.h/.cpp` | `ApiError` (vendor, status, type, message, request id), `NotConfiguredError`, parser for the common error envelopes | +| `config.h/.cpp` | `Config::get`: environment, then `.env.json.local`, then `.env.json`; `MOLO_ENV_FILES=off` for tests | +| `sse.h/.cpp` | `SseDecoder`: named events, `[DONE]`, comments, JSON error bodies in place of a stream | +| `api_client.h/.cpp` | `ApiClient` (bearer or header-key auth, default headers, JSON / raw / multipart / SSE POST) and the `Service` base | +| `chat_types.h/.cpp` | Vendor-neutral messages, tools, request, response, chunk, and the `ChatModel` seam | + +## 2. Static caps (see caps.h) + +| Knob | Value | +|---|---| +| Request body | 32 MiB | +| Response body | 64 MiB | +| Streamed bytes | 64 MiB | +| One SSE event | 4 MiB | +| Upload | 512 MiB | +| Messages / tools per request | 4096 / 256 | +| Page walk | 100 pages | +| Config file | 64 KiB | + +## 3. Contracts + +- No key: `configured()` is false and every call throws `NotConfiguredError` + (503, `NOT_CONFIGURED`) before any network traffic. +- The key travels in a header only, never in a URL, never in an error. +- Every failure is an `ApiError`; HTTP errors keep the vendor's own type or + code and the request id header when present. +- Paths must start with `/` and contain no `..`; ids go through + `Service::valid_id` before they are placed in a path. +- A JSON body where a stream was expected is the vendor's error document and + throws. +- Test runs set `MOLO_ENV_FILES=off`; `MOLO_LIVE_NET=1` re-enables config + files for opt-in live checks; `MOLO_LIVE_PAID=1` additionally allows calls + that cost money. + +## 4. Gaps + +Reported by the vendor modules, owner `src/llm`: + +- `ApiClient::stream` is POST-only: no GET stream (resource event replay), + and it exposes no response headers (rate-limit headers on streams). +- `ApiClient` cannot send an unauthenticated request, and has no per-call + timeout; `post_multipart` only parses JSON replies. +- `ApiError` drops the error `code`/`param` when the type is taken from + elsewhere, drops `Retry-After`, and `error_from_response` loses the + top-level `code` when `error` is a plain string (xAI). +- `SseDecoder` drops comment lines, so keep-alives cannot be counted. +- `ContentPart`/`ChatChunk` have no field for opaque reasoning payloads + (Anthropic redacted thinking, Meta encrypted reasoning, OpenRouter + `reasoning_details`); vendor models carry them on the side. +- `Url` percent-encodes `[` and `]` in query names.