docs(clients): vendor platform integration: llm spec, readmes, example spec, plan gaps, counts
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# LLM vendor foundation - Specification
|
||||
|
||||
Owned directory: `src/llm/`. Namespace `molo::llm`. Depends on `molo::clients`
|
||||
only. Used by the vendor modules `src/xai`, `src/anthropic`, `src/meta`,
|
||||
`src/deepseek`, `src/openai`, `src/openrouter` (plan `plans/17-llm-vendor-platforms.md`).
|
||||
|
||||
## 1. Module map (one concern per file)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings, never autotuned |
|
||||
| `error.h/.cpp` | `ApiError` (vendor, status, type, message, request id), `NotConfiguredError`, parser for the common error envelopes |
|
||||
| `config.h/.cpp` | `Config::get`: environment, then `.env.json.local`, then `.env.json`; `MOLO_ENV_FILES=off` for tests |
|
||||
| `sse.h/.cpp` | `SseDecoder`: named events, `[DONE]`, comments, JSON error bodies in place of a stream |
|
||||
| `api_client.h/.cpp` | `ApiClient` (bearer or header-key auth, default headers, JSON / raw / multipart / SSE POST) and the `Service` base |
|
||||
| `chat_types.h/.cpp` | Vendor-neutral messages, tools, request, response, chunk, and the `ChatModel` seam |
|
||||
|
||||
## 2. Static caps (see caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Request body | 32 MiB |
|
||||
| Response body | 64 MiB |
|
||||
| Streamed bytes | 64 MiB |
|
||||
| One SSE event | 4 MiB |
|
||||
| Upload | 512 MiB |
|
||||
| Messages / tools per request | 4096 / 256 |
|
||||
| Page walk | 100 pages |
|
||||
| Config file | 64 KiB |
|
||||
|
||||
## 3. Contracts
|
||||
|
||||
- No key: `configured()` is false and every call throws `NotConfiguredError`
|
||||
(503, `NOT_CONFIGURED`) before any network traffic.
|
||||
- The key travels in a header only, never in a URL, never in an error.
|
||||
- Every failure is an `ApiError`; HTTP errors keep the vendor's own type or
|
||||
code and the request id header when present.
|
||||
- Paths must start with `/` and contain no `..`; ids go through
|
||||
`Service::valid_id` before they are placed in a path.
|
||||
- A JSON body where a stream was expected is the vendor's error document and
|
||||
throws.
|
||||
- Test runs set `MOLO_ENV_FILES=off`; `MOLO_LIVE_NET=1` re-enables config
|
||||
files for opt-in live checks; `MOLO_LIVE_PAID=1` additionally allows calls
|
||||
that cost money.
|
||||
|
||||
## 4. Gaps
|
||||
|
||||
Reported by the vendor modules, owner `src/llm`:
|
||||
|
||||
- `ApiClient::stream` is POST-only: no GET stream (resource event replay),
|
||||
and it exposes no response headers (rate-limit headers on streams).
|
||||
- `ApiClient` cannot send an unauthenticated request, and has no per-call
|
||||
timeout; `post_multipart` only parses JSON replies.
|
||||
- `ApiError` drops the error `code`/`param` when the type is taken from
|
||||
elsewhere, drops `Retry-After`, and `error_from_response` loses the
|
||||
top-level `code` when `error` is a plain string (xAI).
|
||||
- `SseDecoder` drops comment lines, so keep-alives cannot be counted.
|
||||
- `ContentPart`/`ChatChunk` have no field for opaque reasoning payloads
|
||||
(Anthropic redacted thinking, Meta encrypted reasoning, OpenRouter
|
||||
`reasoning_details`); vendor models carry them on the side.
|
||||
- `Url` percent-encodes `[` and `]` in query names.
|
||||
Reference in New Issue
Block a user