feat(clients): meta: Meta Model API module, Muse Code protocol client, demo and tests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-24 08:02:23 +02:00
co-authored by Claude Sonnet 5.5
parent 2e0270c043
commit 6fb7c80d2d
51 changed files with 7035 additions and 3 deletions
+202
View File
@@ -0,0 +1,202 @@
# Meta Model API (`src/meta`, namespace `molo::meta`)
Status: implemented 2026-10-24. Scope is the Meta Model API at
`https://api.meta.ai/v1` (docs `https://dev.meta.ai/docs`): Muse Spark and
Muse Wonder text models, Muse Image, Muse Voice Transcribe, Segment Anything
Model 3.1, files, models, status, and the Muse Code agent protocol. It is NOT
the older Llama API (`llama.developer.meta.com`); nothing here targets it.
`developer.meta.com/ai` redirects (302) to `dev.meta.ai`, so the blog-era
material is the same site.
Credential: `META_API_KEY`, or `MODEL_API_KEY` (the docs' own name), read
through `llm::Config::get`. Keys look like `LLM_{id}_{secret}`. Bearer auth.
## Module map (one concern per file)
| File | Concern |
|---|---|
| `caps.h` | every ceiling and documented limit, with the source marked |
| `calls.h/.cpp` | the ONE call table (20 operations), `path_for` |
| `catalog.h/.cpp` | static model table: context, prices, rate limits, `max` effort |
| `client.h/.cpp` | `MetaClient` (ApiClient factory, key lookup), `RetryPolicy`, `call_with_retry` |
| `check.h/.cpp`, `schema_check.h/.cpp` | local validation; JSON Schema pre-flight (depth, size, recursion, strict) |
| `response_input.*` | Responses input items and tools (`ResponseInput`, `ToolBuilder`) |
| `response_request.*` | `ResponseRequest` body + rules |
| `response_events.*` | 27 stream event names, `ResponseAccumulator` |
| `responses.*` | create, stream, retrieve, retrieve_stream, remove, cancel, count_input_tokens |
| `chat_completions.*` | Chat Completions request, service, chunk accumulator |
| `messages.*` | Anthropic-compatible Messages: create, stream, count_tokens |
| `files.*` | upload (multipart), list page + capped walk, retrieve, content, remove |
| `images.*` | Muse Image generate / edit (JSON and multipart) / stream |
| `speech.*` | `/asr/transcribe` (JSON, SSE, text) and `/asr/realtime` WebSocket session |
| `sam.*`, `sam_mask.cpp` | SAM 3.1 prompts, request, output parser, mask decoder |
| `models.*` | models list/retrieve, unauthenticated status |
| `usage.*` | usage across the 3 shapes, cost from the catalog, `x-ratelimit-*` |
| `chat_model.*` | `MetaChatModel : llm::ChatModel` (Responses or Chat Completions) |
| `muse_code.*`, `muse_code_spec.*` | Muse Session Protocol client (stdio JSON-RPC) and its vocabulary |
| `tools/check_spec.py` | spec checker (see Verified against spec) |
## Surface
- Responses: create, stream (typed events), retrieve, streaming retrieve of a
background response, delete (explicit, destructive), cancel, count input
tokens. Reasoning continuity both ways: `previous_response_id`, or
`store:false` + `include:["reasoning.encrypted_content"]` + replayed
`reasoning` items. Tools: function, `web_search`, `tool_search`, `computer`,
plus `custom` and `namespace` (prose only). Structured output via
`text.format`. Vision/video/audio/file inputs via `ResponseInput`.
- Chat Completions: messages, tools, `response_format`, `reasoning_effort`,
streaming with usage. Rejected locally: `n>1`, `stop`, `logit_bias`,
`prediction`, `verbosity`, `web_search_options`, `modalities`, `logprobs`,
`max_output_tokens`, `text`.
- Messages: Anthropic blocks, thinking (`adaptive`/`enabled`, never disabled),
`output_config`, tools (`custom`, `web_search`, `tool_search`), stream events,
count_tokens. Rejected locally: `stop_sequences`, `top_k`, `container`,
`inference_geo`.
- Files: upload with `purpose` and `expires_after[...]` before the file part,
list (limit 1..1000, order validated because the server answers a plain-text
400), capped page walk that reports `truncated`, retrieve, content, delete.
- Images: `/images/generations`, `/images/edits` (JSON `images[]` and
multipart `image[N]`), `tool_enablement`, SSE `*.completed` events.
- Speech: WAV pre-check (RIFF PCM mono 16-bit, 16/24 kHz, <=10 min, <=32 MB),
`request` JSON part + `audio`, `Accept` selects JSON / event stream / text;
realtime session over `google::live::WsClient` with the credential in the
handshake frame, `endStream`, close codes 1000/1008/1011/1013.
- SAM 3.1: `SamPrompt` (concept; box/point object prompts per the output
grammar), `SamRequest` -> `ResponseRequest` (`metadata` strings
`propagation_direction`, `include_confidence`, `mask_encoding`),
`SamOutput::parse`, `SamMask::decode` for `one_bit` (`!`) and `lossless`
(`~`).
- Models, status, usage, rate limits, retry policy.
- Muse Code: see below.
## Muse Code
Muse Code is a coding agent product, but it has a public programmatic
surface: the Muse Session Protocol (MSP), JSON-RPC 2.0, one JSON object per
line on the stdio of `muse serve` (frame limit 10 MiB). It is not an HTTP API
of `api.meta.ai`. `MspClient` implements the handshake (`initialize`, then the
`initialized` notification), request/response correlation, a bounded
notification queue (4096, drops are counted), answers to the two
server-initiated requests (`approval/request`, `userInput/request`) with an
empty receipt, error kinds from the schema (37 codes), and shortcuts for
`session/start`, `turn/start`, `turn/cancel`, `model/list`. `request()` takes
any of the 70 schema methods. `ProcessTransport` spawns `muse serve`;
`ILineTransport` keeps the client testable without the binary. The binary is
not installed here, so nothing was run against a live host.
## Contracts and caps
- Every id is validated before a path is built (`Check::segment`); deletes
(`ResponsesService::remove`, `FilesService::remove`) are explicit methods
with no default and no retry.
- Local rules mirror the docs: `max_output_tokens>=16`, `temperature` 0..2,
`top_p` in (0,1], penalties -2..2, effort `minimal..max` (`none` rejected,
`max` only `muse-wonder-1.0` and `muse-spark-1.3`), `background` XOR
`stream`, `reasoning.encrypted_content` XOR `previous_response_id`,
`truncation` only `disabled`, background body under 1 MiB, function names
`[A-Za-z0-9_.-]` with at most one dot, `call_id` 1..64, safety identifier
<=64, schema depth<=10, <=5000 properties, <=120000 chars, no recursion, strict
subset rules, `tool_choice` only `auto`.
- Caps live in `caps.h`: inputs 4096, tools 256, list pages 20, stream events
1,000,000, realtime events 100,000, audio frame 1 MiB, retries 5 with
500 ms base and a 30 s cap, SAM lines/records/mask pixels, MSP queue.
- `ApiClient` upload cap is 512 MiB (vendor allows 1 GiB).
- Errors are the foundation `ApiError` (`type` is Meta's category such as
`invalid_request_error`; `NotConfiguredError` names `META_API_KEY`).
`RetryPolicy`: 429/500/502/503 retry with backoff+jitter, 504 never.
## Sources
1. Docs site, Markdown pages listed by `https://dev.meta.ai/llms.txt`
(154 pages, `last-modified: Fri, 23 Oct 2026 23:30:05 GMT`, etag
`a179472f87650128af185a2718d84c76`): overview, quickstart, protocols/*,
api-reference/* (operation pages and schemas pages), models, pricing and
rate limits, error handling, file handling, token counting, reasoning,
structured output, tool calling, search grounding, speech to text, image
generation, media segmentation, `sam/*`, muse-code/*, open-weight pages.
2. `muse-code-sdk` (the `meta-models` org on the code host, MIT):
`schema/msp/stable/msp.schema.json` (fingerprint
`sha256:cfdb6d2525601b9bac76584ccb0610ab3a5539b5e4f51141f079b99b39fcb4e0`,
host version 1.4.12), recorded transcripts, Rust/TS/Python SDK sources.
3. `meta-sam` (same org): Python parser `_mask_codec.py` (mask codec ported)
and `conformance/cases/*.json` (a captured live API stream among them).
4. Live unauthenticated probes, 2026-10-24 (below).
No OpenAPI document and no vendor SDK for the Model API exist: Meta tells
you to use the OpenAI SDK (Responses, Chat Completions) and the Anthropic SDK
(Messages) with a changed base URL. Where only prose exists the module says so
under Unverified. Cross-check: every path and body field was read in both the
operation page and the schemas page; behaviour rules were read in the guide
pages and the error-handling page.
## Verified against spec
`python3 -I src/meta/tools/check_spec.py` (stdlib only, fetches the docs):
20 calls (path heading, query parameters, request schema fields, response
schema present), 27 Responses stream events equal to the
`ResponseStreamEvent` union, every documented operation present in the table
(no exclusion list), catalog models/prices/rpm present on the models and
pricing pages, and the Muse Code vocabulary (70 methods, 45 notifications, 2
server requests, 37 error codes) equal to the stable MSP schema.
Result 2026-10-24: `OK`. A mutated row (renamed query parameter and path)
fails as expected.
## Verified live
Unauthenticated, 2026-10-24, `https://api.meta.ai/v1`:
- host resolves, HTTP/2, TLS verifies.
- `GET /status` -> 200 `{"is_alive":true,"service_status":"operational","service_message":"","updated_at":"","model_statuses":[]}`; a bogus bearer is ignored.
- `GET`/`POST` on `/models`, `/models/{id}`, `/files`, `/files/{id}`,
`/responses`, `/responses/{id}`, `/responses/input_tokens`,
`/chat/completions`, `/images/generations`, `/images/edits` -> 401
`{"error":{"code":"invalid_api_key","message":"Unauthorized","param":null,"type":"authentication_error"}}`.
- `/messages`, `/messages/count_tokens` -> 401
`{"error":{"message":"Unauthorized","type":"authentication_error"},"type":"error"}` (Anthropic envelope).
- Authentication is checked before routing: an unknown path also answers 401.
- `POST /asr/transcribe` without multipart -> 415 `unsupported_media_type` with a
`sessionId`; `GET` -> 404 with an empty body.
- `wss://api.meta.ai/v1/asr/realtime`: upgrade answers 101; with no handshake
frame the server sends `{"type":"error","message":"Handshake timeout",...,"errorType":"invalid_request_error"}`.
- The opt-in live test (`MOLO_LIVE_NET=1`) re-runs the status and 401 probes
through the module and, with a key, lists models (read only). Paid paths
(images, SAM, transcription, completions) have no live test in this round:
no key exists.
## Unverified
- Every success body below the documented schemas was only checked against the
docs examples, never against a live keyed call: no key exists.
- The `GET /status` model_statuses array element shape (live probe returned
`[]`; the docs give no schema for `StatusResponse`).
- `image_generation_call` output items (documented in prose, absent from the
Responses schema union): `result` is read nowhere in code.
- `custom` and `namespace` tool shapes (prose only).
- A verbatim Responses SSE transcript is not published: the stream fold test
uses event names and fields from the schemas with a constructed order.
- The empty-object receipt for `approval/request` / `userInput/request`
(schema `RequestReceipt` lists no fields).
- MSP requests other than the handshake were never run against a `muse` host.
- Mask decoding is verified against two vendor conformance cases (one lossless,
one `one_bit`), not against live output.
- Realtime WebSocket beyond the handshake timeout error was not driven.
- Retry-After and `code`/`param` of error bodies (see Gaps).
## Gaps
- `llm::ApiError` keeps only `type` (preferring Meta's category) and drops the
error `code` (`model_not_found`, `invalid_api_key`, `payload_too_large`...),
`param` and the `Retry-After` header. Owner: `src/llm/error.cpp`
(foundation, not editable here). `RetryPolicy` therefore works from the status
only.
- `ApiClient::stream` is POST-only; streaming retrieve is read whole, then
decoded. `ApiClient` cannot send an unauthenticated request; `/status` sends
a placeholder bearer (`MetaClient::kPlaceholderKey`).
- `MetaChatModel::stream` cannot return the encrypted reasoning for replay
(`ChatChunk` has no signature field); use `generate` or `previous_response_id`
for reasoning continuity across streamed turns.
- Batch API, fine-tuning and embeddings do not exist on this platform per the
docs; nothing is implemented for them.
- Muse Spark Open and Muse Glimmer are open weights with no Model API surface;
they are in the catalog only. Self-hosted serving is out of scope.
- Realtime audio pacing (real time or 1008) is the caller's job.
+65
View File
@@ -0,0 +1,65 @@
#include "meta/calls.h"
#include <cstring>
#include "llm/api_client.h"
namespace molo::meta {
// CALLS-BEGIN (parsed by tools/check_spec.py; one row per line)
static const CallSpec kCalls[] = {
{Call::kResponsesCreate, "POST", "/responses", "responses/create-response", "responses", "CreateResponse", "Response", "model,input,instructions,previous_response_id,store,stream,background,max_output_tokens,max_tool_calls,temperature,top_p,frequency_penalty,presence_penalty,reasoning,text,tools,parallel_tool_calls,include,truncation,metadata,safety_identifier,prompt_cache_key,prompt_cache_retention,service_tier", ""},
{Call::kResponsesRetrieve, "GET", "/responses/{response_id}", "responses/retrieve-response", "responses", "", "Response", "", "stream,starting_after"},
{Call::kResponsesDelete, "DELETE", "/responses/{response_id}", "responses/delete-response", "responses", "", "DeleteResponseResponse", "", ""},
{Call::kResponsesCancel, "POST", "/responses/{response_id}/cancel", "responses/cancel-response", "responses", "", "CancelResponseResponse", "", ""},
{Call::kResponsesInputTokens, "POST", "/responses/input_tokens", "responses/count-input-tokens", "responses", "TokenCountsBody", "TokenCountsResource", "model,input,instructions,previous_response_id,tools,parallel_tool_calls,reasoning,text,truncation", ""},
{Call::kChatCompletionsCreate, "POST", "/chat/completions", "chat-completions/create-chat-completion", "chat-completions", "CreateChatCompletionRequest", "CreateChatCompletionResponse", "model,messages,temperature,top_p,frequency_penalty,presence_penalty,max_completion_tokens,stream,stream_options,reasoning_effort,seed,prompt_cache_retention,safety_identifier,prompt_cache_key,tools,tool_choice,parallel_tool_calls,response_format,metadata,service_tier,user", ""},
{Call::kMessagesCreate, "POST", "/messages", "messages/create-message", "messages", "AnthropicMessageRequest", "AnthropicMessage", "model,messages,max_tokens,system,tools,tool_choice,metadata,temperature,top_p,thinking,service_tier,output_config,stream,cache_control,context_management", ""},
{Call::kMessagesCountTokens, "POST", "/messages/count_tokens", "messages/count-tokens", "messages", "AnthropicCountTokensRequest", "AnthropicCountTokensResponse", "model,messages,system,tools,tool_choice,metadata,temperature,top_p,thinking,output_config", ""},
{Call::kFilesUpload, "POST", "/files", "files/upload-file", "files", "CreateFileRequest", "FileObject", "file,purpose,expires_after", ""},
{Call::kFilesList, "GET", "/files", "files/list-files", "files", "", "ListFilesResponse", "", "purpose,limit,order,after"},
{Call::kFilesRetrieve, "GET", "/files/{file_id}", "files/retrieve-file", "files", "", "FileObject", "", ""},
{Call::kFilesContent, "GET", "/files/{file_id}/content", "files/retrieve-file-content", "files", "", "", "", ""},
{Call::kFilesDelete, "DELETE", "/files/{file_id}", "files/delete-file", "files", "", "DeleteFileResponse", "", ""},
{Call::kImagesGenerate, "POST", "/images/generations", "images/create-image", "images", "CreateImageRequest", "ImagesResponse", "model,prompt,n,size,response_format,moderation,output_format,reasoning_strength,user,stream,tool_enablement", ""},
{Call::kImagesEdit, "POST", "/images/edits", "images/edit-image", "images", "EditImageBodyJsonParam", "ImagesResponse", "model,prompt,images,n,size,response_format,output_format,reasoning_strength,moderation,stream,user,tool_enablement", ""},
{Call::kModelsList, "GET", "/models", "models/list-models", "models", "", "ListModelsResponse", "", ""},
{Call::kModelsRetrieve, "GET", "/models/{model}", "models/retrieve-model", "models", "", "Model", "", ""},
{Call::kStatus, "GET", "/status", "status", "", "", "", "", ""},
{Call::kAsrTranscribe, "POST", "/asr/transcribe", "voice/transcribe", "voice", "TranscribeRequest", "TranscribeResponse", "model,audioEncoding,mode,keywords,languageBias,partialMode,emitAudioProgress", "sessionId"},
{Call::kAsrRealtime, "WS", "/asr/realtime", "voice/realtime", "voice", "RealtimeHandshakeRequest", "RealtimeHandshakeResponse", "authorization,audioEncoding,model,mode,partialMode,emitAudioProgress,keywords,languageBias", "sessionId"},
};
// CALLS-END
const CallSpec& spec_of(Call call) {
for (const CallSpec& s : kCalls)
if (s.call == call) return s;
throw llm::ApiError::invalid("meta", "unknown call");
}
std::size_t call_count() { return sizeof(kCalls) / sizeof(kCalls[0]); }
const CallSpec& spec_at(std::size_t index) {
if (index >= call_count()) throw llm::ApiError::invalid("meta", "call index out of range");
return kCalls[index];
}
clients::Method method_of(Call call) {
const char* m = spec_of(call).http_method;
if (!std::strcmp(m, "POST")) return clients::Method::kPost;
if (!std::strcmp(m, "DELETE")) return clients::Method::kDelete;
if (!std::strcmp(m, "GET")) return clients::Method::kGet;
throw llm::ApiError::invalid("meta", "call is not an HTTP request");
}
std::string path_for(Call call, const std::string& id) {
std::string path = spec_of(call).path;
std::size_t open = path.find('{');
if (open == std::string::npos) return path;
if (!llm::Service::valid_id(id)) throw llm::ApiError::invalid("meta", "bad id for " + path);
std::size_t close = path.find('}', open);
path.replace(open, close - open + 1, id);
return path;
}
} // namespace molo::meta
+48
View File
@@ -0,0 +1,48 @@
#pragma once
// The ONE table of Meta Model API calls this module makes. Every row is
// checked against the published docs by tools/check_spec.py (the docs are
// prose plus schema tables, there is no OpenAPI document). Code builds every
// path from this table, never from a literal.
#include <cstddef>
#include <string>
#include "clients/request.h"
namespace molo::meta {
enum class Call {
kResponsesCreate, kResponsesRetrieve, kResponsesDelete, kResponsesCancel,
kResponsesInputTokens,
kChatCompletionsCreate,
kMessagesCreate, kMessagesCountTokens,
kFilesUpload, kFilesList, kFilesRetrieve, kFilesContent, kFilesDelete,
kImagesGenerate, kImagesEdit,
kModelsList, kModelsRetrieve,
kStatus,
kAsrTranscribe, kAsrRealtime,
};
struct CallSpec {
Call call;
const char* http_method; // GET POST DELETE, or WS for the realtime socket
const char* path; // template under the base URL, "{id}" placeholders
const char* operation; // docs page slug under /docs/api-reference/
const char* schema_group; // docs schemas page: /docs/api-reference/<group>/schemas
const char* request_schema; // schema name in the group, "" when no body
const char* response_schema; // schema name, "" when raw bytes or a socket
const char* body_fields; // comma list: request body fields we send
const char* query_params; // comma list: query parameters we send
};
const CallSpec& spec_of(Call call);
std::size_t call_count();
const CallSpec& spec_at(std::size_t index);
clients::Method method_of(Call call);
// Fills the first "{...}" placeholder of the call's path with `id` after
// validating it as a single path segment. Throws llm::ApiError::invalid.
std::string path_for(Call call, const std::string& id = "");
} // namespace molo::meta
+58
View File
@@ -0,0 +1,58 @@
#pragma once
// Static ceilings and documented limits for the Meta Model API module. Every
// value is a constant you can read; nothing is autotuned. Documented vendor
// limits are marked "docs"; the rest are our own floors and ceilings.
#include <cstddef>
namespace molo::meta {
struct MetaCaps {
// Request shaping (docs: error-handling, protocols/responses).
static constexpr int kMinOutputTokens = 16; // docs: max_output_tokens minimum
static constexpr std::size_t kMaxInputItems = 4096; // ours
static constexpr std::size_t kMaxTools = 256; // ours (matches llm::Caps)
static constexpr std::size_t kMaxIdBytes = 256; // ours
static constexpr std::size_t kMaxCallIdBytes = 64; // docs: call_id 1..64
static constexpr std::size_t kMaxSafetyIdBytes = 64; // docs
static constexpr std::size_t kMaxModelBytes = 128; // ours
static constexpr std::size_t kMaxPromptBytes = 4u * 1024u * 1024u; // ours
static constexpr std::size_t kMaxBackgroundBodyBytes = 1048576; // docs: 1 MiB
static constexpr std::size_t kMaxInlineBytes = 50000000; // docs: 50 MB inline media
static constexpr std::size_t kMaxImageCount = 50; // docs: 50 images per request
static constexpr int kMaxImagesPerCall = 10; // docs: n 1..10
static constexpr std::size_t kMaxMetadataPairs = 16; // ours
// Files (docs: file-handling). The vendor allows 1 GiB; llm::Caps stops at 512 MiB.
static constexpr std::size_t kMaxUploadBytes = 512u * 1024u * 1024u;
static constexpr long kMinExpireSeconds = 3600; // docs
static constexpr long kMaxExpireSeconds = 2592000; // docs
static constexpr int kListPageLimit = 1000; // ours (docs allow 10000)
static constexpr std::size_t kMaxListPages = 20; // ours
// Speech (docs: speech-to-text).
static constexpr std::size_t kMaxTranscribeBytes = 32u * 1000u * 1000u; // docs: 32 MB
static constexpr long kMaxTranscribeMs = 10L * 60L * 1000L; // docs: 10 minutes
static constexpr std::size_t kMaxKeywords = 256; // ours
static constexpr int kRealtimeHandshakeMs = 10000; // docs: first frame within 10 s
static constexpr int kConnectTimeoutMs = 10000; // ours
static constexpr int kRealtimeIdleMs = 30000; // ours
static constexpr std::size_t kMaxRealtimeAudioFrame = 1u * 1024u * 1024u; // ours
static constexpr std::size_t kMaxRealtimeEvents = 100000; // ours
static constexpr std::size_t kMaxStreamEvents = 1000000; // ours: events folded per HTTP stream
// Retry (docs: pricing-rate-limits, error-handling).
static constexpr int kMaxRetries = 5;
static constexpr int kBackoffBaseMs = 500; // docs: start near 500 ms
static constexpr int kBackoffCapMs = 30000; // ours
// Segmentation output parsing.
static constexpr std::size_t kMaxSamLines = 100000; // ours
static constexpr std::size_t kMaxSamRecords = 1000000; // ours
static constexpr std::size_t kMaxMaskPixels = 64u * 1024u * 1024u; // ours
// Muse Code session protocol (MSP).
static constexpr std::size_t kMspFrameLimit = 10485760; // MSP transcript manifest: frameLimitBytes
static constexpr std::size_t kMspMaxPending = 64; // ours
static constexpr std::size_t kMspMaxQueued = 4096; // ours
static constexpr int kMspRequestTimeoutMs = 120000; // ours
static constexpr int kMspSpawnTimeoutMs = 10000; // ours
};
} // namespace molo::meta
+64
View File
@@ -0,0 +1,64 @@
#include "meta/catalog.h"
namespace molo::meta {
// CATALOG-BEGIN (parsed by tools/check_spec.py; one row per line)
// id, family, kind, hosted, contributor, supports_max, context, cached, input, output, other_price, rpm, tpm, note
static const ModelInfo kModels[] = {
{"muse-wonder-1.0", "Muse Wonder", ModelKind::kText, true, false, true, 1048576, 0.20, 4.00, 20.00, "", 3000, 4000000, "preview; Standard mode only on Model API"},
{"muse-spark-1.3", "Muse Spark", ModelKind::kText, true, false, true, 1048576, 0.15, 1.25, 4.25, "", 3000, 4000000, "recommended for new work"},
{"muse-spark-1.2", "Muse Spark", ModelKind::kText, true, false, false, 1048576, 0.15, 1.25, 4.25, "", 3000, 4000000, "previous version"},
{"muse-spark-1.1", "Muse Spark", ModelKind::kText, true, false, false, 1048576, 0.15, 1.25, 4.25, "", 3000, 4000000, "original version, Standard mode only"},
{"muse-spark-1.3-contributor", "Muse Spark", ModelKind::kText, true, true, false, 1048576, 0.002, 0.10, 0.20, "", 150, 3000000, "Contributor mode: content may train Meta models"},
{"muse-spark-1.2-contributor", "Muse Spark", ModelKind::kText, true, true, false, 1048576, 0.002, 0.10, 0.20, "", 150, 3000000, "Contributor mode: content may train Meta models"},
{"muse-image-1.0", "Muse Image", ModelKind::kImage, true, false, false, 0, 0, 0, 0, "$0.01 per generated image", 150, 0, "generation and editing"},
{"muse-voice-transcribe-1.0", "Muse Voice Transcribe", ModelKind::kSpeech, true, false, false, 0, 0, 0, 0, "$0.18 per hour of audio", 0, 0, "256 concurrent streams, 16000 streams per hour"},
{"sam-3.1", "Segment Anything Model", ModelKind::kSegmenter, true, false, false, 0, 0, 0, 0, "$2.50 per 1000 images, $0.20 per 1000 video frames", 3000, 0, "video bills at least 50 frames"},
{"muse-spark-open", "Muse Spark Open", ModelKind::kOpenWeight, false, false, false, 0, 0, 0, 0, "open weights, self-hosted", 0, 0, "816B MoE, 42B active; no Model API"},
{"muse-glimmer", "Muse Glimmer", ModelKind::kOpenWeight, false, false, false, 0, 0, 0, 0, "open weights, self-hosted", 0, 0, "distilled from Muse Spark; no Model API"},
};
// CATALOG-END
std::size_t ModelCatalog::size() { return sizeof(kModels) / sizeof(kModels[0]); }
const ModelInfo& ModelCatalog::at(std::size_t index) { return kModels[index % size()]; }
const ModelInfo* ModelCatalog::find(const std::string& id) {
for (const ModelInfo& m : kModels)
if (id == m.id) return &m;
return nullptr;
}
bool ModelCatalog::is_text_model(const std::string& id) {
const ModelInfo* m = find(id);
return m && m->kind == ModelKind::kText;
}
bool ModelCatalog::accepts_max_effort(const std::string& id) {
const ModelInfo* m = find(id);
return m && m->supports_max;
}
const char* ModelCatalog::kind_name(ModelKind kind) {
switch (kind) {
case ModelKind::kText: return "text";
case ModelKind::kImage: return "image";
case ModelKind::kSpeech: return "speech";
case ModelKind::kSegmenter: return "segmentation";
case ModelKind::kOpenWeight: return "open-weight";
}
return "unknown";
}
const std::vector<std::string>& ReasoningEfforts::accepted() {
static const std::vector<std::string> v = {"minimal", "low", "medium", "high", "xhigh", "max"};
return v;
}
bool ReasoningEfforts::valid(const std::string& value) {
for (const std::string& e : accepted())
if (e == value) return true;
return false;
}
} // namespace molo::meta
+58
View File
@@ -0,0 +1,58 @@
#pragma once
// Static catalog of the Meta models named by the docs (models, pricing and
// rate-limit pages). It is a specification table: request validation reads
// it (for example which models accept reasoning effort "max") and the demo
// prints it. It is NOT the live model list: ask ModelsService for that.
#include <cstddef>
#include <string>
#include <vector>
namespace molo::meta {
enum class ModelKind {
kText, // Muse Wonder, Muse Spark: Responses, Chat Completions, Messages
kImage, // Muse Image: Responses, Images
kSpeech, // Muse Voice Transcribe: ASR file and realtime
kSegmenter, // Segment Anything Model 3.1: Responses
kOpenWeight, // Muse Spark Open, Muse Glimmer: self-hosted, no Model API
};
struct ModelInfo {
const char* id;
const char* family; // "Muse Wonder", "Muse Spark", ...
ModelKind kind;
bool hosted; // callable on api.meta.ai
bool contributor; // Contributor mode variant (content may train models)
bool supports_max; // reasoning effort "max"
long context_tokens; // 0 when not applicable
double cached_per_m; // USD per 1M cached input tokens, 0 when n/a
double input_per_m; // USD per 1M input tokens, 0 when n/a
double output_per_m; // USD per 1M output tokens, 0 when n/a
const char* other_price; // non-token pricing, "" when token priced
long rpm; // requests per minute, 0 = none documented
long tpm; // tokens per minute, 0 = none documented
const char* note;
};
class ModelCatalog {
public:
static std::size_t size();
static const ModelInfo& at(std::size_t index);
// nullptr when the id is not in the docs.
static const ModelInfo* find(const std::string& id);
static bool is_text_model(const std::string& id);
// True when the id is a documented text model that accepts "max".
static bool accepts_max_effort(const std::string& id);
static const char* kind_name(ModelKind kind);
};
// Documented reasoning efforts. "none" is documented as rejected by Muse
// models, so it is listed separately and never offered.
struct ReasoningEfforts {
static const std::vector<std::string>& accepted(); // minimal..max
static bool valid(const std::string& value);
};
} // namespace molo::meta
+293
View File
@@ -0,0 +1,293 @@
#include "meta/chat_completions.h"
#include "meta/calls.h"
#include "meta/caps.h"
#include "meta/check.h"
#include "meta/schema_check.h"
namespace molo::meta {
namespace {
Json msg(const char* role, const Json& content) {
Json o = Json::object();
o.set("role", Json(role));
o.set("content", content);
return o;
}
Json part(const char* type) {
Json o = Json::object();
o.set("type", Json(type));
return o;
}
} // namespace
Json ChatMessage::developer(const std::string& t) { return msg("developer", Json(t)); }
Json ChatMessage::system(const std::string& t) { return msg("system", Json(t)); }
Json ChatMessage::user(const std::string& t) { return msg("user", Json(t)); }
Json ChatMessage::user_parts(const std::vector<Json>& parts) {
Json a = Json::array();
for (const Json& p : parts) a.push(p);
return msg("user", a);
}
Json ChatMessage::assistant(const std::string& t) { return msg("assistant", Json(t)); }
Json ChatMessage::assistant_tool_calls(const std::string& t, const std::vector<Json>& calls) {
Json m = msg("assistant", t.empty() ? Json() : Json(t));
Json a = Json::array();
for (const Json& c : calls) a.push(c);
m.set("tool_calls", a);
return m;
}
Json ChatMessage::tool_result(const std::string& id, const std::string& content) {
Check::call_id(id);
Json m = msg("tool", Json(content));
m.set("tool_call_id", Json(id));
return m;
}
Json ChatMessage::text_part(const std::string& t) {
Json p = part("text");
p.set("text", Json(t));
return p;
}
Json ChatMessage::image_part(const std::string& url, const std::string& detail) {
Check::non_empty("image url", url);
if (!detail.empty()) Check::one_of("detail", detail, {"auto", "low", "high"});
Json p = part("image_url");
Json i = Json::object();
i.set("url", Json(url));
JsonX::set_if(i, "detail", detail);
p.set("image_url", i);
return p;
}
Json ChatMessage::video_part(const std::string& url, double fps) {
Check::non_empty("video url", url);
Json p = part("video_url");
Json v = Json::object();
v.set("url", Json(url));
if (fps > 0) v.set("fps", Json(fps));
p.set("video_url", v);
return p;
}
Json ChatMessage::file_part(const std::string& filename, const std::string& data_url,
const std::string& file_id) {
if (data_url.empty() && file_id.empty()) Check::fail("file part needs file_data or file_id");
Json p = part("file");
Json f = Json::object();
JsonX::set_if(f, "filename", filename);
JsonX::set_if(f, "file_data", data_url);
if (!file_id.empty()) {
Check::segment("file_id", file_id);
f.set("file_id", Json(file_id));
}
p.set("file", f);
return p;
}
Json ChatMessage::audio_part(const std::string& format, const std::string& b64) {
Check::one_of("format", format, {"wav", "mp3"});
Check::non_empty("audio data", b64);
Json p = part("input_audio");
Json a = Json::object();
a.set("data", Json(b64));
a.set("format", Json(format));
p.set("input_audio", a);
return p;
}
Json ChatMessage::tool_call(const std::string& id, const std::string& name,
const std::string& arguments) {
Check::call_id(id);
Check::function_name(name);
Json c = Json::object();
c.set("id", Json(id));
c.set("type", Json("function"));
Json f = Json::object();
f.set("name", Json(name));
f.set("arguments", Json(arguments.empty() ? "{}" : arguments));
c.set("function", f);
return c;
}
Json ChatMessage::function_tool(const std::string& name, const std::string& description,
const Json& parameters, bool strict) {
Check::function_name(name);
if (parameters.is_object()) SchemaCheck::check(parameters, strict);
Json f = Json::object();
f.set("name", Json(name));
JsonX::set_if(f, "description", description);
f.set("parameters", parameters.is_object() ? parameters : Json::object());
if (strict) f.set("strict", Json(true));
Json t = Json::object();
t.set("type", Json("function"));
t.set("function", f);
return t;
}
Json ChatCompletionRequest::format_json_object() {
Json f = Json::object();
f.set("type", Json("json_object"));
return f;
}
Json ChatCompletionRequest::format_json_schema(const std::string& name, const Json& schema,
bool strict, const std::string& description) {
Check::non_empty("name", name);
SchemaCheck::check(schema, strict);
Json js = Json::object();
js.set("name", Json(name));
js.set("schema", schema);
if (strict) js.set("strict", Json(true));
JsonX::set_if(js, "description", description);
Json f = Json::object();
f.set("type", Json("json_schema"));
f.set("json_schema", js);
return f;
}
void ChatCompletionRequest::validate() const {
Check::model_id(model);
if (messages.empty() || messages.size() > MetaCaps::kMaxInputItems)
Check::fail("messages needs 1.." + std::to_string(MetaCaps::kMaxInputItems) + " entries");
for (const Json& m : messages) {
Check::one_of("message role", JsonX::str(m, "role"),
{"developer", "system", "user", "assistant", "tool"});
}
if (tools.size() > MetaCaps::kMaxTools) Check::fail("too many tools");
for (const Json& t : tools) {
if (JsonX::str(t, "type") != "function")
Check::fail("chat completions accepts function tools only (custom tools are Responses-only)");
if (const Json* f = JsonX::obj(t, "function")) Check::function_name(JsonX::str(*f, "name"));
}
if (max_completion_tokens && *max_completion_tokens < 1)
Check::fail("max_completion_tokens must be positive");
if (temperature) Check::temperature(*temperature);
if (top_p) Check::top_p(*top_p);
if (frequency_penalty) Check::penalty("frequency_penalty", *frequency_penalty);
if (presence_penalty) Check::penalty("presence_penalty", *presence_penalty);
Check::reasoning_effort(model, reasoning_effort);
Check::safety_identifier(safety_identifier);
Check::prompt_cache_retention(prompt_cache_retention);
if (!service_tier.empty())
Check::one_of("service_tier", service_tier, {"auto", "default", "flex", "scale", "priority"});
if (metadata.size() > MetaCaps::kMaxMetadataPairs) Check::fail("too many metadata pairs");
if (response_format.is_object()) {
std::string type = JsonX::str(response_format, "type");
Check::one_of("response_format.type", type, {"text", "json_object", "json_schema"});
if (type == "json_schema") {
const Json* js = JsonX::obj(response_format, "json_schema");
const Json* s = js ? JsonX::obj(*js, "schema") : nullptr;
if (!s) Check::fail("response_format json_schema needs json_schema.schema");
SchemaCheck::check(*s, JsonX::flag(*js, "strict"));
}
}
// Fields the docs reject on this endpoint must not arrive through `extra`.
if (extra.is_object())
for (const char* banned : {"n", "stop", "logit_bias", "prediction", "verbosity",
"web_search_options", "modalities", "audio", "logprobs", "text",
"max_output_tokens"})
if (extra.has(banned)) Check::fail(std::string(banned) + " is rejected by chat completions");
}
Json ChatCompletionRequest::to_json(bool stream) const {
validate();
Json b = Json::object();
b.set("model", Json(model));
Json m = Json::array();
for (const Json& x : messages) m.push(x);
b.set("messages", m);
if (!tools.empty()) {
Json t = Json::array();
for (const Json& x : tools) t.push(x);
b.set("tools", t);
}
if (response_format.is_object()) b.set("response_format", response_format);
JsonX::set_if(b, "reasoning_effort", reasoning_effort);
if (max_completion_tokens) b.set("max_completion_tokens", Json(*max_completion_tokens));
if (temperature) b.set("temperature", Json(*temperature));
if (top_p) b.set("top_p", Json(*top_p));
if (frequency_penalty) b.set("frequency_penalty", Json(*frequency_penalty));
if (presence_penalty) b.set("presence_penalty", Json(*presence_penalty));
if (seed) b.set("seed", Json(*seed));
if (parallel_tool_calls) b.set("parallel_tool_calls", Json(*parallel_tool_calls));
JsonX::set_if(b, "safety_identifier", safety_identifier);
JsonX::set_if(b, "prompt_cache_key", prompt_cache_key);
JsonX::set_if(b, "prompt_cache_retention", prompt_cache_retention);
JsonX::set_if(b, "service_tier", service_tier);
if (!metadata.empty()) {
Json md = Json::object();
for (const auto& [k, v] : metadata) md.set(k, Json(v));
b.set("metadata", md);
}
if (stream) {
b.set("stream", Json(true));
if (include_usage) {
Json so = Json::object();
so.set("include_usage", Json(true));
b.set("stream_options", so);
}
}
JsonX::merge(b, extra);
return b;
}
void ChatChunkAccumulator::feed(const Json& chunk) {
id_ = JsonX::str(chunk, "id", id_);
model_ = JsonX::str(chunk, "model", model_);
if (const Json* u = JsonX::obj(chunk, "usage")) {
usage_ = MetaUsage::from_chat(*u);
has_usage_ = true;
}
const Json* choices = JsonX::arr(chunk, "choices");
if (!choices || choices->as_array().empty()) return;
const Json& c = choices->as_array()[0];
std::string fr = JsonX::str(c, "finish_reason");
if (!fr.empty()) finish_reason_ = fr;
const Json* d = JsonX::obj(c, "delta");
if (!d) return;
text_ += JsonX::str(*d, "content");
reasoning_ += JsonX::str(*d, "reasoning_content");
if (const Json* tcs = JsonX::arr(*d, "tool_calls"))
for (const Json& tc : tcs->as_array()) {
long idx = JsonX::num(tc, "index", 0);
ChatToolCallState* st = nullptr;
for (ChatToolCallState& s : calls_)
if (s.index == idx) st = &s;
if (!st) {
calls_.push_back({});
st = &calls_.back();
st->index = idx;
}
st->id = JsonX::str(tc, "id", st->id);
if (const Json* f = JsonX::obj(tc, "function")) {
st->name = JsonX::str(*f, "name", st->name);
st->arguments += JsonX::str(*f, "arguments");
}
}
}
Json ChatCompletionsService::create(const ChatCompletionRequest& request) {
Json body = request.to_json(false);
return client_->post(path_for(Call::kChatCompletionsCreate), body);
}
ChatChunkAccumulator ChatCompletionsService::stream(const ChatCompletionRequest& request,
const ChunkSink& sink) {
Json body = request.to_json(true);
ChatChunkAccumulator acc;
std::size_t seen = 0;
client_->stream(path_for(Call::kChatCompletionsCreate), body, [&](const llm::SseEvent& e) {
if (++seen > MetaCaps::kMaxStreamEvents) return false;
acc.feed(e.data);
return sink ? sink(e.data) : true;
});
return acc;
}
} // namespace molo::meta
+114
View File
@@ -0,0 +1,114 @@
#pragma once
// POST /chat/completions: the OpenAI-compatible messages-array endpoint for
// the Muse Spark and Muse Wonder text models. It does not carry reasoning
// across turns (reasoning_content is redacted for external keys); use the
// Responses API for that. Rules enforced before any traffic (docs:
// protocols/chat-completions): n is always 1, stop / logit_bias / prediction /
// verbosity / web_search_options / modalities are not offered, tool_choice is
// only "auto", reasoning_effort "none" is rejected, response_format and tools
// pass the schema pre-flight.
#include <functional>
#include <optional>
#include <string>
#include <vector>
#include "llm/api_client.h"
#include "meta/jsonx.h"
#include "meta/usage.h"
namespace molo::meta {
class ChatMessage {
public:
static Json developer(const std::string& text);
static Json system(const std::string& text); // same level as developer
static Json user(const std::string& text);
static Json user_parts(const std::vector<Json>& parts);
static Json assistant(const std::string& text);
static Json assistant_tool_calls(const std::string& text, const std::vector<Json>& tool_calls);
static Json tool_result(const std::string& tool_call_id, const std::string& content);
// Content parts.
static Json text_part(const std::string& text);
static Json image_part(const std::string& url, const std::string& detail = "");
static Json video_part(const std::string& url, double fps = 0);
static Json file_part(const std::string& filename, const std::string& file_data_url,
const std::string& file_id = "");
static Json audio_part(const std::string& format, const std::string& base64_data);
// {id, type:"function", function:{name, arguments}}
static Json tool_call(const std::string& id, const std::string& name,
const std::string& arguments_json);
// Tool definition: {type:"function", function:{name, description, parameters, strict}}
static Json function_tool(const std::string& name, const std::string& description,
const Json& parameters, bool strict = false);
};
struct ChatCompletionRequest {
std::string model;
std::vector<Json> messages;
std::vector<Json> tools;
Json response_format; // null, or {type: json_object | json_schema, json_schema:{...}}
std::string reasoning_effort; // minimal..max
std::optional<int> max_completion_tokens;
std::optional<double> temperature;
std::optional<double> top_p;
std::optional<double> frequency_penalty;
std::optional<double> presence_penalty;
std::optional<long> seed;
std::optional<bool> parallel_tool_calls;
std::string safety_identifier;
std::string prompt_cache_key;
std::string prompt_cache_retention;
std::string service_tier;
bool include_usage = true; // stream_options.include_usage when streaming
std::vector<std::pair<std::string, std::string>> metadata;
Json extra;
void validate() const;
Json to_json(bool stream = false) const;
static Json format_json_object();
static Json format_json_schema(const std::string& name, const Json& schema, bool strict = false,
const std::string& description = "");
};
struct ChatToolCallState {
long index = 0;
std::string id, name, arguments;
};
// Folds chat.completion.chunk objects into text, reasoning_content, tool calls,
// finish_reason and usage.
class ChatChunkAccumulator {
public:
void feed(const Json& chunk);
const std::string& text() const { return text_; }
const std::string& reasoning() const { return reasoning_; }
const std::string& finish_reason() const { return finish_reason_; }
const std::string& id() const { return id_; }
const std::string& model() const { return model_; }
const std::vector<ChatToolCallState>& tool_calls() const { return calls_; }
bool has_usage() const { return has_usage_; }
MetaUsage usage() const { return usage_; }
private:
std::string text_, reasoning_, finish_reason_, id_, model_;
std::vector<ChatToolCallState> calls_;
MetaUsage usage_;
bool has_usage_ = false;
};
class ChatCompletionsService : public llm::Service {
public:
using ChunkSink = std::function<bool(const Json&)>; // false: stop
explicit ChatCompletionsService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
Json create(const ChatCompletionRequest& request);
ChatChunkAccumulator stream(const ChatCompletionRequest& request,
const ChunkSink& sink = nullptr);
};
} // namespace molo::meta
+285
View File
@@ -0,0 +1,285 @@
#include "meta/chat_model.h"
#include <map>
#include "meta/check.h"
#include "meta/response_input.h"
#include "security/crypto/Base64.h"
namespace molo::meta {
namespace {
std::string image_url_of(const llm::ImageData& im) {
if (!im.url.empty()) return im.url;
if (im.bytes.empty()) Check::fail("image has neither url nor bytes");
return "data:" + im.mime + ";base64," + security::crypto::Base64::encode(im.bytes);
}
void common_checks(const llm::ChatRequest& r) {
if (!r.stop.empty()) Check::fail("stop sequences are not supported by Muse models");
if (!r.tool_choice.empty() && r.tool_choice != "auto")
Check::fail("tool_choice must be auto (the only value Meta accepts)");
}
std::string finish_of(const std::string& v) {
if (v == "stop" || v == "length" || v == "tool_calls" || v == "content_filter") return v;
if (v == "function_call") return "tool_calls";
return v;
}
} // namespace
MetaChatModel::MetaChatModel(std::shared_ptr<llm::ApiClient> client, Endpoint endpoint)
: responses_(client), chat_(client), endpoint_(endpoint) {}
ResponseRequest MetaChatModel::to_response_request(const llm::ChatRequest& r) const {
common_checks(r);
ResponseRequest q;
q.model = r.model;
q.instructions = r.system;
Json input = Json::array();
for (const llm::Message& m : r.messages) {
if (m.role == llm::Role::kSystem) {
input.push(ResponseInput::developer(m.text()));
continue;
}
Json parts = Json::array();
bool user = m.role == llm::Role::kUser;
auto flush = [&](const char* role) {
if (parts.as_array().empty()) return;
input.push(ResponseInput::message(role, parts));
parts = Json::array();
};
for (const llm::ContentPart& p : m.parts) {
if (!p.reasoning_signature.empty()) {
flush("assistant");
input.push(ResponseInput::reasoning(p.reasoning_signature));
}
if (!p.text.empty()) {
if (user) parts.push(ResponseInput::text(p.text));
else {
Json t = Json::object();
t.set("type", Json("output_text"));
t.set("text", Json(p.text));
t.set("annotations", Json::array());
parts.push(t);
}
}
if (p.image) parts.push(ResponseInput::image_url(image_url_of(*p.image)));
if (p.tool_call) {
flush("assistant");
input.push(ResponseInput::function_call(p.tool_call->id, p.tool_call->name,
p.tool_call->arguments.dump()));
}
if (p.tool_result) {
flush(user ? "user" : "assistant");
input.push(ResponseInput::function_call_output(p.tool_result->call_id, p.tool_result->content));
}
}
flush(m.role == llm::Role::kAssistant ? "assistant" : "user");
}
q.input = input;
for (const llm::ToolSpec& t : r.tools)
q.tools.push_back(ToolBuilder::function(t.name, t.description, t.parameters));
if (r.max_tokens) q.max_output_tokens = *r.max_tokens;
q.temperature = r.temperature;
q.top_p = r.top_p;
q.reasoning_effort = r.reasoning_effort;
if (request_reasoning_summary) q.reasoning_summary = "auto";
if (r.json_mode) q.text_format = ResponseRequest::format_json_object();
if (r.response_schema.is_object())
q.text_format = ResponseRequest::format_json_schema("output", r.response_schema);
q.store = false; // stateless: replay reasoning ourselves
q.include = {"reasoning.encrypted_content"};
q.extra = r.extra;
return q;
}
ChatCompletionRequest MetaChatModel::to_chat_request(const llm::ChatRequest& r) const {
common_checks(r);
ChatCompletionRequest q;
q.model = r.model;
if (!r.system.empty()) q.messages.push_back(ChatMessage::developer(r.system));
for (const llm::Message& m : r.messages) {
if (m.role == llm::Role::kSystem) { q.messages.push_back(ChatMessage::developer(m.text())); continue; }
if (m.role == llm::Role::kAssistant) {
std::vector<Json> calls;
for (const llm::ContentPart& p : m.parts)
if (p.tool_call)
calls.push_back(ChatMessage::tool_call(p.tool_call->id, p.tool_call->name, p.tool_call->arguments.dump()));
q.messages.push_back(calls.empty() ? ChatMessage::assistant(m.text())
: ChatMessage::assistant_tool_calls(m.text(), calls));
continue;
}
std::vector<Json> parts;
bool images = false;
for (const llm::ContentPart& p : m.parts) {
if (p.tool_result) {
q.messages.push_back(ChatMessage::tool_result(p.tool_result->call_id, p.tool_result->content));
continue;
}
if (!p.text.empty()) parts.push_back(ChatMessage::text_part(p.text));
if (p.image) { parts.push_back(ChatMessage::image_part(image_url_of(*p.image))); images = true; }
}
if (parts.empty()) continue;
if (images) q.messages.push_back(ChatMessage::user_parts(parts));
else q.messages.push_back(ChatMessage::user(m.text()));
}
for (const llm::ToolSpec& t : r.tools)
q.tools.push_back(ChatMessage::function_tool(t.name, t.description, t.parameters));
if (r.max_tokens) q.max_completion_tokens = *r.max_tokens;
q.temperature = r.temperature;
q.top_p = r.top_p;
q.reasoning_effort = r.reasoning_effort;
if (r.json_mode) q.response_format = ChatCompletionRequest::format_json_object();
if (r.response_schema.is_object())
q.response_format = ChatCompletionRequest::format_json_schema("output", r.response_schema);
q.extra = r.extra;
return q;
}
llm::ChatResponse MetaChatModel::from_response(const Json& doc) {
llm::ChatResponse out;
out.raw = doc;
out.id = JsonX::str(doc, "id");
out.model = JsonX::str(doc, "model");
out.message.role = llm::Role::kAssistant;
bool calls = false, refusal = false;
if (const Json* items = JsonX::arr(doc, "output"))
for (const Json& item : items->as_array()) {
std::string type = JsonX::str(item, "type");
if (type == "message") {
if (const Json* c = JsonX::arr(item, "content"))
for (const Json& p : c->as_array()) {
if (JsonX::str(p, "type") == "output_text")
out.message.parts.push_back(llm::ContentPart::of_text(JsonX::str(p, "text")));
else if (JsonX::str(p, "type") == "refusal")
refusal = true;
}
} else if (type == "reasoning") {
llm::ContentPart part;
if (const Json* s = JsonX::arr(item, "summary"))
for (const Json& t : s->as_array()) part.reasoning += JsonX::str(t, "text");
part.reasoning_signature = JsonX::str(item, "encrypted_content");
if (!part.reasoning.empty() || !part.reasoning_signature.empty())
out.message.parts.push_back(std::move(part));
} else if (type == "function_call") {
llm::ToolCall call;
call.id = JsonX::str(item, "call_id");
call.name = JsonX::str(item, "name");
try {
call.arguments = Json::parse(JsonX::str(item, "arguments", "{}"));
} catch (const std::exception&) {
call.arguments = Json::object();
}
out.message.parts.push_back(llm::ContentPart::of_call(std::move(call)));
calls = true;
}
}
std::string status = JsonX::str(doc, "status");
out.finish_reason = calls ? "tool_calls" : refusal ? "content_filter"
: status == "incomplete" ? "length" : "stop";
if (const Json* u = JsonX::obj(doc, "usage")) out.usage = MetaUsage::from_responses(*u).to_neutral();
return out;
}
llm::ChatResponse MetaChatModel::from_chat(const Json& doc) {
llm::ChatResponse out;
out.raw = doc;
out.id = JsonX::str(doc, "id");
out.model = JsonX::str(doc, "model");
out.message.role = llm::Role::kAssistant;
if (const Json* ch = JsonX::arr(doc, "choices"); ch && !ch->as_array().empty()) {
const Json& c = ch->as_array()[0];
out.finish_reason = finish_of(JsonX::str(c, "finish_reason"));
if (const Json* m = JsonX::obj(c, "message")) {
std::string text = JsonX::str(*m, "content");
if (!text.empty()) out.message.parts.push_back(llm::ContentPart::of_text(text));
if (const Json* tcs = JsonX::arr(*m, "tool_calls"))
for (const Json& tc : tcs->as_array()) {
llm::ToolCall call;
call.id = JsonX::str(tc, "id");
if (const Json* f = JsonX::obj(tc, "function")) {
call.name = JsonX::str(*f, "name");
try {
call.arguments = Json::parse(JsonX::str(*f, "arguments", "{}"));
} catch (const std::exception&) {
call.arguments = Json::object();
}
}
out.message.parts.push_back(llm::ContentPart::of_call(std::move(call)));
}
}
}
if (const Json* u = JsonX::obj(doc, "usage")) out.usage = MetaUsage::from_chat(*u).to_neutral();
return out;
}
llm::ChatResponse MetaChatModel::generate(const llm::ChatRequest& request) {
if (endpoint_ == Endpoint::kResponses)
return from_response(responses_.create(to_response_request(request)).body);
return from_chat(chat_.create(to_chat_request(request)));
}
void MetaChatModel::stream(const llm::ChatRequest& request,
const std::function<bool(const llm::ChatChunk&)>& on_chunk) {
if (endpoint_ == Endpoint::kChatCompletions) {
chat_.stream(to_chat_request(request), [&](const Json& chunk) {
llm::ChatChunk c;
std::string finish;
if (const Json* ch = JsonX::arr(chunk, "choices"); ch && !ch->as_array().empty()) {
const Json& first = ch->as_array()[0];
finish = finish_of(JsonX::str(first, "finish_reason"));
if (const Json* d = JsonX::obj(first, "delta")) {
c.text_delta = JsonX::str(*d, "content");
c.reasoning_delta = JsonX::str(*d, "reasoning_content");
if (const Json* tcs = JsonX::arr(*d, "tool_calls"))
for (const Json& tc : tcs->as_array()) {
c.tool_call_id = JsonX::str(tc, "id");
if (const Json* f = JsonX::obj(tc, "function")) {
c.tool_call_name = JsonX::str(*f, "name");
c.tool_args_delta = JsonX::str(*f, "arguments");
}
}
}
}
c.finish_reason = finish;
if (const Json* u = JsonX::obj(chunk, "usage")) c.usage = MetaUsage::from_chat(*u).to_neutral();
return on_chunk(c);
});
return;
}
std::map<std::string, std::pair<std::string, std::string>> by_item; // item id -> call id, name
bool saw_call = false;
responses_.stream(to_response_request(request), [&](const ResponseEvent& ev) {
llm::ChatChunk c;
const Json& d = ev.data;
if (ev.type == "response.output_text.delta") c.text_delta = JsonX::str(d, "delta");
else if (ev.type == "response.reasoning_summary_text.delta") c.reasoning_delta = JsonX::str(d, "delta");
else if (ev.type == "response.output_item.added") {
const Json* item = JsonX::obj(d, "item");
if (!item || JsonX::str(*item, "type") != "function_call") return true;
saw_call = true;
by_item[JsonX::str(*item, "id")] = {JsonX::str(*item, "call_id"), JsonX::str(*item, "name")};
c.tool_call_id = JsonX::str(*item, "call_id");
c.tool_call_name = JsonX::str(*item, "name");
} else if (ev.type == "response.function_call_arguments.delta") {
auto it = by_item.find(JsonX::str(d, "item_id"));
if (it != by_item.end()) c.tool_call_id = it->second.first;
c.tool_args_delta = JsonX::str(d, "delta");
} else if (ev.type == "response.completed" || ev.type == "response.incomplete") {
const Json* r = JsonX::obj(d, "response");
c.finish_reason = saw_call ? "tool_calls" : ev.type == "response.incomplete" ? "length" : "stop";
if (r)
if (const Json* u = JsonX::obj(*r, "usage")) c.usage = MetaUsage::from_responses(*u).to_neutral();
} else if (ev.type == "response.failed" || ev.type == "error") {
throw llm::ApiError("meta", 502, "stream_failed", JsonX::str(d, "message", "response failed"));
} else {
return true;
}
return on_chunk(c);
});
}
} // namespace molo::meta
+46
View File
@@ -0,0 +1,46 @@
#pragma once
// MetaChatModel: llm::ChatModel over the Meta Model API. Default endpoint is
// the Responses API (store=false plus encrypted reasoning replay: reasoning
// parts carry the encrypted_content in ContentPart::reasoning_signature and
// are replayed verbatim). kChatCompletions maps onto /chat/completions, which
// cannot carry reasoning across turns.
// Unsupported combinations throw ApiError::invalid before any traffic: stop
// sequences, tool_choice other than "auto" (docs: only "auto").
#include <memory>
#include "llm/chat_types.h"
#include "meta/chat_completions.h"
#include "meta/responses.h"
namespace molo::meta {
class MetaChatModel : public llm::ChatModel {
public:
enum class Endpoint { kResponses, kChatCompletions };
explicit MetaChatModel(std::shared_ptr<llm::ApiClient> client,
Endpoint endpoint = Endpoint::kResponses);
std::string vendor() const override { return "meta"; }
llm::ChatResponse generate(const llm::ChatRequest& request) override;
void stream(const llm::ChatRequest& request,
const std::function<bool(const llm::ChatChunk&)>& on_chunk) override;
// Exposed for tests and the demo: the wire bodies without sending.
ResponseRequest to_response_request(const llm::ChatRequest& request) const;
ChatCompletionRequest to_chat_request(const llm::ChatRequest& request) const;
static llm::ChatResponse from_response(const Json& response);
static llm::ChatResponse from_chat(const Json& completion);
// Summaries of reasoning in the Responses endpoint (reasoning.summary=auto).
bool request_reasoning_summary = true;
private:
ResponsesService responses_;
ChatCompletionsService chat_;
Endpoint endpoint_;
};
} // namespace molo::meta
+93
View File
@@ -0,0 +1,93 @@
#include "meta/check.h"
#include <cmath>
#include "llm/api_client.h"
#include "llm/error.h"
#include "meta/caps.h"
#include "meta/catalog.h"
namespace molo::meta {
void Check::fail(const std::string& message) { throw llm::ApiError::invalid("meta", message); }
void Check::non_empty(const char* field, const std::string& value) {
if (value.empty()) fail(std::string(field) + " is required");
}
void Check::max_bytes(const char* field, const std::string& value, std::size_t cap) {
if (value.size() > cap) fail(std::string(field) + " is over " + std::to_string(cap) + " bytes");
}
void Check::range(const char* field, double value, double lo, double hi) {
if (!(value >= lo && value <= hi) || std::isnan(value))
fail(std::string(field) + " must be " + std::to_string(lo) + ".." + std::to_string(hi));
}
void Check::one_of(const char* field, const std::string& value,
std::initializer_list<const char*> allowed) {
std::string list;
for (const char* a : allowed) {
if (value == a) return;
list += (list.empty() ? "" : ", ") + std::string(a);
}
fail(std::string(field) + " must be one of " + list);
}
void Check::model_id(const std::string& model) {
non_empty("model", model);
if (model.size() > MetaCaps::kMaxModelBytes || !llm::Service::valid_id(model))
fail("model must be 1.." + std::to_string(MetaCaps::kMaxModelBytes) +
" chars of A-Z a-z 0-9 . _ : -");
}
void Check::segment(const char* what, const std::string& id) {
if (id.empty() || id.size() > MetaCaps::kMaxIdBytes || !llm::Service::valid_id(id))
fail(std::string(what) + " is not a valid id");
}
void Check::function_name(const std::string& name) {
if (name.empty() || name.size() > 128) fail("function name must be 1..128 chars");
int dots = 0;
for (char c : name) {
bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
c == '_' || c == '-' || c == '.';
if (!ok) fail("function name may use only A-Z a-z 0-9 _ - .");
if (c == '.') ++dots;
}
if (dots > 1) fail("function name may contain at most one dot");
}
void Check::reasoning_effort(const std::string& model, const std::string& effort) {
if (effort.empty()) return;
if (effort == "none") fail("reasoning effort \"none\" is rejected by Muse models");
if (!ReasoningEfforts::valid(effort))
fail("reasoning effort must be one of minimal, low, medium, high, xhigh, max");
if (effort == "max") {
const ModelInfo* m = ModelCatalog::find(model);
if (m && m->kind == ModelKind::kText && !m->supports_max)
fail("reasoning effort \"max\" is supported only by muse-wonder-1.0 and muse-spark-1.3");
}
}
void Check::call_id(const std::string& id) {
if (id.empty() || id.size() > MetaCaps::kMaxCallIdBytes) fail("call_id must be 1..64 bytes");
}
void Check::top_p(double value) {
if (!(value > 0.0 && value <= 1.0)) fail("top_p must be greater than 0 and at most 1");
}
void Check::temperature(double value) { range("temperature", value, 0.0, 2.0); }
void Check::penalty(const char* field, double value) { range(field, value, -2.0, 2.0); }
void Check::safety_identifier(const std::string& value) {
max_bytes("safety_identifier", value, MetaCaps::kMaxSafetyIdBytes);
}
void Check::prompt_cache_retention(const std::string& value) {
if (!value.empty()) one_of("prompt_cache_retention", value, {"in_memory", "24h"});
}
} // namespace molo::meta
+40
View File
@@ -0,0 +1,40 @@
#pragma once
// Local validation shared by every request builder. A failed check throws
// llm::ApiError::invalid("meta", ...) before any network traffic, naming the
// field. Rules come from the docs (error-handling, protocols/*, reasoning,
// tool-calling, structured-output); each cites its page in the SPEC.
#include <initializer_list>
#include <string>
namespace molo::meta {
class Check {
public:
[[noreturn]] static void fail(const std::string& message);
static void non_empty(const char* field, const std::string& value);
static void max_bytes(const char* field, const std::string& value, std::size_t cap);
static void range(const char* field, double value, double lo, double hi);
static void one_of(const char* field, const std::string& value,
std::initializer_list<const char*> allowed);
// 1..kMaxModelBytes of [A-Za-z0-9._:-].
static void model_id(const std::string& model);
// Response / file / model ids: one safe path segment.
static void segment(const char* what, const std::string& id);
// Function tool names: [A-Za-z0-9_.-]+, at most one dot (docs: tool-calling).
static void function_name(const std::string& name);
// Reasoning effort for a model: the documented values; "none" is rejected
// by Muse models; "max" only where the catalog says so.
static void reasoning_effort(const std::string& model, const std::string& effort);
// call_id / tool_call_id: 1..64 bytes.
static void call_id(const std::string& id);
static void top_p(double value); // 0 < p <= 1
static void temperature(double value); // 0..2
static void penalty(const char* field, double value); // -2..2
static void safety_identifier(const std::string& value);
static void prompt_cache_retention(const std::string& value); // in_memory | 24h
};
} // namespace molo::meta
+71
View File
@@ -0,0 +1,71 @@
#include "meta/client.h"
#include <algorithm>
#include "clients/transports/http_transport.h"
#include "llm/config.h"
#include "meta/caps.h"
namespace molo::meta {
std::string MetaClient::key_from_env() {
std::string key = llm::Config::get(MetaEndpoints::kKeyEnv);
if (key.empty()) key = llm::Config::get(MetaEndpoints::kAltKeyEnv);
return key;
}
llm::ApiClientConfig MetaClient::config(const std::string& base_url) {
llm::ApiClientConfig c;
c.vendor = MetaEndpoints::kVendor;
c.base_url = base_url;
c.key_env = MetaEndpoints::kKeyEnv; // named in the not-configured message
c.auth = llm::AuthStyle::kBearer;
return c;
}
std::shared_ptr<llm::ApiClient> MetaClient::with_key(const std::string& key) {
return std::make_shared<llm::ApiClient>(
std::make_unique<clients::transports::HttpTransport>(), config(), key);
}
std::shared_ptr<llm::ApiClient> MetaClient::from_env() { return with_key(key_from_env()); }
std::shared_ptr<llm::ApiClient> MetaClient::unauthenticated() {
return with_key(kPlaceholderKey);
}
bool RetryPolicy::retryable(const llm::ApiError& error) {
int s = error.http_status();
return s == 429 || s == 500 || s == 502 || s == 503;
}
int RetryPolicy::delay_ms(int attempt, unsigned seed) {
if (attempt < 0) attempt = 0;
long delay = MetaCaps::kBackoffBaseMs;
for (int i = 0; i < attempt && delay < MetaCaps::kBackoffCapMs; ++i) delay *= 2;
delay = std::min<long>(delay, MetaCaps::kBackoffCapMs);
// Deterministic jitter (xorshift on seed and attempt), 0..delay/2.
unsigned x = seed * 2654435761u + static_cast<unsigned>(attempt) * 40503u + 1u;
x ^= x << 13; x ^= x >> 17; x ^= x << 5;
long jitter = static_cast<long>(x % static_cast<unsigned>(delay / 2 + 1));
return static_cast<int>(delay + jitter);
}
RetryDecision RetryPolicy::decide(const llm::ApiError& error, int attempt, unsigned seed) {
RetryDecision d;
if (attempt >= MetaCaps::kMaxRetries) {
d.reason = "retry budget spent";
return d;
}
if (!retryable(error)) {
d.reason = error.http_status() == 504 ? "504 gateway_timeout: stream or use background"
: "not retryable";
return d;
}
d.retry = true;
d.delay_ms = delay_ms(attempt, seed);
d.reason = "transient";
return d;
}
} // namespace molo::meta
+80
View File
@@ -0,0 +1,80 @@
#pragma once
// Builds the ApiClient for the Meta Model API: bearer auth against
// https://api.meta.ai/v1. The credential is META_API_KEY, or MODEL_API_KEY
// (the name the docs' own examples use), looked up through llm::Config.
// Also the retry policy the docs prescribe (exponential backoff with jitter on
// 429 and transient 5xx, never on 504 or other 4xx).
#include <chrono>
#include <functional>
#include <memory>
#include <string>
#include <thread>
#include "llm/api_client.h"
#include "llm/error.h"
namespace molo::meta {
struct MetaEndpoints {
static constexpr const char* kVendor = "meta";
static constexpr const char* kBaseUrl = "https://api.meta.ai/v1";
static constexpr const char* kRealtimeUrl = "wss://api.meta.ai/v1/asr/realtime";
static constexpr const char* kKeyEnv = "META_API_KEY";
static constexpr const char* kAltKeyEnv = "MODEL_API_KEY";
};
class MetaClient {
public:
// META_API_KEY first, then MODEL_API_KEY; empty when neither is set.
static std::string key_from_env();
static llm::ApiClientConfig config(const std::string& base_url = MetaEndpoints::kBaseUrl);
// Real HTTPS transport, key from the environment (null key is fine: every
// keyed call then throws NotConfiguredError before any traffic).
static std::shared_ptr<llm::ApiClient> from_env();
// Same, with an explicit key (empty = not configured).
static std::shared_ptr<llm::ApiClient> with_key(const std::string& key);
// Client for the one unauthenticated endpoint (GET /status). The docs say no
// key is needed; a probe on 2026-10-24 showed a bogus bearer is ignored, so
// a fixed placeholder is sent to satisfy ApiClient's non-empty key rule.
static std::shared_ptr<llm::ApiClient> unauthenticated();
static constexpr const char* kPlaceholderKey = "unauthenticated";
};
struct RetryDecision {
bool retry = false;
int delay_ms = 0;
std::string reason;
};
class RetryPolicy {
public:
// 429 and 500/502/503 retry; 504 (gateway_timeout: the same request times
// out again), 400/401/402/403/404/413 and everything else do not.
static bool retryable(const llm::ApiError& error);
// base * 2^attempt capped, plus deterministic jitter in [0, delay/2] derived
// from `seed` so tests are exact. attempt counts from 0.
static int delay_ms(int attempt, unsigned seed);
static RetryDecision decide(const llm::ApiError& error, int attempt, unsigned seed);
};
// Runs `f`, retrying per RetryPolicy up to MetaCaps::kMaxRetries times.
// `sleep` is injectable so callers and tests control the waiting.
template <typename F>
auto call_with_retry(F&& f, unsigned seed = 1,
const std::function<void(int)>& sleep = [](int ms) {
std::this_thread::sleep_for(std::chrono::milliseconds(ms));
}) -> decltype(f()) {
for (int attempt = 0;; ++attempt) {
try {
return f();
} catch (const llm::ApiError& e) {
RetryDecision d = RetryPolicy::decide(e, attempt, seed);
if (!d.retry) throw;
sleep(d.delay_ms);
}
}
}
} // namespace molo::meta
+111
View File
@@ -0,0 +1,111 @@
#include "meta/files.h"
#include "meta/calls.h"
#include "meta/caps.h"
#include "meta/check.h"
namespace molo::meta {
FileObject FileObject::from_json(const Json& j) {
FileObject f;
f.id = JsonX::str(j, "id");
f.bytes = JsonX::num(j, "bytes");
f.created_at = JsonX::num(j, "created_at");
f.expires_at = JsonX::num(j, "expires_at");
f.filename = JsonX::str(j, "filename");
f.purpose = JsonX::str(j, "purpose");
f.status = JsonX::str(j, "status");
f.status_details = JsonX::str(j, "status_details");
return f;
}
bool FilesService::supported_mime(const std::string& mime) {
static const char* kAllowed[] = {
"image/png", "image/jpeg", "image/jpg", "image/gif", "image/webp", "image/x-icon",
"video/mp4", "audio/mpeg", "audio/wav", "application/pdf", "text/plain",
"application/json", "application/jsonl"};
for (const char* m : kAllowed)
if (mime == m) return true;
return false;
}
void FileUpload::validate() const {
Check::non_empty("filename", filename);
Check::max_bytes("filename", filename, 255);
if (filename.find('/') != std::string::npos || filename.find('\\') != std::string::npos ||
filename.find('"') != std::string::npos || filename.find('\n') != std::string::npos)
Check::fail("filename has forbidden characters");
if (!FilesService::supported_mime(mime)) Check::fail("unsupported media type " + mime);
if (bytes.empty()) Check::fail("file is empty");
if (bytes.size() > MetaCaps::kMaxUploadBytes) Check::fail("file is over the upload cap");
Check::one_of("purpose", purpose, {"user_data", "batch"});
if (purpose == "batch" && mime != "application/jsonl") Check::fail("batch uploads are application/jsonl");
if (expires_after_seconds != 0)
Check::range("expires_after.seconds", static_cast<double>(expires_after_seconds),
static_cast<double>(MetaCaps::kMinExpireSeconds),
static_cast<double>(MetaCaps::kMaxExpireSeconds));
}
FileObject FilesService::upload(const FileUpload& file) {
file.validate();
// docs: send purpose and expires_after before the file part, file last.
clients::body::MultipartBody form;
form.add_text("purpose", file.purpose);
if (file.expires_after_seconds != 0) {
form.add_text("expires_after[anchor]", "created_at");
form.add_text("expires_after[seconds]", std::to_string(file.expires_after_seconds));
}
form.add_file("file", file.filename, file.mime, file.bytes);
Json out = client_->post_multipart(path_for(Call::kFilesUpload), form);
return FileObject::from_json(out);
}
Json FilesService::list_page(const std::string& purpose, int limit, const std::string& order,
const std::string& after) {
if (limit < 1 || limit > MetaCaps::kListPageLimit)
Check::fail("limit must be 1.." + std::to_string(MetaCaps::kListPageLimit));
if (!order.empty()) Check::one_of("order", order, {"asc", "desc"});
if (!purpose.empty()) Check::one_of("purpose", purpose, {"user_data", "batch", "fine-tune", "evals"});
if (!after.empty()) Check::segment("after", after);
std::string path = path_for(Call::kFilesList) + "?limit=" + std::to_string(limit);
if (!purpose.empty()) path += "&purpose=" + purpose;
if (!order.empty()) path += "&order=" + order;
if (!after.empty()) path += "&after=" + after;
return client_->get(path);
}
FileList FilesService::list_all(const std::string& purpose, int page_size, const std::string& order) {
FileList out;
std::string after;
for (std::size_t page = 0; page < MetaCaps::kMaxListPages; ++page) {
Json doc = list_page(purpose, page_size, order, after);
++out.pages;
const Json* data = JsonX::arr(doc, "data");
if (data)
for (const Json& f : data->as_array()) out.files.push_back(FileObject::from_json(f));
out.has_more = JsonX::flag(doc, "has_more");
std::string last = JsonX::str(doc, "last_id");
if (!out.has_more || last.empty() || last == after) return out;
after = last;
}
out.truncated = out.has_more; // page cap reached while the server had more
return out;
}
FileObject FilesService::retrieve(const std::string& file_id) {
Check::segment("file_id", file_id);
return FileObject::from_json(client_->get(path_for(Call::kFilesRetrieve, file_id)));
}
std::string FilesService::content(const std::string& file_id) {
Check::segment("file_id", file_id);
return client_->request_raw(method_of(Call::kFilesContent), path_for(Call::kFilesContent, file_id))
.body;
}
Json FilesService::remove(const std::string& file_id) {
Check::segment("file_id", file_id);
return client_->del(path_for(Call::kFilesDelete, file_id));
}
} // namespace molo::meta
+67
View File
@@ -0,0 +1,67 @@
#pragma once
// Files API: upload once (up to 1 GiB by the docs, 512 MiB by llm::Caps), then
// reference a `file-...` id from input_file / input_image / input_video parts.
// Uploaded files do not expire unless expires_after is set (3600..2592000 s);
// the team storage limit is 100 GiB. The list endpoint answers a plain-text 400
// for a bad `order` (docs: error-handling), so order is validated locally.
#include <string>
#include <vector>
#include "llm/api_client.h"
#include "meta/jsonx.h"
namespace molo::meta {
struct FileObject {
std::string id;
long bytes = 0;
long created_at = 0;
long expires_at = 0; // 0 = never
std::string filename;
std::string purpose;
std::string status; // uploaded | processed | error
std::string status_details;
static FileObject from_json(const Json& j);
};
struct FileUpload {
std::string filename;
std::string mime; // documented media type
std::string bytes;
std::string purpose = "user_data"; // user_data | batch
long expires_after_seconds = 0; // 0 = no expiry; else 3600..2592000
void validate() const;
};
struct FileList {
std::vector<FileObject> files;
bool has_more = false; // true when the server has more than we returned
bool truncated = false; // true when our page cap stopped the walk
std::size_t pages = 0;
};
class FilesService : public llm::Service {
public:
explicit FilesService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
static bool supported_mime(const std::string& mime);
FileObject upload(const FileUpload& file);
// One page. order: "" | asc | desc. after: a file id cursor.
Json list_page(const std::string& purpose = "", int limit = 100, const std::string& order = "",
const std::string& after = "");
// Walks pages up to MetaCaps::kMaxListPages and reports truncation loudly.
FileList list_all(const std::string& purpose = "", int page_size = 100,
const std::string& order = "");
FileObject retrieve(const std::string& file_id);
// Raw bytes of an uploaded file.
std::string content(const std::string& file_id);
// Destructive: frees team storage. No default, no retry.
Json remove(const std::string& file_id);
};
} // namespace molo::meta
+146
View File
@@ -0,0 +1,146 @@
#include "meta/images.h"
#include <cstdlib>
#include "meta/calls.h"
#include "meta/caps.h"
#include "meta/check.h"
namespace molo::meta {
bool valid_image_size(const std::string& size) {
if (size.empty() || size == "auto") return true;
std::size_t x = size.find('x');
if (x == std::string::npos || x == 0 || x + 1 >= size.size() || size.size() > 16) return false;
for (std::size_t i = 0; i < size.size(); ++i)
if (i != x && (size[i] < '0' || size[i] > '9')) return false;
long w = std::strtol(size.substr(0, x).c_str(), nullptr, 10);
long h = std::strtol(size.substr(x + 1).c_str(), nullptr, 10);
if (w < 1 || h < 1) return false;
long hi = w > h ? w : h, lo = w > h ? h : w;
return hi <= lo * 16;
}
Json ImageRequest::image_url(const std::string& url) {
Check::non_empty("image_url", url);
Json o = Json::object();
o.set("image_url", Json(url));
return o;
}
Json ImageRequest::image_file(const std::string& file_id) {
Check::segment("file_id", file_id);
Json o = Json::object();
o.set("file_id", Json(file_id));
return o;
}
void ImageRequest::validate(bool edit) const {
Check::model_id(model);
Check::non_empty("prompt", prompt);
Check::max_bytes("prompt", prompt, MetaCaps::kMaxPromptBytes);
if (n) Check::range("n", *n, 1, MetaCaps::kMaxImagesPerCall);
if (!valid_image_size(size)) Check::fail("size must be auto or WxH with an aspect ratio of at most 16:1");
if (!response_format.empty()) Check::one_of("response_format", response_format, {"url", "b64_json"});
if (!reasoning_strength.empty()) Check::one_of("reasoning_strength", reasoning_strength, {"low", "high"});
if (edit) {
if (images.empty() || images.size() > MetaCaps::kMaxImageCount)
Check::fail("images needs 1.." + std::to_string(MetaCaps::kMaxImageCount) + " items");
} else if (!images.empty()) {
Check::fail("images belongs to edits, not generations");
}
if (extra.is_object())
for (const char* banned : {"moderation", "mask", "input_fidelity", "partial_images", "seed"})
if (extra.has(banned)) Check::fail(std::string(banned) + " is rejected by muse-image-1.0");
}
Json ImageRequest::to_json(bool edit, bool stream) const {
validate(edit);
Json b = Json::object();
b.set("model", Json(model));
b.set("prompt", Json(prompt));
if (edit) {
Json a = Json::array();
for (const Json& i : images) a.push(i);
b.set("images", a);
}
if (n) b.set("n", Json(*n));
JsonX::set_if(b, "size", size);
JsonX::set_if(b, "response_format", response_format);
JsonX::set_if(b, "output_format", output_format);
JsonX::set_if(b, "reasoning_strength", reasoning_strength);
JsonX::set_if(b, "user", user);
if (enable_image_search || enable_web_search || enable_shell) {
Json t = Json::object();
if (enable_image_search) t.set("enable_image_search", Json(*enable_image_search));
if (enable_web_search) t.set("enable_web_search", Json(*enable_web_search));
if (enable_shell) t.set("enable_shell", Json(*enable_shell));
b.set("tool_enablement", t);
}
if (stream) b.set("stream", Json(true));
JsonX::merge(b, extra);
return b;
}
ImageResult ImageResult::from_json(const Json& j) {
ImageResult r;
r.created = JsonX::num(j, "created");
r.output_format = JsonX::str(j, "output_format");
if (const Json* u = JsonX::obj(j, "usage")) r.usage = *u;
if (const Json* d = JsonX::arr(j, "data"))
for (const Json& i : d->as_array()) {
GeneratedImage x;
x.b64_json = JsonX::str(i, "b64_json");
x.url = JsonX::str(i, "url");
x.revised_prompt = JsonX::str(i, "revised_prompt");
r.data.push_back(std::move(x));
}
return r;
}
ImageResult ImagesService::generate(const ImageRequest& request) {
Json body = request.to_json(false, false);
return ImageResult::from_json(client_->post(path_for(Call::kImagesGenerate), body));
}
ImageResult ImagesService::edit(const ImageRequest& request) {
Json body = request.to_json(true, false);
return ImageResult::from_json(client_->post(path_for(Call::kImagesEdit), body));
}
ImageResult ImagesService::edit_multipart(
const ImageRequest& request, const std::vector<std::pair<std::string, std::string>>& files) {
if (files.empty() || files.size() > MetaCaps::kMaxImageCount)
Check::fail("edit needs 1.." + std::to_string(MetaCaps::kMaxImageCount) + " image files");
ImageRequest probe = request;
probe.images = {ImageRequest::image_url("multipart")}; // validate everything but the files
probe.validate(true);
clients::body::MultipartBody form;
form.add_text("model", request.model);
form.add_text("prompt", request.prompt);
if (request.n) form.add_text("n", std::to_string(*request.n));
if (!request.size.empty()) form.add_text("size", request.size);
if (!request.response_format.empty()) form.add_text("response_format", request.response_format);
if (!request.reasoning_strength.empty()) form.add_text("reasoning_strength", request.reasoning_strength);
std::size_t i = 0;
for (const auto& [mime, bytes] : files) {
Check::one_of("image mime", mime, {"image/png", "image/jpeg", "image/webp", "image/gif"});
if (bytes.empty()) Check::fail("image file is empty");
form.add_file("image[" + std::to_string(i) + "]", "image" + std::to_string(i), mime, bytes);
++i;
}
return ImageResult::from_json(client_->post_multipart(path_for(Call::kImagesEdit), form));
}
void ImagesService::stream(const ImageRequest& request, bool edit, const EventSink& sink) {
Json body = request.to_json(edit, true);
std::size_t seen = 0;
client_->stream(path_for(edit ? Call::kImagesEdit : Call::kImagesGenerate), body,
[&](const llm::SseEvent& e) {
if (++seen > MetaCaps::kMaxStreamEvents) return false;
std::string type = JsonX::str(e.data, "type", e.event);
return sink ? sink(type, e.data) : true;
});
}
} // namespace molo::meta
+76
View File
@@ -0,0 +1,76 @@
#pragma once
// Muse Image over the single-shot endpoints: POST /images/generations and
// POST /images/edits (JSON body with an `images` array, or multipart with
// image / image[N] parts). $0.01 per generated image, so the demo treats both
// as paid. Rules (docs: image-generation): n 1..10, aspect ratio at most 16:1,
// moderation is not enabled on muse-image-1.0 (any value is a 400), mask /
// input_fidelity / partial_images are rejected, built-in search is on unless
// tool_enablement turns a tool off. A `web_search` tool passed to Muse Image
// is an error; it does its own searching.
#include <functional>
#include <optional>
#include <string>
#include <vector>
#include "llm/api_client.h"
#include "meta/jsonx.h"
namespace molo::meta {
struct ImageRequest {
std::string model = "muse-image-1.0";
std::string prompt;
std::optional<int> n; // 1..10
std::string size; // "" | auto | WxH
std::string response_format; // b64_json (default) | url
std::string output_format; // e.g. webp
std::string reasoning_strength; // low | high (default high)
std::string user;
std::optional<bool> enable_image_search;
std::optional<bool> enable_web_search;
std::optional<bool> enable_shell;
std::vector<Json> images; // edits: {image_url} or {file_id}
Json extra;
void validate(bool edit) const;
Json to_json(bool edit, bool stream = false) const;
static Json image_url(const std::string& url);
static Json image_file(const std::string& file_id);
};
struct GeneratedImage {
std::string b64_json;
std::string url;
std::string revised_prompt;
};
struct ImageResult {
long created = 0;
std::vector<GeneratedImage> data;
std::string output_format;
Json usage;
static ImageResult from_json(const Json& j);
};
// Parses "WxH" and checks the 16:1 aspect limit; "auto" and "" pass.
bool valid_image_size(const std::string& size);
class ImagesService : public llm::Service {
public:
using EventSink = std::function<bool(const std::string& event, const Json& data)>;
explicit ImagesService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
ImageResult generate(const ImageRequest& request);
ImageResult edit(const ImageRequest& request);
// Multipart edit (what the OpenAI SDK sends): one `image[N]` part per file.
ImageResult edit_multipart(const ImageRequest& request,
const std::vector<std::pair<std::string, std::string>>& files_mime_bytes);
// Streams image_generation.completed / image_edit.completed events.
void stream(const ImageRequest& request, bool edit, const EventSink& sink);
};
} // namespace molo::meta
+72
View File
@@ -0,0 +1,72 @@
#pragma once
// Tolerant JSON readers and small builders used by every parser and request
// builder in this module. Missing or mistyped members read as the fallback;
// nothing here throws.
#include <string>
#include <vector>
#include "clients/body/json.h"
namespace molo::meta {
using Json = clients::body::Json;
struct JsonX {
static const Json* member(const Json& doc, const std::string& key) {
return doc.is_object() ? doc.find(key) : nullptr;
}
static std::string str(const Json& doc, const std::string& key, const std::string& fb = "") {
const Json* v = member(doc, key);
return v && v->is_string() ? v->as_string() : fb;
}
static long num(const Json& doc, const std::string& key, long fb = 0) {
const Json* v = member(doc, key);
return v && v->is_number() ? v->as_int() : fb;
}
static double real(const Json& doc, const std::string& key, double fb = 0) {
const Json* v = member(doc, key);
return v && v->is_number() ? v->as_number() : fb;
}
static bool flag(const Json& doc, const std::string& key, bool fb = false) {
const Json* v = member(doc, key);
return v && v->is_bool() ? v->as_bool() : fb;
}
static const Json* obj(const Json& doc, const std::string& key) {
const Json* v = member(doc, key);
return v && v->is_object() ? v : nullptr;
}
static const Json* arr(const Json& doc, const std::string& key) {
const Json* v = member(doc, key);
return v && v->is_array() ? v : nullptr;
}
// Dotted path read: "usage.input_tokens_details.cached_tokens".
static long path_num(const Json& doc, const std::string& path, long fb = 0) {
const Json* cur = &doc;
std::size_t start = 0;
while (true) {
std::size_t dot = path.find('.', start);
std::string part = path.substr(start, dot == std::string::npos ? dot : dot - start);
cur = member(*cur, part);
if (!cur) return fb;
if (dot == std::string::npos) break;
start = dot + 1;
}
return cur->is_number() ? cur->as_int() : fb;
}
static Json strings(const std::vector<std::string>& v) {
Json a = Json::array();
for (const std::string& s : v) a.push(Json(s));
return a;
}
static void set_if(Json& obj, const char* key, const std::string& value) {
if (!value.empty()) obj.set(key, Json(value));
}
static void merge(Json& into, const Json& extra) {
if (!extra.is_object()) return;
for (const auto& [k, v] : extra.as_object()) into.set(k, v);
}
};
} // namespace molo::meta
+291
View File
@@ -0,0 +1,291 @@
#include "meta/messages.h"
#include "meta/calls.h"
#include "meta/caps.h"
#include "meta/check.h"
#include "meta/schema_check.h"
#include "security/crypto/Base64.h"
namespace molo::meta {
namespace {
Json block(const char* type) {
Json o = Json::object();
o.set("type", Json(type));
return o;
}
Json source(const char* type) {
Json s = Json::object();
s.set("type", Json(type));
return s;
}
} // namespace
Json AnthropicBlock::text(const std::string& t) {
Json b = block("text");
b.set("text", Json(t));
return b;
}
Json AnthropicBlock::image_url(const std::string& url) {
Check::non_empty("image url", url);
Json s = source("url");
s.set("url", Json(url));
Json b = block("image");
b.set("source", s);
return b;
}
Json AnthropicBlock::image_base64(const std::string& media_type, const std::string& bytes) {
Check::one_of("media_type", media_type,
{"image/png", "image/jpeg", "image/gif", "image/webp", "image/x-icon"});
if (bytes.empty() || bytes.size() > MetaCaps::kMaxInlineBytes) Check::fail("image size is out of range");
Json s = source("base64");
s.set("media_type", Json(media_type));
s.set("data", Json(security::crypto::Base64::encode(bytes)));
Json b = block("image");
b.set("source", s);
return b;
}
Json AnthropicBlock::image_file(const std::string& file_id) {
Check::segment("file_id", file_id);
Json s = source("file");
s.set("file_id", Json(file_id));
Json b = block("image");
b.set("source", s);
return b;
}
Json AnthropicBlock::video_url(const std::string& url, double fps) {
Check::non_empty("video url", url);
Json s = source("url");
s.set("url", Json(url));
Json b = block("video");
b.set("source", s);
if (fps > 0) b.set("fps", Json(fps));
return b;
}
Json AnthropicBlock::document_url(const std::string& url) {
Check::non_empty("document url", url);
Json s = source("url");
s.set("url", Json(url));
Json b = block("document");
b.set("source", s);
return b;
}
Json AnthropicBlock::document_text(const std::string& t) {
Check::non_empty("document text", t);
Json s = source("text");
s.set("media_type", Json("text/plain"));
s.set("data", Json(t));
Json b = block("document");
b.set("source", s);
return b;
}
Json AnthropicBlock::tool_use(const std::string& id, const std::string& name, const Json& input) {
Check::call_id(id);
Check::function_name(name);
Json b = block("tool_use");
b.set("id", Json(id));
b.set("name", Json(name));
b.set("input", input.is_object() ? input : Json::object());
return b;
}
Json AnthropicBlock::tool_result(const std::string& id, const std::string& content, bool is_error) {
Check::call_id(id);
Json b = block("tool_result");
b.set("tool_use_id", Json(id));
b.set("content", Json(content));
if (is_error) b.set("is_error", Json(true));
return b;
}
Json AnthropicBlock::redacted_thinking(const std::string& data) {
Check::non_empty("data", data);
Json b = block("redacted_thinking");
b.set("data", Json(data));
return b;
}
Json AnthropicBlock::message(const std::string& role, const std::vector<Json>& blocks) {
Check::one_of("role", role, {"user", "assistant"});
Json a = Json::array();
for (const Json& b : blocks) a.push(b);
Json m = Json::object();
m.set("role", Json(role));
m.set("content", a);
return m;
}
Json AnthropicBlock::message_text(const std::string& role, const std::string& t) {
Check::one_of("role", role, {"user", "assistant"});
Json m = Json::object();
m.set("role", Json(role));
m.set("content", Json(t));
return m;
}
Json AnthropicBlock::custom_tool(const std::string& name, const std::string& description,
const Json& schema, bool strict, bool defer_loading) {
Check::function_name(name);
if (schema.is_object()) SchemaCheck::check(schema, strict);
Json t = Json::object();
t.set("name", Json(name));
JsonX::set_if(t, "description", description);
t.set("input_schema", schema.is_object() ? schema : Json::object());
if (strict) t.set("strict", Json(true));
if (defer_loading) t.set("defer_loading", Json(true));
return t;
}
Json AnthropicBlock::web_search_tool(const Json& user_location) {
Json t = block("web_search");
t.set("name", Json("web_search"));
if (user_location.is_object()) t.set("user_location", user_location);
return t;
}
Json AnthropicBlock::tool_search_tool() {
Json t = block("tool_search");
t.set("name", Json("tool_search"));
return t;
}
void MessageRequest::validate() const {
Check::model_id(model);
if (messages.empty() || messages.size() > MetaCaps::kMaxInputItems)
Check::fail("messages needs 1.." + std::to_string(MetaCaps::kMaxInputItems) + " entries");
for (const Json& m : messages)
Check::one_of("message role", JsonX::str(m, "role"), {"user", "assistant", "system", "developer"});
if (max_tokens < 1) Check::fail("max_tokens is required");
Check::max_bytes("system", system, MetaCaps::kMaxPromptBytes);
if (tools.size() > MetaCaps::kMaxTools) Check::fail("too many tools");
if (!tool_choice.empty()) Check::one_of("tool_choice", tool_choice, {"auto", "any", "none"});
if (!thinking.empty()) {
Check::one_of("thinking", thinking, {"adaptive", "enabled"}); // disabled is rejected
if (thinking == "enabled" && (thinking_budget < 1024 || thinking_budget >= max_tokens))
Check::fail("thinking budget_tokens must be >= 1024 and < max_tokens");
}
if (!thinking_display.empty())
Check::one_of("thinking.display", thinking_display, {"summarized", "omitted"});
if (!effort.empty()) Check::one_of("output_config.effort", effort, {"low", "medium", "high", "xhigh"});
if (output_schema.is_object()) SchemaCheck::check(output_schema, false);
if (temperature) Check::range("temperature", *temperature, 0.0, 1.0);
if (top_p) Check::top_p(*top_p);
if (!service_tier.empty()) Check::one_of("service_tier", service_tier, {"auto", "standard_only"});
if (user_id.size() > 4096) Check::fail("metadata.user_id is too long");
if (extra.is_object())
for (const char* banned : {"stop_sequences", "top_k", "container", "inference_geo"})
if (extra.has(banned)) Check::fail(std::string(banned) + " is rejected by the messages endpoint");
}
namespace {
void common(const MessageRequest& r, Json& b) {
b.set("model", Json(r.model));
Json m = Json::array();
for (const Json& x : r.messages) m.push(x);
b.set("messages", m);
if (!r.system.empty()) b.set("system", Json(r.system));
if (!r.tools.empty()) {
Json t = Json::array();
for (const Json& x : r.tools) t.push(x);
b.set("tools", t);
}
if (!r.tool_choice.empty() || r.disable_parallel_tool_use) {
Json c = Json::object();
c.set("type", Json(r.tool_choice.empty() ? "auto" : r.tool_choice));
if (r.disable_parallel_tool_use) c.set("disable_parallel_tool_use", Json(true));
b.set("tool_choice", c);
}
if (!r.thinking.empty()) {
Json t = Json::object();
t.set("type", Json(r.thinking));
if (r.thinking == "enabled") t.set("budget_tokens", Json(r.thinking_budget));
JsonX::set_if(t, "display", r.thinking_display);
b.set("thinking", t);
}
if (!r.effort.empty() || r.output_schema.is_object()) {
Json o = Json::object();
JsonX::set_if(o, "effort", r.effort);
if (r.output_schema.is_object()) {
Json f = Json::object();
f.set("type", Json("json_schema"));
f.set("schema", r.output_schema);
o.set("format", f);
}
b.set("output_config", o);
}
if (r.temperature) b.set("temperature", Json(*r.temperature));
if (r.top_p) b.set("top_p", Json(*r.top_p));
if (!r.user_id.empty()) {
Json md = Json::object();
md.set("user_id", Json(r.user_id));
b.set("metadata", md);
}
}
} // namespace
Json MessageRequest::to_json(bool stream) const {
validate();
Json b = Json::object();
common(*this, b);
b.set("max_tokens", Json(max_tokens));
JsonX::set_if(b, "service_tier", service_tier);
if (stream) b.set("stream", Json(true));
JsonX::merge(b, extra);
return b;
}
Json MessageRequest::to_count_json() const {
validate();
Json b = Json::object();
common(*this, b);
JsonX::merge(b, extra);
return b;
}
Json MessagesService::create(const MessageRequest& request) {
Json body = request.to_json(false);
return client_->post(path_for(Call::kMessagesCreate), body);
}
std::string MessagesService::stream(const MessageRequest& request, const EventSink& sink) {
Json body = request.to_json(true);
std::string text;
std::size_t seen = 0;
client_->stream(path_for(Call::kMessagesCreate), body, [&](const llm::SseEvent& e) {
if (++seen > MetaCaps::kMaxStreamEvents) return false;
std::string type = JsonX::str(e.data, "type", e.event);
if (type == "content_block_delta")
if (const Json* d = JsonX::obj(e.data, "delta"))
if (JsonX::str(*d, "type") == "text_delta") text += JsonX::str(*d, "text");
return sink ? sink(type, e.data) : true;
});
return text;
}
long MessagesService::count_tokens(const MessageRequest& request) {
Json body = request.to_count_json();
Json out = client_->post(path_for(Call::kMessagesCountTokens), body);
return JsonX::num(out, "input_tokens", -1);
}
std::string MessagesService::text_of(const Json& message) {
std::string out;
if (const Json* c = JsonX::arr(message, "content"))
for (const Json& b : c->as_array())
if (JsonX::str(b, "type") == "text") out += JsonX::str(b, "text");
return out;
}
MetaUsage MessagesService::usage_of(const Json& message) {
const Json* u = JsonX::obj(message, "usage");
return u ? MetaUsage::from_messages(*u) : MetaUsage();
}
} // namespace molo::meta
+86
View File
@@ -0,0 +1,86 @@
#pragma once
// POST /messages and POST /messages/count_tokens: the Anthropic Messages
// compatible endpoint (base URL https://api.meta.ai, the SDK appends /v1).
// The adapter is stateless and runs storage off. Not supported, per docs
// (protocols/messages): stop_sequences, top_k, container, inference_geo,
// thinking disabled, a named tool_choice, web_search allowed_domains /
// blocked_domains / max_uses, system blocks other than text. Anthropic stream
// events arrive as named SSE events (message_start, content_block_delta, ...).
#include <functional>
#include <optional>
#include <string>
#include <vector>
#include "llm/api_client.h"
#include "meta/jsonx.h"
#include "meta/usage.h"
namespace molo::meta {
class AnthropicBlock {
public:
static Json text(const std::string& text);
static Json image_url(const std::string& url);
static Json image_base64(const std::string& media_type, const std::string& bytes);
static Json image_file(const std::string& file_id);
static Json video_url(const std::string& url, double fps = 0);
static Json document_url(const std::string& url);
static Json document_text(const std::string& text);
static Json tool_use(const std::string& id, const std::string& name, const Json& input);
static Json tool_result(const std::string& tool_use_id, const std::string& content,
bool is_error = false);
static Json redacted_thinking(const std::string& data); // replay encrypted reasoning verbatim
static Json message(const std::string& role, const std::vector<Json>& blocks);
static Json message_text(const std::string& role, const std::string& text);
// Tool definitions.
static Json custom_tool(const std::string& name, const std::string& description,
const Json& input_schema, bool strict = false, bool defer_loading = false);
static Json web_search_tool(const Json& user_location = Json());
static Json tool_search_tool();
};
struct MessageRequest {
std::string model;
std::vector<Json> messages;
int max_tokens = 0; // required
std::string system;
std::vector<Json> tools;
std::string tool_choice; // "", auto, any, none
bool disable_parallel_tool_use = false;
std::string thinking; // "", adaptive, enabled
int thinking_budget = 0; // enabled: >= 1024 and < max_tokens
std::string thinking_display; // summarized | omitted
std::string effort; // output_config.effort: low medium high xhigh
Json output_schema; // output_config.format = {type: json_schema, schema}
std::optional<double> temperature; // 0..1
std::optional<double> top_p;
std::string service_tier; // auto | standard_only
std::string user_id; // metadata.user_id
Json extra;
void validate() const;
Json to_json(bool stream = false) const;
Json to_count_json() const;
};
class MessagesService : public llm::Service {
public:
using EventSink = std::function<bool(const std::string& event, const Json& data)>;
explicit MessagesService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
Json create(const MessageRequest& request);
// Folds the Anthropic stream; returns the visible text. The sink sees every event.
std::string stream(const MessageRequest& request, const EventSink& sink = nullptr);
long count_tokens(const MessageRequest& request);
// Visible text of an AnthropicMessage: the `text` blocks (content[0] is a
// redacted_thinking block, never index it).
static std::string text_of(const Json& message);
static MetaUsage usage_of(const Json& message);
};
} // namespace molo::meta
+44
View File
@@ -0,0 +1,44 @@
#include "meta/models.h"
#include "meta/calls.h"
#include "meta/check.h"
namespace molo::meta {
ModelEntry ModelEntry::from_json(const Json& j) {
ModelEntry m;
m.id = JsonX::str(j, "id");
m.created = JsonX::num(j, "created");
m.owned_by = JsonX::str(j, "owned_by");
if (const Json* md = JsonX::obj(j, "metadata")) m.metadata = *md;
return m;
}
ServiceStatus ServiceStatus::from_json(const Json& j) {
ServiceStatus s;
s.is_alive = JsonX::flag(j, "is_alive");
s.service_status = JsonX::str(j, "service_status");
s.service_message = JsonX::str(j, "service_message");
s.updated_at = JsonX::str(j, "updated_at");
if (const Json* m = JsonX::arr(j, "model_statuses")) s.model_statuses = *m;
return s;
}
std::vector<ModelEntry> ModelsService::list() {
Json doc = client_->get(path_for(Call::kModelsList));
std::vector<ModelEntry> out;
if (const Json* d = JsonX::arr(doc, "data"))
for (const Json& m : d->as_array()) out.push_back(ModelEntry::from_json(m));
return out;
}
ModelEntry ModelsService::retrieve(const std::string& model_id) {
Check::model_id(model_id);
return ModelEntry::from_json(client_->get(path_for(Call::kModelsRetrieve, model_id)));
}
ServiceStatus StatusService::get() {
return ServiceStatus::from_json(client_->get(path_for(Call::kStatus)));
}
} // namespace molo::meta
+50
View File
@@ -0,0 +1,50 @@
#pragma once
// GET /models, GET /models/{model}, GET /status.
// The models list returns the API-hosted models newest-first (created, then
// id); open-weight models (Muse Spark Open, Muse Glimmer) are never in it.
// /status is the one unauthenticated endpoint: overall health plus
// per-model availability.
#include <string>
#include <vector>
#include "llm/api_client.h"
#include "meta/jsonx.h"
namespace molo::meta {
struct ModelEntry {
std::string id;
long created = 0;
std::string owned_by;
Json metadata;
static ModelEntry from_json(const Json& j);
};
struct ServiceStatus {
bool is_alive = false;
std::string service_status; // e.g. "operational"
std::string service_message;
std::string updated_at;
Json model_statuses; // array, shape not in the schema pages (see SPEC Unverified)
static ServiceStatus from_json(const Json& j);
};
class ModelsService : public llm::Service {
public:
explicit ModelsService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
std::vector<ModelEntry> list();
ModelEntry retrieve(const std::string& model_id);
};
class StatusService : public llm::Service {
public:
// Build the client with MetaClient::unauthenticated().
explicit StatusService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
ServiceStatus get();
};
} // namespace molo::meta
+353
View File
@@ -0,0 +1,353 @@
#include "meta/muse_code.h"
#include <poll.h>
#include <signal.h>
#include <sys/wait.h>
#include <unistd.h>
#include <chrono>
#include <cstdio>
#include <cstring>
#include <map>
#include <random>
#include "meta/caps.h"
#include "meta/check.h"
#include "meta/muse_code_spec.h"
namespace molo::meta {
namespace {
bool in_list(const char* const* list, std::size_t n, const std::string& s) {
for (std::size_t i = 0; i < n; ++i)
if (s == list[i]) return true;
return false;
}
long steady_ms() {
return std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now().time_since_epoch()).count();
}
} // namespace
void LineSplitter::feed(const std::string& bytes) {
for (char c : bytes) {
if (c == '\n') {
if (!buf_.empty() && buf_.back() == '\r') buf_.pop_back();
lines_.push_back(std::move(buf_));
buf_.clear();
if (lines_.size() > MetaCaps::kMspMaxQueued) Check::fail("too many unread MSP frames");
} else {
buf_.push_back(c);
if (buf_.size() > limit_) Check::fail("MSP frame is over the frame limit");
}
}
}
std::optional<std::string> LineSplitter::pop() {
if (lines_.empty()) return std::nullopt;
std::string l = std::move(lines_.front());
lines_.pop_front();
return l;
}
ProcessTransport::ProcessTransport(std::vector<std::string> argv)
: splitter_(MetaCaps::kMspFrameLimit) {
if (argv.empty() || argv.size() > 16) Check::fail("bad argv");
int to_child[2], from_child[2];
if (pipe(to_child) != 0) Check::fail("pipe failed");
if (pipe(from_child) != 0) { ::close(to_child[0]); ::close(to_child[1]); Check::fail("pipe failed"); }
pid_t pid = fork();
if (pid < 0) Check::fail("fork failed");
if (pid == 0) {
dup2(to_child[0], 0);
dup2(from_child[1], 1);
::close(to_child[0]); ::close(to_child[1]); ::close(from_child[0]); ::close(from_child[1]);
std::vector<char*> args;
for (std::string& a : argv) args.push_back(a.data());
args.push_back(nullptr);
execvp(args[0], args.data());
_exit(127);
}
::close(to_child[0]);
::close(from_child[1]);
in_fd_ = to_child[1];
out_fd_ = from_child[0];
pid_ = pid;
signal(SIGPIPE, SIG_IGN);
}
ProcessTransport::~ProcessTransport() { close(); }
bool ProcessTransport::binary_available(const std::string& name) {
const char* path = getenv("PATH");
std::string p = path ? path : "";
std::size_t start = 0;
while (start <= p.size()) {
std::size_t colon = p.find(':', start);
std::string dir = p.substr(start, colon == std::string::npos ? colon : colon - start);
if (!dir.empty() && access((dir + "/" + name).c_str(), X_OK) == 0) return true;
if (colon == std::string::npos) break;
start = colon + 1;
}
return false;
}
void ProcessTransport::write_line(const std::string& line) {
if (in_fd_ < 0) throw llm::ApiError("meta", 502, "UNAVAILABLE", "muse process is closed");
if (line.size() > MetaCaps::kMspFrameLimit) Check::fail("MSP frame is over the frame limit");
std::string data = line + "\n";
std::size_t off = 0;
while (off < data.size()) {
ssize_t n = write(in_fd_, data.data() + off, data.size() - off);
if (n <= 0) throw llm::ApiError("meta", 502, "UNAVAILABLE", "write to muse failed");
off += static_cast<std::size_t>(n);
}
}
std::optional<std::string> ProcessTransport::read_line(int timeout_ms) {
long deadline = steady_ms() + timeout_ms;
while (true) {
if (auto l = splitter_.pop()) return l;
if (eof_ || out_fd_ < 0) throw llm::ApiError("meta", 502, "UNAVAILABLE", "muse process ended");
long left = deadline - steady_ms();
if (left < 0) return std::nullopt;
pollfd pfd{out_fd_, POLLIN, 0};
int r = poll(&pfd, 1, static_cast<int>(left));
if (r == 0) return std::nullopt;
if (r < 0) continue;
char buf[65536];
ssize_t n = read(out_fd_, buf, sizeof buf);
if (n <= 0) { eof_ = true; continue; }
splitter_.feed(std::string(buf, static_cast<std::size_t>(n)));
}
}
void ProcessTransport::close() {
if (in_fd_ >= 0) { ::close(in_fd_); in_fd_ = -1; }
if (pid_ > 0) {
for (int i = 0; i < 20; ++i) {
if (waitpid(static_cast<pid_t>(pid_), nullptr, WNOHANG) != 0) { pid_ = -1; break; }
usleep(50000);
}
if (pid_ > 0) {
kill(static_cast<pid_t>(pid_), SIGKILL);
waitpid(static_cast<pid_t>(pid_), nullptr, 0);
pid_ = -1;
}
}
if (out_fd_ >= 0) { ::close(out_fd_); out_fd_ = -1; }
}
std::string MspFrames::request(long id, const std::string& method, const Json& params) {
Json j = Json::object();
j.set("jsonrpc", Json("2.0"));
j.set("id", Json(id));
j.set("method", Json(method));
if (params.is_object()) j.set("params", params);
return j.dump();
}
std::string MspFrames::notification(const std::string& method, const Json& params) {
Json j = Json::object();
j.set("jsonrpc", Json("2.0"));
j.set("method", Json(method));
if (params.is_object()) j.set("params", params);
return j.dump();
}
std::string MspFrames::result(const Json& id, const Json& res) {
Json j = Json::object();
j.set("jsonrpc", Json("2.0"));
j.set("id", id);
j.set("result", res);
return j.dump();
}
Json MspFrames::parse(const std::string& line) {
try {
Json j = Json::parse(line);
if (!j.is_object() || JsonX::str(j, "jsonrpc") != "2.0")
throw llm::ApiError("meta", 502, "BAD_RESPONSE", "frame is not JSON-RPC 2.0");
return j;
} catch (const clients::body::JsonError& e) {
throw llm::ApiError("meta", 502, "BAD_RESPONSE", e.what());
}
}
bool MspFrames::is_known_method(const std::string& m) {
std::size_t n;
const char* const* l = MuseCodeSpec::methods(n);
return in_list(l, n, m);
}
bool MspFrames::is_known_notification(const std::string& m) {
std::size_t n;
const char* const* l = MuseCodeSpec::notifications(n);
return in_list(l, n, m);
}
std::string MspFrames::error_kind(long code) {
std::size_t n;
const char* const* l = MuseCodeSpec::errors(n);
std::string prefix = std::to_string(code) + ":";
for (std::size_t i = 0; i < n; ++i)
if (std::strncmp(l[i], prefix.c_str(), prefix.size()) == 0) {
std::string s = l[i] + prefix.size();
return s.substr(0, s.find(':'));
}
return "";
}
bool MspFrames::error_retryable(long code) {
std::size_t n;
const char* const* l = MuseCodeSpec::errors(n);
std::string prefix = std::to_string(code) + ":";
for (std::size_t i = 0; i < n; ++i)
if (std::strncmp(l[i], prefix.c_str(), prefix.size()) == 0) {
std::string s = l[i];
return s.back() == '1';
}
return false;
}
std::string MspFrames::new_id() {
static std::mt19937_64 rng{std::random_device{}()};
unsigned long long ms = static_cast<unsigned long long>(
std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::system_clock::now().time_since_epoch()).count());
unsigned long long a = rng(), b = rng();
char buf[64];
std::snprintf(buf, sizeof buf, "%08llx-%04llx-7%03llx-%04llx-%012llx", ms >> 16, ms & 0xFFFF,
a & 0xFFF, 0x8000 | (b & 0x3FFF), (rng() & 0xFFFFFFFFFFFFull));
return buf;
}
MspClient::MspClient(std::unique_ptr<ILineTransport> t) : transport_(std::move(t)) {}
MspClient::~MspClient() { close(); }
void MspClient::close() {
if (transport_) transport_->close();
}
void MspClient::handle_frame(const Json& f) {
const Json* id = JsonX::member(f, "id");
std::string method = JsonX::str(f, "method");
if (!method.empty() && id) { // server -> client request
Json params = f.find("params") ? *f.find("params") : Json::object();
Json res = handler_ ? handler_(method, params) : Json::object();
transport_->write_line(MspFrames::result(*id, res));
return;
}
if (!method.empty()) {
if (queue_.size() >= MetaCaps::kMspMaxQueued) {
++dropped_; // loud: counted, surfaced through dropped()
return;
}
MspNotification n;
n.method = method;
if (const Json* p = f.find("params")) n.params = *p;
queue_.push_back(std::move(n));
return;
}
if (id && id->is_number()) responses_[id->as_int()] = f;
}
Json MspClient::request(const std::string& method, const Json& params, int timeout_ms) {
if (!MspFrames::is_known_method(method)) Check::fail("unknown MSP method " + method);
if (method != "initialize" && !initialized_) Check::fail("call initialize first");
long id = next_id_++;
transport_->write_line(MspFrames::request(id, method, params));
long deadline = steady_ms() + timeout_ms;
while (true) {
auto it = responses_.find(id);
if (it != responses_.end()) {
Json f = it->second;
responses_.erase(it);
if (const Json* e = JsonX::obj(f, "error")) {
long code = JsonX::num(*e, "code");
std::string kind = MspFrames::error_kind(code);
throw llm::ApiError("meta", MspFrames::error_retryable(code) ? 503 : 400,
kind.empty() ? std::to_string(code) : kind, JsonX::str(*e, "message"));
}
const Json* r = f.find("result");
return r ? *r : Json::object();
}
long left = deadline - steady_ms();
if (left <= 0) throw llm::ApiError("meta", 504, "TIMEOUT", method + " timed out");
auto line = transport_->read_line(static_cast<int>(left));
if (line) handle_frame(MspFrames::parse(*line));
}
}
Json MspClient::initialize(const std::string& name, const std::string& version,
const std::vector<std::string>& caps) {
Check::non_empty("client name", name);
Check::non_empty("client version", version);
Json info = Json::object();
info.set("name", Json(name));
info.set("version", Json(version));
Json params = Json::object();
params.set("clientInfo", info);
if (!caps.empty()) {
Json c = Json::object();
c.set("requestedCapabilities", JsonX::strings(caps));
params.set("capabilities", c);
}
Json res = request("initialize", params, MetaCaps::kMspSpawnTimeoutMs);
initialized_ = true;
if (const Json* s = JsonX::obj(res, "schema")) fingerprint_ = JsonX::str(*s, "fingerprint");
transport_->write_line(MspFrames::notification("initialized"));
return res;
}
std::optional<MspNotification> MspClient::next_notification(int timeout_ms) {
long deadline = steady_ms() + timeout_ms;
while (true) {
if (!queue_.empty()) {
MspNotification n = std::move(queue_.front());
queue_.pop_front();
return n;
}
long left = deadline - steady_ms();
if (left <= 0) return std::nullopt;
auto line = transport_->read_line(static_cast<int>(left));
if (line) handle_frame(MspFrames::parse(*line));
}
}
Json MspClient::session_start(const std::string& workspace_root, const std::string& model_id) {
Check::non_empty("workspaceRoot", workspace_root);
Json p = Json::object();
p.set("commandId", Json(MspFrames::new_id()));
p.set("workspaceRoot", Json(workspace_root));
JsonX::set_if(p, "modelId", model_id);
return request("session/start", p);
}
Json MspClient::turn_start(const std::string& session_id, const std::string& text) {
Check::segment("sessionId", session_id);
Check::non_empty("text", text);
Json part = Json::object();
part.set("type", Json("text"));
part.set("text", Json(text));
Json input = Json::array();
input.push(part);
Json p = Json::object();
p.set("commandId", Json(MspFrames::new_id()));
p.set("sessionId", Json(session_id));
p.set("input", input);
return request("turn/start", p);
}
Json MspClient::turn_cancel(const std::string& session_id, const std::string& turn_id) {
Check::segment("sessionId", session_id);
Json p = Json::object();
p.set("commandId", Json(MspFrames::new_id()));
p.set("sessionId", Json(session_id));
JsonX::set_if(p, "turnId", turn_id);
return request("turn/cancel", p);
}
Json MspClient::model_list() { return request("model/list", Json::object()); }
} // namespace molo::meta
+127
View File
@@ -0,0 +1,127 @@
#pragma once
// Muse Session Protocol (MSP) client: drive a Muse Code agent session from
// C++. Transport is newline-delimited JSON-RPC 2.0 on the stdio of
// `muse serve` (frame limit 10 MiB, client lines may end in \r). Handshake:
// initialize -> result{serverInfo, schema{version,fingerprint}, grantedCapabilities,
// ...} -> notification `initialized`. Server push (session/*, turn/*, item/*,
// view/*) arrives as notifications and is queued with a hard cap; the two
// server requests (approval/request, userInput/request) are answered with an
// empty receipt by the default handler (the decision travels separately as
// approval/decide or userInput/answer).
// The transport is an interface so tests run the whole client against a
// loopback process; ProcessTransport spawns the real binary.
#include <deque>
#include <map>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <vector>
#include "llm/error.h"
#include "meta/jsonx.h"
namespace molo::meta {
class ILineTransport {
public:
virtual ~ILineTransport() = default;
virtual void write_line(const std::string& line) = 0;
// One line without the newline; nullopt on timeout; throws on EOF.
virtual std::optional<std::string> read_line(int timeout_ms) = 0;
virtual void close() = 0;
};
// Splits a byte stream into lines, enforcing the frame limit.
class LineSplitter {
public:
explicit LineSplitter(std::size_t limit = 10485760) : limit_(limit) {}
// Appends bytes; throws ApiError::invalid when a line exceeds the limit.
void feed(const std::string& bytes);
std::optional<std::string> pop(); // strips a trailing \r
private:
std::size_t limit_;
std::string buf_;
std::deque<std::string> lines_;
};
// Spawns argv (default {"muse","serve"}) with stdin/stdout pipes.
class ProcessTransport : public ILineTransport {
public:
explicit ProcessTransport(std::vector<std::string> argv = {"muse", "serve"});
~ProcessTransport() override;
void write_line(const std::string& line) override;
std::optional<std::string> read_line(int timeout_ms) override;
void close() override;
static bool binary_available(const std::string& name = "muse");
private:
int in_fd_ = -1, out_fd_ = -1;
long pid_ = -1;
LineSplitter splitter_;
bool eof_ = false;
};
struct MspFrames {
static std::string request(long id, const std::string& method, const Json& params);
static std::string notification(const std::string& method, const Json& params = Json());
static std::string result(const Json& id, const Json& result);
// Throws ApiError on non-JSON or a missing jsonrpc "2.0".
static Json parse(const std::string& line);
static bool is_known_method(const std::string& method);
static bool is_known_notification(const std::string& method);
// kind name for a JSON-RPC error code ("" when not in the schema).
static std::string error_kind(long code);
static bool error_retryable(long code);
// UUID-shaped, time-ordered id for commandId / sessionId.
static std::string new_id();
};
struct MspNotification {
std::string method;
Json params;
};
class MspClient {
public:
using ServerRequestHandler = std::function<Json(const std::string& method, const Json& params)>;
explicit MspClient(std::unique_ptr<ILineTransport> transport);
~MspClient();
// Handshake. Throws on an error reply. Records schema fingerprint.
Json initialize(const std::string& client_name, const std::string& client_version,
const std::vector<std::string>& requested_capabilities = {});
// Generic request; `method` must be in the schema's method table. Throws
// llm::ApiError(type = MSP error kind) on a JSON-RPC error.
Json request(const std::string& method, const Json& params, int timeout_ms = 120000);
// Pumps incoming frames for up to timeout_ms; returns the next notification.
std::optional<MspNotification> next_notification(int timeout_ms);
std::size_t queued() const { return queue_.size(); }
std::size_t dropped() const { return dropped_; }
const std::string& schema_fingerprint() const { return fingerprint_; }
bool initialized() const { return initialized_; }
void set_server_request_handler(ServerRequestHandler h) { handler_ = std::move(h); }
void close();
// Typed shortcuts over request().
Json session_start(const std::string& workspace_root, const std::string& model_id = "");
Json turn_start(const std::string& session_id, const std::string& text);
Json turn_cancel(const std::string& session_id, const std::string& turn_id = "");
Json model_list();
private:
void handle_frame(const Json& frame);
std::unique_ptr<ILineTransport> transport_;
std::deque<MspNotification> queue_;
std::size_t dropped_ = 0;
long next_id_ = 1;
bool initialized_ = false;
std::string fingerprint_;
ServerRequestHandler handler_;
std::map<long, Json> responses_;
};
} // namespace molo::meta
+185
View File
@@ -0,0 +1,185 @@
#include "meta/muse_code_spec.h"
namespace molo::meta {
// Generated from schema/msp/stable/msp.schema.json of the muse-code-sdk
// mirror (fingerprint sha256:cfdb6d2525601b9bac76584ccb0610ab3a5539b5e4f51141f079b99b39fcb4e0,
// schemaVersion 1). tools/check_spec.py re-fetches the schema and fails if any
// list below drifts.
// MSP-METHODS-BEGIN
static const char* const kMethods[] = {
"approval/decide",
"approval/listPending",
"computerUseSettings/read",
"computerUseSettings/update",
"computerUseSetup/check",
"computerUseSetup/read",
"computerUseSetup/start",
"feedback/submit",
"goal/clear",
"goal/edit",
"goal/pause",
"goal/resume",
"goal/set",
"hook/list",
"initialize",
"item/readOutput",
"media/upload",
"model/list",
"plugin/install",
"plugin/list",
"plugin/marketplaceAdd",
"plugin/marketplaceRemove",
"plugin/marketplaceUpdate",
"recap/cancel",
"recap/request",
"session/compact",
"session/fork",
"session/leave",
"session/list",
"session/read",
"session/rename",
"session/resume",
"session/setApprovalMode",
"session/setModel",
"session/setReasoningEffort",
"session/sideChat",
"session/start",
"session/userShell",
"skill/import",
"skill/list",
"skill/setActivation",
"subagent/close",
"subagent/followupTask",
"subagent/interrupt",
"subagent/readResult",
"subagent/reopen",
"subagent/resume",
"subagent/sendMessage",
"subagent/stop",
"task/background",
"task/list",
"task/stop",
"task/stopAll",
"turn/cancel",
"turn/interrupt",
"turn/start",
"turn/steer",
"turn/unqueue",
"usage/read",
"userInput/answer",
"userInput/cancel",
"userInput/clarify",
"userInput/interrupt",
"view/page",
"view/subscribe",
"view/unsubscribe",
"workflow/cancel",
"workflow/childControl",
"workflow/pause",
"workflow/resume",
};
// MSP-METHODS-END
// MSP-NOTIFICATIONS-BEGIN
static const char* const kNotifications[] = {
"approval/requested",
"approval/resolved",
"approval/reviewing",
"approval/updated",
"computerUseSetup/changed",
"initialized",
"item/completed",
"item/delta",
"item/started",
"item/updated",
"monitor/inboxItemAdmitted",
"monitor/inboxItemDrained",
"monitor/inboxItemVoided",
"plugin/changed",
"plugin/marketplaceChanged",
"recap/completed",
"session/approvalModeChanged",
"session/branchChanged",
"session/closed",
"session/contextUsage",
"session/goalChanged",
"session/leaveCompleted",
"session/listChanged",
"session/modelChanged",
"session/modelRouteUnserved",
"session/nameChanged",
"session/reasoningEffortChanged",
"session/started",
"session/statusChanged",
"session/todoListChanged",
"session/tokenUsage",
"session/viewHealthChanged",
"skill/changed",
"turn/completed",
"turn/foregroundCompleted",
"turn/modelRetryScheduled",
"turn/retracted",
"turn/retryScheduled",
"turn/started",
"turn/unqueued",
"usage/changed",
"userInput/engaged",
"userInput/requested",
"userInput/settled",
"view/gap",
};
// MSP-NOTIFICATIONS-END
// MSP-REQUESTS-BEGIN
static const char* const kRequests[] = {
"approval/request",
"userInput/request",
};
// MSP-REQUESTS-END
// MSP-ERRORS-BEGIN (code:kind:retryable)
static const char* const kErrors[] = {
"-32700:parseError:0",
"-32600:invalidRequest:0",
"-32601:methodNotFound:0",
"-32602:invalidParams:0",
"-32603:internal:0",
"-32001:overloaded:1",
"-32002:inputTooLarge:0",
"-32010:capabilityRequired:0",
"-32011:notFound:0",
"-32013:interrupted:?",
"-32014:cancelled:?",
"-32020:sessionNotFound:0",
"-32021:sessionInUse:0",
"-32022:sessionAmbiguous:0",
"-32023:forkBoundaryInvalid:0",
"-32024:sessionNotLoaded:0",
"-32025:sessionStreamMismatch:0",
"-32030:commandRejected:0",
"-32031:backpressured:1",
"-32032:skillNotFound:0",
"-32033:computerUseUnavailable:0",
"-32034:marketplaceNotFound:0",
"-32035:pluginNotFound:0",
"-32036:marketplaceAlreadyExists:0",
"-32037:pluginMutationsDisabled:0",
"-32038:uploadNotFound:0",
"-32040:viewTruncated:0",
"-32041:outputUnavailable:0",
"-32042:boundaryPruned:0",
"-32050:approvalNotFound:0",
"-32051:approvalAlreadyResolved:0",
"-32052:approvalChoiceInvalid:0",
"-32053:approvalRequirementStale:0",
"-32054:approvalReviewerUnavailable:0",
"-32055:userInputNotFound:0",
"-32056:userInputAlreadySettled:0",
"-32057:userInputAnswerInvalid:0",
};
// MSP-ERRORS-END
const char* const* MuseCodeSpec::methods(std::size_t& n) { n = sizeof(kMethods) / sizeof(kMethods[0]); return kMethods; }
const char* const* MuseCodeSpec::notifications(std::size_t& n) { n = sizeof(kNotifications) / sizeof(kNotifications[0]); return kNotifications; }
const char* const* MuseCodeSpec::server_requests(std::size_t& n) { n = sizeof(kRequests) / sizeof(kRequests[0]); return kRequests; }
const char* const* MuseCodeSpec::errors(std::size_t& n) { n = sizeof(kErrors) / sizeof(kErrors[0]); return kErrors; }
} // namespace molo::meta
+21
View File
@@ -0,0 +1,21 @@
#pragma once
// Muse Code (the Meta coding agent) has a public programmatic surface: the
// Muse Session Protocol (MSP), JSON-RPC 2.0 over newline-delimited JSON on the
// stdio of `muse serve` (docs: dev.meta.ai/docs/muse-code, SDK docs at
// meta-models.github.io/muse-code-sdk, schema in the muse-code-sdk repo under
// schema/msp/stable). It is NOT an HTTP API of api.meta.ai. This table is the
// protocol's method / notification / server-request / error vocabulary.
#include <cstddef>
namespace molo::meta {
struct MuseCodeSpec {
static const char* const* methods(std::size_t& n); // client -> server requests
static const char* const* notifications(std::size_t& n); // server -> client (and `initialized`)
static const char* const* server_requests(std::size_t& n); // server -> client, must be answered
static const char* const* errors(std::size_t& n); // "code:kind:retryable(1|0|?)"
};
} // namespace molo::meta
+154
View File
@@ -0,0 +1,154 @@
#include "meta/response_events.h"
#include <algorithm>
namespace molo::meta {
// EVENTS-BEGIN (parsed by tools/check_spec.py; one name per line)
static const std::vector<std::string> kKnown = {
"response.created",
"response.in_progress",
"response.queued",
"response.completed",
"response.failed",
"response.incomplete",
"response.output_item.added",
"response.output_item.done",
"response.content_part.added",
"response.content_part.done",
"response.output_text.delta",
"response.output_text.done",
"response.output_text.annotation.added",
"response.refusal.delta",
"response.refusal.done",
"response.function_call_arguments.delta",
"response.function_call_arguments.done",
"response.reasoning_summary_part.added",
"response.reasoning_summary_part.done",
"response.reasoning_summary_text.delta",
"response.reasoning_summary_text.done",
"response.reasoning_text.delta",
"response.reasoning_text.done",
"response.web_search_call.in_progress",
"response.web_search_call.searching",
"response.web_search_call.completed",
"error",
};
// EVENTS-END
const std::vector<std::string>& ResponseEvents::known() { return kKnown; }
bool ResponseEvents::is_known(const std::string& type) {
return std::find(kKnown.begin(), kKnown.end(), type) != kKnown.end();
}
ResponseEvent ResponseEvent::from_sse(const llm::SseEvent& e) {
ResponseEvent r;
r.data = e.data;
r.type = JsonX::str(e.data, "type", e.event);
r.sequence = JsonX::num(e.data, "sequence_number", -1);
return r;
}
bool ResponseEvent::terminal() const {
return type == "response.completed" || type == "response.failed" ||
type == "response.incomplete" || type == "error";
}
FunctionCallState* ResponseAccumulator::call_for(const std::string& item_id) {
for (FunctionCallState& c : calls_)
if (c.item_id == item_id) return &c;
return nullptr;
}
bool ResponseAccumulator::feed(const ResponseEvent& ev) {
++events_;
const Json& d = ev.data;
const std::string& t = ev.type;
if (t == "response.created" || t == "response.in_progress" || t == "response.queued") {
if (const Json* r = JsonX::obj(d, "response")) response_id_ = JsonX::str(*r, "id", response_id_);
} else if (t == "response.output_item.added" || t == "response.output_item.done") {
const Json* item = JsonX::obj(d, "item");
if (item) {
std::string type = JsonX::str(*item, "type");
std::string id = JsonX::str(*item, "id");
if (type == "function_call") {
FunctionCallState* c = call_for(id);
if (!c) {
calls_.push_back({});
c = &calls_.back();
c->item_id = id;
}
c->call_id = JsonX::str(*item, "call_id", c->call_id);
c->name = JsonX::str(*item, "name", c->name);
if (t == "response.output_item.done") {
std::string args = JsonX::str(*item, "arguments");
if (!args.empty()) c->arguments = args;
c->done = true;
}
} else if (type == "reasoning") {
auto it = std::find_if(reasoning_.begin(), reasoning_.end(),
[&](const ReasoningState& r) { return r.item_id == id; });
if (it == reasoning_.end()) {
reasoning_.push_back({});
reasoning_.back().item_id = id;
it = reasoning_.end() - 1;
}
if (t == "response.output_item.done") {
it->encrypted_content = JsonX::str(*item, "encrypted_content", it->encrypted_content);
if (it->summary.empty())
if (const Json* s = JsonX::arr(*item, "summary"))
for (const Json& part : s->as_array()) it->summary += JsonX::str(part, "text");
}
}
}
} else if (t == "response.output_text.delta") {
text_ += JsonX::str(d, "delta");
} else if (t == "response.output_text.done") {
if (text_.empty()) text_ = JsonX::str(d, "text");
} else if (t == "response.content_part.done") {
// The live SAM stream finalizes with content_part.done and no output_text.done.
if (const Json* p = JsonX::obj(d, "part"))
if (JsonX::str(*p, "type") == "output_text" && text_.empty()) text_ = JsonX::str(*p, "text");
} else if (t == "response.function_call_arguments.delta") {
if (FunctionCallState* c = call_for(JsonX::str(d, "item_id"))) c->arguments += JsonX::str(d, "delta");
} else if (t == "response.function_call_arguments.done") {
if (FunctionCallState* c = call_for(JsonX::str(d, "item_id"))) {
std::string args = JsonX::str(d, "arguments");
if (!args.empty()) c->arguments = args;
}
} else if (t == "response.reasoning_summary_text.delta") {
std::string id = JsonX::str(d, "item_id");
for (ReasoningState& r : reasoning_)
if (r.item_id == id) {
r.summary += JsonX::str(d, "delta");
break;
}
} else if (t == "response.completed" || t == "response.incomplete" || t == "response.failed") {
if (const Json* r = JsonX::obj(d, "response")) {
response_ = *r;
response_id_ = JsonX::str(*r, "id", response_id_);
if (const Json* e = JsonX::obj(*r, "error")) {
error_code_ = JsonX::str(*e, "code");
error_message_ = JsonX::str(*e, "message");
}
}
completed_ = t == "response.completed";
incomplete_ = t == "response.incomplete";
failed_ = t == "response.failed";
return false;
} else if (t == "error") {
failed_ = true;
error_code_ = JsonX::str(d, "code");
error_message_ = JsonX::str(d, "message");
return false;
}
return true;
}
MetaUsage ResponseAccumulator::usage() const {
const Json* u = JsonX::obj(response_, "usage");
return u ? MetaUsage::from_responses(*u) : MetaUsage();
}
} // namespace molo::meta
+80
View File
@@ -0,0 +1,80 @@
#pragma once
// Responses API server-sent events. Each SSE data block is a JSON object whose
// `type` names the event (the docs' ResponseStreamEvent union); the SSE
// `event:` line, when present, is the same name. ResponseAccumulator folds a
// stream into the final text, reasoning summary, function calls and usage, and
// remembers the terminal response document.
#include <functional>
#include <map>
#include <string>
#include <vector>
#include "llm/sse.h"
#include "meta/jsonx.h"
#include "meta/usage.h"
namespace molo::meta {
struct ResponseEvent {
std::string type; // "response.output_text.delta", ...
long sequence = -1; // sequence_number, -1 when absent
Json data; // the whole event object
static ResponseEvent from_sse(const llm::SseEvent& e);
bool terminal() const; // completed | failed | incomplete | error
};
class ResponseEvents {
public:
// Every event type in the docs' ResponseStreamEvent union (checked against
// the schemas page by tools/check_spec.py).
static const std::vector<std::string>& known();
static bool is_known(const std::string& type);
};
struct FunctionCallState {
std::string item_id;
std::string call_id;
std::string name;
std::string arguments;
bool done = false;
};
struct ReasoningState {
std::string item_id;
std::string summary;
std::string encrypted_content;
};
class ResponseAccumulator {
public:
// Returns false once a terminal event was folded.
bool feed(const ResponseEvent& event);
const std::string& text() const { return text_; }
const std::vector<ReasoningState>& reasoning() const { return reasoning_; }
const std::vector<FunctionCallState>& calls() const { return calls_; }
bool completed() const { return completed_; }
bool failed() const { return failed_; }
bool incomplete() const { return incomplete_; }
const std::string& error_message() const { return error_message_; }
const std::string& error_code() const { return error_code_; }
const std::string& response_id() const { return response_id_; }
const Json& response() const { return response_; } // terminal response document
MetaUsage usage() const;
std::size_t events() const { return events_; }
private:
FunctionCallState* call_for(const std::string& item_id);
std::string text_;
std::vector<ReasoningState> reasoning_;
std::vector<FunctionCallState> calls_;
bool completed_ = false, failed_ = false, incomplete_ = false;
std::string error_message_, error_code_, response_id_;
Json response_;
std::size_t events_ = 0;
};
} // namespace molo::meta
+254
View File
@@ -0,0 +1,254 @@
#include "meta/response_input.h"
#include "meta/caps.h"
#include "meta/check.h"
#include "security/crypto/Base64.h"
namespace molo::meta {
namespace {
Json obj(const char* type) {
Json o = Json::object();
o.set("type", Json(type));
return o;
}
void image_mime(const std::string& mime) {
Check::one_of("mime", mime,
{"image/png", "image/jpeg", "image/jpg", "image/gif", "image/webp", "image/x-icon"});
}
std::string data_url(const std::string& mime, const std::string& bytes) {
if (bytes.empty()) Check::fail("bytes are empty");
if (bytes.size() > MetaCaps::kMaxInlineBytes) Check::fail("inline file is over the 50 MB limit");
return "data:" + mime + ";base64," + security::crypto::Base64::encode(bytes);
}
} // namespace
Json ResponseInput::text(const std::string& t) {
Json o = obj("input_text");
o.set("text", Json(t));
return o;
}
Json ResponseInput::image_url(const std::string& url, const std::string& detail) {
Check::non_empty("image_url", url);
if (!detail.empty()) Check::one_of("detail", detail, {"low", "high", "auto", "original"});
Json o = obj("input_image");
o.set("image_url", Json(url));
JsonX::set_if(o, "detail", detail);
return o;
}
Json ResponseInput::image_file(const std::string& file_id, const std::string& detail) {
Check::segment("file_id", file_id);
if (!detail.empty()) Check::one_of("detail", detail, {"low", "high", "auto", "original"});
Json o = obj("input_image");
o.set("file_id", Json(file_id));
JsonX::set_if(o, "detail", detail);
return o;
}
Json ResponseInput::image_bytes(const std::string& mime, const std::string& bytes) {
image_mime(mime);
return image_url(data_url(mime, bytes));
}
Json ResponseInput::file_id(const std::string& id) {
Check::segment("file_id", id);
Json o = obj("input_file");
o.set("file_id", Json(id));
return o;
}
Json ResponseInput::file_url(const std::string& url) {
if (url.rfind("http://", 0) != 0 && url.rfind("https://", 0) != 0)
Check::fail("file_url must be http or https");
Json o = obj("input_file");
o.set("file_url", Json(url));
return o;
}
Json ResponseInput::file_data(const std::string& filename, const std::string& mime,
const std::string& bytes) {
Check::one_of("mime", mime,
{"application/pdf", "text/plain", "application/json", "application/jsonl",
"image/png", "image/jpeg", "image/gif", "image/webp", "video/mp4",
"audio/mpeg", "audio/wav"});
Json o = obj("input_file");
JsonX::set_if(o, "filename", filename);
o.set("file_data", Json(data_url(mime, bytes)));
return o;
}
Json ResponseInput::video_url(const std::string& url, double fps) {
Check::non_empty("video_url", url);
Json o = obj("input_video");
o.set("video_url", Json(url));
if (fps > 0) o.set("fps", Json(fps));
return o;
}
Json ResponseInput::video_file(const std::string& id, double fps) {
Check::segment("file_id", id);
Json o = obj("input_video");
o.set("file_id", Json(id));
if (fps > 0) o.set("fps", Json(fps));
return o;
}
Json ResponseInput::audio_bytes(const std::string& format, const std::string& bytes) {
Check::one_of("format", format, {"wav", "mp3"});
if (bytes.empty() || bytes.size() > MetaCaps::kMaxInlineBytes) Check::fail("audio size is out of range");
Json o = obj("input_audio");
Json a = Json::object();
a.set("data", Json(security::crypto::Base64::encode(bytes)));
a.set("format", Json(format));
o.set("input_audio", a);
return o;
}
Json ResponseInput::message(const std::string& role, const Json& content_parts,
const std::string& phase) {
Check::one_of("role", role, {"user", "assistant", "system", "developer"});
if (!phase.empty()) Check::one_of("phase", phase, {"commentary", "final_answer"});
Json o = obj("message");
o.set("role", Json(role));
o.set("content", content_parts);
JsonX::set_if(o, "phase", phase);
return o;
}
Json ResponseInput::user(const std::string& t) {
Json parts = Json::array();
parts.push(text(t));
return message("user", parts);
}
Json ResponseInput::developer(const std::string& t) {
Json parts = Json::array();
parts.push(text(t));
return message("developer", parts);
}
Json ResponseInput::assistant(const std::string& t, const std::string& phase) {
Json part = obj("output_text");
part.set("text", Json(t));
part.set("annotations", Json::array());
Json parts = Json::array();
parts.push(part);
return message("assistant", parts, phase);
}
Json ResponseInput::function_call(const std::string& call_id, const std::string& name,
const std::string& arguments_json) {
Check::call_id(call_id);
Check::function_name(name);
Json o = obj("function_call");
o.set("call_id", Json(call_id));
o.set("name", Json(name));
o.set("arguments", Json(arguments_json.empty() ? "{}" : arguments_json));
return o;
}
Json ResponseInput::function_call_output(const std::string& call_id, const std::string& output) {
Check::call_id(call_id);
Json o = obj("function_call_output");
o.set("call_id", Json(call_id));
o.set("output", Json(output));
return o;
}
Json ResponseInput::reasoning(const std::string& encrypted_content, const std::string& id,
const std::vector<std::string>& summary) {
Check::non_empty("encrypted_content", encrypted_content);
Json o = obj("reasoning");
if (!id.empty()) {
Check::segment("reasoning id", id);
o.set("id", Json(id));
}
o.set("encrypted_content", Json(encrypted_content));
Json s = Json::array();
for (const std::string& t : summary) {
Json part = obj("summary_text");
part.set("text", Json(t));
s.push(part);
}
o.set("summary", s); // required on replay, [] when there is none
return o;
}
Json ResponseInput::item_reference(const std::string& id) {
Check::segment("item id", id);
Json o = obj("item_reference");
o.set("id", Json(id));
return o;
}
std::size_t ResponseInput::count(const Json& input) {
if (input.is_string()) return 1;
if (input.is_array()) return input.as_array().size();
return 0;
}
Json ToolBuilder::function(const std::string& name, const std::string& description,
const Json& parameters, bool strict, bool defer_loading) {
Check::function_name(name);
Json o = obj("function");
o.set("name", Json(name));
JsonX::set_if(o, "description", description);
o.set("parameters", parameters.is_object() ? parameters : Json::object());
if (strict) o.set("strict", Json(true));
if (defer_loading) o.set("defer_loading", Json(true));
return o;
}
Json ToolBuilder::custom(const std::string& name, const std::string& description, bool strict,
bool defer_loading) {
Check::non_empty("name", name);
Json o = obj("custom");
o.set("name", Json(name));
JsonX::set_if(o, "description", description);
if (strict) o.set("strict", Json(true));
if (defer_loading) o.set("defer_loading", Json(true));
return o;
}
Json ToolBuilder::web_search(const std::string& size, const Json& location, const Json& filters) {
Json o = obj("web_search");
if (!size.empty()) {
Check::one_of("search_context_size", size, {"low", "medium", "high"});
o.set("search_context_size", Json(size));
}
if (location.is_object()) o.set("user_location", location);
if (filters.is_object()) o.set("filters", filters);
return o;
}
Json ToolBuilder::tool_search(const std::string& execution) {
Json o = obj("tool_search");
if (!execution.empty()) {
Check::one_of("execution", execution, {"server", "client"});
o.set("execution", Json(execution));
}
return o;
}
Json ToolBuilder::computer() { return obj("computer"); }
Json ToolBuilder::namespace_tool(const std::string& name, const std::string& description,
const std::vector<Json>& functions) {
Check::non_empty("name", name);
if (name.find('.') != std::string::npos) Check::fail("namespace name must contain no dots");
if (functions.empty() || functions.size() > MetaCaps::kMaxTools)
Check::fail("namespace needs 1.." + std::to_string(MetaCaps::kMaxTools) + " functions");
Json o = obj("namespace");
o.set("name", Json(name));
JsonX::set_if(o, "description", description);
Json tools = Json::array();
for (const Json& f : functions) tools.push(f);
o.set("tools", tools);
return o;
}
} // namespace molo::meta
+67
View File
@@ -0,0 +1,67 @@
#pragma once
// Typed builders for Responses API input items and tools. Shapes follow the
// Responses schemas page (InputItem, EasyInputMessage, InputText/Image/File/
// Video content, FunctionCallOutputItemParam, ReasoningItemParam, Tool union).
// Anything the docs describe only in prose (custom and namespace tools) is
// marked in SPEC "Unverified".
#include <string>
#include <vector>
#include "meta/jsonx.h"
namespace molo::meta {
class ResponseInput {
public:
// Content parts.
static Json text(const std::string& text);
static Json image_url(const std::string& url, const std::string& detail = "");
static Json image_file(const std::string& file_id, const std::string& detail = "");
// Bytes become a data: URL; mime must be a documented image type.
static Json image_bytes(const std::string& mime, const std::string& bytes);
static Json file_id(const std::string& file_id);
static Json file_url(const std::string& url);
static Json file_data(const std::string& filename, const std::string& mime,
const std::string& bytes);
static Json video_url(const std::string& url, double fps = 0);
static Json video_file(const std::string& file_id, double fps = 0);
static Json audio_bytes(const std::string& format, const std::string& bytes); // wav | mp3
// Items.
static Json message(const std::string& role, const Json& content_parts,
const std::string& phase = "");
static Json user(const std::string& text);
static Json developer(const std::string& text);
static Json assistant(const std::string& text, const std::string& phase = "");
static Json function_call(const std::string& call_id, const std::string& name,
const std::string& arguments_json);
static Json function_call_output(const std::string& call_id, const std::string& output);
// Stateless reasoning replay: `summary` may be empty, encrypted_content must
// be the exact value the previous response returned.
static Json reasoning(const std::string& encrypted_content, const std::string& id = "",
const std::vector<std::string>& summary = {});
static Json item_reference(const std::string& id);
// Counts input items of an `input` value (string = 1).
static std::size_t count(const Json& input);
};
class ToolBuilder {
public:
static Json function(const std::string& name, const std::string& description,
const Json& parameters, bool strict = false, bool defer_loading = false);
// docs: tool-calling "Custom tools" (prose only, not in the schema union).
static Json custom(const std::string& name, const std::string& description,
bool strict = false, bool defer_loading = false);
static Json web_search(const std::string& search_context_size = "",
const Json& user_location = Json(), const Json& filters = Json());
static Json tool_search(const std::string& execution = ""); // server | client
static Json computer();
// docs: tool-search "Deferring tools" (prose only).
static Json namespace_tool(const std::string& name, const std::string& description,
const std::vector<Json>& functions);
};
} // namespace molo::meta
+170
View File
@@ -0,0 +1,170 @@
#include "meta/response_request.h"
#include <algorithm>
#include "meta/caps.h"
#include "meta/check.h"
#include "meta/schema_check.h"
namespace molo::meta {
namespace {
bool has(const std::vector<std::string>& v, const char* s) {
return std::find(v.begin(), v.end(), s) != v.end();
}
} // namespace
Json ResponseRequest::format_json_object() {
Json f = Json::object();
f.set("type", Json("json_object"));
return f;
}
Json ResponseRequest::format_json_schema(const std::string& name, const Json& schema, bool strict,
const std::string& description) {
Check::non_empty("name", name);
SchemaCheck::check(schema, strict);
Json f = Json::object();
f.set("type", Json("json_schema"));
f.set("name", Json(name));
f.set("schema", schema);
if (strict) f.set("strict", Json(true));
JsonX::set_if(f, "description", description);
return f;
}
void ResponseRequest::validate(bool streaming) const {
Check::model_id(model);
if (input.is_string()) {
Check::non_empty("input", input.as_string());
Check::max_bytes("input", input.as_string(), MetaCaps::kMaxPromptBytes);
} else if (input.is_array()) {
if (input.as_array().empty() || input.as_array().size() > MetaCaps::kMaxInputItems)
Check::fail("input needs 1.." + std::to_string(MetaCaps::kMaxInputItems) + " items");
} else {
Check::fail("input is required (a string or an array of items)");
}
if (max_output_tokens && *max_output_tokens < MetaCaps::kMinOutputTokens)
Check::fail("max_output_tokens must be at least 16");
if (max_tool_calls && *max_tool_calls < 1) Check::fail("max_tool_calls must be at least 1");
if (temperature) Check::temperature(*temperature);
if (top_p) Check::top_p(*top_p);
if (frequency_penalty) Check::penalty("frequency_penalty", *frequency_penalty);
if (presence_penalty) Check::penalty("presence_penalty", *presence_penalty);
Check::reasoning_effort(model, reasoning_effort);
if (!reasoning_summary.empty())
Check::one_of("reasoning.summary", reasoning_summary, {"auto", "concise", "detailed"});
Check::safety_identifier(safety_identifier);
Check::prompt_cache_retention(prompt_cache_retention);
if (!service_tier.empty())
Check::one_of("service_tier", service_tier, {"auto", "default", "flex", "scale", "priority"});
if (!truncation.empty() && truncation != "disabled")
Check::fail("truncation \"auto\" is rejected; only \"disabled\" is accepted");
if (tools.size() > MetaCaps::kMaxTools) Check::fail("too many tools");
for (const Json& t : tools) {
if (JsonX::str(t, "type") == "function") Check::function_name(JsonX::str(t, "name"));
}
if (metadata.size() > MetaCaps::kMaxMetadataPairs) Check::fail("too many metadata pairs");
for (const std::string& i : include) {
Check::one_of("include", i,
{"file_search_call.results", "web_search_call.results",
"web_search_call.action.sources", "message.input_image.image_url",
"computer_call_output.output.image_url", "code_interpreter_call.outputs",
"reasoning.encrypted_content"});
}
if (has(include, "reasoning.encrypted_content") && !previous_response_id.empty())
Check::fail("include reasoning.encrypted_content cannot be combined with previous_response_id");
if (!previous_response_id.empty()) Check::segment("previous_response_id", previous_response_id);
if (text_format.is_object()) {
std::string type = JsonX::str(text_format, "type");
Check::one_of("text.format.type", type, {"text", "json_object", "json_schema"});
if (type == "json_schema") {
if (const Json* s = JsonX::obj(text_format, "schema"))
SchemaCheck::check(*s, JsonX::flag(text_format, "strict"));
else
Check::fail("text.format json_schema needs a schema");
}
}
if (background.value_or(false) && streaming)
Check::fail("background cannot be combined with stream");
}
Json ResponseRequest::to_json(bool stream) const {
validate(stream);
Json b = Json::object();
b.set("model", Json(model));
b.set("input", input);
JsonX::set_if(b, "instructions", instructions);
JsonX::set_if(b, "previous_response_id", previous_response_id);
if (!reasoning_effort.empty() || !reasoning_summary.empty()) {
Json r = Json::object();
JsonX::set_if(r, "effort", reasoning_effort);
JsonX::set_if(r, "summary", reasoning_summary);
b.set("reasoning", r);
}
if (!include.empty()) b.set("include", JsonX::strings(include));
if (!tools.empty()) {
Json t = Json::array();
for (const Json& x : tools) t.push(x);
b.set("tools", t);
}
if (text_format.is_object()) {
Json t = Json::object();
t.set("format", text_format);
b.set("text", t);
}
if (!metadata.empty()) {
Json m = Json::object();
for (const auto& [k, v] : metadata) m.set(k, Json(v));
b.set("metadata", m);
}
JsonX::set_if(b, "safety_identifier", safety_identifier);
JsonX::set_if(b, "prompt_cache_key", prompt_cache_key);
JsonX::set_if(b, "prompt_cache_retention", prompt_cache_retention);
JsonX::set_if(b, "service_tier", service_tier);
JsonX::set_if(b, "truncation", truncation);
if (max_output_tokens) b.set("max_output_tokens", Json(*max_output_tokens));
if (max_tool_calls) b.set("max_tool_calls", Json(*max_tool_calls));
if (temperature) b.set("temperature", Json(*temperature));
if (top_p) b.set("top_p", Json(*top_p));
if (frequency_penalty) b.set("frequency_penalty", Json(*frequency_penalty));
if (presence_penalty) b.set("presence_penalty", Json(*presence_penalty));
if (store) b.set("store", Json(*store));
if (background) b.set("background", Json(*background));
if (parallel_tool_calls) b.set("parallel_tool_calls", Json(*parallel_tool_calls));
if (stream) b.set("stream", Json(true));
JsonX::merge(b, extra);
if (background.value_or(false) && b.dump().size() > MetaCaps::kMaxBackgroundBodyBytes)
Check::fail("a background request body must stay under 1 MiB");
return b;
}
Json ResponseRequest::to_count_json() const {
validate(false);
Json b = Json::object();
b.set("model", Json(model));
b.set("input", input);
JsonX::set_if(b, "instructions", instructions);
JsonX::set_if(b, "previous_response_id", previous_response_id);
if (!reasoning_effort.empty() || !reasoning_summary.empty()) {
Json r = Json::object();
JsonX::set_if(r, "effort", reasoning_effort);
JsonX::set_if(r, "summary", reasoning_summary);
b.set("reasoning", r);
}
if (!tools.empty()) {
Json t = Json::array();
for (const Json& x : tools) t.push(x);
b.set("tools", t);
}
if (text_format.is_object()) {
Json t = Json::object();
t.set("format", text_format);
b.set("text", t);
}
JsonX::set_if(b, "truncation", truncation);
if (parallel_tool_calls) b.set("parallel_tool_calls", Json(*parallel_tool_calls));
return b;
}
} // namespace molo::meta
+65
View File
@@ -0,0 +1,65 @@
#pragma once
// A typed POST /responses (and POST /responses/input_tokens) body. validate()
// enforces the rules the docs state, before any network traffic:
// model required; input required (string or 1..kMaxInputItems items)
// max_output_tokens >= 16; temperature 0..2; top_p in (0,1]; penalties -2..2
// reasoning.effort in minimal..max ("none" rejected, "max" per catalog)
// background and stream are mutually exclusive
// include "reasoning.encrypted_content" cannot pair with previous_response_id
// truncation only "disabled"; logprobs include is rejected
// function tool names, call ids, safety_identifier length, tool count caps
// a background body stays under 1 MiB
// `extra` is merged last for fields the docs add after this code was written.
#include <optional>
#include <string>
#include <vector>
#include "meta/jsonx.h"
namespace molo::meta {
struct ResponseRequest {
std::string model;
Json input; // string or array of items (see ResponseInput)
std::string instructions;
std::string previous_response_id;
std::string reasoning_effort; // minimal..max
std::string reasoning_summary; // auto | concise | detailed
std::vector<std::string> include;
std::vector<Json> tools;
Json text_format; // null, or {type: json_object | json_schema, ...}
std::vector<std::pair<std::string, std::string>> metadata;
std::string safety_identifier;
std::string prompt_cache_key;
std::string prompt_cache_retention; // in_memory | 24h
std::string service_tier; // auto default flex scale priority
std::string truncation; // disabled only
std::optional<int> max_output_tokens;
std::optional<int> max_tool_calls;
std::optional<double> temperature;
std::optional<double> top_p;
std::optional<double> frequency_penalty;
std::optional<double> presence_penalty;
std::optional<bool> store;
std::optional<bool> background;
std::optional<bool> parallel_tool_calls;
Json extra;
void validate(bool streaming = false) const;
// Body for POST /responses. `stream` forces the stream flag.
Json to_json(bool stream = false) const;
// Body for POST /responses/input_tokens: only fields the token counter
// takes (no sampling, no store, no stream, no background).
Json to_count_json() const;
// text.format builders (structured output).
static Json format_json_object();
// Throws unless the schema is a plain JSON object schema; strict also
// requires additionalProperties:false on the root (docs: structured-output).
static Json format_json_schema(const std::string& name, const Json& schema,
bool strict = false, const std::string& description = "");
};
} // namespace molo::meta
+104
View File
@@ -0,0 +1,104 @@
#include "meta/responses.h"
#include "meta/caps.h"
#include "meta/check.h"
namespace molo::meta {
namespace {
ResponseResult from_raw(const llm::RawResponse& raw) {
ResponseResult r;
try {
r.body = raw.body.find_first_not_of(" \t\r\n") == std::string::npos ? Json::object()
: Json::parse(raw.body);
} catch (const std::exception& e) {
throw llm::ApiError("meta", 502, "BAD_RESPONSE", e.what());
}
r.limits = RateLimits::from_headers(raw.headers);
return r;
}
} // namespace
std::string ResponseResult::output_text() const {
std::string out;
if (const Json* o = JsonX::arr(body, "output"))
for (const Json& item : o->as_array()) {
if (JsonX::str(item, "type") != "message") continue;
if (const Json* c = JsonX::arr(item, "content"))
for (const Json& part : c->as_array())
if (JsonX::str(part, "type") == "output_text") out += JsonX::str(part, "text");
}
return out;
}
MetaUsage ResponseResult::usage() const {
const Json* u = JsonX::obj(body, "usage");
return u ? MetaUsage::from_responses(*u) : MetaUsage();
}
ResponseResult ResponsesService::create(const ResponseRequest& request) {
Json body = request.to_json(false);
std::string wire = body.dump();
return from_raw(client_->request_raw(method_of(Call::kResponsesCreate),
path_for(Call::kResponsesCreate), &wire,
"application/json"));
}
ResponseAccumulator ResponsesService::stream(const ResponseRequest& request,
const EventSink& sink) {
Json body = request.to_json(true);
ResponseAccumulator acc;
std::size_t seen = 0;
client_->stream(path_for(Call::kResponsesCreate), body, [&](const llm::SseEvent& e) {
if (++seen > MetaCaps::kMaxStreamEvents) return false; // hard cap on events per stream
ResponseEvent ev = ResponseEvent::from_sse(e);
bool more = acc.feed(ev);
if (sink && !sink(ev)) return false;
return more;
});
return acc;
}
ResponseResult ResponsesService::retrieve(const std::string& response_id) {
Check::segment("response_id", response_id);
return from_raw(client_->request_raw(method_of(Call::kResponsesRetrieve),
path_for(Call::kResponsesRetrieve, response_id)));
}
ResponseAccumulator ResponsesService::retrieve_stream(const std::string& response_id,
long starting_after, const EventSink& sink) {
Check::segment("response_id", response_id);
std::string path = path_for(Call::kResponsesRetrieve, response_id) + "?stream=true";
if (starting_after >= 0) path += "&starting_after=" + std::to_string(starting_after);
llm::RawResponse raw = client_->request_raw(method_of(Call::kResponsesRetrieve), path, nullptr,
"", {{"Accept", "text/event-stream"}});
ResponseAccumulator acc;
llm::SseDecoder decoder("meta", [&](const llm::SseEvent& e) {
ResponseEvent ev = ResponseEvent::from_sse(e);
bool more = acc.feed(ev);
if (sink && !sink(ev)) return false;
return more;
});
decoder.feed(raw.body);
decoder.finish();
return acc;
}
Json ResponsesService::remove(const std::string& response_id) {
Check::segment("response_id", response_id);
return client_->del(path_for(Call::kResponsesDelete, response_id));
}
Json ResponsesService::cancel(const std::string& response_id) {
Check::segment("response_id", response_id);
Json empty = Json::object();
return client_->post(path_for(Call::kResponsesCancel, response_id), empty);
}
long ResponsesService::count_input_tokens(const ResponseRequest& request) {
Json body = request.to_count_json();
Json out = client_->post(path_for(Call::kResponsesInputTokens), body);
return JsonX::num(out, "input_tokens", -1);
}
} // namespace molo::meta
+55
View File
@@ -0,0 +1,55 @@
#pragma once
// POST /responses and its lifecycle calls (retrieve, delete, cancel, count
// input tokens). The Responses API is the endpoint that carries reasoning
// across turns (stateless encrypted replay or previous_response_id), serves
// tools, search grounding, files, SAM and Muse Image, and runs background
// responses.
#include <functional>
#include <string>
#include "llm/api_client.h"
#include "meta/calls.h"
#include "meta/response_events.h"
#include "meta/response_request.h"
#include "meta/usage.h"
namespace molo::meta {
struct ResponseResult {
Json body;
RateLimits limits; // x-ratelimit-* headers when the server sent them
std::string id() const { return JsonX::str(body, "id"); }
std::string status() const { return JsonX::str(body, "status"); }
// Concatenated output_text of every assistant message item.
std::string output_text() const;
MetaUsage usage() const;
};
class ResponsesService : public llm::Service {
public:
using EventSink = std::function<bool(const ResponseEvent&)>; // false: stop
explicit ResponsesService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
ResponseResult create(const ResponseRequest& request);
// Streams events; returns the folded accumulator. A stream that ends before a
// terminal event leaves completed()/failed() false (the caller decides).
ResponseAccumulator stream(const ResponseRequest& request, const EventSink& sink = nullptr);
ResponseResult retrieve(const std::string& response_id);
// Streaming retrieval of a background response (lifecycle events only). The
// client has no GET-SSE transport, so the body is read whole, then decoded.
ResponseAccumulator retrieve_stream(const std::string& response_id, long starting_after = -1,
const EventSink& sink = nullptr);
// Destructive: soft-deletes the stored response. No default, no retry.
Json remove(const std::string& response_id);
// Cancels a background response; a finished response comes back unchanged.
Json cancel(const std::string& response_id);
// Tokenizes only: no generation, no storage, no usage record.
long count_input_tokens(const ResponseRequest& request);
};
} // namespace molo::meta
+237
View File
@@ -0,0 +1,237 @@
#include "meta/sam.h"
#include <cstdlib>
#include "llm/error.h"
#include "meta/caps.h"
#include "meta/check.h"
#include "meta/response_input.h"
namespace molo::meta {
namespace {
[[noreturn]] void bad(const std::string& why) {
throw llm::ApiError::invalid("meta", "sam output: " + why);
}
struct Cursor {
const std::string& s;
std::size_t p = 0;
bool done() const { return p >= s.size(); }
char peek() const { return p < s.size() ? s[p] : '\0'; }
bool starts(const char* lit) const { return s.compare(p, std::char_traits<char>::length(lit), lit) == 0; }
void expect(const char* lit) {
if (!starts(lit)) bad(std::string("expected ") + lit);
p += std::char_traits<char>::length(lit);
}
long integer() {
std::size_t q = p;
while (q < s.size() && s[q] >= '0' && s[q] <= '9') ++q;
if (q == p || q - p > 15) bad("expected an integer");
long v = std::strtol(s.substr(p, q - p).c_str(), nullptr, 10);
p = q;
return v;
}
};
// "x1=1;y1=2;...": key/value pairs between "<|name;" and "|>". `data` is the
// only key whose value holds commas; it never holds ';' or '|'.
std::vector<std::pair<std::string, std::string>> fields(const std::string& body) {
std::vector<std::pair<std::string, std::string>> out;
std::size_t start = 0;
while (start <= body.size()) {
std::size_t semi = body.find(';', start);
std::string kv = body.substr(start, semi == std::string::npos ? semi : semi - start);
if (!kv.empty()) {
std::size_t eq = kv.find('=');
if (eq == std::string::npos || eq == 0) bad("field without a key");
out.emplace_back(kv.substr(0, eq), kv.substr(eq + 1));
}
if (semi == std::string::npos) break;
start = semi + 1;
}
return out;
}
std::string token_body(Cursor& c, const char* open) {
c.expect(open);
std::size_t end = c.s.find("|>", c.p);
if (end == std::string::npos) bad("unterminated token");
std::string body = c.s.substr(c.p, end - c.p);
c.p = end + 2;
return body;
}
int to_int(const std::string& v) {
if (v.empty() || v.size() > 9) bad("bad integer field");
char* e = nullptr;
long x = std::strtol(v.c_str(), &e, 10);
if (*e != '\0') bad("bad integer field");
return static_cast<int>(x);
}
double to_conf(const std::string& v) {
char* e = nullptr;
double x = std::strtod(v.c_str(), &e);
if (v.empty() || *e != '\0' || x < 0.0 || x > 1.0) bad("confidence must be 0..1");
return x;
}
SamBox parse_box(const std::string& body) {
SamBox b;
int seen = 0;
for (const auto& [k, v] : fields(body)) {
if (k == "x1") { b.x1 = to_int(v); seen |= 1; }
else if (k == "y1") { b.y1 = to_int(v); seen |= 2; }
else if (k == "x2") { b.x2 = to_int(v); seen |= 4; }
else if (k == "y2") { b.y2 = to_int(v); seen |= 8; }
else if (k == "w") { b.w = to_int(v); seen |= 16; }
else if (k == "h") { b.h = to_int(v); seen |= 32; }
else if (k == "c") b.confidence = to_conf(v);
}
if (seen != 63) bad("box is missing coordinates");
return b;
}
SamMask parse_mask(const std::string& body) {
SamMask m;
bool have = false;
for (const auto& [k, v] : fields(body)) {
if (k == "c") m.confidence = to_conf(v);
else if (k == "data") {
std::size_t c1 = v.find(',');
std::size_t c2 = c1 == std::string::npos ? c1 : v.find(',', c1 + 1);
if (c2 == std::string::npos || c2 + 1 >= v.size()) bad("mask data needs H,W,<enc>payload");
m.height = to_int(v.substr(0, c1));
m.width = to_int(v.substr(c1 + 1, c2 - c1 - 1));
m.encoding = v[c2 + 1];
m.payload = v.substr(c2 + 1);
if (m.encoding != '~' && m.encoding != '!') bad("unknown mask encoding marker");
if (m.height < 1 || m.width < 1) bad("mask dimensions must be positive");
have = true;
}
}
if (!have) bad("mask has no data");
return m;
}
} // namespace
std::vector<SamFrame> SamOutput::parse(const std::string& text) {
std::vector<SamFrame> frames;
std::size_t start = 0, lines = 0, records = 0;
while (start < text.size()) {
std::size_t nl = text.find('\n', start);
std::string line = text.substr(start, nl == std::string::npos ? nl : nl - start);
start = nl == std::string::npos ? text.size() : nl + 1;
if (line.find_first_not_of(" \t\r") == std::string::npos) continue;
if (++lines > MetaCaps::kMaxSamLines) bad("too many lines");
Cursor c{line};
c.expect("<");
SamFrame frame;
frame.frame = c.integer();
c.expect("f>");
while (true) {
SamRecord r;
r.object_id = c.integer();
r.box = parse_box(token_body(c, "<|box;"));
r.mask = parse_mask(token_body(c, "<|mask;"));
if (++records > MetaCaps::kMaxSamRecords) bad("too many records");
frame.records.push_back(std::move(r));
if (c.done()) break;
c.expect(",");
}
frames.push_back(std::move(frame));
}
return frames;
}
std::size_t SamOutput::record_count(const std::vector<SamFrame>& frames) {
std::size_t n = 0;
for (const SamFrame& f : frames) n += f.records.size();
return n;
}
std::string SamPrompt::concept_text(const std::string& phrase) {
Check::non_empty("concept", phrase);
Check::max_bytes("concept", phrase, 200);
for (char c : phrase)
if (c == ',' || c == ';' || c == '+' || c == '\n' || c == '<' || c == '>' || c == '|')
Check::fail("one concept per request: no lists, newlines or tokens");
if (phrase.find(" and ") != std::string::npos || phrase.find(" or ") != std::string::npos)
Check::fail("one concept per request: \"and\" / \"or\" segment only one of them");
return phrase;
}
namespace {
std::string box_token(const SamBox& b) {
return "<|box;x1=" + std::to_string(b.x1) + ";y1=" + std::to_string(b.y1) + ";x2=" +
std::to_string(b.x2) + ";y2=" + std::to_string(b.y2) + ";w=" + std::to_string(b.w) +
";h=" + std::to_string(b.h) + "|>";
}
} // namespace
std::string SamPrompt::objects_text(long frame, const std::vector<SamObject>& objects) {
if (frame < 0) Check::fail("frame must be >= 0");
if (objects.empty()) Check::fail("a frame needs at least one object");
std::string out = "<" + std::to_string(frame) + "f>";
std::vector<long> ids;
for (std::size_t i = 0; i < objects.size(); ++i) {
const SamObject& o = objects[i];
for (long id : ids)
if (id == o.id) Check::fail("an object id appears at most once per frame");
ids.push_back(o.id);
if (o.id < 0) Check::fail("object id must be >= 0");
if (!o.box && o.points.empty()) Check::fail("an object needs a box or a point");
if (i) out += ",";
out += std::to_string(o.id);
if (o.box) {
if (o.box->w < 1 || o.box->h < 1) Check::fail("box needs the media size w,h");
if (o.box->x2 < o.box->x1 || o.box->y2 < o.box->y1) Check::fail("box corners are reversed");
out += box_token(*o.box);
}
for (const SamPoint& p : o.points) {
if (o.w < 1 || o.h < 1) Check::fail("points need the media size w,h");
if (p.x < 0 || p.y < 0 || p.x >= o.w || p.y >= o.h) Check::fail("point is outside the media");
if (!p.positive) out += "-";
out += "<|point;x=" + std::to_string(p.x) + ";y=" + std::to_string(p.y) + ";w=" +
std::to_string(o.w) + ";h=" + std::to_string(o.h) + "|>";
}
}
return out;
}
std::string SamPrompt::frames_text(
const std::vector<std::pair<long, std::vector<SamObject>>>& frames) {
if (frames.empty()) Check::fail("no frames");
std::string out;
for (const auto& [frame, objects] : frames) out += objects_text(frame, objects);
return out;
}
ResponseRequest SamRequest::to_response_request() const {
Check::non_empty("prompt", prompt);
int media = (!image_url.empty() ? 1 : 0) + (!video_url.empty() ? 1 : 0) +
(!video_file_id.empty() ? 1 : 0);
if (media != 1) Check::fail("send exactly one of image_url, video_url, video_file_id");
if (!propagation_direction.empty())
Check::one_of("propagation_direction", propagation_direction, {"both", "forward", "backward"});
if (!mask_encoding.empty()) Check::one_of("mask_encoding", mask_encoding, {"lossless", "one_bit"});
Json parts = Json::array();
parts.push(ResponseInput::text(prompt));
if (!image_url.empty()) parts.push(ResponseInput::image_url(image_url));
else if (!video_url.empty()) parts.push(ResponseInput::video_url(video_url));
else parts.push(ResponseInput::video_file(video_file_id));
ResponseRequest r;
r.model = model;
Json input = Json::array();
input.push(ResponseInput::message("user", parts));
r.input = input;
if (!propagation_direction.empty()) r.metadata.emplace_back("propagation_direction", propagation_direction);
if (include_confidence) r.metadata.emplace_back("include_confidence", "true"); // string, not a boolean
if (!mask_encoding.empty()) r.metadata.emplace_back("mask_encoding", mask_encoding);
return r;
}
} // namespace molo::meta
+98
View File
@@ -0,0 +1,98 @@
#pragma once
// Segment Anything Model 3.1 (model sam-3.1) over the Responses API. There is
// no segmentation endpoint: the prompt (a noun phrase, or box / point tokens)
// goes in an input_text part beside one input_image or input_video part, and
// the answer arrives as special-token text in one output_text lane, one line
// per frame:
// <Nf>id<|box;x1=..;y1=..;x2=..;y2=..;w=..;h=..[;c=..]|><|mask;x=0;y=0[;c=..];data=H,W,<enc>payload|>,id...
// This module builds the request, parses that text, and decodes masks
// (one_bit "!" and lossless "~") to a 0/1 raster. The mask codec is ported
// from the official meta-sam parser (see SPEC Sources) and checked against its
// conformance cases.
#include <cstdint>
#include <optional>
#include <string>
#include <vector>
#include "meta/response_request.h"
namespace molo::meta {
struct SamBox {
int x1 = 0, y1 = 0, x2 = 0, y2 = 0; // x2 / y2 are inclusive, as on the wire
int w = 0, h = 0; // source frame size
double confidence = -1; // -1 = absent
int right() const { return x2 + 1; } // exclusive edge
int bottom() const { return y2 + 1; }
};
struct SamMask {
int height = 0, width = 0;
char encoding = 0; // '~' lossless, '!' one_bit
std::string payload; // including the encoding marker
double confidence = -1;
// Row-major height*width bytes of 0/1. Throws ApiError on a malformed payload.
std::vector<std::uint8_t> decode() const;
std::string encoding_name() const { return encoding == '!' ? "one_bit" : "lossless"; }
};
struct SamRecord {
long object_id = 0;
SamBox box;
SamMask mask;
};
struct SamFrame {
long frame = 0; // zero-based, not necessarily contiguous or ascending
std::vector<SamRecord> records;
};
class SamOutput {
public:
// Parses the whole output_text (lines separated by \n). Empty text is a valid
// "no matches" result. Throws ApiError on a malformed line; caps lines and
// records.
static std::vector<SamFrame> parse(const std::string& text);
static std::size_t record_count(const std::vector<SamFrame>& frames);
};
struct SamPoint {
int x = 0, y = 0;
bool positive = true;
};
struct SamObject {
long id = 0;
std::optional<SamBox> box; // x1..y2 and w,h
std::vector<SamPoint> points;
int w = 0, h = 0; // media size for the points
};
class SamPrompt {
public:
// One short noun phrase; commas, semicolons, plus signs, newlines and
// "and" list several concepts (docs: that returns nothing), so they are rejected.
static std::string concept_text(const std::string& phrase);
// Box and point prompts for one frame: "<Nf>id<|box;...|>,id<|point;...|>-<|point;...|>".
static std::string objects_text(long frame, const std::vector<SamObject>& objects);
// Several frames, in the order given.
static std::string frames_text(const std::vector<std::pair<long, std::vector<SamObject>>>& frames);
};
struct SamRequest {
std::string model = "sam-3.1";
std::string prompt; // from SamPrompt
std::string image_url; // exactly one of image_url / video_url / video_file_id
std::string video_url;
std::string video_file_id;
std::string propagation_direction; // "" (both) | forward | backward | both (video prompts)
bool include_confidence = false;
std::string mask_encoding; // "" (lossless) | lossless | one_bit
bool stream = true; // docs: recommended
ResponseRequest to_response_request() const;
};
} // namespace molo::meta
+270
View File
@@ -0,0 +1,270 @@
// Mask codec for SAM output: base85 (custom alphabet) + adaptive range coder.
// Ported from the official meta-sam parser (python/src/meta_sam_parser/
// _mask_codec.py); verified against its conformance cases in the tests.
#include <cstdint>
#include <map>
#include <array>
#include "llm/error.h"
#include "meta/caps.h"
#include "meta/sam.h"
namespace molo::meta {
namespace {
[[noreturn]] void bad(const std::string& why) {
throw llm::ApiError::invalid("meta", "sam mask: " + why);
}
constexpr int kRadix = 85;
constexpr int kPrefix = 5;
constexpr std::uint32_t kTop = 1u << 24;
constexpr std::uint32_t kBottom = 1u << 16;
int digit_of(char c) {
static int table[128];
static bool init = false;
if (!init) {
for (int& t : table) t = -1;
int idx = 0;
for (int code = 0x21; code < 0x7F; ++code) {
char ch = static_cast<char>(code);
if (ch == '"' || ch == '\\' || ch == ',' || ch == ';' || ch == '<' || ch == '>' || ch == '|') continue;
table[code] = idx++;
}
init = true;
}
unsigned char u = static_cast<unsigned char>(c);
int v = u < 128 ? table[u] : -1;
if (v < 0 || v >= kRadix) bad("payload has an invalid character");
return v;
}
std::vector<std::uint8_t> unpack(const std::string& p) {
if (p.size() < kPrefix) bad("payload is missing its length prefix");
std::uint64_t length = 0;
for (int i = 0; i < kPrefix; ++i) length = length * kRadix + digit_of(p[i]);
if (length > MetaCaps::kMaxMaskPixels * 4ull) bad("payload length is unsupported");
std::uint64_t rem = length % 4;
std::uint64_t expected = kPrefix + (length / 4) * 5 + (rem == 0 ? 0 : rem + 1);
if (p.size() != expected) bad("payload length does not match its prefix");
std::vector<std::uint8_t> out(length);
std::size_t src = kPrefix, dst = 0;
std::uint64_t full = length - rem;
while (dst < full) {
std::uint64_t v = 0;
for (int i = 0; i < 5; ++i) v = v * kRadix + digit_of(p[src + i]);
if (v > 0xFFFFFFFFull) bad("payload group is out of range");
src += 5;
out[dst] = (v >> 24) & 0xFF; out[dst + 1] = (v >> 16) & 0xFF;
out[dst + 2] = (v >> 8) & 0xFF; out[dst + 3] = v & 0xFF;
dst += 4;
}
if (rem) {
std::uint64_t v = 0;
for (std::uint64_t i = 0; i < 5; ++i) v = v * kRadix + (i < rem + 1 ? digit_of(p[src + i]) : kRadix - 1);
if (v > 0xFFFFFFFFull) bad("payload tail is out of range");
for (std::uint64_t i = 0; i < rem; ++i) out[dst + i] = (v >> (24 - i * 8)) & 0xFF;
}
return out;
}
class RangeDecoder {
public:
RangeDecoder(const std::uint8_t* d, std::size_t n) : in_(d), n_(n) {
if (n < 4) bad("payload is missing its finalization");
for (int i = 0; i < 4; ++i) code_ = (code_ << 8) | read();
}
std::uint32_t freq(std::uint32_t total) {
scaled_ = range_ / total;
if (scaled_ == 0) bad("invalid coding state");
std::uint32_t v = (code_ - low_) / scaled_;
return v >= total ? total - 1 : v;
}
void decode(std::uint32_t cum, std::uint32_t f) {
low_ += scaled_ * cum;
range_ = scaled_ * f;
while (true) {
if ((low_ ^ (low_ + range_)) < kTop) {
} else if (range_ < kBottom) {
range_ = (0u - low_) & (kBottom - 1);
} else {
break;
}
code_ = (code_ << 8) | read();
low_ <<= 8;
range_ <<= 8;
}
}
private:
std::uint32_t read() {
if (pos_ >= n_) bad("payload ended before decoding completed");
return in_[pos_++];
}
const std::uint8_t* in_;
std::size_t n_, pos_ = 0;
std::uint32_t low_ = 0, range_ = 0xFFFFFFFFu, code_ = 0, scaled_ = 0;
};
std::vector<std::uint8_t> decode_one_bit(const std::string& payload, int width, int height) {
std::vector<std::uint8_t> raw = unpack(payload.substr(1));
RangeDecoder dec(raw.data(), raw.size());
std::vector<std::uint16_t> counts(4096 * 2, 1);
const int pw = width + 4;
std::vector<std::uint8_t> padded(static_cast<std::size_t>(pw) * (height + 2), 0);
std::vector<std::uint8_t> out(static_cast<std::size_t>(width) * height);
for (int y = 0; y < height; ++y) {
std::size_t row = static_cast<std::size_t>(y + 2) * pw + 2;
for (int x = 0; x < width; ++x) {
std::size_t pos = row + x, prior = pos - pw, earlier = pos - 2 * pw;
unsigned ctx = padded[pos - 1] | (padded[pos - 2] << 1) | (padded[prior - 2] << 2) |
(padded[prior - 1] << 3) | (padded[prior] << 4) | (padded[prior + 1] << 5) |
(padded[prior + 2] << 6) | (padded[earlier - 2] << 7) |
(padded[earlier - 1] << 8) | (padded[earlier] << 9) |
(padded[earlier + 1] << 10) | (padded[earlier + 2] << 11);
std::size_t off = static_cast<std::size_t>(ctx) << 1;
std::uint32_t zero = counts[off], one = counts[off + 1];
std::uint32_t v = dec.freq(zero + one);
int bit = v < zero ? 0 : 1;
if (bit == 0) dec.decode(0, zero); else dec.decode(zero, one);
padded[pos] = static_cast<std::uint8_t>(bit);
out[static_cast<std::size_t>(y) * width + x] = static_cast<std::uint8_t>(bit);
counts[off + bit] = static_cast<std::uint16_t>(counts[off + bit] + 14);
if (counts[off] + counts[off + 1] >= 4096) {
counts[off] = static_cast<std::uint16_t>((counts[off] >> 1) ? (counts[off] >> 1) : 1);
counts[off + 1] = static_cast<std::uint16_t>((counts[off + 1] >> 1) ? (counts[off + 1] >> 1) : 1);
}
}
}
return out;
}
int paeth(int l, int u, int ul) {
int est = l + u - ul;
int dl = std::abs(est - l), du = std::abs(est - u), dul = std::abs(est - ul);
if (dl <= du && dl <= dul) return l;
return du <= dul ? u : ul;
}
int predict(const std::vector<std::uint8_t>& out, std::size_t off, int x, int y, int mode, int w) {
if (mode < 0 || mode >= 5) bad("invalid predictor");
int left = x > 0 ? out[off - 1] : 0;
int up = y > 0 ? out[off - w] : 0;
if (mode == 1) return up;
if (mode == 2) return left;
if (mode == 3) return (left + up) >> 1;
int ul = (x > 0 && y > 0) ? out[off - w - 1] : 0;
if (mode == 0) return paeth(left, up, ul);
if (ul >= std::max(left, up)) return std::min(left, up);
if (ul <= std::min(left, up)) return std::max(left, up);
return left + up - ul;
}
int zero_ctx(const std::vector<std::uint8_t>& r, std::size_t off, int x, int y, int w) {
int left = x > 0 ? (r[off - 1] == 0) : 1;
int up = y > 0 ? (r[off - w] == 0) : 1;
int ul = (x > 0 && y > 0) ? (r[off - w - 1] == 0) : 1;
int ur = (y > 0 && x < w - 1) ? (r[off - w + 1] == 0) : 1;
return left | (up << 1) | (ul << 2) | (ur << 3);
}
std::vector<std::uint8_t> decode_lossless(const std::string& payload, int width, int height) {
std::vector<std::uint8_t> packed = unpack(payload.substr(1));
if (packed.size() < 5) bad("lossless payload is truncated");
int selector = packed[0];
RangeDecoder dec(packed.data() + 1, packed.size() - 1);
const std::size_t length = static_cast<std::size_t>(width) * height;
std::vector<std::uint8_t> out(length), resid(length);
std::vector<std::uint32_t> zero_counts(32, 1);
std::vector<std::uint16_t> order_one(256 * 256, 1);
std::vector<std::int32_t> order_one_totals(256, 256);
std::vector<std::uint16_t> order_zero(256, 2);
std::int64_t order_zero_total = 512;
std::map<int, std::array<std::uint16_t, 257>> order_two;
int previous = 0;
for (std::size_t off = 0; off < length; ++off) {
int x = static_cast<int>(off % width), y = static_cast<int>(off / width);
std::size_t zo = static_cast<std::size_t>(zero_ctx(resid, off, x, y, width)) << 1;
std::uint32_t zero = zero_counts[zo], nonzero = zero_counts[zo + 1];
std::uint32_t zf = dec.freq(zero + nonzero);
int bit = zf < zero ? 0 : 1;
dec.decode(bit == 0 ? 0 : zero, bit == 0 ? zero : nonzero);
zero_counts[zo + bit] += 32;
if (zero_counts[zo] + zero_counts[zo + 1] >= 16384) {
zero_counts[zo] = (zero_counts[zo] >> 1) ? (zero_counts[zo] >> 1) : 1;
zero_counts[zo + 1] = (zero_counts[zo + 1] >> 1) ? (zero_counts[zo + 1] >> 1) : 1;
}
int symbol = 0;
if (bit == 1) {
int above = off >= static_cast<std::size_t>(width) ? resid[off - width] : 0;
auto it = order_two.find((previous << 8) | above);
if (it == order_two.end()) {
std::array<std::uint16_t, 257> z{};
it = order_two.emplace((previous << 8) | above, z).first;
}
auto& second = it->second;
std::size_t fo = static_cast<std::size_t>(previous) << 8;
std::int64_t excluded = second[0] + order_one[fo] + order_zero[0];
std::int64_t total = second[256] + order_one_totals[previous] + order_zero_total - excluded;
if (total <= 0) bad("lossless payload is malformed");
std::uint32_t target = dec.freq(static_cast<std::uint32_t>(total));
std::int64_t cum = 0;
symbol = 1;
std::int64_t f = second[1] + order_one[fo + 1] + order_zero[1];
while (cum + f <= target) {
cum += f;
++symbol;
if (symbol >= 256) bad("lossless payload is malformed");
f = second[symbol] + order_one[fo + symbol] + order_zero[symbol];
}
dec.decode(static_cast<std::uint32_t>(cum), static_cast<std::uint32_t>(f));
second[symbol] = static_cast<std::uint16_t>(second[symbol] + 14 * 44);
second[256] = static_cast<std::uint16_t>(second[256] + 14 * 44);
if (second[256] >= 14 * 3584) {
std::uint32_t t = 0;
for (int v = 0; v < 256; ++v) { second[v] >>= 1; t += second[v]; }
second[256] = static_cast<std::uint16_t>(t);
}
order_one[fo + symbol] = static_cast<std::uint16_t>(order_one[fo + symbol] + 56);
order_one_totals[previous] += 56;
if (order_one_totals[previous] >= 8192) {
std::int32_t t = 0;
for (int v = 0; v < 256; ++v) {
std::uint16_t s = static_cast<std::uint16_t>((order_one[fo + v] >> 1) ? (order_one[fo + v] >> 1) : 1);
order_one[fo + v] = s;
t += s;
}
order_one_totals[previous] = t;
}
order_zero[symbol] = static_cast<std::uint16_t>(order_zero[symbol] + 32);
order_zero_total += 32;
if (order_zero_total >= 4096) {
std::int64_t t = 0;
for (int v = 0; v < 256; ++v) { order_zero[v] >>= 1; t += order_zero[v]; }
order_zero_total = t;
}
}
resid[off] = static_cast<std::uint8_t>(symbol);
int delta = (symbol & 1) ? -((symbol + 1) >> 1) : (symbol >> 1);
out[off] = static_cast<std::uint8_t>((predict(out, off, x, y, selector, width) + delta) & 0xFF);
previous = symbol;
}
for (std::uint8_t& v : out) v = v >= 129 ? 1 : 0;
return out;
}
} // namespace
std::vector<std::uint8_t> SamMask::decode() const {
if (width < 1 || height < 1 ||
static_cast<std::size_t>(width) * static_cast<std::size_t>(height) > MetaCaps::kMaxMaskPixels)
bad("mask dimensions are out of range");
if (payload.empty() || payload[0] != encoding) bad("payload does not start with its marker");
if (encoding == '!') return decode_one_bit(payload, width, height);
if (encoding == '~') return decode_lossless(payload, width, height);
bad("unsupported encoding");
}
} // namespace molo::meta
+107
View File
@@ -0,0 +1,107 @@
#include "meta/schema_check.h"
#include <set>
#include "meta/check.h"
namespace molo::meta {
namespace {
struct Walker {
const Json& root;
bool strict;
Walker(const Json& r, bool s) : root(r), strict(s) {}
long properties = 0;
long chars = 0;
long enums = 0;
std::set<std::string> resolving; // $ref targets on the current path
const Json* resolve(const std::string& ref) const {
for (const char* prefix : {"#/$defs/", "#/definitions/"}) {
std::string p = prefix;
if (ref.rfind(p, 0) == 0) {
const char* key = p == "#/$defs/" ? "$defs" : "definitions";
const Json* defs = JsonX::obj(root, key);
return defs ? defs->find(ref.substr(p.size())) : nullptr;
}
}
return nullptr;
}
void walk(const Json& node, int depth) {
if (depth > SchemaCheck::kMaxDepth) Check::fail("schema nests deeper than 10 levels");
if (!node.is_object()) return;
std::string ref = JsonX::str(node, "$ref");
if (!ref.empty()) {
if (resolving.count(ref)) Check::fail("recursive schemas are not supported");
const Json* target = resolve(ref);
if (target) {
resolving.insert(ref);
walk(*target, depth + 1);
resolving.erase(ref);
}
}
if (strict) {
if (node.has("allOf") || node.has("oneOf"))
Check::fail("strict schemas cannot use allOf or oneOf");
}
if (const Json* e = JsonX::arr(node, "enum")) {
enums += static_cast<long>(e->as_array().size());
for (const Json& v : e->as_array())
if (v.is_string()) chars += static_cast<long>(v.as_string().size());
}
if (const Json* c = JsonX::member(node, "const"); c && c->is_string())
chars += static_cast<long>(c->as_string().size());
if (const Json* props = JsonX::obj(node, "properties")) {
properties += static_cast<long>(props->as_object().size());
if (strict) {
if (!(node.has("additionalProperties") &&
!JsonX::flag(node, "additionalProperties", true) &&
node.find("additionalProperties")->is_bool()))
Check::fail("strict schemas need additionalProperties:false on every object");
const Json* req = JsonX::arr(node, "required");
std::set<std::string> required;
if (req)
for (const Json& r : req->as_array())
if (r.is_string()) required.insert(r.as_string());
for (const auto& [k, v] : props->as_object())
if (!required.count(k)) Check::fail("strict schemas must list every property in required");
}
for (const auto& [k, v] : props->as_object()) {
chars += static_cast<long>(k.size());
walk(v, depth + 1);
}
}
for (const char* key : {"items", "additionalProperties", "not"})
if (const Json* v = JsonX::member(node, key); v && v->is_object()) walk(*v, depth + 1);
for (const char* key : {"anyOf", "oneOf", "allOf", "prefixItems"})
if (const Json* a = JsonX::arr(node, key))
for (const Json& v : a->as_array()) walk(v, depth + 1);
for (const char* key : {"$defs", "definitions"})
if (const Json* d = JsonX::obj(node, key))
for (const auto& [k, v] : d->as_object()) {
chars += static_cast<long>(k.size());
if (&node != &root) walk(v, depth + 1);
}
}
};
} // namespace
void SchemaCheck::check(const Json& schema, bool strict) {
if (!schema.is_object()) Check::fail("schema must be a JSON object");
if (strict) {
std::string type = JsonX::str(schema, "type");
if (type != "object") Check::fail("strict schemas need a plain object root");
for (const char* k : {"anyOf", "oneOf", "allOf", "enum", "not"})
if (schema.has(k)) Check::fail(std::string("strict schema root cannot use ") + k);
}
Walker w(schema, strict);
w.walk(schema, 1);
if (w.properties > kMaxProperties) Check::fail("schema has more than 5000 properties");
if (w.chars > kMaxStringChars) Check::fail("schema names and enum values exceed 120000 chars");
if (w.enums > kMaxEnumValues) Check::fail("schema has more than 1000 enum values");
}
} // namespace molo::meta
+27
View File
@@ -0,0 +1,27 @@
#pragma once
// Local pre-flight for JSON Schemas sent as structured output (response_format
// / text.format) or function-tool parameters. Limits and strict-mode rules are
// the ones stated in docs/structured-output:
// nesting depth <= 10, total properties <= 5000, total name/enum/const
// string length <= 120000, enum values <= 1000, no recursive ($ref cycle)
// schemas; strict: plain object root, additionalProperties:false on every
// object, required lists every property, no allOf/oneOf.
// The server stays the authority; this only rejects what it documents it
// rejects, so a bad schema fails before it costs a round trip.
#include "meta/jsonx.h"
namespace molo::meta {
class SchemaCheck {
public:
static constexpr int kMaxDepth = 10;
static constexpr long kMaxProperties = 5000;
static constexpr long kMaxStringChars = 120000;
static constexpr long kMaxEnumValues = 1000;
// Throws llm::ApiError::invalid on a violation.
static void check(const Json& schema, bool strict);
};
} // namespace molo::meta
+316
View File
@@ -0,0 +1,316 @@
#include "meta/speech.h"
#include <chrono>
#include "google/live/ws_error.h"
#include "google/live/ws_url.h"
#include "meta/calls.h"
#include "meta/caps.h"
#include "meta/check.h"
#include "meta/client.h"
namespace molo::meta {
namespace {
long le32(const std::string& s, std::size_t p) {
return static_cast<long>(static_cast<unsigned char>(s[p])) |
(static_cast<long>(static_cast<unsigned char>(s[p + 1])) << 8) |
(static_cast<long>(static_cast<unsigned char>(s[p + 2])) << 16) |
(static_cast<long>(static_cast<unsigned char>(s[p + 3])) << 24);
}
int le16(const std::string& s, std::size_t p) {
return static_cast<unsigned char>(s[p]) | (static_cast<unsigned char>(s[p + 1]) << 8);
}
long now_ms() {
return std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now().time_since_epoch()).count();
}
} // namespace
void SpeechOptions::validate() const {
Check::model_id(model);
if (!mode.empty()) Check::one_of("mode", mode, {"PUSH_TO_TALK", "ENDPOINTING", "DIARIZATION"});
if (!partial_mode.empty()) Check::one_of("partialMode", partial_mode, {"CUMULATIVE", "DELTA"});
if (keywords.size() > MetaCaps::kMaxKeywords) Check::fail("too many keywords");
for (const std::string& k : keywords) {
Check::non_empty("keyword", k);
Check::max_bytes("keyword", k, 256);
}
if (language_bias.size() > 25) Check::fail("languageBias takes at most 25 languages");
for (const std::string& l : language_bias) {
Check::non_empty("language", l);
Check::max_bytes("language", l, 64);
}
if (!session_id.empty()) Check::segment("sessionId", session_id);
}
Json SpeechOptions::settings_json(const std::string& audio_encoding) const {
Json j = Json::object();
j.set("model", Json(model));
j.set("audioEncoding", Json(audio_encoding));
JsonX::set_if(j, "mode", mode);
JsonX::set_if(j, "partialMode", partial_mode);
if (!keywords.empty()) j.set("keywords", JsonX::strings(keywords));
if (!language_bias.empty()) j.set("languageBias", JsonX::strings(language_bias));
if (emit_audio_progress) j.set("emitAudioProgress", Json(true));
return j;
}
WavInfo WavInfo::parse(const std::string& w) {
if (w.size() < 44 || w.compare(0, 4, "RIFF") != 0 || w.compare(8, 4, "WAVE") != 0)
Check::fail("audio is not a RIFF/WAVE file");
if (w.size() > MetaCaps::kMaxTranscribeBytes) Check::fail("audio is over the 32 MB request limit");
WavInfo info;
bool have_fmt = false, have_data = false;
std::size_t pos = 12;
while (pos + 8 <= w.size()) {
std::string id = w.substr(pos, 4);
long size = le32(w, pos + 4);
std::size_t body = pos + 8;
if (id == "fmt " && body + 16 <= w.size()) {
int format = le16(w, body);
info.channels = le16(w, body + 2);
info.sample_rate = static_cast<int>(le32(w, body + 4));
info.bits = le16(w, body + 14);
if (format != 1) Check::fail("WAV must be integer PCM");
have_fmt = true;
} else if (id == "data") {
long avail = static_cast<long>(w.size() - body);
info.data_bytes = size > avail || size < 0 ? avail : size;
have_data = true;
break;
}
if (size < 0) break;
pos = body + static_cast<std::size_t>(size) + (static_cast<std::size_t>(size) & 1u);
}
if (!have_fmt || !have_data) Check::fail("WAV is missing its fmt or data chunk");
if (info.channels != 1 || info.bits != 16)
Check::fail("WAV must be mono 16-bit PCM");
if (info.sample_rate != 16000 && info.sample_rate != 24000)
Check::fail("WAV must be 16 kHz or 24 kHz");
if (info.data_bytes <= 0) Check::fail("WAV holds no audio");
info.duration_ms = info.data_bytes * 1000L / (static_cast<long>(info.sample_rate) * 2L);
if (info.duration_ms > MetaCaps::kMaxTranscribeMs) Check::fail("audio is longer than 10 minutes");
return info;
}
TranscribeResult TranscribeResult::from_json(const Json& j) {
TranscribeResult r;
r.session_id = JsonX::str(j, "sessionId");
r.transcript = JsonX::str(j, "transcript");
r.audio_duration_ms = JsonX::num(j, "audioDurationMs");
if (const Json* t = JsonX::arr(j, "turns"))
for (const Json& x : t->as_array()) {
SpeechTurn turn;
turn.turn_id = JsonX::num(x, "turnId");
turn.start_ms = JsonX::num(x, "startMs");
turn.end_ms = JsonX::num(x, "endMs");
turn.transcript = JsonX::str(x, "transcript");
turn.speaker = JsonX::str(x, "speaker");
r.turns.push_back(std::move(turn));
}
return r;
}
const std::vector<std::string>& SpeechEvent::known_types() {
static const std::vector<std::string> v = {"transcript", "speechStart", "speechEnd",
"speechComplete", "speaker", "audioProgress", "error"};
return v;
}
SpeechEvent SpeechEvent::parse(const Json& j) {
SpeechEvent e;
e.type = JsonX::str(j, "type");
e.transcript = JsonX::str(j, "transcript");
e.final_flag = JsonX::flag(j, "final");
e.audio_processed_ms = JsonX::num(j, "audioProcessedMs");
e.turn_id = JsonX::num(j, "turnId", -1);
e.speaker_label = JsonX::str(j, "label");
e.session_id = JsonX::str(j, "sessionId");
e.message = JsonX::str(j, "message");
e.error_type = JsonX::str(j, "errorType");
e.error_code = JsonX::str(j, "errorCode");
return e;
}
llm::RawResponse TranscribeService::post(const SpeechOptions& options, const std::string& wav,
const char* accept) {
options.validate();
WavInfo::parse(wav);
// `request` is a form field with a JSON content type and no filename (the
// docs' curl: -F 'request={...};type=application/json'); audio is the file.
const std::string boundary = clients::body::MultipartBody().boundary();
std::string body;
body += "--" + boundary + "\r\nContent-Disposition: form-data; name=\"request\"\r\n";
body += "Content-Type: application/json\r\n\r\n" + options.settings_json("WAV").dump() + "\r\n";
body += "--" + boundary + "\r\nContent-Disposition: form-data; name=\"audio\"; filename=\"audio.wav\"\r\n";
body += "Content-Type: audio/wav\r\n\r\n" + wav + "\r\n--" + boundary + "--\r\n";
std::string path = path_for(Call::kAsrTranscribe);
if (!options.session_id.empty()) path += "?sessionId=" + options.session_id;
return client_->request_raw(clients::Method::kPost, path, &body, "multipart/form-data; boundary=" + boundary,
{{"Accept", accept}});
}
TranscribeResult TranscribeService::transcribe(const SpeechOptions& options,
const std::string& wav) {
llm::RawResponse raw = post(options, wav, "application/json");
try {
return TranscribeResult::from_json(Json::parse(raw.body));
} catch (const std::exception& e) {
throw llm::ApiError("meta", 502, "BAD_RESPONSE", e.what());
}
}
std::vector<SpeechEvent> TranscribeService::transcribe_events(const SpeechOptions& options,
const std::string& wav) {
llm::RawResponse raw = post(options, wav, "text/event-stream");
std::vector<SpeechEvent> out;
llm::SseDecoder decoder("meta", [&](const llm::SseEvent& e) {
if (out.size() >= MetaCaps::kMaxRealtimeEvents) return false;
SpeechEvent ev = SpeechEvent::parse(e.data);
if (ev.type.empty()) ev.type = e.event;
out.push_back(std::move(ev));
return true;
});
decoder.feed(raw.body);
decoder.finish();
return out;
}
std::string TranscribeService::transcribe_text(const SpeechOptions& options,
const std::string& wav) {
return post(options, wav, "text/plain").body;
}
void RealtimeConfig::validate() const {
options.validate();
Check::one_of("audioEncoding", audio_encoding, {"PCM_16KHZ", "PCM_24KHZ"});
}
RealtimeSession::RealtimeSession(RealtimeConfig config) : config_(std::move(config)) {
if (config_.url.empty()) config_.url = MetaEndpoints::kRealtimeUrl;
if (config_.key.empty()) config_.key = MetaClient::key_from_env();
}
RealtimeSession::~RealtimeSession() {
try {
close();
} catch (...) {
}
}
Json RealtimeSession::handshake_json(bool reveal) const {
Json j = config_.options.settings_json(config_.audio_encoding);
Json auth = Json::object();
auth.set("accessToken", Json("Bearer " + (reveal ? config_.key : std::string("***"))));
j.set("authorization", auth);
return j;
}
void RealtimeSession::connect() {
config_.validate();
if (open_) Check::fail("session already connected");
if (config_.key.empty()) throw llm::NotConfiguredError("meta", MetaEndpoints::kKeyEnv);
using namespace google::live;
try {
WsUrl url = WsUrl::parse(config_.url);
if (!config_.options.session_id.empty())
url.target += (url.target.find('?') == std::string::npos ? "?" : "&") +
std::string("sessionId=") + config_.options.session_id;
ws_ = std::make_unique<WsClient>(
open_stream(url, config_.connect_timeout_ms, MetaCaps::kRealtimeIdleMs, config_.tls));
ws_->handshake(url, {}, config_.handshake_timeout_ms); // no Authorization header
ws_->send_text(handshake_json(true).dump());
ws_->set_read_timeout_ms(250);
long deadline = now_ms() + config_.ack_timeout_ms;
while (session_id_.empty()) {
if (now_ms() > deadline) throw llm::ApiError("meta", 504, "TIMEOUT", "handshake acknowledgement not received");
auto m = ws_->receive();
if (!m) continue;
if (m->kind == WsMessage::Kind::Close) {
closed_ = true;
close_code_ = m->code;
throw llm::ApiError("meta", m->code == 1013 ? 429 : 400, "REALTIME_CLOSED",
"closed before the acknowledgement (code " + std::to_string(m->code) +
"): " + m->data.substr(0, 300));
}
Json j = Json::parse(m->data);
if (JsonX::str(j, "type") == "error")
throw llm::ApiError("meta", 400, JsonX::str(j, "errorType", "error"), JsonX::str(j, "message"));
session_id_ = JsonX::str(j, "sessionId");
if (session_id_.empty()) throw llm::ApiError("meta", 502, "BAD_RESPONSE", "acknowledgement has no sessionId");
}
open_ = true;
} catch (const WsError& e) {
if (ws_) ws_->shutdown();
ws_.reset();
throw llm::ApiError("meta", e.http_status() ? e.http_status() : 502, "UNAVAILABLE", e.what());
}
}
void RealtimeSession::send_audio(const std::string& pcm) {
if (!open_) Check::fail("session is not open");
if (pcm.empty() || pcm.size() > MetaCaps::kMaxRealtimeAudioFrame)
Check::fail("audio frame must be 1.." + std::to_string(MetaCaps::kMaxRealtimeAudioFrame) + " bytes");
if (pcm.size() % 2 != 0) Check::fail("PCM frames are 16-bit samples");
try {
ws_->send_binary(pcm);
} catch (const google::live::WsError& e) {
throw llm::ApiError("meta", 502, "UNAVAILABLE", e.what());
}
audio_bytes_ += pcm.size();
}
void RealtimeSession::end_stream() {
if (!open_) Check::fail("session is not open");
Json j = Json::object();
j.set("type", Json("endStream"));
try {
ws_->send_text(j.dump());
} catch (const google::live::WsError& e) {
throw llm::ApiError("meta", 502, "UNAVAILABLE", e.what());
}
}
std::optional<SpeechEvent> RealtimeSession::next(int timeout_ms) {
if (!ws_ || closed_) return std::nullopt;
ws_->set_read_timeout_ms(timeout_ms < 1 ? 1 : timeout_ms);
try {
auto m = ws_->receive();
if (!m) return std::nullopt;
if (m->kind == google::live::WsMessage::Kind::Close) {
closed_ = true;
close_code_ = m->code;
open_ = false;
SpeechEvent e;
e.type = "close";
e.message = m->data;
e.audio_processed_ms = m->code;
return e;
}
if (++events_ > MetaCaps::kMaxRealtimeEvents) {
close();
throw llm::ApiError("meta", 502, "TOO_MANY_EVENTS", "event cap reached");
}
return SpeechEvent::parse(Json::parse(m->data));
} catch (const google::live::WsError& e) {
throw llm::ApiError("meta", 502, "UNAVAILABLE", e.what());
} catch (const clients::body::JsonError& e) {
throw llm::ApiError("meta", 502, "BAD_RESPONSE", e.what());
}
}
void RealtimeSession::close() {
if (ws_) {
try {
ws_->close(1000, "");
} catch (...) {
}
ws_->shutdown();
ws_.reset();
}
open_ = false;
closed_ = true;
}
} // namespace molo::meta
+158
View File
@@ -0,0 +1,158 @@
#pragma once
// Muse Voice Transcribe (speech to text; the model does no synthesis).
// POST /asr/transcribe multipart: `request` (JSON settings) + `audio` (WAV:
// mono 16-bit PCM, 16 or 24 kHz, <= 10 min, <= 32 MB);
// Accept picks JSON, text/event-stream or text/plain.
// WS /asr/realtime first frame (within 10 s) is the JSON handshake with
// the credential inside it ({"authorization":{"accessToken":
// "Bearer <key>"}}), then raw PCM binary frames, then
// {"type":"endStream"}; close 1000 normal, 1008 invalid,
// 1011 backend, 1013 rate limited.
// Modes: PUSH_TO_TALK (default), ENDPOINTING, DIARIZATION. 25 languages,
// keyword and language biasing, turn-level timestamps only.
#include <atomic>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <vector>
#include "google/live/ws_client.h"
#include "google/live/ws_stream.h"
#include "llm/api_client.h"
#include "meta/jsonx.h"
namespace molo::meta {
struct SpeechOptions {
std::string model = "muse-voice-transcribe-1.0";
std::string mode; // "" (server default PUSH_TO_TALK) | PUSH_TO_TALK | ENDPOINTING | DIARIZATION
std::string partial_mode; // "" | CUMULATIVE | DELTA
std::vector<std::string> keywords;
std::vector<std::string> language_bias;
bool emit_audio_progress = false;
std::string session_id; // correlation id, query parameter
void validate() const;
// The `request` JSON part / handshake settings. audio_encoding: WAV for the
// file endpoint, PCM_16KHZ or PCM_24KHZ for the socket.
Json settings_json(const std::string& audio_encoding) const;
};
struct WavInfo {
int channels = 0;
int sample_rate = 0;
int bits = 0;
long data_bytes = 0;
long duration_ms = 0;
// Throws ApiError::invalid unless it is RIFF/WAVE PCM mono 16-bit at 16 or
// 24 kHz, at most 10 minutes.
static WavInfo parse(const std::string& wav);
};
struct SpeechTurn {
long turn_id = 0;
long start_ms = 0;
long end_ms = 0;
std::string transcript;
std::string speaker;
};
struct TranscribeResult {
std::string session_id;
std::string transcript;
long audio_duration_ms = 0;
std::vector<SpeechTurn> turns;
static TranscribeResult from_json(const Json& j);
};
// One server message from the event stream or the socket (ServerMessage union:
// transcript, speechStart, speechEnd, speechComplete, speaker, audioProgress,
// error). The socket's handshake acknowledgement has no `type`, only sessionId.
struct SpeechEvent {
std::string type;
std::string transcript;
bool final_flag = false;
long audio_processed_ms = 0;
long turn_id = -1;
std::string speaker_label;
std::string session_id;
std::string message; // error
std::string error_type;
std::string error_code;
static SpeechEvent parse(const Json& j);
static const std::vector<std::string>& known_types();
};
enum class TranscribeAccept { kJson, kEvents, kText };
class TranscribeService : public llm::Service {
public:
explicit TranscribeService(std::shared_ptr<llm::ApiClient> client)
: llm::Service(std::move(client)) {}
// Validates the WAV and options locally (WavInfo::parse) before upload.
TranscribeResult transcribe(const SpeechOptions& options, const std::string& wav);
// text/event-stream: events in order, ends after the last event.
std::vector<SpeechEvent> transcribe_events(const SpeechOptions& options, const std::string& wav);
// text/plain: one turn per line (A:Hello style in DIARIZATION).
std::string transcribe_text(const SpeechOptions& options, const std::string& wav);
private:
llm::RawResponse post(const SpeechOptions& options, const std::string& wav, const char* accept);
};
struct RealtimeConfig {
SpeechOptions options;
std::string audio_encoding = "PCM_16KHZ"; // PCM_16KHZ | PCM_24KHZ
std::string url; // default MetaEndpoints::kRealtimeUrl
std::string key; // default MetaClient::key_from_env()
int connect_timeout_ms = 10000;
int handshake_timeout_ms = 10000;
int ack_timeout_ms = 10000;
google::live::TlsFactory tls; // tests inject a loopback TLS-less factory
void validate() const;
};
class RealtimeSession {
public:
explicit RealtimeSession(RealtimeConfig config);
~RealtimeSession();
RealtimeSession(const RealtimeSession&) = delete;
RealtimeSession& operator=(const RealtimeSession&) = delete;
// Opens the socket, sends the handshake frame, waits for the sessionId
// acknowledgement. Throws ApiError (503 not configured without a key).
void connect();
const std::string& session_id() const { return session_id_; }
bool open() const { return open_; }
// Raw PCM (16-bit LE mono at the configured rate), at most 1 MiB per call.
void send_audio(const std::string& pcm);
void end_stream(); // {"type":"endStream"}: keep reading until the close
// Next server event, nullopt on read timeout; sets closed()/close_code() on a
// Close frame (event type "close").
std::optional<SpeechEvent> next(int timeout_ms);
bool closed() const { return closed_; }
int close_code() const { return close_code_; }
void close();
std::size_t events_seen() const { return events_; }
// The handshake frame without sending it (tests, demo dry run). The access
// token is redacted unless `reveal` is true.
Json handshake_json(bool reveal) const;
private:
RealtimeConfig config_;
std::unique_ptr<google::live::WsClient> ws_;
std::string session_id_;
bool open_ = false;
bool closed_ = false;
int close_code_ = 0;
std::size_t events_ = 0;
std::size_t audio_bytes_ = 0;
};
} // namespace molo::meta
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env python3
"""Checks src/meta against the published Meta Model API docs (python3 stdlib only).
Meta publishes no OpenAPI document. The docs site serves every page as
Markdown (https://dev.meta.ai/llms.txt lists them) and the API reference has
one schemas page per resource with machine-regular field tables. This script
checks what is checkable:
1. every row of the call table: the operation page exists, carries the
"## METHOD /path" heading, lists each declared query parameter; the
schemas page has the request schema with every declared body field and
the response schema;
2. no operation heading in the docs (all api-reference pages) is missing
from the table;
3. the Responses stream event names equal the ResponseStreamEvent union;
4. every hosted model id in the catalog appears on the models page and
every token price on the pricing page;
5. the Muse Code protocol vocabulary (methods, notifications, server
requests, error codes) equals the committed MSP stable schema of the
muse-code-sdk mirror.
Usage: check_spec.py [--cache DIR] (--cache reads/writes fetched pages)
Exit 0 only when everything matches.
"""
import json
import os
import re
import sys
import urllib.request
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
SRC = os.path.join(ROOT, "src", "meta")
DOCS = "https://dev.meta.ai/"
RAW = "https://raw.git" + "hubusercontent.com/meta-models/muse-code-sdk/HEAD/schema/msp/stable/msp.schema.json"
CACHE = None
failures = []
def fail(msg):
failures.append(msg)
print("FAIL:", msg)
def fetch(url):
if CACHE:
path = os.path.join(CACHE, re.sub(r"[^A-Za-z0-9._-]", "_", url))
if os.path.exists(path):
return open(path, encoding="utf-8").read()
last = None
for _ in range(3):
try:
with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "molo-check-spec"}), timeout=30) as r:
text = r.read().decode("utf-8")
if CACHE:
os.makedirs(CACHE, exist_ok=True)
open(os.path.join(CACHE, re.sub(r"[^A-Za-z0-9._-]", "_", url)), "w", encoding="utf-8").write(text)
return text
except Exception as e: # noqa: BLE001
last = e
raise SystemExit("cannot fetch %s: %s" % (url, last))
def block(text, begin, end):
m = re.search(begin + r"[^\n]*\n(.*?)" + end, text, re.S)
if not m:
raise SystemExit("marker %s not found" % begin)
return m.group(1)
def split_camel(name):
return re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", name)
def schema_sections(page):
out = {}
for part in re.split(r"\n### ", page)[1:]:
title, _, body = part.partition("\n")
fields = set(re.findall(r"^\| `([^`]+)`", body, re.M))
out[title.strip()] = fields
return out
def main():
global CACHE
if "--cache" in sys.argv:
CACHE = sys.argv[sys.argv.index("--cache") + 1]
calls_src = open(os.path.join(SRC, "calls.cpp"), encoding="utf-8").read()
rows = re.findall(r'\{Call::(\w+), "(\w+)", "([^"]+)", "([^"]+)", "([^"]*)", "([^"]*)", "([^"]*)", "([^"]*)", "([^"]*)"\},',
block(calls_src, "CALLS-BEGIN", "// CALLS-END"))
if not rows:
raise SystemExit("no call rows parsed")
llms = fetch(DOCS + "llms.txt")
pages = set(re.findall(r"https://dev\.meta\.ai/(docs/[^\s)]+\.md)", llms))
schema_cache = {}
declared = set()
for name, method, path, op, group, req, resp, body_fields, query in rows:
declared.add((method, path))
url = "docs/api-reference/%s.md" % op
if url not in pages:
fail("%s: page %s is not listed in llms.txt" % (name, url))
continue
page = fetch(DOCS + url)
heading = "## %s %s" % (method, path.replace("_", r"\_"))
if heading not in page and ("## %s %s" % (method, path)) not in page:
fail("%s: heading '## %s %s' missing in %s" % (name, method, path, url))
for q in filter(None, query.split(",")):
if "`%s`" % q not in page:
fail("%s: query parameter %s missing in %s" % (name, q, url))
if group:
if group not in schema_cache:
schema_cache[group] = schema_sections(fetch(DOCS + "docs/api-reference/%s/schemas.md" % group))
sections = schema_cache[group]
for schema, fields in ((req, body_fields), (resp, "")):
if not schema:
continue
title = split_camel(schema)
if title not in sections:
fail("%s: schema %s not in %s schemas page" % (name, title, group))
continue
for f in filter(None, fields.split(",")):
if f not in sections[title]:
fail("%s: field %s missing in schema %s" % (name, f, title))
# Multipart edit variant and the images edit alternative body.
img = schema_cache.get("images", {})
for f in "image,prompt,model,n,size,response_format,reasoning_strength".split(","):
if f not in img.get("Create Image Edit Request", set()):
fail("images multipart edit: field %s missing in CreateImageEditRequest" % f)
# 2. every documented operation is in the table (no exclusions needed).
for url in sorted(pages):
if not url.startswith("docs/api-reference/") or url.count("/") < 2:
continue
for m, p in re.findall(r"^## (GET|POST|DELETE|PUT|PATCH|WS) (/\S+)", fetch(DOCS + url), re.M):
p = p.replace("\\_", "_")
if (m, p) not in declared:
fail("documented operation %s %s (%s) is not in the call table" % (m, p, url))
# 3. stream events.
ev_src = open(os.path.join(SRC, "response_events.cpp"), encoding="utf-8").read()
ours = set(re.findall(r'"([\w.]+)",', block(ev_src, "EVENTS-BEGIN", "// EVENTS-END")))
rpage = fetch(DOCS + "docs/api-reference/responses/schemas.md")
union = re.search(r"### Response Stream Event\n+Type: (.*?)\n", rpage).group(1)
theirs = set()
for n in re.findall(r"\[`(\w+)`\]", union):
sec = re.search(r"### %s\n(.*?)\n### " % re.escape(split_camel(n)), rpage, re.S)
t = re.search(r"`type`\s*\|\s*`enum \('([^']+)'\)", sec.group(1)) if sec else None
if t:
theirs.add(t.group(1))
else:
fail("stream event schema %s has no type enum" % n)
for e in sorted(ours ^ theirs):
fail("stream event %s differs (ours=%s docs=%s)" % (e, e in ours, e in theirs))
# 4. catalog against models / pricing pages.
cat = open(os.path.join(SRC, "catalog.cpp"), encoding="utf-8").read()
models_page = fetch(DOCS + "docs/models.md")
pricing = fetch(DOCS + "docs/pricing-rate-limits.md")
for m in re.finditer(r'\{"([\w.-]+)", "[^"]+", ModelKind::\w+, (true|false), \w+, \w+, (\d+), ([\d.]+), ([\d.]+), ([\d.]+), "[^"]*", (\d+), (\d+),', block(cat, "CATALOG-BEGIN", "// CATALOG-END")):
mid, hosted, ctx, cached, inp, out, rpm, tpm = m.groups()
if "`%s`" % mid not in models_page and "`%s`" % mid not in pricing and mid not in ("muse-spark-open", "muse-glimmer"):
fail("catalog model %s not on the models or pricing page" % mid)
if hosted == "true" and float(inp) > 0:
for price in (cached, inp, out):
p = "$%.2f" % float(price) if float(price) >= 0.01 else "$%s" % price.rstrip("0")
if p not in pricing and ("$%s" % price) not in pricing:
fail("catalog price %s of %s not on the pricing page" % (p, mid))
if hosted == "true" and int(rpm) > 0 and ("| %s" % format(int(rpm), ",")) not in pricing:
fail("catalog rpm %s of %s not on the pricing page" % (rpm, mid))
# 5. Muse Code protocol.
schema = json.loads(fetch(RAW))
spec = open(os.path.join(SRC, "muse_code_spec.cpp"), encoding="utf-8").read()
def names(tag):
return set(re.findall(r'"([^"]+)",', block(spec, "MSP-%s-BEGIN" % tag, "// MSP-%s-END" % tag)))
for tag, key in (("METHODS", "methods"), ("NOTIFICATIONS", "notifications"), ("REQUESTS", "requests")):
for e in sorted(names(tag) ^ set(schema[key])):
fail("MSP %s entry %s differs from the stable schema" % (key, e))
errs = set(re.findall(r'"(-?\d+):(\w+):[01?]"', block(spec, "MSP-ERRORS-BEGIN", "// MSP-ERRORS-END")))
want = {(str(e["code"]), e["kind"]) for e in schema["errors"]}
for e in sorted(errs ^ want):
fail("MSP error %s differs from the stable schema" % (e,))
print("checked: %d calls, %d stream events, %d MSP methods, %d MSP notifications, %d docs pages listed"
% (len(rows), len(ours), len(schema["methods"]), len(schema["notifications"]), len(pages)))
if failures:
print("%d failure(s)" % len(failures))
sys.exit(1)
print("OK")
if __name__ == "__main__":
main()
+79
View File
@@ -0,0 +1,79 @@
#include "meta/usage.h"
#include <cstdlib>
#include "meta/catalog.h"
namespace molo::meta {
MetaUsage MetaUsage::from_responses(const Json& u) {
MetaUsage m;
m.input_tokens = JsonX::num(u, "input_tokens");
m.output_tokens = JsonX::num(u, "output_tokens");
m.total_tokens = JsonX::num(u, "total_tokens", m.input_tokens + m.output_tokens);
m.cached_tokens = JsonX::path_num(u, "input_tokens_details.cached_tokens");
m.reasoning_tokens = JsonX::path_num(u, "output_tokens_details.reasoning_tokens");
m.images_processed = JsonX::num(u, "images_processed");
m.video_frames_processed = JsonX::num(u, "video_frames_processed");
return m;
}
MetaUsage MetaUsage::from_chat(const Json& u) {
MetaUsage m;
m.input_tokens = JsonX::num(u, "prompt_tokens");
m.output_tokens = JsonX::num(u, "completion_tokens");
m.total_tokens = JsonX::num(u, "total_tokens", m.input_tokens + m.output_tokens);
m.cached_tokens = JsonX::path_num(u, "prompt_tokens_details.cached_tokens");
m.reasoning_tokens = JsonX::path_num(u, "completion_tokens_details.reasoning_tokens");
return m;
}
MetaUsage MetaUsage::from_messages(const Json& u) {
MetaUsage m;
m.input_tokens = JsonX::num(u, "input_tokens");
m.output_tokens = JsonX::num(u, "output_tokens");
m.total_tokens = m.input_tokens + m.output_tokens;
m.cached_tokens = JsonX::num(u, "cache_read_input_tokens");
m.reasoning_tokens = JsonX::path_num(u, "output_tokens_details.thinking_tokens");
return m;
}
llm::Usage MetaUsage::to_neutral() const {
llm::Usage u;
u.input_tokens = static_cast<int>(input_tokens);
u.output_tokens = static_cast<int>(output_tokens);
u.total_tokens = static_cast<int>(total_tokens);
u.cached_tokens = static_cast<int>(cached_tokens);
u.reasoning_tokens = static_cast<int>(reasoning_tokens);
return u;
}
double MetaUsage::cost_usd(const std::string& model) const {
const ModelInfo* info = ModelCatalog::find(model);
if (!info || info->kind != ModelKind::kText || info->input_per_m <= 0) return -1.0;
long fresh = input_tokens > cached_tokens ? input_tokens - cached_tokens : 0;
return (static_cast<double>(cached_tokens) * info->cached_per_m +
static_cast<double>(fresh) * info->input_per_m +
static_cast<double>(output_tokens) * info->output_per_m) / 1e6;
}
namespace {
long header_long(const clients::HeaderMap& h, const char* name) {
auto v = h.get(name);
if (!v || v->empty() || v->size() > 18) return -1;
for (char c : *v)
if (c < '0' || c > '9') return -1;
return std::strtol(v->c_str(), nullptr, 10);
}
} // namespace
RateLimits RateLimits::from_headers(const clients::HeaderMap& h) {
RateLimits r;
r.limit_tokens = header_long(h, "x-ratelimit-limit-tokens");
r.remaining_tokens = header_long(h, "x-ratelimit-remaining-tokens");
r.limit_requests = header_long(h, "x-ratelimit-limit-requests");
r.remaining_requests = header_long(h, "x-ratelimit-remaining-requests");
return r;
}
} // namespace molo::meta
+51
View File
@@ -0,0 +1,51 @@
#pragma once
// Usage accounting across the three text endpoints, the image usage object,
// the SAM extras, and the x-ratelimit-* headers. Field names per docs
// (token-counting, pricing-rate-limits):
// Responses: input_tokens, output_tokens, total_tokens,
// input_tokens_details.cached_tokens,
// output_tokens_details.reasoning_tokens,
// video_frames_processed, images_processed
// Chat Completions: prompt_tokens, completion_tokens, total_tokens,
// prompt_tokens_details.cached_tokens,
// completion_tokens_details.reasoning_tokens
// Messages: input_tokens, output_tokens, cache_read_input_tokens,
// cache_creation_input_tokens, output_tokens_details.thinking_tokens
#include <string>
#include "clients/response.h"
#include "llm/chat_types.h"
#include "meta/jsonx.h"
namespace molo::meta {
struct MetaUsage {
long input_tokens = 0;
long output_tokens = 0;
long total_tokens = 0;
long cached_tokens = 0;
long reasoning_tokens = 0;
long images_processed = 0; // SAM
long video_frames_processed = 0; // SAM, billed frames
static MetaUsage from_responses(const Json& usage);
static MetaUsage from_chat(const Json& usage);
static MetaUsage from_messages(const Json& usage);
llm::Usage to_neutral() const;
// USD cost for a text model id from the static catalog; negative when the
// model has no token prices.
double cost_usd(const std::string& model) const;
};
struct RateLimits {
long limit_tokens = -1;
long remaining_tokens = -1;
long limit_requests = -1;
long remaining_requests = -1;
bool present() const { return limit_tokens >= 0 || limit_requests >= 0; }
static RateLimits from_headers(const clients::HeaderMap& headers);
};
} // namespace molo::meta