models: EndpointAdmin and PublisherAdmin wrap every remaining method of the owned resources (84 call table rows, 132 Discovery methods covered, v1beta1 twins, IAM, operations, responses, export, config, enableModel); planner rejects undeclared body keys; VertexModel::TurnBuilder replays model turns with every thoughtSignature; Lyria WAV mime from bytes; veo-3.1 default. runtime: v1beta1 feedbackEntries, A2A gateway (40 rpcs), a2aTasks, task store records, scoped operations, beta twin flag (103 rows, 214 methods covered); checker verifies twins, a2aEndpoint patterns and coverage. example: admin routes for both parts. SPECs: live results moved from Unverified to Verified live (Veo, Lyria, thought signatures, v1beta1 routes). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
18 KiB
Models (Gemini Enterprise Agent Platform) - Specification
Owned directory: src/google/models/. Namespace molo::google::models.
Depends on the foundation in src/google/ (GoogleClient, Service,
GoogleError, CloudContext, Operation, gemini_types.h). Voice and
Generative are untouched.
1. Scope
Gemini on Vertex AI (generate, stream, count tokens, compute tokens,
embeddings), Model Garden (list, get, publisher model config, EULA, deploy),
partner models (Claude rawPredict and streamRawPredict), endpoint models
(Gemma and any deployed model: list, get, predict family), media generation
(image, music, video via predict and predictLongRunning), and
VertexModel : LanguageModel.
Two addressing modes, one class (VertexTarget):
- express (API key, no project):
https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent - project scoped:
https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:...(globaluses the host without a prefix).
2. Module map (one concern per file)
| File | Concern |
|---|---|
caps.h |
Static ceilings |
calls.h |
THE call table: every call, its Discovery rpc id and request fields |
target.h/.cpp |
VertexTarget: express vs project, id validation |
planner.h/.cpp |
CallPlanner: table row + ids -> exact request (pure, no I/O) |
vertex_service.h/.cpp |
VertexService : Service: runs planned calls |
json_read.h |
tolerant response readers |
gemini.h/.cpp |
Gemini: generate, stream (?alt=sse), count/compute tokens, embedContent, predict embeddings |
vertex_model.h/.cpp |
VertexModel : LanguageModel, TurnBuilder (replayable model turn with every thoughtSignature) |
admin_checks.h |
Input rules shared by the admin classes (object, size, mask, confirm) |
endpoint_admin.h/.cpp |
EndpointAdmin: endpoint lifecycle, remaining inference surfaces, endpoint operations, v1beta1 IAM and list |
publisher_admin.h/.cpp |
PublisherAdmin: publisher v1/responses, serverStreamingPredict, export, config, enableModel, IAM, v1beta1 forms |
model_garden.h/.cpp |
ModelGarden: list, get, fetch config, EULA check/accept, deploy |
claude_models.h/.cpp |
ClaudeModels: rawPredict, streamRawPredict, text_delta |
open_models.h/.cpp |
OpenModels: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke |
media.h/.cpp |
MediaGenerator: image, music, video start/status/wait |
tools/check_discovery.py |
checks calls.h against Google's Discovery documents |
Every call has a plan_* twin returning a PlannedCall (method, endpoint,
path, body) without sending anything; tests and check_discovery.py verify
requests that way.
3. Static caps (caps.h)
Contents per request 200; request text 1,000,000 chars; output tokens 1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072 dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude 200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16, 100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4, music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per result, operation name 512 chars; video polls 120 at 5 s (10 minutes). Admin surface: request body 262144 chars serialized, updateMask 512 chars, operations.wait 1..600 s, operation list pages 1..5, 64 permissions, 5000 streamed events.
4. Contracts
- Every id (publisher, model, endpoint, invoke id) passes
valid_segmentbefore a path is built; operation names andpublisherModelstrings have their own character check. Bad input isGoogleError::invalid(400INVALID_ARGUMENT) before any network traffic. - A call whose table row has
express=falsethrows invalid in express mode, naming the missing project. No credential:NotConfiguredError(503). - Costly or consent calls are explicit and gated:
ModelGarden::deployneedsDeployOptions.confirm,accept_eulaneedsconfirm=true; both refuse otherwise. Nothing in this module deletes. - Streaming uses
GoogleClient::stream_json(SSE via?alt=ssefor Gemini, Anthropic events for Claude); the sink returns false to stop. - Video:
start_video(predictLongRunning),video_status(fetchPredictOperation, a POST),wait_videobounded by the caps.wait_operation(GET) is not used: Discovery has no GET on these operations for publisher models. - Admin bodies are Json whose top-level keys must be declared in the call
table:
CallPlanner::planrejects a stray key (INVALID_ARGUMENT) before any traffic; rows with-(raw HTTP bodies) take any object. EndpointAdmin(beta=true)sends the v1beta1 twin of everyendpoint.row (CallPlanner::plan(..., beta=true));tools/check_discovery.pyverifies each twin (same method, path, rpc id, fields).endpoint.beta_*rows exist in v1beta1 only (list withgdcZone, IAM).- Gated admin calls refuse without
confirm=true: endpoint delete, deployModel, undeployModel, mutateDeployedModel, IAM set, stored response delete, operation record delete, publisher export, setPublisherModelConfig, enableModel. The example routes answer a dry run instead. VertexModel::TurnBuilderbuilds the model turn to replay from stream chunks: text parts merge only while they carry nothoughtSignature, a signature that arrives on an empty part attaches to the part before it, calls and data parts are kept as received.VertexModel::model_turnandreplaydo the same for a whole reply.Gemini::validateaccepts a signature-only part (it is not "empty").- Example (
/agent/models): GET page,/status,/callsneed no credential; every Google call is a POST that answers 503google_not_configuredwithout a credential (checked first), 422 on bad input. Routes: generate, count, embed, claude, list, endpoint, image, music, video, video/status, and the admin routesendpoints(create, patch, update, delete, deploy, undeploy, mutate, list_beta, get_iam, set_iam, test_iam),endpoint/call(compute_tokens, direct_predict, direct_raw_predict, explain, invoke, deployed_invoke, chat, openapi, science, response_get, response_delete),endpoint/operations(get, list, cancel, wait, delete),publisher(stream_predict, responses_*, export, set_config, enable, iam, beta_get, beta_generate, beta_count, beta_post, beta_invoke). The video default model isveo-3.1-generate-001(veo-3.0-generate-001answers 404 for the project).
5. Verified against discovery
python3 -I src/google/models/tools/check_discovery.py fetches
https://aiplatform.googleapis.com/$discovery/rest?version=v1 and v1beta1
and fails when any row of calls.h has a missing method or path, a wrong
rpc id, an absent request body field (dotted names follow the schema) or an
absent query parameter. Result on 2026-10-24: 84 calls checked, 132 owned discovery methods covered, 0 problems.
Coverage is part of the check: every Discovery method whose rpc id starts with
aiplatform.publishers.models., aiplatform.projects.locations.publishers.,
aiplatform.projects.publishers.models., aiplatform.projects.locations.endpoints.,
aiplatform.projects.modelGardenEula. or aiplatform.projects.locations.deployPublisherModel
(v1 and v1beta1, 132 methods) must be a row, the express form of a row, or the
v1beta1 twin of an endpoint. row (verified field by field); the check fails
on any other method until it is wrapped or excluded on purpose (EXCLUDED,
empty today). So a Discovery change that adds a method to these resources is
loud.
The same pass shows what Discovery says about express forms:
generateContent, streamGenerateContent, countTokens, computeTokens, predict,
predictLongRunning, fetchPredictOperation exist without a project;
embedContent, rawPredict, streamRawPredict, publisher invoke do not
(embedContent express answers 404 live). publishers.models.list exists
only in v1beta1; publishers.models.get in v1 and v1beta1.
Request construction is compared to the table in
tests/src/google/models/test_models.cpp and test_models_admin.cpp (every
body key must be a declared field; the planner also rejects a stray key at
run time; every row plans to a concrete URL, express and project scoped).
6. Verified live
Run 2026-10-24 through this module's C++ client
(MOLO_LIVE_NET=1 build/google_test, tests models_live.*), read-only or
tiny (20 tokens). Credentials never printed or stored.
With an express API key (AQ.):
Gemini::generate,stream(SSE),count_tokens,compute_tokens,embed(text-embedding-005 predict) onaiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash: 200.- Project scoped
countTokensat locationglobal: 200. - Refused with 401 "API keys are not supported": Model Garden list/get,
endpoints list, rawPredict, project embedContent. Express
embedContent: 404 (matches Discovery).
With the service account OAuth token (project lisa-448004):
- v1beta1
publishers/google/modelslist (google, anthropic), v1publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash}get incl.view=PUBLISHER_MODEL_VIEW_FULL: 200. - Endpoints list (
us-central1): 200, empty. embedContentproject scopedus-central1gemini-embedding-001: 200 (global and text-embedding-005 not available there: 404/400).- Project scoped
generateContentandcomputeTokensatglobal: 200 (us-central1+ gemini-2.5-flash: 404 "not found or no access"). - Claude
rawPredictatglobal(claude-sonnet-4-6): reached, 429 RESOURCE_EXHAUSTED (project quota);us-east5: 404 no access. Route, auth and body accepted; no completion was obtained. fetchPublisherModelConfig: 404 "PublisherModelConfig not found" (route exists, nothing configured). EULA check forpublishers/google/models/gemma3: 404 model not found (route exists).- Generation calls may answer 429 on the shared project; the live test notes that instead of failing.
Run 2026-10-24 (closing the gaps), service account token, project
lisa-448004, through the new classes (models_live_admin.*) or direct
requests:
EndpointAdminread-only: v1beta1 endpoints list (us-central1): 200{}(an empty list has noendpointskey); v1 and v1beta1 operations list of a missing endpoint: 200{}; v1beta1getIamPolicy,operations.geton a missing endpoint: 404 NOT_FOUND; stored responses (endpoints/{id}/responses/{id}get): 501 UNIMPLEMENTED for the project.PublisherAdmin: v1beta1publishers/google/models/gemini-2.5-flashget: 200; project scoped v1beta1getIamPolicyon a publisher model: 400 INVALID_ARGUMENT (nothing to read there).- Thought signatures:
gemini-3.5-flashatglobalwith a function declaration, streamed: thefunctionCallpart carries athoughtSignature(TurnBuilderkeeps 1 signed part). The stream ends with an empty text part, whichTurnBuilderdrops. The turn replayed with afunctionResponseviaVertexModel::replayis accepted (200, "The weather in Paris is 18°C."). Whether Google rejects a replay that lacks the signature was not tested. - Veo:
veo-3.1-generate-001exists for the project (us-central1 andglobal);veo-3.0-generate-001,veo-3.0-fast-generate-001,veo-2.0-generate-001and every Imagen model tried (imagen-3.0-generate-001/002,imagen-3.0-fast-generate-001,imagen-4.0-generate-001,imagen-4.0-fast-generate-001,imagen-4.0-ultra-generate-001) answer 404 "not found or your project does not have access".predictLongRunningacceptsinstances[{prompt}](an instance withtextinstead: 400 "No inputs provided") and answers{"name": "projects/.../publishers/google/models/veo-3.1-generate-001/operations/<uuid>"}. Parameters are validated inside the operation, which finishesdone:truewitherror{code:3,message}onfetchPredictOperation:sampleCount99 -> "Invalid sample count 99. The sample count should be in the range of [1, 4]" (name and range match our cap 1..4);durationSeconds99 -> "Unsupported output video duration 99 seconds, supported durations are [8,4,6] for feature text_to_video" (name confirmed; the model accepts only 4, 6, 8, our cap 1..8 is wider);storageUri"bogus" -> "Unsupported output storage uri: bogus" (name confirmed);aspectRatio"bogus" is rejected synchronously with 400 "Invalid aspect ratio: bogus". - A finished Veo operation, read with
fetchPredictOperation:done:true,response={"@type":"type.googleapis.com/cloud.ai.large_models.vision.GenerateVideoResponse", "raiMediaFilteredCount":0, "videos":[{"bytesBase64Encoded":"...","mimeType":"video/mp4"}]}(one 8 s clip, about 13.8 MB decoded, under the 32 MiB cap). This is the real shape of the operationresponseAny for video and is whatwait_videoreads. - Lyria:
lyria-002predict(project scoped, us-central1 andglobal):instances[{prompt}]-> 200{"predictions":[{"bytesBase64Encoded":"UklGR..."}]}with nomimeType; the bytes are a RIFF/WAVE file, soitems_fromnamesaudio/wavwhen the mime is missing. An empty prompt or an instance withoutprompt-> 400 "Request contains an invalid argument". - Claude
rawPredictandstreamRawPredictatglobal: still 429 RESOURCE_EXHAUSTED on quotaglobal_online_prediction_requests_per_base_model(anthropic-claude-sonnet-4-6). The streaming call reports the error as a JSON array[{"error":{...}}], not as one object.
Cost note: the Veo and Lyria probes were meant to be rejected requests. Google
accepted several, so the run started 5 Veo operations (4 failed validation
inside the operation and cost nothing, one generateAudio:"str" probe rendered
one 8 s clip) and generated 3 Lyria clips. The probe scripts are not part of
the repo; the live tests in the repo never call a generation method.
Not live-verified (mutating or billable): deploy, EULA accept, image generation (no Imagen model is visible to the project), endpoint create, patch, update, delete, deployModel, undeployModel, mutateDeployedModel, IAM set, publisher export, config and enableModel, the endpoint inference family (no endpoint exists). These are verified by Discovery and by local request tests.
7. Sources
- Discovery documents:
https://aiplatform.googleapis.com/$discovery/rest?version=v1and?version=v1beta1(fetched 2026-10-24). - Claude on Vertex:
https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude(rawPredict/streamRawPredict paths,anthropic_version: vertex-2023-10-16, messages/max_tokens/stream/thinking, response shape). - Reference pages read for semantics:
.../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent},.../reference/rest/v1/publishers.models/get(view enum, languageCode).
8. Unverified (neither Discovery nor docs read, nor live)
Settled on 2026-10-24 and moved to section 6: Veo prompt, sampleCount,
durationSeconds, aspectRatio, storageUri and the finished operation
response; Lyria prompt and the predictions[].bytesBase64Encoded WAV
response; the empty endpoint list shape; thought signatures on function calls.
- Imagen
predictinstance/parameter names (prompt;sampleCount,aspectRatio,negativePrompt) and thepredictions[].bytesBase64Encoded/mimeTyperesponse: no Imagen model is visible to projectlisa-448004(404 for every id tried), so even a rejected request is not informative. - Lyria
negative_prompt,seedandsample_count: a live call accepted them, but it also accepted junk values (seed:"abc",sample_count:99), so acceptance does not prove the names. Whether a clip count above 1 is honored was not checked. - Veo
generateAudioand thegcsUriresult form with a realstorageUribucket (no bucket was used). A missingmimeTypeon Veo output was not seen. - Gemma/vLLM instance shape in
gemma_instance(prompt,max_tokens,temperature): the serving container's, not Google's API; no endpoint exists to ask. - Anthropic streaming event names (
content_block_delta/text_delta) come from Anthropic's format; Claude completions were not obtained live (429 quota on every attempt). deployPublisherModelfield semantics beyond the schema names.- Response shapes of the admin methods that need a real endpoint or a mutation: deployModel, directPredict, directRawPredict, explain, chat completions, openapi, invoke, stored responses (501 for this project), export, setPublisherModelConfig, enableModel. They are returned as raw Json (no field is read), so a different shape cannot break the client.
- Whether Google rejects a Gemini 3 replay that omits the
thoughtSignature(the accepted replay with it is verified).
9. Gaps
- Every Discovery method of the owned resources is in the call table (see
section 5); the only intentional difference is that project-less v1 forms
are the
expressforms of rows and the v1beta1 forms of endpoint rows are twins selected withEndpointAdmin(beta=true). - The admin request bodies are validated by top-level key against the call
table and by caps; nested fields (a
DeployedModel, aPublisherModelConfig) are Json the caller owns. Typed structs exist where the module reads fields (EndpointInfo,Operation,PublisherModel). - Endpoint
generateContentfamily is wired but only Discovery-verified. VertexModelhas no retry; quota 429 surfaces asGoogleError.- A streaming error body shaped as a JSON array (
[{"error":...}], seen on ClaudestreamRawPredict) goes through the foundationStreamDecoder, which is outside this module; not verified that it maps toGoogleError. - Token refresh is the caller's job (foundation gap).