Files
moloapp/src/google/models/SPEC.md
T
retoorandClaude Sonnet 5.5 8e807945a2 feat(clients): models+runtime: close gaps (all Discovery methods wrapped, live re-verification)
models: EndpointAdmin and PublisherAdmin wrap every remaining method of the
owned resources (84 call table rows, 132 Discovery methods covered, v1beta1
twins, IAM, operations, responses, export, config, enableModel); planner
rejects undeclared body keys; VertexModel::TurnBuilder replays model turns
with every thoughtSignature; Lyria WAV mime from bytes; veo-3.1 default.
runtime: v1beta1 feedbackEntries, A2A gateway (40 rpcs), a2aTasks, task store
records, scoped operations, beta twin flag (103 rows, 214 methods covered);
checker verifies twins, a2aEndpoint patterns and coverage.
example: admin routes for both parts. SPECs: live results moved from
Unverified to Verified live (Veo, Lyria, thought signatures, v1beta1 routes).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 05:47:34 +02:00

18 KiB

Models (Gemini Enterprise Agent Platform) - Specification

Owned directory: src/google/models/. Namespace molo::google::models. Depends on the foundation in src/google/ (GoogleClient, Service, GoogleError, CloudContext, Operation, gemini_types.h). Voice and Generative are untouched.

1. Scope

Gemini on Vertex AI (generate, stream, count tokens, compute tokens, embeddings), Model Garden (list, get, publisher model config, EULA, deploy), partner models (Claude rawPredict and streamRawPredict), endpoint models (Gemma and any deployed model: list, get, predict family), media generation (image, music, video via predict and predictLongRunning), and VertexModel : LanguageModel.

Two addressing modes, one class (VertexTarget):

  • express (API key, no project): https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent
  • project scoped: https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:... (global uses the host without a prefix).

2. Module map (one concern per file)

File Concern
caps.h Static ceilings
calls.h THE call table: every call, its Discovery rpc id and request fields
target.h/.cpp VertexTarget: express vs project, id validation
planner.h/.cpp CallPlanner: table row + ids -> exact request (pure, no I/O)
vertex_service.h/.cpp VertexService : Service: runs planned calls
json_read.h tolerant response readers
gemini.h/.cpp Gemini: generate, stream (?alt=sse), count/compute tokens, embedContent, predict embeddings
vertex_model.h/.cpp VertexModel : LanguageModel, TurnBuilder (replayable model turn with every thoughtSignature)
admin_checks.h Input rules shared by the admin classes (object, size, mask, confirm)
endpoint_admin.h/.cpp EndpointAdmin: endpoint lifecycle, remaining inference surfaces, endpoint operations, v1beta1 IAM and list
publisher_admin.h/.cpp PublisherAdmin: publisher v1/responses, serverStreamingPredict, export, config, enableModel, IAM, v1beta1 forms
model_garden.h/.cpp ModelGarden: list, get, fetch config, EULA check/accept, deploy
claude_models.h/.cpp ClaudeModels: rawPredict, streamRawPredict, text_delta
open_models.h/.cpp OpenModels: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke
media.h/.cpp MediaGenerator: image, music, video start/status/wait
tools/check_discovery.py checks calls.h against Google's Discovery documents

Every call has a plan_* twin returning a PlannedCall (method, endpoint, path, body) without sending anything; tests and check_discovery.py verify requests that way.

3. Static caps (caps.h)

Contents per request 200; request text 1,000,000 chars; output tokens 1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072 dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude 200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16, 100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4, music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per result, operation name 512 chars; video polls 120 at 5 s (10 minutes). Admin surface: request body 262144 chars serialized, updateMask 512 chars, operations.wait 1..600 s, operation list pages 1..5, 64 permissions, 5000 streamed events.

4. Contracts

  • Every id (publisher, model, endpoint, invoke id) passes valid_segment before a path is built; operation names and publisherModel strings have their own character check. Bad input is GoogleError::invalid (400 INVALID_ARGUMENT) before any network traffic.
  • A call whose table row has express=false throws invalid in express mode, naming the missing project. No credential: NotConfiguredError (503).
  • Costly or consent calls are explicit and gated: ModelGarden::deploy needs DeployOptions.confirm, accept_eula needs confirm=true; both refuse otherwise. Nothing in this module deletes.
  • Streaming uses GoogleClient::stream_json (SSE via ?alt=sse for Gemini, Anthropic events for Claude); the sink returns false to stop.
  • Video: start_video (predictLongRunning), video_status (fetchPredictOperation, a POST), wait_video bounded by the caps. wait_operation (GET) is not used: Discovery has no GET on these operations for publisher models.
  • Admin bodies are Json whose top-level keys must be declared in the call table: CallPlanner::plan rejects a stray key (INVALID_ARGUMENT) before any traffic; rows with - (raw HTTP bodies) take any object.
  • EndpointAdmin(beta=true) sends the v1beta1 twin of every endpoint. row (CallPlanner::plan(..., beta=true)); tools/check_discovery.py verifies each twin (same method, path, rpc id, fields). endpoint.beta_* rows exist in v1beta1 only (list with gdcZone, IAM).
  • Gated admin calls refuse without confirm=true: endpoint delete, deployModel, undeployModel, mutateDeployedModel, IAM set, stored response delete, operation record delete, publisher export, setPublisherModelConfig, enableModel. The example routes answer a dry run instead.
  • VertexModel::TurnBuilder builds the model turn to replay from stream chunks: text parts merge only while they carry no thoughtSignature, a signature that arrives on an empty part attaches to the part before it, calls and data parts are kept as received. VertexModel::model_turn and replay do the same for a whole reply. Gemini::validate accepts a signature-only part (it is not "empty").
  • Example (/agent/models): GET page, /status, /calls need no credential; every Google call is a POST that answers 503 google_not_configured without a credential (checked first), 422 on bad input. Routes: generate, count, embed, claude, list, endpoint, image, music, video, video/status, and the admin routes endpoints (create, patch, update, delete, deploy, undeploy, mutate, list_beta, get_iam, set_iam, test_iam), endpoint/call (compute_tokens, direct_predict, direct_raw_predict, explain, invoke, deployed_invoke, chat, openapi, science, response_get, response_delete), endpoint/operations (get, list, cancel, wait, delete), publisher (stream_predict, responses_*, export, set_config, enable, iam, beta_get, beta_generate, beta_count, beta_post, beta_invoke). The video default model is veo-3.1-generate-001 (veo-3.0-generate-001 answers 404 for the project).

5. Verified against discovery

python3 -I src/google/models/tools/check_discovery.py fetches https://aiplatform.googleapis.com/$discovery/rest?version=v1 and v1beta1 and fails when any row of calls.h has a missing method or path, a wrong rpc id, an absent request body field (dotted names follow the schema) or an absent query parameter. Result on 2026-10-24: 84 calls checked, 132 owned discovery methods covered, 0 problems.

Coverage is part of the check: every Discovery method whose rpc id starts with aiplatform.publishers.models., aiplatform.projects.locations.publishers., aiplatform.projects.publishers.models., aiplatform.projects.locations.endpoints., aiplatform.projects.modelGardenEula. or aiplatform.projects.locations.deployPublisherModel (v1 and v1beta1, 132 methods) must be a row, the express form of a row, or the v1beta1 twin of an endpoint. row (verified field by field); the check fails on any other method until it is wrapped or excluded on purpose (EXCLUDED, empty today). So a Discovery change that adds a method to these resources is loud. The same pass shows what Discovery says about express forms: generateContent, streamGenerateContent, countTokens, computeTokens, predict, predictLongRunning, fetchPredictOperation exist without a project; embedContent, rawPredict, streamRawPredict, publisher invoke do not (embedContent express answers 404 live). publishers.models.list exists only in v1beta1; publishers.models.get in v1 and v1beta1.

Request construction is compared to the table in tests/src/google/models/test_models.cpp and test_models_admin.cpp (every body key must be a declared field; the planner also rejects a stray key at run time; every row plans to a concrete URL, express and project scoped).

6. Verified live

Run 2026-10-24 through this module's C++ client (MOLO_LIVE_NET=1 build/google_test, tests models_live.*), read-only or tiny (20 tokens). Credentials never printed or stored.

With an express API key (AQ.):

  • Gemini::generate, stream (SSE), count_tokens, compute_tokens, embed (text-embedding-005 predict) on aiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash: 200.
  • Project scoped countTokens at location global: 200.
  • Refused with 401 "API keys are not supported": Model Garden list/get, endpoints list, rawPredict, project embedContent. Express embedContent: 404 (matches Discovery).

With the service account OAuth token (project lisa-448004):

  • v1beta1 publishers/google/models list (google, anthropic), v1 publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash} get incl. view=PUBLISHER_MODEL_VIEW_FULL: 200.
  • Endpoints list (us-central1): 200, empty.
  • embedContent project scoped us-central1 gemini-embedding-001: 200 (global and text-embedding-005 not available there: 404/400).
  • Project scoped generateContent and computeTokens at global: 200 (us-central1 + gemini-2.5-flash: 404 "not found or no access").
  • Claude rawPredict at global (claude-sonnet-4-6): reached, 429 RESOURCE_EXHAUSTED (project quota); us-east5: 404 no access. Route, auth and body accepted; no completion was obtained.
  • fetchPublisherModelConfig: 404 "PublisherModelConfig not found" (route exists, nothing configured). EULA check for publishers/google/models/gemma3: 404 model not found (route exists).
  • Generation calls may answer 429 on the shared project; the live test notes that instead of failing.

Run 2026-10-24 (closing the gaps), service account token, project lisa-448004, through the new classes (models_live_admin.*) or direct requests:

  • EndpointAdmin read-only: v1beta1 endpoints list (us-central1): 200 {} (an empty list has no endpoints key); v1 and v1beta1 operations list of a missing endpoint: 200 {}; v1beta1 getIamPolicy, operations.get on a missing endpoint: 404 NOT_FOUND; stored responses (endpoints/{id}/responses/{id} get): 501 UNIMPLEMENTED for the project.
  • PublisherAdmin: v1beta1 publishers/google/models/gemini-2.5-flash get: 200; project scoped v1beta1 getIamPolicy on a publisher model: 400 INVALID_ARGUMENT (nothing to read there).
  • Thought signatures: gemini-3.5-flash at global with a function declaration, streamed: the functionCall part carries a thoughtSignature (TurnBuilder keeps 1 signed part). The stream ends with an empty text part, which TurnBuilder drops. The turn replayed with a functionResponse via VertexModel::replay is accepted (200, "The weather in Paris is 18°C."). Whether Google rejects a replay that lacks the signature was not tested.
  • Veo: veo-3.1-generate-001 exists for the project (us-central1 and global); veo-3.0-generate-001, veo-3.0-fast-generate-001, veo-2.0-generate-001 and every Imagen model tried (imagen-3.0-generate-001/002, imagen-3.0-fast-generate-001, imagen-4.0-generate-001, imagen-4.0-fast-generate-001, imagen-4.0-ultra-generate-001) answer 404 "not found or your project does not have access". predictLongRunning accepts instances[{prompt}] (an instance with text instead: 400 "No inputs provided") and answers {"name": "projects/.../publishers/google/models/veo-3.1-generate-001/operations/<uuid>"}. Parameters are validated inside the operation, which finishes done:true with error{code:3,message} on fetchPredictOperation: sampleCount 99 -> "Invalid sample count 99. The sample count should be in the range of [1, 4]" (name and range match our cap 1..4); durationSeconds 99 -> "Unsupported output video duration 99 seconds, supported durations are [8,4,6] for feature text_to_video" (name confirmed; the model accepts only 4, 6, 8, our cap 1..8 is wider); storageUri "bogus" -> "Unsupported output storage uri: bogus" (name confirmed); aspectRatio "bogus" is rejected synchronously with 400 "Invalid aspect ratio: bogus".
  • A finished Veo operation, read with fetchPredictOperation: done:true, response = {"@type":"type.googleapis.com/cloud.ai.large_models.vision.GenerateVideoResponse", "raiMediaFilteredCount":0, "videos":[{"bytesBase64Encoded":"...","mimeType":"video/mp4"}]} (one 8 s clip, about 13.8 MB decoded, under the 32 MiB cap). This is the real shape of the operation response Any for video and is what wait_video reads.
  • Lyria: lyria-002 predict (project scoped, us-central1 and global): instances[{prompt}] -> 200 {"predictions":[{"bytesBase64Encoded":"UklGR..."}]} with no mimeType; the bytes are a RIFF/WAVE file, so items_from names audio/wav when the mime is missing. An empty prompt or an instance without prompt -> 400 "Request contains an invalid argument".
  • Claude rawPredict and streamRawPredict at global: still 429 RESOURCE_EXHAUSTED on quota global_online_prediction_requests_per_base_model (anthropic-claude-sonnet-4-6). The streaming call reports the error as a JSON array [{"error":{...}}], not as one object.

Cost note: the Veo and Lyria probes were meant to be rejected requests. Google accepted several, so the run started 5 Veo operations (4 failed validation inside the operation and cost nothing, one generateAudio:"str" probe rendered one 8 s clip) and generated 3 Lyria clips. The probe scripts are not part of the repo; the live tests in the repo never call a generation method.

Not live-verified (mutating or billable): deploy, EULA accept, image generation (no Imagen model is visible to the project), endpoint create, patch, update, delete, deployModel, undeployModel, mutateDeployedModel, IAM set, publisher export, config and enableModel, the endpoint inference family (no endpoint exists). These are verified by Discovery and by local request tests.

7. Sources

  • Discovery documents: https://aiplatform.googleapis.com/$discovery/rest?version=v1 and ?version=v1beta1 (fetched 2026-10-24).
  • Claude on Vertex: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude (rawPredict/streamRawPredict paths, anthropic_version: vertex-2023-10-16, messages/max_tokens/stream/thinking, response shape).
  • Reference pages read for semantics: .../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent}, .../reference/rest/v1/publishers.models/get (view enum, languageCode).

8. Unverified (neither Discovery nor docs read, nor live)

Settled on 2026-10-24 and moved to section 6: Veo prompt, sampleCount, durationSeconds, aspectRatio, storageUri and the finished operation response; Lyria prompt and the predictions[].bytesBase64Encoded WAV response; the empty endpoint list shape; thought signatures on function calls.

  • Imagen predict instance/parameter names (prompt; sampleCount, aspectRatio, negativePrompt) and the predictions[].bytesBase64Encoded / mimeType response: no Imagen model is visible to project lisa-448004 (404 for every id tried), so even a rejected request is not informative.
  • Lyria negative_prompt, seed and sample_count: a live call accepted them, but it also accepted junk values (seed:"abc", sample_count:99), so acceptance does not prove the names. Whether a clip count above 1 is honored was not checked.
  • Veo generateAudio and the gcsUri result form with a real storageUri bucket (no bucket was used). A missing mimeType on Veo output was not seen.
  • Gemma/vLLM instance shape in gemma_instance (prompt, max_tokens, temperature): the serving container's, not Google's API; no endpoint exists to ask.
  • Anthropic streaming event names (content_block_delta / text_delta) come from Anthropic's format; Claude completions were not obtained live (429 quota on every attempt).
  • deployPublisherModel field semantics beyond the schema names.
  • Response shapes of the admin methods that need a real endpoint or a mutation: deployModel, directPredict, directRawPredict, explain, chat completions, openapi, invoke, stored responses (501 for this project), export, setPublisherModelConfig, enableModel. They are returned as raw Json (no field is read), so a different shape cannot break the client.
  • Whether Google rejects a Gemini 3 replay that omits the thoughtSignature (the accepted replay with it is verified).

9. Gaps

  • Every Discovery method of the owned resources is in the call table (see section 5); the only intentional difference is that project-less v1 forms are the express forms of rows and the v1beta1 forms of endpoint rows are twins selected with EndpointAdmin(beta=true).
  • The admin request bodies are validated by top-level key against the call table and by caps; nested fields (a DeployedModel, a PublisherModelConfig) are Json the caller owns. Typed structs exist where the module reads fields (EndpointInfo, Operation, PublisherModel).
  • Endpoint generateContent family is wired but only Discovery-verified.
  • VertexModel has no retry; quota 429 surfaces as GoogleError.
  • A streaming error body shaped as a JSON array ([{"error":...}], seen on Claude streamRawPredict) goes through the foundation StreamDecoder, which is outside this module; not verified that it maps to GoogleError.
  • Token refresh is the caller's job (foundation gap).