# Models (Gemini Enterprise Agent Platform) - Specification Owned directory: `src/google/models/`. Namespace `molo::google::models`. Depends on the foundation in `src/google/` (`GoogleClient`, `Service`, `GoogleError`, `CloudContext`, `Operation`, `gemini_types.h`). Voice and `Generative` are untouched. ## 1. Scope Gemini on Vertex AI (generate, stream, count tokens, compute tokens, embeddings), Model Garden (list, get, publisher model config, EULA, deploy), partner models (Claude rawPredict and streamRawPredict), endpoint models (Gemma and any deployed model: list, get, predict family), media generation (image, music, video via predict and predictLongRunning), and `VertexModel : LanguageModel`. Two addressing modes, one class (`VertexTarget`): - express (API key, no project): `https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent` - project scoped: `https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:...` (`global` uses the host without a prefix). ## 2. Module map (one concern per file) | File | Concern | |---|---| | `caps.h` | Static ceilings | | `calls.h` | THE call table: every call, its Discovery rpc id and request fields | | `target.h/.cpp` | `VertexTarget`: express vs project, id validation | | `planner.h/.cpp` | `CallPlanner`: table row + ids -> exact request (pure, no I/O) | | `vertex_service.h/.cpp` | `VertexService : Service`: runs planned calls | | `json_read.h` | tolerant response readers | | `gemini.h/.cpp` | `Gemini`: generate, stream (`?alt=sse`), count/compute tokens, embedContent, predict embeddings | | `vertex_model.h/.cpp` | `VertexModel : LanguageModel`, `TurnBuilder` (replayable model turn with every `thoughtSignature`) | | `admin_checks.h` | Input rules shared by the admin classes (object, size, mask, confirm) | | `endpoint_admin.h/.cpp` | `EndpointAdmin`: endpoint lifecycle, remaining inference surfaces, endpoint operations, v1beta1 IAM and list | | `publisher_admin.h/.cpp` | `PublisherAdmin`: publisher `v1/responses`, serverStreamingPredict, export, config, enableModel, IAM, v1beta1 forms | | `model_garden.h/.cpp` | `ModelGarden`: list, get, fetch config, EULA check/accept, deploy | | `claude_models.h/.cpp` | `ClaudeModels`: rawPredict, streamRawPredict, `text_delta` | | `open_models.h/.cpp` | `OpenModels`: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke | | `media.h/.cpp` | `MediaGenerator`: image, music, video start/status/wait | | `tools/check_discovery.py` | checks `calls.h` against Google's Discovery documents | Every call has a `plan_*` twin returning a `PlannedCall` (method, endpoint, path, body) without sending anything; tests and `check_discovery.py` verify requests that way. ## 3. Static caps (`caps.h`) Contents per request 200; request text 1,000,000 chars; output tokens 1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072 dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude 200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16, 100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4, music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per result, operation name 512 chars; video polls 120 at 5 s (10 minutes). Admin surface: request body 262144 chars serialized, updateMask 512 chars, operations.wait 1..600 s, operation list pages 1..5, 64 permissions, 5000 streamed events. ## 4. Contracts - Every id (publisher, model, endpoint, invoke id) passes `valid_segment` before a path is built; operation names and `publisherModel` strings have their own character check. Bad input is `GoogleError::invalid` (400 `INVALID_ARGUMENT`) before any network traffic. - A call whose table row has `express=false` throws invalid in express mode, naming the missing project. No credential: `NotConfiguredError` (503). - Costly or consent calls are explicit and gated: `ModelGarden::deploy` needs `DeployOptions.confirm`, `accept_eula` needs `confirm=true`; both refuse otherwise. Nothing in this module deletes. - Streaming uses `GoogleClient::stream_json` (SSE via `?alt=sse` for Gemini, Anthropic events for Claude); the sink returns false to stop. - Video: `start_video` (predictLongRunning), `video_status` (fetchPredictOperation, a POST), `wait_video` bounded by the caps. `wait_operation` (GET) is not used: Discovery has no GET on these operations for publisher models. - Admin bodies are Json whose top-level keys must be declared in the call table: `CallPlanner::plan` rejects a stray key (`INVALID_ARGUMENT`) before any traffic; rows with `-` (raw HTTP bodies) take any object. - `EndpointAdmin(beta=true)` sends the v1beta1 twin of every `endpoint.` row (`CallPlanner::plan(..., beta=true)`); `tools/check_discovery.py` verifies each twin (same method, path, rpc id, fields). `endpoint.beta_*` rows exist in v1beta1 only (list with `gdcZone`, IAM). - Gated admin calls refuse without `confirm=true`: endpoint delete, deployModel, undeployModel, mutateDeployedModel, IAM set, stored response delete, operation record delete, publisher export, setPublisherModelConfig, enableModel. The example routes answer a dry run instead. - `VertexModel::TurnBuilder` builds the model turn to replay from stream chunks: text parts merge only while they carry no `thoughtSignature`, a signature that arrives on an empty part attaches to the part before it, calls and data parts are kept as received. `VertexModel::model_turn` and `replay` do the same for a whole reply. `Gemini::validate` accepts a signature-only part (it is not "empty"). - Example (`/agent/models`): GET page, `/status`, `/calls` need no credential; every Google call is a POST that answers 503 `google_not_configured` without a credential (checked first), 422 on bad input. Routes: generate, count, embed, claude, list, endpoint, image, music, video, video/status, and the admin routes `endpoints` (create, patch, update, delete, deploy, undeploy, mutate, list_beta, get_iam, set_iam, test_iam), `endpoint/call` (compute_tokens, direct_predict, direct_raw_predict, explain, invoke, deployed_invoke, chat, openapi, science, response_get, response_delete), `endpoint/operations` (get, list, cancel, wait, delete), `publisher` (stream_predict, responses_*, export, set_config, enable, iam, beta_get, beta_generate, beta_count, beta_post, beta_invoke). The video default model is `veo-3.1-generate-001` (`veo-3.0-generate-001` answers 404 for the project). ## 5. Verified against discovery `python3 -I src/google/models/tools/check_discovery.py` fetches `https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `v1beta1` and fails when any row of `calls.h` has a missing method or path, a wrong rpc id, an absent request body field (dotted names follow the schema) or an absent query parameter. Result on 2026-10-24: `84 calls checked, 132 owned discovery methods covered, 0 problems`. Coverage is part of the check: every Discovery method whose rpc id starts with `aiplatform.publishers.models.`, `aiplatform.projects.locations.publishers.`, `aiplatform.projects.publishers.models.`, `aiplatform.projects.locations.endpoints.`, `aiplatform.projects.modelGardenEula.` or `aiplatform.projects.locations.deployPublisherModel` (v1 and v1beta1, 132 methods) must be a row, the express form of a row, or the v1beta1 twin of an `endpoint.` row (verified field by field); the check fails on any other method until it is wrapped or excluded on purpose (`EXCLUDED`, empty today). So a Discovery change that adds a method to these resources is loud. The same pass shows what Discovery says about express forms: generateContent, streamGenerateContent, countTokens, computeTokens, predict, predictLongRunning, fetchPredictOperation exist without a project; embedContent, rawPredict, streamRawPredict, publisher invoke do not (`embedContent` express answers 404 live). `publishers.models.list` exists only in v1beta1; `publishers.models.get` in v1 and v1beta1. Request construction is compared to the table in `tests/src/google/models/test_models.cpp` and `test_models_admin.cpp` (every body key must be a declared field; the planner also rejects a stray key at run time; every row plans to a concrete URL, express and project scoped). ## 6. Verified live Run 2026-10-24 through this module's C++ client (`MOLO_LIVE_NET=1 build/google_test`, tests `models_live.*`), read-only or tiny (20 tokens). Credentials never printed or stored. With an express API key (`AQ.`): - `Gemini::generate`, `stream` (SSE), `count_tokens`, `compute_tokens`, `embed` (text-embedding-005 predict) on `aiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash`: 200. - Project scoped `countTokens` at location `global`: 200. - Refused with 401 "API keys are not supported": Model Garden list/get, endpoints list, rawPredict, project embedContent. Express `embedContent`: 404 (matches Discovery). With the service account OAuth token (project `lisa-448004`): - v1beta1 `publishers/google/models` list (google, anthropic), v1 `publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash}` get incl. `view=PUBLISHER_MODEL_VIEW_FULL`: 200. - Endpoints list (`us-central1`): 200, empty. - `embedContent` project scoped `us-central1` `gemini-embedding-001`: 200 (global and text-embedding-005 not available there: 404/400). - Project scoped `generateContent` and `computeTokens` at `global`: 200 (`us-central1` + gemini-2.5-flash: 404 "not found or no access"). - Claude `rawPredict` at `global` (claude-sonnet-4-6): reached, 429 RESOURCE_EXHAUSTED (project quota); `us-east5`: 404 no access. Route, auth and body accepted; no completion was obtained. - `fetchPublisherModelConfig`: 404 "PublisherModelConfig not found" (route exists, nothing configured). EULA check for `publishers/google/models/gemma3`: 404 model not found (route exists). - Generation calls may answer 429 on the shared project; the live test notes that instead of failing. Run 2026-10-24 (closing the gaps), service account token, project `lisa-448004`, through the new classes (`models_live_admin.*`) or direct requests: - `EndpointAdmin` read-only: v1beta1 endpoints list (`us-central1`): 200 `{}` (an empty list has no `endpoints` key); v1 and v1beta1 operations list of a missing endpoint: 200 `{}`; v1beta1 `getIamPolicy`, `operations.get` on a missing endpoint: 404 NOT_FOUND; stored responses (`endpoints/{id}/responses/{id}` get): 501 UNIMPLEMENTED for the project. - `PublisherAdmin`: v1beta1 `publishers/google/models/gemini-2.5-flash` get: 200; project scoped v1beta1 `getIamPolicy` on a publisher model: 400 INVALID_ARGUMENT (nothing to read there). - Thought signatures: `gemini-3.5-flash` at `global` with a function declaration, streamed: the `functionCall` part carries a `thoughtSignature` (`TurnBuilder` keeps 1 signed part). The stream ends with an empty text part, which `TurnBuilder` drops. The turn replayed with a `functionResponse` via `VertexModel::replay` is accepted (200, "The weather in Paris is 18°C."). Whether Google rejects a replay that lacks the signature was not tested. - Veo: `veo-3.1-generate-001` exists for the project (us-central1 and `global`); `veo-3.0-generate-001`, `veo-3.0-fast-generate-001`, `veo-2.0-generate-001` and every Imagen model tried (`imagen-3.0-generate-001/002`, `imagen-3.0-fast-generate-001`, `imagen-4.0-generate-001`, `imagen-4.0-fast-generate-001`, `imagen-4.0-ultra-generate-001`) answer 404 "not found or your project does not have access". `predictLongRunning` accepts `instances[{prompt}]` (an instance with `text` instead: 400 "No inputs provided") and answers `{"name": "projects/.../publishers/google/models/veo-3.1-generate-001/operations/"}`. Parameters are validated inside the operation, which finishes `done:true` with `error{code:3,message}` on `fetchPredictOperation`: `sampleCount` 99 -> "Invalid sample count 99. The sample count should be in the range of [1, 4]" (name and range match our cap 1..4); `durationSeconds` 99 -> "Unsupported output video duration 99 seconds, supported durations are [8,4,6] for feature text_to_video" (name confirmed; the model accepts only 4, 6, 8, our cap 1..8 is wider); `storageUri` "bogus" -> "Unsupported output storage uri: bogus" (name confirmed); `aspectRatio` "bogus" is rejected synchronously with 400 "Invalid aspect ratio: bogus". - A finished Veo operation, read with `fetchPredictOperation`: `done:true`, `response` = `{"@type":"type.googleapis.com/cloud.ai.large_models.vision.GenerateVideoResponse", "raiMediaFilteredCount":0, "videos":[{"bytesBase64Encoded":"...","mimeType":"video/mp4"}]}` (one 8 s clip, about 13.8 MB decoded, under the 32 MiB cap). This is the real shape of the operation `response` Any for video and is what `wait_video` reads. - Lyria: `lyria-002` `predict` (project scoped, us-central1 and `global`): `instances[{prompt}]` -> 200 `{"predictions":[{"bytesBase64Encoded":"UklGR..."}]}` with no `mimeType`; the bytes are a RIFF/WAVE file, so `items_from` names `audio/wav` when the mime is missing. An empty prompt or an instance without `prompt` -> 400 "Request contains an invalid argument". - Claude `rawPredict` and `streamRawPredict` at `global`: still 429 RESOURCE_EXHAUSTED on quota `global_online_prediction_requests_per_base_model` (`anthropic-claude-sonnet-4-6`). The streaming call reports the error as a JSON array `[{"error":{...}}]`, not as one object. Cost note: the Veo and Lyria probes were meant to be rejected requests. Google accepted several, so the run started 5 Veo operations (4 failed validation inside the operation and cost nothing, one `generateAudio:"str"` probe rendered one 8 s clip) and generated 3 Lyria clips. The probe scripts are not part of the repo; the live tests in the repo never call a generation method. Not live-verified (mutating or billable): deploy, EULA accept, image generation (no Imagen model is visible to the project), endpoint create, patch, update, delete, deployModel, undeployModel, mutateDeployedModel, IAM set, publisher export, config and enableModel, the endpoint inference family (no endpoint exists). These are verified by Discovery and by local request tests. ## 7. Sources - Discovery documents: `https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `?version=v1beta1` (fetched 2026-10-24). - Claude on Vertex: `https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude` (rawPredict/streamRawPredict paths, `anthropic_version: vertex-2023-10-16`, messages/max_tokens/stream/thinking, response shape). - Reference pages read for semantics: `.../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent}`, `.../reference/rest/v1/publishers.models/get` (view enum, languageCode). ## 8. Unverified (neither Discovery nor docs read, nor live) Settled on 2026-10-24 and moved to section 6: Veo `prompt`, `sampleCount`, `durationSeconds`, `aspectRatio`, `storageUri` and the finished operation `response`; Lyria `prompt` and the `predictions[].bytesBase64Encoded` WAV response; the empty endpoint list shape; thought signatures on function calls. - Imagen `predict` instance/parameter names (`prompt`; `sampleCount`, `aspectRatio`, `negativePrompt`) and the `predictions[].bytesBase64Encoded` / `mimeType` response: no Imagen model is visible to project `lisa-448004` (404 for every id tried), so even a rejected request is not informative. - Lyria `negative_prompt`, `seed` and `sample_count`: a live call accepted them, but it also accepted junk values (`seed:"abc"`, `sample_count:99`), so acceptance does not prove the names. Whether a clip count above 1 is honored was not checked. - Veo `generateAudio` and the `gcsUri` result form with a real `storageUri` bucket (no bucket was used). A missing `mimeType` on Veo output was not seen. - Gemma/vLLM instance shape in `gemma_instance` (`prompt`, `max_tokens`, `temperature`): the serving container's, not Google's API; no endpoint exists to ask. - Anthropic streaming event names (`content_block_delta` / `text_delta`) come from Anthropic's format; Claude completions were not obtained live (429 quota on every attempt). - `deployPublisherModel` field semantics beyond the schema names. - Response shapes of the admin methods that need a real endpoint or a mutation: deployModel, directPredict, directRawPredict, explain, chat completions, openapi, invoke, stored responses (501 for this project), export, setPublisherModelConfig, enableModel. They are returned as raw Json (no field is read), so a different shape cannot break the client. - Whether Google rejects a Gemini 3 replay that omits the `thoughtSignature` (the accepted replay with it is verified). ## 9. Gaps - Every Discovery method of the owned resources is in the call table (see section 5); the only intentional difference is that project-less v1 forms are the `express` forms of rows and the v1beta1 forms of endpoint rows are twins selected with `EndpointAdmin(beta=true)`. - The admin request bodies are validated by top-level key against the call table and by caps; nested fields (a `DeployedModel`, a `PublisherModelConfig`) are Json the caller owns. Typed structs exist where the module reads fields (`EndpointInfo`, `Operation`, `PublisherModel`). - Endpoint `generateContent` family is wired but only Discovery-verified. - `VertexModel` has no retry; quota 429 surfaces as `GoogleError`. - A streaming error body shaped as a JSON array (`[{"error":...}]`, seen on Claude `streamRawPredict`) goes through the foundation `StreamDecoder`, which is outside this module; not verified that it maps to `GoogleError`. - Token refresh is the caller's job (foundation gap). ## Required APIs Enable these on the project (placeholder project `example-214512124`; full list and a one-line `gcloud` command in `src/google/APIS.md`). A disabled API answers 403 `SERVICE_DISABLED`. - `aiplatform.googleapis.com`: https://console.developers.google.com/apis/api/aiplatform.googleapis.com/overview?project=example-214512124 - `generativelanguage.googleapis.com`: https://console.developers.google.com/apis/api/generativelanguage.googleapis.com/overview?project=example-214512124 (only for the AI Studio key path, not express mode)