164 lines
9.1 KiB
Markdown
164 lines
9.1 KiB
Markdown
# Models (Gemini Enterprise Agent Platform) - Specification
|
|||
|
|
|
||
|
|
Owned directory: `src/google/models/`. Namespace `molo::google::models`.
|
||
|
|
Depends on the foundation in `src/google/` (`GoogleClient`, `Service`,
|
||
|
|
`GoogleError`, `CloudContext`, `Operation`, `gemini_types.h`). Voice and
|
||
|
|
`Generative` are untouched.
|
||
|
|
|
||
|
|
## 1. Scope
|
||
|
|
|
||
|
|
Gemini on Vertex AI (generate, stream, count tokens, compute tokens,
|
||
|
|
embeddings), Model Garden (list, get, publisher model config, EULA, deploy),
|
||
|
|
partner models (Claude rawPredict and streamRawPredict), endpoint models
|
||
|
|
(Gemma and any deployed model: list, get, predict family), media generation
|
||
|
|
(image, music, video via predict and predictLongRunning), and
|
||
|
|
`VertexModel : LanguageModel`.
|
||
|
|
|
||
|
|
Two addressing modes, one class (`VertexTarget`):
|
||
|
|
- express (API key, no project): `https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent`
|
||
|
|
- project scoped: `https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:...`
|
||
|
|
(`global` uses the host without a prefix).
|
||
|
|
|
||
|
|
## 2. Module map (one concern per file)
|
||
|
|
|
||
|
|
| File | Concern |
|
||
|
|
|---|---|
|
||
|
|
| `caps.h` | Static ceilings |
|
||
|
|
| `calls.h` | THE call table: every call, its Discovery rpc id and request fields |
|
||
|
|
| `target.h/.cpp` | `VertexTarget`: express vs project, id validation |
|
||
|
|
| `planner.h/.cpp` | `CallPlanner`: table row + ids -> exact request (pure, no I/O) |
|
||
|
|
| `vertex_service.h/.cpp` | `VertexService : Service`: runs planned calls |
|
||
|
|
| `json_read.h` | tolerant response readers |
|
||
|
|
| `gemini.h/.cpp` | `Gemini`: generate, stream (`?alt=sse`), count/compute tokens, embedContent, predict embeddings |
|
||
|
|
| `vertex_model.h/.cpp` | `VertexModel : LanguageModel` |
|
||
|
|
| `model_garden.h/.cpp` | `ModelGarden`: list, get, fetch config, EULA check/accept, deploy |
|
||
|
|
| `claude_models.h/.cpp` | `ClaudeModels`: rawPredict, streamRawPredict, `text_delta` |
|
||
|
|
| `open_models.h/.cpp` | `OpenModels`: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke |
|
||
|
|
| `media.h/.cpp` | `MediaGenerator`: image, music, video start/status/wait |
|
||
|
|
| `tools/check_discovery.py` | checks `calls.h` against Google's Discovery documents |
|
||
|
|
|
||
|
|
Every call has a `plan_*` twin returning a `PlannedCall` (method, endpoint,
|
||
|
|
path, body) without sending anything; tests and `check_discovery.py` verify
|
||
|
|
requests that way.
|
||
|
|
|
||
|
|
## 3. Static caps (`caps.h`)
|
||
|
|
|
||
|
|
Contents per request 200; request text 1,000,000 chars; output tokens
|
||
|
|
1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072
|
||
|
|
dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude
|
||
|
|
200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16,
|
||
|
|
100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4,
|
||
|
|
music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per
|
||
|
|
result, operation name 512 chars; video polls 120 at 5 s (10 minutes).
|
||
|
|
|
||
|
|
## 4. Contracts
|
||
|
|
|
||
|
|
- Every id (publisher, model, endpoint, invoke id) passes `valid_segment`
|
||
|
|
before a path is built; operation names and `publisherModel` strings have
|
||
|
|
their own character check. Bad input is `GoogleError::invalid` (400
|
||
|
|
`INVALID_ARGUMENT`) before any network traffic.
|
||
|
|
- A call whose table row has `express=false` throws invalid in express mode,
|
||
|
|
naming the missing project. No credential: `NotConfiguredError` (503).
|
||
|
|
- Costly or consent calls are explicit and gated: `ModelGarden::deploy`
|
||
|
|
needs `DeployOptions.confirm`, `accept_eula` needs `confirm=true`; both
|
||
|
|
refuse otherwise. Nothing in this module deletes.
|
||
|
|
- Streaming uses `GoogleClient::stream_json` (SSE via `?alt=sse` for Gemini,
|
||
|
|
Anthropic events for Claude); the sink returns false to stop.
|
||
|
|
- Video: `start_video` (predictLongRunning), `video_status`
|
||
|
|
(fetchPredictOperation, a POST), `wait_video` bounded by the caps.
|
||
|
|
`wait_operation` (GET) is not used: Discovery has no GET on these
|
||
|
|
operations for publisher models.
|
||
|
|
- Example (`/agent/models`): GET page, `/status`, `/calls` need no
|
||
|
|
credential; every Google call is a POST that answers 503
|
||
|
|
`google_not_configured` without a credential (checked first), 422 on bad
|
||
|
|
input. Routes: generate, count, embed, claude, list, endpoint, image,
|
||
|
|
music, video, video/status.
|
||
|
|
|
||
|
|
## 5. Verified against discovery
|
||
|
|
|
||
|
|
`python3 -I src/google/models/tools/check_discovery.py` fetches
|
||
|
|
`https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `v1beta1`
|
||
|
|
and fails when any row of `calls.h` has a missing method or path, a wrong
|
||
|
|
rpc id, an absent request body field (dotted names follow the schema) or an
|
||
|
|
absent query parameter. Result on 2026-10-24: `29 calls checked, 0 problems`.
|
||
|
|
The same pass shows what Discovery says about express forms:
|
||
|
|
generateContent, streamGenerateContent, countTokens, computeTokens, predict,
|
||
|
|
predictLongRunning, fetchPredictOperation exist without a project;
|
||
|
|
embedContent, rawPredict, streamRawPredict, publisher invoke do not
|
||
|
|
(`embedContent` express answers 404 live). `publishers.models.list` exists
|
||
|
|
only in v1beta1; `publishers.models.get` in v1 and v1beta1.
|
||
|
|
|
||
|
|
Request construction is compared to the table in
|
||
|
|
`tests/src/google/models/test_models.cpp` (every body key must be a declared
|
||
|
|
field).
|
||
|
|
|
||
|
|
## 6. Verified live
|
||
|
|
|
||
|
|
Run 2026-10-24 through this module's C++ client
|
||
|
|
(`MOLO_LIVE_NET=1 build/google_test`, tests `models_live.*`), read-only or
|
||
|
|
tiny (20 tokens). Credentials never printed or stored.
|
||
|
|
|
||
|
|
With an express API key (`AQ.`):
|
||
|
|
- `Gemini::generate`, `stream` (SSE), `count_tokens`, `compute_tokens`,
|
||
|
|
`embed` (text-embedding-005 predict) on `aiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash`: 200.
|
||
|
|
- Project scoped `countTokens` at location `global`: 200.
|
||
|
|
- Refused with 401 "API keys are not supported": Model Garden list/get,
|
||
|
|
endpoints list, rawPredict, project embedContent. Express `embedContent`:
|
||
|
|
404 (matches Discovery).
|
||
|
|
|
||
|
|
With the service account OAuth token (project `lisa-448004`):
|
||
|
|
- v1beta1 `publishers/google/models` list (google, anthropic), v1
|
||
|
|
`publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash}` get incl.
|
||
|
|
`view=PUBLISHER_MODEL_VIEW_FULL`: 200.
|
||
|
|
- Endpoints list (`us-central1`): 200, empty.
|
||
|
|
- `embedContent` project scoped `us-central1` `gemini-embedding-001`: 200
|
||
|
|
(global and text-embedding-005 not available there: 404/400).
|
||
|
|
- Project scoped `generateContent` and `computeTokens` at `global`: 200
|
||
|
|
(`us-central1` + gemini-2.5-flash: 404 "not found or no access").
|
||
|
|
- Claude `rawPredict` at `global` (claude-sonnet-4-6): reached, 429
|
||
|
|
RESOURCE_EXHAUSTED (project quota); `us-east5`: 404 no access. Route, auth
|
||
|
|
and body accepted; no completion was obtained.
|
||
|
|
- `fetchPublisherModelConfig`: 404 "PublisherModelConfig not found" (route
|
||
|
|
exists, nothing configured). EULA check for `publishers/google/models/gemma3`:
|
||
|
|
404 model not found (route exists).
|
||
|
|
- Generation calls may answer 429 on the shared project; the live test notes
|
||
|
|
that instead of failing.
|
||
|
|
|
||
|
|
Not live-verified (mutating or billable): deploy, EULA accept, image, music
|
||
|
|
and video generation, endpoint predict family (no endpoint exists), Gemma
|
||
|
|
shapes. These are verified by Discovery only.
|
||
|
|
|
||
|
|
## 7. Sources
|
||
|
|
|
||
|
|
- Discovery documents: `https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `?version=v1beta1` (fetched 2026-10-24).
|
||
|
|
- Claude on Vertex: `https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude` (rawPredict/streamRawPredict paths, `anthropic_version: vertex-2023-10-16`, messages/max_tokens/stream/thinking, response shape).
|
||
|
|
- Reference pages read for semantics: `.../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent}`, `.../reference/rest/v1/publishers.models/get` (view enum, languageCode).
|
||
|
|
|
||
|
|
## 8. Unverified (neither Discovery nor docs read, nor live)
|
||
|
|
|
||
|
|
- Imagen `predict` instance/parameter names (`prompt`; `sampleCount`,
|
||
|
|
`aspectRatio`, `negativePrompt`) and the `predictions[].bytesBase64Encoded`
|
||
|
|
/ `mimeType` response: Discovery types them as free-form `Value`; the doc
|
||
|
|
pages could not be read.
|
||
|
|
- Lyria shape (`prompt`, `negative_prompt`, `seed`; `sample_count`).
|
||
|
|
- Veo shape (`sampleCount`, `durationSeconds`, `aspectRatio`, `storageUri`;
|
||
|
|
final `response.videos[]` with `gcsUri` / `bytesBase64Encoded` / `mimeType`).
|
||
|
|
`fetchPredictOperation` itself is Discovery-verified and reachable live.
|
||
|
|
- Gemma/vLLM instance shape in `gemma_instance` (`prompt`, `max_tokens`,
|
||
|
|
`temperature`): the serving container's, not Google's API.
|
||
|
|
- Anthropic streaming event names (`content_block_delta` / `text_delta`)
|
||
|
|
come from Anthropic's format; Claude completions were not obtained live.
|
||
|
|
- Embedding predict response `embeddings.values` / `statistics.token_count`
|
||
|
|
was observed live for text-embedding-005, not documented-read.
|
||
|
|
- `deployPublisherModel` field semantics beyond the schema names.
|
||
|
|
|
||
|
|
## 9. Gaps
|
||
|
|
|
||
|
|
- Discovery methods deliberately not wrapped (owner: this module): publisher
|
||
|
|
`export`, `setPublisherModelConfig`, `enableModel`, IAM policy, `v1/responses`
|
||
|
|
on publishers, publisher-level `serverStreamingPredict` and v1beta1
|
||
|
|
`publishers.models.*` project-less duplicates.
|
||
|
|
- Endpoint `generateContent` family is wired but only Discovery-verified.
|
||
|
|
- `VertexModel` has no retry; quota 429 surfaces as `GoogleError`.
|
||
|
|
- Token refresh is the caller's job (foundation gap).
|