feat(clients): models: Gemini on Vertex, Model Garden, Claude, endpoint models, media, VertexModel
One call table verified against Discovery (29 calls), live-verified read-only with an express key and a service account token, /agent/models example routes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,163 @@
|
||||
# Models (Gemini Enterprise Agent Platform) - Specification
|
||||
|
||||
Owned directory: `src/google/models/`. Namespace `molo::google::models`.
|
||||
Depends on the foundation in `src/google/` (`GoogleClient`, `Service`,
|
||||
`GoogleError`, `CloudContext`, `Operation`, `gemini_types.h`). Voice and
|
||||
`Generative` are untouched.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
Gemini on Vertex AI (generate, stream, count tokens, compute tokens,
|
||||
embeddings), Model Garden (list, get, publisher model config, EULA, deploy),
|
||||
partner models (Claude rawPredict and streamRawPredict), endpoint models
|
||||
(Gemma and any deployed model: list, get, predict family), media generation
|
||||
(image, music, video via predict and predictLongRunning), and
|
||||
`VertexModel : LanguageModel`.
|
||||
|
||||
Two addressing modes, one class (`VertexTarget`):
|
||||
- express (API key, no project): `https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent`
|
||||
- project scoped: `https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:...`
|
||||
(`global` uses the host without a prefix).
|
||||
|
||||
## 2. Module map (one concern per file)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings |
|
||||
| `calls.h` | THE call table: every call, its Discovery rpc id and request fields |
|
||||
| `target.h/.cpp` | `VertexTarget`: express vs project, id validation |
|
||||
| `planner.h/.cpp` | `CallPlanner`: table row + ids -> exact request (pure, no I/O) |
|
||||
| `vertex_service.h/.cpp` | `VertexService : Service`: runs planned calls |
|
||||
| `json_read.h` | tolerant response readers |
|
||||
| `gemini.h/.cpp` | `Gemini`: generate, stream (`?alt=sse`), count/compute tokens, embedContent, predict embeddings |
|
||||
| `vertex_model.h/.cpp` | `VertexModel : LanguageModel` |
|
||||
| `model_garden.h/.cpp` | `ModelGarden`: list, get, fetch config, EULA check/accept, deploy |
|
||||
| `claude_models.h/.cpp` | `ClaudeModels`: rawPredict, streamRawPredict, `text_delta` |
|
||||
| `open_models.h/.cpp` | `OpenModels`: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke |
|
||||
| `media.h/.cpp` | `MediaGenerator`: image, music, video start/status/wait |
|
||||
| `tools/check_discovery.py` | checks `calls.h` against Google's Discovery documents |
|
||||
|
||||
Every call has a `plan_*` twin returning a `PlannedCall` (method, endpoint,
|
||||
path, body) without sending anything; tests and `check_discovery.py` verify
|
||||
requests that way.
|
||||
|
||||
## 3. Static caps (`caps.h`)
|
||||
|
||||
Contents per request 200; request text 1,000,000 chars; output tokens
|
||||
1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072
|
||||
dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude
|
||||
200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16,
|
||||
100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4,
|
||||
music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per
|
||||
result, operation name 512 chars; video polls 120 at 5 s (10 minutes).
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- Every id (publisher, model, endpoint, invoke id) passes `valid_segment`
|
||||
before a path is built; operation names and `publisherModel` strings have
|
||||
their own character check. Bad input is `GoogleError::invalid` (400
|
||||
`INVALID_ARGUMENT`) before any network traffic.
|
||||
- A call whose table row has `express=false` throws invalid in express mode,
|
||||
naming the missing project. No credential: `NotConfiguredError` (503).
|
||||
- Costly or consent calls are explicit and gated: `ModelGarden::deploy`
|
||||
needs `DeployOptions.confirm`, `accept_eula` needs `confirm=true`; both
|
||||
refuse otherwise. Nothing in this module deletes.
|
||||
- Streaming uses `GoogleClient::stream_json` (SSE via `?alt=sse` for Gemini,
|
||||
Anthropic events for Claude); the sink returns false to stop.
|
||||
- Video: `start_video` (predictLongRunning), `video_status`
|
||||
(fetchPredictOperation, a POST), `wait_video` bounded by the caps.
|
||||
`wait_operation` (GET) is not used: Discovery has no GET on these
|
||||
operations for publisher models.
|
||||
- Example (`/agent/models`): GET page, `/status`, `/calls` need no
|
||||
credential; every Google call is a POST that answers 503
|
||||
`google_not_configured` without a credential (checked first), 422 on bad
|
||||
input. Routes: generate, count, embed, claude, list, endpoint, image,
|
||||
music, video, video/status.
|
||||
|
||||
## 5. Verified against discovery
|
||||
|
||||
`python3 -I src/google/models/tools/check_discovery.py` fetches
|
||||
`https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `v1beta1`
|
||||
and fails when any row of `calls.h` has a missing method or path, a wrong
|
||||
rpc id, an absent request body field (dotted names follow the schema) or an
|
||||
absent query parameter. Result on 2026-10-24: `29 calls checked, 0 problems`.
|
||||
The same pass shows what Discovery says about express forms:
|
||||
generateContent, streamGenerateContent, countTokens, computeTokens, predict,
|
||||
predictLongRunning, fetchPredictOperation exist without a project;
|
||||
embedContent, rawPredict, streamRawPredict, publisher invoke do not
|
||||
(`embedContent` express answers 404 live). `publishers.models.list` exists
|
||||
only in v1beta1; `publishers.models.get` in v1 and v1beta1.
|
||||
|
||||
Request construction is compared to the table in
|
||||
`tests/src/google/models/test_models.cpp` (every body key must be a declared
|
||||
field).
|
||||
|
||||
## 6. Verified live
|
||||
|
||||
Run 2026-10-24 through this module's C++ client
|
||||
(`MOLO_LIVE_NET=1 build/google_test`, tests `models_live.*`), read-only or
|
||||
tiny (20 tokens). Credentials never printed or stored.
|
||||
|
||||
With an express API key (`AQ.`):
|
||||
- `Gemini::generate`, `stream` (SSE), `count_tokens`, `compute_tokens`,
|
||||
`embed` (text-embedding-005 predict) on `aiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash`: 200.
|
||||
- Project scoped `countTokens` at location `global`: 200.
|
||||
- Refused with 401 "API keys are not supported": Model Garden list/get,
|
||||
endpoints list, rawPredict, project embedContent. Express `embedContent`:
|
||||
404 (matches Discovery).
|
||||
|
||||
With the service account OAuth token (project `lisa-448004`):
|
||||
- v1beta1 `publishers/google/models` list (google, anthropic), v1
|
||||
`publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash}` get incl.
|
||||
`view=PUBLISHER_MODEL_VIEW_FULL`: 200.
|
||||
- Endpoints list (`us-central1`): 200, empty.
|
||||
- `embedContent` project scoped `us-central1` `gemini-embedding-001`: 200
|
||||
(global and text-embedding-005 not available there: 404/400).
|
||||
- Project scoped `generateContent` and `computeTokens` at `global`: 200
|
||||
(`us-central1` + gemini-2.5-flash: 404 "not found or no access").
|
||||
- Claude `rawPredict` at `global` (claude-sonnet-4-6): reached, 429
|
||||
RESOURCE_EXHAUSTED (project quota); `us-east5`: 404 no access. Route, auth
|
||||
and body accepted; no completion was obtained.
|
||||
- `fetchPublisherModelConfig`: 404 "PublisherModelConfig not found" (route
|
||||
exists, nothing configured). EULA check for `publishers/google/models/gemma3`:
|
||||
404 model not found (route exists).
|
||||
- Generation calls may answer 429 on the shared project; the live test notes
|
||||
that instead of failing.
|
||||
|
||||
Not live-verified (mutating or billable): deploy, EULA accept, image, music
|
||||
and video generation, endpoint predict family (no endpoint exists), Gemma
|
||||
shapes. These are verified by Discovery only.
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- Discovery documents: `https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `?version=v1beta1` (fetched 2026-10-24).
|
||||
- Claude on Vertex: `https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude` (rawPredict/streamRawPredict paths, `anthropic_version: vertex-2023-10-16`, messages/max_tokens/stream/thinking, response shape).
|
||||
- Reference pages read for semantics: `.../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent}`, `.../reference/rest/v1/publishers.models/get` (view enum, languageCode).
|
||||
|
||||
## 8. Unverified (neither Discovery nor docs read, nor live)
|
||||
|
||||
- Imagen `predict` instance/parameter names (`prompt`; `sampleCount`,
|
||||
`aspectRatio`, `negativePrompt`) and the `predictions[].bytesBase64Encoded`
|
||||
/ `mimeType` response: Discovery types them as free-form `Value`; the doc
|
||||
pages could not be read.
|
||||
- Lyria shape (`prompt`, `negative_prompt`, `seed`; `sample_count`).
|
||||
- Veo shape (`sampleCount`, `durationSeconds`, `aspectRatio`, `storageUri`;
|
||||
final `response.videos[]` with `gcsUri` / `bytesBase64Encoded` / `mimeType`).
|
||||
`fetchPredictOperation` itself is Discovery-verified and reachable live.
|
||||
- Gemma/vLLM instance shape in `gemma_instance` (`prompt`, `max_tokens`,
|
||||
`temperature`): the serving container's, not Google's API.
|
||||
- Anthropic streaming event names (`content_block_delta` / `text_delta`)
|
||||
come from Anthropic's format; Claude completions were not obtained live.
|
||||
- Embedding predict response `embeddings.values` / `statistics.token_count`
|
||||
was observed live for text-embedding-005, not documented-read.
|
||||
- `deployPublisherModel` field semantics beyond the schema names.
|
||||
|
||||
## 9. Gaps
|
||||
|
||||
- Discovery methods deliberately not wrapped (owner: this module): publisher
|
||||
`export`, `setPublisherModelConfig`, `enableModel`, IAM policy, `v1/responses`
|
||||
on publishers, publisher-level `serverStreamingPredict` and v1beta1
|
||||
`publishers.models.*` project-less duplicates.
|
||||
- Endpoint `generateContent` family is wired but only Discovery-verified.
|
||||
- `VertexModel` has no retry; quota 429 surfaces as `GoogleError`.
|
||||
- Token refresh is the caller's job (foundation gap).
|
||||
Reference in New Issue
Block a user