Files
moloapp/src/google/models/SPEC.md
T
retoorandClaude Sonnet 5.5 6b731643bf feat(clients): models: Gemini on Vertex, Model Garden, Claude, endpoint models, media, VertexModel
One call table verified against Discovery (29 calls), live-verified read-only
with an express key and a service account token, /agent/models example routes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 04:40:00 +02:00

9.1 KiB

Models (Gemini Enterprise Agent Platform) - Specification

Owned directory: src/google/models/. Namespace molo::google::models. Depends on the foundation in src/google/ (GoogleClient, Service, GoogleError, CloudContext, Operation, gemini_types.h). Voice and Generative are untouched.

1. Scope

Gemini on Vertex AI (generate, stream, count tokens, compute tokens, embeddings), Model Garden (list, get, publisher model config, EULA, deploy), partner models (Claude rawPredict and streamRawPredict), endpoint models (Gemma and any deployed model: list, get, predict family), media generation (image, music, video via predict and predictLongRunning), and VertexModel : LanguageModel.

Two addressing modes, one class (VertexTarget):

  • express (API key, no project): https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent
  • project scoped: https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:... (global uses the host without a prefix).

2. Module map (one concern per file)

File Concern
caps.h Static ceilings
calls.h THE call table: every call, its Discovery rpc id and request fields
target.h/.cpp VertexTarget: express vs project, id validation
planner.h/.cpp CallPlanner: table row + ids -> exact request (pure, no I/O)
vertex_service.h/.cpp VertexService : Service: runs planned calls
json_read.h tolerant response readers
gemini.h/.cpp Gemini: generate, stream (?alt=sse), count/compute tokens, embedContent, predict embeddings
vertex_model.h/.cpp VertexModel : LanguageModel
model_garden.h/.cpp ModelGarden: list, get, fetch config, EULA check/accept, deploy
claude_models.h/.cpp ClaudeModels: rawPredict, streamRawPredict, text_delta
open_models.h/.cpp OpenModels: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke
media.h/.cpp MediaGenerator: image, music, video start/status/wait
tools/check_discovery.py checks calls.h against Google's Discovery documents

Every call has a plan_* twin returning a PlannedCall (method, endpoint, path, body) without sending anything; tests and check_discovery.py verify requests that way.

3. Static caps (caps.h)

Contents per request 200; request text 1,000,000 chars; output tokens 1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072 dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude 200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16, 100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4, music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per result, operation name 512 chars; video polls 120 at 5 s (10 minutes).

4. Contracts

  • Every id (publisher, model, endpoint, invoke id) passes valid_segment before a path is built; operation names and publisherModel strings have their own character check. Bad input is GoogleError::invalid (400 INVALID_ARGUMENT) before any network traffic.
  • A call whose table row has express=false throws invalid in express mode, naming the missing project. No credential: NotConfiguredError (503).
  • Costly or consent calls are explicit and gated: ModelGarden::deploy needs DeployOptions.confirm, accept_eula needs confirm=true; both refuse otherwise. Nothing in this module deletes.
  • Streaming uses GoogleClient::stream_json (SSE via ?alt=sse for Gemini, Anthropic events for Claude); the sink returns false to stop.
  • Video: start_video (predictLongRunning), video_status (fetchPredictOperation, a POST), wait_video bounded by the caps. wait_operation (GET) is not used: Discovery has no GET on these operations for publisher models.
  • Example (/agent/models): GET page, /status, /calls need no credential; every Google call is a POST that answers 503 google_not_configured without a credential (checked first), 422 on bad input. Routes: generate, count, embed, claude, list, endpoint, image, music, video, video/status.

5. Verified against discovery

python3 -I src/google/models/tools/check_discovery.py fetches https://aiplatform.googleapis.com/$discovery/rest?version=v1 and v1beta1 and fails when any row of calls.h has a missing method or path, a wrong rpc id, an absent request body field (dotted names follow the schema) or an absent query parameter. Result on 2026-10-24: 29 calls checked, 0 problems. The same pass shows what Discovery says about express forms: generateContent, streamGenerateContent, countTokens, computeTokens, predict, predictLongRunning, fetchPredictOperation exist without a project; embedContent, rawPredict, streamRawPredict, publisher invoke do not (embedContent express answers 404 live). publishers.models.list exists only in v1beta1; publishers.models.get in v1 and v1beta1.

Request construction is compared to the table in tests/src/google/models/test_models.cpp (every body key must be a declared field).

6. Verified live

Run 2026-10-24 through this module's C++ client (MOLO_LIVE_NET=1 build/google_test, tests models_live.*), read-only or tiny (20 tokens). Credentials never printed or stored.

With an express API key (AQ.):

  • Gemini::generate, stream (SSE), count_tokens, compute_tokens, embed (text-embedding-005 predict) on aiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash: 200.
  • Project scoped countTokens at location global: 200.
  • Refused with 401 "API keys are not supported": Model Garden list/get, endpoints list, rawPredict, project embedContent. Express embedContent: 404 (matches Discovery).

With the service account OAuth token (project lisa-448004):

  • v1beta1 publishers/google/models list (google, anthropic), v1 publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash} get incl. view=PUBLISHER_MODEL_VIEW_FULL: 200.
  • Endpoints list (us-central1): 200, empty.
  • embedContent project scoped us-central1 gemini-embedding-001: 200 (global and text-embedding-005 not available there: 404/400).
  • Project scoped generateContent and computeTokens at global: 200 (us-central1 + gemini-2.5-flash: 404 "not found or no access").
  • Claude rawPredict at global (claude-sonnet-4-6): reached, 429 RESOURCE_EXHAUSTED (project quota); us-east5: 404 no access. Route, auth and body accepted; no completion was obtained.
  • fetchPublisherModelConfig: 404 "PublisherModelConfig not found" (route exists, nothing configured). EULA check for publishers/google/models/gemma3: 404 model not found (route exists).
  • Generation calls may answer 429 on the shared project; the live test notes that instead of failing.

Not live-verified (mutating or billable): deploy, EULA accept, image, music and video generation, endpoint predict family (no endpoint exists), Gemma shapes. These are verified by Discovery only.

7. Sources

  • Discovery documents: https://aiplatform.googleapis.com/$discovery/rest?version=v1 and ?version=v1beta1 (fetched 2026-10-24).
  • Claude on Vertex: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude (rawPredict/streamRawPredict paths, anthropic_version: vertex-2023-10-16, messages/max_tokens/stream/thinking, response shape).
  • Reference pages read for semantics: .../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent}, .../reference/rest/v1/publishers.models/get (view enum, languageCode).

8. Unverified (neither Discovery nor docs read, nor live)

  • Imagen predict instance/parameter names (prompt; sampleCount, aspectRatio, negativePrompt) and the predictions[].bytesBase64Encoded / mimeType response: Discovery types them as free-form Value; the doc pages could not be read.
  • Lyria shape (prompt, negative_prompt, seed; sample_count).
  • Veo shape (sampleCount, durationSeconds, aspectRatio, storageUri; final response.videos[] with gcsUri / bytesBase64Encoded / mimeType). fetchPredictOperation itself is Discovery-verified and reachable live.
  • Gemma/vLLM instance shape in gemma_instance (prompt, max_tokens, temperature): the serving container's, not Google's API.
  • Anthropic streaming event names (content_block_delta / text_delta) come from Anthropic's format; Claude completions were not obtained live.
  • Embedding predict response embeddings.values / statistics.token_count was observed live for text-embedding-005, not documented-read.
  • deployPublisherModel field semantics beyond the schema names.

9. Gaps

  • Discovery methods deliberately not wrapped (owner: this module): publisher export, setPublisherModelConfig, enableModel, IAM policy, v1/responses on publishers, publisher-level serverStreamingPredict and v1beta1 publishers.models.* project-less duplicates.
  • Endpoint generateContent family is wired but only Discovery-verified.
  • VertexModel has no retry; quota 429 surfaces as GoogleError.
  • Token refresh is the caller's job (foundation gap).