feat(clients): models: Gemini on Vertex, Model Garden, Claude, endpoint models, media, VertexModel

One call table verified against Discovery (29 calls), live-verified read-only
with an express key and a service account token, /agent/models example routes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-24 04:40:00 +02:00
co-authored by Claude Sonnet 5.5
parent 6d744a433f
commit 6b731643bf
30 changed files with 3200 additions and 3 deletions
+163
View File
@@ -0,0 +1,163 @@
# Models (Gemini Enterprise Agent Platform) - Specification
Owned directory: `src/google/models/`. Namespace `molo::google::models`.
Depends on the foundation in `src/google/` (`GoogleClient`, `Service`,
`GoogleError`, `CloudContext`, `Operation`, `gemini_types.h`). Voice and
`Generative` are untouched.
## 1. Scope
Gemini on Vertex AI (generate, stream, count tokens, compute tokens,
embeddings), Model Garden (list, get, publisher model config, EULA, deploy),
partner models (Claude rawPredict and streamRawPredict), endpoint models
(Gemma and any deployed model: list, get, predict family), media generation
(image, music, video via predict and predictLongRunning), and
`VertexModel : LanguageModel`.
Two addressing modes, one class (`VertexTarget`):
- express (API key, no project): `https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent`
- project scoped: `https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:...`
(`global` uses the host without a prefix).
## 2. Module map (one concern per file)
| File | Concern |
|---|---|
| `caps.h` | Static ceilings |
| `calls.h` | THE call table: every call, its Discovery rpc id and request fields |
| `target.h/.cpp` | `VertexTarget`: express vs project, id validation |
| `planner.h/.cpp` | `CallPlanner`: table row + ids -> exact request (pure, no I/O) |
| `vertex_service.h/.cpp` | `VertexService : Service`: runs planned calls |
| `json_read.h` | tolerant response readers |
| `gemini.h/.cpp` | `Gemini`: generate, stream (`?alt=sse`), count/compute tokens, embedContent, predict embeddings |
| `vertex_model.h/.cpp` | `VertexModel : LanguageModel` |
| `model_garden.h/.cpp` | `ModelGarden`: list, get, fetch config, EULA check/accept, deploy |
| `claude_models.h/.cpp` | `ClaudeModels`: rawPredict, streamRawPredict, `text_delta` |
| `open_models.h/.cpp` | `OpenModels`: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke |
| `media.h/.cpp` | `MediaGenerator`: image, music, video start/status/wait |
| `tools/check_discovery.py` | checks `calls.h` against Google's Discovery documents |
Every call has a `plan_*` twin returning a `PlannedCall` (method, endpoint,
path, body) without sending anything; tests and `check_discovery.py` verify
requests that way.
## 3. Static caps (`caps.h`)
Contents per request 200; request text 1,000,000 chars; output tokens
1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072
dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude
200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16,
100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4,
music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per
result, operation name 512 chars; video polls 120 at 5 s (10 minutes).
## 4. Contracts
- Every id (publisher, model, endpoint, invoke id) passes `valid_segment`
before a path is built; operation names and `publisherModel` strings have
their own character check. Bad input is `GoogleError::invalid` (400
`INVALID_ARGUMENT`) before any network traffic.
- A call whose table row has `express=false` throws invalid in express mode,
naming the missing project. No credential: `NotConfiguredError` (503).
- Costly or consent calls are explicit and gated: `ModelGarden::deploy`
needs `DeployOptions.confirm`, `accept_eula` needs `confirm=true`; both
refuse otherwise. Nothing in this module deletes.
- Streaming uses `GoogleClient::stream_json` (SSE via `?alt=sse` for Gemini,
Anthropic events for Claude); the sink returns false to stop.
- Video: `start_video` (predictLongRunning), `video_status`
(fetchPredictOperation, a POST), `wait_video` bounded by the caps.
`wait_operation` (GET) is not used: Discovery has no GET on these
operations for publisher models.
- Example (`/agent/models`): GET page, `/status`, `/calls` need no
credential; every Google call is a POST that answers 503
`google_not_configured` without a credential (checked first), 422 on bad
input. Routes: generate, count, embed, claude, list, endpoint, image,
music, video, video/status.
## 5. Verified against discovery
`python3 -I src/google/models/tools/check_discovery.py` fetches
`https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `v1beta1`
and fails when any row of `calls.h` has a missing method or path, a wrong
rpc id, an absent request body field (dotted names follow the schema) or an
absent query parameter. Result on 2026-10-24: `29 calls checked, 0 problems`.
The same pass shows what Discovery says about express forms:
generateContent, streamGenerateContent, countTokens, computeTokens, predict,
predictLongRunning, fetchPredictOperation exist without a project;
embedContent, rawPredict, streamRawPredict, publisher invoke do not
(`embedContent` express answers 404 live). `publishers.models.list` exists
only in v1beta1; `publishers.models.get` in v1 and v1beta1.
Request construction is compared to the table in
`tests/src/google/models/test_models.cpp` (every body key must be a declared
field).
## 6. Verified live
Run 2026-10-24 through this module's C++ client
(`MOLO_LIVE_NET=1 build/google_test`, tests `models_live.*`), read-only or
tiny (20 tokens). Credentials never printed or stored.
With an express API key (`AQ.`):
- `Gemini::generate`, `stream` (SSE), `count_tokens`, `compute_tokens`,
`embed` (text-embedding-005 predict) on `aiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash`: 200.
- Project scoped `countTokens` at location `global`: 200.
- Refused with 401 "API keys are not supported": Model Garden list/get,
endpoints list, rawPredict, project embedContent. Express `embedContent`:
404 (matches Discovery).
With the service account OAuth token (project `lisa-448004`):
- v1beta1 `publishers/google/models` list (google, anthropic), v1
`publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash}` get incl.
`view=PUBLISHER_MODEL_VIEW_FULL`: 200.
- Endpoints list (`us-central1`): 200, empty.
- `embedContent` project scoped `us-central1` `gemini-embedding-001`: 200
(global and text-embedding-005 not available there: 404/400).
- Project scoped `generateContent` and `computeTokens` at `global`: 200
(`us-central1` + gemini-2.5-flash: 404 "not found or no access").
- Claude `rawPredict` at `global` (claude-sonnet-4-6): reached, 429
RESOURCE_EXHAUSTED (project quota); `us-east5`: 404 no access. Route, auth
and body accepted; no completion was obtained.
- `fetchPublisherModelConfig`: 404 "PublisherModelConfig not found" (route
exists, nothing configured). EULA check for `publishers/google/models/gemma3`:
404 model not found (route exists).
- Generation calls may answer 429 on the shared project; the live test notes
that instead of failing.
Not live-verified (mutating or billable): deploy, EULA accept, image, music
and video generation, endpoint predict family (no endpoint exists), Gemma
shapes. These are verified by Discovery only.
## 7. Sources
- Discovery documents: `https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `?version=v1beta1` (fetched 2026-10-24).
- Claude on Vertex: `https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude` (rawPredict/streamRawPredict paths, `anthropic_version: vertex-2023-10-16`, messages/max_tokens/stream/thinking, response shape).
- Reference pages read for semantics: `.../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent}`, `.../reference/rest/v1/publishers.models/get` (view enum, languageCode).
## 8. Unverified (neither Discovery nor docs read, nor live)
- Imagen `predict` instance/parameter names (`prompt`; `sampleCount`,
`aspectRatio`, `negativePrompt`) and the `predictions[].bytesBase64Encoded`
/ `mimeType` response: Discovery types them as free-form `Value`; the doc
pages could not be read.
- Lyria shape (`prompt`, `negative_prompt`, `seed`; `sample_count`).
- Veo shape (`sampleCount`, `durationSeconds`, `aspectRatio`, `storageUri`;
final `response.videos[]` with `gcsUri` / `bytesBase64Encoded` / `mimeType`).
`fetchPredictOperation` itself is Discovery-verified and reachable live.
- Gemma/vLLM instance shape in `gemma_instance` (`prompt`, `max_tokens`,
`temperature`): the serving container's, not Google's API.
- Anthropic streaming event names (`content_block_delta` / `text_delta`)
come from Anthropic's format; Claude completions were not obtained live.
- Embedding predict response `embeddings.values` / `statistics.token_count`
was observed live for text-embedding-005, not documented-read.
- `deployPublisherModel` field semantics beyond the schema names.
## 9. Gaps
- Discovery methods deliberately not wrapped (owner: this module): publisher
`export`, `setPublisherModelConfig`, `enableModel`, IAM policy, `v1/responses`
on publishers, publisher-level `serverStreamingPredict` and v1beta1
`publishers.models.*` project-less duplicates.
- Endpoint `generateContent` family is wired but only Discovery-verified.
- `VertexModel` has no retry; quota 429 surfaces as `GoogleError`.
- Token refresh is the caller's job (foundation gap).
+75
View File
@@ -0,0 +1,75 @@
#pragma once
// The ONE table of every Google call this module makes. Each row is checked
// against Google's Discovery document by tools/check_discovery.py (see
// SPEC.md "Verified against discovery"). Keep one row per line, in the shape
// {"id", "HTTP", "version", "prefix", "suffix", express, "rpc", "fields"},
// the checker parses exactly this shape.
//
// prefix "L": /projects/{project}/locations/{location} + suffix
// "R": /projects/{project} + suffix
// "" : suffix alone (no project)
// express true: the unprefixed form (suffix alone) also exists in Discovery
// and works with an express API key.
// rpc Discovery method id of the prefixed (or only) form.
// fields space separated: request body field names (dotted: nested via the
// schema, "generationConfig.topP"), "?name" for a query parameter,
// "-" for none (raw HTTP body).
#include <cstddef>
namespace molo::google::models {
struct CallSpec {
const char* id;
const char* http;
const char* version;
const char* prefix;
const char* suffix;
bool express;
const char* rpc;
const char* fields;
};
// clang-format off
inline constexpr CallSpec kCalls[] = {
// Gemini on Vertex AI
{"gemini.generate", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:generateContent", true, "aiplatform.projects.locations.publishers.models.generateContent", "contents systemInstruction tools toolConfig safetySettings generationConfig generationConfig.temperature generationConfig.topP generationConfig.topK generationConfig.maxOutputTokens generationConfig.candidateCount generationConfig.stopSequences generationConfig.responseModalities generationConfig.responseMimeType generationConfig.responseSchema generationConfig.speechConfig generationConfig.thinkingConfig"},
{"gemini.stream", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:streamGenerateContent", true, "aiplatform.projects.locations.publishers.models.streamGenerateContent", "contents systemInstruction tools toolConfig safetySettings generationConfig"},
{"gemini.count_tokens", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:countTokens", true, "aiplatform.projects.locations.publishers.models.countTokens", "contents systemInstruction tools generationConfig"},
{"gemini.compute_tokens", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:computeTokens", true, "aiplatform.projects.locations.publishers.models.computeTokens", "contents"},
{"gemini.embed_content", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:embedContent", false, "aiplatform.projects.locations.publishers.models.embedContent", "content embedContentConfig embedContentConfig.taskType embedContentConfig.title embedContentConfig.outputDimensionality embedContentConfig.autoTruncate"},
{"gemini.embed_predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:predict", true, "aiplatform.projects.locations.publishers.models.predict", "instances parameters"},
// Partner models (Claude): raw JSON body in the model's own format
{"partner.raw_predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:rawPredict", false, "aiplatform.projects.locations.publishers.models.rawPredict", "-"},
{"partner.stream_raw_predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:streamRawPredict", false, "aiplatform.projects.locations.publishers.models.streamRawPredict", "-"},
{"publisher.invoke", "POST", "v1", "L", "/publishers/{publisher}/models/{model}/invoke/{invoke}", false, "aiplatform.projects.locations.publishers.models.invoke.invoke", "-"},
// Media generation
{"media.predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:predict", true, "aiplatform.projects.locations.publishers.models.predict", "instances parameters"},
{"media.predict_long_running", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:predictLongRunning", true, "aiplatform.projects.locations.publishers.models.predictLongRunning", "instances parameters"},
{"media.fetch_predict_operation", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:fetchPredictOperation", true, "aiplatform.projects.locations.publishers.models.fetchPredictOperation", "operationName"},
// Model Garden
{"garden.list", "GET", "v1beta1", "", "/publishers/{publisher}/models", false, "aiplatform.publishers.models.list", "?pageSize ?pageToken ?filter ?view ?orderBy ?languageCode ?listAllVersions"},
{"garden.get", "GET", "v1", "", "/publishers/{publisher}/models/{model}", false, "aiplatform.publishers.models.get", "?view ?languageCode"},
{"garden.fetch_config", "GET", "v1beta1", "L", "/publishers/{publisher}/models/{model}:fetchPublisherModelConfig", false, "aiplatform.projects.locations.publishers.models.fetchPublisherModelConfig", "-"},
{"garden.eula_check", "POST", "v1beta1", "R", "/modelGardenEula:check", false, "aiplatform.projects.modelGardenEula.check", "publisherModel"},
{"garden.eula_accept", "POST", "v1beta1", "R", "/modelGardenEula:accept", false, "aiplatform.projects.modelGardenEula.accept", "publisherModel"},
{"garden.deploy", "POST", "v1beta1", "L", ":deployPublisherModel", false, "aiplatform.projects.locations.deployPublisherModel", "model modelDisplayName endpointDisplayName acceptEula"},
// Endpoints (open models such as Gemma, and any deployed model)
{"endpoint.list", "GET", "v1", "L", "/endpoints", false, "aiplatform.projects.locations.endpoints.list", "?pageSize ?pageToken ?filter ?orderBy ?readMask"},
{"endpoint.get", "GET", "v1", "L", "/endpoints/{endpoint}", false, "aiplatform.projects.locations.endpoints.get", "-"},
{"endpoint.predict", "POST", "v1", "L", "/endpoints/{endpoint}:predict", false, "aiplatform.projects.locations.endpoints.predict", "instances parameters"},
{"endpoint.raw_predict", "POST", "v1", "L", "/endpoints/{endpoint}:rawPredict", false, "aiplatform.projects.locations.endpoints.rawPredict", "-"},
{"endpoint.stream_raw_predict", "POST", "v1", "L", "/endpoints/{endpoint}:streamRawPredict", false, "aiplatform.projects.locations.endpoints.streamRawPredict", "-"},
{"endpoint.server_streaming_predict", "POST", "v1", "L", "/endpoints/{endpoint}:serverStreamingPredict", false, "aiplatform.projects.locations.endpoints.serverStreamingPredict", "inputs parameters"},
{"endpoint.generate_content", "POST", "v1", "L", "/endpoints/{endpoint}:generateContent", false, "aiplatform.projects.locations.endpoints.generateContent", "contents systemInstruction tools generationConfig"},
{"endpoint.stream_generate_content", "POST", "v1", "L", "/endpoints/{endpoint}:streamGenerateContent", false, "aiplatform.projects.locations.endpoints.streamGenerateContent", "contents systemInstruction tools generationConfig"},
{"endpoint.count_tokens", "POST", "v1", "L", "/endpoints/{endpoint}:countTokens", false, "aiplatform.projects.locations.endpoints.countTokens", "contents systemInstruction tools generationConfig"},
{"endpoint.predict_long_running", "POST", "v1", "L", "/endpoints/{endpoint}:predictLongRunning", false, "aiplatform.projects.locations.endpoints.predictLongRunning", "instances parameters"},
{"endpoint.fetch_predict_operation", "POST", "v1", "L", "/endpoints/{endpoint}:fetchPredictOperation", false, "aiplatform.projects.locations.endpoints.fetchPredictOperation", "operationName"},
};
// clang-format on
inline constexpr std::size_t kCallCount = sizeof(kCalls) / sizeof(kCalls[0]);
} // namespace molo::google::models
+45
View File
@@ -0,0 +1,45 @@
#pragma once
// Static ceilings for the models module. Never autotuned.
#include <chrono>
#include <cstddef>
namespace molo::google::models {
struct ModelsCaps {
// Gemini
static constexpr std::size_t kMaxContents = 200;
static constexpr std::size_t kMaxTextChars = 1000000; // all text parts of a request
static constexpr int kMaxOutputTokens = 65536;
static constexpr int kMaxCandidates = 8;
// Embeddings
static constexpr std::size_t kMaxEmbedBatch = 100;
static constexpr std::size_t kMaxEmbedChars = 100000; // per text
static constexpr int kMaxEmbedDimensions = 3072;
// Model Garden
static constexpr std::size_t kMaxListPages = 20;
static constexpr int kMaxPageSize = 100;
static constexpr std::size_t kMaxModelsReturned = 2000;
// Partner (Claude)
static constexpr std::size_t kMaxClaudeMessages = 200;
static constexpr int kMaxClaudeTokens = 32768;
static constexpr std::size_t kMaxClaudeChars = 1000000;
// Open models
static constexpr std::size_t kMaxInstances = 16;
static constexpr std::size_t kMaxInstanceChars = 100000;
static constexpr int kMaxGemmaTokens = 8192;
// Media
static constexpr std::size_t kMaxPromptChars = 4000;
static constexpr int kMaxImages = 4;
static constexpr int kMaxMusicSamples = 4;
static constexpr int kMaxVideoSamples = 4;
static constexpr int kMaxVideoSeconds = 8;
static constexpr std::size_t kMaxMediaBytes = 32u * 1024u * 1024u; // per item
static constexpr std::size_t kMaxMediaItems = 8;
static constexpr std::size_t kMaxOperationName = 512;
static constexpr std::chrono::milliseconds kVideoPollInterval{5000};
static constexpr std::size_t kMaxVideoPolls = 120; // 10 minutes
};
} // namespace molo::google::models
+116
View File
@@ -0,0 +1,116 @@
#include "google/models/claude_models.h"
#include "google/errors.h"
#include "google/models/json_read.h"
#include "google/models/caps.h"
namespace molo::google::models {
using clients::body::Json;
Json ClaudeRequest::to_json(bool stream) const {
Json body = Json::object();
body.set("anthropic_version", Json(ClaudeModels::kAnthropicVersion));
Json msgs = Json::array();
for (const ClaudeMessage& m : messages) {
Json one = Json::object();
one.set("role", Json(m.role));
one.set("content", Json(m.text));
msgs.push(one);
}
body.set("messages", msgs);
body.set("max_tokens", Json(max_tokens));
if (!system.empty()) body.set("system", Json(system));
if (temperature) body.set("temperature", Json(*temperature));
if (thinking_budget > 0) {
Json t = Json::object();
t.set("type", Json("enabled"));
t.set("budget_tokens", Json(thinking_budget));
body.set("thinking", t);
}
if (tools.is_array()) body.set("tools", tools);
if (stream) body.set("stream", Json(true));
return body;
}
ClaudeResponse ClaudeResponse::from_json(const Json& doc) {
ClaudeResponse r;
r.id = json_str(doc, "id");
r.model = json_str(doc, "model");
r.stop_reason = json_str(doc, "stop_reason");
if (const Json* c = doc.find("content"); c && c->is_array()) {
r.content = *c;
for (const Json& block : c->as_array())
if (json_str(block, "type") == "text")
r.text += json_str(block, "text");
}
if (const Json* u = doc.find("usage"); u && u->is_object()) {
r.input_tokens = static_cast<int>(json_num(*u, "input_tokens"));
r.output_tokens = static_cast<int>(json_num(*u, "output_tokens"));
}
return r;
}
ClaudeModels::ClaudeModels(std::shared_ptr<GoogleClient> client, VertexTarget target)
: VertexService(std::move(client), std::move(target), "aiplatform") {}
void ClaudeModels::validate(const ClaudeRequest& r) {
VertexTarget::require_segment(r.model, "model");
if (r.messages.empty()) throw GoogleError::invalid("no messages");
if (r.messages.size() > ModelsCaps::kMaxClaudeMessages)
throw GoogleError::invalid("more than " +
std::to_string(ModelsCaps::kMaxClaudeMessages) +
" messages");
if (r.messages.front().role != "user")
throw GoogleError::invalid("the first message must be from the user");
std::size_t chars = r.system.size();
for (const ClaudeMessage& m : r.messages) {
if (m.role != "user" && m.role != "assistant")
throw GoogleError::invalid("role must be user or assistant");
if (m.text.empty()) throw GoogleError::invalid("message text is empty");
chars += m.text.size();
}
if (chars > ModelsCaps::kMaxClaudeChars)
throw GoogleError::invalid("request text over " +
std::to_string(ModelsCaps::kMaxClaudeChars) +
" characters");
if (r.max_tokens < 1 || r.max_tokens > ModelsCaps::kMaxClaudeTokens)
throw GoogleError::invalid("max_tokens outside 1.." +
std::to_string(ModelsCaps::kMaxClaudeTokens));
if (r.temperature && (*r.temperature < 0.0 || *r.temperature > 1.0))
throw GoogleError::invalid("temperature outside 0..1");
if (r.thinking_budget != 0 &&
(r.thinking_budget < 1024 || r.thinking_budget >= r.max_tokens))
throw GoogleError::invalid("thinking_budget must be >= 1024 and < max_tokens");
}
PlannedCall ClaudeModels::plan_generate(const ClaudeRequest& request) const {
validate(request);
Json body = request.to_json(false);
return planner_.plan("partner.raw_predict", model_args("anthropic", request.model), &body);
}
PlannedCall ClaudeModels::plan_stream(const ClaudeRequest& request) const {
validate(request);
Json body = request.to_json(true);
return planner_.plan("partner.stream_raw_predict", model_args("anthropic", request.model), &body);
}
ClaudeResponse ClaudeModels::generate(const ClaudeRequest& request) const {
return ClaudeResponse::from_json(run(plan_generate(request)));
}
void ClaudeModels::stream(
const ClaudeRequest& request,
const std::function<bool(const Json&)>& sink) const {
run_stream(plan_stream(request), sink);
}
std::string ClaudeModels::text_delta(const Json& event) {
if (json_str(event, "type") != "content_block_delta") return "";
const Json* d = event.find("delta");
if (!d || !d->is_object() || json_str(*d, "type") != "text_delta") return "";
return json_str(*d, "text");
}
} // namespace molo::google::models
+63
View File
@@ -0,0 +1,63 @@
#pragma once
// Partner models: Claude on Vertex AI through publishers/anthropic
// rawPredict and streamRawPredict. Project scoped only (Discovery has no
// express form for the raw predict methods).
#include <functional>
#include <optional>
#include <string>
#include <vector>
#include "google/models/vertex_service.h"
namespace molo::google::models {
struct ClaudeMessage {
std::string role; // "user" | "assistant"
std::string text;
};
struct ClaudeRequest {
std::string model; // e.g. claude-sonnet-4-6
std::string system;
std::vector<ClaudeMessage> messages;
int max_tokens = 1024;
std::optional<double> temperature;
int thinking_budget = 0; // 0: off; else >= 1024 and < max_tokens
clients::body::Json tools; // Anthropic tools array; null: none
clients::body::Json to_json(bool stream) const;
};
struct ClaudeResponse {
std::string id;
std::string model;
std::string stop_reason;
std::string text; // concatenated text blocks
int input_tokens = 0;
int output_tokens = 0;
clients::body::Json content; // raw content blocks (tool_use included)
static ClaudeResponse from_json(const clients::body::Json& doc);
};
class ClaudeModels : public VertexService {
public:
static constexpr const char* kAnthropicVersion = "vertex-2023-10-16";
ClaudeModels(std::shared_ptr<GoogleClient> client, VertexTarget target);
ClaudeResponse generate(const ClaudeRequest& request) const;
// Calls sink per Anthropic stream event (message_start, content_block_delta,
// ...); return false to stop.
void stream(const ClaudeRequest& request,
const std::function<bool(const clients::body::Json&)>& sink) const;
PlannedCall plan_generate(const ClaudeRequest& request) const;
PlannedCall plan_stream(const ClaudeRequest& request) const;
// Text of a content_block_delta / text_delta event, else "".
static std::string text_delta(const clients::body::Json& event);
static void validate(const ClaudeRequest& request);
};
} // namespace molo::google::models
+252
View File
@@ -0,0 +1,252 @@
#include "google/models/gemini.h"
#include "google/errors.h"
#include "google/models/caps.h"
#include "security/crypto/Base64.h"
namespace molo::google::models {
using clients::body::Json;
namespace {
bool token_chars(const std::string& s) {
if (s.size() > 64) return false;
for (char c : s)
if (!((c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '_'))
return false;
return true;
}
Embedding parse_vector(const Json& holder, const Json* stats) {
Embedding e;
if (const Json* v = holder.find("values"); v && v->is_array())
for (const Json& x : v->as_array())
if (x.is_number()) e.values.push_back(x.as_number());
if (stats && stats->is_object()) {
e.token_count = static_cast<int>(json_num(*stats, "token_count"));
if (const Json* t = stats->find("truncated"); t && t->is_bool())
e.truncated = t->as_bool();
}
return e;
}
const char* kPublisher = "google";
} // namespace
Gemini::Gemini(std::shared_ptr<GoogleClient> client, VertexTarget target)
: VertexService(std::move(client), std::move(target), "aiplatform") {}
void Gemini::validate(const GenerateRequest& request) {
if (request.contents.empty()) throw GoogleError::invalid("no contents");
if (request.contents.size() > ModelsCaps::kMaxContents)
throw GoogleError::invalid("more than " +
std::to_string(ModelsCaps::kMaxContents) +
" contents");
std::size_t chars = 0;
for (const Content& c : request.contents) {
if (c.parts.empty()) throw GoogleError::invalid("a content has no parts");
for (const Part& p : c.parts) {
chars += p.text.size();
if (p.text.empty() && !p.inline_data && !p.file_data && !p.function_call &&
!p.function_response)
throw GoogleError::invalid("a part is empty");
}
}
if (request.system)
for (const Part& p : request.system->parts) chars += p.text.size();
if (chars > ModelsCaps::kMaxTextChars)
throw GoogleError::invalid("request text over " +
std::to_string(ModelsCaps::kMaxTextChars) +
" characters");
const auto& cfg = request.config;
if (cfg.max_output_tokens &&
(*cfg.max_output_tokens < 1 || *cfg.max_output_tokens > ModelsCaps::kMaxOutputTokens))
throw GoogleError::invalid("max_output_tokens outside 1.." +
std::to_string(ModelsCaps::kMaxOutputTokens));
if (cfg.candidate_count &&
(*cfg.candidate_count < 1 || *cfg.candidate_count > ModelsCaps::kMaxCandidates))
throw GoogleError::invalid("candidate_count outside 1.." +
std::to_string(ModelsCaps::kMaxCandidates));
}
PlannedCall Gemini::plan_generate(const std::string& model,
const GenerateRequest& r) const {
validate(r);
Json body = r.to_json();
return planner_.plan("gemini.generate", model_args(kPublisher, model), &body);
}
PlannedCall Gemini::plan_stream(const std::string& model,
const GenerateRequest& r) const {
validate(r);
Json body = r.to_json();
return planner_.plan("gemini.stream", model_args(kPublisher, model), &body,
"alt=sse");
}
PlannedCall Gemini::plan_count_tokens(const std::string& model,
const GenerateRequest& r) const {
validate(r);
Json full = r.to_json();
Json body = Json::object();
for (const char* key : {"contents", "systemInstruction", "tools", "generationConfig"})
if (const Json* v = full.find(key)) body.set(key, *v);
return planner_.plan("gemini.count_tokens", model_args(kPublisher, model), &body);
}
PlannedCall Gemini::plan_compute_tokens(const std::string& model,
const std::vector<Content>& contents) const {
if (contents.empty()) throw GoogleError::invalid("no contents");
if (contents.size() > ModelsCaps::kMaxContents)
throw GoogleError::invalid("too many contents");
Json arr = Json::array();
for (const Content& c : contents) arr.push(c.to_json());
Json body = Json::object();
body.set("contents", arr);
return planner_.plan("gemini.compute_tokens", model_args(kPublisher, model), &body);
}
GenerateResponse Gemini::generate(const std::string& model,
const GenerateRequest& request) const {
return GenerateResponse::from_json(run(plan_generate(model, request)));
}
void Gemini::stream(
const std::string& model, const GenerateRequest& request,
const std::function<bool(const GenerateResponse&)>& on_chunk) const {
run_stream(plan_stream(model, request), [&](const Json& event) {
return on_chunk(GenerateResponse::from_json(event));
});
}
CountTokensResult Gemini::count_tokens(const std::string& model,
const GenerateRequest& request) const {
Json reply = run(plan_count_tokens(model, request));
CountTokensResult out;
out.total_tokens = static_cast<int>(json_num(reply, "totalTokens"));
out.billable_characters = static_cast<int>(json_num(reply, "totalBillableCharacters"));
out.raw = reply;
return out;
}
std::vector<TokenInfo> Gemini::compute_tokens(
const std::string& model, const std::vector<Content>& contents) const {
Json reply = run(plan_compute_tokens(model, contents));
std::vector<TokenInfo> out;
const Json* infos = reply.find("tokensInfo");
if (!infos || !infos->is_array()) return out;
for (const Json& i : infos->as_array()) {
if (out.size() >= ModelsCaps::kMaxContents) break;
TokenInfo t;
t.role = json_str(i, "role");
if (const Json* toks = i.find("tokens"); toks && toks->is_array())
for (const Json& x : toks->as_array()) {
std::string raw;
if (x.is_string() && security::crypto::Base64::decode(x.as_string(), raw))
t.tokens.push_back(raw);
}
if (const Json* ids = i.find("tokenIds"); ids && ids->is_array())
for (const Json& x : ids->as_array())
if (x.is_string()) t.token_ids.push_back(x.as_string());
out.push_back(std::move(t));
}
return out;
}
void Gemini::validate_embed(const EmbedOptions& options) {
if (!token_chars(options.task_type))
throw GoogleError::invalid("task_type must be UPPER_SNAKE_CASE");
if (options.output_dimensionality < 0 ||
options.output_dimensionality > ModelsCaps::kMaxEmbedDimensions)
throw GoogleError::invalid("output_dimensionality outside 0.." +
std::to_string(ModelsCaps::kMaxEmbedDimensions));
if (options.title.size() > 1000) throw GoogleError::invalid("title over 1000 characters");
}
PlannedCall Gemini::plan_embed_content(const std::string& model,
const std::string& text,
const EmbedOptions& options) const {
validate_embed(options);
if (text.empty()) throw GoogleError::invalid("text is empty");
if (text.size() > ModelsCaps::kMaxEmbedChars)
throw GoogleError::invalid("text over " +
std::to_string(ModelsCaps::kMaxEmbedChars) +
" characters");
Content content;
content.parts.push_back(Part::of_text(text));
Json body = Json::object();
body.set("content", content.to_json());
Json config = Json::object();
if (!options.task_type.empty()) config.set("taskType", Json(options.task_type));
if (!options.title.empty()) config.set("title", Json(options.title));
if (options.output_dimensionality > 0)
config.set("outputDimensionality", Json(options.output_dimensionality));
if (options.auto_truncate) config.set("autoTruncate", Json(*options.auto_truncate));
if (!config.as_object().empty()) body.set("embedContentConfig", config);
return planner_.plan("gemini.embed_content", model_args(kPublisher, model), &body);
}
Embedding Gemini::embed_content(const std::string& model, const std::string& text,
const EmbedOptions& options) const {
Json reply = run(plan_embed_content(model, text, options));
Embedding e;
if (const Json* emb = reply.find("embedding"); emb && emb->is_object())
e = parse_vector(*emb, nullptr);
if (const Json* u = reply.find("usageMetadata"); u && u->is_object())
e.token_count = static_cast<int>(json_num(*u, "promptTokenCount"));
if (const Json* t = reply.find("truncated"); t && t->is_bool()) e.truncated = t->as_bool();
if (e.values.empty())
throw GoogleError(502, "EMPTY_RESPONSE", "model returned no embedding",
endpoint_.name);
return e;
}
PlannedCall Gemini::plan_embed(const std::string& model,
const std::vector<std::string>& texts,
const EmbedOptions& options) const {
validate_embed(options);
if (texts.empty()) throw GoogleError::invalid("no texts");
if (texts.size() > ModelsCaps::kMaxEmbedBatch)
throw GoogleError::invalid("more than " +
std::to_string(ModelsCaps::kMaxEmbedBatch) + " texts");
Json instances = Json::array();
for (const std::string& t : texts) {
if (t.empty()) throw GoogleError::invalid("text is empty");
if (t.size() > ModelsCaps::kMaxEmbedChars)
throw GoogleError::invalid("text over " +
std::to_string(ModelsCaps::kMaxEmbedChars) +
" characters");
Json one = Json::object();
one.set("content", Json(t));
if (!options.task_type.empty()) one.set("task_type", Json(options.task_type));
if (!options.title.empty()) one.set("title", Json(options.title));
instances.push(one);
}
Json params = Json::object();
if (options.output_dimensionality > 0)
params.set("outputDimensionality", Json(options.output_dimensionality));
if (options.auto_truncate) params.set("autoTruncate", Json(*options.auto_truncate));
Json body = Json::object();
body.set("instances", instances);
if (!params.as_object().empty()) body.set("parameters", params);
return planner_.plan("gemini.embed_predict", model_args(kPublisher, model), &body);
}
std::vector<Embedding> Gemini::embed(const std::string& model,
const std::vector<std::string>& texts,
const EmbedOptions& options) const {
Json reply = run(plan_embed(model, texts, options));
std::vector<Embedding> out;
if (const Json* preds = reply.find("predictions"); preds && preds->is_array())
for (const Json& p : preds->as_array())
if (const Json* emb = p.find("embeddings"); emb && emb->is_object())
out.push_back(parse_vector(*emb, emb->find("statistics")));
if (out.size() != texts.size())
throw GoogleError(502, "BAD_RESPONSE",
"embedding count does not match the request", endpoint_.name);
return out;
}
} // namespace molo::google::models
+81
View File
@@ -0,0 +1,81 @@
#pragma once
// Gemini on Vertex AI: generateContent, streamGenerateContent (?alt=sse),
// countTokens, computeTokens, embedContent (project only) and the predict
// based text embeddings. Every call has a plan_* twin that builds the exact
// request without sending it.
#include <functional>
#include <optional>
#include <string>
#include <vector>
#include "google/gemini_types.h"
#include "google/models/vertex_service.h"
namespace molo::google::models {
struct CountTokensResult {
int total_tokens = 0;
int billable_characters = 0;
clients::body::Json raw;
};
struct TokenInfo {
std::string role;
std::vector<std::string> tokens; // decoded token bytes
std::vector<std::string> token_ids;
};
struct EmbedOptions {
std::string task_type; // e.g. RETRIEVAL_DOCUMENT; empty: model default
std::string title;
int output_dimensionality = 0; // 0: model default
std::optional<bool> auto_truncate;
};
struct Embedding {
std::vector<double> values;
int token_count = 0;
bool truncated = false;
};
class Gemini : public VertexService {
public:
Gemini(std::shared_ptr<GoogleClient> client, VertexTarget target);
GenerateResponse generate(const std::string& model,
const GenerateRequest& request) const;
void stream(const std::string& model, const GenerateRequest& request,
const std::function<bool(const GenerateResponse&)>& on_chunk) const;
CountTokensResult count_tokens(const std::string& model,
const GenerateRequest& request) const;
std::vector<TokenInfo> compute_tokens(const std::string& model,
const std::vector<Content>& contents) const;
// embedContent: project scoped only (Discovery has no express form).
Embedding embed_content(const std::string& model, const std::string& text,
const EmbedOptions& options = {}) const;
// predict based text embeddings: a batch in, one embedding per text out.
std::vector<Embedding> embed(const std::string& model,
const std::vector<std::string>& texts,
const EmbedOptions& options = {}) const;
PlannedCall plan_generate(const std::string& model, const GenerateRequest& r) const;
PlannedCall plan_stream(const std::string& model, const GenerateRequest& r) const;
PlannedCall plan_count_tokens(const std::string& model, const GenerateRequest& r) const;
PlannedCall plan_compute_tokens(const std::string& model,
const std::vector<Content>& contents) const;
PlannedCall plan_embed_content(const std::string& model, const std::string& text,
const EmbedOptions& o) const;
PlannedCall plan_embed(const std::string& model,
const std::vector<std::string>& texts,
const EmbedOptions& o) const;
// Throws GoogleError::invalid on empty or oversized requests.
static void validate(const GenerateRequest& request);
private:
static void validate_embed(const EmbedOptions& options);
};
} // namespace molo::google::models
+24
View File
@@ -0,0 +1,24 @@
#pragma once
// Tolerant readers for response documents (free functions: Service's own are
// protected and structs need them too).
#include <string>
#include "clients/body/json.h"
namespace molo::google::models {
inline std::string json_str(const clients::body::Json& doc, const std::string& key,
const std::string& fallback = "") {
const auto* v = doc.is_object() ? doc.find(key) : nullptr;
return v && v->is_string() ? v->as_string() : fallback;
}
inline double json_num(const clients::body::Json& doc, const std::string& key,
double fallback = 0.0) {
const auto* v = doc.is_object() ? doc.find(key) : nullptr;
return v && v->is_number() ? v->as_number() : fallback;
}
} // namespace molo::google::models
+194
View File
@@ -0,0 +1,194 @@
#include "google/models/media.h"
#include <thread>
#include "google/errors.h"
#include "google/models/caps.h"
#include "google/models/open_models.h"
#include "security/crypto/Base64.h"
namespace molo::google::models {
using clients::body::Json;
namespace {
bool simple_token(const std::string& s, std::size_t max) {
if (s.size() > max) return false;
for (char c : s)
if (!((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') ||
c == ':' || c == '_' || c == '-' || c == '.'))
return false;
return true;
}
} // namespace
MediaGenerator::MediaGenerator(std::shared_ptr<GoogleClient> client,
VertexTarget target)
: VertexService(std::move(client), std::move(target), "aiplatform") {}
void MediaGenerator::check_prompt(const std::string& prompt) {
if (prompt.empty()) throw GoogleError::invalid("prompt is empty");
if (prompt.size() > ModelsCaps::kMaxPromptChars)
throw GoogleError::invalid("prompt over " +
std::to_string(ModelsCaps::kMaxPromptChars) +
" characters");
}
std::vector<MediaItem> MediaGenerator::items_from(const Json& list) {
std::vector<MediaItem> out;
if (!list.is_array()) return out;
for (const Json& e : list.as_array()) {
if (out.size() >= ModelsCaps::kMaxMediaItems) break;
MediaItem item;
item.mime = json_str(e, "mimeType");
item.uri = json_str(e, "gcsUri");
std::string b64 = json_str(e, "bytesBase64Encoded");
if (!b64.empty()) {
if (b64.size() > ModelsCaps::kMaxMediaBytes * 4 / 3 + 8)
throw GoogleError(502, "BAD_RESPONSE", "media item over the size cap");
if (!security::crypto::Base64::decode(b64, item.bytes))
throw GoogleError(502, "BAD_RESPONSE", "media item is not valid base64");
}
if (item.bytes.empty() && item.uri.empty()) continue;
out.push_back(std::move(item));
}
return out;
}
std::vector<MediaItem> MediaGenerator::run_predict(const PlannedCall& call) const {
Json reply = run(call);
const Json* preds = reply.find("predictions");
std::vector<MediaItem> out = items_from(preds ? *preds : Json());
if (out.empty())
throw GoogleError(502, "EMPTY_RESPONSE", "model returned no media", endpoint_.name);
return out;
}
PlannedCall MediaGenerator::plan_image(const std::string& model, const ImageRequest& r) const {
check_prompt(r.prompt);
if (r.count < 1 || r.count > ModelsCaps::kMaxImages)
throw GoogleError::invalid("count outside 1.." + std::to_string(ModelsCaps::kMaxImages));
if (!simple_token(r.aspect_ratio, 8)) throw GoogleError::invalid("bad aspect_ratio");
Json instance = Json::object();
instance.set("prompt", Json(r.prompt));
Json params = Json::object();
params.set("sampleCount", Json(r.count));
if (!r.aspect_ratio.empty()) params.set("aspectRatio", Json(r.aspect_ratio));
if (!r.negative_prompt.empty()) {
if (r.negative_prompt.size() > ModelsCaps::kMaxPromptChars)
throw GoogleError::invalid("negative_prompt too long");
params.set("negativePrompt", Json(r.negative_prompt));
}
Json instances = Json::array();
instances.push(instance);
Json body = Json::object();
body.set("instances", instances);
body.set("parameters", params);
return planner_.plan("media.predict", model_args("google", model), &body);
}
PlannedCall MediaGenerator::plan_music(const std::string& model, const MusicRequest& r) const {
check_prompt(r.prompt);
if (r.count < 1 || r.count > ModelsCaps::kMaxMusicSamples)
throw GoogleError::invalid("count outside 1.." + std::to_string(ModelsCaps::kMaxMusicSamples));
if (r.seed && r.count != 1)
throw GoogleError::invalid("seed and count > 1 cannot be combined");
if (r.negative_prompt.size() > ModelsCaps::kMaxPromptChars)
throw GoogleError::invalid("negative_prompt too long");
Json instance = Json::object();
instance.set("prompt", Json(r.prompt));
if (!r.negative_prompt.empty()) instance.set("negative_prompt", Json(r.negative_prompt));
if (r.seed) instance.set("seed", Json(*r.seed));
Json params = Json::object();
if (!r.seed) params.set("sample_count", Json(r.count));
Json instances = Json::array();
instances.push(instance);
Json body = Json::object();
body.set("instances", instances);
body.set("parameters", params);
return planner_.plan("media.predict", model_args("google", model), &body);
}
PlannedCall MediaGenerator::plan_start_video(const std::string& model, const VideoRequest& r) const {
check_prompt(r.prompt);
if (r.count < 1 || r.count > ModelsCaps::kMaxVideoSamples)
throw GoogleError::invalid("count outside 1.." + std::to_string(ModelsCaps::kMaxVideoSamples));
if (r.duration_seconds < 0 || r.duration_seconds > ModelsCaps::kMaxVideoSeconds)
throw GoogleError::invalid("duration_seconds outside 0.." +
std::to_string(ModelsCaps::kMaxVideoSeconds));
if (!simple_token(r.aspect_ratio, 8)) throw GoogleError::invalid("bad aspect_ratio");
if (!r.storage_uri.empty() &&
(r.storage_uri.rfind("gs://", 0) != 0 || r.storage_uri.size() > 512))
throw GoogleError::invalid("storage_uri must be a gs:// uri");
Json instance = Json::object();
instance.set("prompt", Json(r.prompt));
Json instances = Json::array();
instances.push(instance);
Json params = Json::object();
params.set("sampleCount", Json(r.count));
if (r.duration_seconds > 0) params.set("durationSeconds", Json(r.duration_seconds));
if (!r.aspect_ratio.empty()) params.set("aspectRatio", Json(r.aspect_ratio));
if (!r.storage_uri.empty()) params.set("storageUri", Json(r.storage_uri));
Json body = Json::object();
body.set("instances", instances);
body.set("parameters", params);
return planner_.plan("media.predict_long_running", model_args("google", model), &body);
}
PlannedCall MediaGenerator::plan_video_status(const std::string& model,
const std::string& operation_name) const {
OpenModels::check_operation_name(operation_name);
Json body = Json::object();
body.set("operationName", Json(operation_name));
return planner_.plan("media.fetch_predict_operation", model_args("google", model), &body);
}
std::vector<MediaItem> MediaGenerator::generate_image(const std::string& model,
const ImageRequest& r) const {
return run_predict(plan_image(model, r));
}
std::vector<MediaItem> MediaGenerator::generate_music(const std::string& model,
const MusicRequest& r) const {
return run_predict(plan_music(model, r));
}
std::string MediaGenerator::start_video(const std::string& model, const VideoRequest& r) const {
std::string name = json_str(run(plan_start_video(model, r)), "name");
if (name.empty())
throw GoogleError(502, "BAD_RESPONSE", "no operation name returned", endpoint_.name);
return name;
}
Operation MediaGenerator::video_status(const std::string& model,
const std::string& operation_name) const {
return Operation::from_json(run(plan_video_status(model, operation_name)));
}
std::vector<MediaItem> MediaGenerator::wait_video(const std::string& model,
const std::string& operation_name,
const Sleeper& sleeper) const {
OpenModels::check_operation_name(operation_name);
Sleeper sleep = sleeper ? sleeper : [](std::chrono::milliseconds d) {
std::this_thread::sleep_for(d);
};
for (std::size_t i = 0; i < ModelsCaps::kMaxVideoPolls; ++i) {
Operation op = video_status(model, operation_name);
if (op.done) {
if (op.failed())
throw GoogleError(502, "OPERATION_FAILED", op.error_message, endpoint_.name);
const Json* videos = op.response.find("videos");
std::vector<MediaItem> out = items_from(videos ? *videos : Json());
if (out.empty())
throw GoogleError(502, "EMPTY_RESPONSE", "operation finished without videos",
endpoint_.name);
return out;
}
sleep(ModelsCaps::kVideoPollInterval);
}
throw GoogleError(504, "DEADLINE_EXCEEDED",
"video still rendering at the wait cap: " + operation_name,
endpoint_.name);
}
} // namespace molo::google::models
+78
View File
@@ -0,0 +1,78 @@
#pragma once
// Media generation on Vertex AI publisher models: images (Imagen, predict),
// music (Lyria, predict) and video (Veo, predictLongRunning plus
// fetchPredictOperation polling, bounded).
#include <optional>
#include <string>
#include <vector>
#include "google/operation.h"
#include "google/models/vertex_service.h"
namespace molo::google::models {
struct MediaItem {
std::string mime;
std::string bytes; // decoded; empty when the result lives at `uri`
std::string uri; // gs:// location when the request asked for storage
};
struct ImageRequest {
std::string prompt;
int count = 1; // 1..kMaxImages
std::string aspect_ratio; // "", "1:1", "16:9", ...
std::string negative_prompt;
};
struct MusicRequest {
std::string prompt;
std::string negative_prompt;
int count = 1; // 1..kMaxMusicSamples
std::optional<int> seed;
};
struct VideoRequest {
std::string prompt;
int count = 1; // 1..kMaxVideoSamples
int duration_seconds = 0; // 0: model default; else 1..kMaxVideoSeconds
std::string aspect_ratio; // "", "16:9", "9:16"
std::string storage_uri; // optional gs:// output prefix
};
class MediaGenerator : public VertexService {
public:
MediaGenerator(std::shared_ptr<GoogleClient> client, VertexTarget target);
std::vector<MediaItem> generate_image(const std::string& model,
const ImageRequest& request) const;
std::vector<MediaItem> generate_music(const std::string& model,
const MusicRequest& request) const;
// Starts a video job; returns the operation name.
std::string start_video(const std::string& model,
const VideoRequest& request) const;
// One poll (POST :fetchPredictOperation).
Operation video_status(const std::string& model,
const std::string& operation_name) const;
// Polls until done within kMaxVideoPolls. Throws DEADLINE_EXCEEDED at the
// cap and OPERATION_FAILED when the job failed. sleeper injectable.
std::vector<MediaItem> wait_video(const std::string& model,
const std::string& operation_name,
const Sleeper& sleeper = nullptr) const;
PlannedCall plan_image(const std::string& model, const ImageRequest& r) const;
PlannedCall plan_music(const std::string& model, const MusicRequest& r) const;
PlannedCall plan_start_video(const std::string& model, const VideoRequest& r) const;
PlannedCall plan_video_status(const std::string& model,
const std::string& operation_name) const;
static std::vector<MediaItem> items_from(const clients::body::Json& list);
private:
static void check_prompt(const std::string& prompt);
std::vector<MediaItem> run_predict(const PlannedCall& call) const;
};
} // namespace molo::google::models
+172
View File
@@ -0,0 +1,172 @@
#include "google/models/model_garden.h"
#include "clients/url.h"
#include "google/errors.h"
#include "google/models/caps.h"
#include "google/operation.h"
namespace molo::google::models {
using clients::body::Json;
std::string PublisherModel::publisher() const {
std::size_t a = name.find('/');
std::size_t b = a == std::string::npos ? a : name.find('/', a + 1);
return b == std::string::npos ? "" : name.substr(a + 1, b - a - 1);
}
std::string PublisherModel::model_id() const {
std::size_t p = name.rfind('/');
return p == std::string::npos ? name : name.substr(p + 1);
}
PublisherModel PublisherModel::from_json(const Json& doc) {
PublisherModel m;
m.name = json_str(doc, "name");
m.version_id = json_str(doc, "versionId");
m.open_source_category = json_str(doc, "openSourceCategory");
m.launch_stage = json_str(doc, "launchStage");
m.version_state = json_str(doc, "versionState");
m.publisher_model_template = json_str(doc, "publisherModelTemplate");
if (const Json* a = doc.find("supportedActions"); a && a->is_object())
for (const auto& [k, v] : a->as_object()) {
(void)v;
m.supported_actions.push_back(k);
}
m.raw = doc;
return m;
}
ModelGarden::ModelGarden(std::shared_ptr<GoogleClient> client, VertexTarget target)
: VertexService(std::move(client), std::move(target), "aiplatform") {}
namespace {
bool order_chars(const std::string& s) {
if (s.size() > 64) return false;
for (char c : s)
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
c == '_' || c == ' '))
return false;
return true;
}
} // namespace
PlannedCall ModelGarden::plan_list_page(const std::string& publisher,
const ListModelsOptions& o,
const std::string& page_token) const {
if (o.page_size < 1 || o.page_size > ModelsCaps::kMaxPageSize)
throw GoogleError::invalid("page_size outside 1.." +
std::to_string(ModelsCaps::kMaxPageSize));
if (o.max_pages < 1 || o.max_pages > ModelsCaps::kMaxListPages)
throw GoogleError::invalid("max_pages outside 1.." +
std::to_string(ModelsCaps::kMaxListPages));
if (o.filter.size() > 256) throw GoogleError::invalid("filter over 256 characters");
if (!order_chars(o.order_by)) throw GoogleError::invalid("bad order_by");
std::string q = "pageSize=" + std::to_string(o.page_size);
if (o.full_view) q += "&view=PUBLISHER_MODEL_VIEW_FULL";
if (o.all_versions) q += "&listAllVersions=true";
if (!o.filter.empty()) q += "&filter=" + clients::Url::encode_component(o.filter);
if (!o.order_by.empty()) q += "&orderBy=" + clients::Url::encode_component(o.order_by);
if (!page_token.empty()) q += "&pageToken=" + clients::Url::encode_component(page_token);
return planner_.plan("garden.list", {{"publisher", publisher}}, nullptr, q);
}
std::vector<PublisherModel> ModelGarden::list(const std::string& publisher,
const ListModelsOptions& o) const {
std::vector<PublisherModel> out;
std::string token;
for (std::size_t page = 0; page < o.max_pages; ++page) {
Json reply = run(plan_list_page(publisher, o, token));
if (const Json* items = reply.find("publisherModels"); items && items->is_array())
for (const Json& item : items->as_array()) {
if (out.size() >= ModelsCaps::kMaxModelsReturned) return out;
out.push_back(PublisherModel::from_json(item));
}
token = json_str(reply, "nextPageToken");
if (token.empty()) break;
}
return out;
}
PlannedCall ModelGarden::plan_get(const std::string& publisher,
const std::string& model, bool full_view) const {
return planner_.plan("garden.get", model_args(publisher, model), nullptr,
full_view ? "view=PUBLISHER_MODEL_VIEW_FULL" : "");
}
PublisherModel ModelGarden::get(const std::string& publisher,
const std::string& model, bool full_view) const {
return PublisherModel::from_json(run(plan_get(publisher, model, full_view)));
}
PlannedCall ModelGarden::plan_fetch_config(const std::string& publisher,
const std::string& model) const {
return planner_.plan("garden.fetch_config", model_args(publisher, model));
}
Json ModelGarden::fetch_config(const std::string& publisher,
const std::string& model) const {
return run(plan_fetch_config(publisher, model));
}
PlannedCall ModelGarden::plan_eula(const char* call_id,
const std::string& publisher_model) const {
// publishers/{publisher}/models/{model}[@version]
if (publisher_model.rfind("publishers/", 0) != 0 || publisher_model.size() > 256 ||
publisher_model.find("..") != std::string::npos)
throw GoogleError::invalid("publisher_model must look like publishers/{p}/models/{m}");
for (char c : publisher_model)
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
c == '/' || c == '-' || c == '_' || c == '.' || c == '@'))
throw GoogleError::invalid("publisher_model has invalid characters");
Json body = Json::object();
body.set("publisherModel", Json(publisher_model));
return planner_.plan(call_id, {}, &body);
}
EulaStatus ModelGarden::check_eula(const std::string& publisher_model) const {
Json r = run(plan_eula("garden.eula_check", publisher_model));
EulaStatus s;
if (const Json* a = r.find("publisherModelEulaAcked"); a && a->is_bool())
s.acked = a->as_bool();
s.project_number = json_str(r, "projectNumber");
s.publisher_model = json_str(r, "publisherModel");
return s;
}
EulaStatus ModelGarden::accept_eula(const std::string& publisher_model,
bool confirm) const {
if (!confirm) throw GoogleError::invalid("accept_eula needs confirm=true");
Json r = run(plan_eula("garden.eula_accept", publisher_model));
EulaStatus s;
if (const Json* a = r.find("publisherModelEulaAcked"); a && a->is_bool())
s.acked = a->as_bool();
s.project_number = json_str(r, "projectNumber");
s.publisher_model = json_str(r, "publisherModel");
return s;
}
PlannedCall ModelGarden::plan_deploy(const DeployOptions& o) const {
if (!o.confirm) throw GoogleError::invalid("deploy needs confirm=true (it creates billable resources)");
if (o.publisher_model.empty()) throw GoogleError::invalid("publisher_model is empty");
plan_eula("garden.eula_check", o.publisher_model); // same id validation
if (o.model_display_name.size() > 128 || o.endpoint_display_name.size() > 128)
throw GoogleError::invalid("display name over 128 characters");
Json body = Json::object();
body.set("model", Json(o.publisher_model));
if (!o.model_display_name.empty()) body.set("modelDisplayName", Json(o.model_display_name));
if (!o.endpoint_display_name.empty())
body.set("endpointDisplayName", Json(o.endpoint_display_name));
if (o.accept_eula) body.set("acceptEula", Json(true));
return planner_.plan("garden.deploy", {}, &body);
}
std::string ModelGarden::deploy(const DeployOptions& options) const {
Json r = run(plan_deploy(options));
std::string name = json_str(r, "name");
if (name.empty())
throw GoogleError(502, "BAD_RESPONSE", "no operation name returned", endpoint_.name);
return name;
}
} // namespace molo::google::models
+78
View File
@@ -0,0 +1,78 @@
#pragma once
// Model Garden: publisher models (list: v1beta1, get: v1), publisher model
// config, EULA check and accept, and deploy. Mutating and costly calls
// (accept_eula, deploy) refuse to run unless the caller sets confirm.
#include <string>
#include <vector>
#include "google/models/vertex_service.h"
namespace molo::google::models {
struct PublisherModel {
std::string name; // publishers/{publisher}/models/{model}
std::string version_id;
std::string open_source_category;
std::string launch_stage;
std::string version_state;
std::string publisher_model_template;
std::vector<std::string> supported_actions; // keys of supportedActions
clients::body::Json raw;
std::string publisher() const;
std::string model_id() const;
static PublisherModel from_json(const clients::body::Json& doc);
};
struct ListModelsOptions {
int page_size = 50; // 1..ModelsCaps::kMaxPageSize
std::size_t max_pages = 5; // 1..ModelsCaps::kMaxListPages
std::string filter; // passed through, URL encoded, max 256 chars
std::string order_by; // max 64 chars, [A-Za-z0-9_ ]
bool full_view = false;
bool all_versions = false;
};
struct EulaStatus {
bool acked = false;
std::string project_number;
std::string publisher_model;
};
struct DeployOptions {
std::string publisher_model; // publishers/{publisher}/models/{model}@version
std::string model_display_name;
std::string endpoint_display_name;
bool accept_eula = false;
bool confirm = false; // must be true: deploy creates billable resources
};
class ModelGarden : public VertexService {
public:
ModelGarden(std::shared_ptr<GoogleClient> client, VertexTarget target);
std::vector<PublisherModel> list(const std::string& publisher,
const ListModelsOptions& options = {}) const;
PublisherModel get(const std::string& publisher, const std::string& model,
bool full_view = false) const;
clients::body::Json fetch_config(const std::string& publisher,
const std::string& model) const;
EulaStatus check_eula(const std::string& publisher_model) const;
EulaStatus accept_eula(const std::string& publisher_model, bool confirm) const;
// Returns the long-running operation name; poll it with wait_operation.
std::string deploy(const DeployOptions& options) const;
PlannedCall plan_list_page(const std::string& publisher,
const ListModelsOptions& options,
const std::string& page_token = "") const;
PlannedCall plan_get(const std::string& publisher, const std::string& model,
bool full_view) const;
PlannedCall plan_fetch_config(const std::string& publisher,
const std::string& model) const;
PlannedCall plan_eula(const char* call_id, const std::string& publisher_model) const;
PlannedCall plan_deploy(const DeployOptions& options) const;
};
} // namespace molo::google::models
+235
View File
@@ -0,0 +1,235 @@
#include "google/models/open_models.h"
#include "clients/url.h"
#include "google/errors.h"
#include "google/models/caps.h"
#include "google/models/gemini.h"
namespace molo::google::models {
using clients::body::Json;
EndpointInfo EndpointInfo::from_json(const Json& doc) {
EndpointInfo e;
e.name = json_str(doc, "name");
std::size_t p = e.name.rfind('/');
e.id = p == std::string::npos ? e.name : e.name.substr(p + 1);
e.display_name = json_str(doc, "displayName");
e.dedicated_endpoint_dns = json_str(doc, "dedicatedEndpointDns");
if (const Json* d = doc.find("deployedModels"); d && d->is_array())
e.deployed_models = d->as_array().size();
e.raw = doc;
return e;
}
OpenModels::OpenModels(std::shared_ptr<GoogleClient> client, VertexTarget target)
: VertexService(std::move(client), std::move(target), "aiplatform") {}
void OpenModels::check_operation_name(const std::string& name) {
if (name.empty() || name.size() > ModelsCaps::kMaxOperationName ||
name.find("..") != std::string::npos)
throw GoogleError::invalid("bad operation name");
for (char c : name)
if (!((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') ||
c == '/' || c == '_' || c == '-' || c == '.'))
throw GoogleError::invalid("bad operation name");
}
PlannedCall OpenModels::plan_list_page(std::size_t max_pages, const std::string& filter,
const std::string& page_token) const {
if (max_pages < 1 || max_pages > ModelsCaps::kMaxListPages)
throw GoogleError::invalid("max_pages outside 1.." +
std::to_string(ModelsCaps::kMaxListPages));
if (filter.size() > 256) throw GoogleError::invalid("filter over 256 characters");
std::string q = "pageSize=" + std::to_string(ModelsCaps::kMaxPageSize);
if (!filter.empty()) q += "&filter=" + clients::Url::encode_component(filter);
if (!page_token.empty()) q += "&pageToken=" + clients::Url::encode_component(page_token);
return planner_.plan("endpoint.list", {}, nullptr, q);
}
std::vector<EndpointInfo> OpenModels::list_endpoints(std::size_t max_pages,
const std::string& filter) const {
std::vector<EndpointInfo> out;
std::string token;
for (std::size_t page = 0; page < max_pages; ++page) {
Json reply = run(plan_list_page(max_pages, filter, token));
if (const Json* items = reply.find("endpoints"); items && items->is_array())
for (const Json& item : items->as_array()) {
if (out.size() >= ModelsCaps::kMaxModelsReturned) return out;
out.push_back(EndpointInfo::from_json(item));
}
token = json_str(reply, "nextPageToken");
if (token.empty()) break;
}
return out;
}
PlannedCall OpenModels::plan_get(const std::string& endpoint_id) const {
return planner_.plan("endpoint.get", {{"endpoint", endpoint_id}});
}
EndpointInfo OpenModels::get_endpoint(const std::string& endpoint_id) const {
return EndpointInfo::from_json(run(plan_get(endpoint_id)));
}
PlannedCall OpenModels::plan_predict(const std::string& endpoint_id,
const Json& instances,
const Json& parameters) const {
if (!instances.is_array() || instances.as_array().empty())
throw GoogleError::invalid("instances must be a non-empty array");
if (instances.as_array().size() > ModelsCaps::kMaxInstances)
throw GoogleError::invalid("more than " +
std::to_string(ModelsCaps::kMaxInstances) + " instances");
if (instances.dump().size() > ModelsCaps::kMaxInstanceChars)
throw GoogleError::invalid("instances over " +
std::to_string(ModelsCaps::kMaxInstanceChars) + " characters");
Json body = Json::object();
body.set("instances", instances);
if (parameters.is_object()) body.set("parameters", parameters);
return planner_.plan("endpoint.predict", {{"endpoint", endpoint_id}}, &body);
}
PredictResult OpenModels::predict(const std::string& endpoint_id,
const Json& instances,
const Json& parameters) const {
Json reply = run(plan_predict(endpoint_id, instances, parameters));
PredictResult r;
if (const Json* p = reply.find("predictions")) r.predictions = *p;
r.deployed_model_id = json_str(reply, "deployedModelId");
r.model = json_str(reply, "model");
r.model_display_name = json_str(reply, "modelDisplayName");
r.model_version_id = json_str(reply, "modelVersionId");
if (!r.predictions.is_array())
throw GoogleError(502, "BAD_RESPONSE", "endpoint returned no predictions",
endpoint_.name);
return r;
}
PlannedCall OpenModels::plan_raw(const char* call_id, const std::string& endpoint_id,
const Json& body) const {
if (!body.is_object()) throw GoogleError::invalid("body must be a JSON object");
if (body.dump().size() > ModelsCaps::kMaxInstanceChars)
throw GoogleError::invalid("body over " +
std::to_string(ModelsCaps::kMaxInstanceChars) + " characters");
return planner_.plan(call_id, {{"endpoint", endpoint_id}}, &body);
}
Json OpenModels::raw_predict(const std::string& endpoint_id, const Json& body) const {
return run(plan_raw("endpoint.raw_predict", endpoint_id, body));
}
void OpenModels::stream_raw_predict(
const std::string& endpoint_id, const Json& body,
const std::function<bool(const Json&)>& sink) const {
run_stream(plan_raw("endpoint.stream_raw_predict", endpoint_id, body), sink);
}
PlannedCall OpenModels::plan_server_streaming(const std::string& endpoint_id,
const Json& inputs,
const Json& parameters) const {
if (!inputs.is_array() || inputs.as_array().empty() ||
inputs.as_array().size() > ModelsCaps::kMaxInstances)
throw GoogleError::invalid("inputs must be an array of 1.." +
std::to_string(ModelsCaps::kMaxInstances) + " items");
if (inputs.dump().size() > ModelsCaps::kMaxInstanceChars)
throw GoogleError::invalid("inputs too large");
Json body = Json::object();
body.set("inputs", inputs);
if (parameters.is_object()) body.set("parameters", parameters);
return planner_.plan("endpoint.server_streaming_predict", {{"endpoint", endpoint_id}}, &body);
}
void OpenModels::server_streaming_predict(
const std::string& endpoint_id, const Json& inputs, const Json& parameters,
const std::function<bool(const Json&)>& sink) const {
run_stream(plan_server_streaming(endpoint_id, inputs, parameters), sink);
}
PlannedCall OpenModels::plan_generate(const char* call_id,
const std::string& endpoint_id,
const GenerateRequest& request) const {
Gemini::validate(request);
Json full = request.to_json();
Json body = Json::object();
// Discovery lists these four for the endpoint forms.
for (const char* key : {"contents", "systemInstruction", "tools", "generationConfig"})
if (const Json* v = full.find(key)) body.set(key, *v);
return planner_.plan(call_id, {{"endpoint", endpoint_id}}, &body,
std::string(call_id) == "endpoint.stream_generate_content" ? "alt=sse" : "");
}
GenerateResponse OpenModels::generate_content(const std::string& endpoint_id,
const GenerateRequest& request) const {
return GenerateResponse::from_json(
run(plan_generate("endpoint.generate_content", endpoint_id, request)));
}
void OpenModels::stream_generate_content(
const std::string& endpoint_id, const GenerateRequest& request,
const std::function<bool(const GenerateResponse&)>& on_chunk) const {
run_stream(plan_generate("endpoint.stream_generate_content", endpoint_id, request),
[&](const Json& e) { return on_chunk(GenerateResponse::from_json(e)); });
}
int OpenModels::count_tokens(const std::string& endpoint_id,
const GenerateRequest& request) const {
return static_cast<int>(json_num(
run(plan_generate("endpoint.count_tokens", endpoint_id, request)), "totalTokens"));
}
PlannedCall OpenModels::plan_long_running(const std::string& endpoint_id,
const Json& instances,
const Json& parameters) const {
PlannedCall p = plan_predict(endpoint_id, instances, parameters);
return planner_.plan("endpoint.predict_long_running", {{"endpoint", endpoint_id}}, &p.body);
}
std::string OpenModels::predict_long_running(const std::string& endpoint_id,
const Json& instances,
const Json& parameters) const {
Json r = run(plan_long_running(endpoint_id, instances, parameters));
std::string name = json_str(r, "name");
if (name.empty())
throw GoogleError(502, "BAD_RESPONSE", "no operation name returned", endpoint_.name);
return name;
}
PlannedCall OpenModels::plan_fetch_operation(const std::string& endpoint_id,
const std::string& operation_name) const {
check_operation_name(operation_name);
Json body = Json::object();
body.set("operationName", Json(operation_name));
return planner_.plan("endpoint.fetch_predict_operation", {{"endpoint", endpoint_id}}, &body);
}
Operation OpenModels::fetch_predict_operation(const std::string& endpoint_id,
const std::string& operation_name) const {
return Operation::from_json(run(plan_fetch_operation(endpoint_id, operation_name)));
}
Json OpenModels::invoke_publisher(const std::string& publisher, const std::string& model,
const std::string& invoke_id, const Json& body) const {
if (!body.is_object()) throw GoogleError::invalid("body must be a JSON object");
CallPlanner::Args args = model_args(publisher, model);
args["invoke"] = invoke_id;
return run(planner_.plan("publisher.invoke", args, &body));
}
Json OpenModels::gemma_instance(const std::string& prompt, int max_tokens,
double temperature) {
if (prompt.empty()) throw GoogleError::invalid("prompt is empty");
if (prompt.size() > ModelsCaps::kMaxInstanceChars)
throw GoogleError::invalid("prompt too long");
if (max_tokens < 1 || max_tokens > ModelsCaps::kMaxGemmaTokens)
throw GoogleError::invalid("max_tokens outside 1.." +
std::to_string(ModelsCaps::kMaxGemmaTokens));
if (temperature < 0.0 || temperature > 2.0)
throw GoogleError::invalid("temperature outside 0..2");
Json one = Json::object();
one.set("prompt", Json(prompt));
one.set("max_tokens", Json(max_tokens));
one.set("temperature", Json(temperature));
return one;
}
} // namespace molo::google::models
+101
View File
@@ -0,0 +1,101 @@
#pragma once
// Open models (Gemma and any self-deployed model) served from Vertex AI
// endpoints: list, get, predict, rawPredict, streamRawPredict,
// serverStreamingPredict, generateContent family, predictLongRunning and
// fetchPredictOperation. Project scoped. Also publishers invoke.
#include <functional>
#include <string>
#include <vector>
#include "google/gemini_types.h"
#include "google/models/vertex_service.h"
#include "google/operation.h"
namespace molo::google::models {
struct EndpointInfo {
std::string name; // projects/.../endpoints/{id}
std::string id;
std::string display_name;
std::string dedicated_endpoint_dns;
std::size_t deployed_models = 0;
clients::body::Json raw;
static EndpointInfo from_json(const clients::body::Json& doc);
};
struct PredictResult {
clients::body::Json predictions;
std::string deployed_model_id;
std::string model;
std::string model_display_name;
std::string model_version_id;
};
class OpenModels : public VertexService {
public:
OpenModels(std::shared_ptr<GoogleClient> client, VertexTarget target);
std::vector<EndpointInfo> list_endpoints(std::size_t max_pages = 5,
const std::string& filter = "") const;
EndpointInfo get_endpoint(const std::string& endpoint_id) const;
// instances: array of model specific inputs (see gemma_instance).
PredictResult predict(const std::string& endpoint_id,
const clients::body::Json& instances,
const clients::body::Json& parameters = {}) const;
// Body is the model server's own JSON (Anthropic, vLLM chat, ...).
clients::body::Json raw_predict(const std::string& endpoint_id,
const clients::body::Json& body) const;
void stream_raw_predict(const std::string& endpoint_id,
const clients::body::Json& body,
const std::function<bool(const clients::body::Json&)>& sink) const;
void server_streaming_predict(
const std::string& endpoint_id, const clients::body::Json& inputs,
const clients::body::Json& parameters,
const std::function<bool(const clients::body::Json&)>& sink) const;
GenerateResponse generate_content(const std::string& endpoint_id,
const GenerateRequest& request) const;
void stream_generate_content(
const std::string& endpoint_id, const GenerateRequest& request,
const std::function<bool(const GenerateResponse&)>& on_chunk) const;
int count_tokens(const std::string& endpoint_id, const GenerateRequest& request) const;
std::string predict_long_running(const std::string& endpoint_id,
const clients::body::Json& instances,
const clients::body::Json& parameters = {}) const;
Operation fetch_predict_operation(const std::string& endpoint_id,
const std::string& operation_name) const;
clients::body::Json invoke_publisher(const std::string& publisher,
const std::string& model,
const std::string& invoke_id,
const clients::body::Json& body) const;
PlannedCall plan_list_page(std::size_t max_pages, const std::string& filter,
const std::string& page_token) const;
PlannedCall plan_get(const std::string& endpoint_id) const;
PlannedCall plan_predict(const std::string& endpoint_id,
const clients::body::Json& instances,
const clients::body::Json& parameters) const;
PlannedCall plan_raw(const char* call_id, const std::string& endpoint_id,
const clients::body::Json& body) const;
PlannedCall plan_server_streaming(const std::string& endpoint_id,
const clients::body::Json& inputs,
const clients::body::Json& parameters) const;
PlannedCall plan_generate(const char* call_id, const std::string& endpoint_id,
const GenerateRequest& request) const;
PlannedCall plan_long_running(const std::string& endpoint_id,
const clients::body::Json& instances,
const clients::body::Json& parameters) const;
PlannedCall plan_fetch_operation(const std::string& endpoint_id,
const std::string& operation_name) const;
// The prompt instance Vertex's vLLM serving container accepts for Gemma
// (Unverified: see SPEC; the shape is the container's, not Discovery's).
static clients::body::Json gemma_instance(const std::string& prompt,
int max_tokens = 256,
double temperature = 0.7);
static void check_operation_name(const std::string& name);
};
} // namespace molo::google::models
+82
View File
@@ -0,0 +1,82 @@
#include "google/models/planner.h"
#include <cstring>
#include <stdexcept>
#include "google/errors.h"
namespace molo::google::models {
const CallSpec& CallPlanner::spec(const std::string& call_id) {
for (const CallSpec& s : kCalls)
if (call_id == s.id) return s;
throw std::logic_error("molo::google::models: unknown call id " + call_id);
}
namespace {
clients::Method method_of(const char* http) {
if (!std::strcmp(http, "GET")) return clients::Method::kGet;
if (!std::strcmp(http, "POST")) return clients::Method::kPost;
if (!std::strcmp(http, "PATCH")) return clients::Method::kPatch;
if (!std::strcmp(http, "DELETE")) return clients::Method::kDelete;
throw std::logic_error("molo::google::models: bad http verb in call table");
}
std::string fill(const std::string& tpl, const CallPlanner::Args& args) {
std::string out;
for (std::size_t i = 0; i < tpl.size();) {
if (tpl[i] != '{') {
out += tpl[i++];
continue;
}
std::size_t end = tpl.find('}', i);
std::string key = tpl.substr(i + 1, end - i - 1);
auto it = args.find(key);
if (it == args.end()) throw std::logic_error("missing path argument " + key);
VertexTarget::require_segment(it->second, key);
out += it->second;
i = end + 1;
}
return out;
}
} // namespace
PlannedCall CallPlanner::plan(const std::string& call_id, const Args& args,
const clients::body::Json* body,
const std::string& query) const {
const CallSpec& s = spec(call_id);
PlannedCall p;
p.spec = &s;
p.method = method_of(s.http);
const bool beta = std::strcmp(s.version, "v1beta1") == 0;
const std::string prefix = s.prefix;
std::string path;
if (prefix == "L") {
if (target_.is_express()) {
if (!s.express)
throw GoogleError::invalid(
std::string(call_id) +
" needs a project and an OAuth token (set GOOGLE_CLOUD_PROJECT)");
p.endpoint = beta ? target_.beta_endpoint() : target_.endpoint();
path = fill(s.suffix, args);
} else {
p.endpoint = beta ? target_.beta_endpoint() : target_.endpoint();
path = target_.parent_path() + fill(s.suffix, args);
}
} else if (prefix == "R") {
p.endpoint = beta ? endpoints::aiplatform_beta("global") : endpoints::aiplatform("global");
path = "/projects/" + target_.context().project + fill(s.suffix, args);
} else {
p.endpoint = beta ? endpoints::aiplatform_beta("global") : endpoints::aiplatform("global");
path = fill(s.suffix, args);
}
if (!query.empty()) path += "?" + query;
p.path = std::move(path);
if (body) {
p.body = *body;
p.has_body = true;
}
return p;
}
} // namespace molo::google::models
+49
View File
@@ -0,0 +1,49 @@
#pragma once
// CallPlanner turns a row of the call table (calls.h) plus ids into the exact
// request: method, endpoint, path, body. Pure, no I/O, so request building is
// testable without any Google reply. Every id is validated here.
#include <map>
#include <string>
#include "clients/body/json.h"
#include "clients/request.h"
#include "google/endpoints.h"
#include "google/models/calls.h"
#include "google/models/target.h"
namespace molo::google::models {
struct PlannedCall {
clients::Method method = clients::Method::kGet;
ServiceEndpoint endpoint;
std::string path; // starts with "/", may carry a query string
clients::body::Json body;
bool has_body = false;
const CallSpec* spec = nullptr;
std::string url() const { return endpoint.base_url + path; }
};
class CallPlanner {
public:
using Args = std::map<std::string, std::string>;
explicit CallPlanner(VertexTarget target) : target_(std::move(target)) {}
const VertexTarget& target() const { return target_; }
// Throws std::logic_error for an id missing from the table (a bug), and
// GoogleError::invalid for a bad id value or a project-only call in
// express mode. `query` is appended after "?" when non-empty.
PlannedCall plan(const std::string& call_id, const Args& args,
const clients::body::Json* body = nullptr,
const std::string& query = "") const;
static const CallSpec& spec(const std::string& call_id);
private:
VertexTarget target_;
};
} // namespace molo::google::models
+54
View File
@@ -0,0 +1,54 @@
#include "google/models/target.h"
#include "google/errors.h"
namespace molo::google::models {
VertexTarget VertexTarget::express() { return VertexTarget(); }
VertexTarget VertexTarget::project(CloudContext context) {
context.parent(); // validates project and location
VertexTarget t;
t.context_ = std::move(context);
return t;
}
VertexTarget VertexTarget::from_env() {
CloudContext c = CloudContext::from_env();
return c.project.empty() ? express() : project(std::move(c));
}
const CloudContext& VertexTarget::context() const {
if (!context_)
throw GoogleError::invalid(
"this call needs a project (set GOOGLE_CLOUD_PROJECT); express mode has none");
return *context_;
}
ServiceEndpoint VertexTarget::endpoint() const {
return endpoints::aiplatform(context_ ? context_->location : "global");
}
ServiceEndpoint VertexTarget::beta_endpoint() const {
return endpoints::aiplatform_beta(context_ ? context_->location : "global");
}
void VertexTarget::require_segment(const std::string& value,
const std::string& what) {
if (!valid_segment(value))
throw GoogleError::invalid(what + " must be 1-128 of A-Z a-z 0-9 . _ -");
}
std::string VertexTarget::model_path(const std::string& publisher,
const std::string& model) const {
require_segment(publisher, "publisher");
require_segment(model, "model");
std::string tail = "/publishers/" + publisher + "/models/" + model;
return context_ ? "/" + context_->parent() + tail : tail;
}
std::string VertexTarget::parent_path() const {
return "/" + context().parent();
}
} // namespace molo::google::models
+43
View File
@@ -0,0 +1,43 @@
#pragma once
// Where a Vertex AI call goes. Two shapes, one class:
// - express mode (API key, no project): https://aiplatform.googleapis.com/v1
// paths /publishers/{pub}/models/{model}
// - project scoped (OAuth): https://{loc}-aiplatform.googleapis.com/v1
// paths /projects/{p}/locations/{l}/publishers/{pub}/models/{model}
// Every id is validated before any network traffic.
#include <optional>
#include <string>
#include "google/endpoints.h"
#include "google/resource.h"
namespace molo::google::models {
class VertexTarget {
public:
static VertexTarget express();
static VertexTarget project(CloudContext context);
// Project scoped when GOOGLE_CLOUD_PROJECT is set, express otherwise.
static VertexTarget from_env();
bool is_express() const { return !context_.has_value(); }
const CloudContext& context() const; // throws invalid in express mode
ServiceEndpoint endpoint() const; // v1
ServiceEndpoint beta_endpoint() const; // v1beta1
// "/publishers/{pub}/models/{model}" or the project scoped form.
std::string model_path(const std::string& publisher,
const std::string& model) const;
// "/projects/{p}/locations/{l}"; throws invalid in express mode.
std::string parent_path() const;
static void require_segment(const std::string& value, const std::string& what);
private:
std::optional<CloudContext> context_;
};
} // namespace molo::google::models
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""Checks calls.h (the module's one call table) against Google's Discovery
documents. Python 3 stdlib only, no credentials.
python3 -I src/google/models/tools/check_discovery.py [--cache DIR]
For every row it requires: the HTTP method and path exist in Discovery (the
prefixed form, and the unprefixed express form when the row says express),
the rpc id matches, every body field name exists in the request schema
(dotted names follow $ref), and every "?name" is a query parameter.
Exit 0: all rows verified. Exit 1: at least one mismatch (printed).
"""
import json
import os
import re
import sys
import urllib.request
HERE = os.path.dirname(os.path.abspath(__file__))
TABLE = os.path.join(HERE, "..", "calls.h")
ROW = re.compile(
r'^\s*\{"([^"]+)",\s*"([A-Z]+)",\s*"([^"]+)",\s*"([^"]*)",\s*"([^"]+)",\s*'
r'(true|false),\s*"([^"]+)",\s*"([^"]*)"\},?\s*$')
PREFIX = {"L": "/projects/{}/locations/{}", "R": "/projects/{}", "": ""}
def norm(path):
return re.sub(r"\{[^}]*\}", "{}", path)
def fetch(version, cache):
url = "https://aiplatform.googleapis.com/$discovery/rest?version=" + version
if cache:
os.makedirs(cache, exist_ok=True)
f = os.path.join(cache, "aiplatform_%s.json" % version)
if os.path.exists(f):
return json.load(open(f))
with urllib.request.urlopen(url, timeout=120) as r:
raw = r.read()
if cache:
open(os.path.join(cache, "aiplatform_%s.json" % version), "wb").write(raw)
return json.loads(raw)
def methods(doc):
out = {}
def walk(res):
for r in res.get("resources", {}).values():
for m in r.get("methods", {}).values():
key = (m["httpMethod"], norm("/" + m.get("flatPath", m["path"])))
out[key] = m
walk(r)
walk(doc)
return out
def schema_has(doc, schema_name, dotted):
s = doc["schemas"].get(schema_name)
for part in dotted.split("."):
if s is None or part not in s.get("properties", {}):
return False
prop = s["properties"][part]
ref = prop.get("$ref") or prop.get("items", {}).get("$ref")
s = doc["schemas"].get(ref) if ref else {"properties": {}}
if ref is None:
s = {"properties": {}, "_leaf": True}
return True
def main():
cache = None
if "--cache" in sys.argv:
cache = sys.argv[sys.argv.index("--cache") + 1]
rows = []
for line in open(TABLE):
m = ROW.match(line)
if m:
rows.append(m.groups())
if not rows:
print("no rows parsed from calls.h")
return 1
docs = {}
index = {}
bad = []
for call_id, http, ver, prefix, suffix, express, rpc, fields in rows:
if ver not in docs:
docs[ver] = fetch(ver, cache)
index[ver] = methods(docs[ver])
doc, idx = docs[ver], index[ver]
full = "/" + ver + PREFIX[prefix] + suffix
meth = idx.get((http, norm(full)))
if meth is None:
bad.append("%s: %s %s not in discovery %s" % (call_id, http, full, ver))
continue
if meth["id"] != rpc:
bad.append("%s: rpc id %s, table says %s" % (call_id, meth["id"], rpc))
if express == "true":
ex = idx.get((http, norm("/" + ver + suffix)))
if ex is None:
bad.append("%s: express form %s not in discovery" % (call_id, suffix))
req = meth.get("request", {}).get("$ref")
params = meth.get("parameters", {})
for f in fields.split():
if f == "-":
continue
if f.startswith("?"):
if f[1:] not in params or params[f[1:]]["location"] != "query":
bad.append("%s: query parameter %s absent" % (call_id, f[1:]))
elif req is None or not schema_has(doc, req, f):
bad.append("%s: request field %s absent in %s" % (call_id, f, req))
for b in bad:
print("FAIL " + b)
print("%d calls checked, %d problems" % (len(rows), len(bad)))
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main())
+26
View File
@@ -0,0 +1,26 @@
#include "google/models/vertex_model.h"
#include "google/errors.h"
namespace molo::google::models {
VertexModel::VertexModel(std::shared_ptr<Gemini> gemini, std::string default_model)
: gemini_(std::move(gemini)), default_model_(std::move(default_model)) {
VertexTarget::require_segment(default_model_, "model");
}
const std::string& VertexModel::model_for(const GenerateRequest& request) const {
return request.model.empty() ? default_model_ : request.model;
}
GenerateResponse VertexModel::generate(const GenerateRequest& request) {
return gemini_->generate(model_for(request), request);
}
void VertexModel::stream(
const GenerateRequest& request,
const std::function<bool(const GenerateResponse&)>& on_chunk) {
gemini_->stream(model_for(request), request, on_chunk);
}
} // namespace molo::google::models
+31
View File
@@ -0,0 +1,31 @@
#pragma once
// VertexModel: a LanguageModel over Gemini on Vertex AI. GenerateRequest.model
// overrides the default model when set.
#include <memory>
#include <string>
#include "google/gemini_types.h"
#include "google/models/gemini.h"
namespace molo::google::models {
class VertexModel : public LanguageModel {
public:
VertexModel(std::shared_ptr<Gemini> gemini, std::string default_model);
const std::string& default_model() const { return default_model_; }
GenerateResponse generate(const GenerateRequest& request) override;
void stream(const GenerateRequest& request,
const std::function<bool(const GenerateResponse&)>& on_chunk) override;
private:
const std::string& model_for(const GenerateRequest& request) const;
std::shared_ptr<Gemini> gemini_;
std::string default_model_;
};
} // namespace molo::google::models
+27
View File
@@ -0,0 +1,27 @@
#include "google/models/vertex_service.h"
namespace molo::google::models {
VertexService::VertexService(std::shared_ptr<GoogleClient> client,
VertexTarget target, std::string name)
: Service(std::move(client), ServiceEndpoint{std::move(name), target.endpoint().base_url}),
planner_(std::move(target)) {}
clients::body::Json VertexService::run(const PlannedCall& call) const {
return client_->request(call.method, call.endpoint, call.path,
call.has_body ? &call.body : nullptr);
}
void VertexService::run_stream(const PlannedCall& call,
const GoogleClient::EventSink& sink) const {
client_->stream_json(call.endpoint, call.path,
call.has_body ? call.body : clients::body::Json::object(),
sink);
}
CallPlanner::Args VertexService::model_args(const std::string& publisher,
const std::string& model) {
return {{"publisher", publisher}, {"model", model}};
}
} // namespace molo::google::models
+33
View File
@@ -0,0 +1,33 @@
#pragma once
// Base of every service in this module: a Service that owns a VertexTarget
// and a CallPlanner, and runs planned calls over the shared GoogleClient.
#include <functional>
#include "google/models/json_read.h"
#include "google/models/planner.h"
#include "google/service.h"
namespace molo::google::models {
class VertexService : public Service {
public:
VertexService(std::shared_ptr<GoogleClient> client, VertexTarget target,
std::string name);
const VertexTarget& target() const { return planner_.target(); }
const CallPlanner& planner() const { return planner_; }
protected:
clients::body::Json run(const PlannedCall& call) const;
void run_stream(const PlannedCall& call,
const GoogleClient::EventSink& sink) const;
// Single "{publisher}/{model}" argument map.
static CallPlanner::Args model_args(const std::string& publisher,
const std::string& model);
CallPlanner planner_;
};
} // namespace molo::google::models