Merge models module (Model Garden, Gemini on Vertex, partner and open models, media)
This commit is contained in:
@@ -0,0 +1,163 @@
|
||||
# Models (Gemini Enterprise Agent Platform) - Specification
|
||||
|
||||
Owned directory: `src/google/models/`. Namespace `molo::google::models`.
|
||||
Depends on the foundation in `src/google/` (`GoogleClient`, `Service`,
|
||||
`GoogleError`, `CloudContext`, `Operation`, `gemini_types.h`). Voice and
|
||||
`Generative` are untouched.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
Gemini on Vertex AI (generate, stream, count tokens, compute tokens,
|
||||
embeddings), Model Garden (list, get, publisher model config, EULA, deploy),
|
||||
partner models (Claude rawPredict and streamRawPredict), endpoint models
|
||||
(Gemma and any deployed model: list, get, predict family), media generation
|
||||
(image, music, video via predict and predictLongRunning), and
|
||||
`VertexModel : LanguageModel`.
|
||||
|
||||
Two addressing modes, one class (`VertexTarget`):
|
||||
- express (API key, no project): `https://aiplatform.googleapis.com/v1/publishers/google/models/{m}:generateContent`
|
||||
- project scoped: `https://{loc}-aiplatform.googleapis.com/v1/projects/{p}/locations/{loc}/publishers/{pub}/models/{m}:...`
|
||||
(`global` uses the host without a prefix).
|
||||
|
||||
## 2. Module map (one concern per file)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings |
|
||||
| `calls.h` | THE call table: every call, its Discovery rpc id and request fields |
|
||||
| `target.h/.cpp` | `VertexTarget`: express vs project, id validation |
|
||||
| `planner.h/.cpp` | `CallPlanner`: table row + ids -> exact request (pure, no I/O) |
|
||||
| `vertex_service.h/.cpp` | `VertexService : Service`: runs planned calls |
|
||||
| `json_read.h` | tolerant response readers |
|
||||
| `gemini.h/.cpp` | `Gemini`: generate, stream (`?alt=sse`), count/compute tokens, embedContent, predict embeddings |
|
||||
| `vertex_model.h/.cpp` | `VertexModel : LanguageModel` |
|
||||
| `model_garden.h/.cpp` | `ModelGarden`: list, get, fetch config, EULA check/accept, deploy |
|
||||
| `claude_models.h/.cpp` | `ClaudeModels`: rawPredict, streamRawPredict, `text_delta` |
|
||||
| `open_models.h/.cpp` | `OpenModels`: endpoints list/get, predict, rawPredict, streamRawPredict, serverStreamingPredict, generate/stream/count on endpoints, predictLongRunning, fetchPredictOperation, publisher invoke |
|
||||
| `media.h/.cpp` | `MediaGenerator`: image, music, video start/status/wait |
|
||||
| `tools/check_discovery.py` | checks `calls.h` against Google's Discovery documents |
|
||||
|
||||
Every call has a `plan_*` twin returning a `PlannedCall` (method, endpoint,
|
||||
path, body) without sending anything; tests and `check_discovery.py` verify
|
||||
requests that way.
|
||||
|
||||
## 3. Static caps (`caps.h`)
|
||||
|
||||
Contents per request 200; request text 1,000,000 chars; output tokens
|
||||
1..65536; candidates 1..8; embed batch 100, 100,000 chars per text, 3072
|
||||
dimensions; list page size 1..100, 20 pages, 2000 models returned; Claude
|
||||
200 messages, 1,000,000 chars, max_tokens 1..32768; endpoint instances 16,
|
||||
100,000 chars, Gemma max_tokens 8192; media prompt 4000 chars, images 4,
|
||||
music samples 4, video samples 4, video 8 seconds, 32 MiB and 8 items per
|
||||
result, operation name 512 chars; video polls 120 at 5 s (10 minutes).
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- Every id (publisher, model, endpoint, invoke id) passes `valid_segment`
|
||||
before a path is built; operation names and `publisherModel` strings have
|
||||
their own character check. Bad input is `GoogleError::invalid` (400
|
||||
`INVALID_ARGUMENT`) before any network traffic.
|
||||
- A call whose table row has `express=false` throws invalid in express mode,
|
||||
naming the missing project. No credential: `NotConfiguredError` (503).
|
||||
- Costly or consent calls are explicit and gated: `ModelGarden::deploy`
|
||||
needs `DeployOptions.confirm`, `accept_eula` needs `confirm=true`; both
|
||||
refuse otherwise. Nothing in this module deletes.
|
||||
- Streaming uses `GoogleClient::stream_json` (SSE via `?alt=sse` for Gemini,
|
||||
Anthropic events for Claude); the sink returns false to stop.
|
||||
- Video: `start_video` (predictLongRunning), `video_status`
|
||||
(fetchPredictOperation, a POST), `wait_video` bounded by the caps.
|
||||
`wait_operation` (GET) is not used: Discovery has no GET on these
|
||||
operations for publisher models.
|
||||
- Example (`/agent/models`): GET page, `/status`, `/calls` need no
|
||||
credential; every Google call is a POST that answers 503
|
||||
`google_not_configured` without a credential (checked first), 422 on bad
|
||||
input. Routes: generate, count, embed, claude, list, endpoint, image,
|
||||
music, video, video/status.
|
||||
|
||||
## 5. Verified against discovery
|
||||
|
||||
`python3 -I src/google/models/tools/check_discovery.py` fetches
|
||||
`https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `v1beta1`
|
||||
and fails when any row of `calls.h` has a missing method or path, a wrong
|
||||
rpc id, an absent request body field (dotted names follow the schema) or an
|
||||
absent query parameter. Result on 2026-10-24: `29 calls checked, 0 problems`.
|
||||
The same pass shows what Discovery says about express forms:
|
||||
generateContent, streamGenerateContent, countTokens, computeTokens, predict,
|
||||
predictLongRunning, fetchPredictOperation exist without a project;
|
||||
embedContent, rawPredict, streamRawPredict, publisher invoke do not
|
||||
(`embedContent` express answers 404 live). `publishers.models.list` exists
|
||||
only in v1beta1; `publishers.models.get` in v1 and v1beta1.
|
||||
|
||||
Request construction is compared to the table in
|
||||
`tests/src/google/models/test_models.cpp` (every body key must be a declared
|
||||
field).
|
||||
|
||||
## 6. Verified live
|
||||
|
||||
Run 2026-10-24 through this module's C++ client
|
||||
(`MOLO_LIVE_NET=1 build/google_test`, tests `models_live.*`), read-only or
|
||||
tiny (20 tokens). Credentials never printed or stored.
|
||||
|
||||
With an express API key (`AQ.`):
|
||||
- `Gemini::generate`, `stream` (SSE), `count_tokens`, `compute_tokens`,
|
||||
`embed` (text-embedding-005 predict) on `aiplatform.googleapis.com/v1/publishers/google/models/gemini-3.5-flash`: 200.
|
||||
- Project scoped `countTokens` at location `global`: 200.
|
||||
- Refused with 401 "API keys are not supported": Model Garden list/get,
|
||||
endpoints list, rawPredict, project embedContent. Express `embedContent`:
|
||||
404 (matches Discovery).
|
||||
|
||||
With the service account OAuth token (project `lisa-448004`):
|
||||
- v1beta1 `publishers/google/models` list (google, anthropic), v1
|
||||
`publishers/google/models/{gemini-2.5-flash,gemini-3.5-flash}` get incl.
|
||||
`view=PUBLISHER_MODEL_VIEW_FULL`: 200.
|
||||
- Endpoints list (`us-central1`): 200, empty.
|
||||
- `embedContent` project scoped `us-central1` `gemini-embedding-001`: 200
|
||||
(global and text-embedding-005 not available there: 404/400).
|
||||
- Project scoped `generateContent` and `computeTokens` at `global`: 200
|
||||
(`us-central1` + gemini-2.5-flash: 404 "not found or no access").
|
||||
- Claude `rawPredict` at `global` (claude-sonnet-4-6): reached, 429
|
||||
RESOURCE_EXHAUSTED (project quota); `us-east5`: 404 no access. Route, auth
|
||||
and body accepted; no completion was obtained.
|
||||
- `fetchPublisherModelConfig`: 404 "PublisherModelConfig not found" (route
|
||||
exists, nothing configured). EULA check for `publishers/google/models/gemma3`:
|
||||
404 model not found (route exists).
|
||||
- Generation calls may answer 429 on the shared project; the live test notes
|
||||
that instead of failing.
|
||||
|
||||
Not live-verified (mutating or billable): deploy, EULA accept, image, music
|
||||
and video generation, endpoint predict family (no endpoint exists), Gemma
|
||||
shapes. These are verified by Discovery only.
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- Discovery documents: `https://aiplatform.googleapis.com/$discovery/rest?version=v1` and `?version=v1beta1` (fetched 2026-10-24).
|
||||
- Claude on Vertex: `https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/partner-models/claude/use-claude` (rawPredict/streamRawPredict paths, `anthropic_version: vertex-2023-10-16`, messages/max_tokens/stream/thinking, response shape).
|
||||
- Reference pages read for semantics: `.../reference/rest/v1/projects.locations.publishers.models/{countTokens,embedContent}`, `.../reference/rest/v1/publishers.models/get` (view enum, languageCode).
|
||||
|
||||
## 8. Unverified (neither Discovery nor docs read, nor live)
|
||||
|
||||
- Imagen `predict` instance/parameter names (`prompt`; `sampleCount`,
|
||||
`aspectRatio`, `negativePrompt`) and the `predictions[].bytesBase64Encoded`
|
||||
/ `mimeType` response: Discovery types them as free-form `Value`; the doc
|
||||
pages could not be read.
|
||||
- Lyria shape (`prompt`, `negative_prompt`, `seed`; `sample_count`).
|
||||
- Veo shape (`sampleCount`, `durationSeconds`, `aspectRatio`, `storageUri`;
|
||||
final `response.videos[]` with `gcsUri` / `bytesBase64Encoded` / `mimeType`).
|
||||
`fetchPredictOperation` itself is Discovery-verified and reachable live.
|
||||
- Gemma/vLLM instance shape in `gemma_instance` (`prompt`, `max_tokens`,
|
||||
`temperature`): the serving container's, not Google's API.
|
||||
- Anthropic streaming event names (`content_block_delta` / `text_delta`)
|
||||
come from Anthropic's format; Claude completions were not obtained live.
|
||||
- Embedding predict response `embeddings.values` / `statistics.token_count`
|
||||
was observed live for text-embedding-005, not documented-read.
|
||||
- `deployPublisherModel` field semantics beyond the schema names.
|
||||
|
||||
## 9. Gaps
|
||||
|
||||
- Discovery methods deliberately not wrapped (owner: this module): publisher
|
||||
`export`, `setPublisherModelConfig`, `enableModel`, IAM policy, `v1/responses`
|
||||
on publishers, publisher-level `serverStreamingPredict` and v1beta1
|
||||
`publishers.models.*` project-less duplicates.
|
||||
- Endpoint `generateContent` family is wired but only Discovery-verified.
|
||||
- `VertexModel` has no retry; quota 429 surfaces as `GoogleError`.
|
||||
- Token refresh is the caller's job (foundation gap).
|
||||
@@ -0,0 +1,75 @@
|
||||
#pragma once
|
||||
|
||||
// The ONE table of every Google call this module makes. Each row is checked
|
||||
// against Google's Discovery document by tools/check_discovery.py (see
|
||||
// SPEC.md "Verified against discovery"). Keep one row per line, in the shape
|
||||
// {"id", "HTTP", "version", "prefix", "suffix", express, "rpc", "fields"},
|
||||
// the checker parses exactly this shape.
|
||||
//
|
||||
// prefix "L": /projects/{project}/locations/{location} + suffix
|
||||
// "R": /projects/{project} + suffix
|
||||
// "" : suffix alone (no project)
|
||||
// express true: the unprefixed form (suffix alone) also exists in Discovery
|
||||
// and works with an express API key.
|
||||
// rpc Discovery method id of the prefixed (or only) form.
|
||||
// fields space separated: request body field names (dotted: nested via the
|
||||
// schema, "generationConfig.topP"), "?name" for a query parameter,
|
||||
// "-" for none (raw HTTP body).
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct CallSpec {
|
||||
const char* id;
|
||||
const char* http;
|
||||
const char* version;
|
||||
const char* prefix;
|
||||
const char* suffix;
|
||||
bool express;
|
||||
const char* rpc;
|
||||
const char* fields;
|
||||
};
|
||||
|
||||
// clang-format off
|
||||
inline constexpr CallSpec kCalls[] = {
|
||||
// Gemini on Vertex AI
|
||||
{"gemini.generate", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:generateContent", true, "aiplatform.projects.locations.publishers.models.generateContent", "contents systemInstruction tools toolConfig safetySettings generationConfig generationConfig.temperature generationConfig.topP generationConfig.topK generationConfig.maxOutputTokens generationConfig.candidateCount generationConfig.stopSequences generationConfig.responseModalities generationConfig.responseMimeType generationConfig.responseSchema generationConfig.speechConfig generationConfig.thinkingConfig"},
|
||||
{"gemini.stream", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:streamGenerateContent", true, "aiplatform.projects.locations.publishers.models.streamGenerateContent", "contents systemInstruction tools toolConfig safetySettings generationConfig"},
|
||||
{"gemini.count_tokens", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:countTokens", true, "aiplatform.projects.locations.publishers.models.countTokens", "contents systemInstruction tools generationConfig"},
|
||||
{"gemini.compute_tokens", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:computeTokens", true, "aiplatform.projects.locations.publishers.models.computeTokens", "contents"},
|
||||
{"gemini.embed_content", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:embedContent", false, "aiplatform.projects.locations.publishers.models.embedContent", "content embedContentConfig embedContentConfig.taskType embedContentConfig.title embedContentConfig.outputDimensionality embedContentConfig.autoTruncate"},
|
||||
{"gemini.embed_predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:predict", true, "aiplatform.projects.locations.publishers.models.predict", "instances parameters"},
|
||||
// Partner models (Claude): raw JSON body in the model's own format
|
||||
{"partner.raw_predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:rawPredict", false, "aiplatform.projects.locations.publishers.models.rawPredict", "-"},
|
||||
{"partner.stream_raw_predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:streamRawPredict", false, "aiplatform.projects.locations.publishers.models.streamRawPredict", "-"},
|
||||
{"publisher.invoke", "POST", "v1", "L", "/publishers/{publisher}/models/{model}/invoke/{invoke}", false, "aiplatform.projects.locations.publishers.models.invoke.invoke", "-"},
|
||||
// Media generation
|
||||
{"media.predict", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:predict", true, "aiplatform.projects.locations.publishers.models.predict", "instances parameters"},
|
||||
{"media.predict_long_running", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:predictLongRunning", true, "aiplatform.projects.locations.publishers.models.predictLongRunning", "instances parameters"},
|
||||
{"media.fetch_predict_operation", "POST", "v1", "L", "/publishers/{publisher}/models/{model}:fetchPredictOperation", true, "aiplatform.projects.locations.publishers.models.fetchPredictOperation", "operationName"},
|
||||
// Model Garden
|
||||
{"garden.list", "GET", "v1beta1", "", "/publishers/{publisher}/models", false, "aiplatform.publishers.models.list", "?pageSize ?pageToken ?filter ?view ?orderBy ?languageCode ?listAllVersions"},
|
||||
{"garden.get", "GET", "v1", "", "/publishers/{publisher}/models/{model}", false, "aiplatform.publishers.models.get", "?view ?languageCode"},
|
||||
{"garden.fetch_config", "GET", "v1beta1", "L", "/publishers/{publisher}/models/{model}:fetchPublisherModelConfig", false, "aiplatform.projects.locations.publishers.models.fetchPublisherModelConfig", "-"},
|
||||
{"garden.eula_check", "POST", "v1beta1", "R", "/modelGardenEula:check", false, "aiplatform.projects.modelGardenEula.check", "publisherModel"},
|
||||
{"garden.eula_accept", "POST", "v1beta1", "R", "/modelGardenEula:accept", false, "aiplatform.projects.modelGardenEula.accept", "publisherModel"},
|
||||
{"garden.deploy", "POST", "v1beta1", "L", ":deployPublisherModel", false, "aiplatform.projects.locations.deployPublisherModel", "model modelDisplayName endpointDisplayName acceptEula"},
|
||||
// Endpoints (open models such as Gemma, and any deployed model)
|
||||
{"endpoint.list", "GET", "v1", "L", "/endpoints", false, "aiplatform.projects.locations.endpoints.list", "?pageSize ?pageToken ?filter ?orderBy ?readMask"},
|
||||
{"endpoint.get", "GET", "v1", "L", "/endpoints/{endpoint}", false, "aiplatform.projects.locations.endpoints.get", "-"},
|
||||
{"endpoint.predict", "POST", "v1", "L", "/endpoints/{endpoint}:predict", false, "aiplatform.projects.locations.endpoints.predict", "instances parameters"},
|
||||
{"endpoint.raw_predict", "POST", "v1", "L", "/endpoints/{endpoint}:rawPredict", false, "aiplatform.projects.locations.endpoints.rawPredict", "-"},
|
||||
{"endpoint.stream_raw_predict", "POST", "v1", "L", "/endpoints/{endpoint}:streamRawPredict", false, "aiplatform.projects.locations.endpoints.streamRawPredict", "-"},
|
||||
{"endpoint.server_streaming_predict", "POST", "v1", "L", "/endpoints/{endpoint}:serverStreamingPredict", false, "aiplatform.projects.locations.endpoints.serverStreamingPredict", "inputs parameters"},
|
||||
{"endpoint.generate_content", "POST", "v1", "L", "/endpoints/{endpoint}:generateContent", false, "aiplatform.projects.locations.endpoints.generateContent", "contents systemInstruction tools generationConfig"},
|
||||
{"endpoint.stream_generate_content", "POST", "v1", "L", "/endpoints/{endpoint}:streamGenerateContent", false, "aiplatform.projects.locations.endpoints.streamGenerateContent", "contents systemInstruction tools generationConfig"},
|
||||
{"endpoint.count_tokens", "POST", "v1", "L", "/endpoints/{endpoint}:countTokens", false, "aiplatform.projects.locations.endpoints.countTokens", "contents systemInstruction tools generationConfig"},
|
||||
{"endpoint.predict_long_running", "POST", "v1", "L", "/endpoints/{endpoint}:predictLongRunning", false, "aiplatform.projects.locations.endpoints.predictLongRunning", "instances parameters"},
|
||||
{"endpoint.fetch_predict_operation", "POST", "v1", "L", "/endpoints/{endpoint}:fetchPredictOperation", false, "aiplatform.projects.locations.endpoints.fetchPredictOperation", "operationName"},
|
||||
};
|
||||
// clang-format on
|
||||
|
||||
inline constexpr std::size_t kCallCount = sizeof(kCalls) / sizeof(kCalls[0]);
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,45 @@
|
||||
#pragma once
|
||||
|
||||
// Static ceilings for the models module. Never autotuned.
|
||||
|
||||
#include <chrono>
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct ModelsCaps {
|
||||
// Gemini
|
||||
static constexpr std::size_t kMaxContents = 200;
|
||||
static constexpr std::size_t kMaxTextChars = 1000000; // all text parts of a request
|
||||
static constexpr int kMaxOutputTokens = 65536;
|
||||
static constexpr int kMaxCandidates = 8;
|
||||
// Embeddings
|
||||
static constexpr std::size_t kMaxEmbedBatch = 100;
|
||||
static constexpr std::size_t kMaxEmbedChars = 100000; // per text
|
||||
static constexpr int kMaxEmbedDimensions = 3072;
|
||||
// Model Garden
|
||||
static constexpr std::size_t kMaxListPages = 20;
|
||||
static constexpr int kMaxPageSize = 100;
|
||||
static constexpr std::size_t kMaxModelsReturned = 2000;
|
||||
// Partner (Claude)
|
||||
static constexpr std::size_t kMaxClaudeMessages = 200;
|
||||
static constexpr int kMaxClaudeTokens = 32768;
|
||||
static constexpr std::size_t kMaxClaudeChars = 1000000;
|
||||
// Open models
|
||||
static constexpr std::size_t kMaxInstances = 16;
|
||||
static constexpr std::size_t kMaxInstanceChars = 100000;
|
||||
static constexpr int kMaxGemmaTokens = 8192;
|
||||
// Media
|
||||
static constexpr std::size_t kMaxPromptChars = 4000;
|
||||
static constexpr int kMaxImages = 4;
|
||||
static constexpr int kMaxMusicSamples = 4;
|
||||
static constexpr int kMaxVideoSamples = 4;
|
||||
static constexpr int kMaxVideoSeconds = 8;
|
||||
static constexpr std::size_t kMaxMediaBytes = 32u * 1024u * 1024u; // per item
|
||||
static constexpr std::size_t kMaxMediaItems = 8;
|
||||
static constexpr std::size_t kMaxOperationName = 512;
|
||||
static constexpr std::chrono::milliseconds kVideoPollInterval{5000};
|
||||
static constexpr std::size_t kMaxVideoPolls = 120; // 10 minutes
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,116 @@
|
||||
#include "google/models/claude_models.h"
|
||||
|
||||
#include "google/errors.h"
|
||||
#include "google/models/json_read.h"
|
||||
#include "google/models/caps.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
Json ClaudeRequest::to_json(bool stream) const {
|
||||
Json body = Json::object();
|
||||
body.set("anthropic_version", Json(ClaudeModels::kAnthropicVersion));
|
||||
Json msgs = Json::array();
|
||||
for (const ClaudeMessage& m : messages) {
|
||||
Json one = Json::object();
|
||||
one.set("role", Json(m.role));
|
||||
one.set("content", Json(m.text));
|
||||
msgs.push(one);
|
||||
}
|
||||
body.set("messages", msgs);
|
||||
body.set("max_tokens", Json(max_tokens));
|
||||
if (!system.empty()) body.set("system", Json(system));
|
||||
if (temperature) body.set("temperature", Json(*temperature));
|
||||
if (thinking_budget > 0) {
|
||||
Json t = Json::object();
|
||||
t.set("type", Json("enabled"));
|
||||
t.set("budget_tokens", Json(thinking_budget));
|
||||
body.set("thinking", t);
|
||||
}
|
||||
if (tools.is_array()) body.set("tools", tools);
|
||||
if (stream) body.set("stream", Json(true));
|
||||
return body;
|
||||
}
|
||||
|
||||
ClaudeResponse ClaudeResponse::from_json(const Json& doc) {
|
||||
ClaudeResponse r;
|
||||
r.id = json_str(doc, "id");
|
||||
r.model = json_str(doc, "model");
|
||||
r.stop_reason = json_str(doc, "stop_reason");
|
||||
if (const Json* c = doc.find("content"); c && c->is_array()) {
|
||||
r.content = *c;
|
||||
for (const Json& block : c->as_array())
|
||||
if (json_str(block, "type") == "text")
|
||||
r.text += json_str(block, "text");
|
||||
}
|
||||
if (const Json* u = doc.find("usage"); u && u->is_object()) {
|
||||
r.input_tokens = static_cast<int>(json_num(*u, "input_tokens"));
|
||||
r.output_tokens = static_cast<int>(json_num(*u, "output_tokens"));
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
ClaudeModels::ClaudeModels(std::shared_ptr<GoogleClient> client, VertexTarget target)
|
||||
: VertexService(std::move(client), std::move(target), "aiplatform") {}
|
||||
|
||||
void ClaudeModels::validate(const ClaudeRequest& r) {
|
||||
VertexTarget::require_segment(r.model, "model");
|
||||
if (r.messages.empty()) throw GoogleError::invalid("no messages");
|
||||
if (r.messages.size() > ModelsCaps::kMaxClaudeMessages)
|
||||
throw GoogleError::invalid("more than " +
|
||||
std::to_string(ModelsCaps::kMaxClaudeMessages) +
|
||||
" messages");
|
||||
if (r.messages.front().role != "user")
|
||||
throw GoogleError::invalid("the first message must be from the user");
|
||||
std::size_t chars = r.system.size();
|
||||
for (const ClaudeMessage& m : r.messages) {
|
||||
if (m.role != "user" && m.role != "assistant")
|
||||
throw GoogleError::invalid("role must be user or assistant");
|
||||
if (m.text.empty()) throw GoogleError::invalid("message text is empty");
|
||||
chars += m.text.size();
|
||||
}
|
||||
if (chars > ModelsCaps::kMaxClaudeChars)
|
||||
throw GoogleError::invalid("request text over " +
|
||||
std::to_string(ModelsCaps::kMaxClaudeChars) +
|
||||
" characters");
|
||||
if (r.max_tokens < 1 || r.max_tokens > ModelsCaps::kMaxClaudeTokens)
|
||||
throw GoogleError::invalid("max_tokens outside 1.." +
|
||||
std::to_string(ModelsCaps::kMaxClaudeTokens));
|
||||
if (r.temperature && (*r.temperature < 0.0 || *r.temperature > 1.0))
|
||||
throw GoogleError::invalid("temperature outside 0..1");
|
||||
if (r.thinking_budget != 0 &&
|
||||
(r.thinking_budget < 1024 || r.thinking_budget >= r.max_tokens))
|
||||
throw GoogleError::invalid("thinking_budget must be >= 1024 and < max_tokens");
|
||||
}
|
||||
|
||||
PlannedCall ClaudeModels::plan_generate(const ClaudeRequest& request) const {
|
||||
validate(request);
|
||||
Json body = request.to_json(false);
|
||||
return planner_.plan("partner.raw_predict", model_args("anthropic", request.model), &body);
|
||||
}
|
||||
|
||||
PlannedCall ClaudeModels::plan_stream(const ClaudeRequest& request) const {
|
||||
validate(request);
|
||||
Json body = request.to_json(true);
|
||||
return planner_.plan("partner.stream_raw_predict", model_args("anthropic", request.model), &body);
|
||||
}
|
||||
|
||||
ClaudeResponse ClaudeModels::generate(const ClaudeRequest& request) const {
|
||||
return ClaudeResponse::from_json(run(plan_generate(request)));
|
||||
}
|
||||
|
||||
void ClaudeModels::stream(
|
||||
const ClaudeRequest& request,
|
||||
const std::function<bool(const Json&)>& sink) const {
|
||||
run_stream(plan_stream(request), sink);
|
||||
}
|
||||
|
||||
std::string ClaudeModels::text_delta(const Json& event) {
|
||||
if (json_str(event, "type") != "content_block_delta") return "";
|
||||
const Json* d = event.find("delta");
|
||||
if (!d || !d->is_object() || json_str(*d, "type") != "text_delta") return "";
|
||||
return json_str(*d, "text");
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,63 @@
|
||||
#pragma once
|
||||
|
||||
// Partner models: Claude on Vertex AI through publishers/anthropic
|
||||
// rawPredict and streamRawPredict. Project scoped only (Discovery has no
|
||||
// express form for the raw predict methods).
|
||||
|
||||
#include <functional>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/models/vertex_service.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct ClaudeMessage {
|
||||
std::string role; // "user" | "assistant"
|
||||
std::string text;
|
||||
};
|
||||
|
||||
struct ClaudeRequest {
|
||||
std::string model; // e.g. claude-sonnet-4-6
|
||||
std::string system;
|
||||
std::vector<ClaudeMessage> messages;
|
||||
int max_tokens = 1024;
|
||||
std::optional<double> temperature;
|
||||
int thinking_budget = 0; // 0: off; else >= 1024 and < max_tokens
|
||||
clients::body::Json tools; // Anthropic tools array; null: none
|
||||
clients::body::Json to_json(bool stream) const;
|
||||
};
|
||||
|
||||
struct ClaudeResponse {
|
||||
std::string id;
|
||||
std::string model;
|
||||
std::string stop_reason;
|
||||
std::string text; // concatenated text blocks
|
||||
int input_tokens = 0;
|
||||
int output_tokens = 0;
|
||||
clients::body::Json content; // raw content blocks (tool_use included)
|
||||
static ClaudeResponse from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
class ClaudeModels : public VertexService {
|
||||
public:
|
||||
static constexpr const char* kAnthropicVersion = "vertex-2023-10-16";
|
||||
|
||||
ClaudeModels(std::shared_ptr<GoogleClient> client, VertexTarget target);
|
||||
|
||||
ClaudeResponse generate(const ClaudeRequest& request) const;
|
||||
// Calls sink per Anthropic stream event (message_start, content_block_delta,
|
||||
// ...); return false to stop.
|
||||
void stream(const ClaudeRequest& request,
|
||||
const std::function<bool(const clients::body::Json&)>& sink) const;
|
||||
|
||||
PlannedCall plan_generate(const ClaudeRequest& request) const;
|
||||
PlannedCall plan_stream(const ClaudeRequest& request) const;
|
||||
|
||||
// Text of a content_block_delta / text_delta event, else "".
|
||||
static std::string text_delta(const clients::body::Json& event);
|
||||
static void validate(const ClaudeRequest& request);
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,252 @@
|
||||
#include "google/models/gemini.h"
|
||||
|
||||
#include "google/errors.h"
|
||||
#include "google/models/caps.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
|
||||
bool token_chars(const std::string& s) {
|
||||
if (s.size() > 64) return false;
|
||||
for (char c : s)
|
||||
if (!((c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '_'))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
Embedding parse_vector(const Json& holder, const Json* stats) {
|
||||
Embedding e;
|
||||
if (const Json* v = holder.find("values"); v && v->is_array())
|
||||
for (const Json& x : v->as_array())
|
||||
if (x.is_number()) e.values.push_back(x.as_number());
|
||||
if (stats && stats->is_object()) {
|
||||
e.token_count = static_cast<int>(json_num(*stats, "token_count"));
|
||||
if (const Json* t = stats->find("truncated"); t && t->is_bool())
|
||||
e.truncated = t->as_bool();
|
||||
}
|
||||
return e;
|
||||
}
|
||||
|
||||
const char* kPublisher = "google";
|
||||
|
||||
} // namespace
|
||||
|
||||
Gemini::Gemini(std::shared_ptr<GoogleClient> client, VertexTarget target)
|
||||
: VertexService(std::move(client), std::move(target), "aiplatform") {}
|
||||
|
||||
void Gemini::validate(const GenerateRequest& request) {
|
||||
if (request.contents.empty()) throw GoogleError::invalid("no contents");
|
||||
if (request.contents.size() > ModelsCaps::kMaxContents)
|
||||
throw GoogleError::invalid("more than " +
|
||||
std::to_string(ModelsCaps::kMaxContents) +
|
||||
" contents");
|
||||
std::size_t chars = 0;
|
||||
for (const Content& c : request.contents) {
|
||||
if (c.parts.empty()) throw GoogleError::invalid("a content has no parts");
|
||||
for (const Part& p : c.parts) {
|
||||
chars += p.text.size();
|
||||
if (p.text.empty() && !p.inline_data && !p.file_data && !p.function_call &&
|
||||
!p.function_response)
|
||||
throw GoogleError::invalid("a part is empty");
|
||||
}
|
||||
}
|
||||
if (request.system)
|
||||
for (const Part& p : request.system->parts) chars += p.text.size();
|
||||
if (chars > ModelsCaps::kMaxTextChars)
|
||||
throw GoogleError::invalid("request text over " +
|
||||
std::to_string(ModelsCaps::kMaxTextChars) +
|
||||
" characters");
|
||||
const auto& cfg = request.config;
|
||||
if (cfg.max_output_tokens &&
|
||||
(*cfg.max_output_tokens < 1 || *cfg.max_output_tokens > ModelsCaps::kMaxOutputTokens))
|
||||
throw GoogleError::invalid("max_output_tokens outside 1.." +
|
||||
std::to_string(ModelsCaps::kMaxOutputTokens));
|
||||
if (cfg.candidate_count &&
|
||||
(*cfg.candidate_count < 1 || *cfg.candidate_count > ModelsCaps::kMaxCandidates))
|
||||
throw GoogleError::invalid("candidate_count outside 1.." +
|
||||
std::to_string(ModelsCaps::kMaxCandidates));
|
||||
}
|
||||
|
||||
PlannedCall Gemini::plan_generate(const std::string& model,
|
||||
const GenerateRequest& r) const {
|
||||
validate(r);
|
||||
Json body = r.to_json();
|
||||
return planner_.plan("gemini.generate", model_args(kPublisher, model), &body);
|
||||
}
|
||||
|
||||
PlannedCall Gemini::plan_stream(const std::string& model,
|
||||
const GenerateRequest& r) const {
|
||||
validate(r);
|
||||
Json body = r.to_json();
|
||||
return planner_.plan("gemini.stream", model_args(kPublisher, model), &body,
|
||||
"alt=sse");
|
||||
}
|
||||
|
||||
PlannedCall Gemini::plan_count_tokens(const std::string& model,
|
||||
const GenerateRequest& r) const {
|
||||
validate(r);
|
||||
Json full = r.to_json();
|
||||
Json body = Json::object();
|
||||
for (const char* key : {"contents", "systemInstruction", "tools", "generationConfig"})
|
||||
if (const Json* v = full.find(key)) body.set(key, *v);
|
||||
return planner_.plan("gemini.count_tokens", model_args(kPublisher, model), &body);
|
||||
}
|
||||
|
||||
PlannedCall Gemini::plan_compute_tokens(const std::string& model,
|
||||
const std::vector<Content>& contents) const {
|
||||
if (contents.empty()) throw GoogleError::invalid("no contents");
|
||||
if (contents.size() > ModelsCaps::kMaxContents)
|
||||
throw GoogleError::invalid("too many contents");
|
||||
Json arr = Json::array();
|
||||
for (const Content& c : contents) arr.push(c.to_json());
|
||||
Json body = Json::object();
|
||||
body.set("contents", arr);
|
||||
return planner_.plan("gemini.compute_tokens", model_args(kPublisher, model), &body);
|
||||
}
|
||||
|
||||
GenerateResponse Gemini::generate(const std::string& model,
|
||||
const GenerateRequest& request) const {
|
||||
return GenerateResponse::from_json(run(plan_generate(model, request)));
|
||||
}
|
||||
|
||||
void Gemini::stream(
|
||||
const std::string& model, const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) const {
|
||||
run_stream(plan_stream(model, request), [&](const Json& event) {
|
||||
return on_chunk(GenerateResponse::from_json(event));
|
||||
});
|
||||
}
|
||||
|
||||
CountTokensResult Gemini::count_tokens(const std::string& model,
|
||||
const GenerateRequest& request) const {
|
||||
Json reply = run(plan_count_tokens(model, request));
|
||||
CountTokensResult out;
|
||||
out.total_tokens = static_cast<int>(json_num(reply, "totalTokens"));
|
||||
out.billable_characters = static_cast<int>(json_num(reply, "totalBillableCharacters"));
|
||||
out.raw = reply;
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<TokenInfo> Gemini::compute_tokens(
|
||||
const std::string& model, const std::vector<Content>& contents) const {
|
||||
Json reply = run(plan_compute_tokens(model, contents));
|
||||
std::vector<TokenInfo> out;
|
||||
const Json* infos = reply.find("tokensInfo");
|
||||
if (!infos || !infos->is_array()) return out;
|
||||
for (const Json& i : infos->as_array()) {
|
||||
if (out.size() >= ModelsCaps::kMaxContents) break;
|
||||
TokenInfo t;
|
||||
t.role = json_str(i, "role");
|
||||
if (const Json* toks = i.find("tokens"); toks && toks->is_array())
|
||||
for (const Json& x : toks->as_array()) {
|
||||
std::string raw;
|
||||
if (x.is_string() && security::crypto::Base64::decode(x.as_string(), raw))
|
||||
t.tokens.push_back(raw);
|
||||
}
|
||||
if (const Json* ids = i.find("tokenIds"); ids && ids->is_array())
|
||||
for (const Json& x : ids->as_array())
|
||||
if (x.is_string()) t.token_ids.push_back(x.as_string());
|
||||
out.push_back(std::move(t));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
void Gemini::validate_embed(const EmbedOptions& options) {
|
||||
if (!token_chars(options.task_type))
|
||||
throw GoogleError::invalid("task_type must be UPPER_SNAKE_CASE");
|
||||
if (options.output_dimensionality < 0 ||
|
||||
options.output_dimensionality > ModelsCaps::kMaxEmbedDimensions)
|
||||
throw GoogleError::invalid("output_dimensionality outside 0.." +
|
||||
std::to_string(ModelsCaps::kMaxEmbedDimensions));
|
||||
if (options.title.size() > 1000) throw GoogleError::invalid("title over 1000 characters");
|
||||
}
|
||||
|
||||
PlannedCall Gemini::plan_embed_content(const std::string& model,
|
||||
const std::string& text,
|
||||
const EmbedOptions& options) const {
|
||||
validate_embed(options);
|
||||
if (text.empty()) throw GoogleError::invalid("text is empty");
|
||||
if (text.size() > ModelsCaps::kMaxEmbedChars)
|
||||
throw GoogleError::invalid("text over " +
|
||||
std::to_string(ModelsCaps::kMaxEmbedChars) +
|
||||
" characters");
|
||||
Content content;
|
||||
content.parts.push_back(Part::of_text(text));
|
||||
Json body = Json::object();
|
||||
body.set("content", content.to_json());
|
||||
Json config = Json::object();
|
||||
if (!options.task_type.empty()) config.set("taskType", Json(options.task_type));
|
||||
if (!options.title.empty()) config.set("title", Json(options.title));
|
||||
if (options.output_dimensionality > 0)
|
||||
config.set("outputDimensionality", Json(options.output_dimensionality));
|
||||
if (options.auto_truncate) config.set("autoTruncate", Json(*options.auto_truncate));
|
||||
if (!config.as_object().empty()) body.set("embedContentConfig", config);
|
||||
return planner_.plan("gemini.embed_content", model_args(kPublisher, model), &body);
|
||||
}
|
||||
|
||||
Embedding Gemini::embed_content(const std::string& model, const std::string& text,
|
||||
const EmbedOptions& options) const {
|
||||
Json reply = run(plan_embed_content(model, text, options));
|
||||
Embedding e;
|
||||
if (const Json* emb = reply.find("embedding"); emb && emb->is_object())
|
||||
e = parse_vector(*emb, nullptr);
|
||||
if (const Json* u = reply.find("usageMetadata"); u && u->is_object())
|
||||
e.token_count = static_cast<int>(json_num(*u, "promptTokenCount"));
|
||||
if (const Json* t = reply.find("truncated"); t && t->is_bool()) e.truncated = t->as_bool();
|
||||
if (e.values.empty())
|
||||
throw GoogleError(502, "EMPTY_RESPONSE", "model returned no embedding",
|
||||
endpoint_.name);
|
||||
return e;
|
||||
}
|
||||
|
||||
PlannedCall Gemini::plan_embed(const std::string& model,
|
||||
const std::vector<std::string>& texts,
|
||||
const EmbedOptions& options) const {
|
||||
validate_embed(options);
|
||||
if (texts.empty()) throw GoogleError::invalid("no texts");
|
||||
if (texts.size() > ModelsCaps::kMaxEmbedBatch)
|
||||
throw GoogleError::invalid("more than " +
|
||||
std::to_string(ModelsCaps::kMaxEmbedBatch) + " texts");
|
||||
Json instances = Json::array();
|
||||
for (const std::string& t : texts) {
|
||||
if (t.empty()) throw GoogleError::invalid("text is empty");
|
||||
if (t.size() > ModelsCaps::kMaxEmbedChars)
|
||||
throw GoogleError::invalid("text over " +
|
||||
std::to_string(ModelsCaps::kMaxEmbedChars) +
|
||||
" characters");
|
||||
Json one = Json::object();
|
||||
one.set("content", Json(t));
|
||||
if (!options.task_type.empty()) one.set("task_type", Json(options.task_type));
|
||||
if (!options.title.empty()) one.set("title", Json(options.title));
|
||||
instances.push(one);
|
||||
}
|
||||
Json params = Json::object();
|
||||
if (options.output_dimensionality > 0)
|
||||
params.set("outputDimensionality", Json(options.output_dimensionality));
|
||||
if (options.auto_truncate) params.set("autoTruncate", Json(*options.auto_truncate));
|
||||
Json body = Json::object();
|
||||
body.set("instances", instances);
|
||||
if (!params.as_object().empty()) body.set("parameters", params);
|
||||
return planner_.plan("gemini.embed_predict", model_args(kPublisher, model), &body);
|
||||
}
|
||||
|
||||
std::vector<Embedding> Gemini::embed(const std::string& model,
|
||||
const std::vector<std::string>& texts,
|
||||
const EmbedOptions& options) const {
|
||||
Json reply = run(plan_embed(model, texts, options));
|
||||
std::vector<Embedding> out;
|
||||
if (const Json* preds = reply.find("predictions"); preds && preds->is_array())
|
||||
for (const Json& p : preds->as_array())
|
||||
if (const Json* emb = p.find("embeddings"); emb && emb->is_object())
|
||||
out.push_back(parse_vector(*emb, emb->find("statistics")));
|
||||
if (out.size() != texts.size())
|
||||
throw GoogleError(502, "BAD_RESPONSE",
|
||||
"embedding count does not match the request", endpoint_.name);
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,81 @@
|
||||
#pragma once
|
||||
|
||||
// Gemini on Vertex AI: generateContent, streamGenerateContent (?alt=sse),
|
||||
// countTokens, computeTokens, embedContent (project only) and the predict
|
||||
// based text embeddings. Every call has a plan_* twin that builds the exact
|
||||
// request without sending it.
|
||||
|
||||
#include <functional>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/gemini_types.h"
|
||||
#include "google/models/vertex_service.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct CountTokensResult {
|
||||
int total_tokens = 0;
|
||||
int billable_characters = 0;
|
||||
clients::body::Json raw;
|
||||
};
|
||||
|
||||
struct TokenInfo {
|
||||
std::string role;
|
||||
std::vector<std::string> tokens; // decoded token bytes
|
||||
std::vector<std::string> token_ids;
|
||||
};
|
||||
|
||||
struct EmbedOptions {
|
||||
std::string task_type; // e.g. RETRIEVAL_DOCUMENT; empty: model default
|
||||
std::string title;
|
||||
int output_dimensionality = 0; // 0: model default
|
||||
std::optional<bool> auto_truncate;
|
||||
};
|
||||
|
||||
struct Embedding {
|
||||
std::vector<double> values;
|
||||
int token_count = 0;
|
||||
bool truncated = false;
|
||||
};
|
||||
|
||||
class Gemini : public VertexService {
|
||||
public:
|
||||
Gemini(std::shared_ptr<GoogleClient> client, VertexTarget target);
|
||||
|
||||
GenerateResponse generate(const std::string& model,
|
||||
const GenerateRequest& request) const;
|
||||
void stream(const std::string& model, const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) const;
|
||||
CountTokensResult count_tokens(const std::string& model,
|
||||
const GenerateRequest& request) const;
|
||||
std::vector<TokenInfo> compute_tokens(const std::string& model,
|
||||
const std::vector<Content>& contents) const;
|
||||
// embedContent: project scoped only (Discovery has no express form).
|
||||
Embedding embed_content(const std::string& model, const std::string& text,
|
||||
const EmbedOptions& options = {}) const;
|
||||
// predict based text embeddings: a batch in, one embedding per text out.
|
||||
std::vector<Embedding> embed(const std::string& model,
|
||||
const std::vector<std::string>& texts,
|
||||
const EmbedOptions& options = {}) const;
|
||||
|
||||
PlannedCall plan_generate(const std::string& model, const GenerateRequest& r) const;
|
||||
PlannedCall plan_stream(const std::string& model, const GenerateRequest& r) const;
|
||||
PlannedCall plan_count_tokens(const std::string& model, const GenerateRequest& r) const;
|
||||
PlannedCall plan_compute_tokens(const std::string& model,
|
||||
const std::vector<Content>& contents) const;
|
||||
PlannedCall plan_embed_content(const std::string& model, const std::string& text,
|
||||
const EmbedOptions& o) const;
|
||||
PlannedCall plan_embed(const std::string& model,
|
||||
const std::vector<std::string>& texts,
|
||||
const EmbedOptions& o) const;
|
||||
|
||||
// Throws GoogleError::invalid on empty or oversized requests.
|
||||
static void validate(const GenerateRequest& request);
|
||||
|
||||
private:
|
||||
static void validate_embed(const EmbedOptions& options);
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,24 @@
|
||||
#pragma once
|
||||
|
||||
// Tolerant readers for response documents (free functions: Service's own are
|
||||
// protected and structs need them too).
|
||||
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
inline std::string json_str(const clients::body::Json& doc, const std::string& key,
|
||||
const std::string& fallback = "") {
|
||||
const auto* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_string() ? v->as_string() : fallback;
|
||||
}
|
||||
|
||||
inline double json_num(const clients::body::Json& doc, const std::string& key,
|
||||
double fallback = 0.0) {
|
||||
const auto* v = doc.is_object() ? doc.find(key) : nullptr;
|
||||
return v && v->is_number() ? v->as_number() : fallback;
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,194 @@
|
||||
#include "google/models/media.h"
|
||||
|
||||
#include <thread>
|
||||
|
||||
#include "google/errors.h"
|
||||
#include "google/models/caps.h"
|
||||
#include "google/models/open_models.h"
|
||||
#include "security/crypto/Base64.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
namespace {
|
||||
bool simple_token(const std::string& s, std::size_t max) {
|
||||
if (s.size() > max) return false;
|
||||
for (char c : s)
|
||||
if (!((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') ||
|
||||
c == ':' || c == '_' || c == '-' || c == '.'))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
MediaGenerator::MediaGenerator(std::shared_ptr<GoogleClient> client,
|
||||
VertexTarget target)
|
||||
: VertexService(std::move(client), std::move(target), "aiplatform") {}
|
||||
|
||||
void MediaGenerator::check_prompt(const std::string& prompt) {
|
||||
if (prompt.empty()) throw GoogleError::invalid("prompt is empty");
|
||||
if (prompt.size() > ModelsCaps::kMaxPromptChars)
|
||||
throw GoogleError::invalid("prompt over " +
|
||||
std::to_string(ModelsCaps::kMaxPromptChars) +
|
||||
" characters");
|
||||
}
|
||||
|
||||
std::vector<MediaItem> MediaGenerator::items_from(const Json& list) {
|
||||
std::vector<MediaItem> out;
|
||||
if (!list.is_array()) return out;
|
||||
for (const Json& e : list.as_array()) {
|
||||
if (out.size() >= ModelsCaps::kMaxMediaItems) break;
|
||||
MediaItem item;
|
||||
item.mime = json_str(e, "mimeType");
|
||||
item.uri = json_str(e, "gcsUri");
|
||||
std::string b64 = json_str(e, "bytesBase64Encoded");
|
||||
if (!b64.empty()) {
|
||||
if (b64.size() > ModelsCaps::kMaxMediaBytes * 4 / 3 + 8)
|
||||
throw GoogleError(502, "BAD_RESPONSE", "media item over the size cap");
|
||||
if (!security::crypto::Base64::decode(b64, item.bytes))
|
||||
throw GoogleError(502, "BAD_RESPONSE", "media item is not valid base64");
|
||||
}
|
||||
if (item.bytes.empty() && item.uri.empty()) continue;
|
||||
out.push_back(std::move(item));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<MediaItem> MediaGenerator::run_predict(const PlannedCall& call) const {
|
||||
Json reply = run(call);
|
||||
const Json* preds = reply.find("predictions");
|
||||
std::vector<MediaItem> out = items_from(preds ? *preds : Json());
|
||||
if (out.empty())
|
||||
throw GoogleError(502, "EMPTY_RESPONSE", "model returned no media", endpoint_.name);
|
||||
return out;
|
||||
}
|
||||
|
||||
PlannedCall MediaGenerator::plan_image(const std::string& model, const ImageRequest& r) const {
|
||||
check_prompt(r.prompt);
|
||||
if (r.count < 1 || r.count > ModelsCaps::kMaxImages)
|
||||
throw GoogleError::invalid("count outside 1.." + std::to_string(ModelsCaps::kMaxImages));
|
||||
if (!simple_token(r.aspect_ratio, 8)) throw GoogleError::invalid("bad aspect_ratio");
|
||||
Json instance = Json::object();
|
||||
instance.set("prompt", Json(r.prompt));
|
||||
Json params = Json::object();
|
||||
params.set("sampleCount", Json(r.count));
|
||||
if (!r.aspect_ratio.empty()) params.set("aspectRatio", Json(r.aspect_ratio));
|
||||
if (!r.negative_prompt.empty()) {
|
||||
if (r.negative_prompt.size() > ModelsCaps::kMaxPromptChars)
|
||||
throw GoogleError::invalid("negative_prompt too long");
|
||||
params.set("negativePrompt", Json(r.negative_prompt));
|
||||
}
|
||||
Json instances = Json::array();
|
||||
instances.push(instance);
|
||||
Json body = Json::object();
|
||||
body.set("instances", instances);
|
||||
body.set("parameters", params);
|
||||
return planner_.plan("media.predict", model_args("google", model), &body);
|
||||
}
|
||||
|
||||
PlannedCall MediaGenerator::plan_music(const std::string& model, const MusicRequest& r) const {
|
||||
check_prompt(r.prompt);
|
||||
if (r.count < 1 || r.count > ModelsCaps::kMaxMusicSamples)
|
||||
throw GoogleError::invalid("count outside 1.." + std::to_string(ModelsCaps::kMaxMusicSamples));
|
||||
if (r.seed && r.count != 1)
|
||||
throw GoogleError::invalid("seed and count > 1 cannot be combined");
|
||||
if (r.negative_prompt.size() > ModelsCaps::kMaxPromptChars)
|
||||
throw GoogleError::invalid("negative_prompt too long");
|
||||
Json instance = Json::object();
|
||||
instance.set("prompt", Json(r.prompt));
|
||||
if (!r.negative_prompt.empty()) instance.set("negative_prompt", Json(r.negative_prompt));
|
||||
if (r.seed) instance.set("seed", Json(*r.seed));
|
||||
Json params = Json::object();
|
||||
if (!r.seed) params.set("sample_count", Json(r.count));
|
||||
Json instances = Json::array();
|
||||
instances.push(instance);
|
||||
Json body = Json::object();
|
||||
body.set("instances", instances);
|
||||
body.set("parameters", params);
|
||||
return planner_.plan("media.predict", model_args("google", model), &body);
|
||||
}
|
||||
|
||||
PlannedCall MediaGenerator::plan_start_video(const std::string& model, const VideoRequest& r) const {
|
||||
check_prompt(r.prompt);
|
||||
if (r.count < 1 || r.count > ModelsCaps::kMaxVideoSamples)
|
||||
throw GoogleError::invalid("count outside 1.." + std::to_string(ModelsCaps::kMaxVideoSamples));
|
||||
if (r.duration_seconds < 0 || r.duration_seconds > ModelsCaps::kMaxVideoSeconds)
|
||||
throw GoogleError::invalid("duration_seconds outside 0.." +
|
||||
std::to_string(ModelsCaps::kMaxVideoSeconds));
|
||||
if (!simple_token(r.aspect_ratio, 8)) throw GoogleError::invalid("bad aspect_ratio");
|
||||
if (!r.storage_uri.empty() &&
|
||||
(r.storage_uri.rfind("gs://", 0) != 0 || r.storage_uri.size() > 512))
|
||||
throw GoogleError::invalid("storage_uri must be a gs:// uri");
|
||||
Json instance = Json::object();
|
||||
instance.set("prompt", Json(r.prompt));
|
||||
Json instances = Json::array();
|
||||
instances.push(instance);
|
||||
Json params = Json::object();
|
||||
params.set("sampleCount", Json(r.count));
|
||||
if (r.duration_seconds > 0) params.set("durationSeconds", Json(r.duration_seconds));
|
||||
if (!r.aspect_ratio.empty()) params.set("aspectRatio", Json(r.aspect_ratio));
|
||||
if (!r.storage_uri.empty()) params.set("storageUri", Json(r.storage_uri));
|
||||
Json body = Json::object();
|
||||
body.set("instances", instances);
|
||||
body.set("parameters", params);
|
||||
return planner_.plan("media.predict_long_running", model_args("google", model), &body);
|
||||
}
|
||||
|
||||
PlannedCall MediaGenerator::plan_video_status(const std::string& model,
|
||||
const std::string& operation_name) const {
|
||||
OpenModels::check_operation_name(operation_name);
|
||||
Json body = Json::object();
|
||||
body.set("operationName", Json(operation_name));
|
||||
return planner_.plan("media.fetch_predict_operation", model_args("google", model), &body);
|
||||
}
|
||||
|
||||
std::vector<MediaItem> MediaGenerator::generate_image(const std::string& model,
|
||||
const ImageRequest& r) const {
|
||||
return run_predict(plan_image(model, r));
|
||||
}
|
||||
|
||||
std::vector<MediaItem> MediaGenerator::generate_music(const std::string& model,
|
||||
const MusicRequest& r) const {
|
||||
return run_predict(plan_music(model, r));
|
||||
}
|
||||
|
||||
std::string MediaGenerator::start_video(const std::string& model, const VideoRequest& r) const {
|
||||
std::string name = json_str(run(plan_start_video(model, r)), "name");
|
||||
if (name.empty())
|
||||
throw GoogleError(502, "BAD_RESPONSE", "no operation name returned", endpoint_.name);
|
||||
return name;
|
||||
}
|
||||
|
||||
Operation MediaGenerator::video_status(const std::string& model,
|
||||
const std::string& operation_name) const {
|
||||
return Operation::from_json(run(plan_video_status(model, operation_name)));
|
||||
}
|
||||
|
||||
std::vector<MediaItem> MediaGenerator::wait_video(const std::string& model,
|
||||
const std::string& operation_name,
|
||||
const Sleeper& sleeper) const {
|
||||
OpenModels::check_operation_name(operation_name);
|
||||
Sleeper sleep = sleeper ? sleeper : [](std::chrono::milliseconds d) {
|
||||
std::this_thread::sleep_for(d);
|
||||
};
|
||||
for (std::size_t i = 0; i < ModelsCaps::kMaxVideoPolls; ++i) {
|
||||
Operation op = video_status(model, operation_name);
|
||||
if (op.done) {
|
||||
if (op.failed())
|
||||
throw GoogleError(502, "OPERATION_FAILED", op.error_message, endpoint_.name);
|
||||
const Json* videos = op.response.find("videos");
|
||||
std::vector<MediaItem> out = items_from(videos ? *videos : Json());
|
||||
if (out.empty())
|
||||
throw GoogleError(502, "EMPTY_RESPONSE", "operation finished without videos",
|
||||
endpoint_.name);
|
||||
return out;
|
||||
}
|
||||
sleep(ModelsCaps::kVideoPollInterval);
|
||||
}
|
||||
throw GoogleError(504, "DEADLINE_EXCEEDED",
|
||||
"video still rendering at the wait cap: " + operation_name,
|
||||
endpoint_.name);
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,78 @@
|
||||
#pragma once
|
||||
|
||||
// Media generation on Vertex AI publisher models: images (Imagen, predict),
|
||||
// music (Lyria, predict) and video (Veo, predictLongRunning plus
|
||||
// fetchPredictOperation polling, bounded).
|
||||
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/operation.h"
|
||||
#include "google/models/vertex_service.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct MediaItem {
|
||||
std::string mime;
|
||||
std::string bytes; // decoded; empty when the result lives at `uri`
|
||||
std::string uri; // gs:// location when the request asked for storage
|
||||
};
|
||||
|
||||
struct ImageRequest {
|
||||
std::string prompt;
|
||||
int count = 1; // 1..kMaxImages
|
||||
std::string aspect_ratio; // "", "1:1", "16:9", ...
|
||||
std::string negative_prompt;
|
||||
};
|
||||
|
||||
struct MusicRequest {
|
||||
std::string prompt;
|
||||
std::string negative_prompt;
|
||||
int count = 1; // 1..kMaxMusicSamples
|
||||
std::optional<int> seed;
|
||||
};
|
||||
|
||||
struct VideoRequest {
|
||||
std::string prompt;
|
||||
int count = 1; // 1..kMaxVideoSamples
|
||||
int duration_seconds = 0; // 0: model default; else 1..kMaxVideoSeconds
|
||||
std::string aspect_ratio; // "", "16:9", "9:16"
|
||||
std::string storage_uri; // optional gs:// output prefix
|
||||
};
|
||||
|
||||
class MediaGenerator : public VertexService {
|
||||
public:
|
||||
MediaGenerator(std::shared_ptr<GoogleClient> client, VertexTarget target);
|
||||
|
||||
std::vector<MediaItem> generate_image(const std::string& model,
|
||||
const ImageRequest& request) const;
|
||||
std::vector<MediaItem> generate_music(const std::string& model,
|
||||
const MusicRequest& request) const;
|
||||
|
||||
// Starts a video job; returns the operation name.
|
||||
std::string start_video(const std::string& model,
|
||||
const VideoRequest& request) const;
|
||||
// One poll (POST :fetchPredictOperation).
|
||||
Operation video_status(const std::string& model,
|
||||
const std::string& operation_name) const;
|
||||
// Polls until done within kMaxVideoPolls. Throws DEADLINE_EXCEEDED at the
|
||||
// cap and OPERATION_FAILED when the job failed. sleeper injectable.
|
||||
std::vector<MediaItem> wait_video(const std::string& model,
|
||||
const std::string& operation_name,
|
||||
const Sleeper& sleeper = nullptr) const;
|
||||
|
||||
PlannedCall plan_image(const std::string& model, const ImageRequest& r) const;
|
||||
PlannedCall plan_music(const std::string& model, const MusicRequest& r) const;
|
||||
PlannedCall plan_start_video(const std::string& model, const VideoRequest& r) const;
|
||||
PlannedCall plan_video_status(const std::string& model,
|
||||
const std::string& operation_name) const;
|
||||
|
||||
static std::vector<MediaItem> items_from(const clients::body::Json& list);
|
||||
|
||||
private:
|
||||
static void check_prompt(const std::string& prompt);
|
||||
std::vector<MediaItem> run_predict(const PlannedCall& call) const;
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,172 @@
|
||||
#include "google/models/model_garden.h"
|
||||
|
||||
#include "clients/url.h"
|
||||
#include "google/errors.h"
|
||||
#include "google/models/caps.h"
|
||||
#include "google/operation.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
std::string PublisherModel::publisher() const {
|
||||
std::size_t a = name.find('/');
|
||||
std::size_t b = a == std::string::npos ? a : name.find('/', a + 1);
|
||||
return b == std::string::npos ? "" : name.substr(a + 1, b - a - 1);
|
||||
}
|
||||
|
||||
std::string PublisherModel::model_id() const {
|
||||
std::size_t p = name.rfind('/');
|
||||
return p == std::string::npos ? name : name.substr(p + 1);
|
||||
}
|
||||
|
||||
PublisherModel PublisherModel::from_json(const Json& doc) {
|
||||
PublisherModel m;
|
||||
m.name = json_str(doc, "name");
|
||||
m.version_id = json_str(doc, "versionId");
|
||||
m.open_source_category = json_str(doc, "openSourceCategory");
|
||||
m.launch_stage = json_str(doc, "launchStage");
|
||||
m.version_state = json_str(doc, "versionState");
|
||||
m.publisher_model_template = json_str(doc, "publisherModelTemplate");
|
||||
if (const Json* a = doc.find("supportedActions"); a && a->is_object())
|
||||
for (const auto& [k, v] : a->as_object()) {
|
||||
(void)v;
|
||||
m.supported_actions.push_back(k);
|
||||
}
|
||||
m.raw = doc;
|
||||
return m;
|
||||
}
|
||||
|
||||
ModelGarden::ModelGarden(std::shared_ptr<GoogleClient> client, VertexTarget target)
|
||||
: VertexService(std::move(client), std::move(target), "aiplatform") {}
|
||||
|
||||
namespace {
|
||||
bool order_chars(const std::string& s) {
|
||||
if (s.size() > 64) return false;
|
||||
for (char c : s)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
|
||||
c == '_' || c == ' '))
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
PlannedCall ModelGarden::plan_list_page(const std::string& publisher,
|
||||
const ListModelsOptions& o,
|
||||
const std::string& page_token) const {
|
||||
if (o.page_size < 1 || o.page_size > ModelsCaps::kMaxPageSize)
|
||||
throw GoogleError::invalid("page_size outside 1.." +
|
||||
std::to_string(ModelsCaps::kMaxPageSize));
|
||||
if (o.max_pages < 1 || o.max_pages > ModelsCaps::kMaxListPages)
|
||||
throw GoogleError::invalid("max_pages outside 1.." +
|
||||
std::to_string(ModelsCaps::kMaxListPages));
|
||||
if (o.filter.size() > 256) throw GoogleError::invalid("filter over 256 characters");
|
||||
if (!order_chars(o.order_by)) throw GoogleError::invalid("bad order_by");
|
||||
std::string q = "pageSize=" + std::to_string(o.page_size);
|
||||
if (o.full_view) q += "&view=PUBLISHER_MODEL_VIEW_FULL";
|
||||
if (o.all_versions) q += "&listAllVersions=true";
|
||||
if (!o.filter.empty()) q += "&filter=" + clients::Url::encode_component(o.filter);
|
||||
if (!o.order_by.empty()) q += "&orderBy=" + clients::Url::encode_component(o.order_by);
|
||||
if (!page_token.empty()) q += "&pageToken=" + clients::Url::encode_component(page_token);
|
||||
return planner_.plan("garden.list", {{"publisher", publisher}}, nullptr, q);
|
||||
}
|
||||
|
||||
std::vector<PublisherModel> ModelGarden::list(const std::string& publisher,
|
||||
const ListModelsOptions& o) const {
|
||||
std::vector<PublisherModel> out;
|
||||
std::string token;
|
||||
for (std::size_t page = 0; page < o.max_pages; ++page) {
|
||||
Json reply = run(plan_list_page(publisher, o, token));
|
||||
if (const Json* items = reply.find("publisherModels"); items && items->is_array())
|
||||
for (const Json& item : items->as_array()) {
|
||||
if (out.size() >= ModelsCaps::kMaxModelsReturned) return out;
|
||||
out.push_back(PublisherModel::from_json(item));
|
||||
}
|
||||
token = json_str(reply, "nextPageToken");
|
||||
if (token.empty()) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
PlannedCall ModelGarden::plan_get(const std::string& publisher,
|
||||
const std::string& model, bool full_view) const {
|
||||
return planner_.plan("garden.get", model_args(publisher, model), nullptr,
|
||||
full_view ? "view=PUBLISHER_MODEL_VIEW_FULL" : "");
|
||||
}
|
||||
|
||||
PublisherModel ModelGarden::get(const std::string& publisher,
|
||||
const std::string& model, bool full_view) const {
|
||||
return PublisherModel::from_json(run(plan_get(publisher, model, full_view)));
|
||||
}
|
||||
|
||||
PlannedCall ModelGarden::plan_fetch_config(const std::string& publisher,
|
||||
const std::string& model) const {
|
||||
return planner_.plan("garden.fetch_config", model_args(publisher, model));
|
||||
}
|
||||
|
||||
Json ModelGarden::fetch_config(const std::string& publisher,
|
||||
const std::string& model) const {
|
||||
return run(plan_fetch_config(publisher, model));
|
||||
}
|
||||
|
||||
PlannedCall ModelGarden::plan_eula(const char* call_id,
|
||||
const std::string& publisher_model) const {
|
||||
// publishers/{publisher}/models/{model}[@version]
|
||||
if (publisher_model.rfind("publishers/", 0) != 0 || publisher_model.size() > 256 ||
|
||||
publisher_model.find("..") != std::string::npos)
|
||||
throw GoogleError::invalid("publisher_model must look like publishers/{p}/models/{m}");
|
||||
for (char c : publisher_model)
|
||||
if (!((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') ||
|
||||
c == '/' || c == '-' || c == '_' || c == '.' || c == '@'))
|
||||
throw GoogleError::invalid("publisher_model has invalid characters");
|
||||
Json body = Json::object();
|
||||
body.set("publisherModel", Json(publisher_model));
|
||||
return planner_.plan(call_id, {}, &body);
|
||||
}
|
||||
|
||||
EulaStatus ModelGarden::check_eula(const std::string& publisher_model) const {
|
||||
Json r = run(plan_eula("garden.eula_check", publisher_model));
|
||||
EulaStatus s;
|
||||
if (const Json* a = r.find("publisherModelEulaAcked"); a && a->is_bool())
|
||||
s.acked = a->as_bool();
|
||||
s.project_number = json_str(r, "projectNumber");
|
||||
s.publisher_model = json_str(r, "publisherModel");
|
||||
return s;
|
||||
}
|
||||
|
||||
EulaStatus ModelGarden::accept_eula(const std::string& publisher_model,
|
||||
bool confirm) const {
|
||||
if (!confirm) throw GoogleError::invalid("accept_eula needs confirm=true");
|
||||
Json r = run(plan_eula("garden.eula_accept", publisher_model));
|
||||
EulaStatus s;
|
||||
if (const Json* a = r.find("publisherModelEulaAcked"); a && a->is_bool())
|
||||
s.acked = a->as_bool();
|
||||
s.project_number = json_str(r, "projectNumber");
|
||||
s.publisher_model = json_str(r, "publisherModel");
|
||||
return s;
|
||||
}
|
||||
|
||||
PlannedCall ModelGarden::plan_deploy(const DeployOptions& o) const {
|
||||
if (!o.confirm) throw GoogleError::invalid("deploy needs confirm=true (it creates billable resources)");
|
||||
if (o.publisher_model.empty()) throw GoogleError::invalid("publisher_model is empty");
|
||||
plan_eula("garden.eula_check", o.publisher_model); // same id validation
|
||||
if (o.model_display_name.size() > 128 || o.endpoint_display_name.size() > 128)
|
||||
throw GoogleError::invalid("display name over 128 characters");
|
||||
Json body = Json::object();
|
||||
body.set("model", Json(o.publisher_model));
|
||||
if (!o.model_display_name.empty()) body.set("modelDisplayName", Json(o.model_display_name));
|
||||
if (!o.endpoint_display_name.empty())
|
||||
body.set("endpointDisplayName", Json(o.endpoint_display_name));
|
||||
if (o.accept_eula) body.set("acceptEula", Json(true));
|
||||
return planner_.plan("garden.deploy", {}, &body);
|
||||
}
|
||||
|
||||
std::string ModelGarden::deploy(const DeployOptions& options) const {
|
||||
Json r = run(plan_deploy(options));
|
||||
std::string name = json_str(r, "name");
|
||||
if (name.empty())
|
||||
throw GoogleError(502, "BAD_RESPONSE", "no operation name returned", endpoint_.name);
|
||||
return name;
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,78 @@
|
||||
#pragma once
|
||||
|
||||
// Model Garden: publisher models (list: v1beta1, get: v1), publisher model
|
||||
// config, EULA check and accept, and deploy. Mutating and costly calls
|
||||
// (accept_eula, deploy) refuse to run unless the caller sets confirm.
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/models/vertex_service.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct PublisherModel {
|
||||
std::string name; // publishers/{publisher}/models/{model}
|
||||
std::string version_id;
|
||||
std::string open_source_category;
|
||||
std::string launch_stage;
|
||||
std::string version_state;
|
||||
std::string publisher_model_template;
|
||||
std::vector<std::string> supported_actions; // keys of supportedActions
|
||||
clients::body::Json raw;
|
||||
|
||||
std::string publisher() const;
|
||||
std::string model_id() const;
|
||||
static PublisherModel from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct ListModelsOptions {
|
||||
int page_size = 50; // 1..ModelsCaps::kMaxPageSize
|
||||
std::size_t max_pages = 5; // 1..ModelsCaps::kMaxListPages
|
||||
std::string filter; // passed through, URL encoded, max 256 chars
|
||||
std::string order_by; // max 64 chars, [A-Za-z0-9_ ]
|
||||
bool full_view = false;
|
||||
bool all_versions = false;
|
||||
};
|
||||
|
||||
struct EulaStatus {
|
||||
bool acked = false;
|
||||
std::string project_number;
|
||||
std::string publisher_model;
|
||||
};
|
||||
|
||||
struct DeployOptions {
|
||||
std::string publisher_model; // publishers/{publisher}/models/{model}@version
|
||||
std::string model_display_name;
|
||||
std::string endpoint_display_name;
|
||||
bool accept_eula = false;
|
||||
bool confirm = false; // must be true: deploy creates billable resources
|
||||
};
|
||||
|
||||
class ModelGarden : public VertexService {
|
||||
public:
|
||||
ModelGarden(std::shared_ptr<GoogleClient> client, VertexTarget target);
|
||||
|
||||
std::vector<PublisherModel> list(const std::string& publisher,
|
||||
const ListModelsOptions& options = {}) const;
|
||||
PublisherModel get(const std::string& publisher, const std::string& model,
|
||||
bool full_view = false) const;
|
||||
clients::body::Json fetch_config(const std::string& publisher,
|
||||
const std::string& model) const;
|
||||
EulaStatus check_eula(const std::string& publisher_model) const;
|
||||
EulaStatus accept_eula(const std::string& publisher_model, bool confirm) const;
|
||||
// Returns the long-running operation name; poll it with wait_operation.
|
||||
std::string deploy(const DeployOptions& options) const;
|
||||
|
||||
PlannedCall plan_list_page(const std::string& publisher,
|
||||
const ListModelsOptions& options,
|
||||
const std::string& page_token = "") const;
|
||||
PlannedCall plan_get(const std::string& publisher, const std::string& model,
|
||||
bool full_view) const;
|
||||
PlannedCall plan_fetch_config(const std::string& publisher,
|
||||
const std::string& model) const;
|
||||
PlannedCall plan_eula(const char* call_id, const std::string& publisher_model) const;
|
||||
PlannedCall plan_deploy(const DeployOptions& options) const;
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,235 @@
|
||||
#include "google/models/open_models.h"
|
||||
|
||||
#include "clients/url.h"
|
||||
#include "google/errors.h"
|
||||
#include "google/models/caps.h"
|
||||
#include "google/models/gemini.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
using clients::body::Json;
|
||||
|
||||
EndpointInfo EndpointInfo::from_json(const Json& doc) {
|
||||
EndpointInfo e;
|
||||
e.name = json_str(doc, "name");
|
||||
std::size_t p = e.name.rfind('/');
|
||||
e.id = p == std::string::npos ? e.name : e.name.substr(p + 1);
|
||||
e.display_name = json_str(doc, "displayName");
|
||||
e.dedicated_endpoint_dns = json_str(doc, "dedicatedEndpointDns");
|
||||
if (const Json* d = doc.find("deployedModels"); d && d->is_array())
|
||||
e.deployed_models = d->as_array().size();
|
||||
e.raw = doc;
|
||||
return e;
|
||||
}
|
||||
|
||||
OpenModels::OpenModels(std::shared_ptr<GoogleClient> client, VertexTarget target)
|
||||
: VertexService(std::move(client), std::move(target), "aiplatform") {}
|
||||
|
||||
void OpenModels::check_operation_name(const std::string& name) {
|
||||
if (name.empty() || name.size() > ModelsCaps::kMaxOperationName ||
|
||||
name.find("..") != std::string::npos)
|
||||
throw GoogleError::invalid("bad operation name");
|
||||
for (char c : name)
|
||||
if (!((c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') ||
|
||||
c == '/' || c == '_' || c == '-' || c == '.'))
|
||||
throw GoogleError::invalid("bad operation name");
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_list_page(std::size_t max_pages, const std::string& filter,
|
||||
const std::string& page_token) const {
|
||||
if (max_pages < 1 || max_pages > ModelsCaps::kMaxListPages)
|
||||
throw GoogleError::invalid("max_pages outside 1.." +
|
||||
std::to_string(ModelsCaps::kMaxListPages));
|
||||
if (filter.size() > 256) throw GoogleError::invalid("filter over 256 characters");
|
||||
std::string q = "pageSize=" + std::to_string(ModelsCaps::kMaxPageSize);
|
||||
if (!filter.empty()) q += "&filter=" + clients::Url::encode_component(filter);
|
||||
if (!page_token.empty()) q += "&pageToken=" + clients::Url::encode_component(page_token);
|
||||
return planner_.plan("endpoint.list", {}, nullptr, q);
|
||||
}
|
||||
|
||||
std::vector<EndpointInfo> OpenModels::list_endpoints(std::size_t max_pages,
|
||||
const std::string& filter) const {
|
||||
std::vector<EndpointInfo> out;
|
||||
std::string token;
|
||||
for (std::size_t page = 0; page < max_pages; ++page) {
|
||||
Json reply = run(plan_list_page(max_pages, filter, token));
|
||||
if (const Json* items = reply.find("endpoints"); items && items->is_array())
|
||||
for (const Json& item : items->as_array()) {
|
||||
if (out.size() >= ModelsCaps::kMaxModelsReturned) return out;
|
||||
out.push_back(EndpointInfo::from_json(item));
|
||||
}
|
||||
token = json_str(reply, "nextPageToken");
|
||||
if (token.empty()) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_get(const std::string& endpoint_id) const {
|
||||
return planner_.plan("endpoint.get", {{"endpoint", endpoint_id}});
|
||||
}
|
||||
|
||||
EndpointInfo OpenModels::get_endpoint(const std::string& endpoint_id) const {
|
||||
return EndpointInfo::from_json(run(plan_get(endpoint_id)));
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_predict(const std::string& endpoint_id,
|
||||
const Json& instances,
|
||||
const Json& parameters) const {
|
||||
if (!instances.is_array() || instances.as_array().empty())
|
||||
throw GoogleError::invalid("instances must be a non-empty array");
|
||||
if (instances.as_array().size() > ModelsCaps::kMaxInstances)
|
||||
throw GoogleError::invalid("more than " +
|
||||
std::to_string(ModelsCaps::kMaxInstances) + " instances");
|
||||
if (instances.dump().size() > ModelsCaps::kMaxInstanceChars)
|
||||
throw GoogleError::invalid("instances over " +
|
||||
std::to_string(ModelsCaps::kMaxInstanceChars) + " characters");
|
||||
Json body = Json::object();
|
||||
body.set("instances", instances);
|
||||
if (parameters.is_object()) body.set("parameters", parameters);
|
||||
return planner_.plan("endpoint.predict", {{"endpoint", endpoint_id}}, &body);
|
||||
}
|
||||
|
||||
PredictResult OpenModels::predict(const std::string& endpoint_id,
|
||||
const Json& instances,
|
||||
const Json& parameters) const {
|
||||
Json reply = run(plan_predict(endpoint_id, instances, parameters));
|
||||
PredictResult r;
|
||||
if (const Json* p = reply.find("predictions")) r.predictions = *p;
|
||||
r.deployed_model_id = json_str(reply, "deployedModelId");
|
||||
r.model = json_str(reply, "model");
|
||||
r.model_display_name = json_str(reply, "modelDisplayName");
|
||||
r.model_version_id = json_str(reply, "modelVersionId");
|
||||
if (!r.predictions.is_array())
|
||||
throw GoogleError(502, "BAD_RESPONSE", "endpoint returned no predictions",
|
||||
endpoint_.name);
|
||||
return r;
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_raw(const char* call_id, const std::string& endpoint_id,
|
||||
const Json& body) const {
|
||||
if (!body.is_object()) throw GoogleError::invalid("body must be a JSON object");
|
||||
if (body.dump().size() > ModelsCaps::kMaxInstanceChars)
|
||||
throw GoogleError::invalid("body over " +
|
||||
std::to_string(ModelsCaps::kMaxInstanceChars) + " characters");
|
||||
return planner_.plan(call_id, {{"endpoint", endpoint_id}}, &body);
|
||||
}
|
||||
|
||||
Json OpenModels::raw_predict(const std::string& endpoint_id, const Json& body) const {
|
||||
return run(plan_raw("endpoint.raw_predict", endpoint_id, body));
|
||||
}
|
||||
|
||||
void OpenModels::stream_raw_predict(
|
||||
const std::string& endpoint_id, const Json& body,
|
||||
const std::function<bool(const Json&)>& sink) const {
|
||||
run_stream(plan_raw("endpoint.stream_raw_predict", endpoint_id, body), sink);
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_server_streaming(const std::string& endpoint_id,
|
||||
const Json& inputs,
|
||||
const Json& parameters) const {
|
||||
if (!inputs.is_array() || inputs.as_array().empty() ||
|
||||
inputs.as_array().size() > ModelsCaps::kMaxInstances)
|
||||
throw GoogleError::invalid("inputs must be an array of 1.." +
|
||||
std::to_string(ModelsCaps::kMaxInstances) + " items");
|
||||
if (inputs.dump().size() > ModelsCaps::kMaxInstanceChars)
|
||||
throw GoogleError::invalid("inputs too large");
|
||||
Json body = Json::object();
|
||||
body.set("inputs", inputs);
|
||||
if (parameters.is_object()) body.set("parameters", parameters);
|
||||
return planner_.plan("endpoint.server_streaming_predict", {{"endpoint", endpoint_id}}, &body);
|
||||
}
|
||||
|
||||
void OpenModels::server_streaming_predict(
|
||||
const std::string& endpoint_id, const Json& inputs, const Json& parameters,
|
||||
const std::function<bool(const Json&)>& sink) const {
|
||||
run_stream(plan_server_streaming(endpoint_id, inputs, parameters), sink);
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_generate(const char* call_id,
|
||||
const std::string& endpoint_id,
|
||||
const GenerateRequest& request) const {
|
||||
Gemini::validate(request);
|
||||
Json full = request.to_json();
|
||||
Json body = Json::object();
|
||||
// Discovery lists these four for the endpoint forms.
|
||||
for (const char* key : {"contents", "systemInstruction", "tools", "generationConfig"})
|
||||
if (const Json* v = full.find(key)) body.set(key, *v);
|
||||
return planner_.plan(call_id, {{"endpoint", endpoint_id}}, &body,
|
||||
std::string(call_id) == "endpoint.stream_generate_content" ? "alt=sse" : "");
|
||||
}
|
||||
|
||||
GenerateResponse OpenModels::generate_content(const std::string& endpoint_id,
|
||||
const GenerateRequest& request) const {
|
||||
return GenerateResponse::from_json(
|
||||
run(plan_generate("endpoint.generate_content", endpoint_id, request)));
|
||||
}
|
||||
|
||||
void OpenModels::stream_generate_content(
|
||||
const std::string& endpoint_id, const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) const {
|
||||
run_stream(plan_generate("endpoint.stream_generate_content", endpoint_id, request),
|
||||
[&](const Json& e) { return on_chunk(GenerateResponse::from_json(e)); });
|
||||
}
|
||||
|
||||
int OpenModels::count_tokens(const std::string& endpoint_id,
|
||||
const GenerateRequest& request) const {
|
||||
return static_cast<int>(json_num(
|
||||
run(plan_generate("endpoint.count_tokens", endpoint_id, request)), "totalTokens"));
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_long_running(const std::string& endpoint_id,
|
||||
const Json& instances,
|
||||
const Json& parameters) const {
|
||||
PlannedCall p = plan_predict(endpoint_id, instances, parameters);
|
||||
return planner_.plan("endpoint.predict_long_running", {{"endpoint", endpoint_id}}, &p.body);
|
||||
}
|
||||
|
||||
std::string OpenModels::predict_long_running(const std::string& endpoint_id,
|
||||
const Json& instances,
|
||||
const Json& parameters) const {
|
||||
Json r = run(plan_long_running(endpoint_id, instances, parameters));
|
||||
std::string name = json_str(r, "name");
|
||||
if (name.empty())
|
||||
throw GoogleError(502, "BAD_RESPONSE", "no operation name returned", endpoint_.name);
|
||||
return name;
|
||||
}
|
||||
|
||||
PlannedCall OpenModels::plan_fetch_operation(const std::string& endpoint_id,
|
||||
const std::string& operation_name) const {
|
||||
check_operation_name(operation_name);
|
||||
Json body = Json::object();
|
||||
body.set("operationName", Json(operation_name));
|
||||
return planner_.plan("endpoint.fetch_predict_operation", {{"endpoint", endpoint_id}}, &body);
|
||||
}
|
||||
|
||||
Operation OpenModels::fetch_predict_operation(const std::string& endpoint_id,
|
||||
const std::string& operation_name) const {
|
||||
return Operation::from_json(run(plan_fetch_operation(endpoint_id, operation_name)));
|
||||
}
|
||||
|
||||
Json OpenModels::invoke_publisher(const std::string& publisher, const std::string& model,
|
||||
const std::string& invoke_id, const Json& body) const {
|
||||
if (!body.is_object()) throw GoogleError::invalid("body must be a JSON object");
|
||||
CallPlanner::Args args = model_args(publisher, model);
|
||||
args["invoke"] = invoke_id;
|
||||
return run(planner_.plan("publisher.invoke", args, &body));
|
||||
}
|
||||
|
||||
Json OpenModels::gemma_instance(const std::string& prompt, int max_tokens,
|
||||
double temperature) {
|
||||
if (prompt.empty()) throw GoogleError::invalid("prompt is empty");
|
||||
if (prompt.size() > ModelsCaps::kMaxInstanceChars)
|
||||
throw GoogleError::invalid("prompt too long");
|
||||
if (max_tokens < 1 || max_tokens > ModelsCaps::kMaxGemmaTokens)
|
||||
throw GoogleError::invalid("max_tokens outside 1.." +
|
||||
std::to_string(ModelsCaps::kMaxGemmaTokens));
|
||||
if (temperature < 0.0 || temperature > 2.0)
|
||||
throw GoogleError::invalid("temperature outside 0..2");
|
||||
Json one = Json::object();
|
||||
one.set("prompt", Json(prompt));
|
||||
one.set("max_tokens", Json(max_tokens));
|
||||
one.set("temperature", Json(temperature));
|
||||
return one;
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,101 @@
|
||||
#pragma once
|
||||
|
||||
// Open models (Gemma and any self-deployed model) served from Vertex AI
|
||||
// endpoints: list, get, predict, rawPredict, streamRawPredict,
|
||||
// serverStreamingPredict, generateContent family, predictLongRunning and
|
||||
// fetchPredictOperation. Project scoped. Also publishers invoke.
|
||||
|
||||
#include <functional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "google/gemini_types.h"
|
||||
#include "google/models/vertex_service.h"
|
||||
#include "google/operation.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct EndpointInfo {
|
||||
std::string name; // projects/.../endpoints/{id}
|
||||
std::string id;
|
||||
std::string display_name;
|
||||
std::string dedicated_endpoint_dns;
|
||||
std::size_t deployed_models = 0;
|
||||
clients::body::Json raw;
|
||||
static EndpointInfo from_json(const clients::body::Json& doc);
|
||||
};
|
||||
|
||||
struct PredictResult {
|
||||
clients::body::Json predictions;
|
||||
std::string deployed_model_id;
|
||||
std::string model;
|
||||
std::string model_display_name;
|
||||
std::string model_version_id;
|
||||
};
|
||||
|
||||
class OpenModels : public VertexService {
|
||||
public:
|
||||
OpenModels(std::shared_ptr<GoogleClient> client, VertexTarget target);
|
||||
|
||||
std::vector<EndpointInfo> list_endpoints(std::size_t max_pages = 5,
|
||||
const std::string& filter = "") const;
|
||||
EndpointInfo get_endpoint(const std::string& endpoint_id) const;
|
||||
|
||||
// instances: array of model specific inputs (see gemma_instance).
|
||||
PredictResult predict(const std::string& endpoint_id,
|
||||
const clients::body::Json& instances,
|
||||
const clients::body::Json& parameters = {}) const;
|
||||
// Body is the model server's own JSON (Anthropic, vLLM chat, ...).
|
||||
clients::body::Json raw_predict(const std::string& endpoint_id,
|
||||
const clients::body::Json& body) const;
|
||||
void stream_raw_predict(const std::string& endpoint_id,
|
||||
const clients::body::Json& body,
|
||||
const std::function<bool(const clients::body::Json&)>& sink) const;
|
||||
void server_streaming_predict(
|
||||
const std::string& endpoint_id, const clients::body::Json& inputs,
|
||||
const clients::body::Json& parameters,
|
||||
const std::function<bool(const clients::body::Json&)>& sink) const;
|
||||
GenerateResponse generate_content(const std::string& endpoint_id,
|
||||
const GenerateRequest& request) const;
|
||||
void stream_generate_content(
|
||||
const std::string& endpoint_id, const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) const;
|
||||
int count_tokens(const std::string& endpoint_id, const GenerateRequest& request) const;
|
||||
std::string predict_long_running(const std::string& endpoint_id,
|
||||
const clients::body::Json& instances,
|
||||
const clients::body::Json& parameters = {}) const;
|
||||
Operation fetch_predict_operation(const std::string& endpoint_id,
|
||||
const std::string& operation_name) const;
|
||||
clients::body::Json invoke_publisher(const std::string& publisher,
|
||||
const std::string& model,
|
||||
const std::string& invoke_id,
|
||||
const clients::body::Json& body) const;
|
||||
|
||||
PlannedCall plan_list_page(std::size_t max_pages, const std::string& filter,
|
||||
const std::string& page_token) const;
|
||||
PlannedCall plan_get(const std::string& endpoint_id) const;
|
||||
PlannedCall plan_predict(const std::string& endpoint_id,
|
||||
const clients::body::Json& instances,
|
||||
const clients::body::Json& parameters) const;
|
||||
PlannedCall plan_raw(const char* call_id, const std::string& endpoint_id,
|
||||
const clients::body::Json& body) const;
|
||||
PlannedCall plan_server_streaming(const std::string& endpoint_id,
|
||||
const clients::body::Json& inputs,
|
||||
const clients::body::Json& parameters) const;
|
||||
PlannedCall plan_generate(const char* call_id, const std::string& endpoint_id,
|
||||
const GenerateRequest& request) const;
|
||||
PlannedCall plan_long_running(const std::string& endpoint_id,
|
||||
const clients::body::Json& instances,
|
||||
const clients::body::Json& parameters) const;
|
||||
PlannedCall plan_fetch_operation(const std::string& endpoint_id,
|
||||
const std::string& operation_name) const;
|
||||
|
||||
// The prompt instance Vertex's vLLM serving container accepts for Gemma
|
||||
// (Unverified: see SPEC; the shape is the container's, not Discovery's).
|
||||
static clients::body::Json gemma_instance(const std::string& prompt,
|
||||
int max_tokens = 256,
|
||||
double temperature = 0.7);
|
||||
static void check_operation_name(const std::string& name);
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,82 @@
|
||||
#include "google/models/planner.h"
|
||||
|
||||
#include <cstring>
|
||||
#include <stdexcept>
|
||||
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
const CallSpec& CallPlanner::spec(const std::string& call_id) {
|
||||
for (const CallSpec& s : kCalls)
|
||||
if (call_id == s.id) return s;
|
||||
throw std::logic_error("molo::google::models: unknown call id " + call_id);
|
||||
}
|
||||
|
||||
namespace {
|
||||
clients::Method method_of(const char* http) {
|
||||
if (!std::strcmp(http, "GET")) return clients::Method::kGet;
|
||||
if (!std::strcmp(http, "POST")) return clients::Method::kPost;
|
||||
if (!std::strcmp(http, "PATCH")) return clients::Method::kPatch;
|
||||
if (!std::strcmp(http, "DELETE")) return clients::Method::kDelete;
|
||||
throw std::logic_error("molo::google::models: bad http verb in call table");
|
||||
}
|
||||
|
||||
std::string fill(const std::string& tpl, const CallPlanner::Args& args) {
|
||||
std::string out;
|
||||
for (std::size_t i = 0; i < tpl.size();) {
|
||||
if (tpl[i] != '{') {
|
||||
out += tpl[i++];
|
||||
continue;
|
||||
}
|
||||
std::size_t end = tpl.find('}', i);
|
||||
std::string key = tpl.substr(i + 1, end - i - 1);
|
||||
auto it = args.find(key);
|
||||
if (it == args.end()) throw std::logic_error("missing path argument " + key);
|
||||
VertexTarget::require_segment(it->second, key);
|
||||
out += it->second;
|
||||
i = end + 1;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
PlannedCall CallPlanner::plan(const std::string& call_id, const Args& args,
|
||||
const clients::body::Json* body,
|
||||
const std::string& query) const {
|
||||
const CallSpec& s = spec(call_id);
|
||||
PlannedCall p;
|
||||
p.spec = &s;
|
||||
p.method = method_of(s.http);
|
||||
const bool beta = std::strcmp(s.version, "v1beta1") == 0;
|
||||
const std::string prefix = s.prefix;
|
||||
std::string path;
|
||||
if (prefix == "L") {
|
||||
if (target_.is_express()) {
|
||||
if (!s.express)
|
||||
throw GoogleError::invalid(
|
||||
std::string(call_id) +
|
||||
" needs a project and an OAuth token (set GOOGLE_CLOUD_PROJECT)");
|
||||
p.endpoint = beta ? target_.beta_endpoint() : target_.endpoint();
|
||||
path = fill(s.suffix, args);
|
||||
} else {
|
||||
p.endpoint = beta ? target_.beta_endpoint() : target_.endpoint();
|
||||
path = target_.parent_path() + fill(s.suffix, args);
|
||||
}
|
||||
} else if (prefix == "R") {
|
||||
p.endpoint = beta ? endpoints::aiplatform_beta("global") : endpoints::aiplatform("global");
|
||||
path = "/projects/" + target_.context().project + fill(s.suffix, args);
|
||||
} else {
|
||||
p.endpoint = beta ? endpoints::aiplatform_beta("global") : endpoints::aiplatform("global");
|
||||
path = fill(s.suffix, args);
|
||||
}
|
||||
if (!query.empty()) path += "?" + query;
|
||||
p.path = std::move(path);
|
||||
if (body) {
|
||||
p.body = *body;
|
||||
p.has_body = true;
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,49 @@
|
||||
#pragma once
|
||||
|
||||
// CallPlanner turns a row of the call table (calls.h) plus ids into the exact
|
||||
// request: method, endpoint, path, body. Pure, no I/O, so request building is
|
||||
// testable without any Google reply. Every id is validated here.
|
||||
|
||||
#include <map>
|
||||
#include <string>
|
||||
|
||||
#include "clients/body/json.h"
|
||||
#include "clients/request.h"
|
||||
#include "google/endpoints.h"
|
||||
#include "google/models/calls.h"
|
||||
#include "google/models/target.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
struct PlannedCall {
|
||||
clients::Method method = clients::Method::kGet;
|
||||
ServiceEndpoint endpoint;
|
||||
std::string path; // starts with "/", may carry a query string
|
||||
clients::body::Json body;
|
||||
bool has_body = false;
|
||||
const CallSpec* spec = nullptr;
|
||||
std::string url() const { return endpoint.base_url + path; }
|
||||
};
|
||||
|
||||
class CallPlanner {
|
||||
public:
|
||||
using Args = std::map<std::string, std::string>;
|
||||
|
||||
explicit CallPlanner(VertexTarget target) : target_(std::move(target)) {}
|
||||
|
||||
const VertexTarget& target() const { return target_; }
|
||||
|
||||
// Throws std::logic_error for an id missing from the table (a bug), and
|
||||
// GoogleError::invalid for a bad id value or a project-only call in
|
||||
// express mode. `query` is appended after "?" when non-empty.
|
||||
PlannedCall plan(const std::string& call_id, const Args& args,
|
||||
const clients::body::Json* body = nullptr,
|
||||
const std::string& query = "") const;
|
||||
|
||||
static const CallSpec& spec(const std::string& call_id);
|
||||
|
||||
private:
|
||||
VertexTarget target_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,54 @@
|
||||
#include "google/models/target.h"
|
||||
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
VertexTarget VertexTarget::express() { return VertexTarget(); }
|
||||
|
||||
VertexTarget VertexTarget::project(CloudContext context) {
|
||||
context.parent(); // validates project and location
|
||||
VertexTarget t;
|
||||
t.context_ = std::move(context);
|
||||
return t;
|
||||
}
|
||||
|
||||
VertexTarget VertexTarget::from_env() {
|
||||
CloudContext c = CloudContext::from_env();
|
||||
return c.project.empty() ? express() : project(std::move(c));
|
||||
}
|
||||
|
||||
const CloudContext& VertexTarget::context() const {
|
||||
if (!context_)
|
||||
throw GoogleError::invalid(
|
||||
"this call needs a project (set GOOGLE_CLOUD_PROJECT); express mode has none");
|
||||
return *context_;
|
||||
}
|
||||
|
||||
ServiceEndpoint VertexTarget::endpoint() const {
|
||||
return endpoints::aiplatform(context_ ? context_->location : "global");
|
||||
}
|
||||
|
||||
ServiceEndpoint VertexTarget::beta_endpoint() const {
|
||||
return endpoints::aiplatform_beta(context_ ? context_->location : "global");
|
||||
}
|
||||
|
||||
void VertexTarget::require_segment(const std::string& value,
|
||||
const std::string& what) {
|
||||
if (!valid_segment(value))
|
||||
throw GoogleError::invalid(what + " must be 1-128 of A-Z a-z 0-9 . _ -");
|
||||
}
|
||||
|
||||
std::string VertexTarget::model_path(const std::string& publisher,
|
||||
const std::string& model) const {
|
||||
require_segment(publisher, "publisher");
|
||||
require_segment(model, "model");
|
||||
std::string tail = "/publishers/" + publisher + "/models/" + model;
|
||||
return context_ ? "/" + context_->parent() + tail : tail;
|
||||
}
|
||||
|
||||
std::string VertexTarget::parent_path() const {
|
||||
return "/" + context().parent();
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,43 @@
|
||||
#pragma once
|
||||
|
||||
// Where a Vertex AI call goes. Two shapes, one class:
|
||||
// - express mode (API key, no project): https://aiplatform.googleapis.com/v1
|
||||
// paths /publishers/{pub}/models/{model}
|
||||
// - project scoped (OAuth): https://{loc}-aiplatform.googleapis.com/v1
|
||||
// paths /projects/{p}/locations/{l}/publishers/{pub}/models/{model}
|
||||
// Every id is validated before any network traffic.
|
||||
|
||||
#include <optional>
|
||||
#include <string>
|
||||
|
||||
#include "google/endpoints.h"
|
||||
#include "google/resource.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
class VertexTarget {
|
||||
public:
|
||||
static VertexTarget express();
|
||||
static VertexTarget project(CloudContext context);
|
||||
// Project scoped when GOOGLE_CLOUD_PROJECT is set, express otherwise.
|
||||
static VertexTarget from_env();
|
||||
|
||||
bool is_express() const { return !context_.has_value(); }
|
||||
const CloudContext& context() const; // throws invalid in express mode
|
||||
|
||||
ServiceEndpoint endpoint() const; // v1
|
||||
ServiceEndpoint beta_endpoint() const; // v1beta1
|
||||
|
||||
// "/publishers/{pub}/models/{model}" or the project scoped form.
|
||||
std::string model_path(const std::string& publisher,
|
||||
const std::string& model) const;
|
||||
// "/projects/{p}/locations/{l}"; throws invalid in express mode.
|
||||
std::string parent_path() const;
|
||||
|
||||
static void require_segment(const std::string& value, const std::string& what);
|
||||
|
||||
private:
|
||||
std::optional<CloudContext> context_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Checks calls.h (the module's one call table) against Google's Discovery
|
||||
documents. Python 3 stdlib only, no credentials.
|
||||
|
||||
python3 -I src/google/models/tools/check_discovery.py [--cache DIR]
|
||||
|
||||
For every row it requires: the HTTP method and path exist in Discovery (the
|
||||
prefixed form, and the unprefixed express form when the row says express),
|
||||
the rpc id matches, every body field name exists in the request schema
|
||||
(dotted names follow $ref), and every "?name" is a query parameter.
|
||||
Exit 0: all rows verified. Exit 1: at least one mismatch (printed).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
TABLE = os.path.join(HERE, "..", "calls.h")
|
||||
ROW = re.compile(
|
||||
r'^\s*\{"([^"]+)",\s*"([A-Z]+)",\s*"([^"]+)",\s*"([^"]*)",\s*"([^"]+)",\s*'
|
||||
r'(true|false),\s*"([^"]+)",\s*"([^"]*)"\},?\s*$')
|
||||
PREFIX = {"L": "/projects/{}/locations/{}", "R": "/projects/{}", "": ""}
|
||||
|
||||
|
||||
def norm(path):
|
||||
return re.sub(r"\{[^}]*\}", "{}", path)
|
||||
|
||||
|
||||
def fetch(version, cache):
|
||||
url = "https://aiplatform.googleapis.com/$discovery/rest?version=" + version
|
||||
if cache:
|
||||
os.makedirs(cache, exist_ok=True)
|
||||
f = os.path.join(cache, "aiplatform_%s.json" % version)
|
||||
if os.path.exists(f):
|
||||
return json.load(open(f))
|
||||
with urllib.request.urlopen(url, timeout=120) as r:
|
||||
raw = r.read()
|
||||
if cache:
|
||||
open(os.path.join(cache, "aiplatform_%s.json" % version), "wb").write(raw)
|
||||
return json.loads(raw)
|
||||
|
||||
|
||||
def methods(doc):
|
||||
out = {}
|
||||
|
||||
def walk(res):
|
||||
for r in res.get("resources", {}).values():
|
||||
for m in r.get("methods", {}).values():
|
||||
key = (m["httpMethod"], norm("/" + m.get("flatPath", m["path"])))
|
||||
out[key] = m
|
||||
walk(r)
|
||||
|
||||
walk(doc)
|
||||
return out
|
||||
|
||||
|
||||
def schema_has(doc, schema_name, dotted):
|
||||
s = doc["schemas"].get(schema_name)
|
||||
for part in dotted.split("."):
|
||||
if s is None or part not in s.get("properties", {}):
|
||||
return False
|
||||
prop = s["properties"][part]
|
||||
ref = prop.get("$ref") or prop.get("items", {}).get("$ref")
|
||||
s = doc["schemas"].get(ref) if ref else {"properties": {}}
|
||||
if ref is None:
|
||||
s = {"properties": {}, "_leaf": True}
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
cache = None
|
||||
if "--cache" in sys.argv:
|
||||
cache = sys.argv[sys.argv.index("--cache") + 1]
|
||||
rows = []
|
||||
for line in open(TABLE):
|
||||
m = ROW.match(line)
|
||||
if m:
|
||||
rows.append(m.groups())
|
||||
if not rows:
|
||||
print("no rows parsed from calls.h")
|
||||
return 1
|
||||
docs = {}
|
||||
index = {}
|
||||
bad = []
|
||||
for call_id, http, ver, prefix, suffix, express, rpc, fields in rows:
|
||||
if ver not in docs:
|
||||
docs[ver] = fetch(ver, cache)
|
||||
index[ver] = methods(docs[ver])
|
||||
doc, idx = docs[ver], index[ver]
|
||||
full = "/" + ver + PREFIX[prefix] + suffix
|
||||
meth = idx.get((http, norm(full)))
|
||||
if meth is None:
|
||||
bad.append("%s: %s %s not in discovery %s" % (call_id, http, full, ver))
|
||||
continue
|
||||
if meth["id"] != rpc:
|
||||
bad.append("%s: rpc id %s, table says %s" % (call_id, meth["id"], rpc))
|
||||
if express == "true":
|
||||
ex = idx.get((http, norm("/" + ver + suffix)))
|
||||
if ex is None:
|
||||
bad.append("%s: express form %s not in discovery" % (call_id, suffix))
|
||||
req = meth.get("request", {}).get("$ref")
|
||||
params = meth.get("parameters", {})
|
||||
for f in fields.split():
|
||||
if f == "-":
|
||||
continue
|
||||
if f.startswith("?"):
|
||||
if f[1:] not in params or params[f[1:]]["location"] != "query":
|
||||
bad.append("%s: query parameter %s absent" % (call_id, f[1:]))
|
||||
elif req is None or not schema_has(doc, req, f):
|
||||
bad.append("%s: request field %s absent in %s" % (call_id, f, req))
|
||||
for b in bad:
|
||||
print("FAIL " + b)
|
||||
print("%d calls checked, %d problems" % (len(rows), len(bad)))
|
||||
return 1 if bad else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,26 @@
|
||||
#include "google/models/vertex_model.h"
|
||||
|
||||
#include "google/errors.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
VertexModel::VertexModel(std::shared_ptr<Gemini> gemini, std::string default_model)
|
||||
: gemini_(std::move(gemini)), default_model_(std::move(default_model)) {
|
||||
VertexTarget::require_segment(default_model_, "model");
|
||||
}
|
||||
|
||||
const std::string& VertexModel::model_for(const GenerateRequest& request) const {
|
||||
return request.model.empty() ? default_model_ : request.model;
|
||||
}
|
||||
|
||||
GenerateResponse VertexModel::generate(const GenerateRequest& request) {
|
||||
return gemini_->generate(model_for(request), request);
|
||||
}
|
||||
|
||||
void VertexModel::stream(
|
||||
const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) {
|
||||
gemini_->stream(model_for(request), request, on_chunk);
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
// VertexModel: a LanguageModel over Gemini on Vertex AI. GenerateRequest.model
|
||||
// overrides the default model when set.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "google/gemini_types.h"
|
||||
#include "google/models/gemini.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
class VertexModel : public LanguageModel {
|
||||
public:
|
||||
VertexModel(std::shared_ptr<Gemini> gemini, std::string default_model);
|
||||
|
||||
const std::string& default_model() const { return default_model_; }
|
||||
|
||||
GenerateResponse generate(const GenerateRequest& request) override;
|
||||
void stream(const GenerateRequest& request,
|
||||
const std::function<bool(const GenerateResponse&)>& on_chunk) override;
|
||||
|
||||
private:
|
||||
const std::string& model_for(const GenerateRequest& request) const;
|
||||
|
||||
std::shared_ptr<Gemini> gemini_;
|
||||
std::string default_model_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,27 @@
|
||||
#include "google/models/vertex_service.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
VertexService::VertexService(std::shared_ptr<GoogleClient> client,
|
||||
VertexTarget target, std::string name)
|
||||
: Service(std::move(client), ServiceEndpoint{std::move(name), target.endpoint().base_url}),
|
||||
planner_(std::move(target)) {}
|
||||
|
||||
clients::body::Json VertexService::run(const PlannedCall& call) const {
|
||||
return client_->request(call.method, call.endpoint, call.path,
|
||||
call.has_body ? &call.body : nullptr);
|
||||
}
|
||||
|
||||
void VertexService::run_stream(const PlannedCall& call,
|
||||
const GoogleClient::EventSink& sink) const {
|
||||
client_->stream_json(call.endpoint, call.path,
|
||||
call.has_body ? call.body : clients::body::Json::object(),
|
||||
sink);
|
||||
}
|
||||
|
||||
CallPlanner::Args VertexService::model_args(const std::string& publisher,
|
||||
const std::string& model) {
|
||||
return {{"publisher", publisher}, {"model", model}};
|
||||
}
|
||||
|
||||
} // namespace molo::google::models
|
||||
@@ -0,0 +1,33 @@
|
||||
#pragma once
|
||||
|
||||
// Base of every service in this module: a Service that owns a VertexTarget
|
||||
// and a CallPlanner, and runs planned calls over the shared GoogleClient.
|
||||
|
||||
#include <functional>
|
||||
|
||||
#include "google/models/json_read.h"
|
||||
#include "google/models/planner.h"
|
||||
#include "google/service.h"
|
||||
|
||||
namespace molo::google::models {
|
||||
|
||||
class VertexService : public Service {
|
||||
public:
|
||||
VertexService(std::shared_ptr<GoogleClient> client, VertexTarget target,
|
||||
std::string name);
|
||||
|
||||
const VertexTarget& target() const { return planner_.target(); }
|
||||
const CallPlanner& planner() const { return planner_; }
|
||||
|
||||
protected:
|
||||
clients::body::Json run(const PlannedCall& call) const;
|
||||
void run_stream(const PlannedCall& call,
|
||||
const GoogleClient::EventSink& sink) const;
|
||||
// Single "{publisher}/{model}" argument map.
|
||||
static CallPlanner::Args model_args(const std::string& publisher,
|
||||
const std::string& model);
|
||||
|
||||
CallPlanner planner_;
|
||||
};
|
||||
|
||||
} // namespace molo::google::models
|
||||
Reference in New Issue
Block a user