Files
moloapp/src/google/optimize/SPEC.md
T
retoorandClaude Sonnet 5.5 a3396bfc68 feat(clients): sessions+registry+optimize+armor: close gaps, loud list truncation, shared Gemini adapter
sessions: operations get/list/cancel/delete/wait for engine, session, memory and
serving profile; servingProfiles CRUD. registry: ListResult with truncated flag,
v1alpha twins, gateway/policy operations, authz IAM, deny policy update, boundary
patch/bindings. optimize: make_gemini_model over models::VertexModel (DirectModel
removed), generateSyntheticData, online evaluator operations, trace/log writers.
armor: SecurityAdmin for all SCC v2 methods (attack exposure excluded), documented
agent threat categories, floor settings fixed to location global after live run.
check_discovery.py of all four enforce coverage.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-24 05:43:13 +02:00

15 KiB

Google Agent Platform: optimize - Specification

Owned directory: src/google/optimize/ (namespace molo::google::optimize), tests tests/src/google/optimize/, example example/app/agent/OptimizeDemo.cpp with example/assets/agent_optimize.js, route tests tests/example/agent/optimize/. Depends on the src/google/ foundation only (GoogleClient, Service, CloudContext, Operation, LanguageModel).

1. Scope

Four product areas of the Gemini Enterprise Agent Platform "optimize" family. For each, what is a Google REST API and what is local logic:

Area Google API (typed here) Local logic (named local, runs on any LanguageModel)
Agent Evaluation aiplatform v1: evaluateInstances, evaluateDataset, generateInstanceRubrics, generateLossClusters, evaluationSets, evaluationItems, evaluationMetrics, evaluationRuns, onlineEvaluators (continuous scoring of live traffic) Autorater: multi-turn rubric judge over whole conversations
Agent Simulation aiplatform v1 generateUserScenarios; EvaluationRun inference config (userSimulatorConfig, maxTurn) Simulator, SimulatedUser, VirtualToolbox, ModelAgent: synthetic users and virtualized tools, scoring task success and safety
Agent Observability Cloud Trace v1 traces.list/get, Cloud Logging v2 entries.list ConversationBuilder, summarize (dashboard numbers)
Agent Optimizer none (Google's optimizer is adk optimize, GEPA over a test suite: CLI/SDK, no REST resource) FailureClusterer, InstructionOptimizer

No REST resource exists for simulation tool virtualization or for the optimizer; those are local abstractions and are named local in code and here.

2. Module map (one class or concern per file)

File Concern
caps.h Static ceilings
calls_table.inc, calls.h/.cpp The ONE table of Google calls (verb, path, service, version, body fields, query params), path/query/body checks
optimize_service.h/.cpp OptimizeService::invoke runs a table row after local validation
eval_requests.h/.cpp EvaluationRequests: evaluateInstances bodies (safety, exact match, six trajectory metrics, pointwise, pairwise, autorater config) and result parser
evaluation_api.h/.cpp EvaluationApi: evaluateInstances, evaluateDataset, generateInstanceRubrics, generateUserScenarios, generateLossClusters
eval_resources.h/.cpp EvaluationResources: get/list/create/patch/action/import/delete for the five collections, EvaluationRunSpec, OnlineEvaluatorSpec, EvaluationItemSpec, wait_run
observability.h/.cpp TraceReader, LogReader, Trace, Span, LogEntry, ConversationBuilder, summarize
conversation.h/.cpp Conversation, Turn, transcript parse and render
model_json.h/.cpp JsonModelCall: one JSON object from a LanguageModel, one retry
model_factory.h/.cpp make_gemini_model: the one Gemini adapter, models::VertexModel from src/google/models, target picked from the credentials (the former private DirectModel is gone)
operations_api.h/.cpp OnlineEvaluatorOperations: get, list (loud truncation flag), cancel, delete, server-side wait for online evaluator operations
telemetry_writer.h/.cpp TraceWriter (patchTraces, batchWrite, createSpan) and LogWriter (entries:write with dryRun, entries:copy)
autorater.h/.cpp Autorater, Rubric, Score
scenario.h/.cpp Scenario (mirrors Google's UserScenario)
agent.h/.cpp Agent seam, ModelAgent
virtual_tools.h/.cpp VirtualToolbox
simulated_user.h/.cpp SimulatedUser
simulator.h/.cpp Simulator, SimulationReport
clusterer.h/.cpp FailureClusterer
optimizer.h/.cpp InstructionOptimizer
tools/check_discovery.py Discovery verifier (stdlib only)

3. Static caps (see caps.h)

Knob Value
Text per prompt, response, field 20000 chars
Rendered conversation 60000 chars, cut with a marker
Turns per conversation 50
System instruction 8000 chars
Rubrics per score / rubric text 16 / 500 chars
Trajectory calls 100
evaluateInstances instance fields 16
Scenarios per generate call 100 (Google's ceiling)
Run metrics, online evaluator metrics 16
List pages / items 20 pages (default 5), 1000 items
wait_run polls 200 at 2 s; operations use foundation caps (400 polls, 600 s)
Simulation turns default 5, max 12
Tool rounds per turn / tool calls per session 5 / 40
Traces / spans per trace 500 / 2000
Log entries / page / resource names / filter 1000 / 100 / 100 / 20000 chars
Failures analyzed / clusters 200 (rest counted as dropped) / 10
Optimizer rounds 3
Model JSON retries 1
Online evaluator sampling 1..100 percent; maxEvaluatedSamplesPerRun 1..100000 (0 means unbounded at Google and is refused)

4. Contracts

  • Every Google call is a row in calls_table.inc. OptimizeService::invoke expands {x} and {+x} path parameters (each segment valid_segment, resource names projects/..., max 12 segments), rejects query parameters and top-level body fields the row does not list, and only then sends. Invalid input is GoogleError 400 INVALID_ARGUMENT before any traffic; no credential is NotConfiguredError 503 before any traffic.
  • Resource names are checked against their collection (projects/p/locations/l/{collection}/id). A verb a collection lacks in Discovery (for example patch on evaluationRuns) is refused locally.
  • Reads come before mutations. Delete is EvaluationResources::delete_resource only, long-running, wait=false by default.
  • Long-running calls return an Operation; wait=true uses the foundation wait_operation (injectable sleeper).
  • int64 fields (userScenarioCount, maxEvaluatedSamplesPerRun) are sent as JSON strings, as Discovery's JSON mapping requires.
  • Autorater: one model call scores the whole conversation against rubrics. A rubric the model did not answer fails with "no verdict returned"; it is never silently passed.
  • Clusterer output is verified: unknown and duplicate ids are discarded, clusters above the cap are cut, unplaced failures are listed in unassigned.
  • Optimizer optimize() returns the original instruction unless a scorer proves a strict improvement; suggest() never claims improved.
  • Simulator loops are bounded by turns, tool rounds and tool calls; the stop reason is reported (user_done, max_turns, tool_round_cap, tool_call_cap).
  • Credentials: this module never reads or prints them. make_gemini_model builds a models::VertexModel (so there is one Gemini adapter in the platform): an OAuth token (GOOGLE_ACCESS_TOKEN) with a valid GOOGLE_CLOUD_PROJECT gives the project scoped Vertex path in location global, anything else Vertex express mode (publisher path). Behavior change from DirectModel: the Generative Language endpoint (non Vertex API keys) is no longer used; such a key now fails at Google on the Vertex path. Streaming is real (?alt=sse) instead of one chunk.
  • Write calls are explicit classes (TraceWriter, LogWriter), bodies are raw Json checked against the table and capped (1 MiB, 1000 log entries, 2000 spans). LogWriter::write supports dryRun; the demo route is a dry run unless confirm is write. entries.tail is excluded: an endless server stream has no cap. OnlineEvaluatorOperations::list stops at 20 pages / 1000 items and says truncated with the next page token.

5. Example routes (/agent/optimize)

GET /agent/optimize (page, matrix row), GET /agent/optimize/status. POST (503 google_not_configured without credentials or a valid GOOGLE_CLOUD_PROJECT, 422 on bad input, 502 google_error upstream): evaluate, scenarios, runs, traces, logs, synthetic, operations/list, operations/get, operations/wait, operations/cancel, logs/write (Google APIs; DELETE operations?name= removes an operation record) and score, cluster, suggest, simulate (local logic on a Gemini model).

6. Verified against discovery

python3 -I src/google/optimize/tools/check_discovery.py, run 2026-10-24 (gap-closing round): OK: 46 calls and 64 schemas match discovery. It also enforces coverage: every aiplatform v1 evaluat*, generate* and onlineEvaluators* method, every Cloud Trace v1 and v2 method and every logging.entries* method must be a table row or be listed in EXCLUDED (only logging.entries.tail, endless stream). It fetches https://{aiplatform,cloudtrace,logging}.googleapis.com/$discovery/rest (aiplatform v1, cloudtrace v1 and v2, logging v2) and fails on any declared call absent or different in verb, path, request fields or query parameters, and on any schema field or enum value the builders and parsers rely on.

7. Verified live

Date 2026-10-24, project from GOOGLE_CLOUD_PROJECT (not recorded here), opt-in test tests/src/google/optimize/test_live.cpp (MOLO_LIVE_NET=1, loud SKIP otherwise).

Call Credential Result
POST locations/us-central1:evaluateInstances exactMatch, trajectoryExactMatch express key 200, scores parsed
same, safetyInput (LLM judged) express key and OAuth 404 NOT_FOUND: "Failed to make GenerateContent request to autorater model ... gemini-2.5-flash". The credential is accepted; the project cannot use the default autorater model. LLM-judged metrics are therefore not verified end to end.
GET evaluationSets, evaluationItems, evaluationMetrics, evaluationRuns, onlineEvaluators ?pageSize=1 express key 401 UNAUTHENTICATED (API keys not supported)
same five lists OAuth (service account) 200, empty lists
POST logging v2 entries:list pageSize 1 express key 401
same OAuth 200, 1 entry
GET cloudtrace v1 traces?pageSize=1 express key 401
same OAuth 200, 0 traces
Autorater and DirectModel against real generateContent (express key, Vertex publisher path; and OAuth, project path) both verdicts parsed, score 1.0

One run of the OAuth live suite showed 1 failed test of 5 (name not captured); the immediate rerun and a later run passed 5 of 5. Treat as an unexplained transient until reproduced.

Gap-closing round, 2026-10-24, service-account token and express key, read-only (optimize_live_gaps.*, optimize_live.*):

Call Result
GET onlineEvaluators/1/operations?pageSize=1 (absent evaluator) 200, no operations
GET onlineEvaluators/probe/operations 400 INVALID_ARGUMENT: the LRO layer wants a numeric evaluator id ("must be a Long")
make_gemini_model (models::VertexModel) generate, OAuth token, project path, location global 200, a reply
Autorater through make_gemini_model with the express key (no token) verdicts parsed, score 1
Autorater through make_gemini_model with the token verdicts parsed, score 1
evaluate_instances exactMatch / trajectory (key), five resource lists (token) unchanged from above: 200

Not run live (would create or change resources): generateSyntheticData, the online evaluator operation cancel, delete and wait, patchTraces, batchWrite, createSpan, entries:write, entries:copy, create/patch/delete/import of sets, items, metrics, runs, online evaluators, evaluateDataset, generateInstanceRubrics, generateUserScenarios, generateLossClusters. These are verified by discovery only.

8. Unverified

  • Neither discovery nor docs state: the predefinedMetricSpec.metricSpecName values, the agentResource name format of an online evaluator, the response type of generateLossClusters (discovery types it as a generic long-running response, so it is returned as raw Json), and what cloudObservability requires beyond openTelemetry.semconvVersion.
  • Cloud Trace span labels and Cloud Logging jsonPayload layout of agent telemetry (OTel GenAI semantic conventions) are not specified on the pages read. ConversationBuilder is a local heuristic (jsonPayload.role of user|model|assistant|agent plus content, text or message); Trace reads labels generically.
  • Docs mention is_agentic=true and principal:// / spiffe:// attribution in Cloud Logging; the exact log field was not read, so no filter helper uses it.
  • The default autorater model for LLM-judged metrics is unavailable to the test project (see 7); AutoraterOptions::model is the workaround, untested.
  • v1beta1 differences are not checked.
  • generateSyntheticData: request and reply shapes are raw Json; the reply type is not typed here and was never seen.
  • Online evaluator operation names: the id of an evaluator in an operation name must be numeric at Google's LRO layer (400 seen); evaluator ids Google assigns were never observed because none exists.
  • entries:copy destination string format and entries:write resource labels are passed through, only top-level fields are checked.
  • The shared adapter is Vertex only: a non Vertex API key (Generative Language AIza key) is not routed to the Generative Language endpoint any more.

9. Gaps

  • logging.entries.tail (endless server stream) is not wrapped; the write calls (TraceWriter, LogWriter) exist for tools, agents normally write through OTel exporters.
  • Trace and log lists (TraceReader::list, LogReader::entries) still stop at their caps without a truncated flag; only the new operations list reports truncation. Owner: observability.cpp.
  • Dashboards: only numbers (DashboardData); no Cloud Monitoring reads.
  • evaluateDataset, generateInstanceRubrics, evaluationMetrics and item bodies are passed as raw Json (checked at the top level against Discovery) rather than fully typed.
  • Simulation tool virtualization and the optimizer are local approximations, not Google's. Owner: simulator.cpp, optimizer.cpp.
  • Token refresh is the caller's job (foundation gap).

Sources