Files
moloapp/src/google/optimize/SPEC.md
T

192 lines
12 KiB
Markdown
Raw Normal View History

# Google Agent Platform: optimize - Specification
Owned directory: `src/google/optimize/` (namespace `molo::google::optimize`),
tests `tests/src/google/optimize/`, example `example/app/agent/OptimizeDemo.cpp`
with `example/assets/agent_optimize.js`, route tests
`tests/example/agent/optimize/`. Depends on the `src/google/` foundation only
(`GoogleClient`, `Service`, `CloudContext`, `Operation`, `LanguageModel`).
## 1. Scope
Four product areas of the Gemini Enterprise Agent Platform "optimize" family.
For each, what is a Google REST API and what is local logic:
| Area | Google API (typed here) | Local logic (named local, runs on any `LanguageModel`) |
|---|---|---|
| Agent Evaluation | aiplatform v1: `evaluateInstances`, `evaluateDataset`, `generateInstanceRubrics`, `generateLossClusters`, `evaluationSets`, `evaluationItems`, `evaluationMetrics`, `evaluationRuns`, `onlineEvaluators` (continuous scoring of live traffic) | `Autorater`: multi-turn rubric judge over whole conversations |
| Agent Simulation | aiplatform v1 `generateUserScenarios`; `EvaluationRun` inference config (`userSimulatorConfig`, `maxTurn`) | `Simulator`, `SimulatedUser`, `VirtualToolbox`, `ModelAgent`: synthetic users and virtualized tools, scoring task success and safety |
| Agent Observability | Cloud Trace v1 `traces.list/get`, Cloud Logging v2 `entries.list` | `ConversationBuilder`, `summarize` (dashboard numbers) |
| Agent Optimizer | none (Google's optimizer is `adk optimize`, GEPA over a test suite: CLI/SDK, no REST resource) | `FailureClusterer`, `InstructionOptimizer` |
No REST resource exists for simulation tool virtualization or for the
optimizer; those are local abstractions and are named local in code and here.
## 2. Module map (one class or concern per file)
| File | Concern |
|---|---|
| `caps.h` | Static ceilings |
| `calls_table.inc`, `calls.h/.cpp` | The ONE table of Google calls (verb, path, service, version, body fields, query params), path/query/body checks |
| `optimize_service.h/.cpp` | `OptimizeService::invoke` runs a table row after local validation |
| `eval_requests.h/.cpp` | `EvaluationRequests`: evaluateInstances bodies (safety, exact match, six trajectory metrics, pointwise, pairwise, autorater config) and result parser |
| `evaluation_api.h/.cpp` | `EvaluationApi`: evaluateInstances, evaluateDataset, generateInstanceRubrics, generateUserScenarios, generateLossClusters |
| `eval_resources.h/.cpp` | `EvaluationResources`: get/list/create/patch/action/import/delete for the five collections, `EvaluationRunSpec`, `OnlineEvaluatorSpec`, `EvaluationItemSpec`, `wait_run` |
| `observability.h/.cpp` | `TraceReader`, `LogReader`, `Trace`, `Span`, `LogEntry`, `ConversationBuilder`, `summarize` |
| `conversation.h/.cpp` | `Conversation`, `Turn`, transcript parse and render |
| `model_json.h/.cpp` | `JsonModelCall`: one JSON object from a `LanguageModel`, one retry |
| `direct_model.h/.cpp` | `DirectModel : LanguageModel` over `GoogleClient` (private to this module) |
| `autorater.h/.cpp` | `Autorater`, `Rubric`, `Score` |
| `scenario.h/.cpp` | `Scenario` (mirrors Google's `UserScenario`) |
| `agent.h/.cpp` | `Agent` seam, `ModelAgent` |
| `virtual_tools.h/.cpp` | `VirtualToolbox` |
| `simulated_user.h/.cpp` | `SimulatedUser` |
| `simulator.h/.cpp` | `Simulator`, `SimulationReport` |
| `clusterer.h/.cpp` | `FailureClusterer` |
| `optimizer.h/.cpp` | `InstructionOptimizer` |
| `tools/check_discovery.py` | Discovery verifier (stdlib only) |
## 3. Static caps (see caps.h)
| Knob | Value |
|---|---|
| Text per prompt, response, field | 20000 chars |
| Rendered conversation | 60000 chars, cut with a marker |
| Turns per conversation | 50 |
| System instruction | 8000 chars |
| Rubrics per score / rubric text | 16 / 500 chars |
| Trajectory calls | 100 |
| evaluateInstances instance fields | 16 |
| Scenarios per generate call | 100 (Google's ceiling) |
| Run metrics, online evaluator metrics | 16 |
| List pages / items | 20 pages (default 5), 1000 items |
| `wait_run` polls | 200 at 2 s; operations use foundation caps (400 polls, 600 s) |
| Simulation turns | default 5, max 12 |
| Tool rounds per turn / tool calls per session | 5 / 40 |
| Traces / spans per trace | 500 / 2000 |
| Log entries / page / resource names / filter | 1000 / 100 / 100 / 20000 chars |
| Failures analyzed / clusters | 200 (rest counted as `dropped`) / 10 |
| Optimizer rounds | 3 |
| Model JSON retries | 1 |
| Online evaluator sampling | 1..100 percent; `maxEvaluatedSamplesPerRun` 1..100000 (0 means unbounded at Google and is refused) |
## 4. Contracts
- Every Google call is a row in `calls_table.inc`. `OptimizeService::invoke`
expands `{x}` and `{+x}` path parameters (each segment `valid_segment`,
resource names `projects/...`, max 12 segments), rejects query parameters
and top-level body fields the row does not list, and only then sends.
Invalid input is `GoogleError` 400 `INVALID_ARGUMENT` before any traffic;
no credential is `NotConfiguredError` 503 before any traffic.
- Resource names are checked against their collection
(`projects/p/locations/l/{collection}/id`). A verb a collection lacks in
Discovery (for example patch on `evaluationRuns`) is refused locally.
- Reads come before mutations. Delete is `EvaluationResources::delete_resource`
only, long-running, `wait=false` by default.
- Long-running calls return an `Operation`; `wait=true` uses the foundation
`wait_operation` (injectable sleeper).
- `int64` fields (`userScenarioCount`, `maxEvaluatedSamplesPerRun`) are sent
as JSON strings, as Discovery's JSON mapping requires.
- Autorater: one model call scores the whole conversation against rubrics. A
rubric the model did not answer fails with "no verdict returned"; it is
never silently passed.
- Clusterer output is verified: unknown and duplicate ids are discarded,
clusters above the cap are cut, unplaced failures are listed in
`unassigned`.
- Optimizer `optimize()` returns the original instruction unless a scorer
proves a strict improvement; `suggest()` never claims `improved`.
- Simulator loops are bounded by turns, tool rounds and tool calls; the
stop reason is reported (`user_done`, `max_turns`, `tool_round_cap`,
`tool_call_cap`).
- Credentials: this module never reads or prints them. `DirectModel::from_env`
picks the endpoint: OAuth token plus project gives the project-scoped
Vertex path (global location), an express key the Vertex publisher path,
otherwise Generative Language.
## 5. Example routes (`/agent/optimize`)
GET `/agent/optimize` (page, matrix row), GET `/agent/optimize/status`.
POST (503 `google_not_configured` without credentials or a valid
`GOOGLE_CLOUD_PROJECT`, 422 on bad input, 502 `google_error` upstream):
`evaluate`, `scenarios`, `runs`, `traces`, `logs` (Google APIs) and `score`,
`cluster`, `suggest`, `simulate` (local logic on a Gemini model).
## 6. Verified against discovery
`python3 -I src/google/optimize/tools/check_discovery.py`, run 2026-10-24:
`OK: 38 calls and 64 schemas match discovery`. It fetches
`https://{aiplatform,cloudtrace,logging}.googleapis.com/$discovery/rest`
(aiplatform v1, cloudtrace v1 and v2, logging v2) and fails on any declared
call absent or different in verb, path, request fields or query parameters,
and on any schema field or enum value the builders and parsers rely on.
## 7. Verified live
Date 2026-10-24, project from `GOOGLE_CLOUD_PROJECT` (not recorded here),
opt-in test `tests/src/google/optimize/test_live.cpp` (`MOLO_LIVE_NET=1`,
loud SKIP otherwise).
| Call | Credential | Result |
|---|---|---|
| `POST locations/us-central1:evaluateInstances` exactMatch, trajectoryExactMatch | express key | 200, scores parsed |
| same, `safetyInput` (LLM judged) | express key and OAuth | 404 NOT_FOUND: "Failed to make GenerateContent request to autorater model ... gemini-2.5-flash". The credential is accepted; the project cannot use the default autorater model. LLM-judged metrics are therefore not verified end to end. |
| `GET evaluationSets, evaluationItems, evaluationMetrics, evaluationRuns, onlineEvaluators ?pageSize=1` | express key | 401 UNAUTHENTICATED (API keys not supported) |
| same five lists | OAuth (service account) | 200, empty lists |
| `POST logging v2 entries:list` pageSize 1 | express key | 401 |
| same | OAuth | 200, 1 entry |
| `GET cloudtrace v1 traces?pageSize=1` | express key | 401 |
| same | OAuth | 200, 0 traces |
| Autorater and `DirectModel` against real `generateContent` (express key, Vertex publisher path; and OAuth, project path) | both | verdicts parsed, score 1.0 |
One run of the OAuth live suite showed 1 failed test of 5 (name not captured);
the immediate rerun and a later run passed 5 of 5. Treat as an unexplained
transient until reproduced.
Not run live (would create or change resources): create/patch/delete/import
of sets, items, metrics, runs, online evaluators, `evaluateDataset`,
`generateInstanceRubrics`, `generateUserScenarios`, `generateLossClusters`.
These are verified by discovery only.
## 8. Unverified
- Neither discovery nor docs state: the `predefinedMetricSpec.metricSpecName`
values, the `agentResource` name format of an online evaluator, the response
type of `generateLossClusters` (discovery types it as a generic long-running
response, so it is returned as raw `Json`), and what `cloudObservability`
requires beyond `openTelemetry.semconvVersion`.
- Cloud Trace span labels and Cloud Logging `jsonPayload` layout of agent
telemetry (OTel GenAI semantic conventions) are not specified on the pages
read. `ConversationBuilder` is a local heuristic (`jsonPayload.role` of
user|model|assistant|agent plus `content`, `text` or `message`); `Trace`
reads labels generically.
- Docs mention `is_agentic=true` and `principal://` / `spiffe://` attribution
in Cloud Logging; the exact log field was not read, so no filter helper
uses it.
- The default autorater model for LLM-judged metrics is unavailable to the
test project (see 7); `AutoraterOptions::model` is the workaround, untested.
- v1beta1 differences are not checked.
## 9. Gaps
- Cloud Trace v2 `batchWrite` and `createSpan` and Cloud Logging
`entries.write/tail` are not wrapped: agents write through OTel exporters.
The v2 rows are not in the table because nothing calls them.
- Dashboards: only numbers (`DashboardData`); no Cloud Monitoring reads.
- `evaluateDataset`, `generateInstanceRubrics`, `evaluationMetrics` and item
bodies are passed as raw `Json` (checked at the top level against
Discovery) rather than fully typed.
- Simulation tool virtualization and the optimizer are local approximations,
not Google's. Owner: `simulator.cpp`, `optimizer.cpp`.
- Token refresh is the caller's job (foundation gap).
## Sources
- aiplatform v1 Discovery: https://aiplatform.googleapis.com/$discovery/rest?version=v1
- Cloud Trace Discovery: https://cloudtrace.googleapis.com/$discovery/rest?version=v1 and `version=v2`
- Cloud Logging Discovery: https://logging.googleapis.com/$discovery/rest?version=v2
- https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/evaluate-simulated
- https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/optimize-agent
- https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/observability/overview
- https://docs.cloud.google.com/trace/docs/reference/v1/rest and `.../v1/rest/v1/projects.traces/list`
- https://docs.cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list
- The agent platform REST reference page for `projects.locations.evaluateInstances` was fetched but its body was truncated; the request and response shapes come from Discovery.