feat(clients): optimize: evaluation, simulation, observability and optimizer on a discovery-checked call table
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,191 @@
|
||||
# Google Agent Platform: optimize - Specification
|
||||
|
||||
Owned directory: `src/google/optimize/` (namespace `molo::google::optimize`),
|
||||
tests `tests/src/google/optimize/`, example `example/app/agent/OptimizeDemo.cpp`
|
||||
with `example/assets/agent_optimize.js`, route tests
|
||||
`tests/example/agent/optimize/`. Depends on the `src/google/` foundation only
|
||||
(`GoogleClient`, `Service`, `CloudContext`, `Operation`, `LanguageModel`).
|
||||
|
||||
## 1. Scope
|
||||
|
||||
Four product areas of the Gemini Enterprise Agent Platform "optimize" family.
|
||||
For each, what is a Google REST API and what is local logic:
|
||||
|
||||
| Area | Google API (typed here) | Local logic (named local, runs on any `LanguageModel`) |
|
||||
|---|---|---|
|
||||
| Agent Evaluation | aiplatform v1: `evaluateInstances`, `evaluateDataset`, `generateInstanceRubrics`, `generateLossClusters`, `evaluationSets`, `evaluationItems`, `evaluationMetrics`, `evaluationRuns`, `onlineEvaluators` (continuous scoring of live traffic) | `Autorater`: multi-turn rubric judge over whole conversations |
|
||||
| Agent Simulation | aiplatform v1 `generateUserScenarios`; `EvaluationRun` inference config (`userSimulatorConfig`, `maxTurn`) | `Simulator`, `SimulatedUser`, `VirtualToolbox`, `ModelAgent`: synthetic users and virtualized tools, scoring task success and safety |
|
||||
| Agent Observability | Cloud Trace v1 `traces.list/get`, Cloud Logging v2 `entries.list` | `ConversationBuilder`, `summarize` (dashboard numbers) |
|
||||
| Agent Optimizer | none (Google's optimizer is `adk optimize`, GEPA over a test suite: CLI/SDK, no REST resource) | `FailureClusterer`, `InstructionOptimizer` |
|
||||
|
||||
No REST resource exists for simulation tool virtualization or for the
|
||||
optimizer; those are local abstractions and are named local in code and here.
|
||||
|
||||
## 2. Module map (one class or concern per file)
|
||||
|
||||
| File | Concern |
|
||||
|---|---|
|
||||
| `caps.h` | Static ceilings |
|
||||
| `calls_table.inc`, `calls.h/.cpp` | The ONE table of Google calls (verb, path, service, version, body fields, query params), path/query/body checks |
|
||||
| `optimize_service.h/.cpp` | `OptimizeService::invoke` runs a table row after local validation |
|
||||
| `eval_requests.h/.cpp` | `EvaluationRequests`: evaluateInstances bodies (safety, exact match, six trajectory metrics, pointwise, pairwise, autorater config) and result parser |
|
||||
| `evaluation_api.h/.cpp` | `EvaluationApi`: evaluateInstances, evaluateDataset, generateInstanceRubrics, generateUserScenarios, generateLossClusters |
|
||||
| `eval_resources.h/.cpp` | `EvaluationResources`: get/list/create/patch/action/import/delete for the five collections, `EvaluationRunSpec`, `OnlineEvaluatorSpec`, `EvaluationItemSpec`, `wait_run` |
|
||||
| `observability.h/.cpp` | `TraceReader`, `LogReader`, `Trace`, `Span`, `LogEntry`, `ConversationBuilder`, `summarize` |
|
||||
| `conversation.h/.cpp` | `Conversation`, `Turn`, transcript parse and render |
|
||||
| `model_json.h/.cpp` | `JsonModelCall`: one JSON object from a `LanguageModel`, one retry |
|
||||
| `direct_model.h/.cpp` | `DirectModel : LanguageModel` over `GoogleClient` (private to this module) |
|
||||
| `autorater.h/.cpp` | `Autorater`, `Rubric`, `Score` |
|
||||
| `scenario.h/.cpp` | `Scenario` (mirrors Google's `UserScenario`) |
|
||||
| `agent.h/.cpp` | `Agent` seam, `ModelAgent` |
|
||||
| `virtual_tools.h/.cpp` | `VirtualToolbox` |
|
||||
| `simulated_user.h/.cpp` | `SimulatedUser` |
|
||||
| `simulator.h/.cpp` | `Simulator`, `SimulationReport` |
|
||||
| `clusterer.h/.cpp` | `FailureClusterer` |
|
||||
| `optimizer.h/.cpp` | `InstructionOptimizer` |
|
||||
| `tools/check_discovery.py` | Discovery verifier (stdlib only) |
|
||||
|
||||
## 3. Static caps (see caps.h)
|
||||
|
||||
| Knob | Value |
|
||||
|---|---|
|
||||
| Text per prompt, response, field | 20000 chars |
|
||||
| Rendered conversation | 60000 chars, cut with a marker |
|
||||
| Turns per conversation | 50 |
|
||||
| System instruction | 8000 chars |
|
||||
| Rubrics per score / rubric text | 16 / 500 chars |
|
||||
| Trajectory calls | 100 |
|
||||
| evaluateInstances instance fields | 16 |
|
||||
| Scenarios per generate call | 100 (Google's ceiling) |
|
||||
| Run metrics, online evaluator metrics | 16 |
|
||||
| List pages / items | 20 pages (default 5), 1000 items |
|
||||
| `wait_run` polls | 200 at 2 s; operations use foundation caps (400 polls, 600 s) |
|
||||
| Simulation turns | default 5, max 12 |
|
||||
| Tool rounds per turn / tool calls per session | 5 / 40 |
|
||||
| Traces / spans per trace | 500 / 2000 |
|
||||
| Log entries / page / resource names / filter | 1000 / 100 / 100 / 20000 chars |
|
||||
| Failures analyzed / clusters | 200 (rest counted as `dropped`) / 10 |
|
||||
| Optimizer rounds | 3 |
|
||||
| Model JSON retries | 1 |
|
||||
| Online evaluator sampling | 1..100 percent; `maxEvaluatedSamplesPerRun` 1..100000 (0 means unbounded at Google and is refused) |
|
||||
|
||||
## 4. Contracts
|
||||
|
||||
- Every Google call is a row in `calls_table.inc`. `OptimizeService::invoke`
|
||||
expands `{x}` and `{+x}` path parameters (each segment `valid_segment`,
|
||||
resource names `projects/...`, max 12 segments), rejects query parameters
|
||||
and top-level body fields the row does not list, and only then sends.
|
||||
Invalid input is `GoogleError` 400 `INVALID_ARGUMENT` before any traffic;
|
||||
no credential is `NotConfiguredError` 503 before any traffic.
|
||||
- Resource names are checked against their collection
|
||||
(`projects/p/locations/l/{collection}/id`). A verb a collection lacks in
|
||||
Discovery (for example patch on `evaluationRuns`) is refused locally.
|
||||
- Reads come before mutations. Delete is `EvaluationResources::delete_resource`
|
||||
only, long-running, `wait=false` by default.
|
||||
- Long-running calls return an `Operation`; `wait=true` uses the foundation
|
||||
`wait_operation` (injectable sleeper).
|
||||
- `int64` fields (`userScenarioCount`, `maxEvaluatedSamplesPerRun`) are sent
|
||||
as JSON strings, as Discovery's JSON mapping requires.
|
||||
- Autorater: one model call scores the whole conversation against rubrics. A
|
||||
rubric the model did not answer fails with "no verdict returned"; it is
|
||||
never silently passed.
|
||||
- Clusterer output is verified: unknown and duplicate ids are discarded,
|
||||
clusters above the cap are cut, unplaced failures are listed in
|
||||
`unassigned`.
|
||||
- Optimizer `optimize()` returns the original instruction unless a scorer
|
||||
proves a strict improvement; `suggest()` never claims `improved`.
|
||||
- Simulator loops are bounded by turns, tool rounds and tool calls; the
|
||||
stop reason is reported (`user_done`, `max_turns`, `tool_round_cap`,
|
||||
`tool_call_cap`).
|
||||
- Credentials: this module never reads or prints them. `DirectModel::from_env`
|
||||
picks the endpoint: OAuth token plus project gives the project-scoped
|
||||
Vertex path (global location), an express key the Vertex publisher path,
|
||||
otherwise Generative Language.
|
||||
|
||||
## 5. Example routes (`/agent/optimize`)
|
||||
|
||||
GET `/agent/optimize` (page, matrix row), GET `/agent/optimize/status`.
|
||||
POST (503 `google_not_configured` without credentials or a valid
|
||||
`GOOGLE_CLOUD_PROJECT`, 422 on bad input, 502 `google_error` upstream):
|
||||
`evaluate`, `scenarios`, `runs`, `traces`, `logs` (Google APIs) and `score`,
|
||||
`cluster`, `suggest`, `simulate` (local logic on a Gemini model).
|
||||
|
||||
## 6. Verified against discovery
|
||||
|
||||
`python3 -I src/google/optimize/tools/check_discovery.py`, run 2026-10-24:
|
||||
`OK: 38 calls and 64 schemas match discovery`. It fetches
|
||||
`https://{aiplatform,cloudtrace,logging}.googleapis.com/$discovery/rest`
|
||||
(aiplatform v1, cloudtrace v1 and v2, logging v2) and fails on any declared
|
||||
call absent or different in verb, path, request fields or query parameters,
|
||||
and on any schema field or enum value the builders and parsers rely on.
|
||||
|
||||
## 7. Verified live
|
||||
|
||||
Date 2026-10-24, project from `GOOGLE_CLOUD_PROJECT` (not recorded here),
|
||||
opt-in test `tests/src/google/optimize/test_live.cpp` (`MOLO_LIVE_NET=1`,
|
||||
loud SKIP otherwise).
|
||||
|
||||
| Call | Credential | Result |
|
||||
|---|---|---|
|
||||
| `POST locations/us-central1:evaluateInstances` exactMatch, trajectoryExactMatch | express key | 200, scores parsed |
|
||||
| same, `safetyInput` (LLM judged) | express key and OAuth | 404 NOT_FOUND: "Failed to make GenerateContent request to autorater model ... gemini-2.5-flash". The credential is accepted; the project cannot use the default autorater model. LLM-judged metrics are therefore not verified end to end. |
|
||||
| `GET evaluationSets, evaluationItems, evaluationMetrics, evaluationRuns, onlineEvaluators ?pageSize=1` | express key | 401 UNAUTHENTICATED (API keys not supported) |
|
||||
| same five lists | OAuth (service account) | 200, empty lists |
|
||||
| `POST logging v2 entries:list` pageSize 1 | express key | 401 |
|
||||
| same | OAuth | 200, 1 entry |
|
||||
| `GET cloudtrace v1 traces?pageSize=1` | express key | 401 |
|
||||
| same | OAuth | 200, 0 traces |
|
||||
| Autorater and `DirectModel` against real `generateContent` (express key, Vertex publisher path; and OAuth, project path) | both | verdicts parsed, score 1.0 |
|
||||
|
||||
One run of the OAuth live suite showed 1 failed test of 5 (name not captured);
|
||||
the immediate rerun and a later run passed 5 of 5. Treat as an unexplained
|
||||
transient until reproduced.
|
||||
|
||||
Not run live (would create or change resources): create/patch/delete/import
|
||||
of sets, items, metrics, runs, online evaluators, `evaluateDataset`,
|
||||
`generateInstanceRubrics`, `generateUserScenarios`, `generateLossClusters`.
|
||||
These are verified by discovery only.
|
||||
|
||||
## 8. Unverified
|
||||
|
||||
- Neither discovery nor docs state: the `predefinedMetricSpec.metricSpecName`
|
||||
values, the `agentResource` name format of an online evaluator, the response
|
||||
type of `generateLossClusters` (discovery types it as a generic long-running
|
||||
response, so it is returned as raw `Json`), and what `cloudObservability`
|
||||
requires beyond `openTelemetry.semconvVersion`.
|
||||
- Cloud Trace span labels and Cloud Logging `jsonPayload` layout of agent
|
||||
telemetry (OTel GenAI semantic conventions) are not specified on the pages
|
||||
read. `ConversationBuilder` is a local heuristic (`jsonPayload.role` of
|
||||
user|model|assistant|agent plus `content`, `text` or `message`); `Trace`
|
||||
reads labels generically.
|
||||
- Docs mention `is_agentic=true` and `principal://` / `spiffe://` attribution
|
||||
in Cloud Logging; the exact log field was not read, so no filter helper
|
||||
uses it.
|
||||
- The default autorater model for LLM-judged metrics is unavailable to the
|
||||
test project (see 7); `AutoraterOptions::model` is the workaround, untested.
|
||||
- v1beta1 differences are not checked.
|
||||
|
||||
## 9. Gaps
|
||||
|
||||
- Cloud Trace v2 `batchWrite` and `createSpan` and Cloud Logging
|
||||
`entries.write/tail` are not wrapped: agents write through OTel exporters.
|
||||
The v2 rows are not in the table because nothing calls them.
|
||||
- Dashboards: only numbers (`DashboardData`); no Cloud Monitoring reads.
|
||||
- `evaluateDataset`, `generateInstanceRubrics`, `evaluationMetrics` and item
|
||||
bodies are passed as raw `Json` (checked at the top level against
|
||||
Discovery) rather than fully typed.
|
||||
- Simulation tool virtualization and the optimizer are local approximations,
|
||||
not Google's. Owner: `simulator.cpp`, `optimizer.cpp`.
|
||||
- Token refresh is the caller's job (foundation gap).
|
||||
|
||||
## Sources
|
||||
|
||||
- aiplatform v1 Discovery: https://aiplatform.googleapis.com/$discovery/rest?version=v1
|
||||
- Cloud Trace Discovery: https://cloudtrace.googleapis.com/$discovery/rest?version=v1 and `version=v2`
|
||||
- Cloud Logging Discovery: https://logging.googleapis.com/$discovery/rest?version=v2
|
||||
- https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/evaluate-simulated
|
||||
- https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/optimize-agent
|
||||
- https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/observability/overview
|
||||
- https://docs.cloud.google.com/trace/docs/reference/v1/rest and `.../v1/rest/v1/projects.traces/list`
|
||||
- https://docs.cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list
|
||||
- The agent platform REST reference page for `projects.locations.evaluateInstances` was fetched but its body was truncated; the request and response shapes come from Discovery.
|
||||
Reference in New Issue
Block a user