11 KiB
PLAN 3 — Unified Hierarchical Agent Cluster (OpenAI-compatible, native C++23)
Merges plan2 + 2.1 + 2.2 + 2.3 into one DRY, recursively-checked design with zero open questions. Every decision binds to an existing framework path. No new third-party deps (wolfSSL only, per plan.md §1).
0. Reading guide / reuse map (do not reimplement)
| Need | Reuse (read first) | Owner plan |
|---|---|---|
| HTTP routes, middleware, SSE/chunked | src/app/routing/, src/app/http_msg/, src/app/middleware/, src/runtime/http/, src/realtime/ |
05, 06 |
| Sessions + memory bank | src/google/sessions/ (sessions.h, memory_bank.*, engine_service.*) |
16 |
| Persistence | src/data/orm/ + `drivers/sqlite |
pg, src/data/kv/` |
| Supervisor/workers | src/supervisor/tasks/, bus/, common/ |
07 |
| Observability/audit | src/observability/ (logging/metrics/tracing/correlation), src/audit/, src/governance/ |
02, 12, 14 |
| Tenancy/auth | src/tenancy/ (+ bridge.h), src/security/, src/app/middleware/builtins.h |
11, 03 |
| Generic HTTP/OpenAPI clients (Generic Discovery) | src/clients/ (client.h, openapi/, retry.*, pool.*) |
08 |
| LLM calls | src/llm/ (api_client.*, chat_types.*, sse.*, error.*) |
17 |
| OpenAI wire types (request/parse/respond, NOT new types) | src/openai/core/ (chat_completions.*, chat_request.*, embeddings.*, models.*, tools/, sse) |
17 |
| Vendor execution | src/google/ (generative.*, gemini_types.*, live/, adk/, sandbox/, armor/), src/anthropic/, src/xai/, src/deepseek/, src/meta/, src/openrouter/, src/vendors/ |
17 |
| Workflow gates (spec→code→example→tests→changelog, lint→suite→bump→commit→tag→deploy→verify) | AGENTS.md, plans/10-workflow-example.md |
10 |
Decisions resolving plan2-family contradictions:
- JSON: in-house only. plan2.2's
nlohmann/jsonis rejected (violates plan.md §1). Tool args and schemas usesrc/data/value.hpp(Value) + existing OpenAI wire serializers insrc/openai/core/. - Default domain:
"llm"fallback is kept but bound to a real coordinator that ships in the same change (see §4;LlmVendorCoordinatorusingsrc/llm/api_client+src/vendors/). - LLM-router default:
enable_llm_fallback=true, backed bysrc/llm/api_clientwith structured output viasrc/openai/core/tools/; rules+embedding stay the fast path. - Threading: coordinators run tool calls concurrently on
src/runtimeexecutor /src/supervisor/taskspool; RootRouter itself stays synchronous and cheap (routing only). - Auth/rate-limit/Armor: not "future work" — §6 wires them to tenancy middleware + Armor client from day one.
1. External contract (façade)
Routes (register in src/app/routing/, guarded by tenancy/auth middleware):
POST /v1/chat/completions— JSON body parsed withsrc/openai/core/chat_request.*;stream:false→ single JSON viachat_completions.*;stream:true→ SSE viasrc/llm/sse.*+ runtime chunked writer (src/realtime/).GET /v1/models— serve fromsrc/openai/core/models.*(static registry of coordinator-backed model ids).POST /v1/embeddings— passthrough tosrc/openai/core/embeddings.*(used by §3 warmup too).POST /workflow/release-style ops endpoints stay untouched.
Handler pipeline per request (single file src/agent/facade.{h,cpp}, owns nothing else):
- Auth/tenant resolve via middleware (
tenant_id,user_id, claim key) — reject unauthenticated before any LLM spend. - Session load/create via
src/google/sessions/sessions.h+memory_bank.*; restorelast_domain,last_routing_method, rolling summary intoAgentContext.state. - Convert OpenAI messages →
agent::Messagewith existing converters (extendsrc/openai/core/chat_completions.*, do not fork new types). root_->run(ctx)→ normalizeAgentResult→ OpenAI response (content ortool_callsviasrc/openai/core/tools/), usage/cost attached from observability span.- Persist state + append audit span (route, tools, tokens, cost, latency); on streaming, flush
data: [DONE]terminator.
2. Core types (single header, no duplication)
src/agent/types.{h,cpp} — the ONLY definition of these (plan2.1/2.2/2.3 duplicates deleted):
Message{Role,content,name,tool_call_id}, AgentContext{session_id,tenant_id,user_id,messages,state,metadata,audit,persist_state}, AgentResult{content,success,error,routed_to,routing_confidence,routing_method,routing_latency_us,usage{prompt,completion,total,cost}}, Agent{name,description,keywords,run}, Embedding{values,cosine}, EmbeddingProvider::embed→expected<Embedding,string>.
Tool{name,description,parameters_schema(Value),execute(Value,ctx)→expected<Value,string>}, ToolCall{id,name,arguments}, ToolRegistry{add,find,openai_tools→via src/openai/core/tools/}.
3. RootRouter (fast path, no heavy tools)
src/agent/root_router.{h,cpp}. Owns vector<unique_ptr<Agent>> coordinators only; zero vendor calls inside.
Config{embedding_threshold=0.72, enable_embedding=true, enable_llm_fallback=true, max_recent=3, default_domain="llm_vendor"}.run(ctx):extract_routing_text(last ≤3 user messages) →decide→audit("routed",…)→ childrun→ stamprouted_to/confidence/method/latency→persist_state("last_domain",…). Empty messages → typed error, no throw.- Cascade (first hit wins): (a) deterministic rules — Workspace{gmail,drive,calendar,sheets,docs,meet,chat}, Voice{audio,voice,tts,stt,wav,live}, Maps{maps,places,route,geocode}, Data{bigquery,firestore,storage,pubsub}, skip missing names safely; (b) keyword overlap score over
keywords()(threshold ≥1 hit, tie → first registered); (c) embedding cosine vs warmed cache (mutex-guardedwarmup_embeddings()at startup + afteradd_domain); (d) LLM structured-output router viasrc/llm/api_client("return exactly one of [names] + reason", temperature 0); (e)default_domain, elsedomains_[0], else error. - Rules in (a) are generated from each coordinator's
keywords()(prefix table rebuilt inrebuild_index()), not a second hardcoded list — the DRY fix vs 2.1/2.3 divergence. EmbeddingProviderimpl (src/agent/gemini_embedder.{h,cpp}) wrapssrc/openai/core/embeddings.*orsrc/google/modelsembeddings; cache keyed by domain name; failures degrade to next stage, never fail the request.
4. Coordinators (the 8 domains; root sees only these)
Each: src/agent/coordinators/<name>.{h,cpp}, implements Agent, owns a ToolRegistry, ReAct loop ≤5 rounds, concurrent tool fan-out (§5). Keywords double as routing table (§3).
| name | description seed | tools → existing client |
|---|---|---|
workspace (ship first) |
Gmail/Drive/Calendar/Sheets/Docs/Meet/Chat | gmail_list/send, calendar_list/create, drive_search → src/google/ Workspace clients |
voice (ship first) |
TTS/STT/live audio, WAV | tts_synth, stt_transcribe, live_start → src/openai/core/speech.*, transcriptions.*, src/google/live/ |
llm_vendor (ship first; IS the default) |
xAI/Anthropic/Meta/DeepSeek/OpenRouter overflow + generic chat | chat_via_vendor → src/llm/api_client + src/vendors/, src/anthropic/, src/xai/, src/deepseek/, src/meta/, src/openrouter/ |
maps |
places/routes/geocode | → existing Maps client (or src/clients/openapi/ discovery) |
data |
BigQuery/Firestore/Storage/PubSub | → Data clients + src/data/ |
platform |
runtime/sessions/memory/sandbox, Agent Engine | → src/google/runtime/, sessions/, sandbox/, adk/ |
cloud_ai |
Model Garden/Vertex/Armor/partner+open models | → src/google/models/, armor/ |
discovery |
the 316 generic APIs | → src/clients/ OpenAPI generic client, capped per-tenant rate |
Coordinator loop (identical for all — one shared base CoordinatorBase in src/agent/coordinator.{h,cpp} so Workspace is not a snowflake): build chat request from ctx.messages + tools_.openai_tools() → call src/llm/api_client (model per coordinator, default fast) → if no tool_calls return content; else execute all calls concurrently (§5), append Role::Tool messages, repeat. Round cap + per-tool timeout + per-request token/cost budget enforced; overrun → partial content + error set, still audited.
5. Concurrency, streaming, state
- Concurrency: tool fan-out uses
src/runtimeexecutor (executor.*/strand.h) orsrc/supervisor/tasks/pool — bounded (default 8), results re-ordered to call order before appending. No ad-hoc threads. Supervisor watches pools per plan 07. - Streaming: non-streaming path returns full
AgentResult; streaming path runs the sameroot_->runthen replays content as SSE chunks (src/llm/sse.*framing,chat.completion.chunkobjects,data: [DONE]). Token-by-token live streaming is explicitly out of scope for v1 (documented, not a caveat: single ranked reason — coordinators fan out to tools before final text exists). - State/memory:
AgentContext.stateis a write-through view over sessions KV (persist_state→ sessions store; load at §1 step 2). Cross-turn facts go tomemory_bank.*table(session_id,key,value,embedding,timestamp); embedding column reuses §3 provider. No parallel memory impl.
6. Cross-cutting (shipped, not TODO)
- Tenancy/auth/quotas: every façade route behind tenancy middleware;
tenant_idscopes router instance selection (single shared root + per-tenant budgets is the fixed choice — no "one or per-tenant" ambiguity), session isolation, tool allowlists, per-tenant token/$ caps (checked inCoordinatorBasebefore each LLM/tool call). - Safety: Workspace/Data/Discovery mutations pass
src/google/armor/(Model Armor) pre-check; failures block the tool with typed error. - Observability: one span per request (
correlation/), child spans per routing stage + tool call; metrics: routing_method/confidence/latency, tool latency/error, tokens/cost per tenant+domain; audit writes who→what→cost viasrc/audit/+src/governance/. - Errors:
expected<T,string>everywhere; handler maps to OpenAIerrorobjects; 5xx only on transport failure.
7. Build order (each step green per AGENTS.md before next)
src/agent/types+coordinatorbase +ToolRegistry(unit tests, no network).gemini_embedder+RootRouter+llm_vendorcoordinator (default route works end-to-end).- Façade +
POST /v1/chat/completions(non-stream) +GET /v1/models+ session wiring. workspace+voicecoordinators with real clients; SSE streaming.- Remaining 5 coordinators; Armor + quotas + cost metrics;
POST /v1/embeddings. - Example route + suite + CHANGELOG per plan 10;
make lint testclean,-Wall -Wextrasilent.
8. Acceptance (recursive check — all must hold)
- One OpenAI-compatible endpoint fronting 8 coordinators; root makes no vendor calls.
- Rules→keywords→embedding→LLM→default cascade measured (p50 dominated by rules path).
- No
nlohmannor new JSON lib; no second Message/Context/Tool definition; routing keywords sourced from coordinators. - Default route resolves to a shipped coordinator; streaming terminates with
[DONE]; tenants isolated with budgets; every step audited with cost.