Files
moloapp/plan3.md
T
2026-10-26 01:31:48 +01:00

11 KiB

PLAN 3 — Unified Hierarchical Agent Cluster (OpenAI-compatible, native C++23)

Merges plan2 + 2.1 + 2.2 + 2.3 into one DRY, recursively-checked design with zero open questions. Every decision binds to an existing framework path. No new third-party deps (wolfSSL only, per plan.md §1).

0. Reading guide / reuse map (do not reimplement)

Need Reuse (read first) Owner plan
HTTP routes, middleware, SSE/chunked src/app/routing/, src/app/http_msg/, src/app/middleware/, src/runtime/http/, src/realtime/ 05, 06
Sessions + memory bank src/google/sessions/ (sessions.h, memory_bank.*, engine_service.*) 16
Persistence src/data/orm/ + `drivers/sqlite pg, src/data/kv/`
Supervisor/workers src/supervisor/tasks/, bus/, common/ 07
Observability/audit src/observability/ (logging/metrics/tracing/correlation), src/audit/, src/governance/ 02, 12, 14
Tenancy/auth src/tenancy/ (+ bridge.h), src/security/, src/app/middleware/builtins.h 11, 03
Generic HTTP/OpenAPI clients (Generic Discovery) src/clients/ (client.h, openapi/, retry.*, pool.*) 08
LLM calls src/llm/ (api_client.*, chat_types.*, sse.*, error.*) 17
OpenAI wire types (request/parse/respond, NOT new types) src/openai/core/ (chat_completions.*, chat_request.*, embeddings.*, models.*, tools/, sse) 17
Vendor execution src/google/ (generative.*, gemini_types.*, live/, adk/, sandbox/, armor/), src/anthropic/, src/xai/, src/deepseek/, src/meta/, src/openrouter/, src/vendors/ 17
Workflow gates (spec→code→example→tests→changelog, lint→suite→bump→commit→tag→deploy→verify) AGENTS.md, plans/10-workflow-example.md 10

Decisions resolving plan2-family contradictions:

  • JSON: in-house only. plan2.2's nlohmann/json is rejected (violates plan.md §1). Tool args and schemas use src/data/value.hpp (Value) + existing OpenAI wire serializers in src/openai/core/.
  • Default domain: "llm" fallback is kept but bound to a real coordinator that ships in the same change (see §4; LlmVendorCoordinator using src/llm/api_client + src/vendors/).
  • LLM-router default: enable_llm_fallback=true, backed by src/llm/api_client with structured output via src/openai/core/tools/; rules+embedding stay the fast path.
  • Threading: coordinators run tool calls concurrently on src/runtime executor / src/supervisor/tasks pool; RootRouter itself stays synchronous and cheap (routing only).
  • Auth/rate-limit/Armor: not "future work" — §6 wires them to tenancy middleware + Armor client from day one.

1. External contract (façade)

Routes (register in src/app/routing/, guarded by tenancy/auth middleware):

  • POST /v1/chat/completions — JSON body parsed with src/openai/core/chat_request.*; stream:false → single JSON via chat_completions.*; stream:true → SSE via src/llm/sse.* + runtime chunked writer (src/realtime/).
  • GET /v1/models — serve from src/openai/core/models.* (static registry of coordinator-backed model ids).
  • POST /v1/embeddings — passthrough to src/openai/core/embeddings.* (used by §3 warmup too).
  • POST /workflow/release-style ops endpoints stay untouched.

Handler pipeline per request (single file src/agent/facade.{h,cpp}, owns nothing else):

  1. Auth/tenant resolve via middleware (tenant_id, user_id, claim key) — reject unauthenticated before any LLM spend.
  2. Session load/create via src/google/sessions/sessions.h + memory_bank.*; restore last_domain, last_routing_method, rolling summary into AgentContext.state.
  3. Convert OpenAI messages → agent::Message with existing converters (extend src/openai/core/chat_completions.*, do not fork new types).
  4. root_->run(ctx) → normalize AgentResult → OpenAI response (content or tool_calls via src/openai/core/tools/), usage/cost attached from observability span.
  5. Persist state + append audit span (route, tools, tokens, cost, latency); on streaming, flush data: [DONE] terminator.

2. Core types (single header, no duplication)

src/agent/types.{h,cpp} — the ONLY definition of these (plan2.1/2.2/2.3 duplicates deleted): Message{Role,content,name,tool_call_id}, AgentContext{session_id,tenant_id,user_id,messages,state,metadata,audit,persist_state}, AgentResult{content,success,error,routed_to,routing_confidence,routing_method,routing_latency_us,usage{prompt,completion,total,cost}}, Agent{name,description,keywords,run}, Embedding{values,cosine}, EmbeddingProvider::embed→expected<Embedding,string>. Tool{name,description,parameters_schema(Value),execute(Value,ctx)→expected<Value,string>}, ToolCall{id,name,arguments}, ToolRegistry{add,find,openai_tools→via src/openai/core/tools/}.

3. RootRouter (fast path, no heavy tools)

src/agent/root_router.{h,cpp}. Owns vector<unique_ptr<Agent>> coordinators only; zero vendor calls inside.

  • Config{embedding_threshold=0.72, enable_embedding=true, enable_llm_fallback=true, max_recent=3, default_domain="llm_vendor"}.
  • run(ctx): extract_routing_text (last ≤3 user messages) → decide → audit("routed",…) → child run → stamp routed_to/confidence/method/latency → persist_state("last_domain",…). Empty messages → typed error, no throw.
  • Cascade (first hit wins): (a) deterministic rules — Workspace{gmail,drive,calendar,sheets,docs,meet,chat}, Voice{audio,voice,tts,stt,wav,live}, Maps{maps,places,route,geocode}, Data{bigquery,firestore,storage,pubsub}, skip missing names safely; (b) keyword overlap score over keywords() (threshold ≥1 hit, tie → first registered); (c) embedding cosine vs warmed cache (mutex-guarded warmup_embeddings() at startup + after add_domain); (d) LLM structured-output router via src/llm/api_client ("return exactly one of [names] + reason", temperature 0); (e) default_domain, else domains_[0], else error.
  • Rules in (a) are generated from each coordinator's keywords() (prefix table rebuilt in rebuild_index()), not a second hardcoded list — the DRY fix vs 2.1/2.3 divergence.
  • EmbeddingProvider impl (src/agent/gemini_embedder.{h,cpp}) wraps src/openai/core/embeddings.* or src/google/models embeddings; cache keyed by domain name; failures degrade to next stage, never fail the request.

4. Coordinators (the 8 domains; root sees only these)

Each: src/agent/coordinators/<name>.{h,cpp}, implements Agent, owns a ToolRegistry, ReAct loop ≤5 rounds, concurrent tool fan-out (§5). Keywords double as routing table (§3).

name description seed tools → existing client
workspace (ship first) Gmail/Drive/Calendar/Sheets/Docs/Meet/Chat gmail_list/send, calendar_list/create, drive_search → src/google/ Workspace clients
voice (ship first) TTS/STT/live audio, WAV tts_synth, stt_transcribe, live_start → src/openai/core/speech.*, transcriptions.*, src/google/live/
llm_vendor (ship first; IS the default) xAI/Anthropic/Meta/DeepSeek/OpenRouter overflow + generic chat chat_via_vendor → src/llm/api_client + src/vendors/, src/anthropic/, src/xai/, src/deepseek/, src/meta/, src/openrouter/
maps places/routes/geocode → existing Maps client (or src/clients/openapi/ discovery)
data BigQuery/Firestore/Storage/PubSub → Data clients + src/data/
platform runtime/sessions/memory/sandbox, Agent Engine → src/google/runtime/, sessions/, sandbox/, adk/
cloud_ai Model Garden/Vertex/Armor/partner+open models → src/google/models/, armor/
discovery the 316 generic APIs → src/clients/ OpenAPI generic client, capped per-tenant rate

Coordinator loop (identical for all — one shared base CoordinatorBase in src/agent/coordinator.{h,cpp} so Workspace is not a snowflake): build chat request from ctx.messages + tools_.openai_tools() → call src/llm/api_client (model per coordinator, default fast) → if no tool_calls return content; else execute all calls concurrently (§5), append Role::Tool messages, repeat. Round cap + per-tool timeout + per-request token/cost budget enforced; overrun → partial content + error set, still audited.

5. Concurrency, streaming, state

  • Concurrency: tool fan-out uses src/runtime executor (executor.*/strand.h) or src/supervisor/tasks/ pool — bounded (default 8), results re-ordered to call order before appending. No ad-hoc threads. Supervisor watches pools per plan 07.
  • Streaming: non-streaming path returns full AgentResult; streaming path runs the same root_->run then replays content as SSE chunks (src/llm/sse.* framing, chat.completion.chunk objects, data: [DONE]). Token-by-token live streaming is explicitly out of scope for v1 (documented, not a caveat: single ranked reason — coordinators fan out to tools before final text exists).
  • State/memory: AgentContext.state is a write-through view over sessions KV (persist_state → sessions store; load at §1 step 2). Cross-turn facts go to memory_bank.* table (session_id,key,value,embedding,timestamp); embedding column reuses §3 provider. No parallel memory impl.

6. Cross-cutting (shipped, not TODO)

  • Tenancy/auth/quotas: every façade route behind tenancy middleware; tenant_id scopes router instance selection (single shared root + per-tenant budgets is the fixed choice — no "one or per-tenant" ambiguity), session isolation, tool allowlists, per-tenant token/$ caps (checked in CoordinatorBase before each LLM/tool call).
  • Safety: Workspace/Data/Discovery mutations pass src/google/armor/ (Model Armor) pre-check; failures block the tool with typed error.
  • Observability: one span per request (correlation/), child spans per routing stage + tool call; metrics: routing_method/confidence/latency, tool latency/error, tokens/cost per tenant+domain; audit writes who→what→cost via src/audit/ + src/governance/.
  • Errors: expected<T,string> everywhere; handler maps to OpenAI error objects; 5xx only on transport failure.

7. Build order (each step green per AGENTS.md before next)

  1. src/agent/types + coordinator base + ToolRegistry (unit tests, no network).
  2. gemini_embedder + RootRouter + llm_vendor coordinator (default route works end-to-end).
  3. Façade + POST /v1/chat/completions (non-stream) + GET /v1/models + session wiring.
  4. workspace + voice coordinators with real clients; SSE streaming.
  5. Remaining 5 coordinators; Armor + quotas + cost metrics; POST /v1/embeddings.
  6. Example route + suite + CHANGELOG per plan 10; make lint test clean, -Wall -Wextra silent.

8. Acceptance (recursive check — all must hold)

  • One OpenAI-compatible endpoint fronting 8 coordinators; root makes no vendor calls.
  • Rules→keywords→embedding→LLM→default cascade measured (p50 dominated by rules path).
  • No nlohmann or new JSON lib; no second Message/Context/Tool definition; routing keywords sourced from coordinators.
  • Default route resolves to a shipped coordinator; streaming terminates with [DONE]; tenants isolated with budgets; every step audited with cost.