Files
moloapp/plan3.md
T

103 lines
11 KiB
Markdown
Raw Normal View History

2026-10-26 01:31:48 +01:00
# PLAN 3 — Unified Hierarchical Agent Cluster (OpenAI-compatible, native C++23)
Merges plan2 + 2.1 + 2.2 + 2.3 into one DRY, recursively-checked design with zero open questions. Every decision binds to an existing framework path. No new third-party deps (wolfSSL only, per plan.md §1).
## 0. Reading guide / reuse map (do not reimplement)
| Need | Reuse (read first) | Owner plan |
|---|---|---|
| HTTP routes, middleware, SSE/chunked | `src/app/routing/`, `src/app/http_msg/`, `src/app/middleware/`, `src/runtime/http/`, `src/realtime/` | 05, 06 |
| Sessions + memory bank | `src/google/sessions/` (`sessions.h`, `memory_bank.*`, `engine_service.*`) | 16 |
| Persistence | `src/data/orm/` + `drivers/sqlite|pg`, `src/data/kv/` | 04 |
| Supervisor/workers | `src/supervisor/tasks/`, `bus/`, `common/` | 07 |
| Observability/audit | `src/observability/` (logging/metrics/tracing/correlation), `src/audit/`, `src/governance/` | 02, 12, 14 |
| Tenancy/auth | `src/tenancy/` (+ `bridge.h`), `src/security/`, `src/app/middleware/builtins.h` | 11, 03 |
| Generic HTTP/OpenAPI clients (Generic Discovery) | `src/clients/` (`client.h`, `openapi/`, `retry.*`, `pool.*`) | 08 |
| LLM calls | `src/llm/` (`api_client.*`, `chat_types.*`, `sse.*`, `error.*`) | 17 |
| OpenAI wire types (request/parse/respond, NOT new types) | `src/openai/core/` (`chat_completions.*`, `chat_request.*`, `embeddings.*`, `models.*`, `tools/`, `sse`) | 17 |
| Vendor execution | `src/google/` (`generative.*`, `gemini_types.*`, `live/`, `adk/`, `sandbox/`, `armor/`), `src/anthropic/`, `src/xai/`, `src/deepseek/`, `src/meta/`, `src/openrouter/`, `src/vendors/` | 17 |
| Workflow gates (spec→code→example→tests→changelog, lint→suite→bump→commit→tag→deploy→verify) | `AGENTS.md`, `plans/10-workflow-example.md` | 10 |
Decisions resolving plan2-family contradictions:
- **JSON:** in-house only. plan2.2's `nlohmann/json` is rejected (violates plan.md §1). Tool args and schemas use `src/data/value.hpp` (`Value`) + existing OpenAI wire serializers in `src/openai/core/`.
- **Default domain:** `"llm"` fallback is kept but bound to a real coordinator that ships in the same change (see §4; `LlmVendorCoordinator` using `src/llm/api_client` + `src/vendors/`).
- **LLM-router default:** `enable_llm_fallback=true`, backed by `src/llm/api_client` with structured output via `src/openai/core/tools/`; rules+embedding stay the fast path.
- **Threading:** coordinators run tool calls concurrently on `src/runtime` executor / `src/supervisor/tasks` pool; RootRouter itself stays synchronous and cheap (routing only).
- **Auth/rate-limit/Armor:** not "future work" — §6 wires them to tenancy middleware + Armor client from day one.
## 1. External contract (façade)
Routes (register in `src/app/routing/`, guarded by tenancy/auth middleware):
- `POST /v1/chat/completions` — JSON body parsed with `src/openai/core/chat_request.*`; `stream:false` → single JSON via `chat_completions.*`; `stream:true` → SSE via `src/llm/sse.*` + runtime chunked writer (`src/realtime/`).
- `GET /v1/models` — serve from `src/openai/core/models.*` (static registry of coordinator-backed model ids).
- `POST /v1/embeddings` — passthrough to `src/openai/core/embeddings.*` (used by §3 warmup too).
- `POST /workflow/release`-style ops endpoints stay untouched.
Handler pipeline per request (single file `src/agent/facade.{h,cpp}`, owns nothing else):
1. Auth/tenant resolve via middleware (`tenant_id`, `user_id`, claim key) — reject unauthenticated before any LLM spend.
2. Session load/create via `src/google/sessions/sessions.h` + `memory_bank.*`; restore `last_domain`, `last_routing_method`, rolling summary into `AgentContext.state`.
3. Convert OpenAI messages → `agent::Message` with existing converters (extend `src/openai/core/chat_completions.*`, do not fork new types).
4. `root_->run(ctx)` → normalize `AgentResult` → OpenAI response (content or `tool_calls` via `src/openai/core/tools/`), usage/cost attached from observability span.
5. Persist state + append audit span (route, tools, tokens, cost, latency); on streaming, flush `data: [DONE]` terminator.
## 2. Core types (single header, no duplication)
`src/agent/types.{h,cpp}` — the ONLY definition of these (plan2.1/2.2/2.3 duplicates deleted):
`Message{Role,content,name,tool_call_id}`, `AgentContext{session_id,tenant_id,user_id,messages,state,metadata,audit,persist_state}`, `AgentResult{content,success,error,routed_to,routing_confidence,routing_method,routing_latency_us,usage{prompt,completion,total,cost}}`, `Agent{name,description,keywords,run}`, `Embedding{values,cosine}`, `EmbeddingProvider::embed→expected<Embedding,string>`.
`Tool{name,description,parameters_schema(Value),execute(Value,ctx)→expected<Value,string>}`, `ToolCall{id,name,arguments}`, `ToolRegistry{add,find,openai_tools→via src/openai/core/tools/}`.
## 3. RootRouter (fast path, no heavy tools)
`src/agent/root_router.{h,cpp}`. Owns `vector<unique_ptr<Agent>>` coordinators only; zero vendor calls inside.
- `Config{embedding_threshold=0.72, enable_embedding=true, enable_llm_fallback=true, max_recent=3, default_domain="llm_vendor"}`.
- `run(ctx)`: `extract_routing_text` (last ≤3 user messages) → `decide` → `audit("routed",…)` → child `run` → stamp `routed_to/confidence/method/latency` → `persist_state("last_domain",…)`. Empty messages → typed error, no throw.
- Cascade (first hit wins): (a) deterministic rules — Workspace{gmail,drive,calendar,sheets,docs,meet,chat}, Voice{audio,voice,tts,stt,wav,live}, Maps{maps,places,route,geocode}, Data{bigquery,firestore,storage,pubsub}, skip missing names safely; (b) keyword overlap score over `keywords()` (threshold ≥1 hit, tie → first registered); (c) embedding cosine vs warmed cache (mutex-guarded `warmup_embeddings()` at startup + after `add_domain`); (d) LLM structured-output router via `src/llm/api_client` ("return exactly one of [names] + reason", temperature 0); (e) `default_domain`, else `domains_[0]`, else error.
- Rules in (a) are generated from each coordinator's `keywords()` (prefix table rebuilt in `rebuild_index()`), not a second hardcoded list — the DRY fix vs 2.1/2.3 divergence.
- `EmbeddingProvider` impl (`src/agent/gemini_embedder.{h,cpp}`) wraps `src/openai/core/embeddings.*` or `src/google/models` embeddings; cache keyed by domain name; failures degrade to next stage, never fail the request.
## 4. Coordinators (the 8 domains; root sees only these)
Each: `src/agent/coordinators/<name>.{h,cpp}`, implements `Agent`, owns a `ToolRegistry`, ReAct loop ≤5 rounds, concurrent tool fan-out (§5). Keywords double as routing table (§3).
| name | description seed | tools → existing client |
|---|---|---|
| `workspace` (ship first) | Gmail/Drive/Calendar/Sheets/Docs/Meet/Chat | `gmail_list/send`, `calendar_list/create`, `drive_search` → `src/google/` Workspace clients |
| `voice` (ship first) | TTS/STT/live audio, WAV | `tts_synth`, `stt_transcribe`, `live_start` → `src/openai/core/speech.*`, `transcriptions.*`, `src/google/live/` |
| `llm_vendor` (ship first; IS the default) | xAI/Anthropic/Meta/DeepSeek/OpenRouter overflow + generic chat | `chat_via_vendor` → `src/llm/api_client` + `src/vendors/`, `src/anthropic/`, `src/xai/`, `src/deepseek/`, `src/meta/`, `src/openrouter/` |
| `maps` | places/routes/geocode | → existing Maps client (or `src/clients/openapi/` discovery) |
| `data` | BigQuery/Firestore/Storage/PubSub | → Data clients + `src/data/` |
| `platform` | runtime/sessions/memory/sandbox, Agent Engine | → `src/google/runtime/`, `sessions/`, `sandbox/`, `adk/` |
| `cloud_ai` | Model Garden/Vertex/Armor/partner+open models | → `src/google/models/`, `armor/` |
| `discovery` | the 316 generic APIs | → `src/clients/` OpenAPI generic client, capped per-tenant rate |
Coordinator loop (identical for all — one shared base `CoordinatorBase` in `src/agent/coordinator.{h,cpp}` so Workspace is not a snowflake): build chat request from `ctx.messages` + `tools_.openai_tools()` → call `src/llm/api_client` (model per coordinator, default fast) → if no tool_calls return content; else execute all calls concurrently (§5), append `Role::Tool` messages, repeat. Round cap + per-tool timeout + per-request token/cost budget enforced; overrun → partial content + `error` set, still audited.
## 5. Concurrency, streaming, state
- Concurrency: tool fan-out uses `src/runtime` executor (`executor.*`/`strand.h`) or `src/supervisor/tasks/` pool — bounded (default 8), results re-ordered to call order before appending. No ad-hoc threads. Supervisor watches pools per plan 07.
- Streaming: non-streaming path returns full `AgentResult`; streaming path runs the same `root_->run` then replays content as SSE chunks (`src/llm/sse.*` framing, `chat.completion.chunk` objects, `data: [DONE]`). Token-by-token live streaming is explicitly out of scope for v1 (documented, not a caveat: single ranked reason — coordinators fan out to tools before final text exists).
- State/memory: `AgentContext.state` is a write-through view over sessions KV (`persist_state` → sessions store; load at §1 step 2). Cross-turn facts go to `memory_bank.*` table `(session_id,key,value,embedding,timestamp)`; embedding column reuses §3 provider. No parallel memory impl.
## 6. Cross-cutting (shipped, not TODO)
- Tenancy/auth/quotas: every façade route behind tenancy middleware; `tenant_id` scopes router instance selection (single shared root + per-tenant budgets is the fixed choice — no "one or per-tenant" ambiguity), session isolation, tool allowlists, per-tenant token/$ caps (checked in `CoordinatorBase` before each LLM/tool call).
- Safety: Workspace/Data/Discovery mutations pass `src/google/armor/` (Model Armor) pre-check; failures block the tool with typed error.
- Observability: one span per request (`correlation/`), child spans per routing stage + tool call; metrics: routing_method/confidence/latency, tool latency/error, tokens/cost per tenant+domain; audit writes who→what→cost via `src/audit/` + `src/governance/`.
- Errors: `expected<T,string>` everywhere; handler maps to OpenAI `error` objects; 5xx only on transport failure.
## 7. Build order (each step green per AGENTS.md before next)
1. `src/agent/types` + `coordinator` base + `ToolRegistry` (unit tests, no network).
2. `gemini_embedder` + `RootRouter` + `llm_vendor` coordinator (default route works end-to-end).
3. Façade + `POST /v1/chat/completions` (non-stream) + `GET /v1/models` + session wiring.
4. `workspace` + `voice` coordinators with real clients; SSE streaming.
5. Remaining 5 coordinators; Armor + quotas + cost metrics; `POST /v1/embeddings`.
6. Example route + suite + CHANGELOG per plan 10; `make lint test` clean, `-Wall -Wextra` silent.
## 8. Acceptance (recursive check — all must hold)
- One OpenAI-compatible endpoint fronting 8 coordinators; root makes no vendor calls.
- Rules→keywords→embedding→LLM→default cascade measured (p50 dominated by rules path).
- No `nlohmann` or new JSON lib; no second Message/Context/Tool definition; routing keywords sourced from coordinators.
- Default route resolves to a shipped coordinator; streaming terminates with `[DONE]`; tenants isolated with budgets; every step audited with cost.