Files
moloapp/src/supervisor/TOPOLOGY.md
T
retoor 3c4334cec9 feat: implement MoloApp plans 01-15 with integration
Full C++23 framework tree: runtime, observability, security,
data, app, realtime, supervisor, clients, CLI, example plus
enterprise tenancy, audit, operability, governance,
supplychain. Postgres via docker compose for tests.
2026-10-23 18:20:04 +02:00

3.9 KiB

Multi-instance topology (S7-9, E7-8)

N stateless app instances behind a load balancer, one shared durable bus, one shared session and KV store, and one scheduler leader lease in shared KV. Anything needed by more than one instance lives in a shared backend; local memory is never treated as global (R7-8).

Pieces owned by this part

  • molo::bus::TopicBus: ephemeral broadcast fan-out (at-most-once).
  • molo::bus::StreamBus: durable streams for catch-up replay (at-least-once).
  • molo::tasks::Scheduler: single leader via the shared-KV lease, passive followers, last-fire records in shared KV, idempotency-keyed fires.
  • molo::replication::Replicator plus SyncSession: oplog push plus catch-up pull between instances with per-model conflict policies.
  • molo::supervisor::Supervisor::status_json(): health input for endpoints.

Pieces owned elsewhere

  • 04-data.md: the shared session store and shared KV backend. This part programs against molo::common::KvStore; InMemoryKvStore is the single-node implementation. Multi-instance deployments swap in the shared backend without touching scheduler or queue code.
  • 06-realtime.md: room and broadcast semantics. The rule is: broadcast emit publishes once to the bus; every node re-emits to its local sockets. See tests/src/supervisor/test_multi_instance.cpp.
  • 09-cli-ops.md and 05-request-routing.md: health/readiness endpoints and the CLI surface. They read status_json() and drive Supervisor plus TaskQueue only through the declared APIs (R7-10).

Load balancer: sticky sessions

WebSocket and Socket.IO long-polling transports must pin to one node (BP-6, S7-9), otherwise frames for one connection land on different nodes. Plain WS and SSE may pin or re-resolve per route (documented in 06-realtime.md).

Example (haproxy, cookie-based stickiness):

backend molo_nodes
  balance roundrobin
  cookie MOLO_NODE insert indirect nocache
  server node1 10.0.0.11:8080 cookie n1 check
  server node2 10.0.0.12:8080 cookie n2 check

Example (nginx, ip_hash pinning):

upstream molo_nodes {
  ip_hash;
  server 10.0.0.11:8080;
  server 10.0.0.12:8080;
}

Health checks should hit the readiness endpoint (wired by 09-cli-ops.md from status_json(), queue depth, and sync lag gauges) and shed a node whose supervisor reports fatal workers or whose lag passes 10000 (S7-7).

Broadcast and catch-up pattern

// Emit once (any node):
live.publish("room:lobby", payload);  // ephemeral fan-out
bus.publish("roomlog", payload);      // durable copy for reconnects

// Every live node re-emits to its local sockets:
for (auto& m : live.poll(sub, 64)) local_emit(m.payload);

// A restarted node catches up what it missed while dead:
for (auto& m : bus.poll("roomlog", node_id, 64)) { replay(m); ack(m.seq); }

Scheduler leadership

One leader per deployment (S7-5): lease key molo/scheduler/leader, 10s TTL, 3s renewal. Follower takeover is bounded by the TTL (proven in test_scheduler.cpp). Dual-leader overlap during partitions is accepted with bounds (O7-4): overlapping fires carry the same idempotency key, so the queue dedupes them and no duplicate run happens.

Known v1 limits

  • Thread workers only; OS-process isolation is a v2 transport behind the same WorkFunc interface (O7-2).
  • In-house backends only; no Redis-protocol compatibility (O7-1).
  • At-least-once everywhere; no exactly-once primitive (O7-3). Handlers must be idempotent (R7-2).
  • The dead-letter queue is in-memory per process; entries do not survive a process restart yet. Queued (non-dead) work does survive via the durable op log.
  • The oplog keeps full history per instance; operators archive or trim oplog.log files on a schedule. Bounded per-round sync batches keep memory flat regardless of log size.
  • Durable streams use a dedicated append-only segment file per stream. Sharing the 04-data.md SQLite WAL path instead is still an open joint decision (O7-5); the SegmentLog seam keeps the swap local.