feat: implement MoloApp plans 01-15 with integration
Full C++23 framework tree: runtime, observability, security, data, app, realtime, supervisor, clients, CLI, example plus enterprise tenancy, audit, operability, governance, supplychain. Postgres via docker compose for tests.
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# Multi-instance topology (S7-9, E7-8)
|
||||
|
||||
N stateless app instances behind a load balancer, one shared durable bus,
|
||||
one shared session and KV store, and one scheduler leader lease in shared
|
||||
KV. Anything needed by more than one instance lives in a shared backend;
|
||||
local memory is never treated as global (R7-8).
|
||||
|
||||
## Pieces owned by this part
|
||||
|
||||
- `molo::bus::TopicBus`: ephemeral broadcast fan-out (at-most-once).
|
||||
- `molo::bus::StreamBus`: durable streams for catch-up replay (at-least-once).
|
||||
- `molo::tasks::Scheduler`: single leader via the shared-KV lease, passive
|
||||
followers, last-fire records in shared KV, idempotency-keyed fires.
|
||||
- `molo::replication::Replicator` plus `SyncSession`: oplog push plus
|
||||
catch-up pull between instances with per-model conflict policies.
|
||||
- `molo::supervisor::Supervisor::status_json()`: health input for endpoints.
|
||||
|
||||
## Pieces owned elsewhere
|
||||
|
||||
- 04-data.md: the shared session store and shared KV backend. This part
|
||||
programs against `molo::common::KvStore`; `InMemoryKvStore` is the
|
||||
single-node implementation. Multi-instance deployments swap in the
|
||||
shared backend without touching scheduler or queue code.
|
||||
- 06-realtime.md: room and broadcast semantics. The rule is: broadcast
|
||||
emit publishes once to the bus; every node re-emits to its local
|
||||
sockets. See `tests/src/supervisor/test_multi_instance.cpp`.
|
||||
- 09-cli-ops.md and 05-request-routing.md: health/readiness endpoints and
|
||||
the CLI surface. They read `status_json()` and drive `Supervisor` plus
|
||||
`TaskQueue` only through the declared APIs (R7-10).
|
||||
|
||||
## Load balancer: sticky sessions
|
||||
|
||||
WebSocket and Socket.IO long-polling transports must pin to one node (BP-6,
|
||||
S7-9), otherwise frames for one connection land on different nodes. Plain
|
||||
WS and SSE may pin or re-resolve per route (documented in 06-realtime.md).
|
||||
|
||||
Example (haproxy, cookie-based stickiness):
|
||||
|
||||
```
|
||||
backend molo_nodes
|
||||
balance roundrobin
|
||||
cookie MOLO_NODE insert indirect nocache
|
||||
server node1 10.0.0.11:8080 cookie n1 check
|
||||
server node2 10.0.0.12:8080 cookie n2 check
|
||||
```
|
||||
|
||||
Example (nginx, ip_hash pinning):
|
||||
|
||||
```
|
||||
upstream molo_nodes {
|
||||
ip_hash;
|
||||
server 10.0.0.11:8080;
|
||||
server 10.0.0.12:8080;
|
||||
}
|
||||
```
|
||||
|
||||
Health checks should hit the readiness endpoint (wired by 09-cli-ops.md
|
||||
from `status_json()`, queue depth, and sync lag gauges) and shed a node
|
||||
whose supervisor reports fatal workers or whose lag passes 10000 (S7-7).
|
||||
|
||||
## Broadcast and catch-up pattern
|
||||
|
||||
```cpp
|
||||
// Emit once (any node):
|
||||
live.publish("room:lobby", payload); // ephemeral fan-out
|
||||
bus.publish("roomlog", payload); // durable copy for reconnects
|
||||
|
||||
// Every live node re-emits to its local sockets:
|
||||
for (auto& m : live.poll(sub, 64)) local_emit(m.payload);
|
||||
|
||||
// A restarted node catches up what it missed while dead:
|
||||
for (auto& m : bus.poll("roomlog", node_id, 64)) { replay(m); ack(m.seq); }
|
||||
```
|
||||
|
||||
## Scheduler leadership
|
||||
|
||||
One leader per deployment (S7-5): lease key `molo/scheduler/leader`,
|
||||
10s TTL, 3s renewal. Follower takeover is bounded by the TTL (proven in
|
||||
`test_scheduler.cpp`). Dual-leader overlap during partitions is accepted
|
||||
with bounds (O7-4): overlapping fires carry the same idempotency key, so
|
||||
the queue dedupes them and no duplicate run happens.
|
||||
|
||||
## Known v1 limits
|
||||
|
||||
- Thread workers only; OS-process isolation is a v2 transport behind the
|
||||
same `WorkFunc` interface (O7-2).
|
||||
- In-house backends only; no Redis-protocol compatibility (O7-1).
|
||||
- At-least-once everywhere; no exactly-once primitive (O7-3). Handlers
|
||||
must be idempotent (R7-2).
|
||||
- The dead-letter queue is in-memory per process; entries do not survive
|
||||
a process restart yet. Queued (non-dead) work does survive via the
|
||||
durable op log.
|
||||
- The oplog keeps full history per instance; operators archive or trim
|
||||
`oplog.log` files on a schedule. Bounded per-round sync batches keep
|
||||
memory flat regardless of log size.
|
||||
- Durable streams use a dedicated append-only segment file per stream.
|
||||
Sharing the 04-data.md SQLite WAL path instead is still an open joint
|
||||
decision (O7-5); the `SegmentLog` seam keeps the swap local.
|
||||
Reference in New Issue
Block a user