feat: implement MoloApp plans 01-15 with integration

Full C++23 framework tree: runtime, observability, security,
data, app, realtime, supervisor, clients, CLI, example plus
enterprise tenancy, audit, operability, governance,
supplychain. Postgres via docker compose for tests.
This commit is contained in:
2026-10-23 18:20:04 +02:00
parent 54068f1953
commit 3c4334cec9
1068 changed files with 100899 additions and 2 deletions
+334
View File
@@ -0,0 +1,334 @@
# Part 13 operability: required base-tree edits
This file lists every edit the base tree needs to integrate the
`src/operability/` part. Paths are repo-relative. Hunks are unified diffs
anchored on the seam symbols; this worktree does not contain the base `src/`
tree, so context line numbers are symbolic and the integrator anchors each
hunk on the named symbol. No hunk changes existing behavior when the new
code paths are not taken.
Conventions used by the new code:
- Includes use `-Isrc`: `"operability/..."`, `"supervisor/..."`, etc.
- Namespaces are the leaf names from the seam spec: `molo::deploy`,
`molo::coord`, `molo::resilience`, `molo::chaos`, `molo::debug`, with shared
caps in `molo::operability::Caps` (`src/operability/caps.h`).
- Production builds use `-Isrc` only. The `ADAPTER` shims under
`src/operability/adapters/include/` are standalone-only (see
`src/operability.mk`, `OPERABILITY_STANDALONE=1`) and must never be on the
production include path.
- All new logic takes `molo::common::Clock` and `molo::common::Observer` by
reference for frozen-clock tests.
- Static caps live in `src/operability/caps.h`. Nothing autotunes.
## 1. `Makefile`: include the operability fragment
Anchor: the per-part `.mk` include block (next to the other `-include`
lines such as the supervisor fragment).
```diff
--- a/Makefile
+++ b/Makefile
@@ -1,0 +1,1 @@
-include src/cli.mk
+-include src/operability.mk
```
Effect: builds `build/liboperability.a` and the `operability`,
`operability-tests`, `operability-check` targets.
## 2. `src/supervisor/common/kv_store.hpp`: fencing CAS extension
`molo::coord::Fence` and `molo::coord::DistLock` need compare-and-swap on
(key, expected token). This adds one virtual plus the in-memory
implementation; existing `get/set/erase/acquire/renew` signatures are
untouched.
```diff
--- a/src/supervisor/common/kv_store.hpp
+++ b/src/supervisor/common/kv_store.hpp
@@ class KvStore
virtual bool acquire(const std::string& key, const std::string& value, std::int64_t ttl_ms) = 0;
virtual bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) = 0;
+ // Fencing extension: set key to next only when its current value equals
+ // expected. Empty expected matches a missing key.
+ virtual bool cas(const std::string& key, const std::string& expected, const std::string& next,
+ std::int64_t ttl_ms) = 0;
```
```diff
--- a/src/supervisor/common/kv_store.hpp
+++ b/src/supervisor/common/kv_store.hpp
@@ class InMemoryKvStore
bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) override;
+ bool cas(const std::string& key, const std::string& expected, const std::string& next,
+ std::int64_t ttl_ms) override;
```
The `InMemoryKvStore::cas` body follows `renew` but stores `next` on match
and inserts when the key is missing and `expected` is empty (mirror the
ADAPTER shim at
`src/operability/adapters/include/supervisor/common/kv_store.hpp`, which is
the reference implementation). Every other `KvStore` backend must implement
`cas` with the same semantics; a backend that cannot do atomic CAS must
return false so callers treat fencing as unavailable.
## 3. `src/cli/ops/ops_routes.hpp`: debug handler declaration
Anchor: `molo::ops::OpsRoutes`, next to `healthz/readyz/startupz/metrics`.
```diff
--- a/src/cli/ops/ops_routes.hpp
+++ b/src/cli/ops/ops_routes.hpp
@@ class OpsRoutes
OpsResponse healthz() const;
OpsResponse readyz() const;
OpsResponse startupz() const;
OpsResponse metrics(bool gzip) const;
+ // Admin-only debug surface. Returns 404 unless the process was built with
+ // MOLO_ENABLE_DEBUG and the caller is the admin listener (see the .cpp).
+ OpsResponse debug(const std::string& path, const std::string& token, bool admin_listener) const;
```
`OpsResponse` needs no change (`{status, content_type, body, gzip}` covers
the debug responses; debug bodies are never gzipped by this handler).
## 4. `src/cli/ops/ops_routes.cpp`: debug handler definition
Anchor: after `OpsRoutes::metrics`. Delegates to `molo::debug::Endpoint`;
public listeners always get 404, and builds without `MOLO_ENABLE_DEBUG`
always get 404.
```diff
--- a/src/cli/ops/ops_routes.cpp
+++ b/src/cli/ops/ops_routes.cpp
@@
#include "cli/ops/ops_routes.hpp"
+#include "operability/debug/endpoint.h"
+#include "operability/debug/snapshot.h"
@@ OpsResponse OpsRoutes::metrics(bool gzip) const
...
+OpsResponse OpsRoutes::debug(const std::string& path, const std::string& token,
+ bool admin_listener) const {
+ OpsResponse r;
+ r.status = 404;
+ r.content_type = "text/plain";
+ r.body = "not found";
+ r.gzip = false;
+#ifdef MOLO_ENABLE_DEBUG
+ if (!admin_listener) return r;
+ // Auth gate: reuse the ops auth check used by the admin listener. The
+ // predicate below is the integration point, see note in section 7.
+ molo::debug::Endpoint ep([this] { return this->debug_snapshot(); },
+ [](const std::string& t) { return ops_admin_auth(t); });
+ const auto out = ep.handle(path, token);
+ r.status = out.status;
+ r.content_type = out.content_type;
+ r.body = out.body;
+#else
+ (void)path;
+ (void)token;
+ (void)admin_listener;
+#endif
+ return r;
+}
```
Notes:
- `debug_snapshot()` is a small new private helper on `OpsRoutes` (declare
in the hpp next to `debug`) that builds a `molo::debug::Snapshot`,
registers sections, and returns `capture()`. One suggested section per
subsystem; keep each section small, the snapshot caps at 1 MB.
- `ops_admin_auth` stands for the existing admin listener auth check (03
security gate owner). Wire the real predicate; never accept an empty
token.
- Routing rule, to be enforced at the listener setup call site (exact mount
file unconfirmed in the seam spec, see section 7): `GET /debug/*` routes
to `OpsRoutes::debug` with `admin_listener=true` on the admin listener
only. The public listener must not route `/debug/*` at all (404).
## 5. `src/supervisor/tasks/scheduler.hpp`: elect via coord
Anchor: `molo::tasks::Scheduler` private lease members. The scheduler keeps
its public surface (`tick_once()`, `is_leader()`); only the mechanism moves
onto `molo::coord::LeaderElection`.
```diff
--- a/src/supervisor/tasks/scheduler.hpp
+++ b/src/supervisor/tasks/scheduler.hpp
@@
#include "supervisor/tasks/scheduler.hpp"
+#include "operability/coord/election.h"
@@ class Scheduler
private:
- // old internal lease fields (leader key, ttl, renew deadline)
+ molo::coord::LeaderElection election_;
```
Construct `election_` with the existing queue/kv/clock/observer members:
key `"molo/scheduler/leader"` (keep the current `Options::leader_key`
default), ttl 10000, renew 3000, node id from the scheduler options or
worker id.
## 6. `src/supervisor/tasks/scheduler.cpp`: delegate to the election
Anchor: `Scheduler::tick_once` and `Scheduler::is_leader`.
```diff
--- a/src/supervisor/tasks/scheduler.cpp
+++ b/src/supervisor/tasks/scheduler.cpp
@@ bool Scheduler::tick_once()
- // old elect_locked lease handling
- ...
+ const bool leader = election_.tick();
+ if (!leader) return false; // follower: no dispatch this tick
... // existing dispatch work, unchanged
@@ bool Scheduler::is_leader() const
- return lease_held_locked();
+ return election_.is_leader();
```
Delete the replaced private lease helpers. Observer metrics keep their
existing names; the election additionally emits `election.won/lost` events
and `election_renewed_total`.
## 7. `src/clients/request.h`: deadline header helper
Anchor: `molo::clients::ClientRequest`, next to `header()`.
```diff
--- a/src/clients/request.h
+++ b/src/clients/request.h
@@ class ClientRequest
// existing header/timeouts accessors
+ // Absolute X-Deadline-Ms value when present and well formed.
+ std::optional<std::int64_t> deadline_ms() const;
```
Implement in the existing request source file with
`molo::resilience::Deadline::parse_header` on the `X-Deadline-Ms` header:
```cpp
std::optional<std::int64_t> ClientRequest::deadline_ms() const {
const auto v = header("X-Deadline-Ms"); // existing accessor
if (!v) return std::nullopt;
return molo::resilience::Deadline::parse_header(*v);
}
```
(Adjust to the real `header()` return type: string, optional, or out-param.)
## 8. `src/clients/client.cpp`: breaker plus deadline propagation
Anchor: `molo::clients::Client::send`, before `send_inner`. Order:
deadline outermost, then shedder/bulkhead/breaker, per the resilience
wiring (exact middleware ordering slot in `src/app` is unconfirmed, see
section 9; the client-side half below is exact).
```diff
--- a/src/clients/client.cpp
+++ b/src/clients/client.cpp
@@
#include "clients/client.h"
+#include "operability/resilience/breaker.h"
+#include "operability/resilience/deadline.h"
@@ Task<ClientResponse> Client::send(ClientRequest req)
+ // 1. Breaker gate, innermost stage: never retry into an open breaker.
+ // `breaker_` is a per-host molo::resilience::Breaker owned by Client.
+ if (!breaker_.allow()) {
+ co_return ClientResponse::failure(503, "breaker open");
+ }
+ // 2. Deadline propagation: convert the absolute inbound X-Deadline-Ms to
+ // remaining budget minus the 50 ms floor margin for the outbound hop.
+ if (const auto abs = req.deadline_ms()) {
+ molo::resilience::Deadline dl(clock_);
+ const std::int64_t budget = dl.remaining_ms(*abs);
+ if (budget == 0) {
+ breaker_.on_failure();
+ co_return ClientResponse::failure(504, "deadline exceeded");
+ }
+ req.set_header("X-Deadline-Ms",
+ molo::resilience::Deadline::format_header(clock_.wall_ms() + budget));
+ reqmutable_timeouts_for(budget); // clamp hop timeouts to budget
+ }
auto resp = co_await send_inner(std::move(req));
+ if (resp.ok()) {
+ breaker_.on_success();
+ } else if (resp.status() >= 500 || resp.transport_error()) {
+ breaker_.on_failure();
+ }
co_return resp;
```
Notes:
- `breaker_` needs a `molo::common::Clock` and `molo::common::Observer` at
`Client` construction; add them to the existing client config or
constructor the same way timeouts are threaded today.
- `ClientResponse::failure`, `resp.ok()`, `resp.status()`,
`resp.transport_error()` are placeholders for the real response API:
count transport errors and 5xx as breaker failures, everything else as
success. Never report a shed 503 from downstream as a local failure
twice; the downstream already counted it.
- Retry policy (`RetryPolicy::should_retry_status/backoff_for`) must skip
retries while `breaker_.allow()` is false and must honor downstream
`Retry-After` via `RetryPolicy::parse_retry_after`.
- `reqmutable_timeouts_for` stands for clamping the outbound hop timeouts
(`Timeouts{connect,read,write,pool_acquire,total}`) to the propagated
budget. Keep `Deadline::budget_for(phase)` shares proportional.
## 9. Unconfirmed sites (from the seam spec, kept as gaps)
These need the base tree to pin the exact file; the hunks above are written
so they apply once the site is known:
- Middleware ordering slot for deadline outermost then
shedder/bulkhead/breaker in `src/app` (05 request routing). The
per-request wiring is: `Deadline` check, `Shedder::should_shed` (503 plus
`Retry-After` on shed), `Bulkhead::try_acquire` (shed on `kRejected`),
then dispatch; `Breaker` stays innermost at the client (section 8).
- Admin listener bind site that mounts `OpsRoutes` (section 4 routing
rule). Public listener must 404 `/debug/*`.
- Realtime close path (06, connection close 1001) and auth gate owner
(03 security): `Lifecycle::run_drain` phase 2 should close realtime
connections with 1001; the debug auth predicate must be the 03 gate.
## 10. CLI wiring (rollout, rollback, experiment, debug)
`molo::cli::Command(name, brief)` with `flag()/child()/handler()` is the
seam (`src/cli/cli/command.hpp`). Register four commands at the existing
command registry (same site that registers supervisor/task commands):
- `rollout start --strategy rolling|blue-green --batch N --max-unavailable M`
driving `molo::deploy::Rolling` / `molo::deploy::BlueGreen` step loops
with readiness probes polling `readyz/startupz`.
- `rollout abort` and `rollback` driving `Strategy::abort` and
`BlueGreen::rollback`.
- `experiment start|abort --blast N --duration MS --max-burn X` driving
`molo::chaos::Experiment` plus `molo::chaos::Injector` hook setup.
- `debug snapshot` printing `molo::debug::Snapshot::capture()`.
Handlers return `ExitCode` and write through `Output&`; no new CLI
framework code is needed. Exact registry file to be confirmed against the
base tree.
## 11. Verification
Standalone (this worktree, no base tree):
```sh
make -f src/operability.mk OPERABILITY_STANDALONE=1 operability-check
```
This builds `build/liboperability.a` with
`g++ -std=c++23 -O2 -Wall -Wextra -Wpedantic -Werror -pthread` and runs
`build/operability_tests` (with `MOLO_ENABLE_DEBUG`) plus
`build/operability_tests_nodebug` (without it). All eight suites
(coord, lifecycle, breaker/bulkhead, shed/deadline, deploy, slo, chaos,
debug) run on frozen clocks with no real waiting.
Production (base tree present): drop `OPERABILITY_STANDALONE=1` so only
`-Isrc` is used and the real
`supervisor/common/{clock,kv_store,observer}.hpp` headers satisfy the
includes. The ADAPTER shims must not be on the include path.