feat: implement MoloApp plans 01-15 with integration
Full C++23 framework tree: runtime, observability, security, data, app, realtime, supervisor, clients, CLI, example plus enterprise tenancy, audit, operability, governance, supplychain. Postgres via docker compose for tests.
This commit is contained in:
@@ -0,0 +1,334 @@
|
||||
# Part 13 operability: required base-tree edits
|
||||
|
||||
This file lists every edit the base tree needs to integrate the
|
||||
`src/operability/` part. Paths are repo-relative. Hunks are unified diffs
|
||||
anchored on the seam symbols; this worktree does not contain the base `src/`
|
||||
tree, so context line numbers are symbolic and the integrator anchors each
|
||||
hunk on the named symbol. No hunk changes existing behavior when the new
|
||||
code paths are not taken.
|
||||
|
||||
Conventions used by the new code:
|
||||
|
||||
- Includes use `-Isrc`: `"operability/..."`, `"supervisor/..."`, etc.
|
||||
- Namespaces are the leaf names from the seam spec: `molo::deploy`,
|
||||
`molo::coord`, `molo::resilience`, `molo::chaos`, `molo::debug`, with shared
|
||||
caps in `molo::operability::Caps` (`src/operability/caps.h`).
|
||||
- Production builds use `-Isrc` only. The `ADAPTER` shims under
|
||||
`src/operability/adapters/include/` are standalone-only (see
|
||||
`src/operability.mk`, `OPERABILITY_STANDALONE=1`) and must never be on the
|
||||
production include path.
|
||||
- All new logic takes `molo::common::Clock` and `molo::common::Observer` by
|
||||
reference for frozen-clock tests.
|
||||
- Static caps live in `src/operability/caps.h`. Nothing autotunes.
|
||||
|
||||
## 1. `Makefile`: include the operability fragment
|
||||
|
||||
Anchor: the per-part `.mk` include block (next to the other `-include`
|
||||
lines such as the supervisor fragment).
|
||||
|
||||
```diff
|
||||
--- a/Makefile
|
||||
+++ b/Makefile
|
||||
@@ -1,0 +1,1 @@
|
||||
-include src/cli.mk
|
||||
+-include src/operability.mk
|
||||
```
|
||||
|
||||
Effect: builds `build/liboperability.a` and the `operability`,
|
||||
`operability-tests`, `operability-check` targets.
|
||||
|
||||
## 2. `src/supervisor/common/kv_store.hpp`: fencing CAS extension
|
||||
|
||||
`molo::coord::Fence` and `molo::coord::DistLock` need compare-and-swap on
|
||||
(key, expected token). This adds one virtual plus the in-memory
|
||||
implementation; existing `get/set/erase/acquire/renew` signatures are
|
||||
untouched.
|
||||
|
||||
```diff
|
||||
--- a/src/supervisor/common/kv_store.hpp
|
||||
+++ b/src/supervisor/common/kv_store.hpp
|
||||
@@ class KvStore
|
||||
virtual bool acquire(const std::string& key, const std::string& value, std::int64_t ttl_ms) = 0;
|
||||
virtual bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) = 0;
|
||||
+ // Fencing extension: set key to next only when its current value equals
|
||||
+ // expected. Empty expected matches a missing key.
|
||||
+ virtual bool cas(const std::string& key, const std::string& expected, const std::string& next,
|
||||
+ std::int64_t ttl_ms) = 0;
|
||||
```
|
||||
|
||||
```diff
|
||||
--- a/src/supervisor/common/kv_store.hpp
|
||||
+++ b/src/supervisor/common/kv_store.hpp
|
||||
@@ class InMemoryKvStore
|
||||
bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) override;
|
||||
+ bool cas(const std::string& key, const std::string& expected, const std::string& next,
|
||||
+ std::int64_t ttl_ms) override;
|
||||
```
|
||||
|
||||
The `InMemoryKvStore::cas` body follows `renew` but stores `next` on match
|
||||
and inserts when the key is missing and `expected` is empty (mirror the
|
||||
ADAPTER shim at
|
||||
`src/operability/adapters/include/supervisor/common/kv_store.hpp`, which is
|
||||
the reference implementation). Every other `KvStore` backend must implement
|
||||
`cas` with the same semantics; a backend that cannot do atomic CAS must
|
||||
return false so callers treat fencing as unavailable.
|
||||
|
||||
## 3. `src/cli/ops/ops_routes.hpp`: debug handler declaration
|
||||
|
||||
Anchor: `molo::ops::OpsRoutes`, next to `healthz/readyz/startupz/metrics`.
|
||||
|
||||
```diff
|
||||
--- a/src/cli/ops/ops_routes.hpp
|
||||
+++ b/src/cli/ops/ops_routes.hpp
|
||||
@@ class OpsRoutes
|
||||
OpsResponse healthz() const;
|
||||
OpsResponse readyz() const;
|
||||
OpsResponse startupz() const;
|
||||
OpsResponse metrics(bool gzip) const;
|
||||
+ // Admin-only debug surface. Returns 404 unless the process was built with
|
||||
+ // MOLO_ENABLE_DEBUG and the caller is the admin listener (see the .cpp).
|
||||
+ OpsResponse debug(const std::string& path, const std::string& token, bool admin_listener) const;
|
||||
```
|
||||
|
||||
`OpsResponse` needs no change (`{status, content_type, body, gzip}` covers
|
||||
the debug responses; debug bodies are never gzipped by this handler).
|
||||
|
||||
## 4. `src/cli/ops/ops_routes.cpp`: debug handler definition
|
||||
|
||||
Anchor: after `OpsRoutes::metrics`. Delegates to `molo::debug::Endpoint`;
|
||||
public listeners always get 404, and builds without `MOLO_ENABLE_DEBUG`
|
||||
always get 404.
|
||||
|
||||
```diff
|
||||
--- a/src/cli/ops/ops_routes.cpp
|
||||
+++ b/src/cli/ops/ops_routes.cpp
|
||||
@@
|
||||
#include "cli/ops/ops_routes.hpp"
|
||||
+#include "operability/debug/endpoint.h"
|
||||
+#include "operability/debug/snapshot.h"
|
||||
@@ OpsResponse OpsRoutes::metrics(bool gzip) const
|
||||
...
|
||||
+OpsResponse OpsRoutes::debug(const std::string& path, const std::string& token,
|
||||
+ bool admin_listener) const {
|
||||
+ OpsResponse r;
|
||||
+ r.status = 404;
|
||||
+ r.content_type = "text/plain";
|
||||
+ r.body = "not found";
|
||||
+ r.gzip = false;
|
||||
+#ifdef MOLO_ENABLE_DEBUG
|
||||
+ if (!admin_listener) return r;
|
||||
+ // Auth gate: reuse the ops auth check used by the admin listener. The
|
||||
+ // predicate below is the integration point, see note in section 7.
|
||||
+ molo::debug::Endpoint ep([this] { return this->debug_snapshot(); },
|
||||
+ [](const std::string& t) { return ops_admin_auth(t); });
|
||||
+ const auto out = ep.handle(path, token);
|
||||
+ r.status = out.status;
|
||||
+ r.content_type = out.content_type;
|
||||
+ r.body = out.body;
|
||||
+#else
|
||||
+ (void)path;
|
||||
+ (void)token;
|
||||
+ (void)admin_listener;
|
||||
+#endif
|
||||
+ return r;
|
||||
+}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `debug_snapshot()` is a small new private helper on `OpsRoutes` (declare
|
||||
in the hpp next to `debug`) that builds a `molo::debug::Snapshot`,
|
||||
registers sections, and returns `capture()`. One suggested section per
|
||||
subsystem; keep each section small, the snapshot caps at 1 MB.
|
||||
- `ops_admin_auth` stands for the existing admin listener auth check (03
|
||||
security gate owner). Wire the real predicate; never accept an empty
|
||||
token.
|
||||
- Routing rule, to be enforced at the listener setup call site (exact mount
|
||||
file unconfirmed in the seam spec, see section 7): `GET /debug/*` routes
|
||||
to `OpsRoutes::debug` with `admin_listener=true` on the admin listener
|
||||
only. The public listener must not route `/debug/*` at all (404).
|
||||
|
||||
## 5. `src/supervisor/tasks/scheduler.hpp`: elect via coord
|
||||
|
||||
Anchor: `molo::tasks::Scheduler` private lease members. The scheduler keeps
|
||||
its public surface (`tick_once()`, `is_leader()`); only the mechanism moves
|
||||
onto `molo::coord::LeaderElection`.
|
||||
|
||||
```diff
|
||||
--- a/src/supervisor/tasks/scheduler.hpp
|
||||
+++ b/src/supervisor/tasks/scheduler.hpp
|
||||
@@
|
||||
#include "supervisor/tasks/scheduler.hpp"
|
||||
+#include "operability/coord/election.h"
|
||||
@@ class Scheduler
|
||||
private:
|
||||
- // old internal lease fields (leader key, ttl, renew deadline)
|
||||
+ molo::coord::LeaderElection election_;
|
||||
```
|
||||
|
||||
Construct `election_` with the existing queue/kv/clock/observer members:
|
||||
key `"molo/scheduler/leader"` (keep the current `Options::leader_key`
|
||||
default), ttl 10000, renew 3000, node id from the scheduler options or
|
||||
worker id.
|
||||
|
||||
## 6. `src/supervisor/tasks/scheduler.cpp`: delegate to the election
|
||||
|
||||
Anchor: `Scheduler::tick_once` and `Scheduler::is_leader`.
|
||||
|
||||
```diff
|
||||
--- a/src/supervisor/tasks/scheduler.cpp
|
||||
+++ b/src/supervisor/tasks/scheduler.cpp
|
||||
@@ bool Scheduler::tick_once()
|
||||
- // old elect_locked lease handling
|
||||
- ...
|
||||
+ const bool leader = election_.tick();
|
||||
+ if (!leader) return false; // follower: no dispatch this tick
|
||||
... // existing dispatch work, unchanged
|
||||
@@ bool Scheduler::is_leader() const
|
||||
- return lease_held_locked();
|
||||
+ return election_.is_leader();
|
||||
```
|
||||
|
||||
Delete the replaced private lease helpers. Observer metrics keep their
|
||||
existing names; the election additionally emits `election.won/lost` events
|
||||
and `election_renewed_total`.
|
||||
|
||||
## 7. `src/clients/request.h`: deadline header helper
|
||||
|
||||
Anchor: `molo::clients::ClientRequest`, next to `header()`.
|
||||
|
||||
```diff
|
||||
--- a/src/clients/request.h
|
||||
+++ b/src/clients/request.h
|
||||
@@ class ClientRequest
|
||||
// existing header/timeouts accessors
|
||||
+ // Absolute X-Deadline-Ms value when present and well formed.
|
||||
+ std::optional<std::int64_t> deadline_ms() const;
|
||||
```
|
||||
|
||||
Implement in the existing request source file with
|
||||
`molo::resilience::Deadline::parse_header` on the `X-Deadline-Ms` header:
|
||||
|
||||
```cpp
|
||||
std::optional<std::int64_t> ClientRequest::deadline_ms() const {
|
||||
const auto v = header("X-Deadline-Ms"); // existing accessor
|
||||
if (!v) return std::nullopt;
|
||||
return molo::resilience::Deadline::parse_header(*v);
|
||||
}
|
||||
```
|
||||
|
||||
(Adjust to the real `header()` return type: string, optional, or out-param.)
|
||||
|
||||
## 8. `src/clients/client.cpp`: breaker plus deadline propagation
|
||||
|
||||
Anchor: `molo::clients::Client::send`, before `send_inner`. Order:
|
||||
deadline outermost, then shedder/bulkhead/breaker, per the resilience
|
||||
wiring (exact middleware ordering slot in `src/app` is unconfirmed, see
|
||||
section 9; the client-side half below is exact).
|
||||
|
||||
```diff
|
||||
--- a/src/clients/client.cpp
|
||||
+++ b/src/clients/client.cpp
|
||||
@@
|
||||
#include "clients/client.h"
|
||||
+#include "operability/resilience/breaker.h"
|
||||
+#include "operability/resilience/deadline.h"
|
||||
@@ Task<ClientResponse> Client::send(ClientRequest req)
|
||||
+ // 1. Breaker gate, innermost stage: never retry into an open breaker.
|
||||
+ // `breaker_` is a per-host molo::resilience::Breaker owned by Client.
|
||||
+ if (!breaker_.allow()) {
|
||||
+ co_return ClientResponse::failure(503, "breaker open");
|
||||
+ }
|
||||
+ // 2. Deadline propagation: convert the absolute inbound X-Deadline-Ms to
|
||||
+ // remaining budget minus the 50 ms floor margin for the outbound hop.
|
||||
+ if (const auto abs = req.deadline_ms()) {
|
||||
+ molo::resilience::Deadline dl(clock_);
|
||||
+ const std::int64_t budget = dl.remaining_ms(*abs);
|
||||
+ if (budget == 0) {
|
||||
+ breaker_.on_failure();
|
||||
+ co_return ClientResponse::failure(504, "deadline exceeded");
|
||||
+ }
|
||||
+ req.set_header("X-Deadline-Ms",
|
||||
+ molo::resilience::Deadline::format_header(clock_.wall_ms() + budget));
|
||||
+ reqmutable_timeouts_for(budget); // clamp hop timeouts to budget
|
||||
+ }
|
||||
auto resp = co_await send_inner(std::move(req));
|
||||
+ if (resp.ok()) {
|
||||
+ breaker_.on_success();
|
||||
+ } else if (resp.status() >= 500 || resp.transport_error()) {
|
||||
+ breaker_.on_failure();
|
||||
+ }
|
||||
co_return resp;
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `breaker_` needs a `molo::common::Clock` and `molo::common::Observer` at
|
||||
`Client` construction; add them to the existing client config or
|
||||
constructor the same way timeouts are threaded today.
|
||||
- `ClientResponse::failure`, `resp.ok()`, `resp.status()`,
|
||||
`resp.transport_error()` are placeholders for the real response API:
|
||||
count transport errors and 5xx as breaker failures, everything else as
|
||||
success. Never report a shed 503 from downstream as a local failure
|
||||
twice; the downstream already counted it.
|
||||
- Retry policy (`RetryPolicy::should_retry_status/backoff_for`) must skip
|
||||
retries while `breaker_.allow()` is false and must honor downstream
|
||||
`Retry-After` via `RetryPolicy::parse_retry_after`.
|
||||
- `reqmutable_timeouts_for` stands for clamping the outbound hop timeouts
|
||||
(`Timeouts{connect,read,write,pool_acquire,total}`) to the propagated
|
||||
budget. Keep `Deadline::budget_for(phase)` shares proportional.
|
||||
|
||||
## 9. Unconfirmed sites (from the seam spec, kept as gaps)
|
||||
|
||||
These need the base tree to pin the exact file; the hunks above are written
|
||||
so they apply once the site is known:
|
||||
|
||||
- Middleware ordering slot for deadline outermost then
|
||||
shedder/bulkhead/breaker in `src/app` (05 request routing). The
|
||||
per-request wiring is: `Deadline` check, `Shedder::should_shed` (503 plus
|
||||
`Retry-After` on shed), `Bulkhead::try_acquire` (shed on `kRejected`),
|
||||
then dispatch; `Breaker` stays innermost at the client (section 8).
|
||||
- Admin listener bind site that mounts `OpsRoutes` (section 4 routing
|
||||
rule). Public listener must 404 `/debug/*`.
|
||||
- Realtime close path (06, connection close 1001) and auth gate owner
|
||||
(03 security): `Lifecycle::run_drain` phase 2 should close realtime
|
||||
connections with 1001; the debug auth predicate must be the 03 gate.
|
||||
|
||||
## 10. CLI wiring (rollout, rollback, experiment, debug)
|
||||
|
||||
`molo::cli::Command(name, brief)` with `flag()/child()/handler()` is the
|
||||
seam (`src/cli/cli/command.hpp`). Register four commands at the existing
|
||||
command registry (same site that registers supervisor/task commands):
|
||||
|
||||
- `rollout start --strategy rolling|blue-green --batch N --max-unavailable M`
|
||||
driving `molo::deploy::Rolling` / `molo::deploy::BlueGreen` step loops
|
||||
with readiness probes polling `readyz/startupz`.
|
||||
- `rollout abort` and `rollback` driving `Strategy::abort` and
|
||||
`BlueGreen::rollback`.
|
||||
- `experiment start|abort --blast N --duration MS --max-burn X` driving
|
||||
`molo::chaos::Experiment` plus `molo::chaos::Injector` hook setup.
|
||||
- `debug snapshot` printing `molo::debug::Snapshot::capture()`.
|
||||
|
||||
Handlers return `ExitCode` and write through `Output&`; no new CLI
|
||||
framework code is needed. Exact registry file to be confirmed against the
|
||||
base tree.
|
||||
|
||||
## 11. Verification
|
||||
|
||||
Standalone (this worktree, no base tree):
|
||||
|
||||
```sh
|
||||
make -f src/operability.mk OPERABILITY_STANDALONE=1 operability-check
|
||||
```
|
||||
|
||||
This builds `build/liboperability.a` with
|
||||
`g++ -std=c++23 -O2 -Wall -Wextra -Wpedantic -Werror -pthread` and runs
|
||||
`build/operability_tests` (with `MOLO_ENABLE_DEBUG`) plus
|
||||
`build/operability_tests_nodebug` (without it). All eight suites
|
||||
(coord, lifecycle, breaker/bulkhead, shed/deadline, deploy, slo, chaos,
|
||||
debug) run on frozen clocks with no real waiting.
|
||||
|
||||
Production (base tree present): drop `OPERABILITY_STANDALONE=1` so only
|
||||
`-Isrc` is used and the real
|
||||
`supervisor/common/{clock,kv_store,observer}.hpp` headers satisfy the
|
||||
includes. The ADAPTER shims must not be on the include path.
|
||||
Reference in New Issue
Block a user