Full C++23 framework tree: runtime, observability, security, data, app, realtime, supervisor, clients, CLI, example plus enterprise tenancy, audit, operability, governance, supplychain. Postgres via docker compose for tests.
13 KiB
Part 13 operability: required base-tree edits
This file lists every edit the base tree needs to integrate the
src/operability/ part. Paths are repo-relative. Hunks are unified diffs
anchored on the seam symbols; this worktree does not contain the base src/
tree, so context line numbers are symbolic and the integrator anchors each
hunk on the named symbol. No hunk changes existing behavior when the new
code paths are not taken.
Conventions used by the new code:
- Includes use
-Isrc:"operability/...","supervisor/...", etc. - Namespaces are the leaf names from the seam spec:
molo::deploy,molo::coord,molo::resilience,molo::chaos,molo::debug, with shared caps inmolo::operability::Caps(src/operability/caps.h). - Production builds use
-Isrconly. TheADAPTERshims undersrc/operability/adapters/include/are standalone-only (seesrc/operability.mk,OPERABILITY_STANDALONE=1) and must never be on the production include path. - All new logic takes
molo::common::Clockandmolo::common::Observerby reference for frozen-clock tests. - Static caps live in
src/operability/caps.h. Nothing autotunes.
1. Makefile: include the operability fragment
Anchor: the per-part .mk include block (next to the other -include
lines such as the supervisor fragment).
--- a/Makefile
+++ b/Makefile
@@ -1,0 +1,1 @@
-include src/cli.mk
+-include src/operability.mk
Effect: builds build/liboperability.a and the operability,
operability-tests, operability-check targets.
2. src/supervisor/common/kv_store.hpp: fencing CAS extension
molo::coord::Fence and molo::coord::DistLock need compare-and-swap on
(key, expected token). This adds one virtual plus the in-memory
implementation; existing get/set/erase/acquire/renew signatures are
untouched.
--- a/src/supervisor/common/kv_store.hpp
+++ b/src/supervisor/common/kv_store.hpp
@@ class KvStore
virtual bool acquire(const std::string& key, const std::string& value, std::int64_t ttl_ms) = 0;
virtual bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) = 0;
+ // Fencing extension: set key to next only when its current value equals
+ // expected. Empty expected matches a missing key.
+ virtual bool cas(const std::string& key, const std::string& expected, const std::string& next,
+ std::int64_t ttl_ms) = 0;
--- a/src/supervisor/common/kv_store.hpp
+++ b/src/supervisor/common/kv_store.hpp
@@ class InMemoryKvStore
bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) override;
+ bool cas(const std::string& key, const std::string& expected, const std::string& next,
+ std::int64_t ttl_ms) override;
The InMemoryKvStore::cas body follows renew but stores next on match
and inserts when the key is missing and expected is empty (mirror the
ADAPTER shim at
src/operability/adapters/include/supervisor/common/kv_store.hpp, which is
the reference implementation). Every other KvStore backend must implement
cas with the same semantics; a backend that cannot do atomic CAS must
return false so callers treat fencing as unavailable.
3. src/cli/ops/ops_routes.hpp: debug handler declaration
Anchor: molo::ops::OpsRoutes, next to healthz/readyz/startupz/metrics.
--- a/src/cli/ops/ops_routes.hpp
+++ b/src/cli/ops/ops_routes.hpp
@@ class OpsRoutes
OpsResponse healthz() const;
OpsResponse readyz() const;
OpsResponse startupz() const;
OpsResponse metrics(bool gzip) const;
+ // Admin-only debug surface. Returns 404 unless the process was built with
+ // MOLO_ENABLE_DEBUG and the caller is the admin listener (see the .cpp).
+ OpsResponse debug(const std::string& path, const std::string& token, bool admin_listener) const;
OpsResponse needs no change ({status, content_type, body, gzip} covers
the debug responses; debug bodies are never gzipped by this handler).
4. src/cli/ops/ops_routes.cpp: debug handler definition
Anchor: after OpsRoutes::metrics. Delegates to molo::debug::Endpoint;
public listeners always get 404, and builds without MOLO_ENABLE_DEBUG
always get 404.
--- a/src/cli/ops/ops_routes.cpp
+++ b/src/cli/ops/ops_routes.cpp
@@
#include "cli/ops/ops_routes.hpp"
+#include "operability/debug/endpoint.h"
+#include "operability/debug/snapshot.h"
@@ OpsResponse OpsRoutes::metrics(bool gzip) const
...
+OpsResponse OpsRoutes::debug(const std::string& path, const std::string& token,
+ bool admin_listener) const {
+ OpsResponse r;
+ r.status = 404;
+ r.content_type = "text/plain";
+ r.body = "not found";
+ r.gzip = false;
+#ifdef MOLO_ENABLE_DEBUG
+ if (!admin_listener) return r;
+ // Auth gate: reuse the ops auth check used by the admin listener. The
+ // predicate below is the integration point, see note in section 7.
+ molo::debug::Endpoint ep([this] { return this->debug_snapshot(); },
+ [](const std::string& t) { return ops_admin_auth(t); });
+ const auto out = ep.handle(path, token);
+ r.status = out.status;
+ r.content_type = out.content_type;
+ r.body = out.body;
+#else
+ (void)path;
+ (void)token;
+ (void)admin_listener;
+#endif
+ return r;
+}
Notes:
debug_snapshot()is a small new private helper onOpsRoutes(declare in the hpp next todebug) that builds amolo::debug::Snapshot, registers sections, and returnscapture(). One suggested section per subsystem; keep each section small, the snapshot caps at 1 MB.ops_admin_authstands for the existing admin listener auth check (03 security gate owner). Wire the real predicate; never accept an empty token.- Routing rule, to be enforced at the listener setup call site (exact mount
file unconfirmed in the seam spec, see section 7):
GET /debug/*routes toOpsRoutes::debugwithadmin_listener=trueon the admin listener only. The public listener must not route/debug/*at all (404).
5. src/supervisor/tasks/scheduler.hpp: elect via coord
Anchor: molo::tasks::Scheduler private lease members. The scheduler keeps
its public surface (tick_once(), is_leader()); only the mechanism moves
onto molo::coord::LeaderElection.
--- a/src/supervisor/tasks/scheduler.hpp
+++ b/src/supervisor/tasks/scheduler.hpp
@@
#include "supervisor/tasks/scheduler.hpp"
+#include "operability/coord/election.h"
@@ class Scheduler
private:
- // old internal lease fields (leader key, ttl, renew deadline)
+ molo::coord::LeaderElection election_;
Construct election_ with the existing queue/kv/clock/observer members:
key "molo/scheduler/leader" (keep the current Options::leader_key
default), ttl 10000, renew 3000, node id from the scheduler options or
worker id.
6. src/supervisor/tasks/scheduler.cpp: delegate to the election
Anchor: Scheduler::tick_once and Scheduler::is_leader.
--- a/src/supervisor/tasks/scheduler.cpp
+++ b/src/supervisor/tasks/scheduler.cpp
@@ bool Scheduler::tick_once()
- // old elect_locked lease handling
- ...
+ const bool leader = election_.tick();
+ if (!leader) return false; // follower: no dispatch this tick
... // existing dispatch work, unchanged
@@ bool Scheduler::is_leader() const
- return lease_held_locked();
+ return election_.is_leader();
Delete the replaced private lease helpers. Observer metrics keep their
existing names; the election additionally emits election.won/lost events
and election_renewed_total.
7. src/clients/request.h: deadline header helper
Anchor: molo::clients::ClientRequest, next to header().
--- a/src/clients/request.h
+++ b/src/clients/request.h
@@ class ClientRequest
// existing header/timeouts accessors
+ // Absolute X-Deadline-Ms value when present and well formed.
+ std::optional<std::int64_t> deadline_ms() const;
Implement in the existing request source file with
molo::resilience::Deadline::parse_header on the X-Deadline-Ms header:
std::optional<std::int64_t> ClientRequest::deadline_ms() const {
const auto v = header("X-Deadline-Ms"); // existing accessor
if (!v) return std::nullopt;
return molo::resilience::Deadline::parse_header(*v);
}
(Adjust to the real header() return type: string, optional, or out-param.)
8. src/clients/client.cpp: breaker plus deadline propagation
Anchor: molo::clients::Client::send, before send_inner. Order:
deadline outermost, then shedder/bulkhead/breaker, per the resilience
wiring (exact middleware ordering slot in src/app is unconfirmed, see
section 9; the client-side half below is exact).
--- a/src/clients/client.cpp
+++ b/src/clients/client.cpp
@@
#include "clients/client.h"
+#include "operability/resilience/breaker.h"
+#include "operability/resilience/deadline.h"
@@ Task<ClientResponse> Client::send(ClientRequest req)
+ // 1. Breaker gate, innermost stage: never retry into an open breaker.
+ // `breaker_` is a per-host molo::resilience::Breaker owned by Client.
+ if (!breaker_.allow()) {
+ co_return ClientResponse::failure(503, "breaker open");
+ }
+ // 2. Deadline propagation: convert the absolute inbound X-Deadline-Ms to
+ // remaining budget minus the 50 ms floor margin for the outbound hop.
+ if (const auto abs = req.deadline_ms()) {
+ molo::resilience::Deadline dl(clock_);
+ const std::int64_t budget = dl.remaining_ms(*abs);
+ if (budget == 0) {
+ breaker_.on_failure();
+ co_return ClientResponse::failure(504, "deadline exceeded");
+ }
+ req.set_header("X-Deadline-Ms",
+ molo::resilience::Deadline::format_header(clock_.wall_ms() + budget));
+ reqmutable_timeouts_for(budget); // clamp hop timeouts to budget
+ }
auto resp = co_await send_inner(std::move(req));
+ if (resp.ok()) {
+ breaker_.on_success();
+ } else if (resp.status() >= 500 || resp.transport_error()) {
+ breaker_.on_failure();
+ }
co_return resp;
Notes:
breaker_needs amolo::common::Clockandmolo::common::ObserveratClientconstruction; add them to the existing client config or constructor the same way timeouts are threaded today.ClientResponse::failure,resp.ok(),resp.status(),resp.transport_error()are placeholders for the real response API: count transport errors and 5xx as breaker failures, everything else as success. Never report a shed 503 from downstream as a local failure twice; the downstream already counted it.- Retry policy (
RetryPolicy::should_retry_status/backoff_for) must skip retries whilebreaker_.allow()is false and must honor downstreamRetry-AfterviaRetryPolicy::parse_retry_after. reqmutable_timeouts_forstands for clamping the outbound hop timeouts (Timeouts{connect,read,write,pool_acquire,total}) to the propagated budget. KeepDeadline::budget_for(phase)shares proportional.
9. Unconfirmed sites (from the seam spec, kept as gaps)
These need the base tree to pin the exact file; the hunks above are written so they apply once the site is known:
- Middleware ordering slot for deadline outermost then
shedder/bulkhead/breaker in
src/app(05 request routing). The per-request wiring is:Deadlinecheck,Shedder::should_shed(503 plusRetry-Afteron shed),Bulkhead::try_acquire(shed onkRejected), then dispatch;Breakerstays innermost at the client (section 8). - Admin listener bind site that mounts
OpsRoutes(section 4 routing rule). Public listener must 404/debug/*. - Realtime close path (06, connection close 1001) and auth gate owner
(03 security):
Lifecycle::run_drainphase 2 should close realtime connections with 1001; the debug auth predicate must be the 03 gate.
10. CLI wiring (rollout, rollback, experiment, debug)
molo::cli::Command(name, brief) with flag()/child()/handler() is the
seam (src/cli/cli/command.hpp). Register four commands at the existing
command registry (same site that registers supervisor/task commands):
rollout start --strategy rolling|blue-green --batch N --max-unavailable Mdrivingmolo::deploy::Rolling/molo::deploy::BlueGreenstep loops with readiness probes pollingreadyz/startupz.rollout abortandrollbackdrivingStrategy::abortandBlueGreen::rollback.experiment start|abort --blast N --duration MS --max-burn Xdrivingmolo::chaos::Experimentplusmolo::chaos::Injectorhook setup.debug snapshotprintingmolo::debug::Snapshot::capture().
Handlers return ExitCode and write through Output&; no new CLI
framework code is needed. Exact registry file to be confirmed against the
base tree.
11. Verification
Standalone (this worktree, no base tree):
make -f src/operability.mk OPERABILITY_STANDALONE=1 operability-check
This builds build/liboperability.a with
g++ -std=c++23 -O2 -Wall -Wextra -Wpedantic -Werror -pthread and runs
build/operability_tests (with MOLO_ENABLE_DEBUG) plus
build/operability_tests_nodebug (without it). All eight suites
(coord, lifecycle, breaker/bulkhead, shed/deadline, deploy, slo, chaos,
debug) run on frozen clocks with no real waiting.
Production (base tree present): drop OPERABILITY_STANDALONE=1 so only
-Isrc is used and the real
supervisor/common/{clock,kv_store,observer}.hpp headers satisfy the
includes. The ADAPTER shims must not be on the include path.