# Part 13 operability: required base-tree edits This file lists every edit the base tree needs to integrate the `src/operability/` part. Paths are repo-relative. Hunks are unified diffs anchored on the seam symbols; this worktree does not contain the base `src/` tree, so context line numbers are symbolic and the integrator anchors each hunk on the named symbol. No hunk changes existing behavior when the new code paths are not taken. Conventions used by the new code: - Includes use `-Isrc`: `"operability/..."`, `"supervisor/..."`, etc. - Namespaces are the leaf names from the seam spec: `molo::deploy`, `molo::coord`, `molo::resilience`, `molo::chaos`, `molo::debug`, with shared caps in `molo::operability::Caps` (`src/operability/caps.h`). - Production builds use `-Isrc` only. The `ADAPTER` shims under `src/operability/adapters/include/` are standalone-only (see `src/operability.mk`, `OPERABILITY_STANDALONE=1`) and must never be on the production include path. - All new logic takes `molo::common::Clock` and `molo::common::Observer` by reference for frozen-clock tests. - Static caps live in `src/operability/caps.h`. Nothing autotunes. ## 1. `Makefile`: include the operability fragment Anchor: the per-part `.mk` include block (next to the other `-include` lines such as the supervisor fragment). ```diff --- a/Makefile +++ b/Makefile @@ -1,0 +1,1 @@ -include src/cli.mk +-include src/operability.mk ``` Effect: builds `build/liboperability.a` and the `operability`, `operability-tests`, `operability-check` targets. ## 2. `src/supervisor/common/kv_store.hpp`: fencing CAS extension `molo::coord::Fence` and `molo::coord::DistLock` need compare-and-swap on (key, expected token). This adds one virtual plus the in-memory implementation; existing `get/set/erase/acquire/renew` signatures are untouched. ```diff --- a/src/supervisor/common/kv_store.hpp +++ b/src/supervisor/common/kv_store.hpp @@ class KvStore virtual bool acquire(const std::string& key, const std::string& value, std::int64_t ttl_ms) = 0; virtual bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) = 0; + // Fencing extension: set key to next only when its current value equals + // expected. Empty expected matches a missing key. + virtual bool cas(const std::string& key, const std::string& expected, const std::string& next, + std::int64_t ttl_ms) = 0; ``` ```diff --- a/src/supervisor/common/kv_store.hpp +++ b/src/supervisor/common/kv_store.hpp @@ class InMemoryKvStore bool renew(const std::string& key, const std::string& expected, std::int64_t ttl_ms) override; + bool cas(const std::string& key, const std::string& expected, const std::string& next, + std::int64_t ttl_ms) override; ``` The `InMemoryKvStore::cas` body follows `renew` but stores `next` on match and inserts when the key is missing and `expected` is empty (mirror the ADAPTER shim at `src/operability/adapters/include/supervisor/common/kv_store.hpp`, which is the reference implementation). Every other `KvStore` backend must implement `cas` with the same semantics; a backend that cannot do atomic CAS must return false so callers treat fencing as unavailable. ## 3. `src/cli/ops/ops_routes.hpp`: debug handler declaration Anchor: `molo::ops::OpsRoutes`, next to `healthz/readyz/startupz/metrics`. ```diff --- a/src/cli/ops/ops_routes.hpp +++ b/src/cli/ops/ops_routes.hpp @@ class OpsRoutes OpsResponse healthz() const; OpsResponse readyz() const; OpsResponse startupz() const; OpsResponse metrics(bool gzip) const; + // Admin-only debug surface. Returns 404 unless the process was built with + // MOLO_ENABLE_DEBUG and the caller is the admin listener (see the .cpp). + OpsResponse debug(const std::string& path, const std::string& token, bool admin_listener) const; ``` `OpsResponse` needs no change (`{status, content_type, body, gzip}` covers the debug responses; debug bodies are never gzipped by this handler). ## 4. `src/cli/ops/ops_routes.cpp`: debug handler definition Anchor: after `OpsRoutes::metrics`. Delegates to `molo::debug::Endpoint`; public listeners always get 404, and builds without `MOLO_ENABLE_DEBUG` always get 404. ```diff --- a/src/cli/ops/ops_routes.cpp +++ b/src/cli/ops/ops_routes.cpp @@ #include "cli/ops/ops_routes.hpp" +#include "operability/debug/endpoint.h" +#include "operability/debug/snapshot.h" @@ OpsResponse OpsRoutes::metrics(bool gzip) const ... +OpsResponse OpsRoutes::debug(const std::string& path, const std::string& token, + bool admin_listener) const { + OpsResponse r; + r.status = 404; + r.content_type = "text/plain"; + r.body = "not found"; + r.gzip = false; +#ifdef MOLO_ENABLE_DEBUG + if (!admin_listener) return r; + // Auth gate: reuse the ops auth check used by the admin listener. The + // predicate below is the integration point, see note in section 7. + molo::debug::Endpoint ep([this] { return this->debug_snapshot(); }, + [](const std::string& t) { return ops_admin_auth(t); }); + const auto out = ep.handle(path, token); + r.status = out.status; + r.content_type = out.content_type; + r.body = out.body; +#else + (void)path; + (void)token; + (void)admin_listener; +#endif + return r; +} ``` Notes: - `debug_snapshot()` is a small new private helper on `OpsRoutes` (declare in the hpp next to `debug`) that builds a `molo::debug::Snapshot`, registers sections, and returns `capture()`. One suggested section per subsystem; keep each section small, the snapshot caps at 1 MB. - `ops_admin_auth` stands for the existing admin listener auth check (03 security gate owner). Wire the real predicate; never accept an empty token. - Routing rule, to be enforced at the listener setup call site (exact mount file unconfirmed in the seam spec, see section 7): `GET /debug/*` routes to `OpsRoutes::debug` with `admin_listener=true` on the admin listener only. The public listener must not route `/debug/*` at all (404). ## 5. `src/supervisor/tasks/scheduler.hpp`: elect via coord Anchor: `molo::tasks::Scheduler` private lease members. The scheduler keeps its public surface (`tick_once()`, `is_leader()`); only the mechanism moves onto `molo::coord::LeaderElection`. ```diff --- a/src/supervisor/tasks/scheduler.hpp +++ b/src/supervisor/tasks/scheduler.hpp @@ #include "supervisor/tasks/scheduler.hpp" +#include "operability/coord/election.h" @@ class Scheduler private: - // old internal lease fields (leader key, ttl, renew deadline) + molo::coord::LeaderElection election_; ``` Construct `election_` with the existing queue/kv/clock/observer members: key `"molo/scheduler/leader"` (keep the current `Options::leader_key` default), ttl 10000, renew 3000, node id from the scheduler options or worker id. ## 6. `src/supervisor/tasks/scheduler.cpp`: delegate to the election Anchor: `Scheduler::tick_once` and `Scheduler::is_leader`. ```diff --- a/src/supervisor/tasks/scheduler.cpp +++ b/src/supervisor/tasks/scheduler.cpp @@ bool Scheduler::tick_once() - // old elect_locked lease handling - ... + const bool leader = election_.tick(); + if (!leader) return false; // follower: no dispatch this tick ... // existing dispatch work, unchanged @@ bool Scheduler::is_leader() const - return lease_held_locked(); + return election_.is_leader(); ``` Delete the replaced private lease helpers. Observer metrics keep their existing names; the election additionally emits `election.won/lost` events and `election_renewed_total`. ## 7. `src/clients/request.h`: deadline header helper Anchor: `molo::clients::ClientRequest`, next to `header()`. ```diff --- a/src/clients/request.h +++ b/src/clients/request.h @@ class ClientRequest // existing header/timeouts accessors + // Absolute X-Deadline-Ms value when present and well formed. + std::optional deadline_ms() const; ``` Implement in the existing request source file with `molo::resilience::Deadline::parse_header` on the `X-Deadline-Ms` header: ```cpp std::optional ClientRequest::deadline_ms() const { const auto v = header("X-Deadline-Ms"); // existing accessor if (!v) return std::nullopt; return molo::resilience::Deadline::parse_header(*v); } ``` (Adjust to the real `header()` return type: string, optional, or out-param.) ## 8. `src/clients/client.cpp`: breaker plus deadline propagation Anchor: `molo::clients::Client::send`, before `send_inner`. Order: deadline outermost, then shedder/bulkhead/breaker, per the resilience wiring (exact middleware ordering slot in `src/app` is unconfirmed, see section 9; the client-side half below is exact). ```diff --- a/src/clients/client.cpp +++ b/src/clients/client.cpp @@ #include "clients/client.h" +#include "operability/resilience/breaker.h" +#include "operability/resilience/deadline.h" @@ Task Client::send(ClientRequest req) + // 1. Breaker gate, innermost stage: never retry into an open breaker. + // `breaker_` is a per-host molo::resilience::Breaker owned by Client. + if (!breaker_.allow()) { + co_return ClientResponse::failure(503, "breaker open"); + } + // 2. Deadline propagation: convert the absolute inbound X-Deadline-Ms to + // remaining budget minus the 50 ms floor margin for the outbound hop. + if (const auto abs = req.deadline_ms()) { + molo::resilience::Deadline dl(clock_); + const std::int64_t budget = dl.remaining_ms(*abs); + if (budget == 0) { + breaker_.on_failure(); + co_return ClientResponse::failure(504, "deadline exceeded"); + } + req.set_header("X-Deadline-Ms", + molo::resilience::Deadline::format_header(clock_.wall_ms() + budget)); + reqmutable_timeouts_for(budget); // clamp hop timeouts to budget + } auto resp = co_await send_inner(std::move(req)); + if (resp.ok()) { + breaker_.on_success(); + } else if (resp.status() >= 500 || resp.transport_error()) { + breaker_.on_failure(); + } co_return resp; ``` Notes: - `breaker_` needs a `molo::common::Clock` and `molo::common::Observer` at `Client` construction; add them to the existing client config or constructor the same way timeouts are threaded today. - `ClientResponse::failure`, `resp.ok()`, `resp.status()`, `resp.transport_error()` are placeholders for the real response API: count transport errors and 5xx as breaker failures, everything else as success. Never report a shed 503 from downstream as a local failure twice; the downstream already counted it. - Retry policy (`RetryPolicy::should_retry_status/backoff_for`) must skip retries while `breaker_.allow()` is false and must honor downstream `Retry-After` via `RetryPolicy::parse_retry_after`. - `reqmutable_timeouts_for` stands for clamping the outbound hop timeouts (`Timeouts{connect,read,write,pool_acquire,total}`) to the propagated budget. Keep `Deadline::budget_for(phase)` shares proportional. ## 9. Unconfirmed sites (from the seam spec, kept as gaps) These need the base tree to pin the exact file; the hunks above are written so they apply once the site is known: - Middleware ordering slot for deadline outermost then shedder/bulkhead/breaker in `src/app` (05 request routing). The per-request wiring is: `Deadline` check, `Shedder::should_shed` (503 plus `Retry-After` on shed), `Bulkhead::try_acquire` (shed on `kRejected`), then dispatch; `Breaker` stays innermost at the client (section 8). - Admin listener bind site that mounts `OpsRoutes` (section 4 routing rule). Public listener must 404 `/debug/*`. - Realtime close path (06, connection close 1001) and auth gate owner (03 security): `Lifecycle::run_drain` phase 2 should close realtime connections with 1001; the debug auth predicate must be the 03 gate. ## 10. CLI wiring (rollout, rollback, experiment, debug) `molo::cli::Command(name, brief)` with `flag()/child()/handler()` is the seam (`src/cli/cli/command.hpp`). Register four commands at the existing command registry (same site that registers supervisor/task commands): - `rollout start --strategy rolling|blue-green --batch N --max-unavailable M` driving `molo::deploy::Rolling` / `molo::deploy::BlueGreen` step loops with readiness probes polling `readyz/startupz`. - `rollout abort` and `rollback` driving `Strategy::abort` and `BlueGreen::rollback`. - `experiment start|abort --blast N --duration MS --max-burn X` driving `molo::chaos::Experiment` plus `molo::chaos::Injector` hook setup. - `debug snapshot` printing `molo::debug::Snapshot::capture()`. Handlers return `ExitCode` and write through `Output&`; no new CLI framework code is needed. Exact registry file to be confirmed against the base tree. ## 11. Verification Standalone (this worktree, no base tree): ```sh make -f src/operability.mk OPERABILITY_STANDALONE=1 operability-check ``` This builds `build/liboperability.a` with `g++ -std=c++23 -O2 -Wall -Wextra -Wpedantic -Werror -pthread` and runs `build/operability_tests` (with `MOLO_ENABLE_DEBUG`) plus `build/operability_tests_nodebug` (without it). All eight suites (coord, lifecycle, breaker/bulkhead, shed/deadline, deploy, slo, chaos, debug) run on frozen clocks with no real waiting. Production (base tree present): drop `OPERABILITY_STANDALONE=1` so only `-Isrc` is used and the real `supervisor/common/{clock,kv_store,observer}.hpp` headers satisfy the includes. The ADAPTER shims must not be on the include path.