retoor/moloapp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2488651d810c1934aeb233200151f665571a46ec
moloapp/plans/13-operability.md
T
retoor 3c4334cec9 feat: implement MoloApp plans 01-15 with integration 
Full C++23 framework tree: runtime, observability, security,
data, app, realtime, supervisor, clients, CLI, example plus
enterprise tenancy, audit, operability, governance,
supplychain. Postgres via docker compose for tests.
2026-10-23 18:20:04 +02:00

20 KiB
Raw Blame History

13 - Operability at Scale

Title: Operability at Scale Scope: Rolling and blue-green deploys, startup/shutdown drains, distributed locks, leader election, circuit breakers, bulkheads, load shedding, client deadlines, chaos hooks, SLO and error-budget metrics, debug endpoints.

1. Purpose and philosophy

This file owns what keeps MoloApp alive under real production pressure: safe deploys, clean startup and shutdown, single-owner coordination across instances, fast failure under overload, propagated deadlines, deliberate fault injection, SLO accounting, and debug introspection. It does not own the primitives it builds on: probes live in 09-cli-ops.md, metric and trace primitives live in 02-observability.md, retry and timeout phases live in 08-clients-openapi.md, scheduler leadership and replication live in 07-supervisor-replication.md, connection draining mechanics live in 01-runtime.md.

Philosophy:

  • Fail fast, shed early, degrade loudly. It is nearly always better to fail quickly and apply back pressure downstream as soon as possible (Envoy circuit breaking docs). Overload is handled by policy, never by unbounded queues.
  • Deploys are boring by construction. Health-gated, reversible, and version-tolerant: two versions serve side by side during every rollout, and shared state never assumes a single version.
  • Coordination needs a lease and a fence. Every lock and leadership claim expires on its own and carries a fencing token, so a slow holder can never corrupt state after losing ownership.
  • Deadlines flow downhill. Every inbound request carries a deadline, and every outbound call inherits what remains, converted to a remaining-time budget so clock skew cannot stretch it.
  • Chaos is a first-class test input. Fault hooks are compiled in, disabled by default, authenticated always, and every experiment states its steady-state hypothesis up front.

2. Best practices (from research)

  1. Error budgets make reliability negotiable. An error budget is 1 minus the SLO of the service: a 99.9% SLO service has a 0.1% error budget, so 1,000,000 requests in four weeks allow 1,000 errors (Source: https://sre.google/sre-book/embracing-risk, retrieved 2026-10-23). SLOs are the target, the budget is the spendable allowance for deploys and experiments.
  2. Rolling updates gate on readiness. Kubernetes RollingUpdate strategy exposes maxUnavailable (pods allowed down) and maxSurge (extra pods allowed above desired), both defaulting to 25%, and with maxUnavailable 0 a new pod must pass its readiness probe before an old one is terminated (Source: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/, retrieved 2026-10-23; parameter table at https://kubernetes.io/docs/tasks/run-application/update-deployment-rolling/, retrieved 2026-10-23). MoloApp deploys mirror this: readiness gates traffic, startup gates protect slow boots.
  3. Circuit breaking is distributed and fail-fast. Envoy enforces circuit breaking limits at the network level with fully distributed (not coordinated) breakers: cluster maximum connections, pending requests, active requests, and retries, each with overflow counters (Source: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking, retrieved 2026-10-23). Triangulates with the SRE overload chapter: gracefully handling overload is fundamental to a reliable serving system, via degraded responses and client-side throttling (Source: https://sre.google/sre-book/handling-overload/, retrieved 2026-10-23).
  4. Deadlines are absolute, propagation is relative. A deadline is a point in time past which a client is unwilling to wait; because clocks differ, gRPC converts the deadline to a remaining timeout with elapsed time already deducted before propagating it downstream, and expiry surfaces as DEADLINE_EXCEEDED (Source: https://grpc.io/docs/guides/deadlines/, retrieved 2026-10-23). MoloApp copies this: absolute on the wire, remaining-budget on forward.
  5. Locks and elections ride on lease-bound sessions. etcd concurrency builds Mutex and Election on Sessions: multiple sessions may contend for one election prefix but only one can be leader at a time, and session loss releases everything it held (Source: https://pkg.go.dev/go.etcd.io/etcd/client/v3/concurrency, retrieved 2026-10-23). MoloApp implements the same shape over the shared KV from 04-data.md, which answers open question O7-4 in 07-supervisor-replication.md by adding fencing tokens in v1.
  6. Chaos experiments follow four steps: define steady state as measurable output, hypothesize it holds for control and experimental groups, inject real-world variables (crashes, severed links, failing disks), and try to disprove the hypothesis by comparing the groups (Source: https://principlesofchaos.org/, retrieved 2026-10-23). MoloApp chaos hooks encode exactly this loop: steady-state metric, hypothesis record, fault action, comparison verdict.
  7. Debug endpoints are separate, conventional, and cheap. Go exposes pprof under /debug/pprof/ with heap, CPU profile, trace, goroutine, block, and mutex views (Source: https://pkg.go.dev/net/http/pprof, retrieved 2026-10-23). MoloApp mirrors the convention under /debug/ with its own auth, caps, and opt-in compile flag, never on the public listener by default.

3. Structure

Root: src/ops/ continues in 09-cli-ops.md; this file owns src/resilience/ plus deploy and debug surface. One class or concern per file, deep over flat.

src/deploy/
  strategy.h               molo::deploy::Strategy (rolling, blue_green)
  rolling.h/.cpp           molo::deploy::RollingUpdater (surge, unavailable, gates)
  blue_green.h/.cpp        molo::deploy::BlueGreenSwitcher (standby, cutover, rollback)
  lifecycle.h/.cpp         molo::deploy::Lifecycle (startup gate, drain phases, hooks)
src/coord/
  lease.h/.cpp             molo::coord::Lease (TTL, renewal, expiry callback)
  dist_lock.h/.cpp         molo::coord::DistributedLock (acquire, fencing token)
  election.h/.cpp          molo::coord::LeaderElection (campaign, observe, resign)
  fence.h                  molo::coord::FencingToken (monotonic, compare)
src/resilience/
  breaker.h/.cpp           molo::resilience::CircuitBreaker (closed, open, half_open)
  bulkhead.h/.cpp          molo::resilience::Bulkhead (partition, semaphore, queue)
  shedder.h/.cpp           molo::resilience::LoadShedder (priority, overload policy)
  deadline.h/.cpp          molo::resilience::Deadline (absolute, remaining, propagate)
  slo.h/.cpp               molo::resilience::SloTracker (SLI windows, burn rate, budget)
src/chaos/
  hook.h                   molo::chaos::FaultHook (interface: fault points)
  injector.h/.cpp          molo::chaos::Injector (latency, error, kill, partition)
  experiment.h/.cpp        molo::chaos::Experiment (hypothesis, steady state, verdict)
src/debug/
  endpoint.h/.cpp          molo::debug::DebugEndpoint (route table, auth gate)
  snapshot.h/.cpp          molo::debug::StateSnapshot (loops, pools, breakers, locks)

Namespaces: molo::deploy, molo::coord, molo::resilience, molo::chaos, molo::debug. No sibling namespace may implement its own breaker, lock, or deadline; they consume these.

4. Rules (hard, numbered)

  1. Every deploy keeps the old version serving until the new version passes readiness; traffic never moves to an unreadied instance.
  2. Mixed versions must interoperate: shared state formats (oplog, envelopes, sessions) stay backward compatible across one minor version, old readers ignore unknown fields.
  3. Shutdown always drains: stop accepting, finish in-flight work within the deadline, then exit; dropped work lands in the dead-letter queue with a reason, never vanishes silently (mechanics in 01-runtime.md and 07-supervisor-replication.md, orchestration here).
  4. Every lock acquisition and leadership claim has a TTL and returns a fencing token; writes without a current token are rejected by shared stores.
  5. Only one leader per election name at a time; leadership loss is delivered as an event within one renewal interval, and the loser stops acting immediately.
  6. Circuit breakers fail fast when open: no queuing behind an open breaker, no retries into it, every rejection counted and logged at debug.
  7. Bulkheads partition by downstream dependency and by priority class; one saturated partition never starves another.
  8. Load shedding drops lowest priority first by declared policy; shed requests return 503 with Retry-After, never hang.
  9. Every request has an effective deadline; outbound calls inherit the remaining budget minus a safety margin, and expiry returns 504 (or DEADLINE_EXCEEDED equivalent on realtime paths).
  10. Chaos hooks are disabled by default, require explicit enable plus auth, and every experiment records hypothesis, blast radius, and verdict.
  11. Debug endpoints bind to the admin listener only, require auth, and are compiled out unless MOLO_ENABLE_DEBUG is set.
  12. SLO burn-rate alerts fire on budget consumption rate, not on raw error counts; paging alerts require both fast-burn and slow-burn windows to agree.

5. Development guidelines

  • Build coordination first (lease, lock, election over the 04-data.md shared KV), then drains, then breakers and bulkheads, then deadlines, then deploy strategies, then SLO tracking, then chaos and debug last.
  • State machines everywhere: breaker (closed, open, half_open), election (candidate, leader, follower), lifecycle (booting, serving, draining, stopped). Explicit transitions, logged, with metrics per transition.
  • Time comes from an injectable clock (see 08-clients-openapi.md frozen clock) so TTLs, backoffs, and burn windows are deterministic in tests.
  • Fencing tokens are monotonic per resource (uint64 from the shared store); compare with greater-than, never equality, and persist the highest seen token alongside the data.
  • Prefer middleware composition for resilience: breaker, bulkhead, shedder, and deadline middlewares wrap handlers in a fixed order (deadline outermost, then shedder, bulkhead, breaker innermost near the call).
  • Chaos faults implement one interface with dry-run support: every fault can report what it would do without doing it, and dry-run is the default first step of every experiment.
  • Debug output is snapshots, not live references: serialize state under a short lock, then render; a slow debugger must never stall the loop.

6. Specifications (concrete behaviors, caps, formats)

Deploy:

  • Rolling: max_unavailable default 0, max_surge default 1; per-instance steps are wait_ready (cap 120 s) then terminate_old; revision history kept for last 5 deploys; automatic rollback when new-version error rate exceeds 5x baseline over 2 minutes.
  • Blue-green: standby pool warms to ready, cutover flips router weight in 10% steps every 30 s, instant rollback flips back in one step; both pools share the same shared-KV and durable bus (no split brain).
  • Startup gate: /startupz (09-cli-ops.md) returns 503 until lifecycle reaches serving; slow boot budget 60 s, then process exits nonzero so the orchestrator retries.
  • Drain phases on SIGTERM: phase 1 stop accept plus fail /readyz (immediate); phase 2 finish in-flight requests (cap 30 s); phase 3 close realtime connections with code 1001 (cap 10 s); phase 4 flush tasks and spans (cap 10 s); total cap 60 s, then hard exit with exit code 0 and a loud log of anything dropped.

Coordination:

  • Lease: TTL default 10 s, renewal every 3 s, expiry callback fires within 1 renewal interval; max 1024 live leases per process.
  • DistributedLock: acquire timeout default 5 s, hold TTL 10 s with auto-renew while held; returns FencingToken uint64 strictly increasing per lock name; unlock with stale token is a no-op counted in metrics.
  • LeaderElection: one name, one leader; campaign blocks until elected or cancelled; heartbeat 3 s; followers get leader-change events; resign hands over within 1 s. Scheduler leadership in 07-supervisor-replication.md is the first consumer.
  • Fencing enforcement: shared KV compare-and-set on (key, token) rejects writes with token older than stored; rejection error carries winner token.

Resilience:

  • CircuitBreaker per downstream: opens after 20 failures or 50% failure rate over a 10 s sliding window (min 10 calls); open duration 30 s; half_open allows 3 probe calls, closes on 3 successes, reopens on 1 failure. States exported as gauge plus transition counter.
  • Bulkhead: max 32 partitions per process; each partition has max_concurrent (default 64) plus max_queue (default 128); overflow sheds with 503; partitions named per downstream host or priority class.
  • LoadShedder: priority classes critical, normal, best_effort; when CPU queue depth or memory pressure crosses thresholds, shed best_effort first; shed decision under 1 ms; every shed emits metric with class and reason.
  • Deadline: header X-Deadline-Ms (absolute Unix millis) on inbound; outbound propagation sends remaining minus 50 ms floor-margin; default request budget 30 s, max 300 s; realtime paths carry deadline in the connection context from 06-realtime.md.

SLO and error budget:

  • SLI windows: 30-day rolling for budget accounting, plus 1 h and 6 h burn-rate windows for alerting; availability SLI = good_events / total_events per route template (never raw path).
  • Budget: budget = 1 - SLO; default SLO 99.9% per service; burn_rate = observed_error_ratio / budget_ratio; fast-burn page at 14x over 1 h, slow-burn ticket at 6x over 6 h; both must fire for a page.
  • Metrics: molo_slo_budget_remaining_ratio, molo_slo_burn_rate{window}, molo_slo_bad_events_total{route,reason}; cardinality capped per 02-observability.md rules.

Chaos and debug:

  • Fault hooks: latency_add_ms (cap 10000), error_rate (cap 1.0), conn_kill, partition_downstream, clock_skew_ms (cap 60000, test only); each hook scoped to route template or downstream name, max 64 active hooks.
  • Experiment record: {id, hypothesis, steady_state_metric, threshold, duration_cap 30 min, blast_radius, verdict}; auto-abort when steady state breaches threshold; all actions audit-logged (feeds 12-audit-compliance.md).
  • Debug endpoints (admin listener, auth required): GET /debug/snapshot (loops, pools, breakers, locks, elections), GET /debug/breakers, GET /debug/elections, GET /debug/slo; response cap 1 MB; pprof-style CPU and heap views where the platform allows, same path convention /debug/pprof/.

7. Execution steps (in build order)

  1. Coordination core: Lease, DistributedLock with fencing tokens, LeaderElection over shared KV; migrate the 07-supervisor-replication.md scheduler lease onto it.
  2. Lifecycle orchestration: startup gate, four-phase drain, signal wiring with 01-runtime.md, drop accounting into dead-letter queue.
  3. Breaker plus bulkhead: per-downstream breaker state machines, partitions, middleware wrappers, overflow metrics.
  4. Shedder plus deadlines: priority shedder, X-Deadline-Ms parse and propagate through 08-clients-openapi.md client and 06-realtime.md contexts.
  5. Deploy strategies: rolling updater with readiness gates and auto-rollback, blue-green switcher with weighted cutover, revision history.
  6. SLO tracking: SLI windows, budget math, burn-rate alerts, Prometheus series for budget and burn.
  7. Chaos hooks: fault interface, injector actions, experiment runner with auto-abort and audit records.
  8. Debug endpoints: admin-listener routes, snapshot serializers, auth gate, compile-flag removal check.

8. Validation (how to verify each step)

  • Step 1: two processes campaign for one election, kill the leader, follower takes over within 5 s; stale-token write rejected with winner token; lease expiry releases lock within 4 s of holder death (frozen-clock plus real-clock runs).
  • Step 2: SIGTERM during 1000 in-flight requests drains with zero silent drops; /readyz flips to 503 within 100 ms; realtime peers receive close 1001; hard cap enforced at 60 s with loud log.
  • Step 3: failing downstream opens breaker within window, calls fail fast under 1 ms with rejection counter; bulkhead saturation on one host leaves other hosts at full speed; half-open probes close the breaker after recovery.
  • Step 4: low-priority flood sheds best_effort first while critical stays green; propagated deadline expires downstream first and returns 504 upstream with matching request IDs; margin floor never sends negative budgets.
  • Step 5: rolling deploy with max_unavailable 0 shows zero failed requests across cutover; injected 10% new-version errors trigger auto-rollback within 3 minutes; blue-green cutover and instant rollback both verified through the real router.
  • Step 6: recorded error rates reproduce budget math within rounding; 14x fast-burn plus 6x slow-burn fixtures page, single-window fixtures do not; series cardinality stays under caps at 10k routes.
  • Step 7: experiment with latency fault shows steady-state breach, auto-abort, and verdict record; dry-run changes nothing; unauthenticated chaos call rejected; disabled-by-default build contains no reachable fault path.
  • Step 8: debug routes 404 on the public listener and 200 on admin with auth; snapshot under 1 MB renders in under 100 ms; MOLO_ENABLE_DEBUG=0 build has zero debug symbols in the binary.

9. References (sibling files this depends on or feeds)

  • Depends on 01-runtime.md: signal handling, connection drain mechanics, H2 GOAWAY, loop introspection for snapshots.
  • Depends on 02-observability.md: metric, event, and span primitives; label cardinality rules; zero-cost gating for all resilience hooks.
  • Depends on 04-data.md: shared KV for leases, locks, elections, and fencing compare-and-set.
  • Depends on 05-request-routing.md: middleware ordering slot for deadline, shedder, bulkhead, breaker layers.
  • Depends on 06-realtime.md: connection context for deadlines, close codes for drain, broadcast behavior during deploys.
  • Depends on 07-supervisor-replication.md: drain deadlines, dead-letter queue, scheduler leadership migration, multi-instance topology.
  • Depends on 08-clients-openapi.md: timeout phases, retry policy interplay with breakers, frozen clock, deadline propagation in the HTTP client.
  • Depends on 09-cli-ops.md: /healthz, /readyz, /startupz, /metrics endpoints consumed as deploy gates; admin listener shape.
  • Feeds 09-cli-ops.md: deploy and chaos CLI surface (rollout status, rollback, experiment run/abort), debug endpoint hosting.
  • Feeds 10-workflow-example.md: living example must demo rolling deploy, breaker trip, deadline expiry, and one chaos experiment.
  • Feeds 12-audit-compliance.md: chaos action and debug access audit records (no duplication of audit formats).
  • Boundary with 03-security.md: auth for debug and chaos endpoints is owned there; this file only names the gate.

10. Open questions

  1. Should blue-green weight steps be time-based (10% per 30 s) or metric-gated per step (advance only when error rate holds)? Proposal: metric-gated with time cap.
  2. Fencing token scope: per lock name or global per store? Proposal: per lock name for v1, global optional later.
  3. Default SLO: single 99.9% for the whole service, or per-route SLOs from day one? Proposal: service default plus opt-in per-route overrides.
  4. Chaos in production: allow guarded production experiments in v1, or restrict to staging by config? Proposal: staging-only until the audit trail in 12-audit-compliance.md lands.
  5. Debug CPU and heap profiling: in-house sampler or platform hooks only? Proposal: platform hooks where available, documented gap elsewhere.

11. Sources (exact URLs with retrieval dates)

Reference in New Issue View Git Blame Copy Permalink