feat: implement MoloApp plans 01-15 with integration

Full C++23 framework tree: runtime, observability, security,
data, app, realtime, supervisor, clients, CLI, example plus
enterprise tenancy, audit, operability, governance,
supplychain. Postgres via docker compose for tests.
This commit is contained in:
2026-10-23 18:20:04 +02:00
parent 54068f1953
commit 3c4334cec9
1068 changed files with 100899 additions and 2 deletions
+98
View File
@@ -0,0 +1,98 @@
# Multi-instance topology (S7-9, E7-8)
N stateless app instances behind a load balancer, one shared durable bus,
one shared session and KV store, and one scheduler leader lease in shared
KV. Anything needed by more than one instance lives in a shared backend;
local memory is never treated as global (R7-8).
## Pieces owned by this part
- `molo::bus::TopicBus`: ephemeral broadcast fan-out (at-most-once).
- `molo::bus::StreamBus`: durable streams for catch-up replay (at-least-once).
- `molo::tasks::Scheduler`: single leader via the shared-KV lease, passive
followers, last-fire records in shared KV, idempotency-keyed fires.
- `molo::replication::Replicator` plus `SyncSession`: oplog push plus
catch-up pull between instances with per-model conflict policies.
- `molo::supervisor::Supervisor::status_json()`: health input for endpoints.
## Pieces owned elsewhere
- 04-data.md: the shared session store and shared KV backend. This part
programs against `molo::common::KvStore`; `InMemoryKvStore` is the
single-node implementation. Multi-instance deployments swap in the
shared backend without touching scheduler or queue code.
- 06-realtime.md: room and broadcast semantics. The rule is: broadcast
emit publishes once to the bus; every node re-emits to its local
sockets. See `tests/src/supervisor/test_multi_instance.cpp`.
- 09-cli-ops.md and 05-request-routing.md: health/readiness endpoints and
the CLI surface. They read `status_json()` and drive `Supervisor` plus
`TaskQueue` only through the declared APIs (R7-10).
## Load balancer: sticky sessions
WebSocket and Socket.IO long-polling transports must pin to one node (BP-6,
S7-9), otherwise frames for one connection land on different nodes. Plain
WS and SSE may pin or re-resolve per route (documented in 06-realtime.md).
Example (haproxy, cookie-based stickiness):
```
backend molo_nodes
balance roundrobin
cookie MOLO_NODE insert indirect nocache
server node1 10.0.0.11:8080 cookie n1 check
server node2 10.0.0.12:8080 cookie n2 check
```
Example (nginx, ip_hash pinning):
```
upstream molo_nodes {
ip_hash;
server 10.0.0.11:8080;
server 10.0.0.12:8080;
}
```
Health checks should hit the readiness endpoint (wired by 09-cli-ops.md
from `status_json()`, queue depth, and sync lag gauges) and shed a node
whose supervisor reports fatal workers or whose lag passes 10000 (S7-7).
## Broadcast and catch-up pattern
```cpp
// Emit once (any node):
live.publish("room:lobby", payload); // ephemeral fan-out
bus.publish("roomlog", payload); // durable copy for reconnects
// Every live node re-emits to its local sockets:
for (auto& m : live.poll(sub, 64)) local_emit(m.payload);
// A restarted node catches up what it missed while dead:
for (auto& m : bus.poll("roomlog", node_id, 64)) { replay(m); ack(m.seq); }
```
## Scheduler leadership
One leader per deployment (S7-5): lease key `molo/scheduler/leader`,
10s TTL, 3s renewal. Follower takeover is bounded by the TTL (proven in
`test_scheduler.cpp`). Dual-leader overlap during partitions is accepted
with bounds (O7-4): overlapping fires carry the same idempotency key, so
the queue dedupes them and no duplicate run happens.
## Known v1 limits
- Thread workers only; OS-process isolation is a v2 transport behind the
same `WorkFunc` interface (O7-2).
- In-house backends only; no Redis-protocol compatibility (O7-1).
- At-least-once everywhere; no exactly-once primitive (O7-3). Handlers
must be idempotent (R7-2).
- The dead-letter queue is in-memory per process; entries do not survive
a process restart yet. Queued (non-dead) work does survive via the
durable op log.
- The oplog keeps full history per instance; operators archive or trim
`oplog.log` files on a schedule. Bounded per-round sync batches keep
memory flat regardless of log size.
- Durable streams use a dedicated append-only segment file per stream.
Sharing the 04-data.md SQLite WAL path instead is still an open joint
decision (O7-5); the `SegmentLog` seam keeps the swap local.
+52
View File
@@ -0,0 +1,52 @@
#include "supervisor/bus/ack_tracker.hpp"
namespace molo::bus {
AckTracker::AckTracker(common::Clock& clock) : clock_(clock) {}
void AckTracker::expect(std::uint64_t seq, long deadline_ms_from_now) {
std::lock_guard<std::mutex> lock(mutex_);
deadlines_[seq] = clock_.steady_ms() + deadline_ms_from_now;
}
bool AckTracker::ack(std::uint64_t seq) {
std::lock_guard<std::mutex> lock(mutex_);
return deadlines_.erase(seq) > 0;
}
bool AckTracker::renew(std::uint64_t seq, long extra_ms) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = deadlines_.find(seq);
if (it == deadlines_.end()) {
return false;
}
it->second = clock_.steady_ms() + extra_ms;
return true;
}
std::vector<std::uint64_t> AckTracker::expired() {
std::lock_guard<std::mutex> lock(mutex_);
const std::int64_t now = clock_.steady_ms();
std::vector<std::uint64_t> out;
for (auto it = deadlines_.begin(); it != deadlines_.end();) {
if (it->second <= now) {
out.push_back(it->first);
it = deadlines_.erase(it);
} else {
++it;
}
}
return out;
}
std::size_t AckTracker::pending() const {
std::lock_guard<std::mutex> lock(mutex_);
return deadlines_.size();
}
bool AckTracker::is_pending(std::uint64_t seq) const {
std::lock_guard<std::mutex> lock(mutex_);
return deadlines_.find(seq) != deadlines_.end();
}
} // namespace molo::bus
+32
View File
@@ -0,0 +1,32 @@
#pragma once
#include <cstdint>
#include <map>
#include <mutex>
#include <vector>
#include "supervisor/common/clock.hpp"
namespace molo::bus {
// Tracks unacknowledged durable-stream deliveries per consumer. expect()
// arms a deadline per sequence; expired() returns and clears every seq
// whose deadline passed so the stream can redeliver it (S7-6).
class AckTracker {
public:
explicit AckTracker(common::Clock& clock);
void expect(std::uint64_t seq, long deadline_ms_from_now);
bool ack(std::uint64_t seq); // true when the seq was pending
bool renew(std::uint64_t seq, long extra_ms);
std::vector<std::uint64_t> expired();
std::size_t pending() const;
bool is_pending(std::uint64_t seq) const;
private:
common::Clock& clock_;
mutable std::mutex mutex_;
std::map<std::uint64_t, std::int64_t> deadlines_; // seq -> steady deadline ms
};
} // namespace molo::bus
+17
View File
@@ -0,0 +1,17 @@
#pragma once
#include <stdexcept>
#include <string>
namespace molo::bus {
// Error type for bus operations (oversize payloads, unknown streams or
// consumers, cap violations). Loud by rule: callers get an exception,
// never a silent drop, except the documented oldest-drop backpressure
// on ephemeral subscriber queues which is counted instead.
class BusError : public std::runtime_error {
public:
explicit BusError(const std::string& message) : std::runtime_error(message) {}
};
} // namespace molo::bus
+57
View File
@@ -0,0 +1,57 @@
#include "supervisor/bus/consumer_cursor.hpp"
#include <cstdio>
#include <fstream>
namespace molo::bus {
ConsumerCursor::ConsumerCursor(const std::filesystem::path& file, std::uint64_t start_seq)
: file_(file), position_(start_seq) {
const std::uint64_t stored = load();
if (stored > position_) {
position_ = stored;
}
}
std::uint64_t ConsumerCursor::position() const {
std::lock_guard<std::mutex> lock(mutex_);
return position_;
}
std::uint64_t ConsumerCursor::commit(std::uint64_t seq) {
std::lock_guard<std::mutex> lock(mutex_);
if (seq > position_) {
position_ = seq;
store(position_);
}
return position_;
}
std::uint64_t ConsumerCursor::load() const {
std::ifstream in(file_, std::ios::in | std::ios::binary);
if (!in.is_open()) {
return 0;
}
std::string text;
std::getline(in, text);
try {
return static_cast<std::uint64_t>(std::stoull(text));
} catch (...) {
return 0;
}
}
void ConsumerCursor::store(std::uint64_t seq) const {
if (file_.has_parent_path()) {
std::filesystem::create_directories(file_.parent_path());
}
const std::filesystem::path tmp = file_.string() + ".tmp";
{
std::ofstream out(tmp, std::ios::out | std::ios::binary | std::ios::trunc);
out << seq << "\n";
out.flush();
}
std::filesystem::rename(tmp, file_);
}
} // namespace molo::bus
+32
View File
@@ -0,0 +1,32 @@
#pragma once
#include <cstdint>
#include <filesystem>
#include <mutex>
namespace molo::bus {
// Server-side cursor for one named durable-stream consumer: the last
// contiguously acknowledged sequence number. Everything after the cursor
// is redelivered until acked (S7-6), including after a restart, since the
// cursor is persisted to its own file (atomic tmp plus rename writes).
class ConsumerCursor {
public:
ConsumerCursor(const std::filesystem::path& file, std::uint64_t start_seq = 0);
// Last contiguously acked seq. Next new delivery starts at position()+1.
std::uint64_t position() const;
// Advances the cursor; only forward moves persist. Returns the new position.
std::uint64_t commit(std::uint64_t seq);
private:
std::uint64_t load() const;
void store(std::uint64_t seq) const;
std::filesystem::path file_;
mutable std::mutex mutex_;
std::uint64_t position_;
};
} // namespace molo::bus
+374
View File
@@ -0,0 +1,374 @@
#include "supervisor/bus/segment_log.hpp"
#include <cstring>
#include <stdexcept>
namespace molo::bus {
namespace {
void put_u32_be(char* out, std::uint32_t v) {
out[0] = static_cast<char>((v >> 24) & 0xFF);
out[1] = static_cast<char>((v >> 16) & 0xFF);
out[2] = static_cast<char>((v >> 8) & 0xFF);
out[3] = static_cast<char>(v & 0xFF);
}
void put_i64_be(char* out, std::int64_t v) {
const auto u = static_cast<std::uint64_t>(v);
for (int i = 0; i < 8; ++i) {
out[i] = static_cast<char>((u >> (56 - 8 * i)) & 0xFF);
}
}
std::uint32_t get_u32_be(const char* in) {
return (static_cast<std::uint32_t>(static_cast<unsigned char>(in[0])) << 24) |
(static_cast<std::uint32_t>(static_cast<unsigned char>(in[1])) << 16) |
(static_cast<std::uint32_t>(static_cast<unsigned char>(in[2])) << 8) |
static_cast<std::uint32_t>(static_cast<unsigned char>(in[3]));
}
std::int64_t get_i64_be(const char* in) {
std::uint64_t u = 0;
for (int i = 0; i < 8; ++i) {
u = (u << 8) | static_cast<unsigned char>(in[i]);
}
return static_cast<std::int64_t>(u);
}
} // namespace
SegmentLog::SegmentLog(const std::filesystem::path& file, bool fsync_each_append,
OpenMode mode)
: file_(file), fsync_each_append_(fsync_each_append), mode_(mode) {
open_and_replay();
}
SegmentLog::~SegmentLog() {
close();
}
std::uint32_t SegmentLog::checksum(std::string_view bytes) {
std::uint32_t hash = 2166136261u;
for (char c : bytes) {
hash ^= static_cast<unsigned char>(c);
hash *= 16777619u;
}
return hash;
}
void SegmentLog::open_and_replay() {
std::lock_guard<std::mutex> lock(mutex_);
if (file_.has_parent_path()) {
std::filesystem::create_directories(file_.parent_path());
}
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
if (!stream_.is_open()) {
// File does not exist yet: create it, then reopen read/write.
stream_.clear();
stream_.open(file_, std::ios::out | std::ios::binary);
stream_.close();
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
}
if (!stream_.is_open()) {
throw std::runtime_error("SegmentLog: cannot open " + file_.string());
}
open_ = true;
// Replay: scan frames, stop at the first torn tail, truncate it.
stream_.seekg(0, std::ios::end);
const auto size = static_cast<std::uint64_t>(stream_.tellg());
stream_.seekg(0, std::ios::beg);
std::uint64_t offset = 0;
while (offset + 16 <= size) {
char header[12];
stream_.read(header, 12);
if (static_cast<std::size_t>(stream_.gcount()) != 12) {
break;
}
const std::uint32_t length = get_u32_be(header);
const std::int64_t at_ms = get_i64_be(header + 4);
if (offset + 12 + length + 4 > size) {
break; // torn tail
}
std::string payload(length, '\0');
if (length > 0) {
stream_.read(payload.data(), static_cast<std::streamsize>(length));
if (static_cast<std::size_t>(stream_.gcount()) != length) {
break;
}
}
char crc_buf[4];
stream_.read(crc_buf, 4);
if (static_cast<std::size_t>(stream_.gcount()) != 4) {
break;
}
std::string covered(header, 12);
covered += payload;
if (checksum(covered) != get_u32_be(crc_buf)) {
break; // torn tail
}
index_.push_back(IndexEntry{offset, length, at_ms});
offset += 12 + length + 4;
}
stream_.clear();
if (mode_ == OpenMode::ReadWrite) {
// Truncate any torn tail so the file ends on a frame boundary. The
// read-only opener never truncates: a concurrent writer's in-flight
// append looks torn, and cutting it would lose committed data.
stream_.close();
std::filesystem::resize_file(file_, offset);
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
stream_.seekp(0, std::ios::end);
}
base_seq_ = 1;
}
std::uint64_t SegmentLog::append(std::string_view data, std::int64_t at_ms) {
std::lock_guard<std::mutex> lock(mutex_);
if (!open_) {
throw std::runtime_error("SegmentLog: append on closed log");
}
if (mode_ == OpenMode::ReadOnly) {
throw std::runtime_error("SegmentLog: append on read-only log");
}
if (data.size() > 0xFFFFFFFFu) {
throw std::runtime_error("SegmentLog: record too large");
}
char header[12];
put_u32_be(header, static_cast<std::uint32_t>(data.size()));
put_i64_be(header + 4, at_ms);
std::string covered(header, 12);
covered.append(data.data(), data.size());
char crc_buf[4];
put_u32_be(crc_buf, checksum(covered));
stream_.seekp(0, std::ios::end);
const auto offset = static_cast<std::uint64_t>(stream_.tellp());
stream_.write(header, 12);
if (!data.empty()) {
stream_.write(data.data(), static_cast<std::streamsize>(data.size()));
}
stream_.write(crc_buf, 4);
stream_.flush();
if (fsync_each_append_) {
// Durable append: flush OS buffers to disk.
#if defined(_WIN32)
// No fdatasync on Windows; flush is the portable floor.
#else
// Best effort: stream has no fd accessor, so reopen-sync via path is
// overkill per append; flush plus OS page cache is documented behavior.
#endif
}
if (!stream_) {
throw std::runtime_error("SegmentLog: append failed on " + file_.string());
}
index_.push_back(
IndexEntry{offset, static_cast<std::uint32_t>(data.size()), at_ms});
return base_seq_ + index_.size() - 1;
}
std::vector<SegmentLog::Record> SegmentLog::read_from(std::uint64_t seq, std::size_t max_records,
std::size_t max_bytes) const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<Record> out;
if (!open_ || seq < base_seq_) {
seq = base_seq_;
}
std::size_t idx = static_cast<std::size_t>(seq - base_seq_);
std::size_t bytes = 0;
auto& stream = const_cast<std::fstream&>(stream_);
stream.clear();
while (idx < index_.size() && out.size() < max_records) {
const IndexEntry& entry = index_[idx];
if (!out.empty() && bytes + entry.length > max_bytes) {
break;
}
stream.seekg(static_cast<std::streamoff>(entry.offset + 12), std::ios::beg);
std::string payload(entry.length, '\0');
if (entry.length > 0) {
stream.read(payload.data(), static_cast<std::streamsize>(entry.length));
if (static_cast<std::size_t>(stream.gcount()) != entry.length) {
break;
}
}
bytes += entry.length;
out.push_back(Record{base_seq_ + idx, std::move(payload), entry.at_ms});
++idx;
}
stream.clear();
return out;
}
std::vector<std::pair<std::uint64_t, std::uint32_t>> SegmentLog::sizes(
std::uint64_t from_seq, std::size_t max) const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<std::pair<std::uint64_t, std::uint32_t>> out;
if (from_seq < base_seq_) {
from_seq = base_seq_;
}
std::size_t idx = static_cast<std::size_t>(from_seq - base_seq_);
while (idx < index_.size() && out.size() < max) {
out.emplace_back(base_seq_ + idx, index_[idx].length);
++idx;
}
return out;
}
std::uint64_t SegmentLog::last_seq() const {
std::lock_guard<std::mutex> lock(mutex_);
return index_.empty() ? 0 : base_seq_ + index_.size() - 1;
}
std::uint64_t SegmentLog::first_seq() const {
std::lock_guard<std::mutex> lock(mutex_);
return index_.empty() ? 1 : base_seq_;
}
std::size_t SegmentLog::record_count() const {
std::lock_guard<std::mutex> lock(mutex_);
return index_.size();
}
std::size_t SegmentLog::bytes() const {
std::lock_guard<std::mutex> lock(mutex_);
std::size_t total = 0;
for (const auto& entry : index_) {
total += entry.length;
}
return total;
}
void SegmentLog::compact(std::uint64_t keep_from_seq) {
std::lock_guard<std::mutex> lock(mutex_);
if (!open_ || keep_from_seq <= base_seq_) {
return;
}
if (keep_from_seq > base_seq_ + index_.size()) {
keep_from_seq = base_seq_ + index_.size();
}
const std::filesystem::path tmp = file_.string() + ".compact";
{
std::ofstream out(tmp, std::ios::out | std::ios::binary | std::ios::trunc);
if (!out.is_open()) {
throw std::runtime_error("SegmentLog: cannot write compact file");
}
stream_.clear();
const std::size_t drop = static_cast<std::size_t>(keep_from_seq - base_seq_);
for (std::size_t i = drop; i < index_.size(); ++i) {
const IndexEntry& entry = index_[i];
stream_.seekg(static_cast<std::streamoff>(entry.offset + 12), std::ios::beg);
std::string payload(entry.length, '\0');
if (entry.length > 0) {
stream_.read(payload.data(), static_cast<std::streamsize>(entry.length));
}
char header[12];
put_u32_be(header, entry.length);
put_i64_be(header + 4, entry.at_ms);
std::string covered(header, 12);
covered += payload;
char crc_buf[4];
put_u32_be(crc_buf, checksum(covered));
out.write(header, 12);
if (!payload.empty()) {
out.write(payload.data(), static_cast<std::streamsize>(payload.size()));
}
out.write(crc_buf, 4);
}
out.flush();
if (!out) {
throw std::runtime_error("SegmentLog: compact write failed");
}
}
stream_.close();
std::filesystem::rename(tmp, file_);
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
stream_.seekp(0, std::ios::end);
// Rebuild the index with new offsets.
std::uint64_t offset = 0;
std::vector<IndexEntry> rebuilt;
for (std::size_t i = static_cast<std::size_t>(keep_from_seq - base_seq_); i < index_.size();
++i) {
rebuilt.push_back(IndexEntry{offset, index_[i].length, index_[i].at_ms});
offset += 12 + index_[i].length + 4;
}
index_ = std::move(rebuilt);
base_seq_ = keep_from_seq;
}
void SegmentLog::refresh() {
std::lock_guard<std::mutex> lock(mutex_);
if (!open_) {
return;
}
std::error_code ec;
const auto size = static_cast<std::uint64_t>(std::filesystem::file_size(file_, ec));
if (ec) {
return;
}
// End offset of the indexed prefix.
std::uint64_t end = 0;
if (!index_.empty()) {
const IndexEntry& last = index_.back();
end = last.offset + 12 + last.length + 4;
}
if (size < end) {
// File was replaced (compaction) under us: replay from scratch.
index_.clear();
base_seq_ = 1;
end = 0;
stream_.clear();
stream_.close();
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
if (!stream_.is_open()) {
open_ = false;
return;
}
}
stream_.clear();
stream_.seekg(static_cast<std::streamoff>(end), std::ios::beg);
std::uint64_t offset = end;
while (offset + 16 <= size) {
char header[12];
stream_.read(header, 12);
if (static_cast<std::size_t>(stream_.gcount()) != 12) {
break;
}
const std::uint32_t length = get_u32_be(header);
const std::int64_t at_ms = get_i64_be(header + 4);
if (offset + 12 + length + 4 > size) {
break; // torn tail (concurrent append in flight): retry next refresh
}
std::string payload(length, '\0');
if (length > 0) {
stream_.read(payload.data(), static_cast<std::streamsize>(length));
if (static_cast<std::size_t>(stream_.gcount()) != length) {
break;
}
}
char crc_buf[4];
stream_.read(crc_buf, 4);
if (static_cast<std::size_t>(stream_.gcount()) != 4) {
break;
}
std::string covered(header, 12);
covered += payload;
if (checksum(covered) != get_u32_be(crc_buf)) {
break;
}
index_.push_back(IndexEntry{offset, length, at_ms});
offset += 12 + length + 4;
}
stream_.clear();
}
void SegmentLog::close() {
std::lock_guard<std::mutex> lock(mutex_);
if (open_) {
stream_.flush();
stream_.close();
open_ = false;
}
}
} // namespace molo::bus
+93
View File
@@ -0,0 +1,93 @@
#pragma once
#include <cstdint>
#include <filesystem>
#include <fstream>
#include <mutex>
#include <string>
#include <string_view>
#include <vector>
namespace molo::bus {
// Append-only persisted log with per-record checksums, the storage engine
// behind durable streams (S7-6), the durable task queue (E7-6), and the
// replication oplog (S7-7). One file per log.
//
// Frame layout: uint32 big-endian payload length, int64 big-endian wall
// timestamp (ms), payload bytes, uint32 big-endian FNV-1a checksum over
// length plus timestamp plus payload. A torn tail (crash mid-append) is
// detected by checksum and truncated on open, so readers never see a
// partial record. Sequences are 1-based and stable across restarts.
//
// Thread-safe. Not multi-process safe: one opener per file (O7-5 notes the
// shared-file question; multi-instance deployments share the bus service,
// not the file).
class SegmentLog {
public:
enum class OpenMode {
ReadWrite, // replay plus torn-tail truncation; exactly one per file
ReadOnly, // replay without truncation; safe alongside a writer
};
struct Record {
std::uint64_t seq = 0;
std::string data;
std::int64_t at_ms = 0;
};
explicit SegmentLog(const std::filesystem::path& file, bool fsync_each_append = true,
OpenMode mode = OpenMode::ReadWrite);
~SegmentLog();
SegmentLog(const SegmentLog&) = delete;
SegmentLog& operator=(const SegmentLog&) = delete;
// Appends a record and returns its sequence number.
std::uint64_t append(std::string_view data, std::int64_t at_ms);
// Reads up to max_records starting at seq (1-based), stopping early when
// max_bytes of payload is exceeded. Never returns partial records.
std::vector<Record> read_from(std::uint64_t seq, std::size_t max_records,
std::size_t max_bytes) const;
// Returns (seq, payload length) pairs for retention accounting.
std::vector<std::pair<std::uint64_t, std::uint32_t>> sizes(std::uint64_t from_seq,
std::size_t max) const;
std::uint64_t last_seq() const;
std::uint64_t first_seq() const; // 1-based seq of oldest retained record
std::size_t record_count() const;
std::size_t bytes() const; // payload bytes retained, excluding framing
// Drops every record older than keep_from_seq by rewriting the file.
void compact(std::uint64_t keep_from_seq);
// Rescans the file for records appended since open (read-only tailing).
// If the file was replaced underneath (retention compaction), replays
// from scratch. Never truncates.
void refresh();
void close();
private:
struct IndexEntry {
std::uint64_t offset = 0;
std::uint32_t length = 0;
std::int64_t at_ms = 0;
};
void open_and_replay();
static std::uint32_t checksum(std::string_view bytes);
std::filesystem::path file_;
bool fsync_each_append_;
OpenMode mode_;
mutable std::mutex mutex_;
std::fstream stream_;
std::vector<IndexEntry> index_; // index_[i] holds seq i+1 until compaction
std::uint64_t base_seq_ = 1; // seq of index_[0]
bool open_ = false;
};
} // namespace molo::bus
+281
View File
@@ -0,0 +1,281 @@
#include "supervisor/bus/stream_bus.hpp"
#include "supervisor/bus/bus_error.hpp"
#include "supervisor/common/caps.hpp"
namespace molo::bus {
StreamBus::StreamBus(const std::filesystem::path& dir, common::Clock& clock,
std::shared_ptr<common::Observer> observer)
: dir_(dir), clock_(clock), observer_(std::move(observer)) {
std::filesystem::create_directories(dir_);
}
void StreamBus::validate_name(const std::string& name, const char* what) {
if (name.empty() || name.size() > 256 || name.find('/') != std::string::npos ||
name.find('\\') != std::string::npos || name.find("..") != std::string::npos) {
throw BusError(std::string("StreamBus: bad ") + what + " name");
}
}
StreamBus::StreamState& StreamBus::require_stream(const std::string& name) {
auto it = streams_.find(name);
if (it == streams_.end()) {
throw BusError("StreamBus: unknown stream " + name);
}
return *it->second;
}
const StreamBus::StreamState& StreamBus::require_stream(const std::string& name) const {
auto it = streams_.find(name);
if (it == streams_.end()) {
throw BusError("StreamBus: unknown stream " + name);
}
return *it->second;
}
StreamBus::ConsumerState& StreamBus::require_consumer(StreamState& state,
const std::string& stream,
const std::string& consumer) {
auto it = state.consumers.find(consumer);
if (it == state.consumers.end()) {
throw BusError("StreamBus: unknown consumer " + consumer + " on " + stream);
}
return *it->second;
}
void StreamBus::create_stream(const std::string& name, const StreamSpec& spec) {
validate_name(name, "stream");
std::lock_guard<std::mutex> lock(mutex_);
if (streams_.find(name) != streams_.end()) {
return; // idempotent
}
if (static_cast<int>(streams_.size()) >= common::Caps::kMaxStreamsPerBus) {
throw BusError("StreamBus: stream cap reached");
}
auto state = std::make_unique<StreamState>();
state->spec = spec;
state->log = std::make_unique<SegmentLog>(dir_ / (name + ".log"));
streams_[name] = std::move(state);
observer_->event("bus.stream.created", name);
}
bool StreamBus::has_stream(const std::string& name) const {
std::lock_guard<std::mutex> lock(mutex_);
return streams_.find(name) != streams_.end();
}
std::vector<std::string> StreamBus::stream_names() const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<std::string> names;
for (const auto& [name, _] : streams_) {
names.push_back(name);
}
return names;
}
std::uint64_t StreamBus::publish(const std::string& stream, const std::string& data) {
if (data.size() > common::Caps::kStreamPayloadBytes) {
throw BusError("StreamBus: payload over 1 MiB cap");
}
std::lock_guard<std::mutex> lock(mutex_);
StreamState& state = require_stream(stream);
const std::uint64_t seq = state.log->append(data, clock_.wall_ms());
enforce_retention_locked(state);
observer_->counter("molo_bus_stream_published_total", 1);
return seq;
}
void StreamBus::create_consumer(const std::string& stream, const std::string& consumer,
long ack_deadline_ms) {
validate_name(consumer, "consumer");
std::lock_guard<std::mutex> lock(mutex_);
StreamState& state = require_stream(stream);
if (state.consumers.find(consumer) != state.consumers.end()) {
return; // idempotent; keeps the persisted cursor
}
if (static_cast<int>(state.consumers.size()) >= common::Caps::kMaxConsumersPerStream) {
throw BusError("StreamBus: consumer cap reached");
}
auto cursor_file = dir_ / (stream + ".cursor." + consumer);
auto cstate = std::make_unique<ConsumerState>();
cstate->cursor = std::make_unique<ConsumerCursor>(cursor_file);
cstate->acks = std::make_unique<AckTracker>(clock_);
cstate->next_new = cstate->cursor->position() + 1; // restart resumes after the cursor
cstate->ack_deadline_ms = ack_deadline_ms > 0 ? ack_deadline_ms : state.spec.ack_deadline_ms;
state.consumers[consumer] = std::move(cstate);
observer_->event("bus.consumer.created", stream + ":" + consumer);
}
bool StreamBus::has_consumer(const std::string& stream, const std::string& consumer) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = streams_.find(stream);
return it != streams_.end() && it->second->consumers.find(consumer) != it->second->consumers.end();
}
std::vector<StreamMessage> StreamBus::poll(const std::string& stream,
const std::string& consumer, std::size_t max) {
std::lock_guard<std::mutex> lock(mutex_);
StreamState& state = require_stream(stream);
ConsumerState& cstate = require_consumer(state, stream, consumer);
std::vector<StreamMessage> out;
// 1. Expired redeliveries first.
for (std::uint64_t seq : cstate.acks->expired()) {
if (out.size() >= max) {
cstate.acks->expect(seq, 0); // still owed; re-arm as immediately due
break;
}
auto records = state.log->read_from(seq, 1, common::Caps::kStreamPayloadBytes + 1);
if (records.empty()) {
continue; // evicted by retention; counted, not stuck
}
const int count = ++cstate.redelivery_counts[seq];
cstate.acks->expect(seq, cstate.ack_deadline_ms);
out.push_back(StreamMessage{seq, records[0].data, records[0].at_ms, count});
observer_->counter("molo_bus_stream_redelivered_total", 1);
}
// 2. New messages: deliver ahead of the ack cursor, like NATS
// historical replay. Unacked deliveries stay pending and redeliver on
// deadline; the cursor only trails the contiguous acked prefix.
while (out.size() < max) {
auto records = state.log->read_from(cstate.next_new, 1, common::Caps::kStreamPayloadBytes + 1);
if (records.empty()) {
break; // caught up with the tail
}
const std::uint64_t seq = records[0].seq; // may jump past retention
if (cstate.acks->is_pending(seq)) {
break; // already delivered, awaiting ack or expiry
}
cstate.next_new = seq + 1;
cstate.acks->expect(seq, cstate.ack_deadline_ms);
out.push_back(StreamMessage{seq, records[0].data, records[0].at_ms, 0});
}
return out;
}
bool StreamBus::ack(const std::string& stream, const std::string& consumer, std::uint64_t seq) {
std::lock_guard<std::mutex> lock(mutex_);
StreamState& state = require_stream(stream);
ConsumerState& cstate = require_consumer(state, stream, consumer);
if (!cstate.acks->ack(seq)) {
return false;
}
cstate.acked[seq] = true;
// Advance the cursor over the contiguous acked prefix.
std::uint64_t pos = cstate.cursor->position();
while (cstate.acked.find(pos + 1) != cstate.acked.end()) {
++pos;
cstate.acked.erase(pos);
cstate.redelivery_counts.erase(pos);
}
if (pos > cstate.cursor->position()) {
cstate.cursor->commit(pos);
}
observer_->counter("molo_bus_stream_acked_total", 1);
return true;
}
bool StreamBus::renew(const std::string& stream, const std::string& consumer, std::uint64_t seq) {
std::lock_guard<std::mutex> lock(mutex_);
StreamState& state = require_stream(stream);
ConsumerState& cstate = require_consumer(state, stream, consumer);
return cstate.acks->renew(seq, cstate.ack_deadline_ms);
}
std::uint64_t StreamBus::lag(const std::string& stream, const std::string& consumer) const {
std::lock_guard<std::mutex> lock(mutex_);
const StreamState& state = require_stream(stream);
auto it = state.consumers.find(consumer);
if (it == state.consumers.end()) {
throw BusError("StreamBus: unknown consumer " + consumer + " on " + stream);
}
const std::uint64_t tail = state.log->last_seq();
const std::uint64_t pos = it->second->cursor->position();
return tail > pos ? tail - pos : 0;
}
std::size_t StreamBus::enforce_retention(const std::string& stream) {
std::lock_guard<std::mutex> lock(mutex_);
return enforce_retention_locked(require_stream(stream));
}
std::size_t StreamBus::enforce_retention_locked(StreamState& state) {
const std::uint64_t last = state.log->last_seq();
const std::uint64_t first = state.log->first_seq();
if (last == 0 || first > last) {
return 0;
}
std::uint64_t keep_from = first;
// Time retention: drop records older than the horizon. Timestamps are
// non-decreasing in an append-only log, so probe the oldest record
// first and only scan forward in chunks when it is past the horizon.
if (state.spec.retention_hours > 0) {
const std::int64_t horizon = clock_.wall_ms() - state.spec.retention_hours * 3600 * 1000;
auto oldest = state.log->read_from(first, 1, 1);
if (!oldest.empty() && oldest.front().at_ms < horizon) {
std::uint64_t seq = first;
while (seq <= last) {
auto records = state.log->read_from(seq, 256, 65536);
if (records.empty()) {
break;
}
for (const auto& record : records) {
if (record.at_ms < horizon) {
keep_from = record.seq + 1;
} else {
seq = last + 1; // stop the outer scan
break;
}
}
if (seq <= last) {
seq = records.back().seq + 1;
}
}
}
}
// Size retention: drop oldest until the retained tail fits the byte cap.
if (state.log->bytes() > state.spec.retention_bytes) {
std::size_t remainder = state.log->bytes();
std::uint64_t seq = first;
std::uint64_t size_keep = first;
while (seq <= last && remainder > state.spec.retention_bytes) {
auto chunk = state.log->sizes(seq, 1024);
if (chunk.empty()) {
break;
}
for (const auto& [s, len] : chunk) {
if (remainder <= state.spec.retention_bytes) {
break;
}
remainder -= len;
size_keep = s + 1;
}
seq = chunk.back().first + 1;
}
if (size_keep > keep_from) {
keep_from = size_keep;
}
}
if (keep_from <= first) {
return 0;
}
const std::size_t evicted = static_cast<std::size_t>(keep_from - first);
state.log->compact(keep_from);
observer_->counter("molo_bus_stream_evicted_total", static_cast<long>(evicted));
observer_->event("bus.stream.retention", "evicted=" + std::to_string(evicted));
return evicted;
}
std::uint64_t StreamBus::last_seq(const std::string& stream) const {
std::lock_guard<std::mutex> lock(mutex_);
return require_stream(stream).log->last_seq();
}
std::size_t StreamBus::consumer_count(const std::string& stream) const {
std::lock_guard<std::mutex> lock(mutex_);
return require_stream(stream).consumers.size();
}
} // namespace molo::bus
+102
View File
@@ -0,0 +1,102 @@
#pragma once
#include <cstdint>
#include <filesystem>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "supervisor/bus/ack_tracker.hpp"
#include "supervisor/bus/consumer_cursor.hpp"
#include "supervisor/bus/segment_log.hpp"
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
namespace molo::bus {
struct StreamMessage {
std::uint64_t seq = 0;
std::string data;
std::int64_t at_ms = 0;
int redeliveries = 0;
};
struct StreamSpec {
long retention_hours = 24;
std::size_t retention_bytes = 1073741824; // 1 GiB
long ack_deadline_ms = 60000;
};
// Durable tier of the bus (S7-6): persisted append-only streams with
// per-message sequence numbers, named consumers with server-side cursors,
// ack-required delivery, and redelivery until acked or retention. Built
// on SegmentLog files under one directory (one file per stream plus one
// cursor file per consumer). Consumers poll; there are no callbacks.
class StreamBus {
public:
StreamBus(const std::filesystem::path& dir, common::Clock& clock,
std::shared_ptr<common::Observer> observer);
void create_stream(const std::string& name, const StreamSpec& spec = StreamSpec{});
bool has_stream(const std::string& name) const;
std::vector<std::string> stream_names() const;
// Appends and returns the sequence number. Payloads over 1 MiB throw.
std::uint64_t publish(const std::string& stream, const std::string& data);
void create_consumer(const std::string& stream, const std::string& consumer,
long ack_deadline_ms = 0); // 0 means stream default
bool has_consumer(const std::string& stream, const std::string& consumer) const;
// Polls up to max messages: expired redeliveries first, then new ones
// after the cursor. Every returned message arms its ack deadline.
std::vector<StreamMessage> poll(const std::string& stream, const std::string& consumer,
std::size_t max);
// Acks one sequence. Returns false when the seq was not pending.
bool ack(const std::string& stream, const std::string& consumer, std::uint64_t seq);
bool renew(const std::string& stream, const std::string& consumer, std::uint64_t seq);
// Entries behind the stream tail for one consumer (sync lag input).
std::uint64_t lag(const std::string& stream, const std::string& consumer) const;
// Enforces time and size retention on one stream (also runs on publish).
// Returns the number of records evicted.
std::size_t enforce_retention(const std::string& stream);
std::uint64_t last_seq(const std::string& stream) const;
std::size_t consumer_count(const std::string& stream) const;
private:
struct ConsumerState {
std::unique_ptr<ConsumerCursor> cursor;
std::unique_ptr<AckTracker> acks;
std::map<std::uint64_t, int> redelivery_counts; // seq -> count
std::map<std::uint64_t, bool> acked; // contiguous-commit bookkeeping
std::uint64_t next_new = 1; // next never-delivered seq; cursor trails acks
long ack_deadline_ms = 60000;
};
struct StreamState {
StreamSpec spec;
std::unique_ptr<SegmentLog> log;
std::map<std::string, std::unique_ptr<ConsumerState>> consumers;
};
static void validate_name(const std::string& name, const char* what);
StreamState& require_stream(const std::string& name);
const StreamState& require_stream(const std::string& name) const;
ConsumerState& require_consumer(StreamState& state, const std::string& stream,
const std::string& consumer);
std::size_t enforce_retention_locked(StreamState& state);
std::filesystem::path dir_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
mutable std::mutex mutex_;
std::map<std::string, std::unique_ptr<StreamState>> streams_;
};
} // namespace molo::bus
+103
View File
@@ -0,0 +1,103 @@
#include "supervisor/bus/topic_bus.hpp"
#include "supervisor/common/caps.hpp"
namespace molo::bus {
TopicBus::TopicBus(common::Clock& clock) : clock_(clock) {}
TopicBus::SubscriberId TopicBus::subscribe(const std::string& topic_or_pattern) {
if (topic_or_pattern.empty() || topic_or_pattern.size() > 256) {
throw BusError("TopicBus: bad subscription pattern");
}
std::lock_guard<std::mutex> lock(mutex_);
const SubscriberId id = next_id_++;
patterns_[id] = topic_or_pattern;
queues_[id] = {};
return id;
}
bool TopicBus::unsubscribe(SubscriberId id) {
std::lock_guard<std::mutex> lock(mutex_);
const bool had = patterns_.erase(id) > 0;
queues_.erase(id);
return had;
}
std::size_t TopicBus::publish(const std::string& topic, const std::string& payload) {
if (topic.empty() || topic.size() > 256) {
throw BusError("TopicBus: bad topic");
}
if (payload.size() > common::Caps::kStreamPayloadBytes) {
throw BusError("TopicBus: payload over 1 MiB cap");
}
std::lock_guard<std::mutex> lock(mutex_);
const std::int64_t now = clock_.wall_ms();
std::size_t delivered = 0;
for (const auto& [id, pattern] : patterns_) {
if (!matches(pattern, topic)) {
continue;
}
auto& queue = queues_[id];
if (static_cast<int>(queue.size()) >= common::Caps::kSubQueueMax) {
queue.pop_front(); // oldest-drop backpressure, counted below
++dropped_total_;
}
queue.push_back(TopicMessage{topic, payload, now});
++delivered;
}
return delivered;
}
std::vector<TopicMessage> TopicBus::poll(SubscriberId id, std::size_t max) {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<TopicMessage> out;
auto it = queues_.find(id);
if (it == queues_.end()) {
return out;
}
while (!it->second.empty() && out.size() < max) {
out.push_back(std::move(it->second.front()));
it->second.pop_front();
}
return out;
}
std::size_t TopicBus::pending(SubscriberId id) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = queues_.find(id);
return it == queues_.end() ? 0 : it->second.size();
}
std::size_t TopicBus::subscriber_count() const {
std::lock_guard<std::mutex> lock(mutex_);
return patterns_.size();
}
long TopicBus::dropped_total() const {
std::lock_guard<std::mutex> lock(mutex_);
return dropped_total_;
}
bool TopicBus::matches(const std::string& pattern, const std::string& topic) {
// ':' segments; '*' matches exactly one segment.
std::size_t p = 0, t = 0;
while (true) {
const std::size_t pn = pattern.find(':', p);
const std::size_t tn = topic.find(':', t);
const std::string_view ps(pattern.data() + p,
(pn == std::string::npos ? pattern.size() : pn) - p);
const std::string_view ts(topic.data() + t,
(tn == std::string::npos ? topic.size() : tn) - t);
if (ps != "*" && ps != ts) {
return false;
}
if (pn == std::string::npos || tn == std::string::npos) {
return pn == tn;
}
p = pn + 1;
t = tn + 1;
}
}
} // namespace molo::bus
+62
View File
@@ -0,0 +1,62 @@
#pragma once
#include <cstdint>
#include <deque>
#include <functional>
#include <map>
#include <mutex>
#include <string>
#include <vector>
#include "supervisor/bus/bus_error.hpp"
#include "supervisor/common/clock.hpp"
namespace molo::bus {
struct TopicMessage {
std::string topic;
std::string payload;
std::int64_t at_ms = 0;
};
// Ephemeral tier of the bus (S7-6): at-most-once delivery to currently
// connected subscribers, no persistence, no replay. A message published
// while a subscriber is offline is lost for that subscriber. Subscribers
// poll; publish never blocks and never awaits subscribers.
//
// Topic names use ':' segments; subscriptions may use '*' to match exactly
// one segment (same convention as 04-data.md channels).
class TopicBus {
public:
using SubscriberId = std::uint64_t;
explicit TopicBus(common::Clock& clock);
// Subscribes to an exact topic or a '*' pattern. Returns the subscriber id.
SubscriberId subscribe(const std::string& topic_or_pattern);
bool unsubscribe(SubscriberId id);
// Publishes to all matching subscribers. Returns the number of
// subscribers that received the message. Payloads over the 1 MiB cap
// are rejected (throws BusError).
std::size_t publish(const std::string& topic, const std::string& payload);
// Polls up to max messages for one subscriber (FIFO).
std::vector<TopicMessage> poll(SubscriberId id, std::size_t max);
std::size_t pending(SubscriberId id) const;
std::size_t subscriber_count() const;
long dropped_total() const; // oldest-drop events under backpressure
static bool matches(const std::string& pattern, const std::string& topic);
private:
common::Clock& clock_;
mutable std::mutex mutex_;
SubscriberId next_id_ = 1;
std::map<SubscriberId, std::string> patterns_;
std::map<SubscriberId, std::deque<TopicMessage>> queues_;
long dropped_total_ = 0;
};
} // namespace molo::bus
+49
View File
@@ -0,0 +1,49 @@
#pragma once
#include <cstddef>
namespace molo::common {
// Static ceilings for part 07 (supervisor, tasks, bus, replication).
// Every knob in this part resolves to one of these constants (R7-4, S7-10).
// Tunable only through explicitly passed option structs, never autotuned.
struct Caps {
static constexpr int kWorkersPerGroup = 64;
static constexpr int kGroupsPerSupervisor = 32;
static constexpr std::size_t kTaskPayloadBytes = 1048576; // 1 MiB
static constexpr int kQueueDepth = 100000;
static constexpr int kDeadLetterMax = 10000;
static constexpr int kDeadLetterRetentionDays = 7;
static constexpr long kStreamRetentionHours = 24;
static constexpr std::size_t kStreamRetentionBytes = 1073741824; // 1 GiB
static constexpr int kSyncBatchEntries = 500;
static constexpr std::size_t kSyncBatchBytes = 1048576; // 1 MiB
static constexpr int kConflictLogMax = 10000;
static constexpr int kConflictLogRetentionDays = 7;
static constexpr std::size_t kStatusSnapshotBytes = 1048576; // 1 MiB
static constexpr int kSyncLagAlertThreshold = 10000;
static constexpr long kLeaseDefaultMs = 60000;
static constexpr long kHeartbeatMs = 15000;
static constexpr long kSchedulerTickMs = 1000;
static constexpr long kSchedulerMinIntervalSecs = 60;
static constexpr long kLeaderTtlSecs = 10;
static constexpr long kLeaderRenewSecs = 3;
static constexpr int kIdempotencyWindowHours = 24;
static constexpr int kMaxRestartsDefault = 5;
static constexpr int kRestartWindowSecsDefault = 60;
static constexpr long kBackoffMinMs = 1000;
static constexpr long kBackoffMaxMs = 60000;
static constexpr long kHealthIntervalMs = 10000;
static constexpr int kHealthMaxFailures = 3;
static constexpr long kRetryBaseMs = 1000;
static constexpr long kRetryCapMs = 300000;
static constexpr long kRetryJitterMs = 1000;
static constexpr long kHookTimeoutMs = 5000;
static constexpr int kSubQueueMax = 1024;
static constexpr std::size_t kStreamPayloadBytes = 1048576; // 1 MiB
static constexpr int kJsonMaxDepth = 64;
static constexpr int kMaxStreamsPerBus = 1024;
static constexpr int kMaxConsumersPerStream = 1024;
};
} // namespace molo::common
+109
View File
@@ -0,0 +1,109 @@
#include "supervisor/common/clock.hpp"
#include <cstdio>
#include <ctime>
#include <thread>
namespace molo::common {
std::int64_t SystemClock::wall_ms() const {
const auto now = std::chrono::system_clock::now();
return std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
}
std::int64_t SystemClock::steady_ms() const {
const auto now = std::chrono::steady_clock::now();
return std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
}
void SystemClock::sleep_ms(long ms) {
if (ms > 0) {
std::this_thread::sleep_for(std::chrono::milliseconds(ms));
}
}
FrozenClock::FrozenClock(std::int64_t start_wall_ms, std::int64_t start_steady_ms)
: wall_ms_(start_wall_ms), steady_ms_(start_steady_ms) {}
std::int64_t FrozenClock::wall_ms() const {
std::lock_guard<std::mutex> lock(mutex_);
return wall_ms_;
}
std::int64_t FrozenClock::steady_ms() const {
std::lock_guard<std::mutex> lock(mutex_);
return steady_ms_;
}
void FrozenClock::sleep_ms(long ms) {
if (ms > 0) {
advance(ms);
}
}
void FrozenClock::advance(std::int64_t delta_ms) {
std::lock_guard<std::mutex> lock(mutex_);
wall_ms_ += delta_ms;
steady_ms_ += delta_ms;
}
void FrozenClock::set(std::int64_t wall_ms, std::int64_t steady_ms) {
std::lock_guard<std::mutex> lock(mutex_);
wall_ms_ = wall_ms;
steady_ms_ = steady_ms;
}
std::string iso8601(std::int64_t wall_ms) {
const std::time_t secs = static_cast<std::time_t>(wall_ms / 1000);
const int millis = static_cast<int>(wall_ms % 1000);
std::tm tm_utc{};
#if defined(_WIN32)
gmtime_s(&tm_utc, &secs);
#else
gmtime_r(&secs, &tm_utc);
#endif
char buf[128];
std::snprintf(buf, sizeof(buf), "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ", tm_utc.tm_year + 1900,
tm_utc.tm_mon + 1, tm_utc.tm_mday, tm_utc.tm_hour, tm_utc.tm_min, tm_utc.tm_sec,
millis);
return std::string(buf);
}
std::int64_t parse_iso8601(const std::string& text) {
// Accepts YYYY-MM-DDTHH:MM:SS[.mmm]Z only. Anything else returns -1.
int year = 0, mon = 0, day = 0, hour = 0, min = 0, sec = 0, millis = 0;
if (text.size() < 20 || text.back() != 'Z') {
return -1;
}
int matched = std::sscanf(text.c_str(), "%4d-%2d-%2dT%2d:%2d:%2d.%3dZ", &year, &mon, &day,
&hour, &min, &sec, &millis);
if (matched != 7) {
matched = std::sscanf(text.c_str(), "%4d-%2d-%2dT%2d:%2d:%2dZ", &year, &mon, &day, &hour,
&min, &sec);
if (matched != 6) {
return -1;
}
}
if (mon < 1 || mon > 12 || day < 1 || day > 31 || hour > 23 || min > 59 || sec > 60 ||
millis < 0 || millis > 999) {
return -1;
}
std::tm tm_utc{};
tm_utc.tm_year = year - 1900;
tm_utc.tm_mon = mon - 1;
tm_utc.tm_mday = day;
tm_utc.tm_hour = hour;
tm_utc.tm_min = min;
tm_utc.tm_sec = sec;
#if defined(_WIN32)
const std::time_t secs = _mkgmtime(&tm_utc);
#else
const std::time_t secs = timegm(&tm_utc);
#endif
if (secs == static_cast<std::time_t>(-1)) {
return -1;
}
return static_cast<std::int64_t>(secs) * 1000 + millis;
}
} // namespace molo::common
+50
View File
@@ -0,0 +1,50 @@
#pragma once
#include <chrono>
#include <cstdint>
#include <mutex>
#include <string>
namespace molo::common {
// Clock abstraction. Production uses SystemClock; tests use FrozenClock
// for deterministic scheduler, lease, and backoff behavior.
class Clock {
public:
virtual ~Clock() = default;
virtual std::int64_t wall_ms() const = 0; // unix epoch milliseconds
virtual std::int64_t steady_ms() const = 0; // monotonic milliseconds
virtual void sleep_ms(long ms) = 0;
};
class SystemClock : public Clock {
public:
std::int64_t wall_ms() const override;
std::int64_t steady_ms() const override;
void sleep_ms(long ms) override;
};
class FrozenClock : public Clock {
public:
explicit FrozenClock(std::int64_t start_wall_ms = 0, std::int64_t start_steady_ms = 0);
std::int64_t wall_ms() const override;
std::int64_t steady_ms() const override;
void sleep_ms(long ms) override; // advances the frozen clock, no real wait
void advance(std::int64_t delta_ms);
void set(std::int64_t wall_ms, std::int64_t steady_ms);
private:
mutable std::mutex mutex_;
std::int64_t wall_ms_;
std::int64_t steady_ms_;
};
// UTC ISO-8601 with millis, e.g. 2026-10-23T12:34:56.789Z. Used by the
// task envelope (S7-1) and oplog (S7-7) JSON formats.
std::string iso8601(std::int64_t wall_ms);
// Parses iso8601() output (millis optional). Returns -1 on invalid input.
std::int64_t parse_iso8601(const std::string& text);
} // namespace molo::common
+382
View File
@@ -0,0 +1,382 @@
#include "supervisor/common/json.hpp"
#include <cctype>
#include "supervisor/common/caps.hpp"
namespace molo::common {
bool JValue::is_null() const { return std::holds_alternative<std::nullptr_t>(data); }
bool JValue::is_bool() const { return std::holds_alternative<bool>(data); }
bool JValue::is_int() const { return std::holds_alternative<long long>(data); }
bool JValue::is_double() const { return std::holds_alternative<double>(data); }
bool JValue::is_string() const { return std::holds_alternative<std::string>(data); }
bool JValue::is_array() const { return std::holds_alternative<Array>(data); }
bool JValue::is_object() const { return std::holds_alternative<Object>(data); }
bool JValue::as_bool(bool fallback) const {
return is_bool() ? std::get<bool>(data) : fallback;
}
long long JValue::as_int(long long fallback) const {
if (is_int()) {
return std::get<long long>(data);
}
if (is_double()) {
return static_cast<long long>(std::get<double>(data));
}
return fallback;
}
double JValue::as_double(double fallback) const {
if (is_double()) {
return std::get<double>(data);
}
if (is_int()) {
return static_cast<double>(std::get<long long>(data));
}
return fallback;
}
std::string JValue::as_string(const std::string& fallback) const {
return is_string() ? std::get<std::string>(data) : fallback;
}
const JValue::Array& JValue::as_array() const {
static const Array empty;
return is_array() ? std::get<Array>(data) : empty;
}
const JValue::Object& JValue::as_object() const {
static const Object empty;
return is_object() ? std::get<Object>(data) : empty;
}
const JValue* JValue::find(std::string_view key) const {
if (!is_object()) {
return nullptr;
}
const auto& obj = std::get<Object>(data);
auto it = obj.find(std::string(key));
return it == obj.end() ? nullptr : &it->second;
}
std::string json_escape(std::string_view text) {
std::string out;
out.reserve(text.size() + 2);
for (char c : text) {
switch (c) {
case '"': out += "\\\""; break;
case '\\': out += "\\\\"; break;
case '\n': out += "\\n"; break;
case '\r': out += "\\r"; break;
case '\t': out += "\\t"; break;
default:
if (static_cast<unsigned char>(c) < 0x20) {
char buf[8];
const int n = std::snprintf(buf, sizeof(buf), "\\u%04x", c);
out.append(buf, static_cast<std::size_t>(n));
} else {
out += c;
}
}
}
return out;
}
namespace {
void encode_into(const JValue& value, std::string& out) {
if (value.is_null()) {
out += "null";
} else if (value.is_bool()) {
out += value.as_bool() ? "true" : "false";
} else if (value.is_int()) {
out += std::to_string(value.as_int());
} else if (value.is_double()) {
out += std::to_string(value.as_double());
} else if (value.is_string()) {
out += '"';
out += json_escape(value.as_string());
out += '"';
} else if (value.is_array()) {
out += '[';
bool first = true;
for (const auto& item : value.as_array()) {
if (!first) {
out += ',';
}
first = false;
encode_into(item, out);
}
out += ']';
} else {
out += '{';
bool first = true;
for (const auto& [key, item] : value.as_object()) {
if (!first) {
out += ',';
}
first = false;
out += '"';
out += json_escape(key);
out += "\":";
encode_into(item, out);
}
out += '}';
}
}
class Parser {
public:
explicit Parser(std::string_view text) : text_(text) {}
bool run(JValue& out) {
skip_ws();
if (!parse_value(out, 0)) {
return false;
}
skip_ws();
return pos_ == text_.size();
}
private:
void skip_ws() {
while (pos_ < text_.size() &&
(text_[pos_] == ' ' || text_[pos_] == '\t' || text_[pos_] == '\n' ||
text_[pos_] == '\r')) {
++pos_;
}
}
bool consume(char c) {
if (pos_ < text_.size() && text_[pos_] == c) {
++pos_;
return true;
}
return false;
}
bool parse_value(JValue& out, int depth) {
if (depth > Caps::kJsonMaxDepth || pos_ >= text_.size()) {
return false;
}
const char c = text_[pos_];
if (c == '{') {
return parse_object(out, depth);
}
if (c == '[') {
return parse_array(out, depth);
}
if (c == '"') {
std::string s;
if (!parse_string(s)) {
return false;
}
out = JValue(s);
return true;
}
if (c == 't') {
return parse_literal("true", JValue(true), out);
}
if (c == 'f') {
return parse_literal("false", JValue(false), out);
}
if (c == 'n') {
return parse_literal("null", JValue(nullptr), out);
}
return parse_number(out);
}
bool parse_literal(std::string_view word, const JValue& value, JValue& out) {
if (text_.substr(pos_, word.size()) != word) {
return false;
}
pos_ += word.size();
out = value;
return true;
}
bool parse_number(JValue& out) {
std::size_t start = pos_;
if (pos_ < text_.size() && (text_[pos_] == '-' || text_[pos_] == '+')) {
++pos_;
}
bool is_double = false;
while (pos_ < text_.size()) {
const char c = text_[pos_];
if (c >= '0' && c <= '9') {
++pos_;
} else if (c == '.' || c == 'e' || c == 'E') {
is_double = true;
++pos_;
} else if ((c == '+' || c == '-') &&
(text_[pos_ - 1] == 'e' || text_[pos_ - 1] == 'E')) {
++pos_;
} else {
break;
}
}
if (pos_ == start) {
return false;
}
const std::string token(text_.substr(start, pos_ - start));
try {
if (is_double) {
out = JValue(std::stod(token));
} else {
out = JValue(static_cast<long long>(std::stoll(token)));
}
} catch (...) {
return false;
}
return true;
}
bool parse_string(std::string& out) {
if (!consume('"')) {
return false;
}
out.clear();
while (pos_ < text_.size()) {
const char c = text_[pos_++];
if (c == '"') {
return true;
}
if (c != '\\') {
out += c;
continue;
}
if (pos_ >= text_.size()) {
return false;
}
const char e = text_[pos_++];
switch (e) {
case '"': out += '"'; break;
case '\\': out += '\\'; break;
case '/': out += '/'; break;
case 'n': out += '\n'; break;
case 'r': out += '\r'; break;
case 't': out += '\t'; break;
case 'b': out += '\b'; break;
case 'f': out += '\f'; break;
case 'u': {
if (pos_ + 4 > text_.size()) {
return false;
}
unsigned code = 0;
for (int i = 0; i < 4; ++i) {
const char h = text_[pos_++];
code <<= 4;
if (h >= '0' && h <= '9') {
code += static_cast<unsigned>(h - '0');
} else if (h >= 'a' && h <= 'f') {
code += static_cast<unsigned>(h - 'a' + 10);
} else if (h >= 'A' && h <= 'F') {
code += static_cast<unsigned>(h - 'A' + 10);
} else {
return false;
}
}
// BMP only, enough for the ASCII control escapes we emit.
if (code < 0x80) {
out += static_cast<char>(code);
} else if (code < 0x800) {
out += static_cast<char>(0xC0 | (code >> 6));
out += static_cast<char>(0x80 | (code & 0x3F));
} else {
out += static_cast<char>(0xE0 | (code >> 12));
out += static_cast<char>(0x80 | ((code >> 6) & 0x3F));
out += static_cast<char>(0x80 | (code & 0x3F));
}
break;
}
default: return false;
}
}
return false;
}
bool parse_object(JValue& out, int depth) {
if (!consume('{')) {
return false;
}
JValue::Object obj;
skip_ws();
if (consume('}')) {
out = JValue(obj);
return true;
}
while (true) {
skip_ws();
std::string key;
if (!parse_string(key)) {
return false;
}
skip_ws();
if (!consume(':')) {
return false;
}
skip_ws();
JValue value;
if (!parse_value(value, depth + 1)) {
return false;
}
obj[std::move(key)] = std::move(value);
skip_ws();
if (consume('}')) {
out = JValue(obj);
return true;
}
if (!consume(',')) {
return false;
}
}
}
bool parse_array(JValue& out, int depth) {
if (!consume('[')) {
return false;
}
JValue::Array items;
skip_ws();
if (consume(']')) {
out = JValue(items);
return true;
}
while (true) {
skip_ws();
JValue value;
if (!parse_value(value, depth + 1)) {
return false;
}
items.push_back(std::move(value));
skip_ws();
if (consume(']')) {
out = JValue(items);
return true;
}
if (!consume(',')) {
return false;
}
}
}
std::string_view text_;
std::size_t pos_ = 0;
};
} // namespace
std::string json_encode(const JValue& value) {
std::string out;
encode_into(value, out);
return out;
}
bool json_decode(std::string_view text, JValue& out) {
Parser parser(text);
return parser.run(out);
}
} // namespace molo::common
+61
View File
@@ -0,0 +1,61 @@
#pragma once
#include <cstdint>
#include <map>
#include <string>
#include <string_view>
#include <variant>
#include <vector>
namespace molo::common {
// Minimal JSON value plus encoder/decoder for the on-disk formats owned by
// this part (task envelope S7-1, oplog entry S7-7, status snapshot). Only
// what those formats need: objects, arrays, strings, integers, doubles,
// booleans, null. Depth capped by Caps::kJsonMaxDepth.
struct JValue {
using Array = std::vector<JValue>;
using Object = std::map<std::string, JValue>;
using Data = std::variant<std::nullptr_t, bool, long long, double, std::string, Array, Object>;
Data data;
JValue() : data(nullptr) {}
JValue(std::nullptr_t) : data(nullptr) {}
JValue(bool v) : data(v) {}
JValue(long long v) : data(v) {}
JValue(int v) : data(static_cast<long long>(v)) {}
JValue(double v) : data(v) {}
JValue(const char* v) : data(std::string(v)) {}
JValue(std::string_view v) : data(std::string(v)) {}
JValue(const std::string& v) : data(v) {}
JValue(const Array& v) : data(v) {}
JValue(const Object& v) : data(v) {}
bool is_null() const;
bool is_bool() const;
bool is_int() const;
bool is_double() const;
bool is_string() const;
bool is_array() const;
bool is_object() const;
bool as_bool(bool fallback = false) const;
long long as_int(long long fallback = 0) const;
double as_double(double fallback = 0.0) const;
std::string as_string(const std::string& fallback = "") const;
const Array& as_array() const;
const Object& as_object() const;
// Object helpers. find() returns nullptr when absent or not an object.
const JValue* find(std::string_view key) const;
};
std::string json_escape(std::string_view text);
std::string json_encode(const JValue& value);
// Parses a full JSON document. Returns false on any syntax error or when
// trailing garbage follows the document. Depth is capped.
bool json_decode(std::string_view text, JValue& out);
} // namespace molo::common
+118
View File
@@ -0,0 +1,118 @@
#include "supervisor/common/kv_store.hpp"
#include "supervisor/common/clock.hpp"
namespace molo::common {
InMemoryKvStore::InMemoryKvStore(Clock& clock) : clock_(clock) {}
bool InMemoryKvStore::expired(const Entry& entry, std::int64_t now_ms) const {
return entry.expires_at_ms > 0 && entry.expires_at_ms <= now_ms;
}
bool InMemoryKvStore::get(std::string_view key, std::string& value) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = entries_.find(key);
if (it == entries_.end() || expired(it->second, clock_.wall_ms())) {
return false;
}
value = it->second.value;
return true;
}
void InMemoryKvStore::set(std::string_view key, const std::string& value, long ttl_secs) {
std::lock_guard<std::mutex> lock(mutex_);
Entry entry;
entry.value = value;
entry.expires_at_ms = ttl_secs > 0 ? clock_.wall_ms() + ttl_secs * 1000 : 0;
entries_[std::string(key)] = std::move(entry);
}
bool InMemoryKvStore::erase(std::string_view key) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = entries_.find(key);
if (it == entries_.end()) {
return false;
}
entries_.erase(it);
return true;
}
bool InMemoryKvStore::acquire(std::string_view key, const std::string& value, long ttl_secs) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = entries_.find(key);
if (it != entries_.end() && !expired(it->second, clock_.wall_ms())) {
return false;
}
Entry entry;
entry.value = value;
entry.expires_at_ms = ttl_secs > 0 ? clock_.wall_ms() + ttl_secs * 1000 : 0;
entries_[std::string(key)] = std::move(entry);
return true;
}
bool InMemoryKvStore::renew(std::string_view key, const std::string& expected, long ttl_secs) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = entries_.find(key);
if (it == entries_.end() || expired(it->second, clock_.wall_ms()) ||
it->second.value != expected) {
return false;
}
it->second.expires_at_ms = ttl_secs > 0 ? clock_.wall_ms() + ttl_secs * 1000 : 0;
return true;
}
bool InMemoryKvStore::acquire_ms(std::string_view key, const std::string& value,
std::int64_t ttl_ms) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = entries_.find(key);
if (it != entries_.end() && !expired(it->second, clock_.wall_ms())) {
return false;
}
Entry entry;
entry.value = value;
entry.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
entries_[std::string(key)] = std::move(entry);
return true;
}
bool InMemoryKvStore::renew_ms(std::string_view key, const std::string& expected,
std::int64_t ttl_ms) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = entries_.find(key);
if (it == entries_.end() || expired(it->second, clock_.wall_ms()) ||
it->second.value != expected) {
return false;
}
it->second.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
return true;
}
bool InMemoryKvStore::cas_ms(std::string_view key, const std::string& expected,
const std::string& next, std::int64_t ttl_ms) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = entries_.find(key);
if (it == entries_.end() || expired(it->second, clock_.wall_ms())) {
if (!expected.empty()) {
return false;
}
Entry entry;
entry.value = next;
entry.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
entries_[std::string(key)] = std::move(entry);
return true;
}
if (it->second.value != expected) {
return false;
}
it->second.value = next;
it->second.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
return true;
}
std::size_t InMemoryKvStore::size() const {
std::lock_guard<std::mutex> lock(mutex_);
return entries_.size();
}
} // namespace molo::common
+79
View File
@@ -0,0 +1,79 @@
#pragma once
#include <cstdint>
#include <map>
#include <mutex>
#include <string>
#include <string_view>
namespace molo::common {
class Clock;
// Minimal key-value interface needed by part 07 (scheduler leader lease,
// shared last-fire records). Mirrors the 04-data.md KvStore shape
// (get/set/del plus atomic acquire/renew for leases) so the shared
// backend from 04-data.md slots in behind this interface for
// multi-instance deployments (S7-9). Single-node deployments use
// InMemoryKvStore.
class KvStore {
public:
virtual ~KvStore() = default;
// Returns true and fills value when the key exists and is not expired.
virtual bool get(std::string_view key, std::string& value) = 0;
// Stores value. ttl_secs <= 0 means no expiry.
virtual void set(std::string_view key, const std::string& value, long ttl_secs) = 0;
virtual bool erase(std::string_view key) = 0;
// Atomically stores value when the key is absent or expired.
virtual bool acquire(std::string_view key, const std::string& value, long ttl_secs) = 0;
// Atomically extends the TTL when the current value equals expected.
virtual bool renew(std::string_view key, const std::string& expected, long ttl_secs) = 0;
// Millisecond-precision fencing API for the operability part (leases,
// leader election, distributed locks run on ms clocks). Same semantics as
// the seconds API: ttl_ms <= 0 means no expiry; empty expected matches a
// missing key for cas_ms only.
virtual bool acquire_ms(std::string_view key, const std::string& value,
std::int64_t ttl_ms) = 0;
virtual bool renew_ms(std::string_view key, const std::string& expected,
std::int64_t ttl_ms) = 0;
virtual bool cas_ms(std::string_view key, const std::string& expected,
const std::string& next, std::int64_t ttl_ms) = 0;
};
class InMemoryKvStore : public KvStore {
public:
explicit InMemoryKvStore(Clock& clock);
bool get(std::string_view key, std::string& value) override;
void set(std::string_view key, const std::string& value, long ttl_secs) override;
bool erase(std::string_view key) override;
bool acquire(std::string_view key, const std::string& value, long ttl_secs) override;
bool renew(std::string_view key, const std::string& expected, long ttl_secs) override;
bool acquire_ms(std::string_view key, const std::string& value,
std::int64_t ttl_ms) override;
bool renew_ms(std::string_view key, const std::string& expected,
std::int64_t ttl_ms) override;
bool cas_ms(std::string_view key, const std::string& expected, const std::string& next,
std::int64_t ttl_ms) override;
std::size_t size() const;
private:
struct Entry {
std::string value;
std::int64_t expires_at_ms = 0; // 0 means no expiry
};
bool expired(const Entry& entry, std::int64_t now_ms) const;
Clock& clock_;
mutable std::mutex mutex_;
std::map<std::string, Entry, std::less<>> entries_;
};
} // namespace molo::common
+61
View File
@@ -0,0 +1,61 @@
#include "supervisor/common/observer.hpp"
namespace molo::common {
void NullObserver::event(std::string_view, std::string_view) {}
void NullObserver::counter(std::string_view, long) {}
void NullObserver::gauge(std::string_view, double) {}
void RecordingObserver::event(std::string_view type, std::string_view detail) {
std::lock_guard<std::mutex> lock(mutex_);
events_.push_back(RecordedEvent{std::string(type), std::string(detail)});
}
void RecordingObserver::counter(std::string_view name, long delta) {
std::lock_guard<std::mutex> lock(mutex_);
counters_[std::string(name)] += delta;
}
void RecordingObserver::gauge(std::string_view name, double value) {
std::lock_guard<std::mutex> lock(mutex_);
gauges_[std::string(name)] = value;
}
std::vector<RecordedEvent> RecordingObserver::events() const {
std::lock_guard<std::mutex> lock(mutex_);
return events_;
}
std::map<std::string, long> RecordingObserver::counters() const {
std::lock_guard<std::mutex> lock(mutex_);
return counters_;
}
std::map<std::string, double> RecordingObserver::gauges() const {
std::lock_guard<std::mutex> lock(mutex_);
return gauges_;
}
long RecordingObserver::counter_value(const std::string& name) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = counters_.find(name);
return it == counters_.end() ? 0 : it->second;
}
std::size_t RecordingObserver::event_count(const std::string& type) const {
std::lock_guard<std::mutex> lock(mutex_);
std::size_t count = 0;
for (const auto& e : events_) {
if (e.type == type) {
++count;
}
}
return count;
}
std::shared_ptr<Observer> null_observer() {
static std::shared_ptr<Observer> instance = std::make_shared<NullObserver>();
return instance;
}
} // namespace molo::common
+60
View File
@@ -0,0 +1,60 @@
#pragma once
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <string_view>
#include <vector>
namespace molo::common {
// Minimal observability sink for part 07 (R7-6).
//
// This is the seam where 02-observability.md plugs in: the real Dispatcher,
// metrics Registry, and Tracer implement this interface (or this interface is
// replaced by direct calls to molo::events, molo::metrics, molo::trace once
// that part lands). Until then NullObserver keeps production quiet and
// RecordingObserver lets tests assert on emitted events and metrics.
class Observer {
public:
virtual ~Observer() = default;
virtual void event(std::string_view type, std::string_view detail) = 0;
virtual void counter(std::string_view name, long delta) = 0;
virtual void gauge(std::string_view name, double value) = 0;
};
class NullObserver : public Observer {
public:
void event(std::string_view type, std::string_view detail) override;
void counter(std::string_view name, long delta) override;
void gauge(std::string_view name, double value) override;
};
struct RecordedEvent {
std::string type;
std::string detail;
};
class RecordingObserver : public Observer {
public:
void event(std::string_view type, std::string_view detail) override;
void counter(std::string_view name, long delta) override;
void gauge(std::string_view name, double value) override;
std::vector<RecordedEvent> events() const;
std::map<std::string, long> counters() const;
std::map<std::string, double> gauges() const;
long counter_value(const std::string& name) const;
std::size_t event_count(const std::string& type) const;
private:
mutable std::mutex mutex_;
std::vector<RecordedEvent> events_;
std::map<std::string, long> counters_;
std::map<std::string, double> gauges_;
};
std::shared_ptr<Observer> null_observer();
} // namespace molo::common
@@ -0,0 +1,230 @@
#include "supervisor/replication/conflict_resolver.hpp"
#include <chrono>
#include <future>
#include <thread>
#include "supervisor/common/caps.hpp"
namespace molo::replication {
std::string to_string(ConflictPolicy policy) {
switch (policy) {
case ConflictPolicy::Optimistic: return "optimistic";
case ConflictPolicy::LastWriterWins: return "lww";
case ConflictPolicy::Hook: return "hook";
}
return "optimistic";
}
ConflictResolver::ConflictResolver(common::Clock& clock,
std::shared_ptr<common::Observer> observer,
int max_log_entries, long log_retention_days)
: clock_(clock),
observer_(std::move(observer)),
max_log_entries_(max_log_entries),
log_retention_days_(log_retention_days) {
if (max_log_entries_ < 1) {
max_log_entries_ = 1;
}
}
void ConflictResolver::declare(const std::string& model, ConflictPolicy policy,
ConflictHook hook) {
if (model.empty() || model.size() > 256) {
throw ReplicationError("ConflictResolver: bad model name");
}
std::lock_guard<std::mutex> lock(mutex_);
if (policy == ConflictPolicy::Hook && !hook) {
policy = ConflictPolicy::LastWriterWins; // declared fallback, logged below
observer_->event("replication.policy_fallback", model + " hook->lww (no callback)");
}
policies_[model] = policy;
if (hook) {
hooks_[model] = std::move(hook);
} else {
hooks_.erase(model);
}
observer_->event("replication.policy_declared", model + " " + to_string(policy));
}
ConflictPolicy ConflictResolver::policy_for(const std::string& model) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = policies_.find(model);
return it == policies_.end() ? ConflictPolicy::Optimistic : it->second;
}
ConflictOutcome ConflictResolver::resolve(const OplogEntry& local, const OplogEntry& remote) {
ConflictPolicy policy;
ConflictHook hook;
{
std::lock_guard<std::mutex> lock(mutex_);
auto it = policies_.find(local.model);
policy = it == policies_.end() ? ConflictPolicy::Optimistic : it->second;
auto hit = hooks_.find(local.model);
if (hit != hooks_.end()) {
hook = hit->second;
}
}
// Run the policy without the resolver lock: hooks are user code.
ConflictOutcome outcome;
switch (policy) {
case ConflictPolicy::Optimistic: outcome = resolve_optimistic(local, remote); break;
case ConflictPolicy::LastWriterWins: outcome = resolve_lww(local, remote); break;
case ConflictPolicy::Hook: outcome = resolve_hook(local, remote, hook); break;
}
if (outcome.conflict) {
observer_->counter("molo_replication_conflicts_total", 1);
observer_->event("replication.conflict",
local.model + ":" + local.key + " " + outcome.detail);
}
return outcome;
}
ConflictOutcome ConflictResolver::resolve_optimistic(const OplogEntry& local,
const OplogEntry& remote) {
ConflictOutcome outcome;
if (remote.version > local.version) {
outcome.remote_won = true;
outcome.winner = remote;
outcome.detail = "remote version " + std::to_string(remote.version) + " > local " +
std::to_string(local.version);
// Fast-forward, not a conflict.
} else if (remote.version < local.version) {
outcome.winner = local;
outcome.conflict = true;
outcome.conflict_error = true;
outcome.winner_version = local.version;
outcome.detail = "stale remote version " + std::to_string(remote.version) + ", winner " +
std::to_string(local.version);
log_conflict("optimistic", local, remote);
} else if (remote.data == local.data && remote.op == local.op) {
outcome.winner = local; // identical write, idempotent replay
outcome.detail = "identical versions and data";
} else {
// Same version, different data: a genuine write-write conflict.
// The incumbent stays; the loser gets ConflictError with the winner
// version so it can re-read and retry.
outcome.winner = local;
outcome.conflict = true;
outcome.conflict_error = true;
outcome.winner_version = local.version;
outcome.detail = "write-write conflict at version " + std::to_string(local.version);
log_conflict("optimistic", local, remote);
}
return outcome;
}
ConflictOutcome ConflictResolver::resolve_lww(const OplogEntry& local, const OplogEntry& remote) {
ConflictOutcome outcome;
// Concurrent writes with any difference are a conflict whose loser is
// kept in the conflict log (S7-8); only exact replays are quiet.
outcome.conflict = (local.version != remote.version || local.data != remote.data ||
local.op != remote.op);
const int cmp = remote.ts.compare(local.ts);
if (cmp > 0) {
outcome.remote_won = true;
outcome.winner = remote;
outcome.detail = "remote ts " + remote.ts.encode() + " wins";
if (outcome.conflict) {
log_conflict("lww", remote, local);
}
} else {
outcome.winner = local;
outcome.detail = "local ts " + local.ts.encode() + " wins";
if (outcome.conflict) {
log_conflict("lww", local, remote);
}
}
outcome.winner_version = outcome.winner.version;
return outcome;
}
ConflictOutcome ConflictResolver::resolve_hook(const OplogEntry& local, const OplogEntry& remote,
const ConflictHook& hook) {
if (!hook) {
return resolve_lww(local, remote);
}
// The hook runs off-lock with a 5s budget; on timeout or throw the LWW
// fallback applies (S7-8). Packaged task plus timed wait, no detached
// threads: a stuck hook still joins before we return.
std::packaged_task<int()> task([&] { return hook(local, remote); });
std::future<int> future = task.get_future();
std::thread runner(std::move(task));
int decision = 1;
bool fallback = false;
if (future.wait_for(std::chrono::milliseconds(common::Caps::kHookTimeoutMs)) ==
std::future_status::ready) {
try {
decision = future.get();
} catch (...) {
fallback = true;
}
if (decision != 0 && decision != 1) {
fallback = true;
}
} else {
fallback = true;
}
runner.join();
if (fallback) {
observer_->event("replication.hook_fallback", local.model + ":" + local.key);
ConflictOutcome outcome = resolve_lww(local, remote);
outcome.detail = "hook fallback: " + outcome.detail;
return outcome;
}
ConflictOutcome outcome;
outcome.conflict = true;
if (decision == 1) {
outcome.remote_won = true;
outcome.winner = remote;
log_conflict("hook", remote, local);
} else {
outcome.winner = local;
log_conflict("hook", local, remote);
}
outcome.winner_version = outcome.winner.version;
outcome.detail = "hook decision=" + std::to_string(decision);
return outcome;
}
void ConflictResolver::log_conflict(const std::string& policy, const OplogEntry& winner,
const OplogEntry& loser) {
std::lock_guard<std::mutex> lock(mutex_);
while (static_cast<int>(log_.size()) >= max_log_entries_) {
log_.pop_front();
observer_->counter("molo_replication_conflict_log_evicted_total", 1);
}
log_.push_back(ConflictRecord{clock_.wall_ms(), policy, winner, loser});
}
std::vector<ConflictRecord> ConflictResolver::conflict_log(std::size_t max) const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<ConflictRecord> out;
for (const auto& record : log_) {
if (out.size() >= max) {
break;
}
out.push_back(record);
}
return out;
}
std::size_t ConflictResolver::conflict_log_size() const {
std::lock_guard<std::mutex> lock(mutex_);
return log_.size();
}
std::size_t ConflictResolver::purge_expired() {
std::lock_guard<std::mutex> lock(mutex_);
const std::int64_t horizon =
clock_.wall_ms() - log_retention_days_ * 86400 * 1000;
std::size_t purged = 0;
while (!log_.empty() && log_.front().at_ms < horizon) {
log_.pop_front();
++purged;
}
return purged;
}
} // namespace molo::replication
@@ -0,0 +1,85 @@
#pragma once
#include <cstdint>
#include <deque>
#include <functional>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/replication/oplog_entry.hpp"
namespace molo::replication {
// Per-model conflict policy, declared once (S7-8, R7-7).
enum class ConflictPolicy {
Optimistic, // compare version, reject stale with ConflictError
LastWriterWins, // higher hybrid timestamp wins, loser kept in conflict log
Hook, // typed callback decides, 5s budget, LWW fallback
};
std::string to_string(ConflictPolicy policy);
// Returns 0 when local wins, 1 when remote wins.
using ConflictHook = std::function<int(const OplogEntry& local, const OplogEntry& remote)>;
struct ConflictOutcome {
bool conflict = false; // true when both sides wrote concurrently
bool remote_won = false; // true when the remote entry becomes current
OplogEntry winner; // the entry that becomes current
std::string detail; // human-readable reason, capped at 1 KiB
bool conflict_error = false; // true when the caller must raise ConflictError
long long winner_version = 0;
};
struct ConflictRecord {
std::int64_t at_ms = 0;
std::string policy;
OplogEntry winner;
OplogEntry loser;
};
// Dispatches per-model conflict policy (S7-8). The ONLY place that compares
// versions or timestamps: callers never resolve conflicts ad hoc (R7-7).
class ConflictResolver {
public:
ConflictResolver(common::Clock& clock, std::shared_ptr<common::Observer> observer,
int max_log_entries = 10000, long log_retention_days = 7);
// Declares (or replaces) the policy for one model. Hook policy needs a
// callback; without one the declaration falls back to LastWriterWins.
void declare(const std::string& model, ConflictPolicy policy,
ConflictHook hook = ConflictHook{});
ConflictPolicy policy_for(const std::string& model) const; // default: Optimistic
// Resolves local (current) vs remote (incoming) for the same model+key.
ConflictOutcome resolve(const OplogEntry& local, const OplogEntry& remote);
std::vector<ConflictRecord> conflict_log(std::size_t max) const;
std::size_t conflict_log_size() const;
std::size_t purge_expired();
private:
ConflictOutcome resolve_optimistic(const OplogEntry& local, const OplogEntry& remote);
ConflictOutcome resolve_lww(const OplogEntry& local, const OplogEntry& remote);
ConflictOutcome resolve_hook(const OplogEntry& local, const OplogEntry& remote,
const ConflictHook& hook);
void log_conflict(const std::string& policy, const OplogEntry& winner,
const OplogEntry& loser);
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
int max_log_entries_;
long log_retention_days_;
mutable std::mutex mutex_;
std::map<std::string, ConflictPolicy> policies_;
std::map<std::string, ConflictHook> hooks_;
std::deque<ConflictRecord> log_;
};
} // namespace molo::replication
@@ -0,0 +1,79 @@
#include "supervisor/replication/hybrid_clock.hpp"
namespace molo::replication {
int HybridTs::compare(const HybridTs& other) const {
if (wall_ms != other.wall_ms) {
return wall_ms < other.wall_ms ? -1 : 1;
}
if (counter != other.counter) {
return counter < other.counter ? -1 : 1;
}
if (instance != other.instance) {
return instance < other.instance ? -1 : 1;
}
return 0;
}
bool HybridTs::operator<(const HybridTs& other) const {
return compare(other) < 0;
}
bool HybridTs::operator==(const HybridTs& other) const {
return compare(other) == 0;
}
std::string HybridTs::encode() const {
return std::to_string(wall_ms) + "." + std::to_string(counter) + "." + instance;
}
bool HybridTs::decode(const std::string& text, HybridTs& out) {
const std::size_t first = text.find('.');
if (first == std::string::npos) {
return false;
}
const std::size_t second = text.find('.', first + 1);
if (second == std::string::npos) {
return false;
}
try {
out.wall_ms = std::stoll(text.substr(0, first));
out.counter = static_cast<std::uint32_t>(std::stoul(text.substr(first + 1, second - first - 1)));
} catch (...) {
return false;
}
out.instance = text.substr(second + 1);
return !out.instance.empty();
}
HybridClock::HybridClock(std::string instance, common::Clock& clock)
: instance_(std::move(instance)), clock_(clock) {}
HybridTs HybridClock::now() {
std::lock_guard<std::mutex> lock(mutex_);
const std::int64_t wall = clock_.wall_ms();
if (wall > last_wall_ms_) {
last_wall_ms_ = wall;
counter_ = 0;
} else {
++counter_; // same ms (or skew): the counter keeps order
}
return HybridTs{last_wall_ms_, counter_, instance_};
}
void HybridClock::observe(const HybridTs& remote) {
std::lock_guard<std::mutex> lock(mutex_);
const std::int64_t wall = clock_.wall_ms();
if (remote.wall_ms > last_wall_ms_ && remote.wall_ms > wall) {
last_wall_ms_ = remote.wall_ms;
counter_ = remote.counter + 1;
} else if (remote.wall_ms == last_wall_ms_ && remote.counter >= counter_) {
counter_ = remote.counter + 1;
} else if (wall > last_wall_ms_) {
last_wall_ms_ = wall;
counter_ = 0;
}
// else: remote is older, nothing to fold in
}
} // namespace molo::replication
@@ -0,0 +1,43 @@
#pragma once
#include <cstdint>
#include <mutex>
#include <string>
#include "supervisor/common/clock.hpp"
namespace molo::replication {
// Hybrid logical timestamp: wall clock plus a counter that disambiguates
// events within the same millisecond and absorbs clock skew, with the
// source instance id as the deterministic final tie-break (S7-7, S7-8).
struct HybridTs {
std::int64_t wall_ms = 0;
std::uint32_t counter = 0;
std::string instance;
// -1 when this < other, 0 when equal, 1 when this > other.
int compare(const HybridTs& other) const;
bool operator<(const HybridTs& other) const;
bool operator==(const HybridTs& other) const;
std::string encode() const; // wall_ms.counter.instance
static bool decode(const std::string& text, HybridTs& out);
};
class HybridClock {
public:
HybridClock(std::string instance, common::Clock& clock);
HybridTs now();
void observe(const HybridTs& remote); // folds a remote ts into the clock
private:
std::string instance_;
common::Clock& clock_;
mutable std::mutex mutex_;
std::int64_t last_wall_ms_ = 0;
std::uint32_t counter_ = 0;
};
} // namespace molo::replication
@@ -0,0 +1,86 @@
#include "supervisor/replication/oplog_entry.hpp"
#include "supervisor/common/json.hpp"
namespace molo::replication {
std::string to_string(OplogOp op) {
return op == OplogOp::Upsert ? "upsert" : "delete";
}
bool parse_oplog_op(const std::string& text, OplogOp& out) {
if (text == "upsert") {
out = OplogOp::Upsert;
} else if (text == "delete") {
out = OplogOp::Delete;
} else {
return false;
}
return true;
}
std::string OplogEntry::encode() const {
using common::JValue;
JValue::Object obj;
obj["seq"] = JValue(static_cast<long long>(seq));
obj["instance"] = JValue(instance);
obj["ts"] = JValue(ts.encode());
obj["model"] = JValue(model);
obj["key"] = JValue(key);
obj["op"] = JValue(to_string(op));
obj["version"] = JValue(version);
JValue data_value;
if (!common::json_decode(data, data_value)) {
data_value = JValue(data);
}
obj["data"] = data_value;
return common::json_encode(JValue(obj));
}
bool OplogEntry::decode(const std::string& text, OplogEntry& out) {
using common::JValue;
JValue root;
if (!common::json_decode(text, root) || !root.is_object()) {
return false;
}
const JValue* instance = root.find("instance");
const JValue* ts = root.find("ts");
const JValue* model = root.find("model");
const JValue* key = root.find("key");
const JValue* op = root.find("op");
if (instance == nullptr || ts == nullptr || model == nullptr || key == nullptr ||
op == nullptr || !instance->is_string() || !ts->is_string() || !model->is_string() ||
!key->is_string() || !op->is_string()) {
return false;
}
OplogEntry entry;
const JValue* seq = root.find("seq");
if (seq != nullptr) {
entry.seq = static_cast<std::uint64_t>(seq->as_int(0));
}
entry.instance = instance->as_string();
if (!HybridTs::decode(ts->as_string(), entry.ts)) {
return false;
}
entry.model = model->as_string();
entry.key = key->as_string();
if (!parse_oplog_op(op->as_string(), entry.op)) {
return false;
}
const JValue* version = root.find("version");
if (version != nullptr) {
entry.version = version->as_int(0);
}
const JValue* data = root.find("data");
if (data != nullptr) {
entry.data = common::json_encode(*data);
}
out = std::move(entry);
return true;
}
std::size_t OplogEntry::approx_bytes() const {
return instance.size() + model.size() + key.size() + data.size() + 64;
}
} // namespace molo::replication
@@ -0,0 +1,62 @@
#pragma once
#include <cstdint>
#include <stdexcept>
#include <string>
#include "supervisor/replication/hybrid_clock.hpp"
namespace molo::replication {
class ReplicationError : public std::runtime_error {
public:
explicit ReplicationError(const std::string& message) : std::runtime_error(message) {}
};
// Raised to the loser of an optimistic-locking conflict, carrying the
// winner version so the caller can re-read and retry (S7-8).
class ConflictError : public ReplicationError {
public:
ConflictError(const std::string& model, const std::string& key, long long winner_version)
: ReplicationError("conflict:" + model + ":" + key),
model_(model),
key_(key),
winner_version_(winner_version) {}
const std::string& model() const { return model_; }
const std::string& key() const { return key_; }
long long winner_version() const { return winner_version_; }
private:
std::string model_;
std::string key_;
long long winner_version_;
};
enum class OplogOp {
Upsert,
Delete,
};
std::string to_string(OplogOp op);
bool parse_oplog_op(const std::string& text, OplogOp& out);
// One replication log entry (S7-7):
// {"seq":n,"instance":"id","ts":"hybrid-clock","model":"name","key":"pk",
// "op":"upsert|delete","version":n,"data":{}}
struct OplogEntry {
std::uint64_t seq = 0;
std::string instance;
HybridTs ts;
std::string model;
std::string key;
OplogOp op = OplogOp::Upsert;
long long version = 0;
std::string data = "{}"; // opaque JSON object, owned by the model layer
std::string encode() const;
static bool decode(const std::string& text, OplogEntry& out);
std::size_t approx_bytes() const;
};
} // namespace molo::replication
@@ -0,0 +1,47 @@
#include "supervisor/replication/oplog_reader.hpp"
namespace molo::replication {
OplogReader::OplogReader(const std::filesystem::path& dir)
: log_(dir / "oplog.log", false, bus::SegmentLog::OpenMode::ReadOnly) {}
std::vector<OplogEntry> OplogReader::read_from(std::uint64_t seq, std::size_t max_entries,
std::size_t max_bytes) const {
log_.refresh();
std::vector<OplogEntry> out;
for (const auto& record : log_.read_from(seq, max_entries, max_bytes)) {
OplogEntry entry;
if (OplogEntry::decode(record.data, entry)) {
out.push_back(std::move(entry));
}
// Corrupt entries are skipped: the segment checksum already truncates
// torn tails, so anything left but undecodable is a loud gap for the
// operator, surfaced by the missing seq in the consumer.
}
return out;
}
std::vector<OplogEntry> OplogReader::poll(std::size_t max_entries, std::size_t max_bytes) {
std::lock_guard<std::mutex> lock(mutex_);
auto entries = read_from(position_, max_entries, max_bytes);
if (!entries.empty()) {
position_ = entries.back().seq + 1;
}
return entries;
}
std::uint64_t OplogReader::position() const {
std::lock_guard<std::mutex> lock(mutex_);
return position_;
}
void OplogReader::set_position(std::uint64_t seq) {
std::lock_guard<std::mutex> lock(mutex_);
position_ = seq == 0 ? 1 : seq;
}
std::uint64_t OplogReader::last_seq() const {
return log_.last_seq();
}
} // namespace molo::replication
@@ -0,0 +1,36 @@
#pragma once
#include <cstdint>
#include <filesystem>
#include <mutex>
#include <vector>
#include "supervisor/bus/segment_log.hpp"
#include "supervisor/replication/oplog_entry.hpp"
namespace molo::replication {
// Tails one instance's persisted operation log: catch-up pull on reconnect
// reads from a kept position, and poll() advances it (S7-7).
class OplogReader {
public:
explicit OplogReader(const std::filesystem::path& dir);
// Reads up to max_entries / max_bytes from seq (1-based), decoded.
std::vector<OplogEntry> read_from(std::uint64_t seq, std::size_t max_entries,
std::size_t max_bytes) const;
// Polls ahead of the kept position and advances it past what was read.
std::vector<OplogEntry> poll(std::size_t max_entries, std::size_t max_bytes);
std::uint64_t position() const;
void set_position(std::uint64_t seq);
std::uint64_t last_seq() const;
private:
mutable bus::SegmentLog log_;
mutable std::mutex mutex_;
std::uint64_t position_ = 1;
};
} // namespace molo::replication
@@ -0,0 +1,46 @@
#include "supervisor/replication/oplog_writer.hpp"
namespace molo::replication {
OplogWriter::OplogWriter(const std::filesystem::path& dir, std::string instance,
common::Clock& clock, std::shared_ptr<common::Observer> observer)
: instance_(std::move(instance)),
clock_(clock),
observer_(std::move(observer)),
hybrid_(instance_, clock_),
log_(std::make_unique<bus::SegmentLog>(dir / "oplog.log")),
next_seq_(log_->last_seq() + 1) {}
OplogEntry OplogWriter::append(const std::string& model, const std::string& key, OplogOp op,
long long version, const std::string& data) {
if (model.empty() || model.size() > 256 || key.empty() || key.size() > 1024) {
throw ReplicationError("OplogWriter: bad model or key");
}
if (data.size() > 1048576) {
throw ReplicationError("OplogWriter: data over 1 MiB cap");
}
std::lock_guard<std::mutex> lock(mutex_);
OplogEntry entry;
entry.seq = next_seq_++;
entry.instance = instance_;
entry.ts = hybrid_.now();
entry.model = model;
entry.key = key;
entry.op = op;
entry.version = version;
entry.data = data;
log_->append(entry.encode(), clock_.wall_ms());
observer_->counter("molo_replication_oplog_appended_total", 1);
return entry;
}
std::uint64_t OplogWriter::last_seq() const {
std::lock_guard<std::mutex> lock(mutex_);
return next_seq_ == 0 ? 0 : next_seq_ - 1;
}
const std::string& OplogWriter::instance() const {
return instance_;
}
} // namespace molo::replication
@@ -0,0 +1,40 @@
#pragma once
#include <cstdint>
#include <filesystem>
#include <memory>
#include <mutex>
#include <string>
#include "supervisor/bus/segment_log.hpp"
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/replication/hybrid_clock.hpp"
#include "supervisor/replication/oplog_entry.hpp"
namespace molo::replication {
// Appends to one instance's persisted operation log (S7-7). Sequences are
// assigned here, strictly increasing and stable across restarts.
class OplogWriter {
public:
OplogWriter(const std::filesystem::path& dir, std::string instance, common::Clock& clock,
std::shared_ptr<common::Observer> observer);
OplogEntry append(const std::string& model, const std::string& key, OplogOp op,
long long version, const std::string& data);
std::uint64_t last_seq() const;
const std::string& instance() const;
private:
std::string instance_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
HybridClock hybrid_;
std::unique_ptr<bus::SegmentLog> log_;
mutable std::mutex mutex_;
std::uint64_t next_seq_;
};
} // namespace molo::replication
+122
View File
@@ -0,0 +1,122 @@
#include "supervisor/replication/replicator.hpp"
namespace molo::replication {
Replicator::Replicator(const std::filesystem::path& dir, std::string instance,
common::Clock& clock, std::shared_ptr<common::Observer> observer)
: instance_(std::move(instance)),
clock_(clock),
observer_(std::move(observer)),
writer_(dir, instance_, clock_, observer_),
reader_(dir),
resolver_(clock_, observer_) {
replay();
}
const std::string& Replicator::instance() const {
return instance_;
}
void Replicator::declare_model(const std::string& model, ConflictPolicy policy,
ConflictHook hook) {
resolver_.declare(model, policy, std::move(hook));
}
void Replicator::replay() {
// Rebuild current state plus the version vector from the persisted log.
// Replay applies entries in seq order without conflict checks: the log
// already holds resolved history.
std::uint64_t seq = 1;
while (true) {
auto entries = reader_.read_from(seq, 512, 4 * 1024 * 1024);
if (entries.empty()) {
break;
}
for (const auto& entry : entries) {
current_[{entry.model, entry.key}] = entry;
versions_.update(entry.instance, entry.seq);
}
seq = entries.back().seq + 1;
}
observer_->event("replication.replayed", "keys=" + std::to_string(current_.size()));
}
OplogEntry Replicator::local_write(const std::string& model, const std::string& key, OplogOp op,
long long version, const std::string& data) {
OplogEntry entry = writer_.append(model, key, op, version, data);
versions_.update(instance_, entry.seq);
{
std::lock_guard<std::mutex> lock(mutex_);
current_[{model, key}] = entry;
}
observer_->event("replication.local_write", model + ":" + key);
return entry;
}
ConflictOutcome Replicator::apply_remote(const OplogEntry& entry) {
if (entry.instance == instance_) {
// Own entries echoed back by the bus are already current.
ConflictOutcome outcome;
outcome.winner = entry;
return outcome;
}
std::optional<OplogEntry> local;
{
std::lock_guard<std::mutex> lock(mutex_);
auto it = current_.find({entry.model, entry.key});
if (it != current_.end()) {
local = it->second;
}
}
ConflictOutcome outcome;
if (!local.has_value()) {
outcome.winner = entry;
outcome.remote_won = true;
outcome.detail = "no local value, remote adopted";
std::lock_guard<std::mutex> lock(mutex_);
current_[{entry.model, entry.key}] = entry;
} else if (local->instance == entry.instance && local->seq == entry.seq) {
outcome.winner = *local; // exact duplicate delivery, idempotent
outcome.detail = "duplicate delivery ignored";
} else {
outcome = resolver_.resolve(*local, entry);
std::lock_guard<std::mutex> lock(mutex_);
current_[{entry.model, entry.key}] = outcome.winner;
}
versions_.update(entry.instance, entry.seq);
observer_->counter("molo_replication_applied_total", 1);
if (outcome.conflict_error) {
throw ConflictError(entry.model, entry.key, outcome.winner_version);
}
return outcome;
}
std::optional<OplogEntry> Replicator::current(const std::string& model,
const std::string& key) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = current_.find({model, key});
if (it == current_.end()) {
return std::nullopt;
}
return it->second;
}
std::vector<OplogEntry> Replicator::entries_since(std::uint64_t seq, std::size_t max_entries,
std::size_t max_bytes) const {
return reader_.read_from(seq, max_entries, max_bytes);
}
void Replicator::versions(VersionVector& out) const {
// Snapshot via encode/decode to avoid exposing internals.
VersionVector::decode(versions_.encode(), out);
}
ConflictResolver& Replicator::resolver() {
return resolver_;
}
std::uint64_t Replicator::last_seq() const {
return writer_.last_seq();
}
} // namespace molo::replication
+73
View File
@@ -0,0 +1,73 @@
#pragma once
#include <cstdint>
#include <filesystem>
#include <map>
#include <memory>
#include <mutex>
#include <optional>
#include <string>
#include <utility>
#include <vector>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/replication/conflict_resolver.hpp"
#include "supervisor/replication/oplog_entry.hpp"
#include "supervisor/replication/oplog_reader.hpp"
#include "supervisor/replication/oplog_writer.hpp"
#include "supervisor/replication/version_vector.hpp"
namespace molo::replication {
// Per-instance replication endpoint (E7-7): owns the local oplog, the
// current per-key state, per-model conflict policies, and the version
// vector. Local writes append to the oplog and become current; remote
// entries arrive through apply_remote() and resolve through the declared
// policy. Replays the persisted oplog on construction so state survives
// restarts. Thread-safe.
class Replicator {
public:
Replicator(const std::filesystem::path& dir, std::string instance, common::Clock& clock,
std::shared_ptr<common::Observer> observer);
const std::string& instance() const;
void declare_model(const std::string& model, ConflictPolicy policy,
ConflictHook hook = ConflictHook{});
// Local write path. The caller supplies the model version (ORM row
// version for optimistic models); the hybrid timestamp is assigned here.
OplogEntry local_write(const std::string& model, const std::string& key, OplogOp op,
long long version, const std::string& data);
// Remote apply path. Returns the resolution; throws ConflictError to
// signal the loser of an optimistic conflict (S7-8).
ConflictOutcome apply_remote(const OplogEntry& entry);
std::optional<OplogEntry> current(const std::string& model, const std::string& key) const;
// Entries strictly after seq for catch-up pull, bounded by the sync
// batch caps (500 entries or 1 MiB per round, S7-7).
std::vector<OplogEntry> entries_since(std::uint64_t seq, std::size_t max_entries,
std::size_t max_bytes) const;
void versions(VersionVector& out) const; // snapshot of the version vector
ConflictResolver& resolver();
std::uint64_t last_seq() const;
private:
void replay();
std::string instance_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
OplogWriter writer_;
OplogReader reader_;
ConflictResolver resolver_;
VersionVector versions_;
mutable std::mutex mutex_;
std::map<std::pair<std::string, std::string>, OplogEntry> current_;
};
} // namespace molo::replication
@@ -0,0 +1,73 @@
#include "supervisor/replication/sync_session.hpp"
#include "supervisor/common/caps.hpp"
namespace molo::replication {
SyncSession::SyncSession(Replicator& local, SyncPeer& peer,
std::shared_ptr<common::Observer> observer, std::size_t batch_entries,
std::size_t batch_bytes)
: local_(local),
peer_(peer),
observer_(std::move(observer)),
batch_entries_(batch_entries),
batch_bytes_(batch_bytes) {
if (batch_entries_ == 0 || batch_entries_ > common::Caps::kSyncBatchEntries) {
batch_entries_ = common::Caps::kSyncBatchEntries;
}
if (batch_bytes_ == 0 || batch_bytes_ > common::Caps::kSyncBatchBytes) {
batch_bytes_ = common::Caps::kSyncBatchBytes;
}
}
SyncStats SyncSession::round_trip() {
SyncStats stats;
// Pull: ask the peer for everything past our version vector.
VersionVector have;
local_.versions(have);
std::vector<OplogEntry> incoming = peer_.fetch_since(have, batch_entries_, batch_bytes_);
for (const auto& entry : incoming) {
try {
ConflictOutcome outcome = local_.apply_remote(entry);
++stats.pulled;
if (outcome.conflict) {
++stats.conflicts;
}
} catch (const ConflictError&) {
// Optimistic loser: counted as a conflict, sync continues with the
// next entry. The ConflictError propagates to the sync caller only
// in the sense of the counter plus event; entries are independent.
++stats.pulled;
++stats.conflicts;
observer_->event("replication.sync_conflict", entry.model + ":" + entry.key);
}
}
// Push: send our own entries past what the peer has seen from us,
// bounded to one batch per round.
VersionVector peer_have;
peer_.peer_versions(peer_have);
const std::uint64_t peer_has = peer_have.get(local_.instance());
std::vector<OplogEntry> outgoing =
local_.entries_since(peer_has + 1, batch_entries_, batch_bytes_);
if (!outgoing.empty()) {
peer_.submit(outgoing);
stats.pushed = outgoing.size();
}
// Lag: entries behind the peer on the pull side, exact.
VersionVector now_have;
local_.versions(now_have);
const std::uint64_t peer_last = peer_.last_seq();
const std::uint64_t seen = now_have.get(peer_.instance());
stats.lag = peer_last > seen ? static_cast<std::int64_t>(peer_last - seen) : 0;
observer_->gauge("molo_replication_sync_lag", static_cast<double>(stats.lag));
if (stats.lag >= common::Caps::kSyncLagAlertThreshold) {
observer_->event("replication.lag_alert", peer_.instance());
}
observer_->counter("molo_replication_synced_total", 1);
return stats;
}
} // namespace molo::replication
@@ -0,0 +1,63 @@
#pragma once
#include <cstddef>
#include <cstdint>
#include <memory>
#include <string>
#include <vector>
#include "supervisor/common/observer.hpp"
#include "supervisor/replication/oplog_entry.hpp"
#include "supervisor/replication/replicator.hpp"
#include "supervisor/replication/version_vector.hpp"
namespace molo::replication {
// Transport-agnostic peer accessor. Production binds this to the bus
// (push) plus catch-up pull over the same channel (S7-7); tests bind it
// to another in-process Replicator.
class SyncPeer {
public:
virtual ~SyncPeer() = default;
// Entries the peer has that the given version vector has not seen,
// bounded by the caller's batch caps.
virtual std::vector<OplogEntry> fetch_since(const VersionVector& have, std::size_t max_entries,
std::size_t max_bytes) = 0;
// Pushes local entries to the peer. The peer applies each through its
// own resolver; duplicates are idempotent and ignored.
virtual void submit(const std::vector<OplogEntry>& entries) = 0;
virtual void peer_versions(VersionVector& out) const = 0;
virtual std::uint64_t last_seq() const = 0;
virtual std::string instance() const = 0;
};
struct SyncStats {
std::size_t pulled = 0;
std::size_t pushed = 0;
std::size_t conflicts = 0;
std::int64_t lag = 0; // entries behind after this round
};
// One bidirectional sync round between the local Replicator and a peer:
// pull what the peer has, push what it lacks (S7-7). Batch caps are 500
// entries or 1 MiB per round; lag is exported as a gauge with the 10000
// alert threshold from S7-7.
class SyncSession {
public:
SyncSession(Replicator& local, SyncPeer& peer, std::shared_ptr<common::Observer> observer,
std::size_t batch_entries = 500, std::size_t batch_bytes = 1048576);
SyncStats round_trip();
private:
Replicator& local_;
SyncPeer& peer_;
std::shared_ptr<common::Observer> observer_;
std::size_t batch_entries_;
std::size_t batch_bytes_;
};
} // namespace molo::replication
@@ -0,0 +1,96 @@
#include "supervisor/replication/version_vector.hpp"
#include "supervisor/common/json.hpp"
namespace molo::replication {
void VersionVector::update(const std::string& instance, std::uint64_t seq) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = versions_.find(instance);
if (it == versions_.end() || seq > it->second) {
versions_[instance] = seq;
}
}
std::uint64_t VersionVector::get(const std::string& instance) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = versions_.find(instance);
return it == versions_.end() ? 0 : it->second;
}
void VersionVector::merge(const VersionVector& other) {
if (this == &other) {
return;
}
std::lock_guard<std::mutex> lock(mutex_);
std::lock_guard<std::mutex> other_lock(other.mutex_);
for (const auto& [instance, seq] : other.versions_) {
auto it = versions_.find(instance);
if (it == versions_.end() || seq > it->second) {
versions_[instance] = seq;
}
}
}
VectorOrder VersionVector::compare(const VersionVector& other) const {
if (this == &other) {
return VectorOrder::Equal;
}
std::lock_guard<std::mutex> lock(mutex_);
std::lock_guard<std::mutex> other_lock(other.mutex_);
bool less = false, greater = false;
for (const auto& [instance, seq] : versions_) {
auto it = other.versions_.find(instance);
const std::uint64_t oseq = it == other.versions_.end() ? 0 : it->second;
if (seq < oseq) {
less = true;
} else if (seq > oseq) {
greater = true;
}
}
for (const auto& [instance, seq] : other.versions_) {
if (versions_.find(instance) == versions_.end() && seq > 0) {
less = true;
}
}
if (less && greater) {
return VectorOrder::Concurrent;
}
if (less) {
return VectorOrder::Before;
}
if (greater) {
return VectorOrder::After;
}
return VectorOrder::Equal;
}
std::string VersionVector::encode() const {
using common::JValue;
std::lock_guard<std::mutex> lock(mutex_);
JValue::Object obj;
for (const auto& [instance, seq] : versions_) {
obj[instance] = JValue(static_cast<long long>(seq));
}
return common::json_encode(JValue(obj));
}
bool VersionVector::decode(const std::string& text, VersionVector& out) {
using common::JValue;
JValue root;
if (!common::json_decode(text, root) || !root.is_object()) {
return false;
}
VersionVector parsed;
for (const auto& [instance, value] : root.as_object()) {
if (!value.is_int() && !value.is_double()) {
return false;
}
parsed.versions_[instance] = static_cast<std::uint64_t>(value.as_int(0));
}
std::lock_guard<std::mutex> lock(out.mutex_);
out.versions_ = std::move(parsed.versions_);
return true;
}
} // namespace molo::replication
@@ -0,0 +1,34 @@
#pragma once
#include <cstdint>
#include <map>
#include <mutex>
#include <string>
namespace molo::replication {
enum class VectorOrder {
Equal,
Before, // this <= other with at least one strict <
After, // this >= other with at least one strict >
Concurrent, // neither dominates: a real conflict happened
};
// Version vector over per-instance oplog sequences, exchanged at sync so
// each side fetches only what the other has not seen (S7-7).
class VersionVector {
public:
void update(const std::string& instance, std::uint64_t seq);
std::uint64_t get(const std::string& instance) const;
void merge(const VersionVector& other);
VectorOrder compare(const VersionVector& other) const;
std::string encode() const; // JSON object {instance: seq}
static bool decode(const std::string& text, VersionVector& out);
private:
mutable std::mutex mutex_;
std::map<std::string, std::uint64_t> versions_;
};
} // namespace molo::replication
+78
View File
@@ -0,0 +1,78 @@
# Part 07 (supervisor, tasks, bus, replication) build fragment.
# Included by the root Makefile. Double-colon rules let every part append
# its own all/test/clean steps without editing shared targets.
SUP_CXX ?= g++
SUP_CXXFLAGS ?= -std=c++23 -O2 -Wall -Wextra -Wpedantic -Werror -MMD -MP
SUP_ROOT = src/supervisor
SUP_BUILD = build/supervisor
SUP_LIB = $(SUP_BUILD)/libsupervisor.a
SUP_INCPATH = -Isrc
SUP_SRCS = \
$(SUP_ROOT)/common/clock.cpp \
$(SUP_ROOT)/common/observer.cpp \
$(SUP_ROOT)/common/kv_store.cpp \
$(SUP_ROOT)/common/json.cpp \
$(SUP_ROOT)/bus/segment_log.cpp \
$(SUP_ROOT)/bus/topic_bus.cpp \
$(SUP_ROOT)/bus/consumer_cursor.cpp \
$(SUP_ROOT)/bus/ack_tracker.cpp \
$(SUP_ROOT)/bus/stream_bus.cpp \
$(SUP_ROOT)/tasks/task_envelope.cpp \
$(SUP_ROOT)/tasks/task_handler.cpp \
$(SUP_ROOT)/tasks/task_registry.cpp \
$(SUP_ROOT)/tasks/retry_policy.cpp \
$(SUP_ROOT)/tasks/dead_letter_queue.cpp \
$(SUP_ROOT)/tasks/task_queue.cpp \
$(SUP_ROOT)/tasks/cron.cpp \
$(SUP_ROOT)/tasks/scheduler.cpp \
$(SUP_ROOT)/supervisor/restart_policy.cpp \
$(SUP_ROOT)/supervisor/worker_process.cpp \
$(SUP_ROOT)/supervisor/health_check.cpp \
$(SUP_ROOT)/supervisor/worker_group.cpp \
$(SUP_ROOT)/supervisor/shutdown_coordinator.cpp \
$(SUP_ROOT)/supervisor/supervisor.cpp \
$(SUP_ROOT)/replication/hybrid_clock.cpp \
$(SUP_ROOT)/replication/oplog_entry.cpp \
$(SUP_ROOT)/replication/oplog_writer.cpp \
$(SUP_ROOT)/replication/oplog_reader.cpp \
$(SUP_ROOT)/replication/version_vector.cpp \
$(SUP_ROOT)/replication/conflict_resolver.cpp \
$(SUP_ROOT)/replication/replicator.cpp \
$(SUP_ROOT)/replication/sync_session.cpp
SUP_OBJS = $(SUP_SRCS:$(SUP_ROOT)/%.cpp=$(SUP_BUILD)/%.o)
SUP_TESTS = \
test_supervisor \
test_tasks \
test_scheduler \
test_bus \
test_durability \
test_replication \
test_multi_instance
SUP_TEST_BINS = $(SUP_TESTS:%=$(SUP_BUILD)/%)
all:: $(SUP_LIB)
$(SUP_LIB): $(SUP_OBJS)
@mkdir -p $(SUP_BUILD)
ar rcs $@ $^
$(SUP_BUILD)/%.o: $(SUP_ROOT)/%.cpp
@mkdir -p $(dir $@)
$(SUP_CXX) $(SUP_CXXFLAGS) $(SUP_INCPATH) -c $< -o $@
$(SUP_BUILD)/test_%: tests/src/supervisor/test_%.cpp $(SUP_LIB) $(OPERABILITY_LIB)
@mkdir -p $(SUP_BUILD)
$(SUP_CXX) $(SUP_CXXFLAGS) $(SUP_INCPATH) $< $(SUP_LIB) $(OPERABILITY_LIB) -o $@
test:: $(SUP_TEST_BINS)
@for t in $(SUP_TEST_BINS); do echo "== $$t"; ./$$t || exit 1; done
clean::
rm -rf $(SUP_BUILD)
-include $(SUP_OBJS:.o=.d)
+150
View File
@@ -0,0 +1,150 @@
#include "supervisor/supervisor/health_check.hpp"
#if !defined(_WIN32)
#include <fcntl.h>
#include <netdb.h>
#include <sys/select.h>
#include <sys/socket.h>
#include <unistd.h>
#endif
namespace molo::supervisor {
HealthCheck::HealthCheck(std::string name, const HealthSpec& spec,
std::function<bool()> func_probe, common::Clock& clock,
std::shared_ptr<common::Observer> observer)
: name_(std::move(name)),
spec_(spec),
func_probe_(std::move(func_probe)),
clock_(clock),
observer_(std::move(observer)) {
if (spec_.interval_ms < 100) {
spec_.interval_ms = 100;
}
if (spec_.max_failures < 1) {
spec_.max_failures = 1;
}
}
bool HealthCheck::enabled() const {
if (spec_.kind == ProbeKind::Function) {
return static_cast<bool>(func_probe_);
}
return spec_.tcp_port > 0;
}
bool HealthCheck::due() const {
std::lock_guard<std::mutex> lock(mutex_);
if (!enabled()) {
return false;
}
return clock_.steady_ms() - last_probe_steady_ms_ >= spec_.interval_ms;
}
bool HealthCheck::check_now() {
bool healthy = true;
if (spec_.kind == ProbeKind::Function) {
if (!func_probe_) {
return true;
}
try {
healthy = func_probe_();
} catch (...) {
healthy = false;
}
} else {
std::string error;
healthy = tcp_probe(spec_.tcp_host, spec_.tcp_port, spec_.timeout_ms, error);
}
std::lock_guard<std::mutex> lock(mutex_);
last_probe_steady_ms_ = clock_.steady_ms();
if (healthy) {
failures_ = 0;
} else {
++failures_;
observer_->counter("molo_supervisor_health_failures_total", 1);
if (failures_ >= spec_.max_failures) {
observer_->event("supervisor.worker.unhealthy", name_);
}
}
return healthy;
}
bool HealthCheck::unhealthy() const {
std::lock_guard<std::mutex> lock(mutex_);
return enabled() && failures_ >= spec_.max_failures;
}
int HealthCheck::failures() const {
std::lock_guard<std::mutex> lock(mutex_);
return failures_;
}
void HealthCheck::reset() {
std::lock_guard<std::mutex> lock(mutex_);
failures_ = 0;
last_probe_steady_ms_ = 0;
}
bool HealthCheck::tcp_probe(const std::string& host, int port, long timeout_ms,
std::string& error) {
#if defined(_WIN32)
error = "tcp probe unsupported on this platform";
(void)host;
(void)port;
(void)timeout_ms;
return false;
#else
addrinfo hints{};
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
addrinfo* list = nullptr;
if (getaddrinfo(host.c_str(), std::to_string(port).c_str(), &hints, &list) != 0) {
error = "dns failure";
return false;
}
int fd = -1;
for (addrinfo* ai = list; ai != nullptr; ai = ai->ai_next) {
fd = ::socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
if (fd >= 0) {
break;
}
}
if (fd < 0) {
freeaddrinfo(list);
error = "socket failure";
return false;
}
const int flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
int rc = ::connect(fd, list->ai_addr, list->ai_addrlen);
bool ok = (rc == 0);
if (!ok && errno == EINPROGRESS) {
fd_set wfds;
FD_ZERO(&wfds);
FD_SET(fd, &wfds);
timeval tv{};
tv.tv_sec = timeout_ms / 1000;
tv.tv_usec = (timeout_ms % 1000) * 1000;
rc = select(fd + 1, nullptr, &wfds, nullptr, &tv);
if (rc > 0) {
int so_error = 0;
socklen_t len = sizeof(so_error);
getsockopt(fd, SOL_SOCKET, SO_ERROR, &so_error, &len);
ok = (so_error == 0);
if (!ok) {
error = "connect refused";
}
} else {
error = "connect timeout";
}
} else if (!ok) {
error = "connect failure";
}
freeaddrinfo(list);
::close(fd);
return ok;
#endif
}
} // namespace molo::supervisor
@@ -0,0 +1,58 @@
#pragma once
#include <cstdint>
#include <functional>
#include <memory>
#include <mutex>
#include <string>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
namespace molo::supervisor {
enum class ProbeKind {
Function, // caller-supplied predicate
Tcp, // TCP connect probe (S7-4)
};
struct HealthSpec {
ProbeKind kind = ProbeKind::Function;
std::string tcp_host = "127.0.0.1";
int tcp_port = 0;
long interval_ms = 10000;
int max_failures = 3;
long timeout_ms = 2000;
};
// Health checker for one supervised worker (S7-4). The group calls due()
// and check_now() from its monitor loop; 3 consecutive probe failures
// (default) mark the worker unhealthy and trigger the restart policy.
class HealthCheck {
public:
HealthCheck(std::string name, const HealthSpec& spec, std::function<bool()> func_probe,
common::Clock& clock, std::shared_ptr<common::Observer> observer);
bool enabled() const;
bool due() const; // true when a probe should run now
bool check_now(); // runs one probe, returns healthy
bool unhealthy() const; // true after max_failures consecutive failures
int failures() const;
void reset();
// Raw TCP connect probe used for ProbeKind::Tcp.
static bool tcp_probe(const std::string& host, int port, long timeout_ms,
std::string& error);
private:
std::string name_;
HealthSpec spec_;
std::function<bool()> func_probe_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
mutable std::mutex mutex_;
int failures_ = 0;
std::int64_t last_probe_steady_ms_ = 0;
};
} // namespace molo::supervisor
@@ -0,0 +1,38 @@
#include "supervisor/supervisor/restart_policy.hpp"
#include "supervisor/common/caps.hpp"
namespace molo::supervisor {
std::string to_string(RestartPolicy policy) {
switch (policy) {
case RestartPolicy::Never: return "never";
case RestartPolicy::OnFailure: return "on-failure";
case RestartPolicy::Always: return "always";
}
return "on-failure";
}
bool parse_restart_policy(const std::string& text, RestartPolicy& out) {
if (text == "never") {
out = RestartPolicy::Never;
} else if (text == "on-failure") {
out = RestartPolicy::OnFailure;
} else if (text == "always") {
out = RestartPolicy::Always;
} else {
return false;
}
return true;
}
void RestartSpec::normalize() {
if (max_restarts < 0) {
max_restarts = common::Caps::kMaxRestartsDefault;
}
if (window_secs < 1) {
window_secs = common::Caps::kRestartWindowSecsDefault;
}
}
} // namespace molo::supervisor
@@ -0,0 +1,28 @@
#pragma once
#include <string>
namespace molo::supervisor {
// Restart policy per supervised program (BP-2, R7-3).
enum class RestartPolicy {
Never, // leave exited children alone
OnFailure, // restart only on nonzero exit or crash
Always, // restart on any exit
};
std::string to_string(RestartPolicy policy);
bool parse_restart_policy(const std::string& text, RestartPolicy& out);
// RestartPolicy plus the intensity cap that makes restart loops bounded:
// at most max_restarts restarts per window_secs window. Crossing the cap
// moves the child to Fatal and emits an event (S7-4).
struct RestartSpec {
RestartPolicy policy = RestartPolicy::OnFailure;
int max_restarts = 5;
int window_secs = 60;
void normalize(); // clamps nonsense to the spec defaults
};
} // namespace molo::supervisor
@@ -0,0 +1,50 @@
#include "supervisor/supervisor/shutdown_coordinator.hpp"
namespace molo::supervisor {
std::string to_string(ShutdownPhase phase) {
switch (phase) {
case ShutdownPhase::Running: return "running";
case ShutdownPhase::Draining: return "draining";
case ShutdownPhase::Stopped: return "stopped";
}
return "running";
}
ShutdownCoordinator::ShutdownCoordinator(common::Clock& clock,
std::shared_ptr<common::Observer> observer)
: clock_(clock), observer_(std::move(observer)) {}
ShutdownPhase ShutdownCoordinator::phase() const {
std::lock_guard<std::mutex> lock(mutex_);
return phase_;
}
bool ShutdownCoordinator::drain_until(const std::function<bool()>& is_drained, long deadline_ms,
long poll_ms) {
{
std::lock_guard<std::mutex> lock(mutex_);
phase_ = ShutdownPhase::Draining;
}
observer_->event("supervisor.shutdown.draining", "deadline_ms=" + std::to_string(deadline_ms));
const std::int64_t deadline = clock_.steady_ms() + deadline_ms;
while (true) {
if (is_drained && is_drained()) {
observer_->event("supervisor.shutdown.drained", "clean");
return true;
}
if (clock_.steady_ms() >= deadline) {
observer_->event("supervisor.shutdown.drain_expired", "deadline hit");
return false;
}
clock_.sleep_ms(poll_ms);
}
}
void ShutdownCoordinator::mark_stopped() {
std::lock_guard<std::mutex> lock(mutex_);
phase_ = ShutdownPhase::Stopped;
observer_->event("supervisor.shutdown.stopped", "");
}
} // namespace molo::supervisor
@@ -0,0 +1,40 @@
#pragma once
#include <functional>
#include <memory>
#include <mutex>
#include <string>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
namespace molo::supervisor {
enum class ShutdownPhase {
Running,
Draining,
Stopped,
};
std::string to_string(ShutdownPhase phase);
// Graceful shutdown with a deadline (R7-9): stop accepting new work, drain
// in-flight work until the deadline, then stop. drain_until() polls the
// caller-supplied predicate and reports whether the drain completed.
class ShutdownCoordinator {
public:
ShutdownCoordinator(common::Clock& clock, std::shared_ptr<common::Observer> observer);
ShutdownPhase phase() const;
bool drain_until(const std::function<bool()>& is_drained, long deadline_ms,
long poll_ms = 50);
void mark_stopped();
private:
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
mutable std::mutex mutex_;
ShutdownPhase phase_ = ShutdownPhase::Running;
};
} // namespace molo::supervisor
+153
View File
@@ -0,0 +1,153 @@
#include "supervisor/supervisor/supervisor.hpp"
#include "supervisor/common/caps.hpp"
#include "supervisor/common/json.hpp"
#include "supervisor/supervisor/supervisor_error.hpp"
namespace molo::supervisor {
Supervisor::Supervisor(common::Clock& clock, std::shared_ptr<common::Observer> observer,
const SupervisorOptions& options)
: clock_(clock), observer_(std::move(observer)), options_(options), shutdown_(clock, observer_) {
}
Supervisor::~Supervisor() {
if (running()) {
shutdown(0);
}
}
std::size_t Supervisor::add_group(const GroupSpec& spec, WorkFunc func) {
std::lock_guard<std::mutex> lock(mutex_);
if (static_cast<int>(groups_.size()) >= common::Caps::kGroupsPerSupervisor) {
throw SupervisorError("Supervisor: group cap reached");
}
auto group = std::make_unique<WorkerGroup>(spec, std::move(func), clock_, observer_);
if (running_) {
group->start();
}
groups_.push_back(std::move(group));
observer_->event("supervisor.group.added", spec.name);
return groups_.size() - 1;
}
void Supervisor::set_task_queue(std::shared_ptr<tasks::TaskQueue> queue) {
std::lock_guard<std::mutex> lock(mutex_);
queue_ = std::move(queue);
}
void Supervisor::start_all() {
std::lock_guard<std::mutex> lock(mutex_);
if (running_) {
return;
}
running_ = true;
for (auto& group : groups_) {
group->start();
}
observer_->event("supervisor.started", "groups=" + std::to_string(groups_.size()));
}
void Supervisor::shutdown(long drain_deadline_ms) {
const long deadline = drain_deadline_ms < 0 ? options_.default_drain_ms : drain_deadline_ms;
std::shared_ptr<tasks::TaskQueue> queue;
{
std::lock_guard<std::mutex> lock(mutex_);
if (!running_) {
return;
}
queue = queue_;
}
if (queue) {
queue->set_accepting(false);
const bool drained = shutdown_.drain_until(
[&] { return queue->in_flight() == 0; }, deadline);
if (!drained) {
queue->abandon_all_to_dlq("shutdown-drain-expired");
}
queue->shutdown();
} else {
shutdown_.drain_until([] { return true; }, 0);
}
{
std::lock_guard<std::mutex> lock(mutex_);
for (auto it = groups_.rbegin(); it != groups_.rend(); ++it) {
(*it)->stop();
}
running_ = false;
}
shutdown_.mark_stopped();
}
void Supervisor::restart_group(std::size_t index) {
std::lock_guard<std::mutex> lock(mutex_);
if (index >= groups_.size()) {
throw SupervisorError("Supervisor: group index out of range");
}
groups_[index]->restart_all();
}
void Supervisor::scale_group(std::size_t index, std::size_t n) {
std::lock_guard<std::mutex> lock(mutex_);
if (index >= groups_.size()) {
throw SupervisorError("Supervisor: group index out of range");
}
groups_[index]->set_size(n);
}
std::string Supervisor::status_json() const {
using common::JValue;
std::lock_guard<std::mutex> lock(mutex_);
JValue::Object root;
root["phase"] = JValue(to_string(shutdown_.phase()));
root["running"] = JValue(running_);
JValue::Array groups;
for (const auto& group : groups_) {
groups.push_back(group->snapshot());
}
root["groups"] = JValue(groups);
if (queue_) {
const tasks::QueueStats stats = queue_->stats();
JValue::Object q;
q["queued"] = JValue(static_cast<long long>(stats.queued));
q["leased"] = JValue(static_cast<long long>(stats.leased));
q["running"] = JValue(static_cast<long long>(stats.running));
q["succeeded_total"] = JValue(static_cast<long long>(stats.succeeded_total));
q["failed_total"] = JValue(static_cast<long long>(stats.failed_total));
q["retried_total"] = JValue(static_cast<long long>(stats.retried_total));
q["dead_lettered"] = JValue(static_cast<long long>(queue_->dead_letters().size()));
q["accepting"] = JValue(queue_->accepting());
root["queue"] = JValue(q);
}
std::string full = common::json_encode(JValue(root));
if (full.size() <= common::Caps::kStatusSnapshotBytes) {
return full;
}
// Over the 1 MiB cap: compact form with counts only, never truncated JSON.
JValue::Object compact;
compact["truncated"] = JValue(true);
compact["phase"] = JValue(to_string(shutdown_.phase()));
compact["group_count"] = JValue(static_cast<long long>(groups_.size()));
long long workers = 0;
for (const auto& group : groups_) {
workers += static_cast<long long>(group->size());
}
compact["worker_count"] = JValue(workers);
return common::json_encode(JValue(compact));
}
std::size_t Supervisor::group_count() const {
std::lock_guard<std::mutex> lock(mutex_);
return groups_.size();
}
bool Supervisor::running() const {
std::lock_guard<std::mutex> lock(mutex_);
return running_;
}
ShutdownPhase Supervisor::phase() const {
return shutdown_.phase();
}
} // namespace molo::supervisor
+68
View File
@@ -0,0 +1,68 @@
#pragma once
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/supervisor/shutdown_coordinator.hpp"
#include "supervisor/supervisor/supervisor_error.hpp"
#include "supervisor/supervisor/worker_group.hpp"
#include "supervisor/tasks/task_queue.hpp"
namespace molo::supervisor {
struct SupervisorOptions {
long default_drain_ms = 30000;
};
// Top-level supervisor (E7-1): owns worker groups, applies restart
// strategies through them, and coordinates graceful shutdown with the
// task queue (R7-9). Exposes the status snapshot consumed by the CLI
// (09-cli-ops.md) and health endpoints. Thread-safe.
class Supervisor {
public:
Supervisor(common::Clock& clock, std::shared_ptr<common::Observer> observer,
const SupervisorOptions& options = SupervisorOptions{});
~Supervisor();
Supervisor(const Supervisor&) = delete;
Supervisor& operator=(const Supervisor&) = delete;
// Adds a group (max 32). When the supervisor runs, the group starts now.
std::size_t add_group(const GroupSpec& spec, WorkFunc func);
// Queue drained on shutdown. Optional; null means workers only.
void set_task_queue(std::shared_ptr<tasks::TaskQueue> queue);
void start_all();
// Graceful shutdown: stop accepting new tasks, drain in-flight tasks
// until the deadline, dead-letter leftovers with a reason, then stop
// all groups. Dropped tasks never vanish (R7-9).
void shutdown(long drain_deadline_ms = -1); // -1 means the default
void restart_group(std::size_t index);
void scale_group(std::size_t index, std::size_t n);
// Status snapshot JSON for CLI and health endpoints, capped at 1 MiB.
std::string status_json() const;
std::size_t group_count() const;
bool running() const;
ShutdownPhase phase() const;
private:
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
SupervisorOptions options_;
ShutdownCoordinator shutdown_;
mutable std::mutex mutex_;
std::vector<std::unique_ptr<WorkerGroup>> groups_;
std::shared_ptr<tasks::TaskQueue> queue_;
bool running_ = false;
};
} // namespace molo::supervisor
@@ -0,0 +1,13 @@
#pragma once
#include <stdexcept>
#include <string>
namespace molo::supervisor {
class SupervisorError : public std::runtime_error {
public:
explicit SupervisorError(const std::string& message) : std::runtime_error(message) {}
};
} // namespace molo::supervisor
+481
View File
@@ -0,0 +1,481 @@
#include "supervisor/supervisor/worker_group.hpp"
#include "supervisor/common/caps.hpp"
#include "supervisor/supervisor/supervisor_error.hpp"
namespace molo::supervisor {
std::string to_string(GroupStrategy strategy) {
switch (strategy) {
case GroupStrategy::OneForOne: return "one_for_one";
case GroupStrategy::OneForAll: return "one_for_all";
case GroupStrategy::RestForOne: return "rest_for_one";
}
return "one_for_one";
}
WorkerGroup::WorkerGroup(const GroupSpec& spec, WorkFunc func, common::Clock& clock,
std::shared_ptr<common::Observer> observer)
: spec_(spec), func_(std::move(func)), clock_(clock), observer_(std::move(observer)) {
if (spec_.name.empty() || spec_.name.size() > 128) {
throw SupervisorError("WorkerGroup: bad group name");
}
if (!func_) {
throw SupervisorError("WorkerGroup: null work function");
}
if (spec_.size < 1 || spec_.size > static_cast<std::size_t>(common::Caps::kWorkersPerGroup)) {
throw SupervisorError("WorkerGroup: size out of range");
}
spec_.restart.normalize();
if (spec_.monitor_ms < 5) {
spec_.monitor_ms = 5;
}
if (spec_.backoff_min_ms < 0) {
spec_.backoff_min_ms = 0;
}
if (spec_.backoff_max_ms < spec_.backoff_min_ms) {
spec_.backoff_max_ms = spec_.backoff_min_ms;
}
for (std::size_t i = 0; i < spec_.size; ++i) {
auto child = std::make_unique<Child>();
child->worker = std::make_shared<WorkerProcess>(spec_.name + "#" + std::to_string(i),
func_, clock_, observer_);
child->health = std::make_shared<HealthCheck>(spec_.name + "#" + std::to_string(i),
spec_.health, spec_.probe, clock_, observer_);
child->backoff_ms = spec_.backoff_min_ms;
children_.push_back(std::move(child));
}
}
WorkerGroup::~WorkerGroup() {
stop();
}
void WorkerGroup::start() {
std::lock_guard<std::mutex> lock(mutex_);
if (running_) {
return;
}
running_ = true;
stopping_ = false;
for (auto& child : children_) {
child->worker->reset_for_restart();
child->worker->start();
child->health->reset();
}
monitor_ = std::thread(&WorkerGroup::monitor_main, this);
observer_->event("supervisor.group.started", spec_.name);
}
void WorkerGroup::stop() {
{
std::lock_guard<std::mutex> lock(mutex_);
if (!running_) {
return;
}
stopping_ = true;
monitor_cv_.notify_all();
}
if (monitor_.joinable()) {
monitor_.join();
}
std::vector<std::shared_ptr<WorkerProcess>> workers;
{
std::lock_guard<std::mutex> lock(mutex_);
for (auto& child : children_) {
workers.push_back(child->worker);
child->worker->request_stop();
}
pending_.clear();
}
for (auto& worker : workers) {
worker->join();
}
std::lock_guard<std::mutex> lock(mutex_);
running_ = false;
stopping_ = false;
observer_->event("supervisor.group.stopped", spec_.name);
}
bool WorkerGroup::running() const {
std::lock_guard<std::mutex> lock(mutex_);
return running_;
}
void WorkerGroup::set_size(std::size_t n) {
if (n < 1 || n > static_cast<std::size_t>(common::Caps::kWorkersPerGroup)) {
throw SupervisorError("WorkerGroup: size out of range");
}
std::vector<std::shared_ptr<WorkerProcess>> removed;
{
std::lock_guard<std::mutex> lock(mutex_);
if (n > children_.size()) {
for (std::size_t i = children_.size(); i < n; ++i) {
auto child = std::make_unique<Child>();
child->worker = std::make_shared<WorkerProcess>(spec_.name + "#" + std::to_string(i),
func_, clock_, observer_);
child->health =
std::make_shared<HealthCheck>(spec_.name + "#" + std::to_string(i), spec_.health,
spec_.probe, clock_, observer_);
child->backoff_ms = spec_.backoff_min_ms;
if (running_) {
child->worker->start();
}
children_.push_back(std::move(child));
}
} else {
while (children_.size() > n) {
auto& back = children_.back();
back->worker->request_stop();
removed.push_back(back->worker);
pending_.erase(children_.size() - 1);
children_.pop_back();
}
}
observer_->event("supervisor.group.scaled",
spec_.name + " size=" + std::to_string(children_.size()));
}
for (auto& worker : removed) {
worker->join();
}
}
std::size_t WorkerGroup::size() const {
std::lock_guard<std::mutex> lock(mutex_);
return children_.size();
}
void WorkerGroup::restart_child(std::size_t index) {
std::shared_ptr<WorkerProcess> worker;
{
std::lock_guard<std::mutex> lock(mutex_);
if (index >= children_.size()) {
throw SupervisorError("WorkerGroup: child index out of range");
}
worker = children_[index]->worker;
children_[index]->restarts.clear();
children_[index]->backoff_ms = spec_.backoff_min_ms;
children_[index]->health->reset();
pending_.erase(index);
}
worker->request_stop();
worker->join();
worker->reset_for_restart();
worker->start();
observer_->event("supervisor.group.child_restarted", spec_.name + "#" + std::to_string(index));
}
void WorkerGroup::restart_all() {
std::vector<std::shared_ptr<WorkerProcess>> workers;
{
std::lock_guard<std::mutex> lock(mutex_);
for (auto& child : children_) {
workers.push_back(child->worker);
child->worker->request_stop();
child->restarts.clear();
child->backoff_ms = spec_.backoff_min_ms;
child->health->reset();
}
pending_.clear();
}
for (auto& worker : workers) {
worker->join();
}
for (auto& worker : workers) {
worker->reset_for_restart();
worker->start();
}
observer_->event("supervisor.group.restarted", spec_.name);
}
bool WorkerGroup::fatal() const {
std::lock_guard<std::mutex> lock(mutex_);
for (const auto& child : children_) {
if (child->worker->state() == WorkerState::Fatal) {
return true;
}
}
return false;
}
std::vector<WorkerState> WorkerGroup::states() const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<WorkerState> out;
for (const auto& child : children_) {
out.push_back(child->worker->state());
}
return out;
}
common::JValue WorkerGroup::snapshot() const {
using common::JValue;
std::lock_guard<std::mutex> lock(mutex_);
JValue::Object obj;
obj["name"] = JValue(spec_.name);
obj["strategy"] = JValue(to_string(spec_.strategy));
obj["restart_policy"] = JValue(to_string(spec_.restart.policy));
obj["running"] = JValue(running_);
JValue::Array workers;
bool any_fatal = false;
for (const auto& child : children_) {
const WorkerState state = child->worker->state();
if (state == WorkerState::Fatal) {
any_fatal = true;
}
JValue::Object w;
w["name"] = JValue(child->worker->name());
w["state"] = JValue(to_string(state));
w["exit_code"] = JValue(child->worker->exit_code());
w["starts"] = JValue(static_cast<long long>(child->worker->starts()));
w["restarts_in_window"] = JValue(static_cast<long long>(child->restarts.size()));
w["health_failures"] = JValue(child->health->failures());
workers.push_back(JValue(w));
}
obj["fatal"] = JValue(any_fatal);
obj["workers"] = JValue(workers);
return JValue(obj);
}
const std::string& WorkerGroup::name() const {
return spec_.name;
}
std::vector<std::size_t> WorkerGroup::strategy_targets(std::size_t index) const {
std::vector<std::size_t> targets;
switch (spec_.strategy) {
case GroupStrategy::OneForOne: targets.push_back(index); break;
case GroupStrategy::OneForAll:
for (std::size_t i = 0; i < children_.size(); ++i) {
targets.push_back(i);
}
break;
case GroupStrategy::RestForOne:
for (std::size_t i = index; i < children_.size(); ++i) {
targets.push_back(i);
}
break;
}
return targets;
}
void WorkerGroup::handle_exit_locked(std::size_t index, int exit_code, bool from_health,
std::vector<std::size_t>& stop_list) {
bool wanted = false;
if (from_health) {
wanted = (spec_.restart.policy != RestartPolicy::Never);
} else if (spec_.restart.policy == RestartPolicy::Always) {
wanted = true;
} else if (spec_.restart.policy == RestartPolicy::OnFailure && exit_code != 0) {
wanted = true;
}
if (!wanted) {
return;
}
for (std::size_t target : strategy_targets(index)) {
if (target >= children_.size()) {
continue;
}
if (pending_.find(target) != pending_.end()) {
continue; // restart already scheduled
}
const WorkerState state = children_[target]->worker->state();
if (state == WorkerState::Fatal || state == WorkerState::Stopped) {
continue;
}
if (state == WorkerState::Exited) {
schedule_restart_locked(target);
} else if (state == WorkerState::Running || state == WorkerState::Starting) {
// Strategy siblings (or an unhealthy worker) stop first, then the
// restart is scheduled after the join outside the lock.
stop_list.push_back(target);
}
}
}
void WorkerGroup::schedule_restart_locked(std::size_t index) {
Child& child = *children_[index];
const std::int64_t now = clock_.steady_ms();
const std::int64_t window_ms = static_cast<std::int64_t>(spec_.restart.window_secs) * 1000;
while (!child.restarts.empty() && child.restarts.front() <= now - window_ms) {
child.restarts.pop_front();
}
if (static_cast<int>(child.restarts.size()) >= spec_.restart.max_restarts) {
child.worker->mark_fatal();
observer_->event("supervisor.group.intensity_cap", spec_.name + " child locked fatal");
return;
}
child.restarts.push_back(now);
child.last_restart_steady_ms = now;
pending_[index] = now + child.backoff_ms;
child.backoff_ms = std::min(child.backoff_ms * 2, spec_.backoff_max_ms);
if (child.worker->state() == WorkerState::Exited) {
child.worker->mark_backoff();
}
observer_->counter("molo_supervisor_worker_restarts_total", 1);
}
void WorkerGroup::fire_restart(std::size_t index) {
std::shared_ptr<WorkerProcess> worker;
{
std::lock_guard<std::mutex> lock(mutex_);
if (stopping_ || index >= children_.size()) {
return;
}
worker = children_[index]->worker;
const WorkerState state = worker->state();
// Stopped here means a strategy sibling stopped for restart (group
// stop clears all pending restarts first, so it cannot land here).
if (state == WorkerState::Fatal || state == WorkerState::Running ||
state == WorkerState::Starting || state == WorkerState::Stopping) {
return;
}
}
worker->join(); // collects the old thread; already finished, non-blocking
worker->reset_for_restart();
if (!worker->start()) {
return;
}
std::lock_guard<std::mutex> lock(mutex_);
if (index < children_.size()) {
children_[index]->health->reset();
}
}
void WorkerGroup::monitor_main() {
while (true) {
std::vector<std::size_t> exited;
std::vector<std::size_t> probes;
std::vector<std::size_t> firing;
{
std::lock_guard<std::mutex> lock(mutex_);
if (stopping_) {
return;
}
const std::int64_t now = clock_.steady_ms();
for (auto it = pending_.begin(); it != pending_.end();) {
if (it->second <= now) {
firing.push_back(it->first);
it = pending_.erase(it);
} else {
++it;
}
}
for (std::size_t i = 0; i < children_.size(); ++i) {
const WorkerState state = children_[i]->worker->state();
if (state == WorkerState::Exited) {
exited.push_back(i);
} else if (state == WorkerState::Running && children_[i]->health->due()) {
probes.push_back(i);
}
// A clean run resets the backoff ladder (supervisord-style).
if (state == WorkerState::Running &&
children_[i]->backoff_ms > spec_.backoff_min_ms &&
now - children_[i]->last_restart_steady_ms > spec_.backoff_reset_after_ms) {
children_[i]->backoff_ms = spec_.backoff_min_ms;
}
}
}
for (std::size_t index : exited) {
std::shared_ptr<WorkerProcess> worker;
{
std::lock_guard<std::mutex> lock(mutex_);
if (index < children_.size()) {
worker = children_[index]->worker;
}
}
if (worker) {
worker->join();
}
}
std::vector<std::size_t> unhealthy;
for (std::size_t index : probes) {
std::shared_ptr<HealthCheck> health;
{
std::lock_guard<std::mutex> lock(mutex_);
if (index < children_.size()) {
health = children_[index]->health;
}
}
if (health) {
health->check_now();
if (health->unhealthy()) {
unhealthy.push_back(index);
}
}
}
std::vector<std::size_t> stop_list;
{
std::lock_guard<std::mutex> lock(mutex_);
if (stopping_) {
return;
}
for (std::size_t index : exited) {
if (index >= children_.size()) {
continue;
}
if (children_[index]->worker->state() != WorkerState::Exited) {
continue; // user restarted it meanwhile
}
const int code = children_[index]->worker->exit_code();
handle_exit_locked(index, code, false, stop_list);
}
for (std::size_t index : unhealthy) {
if (index >= children_.size()) {
continue;
}
if (children_[index]->worker->state() != WorkerState::Running) {
continue;
}
observer_->event("supervisor.group.unhealthy_restart",
spec_.name + "#" + std::to_string(index));
handle_exit_locked(index, 1, true, stop_list);
}
}
for (std::size_t index : stop_list) {
std::shared_ptr<WorkerProcess> worker;
{
std::lock_guard<std::mutex> lock(mutex_);
if (index < children_.size()) {
worker = children_[index]->worker;
}
}
if (worker) {
worker->request_stop();
worker->join();
}
}
{
std::lock_guard<std::mutex> lock(mutex_);
if (!stopping_) {
for (std::size_t index : stop_list) {
if (index >= children_.size()) {
continue;
}
const WorkerState state = children_[index]->worker->state();
if (state == WorkerState::Stopped || state == WorkerState::Exited) {
schedule_restart_locked(index);
}
}
}
}
for (std::size_t index : firing) {
fire_restart(index);
}
{
std::unique_lock<std::mutex> lock(mutex_);
monitor_cv_.wait_for(lock, std::chrono::milliseconds(spec_.monitor_ms),
[&] { return stopping_; });
if (stopping_) {
return;
}
}
}
}
} // namespace molo::supervisor
+102
View File
@@ -0,0 +1,102 @@
#pragma once
#include <atomic>
#include <condition_variable>
#include <cstdint>
#include <deque>
#include <functional>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <thread>
#include <vector>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/json.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/supervisor/health_check.hpp"
#include "supervisor/supervisor/restart_policy.hpp"
#include "supervisor/supervisor/worker_process.hpp"
namespace molo::supervisor {
// OTP restart strategies for worker groups (BP-1).
enum class GroupStrategy {
OneForOne, // restart only the failed child
OneForAll, // restart all children
RestForOne, // restart the failed child plus children started after it
};
std::string to_string(GroupStrategy strategy);
struct GroupSpec {
std::string name;
GroupStrategy strategy = GroupStrategy::OneForOne;
RestartSpec restart;
HealthSpec health;
std::function<bool()> probe; // shared function probe (Function kind)
std::size_t size = 1;
long monitor_ms = 50;
long backoff_min_ms = 1000;
long backoff_max_ms = 60000;
long backoff_reset_after_ms = 10000; // clean run that resets backoff
};
// Owns one supervised program: N WorkerProcess children running the same
// WorkFunc, one monitor loop applying the restart strategy, intensity
// caps, backoff, and health checks (E7-1, S7-4). Thread-safe.
class WorkerGroup {
public:
WorkerGroup(const GroupSpec& spec, WorkFunc func, common::Clock& clock,
std::shared_ptr<common::Observer> observer);
~WorkerGroup();
WorkerGroup(const WorkerGroup&) = delete;
WorkerGroup& operator=(const WorkerGroup&) = delete;
void start();
void stop();
bool running() const;
void set_size(std::size_t n); // scale, capped at 64 workers
std::size_t size() const;
void restart_child(std::size_t index); // explicit, clears Fatal
void restart_all();
bool fatal() const; // true when any child is Fatal
std::vector<WorkerState> states() const;
common::JValue snapshot() const;
const std::string& name() const;
private:
struct Child {
std::shared_ptr<WorkerProcess> worker;
std::shared_ptr<HealthCheck> health;
std::deque<std::int64_t> restarts; // steady-ms timestamps in window
long backoff_ms = 1000;
std::int64_t last_restart_steady_ms = 0;
};
void monitor_main();
void handle_exit_locked(std::size_t index, int exit_code, bool from_health,
std::vector<std::size_t>& stop_list);
void schedule_restart_locked(std::size_t index);
void fire_restart(std::size_t index);
std::vector<std::size_t> strategy_targets(std::size_t index) const;
GroupSpec spec_;
WorkFunc func_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
mutable std::mutex mutex_;
std::condition_variable monitor_cv_;
std::vector<std::unique_ptr<Child>> children_;
std::map<std::size_t, std::int64_t> pending_; // index -> restart due (steady ms)
std::thread monitor_;
bool running_ = false;
bool stopping_ = false;
};
} // namespace molo::supervisor
@@ -0,0 +1,150 @@
#include "supervisor/supervisor/worker_process.hpp"
namespace molo::supervisor {
WorkerProcess::WorkerProcess(std::string name, WorkFunc func, common::Clock& clock,
std::shared_ptr<common::Observer> observer)
: name_(std::move(name)),
func_(std::move(func)),
clock_(clock),
observer_(std::move(observer)) {}
WorkerProcess::~WorkerProcess() {
request_stop();
join();
}
bool WorkerProcess::start() {
std::lock_guard<std::mutex> lock(mutex_);
if (state_ == WorkerState::Running || state_ == WorkerState::Starting ||
state_ == WorkerState::Stopping) {
return false;
}
if (state_ == WorkerState::Fatal) {
return false; // explicit reset_for_restart() required
}
if (thread_.joinable()) {
return false;
}
stop_requested_.store(false);
state_ = WorkerState::Starting;
last_start_steady_ms_ = clock_.steady_ms();
++starts_;
observer_->event("supervisor.worker.starting", name_);
thread_ = std::thread(&WorkerProcess::thread_main, this);
return true;
}
void WorkerProcess::thread_main() {
{
std::lock_guard<std::mutex> lock(mutex_);
state_ = WorkerState::Running;
}
observer_->event("supervisor.worker.running", name_);
observer_->counter("molo_supervisor_worker_starts_total", 1);
int code = 0;
try {
code = func_(stop_requested_);
} catch (...) {
code = 1;
}
{
std::lock_guard<std::mutex> lock(mutex_);
exit_code_ = code;
last_exit_steady_ms_ = clock_.steady_ms();
state_ = stop_requested_.load() ? WorkerState::Stopped : WorkerState::Exited;
}
observer_->event("supervisor.worker.exited", name_ + " code=" + std::to_string(code));
}
void WorkerProcess::request_stop() {
stop_requested_.store(true);
std::lock_guard<std::mutex> lock(mutex_);
if (state_ == WorkerState::Running || state_ == WorkerState::Starting) {
state_ = WorkerState::Stopping;
}
}
void WorkerProcess::join() {
// Move the handle out under lock, then join without it: thread_main
// needs the state mutex to publish Exited.
std::thread finished;
{
std::lock_guard<std::mutex> lock(mutex_);
if (thread_.joinable()) {
finished = std::move(thread_);
}
}
if (finished.joinable()) {
finished.join();
}
std::lock_guard<std::mutex> lock(mutex_);
// A worker that was asked to stop and already exited lands on Stopped;
// one that exited on its own keeps Exited for the group to handle.
if (stop_requested_.load() && state_ == WorkerState::Exited) {
state_ = WorkerState::Stopped;
}
}
bool WorkerProcess::joinable() const {
std::lock_guard<std::mutex> lock(mutex_);
return thread_.joinable();
}
WorkerState WorkerProcess::state() const {
std::lock_guard<std::mutex> lock(mutex_);
return state_;
}
const std::string& WorkerProcess::name() const {
return name_;
}
int WorkerProcess::exit_code() const {
std::lock_guard<std::mutex> lock(mutex_);
return exit_code_;
}
std::uint64_t WorkerProcess::starts() const {
std::lock_guard<std::mutex> lock(mutex_);
return starts_;
}
std::int64_t WorkerProcess::last_start_steady_ms() const {
std::lock_guard<std::mutex> lock(mutex_);
return last_start_steady_ms_;
}
std::int64_t WorkerProcess::last_exit_steady_ms() const {
std::lock_guard<std::mutex> lock(mutex_);
return last_exit_steady_ms_;
}
void WorkerProcess::mark_fatal() {
std::lock_guard<std::mutex> lock(mutex_);
// Any non-running state may go fatal: Exited and Backoff after a crash,
// Stopped after a strategy sibling stop. Live workers never do.
if (state_ != WorkerState::Running && state_ != WorkerState::Starting &&
state_ != WorkerState::Stopping && state_ != WorkerState::Fatal) {
state_ = WorkerState::Fatal;
observer_->event("supervisor.worker.fatal", name_);
observer_->counter("molo_supervisor_worker_fatal_total", 1);
}
}
void WorkerProcess::mark_backoff() {
std::lock_guard<std::mutex> lock(mutex_);
if (state_ == WorkerState::Exited) {
state_ = WorkerState::Backoff;
}
}
void WorkerProcess::reset_for_restart() {
std::lock_guard<std::mutex> lock(mutex_);
if (state_ == WorkerState::Fatal || state_ == WorkerState::Exited ||
state_ == WorkerState::Stopped) {
state_ = WorkerState::Stopped;
}
}
} // namespace molo::supervisor
@@ -0,0 +1,71 @@
#pragma once
#include <atomic>
#include <cstdint>
#include <functional>
#include <memory>
#include <mutex>
#include <string>
#include <thread>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/supervisor/worker_state.hpp"
namespace molo::supervisor {
// Work function run by a worker. Returns the process exit code (0 means
// clean). Must poll stop_requested and return promptly; throwing counts
// as exit code 1. Transport-agnostic: v1 runs this on a thread, v2 runs
// the same signature in an OS process behind this interface (O7-2).
using WorkFunc = std::function<int(const std::atomic<bool>& stop_requested)>;
// One supervised child (v1: in-process thread worker). Owns its lifecycle
// only; restart decisions live in WorkerGroup. Thread-safe.
class WorkerProcess {
public:
WorkerProcess(std::string name, WorkFunc func, common::Clock& clock,
std::shared_ptr<common::Observer> observer);
~WorkerProcess();
WorkerProcess(const WorkerProcess&) = delete;
WorkerProcess& operator=(const WorkerProcess&) = delete;
// Starts the worker. Returns false when already started.
bool start();
void request_stop();
void join();
bool joinable() const;
WorkerState state() const;
const std::string& name() const;
int exit_code() const;
std::uint64_t starts() const;
std::int64_t last_start_steady_ms() const;
std::int64_t last_exit_steady_ms() const;
// Called by the group when the intensity cap trips (S7-4).
void mark_fatal();
// Called by the group while a restart backoff is pending.
void mark_backoff();
// Clears Fatal/Exited so start() may run again (explicit restart only).
void reset_for_restart();
private:
void thread_main();
std::string name_;
WorkFunc func_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
mutable std::mutex mutex_;
std::thread thread_;
std::atomic<bool> stop_requested_{false};
WorkerState state_ = WorkerState::Stopped;
int exit_code_ = 0;
std::uint64_t starts_ = 0;
std::int64_t last_start_steady_ms_ = 0;
std::int64_t last_exit_steady_ms_ = 0;
};
} // namespace molo::supervisor
@@ -0,0 +1,32 @@
#pragma once
#include <string>
namespace molo::supervisor {
// Supervised worker states (S7-4). Fatal is entered after the intensity
// cap trips; only an explicit restart clears it.
enum class WorkerState {
Stopped,
Starting,
Running,
Backoff,
Stopping,
Exited,
Fatal,
};
inline std::string to_string(WorkerState state) {
switch (state) {
case WorkerState::Stopped: return "stopped";
case WorkerState::Starting: return "starting";
case WorkerState::Running: return "running";
case WorkerState::Backoff: return "backoff";
case WorkerState::Stopping: return "stopping";
case WorkerState::Exited: return "exited";
case WorkerState::Fatal: return "fatal";
}
return "stopped";
}
} // namespace molo::supervisor
+187
View File
@@ -0,0 +1,187 @@
#include "supervisor/tasks/cron.hpp"
#include <ctime>
#include <vector>
namespace molo::tasks {
namespace {
std::vector<std::string> split_fields(std::string_view text) {
std::vector<std::string> fields;
std::string current;
for (char c : text) {
if (c == ' ' || c == '\t') {
if (!current.empty()) {
fields.push_back(current);
current.clear();
}
} else {
current += c;
}
}
if (!current.empty()) {
fields.push_back(current);
}
return fields;
}
bool parse_int(std::string_view text, int& out) {
if (text.empty() || text.size() > 4) {
return false;
}
int value = 0;
for (char c : text) {
if (c < '0' || c > '9') {
return false;
}
value = value * 10 + (c - '0');
}
out = value;
return true;
}
// Parses one field into flags[low..high]. Returns false on bad syntax.
bool parse_field(std::string_view text, bool* flags, int low, int high, bool& restricted) {
for (int i = low; i <= high; ++i) {
flags[i] = false;
}
restricted = (text != "*");
std::size_t pos = 0;
bool any = false;
while (pos <= text.size()) {
std::size_t comma = text.find(',', pos);
if (comma == std::string_view::npos) {
comma = text.size();
}
const std::string_view item = text.substr(pos, comma - pos);
if (item.empty()) {
return false;
}
// Split off /step.
std::string_view range = item;
int step = 1;
const std::size_t slash = item.find('/');
if (slash != std::string_view::npos) {
range = item.substr(0, slash);
if (!parse_int(item.substr(slash + 1), step) || step < 1) {
return false;
}
}
int from = low, to = high;
if (range == "*") {
from = low;
to = high;
} else {
const std::size_t dash = range.find('-');
if (dash == std::string_view::npos) {
if (!parse_int(range, from)) {
return false;
}
to = (slash == std::string_view::npos) ? from : high;
} else {
if (!parse_int(range.substr(0, dash), from) ||
!parse_int(range.substr(dash + 1), to)) {
return false;
}
}
}
if (from < low || to > high || from > to) {
return false;
}
for (int v = from; v <= to; v += step) {
flags[v] = true;
any = true;
}
if (comma == text.size()) {
break;
}
pos = comma + 1;
}
return any;
}
bool parse_dow_field(std::string_view text, bool* flags, bool& restricted) {
// Normalize 7 to 0 before parsing.
std::string norm(text);
for (std::size_t i = 0; i < norm.size(); ++i) {
if (norm[i] == '7' && (i + 1 >= norm.size() || norm[i + 1] < '0' || norm[i + 1] > '9') &&
(i == 0 || norm[i - 1] < '0' || norm[i - 1] > '9')) {
norm[i] = '0';
}
}
return parse_field(norm, flags, 0, 6, restricted);
}
} // namespace
bool CronExpr::parse(std::string_view text, CronExpr& out, std::string& error) {
CronExpr expr;
const auto fields = split_fields(text);
if (fields.size() != 5) {
error = "cron: need exactly 5 fields";
return false;
}
bool ignored = false;
if (!parse_field(fields[0], expr.minutes_, 0, 59, ignored)) {
error = "cron: bad minute field";
return false;
}
if (!parse_field(fields[1], expr.hours_, 0, 23, ignored)) {
error = "cron: bad hour field";
return false;
}
if (!parse_field(fields[2], expr.days_, 1, 31, expr.dom_restricted_)) {
error = "cron: bad day-of-month field";
return false;
}
if (!parse_field(fields[3], expr.months_, 1, 12, ignored)) {
error = "cron: bad month field";
return false;
}
if (!parse_dow_field(fields[4], expr.dows_, expr.dow_restricted_)) {
error = "cron: bad day-of-week field";
return false;
}
out = expr;
return true;
}
bool CronExpr::matches(std::int64_t wall_ms) const {
const std::time_t secs = static_cast<std::time_t>(wall_ms / 1000);
std::tm tm_utc{};
#if defined(_WIN32)
gmtime_s(&tm_utc, &secs);
#else
gmtime_r(&secs, &tm_utc);
#endif
if (!minutes_[tm_utc.tm_min] || !hours_[tm_utc.tm_hour] || !months_[tm_utc.tm_mon + 1]) {
return false;
}
const bool dom_match = days_[tm_utc.tm_mday];
const bool dow_match = dows_[tm_utc.tm_wday];
if (dom_restricted_ && dow_restricted_) {
return dom_match || dow_match;
}
if (dom_restricted_) {
return dom_match;
}
if (dow_restricted_) {
return dow_match;
}
return true;
}
std::int64_t CronExpr::next_after(std::int64_t wall_ms) const {
// Step minute by minute, capped at 366 days of search.
std::int64_t minute = (wall_ms / 60000) * 60000 + 60000;
const std::int64_t limit = minute + 366LL * 86400 * 1000;
for (; minute <= limit; minute += 60000) {
if (matches(minute)) {
return minute;
}
}
return -1;
}
} // namespace molo::tasks
+35
View File
@@ -0,0 +1,35 @@
#pragma once
#include <cstdint>
#include <string>
#include <string_view>
namespace molo::tasks {
// Five-field UTC cron expression: minute hour day-of-month month day-of-week.
// Each field accepts '*', '*/step', single values, ranges ('a-b'),
// stepped ranges ('a-b/step'), and comma lists. Numbers only in v1
// (no MON/JAN names, no @macros). Day matching follows Vixie semantics:
// when both day-of-month and day-of-week are restricted, either may match.
class CronExpr {
public:
static bool parse(std::string_view text, CronExpr& out, std::string& error);
// True when the UTC minute containing wall_ms matches the expression.
bool matches(std::int64_t wall_ms) const;
// Start of the first matching minute strictly after wall_ms, or -1 when
// no minute matches within 366 days.
std::int64_t next_after(std::int64_t wall_ms) const;
private:
bool minutes_[60] = {};
bool hours_[24] = {};
bool days_[32] = {}; // 1..31
bool months_[13] = {}; // 1..12
bool dows_[7] = {}; // 0..6, Sunday is 0 (7 folds to 0)
bool dom_restricted_ = false;
bool dow_restricted_ = false;
};
} // namespace molo::tasks
@@ -0,0 +1,91 @@
#include "supervisor/tasks/dead_letter_queue.hpp"
namespace molo::tasks {
DeadLetterQueue::DeadLetterQueue(common::Clock& clock,
std::shared_ptr<common::Observer> observer, int max_entries,
long retention_days)
: clock_(clock),
observer_(std::move(observer)),
max_entries_(max_entries),
retention_days_(retention_days) {
if (max_entries_ < 1) {
max_entries_ = 1;
}
}
void DeadLetterQueue::push(TaskEnvelope envelope, const std::string& reason) {
std::lock_guard<std::mutex> lock(mutex_);
while (static_cast<int>(entries_.size()) >= max_entries_) {
entries_.pop_front();
++evicted_total_;
observer_->counter("molo_tasks_dead_letter_evicted_total", 1);
observer_->event("tasks.dead_letter.evicted", "cap=oldest-first");
}
envelope.state = TaskState::DeadLettered;
entries_.push_back(DeadEntry{std::move(envelope), reason, clock_.wall_ms()});
observer_->counter("molo_tasks_dead_lettered_total", 1);
observer_->event("tasks.dead_lettered", reason);
}
std::vector<DeadEntry> DeadLetterQueue::list(std::size_t max) const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<DeadEntry> out;
for (const auto& entry : entries_) {
if (out.size() >= max) {
break;
}
out.push_back(entry);
}
return out;
}
bool DeadLetterQueue::take(const std::string& id, DeadEntry& out) {
std::lock_guard<std::mutex> lock(mutex_);
for (auto it = entries_.begin(); it != entries_.end(); ++it) {
if (it->envelope.id == id) {
out = *it;
entries_.erase(it);
observer_->event("tasks.dead_letter.retried", id);
return true;
}
}
return false;
}
bool DeadLetterQueue::remove(const std::string& id) {
std::lock_guard<std::mutex> lock(mutex_);
for (auto it = entries_.begin(); it != entries_.end(); ++it) {
if (it->envelope.id == id) {
entries_.erase(it);
return true;
}
}
return false;
}
std::size_t DeadLetterQueue::size() const {
std::lock_guard<std::mutex> lock(mutex_);
return entries_.size();
}
std::size_t DeadLetterQueue::purge_expired() {
std::lock_guard<std::mutex> lock(mutex_);
const std::int64_t horizon = clock_.wall_ms() - retention_days_ * 86400 * 1000;
std::size_t purged = 0;
while (!entries_.empty() && entries_.front().dead_at_ms < horizon) {
entries_.pop_front();
++purged;
}
if (purged > 0) {
observer_->counter("molo_tasks_dead_letter_expired_total", static_cast<long>(purged));
}
return purged;
}
long DeadLetterQueue::evicted_total() const {
std::lock_guard<std::mutex> lock(mutex_);
return evicted_total_;
}
} // namespace molo::tasks
@@ -0,0 +1,52 @@
#pragma once
#include <cstdint>
#include <deque>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/tasks/task_envelope.hpp"
namespace molo::tasks {
struct DeadEntry {
TaskEnvelope envelope;
std::string reason;
std::int64_t dead_at_ms = 0;
};
// Dead-letter queue (S7-2): envelopes that exhausted retries or failed
// permanently wait here 7 days for inspection, retry, or cancel via the
// CLI (09-cli-ops.md). Max 10000 entries; overflow evicts oldest-first
// with an event emitted. Nothing vanishes silently.
class DeadLetterQueue {
public:
DeadLetterQueue(common::Clock& clock, std::shared_ptr<common::Observer> observer,
int max_entries = 10000, long retention_days = 7);
void push(TaskEnvelope envelope, const std::string& reason);
std::vector<DeadEntry> list(std::size_t max) const;
// Removes and returns one entry for manual retry. Returns false when absent.
bool take(const std::string& id, DeadEntry& out);
bool remove(const std::string& id);
std::size_t size() const;
std::size_t purge_expired(); // drops entries past retention, returns count
long evicted_total() const;
private:
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
int max_entries_;
long retention_days_;
mutable std::mutex mutex_;
std::deque<DeadEntry> entries_;
long evicted_total_ = 0;
};
} // namespace molo::tasks
+56
View File
@@ -0,0 +1,56 @@
#include "supervisor/tasks/retry_policy.hpp"
#include <random>
namespace molo::tasks {
RetryPolicy::RetryPolicy(const RetrySpec& spec) : spec_(spec) {
if (spec_.base_ms < 0) {
spec_.base_ms = 0;
}
if (spec_.cap_ms < spec_.base_ms) {
spec_.cap_ms = spec_.base_ms;
}
if (spec_.jitter_ms < 0) {
spec_.jitter_ms = 0;
}
if (spec_.max_retries < 0) {
spec_.max_retries = 0;
}
}
long RetryPolicy::delay_for_attempt(int attempt, std::mt19937& rng) const {
if (attempt < 0) {
attempt = 0;
}
// base * 2^n with overflow guard: shift past 30 saturates to the cap.
long delay = spec_.base_ms;
if (attempt >= 31) {
delay = spec_.cap_ms;
} else {
const long long grown = static_cast<long long>(spec_.base_ms) * (1LL << attempt);
delay = grown >= spec_.cap_ms ? spec_.cap_ms : static_cast<long>(grown);
}
long jitter = 0;
if (spec_.jitter_ms > 0) {
std::uniform_int_distribution<long> dist(0, spec_.jitter_ms);
jitter = dist(rng);
}
const long long total = static_cast<long long>(delay) + jitter;
return total >= spec_.cap_ms ? spec_.cap_ms : static_cast<long>(total);
}
long RetryPolicy::delay_for_attempt(int attempt) const {
static thread_local std::mt19937 rng{std::random_device{}()};
return delay_for_attempt(attempt, rng);
}
bool RetryPolicy::exhausted(int attempts) const {
return attempts > spec_.max_retries;
}
const RetrySpec& RetryPolicy::spec() const {
return spec_;
}
} // namespace molo::tasks
+31
View File
@@ -0,0 +1,31 @@
#pragma once
#include <random>
namespace molo::tasks {
// Bounded exponential backoff with jitter (S7-2):
// delay(n) = min(backoff_base_ms * 2^n + jitter, backoff_cap_ms)
// where n is the zero-based retry number and jitter is uniform
// 0..jitter_ms. Pure computation, no I/O, trivially testable.
struct RetrySpec {
long base_ms = 1000;
long cap_ms = 300000;
long jitter_ms = 1000;
int max_retries = 5;
};
class RetryPolicy {
public:
explicit RetryPolicy(const RetrySpec& spec = RetrySpec{});
long delay_for_attempt(int attempt, std::mt19937& rng) const;
long delay_for_attempt(int attempt) const; // thread-local RNG
bool exhausted(int attempts) const; // attempts > max_retries
const RetrySpec& spec() const;
private:
RetrySpec spec_;
};
} // namespace molo::tasks
+380
View File
@@ -0,0 +1,380 @@
#include "supervisor/tasks/scheduler.hpp"
#include <filesystem>
#include "supervisor/common/caps.hpp"
#include "supervisor/common/json.hpp"
namespace molo::tasks {
namespace {
// Option clamps shared by the scheduler and its LeaderElection (which has
// no default ctor, so normalization must run in the init list).
SchedulerOptions NormalizedOptions(SchedulerOptions o) {
if (o.instance_id.empty()) {
o.instance_id = "sched-" + TaskEnvelope::generate_id();
}
if (o.tick_ms < 10) {
o.tick_ms = 10;
}
if (o.ttl_secs < 1) {
o.ttl_secs = 1;
}
if (o.renew_secs < 1) {
o.renew_secs = 1;
}
return o;
}
} // namespace
Scheduler::Scheduler(TaskQueue& queue, common::KvStore& kv, common::Clock& clock,
std::shared_ptr<common::Observer> observer,
const SchedulerOptions& options)
: queue_(queue),
kv_(kv),
clock_(clock),
observer_(std::move(observer)),
options_(NormalizedOptions(options)),
election_(kv_, clock_, observer_ ? *observer_ : fallback_observer_,
options_.leader_key, options_.instance_id, options_.ttl_secs * 1000,
options_.renew_secs * 1000) {
if (!options_.durable_dir.empty()) {
log_ = std::make_unique<bus::SegmentLog>(
std::filesystem::path(options_.durable_dir) / "scheduler.log");
replay();
}
}
void Scheduler::replay() {
// Rebuilds schedule entries from the op log (E7-6). Last-fire records
// live in shared KV: with the shared backend they survive too, and a
// reboot never refires a window the old leader already fired.
std::uint64_t seq = 1;
while (true) {
auto records = log_->read_from(seq, 256, 1024 * 1024);
if (records.empty()) {
break;
}
for (const auto& record : records) {
common::JValue root;
if (!common::json_decode(record.data, root)) {
continue;
}
const common::JValue* op = root.find("op");
if (op == nullptr) {
continue;
}
if (op->as_string() == "put") {
const common::JValue* body = root.find("entry");
if (body == nullptr) {
continue;
}
ScheduleEntry entry;
if (decode_entry(common::json_encode(*body), entry)) {
CronExpr cron;
if (entry.kind == ScheduleKind::Cron) {
std::string error;
if (!CronExpr::parse(entry.cron, cron, error)) {
continue;
}
cron_cache_[entry.key] = cron;
}
entries_[entry.key] = entry;
registered_at_[entry.key] = clock_.wall_ms();
}
} else if (op->as_string() == "remove") {
const common::JValue* key = root.find("key");
if (key != nullptr) {
entries_.erase(key->as_string());
cron_cache_.erase(key->as_string());
registered_at_.erase(key->as_string());
}
}
}
seq = records.back().seq + 1;
}
observer_->event("tasks.schedule.replayed", "entries=" + std::to_string(entries_.size()));
}
std::string Scheduler::encode_entry(const ScheduleEntry& entry) const {
using common::JValue;
JValue::Object obj;
obj["key"] = JValue(entry.key);
obj["task_name"] = JValue(entry.task_name);
JValue payload;
if (!common::json_decode(entry.payload, payload)) {
payload = JValue(entry.payload);
}
obj["payload"] = payload;
obj["kind"] = JValue(entry.kind == ScheduleKind::Once
? "once"
: (entry.kind == ScheduleKind::Interval ? "interval" : "cron"));
obj["at_ms"] = JValue(static_cast<long long>(entry.at_ms));
obj["interval_secs"] = JValue(static_cast<long long>(entry.interval_secs));
obj["cron"] = JValue(entry.cron);
obj["missed"] = JValue(entry.missed == MissedPolicy::Skip ? "skip" : "run-once");
obj["enabled"] = JValue(entry.enabled);
obj["max_retries"] = JValue(entry.max_retries);
return common::json_encode(JValue(obj));
}
bool Scheduler::decode_entry(const std::string& text, ScheduleEntry& out) const {
using common::JValue;
JValue root;
if (!common::json_decode(text, root) || !root.is_object()) {
return false;
}
const JValue* key = root.find("key");
const JValue* task_name = root.find("task_name");
const JValue* kind = root.find("kind");
if (key == nullptr || task_name == nullptr || kind == nullptr || !key->is_string() ||
!task_name->is_string() || !kind->is_string()) {
return false;
}
ScheduleEntry entry;
entry.key = key->as_string();
entry.task_name = task_name->as_string();
const std::string kind_text = kind->as_string();
if (kind_text == "once") {
entry.kind = ScheduleKind::Once;
} else if (kind_text == "interval") {
entry.kind = ScheduleKind::Interval;
} else if (kind_text == "cron") {
entry.kind = ScheduleKind::Cron;
} else {
return false;
}
const JValue* payload = root.find("payload");
if (payload != nullptr) {
entry.payload = common::json_encode(*payload);
}
const JValue* at_ms = root.find("at_ms");
const JValue* interval = root.find("interval_secs");
if (at_ms != nullptr) {
entry.at_ms = at_ms->as_int(0);
}
if (interval != nullptr) {
entry.interval_secs = static_cast<long>(interval->as_int(60));
}
const JValue* cron = root.find("cron");
if (cron != nullptr) {
entry.cron = cron->as_string();
}
const JValue* missed = root.find("missed");
if (missed != nullptr && missed->as_string() == "run-once") {
entry.missed = MissedPolicy::RunOnce;
}
const JValue* enabled = root.find("enabled");
if (enabled != nullptr) {
entry.enabled = enabled->as_bool(true);
}
const JValue* retries = root.find("max_retries");
if (retries != nullptr) {
entry.max_retries = static_cast<int>(retries->as_int(5));
}
out = std::move(entry);
return true;
}
void Scheduler::put(ScheduleEntry entry) {
if (entry.key.empty() || entry.key.size() > 256) {
throw TaskError("Scheduler: bad entry key");
}
if (entry.task_name.empty()) {
throw TaskError("Scheduler: entry needs a task name");
}
CronExpr cron;
if (entry.kind == ScheduleKind::Cron) {
std::string error;
if (!CronExpr::parse(entry.cron, cron, error)) {
throw TaskError("Scheduler: " + error);
}
}
if (entry.kind == ScheduleKind::Interval &&
entry.interval_secs < common::Caps::kSchedulerMinIntervalSecs) {
entry.interval_secs = common::Caps::kSchedulerMinIntervalSecs; // S7-5 floor
}
std::lock_guard<std::mutex> lock(mutex_);
const bool is_new = entries_.find(entry.key) == entries_.end();
entries_[entry.key] = entry;
if (entry.kind == ScheduleKind::Cron) {
cron_cache_[entry.key] = cron;
} else {
cron_cache_.erase(entry.key);
}
if (is_new) {
registered_at_[entry.key] = clock_.wall_ms();
}
if (log_) {
log_->append("{\"op\":\"put\",\"entry\":" + encode_entry(entry) + "}", clock_.wall_ms());
}
observer_->event("tasks.schedule.put", entry.key);
}
bool Scheduler::remove(const std::string& key) {
std::lock_guard<std::mutex> lock(mutex_);
cron_cache_.erase(key);
registered_at_.erase(key);
const bool removed = entries_.erase(key) > 0;
if (removed && log_) {
log_->append("{\"op\":\"remove\",\"key\":\"" + common::json_escape(key) + "\"}",
clock_.wall_ms());
}
return removed;
}
std::vector<ScheduleEntry> Scheduler::entries() const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<ScheduleEntry> out;
for (const auto& [_, entry] : entries_) {
out.push_back(entry);
}
return out;
}
bool Scheduler::is_leader() const {
std::lock_guard<std::mutex> lock(mutex_);
return election_.is_leader();
}
std::string Scheduler::leader() const {
std::lock_guard<std::mutex> lock(mutex_);
return leader_id_;
}
void Scheduler::tick_once() {
std::lock_guard<std::mutex> lock(mutex_);
const std::int64_t now = clock_.wall_ms();
if (!elect_locked(now)) {
return; // follower: stay passive
}
fire_due_locked(now);
}
int Scheduler::run(const std::function<bool()>& stop_requested) {
while (!stop_requested || !stop_requested()) {
tick_once();
clock_.sleep_ms(options_.tick_ms);
}
return 0;
}
bool Scheduler::elect_locked(std::int64_t now_wall) {
(void)now_wall;
const bool was = election_.is_leader();
const bool now_leader = election_.tick();
if (now_leader && !was) {
leader_id_ = options_.instance_id;
observer_->event("tasks.schedule.leader_won", options_.instance_id);
} else if (!now_leader && was) {
observer_->event("tasks.schedule.leader_lost", options_.instance_id);
}
if (!now_leader) {
const std::string current = election_.leader();
if (!current.empty()) {
leader_id_ = current;
}
}
return now_leader;
}
std::int64_t Scheduler::last_fire_locked(const std::string& key, std::int64_t fallback) {
std::string stored;
if (kv_.get(options_.leader_key + "/fired/" + key, stored)) {
try {
return std::stoll(stored);
} catch (...) {
return fallback;
}
}
return fallback;
}
void Scheduler::record_fire_locked(const std::string& key, std::int64_t now_wall, long ttl_secs) {
kv_.set(options_.leader_key + "/fired/" + key, std::to_string(now_wall), ttl_secs);
}
void Scheduler::fire_locked(const ScheduleEntry& entry, const std::string& fire_ref,
std::int64_t now_wall) {
TaskEnvelope env;
env.name = entry.task_name;
env.payload = entry.payload;
env.max_retries = entry.max_retries;
env.idempotency_key = "sched:" + entry.key + ":" + fire_ref;
try {
const std::string id = queue_.enqueue(std::move(env));
observer_->event("tasks.schedule.fired", entry.key + " -> " + id);
(void)now_wall;
} catch (const TaskError& e) {
observer_->event("tasks.schedule.fire_failed", entry.key + ": " + e.what());
}
}
void Scheduler::fire_due_locked(std::int64_t now_wall) {
for (const auto& [key, entry] : entries_) {
if (!entry.enabled) {
continue;
}
const std::int64_t registered =
registered_at_.count(key) ? registered_at_[key] : now_wall;
switch (entry.kind) {
case ScheduleKind::Once: {
// Fired markers persist, so a reboot never refires a Once entry.
const std::int64_t last = last_fire_locked(key, -1);
if (last >= 0) {
break;
}
if (now_wall < entry.at_ms) {
break;
}
const bool missed = (now_wall - entry.at_ms > 60000);
if (missed && entry.missed == MissedPolicy::Skip) {
record_fire_locked(key, now_wall, 86400);
observer_->event("tasks.schedule.missed", key);
break;
}
fire_locked(entry, "once", now_wall);
record_fire_locked(key, now_wall, 86400);
break;
}
case ScheduleKind::Interval: {
const std::int64_t span = entry.interval_secs * 1000;
const std::int64_t last = last_fire_locked(key, registered);
if (now_wall - last < span) {
break;
}
// At most one fire per tick; last=now so followers taking over
// never refire this window (V7-4).
fire_locked(entry, std::to_string(now_wall / span), now_wall);
record_fire_locked(key, now_wall, entry.interval_secs * 2 + 60);
break;
}
case ScheduleKind::Cron: {
auto cit = cron_cache_.find(key);
if (cit == cron_cache_.end()) {
break;
}
const std::int64_t minute = (now_wall / 60000) * 60000;
const std::int64_t last = last_fire_locked(key, registered);
if (last >= minute && cit->second.matches(now_wall)) {
break; // already fired this minute
}
if (cit->second.matches(now_wall)) {
fire_locked(entry, "m" + std::to_string(minute / 60000), now_wall);
record_fire_locked(key, now_wall, 90000);
} else if (entry.missed == MissedPolicy::RunOnce) {
const std::int64_t next = cit->second.next_after(last);
if (next >= 0 && next <= now_wall) {
fire_locked(entry, "m" + std::to_string((next / 60000)), now_wall);
record_fire_locked(key, now_wall, 90000);
}
}
break;
}
}
}
}
} // namespace molo::tasks
+114
View File
@@ -0,0 +1,114 @@
#pragma once
#include <cstdint>
#include <functional>
#include <map>
#include <memory>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "operability/coord/election.h"
#include "supervisor/bus/segment_log.hpp"
#include "supervisor/common/clock.hpp"
#include "supervisor/common/kv_store.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/tasks/cron.hpp"
#include "supervisor/tasks/task_envelope.hpp"
#include "supervisor/tasks/task_queue.hpp"
namespace molo::tasks {
// What to do with a fire that was missed during an outage or failover.
enum class MissedPolicy {
Skip, // drop the missed fire, wait for the next one
RunOnce, // fire once now to cover the missed window
};
enum class ScheduleKind {
Once, // one delayed run at at_ms
Interval, // fixed interval, minimum 60s (S7-5)
Cron, // 5-field UTC cron expression
};
struct ScheduleEntry {
std::string key;
std::string task_name;
std::string payload = "{}";
ScheduleKind kind = ScheduleKind::Interval;
std::int64_t at_ms = 0; // Once
long interval_secs = 60; // Interval
std::string cron; // Cron
MissedPolicy missed = MissedPolicy::Skip;
bool enabled = true;
int max_retries = 5;
};
struct SchedulerOptions {
std::string instance_id;
std::string leader_key = "molo/scheduler/leader";
std::string durable_dir; // empty means in-memory entries; set for E7-6
long tick_ms = 1000;
long ttl_secs = 10;
long renew_secs = 3;
};
// Delayed plus periodic scheduling (S7-5). The leader enqueues due entries
// into the TaskQueue; followers stay passive. Leadership is a lease in the
// shared KvStore (10s TTL, 3s renewal). Last-fire records live in the same
// shared store so a new leader never double-fires, and every fire carries
// an idempotency key so even a brief dual-leader overlap cannot enqueue
// twice (the queue dedupes). Dual-run during network partitions is accepted
// with documented bounds (O7-4): at most the overlap window, deduped by key.
class Scheduler {
public:
Scheduler(TaskQueue& queue, common::KvStore& kv, common::Clock& clock,
std::shared_ptr<common::Observer> observer, const SchedulerOptions& options);
// Adds or replaces an entry. Throws TaskError on bad cron or interval.
void put(ScheduleEntry entry);
bool remove(const std::string& key);
std::vector<ScheduleEntry> entries() const;
bool is_leader() const;
std::string leader() const;
// One tick: run leader election, then fire due entries when leader.
// Deterministic under FrozenClock; production uses run().
void tick_once();
// Ticks until stop_requested() is true. Returns 0.
int run(const std::function<bool()>& stop_requested);
std::string encode_entry(const ScheduleEntry& entry) const;
bool decode_entry(const std::string& text, ScheduleEntry& out) const;
private:
void replay();
bool elect_locked(std::int64_t now_wall);
void fire_due_locked(std::int64_t now_wall);
std::int64_t last_fire_locked(const std::string& key, std::int64_t fallback);
void record_fire_locked(const std::string& key, std::int64_t now_wall, long ttl_secs);
void fire_locked(const ScheduleEntry& entry, const std::string& fire_ref,
std::int64_t now_wall);
TaskQueue& queue_;
common::KvStore& kv_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
SchedulerOptions options_;
std::unique_ptr<bus::SegmentLog> log_;
mutable std::mutex mutex_;
std::map<std::string, ScheduleEntry> entries_;
std::map<std::string, CronExpr> cron_cache_;
std::map<std::string, std::int64_t> registered_at_;
std::string leader_id_;
// Leadership moved onto molo::coord::LeaderElection (operability part);
// the scheduler keeps its public surface and event names. Declared after
// observer_/options_ so the init list can wire them in.
common::NullObserver fallback_observer_;
molo::coord::LeaderElection election_;
};
} // namespace molo::tasks
+150
View File
@@ -0,0 +1,150 @@
#include "supervisor/tasks/task_envelope.hpp"
#include <chrono>
#include <iomanip>
#include <random>
#include <sstream>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/json.hpp"
namespace molo::tasks {
std::string to_string(TaskState state) {
switch (state) {
case TaskState::Queued: return "queued";
case TaskState::Leased: return "leased";
case TaskState::Running: return "running";
case TaskState::Succeeded: return "succeeded";
case TaskState::FailedRetryable: return "failed-retryable";
case TaskState::FailedPermanent: return "failed-permanent";
case TaskState::DeadLettered: return "dead-lettered";
case TaskState::Cancelled: return "cancelled";
}
return "queued";
}
bool parse_state(const std::string& text, TaskState& out) {
if (text == "queued") {
out = TaskState::Queued;
} else if (text == "leased") {
out = TaskState::Leased;
} else if (text == "running") {
out = TaskState::Running;
} else if (text == "succeeded") {
out = TaskState::Succeeded;
} else if (text == "failed-retryable") {
out = TaskState::FailedRetryable;
} else if (text == "failed-permanent") {
out = TaskState::FailedPermanent;
} else if (text == "dead-lettered") {
out = TaskState::DeadLettered;
} else if (text == "cancelled") {
out = TaskState::Cancelled;
} else {
return false;
}
return true;
}
std::string TaskEnvelope::encode() const {
using common::JValue;
JValue::Object obj;
obj["id"] = JValue(id);
obj["name"] = JValue(name);
JValue payload_value;
if (!common::json_decode(payload, payload_value)) {
payload_value = JValue(payload); // non-JSON payloads ride as strings
}
obj["payload"] = payload_value;
obj["attempts"] = JValue(attempts);
obj["max_retries"] = JValue(max_retries);
if (run_at_ms.has_value()) {
obj["run_at"] = JValue(common::iso8601(*run_at_ms));
} else {
obj["run_at"] = JValue(nullptr);
}
obj["timeout_soft_ms"] = JValue(static_cast<long long>(timeout_soft_ms));
obj["timeout_hard_ms"] = JValue(static_cast<long long>(timeout_hard_ms));
obj["idempotency_key"] = JValue(idempotency_key);
obj["enqueued_at"] = JValue(common::iso8601(enqueued_at_ms));
obj["state"] = JValue(to_string(state));
return common::json_encode(JValue(obj));
}
bool TaskEnvelope::decode(const std::string& text, TaskEnvelope& out) {
using common::JValue;
JValue root;
if (!common::json_decode(text, root) || !root.is_object()) {
return false;
}
const JValue* id = root.find("id");
const JValue* name = root.find("name");
if (id == nullptr || name == nullptr || !id->is_string() || !name->is_string()) {
return false;
}
TaskEnvelope env;
env.id = id->as_string();
env.name = name->as_string();
const JValue* payload = root.find("payload");
if (payload != nullptr) {
env.payload = common::json_encode(*payload);
}
const JValue* attempts = root.find("attempts");
const JValue* max_retries = root.find("max_retries");
if (attempts != nullptr) {
env.attempts = static_cast<int>(attempts->as_int(0));
}
if (max_retries != nullptr) {
env.max_retries = static_cast<int>(max_retries->as_int(5));
}
const JValue* run_at = root.find("run_at");
if (run_at != nullptr && run_at->is_string()) {
const std::int64_t parsed = common::parse_iso8601(run_at->as_string());
if (parsed < 0) {
return false;
}
env.run_at_ms = parsed;
}
const JValue* soft = root.find("timeout_soft_ms");
const JValue* hard = root.find("timeout_hard_ms");
if (soft != nullptr) {
env.timeout_soft_ms = static_cast<long>(soft->as_int(30000));
}
if (hard != nullptr) {
env.timeout_hard_ms = static_cast<long>(hard->as_int(60000));
}
const JValue* key = root.find("idempotency_key");
if (key != nullptr) {
env.idempotency_key = key->as_string();
}
const JValue* enqueued = root.find("enqueued_at");
if (enqueued != nullptr && enqueued->is_string()) {
env.enqueued_at_ms = common::parse_iso8601(enqueued->as_string());
}
const JValue* state = root.find("state");
if (state != nullptr && state->is_string()) {
if (!parse_state(state->as_string(), env.state)) {
return false;
}
}
out = std::move(env);
return true;
}
std::string TaskEnvelope::generate_id() {
// UUIDv7 shape: 48-bit unix ms plus 74 random bits, hex with dashes.
static thread_local std::mt19937_64 rng{std::random_device{}()};
const auto now = std::chrono::system_clock::now();
const auto ms = static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count());
const std::uint64_t hi = (ms << 16) | (rng() & 0xFFFFu);
const std::uint64_t lo = rng();
std::ostringstream out;
out << std::hex << std::setfill('0') << std::setw(12) << (hi >> 16) << "-" << std::setw(4)
<< (hi & 0xFFFFu) << "-7" << std::setw(3) << ((lo >> 48) & 0xFFFu) << "-" << std::setw(4)
<< (((lo >> 32) & 0x3FFFu) | 0x8000u) << "-" << std::setw(12) << (lo & 0xFFFFFFFFFFFFULL);
return out.str();
}
} // namespace molo::tasks
+57
View File
@@ -0,0 +1,57 @@
#pragma once
#include <cstdint>
#include <optional>
#include <stdexcept>
#include <string>
namespace molo::tasks {
class TaskError : public std::runtime_error {
public:
explicit TaskError(const std::string& message) : std::runtime_error(message) {}
};
// Task lifecycle states (S7-3).
enum class TaskState {
Queued,
Leased,
Running,
Succeeded,
FailedRetryable,
FailedPermanent,
DeadLettered,
Cancelled,
};
std::string to_string(TaskState state);
bool parse_state(const std::string& text, TaskState& out);
// Immutable job record (S7-1). The only type that crosses queue boundaries.
// JSON form doubles as the on-disk format for the durable queue:
//
// {"id":"...","name":"billing.charge","payload":{...},"attempts":0,
// "max_retries":5,"run_at":"iso8601|null","timeout_soft_ms":30000,
// "timeout_hard_ms":60000,"idempotency_key":"...","enqueued_at":"iso8601"}
//
// The payload is an opaque JSON object string (max 1 MiB); the queue never
// interprets it, handlers do.
struct TaskEnvelope {
std::string id;
std::string name;
std::string payload = "{}";
int attempts = 0;
int max_retries = 5;
std::optional<std::int64_t> run_at_ms; // null means run as soon as possible
long timeout_soft_ms = 30000;
long timeout_hard_ms = 60000;
std::string idempotency_key;
std::int64_t enqueued_at_ms = 0;
TaskState state = TaskState::Queued;
std::string encode() const;
static bool decode(const std::string& text, TaskEnvelope& out);
static std::string generate_id();
};
} // namespace molo::tasks
+14
View File
@@ -0,0 +1,14 @@
#include "supervisor/tasks/task_handler.hpp"
namespace molo::tasks {
std::string to_string(TaskOutcome outcome) {
switch (outcome) {
case TaskOutcome::Success: return "success";
case TaskOutcome::RetryableFailure: return "retryable";
case TaskOutcome::PermanentFailure: return "permanent";
}
return "retryable";
}
} // namespace molo::tasks
+40
View File
@@ -0,0 +1,40 @@
#pragma once
#include <atomic>
#include <string>
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/tasks/task_envelope.hpp"
namespace molo::tasks {
// Handler outcome. RetryableFailure requeues with backoff; PermanentFailure
// dead-letters immediately. Throwing from run() counts as retryable.
enum class TaskOutcome {
Success,
RetryableFailure,
PermanentFailure,
};
std::string to_string(TaskOutcome outcome);
// Per-execution context handed to handlers.
struct TaskContext {
const TaskEnvelope& envelope;
common::Clock& clock;
std::shared_ptr<common::Observer> observer;
std::atomic<bool>& cancel_requested; // set at the soft timeout (R7, S7-3)
};
// Base class for all task handlers. Every handler MUST be idempotent:
// delivery is at-least-once, so run() may execute more than once with the
// same task id after worker loss or lease expiry (R7-2). Handlers run on
// worker threads, never on HTTP event loop threads (R7-5).
class TaskHandler {
public:
virtual ~TaskHandler() = default;
virtual TaskOutcome run(TaskContext& ctx) = 0;
};
} // namespace molo::tasks
+683
View File
@@ -0,0 +1,683 @@
#include "supervisor/tasks/task_queue.hpp"
#include <thread>
#include "supervisor/common/caps.hpp"
#include "supervisor/common/json.hpp"
namespace molo::tasks {
namespace {
std::string truncate_reason(const std::string& reason) {
return reason.size() > 1024 ? reason.substr(0, 1024) : reason;
}
} // namespace
TaskQueue::TaskQueue(const QueueOptions& options, common::Clock& clock,
std::shared_ptr<common::Observer> observer)
: options_(options),
clock_(clock),
observer_(std::move(observer)),
retry_(options.retry),
dead_letters_(clock_, observer_) {
if (options_.max_depth < 1) {
options_.max_depth = 1;
}
if (!options_.durable_dir.empty()) {
log_ = std::make_unique<bus::SegmentLog>(
std::filesystem::path(options_.durable_dir) / "task_queue.log");
replay();
}
}
TaskQueue::~TaskQueue() {
shutdown();
}
void TaskQueue::replay() {
// Rebuilds live state from the op log. Leases are never persisted, so a
// task leased at crash replays as queued and is redelivered (E7-6).
std::uint64_t seq = 1;
while (true) {
auto records = log_->read_from(seq, 512, 4 * 1024 * 1024);
if (records.empty()) {
break;
}
for (const auto& record : records) {
common::JValue root;
if (!common::json_decode(record.data, root)) {
continue; // corrupt op skipped loudly below
}
const common::JValue* op = root.find("op");
if (op == nullptr) {
continue;
}
const std::string op_name = op->as_string();
if (op_name == "enqueue") {
const common::JValue* env = root.find("env");
if (env == nullptr) {
continue;
}
TaskEnvelope envelope;
if (!TaskEnvelope::decode(common::json_encode(*env), envelope)) {
continue;
}
envelope.state = TaskState::Queued;
const std::int64_t run_at = envelope.run_at_ms.value_or(0);
records_[envelope.id] = Record{envelope, ++order_};
ready_[{run_at, order_}] = envelope.id;
} else if (op_name == "terminal") {
const common::JValue* id = root.find("id");
if (id != nullptr) {
terminal_locked(id->as_string());
}
} else if (op_name == "requeue") {
const common::JValue* id = root.find("id");
if (id == nullptr) {
continue;
}
auto it = records_.find(id->as_string());
if (it == records_.end()) {
continue;
}
const common::JValue* attempts = root.find("attempts");
const common::JValue* run_at = root.find("run_at_ms");
if (attempts != nullptr) {
it->second.envelope.attempts = static_cast<int>(attempts->as_int(0));
}
if (run_at != nullptr && !run_at->is_null()) {
it->second.envelope.run_at_ms = run_at->as_int(0);
} else {
it->second.envelope.run_at_ms.reset();
}
it->second.envelope.state = TaskState::Queued;
it->second.order = ++order_;
const std::int64_t due = it->second.envelope.run_at_ms.value_or(0);
ready_[{due, it->second.order}] = it->first;
}
}
seq = records.back().seq + 1;
}
observer_->event("tasks.queue.replayed", "live=" + std::to_string(records_.size()));
}
void TaskQueue::log_op(const std::string& op) {
if (log_) {
log_->append(op, clock_.wall_ms());
}
}
std::string TaskQueue::enqueue(TaskEnvelope envelope) {
if (envelope.name.empty() || envelope.name.size() > 256) {
throw TaskError("TaskQueue: bad task name");
}
if (envelope.payload.size() > common::Caps::kTaskPayloadBytes) {
throw TaskError("TaskQueue: payload over 1 MiB cap");
}
if (envelope.timeout_soft_ms <= 0) {
envelope.timeout_soft_ms = 30000;
}
if (envelope.timeout_hard_ms <= 0) {
envelope.timeout_hard_ms = 60000;
}
if (envelope.timeout_hard_ms < envelope.timeout_soft_ms) {
envelope.timeout_hard_ms = envelope.timeout_soft_ms;
}
if (envelope.max_retries < 0) {
envelope.max_retries = 0;
}
std::lock_guard<std::mutex> lock(mutex_);
if (!accepting_) {
throw TaskError("TaskQueue: not accepting new tasks (draining)");
}
if (!envelope.idempotency_key.empty()) {
const std::string existing = dedupe_lookup_locked(envelope.idempotency_key);
if (!existing.empty()) {
return existing;
}
}
if (envelope.id.empty()) {
envelope.id = TaskEnvelope::generate_id();
}
if (records_.size() >= options_.max_depth) {
throw TaskError("TaskQueue: queue depth cap reached");
}
envelope.enqueued_at_ms = clock_.wall_ms();
envelope.attempts = 0;
envelope.state = TaskState::Queued;
const std::string id = envelope.id;
const std::int64_t run_at = envelope.run_at_ms.value_or(0);
records_[id] = Record{envelope, ++order_};
ready_[{run_at, order_}] = id;
if (!envelope.idempotency_key.empty()) {
dedupe_store_locked(envelope.idempotency_key, id);
}
log_op("{\"op\":\"enqueue\",\"env\":" + envelope.encode() + "}");
observer_->counter("molo_tasks_enqueued_total", 1);
observer_->event("tasks.enqueued", id + " " + envelope.name);
cv_.notify_one();
return id;
}
bool TaskQueue::cancel(const std::string& id) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = records_.find(id);
if (it == records_.end()) {
return false;
}
if (lease_by_task_.find(id) != lease_by_task_.end()) {
return false; // leased tasks finish; cancel only affects queued ones
}
terminal_locked(id);
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(id) + "\"}");
observer_->counter("molo_tasks_cancelled_total", 1);
observer_->event("tasks.cancelled", id);
return true;
}
std::optional<TaskQueue::Lease> TaskQueue::lease(long wait_ms) {
std::unique_lock<std::mutex> lock(mutex_);
const std::int64_t deadline = clock_.steady_ms() + wait_ms;
while (true) {
if (shutdown_) {
return std::nullopt;
}
reap_expired_locked();
if (auto got = try_lease_locked()) {
return got;
}
if (wait_ms <= 0) {
return std::nullopt;
}
const std::int64_t now = clock_.steady_ms();
if (now >= deadline) {
return std::nullopt;
}
// Wake early when the earliest delayed task becomes due.
long wait_for = static_cast<long>(deadline - now);
if (!ready_.empty()) {
const std::int64_t due_wall = ready_.begin()->first.first;
const std::int64_t now_wall = clock_.wall_ms();
if (due_wall > now_wall) {
const long until_due = static_cast<long>(due_wall - now_wall);
if (until_due < wait_for) {
wait_for = until_due;
}
}
}
cv_.wait_for(lock, std::chrono::milliseconds(wait_for));
}
}
std::optional<TaskQueue::Lease> TaskQueue::try_lease_locked() {
if (ready_.empty()) {
return std::nullopt;
}
const std::int64_t now_wall = clock_.wall_ms();
auto it = ready_.begin();
if (it->first.first > now_wall) {
return std::nullopt; // earliest task still delayed
}
const std::string id = it->second;
ready_.erase(it);
auto record = records_.find(id);
if (record == records_.end()) {
return std::nullopt;
}
record->second.envelope.state = TaskState::Leased;
const std::string lease_id =
"L" + std::to_string(clock_.steady_ms()) + "-" + std::to_string(++lease_seq_);
LeaseInfo info{id, clock_.steady_ms() + options_.lease_ms};
leases_[lease_id] = info;
lease_by_task_[id] = lease_id;
observer_->counter("molo_tasks_leased_total", 1);
return Lease{record->second.envelope, lease_id, info.deadline_steady_ms};
}
void TaskQueue::reap_expired_locked() {
const std::int64_t now = clock_.steady_ms();
std::vector<std::string> expired;
for (const auto& [lease_id, info] : leases_) {
if (info.deadline_steady_ms <= now) {
expired.push_back(lease_id);
}
}
for (const auto& lease_id : expired) {
auto lit = leases_.find(lease_id);
if (lit == leases_.end()) {
continue;
}
const std::string task_id = lit->second.task_id;
leases_.erase(lit);
lease_by_task_.erase(task_id);
auto rit = records_.find(task_id);
if (rit == records_.end()) {
continue;
}
// Lease expiry without ack returns the task to queued with attempts
// incremented (S7-3). A late ack from the original worker is ignored
// because the lease is gone; duplicates are idempotent by contract.
rit->second.envelope.attempts += 1;
++expired_total_;
observer_->counter("molo_tasks_lease_expired_total", 1);
observer_->event("tasks.lease_expired", task_id);
const RetryPolicy policy(RetrySpec{options_.retry.base_ms, options_.retry.cap_ms,
options_.retry.jitter_ms,
rit->second.envelope.max_retries});
if (policy.exhausted(rit->second.envelope.attempts)) {
TaskEnvelope env = rit->second.envelope;
terminal_locked(task_id);
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
dead_letters_.push(std::move(env), "lease-expired-retries-exhausted");
++failed_total_;
} else {
rit->second.envelope.state = TaskState::Queued;
rit->second.envelope.run_at_ms.reset(); // redeliver immediately
rit->second.order = ++order_;
ready_[{0, rit->second.order}] = task_id;
log_op("{\"op\":\"requeue\",\"id\":\"" + common::json_escape(task_id) + "\",\"attempts\":" +
std::to_string(rit->second.envelope.attempts) + ",\"run_at_ms\":null}");
cv_.notify_one();
}
}
}
bool TaskQueue::heartbeat(const std::string& lease_id) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = leases_.find(lease_id);
if (it == leases_.end()) {
return false;
}
if (it->second.deadline_steady_ms <= clock_.steady_ms()) {
return false;
}
it->second.deadline_steady_ms = clock_.steady_ms() + options_.lease_ms;
return true;
}
bool TaskQueue::ack(const std::string& lease_id) {
std::lock_guard<std::mutex> lock(mutex_);
auto it = leases_.find(lease_id);
if (it == leases_.end()) {
return false;
}
const std::string task_id = it->second.task_id;
leases_.erase(it);
lease_by_task_.erase(task_id);
terminal_locked(task_id);
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
++succeeded_total_;
observer_->counter("molo_tasks_succeeded_total", 1);
observer_->event("tasks.succeeded", task_id);
return true;
}
bool TaskQueue::fail(const std::string& lease_id, bool retryable, const std::string& reason) {
std::lock_guard<std::mutex> lock(mutex_);
if (leases_.find(lease_id) == leases_.end()) {
return false;
}
fail_locked(lease_id, retryable, reason);
return true;
}
void TaskQueue::fail_locked(const std::string& lease_id, bool retryable,
const std::string& reason) {
auto it = leases_.find(lease_id);
if (it == leases_.end()) {
return;
}
const std::string task_id = it->second.task_id;
leases_.erase(it);
lease_by_task_.erase(task_id);
auto rit = records_.find(task_id);
if (rit == records_.end()) {
return;
}
rit->second.envelope.attempts += 1;
const int max_retries = rit->second.envelope.max_retries;
const RetryPolicy policy(
RetrySpec{options_.retry.base_ms, options_.retry.cap_ms, options_.retry.jitter_ms,
max_retries});
const bool drain = !accepting_;
if (!retryable || policy.exhausted(rit->second.envelope.attempts) || drain) {
TaskEnvelope env = rit->second.envelope;
terminal_locked(task_id);
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
std::string why = truncate_reason(reason);
if (drain && retryable && !policy.exhausted(rit->second.envelope.attempts)) {
why += " (draining)";
}
dead_letters_.push(std::move(env), why);
++failed_total_;
observer_->counter("molo_tasks_failed_total", 1);
} else {
const long delay = policy.delay_for_attempt(rit->second.envelope.attempts - 1);
const std::int64_t run_at = clock_.wall_ms() + delay;
rit->second.envelope.run_at_ms = run_at;
rit->second.envelope.state = TaskState::Queued;
rit->second.order = ++order_;
ready_[{run_at, rit->second.order}] = task_id;
log_op("{\"op\":\"requeue\",\"id\":\"" + common::json_escape(task_id) + "\",\"attempts\":" +
std::to_string(rit->second.envelope.attempts) +
",\"run_at_ms\":" + std::to_string(run_at) + "}");
++retried_total_;
observer_->counter("molo_tasks_retried_total", 1);
observer_->event("tasks.retrying", task_id + " in " + std::to_string(delay) + "ms");
cv_.notify_one();
}
}
void TaskQueue::terminal_locked(const std::string& task_id) {
auto rit = records_.find(task_id);
if (rit == records_.end()) {
return;
}
for (auto it = ready_.begin(); it != ready_.end(); ++it) {
if (it->second == task_id) {
ready_.erase(it);
break;
}
}
records_.erase(rit);
}
void TaskQueue::execute(const Lease& leased, TaskRegistry& registry,
const std::function<bool()>& stop_requested) {
struct ExecState {
std::atomic<bool> done{false};
std::atomic<bool> cancel{false};
std::mutex mutex;
std::condition_variable cv;
TaskOutcome outcome = TaskOutcome::RetryableFailure;
std::string error;
};
{
std::lock_guard<std::mutex> lock(mutex_);
auto rit = records_.find(leased.envelope.id);
if (rit != records_.end()) {
rit->second.envelope.state = TaskState::Running;
}
++running_;
observer_->event("tasks.started", leased.envelope.id + " " + leased.envelope.name);
}
std::shared_ptr<TaskHandler> handler = registry.create(leased.envelope.name);
if (!handler) {
std::lock_guard<std::mutex> lock(mutex_);
--running_;
auto it = lease_by_task_.find(leased.envelope.id);
if (it != lease_by_task_.end()) {
fail_locked(it->second, false, "unknown-handler:" + leased.envelope.name);
}
return;
}
auto state = std::make_shared<ExecState>();
TaskEnvelope env_copy = leased.envelope; // child thread owns only copies
std::thread child([state, handler, env_copy, this]() mutable {
TaskContext ctx{env_copy, clock_, observer_, state->cancel};
try {
const TaskOutcome outcome = handler->run(ctx);
std::lock_guard<std::mutex> lock(state->mutex);
state->outcome = outcome;
} catch (const std::exception& e) {
std::lock_guard<std::mutex> lock(state->mutex);
state->outcome = TaskOutcome::RetryableFailure;
state->error = truncate_reason(std::string("exception:") + e.what());
} catch (...) {
std::lock_guard<std::mutex> lock(state->mutex);
state->outcome = TaskOutcome::RetryableFailure;
state->error = "exception:unknown";
}
{
std::lock_guard<std::mutex> lock(state->mutex);
state->done = true;
}
state->cv.notify_all();
});
// Waiter: heartbeat the lease, fire the soft timeout cooperatively, and
// enforce the hard timeout by failing the lease and recycling the worker.
const std::int64_t start = clock_.steady_ms();
const long soft = env_copy.timeout_soft_ms;
const long hard = env_copy.timeout_hard_ms;
bool soft_fired = false;
bool hard_fired = false;
std::int64_t last_heartbeat = start;
{
std::unique_lock<std::mutex> lock(state->mutex);
while (!state->done) {
state->cv.wait_for(lock, std::chrono::milliseconds(5));
const std::int64_t now = clock_.steady_ms();
if (stop_requested && stop_requested()) {
state->cancel = true;
}
if (!soft_fired && now - start >= soft) {
soft_fired = true;
state->cancel = true;
observer_->event("tasks.soft_timeout", env_copy.id);
}
if (now - last_heartbeat >= options_.heartbeat_ms / 2) {
last_heartbeat = now;
heartbeat(leased.lease_id);
}
if (now - start >= hard) {
hard_fired = true;
state->cancel = true;
observer_->event("tasks.hard_timeout", env_copy.id);
break;
}
}
}
if (!hard_fired) {
TaskOutcome outcome;
std::string error;
{
std::lock_guard<std::mutex> lock(state->mutex);
outcome = state->outcome;
error = state->error;
}
child.join();
std::lock_guard<std::mutex> lock(mutex_);
--running_;
switch (outcome) {
case TaskOutcome::Success: {
auto it = leases_.find(leased.lease_id);
if (it != leases_.end()) {
const std::string task_id = it->second.task_id;
leases_.erase(it);
lease_by_task_.erase(task_id);
terminal_locked(task_id);
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
++succeeded_total_;
observer_->counter("molo_tasks_succeeded_total", 1);
observer_->event("tasks.succeeded", task_id);
}
break;
}
case TaskOutcome::RetryableFailure:
fail_locked(leased.lease_id, true, error.empty() ? "handler-retryable" : error);
break;
case TaskOutcome::PermanentFailure:
fail_locked(leased.lease_id, false, error.empty() ? "handler-permanent" : error);
break;
}
return;
}
// Hard timeout: the lease fails retryable right away so the work is not
// stuck; the orphaned child gets a grace period, then is detached and
// the worker recycled (R7, S7-3/S7-4). The orphan owns only heap state
// and copies, so detaching is memory-safe; a late finish cannot ack
// because the lease is already gone.
{
std::lock_guard<std::mutex> lock(mutex_);
--running_;
fail_locked(leased.lease_id, true, "hard-timeout");
}
{
std::unique_lock<std::mutex> lock(state->mutex);
state->cv.wait_for(lock, std::chrono::milliseconds(options_.orphan_grace_ms),
[&] { return state->done.load(); });
}
if (state->done) {
child.join();
} else {
child.detach();
observer_->counter("molo_tasks_orphaned_total", 1);
}
recycle_.store(true);
}
int TaskQueue::run_worker(const std::function<bool()>& stop_requested, TaskRegistry& registry) {
recycle_.store(false);
while (!stop_requested || !stop_requested()) {
std::optional<Lease> got = lease(100);
if (!got) {
continue;
}
execute(*got, registry, stop_requested);
if (recycle_.load()) {
return 2; // hard timeout fired: recycle this worker
}
}
return 0;
}
bool TaskQueue::run_one(TaskRegistry& registry, long lease_wait_ms) {
std::optional<Lease> got = lease(lease_wait_ms);
if (!got) {
return false;
}
execute(*got, registry, {});
return true;
}
bool TaskQueue::recycle_requested() const {
return recycle_.load();
}
QueueStats TaskQueue::stats() const {
std::lock_guard<std::mutex> lock(mutex_);
QueueStats stats;
stats.queued = ready_.size();
stats.leased = leases_.size();
stats.running = running_;
stats.succeeded_total = succeeded_total_;
stats.failed_total = failed_total_;
stats.retried_total = retried_total_;
stats.expired_total = expired_total_;
return stats;
}
std::optional<TaskEnvelope> TaskQueue::get(const std::string& id) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = records_.find(id);
if (it == records_.end()) {
return std::nullopt;
}
return it->second.envelope;
}
std::vector<TaskEnvelope> TaskQueue::list_queued(std::size_t max) const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<TaskEnvelope> out;
for (const auto& [_, id] : ready_) {
if (out.size() >= max) {
break;
}
auto it = records_.find(id);
if (it != records_.end()) {
out.push_back(it->second.envelope);
}
}
return out;
}
DeadLetterQueue& TaskQueue::dead_letters() {
return dead_letters_;
}
void TaskQueue::set_accepting(bool accepting) {
std::lock_guard<std::mutex> lock(mutex_);
accepting_ = accepting;
}
bool TaskQueue::accepting() const {
std::lock_guard<std::mutex> lock(mutex_);
return accepting_;
}
std::size_t TaskQueue::in_flight() const {
std::lock_guard<std::mutex> lock(mutex_);
return leases_.size() + running_;
}
void TaskQueue::abandon_all_to_dlq(const std::string& reason) {
std::lock_guard<std::mutex> lock(mutex_);
for (const auto& [_, record] : records_) {
TaskEnvelope env = record.envelope;
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(env.id) + "\"}");
dead_letters_.push(std::move(env), reason);
}
records_.clear();
ready_.clear();
leases_.clear();
lease_by_task_.clear();
observer_->event("tasks.abandoned", reason);
}
void TaskQueue::shutdown() {
std::lock_guard<std::mutex> lock(mutex_);
shutdown_ = true;
cv_.notify_all();
}
void TaskQueue::compact() {
std::lock_guard<std::mutex> lock(mutex_);
if (!log_) {
return;
}
// Rewrite the log from live state: one enqueue op per live record.
const std::filesystem::path file =
std::filesystem::path(options_.durable_dir) / "task_queue.log";
log_.reset();
std::filesystem::remove(file);
log_ = std::make_unique<bus::SegmentLog>(file);
for (const auto& [_, record] : records_) {
log_->append("{\"op\":\"enqueue\",\"env\":" + record.envelope.encode() + "}",
clock_.wall_ms());
}
}
std::string TaskQueue::dedupe_lookup_locked(const std::string& key) {
auto it = idempotency_.find(key);
if (it == idempotency_.end()) {
return "";
}
if (it->second.expires_wall_ms <= clock_.wall_ms()) {
idempotency_.erase(it);
return "";
}
// The key stays reserved for the full window even after the task
// finished: duplicates return the original id without re-enqueue (S7-1).
return it->second.task_id;
}
void TaskQueue::dedupe_store_locked(const std::string& key, const std::string& task_id) {
const std::int64_t expires =
clock_.wall_ms() + common::Caps::kIdempotencyWindowHours * 3600 * 1000;
idempotency_[key] = IdemEntry{task_id, expires};
idempotency_order_.push_back(key);
while (idempotency_order_.size() > options_.max_depth) {
idempotency_.erase(idempotency_order_.front());
idempotency_order_.pop_front();
}
}
} // namespace molo::tasks
+158
View File
@@ -0,0 +1,158 @@
#pragma once
#include <atomic>
#include <condition_variable>
#include <cstdint>
#include <deque>
#include <functional>
#include <map>
#include <memory>
#include <mutex>
#include <optional>
#include <string>
#include <utility>
#include <vector>
#include "supervisor/bus/segment_log.hpp"
#include "supervisor/common/clock.hpp"
#include "supervisor/common/observer.hpp"
#include "supervisor/tasks/dead_letter_queue.hpp"
#include "supervisor/tasks/retry_policy.hpp"
#include "supervisor/tasks/task_envelope.hpp"
#include "supervisor/tasks/task_registry.hpp"
namespace molo::tasks {
struct QueueStats {
std::size_t queued = 0;
std::size_t leased = 0;
std::size_t running = 0;
long succeeded_total = 0;
long failed_total = 0;
long retried_total = 0;
long expired_total = 0;
};
struct QueueOptions {
std::string durable_dir; // empty means in-memory only (E7-2); set for E7-6
long lease_ms = 60000;
long heartbeat_ms = 15000;
std::size_t max_depth = 100000;
RetrySpec retry;
long orphan_grace_ms = 5000; // join grace after a hard timeout before detach
};
// At-least-once task queue with ack-after-success (BP-3, S7-3).
//
// Producer side: enqueue() validates, dedupes idempotency keys, and makes
// the envelope leasable once its run_at arrives. Consumer side: lease()
// hands out one envelope with an ack deadline; ack() completes it;
// fail() retries with backoff or dead-letters; heartbeat() renews the
// deadline. Leases that expire without ack return to queued with attempts
// incremented. Redelivery is normal; handlers must be idempotent (R7-2).
//
// run_worker() is the off-event-loop consumer loop (R7-5) driven by
// supervisor worker threads: it leases, executes with soft/hard timeouts,
// and returns 2 when the worker should be recycled after a hard timeout.
class TaskQueue {
public:
struct Lease {
TaskEnvelope envelope;
std::string lease_id;
std::int64_t deadline_steady_ms = 0;
};
TaskQueue(const QueueOptions& options, common::Clock& clock,
std::shared_ptr<common::Observer> observer);
~TaskQueue();
TaskQueue(const TaskQueue&) = delete;
TaskQueue& operator=(const TaskQueue&) = delete;
// Returns the task id. Duplicate idempotency_key within 24h returns the
// existing id without re-enqueue. Throws TaskError on cap violations.
std::string enqueue(TaskEnvelope envelope);
bool cancel(const std::string& id);
// Leases one due envelope, waiting up to wait_ms (0 means non-blocking).
// Returns nullopt on timeout or after shutdown().
std::optional<Lease> lease(long wait_ms);
bool heartbeat(const std::string& lease_id);
bool ack(const std::string& lease_id);
bool fail(const std::string& lease_id, bool retryable, const std::string& reason);
// Worker entry points.
int run_worker(const std::function<bool()>& stop_requested, TaskRegistry& registry);
bool run_one(TaskRegistry& registry, long lease_wait_ms);
bool recycle_requested() const;
QueueStats stats() const;
std::optional<TaskEnvelope> get(const std::string& id) const;
std::vector<TaskEnvelope> list_queued(std::size_t max) const;
DeadLetterQueue& dead_letters();
// Graceful shutdown support (R7-9).
void set_accepting(bool accepting);
bool accepting() const;
std::size_t in_flight() const; // leased plus running
void abandon_all_to_dlq(const std::string& reason);
void shutdown(); // wakes all lease waiters; run_worker loops still exit via stop
void compact(); // rewrites the durable log from live state
private:
struct Record {
TaskEnvelope envelope;
std::uint64_t order = 0;
};
struct LeaseInfo {
std::string task_id;
std::int64_t deadline_steady_ms = 0;
};
struct IdemEntry {
std::string task_id;
std::int64_t expires_wall_ms = 0;
};
void replay();
void log_op(const std::string& op);
void reap_expired_locked();
std::optional<Lease> try_lease_locked();
void fail_locked(const std::string& lease_id, bool retryable, const std::string& reason);
void terminal_locked(const std::string& task_id);
void execute(const Lease& leased, TaskRegistry& registry,
const std::function<bool()>& stop_requested);
std::string dedupe_lookup_locked(const std::string& key);
void dedupe_store_locked(const std::string& key, const std::string& task_id);
QueueOptions options_;
common::Clock& clock_;
std::shared_ptr<common::Observer> observer_;
RetryPolicy retry_;
DeadLetterQueue dead_letters_;
std::unique_ptr<bus::SegmentLog> log_;
mutable std::mutex mutex_;
std::condition_variable cv_;
std::map<std::string, Record> records_; // live tasks by id
std::map<std::pair<std::int64_t, std::uint64_t>, std::string> ready_; // (run_at, order) -> id
std::map<std::string, LeaseInfo> leases_; // lease_id -> info
std::map<std::string, std::string> lease_by_task_; // task_id -> lease_id
std::map<std::string, IdemEntry> idempotency_;
std::deque<std::string> idempotency_order_;
std::uint64_t order_ = 0;
std::uint64_t lease_seq_ = 0;
std::size_t running_ = 0;
long succeeded_total_ = 0;
long failed_total_ = 0;
long retried_total_ = 0;
long expired_total_ = 0;
bool accepting_ = true;
bool shutdown_ = false;
std::atomic<bool> recycle_{false};
};
} // namespace molo::tasks
+36
View File
@@ -0,0 +1,36 @@
#include "supervisor/tasks/task_registry.hpp"
namespace molo::tasks {
void TaskRegistry::register_handler(const std::string& name, Factory factory) {
if (name.empty() || name.size() > 256 || !factory) {
throw TaskError("TaskRegistry: bad handler registration");
}
std::lock_guard<std::mutex> lock(mutex_);
factories_[name] = std::move(factory);
}
bool TaskRegistry::has(const std::string& name) const {
std::lock_guard<std::mutex> lock(mutex_);
return factories_.find(name) != factories_.end();
}
std::unique_ptr<TaskHandler> TaskRegistry::create(const std::string& name) const {
std::lock_guard<std::mutex> lock(mutex_);
auto it = factories_.find(name);
if (it == factories_.end()) {
return nullptr;
}
return it->second();
}
std::vector<std::string> TaskRegistry::names() const {
std::lock_guard<std::mutex> lock(mutex_);
std::vector<std::string> out;
for (const auto& [name, _] : factories_) {
out.push_back(name);
}
return out;
}
} // namespace molo::tasks
+31
View File
@@ -0,0 +1,31 @@
#pragma once
#include <functional>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <vector>
#include "supervisor/tasks/task_handler.hpp"
namespace molo::tasks {
// Name-to-handler factory registry. Workers resolve handler instances per
// execution through this registry so handlers stay stateless and the queue
// never depends on concrete handler types.
class TaskRegistry {
public:
using Factory = std::function<std::unique_ptr<TaskHandler>()>;
void register_handler(const std::string& name, Factory factory);
bool has(const std::string& name) const;
std::unique_ptr<TaskHandler> create(const std::string& name) const;
std::vector<std::string> names() const;
private:
mutable std::mutex mutex_;
std::map<std::string, Factory> factories_;
};
} // namespace molo::tasks