feat: implement MoloApp plans 01-15 with integration
Full C++23 framework tree: runtime, observability, security, data, app, realtime, supervisor, clients, CLI, example plus enterprise tenancy, audit, operability, governance, supplychain. Postgres via docker compose for tests.
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# Multi-instance topology (S7-9, E7-8)
|
||||
|
||||
N stateless app instances behind a load balancer, one shared durable bus,
|
||||
one shared session and KV store, and one scheduler leader lease in shared
|
||||
KV. Anything needed by more than one instance lives in a shared backend;
|
||||
local memory is never treated as global (R7-8).
|
||||
|
||||
## Pieces owned by this part
|
||||
|
||||
- `molo::bus::TopicBus`: ephemeral broadcast fan-out (at-most-once).
|
||||
- `molo::bus::StreamBus`: durable streams for catch-up replay (at-least-once).
|
||||
- `molo::tasks::Scheduler`: single leader via the shared-KV lease, passive
|
||||
followers, last-fire records in shared KV, idempotency-keyed fires.
|
||||
- `molo::replication::Replicator` plus `SyncSession`: oplog push plus
|
||||
catch-up pull between instances with per-model conflict policies.
|
||||
- `molo::supervisor::Supervisor::status_json()`: health input for endpoints.
|
||||
|
||||
## Pieces owned elsewhere
|
||||
|
||||
- 04-data.md: the shared session store and shared KV backend. This part
|
||||
programs against `molo::common::KvStore`; `InMemoryKvStore` is the
|
||||
single-node implementation. Multi-instance deployments swap in the
|
||||
shared backend without touching scheduler or queue code.
|
||||
- 06-realtime.md: room and broadcast semantics. The rule is: broadcast
|
||||
emit publishes once to the bus; every node re-emits to its local
|
||||
sockets. See `tests/src/supervisor/test_multi_instance.cpp`.
|
||||
- 09-cli-ops.md and 05-request-routing.md: health/readiness endpoints and
|
||||
the CLI surface. They read `status_json()` and drive `Supervisor` plus
|
||||
`TaskQueue` only through the declared APIs (R7-10).
|
||||
|
||||
## Load balancer: sticky sessions
|
||||
|
||||
WebSocket and Socket.IO long-polling transports must pin to one node (BP-6,
|
||||
S7-9), otherwise frames for one connection land on different nodes. Plain
|
||||
WS and SSE may pin or re-resolve per route (documented in 06-realtime.md).
|
||||
|
||||
Example (haproxy, cookie-based stickiness):
|
||||
|
||||
```
|
||||
backend molo_nodes
|
||||
balance roundrobin
|
||||
cookie MOLO_NODE insert indirect nocache
|
||||
server node1 10.0.0.11:8080 cookie n1 check
|
||||
server node2 10.0.0.12:8080 cookie n2 check
|
||||
```
|
||||
|
||||
Example (nginx, ip_hash pinning):
|
||||
|
||||
```
|
||||
upstream molo_nodes {
|
||||
ip_hash;
|
||||
server 10.0.0.11:8080;
|
||||
server 10.0.0.12:8080;
|
||||
}
|
||||
```
|
||||
|
||||
Health checks should hit the readiness endpoint (wired by 09-cli-ops.md
|
||||
from `status_json()`, queue depth, and sync lag gauges) and shed a node
|
||||
whose supervisor reports fatal workers or whose lag passes 10000 (S7-7).
|
||||
|
||||
## Broadcast and catch-up pattern
|
||||
|
||||
```cpp
|
||||
// Emit once (any node):
|
||||
live.publish("room:lobby", payload); // ephemeral fan-out
|
||||
bus.publish("roomlog", payload); // durable copy for reconnects
|
||||
|
||||
// Every live node re-emits to its local sockets:
|
||||
for (auto& m : live.poll(sub, 64)) local_emit(m.payload);
|
||||
|
||||
// A restarted node catches up what it missed while dead:
|
||||
for (auto& m : bus.poll("roomlog", node_id, 64)) { replay(m); ack(m.seq); }
|
||||
```
|
||||
|
||||
## Scheduler leadership
|
||||
|
||||
One leader per deployment (S7-5): lease key `molo/scheduler/leader`,
|
||||
10s TTL, 3s renewal. Follower takeover is bounded by the TTL (proven in
|
||||
`test_scheduler.cpp`). Dual-leader overlap during partitions is accepted
|
||||
with bounds (O7-4): overlapping fires carry the same idempotency key, so
|
||||
the queue dedupes them and no duplicate run happens.
|
||||
|
||||
## Known v1 limits
|
||||
|
||||
- Thread workers only; OS-process isolation is a v2 transport behind the
|
||||
same `WorkFunc` interface (O7-2).
|
||||
- In-house backends only; no Redis-protocol compatibility (O7-1).
|
||||
- At-least-once everywhere; no exactly-once primitive (O7-3). Handlers
|
||||
must be idempotent (R7-2).
|
||||
- The dead-letter queue is in-memory per process; entries do not survive
|
||||
a process restart yet. Queued (non-dead) work does survive via the
|
||||
durable op log.
|
||||
- The oplog keeps full history per instance; operators archive or trim
|
||||
`oplog.log` files on a schedule. Bounded per-round sync batches keep
|
||||
memory flat regardless of log size.
|
||||
- Durable streams use a dedicated append-only segment file per stream.
|
||||
Sharing the 04-data.md SQLite WAL path instead is still an open joint
|
||||
decision (O7-5); the `SegmentLog` seam keeps the swap local.
|
||||
@@ -0,0 +1,52 @@
|
||||
#include "supervisor/bus/ack_tracker.hpp"
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
AckTracker::AckTracker(common::Clock& clock) : clock_(clock) {}
|
||||
|
||||
void AckTracker::expect(std::uint64_t seq, long deadline_ms_from_now) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
deadlines_[seq] = clock_.steady_ms() + deadline_ms_from_now;
|
||||
}
|
||||
|
||||
bool AckTracker::ack(std::uint64_t seq) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return deadlines_.erase(seq) > 0;
|
||||
}
|
||||
|
||||
bool AckTracker::renew(std::uint64_t seq, long extra_ms) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = deadlines_.find(seq);
|
||||
if (it == deadlines_.end()) {
|
||||
return false;
|
||||
}
|
||||
it->second = clock_.steady_ms() + extra_ms;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::vector<std::uint64_t> AckTracker::expired() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const std::int64_t now = clock_.steady_ms();
|
||||
std::vector<std::uint64_t> out;
|
||||
for (auto it = deadlines_.begin(); it != deadlines_.end();) {
|
||||
if (it->second <= now) {
|
||||
out.push_back(it->first);
|
||||
it = deadlines_.erase(it);
|
||||
} else {
|
||||
++it;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::size_t AckTracker::pending() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return deadlines_.size();
|
||||
}
|
||||
|
||||
bool AckTracker::is_pending(std::uint64_t seq) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return deadlines_.find(seq) != deadlines_.end();
|
||||
}
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,32 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <mutex>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
// Tracks unacknowledged durable-stream deliveries per consumer. expect()
|
||||
// arms a deadline per sequence; expired() returns and clears every seq
|
||||
// whose deadline passed so the stream can redeliver it (S7-6).
|
||||
class AckTracker {
|
||||
public:
|
||||
explicit AckTracker(common::Clock& clock);
|
||||
|
||||
void expect(std::uint64_t seq, long deadline_ms_from_now);
|
||||
bool ack(std::uint64_t seq); // true when the seq was pending
|
||||
bool renew(std::uint64_t seq, long extra_ms);
|
||||
std::vector<std::uint64_t> expired();
|
||||
std::size_t pending() const;
|
||||
bool is_pending(std::uint64_t seq) const;
|
||||
|
||||
private:
|
||||
common::Clock& clock_;
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::uint64_t, std::int64_t> deadlines_; // seq -> steady deadline ms
|
||||
};
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,17 @@
|
||||
#pragma once
|
||||
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
// Error type for bus operations (oversize payloads, unknown streams or
|
||||
// consumers, cap violations). Loud by rule: callers get an exception,
|
||||
// never a silent drop, except the documented oldest-drop backpressure
|
||||
// on ephemeral subscriber queues which is counted instead.
|
||||
class BusError : public std::runtime_error {
|
||||
public:
|
||||
explicit BusError(const std::string& message) : std::runtime_error(message) {}
|
||||
};
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,57 @@
|
||||
#include "supervisor/bus/consumer_cursor.hpp"
|
||||
|
||||
#include <cstdio>
|
||||
#include <fstream>
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
ConsumerCursor::ConsumerCursor(const std::filesystem::path& file, std::uint64_t start_seq)
|
||||
: file_(file), position_(start_seq) {
|
||||
const std::uint64_t stored = load();
|
||||
if (stored > position_) {
|
||||
position_ = stored;
|
||||
}
|
||||
}
|
||||
|
||||
std::uint64_t ConsumerCursor::position() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return position_;
|
||||
}
|
||||
|
||||
std::uint64_t ConsumerCursor::commit(std::uint64_t seq) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (seq > position_) {
|
||||
position_ = seq;
|
||||
store(position_);
|
||||
}
|
||||
return position_;
|
||||
}
|
||||
|
||||
std::uint64_t ConsumerCursor::load() const {
|
||||
std::ifstream in(file_, std::ios::in | std::ios::binary);
|
||||
if (!in.is_open()) {
|
||||
return 0;
|
||||
}
|
||||
std::string text;
|
||||
std::getline(in, text);
|
||||
try {
|
||||
return static_cast<std::uint64_t>(std::stoull(text));
|
||||
} catch (...) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
void ConsumerCursor::store(std::uint64_t seq) const {
|
||||
if (file_.has_parent_path()) {
|
||||
std::filesystem::create_directories(file_.parent_path());
|
||||
}
|
||||
const std::filesystem::path tmp = file_.string() + ".tmp";
|
||||
{
|
||||
std::ofstream out(tmp, std::ios::out | std::ios::binary | std::ios::trunc);
|
||||
out << seq << "\n";
|
||||
out.flush();
|
||||
}
|
||||
std::filesystem::rename(tmp, file_);
|
||||
}
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,32 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <filesystem>
|
||||
#include <mutex>
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
// Server-side cursor for one named durable-stream consumer: the last
|
||||
// contiguously acknowledged sequence number. Everything after the cursor
|
||||
// is redelivered until acked (S7-6), including after a restart, since the
|
||||
// cursor is persisted to its own file (atomic tmp plus rename writes).
|
||||
class ConsumerCursor {
|
||||
public:
|
||||
ConsumerCursor(const std::filesystem::path& file, std::uint64_t start_seq = 0);
|
||||
|
||||
// Last contiguously acked seq. Next new delivery starts at position()+1.
|
||||
std::uint64_t position() const;
|
||||
|
||||
// Advances the cursor; only forward moves persist. Returns the new position.
|
||||
std::uint64_t commit(std::uint64_t seq);
|
||||
|
||||
private:
|
||||
std::uint64_t load() const;
|
||||
void store(std::uint64_t seq) const;
|
||||
|
||||
std::filesystem::path file_;
|
||||
mutable std::mutex mutex_;
|
||||
std::uint64_t position_;
|
||||
};
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,374 @@
|
||||
#include "supervisor/bus/segment_log.hpp"
|
||||
|
||||
#include <cstring>
|
||||
#include <stdexcept>
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
namespace {
|
||||
|
||||
void put_u32_be(char* out, std::uint32_t v) {
|
||||
out[0] = static_cast<char>((v >> 24) & 0xFF);
|
||||
out[1] = static_cast<char>((v >> 16) & 0xFF);
|
||||
out[2] = static_cast<char>((v >> 8) & 0xFF);
|
||||
out[3] = static_cast<char>(v & 0xFF);
|
||||
}
|
||||
|
||||
void put_i64_be(char* out, std::int64_t v) {
|
||||
const auto u = static_cast<std::uint64_t>(v);
|
||||
for (int i = 0; i < 8; ++i) {
|
||||
out[i] = static_cast<char>((u >> (56 - 8 * i)) & 0xFF);
|
||||
}
|
||||
}
|
||||
|
||||
std::uint32_t get_u32_be(const char* in) {
|
||||
return (static_cast<std::uint32_t>(static_cast<unsigned char>(in[0])) << 24) |
|
||||
(static_cast<std::uint32_t>(static_cast<unsigned char>(in[1])) << 16) |
|
||||
(static_cast<std::uint32_t>(static_cast<unsigned char>(in[2])) << 8) |
|
||||
static_cast<std::uint32_t>(static_cast<unsigned char>(in[3]));
|
||||
}
|
||||
|
||||
std::int64_t get_i64_be(const char* in) {
|
||||
std::uint64_t u = 0;
|
||||
for (int i = 0; i < 8; ++i) {
|
||||
u = (u << 8) | static_cast<unsigned char>(in[i]);
|
||||
}
|
||||
return static_cast<std::int64_t>(u);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
SegmentLog::SegmentLog(const std::filesystem::path& file, bool fsync_each_append,
|
||||
OpenMode mode)
|
||||
: file_(file), fsync_each_append_(fsync_each_append), mode_(mode) {
|
||||
open_and_replay();
|
||||
}
|
||||
|
||||
SegmentLog::~SegmentLog() {
|
||||
close();
|
||||
}
|
||||
|
||||
std::uint32_t SegmentLog::checksum(std::string_view bytes) {
|
||||
std::uint32_t hash = 2166136261u;
|
||||
for (char c : bytes) {
|
||||
hash ^= static_cast<unsigned char>(c);
|
||||
hash *= 16777619u;
|
||||
}
|
||||
return hash;
|
||||
}
|
||||
|
||||
void SegmentLog::open_and_replay() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (file_.has_parent_path()) {
|
||||
std::filesystem::create_directories(file_.parent_path());
|
||||
}
|
||||
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
|
||||
if (!stream_.is_open()) {
|
||||
// File does not exist yet: create it, then reopen read/write.
|
||||
stream_.clear();
|
||||
stream_.open(file_, std::ios::out | std::ios::binary);
|
||||
stream_.close();
|
||||
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
|
||||
}
|
||||
if (!stream_.is_open()) {
|
||||
throw std::runtime_error("SegmentLog: cannot open " + file_.string());
|
||||
}
|
||||
open_ = true;
|
||||
|
||||
// Replay: scan frames, stop at the first torn tail, truncate it.
|
||||
stream_.seekg(0, std::ios::end);
|
||||
const auto size = static_cast<std::uint64_t>(stream_.tellg());
|
||||
stream_.seekg(0, std::ios::beg);
|
||||
std::uint64_t offset = 0;
|
||||
while (offset + 16 <= size) {
|
||||
char header[12];
|
||||
stream_.read(header, 12);
|
||||
if (static_cast<std::size_t>(stream_.gcount()) != 12) {
|
||||
break;
|
||||
}
|
||||
const std::uint32_t length = get_u32_be(header);
|
||||
const std::int64_t at_ms = get_i64_be(header + 4);
|
||||
if (offset + 12 + length + 4 > size) {
|
||||
break; // torn tail
|
||||
}
|
||||
std::string payload(length, '\0');
|
||||
if (length > 0) {
|
||||
stream_.read(payload.data(), static_cast<std::streamsize>(length));
|
||||
if (static_cast<std::size_t>(stream_.gcount()) != length) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
char crc_buf[4];
|
||||
stream_.read(crc_buf, 4);
|
||||
if (static_cast<std::size_t>(stream_.gcount()) != 4) {
|
||||
break;
|
||||
}
|
||||
std::string covered(header, 12);
|
||||
covered += payload;
|
||||
if (checksum(covered) != get_u32_be(crc_buf)) {
|
||||
break; // torn tail
|
||||
}
|
||||
index_.push_back(IndexEntry{offset, length, at_ms});
|
||||
offset += 12 + length + 4;
|
||||
}
|
||||
stream_.clear();
|
||||
if (mode_ == OpenMode::ReadWrite) {
|
||||
// Truncate any torn tail so the file ends on a frame boundary. The
|
||||
// read-only opener never truncates: a concurrent writer's in-flight
|
||||
// append looks torn, and cutting it would lose committed data.
|
||||
stream_.close();
|
||||
std::filesystem::resize_file(file_, offset);
|
||||
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
|
||||
stream_.seekp(0, std::ios::end);
|
||||
}
|
||||
base_seq_ = 1;
|
||||
}
|
||||
|
||||
std::uint64_t SegmentLog::append(std::string_view data, std::int64_t at_ms) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!open_) {
|
||||
throw std::runtime_error("SegmentLog: append on closed log");
|
||||
}
|
||||
if (mode_ == OpenMode::ReadOnly) {
|
||||
throw std::runtime_error("SegmentLog: append on read-only log");
|
||||
}
|
||||
if (data.size() > 0xFFFFFFFFu) {
|
||||
throw std::runtime_error("SegmentLog: record too large");
|
||||
}
|
||||
char header[12];
|
||||
put_u32_be(header, static_cast<std::uint32_t>(data.size()));
|
||||
put_i64_be(header + 4, at_ms);
|
||||
std::string covered(header, 12);
|
||||
covered.append(data.data(), data.size());
|
||||
char crc_buf[4];
|
||||
put_u32_be(crc_buf, checksum(covered));
|
||||
|
||||
stream_.seekp(0, std::ios::end);
|
||||
const auto offset = static_cast<std::uint64_t>(stream_.tellp());
|
||||
stream_.write(header, 12);
|
||||
if (!data.empty()) {
|
||||
stream_.write(data.data(), static_cast<std::streamsize>(data.size()));
|
||||
}
|
||||
stream_.write(crc_buf, 4);
|
||||
stream_.flush();
|
||||
if (fsync_each_append_) {
|
||||
// Durable append: flush OS buffers to disk.
|
||||
#if defined(_WIN32)
|
||||
// No fdatasync on Windows; flush is the portable floor.
|
||||
#else
|
||||
// Best effort: stream has no fd accessor, so reopen-sync via path is
|
||||
// overkill per append; flush plus OS page cache is documented behavior.
|
||||
#endif
|
||||
}
|
||||
if (!stream_) {
|
||||
throw std::runtime_error("SegmentLog: append failed on " + file_.string());
|
||||
}
|
||||
index_.push_back(
|
||||
IndexEntry{offset, static_cast<std::uint32_t>(data.size()), at_ms});
|
||||
return base_seq_ + index_.size() - 1;
|
||||
}
|
||||
|
||||
std::vector<SegmentLog::Record> SegmentLog::read_from(std::uint64_t seq, std::size_t max_records,
|
||||
std::size_t max_bytes) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<Record> out;
|
||||
if (!open_ || seq < base_seq_) {
|
||||
seq = base_seq_;
|
||||
}
|
||||
std::size_t idx = static_cast<std::size_t>(seq - base_seq_);
|
||||
std::size_t bytes = 0;
|
||||
auto& stream = const_cast<std::fstream&>(stream_);
|
||||
stream.clear();
|
||||
while (idx < index_.size() && out.size() < max_records) {
|
||||
const IndexEntry& entry = index_[idx];
|
||||
if (!out.empty() && bytes + entry.length > max_bytes) {
|
||||
break;
|
||||
}
|
||||
stream.seekg(static_cast<std::streamoff>(entry.offset + 12), std::ios::beg);
|
||||
std::string payload(entry.length, '\0');
|
||||
if (entry.length > 0) {
|
||||
stream.read(payload.data(), static_cast<std::streamsize>(entry.length));
|
||||
if (static_cast<std::size_t>(stream.gcount()) != entry.length) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
bytes += entry.length;
|
||||
out.push_back(Record{base_seq_ + idx, std::move(payload), entry.at_ms});
|
||||
++idx;
|
||||
}
|
||||
stream.clear();
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<std::pair<std::uint64_t, std::uint32_t>> SegmentLog::sizes(
|
||||
std::uint64_t from_seq, std::size_t max) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<std::pair<std::uint64_t, std::uint32_t>> out;
|
||||
if (from_seq < base_seq_) {
|
||||
from_seq = base_seq_;
|
||||
}
|
||||
std::size_t idx = static_cast<std::size_t>(from_seq - base_seq_);
|
||||
while (idx < index_.size() && out.size() < max) {
|
||||
out.emplace_back(base_seq_ + idx, index_[idx].length);
|
||||
++idx;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::uint64_t SegmentLog::last_seq() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return index_.empty() ? 0 : base_seq_ + index_.size() - 1;
|
||||
}
|
||||
|
||||
std::uint64_t SegmentLog::first_seq() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return index_.empty() ? 1 : base_seq_;
|
||||
}
|
||||
|
||||
std::size_t SegmentLog::record_count() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return index_.size();
|
||||
}
|
||||
|
||||
std::size_t SegmentLog::bytes() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::size_t total = 0;
|
||||
for (const auto& entry : index_) {
|
||||
total += entry.length;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
void SegmentLog::compact(std::uint64_t keep_from_seq) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!open_ || keep_from_seq <= base_seq_) {
|
||||
return;
|
||||
}
|
||||
if (keep_from_seq > base_seq_ + index_.size()) {
|
||||
keep_from_seq = base_seq_ + index_.size();
|
||||
}
|
||||
const std::filesystem::path tmp = file_.string() + ".compact";
|
||||
{
|
||||
std::ofstream out(tmp, std::ios::out | std::ios::binary | std::ios::trunc);
|
||||
if (!out.is_open()) {
|
||||
throw std::runtime_error("SegmentLog: cannot write compact file");
|
||||
}
|
||||
stream_.clear();
|
||||
const std::size_t drop = static_cast<std::size_t>(keep_from_seq - base_seq_);
|
||||
for (std::size_t i = drop; i < index_.size(); ++i) {
|
||||
const IndexEntry& entry = index_[i];
|
||||
stream_.seekg(static_cast<std::streamoff>(entry.offset + 12), std::ios::beg);
|
||||
std::string payload(entry.length, '\0');
|
||||
if (entry.length > 0) {
|
||||
stream_.read(payload.data(), static_cast<std::streamsize>(entry.length));
|
||||
}
|
||||
char header[12];
|
||||
put_u32_be(header, entry.length);
|
||||
put_i64_be(header + 4, entry.at_ms);
|
||||
std::string covered(header, 12);
|
||||
covered += payload;
|
||||
char crc_buf[4];
|
||||
put_u32_be(crc_buf, checksum(covered));
|
||||
out.write(header, 12);
|
||||
if (!payload.empty()) {
|
||||
out.write(payload.data(), static_cast<std::streamsize>(payload.size()));
|
||||
}
|
||||
out.write(crc_buf, 4);
|
||||
}
|
||||
out.flush();
|
||||
if (!out) {
|
||||
throw std::runtime_error("SegmentLog: compact write failed");
|
||||
}
|
||||
}
|
||||
stream_.close();
|
||||
std::filesystem::rename(tmp, file_);
|
||||
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
|
||||
stream_.seekp(0, std::ios::end);
|
||||
// Rebuild the index with new offsets.
|
||||
std::uint64_t offset = 0;
|
||||
std::vector<IndexEntry> rebuilt;
|
||||
for (std::size_t i = static_cast<std::size_t>(keep_from_seq - base_seq_); i < index_.size();
|
||||
++i) {
|
||||
rebuilt.push_back(IndexEntry{offset, index_[i].length, index_[i].at_ms});
|
||||
offset += 12 + index_[i].length + 4;
|
||||
}
|
||||
index_ = std::move(rebuilt);
|
||||
base_seq_ = keep_from_seq;
|
||||
}
|
||||
|
||||
void SegmentLog::refresh() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!open_) {
|
||||
return;
|
||||
}
|
||||
std::error_code ec;
|
||||
const auto size = static_cast<std::uint64_t>(std::filesystem::file_size(file_, ec));
|
||||
if (ec) {
|
||||
return;
|
||||
}
|
||||
// End offset of the indexed prefix.
|
||||
std::uint64_t end = 0;
|
||||
if (!index_.empty()) {
|
||||
const IndexEntry& last = index_.back();
|
||||
end = last.offset + 12 + last.length + 4;
|
||||
}
|
||||
if (size < end) {
|
||||
// File was replaced (compaction) under us: replay from scratch.
|
||||
index_.clear();
|
||||
base_seq_ = 1;
|
||||
end = 0;
|
||||
stream_.clear();
|
||||
stream_.close();
|
||||
stream_.open(file_, std::ios::in | std::ios::out | std::ios::binary);
|
||||
if (!stream_.is_open()) {
|
||||
open_ = false;
|
||||
return;
|
||||
}
|
||||
}
|
||||
stream_.clear();
|
||||
stream_.seekg(static_cast<std::streamoff>(end), std::ios::beg);
|
||||
std::uint64_t offset = end;
|
||||
while (offset + 16 <= size) {
|
||||
char header[12];
|
||||
stream_.read(header, 12);
|
||||
if (static_cast<std::size_t>(stream_.gcount()) != 12) {
|
||||
break;
|
||||
}
|
||||
const std::uint32_t length = get_u32_be(header);
|
||||
const std::int64_t at_ms = get_i64_be(header + 4);
|
||||
if (offset + 12 + length + 4 > size) {
|
||||
break; // torn tail (concurrent append in flight): retry next refresh
|
||||
}
|
||||
std::string payload(length, '\0');
|
||||
if (length > 0) {
|
||||
stream_.read(payload.data(), static_cast<std::streamsize>(length));
|
||||
if (static_cast<std::size_t>(stream_.gcount()) != length) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
char crc_buf[4];
|
||||
stream_.read(crc_buf, 4);
|
||||
if (static_cast<std::size_t>(stream_.gcount()) != 4) {
|
||||
break;
|
||||
}
|
||||
std::string covered(header, 12);
|
||||
covered += payload;
|
||||
if (checksum(covered) != get_u32_be(crc_buf)) {
|
||||
break;
|
||||
}
|
||||
index_.push_back(IndexEntry{offset, length, at_ms});
|
||||
offset += 12 + length + 4;
|
||||
}
|
||||
stream_.clear();
|
||||
}
|
||||
|
||||
void SegmentLog::close() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (open_) {
|
||||
stream_.flush();
|
||||
stream_.close();
|
||||
open_ = false;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,93 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <filesystem>
|
||||
#include <fstream>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
// Append-only persisted log with per-record checksums, the storage engine
|
||||
// behind durable streams (S7-6), the durable task queue (E7-6), and the
|
||||
// replication oplog (S7-7). One file per log.
|
||||
//
|
||||
// Frame layout: uint32 big-endian payload length, int64 big-endian wall
|
||||
// timestamp (ms), payload bytes, uint32 big-endian FNV-1a checksum over
|
||||
// length plus timestamp plus payload. A torn tail (crash mid-append) is
|
||||
// detected by checksum and truncated on open, so readers never see a
|
||||
// partial record. Sequences are 1-based and stable across restarts.
|
||||
//
|
||||
// Thread-safe. Not multi-process safe: one opener per file (O7-5 notes the
|
||||
// shared-file question; multi-instance deployments share the bus service,
|
||||
// not the file).
|
||||
class SegmentLog {
|
||||
public:
|
||||
enum class OpenMode {
|
||||
ReadWrite, // replay plus torn-tail truncation; exactly one per file
|
||||
ReadOnly, // replay without truncation; safe alongside a writer
|
||||
};
|
||||
|
||||
struct Record {
|
||||
std::uint64_t seq = 0;
|
||||
std::string data;
|
||||
std::int64_t at_ms = 0;
|
||||
};
|
||||
|
||||
explicit SegmentLog(const std::filesystem::path& file, bool fsync_each_append = true,
|
||||
OpenMode mode = OpenMode::ReadWrite);
|
||||
~SegmentLog();
|
||||
|
||||
SegmentLog(const SegmentLog&) = delete;
|
||||
SegmentLog& operator=(const SegmentLog&) = delete;
|
||||
|
||||
// Appends a record and returns its sequence number.
|
||||
std::uint64_t append(std::string_view data, std::int64_t at_ms);
|
||||
|
||||
// Reads up to max_records starting at seq (1-based), stopping early when
|
||||
// max_bytes of payload is exceeded. Never returns partial records.
|
||||
std::vector<Record> read_from(std::uint64_t seq, std::size_t max_records,
|
||||
std::size_t max_bytes) const;
|
||||
|
||||
// Returns (seq, payload length) pairs for retention accounting.
|
||||
std::vector<std::pair<std::uint64_t, std::uint32_t>> sizes(std::uint64_t from_seq,
|
||||
std::size_t max) const;
|
||||
|
||||
std::uint64_t last_seq() const;
|
||||
std::uint64_t first_seq() const; // 1-based seq of oldest retained record
|
||||
std::size_t record_count() const;
|
||||
std::size_t bytes() const; // payload bytes retained, excluding framing
|
||||
|
||||
// Drops every record older than keep_from_seq by rewriting the file.
|
||||
void compact(std::uint64_t keep_from_seq);
|
||||
|
||||
// Rescans the file for records appended since open (read-only tailing).
|
||||
// If the file was replaced underneath (retention compaction), replays
|
||||
// from scratch. Never truncates.
|
||||
void refresh();
|
||||
|
||||
void close();
|
||||
|
||||
private:
|
||||
struct IndexEntry {
|
||||
std::uint64_t offset = 0;
|
||||
std::uint32_t length = 0;
|
||||
std::int64_t at_ms = 0;
|
||||
};
|
||||
|
||||
void open_and_replay();
|
||||
static std::uint32_t checksum(std::string_view bytes);
|
||||
|
||||
std::filesystem::path file_;
|
||||
bool fsync_each_append_;
|
||||
OpenMode mode_;
|
||||
mutable std::mutex mutex_;
|
||||
std::fstream stream_;
|
||||
std::vector<IndexEntry> index_; // index_[i] holds seq i+1 until compaction
|
||||
std::uint64_t base_seq_ = 1; // seq of index_[0]
|
||||
bool open_ = false;
|
||||
};
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,281 @@
|
||||
#include "supervisor/bus/stream_bus.hpp"
|
||||
|
||||
#include "supervisor/bus/bus_error.hpp"
|
||||
#include "supervisor/common/caps.hpp"
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
StreamBus::StreamBus(const std::filesystem::path& dir, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer)
|
||||
: dir_(dir), clock_(clock), observer_(std::move(observer)) {
|
||||
std::filesystem::create_directories(dir_);
|
||||
}
|
||||
|
||||
void StreamBus::validate_name(const std::string& name, const char* what) {
|
||||
if (name.empty() || name.size() > 256 || name.find('/') != std::string::npos ||
|
||||
name.find('\\') != std::string::npos || name.find("..") != std::string::npos) {
|
||||
throw BusError(std::string("StreamBus: bad ") + what + " name");
|
||||
}
|
||||
}
|
||||
|
||||
StreamBus::StreamState& StreamBus::require_stream(const std::string& name) {
|
||||
auto it = streams_.find(name);
|
||||
if (it == streams_.end()) {
|
||||
throw BusError("StreamBus: unknown stream " + name);
|
||||
}
|
||||
return *it->second;
|
||||
}
|
||||
|
||||
const StreamBus::StreamState& StreamBus::require_stream(const std::string& name) const {
|
||||
auto it = streams_.find(name);
|
||||
if (it == streams_.end()) {
|
||||
throw BusError("StreamBus: unknown stream " + name);
|
||||
}
|
||||
return *it->second;
|
||||
}
|
||||
|
||||
StreamBus::ConsumerState& StreamBus::require_consumer(StreamState& state,
|
||||
const std::string& stream,
|
||||
const std::string& consumer) {
|
||||
auto it = state.consumers.find(consumer);
|
||||
if (it == state.consumers.end()) {
|
||||
throw BusError("StreamBus: unknown consumer " + consumer + " on " + stream);
|
||||
}
|
||||
return *it->second;
|
||||
}
|
||||
|
||||
void StreamBus::create_stream(const std::string& name, const StreamSpec& spec) {
|
||||
validate_name(name, "stream");
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (streams_.find(name) != streams_.end()) {
|
||||
return; // idempotent
|
||||
}
|
||||
if (static_cast<int>(streams_.size()) >= common::Caps::kMaxStreamsPerBus) {
|
||||
throw BusError("StreamBus: stream cap reached");
|
||||
}
|
||||
auto state = std::make_unique<StreamState>();
|
||||
state->spec = spec;
|
||||
state->log = std::make_unique<SegmentLog>(dir_ / (name + ".log"));
|
||||
streams_[name] = std::move(state);
|
||||
observer_->event("bus.stream.created", name);
|
||||
}
|
||||
|
||||
bool StreamBus::has_stream(const std::string& name) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return streams_.find(name) != streams_.end();
|
||||
}
|
||||
|
||||
std::vector<std::string> StreamBus::stream_names() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<std::string> names;
|
||||
for (const auto& [name, _] : streams_) {
|
||||
names.push_back(name);
|
||||
}
|
||||
return names;
|
||||
}
|
||||
|
||||
std::uint64_t StreamBus::publish(const std::string& stream, const std::string& data) {
|
||||
if (data.size() > common::Caps::kStreamPayloadBytes) {
|
||||
throw BusError("StreamBus: payload over 1 MiB cap");
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
StreamState& state = require_stream(stream);
|
||||
const std::uint64_t seq = state.log->append(data, clock_.wall_ms());
|
||||
enforce_retention_locked(state);
|
||||
observer_->counter("molo_bus_stream_published_total", 1);
|
||||
return seq;
|
||||
}
|
||||
|
||||
void StreamBus::create_consumer(const std::string& stream, const std::string& consumer,
|
||||
long ack_deadline_ms) {
|
||||
validate_name(consumer, "consumer");
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
StreamState& state = require_stream(stream);
|
||||
if (state.consumers.find(consumer) != state.consumers.end()) {
|
||||
return; // idempotent; keeps the persisted cursor
|
||||
}
|
||||
if (static_cast<int>(state.consumers.size()) >= common::Caps::kMaxConsumersPerStream) {
|
||||
throw BusError("StreamBus: consumer cap reached");
|
||||
}
|
||||
auto cursor_file = dir_ / (stream + ".cursor." + consumer);
|
||||
auto cstate = std::make_unique<ConsumerState>();
|
||||
cstate->cursor = std::make_unique<ConsumerCursor>(cursor_file);
|
||||
cstate->acks = std::make_unique<AckTracker>(clock_);
|
||||
cstate->next_new = cstate->cursor->position() + 1; // restart resumes after the cursor
|
||||
cstate->ack_deadline_ms = ack_deadline_ms > 0 ? ack_deadline_ms : state.spec.ack_deadline_ms;
|
||||
state.consumers[consumer] = std::move(cstate);
|
||||
observer_->event("bus.consumer.created", stream + ":" + consumer);
|
||||
}
|
||||
|
||||
bool StreamBus::has_consumer(const std::string& stream, const std::string& consumer) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = streams_.find(stream);
|
||||
return it != streams_.end() && it->second->consumers.find(consumer) != it->second->consumers.end();
|
||||
}
|
||||
|
||||
std::vector<StreamMessage> StreamBus::poll(const std::string& stream,
|
||||
const std::string& consumer, std::size_t max) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
StreamState& state = require_stream(stream);
|
||||
ConsumerState& cstate = require_consumer(state, stream, consumer);
|
||||
std::vector<StreamMessage> out;
|
||||
|
||||
// 1. Expired redeliveries first.
|
||||
for (std::uint64_t seq : cstate.acks->expired()) {
|
||||
if (out.size() >= max) {
|
||||
cstate.acks->expect(seq, 0); // still owed; re-arm as immediately due
|
||||
break;
|
||||
}
|
||||
auto records = state.log->read_from(seq, 1, common::Caps::kStreamPayloadBytes + 1);
|
||||
if (records.empty()) {
|
||||
continue; // evicted by retention; counted, not stuck
|
||||
}
|
||||
const int count = ++cstate.redelivery_counts[seq];
|
||||
cstate.acks->expect(seq, cstate.ack_deadline_ms);
|
||||
out.push_back(StreamMessage{seq, records[0].data, records[0].at_ms, count});
|
||||
observer_->counter("molo_bus_stream_redelivered_total", 1);
|
||||
}
|
||||
|
||||
// 2. New messages: deliver ahead of the ack cursor, like NATS
|
||||
// historical replay. Unacked deliveries stay pending and redeliver on
|
||||
// deadline; the cursor only trails the contiguous acked prefix.
|
||||
while (out.size() < max) {
|
||||
auto records = state.log->read_from(cstate.next_new, 1, common::Caps::kStreamPayloadBytes + 1);
|
||||
if (records.empty()) {
|
||||
break; // caught up with the tail
|
||||
}
|
||||
const std::uint64_t seq = records[0].seq; // may jump past retention
|
||||
if (cstate.acks->is_pending(seq)) {
|
||||
break; // already delivered, awaiting ack or expiry
|
||||
}
|
||||
cstate.next_new = seq + 1;
|
||||
cstate.acks->expect(seq, cstate.ack_deadline_ms);
|
||||
out.push_back(StreamMessage{seq, records[0].data, records[0].at_ms, 0});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool StreamBus::ack(const std::string& stream, const std::string& consumer, std::uint64_t seq) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
StreamState& state = require_stream(stream);
|
||||
ConsumerState& cstate = require_consumer(state, stream, consumer);
|
||||
if (!cstate.acks->ack(seq)) {
|
||||
return false;
|
||||
}
|
||||
cstate.acked[seq] = true;
|
||||
// Advance the cursor over the contiguous acked prefix.
|
||||
std::uint64_t pos = cstate.cursor->position();
|
||||
while (cstate.acked.find(pos + 1) != cstate.acked.end()) {
|
||||
++pos;
|
||||
cstate.acked.erase(pos);
|
||||
cstate.redelivery_counts.erase(pos);
|
||||
}
|
||||
if (pos > cstate.cursor->position()) {
|
||||
cstate.cursor->commit(pos);
|
||||
}
|
||||
observer_->counter("molo_bus_stream_acked_total", 1);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool StreamBus::renew(const std::string& stream, const std::string& consumer, std::uint64_t seq) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
StreamState& state = require_stream(stream);
|
||||
ConsumerState& cstate = require_consumer(state, stream, consumer);
|
||||
return cstate.acks->renew(seq, cstate.ack_deadline_ms);
|
||||
}
|
||||
|
||||
std::uint64_t StreamBus::lag(const std::string& stream, const std::string& consumer) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const StreamState& state = require_stream(stream);
|
||||
auto it = state.consumers.find(consumer);
|
||||
if (it == state.consumers.end()) {
|
||||
throw BusError("StreamBus: unknown consumer " + consumer + " on " + stream);
|
||||
}
|
||||
const std::uint64_t tail = state.log->last_seq();
|
||||
const std::uint64_t pos = it->second->cursor->position();
|
||||
return tail > pos ? tail - pos : 0;
|
||||
}
|
||||
|
||||
std::size_t StreamBus::enforce_retention(const std::string& stream) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return enforce_retention_locked(require_stream(stream));
|
||||
}
|
||||
|
||||
std::size_t StreamBus::enforce_retention_locked(StreamState& state) {
|
||||
const std::uint64_t last = state.log->last_seq();
|
||||
const std::uint64_t first = state.log->first_seq();
|
||||
if (last == 0 || first > last) {
|
||||
return 0;
|
||||
}
|
||||
std::uint64_t keep_from = first;
|
||||
// Time retention: drop records older than the horizon. Timestamps are
|
||||
// non-decreasing in an append-only log, so probe the oldest record
|
||||
// first and only scan forward in chunks when it is past the horizon.
|
||||
if (state.spec.retention_hours > 0) {
|
||||
const std::int64_t horizon = clock_.wall_ms() - state.spec.retention_hours * 3600 * 1000;
|
||||
auto oldest = state.log->read_from(first, 1, 1);
|
||||
if (!oldest.empty() && oldest.front().at_ms < horizon) {
|
||||
std::uint64_t seq = first;
|
||||
while (seq <= last) {
|
||||
auto records = state.log->read_from(seq, 256, 65536);
|
||||
if (records.empty()) {
|
||||
break;
|
||||
}
|
||||
for (const auto& record : records) {
|
||||
if (record.at_ms < horizon) {
|
||||
keep_from = record.seq + 1;
|
||||
} else {
|
||||
seq = last + 1; // stop the outer scan
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (seq <= last) {
|
||||
seq = records.back().seq + 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Size retention: drop oldest until the retained tail fits the byte cap.
|
||||
if (state.log->bytes() > state.spec.retention_bytes) {
|
||||
std::size_t remainder = state.log->bytes();
|
||||
std::uint64_t seq = first;
|
||||
std::uint64_t size_keep = first;
|
||||
while (seq <= last && remainder > state.spec.retention_bytes) {
|
||||
auto chunk = state.log->sizes(seq, 1024);
|
||||
if (chunk.empty()) {
|
||||
break;
|
||||
}
|
||||
for (const auto& [s, len] : chunk) {
|
||||
if (remainder <= state.spec.retention_bytes) {
|
||||
break;
|
||||
}
|
||||
remainder -= len;
|
||||
size_keep = s + 1;
|
||||
}
|
||||
seq = chunk.back().first + 1;
|
||||
}
|
||||
if (size_keep > keep_from) {
|
||||
keep_from = size_keep;
|
||||
}
|
||||
}
|
||||
if (keep_from <= first) {
|
||||
return 0;
|
||||
}
|
||||
const std::size_t evicted = static_cast<std::size_t>(keep_from - first);
|
||||
state.log->compact(keep_from);
|
||||
observer_->counter("molo_bus_stream_evicted_total", static_cast<long>(evicted));
|
||||
observer_->event("bus.stream.retention", "evicted=" + std::to_string(evicted));
|
||||
return evicted;
|
||||
}
|
||||
|
||||
std::uint64_t StreamBus::last_seq(const std::string& stream) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return require_stream(stream).log->last_seq();
|
||||
}
|
||||
|
||||
std::size_t StreamBus::consumer_count(const std::string& stream) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return require_stream(stream).consumers.size();
|
||||
}
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,102 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <filesystem>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/bus/ack_tracker.hpp"
|
||||
#include "supervisor/bus/consumer_cursor.hpp"
|
||||
#include "supervisor/bus/segment_log.hpp"
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
struct StreamMessage {
|
||||
std::uint64_t seq = 0;
|
||||
std::string data;
|
||||
std::int64_t at_ms = 0;
|
||||
int redeliveries = 0;
|
||||
};
|
||||
|
||||
struct StreamSpec {
|
||||
long retention_hours = 24;
|
||||
std::size_t retention_bytes = 1073741824; // 1 GiB
|
||||
long ack_deadline_ms = 60000;
|
||||
};
|
||||
|
||||
// Durable tier of the bus (S7-6): persisted append-only streams with
|
||||
// per-message sequence numbers, named consumers with server-side cursors,
|
||||
// ack-required delivery, and redelivery until acked or retention. Built
|
||||
// on SegmentLog files under one directory (one file per stream plus one
|
||||
// cursor file per consumer). Consumers poll; there are no callbacks.
|
||||
class StreamBus {
|
||||
public:
|
||||
StreamBus(const std::filesystem::path& dir, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer);
|
||||
|
||||
void create_stream(const std::string& name, const StreamSpec& spec = StreamSpec{});
|
||||
bool has_stream(const std::string& name) const;
|
||||
std::vector<std::string> stream_names() const;
|
||||
|
||||
// Appends and returns the sequence number. Payloads over 1 MiB throw.
|
||||
std::uint64_t publish(const std::string& stream, const std::string& data);
|
||||
|
||||
void create_consumer(const std::string& stream, const std::string& consumer,
|
||||
long ack_deadline_ms = 0); // 0 means stream default
|
||||
bool has_consumer(const std::string& stream, const std::string& consumer) const;
|
||||
|
||||
// Polls up to max messages: expired redeliveries first, then new ones
|
||||
// after the cursor. Every returned message arms its ack deadline.
|
||||
std::vector<StreamMessage> poll(const std::string& stream, const std::string& consumer,
|
||||
std::size_t max);
|
||||
|
||||
// Acks one sequence. Returns false when the seq was not pending.
|
||||
bool ack(const std::string& stream, const std::string& consumer, std::uint64_t seq);
|
||||
bool renew(const std::string& stream, const std::string& consumer, std::uint64_t seq);
|
||||
|
||||
// Entries behind the stream tail for one consumer (sync lag input).
|
||||
std::uint64_t lag(const std::string& stream, const std::string& consumer) const;
|
||||
|
||||
// Enforces time and size retention on one stream (also runs on publish).
|
||||
// Returns the number of records evicted.
|
||||
std::size_t enforce_retention(const std::string& stream);
|
||||
|
||||
std::uint64_t last_seq(const std::string& stream) const;
|
||||
std::size_t consumer_count(const std::string& stream) const;
|
||||
|
||||
private:
|
||||
struct ConsumerState {
|
||||
std::unique_ptr<ConsumerCursor> cursor;
|
||||
std::unique_ptr<AckTracker> acks;
|
||||
std::map<std::uint64_t, int> redelivery_counts; // seq -> count
|
||||
std::map<std::uint64_t, bool> acked; // contiguous-commit bookkeeping
|
||||
std::uint64_t next_new = 1; // next never-delivered seq; cursor trails acks
|
||||
long ack_deadline_ms = 60000;
|
||||
};
|
||||
|
||||
struct StreamState {
|
||||
StreamSpec spec;
|
||||
std::unique_ptr<SegmentLog> log;
|
||||
std::map<std::string, std::unique_ptr<ConsumerState>> consumers;
|
||||
};
|
||||
|
||||
static void validate_name(const std::string& name, const char* what);
|
||||
StreamState& require_stream(const std::string& name);
|
||||
const StreamState& require_stream(const std::string& name) const;
|
||||
ConsumerState& require_consumer(StreamState& state, const std::string& stream,
|
||||
const std::string& consumer);
|
||||
std::size_t enforce_retention_locked(StreamState& state);
|
||||
|
||||
std::filesystem::path dir_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::string, std::unique_ptr<StreamState>> streams_;
|
||||
};
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,103 @@
|
||||
#include "supervisor/bus/topic_bus.hpp"
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
TopicBus::TopicBus(common::Clock& clock) : clock_(clock) {}
|
||||
|
||||
TopicBus::SubscriberId TopicBus::subscribe(const std::string& topic_or_pattern) {
|
||||
if (topic_or_pattern.empty() || topic_or_pattern.size() > 256) {
|
||||
throw BusError("TopicBus: bad subscription pattern");
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const SubscriberId id = next_id_++;
|
||||
patterns_[id] = topic_or_pattern;
|
||||
queues_[id] = {};
|
||||
return id;
|
||||
}
|
||||
|
||||
bool TopicBus::unsubscribe(SubscriberId id) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const bool had = patterns_.erase(id) > 0;
|
||||
queues_.erase(id);
|
||||
return had;
|
||||
}
|
||||
|
||||
std::size_t TopicBus::publish(const std::string& topic, const std::string& payload) {
|
||||
if (topic.empty() || topic.size() > 256) {
|
||||
throw BusError("TopicBus: bad topic");
|
||||
}
|
||||
if (payload.size() > common::Caps::kStreamPayloadBytes) {
|
||||
throw BusError("TopicBus: payload over 1 MiB cap");
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const std::int64_t now = clock_.wall_ms();
|
||||
std::size_t delivered = 0;
|
||||
for (const auto& [id, pattern] : patterns_) {
|
||||
if (!matches(pattern, topic)) {
|
||||
continue;
|
||||
}
|
||||
auto& queue = queues_[id];
|
||||
if (static_cast<int>(queue.size()) >= common::Caps::kSubQueueMax) {
|
||||
queue.pop_front(); // oldest-drop backpressure, counted below
|
||||
++dropped_total_;
|
||||
}
|
||||
queue.push_back(TopicMessage{topic, payload, now});
|
||||
++delivered;
|
||||
}
|
||||
return delivered;
|
||||
}
|
||||
|
||||
std::vector<TopicMessage> TopicBus::poll(SubscriberId id, std::size_t max) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<TopicMessage> out;
|
||||
auto it = queues_.find(id);
|
||||
if (it == queues_.end()) {
|
||||
return out;
|
||||
}
|
||||
while (!it->second.empty() && out.size() < max) {
|
||||
out.push_back(std::move(it->second.front()));
|
||||
it->second.pop_front();
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::size_t TopicBus::pending(SubscriberId id) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = queues_.find(id);
|
||||
return it == queues_.end() ? 0 : it->second.size();
|
||||
}
|
||||
|
||||
std::size_t TopicBus::subscriber_count() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return patterns_.size();
|
||||
}
|
||||
|
||||
long TopicBus::dropped_total() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return dropped_total_;
|
||||
}
|
||||
|
||||
bool TopicBus::matches(const std::string& pattern, const std::string& topic) {
|
||||
// ':' segments; '*' matches exactly one segment.
|
||||
std::size_t p = 0, t = 0;
|
||||
while (true) {
|
||||
const std::size_t pn = pattern.find(':', p);
|
||||
const std::size_t tn = topic.find(':', t);
|
||||
const std::string_view ps(pattern.data() + p,
|
||||
(pn == std::string::npos ? pattern.size() : pn) - p);
|
||||
const std::string_view ts(topic.data() + t,
|
||||
(tn == std::string::npos ? topic.size() : tn) - t);
|
||||
if (ps != "*" && ps != ts) {
|
||||
return false;
|
||||
}
|
||||
if (pn == std::string::npos || tn == std::string::npos) {
|
||||
return pn == tn;
|
||||
}
|
||||
p = pn + 1;
|
||||
t = tn + 1;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,62 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <deque>
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/bus/bus_error.hpp"
|
||||
#include "supervisor/common/clock.hpp"
|
||||
|
||||
namespace molo::bus {
|
||||
|
||||
struct TopicMessage {
|
||||
std::string topic;
|
||||
std::string payload;
|
||||
std::int64_t at_ms = 0;
|
||||
};
|
||||
|
||||
// Ephemeral tier of the bus (S7-6): at-most-once delivery to currently
|
||||
// connected subscribers, no persistence, no replay. A message published
|
||||
// while a subscriber is offline is lost for that subscriber. Subscribers
|
||||
// poll; publish never blocks and never awaits subscribers.
|
||||
//
|
||||
// Topic names use ':' segments; subscriptions may use '*' to match exactly
|
||||
// one segment (same convention as 04-data.md channels).
|
||||
class TopicBus {
|
||||
public:
|
||||
using SubscriberId = std::uint64_t;
|
||||
|
||||
explicit TopicBus(common::Clock& clock);
|
||||
|
||||
// Subscribes to an exact topic or a '*' pattern. Returns the subscriber id.
|
||||
SubscriberId subscribe(const std::string& topic_or_pattern);
|
||||
bool unsubscribe(SubscriberId id);
|
||||
|
||||
// Publishes to all matching subscribers. Returns the number of
|
||||
// subscribers that received the message. Payloads over the 1 MiB cap
|
||||
// are rejected (throws BusError).
|
||||
std::size_t publish(const std::string& topic, const std::string& payload);
|
||||
|
||||
// Polls up to max messages for one subscriber (FIFO).
|
||||
std::vector<TopicMessage> poll(SubscriberId id, std::size_t max);
|
||||
|
||||
std::size_t pending(SubscriberId id) const;
|
||||
std::size_t subscriber_count() const;
|
||||
long dropped_total() const; // oldest-drop events under backpressure
|
||||
|
||||
static bool matches(const std::string& pattern, const std::string& topic);
|
||||
|
||||
private:
|
||||
common::Clock& clock_;
|
||||
mutable std::mutex mutex_;
|
||||
SubscriberId next_id_ = 1;
|
||||
std::map<SubscriberId, std::string> patterns_;
|
||||
std::map<SubscriberId, std::deque<TopicMessage>> queues_;
|
||||
long dropped_total_ = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::bus
|
||||
@@ -0,0 +1,49 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstddef>
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
// Static ceilings for part 07 (supervisor, tasks, bus, replication).
|
||||
// Every knob in this part resolves to one of these constants (R7-4, S7-10).
|
||||
// Tunable only through explicitly passed option structs, never autotuned.
|
||||
struct Caps {
|
||||
static constexpr int kWorkersPerGroup = 64;
|
||||
static constexpr int kGroupsPerSupervisor = 32;
|
||||
static constexpr std::size_t kTaskPayloadBytes = 1048576; // 1 MiB
|
||||
static constexpr int kQueueDepth = 100000;
|
||||
static constexpr int kDeadLetterMax = 10000;
|
||||
static constexpr int kDeadLetterRetentionDays = 7;
|
||||
static constexpr long kStreamRetentionHours = 24;
|
||||
static constexpr std::size_t kStreamRetentionBytes = 1073741824; // 1 GiB
|
||||
static constexpr int kSyncBatchEntries = 500;
|
||||
static constexpr std::size_t kSyncBatchBytes = 1048576; // 1 MiB
|
||||
static constexpr int kConflictLogMax = 10000;
|
||||
static constexpr int kConflictLogRetentionDays = 7;
|
||||
static constexpr std::size_t kStatusSnapshotBytes = 1048576; // 1 MiB
|
||||
static constexpr int kSyncLagAlertThreshold = 10000;
|
||||
static constexpr long kLeaseDefaultMs = 60000;
|
||||
static constexpr long kHeartbeatMs = 15000;
|
||||
static constexpr long kSchedulerTickMs = 1000;
|
||||
static constexpr long kSchedulerMinIntervalSecs = 60;
|
||||
static constexpr long kLeaderTtlSecs = 10;
|
||||
static constexpr long kLeaderRenewSecs = 3;
|
||||
static constexpr int kIdempotencyWindowHours = 24;
|
||||
static constexpr int kMaxRestartsDefault = 5;
|
||||
static constexpr int kRestartWindowSecsDefault = 60;
|
||||
static constexpr long kBackoffMinMs = 1000;
|
||||
static constexpr long kBackoffMaxMs = 60000;
|
||||
static constexpr long kHealthIntervalMs = 10000;
|
||||
static constexpr int kHealthMaxFailures = 3;
|
||||
static constexpr long kRetryBaseMs = 1000;
|
||||
static constexpr long kRetryCapMs = 300000;
|
||||
static constexpr long kRetryJitterMs = 1000;
|
||||
static constexpr long kHookTimeoutMs = 5000;
|
||||
static constexpr int kSubQueueMax = 1024;
|
||||
static constexpr std::size_t kStreamPayloadBytes = 1048576; // 1 MiB
|
||||
static constexpr int kJsonMaxDepth = 64;
|
||||
static constexpr int kMaxStreamsPerBus = 1024;
|
||||
static constexpr int kMaxConsumersPerStream = 1024;
|
||||
};
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,109 @@
|
||||
#include "supervisor/common/clock.hpp"
|
||||
|
||||
#include <cstdio>
|
||||
#include <ctime>
|
||||
#include <thread>
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
std::int64_t SystemClock::wall_ms() const {
|
||||
const auto now = std::chrono::system_clock::now();
|
||||
return std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
|
||||
}
|
||||
|
||||
std::int64_t SystemClock::steady_ms() const {
|
||||
const auto now = std::chrono::steady_clock::now();
|
||||
return std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count();
|
||||
}
|
||||
|
||||
void SystemClock::sleep_ms(long ms) {
|
||||
if (ms > 0) {
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(ms));
|
||||
}
|
||||
}
|
||||
|
||||
FrozenClock::FrozenClock(std::int64_t start_wall_ms, std::int64_t start_steady_ms)
|
||||
: wall_ms_(start_wall_ms), steady_ms_(start_steady_ms) {}
|
||||
|
||||
std::int64_t FrozenClock::wall_ms() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return wall_ms_;
|
||||
}
|
||||
|
||||
std::int64_t FrozenClock::steady_ms() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return steady_ms_;
|
||||
}
|
||||
|
||||
void FrozenClock::sleep_ms(long ms) {
|
||||
if (ms > 0) {
|
||||
advance(ms);
|
||||
}
|
||||
}
|
||||
|
||||
void FrozenClock::advance(std::int64_t delta_ms) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
wall_ms_ += delta_ms;
|
||||
steady_ms_ += delta_ms;
|
||||
}
|
||||
|
||||
void FrozenClock::set(std::int64_t wall_ms, std::int64_t steady_ms) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
wall_ms_ = wall_ms;
|
||||
steady_ms_ = steady_ms;
|
||||
}
|
||||
|
||||
std::string iso8601(std::int64_t wall_ms) {
|
||||
const std::time_t secs = static_cast<std::time_t>(wall_ms / 1000);
|
||||
const int millis = static_cast<int>(wall_ms % 1000);
|
||||
std::tm tm_utc{};
|
||||
#if defined(_WIN32)
|
||||
gmtime_s(&tm_utc, &secs);
|
||||
#else
|
||||
gmtime_r(&secs, &tm_utc);
|
||||
#endif
|
||||
char buf[128];
|
||||
std::snprintf(buf, sizeof(buf), "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ", tm_utc.tm_year + 1900,
|
||||
tm_utc.tm_mon + 1, tm_utc.tm_mday, tm_utc.tm_hour, tm_utc.tm_min, tm_utc.tm_sec,
|
||||
millis);
|
||||
return std::string(buf);
|
||||
}
|
||||
|
||||
std::int64_t parse_iso8601(const std::string& text) {
|
||||
// Accepts YYYY-MM-DDTHH:MM:SS[.mmm]Z only. Anything else returns -1.
|
||||
int year = 0, mon = 0, day = 0, hour = 0, min = 0, sec = 0, millis = 0;
|
||||
if (text.size() < 20 || text.back() != 'Z') {
|
||||
return -1;
|
||||
}
|
||||
int matched = std::sscanf(text.c_str(), "%4d-%2d-%2dT%2d:%2d:%2d.%3dZ", &year, &mon, &day,
|
||||
&hour, &min, &sec, &millis);
|
||||
if (matched != 7) {
|
||||
matched = std::sscanf(text.c_str(), "%4d-%2d-%2dT%2d:%2d:%2dZ", &year, &mon, &day, &hour,
|
||||
&min, &sec);
|
||||
if (matched != 6) {
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
if (mon < 1 || mon > 12 || day < 1 || day > 31 || hour > 23 || min > 59 || sec > 60 ||
|
||||
millis < 0 || millis > 999) {
|
||||
return -1;
|
||||
}
|
||||
std::tm tm_utc{};
|
||||
tm_utc.tm_year = year - 1900;
|
||||
tm_utc.tm_mon = mon - 1;
|
||||
tm_utc.tm_mday = day;
|
||||
tm_utc.tm_hour = hour;
|
||||
tm_utc.tm_min = min;
|
||||
tm_utc.tm_sec = sec;
|
||||
#if defined(_WIN32)
|
||||
const std::time_t secs = _mkgmtime(&tm_utc);
|
||||
#else
|
||||
const std::time_t secs = timegm(&tm_utc);
|
||||
#endif
|
||||
if (secs == static_cast<std::time_t>(-1)) {
|
||||
return -1;
|
||||
}
|
||||
return static_cast<std::int64_t>(secs) * 1000 + millis;
|
||||
}
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,50 @@
|
||||
#pragma once
|
||||
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
// Clock abstraction. Production uses SystemClock; tests use FrozenClock
|
||||
// for deterministic scheduler, lease, and backoff behavior.
|
||||
class Clock {
|
||||
public:
|
||||
virtual ~Clock() = default;
|
||||
virtual std::int64_t wall_ms() const = 0; // unix epoch milliseconds
|
||||
virtual std::int64_t steady_ms() const = 0; // monotonic milliseconds
|
||||
virtual void sleep_ms(long ms) = 0;
|
||||
};
|
||||
|
||||
class SystemClock : public Clock {
|
||||
public:
|
||||
std::int64_t wall_ms() const override;
|
||||
std::int64_t steady_ms() const override;
|
||||
void sleep_ms(long ms) override;
|
||||
};
|
||||
|
||||
class FrozenClock : public Clock {
|
||||
public:
|
||||
explicit FrozenClock(std::int64_t start_wall_ms = 0, std::int64_t start_steady_ms = 0);
|
||||
|
||||
std::int64_t wall_ms() const override;
|
||||
std::int64_t steady_ms() const override;
|
||||
void sleep_ms(long ms) override; // advances the frozen clock, no real wait
|
||||
void advance(std::int64_t delta_ms);
|
||||
void set(std::int64_t wall_ms, std::int64_t steady_ms);
|
||||
|
||||
private:
|
||||
mutable std::mutex mutex_;
|
||||
std::int64_t wall_ms_;
|
||||
std::int64_t steady_ms_;
|
||||
};
|
||||
|
||||
// UTC ISO-8601 with millis, e.g. 2026-10-23T12:34:56.789Z. Used by the
|
||||
// task envelope (S7-1) and oplog (S7-7) JSON formats.
|
||||
std::string iso8601(std::int64_t wall_ms);
|
||||
|
||||
// Parses iso8601() output (millis optional). Returns -1 on invalid input.
|
||||
std::int64_t parse_iso8601(const std::string& text);
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,382 @@
|
||||
#include "supervisor/common/json.hpp"
|
||||
|
||||
#include <cctype>
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
bool JValue::is_null() const { return std::holds_alternative<std::nullptr_t>(data); }
|
||||
bool JValue::is_bool() const { return std::holds_alternative<bool>(data); }
|
||||
bool JValue::is_int() const { return std::holds_alternative<long long>(data); }
|
||||
bool JValue::is_double() const { return std::holds_alternative<double>(data); }
|
||||
bool JValue::is_string() const { return std::holds_alternative<std::string>(data); }
|
||||
bool JValue::is_array() const { return std::holds_alternative<Array>(data); }
|
||||
bool JValue::is_object() const { return std::holds_alternative<Object>(data); }
|
||||
|
||||
bool JValue::as_bool(bool fallback) const {
|
||||
return is_bool() ? std::get<bool>(data) : fallback;
|
||||
}
|
||||
|
||||
long long JValue::as_int(long long fallback) const {
|
||||
if (is_int()) {
|
||||
return std::get<long long>(data);
|
||||
}
|
||||
if (is_double()) {
|
||||
return static_cast<long long>(std::get<double>(data));
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
double JValue::as_double(double fallback) const {
|
||||
if (is_double()) {
|
||||
return std::get<double>(data);
|
||||
}
|
||||
if (is_int()) {
|
||||
return static_cast<double>(std::get<long long>(data));
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
std::string JValue::as_string(const std::string& fallback) const {
|
||||
return is_string() ? std::get<std::string>(data) : fallback;
|
||||
}
|
||||
|
||||
const JValue::Array& JValue::as_array() const {
|
||||
static const Array empty;
|
||||
return is_array() ? std::get<Array>(data) : empty;
|
||||
}
|
||||
|
||||
const JValue::Object& JValue::as_object() const {
|
||||
static const Object empty;
|
||||
return is_object() ? std::get<Object>(data) : empty;
|
||||
}
|
||||
|
||||
const JValue* JValue::find(std::string_view key) const {
|
||||
if (!is_object()) {
|
||||
return nullptr;
|
||||
}
|
||||
const auto& obj = std::get<Object>(data);
|
||||
auto it = obj.find(std::string(key));
|
||||
return it == obj.end() ? nullptr : &it->second;
|
||||
}
|
||||
|
||||
std::string json_escape(std::string_view text) {
|
||||
std::string out;
|
||||
out.reserve(text.size() + 2);
|
||||
for (char c : text) {
|
||||
switch (c) {
|
||||
case '"': out += "\\\""; break;
|
||||
case '\\': out += "\\\\"; break;
|
||||
case '\n': out += "\\n"; break;
|
||||
case '\r': out += "\\r"; break;
|
||||
case '\t': out += "\\t"; break;
|
||||
default:
|
||||
if (static_cast<unsigned char>(c) < 0x20) {
|
||||
char buf[8];
|
||||
const int n = std::snprintf(buf, sizeof(buf), "\\u%04x", c);
|
||||
out.append(buf, static_cast<std::size_t>(n));
|
||||
} else {
|
||||
out += c;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
void encode_into(const JValue& value, std::string& out) {
|
||||
if (value.is_null()) {
|
||||
out += "null";
|
||||
} else if (value.is_bool()) {
|
||||
out += value.as_bool() ? "true" : "false";
|
||||
} else if (value.is_int()) {
|
||||
out += std::to_string(value.as_int());
|
||||
} else if (value.is_double()) {
|
||||
out += std::to_string(value.as_double());
|
||||
} else if (value.is_string()) {
|
||||
out += '"';
|
||||
out += json_escape(value.as_string());
|
||||
out += '"';
|
||||
} else if (value.is_array()) {
|
||||
out += '[';
|
||||
bool first = true;
|
||||
for (const auto& item : value.as_array()) {
|
||||
if (!first) {
|
||||
out += ',';
|
||||
}
|
||||
first = false;
|
||||
encode_into(item, out);
|
||||
}
|
||||
out += ']';
|
||||
} else {
|
||||
out += '{';
|
||||
bool first = true;
|
||||
for (const auto& [key, item] : value.as_object()) {
|
||||
if (!first) {
|
||||
out += ',';
|
||||
}
|
||||
first = false;
|
||||
out += '"';
|
||||
out += json_escape(key);
|
||||
out += "\":";
|
||||
encode_into(item, out);
|
||||
}
|
||||
out += '}';
|
||||
}
|
||||
}
|
||||
|
||||
class Parser {
|
||||
public:
|
||||
explicit Parser(std::string_view text) : text_(text) {}
|
||||
|
||||
bool run(JValue& out) {
|
||||
skip_ws();
|
||||
if (!parse_value(out, 0)) {
|
||||
return false;
|
||||
}
|
||||
skip_ws();
|
||||
return pos_ == text_.size();
|
||||
}
|
||||
|
||||
private:
|
||||
void skip_ws() {
|
||||
while (pos_ < text_.size() &&
|
||||
(text_[pos_] == ' ' || text_[pos_] == '\t' || text_[pos_] == '\n' ||
|
||||
text_[pos_] == '\r')) {
|
||||
++pos_;
|
||||
}
|
||||
}
|
||||
|
||||
bool consume(char c) {
|
||||
if (pos_ < text_.size() && text_[pos_] == c) {
|
||||
++pos_;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool parse_value(JValue& out, int depth) {
|
||||
if (depth > Caps::kJsonMaxDepth || pos_ >= text_.size()) {
|
||||
return false;
|
||||
}
|
||||
const char c = text_[pos_];
|
||||
if (c == '{') {
|
||||
return parse_object(out, depth);
|
||||
}
|
||||
if (c == '[') {
|
||||
return parse_array(out, depth);
|
||||
}
|
||||
if (c == '"') {
|
||||
std::string s;
|
||||
if (!parse_string(s)) {
|
||||
return false;
|
||||
}
|
||||
out = JValue(s);
|
||||
return true;
|
||||
}
|
||||
if (c == 't') {
|
||||
return parse_literal("true", JValue(true), out);
|
||||
}
|
||||
if (c == 'f') {
|
||||
return parse_literal("false", JValue(false), out);
|
||||
}
|
||||
if (c == 'n') {
|
||||
return parse_literal("null", JValue(nullptr), out);
|
||||
}
|
||||
return parse_number(out);
|
||||
}
|
||||
|
||||
bool parse_literal(std::string_view word, const JValue& value, JValue& out) {
|
||||
if (text_.substr(pos_, word.size()) != word) {
|
||||
return false;
|
||||
}
|
||||
pos_ += word.size();
|
||||
out = value;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool parse_number(JValue& out) {
|
||||
std::size_t start = pos_;
|
||||
if (pos_ < text_.size() && (text_[pos_] == '-' || text_[pos_] == '+')) {
|
||||
++pos_;
|
||||
}
|
||||
bool is_double = false;
|
||||
while (pos_ < text_.size()) {
|
||||
const char c = text_[pos_];
|
||||
if (c >= '0' && c <= '9') {
|
||||
++pos_;
|
||||
} else if (c == '.' || c == 'e' || c == 'E') {
|
||||
is_double = true;
|
||||
++pos_;
|
||||
} else if ((c == '+' || c == '-') &&
|
||||
(text_[pos_ - 1] == 'e' || text_[pos_ - 1] == 'E')) {
|
||||
++pos_;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (pos_ == start) {
|
||||
return false;
|
||||
}
|
||||
const std::string token(text_.substr(start, pos_ - start));
|
||||
try {
|
||||
if (is_double) {
|
||||
out = JValue(std::stod(token));
|
||||
} else {
|
||||
out = JValue(static_cast<long long>(std::stoll(token)));
|
||||
}
|
||||
} catch (...) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool parse_string(std::string& out) {
|
||||
if (!consume('"')) {
|
||||
return false;
|
||||
}
|
||||
out.clear();
|
||||
while (pos_ < text_.size()) {
|
||||
const char c = text_[pos_++];
|
||||
if (c == '"') {
|
||||
return true;
|
||||
}
|
||||
if (c != '\\') {
|
||||
out += c;
|
||||
continue;
|
||||
}
|
||||
if (pos_ >= text_.size()) {
|
||||
return false;
|
||||
}
|
||||
const char e = text_[pos_++];
|
||||
switch (e) {
|
||||
case '"': out += '"'; break;
|
||||
case '\\': out += '\\'; break;
|
||||
case '/': out += '/'; break;
|
||||
case 'n': out += '\n'; break;
|
||||
case 'r': out += '\r'; break;
|
||||
case 't': out += '\t'; break;
|
||||
case 'b': out += '\b'; break;
|
||||
case 'f': out += '\f'; break;
|
||||
case 'u': {
|
||||
if (pos_ + 4 > text_.size()) {
|
||||
return false;
|
||||
}
|
||||
unsigned code = 0;
|
||||
for (int i = 0; i < 4; ++i) {
|
||||
const char h = text_[pos_++];
|
||||
code <<= 4;
|
||||
if (h >= '0' && h <= '9') {
|
||||
code += static_cast<unsigned>(h - '0');
|
||||
} else if (h >= 'a' && h <= 'f') {
|
||||
code += static_cast<unsigned>(h - 'a' + 10);
|
||||
} else if (h >= 'A' && h <= 'F') {
|
||||
code += static_cast<unsigned>(h - 'A' + 10);
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
// BMP only, enough for the ASCII control escapes we emit.
|
||||
if (code < 0x80) {
|
||||
out += static_cast<char>(code);
|
||||
} else if (code < 0x800) {
|
||||
out += static_cast<char>(0xC0 | (code >> 6));
|
||||
out += static_cast<char>(0x80 | (code & 0x3F));
|
||||
} else {
|
||||
out += static_cast<char>(0xE0 | (code >> 12));
|
||||
out += static_cast<char>(0x80 | ((code >> 6) & 0x3F));
|
||||
out += static_cast<char>(0x80 | (code & 0x3F));
|
||||
}
|
||||
break;
|
||||
}
|
||||
default: return false;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool parse_object(JValue& out, int depth) {
|
||||
if (!consume('{')) {
|
||||
return false;
|
||||
}
|
||||
JValue::Object obj;
|
||||
skip_ws();
|
||||
if (consume('}')) {
|
||||
out = JValue(obj);
|
||||
return true;
|
||||
}
|
||||
while (true) {
|
||||
skip_ws();
|
||||
std::string key;
|
||||
if (!parse_string(key)) {
|
||||
return false;
|
||||
}
|
||||
skip_ws();
|
||||
if (!consume(':')) {
|
||||
return false;
|
||||
}
|
||||
skip_ws();
|
||||
JValue value;
|
||||
if (!parse_value(value, depth + 1)) {
|
||||
return false;
|
||||
}
|
||||
obj[std::move(key)] = std::move(value);
|
||||
skip_ws();
|
||||
if (consume('}')) {
|
||||
out = JValue(obj);
|
||||
return true;
|
||||
}
|
||||
if (!consume(',')) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bool parse_array(JValue& out, int depth) {
|
||||
if (!consume('[')) {
|
||||
return false;
|
||||
}
|
||||
JValue::Array items;
|
||||
skip_ws();
|
||||
if (consume(']')) {
|
||||
out = JValue(items);
|
||||
return true;
|
||||
}
|
||||
while (true) {
|
||||
skip_ws();
|
||||
JValue value;
|
||||
if (!parse_value(value, depth + 1)) {
|
||||
return false;
|
||||
}
|
||||
items.push_back(std::move(value));
|
||||
skip_ws();
|
||||
if (consume(']')) {
|
||||
out = JValue(items);
|
||||
return true;
|
||||
}
|
||||
if (!consume(',')) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
std::string_view text_;
|
||||
std::size_t pos_ = 0;
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string json_encode(const JValue& value) {
|
||||
std::string out;
|
||||
encode_into(value, out);
|
||||
return out;
|
||||
}
|
||||
|
||||
bool json_decode(std::string_view text, JValue& out) {
|
||||
Parser parser(text);
|
||||
return parser.run(out);
|
||||
}
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,61 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <variant>
|
||||
#include <vector>
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
// Minimal JSON value plus encoder/decoder for the on-disk formats owned by
|
||||
// this part (task envelope S7-1, oplog entry S7-7, status snapshot). Only
|
||||
// what those formats need: objects, arrays, strings, integers, doubles,
|
||||
// booleans, null. Depth capped by Caps::kJsonMaxDepth.
|
||||
struct JValue {
|
||||
using Array = std::vector<JValue>;
|
||||
using Object = std::map<std::string, JValue>;
|
||||
using Data = std::variant<std::nullptr_t, bool, long long, double, std::string, Array, Object>;
|
||||
|
||||
Data data;
|
||||
|
||||
JValue() : data(nullptr) {}
|
||||
JValue(std::nullptr_t) : data(nullptr) {}
|
||||
JValue(bool v) : data(v) {}
|
||||
JValue(long long v) : data(v) {}
|
||||
JValue(int v) : data(static_cast<long long>(v)) {}
|
||||
JValue(double v) : data(v) {}
|
||||
JValue(const char* v) : data(std::string(v)) {}
|
||||
JValue(std::string_view v) : data(std::string(v)) {}
|
||||
JValue(const std::string& v) : data(v) {}
|
||||
JValue(const Array& v) : data(v) {}
|
||||
JValue(const Object& v) : data(v) {}
|
||||
|
||||
bool is_null() const;
|
||||
bool is_bool() const;
|
||||
bool is_int() const;
|
||||
bool is_double() const;
|
||||
bool is_string() const;
|
||||
bool is_array() const;
|
||||
bool is_object() const;
|
||||
|
||||
bool as_bool(bool fallback = false) const;
|
||||
long long as_int(long long fallback = 0) const;
|
||||
double as_double(double fallback = 0.0) const;
|
||||
std::string as_string(const std::string& fallback = "") const;
|
||||
const Array& as_array() const;
|
||||
const Object& as_object() const;
|
||||
|
||||
// Object helpers. find() returns nullptr when absent or not an object.
|
||||
const JValue* find(std::string_view key) const;
|
||||
};
|
||||
|
||||
std::string json_escape(std::string_view text);
|
||||
std::string json_encode(const JValue& value);
|
||||
|
||||
// Parses a full JSON document. Returns false on any syntax error or when
|
||||
// trailing garbage follows the document. Depth is capped.
|
||||
bool json_decode(std::string_view text, JValue& out);
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,118 @@
|
||||
#include "supervisor/common/kv_store.hpp"
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
InMemoryKvStore::InMemoryKvStore(Clock& clock) : clock_(clock) {}
|
||||
|
||||
bool InMemoryKvStore::expired(const Entry& entry, std::int64_t now_ms) const {
|
||||
return entry.expires_at_ms > 0 && entry.expires_at_ms <= now_ms;
|
||||
}
|
||||
|
||||
bool InMemoryKvStore::get(std::string_view key, std::string& value) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(key);
|
||||
if (it == entries_.end() || expired(it->second, clock_.wall_ms())) {
|
||||
return false;
|
||||
}
|
||||
value = it->second.value;
|
||||
return true;
|
||||
}
|
||||
|
||||
void InMemoryKvStore::set(std::string_view key, const std::string& value, long ttl_secs) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
Entry entry;
|
||||
entry.value = value;
|
||||
entry.expires_at_ms = ttl_secs > 0 ? clock_.wall_ms() + ttl_secs * 1000 : 0;
|
||||
entries_[std::string(key)] = std::move(entry);
|
||||
}
|
||||
|
||||
bool InMemoryKvStore::erase(std::string_view key) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(key);
|
||||
if (it == entries_.end()) {
|
||||
return false;
|
||||
}
|
||||
entries_.erase(it);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool InMemoryKvStore::acquire(std::string_view key, const std::string& value, long ttl_secs) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(key);
|
||||
if (it != entries_.end() && !expired(it->second, clock_.wall_ms())) {
|
||||
return false;
|
||||
}
|
||||
Entry entry;
|
||||
entry.value = value;
|
||||
entry.expires_at_ms = ttl_secs > 0 ? clock_.wall_ms() + ttl_secs * 1000 : 0;
|
||||
entries_[std::string(key)] = std::move(entry);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool InMemoryKvStore::renew(std::string_view key, const std::string& expected, long ttl_secs) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(key);
|
||||
if (it == entries_.end() || expired(it->second, clock_.wall_ms()) ||
|
||||
it->second.value != expected) {
|
||||
return false;
|
||||
}
|
||||
it->second.expires_at_ms = ttl_secs > 0 ? clock_.wall_ms() + ttl_secs * 1000 : 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool InMemoryKvStore::acquire_ms(std::string_view key, const std::string& value,
|
||||
std::int64_t ttl_ms) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(key);
|
||||
if (it != entries_.end() && !expired(it->second, clock_.wall_ms())) {
|
||||
return false;
|
||||
}
|
||||
Entry entry;
|
||||
entry.value = value;
|
||||
entry.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
|
||||
entries_[std::string(key)] = std::move(entry);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool InMemoryKvStore::renew_ms(std::string_view key, const std::string& expected,
|
||||
std::int64_t ttl_ms) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(key);
|
||||
if (it == entries_.end() || expired(it->second, clock_.wall_ms()) ||
|
||||
it->second.value != expected) {
|
||||
return false;
|
||||
}
|
||||
it->second.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool InMemoryKvStore::cas_ms(std::string_view key, const std::string& expected,
|
||||
const std::string& next, std::int64_t ttl_ms) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = entries_.find(key);
|
||||
if (it == entries_.end() || expired(it->second, clock_.wall_ms())) {
|
||||
if (!expected.empty()) {
|
||||
return false;
|
||||
}
|
||||
Entry entry;
|
||||
entry.value = next;
|
||||
entry.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
|
||||
entries_[std::string(key)] = std::move(entry);
|
||||
return true;
|
||||
}
|
||||
if (it->second.value != expected) {
|
||||
return false;
|
||||
}
|
||||
it->second.value = next;
|
||||
it->second.expires_at_ms = ttl_ms > 0 ? clock_.wall_ms() + ttl_ms : 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
std::size_t InMemoryKvStore::size() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return entries_.size();
|
||||
}
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,79 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
class Clock;
|
||||
|
||||
// Minimal key-value interface needed by part 07 (scheduler leader lease,
|
||||
// shared last-fire records). Mirrors the 04-data.md KvStore shape
|
||||
// (get/set/del plus atomic acquire/renew for leases) so the shared
|
||||
// backend from 04-data.md slots in behind this interface for
|
||||
// multi-instance deployments (S7-9). Single-node deployments use
|
||||
// InMemoryKvStore.
|
||||
class KvStore {
|
||||
public:
|
||||
virtual ~KvStore() = default;
|
||||
|
||||
// Returns true and fills value when the key exists and is not expired.
|
||||
virtual bool get(std::string_view key, std::string& value) = 0;
|
||||
|
||||
// Stores value. ttl_secs <= 0 means no expiry.
|
||||
virtual void set(std::string_view key, const std::string& value, long ttl_secs) = 0;
|
||||
|
||||
virtual bool erase(std::string_view key) = 0;
|
||||
|
||||
// Atomically stores value when the key is absent or expired.
|
||||
virtual bool acquire(std::string_view key, const std::string& value, long ttl_secs) = 0;
|
||||
|
||||
// Atomically extends the TTL when the current value equals expected.
|
||||
virtual bool renew(std::string_view key, const std::string& expected, long ttl_secs) = 0;
|
||||
|
||||
// Millisecond-precision fencing API for the operability part (leases,
|
||||
// leader election, distributed locks run on ms clocks). Same semantics as
|
||||
// the seconds API: ttl_ms <= 0 means no expiry; empty expected matches a
|
||||
// missing key for cas_ms only.
|
||||
virtual bool acquire_ms(std::string_view key, const std::string& value,
|
||||
std::int64_t ttl_ms) = 0;
|
||||
virtual bool renew_ms(std::string_view key, const std::string& expected,
|
||||
std::int64_t ttl_ms) = 0;
|
||||
virtual bool cas_ms(std::string_view key, const std::string& expected,
|
||||
const std::string& next, std::int64_t ttl_ms) = 0;
|
||||
};
|
||||
|
||||
class InMemoryKvStore : public KvStore {
|
||||
public:
|
||||
explicit InMemoryKvStore(Clock& clock);
|
||||
|
||||
bool get(std::string_view key, std::string& value) override;
|
||||
void set(std::string_view key, const std::string& value, long ttl_secs) override;
|
||||
bool erase(std::string_view key) override;
|
||||
bool acquire(std::string_view key, const std::string& value, long ttl_secs) override;
|
||||
bool renew(std::string_view key, const std::string& expected, long ttl_secs) override;
|
||||
bool acquire_ms(std::string_view key, const std::string& value,
|
||||
std::int64_t ttl_ms) override;
|
||||
bool renew_ms(std::string_view key, const std::string& expected,
|
||||
std::int64_t ttl_ms) override;
|
||||
bool cas_ms(std::string_view key, const std::string& expected, const std::string& next,
|
||||
std::int64_t ttl_ms) override;
|
||||
std::size_t size() const;
|
||||
|
||||
private:
|
||||
struct Entry {
|
||||
std::string value;
|
||||
std::int64_t expires_at_ms = 0; // 0 means no expiry
|
||||
};
|
||||
|
||||
bool expired(const Entry& entry, std::int64_t now_ms) const;
|
||||
|
||||
Clock& clock_;
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::string, Entry, std::less<>> entries_;
|
||||
};
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,61 @@
|
||||
#include "supervisor/common/observer.hpp"
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
void NullObserver::event(std::string_view, std::string_view) {}
|
||||
void NullObserver::counter(std::string_view, long) {}
|
||||
void NullObserver::gauge(std::string_view, double) {}
|
||||
|
||||
void RecordingObserver::event(std::string_view type, std::string_view detail) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
events_.push_back(RecordedEvent{std::string(type), std::string(detail)});
|
||||
}
|
||||
|
||||
void RecordingObserver::counter(std::string_view name, long delta) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
counters_[std::string(name)] += delta;
|
||||
}
|
||||
|
||||
void RecordingObserver::gauge(std::string_view name, double value) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
gauges_[std::string(name)] = value;
|
||||
}
|
||||
|
||||
std::vector<RecordedEvent> RecordingObserver::events() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return events_;
|
||||
}
|
||||
|
||||
std::map<std::string, long> RecordingObserver::counters() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return counters_;
|
||||
}
|
||||
|
||||
std::map<std::string, double> RecordingObserver::gauges() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return gauges_;
|
||||
}
|
||||
|
||||
long RecordingObserver::counter_value(const std::string& name) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = counters_.find(name);
|
||||
return it == counters_.end() ? 0 : it->second;
|
||||
}
|
||||
|
||||
std::size_t RecordingObserver::event_count(const std::string& type) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::size_t count = 0;
|
||||
for (const auto& e : events_) {
|
||||
if (e.type == type) {
|
||||
++count;
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
std::shared_ptr<Observer> null_observer() {
|
||||
static std::shared_ptr<Observer> instance = std::make_shared<NullObserver>();
|
||||
return instance;
|
||||
}
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,60 @@
|
||||
#pragma once
|
||||
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <vector>
|
||||
|
||||
namespace molo::common {
|
||||
|
||||
// Minimal observability sink for part 07 (R7-6).
|
||||
//
|
||||
// This is the seam where 02-observability.md plugs in: the real Dispatcher,
|
||||
// metrics Registry, and Tracer implement this interface (or this interface is
|
||||
// replaced by direct calls to molo::events, molo::metrics, molo::trace once
|
||||
// that part lands). Until then NullObserver keeps production quiet and
|
||||
// RecordingObserver lets tests assert on emitted events and metrics.
|
||||
class Observer {
|
||||
public:
|
||||
virtual ~Observer() = default;
|
||||
virtual void event(std::string_view type, std::string_view detail) = 0;
|
||||
virtual void counter(std::string_view name, long delta) = 0;
|
||||
virtual void gauge(std::string_view name, double value) = 0;
|
||||
};
|
||||
|
||||
class NullObserver : public Observer {
|
||||
public:
|
||||
void event(std::string_view type, std::string_view detail) override;
|
||||
void counter(std::string_view name, long delta) override;
|
||||
void gauge(std::string_view name, double value) override;
|
||||
};
|
||||
|
||||
struct RecordedEvent {
|
||||
std::string type;
|
||||
std::string detail;
|
||||
};
|
||||
|
||||
class RecordingObserver : public Observer {
|
||||
public:
|
||||
void event(std::string_view type, std::string_view detail) override;
|
||||
void counter(std::string_view name, long delta) override;
|
||||
void gauge(std::string_view name, double value) override;
|
||||
|
||||
std::vector<RecordedEvent> events() const;
|
||||
std::map<std::string, long> counters() const;
|
||||
std::map<std::string, double> gauges() const;
|
||||
long counter_value(const std::string& name) const;
|
||||
std::size_t event_count(const std::string& type) const;
|
||||
|
||||
private:
|
||||
mutable std::mutex mutex_;
|
||||
std::vector<RecordedEvent> events_;
|
||||
std::map<std::string, long> counters_;
|
||||
std::map<std::string, double> gauges_;
|
||||
};
|
||||
|
||||
std::shared_ptr<Observer> null_observer();
|
||||
|
||||
} // namespace molo::common
|
||||
@@ -0,0 +1,230 @@
|
||||
#include "supervisor/replication/conflict_resolver.hpp"
|
||||
|
||||
#include <chrono>
|
||||
#include <future>
|
||||
#include <thread>
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
std::string to_string(ConflictPolicy policy) {
|
||||
switch (policy) {
|
||||
case ConflictPolicy::Optimistic: return "optimistic";
|
||||
case ConflictPolicy::LastWriterWins: return "lww";
|
||||
case ConflictPolicy::Hook: return "hook";
|
||||
}
|
||||
return "optimistic";
|
||||
}
|
||||
|
||||
ConflictResolver::ConflictResolver(common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer,
|
||||
int max_log_entries, long log_retention_days)
|
||||
: clock_(clock),
|
||||
observer_(std::move(observer)),
|
||||
max_log_entries_(max_log_entries),
|
||||
log_retention_days_(log_retention_days) {
|
||||
if (max_log_entries_ < 1) {
|
||||
max_log_entries_ = 1;
|
||||
}
|
||||
}
|
||||
|
||||
void ConflictResolver::declare(const std::string& model, ConflictPolicy policy,
|
||||
ConflictHook hook) {
|
||||
if (model.empty() || model.size() > 256) {
|
||||
throw ReplicationError("ConflictResolver: bad model name");
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (policy == ConflictPolicy::Hook && !hook) {
|
||||
policy = ConflictPolicy::LastWriterWins; // declared fallback, logged below
|
||||
observer_->event("replication.policy_fallback", model + " hook->lww (no callback)");
|
||||
}
|
||||
policies_[model] = policy;
|
||||
if (hook) {
|
||||
hooks_[model] = std::move(hook);
|
||||
} else {
|
||||
hooks_.erase(model);
|
||||
}
|
||||
observer_->event("replication.policy_declared", model + " " + to_string(policy));
|
||||
}
|
||||
|
||||
ConflictPolicy ConflictResolver::policy_for(const std::string& model) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = policies_.find(model);
|
||||
return it == policies_.end() ? ConflictPolicy::Optimistic : it->second;
|
||||
}
|
||||
|
||||
ConflictOutcome ConflictResolver::resolve(const OplogEntry& local, const OplogEntry& remote) {
|
||||
ConflictPolicy policy;
|
||||
ConflictHook hook;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = policies_.find(local.model);
|
||||
policy = it == policies_.end() ? ConflictPolicy::Optimistic : it->second;
|
||||
auto hit = hooks_.find(local.model);
|
||||
if (hit != hooks_.end()) {
|
||||
hook = hit->second;
|
||||
}
|
||||
}
|
||||
// Run the policy without the resolver lock: hooks are user code.
|
||||
ConflictOutcome outcome;
|
||||
switch (policy) {
|
||||
case ConflictPolicy::Optimistic: outcome = resolve_optimistic(local, remote); break;
|
||||
case ConflictPolicy::LastWriterWins: outcome = resolve_lww(local, remote); break;
|
||||
case ConflictPolicy::Hook: outcome = resolve_hook(local, remote, hook); break;
|
||||
}
|
||||
if (outcome.conflict) {
|
||||
observer_->counter("molo_replication_conflicts_total", 1);
|
||||
observer_->event("replication.conflict",
|
||||
local.model + ":" + local.key + " " + outcome.detail);
|
||||
}
|
||||
return outcome;
|
||||
}
|
||||
|
||||
ConflictOutcome ConflictResolver::resolve_optimistic(const OplogEntry& local,
|
||||
const OplogEntry& remote) {
|
||||
ConflictOutcome outcome;
|
||||
if (remote.version > local.version) {
|
||||
outcome.remote_won = true;
|
||||
outcome.winner = remote;
|
||||
outcome.detail = "remote version " + std::to_string(remote.version) + " > local " +
|
||||
std::to_string(local.version);
|
||||
// Fast-forward, not a conflict.
|
||||
} else if (remote.version < local.version) {
|
||||
outcome.winner = local;
|
||||
outcome.conflict = true;
|
||||
outcome.conflict_error = true;
|
||||
outcome.winner_version = local.version;
|
||||
outcome.detail = "stale remote version " + std::to_string(remote.version) + ", winner " +
|
||||
std::to_string(local.version);
|
||||
log_conflict("optimistic", local, remote);
|
||||
} else if (remote.data == local.data && remote.op == local.op) {
|
||||
outcome.winner = local; // identical write, idempotent replay
|
||||
outcome.detail = "identical versions and data";
|
||||
} else {
|
||||
// Same version, different data: a genuine write-write conflict.
|
||||
// The incumbent stays; the loser gets ConflictError with the winner
|
||||
// version so it can re-read and retry.
|
||||
outcome.winner = local;
|
||||
outcome.conflict = true;
|
||||
outcome.conflict_error = true;
|
||||
outcome.winner_version = local.version;
|
||||
outcome.detail = "write-write conflict at version " + std::to_string(local.version);
|
||||
log_conflict("optimistic", local, remote);
|
||||
}
|
||||
return outcome;
|
||||
}
|
||||
|
||||
ConflictOutcome ConflictResolver::resolve_lww(const OplogEntry& local, const OplogEntry& remote) {
|
||||
ConflictOutcome outcome;
|
||||
// Concurrent writes with any difference are a conflict whose loser is
|
||||
// kept in the conflict log (S7-8); only exact replays are quiet.
|
||||
outcome.conflict = (local.version != remote.version || local.data != remote.data ||
|
||||
local.op != remote.op);
|
||||
const int cmp = remote.ts.compare(local.ts);
|
||||
if (cmp > 0) {
|
||||
outcome.remote_won = true;
|
||||
outcome.winner = remote;
|
||||
outcome.detail = "remote ts " + remote.ts.encode() + " wins";
|
||||
if (outcome.conflict) {
|
||||
log_conflict("lww", remote, local);
|
||||
}
|
||||
} else {
|
||||
outcome.winner = local;
|
||||
outcome.detail = "local ts " + local.ts.encode() + " wins";
|
||||
if (outcome.conflict) {
|
||||
log_conflict("lww", local, remote);
|
||||
}
|
||||
}
|
||||
outcome.winner_version = outcome.winner.version;
|
||||
return outcome;
|
||||
}
|
||||
|
||||
ConflictOutcome ConflictResolver::resolve_hook(const OplogEntry& local, const OplogEntry& remote,
|
||||
const ConflictHook& hook) {
|
||||
if (!hook) {
|
||||
return resolve_lww(local, remote);
|
||||
}
|
||||
// The hook runs off-lock with a 5s budget; on timeout or throw the LWW
|
||||
// fallback applies (S7-8). Packaged task plus timed wait, no detached
|
||||
// threads: a stuck hook still joins before we return.
|
||||
std::packaged_task<int()> task([&] { return hook(local, remote); });
|
||||
std::future<int> future = task.get_future();
|
||||
std::thread runner(std::move(task));
|
||||
int decision = 1;
|
||||
bool fallback = false;
|
||||
if (future.wait_for(std::chrono::milliseconds(common::Caps::kHookTimeoutMs)) ==
|
||||
std::future_status::ready) {
|
||||
try {
|
||||
decision = future.get();
|
||||
} catch (...) {
|
||||
fallback = true;
|
||||
}
|
||||
if (decision != 0 && decision != 1) {
|
||||
fallback = true;
|
||||
}
|
||||
} else {
|
||||
fallback = true;
|
||||
}
|
||||
runner.join();
|
||||
if (fallback) {
|
||||
observer_->event("replication.hook_fallback", local.model + ":" + local.key);
|
||||
ConflictOutcome outcome = resolve_lww(local, remote);
|
||||
outcome.detail = "hook fallback: " + outcome.detail;
|
||||
return outcome;
|
||||
}
|
||||
ConflictOutcome outcome;
|
||||
outcome.conflict = true;
|
||||
if (decision == 1) {
|
||||
outcome.remote_won = true;
|
||||
outcome.winner = remote;
|
||||
log_conflict("hook", remote, local);
|
||||
} else {
|
||||
outcome.winner = local;
|
||||
log_conflict("hook", local, remote);
|
||||
}
|
||||
outcome.winner_version = outcome.winner.version;
|
||||
outcome.detail = "hook decision=" + std::to_string(decision);
|
||||
return outcome;
|
||||
}
|
||||
|
||||
void ConflictResolver::log_conflict(const std::string& policy, const OplogEntry& winner,
|
||||
const OplogEntry& loser) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
while (static_cast<int>(log_.size()) >= max_log_entries_) {
|
||||
log_.pop_front();
|
||||
observer_->counter("molo_replication_conflict_log_evicted_total", 1);
|
||||
}
|
||||
log_.push_back(ConflictRecord{clock_.wall_ms(), policy, winner, loser});
|
||||
}
|
||||
|
||||
std::vector<ConflictRecord> ConflictResolver::conflict_log(std::size_t max) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<ConflictRecord> out;
|
||||
for (const auto& record : log_) {
|
||||
if (out.size() >= max) {
|
||||
break;
|
||||
}
|
||||
out.push_back(record);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::size_t ConflictResolver::conflict_log_size() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return log_.size();
|
||||
}
|
||||
|
||||
std::size_t ConflictResolver::purge_expired() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const std::int64_t horizon =
|
||||
clock_.wall_ms() - log_retention_days_ * 86400 * 1000;
|
||||
std::size_t purged = 0;
|
||||
while (!log_.empty() && log_.front().at_ms < horizon) {
|
||||
log_.pop_front();
|
||||
++purged;
|
||||
}
|
||||
return purged;
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,85 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <deque>
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/replication/oplog_entry.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
// Per-model conflict policy, declared once (S7-8, R7-7).
|
||||
enum class ConflictPolicy {
|
||||
Optimistic, // compare version, reject stale with ConflictError
|
||||
LastWriterWins, // higher hybrid timestamp wins, loser kept in conflict log
|
||||
Hook, // typed callback decides, 5s budget, LWW fallback
|
||||
};
|
||||
|
||||
std::string to_string(ConflictPolicy policy);
|
||||
|
||||
// Returns 0 when local wins, 1 when remote wins.
|
||||
using ConflictHook = std::function<int(const OplogEntry& local, const OplogEntry& remote)>;
|
||||
|
||||
struct ConflictOutcome {
|
||||
bool conflict = false; // true when both sides wrote concurrently
|
||||
bool remote_won = false; // true when the remote entry becomes current
|
||||
OplogEntry winner; // the entry that becomes current
|
||||
std::string detail; // human-readable reason, capped at 1 KiB
|
||||
bool conflict_error = false; // true when the caller must raise ConflictError
|
||||
long long winner_version = 0;
|
||||
};
|
||||
|
||||
struct ConflictRecord {
|
||||
std::int64_t at_ms = 0;
|
||||
std::string policy;
|
||||
OplogEntry winner;
|
||||
OplogEntry loser;
|
||||
};
|
||||
|
||||
// Dispatches per-model conflict policy (S7-8). The ONLY place that compares
|
||||
// versions or timestamps: callers never resolve conflicts ad hoc (R7-7).
|
||||
class ConflictResolver {
|
||||
public:
|
||||
ConflictResolver(common::Clock& clock, std::shared_ptr<common::Observer> observer,
|
||||
int max_log_entries = 10000, long log_retention_days = 7);
|
||||
|
||||
// Declares (or replaces) the policy for one model. Hook policy needs a
|
||||
// callback; without one the declaration falls back to LastWriterWins.
|
||||
void declare(const std::string& model, ConflictPolicy policy,
|
||||
ConflictHook hook = ConflictHook{});
|
||||
|
||||
ConflictPolicy policy_for(const std::string& model) const; // default: Optimistic
|
||||
|
||||
// Resolves local (current) vs remote (incoming) for the same model+key.
|
||||
ConflictOutcome resolve(const OplogEntry& local, const OplogEntry& remote);
|
||||
|
||||
std::vector<ConflictRecord> conflict_log(std::size_t max) const;
|
||||
std::size_t conflict_log_size() const;
|
||||
std::size_t purge_expired();
|
||||
|
||||
private:
|
||||
ConflictOutcome resolve_optimistic(const OplogEntry& local, const OplogEntry& remote);
|
||||
ConflictOutcome resolve_lww(const OplogEntry& local, const OplogEntry& remote);
|
||||
ConflictOutcome resolve_hook(const OplogEntry& local, const OplogEntry& remote,
|
||||
const ConflictHook& hook);
|
||||
void log_conflict(const std::string& policy, const OplogEntry& winner,
|
||||
const OplogEntry& loser);
|
||||
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
int max_log_entries_;
|
||||
long log_retention_days_;
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::string, ConflictPolicy> policies_;
|
||||
std::map<std::string, ConflictHook> hooks_;
|
||||
std::deque<ConflictRecord> log_;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,79 @@
|
||||
#include "supervisor/replication/hybrid_clock.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
int HybridTs::compare(const HybridTs& other) const {
|
||||
if (wall_ms != other.wall_ms) {
|
||||
return wall_ms < other.wall_ms ? -1 : 1;
|
||||
}
|
||||
if (counter != other.counter) {
|
||||
return counter < other.counter ? -1 : 1;
|
||||
}
|
||||
if (instance != other.instance) {
|
||||
return instance < other.instance ? -1 : 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
bool HybridTs::operator<(const HybridTs& other) const {
|
||||
return compare(other) < 0;
|
||||
}
|
||||
|
||||
bool HybridTs::operator==(const HybridTs& other) const {
|
||||
return compare(other) == 0;
|
||||
}
|
||||
|
||||
std::string HybridTs::encode() const {
|
||||
return std::to_string(wall_ms) + "." + std::to_string(counter) + "." + instance;
|
||||
}
|
||||
|
||||
bool HybridTs::decode(const std::string& text, HybridTs& out) {
|
||||
const std::size_t first = text.find('.');
|
||||
if (first == std::string::npos) {
|
||||
return false;
|
||||
}
|
||||
const std::size_t second = text.find('.', first + 1);
|
||||
if (second == std::string::npos) {
|
||||
return false;
|
||||
}
|
||||
try {
|
||||
out.wall_ms = std::stoll(text.substr(0, first));
|
||||
out.counter = static_cast<std::uint32_t>(std::stoul(text.substr(first + 1, second - first - 1)));
|
||||
} catch (...) {
|
||||
return false;
|
||||
}
|
||||
out.instance = text.substr(second + 1);
|
||||
return !out.instance.empty();
|
||||
}
|
||||
|
||||
HybridClock::HybridClock(std::string instance, common::Clock& clock)
|
||||
: instance_(std::move(instance)), clock_(clock) {}
|
||||
|
||||
HybridTs HybridClock::now() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const std::int64_t wall = clock_.wall_ms();
|
||||
if (wall > last_wall_ms_) {
|
||||
last_wall_ms_ = wall;
|
||||
counter_ = 0;
|
||||
} else {
|
||||
++counter_; // same ms (or skew): the counter keeps order
|
||||
}
|
||||
return HybridTs{last_wall_ms_, counter_, instance_};
|
||||
}
|
||||
|
||||
void HybridClock::observe(const HybridTs& remote) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const std::int64_t wall = clock_.wall_ms();
|
||||
if (remote.wall_ms > last_wall_ms_ && remote.wall_ms > wall) {
|
||||
last_wall_ms_ = remote.wall_ms;
|
||||
counter_ = remote.counter + 1;
|
||||
} else if (remote.wall_ms == last_wall_ms_ && remote.counter >= counter_) {
|
||||
counter_ = remote.counter + 1;
|
||||
} else if (wall > last_wall_ms_) {
|
||||
last_wall_ms_ = wall;
|
||||
counter_ = 0;
|
||||
}
|
||||
// else: remote is older, nothing to fold in
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,43 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
// Hybrid logical timestamp: wall clock plus a counter that disambiguates
|
||||
// events within the same millisecond and absorbs clock skew, with the
|
||||
// source instance id as the deterministic final tie-break (S7-7, S7-8).
|
||||
struct HybridTs {
|
||||
std::int64_t wall_ms = 0;
|
||||
std::uint32_t counter = 0;
|
||||
std::string instance;
|
||||
|
||||
// -1 when this < other, 0 when equal, 1 when this > other.
|
||||
int compare(const HybridTs& other) const;
|
||||
bool operator<(const HybridTs& other) const;
|
||||
bool operator==(const HybridTs& other) const;
|
||||
|
||||
std::string encode() const; // wall_ms.counter.instance
|
||||
static bool decode(const std::string& text, HybridTs& out);
|
||||
};
|
||||
|
||||
class HybridClock {
|
||||
public:
|
||||
HybridClock(std::string instance, common::Clock& clock);
|
||||
|
||||
HybridTs now();
|
||||
void observe(const HybridTs& remote); // folds a remote ts into the clock
|
||||
|
||||
private:
|
||||
std::string instance_;
|
||||
common::Clock& clock_;
|
||||
mutable std::mutex mutex_;
|
||||
std::int64_t last_wall_ms_ = 0;
|
||||
std::uint32_t counter_ = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,86 @@
|
||||
#include "supervisor/replication/oplog_entry.hpp"
|
||||
|
||||
#include "supervisor/common/json.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
std::string to_string(OplogOp op) {
|
||||
return op == OplogOp::Upsert ? "upsert" : "delete";
|
||||
}
|
||||
|
||||
bool parse_oplog_op(const std::string& text, OplogOp& out) {
|
||||
if (text == "upsert") {
|
||||
out = OplogOp::Upsert;
|
||||
} else if (text == "delete") {
|
||||
out = OplogOp::Delete;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string OplogEntry::encode() const {
|
||||
using common::JValue;
|
||||
JValue::Object obj;
|
||||
obj["seq"] = JValue(static_cast<long long>(seq));
|
||||
obj["instance"] = JValue(instance);
|
||||
obj["ts"] = JValue(ts.encode());
|
||||
obj["model"] = JValue(model);
|
||||
obj["key"] = JValue(key);
|
||||
obj["op"] = JValue(to_string(op));
|
||||
obj["version"] = JValue(version);
|
||||
JValue data_value;
|
||||
if (!common::json_decode(data, data_value)) {
|
||||
data_value = JValue(data);
|
||||
}
|
||||
obj["data"] = data_value;
|
||||
return common::json_encode(JValue(obj));
|
||||
}
|
||||
|
||||
bool OplogEntry::decode(const std::string& text, OplogEntry& out) {
|
||||
using common::JValue;
|
||||
JValue root;
|
||||
if (!common::json_decode(text, root) || !root.is_object()) {
|
||||
return false;
|
||||
}
|
||||
const JValue* instance = root.find("instance");
|
||||
const JValue* ts = root.find("ts");
|
||||
const JValue* model = root.find("model");
|
||||
const JValue* key = root.find("key");
|
||||
const JValue* op = root.find("op");
|
||||
if (instance == nullptr || ts == nullptr || model == nullptr || key == nullptr ||
|
||||
op == nullptr || !instance->is_string() || !ts->is_string() || !model->is_string() ||
|
||||
!key->is_string() || !op->is_string()) {
|
||||
return false;
|
||||
}
|
||||
OplogEntry entry;
|
||||
const JValue* seq = root.find("seq");
|
||||
if (seq != nullptr) {
|
||||
entry.seq = static_cast<std::uint64_t>(seq->as_int(0));
|
||||
}
|
||||
entry.instance = instance->as_string();
|
||||
if (!HybridTs::decode(ts->as_string(), entry.ts)) {
|
||||
return false;
|
||||
}
|
||||
entry.model = model->as_string();
|
||||
entry.key = key->as_string();
|
||||
if (!parse_oplog_op(op->as_string(), entry.op)) {
|
||||
return false;
|
||||
}
|
||||
const JValue* version = root.find("version");
|
||||
if (version != nullptr) {
|
||||
entry.version = version->as_int(0);
|
||||
}
|
||||
const JValue* data = root.find("data");
|
||||
if (data != nullptr) {
|
||||
entry.data = common::json_encode(*data);
|
||||
}
|
||||
out = std::move(entry);
|
||||
return true;
|
||||
}
|
||||
|
||||
std::size_t OplogEntry::approx_bytes() const {
|
||||
return instance.size() + model.size() + key.size() + data.size() + 64;
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,62 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
|
||||
#include "supervisor/replication/hybrid_clock.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
class ReplicationError : public std::runtime_error {
|
||||
public:
|
||||
explicit ReplicationError(const std::string& message) : std::runtime_error(message) {}
|
||||
};
|
||||
|
||||
// Raised to the loser of an optimistic-locking conflict, carrying the
|
||||
// winner version so the caller can re-read and retry (S7-8).
|
||||
class ConflictError : public ReplicationError {
|
||||
public:
|
||||
ConflictError(const std::string& model, const std::string& key, long long winner_version)
|
||||
: ReplicationError("conflict:" + model + ":" + key),
|
||||
model_(model),
|
||||
key_(key),
|
||||
winner_version_(winner_version) {}
|
||||
|
||||
const std::string& model() const { return model_; }
|
||||
const std::string& key() const { return key_; }
|
||||
long long winner_version() const { return winner_version_; }
|
||||
|
||||
private:
|
||||
std::string model_;
|
||||
std::string key_;
|
||||
long long winner_version_;
|
||||
};
|
||||
|
||||
enum class OplogOp {
|
||||
Upsert,
|
||||
Delete,
|
||||
};
|
||||
|
||||
std::string to_string(OplogOp op);
|
||||
bool parse_oplog_op(const std::string& text, OplogOp& out);
|
||||
|
||||
// One replication log entry (S7-7):
|
||||
// {"seq":n,"instance":"id","ts":"hybrid-clock","model":"name","key":"pk",
|
||||
// "op":"upsert|delete","version":n,"data":{}}
|
||||
struct OplogEntry {
|
||||
std::uint64_t seq = 0;
|
||||
std::string instance;
|
||||
HybridTs ts;
|
||||
std::string model;
|
||||
std::string key;
|
||||
OplogOp op = OplogOp::Upsert;
|
||||
long long version = 0;
|
||||
std::string data = "{}"; // opaque JSON object, owned by the model layer
|
||||
|
||||
std::string encode() const;
|
||||
static bool decode(const std::string& text, OplogEntry& out);
|
||||
std::size_t approx_bytes() const;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,47 @@
|
||||
#include "supervisor/replication/oplog_reader.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
OplogReader::OplogReader(const std::filesystem::path& dir)
|
||||
: log_(dir / "oplog.log", false, bus::SegmentLog::OpenMode::ReadOnly) {}
|
||||
|
||||
std::vector<OplogEntry> OplogReader::read_from(std::uint64_t seq, std::size_t max_entries,
|
||||
std::size_t max_bytes) const {
|
||||
log_.refresh();
|
||||
std::vector<OplogEntry> out;
|
||||
for (const auto& record : log_.read_from(seq, max_entries, max_bytes)) {
|
||||
OplogEntry entry;
|
||||
if (OplogEntry::decode(record.data, entry)) {
|
||||
out.push_back(std::move(entry));
|
||||
}
|
||||
// Corrupt entries are skipped: the segment checksum already truncates
|
||||
// torn tails, so anything left but undecodable is a loud gap for the
|
||||
// operator, surfaced by the missing seq in the consumer.
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<OplogEntry> OplogReader::poll(std::size_t max_entries, std::size_t max_bytes) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto entries = read_from(position_, max_entries, max_bytes);
|
||||
if (!entries.empty()) {
|
||||
position_ = entries.back().seq + 1;
|
||||
}
|
||||
return entries;
|
||||
}
|
||||
|
||||
std::uint64_t OplogReader::position() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return position_;
|
||||
}
|
||||
|
||||
void OplogReader::set_position(std::uint64_t seq) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
position_ = seq == 0 ? 1 : seq;
|
||||
}
|
||||
|
||||
std::uint64_t OplogReader::last_seq() const {
|
||||
return log_.last_seq();
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,36 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <filesystem>
|
||||
#include <mutex>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/bus/segment_log.hpp"
|
||||
#include "supervisor/replication/oplog_entry.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
// Tails one instance's persisted operation log: catch-up pull on reconnect
|
||||
// reads from a kept position, and poll() advances it (S7-7).
|
||||
class OplogReader {
|
||||
public:
|
||||
explicit OplogReader(const std::filesystem::path& dir);
|
||||
|
||||
// Reads up to max_entries / max_bytes from seq (1-based), decoded.
|
||||
std::vector<OplogEntry> read_from(std::uint64_t seq, std::size_t max_entries,
|
||||
std::size_t max_bytes) const;
|
||||
|
||||
// Polls ahead of the kept position and advances it past what was read.
|
||||
std::vector<OplogEntry> poll(std::size_t max_entries, std::size_t max_bytes);
|
||||
|
||||
std::uint64_t position() const;
|
||||
void set_position(std::uint64_t seq);
|
||||
std::uint64_t last_seq() const;
|
||||
|
||||
private:
|
||||
mutable bus::SegmentLog log_;
|
||||
mutable std::mutex mutex_;
|
||||
std::uint64_t position_ = 1;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,46 @@
|
||||
#include "supervisor/replication/oplog_writer.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
OplogWriter::OplogWriter(const std::filesystem::path& dir, std::string instance,
|
||||
common::Clock& clock, std::shared_ptr<common::Observer> observer)
|
||||
: instance_(std::move(instance)),
|
||||
clock_(clock),
|
||||
observer_(std::move(observer)),
|
||||
hybrid_(instance_, clock_),
|
||||
log_(std::make_unique<bus::SegmentLog>(dir / "oplog.log")),
|
||||
next_seq_(log_->last_seq() + 1) {}
|
||||
|
||||
OplogEntry OplogWriter::append(const std::string& model, const std::string& key, OplogOp op,
|
||||
long long version, const std::string& data) {
|
||||
if (model.empty() || model.size() > 256 || key.empty() || key.size() > 1024) {
|
||||
throw ReplicationError("OplogWriter: bad model or key");
|
||||
}
|
||||
if (data.size() > 1048576) {
|
||||
throw ReplicationError("OplogWriter: data over 1 MiB cap");
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
OplogEntry entry;
|
||||
entry.seq = next_seq_++;
|
||||
entry.instance = instance_;
|
||||
entry.ts = hybrid_.now();
|
||||
entry.model = model;
|
||||
entry.key = key;
|
||||
entry.op = op;
|
||||
entry.version = version;
|
||||
entry.data = data;
|
||||
log_->append(entry.encode(), clock_.wall_ms());
|
||||
observer_->counter("molo_replication_oplog_appended_total", 1);
|
||||
return entry;
|
||||
}
|
||||
|
||||
std::uint64_t OplogWriter::last_seq() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return next_seq_ == 0 ? 0 : next_seq_ - 1;
|
||||
}
|
||||
|
||||
const std::string& OplogWriter::instance() const {
|
||||
return instance_;
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,40 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <filesystem>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
|
||||
#include "supervisor/bus/segment_log.hpp"
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/replication/hybrid_clock.hpp"
|
||||
#include "supervisor/replication/oplog_entry.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
// Appends to one instance's persisted operation log (S7-7). Sequences are
|
||||
// assigned here, strictly increasing and stable across restarts.
|
||||
class OplogWriter {
|
||||
public:
|
||||
OplogWriter(const std::filesystem::path& dir, std::string instance, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer);
|
||||
|
||||
OplogEntry append(const std::string& model, const std::string& key, OplogOp op,
|
||||
long long version, const std::string& data);
|
||||
|
||||
std::uint64_t last_seq() const;
|
||||
const std::string& instance() const;
|
||||
|
||||
private:
|
||||
std::string instance_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
HybridClock hybrid_;
|
||||
std::unique_ptr<bus::SegmentLog> log_;
|
||||
mutable std::mutex mutex_;
|
||||
std::uint64_t next_seq_;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,122 @@
|
||||
#include "supervisor/replication/replicator.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
Replicator::Replicator(const std::filesystem::path& dir, std::string instance,
|
||||
common::Clock& clock, std::shared_ptr<common::Observer> observer)
|
||||
: instance_(std::move(instance)),
|
||||
clock_(clock),
|
||||
observer_(std::move(observer)),
|
||||
writer_(dir, instance_, clock_, observer_),
|
||||
reader_(dir),
|
||||
resolver_(clock_, observer_) {
|
||||
replay();
|
||||
}
|
||||
|
||||
const std::string& Replicator::instance() const {
|
||||
return instance_;
|
||||
}
|
||||
|
||||
void Replicator::declare_model(const std::string& model, ConflictPolicy policy,
|
||||
ConflictHook hook) {
|
||||
resolver_.declare(model, policy, std::move(hook));
|
||||
}
|
||||
|
||||
void Replicator::replay() {
|
||||
// Rebuild current state plus the version vector from the persisted log.
|
||||
// Replay applies entries in seq order without conflict checks: the log
|
||||
// already holds resolved history.
|
||||
std::uint64_t seq = 1;
|
||||
while (true) {
|
||||
auto entries = reader_.read_from(seq, 512, 4 * 1024 * 1024);
|
||||
if (entries.empty()) {
|
||||
break;
|
||||
}
|
||||
for (const auto& entry : entries) {
|
||||
current_[{entry.model, entry.key}] = entry;
|
||||
versions_.update(entry.instance, entry.seq);
|
||||
}
|
||||
seq = entries.back().seq + 1;
|
||||
}
|
||||
observer_->event("replication.replayed", "keys=" + std::to_string(current_.size()));
|
||||
}
|
||||
|
||||
OplogEntry Replicator::local_write(const std::string& model, const std::string& key, OplogOp op,
|
||||
long long version, const std::string& data) {
|
||||
OplogEntry entry = writer_.append(model, key, op, version, data);
|
||||
versions_.update(instance_, entry.seq);
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
current_[{model, key}] = entry;
|
||||
}
|
||||
observer_->event("replication.local_write", model + ":" + key);
|
||||
return entry;
|
||||
}
|
||||
|
||||
ConflictOutcome Replicator::apply_remote(const OplogEntry& entry) {
|
||||
if (entry.instance == instance_) {
|
||||
// Own entries echoed back by the bus are already current.
|
||||
ConflictOutcome outcome;
|
||||
outcome.winner = entry;
|
||||
return outcome;
|
||||
}
|
||||
std::optional<OplogEntry> local;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = current_.find({entry.model, entry.key});
|
||||
if (it != current_.end()) {
|
||||
local = it->second;
|
||||
}
|
||||
}
|
||||
ConflictOutcome outcome;
|
||||
if (!local.has_value()) {
|
||||
outcome.winner = entry;
|
||||
outcome.remote_won = true;
|
||||
outcome.detail = "no local value, remote adopted";
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
current_[{entry.model, entry.key}] = entry;
|
||||
} else if (local->instance == entry.instance && local->seq == entry.seq) {
|
||||
outcome.winner = *local; // exact duplicate delivery, idempotent
|
||||
outcome.detail = "duplicate delivery ignored";
|
||||
} else {
|
||||
outcome = resolver_.resolve(*local, entry);
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
current_[{entry.model, entry.key}] = outcome.winner;
|
||||
}
|
||||
versions_.update(entry.instance, entry.seq);
|
||||
observer_->counter("molo_replication_applied_total", 1);
|
||||
if (outcome.conflict_error) {
|
||||
throw ConflictError(entry.model, entry.key, outcome.winner_version);
|
||||
}
|
||||
return outcome;
|
||||
}
|
||||
|
||||
std::optional<OplogEntry> Replicator::current(const std::string& model,
|
||||
const std::string& key) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = current_.find({model, key});
|
||||
if (it == current_.end()) {
|
||||
return std::nullopt;
|
||||
}
|
||||
return it->second;
|
||||
}
|
||||
|
||||
std::vector<OplogEntry> Replicator::entries_since(std::uint64_t seq, std::size_t max_entries,
|
||||
std::size_t max_bytes) const {
|
||||
return reader_.read_from(seq, max_entries, max_bytes);
|
||||
}
|
||||
|
||||
void Replicator::versions(VersionVector& out) const {
|
||||
// Snapshot via encode/decode to avoid exposing internals.
|
||||
VersionVector::decode(versions_.encode(), out);
|
||||
}
|
||||
|
||||
ConflictResolver& Replicator::resolver() {
|
||||
return resolver_;
|
||||
}
|
||||
|
||||
std::uint64_t Replicator::last_seq() const {
|
||||
return writer_.last_seq();
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,73 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <filesystem>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/replication/conflict_resolver.hpp"
|
||||
#include "supervisor/replication/oplog_entry.hpp"
|
||||
#include "supervisor/replication/oplog_reader.hpp"
|
||||
#include "supervisor/replication/oplog_writer.hpp"
|
||||
#include "supervisor/replication/version_vector.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
// Per-instance replication endpoint (E7-7): owns the local oplog, the
|
||||
// current per-key state, per-model conflict policies, and the version
|
||||
// vector. Local writes append to the oplog and become current; remote
|
||||
// entries arrive through apply_remote() and resolve through the declared
|
||||
// policy. Replays the persisted oplog on construction so state survives
|
||||
// restarts. Thread-safe.
|
||||
class Replicator {
|
||||
public:
|
||||
Replicator(const std::filesystem::path& dir, std::string instance, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer);
|
||||
|
||||
const std::string& instance() const;
|
||||
|
||||
void declare_model(const std::string& model, ConflictPolicy policy,
|
||||
ConflictHook hook = ConflictHook{});
|
||||
|
||||
// Local write path. The caller supplies the model version (ORM row
|
||||
// version for optimistic models); the hybrid timestamp is assigned here.
|
||||
OplogEntry local_write(const std::string& model, const std::string& key, OplogOp op,
|
||||
long long version, const std::string& data);
|
||||
|
||||
// Remote apply path. Returns the resolution; throws ConflictError to
|
||||
// signal the loser of an optimistic conflict (S7-8).
|
||||
ConflictOutcome apply_remote(const OplogEntry& entry);
|
||||
|
||||
std::optional<OplogEntry> current(const std::string& model, const std::string& key) const;
|
||||
|
||||
// Entries strictly after seq for catch-up pull, bounded by the sync
|
||||
// batch caps (500 entries or 1 MiB per round, S7-7).
|
||||
std::vector<OplogEntry> entries_since(std::uint64_t seq, std::size_t max_entries,
|
||||
std::size_t max_bytes) const;
|
||||
|
||||
void versions(VersionVector& out) const; // snapshot of the version vector
|
||||
ConflictResolver& resolver();
|
||||
std::uint64_t last_seq() const;
|
||||
|
||||
private:
|
||||
void replay();
|
||||
|
||||
std::string instance_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
OplogWriter writer_;
|
||||
OplogReader reader_;
|
||||
ConflictResolver resolver_;
|
||||
VersionVector versions_;
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::pair<std::string, std::string>, OplogEntry> current_;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,73 @@
|
||||
#include "supervisor/replication/sync_session.hpp"
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
SyncSession::SyncSession(Replicator& local, SyncPeer& peer,
|
||||
std::shared_ptr<common::Observer> observer, std::size_t batch_entries,
|
||||
std::size_t batch_bytes)
|
||||
: local_(local),
|
||||
peer_(peer),
|
||||
observer_(std::move(observer)),
|
||||
batch_entries_(batch_entries),
|
||||
batch_bytes_(batch_bytes) {
|
||||
if (batch_entries_ == 0 || batch_entries_ > common::Caps::kSyncBatchEntries) {
|
||||
batch_entries_ = common::Caps::kSyncBatchEntries;
|
||||
}
|
||||
if (batch_bytes_ == 0 || batch_bytes_ > common::Caps::kSyncBatchBytes) {
|
||||
batch_bytes_ = common::Caps::kSyncBatchBytes;
|
||||
}
|
||||
}
|
||||
|
||||
SyncStats SyncSession::round_trip() {
|
||||
SyncStats stats;
|
||||
|
||||
// Pull: ask the peer for everything past our version vector.
|
||||
VersionVector have;
|
||||
local_.versions(have);
|
||||
std::vector<OplogEntry> incoming = peer_.fetch_since(have, batch_entries_, batch_bytes_);
|
||||
for (const auto& entry : incoming) {
|
||||
try {
|
||||
ConflictOutcome outcome = local_.apply_remote(entry);
|
||||
++stats.pulled;
|
||||
if (outcome.conflict) {
|
||||
++stats.conflicts;
|
||||
}
|
||||
} catch (const ConflictError&) {
|
||||
// Optimistic loser: counted as a conflict, sync continues with the
|
||||
// next entry. The ConflictError propagates to the sync caller only
|
||||
// in the sense of the counter plus event; entries are independent.
|
||||
++stats.pulled;
|
||||
++stats.conflicts;
|
||||
observer_->event("replication.sync_conflict", entry.model + ":" + entry.key);
|
||||
}
|
||||
}
|
||||
|
||||
// Push: send our own entries past what the peer has seen from us,
|
||||
// bounded to one batch per round.
|
||||
VersionVector peer_have;
|
||||
peer_.peer_versions(peer_have);
|
||||
const std::uint64_t peer_has = peer_have.get(local_.instance());
|
||||
std::vector<OplogEntry> outgoing =
|
||||
local_.entries_since(peer_has + 1, batch_entries_, batch_bytes_);
|
||||
if (!outgoing.empty()) {
|
||||
peer_.submit(outgoing);
|
||||
stats.pushed = outgoing.size();
|
||||
}
|
||||
|
||||
// Lag: entries behind the peer on the pull side, exact.
|
||||
VersionVector now_have;
|
||||
local_.versions(now_have);
|
||||
const std::uint64_t peer_last = peer_.last_seq();
|
||||
const std::uint64_t seen = now_have.get(peer_.instance());
|
||||
stats.lag = peer_last > seen ? static_cast<std::int64_t>(peer_last - seen) : 0;
|
||||
observer_->gauge("molo_replication_sync_lag", static_cast<double>(stats.lag));
|
||||
if (stats.lag >= common::Caps::kSyncLagAlertThreshold) {
|
||||
observer_->event("replication.lag_alert", peer_.instance());
|
||||
}
|
||||
observer_->counter("molo_replication_synced_total", 1);
|
||||
return stats;
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,63 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/replication/oplog_entry.hpp"
|
||||
#include "supervisor/replication/replicator.hpp"
|
||||
#include "supervisor/replication/version_vector.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
// Transport-agnostic peer accessor. Production binds this to the bus
|
||||
// (push) plus catch-up pull over the same channel (S7-7); tests bind it
|
||||
// to another in-process Replicator.
|
||||
class SyncPeer {
|
||||
public:
|
||||
virtual ~SyncPeer() = default;
|
||||
|
||||
// Entries the peer has that the given version vector has not seen,
|
||||
// bounded by the caller's batch caps.
|
||||
virtual std::vector<OplogEntry> fetch_since(const VersionVector& have, std::size_t max_entries,
|
||||
std::size_t max_bytes) = 0;
|
||||
|
||||
// Pushes local entries to the peer. The peer applies each through its
|
||||
// own resolver; duplicates are idempotent and ignored.
|
||||
virtual void submit(const std::vector<OplogEntry>& entries) = 0;
|
||||
|
||||
virtual void peer_versions(VersionVector& out) const = 0;
|
||||
virtual std::uint64_t last_seq() const = 0;
|
||||
virtual std::string instance() const = 0;
|
||||
};
|
||||
|
||||
struct SyncStats {
|
||||
std::size_t pulled = 0;
|
||||
std::size_t pushed = 0;
|
||||
std::size_t conflicts = 0;
|
||||
std::int64_t lag = 0; // entries behind after this round
|
||||
};
|
||||
|
||||
// One bidirectional sync round between the local Replicator and a peer:
|
||||
// pull what the peer has, push what it lacks (S7-7). Batch caps are 500
|
||||
// entries or 1 MiB per round; lag is exported as a gauge with the 10000
|
||||
// alert threshold from S7-7.
|
||||
class SyncSession {
|
||||
public:
|
||||
SyncSession(Replicator& local, SyncPeer& peer, std::shared_ptr<common::Observer> observer,
|
||||
std::size_t batch_entries = 500, std::size_t batch_bytes = 1048576);
|
||||
|
||||
SyncStats round_trip();
|
||||
|
||||
private:
|
||||
Replicator& local_;
|
||||
SyncPeer& peer_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
std::size_t batch_entries_;
|
||||
std::size_t batch_bytes_;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,96 @@
|
||||
#include "supervisor/replication/version_vector.hpp"
|
||||
|
||||
#include "supervisor/common/json.hpp"
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
void VersionVector::update(const std::string& instance, std::uint64_t seq) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = versions_.find(instance);
|
||||
if (it == versions_.end() || seq > it->second) {
|
||||
versions_[instance] = seq;
|
||||
}
|
||||
}
|
||||
|
||||
std::uint64_t VersionVector::get(const std::string& instance) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = versions_.find(instance);
|
||||
return it == versions_.end() ? 0 : it->second;
|
||||
}
|
||||
|
||||
void VersionVector::merge(const VersionVector& other) {
|
||||
if (this == &other) {
|
||||
return;
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::lock_guard<std::mutex> other_lock(other.mutex_);
|
||||
for (const auto& [instance, seq] : other.versions_) {
|
||||
auto it = versions_.find(instance);
|
||||
if (it == versions_.end() || seq > it->second) {
|
||||
versions_[instance] = seq;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
VectorOrder VersionVector::compare(const VersionVector& other) const {
|
||||
if (this == &other) {
|
||||
return VectorOrder::Equal;
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::lock_guard<std::mutex> other_lock(other.mutex_);
|
||||
bool less = false, greater = false;
|
||||
for (const auto& [instance, seq] : versions_) {
|
||||
auto it = other.versions_.find(instance);
|
||||
const std::uint64_t oseq = it == other.versions_.end() ? 0 : it->second;
|
||||
if (seq < oseq) {
|
||||
less = true;
|
||||
} else if (seq > oseq) {
|
||||
greater = true;
|
||||
}
|
||||
}
|
||||
for (const auto& [instance, seq] : other.versions_) {
|
||||
if (versions_.find(instance) == versions_.end() && seq > 0) {
|
||||
less = true;
|
||||
}
|
||||
}
|
||||
if (less && greater) {
|
||||
return VectorOrder::Concurrent;
|
||||
}
|
||||
if (less) {
|
||||
return VectorOrder::Before;
|
||||
}
|
||||
if (greater) {
|
||||
return VectorOrder::After;
|
||||
}
|
||||
return VectorOrder::Equal;
|
||||
}
|
||||
|
||||
std::string VersionVector::encode() const {
|
||||
using common::JValue;
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
JValue::Object obj;
|
||||
for (const auto& [instance, seq] : versions_) {
|
||||
obj[instance] = JValue(static_cast<long long>(seq));
|
||||
}
|
||||
return common::json_encode(JValue(obj));
|
||||
}
|
||||
|
||||
bool VersionVector::decode(const std::string& text, VersionVector& out) {
|
||||
using common::JValue;
|
||||
JValue root;
|
||||
if (!common::json_decode(text, root) || !root.is_object()) {
|
||||
return false;
|
||||
}
|
||||
VersionVector parsed;
|
||||
for (const auto& [instance, value] : root.as_object()) {
|
||||
if (!value.is_int() && !value.is_double()) {
|
||||
return false;
|
||||
}
|
||||
parsed.versions_[instance] = static_cast<std::uint64_t>(value.as_int(0));
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(out.mutex_);
|
||||
out.versions_ = std::move(parsed.versions_);
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,34 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
|
||||
namespace molo::replication {
|
||||
|
||||
enum class VectorOrder {
|
||||
Equal,
|
||||
Before, // this <= other with at least one strict <
|
||||
After, // this >= other with at least one strict >
|
||||
Concurrent, // neither dominates: a real conflict happened
|
||||
};
|
||||
|
||||
// Version vector over per-instance oplog sequences, exchanged at sync so
|
||||
// each side fetches only what the other has not seen (S7-7).
|
||||
class VersionVector {
|
||||
public:
|
||||
void update(const std::string& instance, std::uint64_t seq);
|
||||
std::uint64_t get(const std::string& instance) const;
|
||||
void merge(const VersionVector& other);
|
||||
VectorOrder compare(const VersionVector& other) const;
|
||||
|
||||
std::string encode() const; // JSON object {instance: seq}
|
||||
static bool decode(const std::string& text, VersionVector& out);
|
||||
|
||||
private:
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::string, std::uint64_t> versions_;
|
||||
};
|
||||
|
||||
} // namespace molo::replication
|
||||
@@ -0,0 +1,78 @@
|
||||
# Part 07 (supervisor, tasks, bus, replication) build fragment.
|
||||
# Included by the root Makefile. Double-colon rules let every part append
|
||||
# its own all/test/clean steps without editing shared targets.
|
||||
|
||||
SUP_CXX ?= g++
|
||||
SUP_CXXFLAGS ?= -std=c++23 -O2 -Wall -Wextra -Wpedantic -Werror -MMD -MP
|
||||
SUP_ROOT = src/supervisor
|
||||
SUP_BUILD = build/supervisor
|
||||
SUP_LIB = $(SUP_BUILD)/libsupervisor.a
|
||||
SUP_INCPATH = -Isrc
|
||||
|
||||
SUP_SRCS = \
|
||||
$(SUP_ROOT)/common/clock.cpp \
|
||||
$(SUP_ROOT)/common/observer.cpp \
|
||||
$(SUP_ROOT)/common/kv_store.cpp \
|
||||
$(SUP_ROOT)/common/json.cpp \
|
||||
$(SUP_ROOT)/bus/segment_log.cpp \
|
||||
$(SUP_ROOT)/bus/topic_bus.cpp \
|
||||
$(SUP_ROOT)/bus/consumer_cursor.cpp \
|
||||
$(SUP_ROOT)/bus/ack_tracker.cpp \
|
||||
$(SUP_ROOT)/bus/stream_bus.cpp \
|
||||
$(SUP_ROOT)/tasks/task_envelope.cpp \
|
||||
$(SUP_ROOT)/tasks/task_handler.cpp \
|
||||
$(SUP_ROOT)/tasks/task_registry.cpp \
|
||||
$(SUP_ROOT)/tasks/retry_policy.cpp \
|
||||
$(SUP_ROOT)/tasks/dead_letter_queue.cpp \
|
||||
$(SUP_ROOT)/tasks/task_queue.cpp \
|
||||
$(SUP_ROOT)/tasks/cron.cpp \
|
||||
$(SUP_ROOT)/tasks/scheduler.cpp \
|
||||
$(SUP_ROOT)/supervisor/restart_policy.cpp \
|
||||
$(SUP_ROOT)/supervisor/worker_process.cpp \
|
||||
$(SUP_ROOT)/supervisor/health_check.cpp \
|
||||
$(SUP_ROOT)/supervisor/worker_group.cpp \
|
||||
$(SUP_ROOT)/supervisor/shutdown_coordinator.cpp \
|
||||
$(SUP_ROOT)/supervisor/supervisor.cpp \
|
||||
$(SUP_ROOT)/replication/hybrid_clock.cpp \
|
||||
$(SUP_ROOT)/replication/oplog_entry.cpp \
|
||||
$(SUP_ROOT)/replication/oplog_writer.cpp \
|
||||
$(SUP_ROOT)/replication/oplog_reader.cpp \
|
||||
$(SUP_ROOT)/replication/version_vector.cpp \
|
||||
$(SUP_ROOT)/replication/conflict_resolver.cpp \
|
||||
$(SUP_ROOT)/replication/replicator.cpp \
|
||||
$(SUP_ROOT)/replication/sync_session.cpp
|
||||
|
||||
SUP_OBJS = $(SUP_SRCS:$(SUP_ROOT)/%.cpp=$(SUP_BUILD)/%.o)
|
||||
|
||||
SUP_TESTS = \
|
||||
test_supervisor \
|
||||
test_tasks \
|
||||
test_scheduler \
|
||||
test_bus \
|
||||
test_durability \
|
||||
test_replication \
|
||||
test_multi_instance
|
||||
|
||||
SUP_TEST_BINS = $(SUP_TESTS:%=$(SUP_BUILD)/%)
|
||||
|
||||
all:: $(SUP_LIB)
|
||||
|
||||
$(SUP_LIB): $(SUP_OBJS)
|
||||
@mkdir -p $(SUP_BUILD)
|
||||
ar rcs $@ $^
|
||||
|
||||
$(SUP_BUILD)/%.o: $(SUP_ROOT)/%.cpp
|
||||
@mkdir -p $(dir $@)
|
||||
$(SUP_CXX) $(SUP_CXXFLAGS) $(SUP_INCPATH) -c $< -o $@
|
||||
|
||||
$(SUP_BUILD)/test_%: tests/src/supervisor/test_%.cpp $(SUP_LIB) $(OPERABILITY_LIB)
|
||||
@mkdir -p $(SUP_BUILD)
|
||||
$(SUP_CXX) $(SUP_CXXFLAGS) $(SUP_INCPATH) $< $(SUP_LIB) $(OPERABILITY_LIB) -o $@
|
||||
|
||||
test:: $(SUP_TEST_BINS)
|
||||
@for t in $(SUP_TEST_BINS); do echo "== $$t"; ./$$t || exit 1; done
|
||||
|
||||
clean::
|
||||
rm -rf $(SUP_BUILD)
|
||||
|
||||
-include $(SUP_OBJS:.o=.d)
|
||||
@@ -0,0 +1,150 @@
|
||||
#include "supervisor/supervisor/health_check.hpp"
|
||||
|
||||
#if !defined(_WIN32)
|
||||
#include <fcntl.h>
|
||||
#include <netdb.h>
|
||||
#include <sys/select.h>
|
||||
#include <sys/socket.h>
|
||||
#include <unistd.h>
|
||||
#endif
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
HealthCheck::HealthCheck(std::string name, const HealthSpec& spec,
|
||||
std::function<bool()> func_probe, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer)
|
||||
: name_(std::move(name)),
|
||||
spec_(spec),
|
||||
func_probe_(std::move(func_probe)),
|
||||
clock_(clock),
|
||||
observer_(std::move(observer)) {
|
||||
if (spec_.interval_ms < 100) {
|
||||
spec_.interval_ms = 100;
|
||||
}
|
||||
if (spec_.max_failures < 1) {
|
||||
spec_.max_failures = 1;
|
||||
}
|
||||
}
|
||||
|
||||
bool HealthCheck::enabled() const {
|
||||
if (spec_.kind == ProbeKind::Function) {
|
||||
return static_cast<bool>(func_probe_);
|
||||
}
|
||||
return spec_.tcp_port > 0;
|
||||
}
|
||||
|
||||
bool HealthCheck::due() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!enabled()) {
|
||||
return false;
|
||||
}
|
||||
return clock_.steady_ms() - last_probe_steady_ms_ >= spec_.interval_ms;
|
||||
}
|
||||
|
||||
bool HealthCheck::check_now() {
|
||||
bool healthy = true;
|
||||
if (spec_.kind == ProbeKind::Function) {
|
||||
if (!func_probe_) {
|
||||
return true;
|
||||
}
|
||||
try {
|
||||
healthy = func_probe_();
|
||||
} catch (...) {
|
||||
healthy = false;
|
||||
}
|
||||
} else {
|
||||
std::string error;
|
||||
healthy = tcp_probe(spec_.tcp_host, spec_.tcp_port, spec_.timeout_ms, error);
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
last_probe_steady_ms_ = clock_.steady_ms();
|
||||
if (healthy) {
|
||||
failures_ = 0;
|
||||
} else {
|
||||
++failures_;
|
||||
observer_->counter("molo_supervisor_health_failures_total", 1);
|
||||
if (failures_ >= spec_.max_failures) {
|
||||
observer_->event("supervisor.worker.unhealthy", name_);
|
||||
}
|
||||
}
|
||||
return healthy;
|
||||
}
|
||||
|
||||
bool HealthCheck::unhealthy() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return enabled() && failures_ >= spec_.max_failures;
|
||||
}
|
||||
|
||||
int HealthCheck::failures() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return failures_;
|
||||
}
|
||||
|
||||
void HealthCheck::reset() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
failures_ = 0;
|
||||
last_probe_steady_ms_ = 0;
|
||||
}
|
||||
|
||||
bool HealthCheck::tcp_probe(const std::string& host, int port, long timeout_ms,
|
||||
std::string& error) {
|
||||
#if defined(_WIN32)
|
||||
error = "tcp probe unsupported on this platform";
|
||||
(void)host;
|
||||
(void)port;
|
||||
(void)timeout_ms;
|
||||
return false;
|
||||
#else
|
||||
addrinfo hints{};
|
||||
hints.ai_family = AF_UNSPEC;
|
||||
hints.ai_socktype = SOCK_STREAM;
|
||||
addrinfo* list = nullptr;
|
||||
if (getaddrinfo(host.c_str(), std::to_string(port).c_str(), &hints, &list) != 0) {
|
||||
error = "dns failure";
|
||||
return false;
|
||||
}
|
||||
int fd = -1;
|
||||
for (addrinfo* ai = list; ai != nullptr; ai = ai->ai_next) {
|
||||
fd = ::socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
|
||||
if (fd >= 0) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (fd < 0) {
|
||||
freeaddrinfo(list);
|
||||
error = "socket failure";
|
||||
return false;
|
||||
}
|
||||
const int flags = fcntl(fd, F_GETFL, 0);
|
||||
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
|
||||
int rc = ::connect(fd, list->ai_addr, list->ai_addrlen);
|
||||
bool ok = (rc == 0);
|
||||
if (!ok && errno == EINPROGRESS) {
|
||||
fd_set wfds;
|
||||
FD_ZERO(&wfds);
|
||||
FD_SET(fd, &wfds);
|
||||
timeval tv{};
|
||||
tv.tv_sec = timeout_ms / 1000;
|
||||
tv.tv_usec = (timeout_ms % 1000) * 1000;
|
||||
rc = select(fd + 1, nullptr, &wfds, nullptr, &tv);
|
||||
if (rc > 0) {
|
||||
int so_error = 0;
|
||||
socklen_t len = sizeof(so_error);
|
||||
getsockopt(fd, SOL_SOCKET, SO_ERROR, &so_error, &len);
|
||||
ok = (so_error == 0);
|
||||
if (!ok) {
|
||||
error = "connect refused";
|
||||
}
|
||||
} else {
|
||||
error = "connect timeout";
|
||||
}
|
||||
} else if (!ok) {
|
||||
error = "connect failure";
|
||||
}
|
||||
freeaddrinfo(list);
|
||||
::close(fd);
|
||||
return ok;
|
||||
#endif
|
||||
}
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,58 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
enum class ProbeKind {
|
||||
Function, // caller-supplied predicate
|
||||
Tcp, // TCP connect probe (S7-4)
|
||||
};
|
||||
|
||||
struct HealthSpec {
|
||||
ProbeKind kind = ProbeKind::Function;
|
||||
std::string tcp_host = "127.0.0.1";
|
||||
int tcp_port = 0;
|
||||
long interval_ms = 10000;
|
||||
int max_failures = 3;
|
||||
long timeout_ms = 2000;
|
||||
};
|
||||
|
||||
// Health checker for one supervised worker (S7-4). The group calls due()
|
||||
// and check_now() from its monitor loop; 3 consecutive probe failures
|
||||
// (default) mark the worker unhealthy and trigger the restart policy.
|
||||
class HealthCheck {
|
||||
public:
|
||||
HealthCheck(std::string name, const HealthSpec& spec, std::function<bool()> func_probe,
|
||||
common::Clock& clock, std::shared_ptr<common::Observer> observer);
|
||||
|
||||
bool enabled() const;
|
||||
bool due() const; // true when a probe should run now
|
||||
bool check_now(); // runs one probe, returns healthy
|
||||
bool unhealthy() const; // true after max_failures consecutive failures
|
||||
int failures() const;
|
||||
void reset();
|
||||
|
||||
// Raw TCP connect probe used for ProbeKind::Tcp.
|
||||
static bool tcp_probe(const std::string& host, int port, long timeout_ms,
|
||||
std::string& error);
|
||||
|
||||
private:
|
||||
std::string name_;
|
||||
HealthSpec spec_;
|
||||
std::function<bool()> func_probe_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
mutable std::mutex mutex_;
|
||||
int failures_ = 0;
|
||||
std::int64_t last_probe_steady_ms_ = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,38 @@
|
||||
#include "supervisor/supervisor/restart_policy.hpp"
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
std::string to_string(RestartPolicy policy) {
|
||||
switch (policy) {
|
||||
case RestartPolicy::Never: return "never";
|
||||
case RestartPolicy::OnFailure: return "on-failure";
|
||||
case RestartPolicy::Always: return "always";
|
||||
}
|
||||
return "on-failure";
|
||||
}
|
||||
|
||||
bool parse_restart_policy(const std::string& text, RestartPolicy& out) {
|
||||
if (text == "never") {
|
||||
out = RestartPolicy::Never;
|
||||
} else if (text == "on-failure") {
|
||||
out = RestartPolicy::OnFailure;
|
||||
} else if (text == "always") {
|
||||
out = RestartPolicy::Always;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void RestartSpec::normalize() {
|
||||
if (max_restarts < 0) {
|
||||
max_restarts = common::Caps::kMaxRestartsDefault;
|
||||
}
|
||||
if (window_secs < 1) {
|
||||
window_secs = common::Caps::kRestartWindowSecsDefault;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,28 @@
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
// Restart policy per supervised program (BP-2, R7-3).
|
||||
enum class RestartPolicy {
|
||||
Never, // leave exited children alone
|
||||
OnFailure, // restart only on nonzero exit or crash
|
||||
Always, // restart on any exit
|
||||
};
|
||||
|
||||
std::string to_string(RestartPolicy policy);
|
||||
bool parse_restart_policy(const std::string& text, RestartPolicy& out);
|
||||
|
||||
// RestartPolicy plus the intensity cap that makes restart loops bounded:
|
||||
// at most max_restarts restarts per window_secs window. Crossing the cap
|
||||
// moves the child to Fatal and emits an event (S7-4).
|
||||
struct RestartSpec {
|
||||
RestartPolicy policy = RestartPolicy::OnFailure;
|
||||
int max_restarts = 5;
|
||||
int window_secs = 60;
|
||||
|
||||
void normalize(); // clamps nonsense to the spec defaults
|
||||
};
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,50 @@
|
||||
#include "supervisor/supervisor/shutdown_coordinator.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
std::string to_string(ShutdownPhase phase) {
|
||||
switch (phase) {
|
||||
case ShutdownPhase::Running: return "running";
|
||||
case ShutdownPhase::Draining: return "draining";
|
||||
case ShutdownPhase::Stopped: return "stopped";
|
||||
}
|
||||
return "running";
|
||||
}
|
||||
|
||||
ShutdownCoordinator::ShutdownCoordinator(common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer)
|
||||
: clock_(clock), observer_(std::move(observer)) {}
|
||||
|
||||
ShutdownPhase ShutdownCoordinator::phase() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return phase_;
|
||||
}
|
||||
|
||||
bool ShutdownCoordinator::drain_until(const std::function<bool()>& is_drained, long deadline_ms,
|
||||
long poll_ms) {
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
phase_ = ShutdownPhase::Draining;
|
||||
}
|
||||
observer_->event("supervisor.shutdown.draining", "deadline_ms=" + std::to_string(deadline_ms));
|
||||
const std::int64_t deadline = clock_.steady_ms() + deadline_ms;
|
||||
while (true) {
|
||||
if (is_drained && is_drained()) {
|
||||
observer_->event("supervisor.shutdown.drained", "clean");
|
||||
return true;
|
||||
}
|
||||
if (clock_.steady_ms() >= deadline) {
|
||||
observer_->event("supervisor.shutdown.drain_expired", "deadline hit");
|
||||
return false;
|
||||
}
|
||||
clock_.sleep_ms(poll_ms);
|
||||
}
|
||||
}
|
||||
|
||||
void ShutdownCoordinator::mark_stopped() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
phase_ = ShutdownPhase::Stopped;
|
||||
observer_->event("supervisor.shutdown.stopped", "");
|
||||
}
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,40 @@
|
||||
#pragma once
|
||||
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
enum class ShutdownPhase {
|
||||
Running,
|
||||
Draining,
|
||||
Stopped,
|
||||
};
|
||||
|
||||
std::string to_string(ShutdownPhase phase);
|
||||
|
||||
// Graceful shutdown with a deadline (R7-9): stop accepting new work, drain
|
||||
// in-flight work until the deadline, then stop. drain_until() polls the
|
||||
// caller-supplied predicate and reports whether the drain completed.
|
||||
class ShutdownCoordinator {
|
||||
public:
|
||||
ShutdownCoordinator(common::Clock& clock, std::shared_ptr<common::Observer> observer);
|
||||
|
||||
ShutdownPhase phase() const;
|
||||
bool drain_until(const std::function<bool()>& is_drained, long deadline_ms,
|
||||
long poll_ms = 50);
|
||||
void mark_stopped();
|
||||
|
||||
private:
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
mutable std::mutex mutex_;
|
||||
ShutdownPhase phase_ = ShutdownPhase::Running;
|
||||
};
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,153 @@
|
||||
#include "supervisor/supervisor/supervisor.hpp"
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
#include "supervisor/common/json.hpp"
|
||||
#include "supervisor/supervisor/supervisor_error.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
Supervisor::Supervisor(common::Clock& clock, std::shared_ptr<common::Observer> observer,
|
||||
const SupervisorOptions& options)
|
||||
: clock_(clock), observer_(std::move(observer)), options_(options), shutdown_(clock, observer_) {
|
||||
}
|
||||
|
||||
Supervisor::~Supervisor() {
|
||||
if (running()) {
|
||||
shutdown(0);
|
||||
}
|
||||
}
|
||||
|
||||
std::size_t Supervisor::add_group(const GroupSpec& spec, WorkFunc func) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (static_cast<int>(groups_.size()) >= common::Caps::kGroupsPerSupervisor) {
|
||||
throw SupervisorError("Supervisor: group cap reached");
|
||||
}
|
||||
auto group = std::make_unique<WorkerGroup>(spec, std::move(func), clock_, observer_);
|
||||
if (running_) {
|
||||
group->start();
|
||||
}
|
||||
groups_.push_back(std::move(group));
|
||||
observer_->event("supervisor.group.added", spec.name);
|
||||
return groups_.size() - 1;
|
||||
}
|
||||
|
||||
void Supervisor::set_task_queue(std::shared_ptr<tasks::TaskQueue> queue) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
queue_ = std::move(queue);
|
||||
}
|
||||
|
||||
void Supervisor::start_all() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (running_) {
|
||||
return;
|
||||
}
|
||||
running_ = true;
|
||||
for (auto& group : groups_) {
|
||||
group->start();
|
||||
}
|
||||
observer_->event("supervisor.started", "groups=" + std::to_string(groups_.size()));
|
||||
}
|
||||
|
||||
void Supervisor::shutdown(long drain_deadline_ms) {
|
||||
const long deadline = drain_deadline_ms < 0 ? options_.default_drain_ms : drain_deadline_ms;
|
||||
std::shared_ptr<tasks::TaskQueue> queue;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!running_) {
|
||||
return;
|
||||
}
|
||||
queue = queue_;
|
||||
}
|
||||
if (queue) {
|
||||
queue->set_accepting(false);
|
||||
const bool drained = shutdown_.drain_until(
|
||||
[&] { return queue->in_flight() == 0; }, deadline);
|
||||
if (!drained) {
|
||||
queue->abandon_all_to_dlq("shutdown-drain-expired");
|
||||
}
|
||||
queue->shutdown();
|
||||
} else {
|
||||
shutdown_.drain_until([] { return true; }, 0);
|
||||
}
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (auto it = groups_.rbegin(); it != groups_.rend(); ++it) {
|
||||
(*it)->stop();
|
||||
}
|
||||
running_ = false;
|
||||
}
|
||||
shutdown_.mark_stopped();
|
||||
}
|
||||
|
||||
void Supervisor::restart_group(std::size_t index) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (index >= groups_.size()) {
|
||||
throw SupervisorError("Supervisor: group index out of range");
|
||||
}
|
||||
groups_[index]->restart_all();
|
||||
}
|
||||
|
||||
void Supervisor::scale_group(std::size_t index, std::size_t n) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (index >= groups_.size()) {
|
||||
throw SupervisorError("Supervisor: group index out of range");
|
||||
}
|
||||
groups_[index]->set_size(n);
|
||||
}
|
||||
|
||||
std::string Supervisor::status_json() const {
|
||||
using common::JValue;
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
JValue::Object root;
|
||||
root["phase"] = JValue(to_string(shutdown_.phase()));
|
||||
root["running"] = JValue(running_);
|
||||
JValue::Array groups;
|
||||
for (const auto& group : groups_) {
|
||||
groups.push_back(group->snapshot());
|
||||
}
|
||||
root["groups"] = JValue(groups);
|
||||
if (queue_) {
|
||||
const tasks::QueueStats stats = queue_->stats();
|
||||
JValue::Object q;
|
||||
q["queued"] = JValue(static_cast<long long>(stats.queued));
|
||||
q["leased"] = JValue(static_cast<long long>(stats.leased));
|
||||
q["running"] = JValue(static_cast<long long>(stats.running));
|
||||
q["succeeded_total"] = JValue(static_cast<long long>(stats.succeeded_total));
|
||||
q["failed_total"] = JValue(static_cast<long long>(stats.failed_total));
|
||||
q["retried_total"] = JValue(static_cast<long long>(stats.retried_total));
|
||||
q["dead_lettered"] = JValue(static_cast<long long>(queue_->dead_letters().size()));
|
||||
q["accepting"] = JValue(queue_->accepting());
|
||||
root["queue"] = JValue(q);
|
||||
}
|
||||
std::string full = common::json_encode(JValue(root));
|
||||
if (full.size() <= common::Caps::kStatusSnapshotBytes) {
|
||||
return full;
|
||||
}
|
||||
// Over the 1 MiB cap: compact form with counts only, never truncated JSON.
|
||||
JValue::Object compact;
|
||||
compact["truncated"] = JValue(true);
|
||||
compact["phase"] = JValue(to_string(shutdown_.phase()));
|
||||
compact["group_count"] = JValue(static_cast<long long>(groups_.size()));
|
||||
long long workers = 0;
|
||||
for (const auto& group : groups_) {
|
||||
workers += static_cast<long long>(group->size());
|
||||
}
|
||||
compact["worker_count"] = JValue(workers);
|
||||
return common::json_encode(JValue(compact));
|
||||
}
|
||||
|
||||
std::size_t Supervisor::group_count() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return groups_.size();
|
||||
}
|
||||
|
||||
bool Supervisor::running() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return running_;
|
||||
}
|
||||
|
||||
ShutdownPhase Supervisor::phase() const {
|
||||
return shutdown_.phase();
|
||||
}
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,68 @@
|
||||
#pragma once
|
||||
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/supervisor/shutdown_coordinator.hpp"
|
||||
#include "supervisor/supervisor/supervisor_error.hpp"
|
||||
#include "supervisor/supervisor/worker_group.hpp"
|
||||
#include "supervisor/tasks/task_queue.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
struct SupervisorOptions {
|
||||
long default_drain_ms = 30000;
|
||||
};
|
||||
|
||||
// Top-level supervisor (E7-1): owns worker groups, applies restart
|
||||
// strategies through them, and coordinates graceful shutdown with the
|
||||
// task queue (R7-9). Exposes the status snapshot consumed by the CLI
|
||||
// (09-cli-ops.md) and health endpoints. Thread-safe.
|
||||
class Supervisor {
|
||||
public:
|
||||
Supervisor(common::Clock& clock, std::shared_ptr<common::Observer> observer,
|
||||
const SupervisorOptions& options = SupervisorOptions{});
|
||||
~Supervisor();
|
||||
|
||||
Supervisor(const Supervisor&) = delete;
|
||||
Supervisor& operator=(const Supervisor&) = delete;
|
||||
|
||||
// Adds a group (max 32). When the supervisor runs, the group starts now.
|
||||
std::size_t add_group(const GroupSpec& spec, WorkFunc func);
|
||||
|
||||
// Queue drained on shutdown. Optional; null means workers only.
|
||||
void set_task_queue(std::shared_ptr<tasks::TaskQueue> queue);
|
||||
|
||||
void start_all();
|
||||
|
||||
// Graceful shutdown: stop accepting new tasks, drain in-flight tasks
|
||||
// until the deadline, dead-letter leftovers with a reason, then stop
|
||||
// all groups. Dropped tasks never vanish (R7-9).
|
||||
void shutdown(long drain_deadline_ms = -1); // -1 means the default
|
||||
|
||||
void restart_group(std::size_t index);
|
||||
void scale_group(std::size_t index, std::size_t n);
|
||||
|
||||
// Status snapshot JSON for CLI and health endpoints, capped at 1 MiB.
|
||||
std::string status_json() const;
|
||||
|
||||
std::size_t group_count() const;
|
||||
bool running() const;
|
||||
ShutdownPhase phase() const;
|
||||
|
||||
private:
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
SupervisorOptions options_;
|
||||
ShutdownCoordinator shutdown_;
|
||||
mutable std::mutex mutex_;
|
||||
std::vector<std::unique_ptr<WorkerGroup>> groups_;
|
||||
std::shared_ptr<tasks::TaskQueue> queue_;
|
||||
bool running_ = false;
|
||||
};
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,13 @@
|
||||
#pragma once
|
||||
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
class SupervisorError : public std::runtime_error {
|
||||
public:
|
||||
explicit SupervisorError(const std::string& message) : std::runtime_error(message) {}
|
||||
};
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,481 @@
|
||||
#include "supervisor/supervisor/worker_group.hpp"
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
#include "supervisor/supervisor/supervisor_error.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
std::string to_string(GroupStrategy strategy) {
|
||||
switch (strategy) {
|
||||
case GroupStrategy::OneForOne: return "one_for_one";
|
||||
case GroupStrategy::OneForAll: return "one_for_all";
|
||||
case GroupStrategy::RestForOne: return "rest_for_one";
|
||||
}
|
||||
return "one_for_one";
|
||||
}
|
||||
|
||||
WorkerGroup::WorkerGroup(const GroupSpec& spec, WorkFunc func, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer)
|
||||
: spec_(spec), func_(std::move(func)), clock_(clock), observer_(std::move(observer)) {
|
||||
if (spec_.name.empty() || spec_.name.size() > 128) {
|
||||
throw SupervisorError("WorkerGroup: bad group name");
|
||||
}
|
||||
if (!func_) {
|
||||
throw SupervisorError("WorkerGroup: null work function");
|
||||
}
|
||||
if (spec_.size < 1 || spec_.size > static_cast<std::size_t>(common::Caps::kWorkersPerGroup)) {
|
||||
throw SupervisorError("WorkerGroup: size out of range");
|
||||
}
|
||||
spec_.restart.normalize();
|
||||
if (spec_.monitor_ms < 5) {
|
||||
spec_.monitor_ms = 5;
|
||||
}
|
||||
if (spec_.backoff_min_ms < 0) {
|
||||
spec_.backoff_min_ms = 0;
|
||||
}
|
||||
if (spec_.backoff_max_ms < spec_.backoff_min_ms) {
|
||||
spec_.backoff_max_ms = spec_.backoff_min_ms;
|
||||
}
|
||||
for (std::size_t i = 0; i < spec_.size; ++i) {
|
||||
auto child = std::make_unique<Child>();
|
||||
child->worker = std::make_shared<WorkerProcess>(spec_.name + "#" + std::to_string(i),
|
||||
func_, clock_, observer_);
|
||||
child->health = std::make_shared<HealthCheck>(spec_.name + "#" + std::to_string(i),
|
||||
spec_.health, spec_.probe, clock_, observer_);
|
||||
child->backoff_ms = spec_.backoff_min_ms;
|
||||
children_.push_back(std::move(child));
|
||||
}
|
||||
}
|
||||
|
||||
WorkerGroup::~WorkerGroup() {
|
||||
stop();
|
||||
}
|
||||
|
||||
void WorkerGroup::start() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (running_) {
|
||||
return;
|
||||
}
|
||||
running_ = true;
|
||||
stopping_ = false;
|
||||
for (auto& child : children_) {
|
||||
child->worker->reset_for_restart();
|
||||
child->worker->start();
|
||||
child->health->reset();
|
||||
}
|
||||
monitor_ = std::thread(&WorkerGroup::monitor_main, this);
|
||||
observer_->event("supervisor.group.started", spec_.name);
|
||||
}
|
||||
|
||||
void WorkerGroup::stop() {
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!running_) {
|
||||
return;
|
||||
}
|
||||
stopping_ = true;
|
||||
monitor_cv_.notify_all();
|
||||
}
|
||||
if (monitor_.joinable()) {
|
||||
monitor_.join();
|
||||
}
|
||||
std::vector<std::shared_ptr<WorkerProcess>> workers;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (auto& child : children_) {
|
||||
workers.push_back(child->worker);
|
||||
child->worker->request_stop();
|
||||
}
|
||||
pending_.clear();
|
||||
}
|
||||
for (auto& worker : workers) {
|
||||
worker->join();
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
running_ = false;
|
||||
stopping_ = false;
|
||||
observer_->event("supervisor.group.stopped", spec_.name);
|
||||
}
|
||||
|
||||
bool WorkerGroup::running() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return running_;
|
||||
}
|
||||
|
||||
void WorkerGroup::set_size(std::size_t n) {
|
||||
if (n < 1 || n > static_cast<std::size_t>(common::Caps::kWorkersPerGroup)) {
|
||||
throw SupervisorError("WorkerGroup: size out of range");
|
||||
}
|
||||
std::vector<std::shared_ptr<WorkerProcess>> removed;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (n > children_.size()) {
|
||||
for (std::size_t i = children_.size(); i < n; ++i) {
|
||||
auto child = std::make_unique<Child>();
|
||||
child->worker = std::make_shared<WorkerProcess>(spec_.name + "#" + std::to_string(i),
|
||||
func_, clock_, observer_);
|
||||
child->health =
|
||||
std::make_shared<HealthCheck>(spec_.name + "#" + std::to_string(i), spec_.health,
|
||||
spec_.probe, clock_, observer_);
|
||||
child->backoff_ms = spec_.backoff_min_ms;
|
||||
if (running_) {
|
||||
child->worker->start();
|
||||
}
|
||||
children_.push_back(std::move(child));
|
||||
}
|
||||
} else {
|
||||
while (children_.size() > n) {
|
||||
auto& back = children_.back();
|
||||
back->worker->request_stop();
|
||||
removed.push_back(back->worker);
|
||||
pending_.erase(children_.size() - 1);
|
||||
children_.pop_back();
|
||||
}
|
||||
}
|
||||
observer_->event("supervisor.group.scaled",
|
||||
spec_.name + " size=" + std::to_string(children_.size()));
|
||||
}
|
||||
for (auto& worker : removed) {
|
||||
worker->join();
|
||||
}
|
||||
}
|
||||
|
||||
std::size_t WorkerGroup::size() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return children_.size();
|
||||
}
|
||||
|
||||
void WorkerGroup::restart_child(std::size_t index) {
|
||||
std::shared_ptr<WorkerProcess> worker;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (index >= children_.size()) {
|
||||
throw SupervisorError("WorkerGroup: child index out of range");
|
||||
}
|
||||
worker = children_[index]->worker;
|
||||
children_[index]->restarts.clear();
|
||||
children_[index]->backoff_ms = spec_.backoff_min_ms;
|
||||
children_[index]->health->reset();
|
||||
pending_.erase(index);
|
||||
}
|
||||
worker->request_stop();
|
||||
worker->join();
|
||||
worker->reset_for_restart();
|
||||
worker->start();
|
||||
observer_->event("supervisor.group.child_restarted", spec_.name + "#" + std::to_string(index));
|
||||
}
|
||||
|
||||
void WorkerGroup::restart_all() {
|
||||
std::vector<std::shared_ptr<WorkerProcess>> workers;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (auto& child : children_) {
|
||||
workers.push_back(child->worker);
|
||||
child->worker->request_stop();
|
||||
child->restarts.clear();
|
||||
child->backoff_ms = spec_.backoff_min_ms;
|
||||
child->health->reset();
|
||||
}
|
||||
pending_.clear();
|
||||
}
|
||||
for (auto& worker : workers) {
|
||||
worker->join();
|
||||
}
|
||||
for (auto& worker : workers) {
|
||||
worker->reset_for_restart();
|
||||
worker->start();
|
||||
}
|
||||
observer_->event("supervisor.group.restarted", spec_.name);
|
||||
}
|
||||
|
||||
bool WorkerGroup::fatal() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (const auto& child : children_) {
|
||||
if (child->worker->state() == WorkerState::Fatal) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
std::vector<WorkerState> WorkerGroup::states() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<WorkerState> out;
|
||||
for (const auto& child : children_) {
|
||||
out.push_back(child->worker->state());
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
common::JValue WorkerGroup::snapshot() const {
|
||||
using common::JValue;
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
JValue::Object obj;
|
||||
obj["name"] = JValue(spec_.name);
|
||||
obj["strategy"] = JValue(to_string(spec_.strategy));
|
||||
obj["restart_policy"] = JValue(to_string(spec_.restart.policy));
|
||||
obj["running"] = JValue(running_);
|
||||
JValue::Array workers;
|
||||
bool any_fatal = false;
|
||||
for (const auto& child : children_) {
|
||||
const WorkerState state = child->worker->state();
|
||||
if (state == WorkerState::Fatal) {
|
||||
any_fatal = true;
|
||||
}
|
||||
JValue::Object w;
|
||||
w["name"] = JValue(child->worker->name());
|
||||
w["state"] = JValue(to_string(state));
|
||||
w["exit_code"] = JValue(child->worker->exit_code());
|
||||
w["starts"] = JValue(static_cast<long long>(child->worker->starts()));
|
||||
w["restarts_in_window"] = JValue(static_cast<long long>(child->restarts.size()));
|
||||
w["health_failures"] = JValue(child->health->failures());
|
||||
workers.push_back(JValue(w));
|
||||
}
|
||||
obj["fatal"] = JValue(any_fatal);
|
||||
obj["workers"] = JValue(workers);
|
||||
return JValue(obj);
|
||||
}
|
||||
|
||||
const std::string& WorkerGroup::name() const {
|
||||
return spec_.name;
|
||||
}
|
||||
|
||||
std::vector<std::size_t> WorkerGroup::strategy_targets(std::size_t index) const {
|
||||
std::vector<std::size_t> targets;
|
||||
switch (spec_.strategy) {
|
||||
case GroupStrategy::OneForOne: targets.push_back(index); break;
|
||||
case GroupStrategy::OneForAll:
|
||||
for (std::size_t i = 0; i < children_.size(); ++i) {
|
||||
targets.push_back(i);
|
||||
}
|
||||
break;
|
||||
case GroupStrategy::RestForOne:
|
||||
for (std::size_t i = index; i < children_.size(); ++i) {
|
||||
targets.push_back(i);
|
||||
}
|
||||
break;
|
||||
}
|
||||
return targets;
|
||||
}
|
||||
|
||||
void WorkerGroup::handle_exit_locked(std::size_t index, int exit_code, bool from_health,
|
||||
std::vector<std::size_t>& stop_list) {
|
||||
bool wanted = false;
|
||||
if (from_health) {
|
||||
wanted = (spec_.restart.policy != RestartPolicy::Never);
|
||||
} else if (spec_.restart.policy == RestartPolicy::Always) {
|
||||
wanted = true;
|
||||
} else if (spec_.restart.policy == RestartPolicy::OnFailure && exit_code != 0) {
|
||||
wanted = true;
|
||||
}
|
||||
if (!wanted) {
|
||||
return;
|
||||
}
|
||||
for (std::size_t target : strategy_targets(index)) {
|
||||
if (target >= children_.size()) {
|
||||
continue;
|
||||
}
|
||||
if (pending_.find(target) != pending_.end()) {
|
||||
continue; // restart already scheduled
|
||||
}
|
||||
const WorkerState state = children_[target]->worker->state();
|
||||
if (state == WorkerState::Fatal || state == WorkerState::Stopped) {
|
||||
continue;
|
||||
}
|
||||
if (state == WorkerState::Exited) {
|
||||
schedule_restart_locked(target);
|
||||
} else if (state == WorkerState::Running || state == WorkerState::Starting) {
|
||||
// Strategy siblings (or an unhealthy worker) stop first, then the
|
||||
// restart is scheduled after the join outside the lock.
|
||||
stop_list.push_back(target);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void WorkerGroup::schedule_restart_locked(std::size_t index) {
|
||||
Child& child = *children_[index];
|
||||
const std::int64_t now = clock_.steady_ms();
|
||||
const std::int64_t window_ms = static_cast<std::int64_t>(spec_.restart.window_secs) * 1000;
|
||||
while (!child.restarts.empty() && child.restarts.front() <= now - window_ms) {
|
||||
child.restarts.pop_front();
|
||||
}
|
||||
if (static_cast<int>(child.restarts.size()) >= spec_.restart.max_restarts) {
|
||||
child.worker->mark_fatal();
|
||||
observer_->event("supervisor.group.intensity_cap", spec_.name + " child locked fatal");
|
||||
return;
|
||||
}
|
||||
child.restarts.push_back(now);
|
||||
child.last_restart_steady_ms = now;
|
||||
pending_[index] = now + child.backoff_ms;
|
||||
child.backoff_ms = std::min(child.backoff_ms * 2, spec_.backoff_max_ms);
|
||||
if (child.worker->state() == WorkerState::Exited) {
|
||||
child.worker->mark_backoff();
|
||||
}
|
||||
observer_->counter("molo_supervisor_worker_restarts_total", 1);
|
||||
}
|
||||
|
||||
void WorkerGroup::fire_restart(std::size_t index) {
|
||||
std::shared_ptr<WorkerProcess> worker;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (stopping_ || index >= children_.size()) {
|
||||
return;
|
||||
}
|
||||
worker = children_[index]->worker;
|
||||
const WorkerState state = worker->state();
|
||||
// Stopped here means a strategy sibling stopped for restart (group
|
||||
// stop clears all pending restarts first, so it cannot land here).
|
||||
if (state == WorkerState::Fatal || state == WorkerState::Running ||
|
||||
state == WorkerState::Starting || state == WorkerState::Stopping) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
worker->join(); // collects the old thread; already finished, non-blocking
|
||||
worker->reset_for_restart();
|
||||
if (!worker->start()) {
|
||||
return;
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (index < children_.size()) {
|
||||
children_[index]->health->reset();
|
||||
}
|
||||
}
|
||||
|
||||
void WorkerGroup::monitor_main() {
|
||||
while (true) {
|
||||
std::vector<std::size_t> exited;
|
||||
std::vector<std::size_t> probes;
|
||||
std::vector<std::size_t> firing;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (stopping_) {
|
||||
return;
|
||||
}
|
||||
const std::int64_t now = clock_.steady_ms();
|
||||
for (auto it = pending_.begin(); it != pending_.end();) {
|
||||
if (it->second <= now) {
|
||||
firing.push_back(it->first);
|
||||
it = pending_.erase(it);
|
||||
} else {
|
||||
++it;
|
||||
}
|
||||
}
|
||||
for (std::size_t i = 0; i < children_.size(); ++i) {
|
||||
const WorkerState state = children_[i]->worker->state();
|
||||
if (state == WorkerState::Exited) {
|
||||
exited.push_back(i);
|
||||
} else if (state == WorkerState::Running && children_[i]->health->due()) {
|
||||
probes.push_back(i);
|
||||
}
|
||||
// A clean run resets the backoff ladder (supervisord-style).
|
||||
if (state == WorkerState::Running &&
|
||||
children_[i]->backoff_ms > spec_.backoff_min_ms &&
|
||||
now - children_[i]->last_restart_steady_ms > spec_.backoff_reset_after_ms) {
|
||||
children_[i]->backoff_ms = spec_.backoff_min_ms;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (std::size_t index : exited) {
|
||||
std::shared_ptr<WorkerProcess> worker;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (index < children_.size()) {
|
||||
worker = children_[index]->worker;
|
||||
}
|
||||
}
|
||||
if (worker) {
|
||||
worker->join();
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<std::size_t> unhealthy;
|
||||
for (std::size_t index : probes) {
|
||||
std::shared_ptr<HealthCheck> health;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (index < children_.size()) {
|
||||
health = children_[index]->health;
|
||||
}
|
||||
}
|
||||
if (health) {
|
||||
health->check_now();
|
||||
if (health->unhealthy()) {
|
||||
unhealthy.push_back(index);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<std::size_t> stop_list;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (stopping_) {
|
||||
return;
|
||||
}
|
||||
for (std::size_t index : exited) {
|
||||
if (index >= children_.size()) {
|
||||
continue;
|
||||
}
|
||||
if (children_[index]->worker->state() != WorkerState::Exited) {
|
||||
continue; // user restarted it meanwhile
|
||||
}
|
||||
const int code = children_[index]->worker->exit_code();
|
||||
handle_exit_locked(index, code, false, stop_list);
|
||||
}
|
||||
for (std::size_t index : unhealthy) {
|
||||
if (index >= children_.size()) {
|
||||
continue;
|
||||
}
|
||||
if (children_[index]->worker->state() != WorkerState::Running) {
|
||||
continue;
|
||||
}
|
||||
observer_->event("supervisor.group.unhealthy_restart",
|
||||
spec_.name + "#" + std::to_string(index));
|
||||
handle_exit_locked(index, 1, true, stop_list);
|
||||
}
|
||||
}
|
||||
|
||||
for (std::size_t index : stop_list) {
|
||||
std::shared_ptr<WorkerProcess> worker;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (index < children_.size()) {
|
||||
worker = children_[index]->worker;
|
||||
}
|
||||
}
|
||||
if (worker) {
|
||||
worker->request_stop();
|
||||
worker->join();
|
||||
}
|
||||
}
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!stopping_) {
|
||||
for (std::size_t index : stop_list) {
|
||||
if (index >= children_.size()) {
|
||||
continue;
|
||||
}
|
||||
const WorkerState state = children_[index]->worker->state();
|
||||
if (state == WorkerState::Stopped || state == WorkerState::Exited) {
|
||||
schedule_restart_locked(index);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (std::size_t index : firing) {
|
||||
fire_restart(index);
|
||||
}
|
||||
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(mutex_);
|
||||
monitor_cv_.wait_for(lock, std::chrono::milliseconds(spec_.monitor_ms),
|
||||
[&] { return stopping_; });
|
||||
if (stopping_) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,102 @@
|
||||
#pragma once
|
||||
|
||||
#include <atomic>
|
||||
#include <condition_variable>
|
||||
#include <cstdint>
|
||||
#include <deque>
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/json.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/supervisor/health_check.hpp"
|
||||
#include "supervisor/supervisor/restart_policy.hpp"
|
||||
#include "supervisor/supervisor/worker_process.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
// OTP restart strategies for worker groups (BP-1).
|
||||
enum class GroupStrategy {
|
||||
OneForOne, // restart only the failed child
|
||||
OneForAll, // restart all children
|
||||
RestForOne, // restart the failed child plus children started after it
|
||||
};
|
||||
|
||||
std::string to_string(GroupStrategy strategy);
|
||||
|
||||
struct GroupSpec {
|
||||
std::string name;
|
||||
GroupStrategy strategy = GroupStrategy::OneForOne;
|
||||
RestartSpec restart;
|
||||
HealthSpec health;
|
||||
std::function<bool()> probe; // shared function probe (Function kind)
|
||||
std::size_t size = 1;
|
||||
long monitor_ms = 50;
|
||||
long backoff_min_ms = 1000;
|
||||
long backoff_max_ms = 60000;
|
||||
long backoff_reset_after_ms = 10000; // clean run that resets backoff
|
||||
};
|
||||
|
||||
// Owns one supervised program: N WorkerProcess children running the same
|
||||
// WorkFunc, one monitor loop applying the restart strategy, intensity
|
||||
// caps, backoff, and health checks (E7-1, S7-4). Thread-safe.
|
||||
class WorkerGroup {
|
||||
public:
|
||||
WorkerGroup(const GroupSpec& spec, WorkFunc func, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer);
|
||||
~WorkerGroup();
|
||||
|
||||
WorkerGroup(const WorkerGroup&) = delete;
|
||||
WorkerGroup& operator=(const WorkerGroup&) = delete;
|
||||
|
||||
void start();
|
||||
void stop();
|
||||
bool running() const;
|
||||
|
||||
void set_size(std::size_t n); // scale, capped at 64 workers
|
||||
std::size_t size() const;
|
||||
|
||||
void restart_child(std::size_t index); // explicit, clears Fatal
|
||||
void restart_all();
|
||||
bool fatal() const; // true when any child is Fatal
|
||||
|
||||
std::vector<WorkerState> states() const;
|
||||
common::JValue snapshot() const;
|
||||
const std::string& name() const;
|
||||
|
||||
private:
|
||||
struct Child {
|
||||
std::shared_ptr<WorkerProcess> worker;
|
||||
std::shared_ptr<HealthCheck> health;
|
||||
std::deque<std::int64_t> restarts; // steady-ms timestamps in window
|
||||
long backoff_ms = 1000;
|
||||
std::int64_t last_restart_steady_ms = 0;
|
||||
};
|
||||
|
||||
void monitor_main();
|
||||
void handle_exit_locked(std::size_t index, int exit_code, bool from_health,
|
||||
std::vector<std::size_t>& stop_list);
|
||||
void schedule_restart_locked(std::size_t index);
|
||||
void fire_restart(std::size_t index);
|
||||
std::vector<std::size_t> strategy_targets(std::size_t index) const;
|
||||
|
||||
GroupSpec spec_;
|
||||
WorkFunc func_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
mutable std::mutex mutex_;
|
||||
std::condition_variable monitor_cv_;
|
||||
std::vector<std::unique_ptr<Child>> children_;
|
||||
std::map<std::size_t, std::int64_t> pending_; // index -> restart due (steady ms)
|
||||
std::thread monitor_;
|
||||
bool running_ = false;
|
||||
bool stopping_ = false;
|
||||
};
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,150 @@
|
||||
#include "supervisor/supervisor/worker_process.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
WorkerProcess::WorkerProcess(std::string name, WorkFunc func, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer)
|
||||
: name_(std::move(name)),
|
||||
func_(std::move(func)),
|
||||
clock_(clock),
|
||||
observer_(std::move(observer)) {}
|
||||
|
||||
WorkerProcess::~WorkerProcess() {
|
||||
request_stop();
|
||||
join();
|
||||
}
|
||||
|
||||
bool WorkerProcess::start() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (state_ == WorkerState::Running || state_ == WorkerState::Starting ||
|
||||
state_ == WorkerState::Stopping) {
|
||||
return false;
|
||||
}
|
||||
if (state_ == WorkerState::Fatal) {
|
||||
return false; // explicit reset_for_restart() required
|
||||
}
|
||||
if (thread_.joinable()) {
|
||||
return false;
|
||||
}
|
||||
stop_requested_.store(false);
|
||||
state_ = WorkerState::Starting;
|
||||
last_start_steady_ms_ = clock_.steady_ms();
|
||||
++starts_;
|
||||
observer_->event("supervisor.worker.starting", name_);
|
||||
thread_ = std::thread(&WorkerProcess::thread_main, this);
|
||||
return true;
|
||||
}
|
||||
|
||||
void WorkerProcess::thread_main() {
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
state_ = WorkerState::Running;
|
||||
}
|
||||
observer_->event("supervisor.worker.running", name_);
|
||||
observer_->counter("molo_supervisor_worker_starts_total", 1);
|
||||
int code = 0;
|
||||
try {
|
||||
code = func_(stop_requested_);
|
||||
} catch (...) {
|
||||
code = 1;
|
||||
}
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
exit_code_ = code;
|
||||
last_exit_steady_ms_ = clock_.steady_ms();
|
||||
state_ = stop_requested_.load() ? WorkerState::Stopped : WorkerState::Exited;
|
||||
}
|
||||
observer_->event("supervisor.worker.exited", name_ + " code=" + std::to_string(code));
|
||||
}
|
||||
|
||||
void WorkerProcess::request_stop() {
|
||||
stop_requested_.store(true);
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (state_ == WorkerState::Running || state_ == WorkerState::Starting) {
|
||||
state_ = WorkerState::Stopping;
|
||||
}
|
||||
}
|
||||
|
||||
void WorkerProcess::join() {
|
||||
// Move the handle out under lock, then join without it: thread_main
|
||||
// needs the state mutex to publish Exited.
|
||||
std::thread finished;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (thread_.joinable()) {
|
||||
finished = std::move(thread_);
|
||||
}
|
||||
}
|
||||
if (finished.joinable()) {
|
||||
finished.join();
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
// A worker that was asked to stop and already exited lands on Stopped;
|
||||
// one that exited on its own keeps Exited for the group to handle.
|
||||
if (stop_requested_.load() && state_ == WorkerState::Exited) {
|
||||
state_ = WorkerState::Stopped;
|
||||
}
|
||||
}
|
||||
|
||||
bool WorkerProcess::joinable() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return thread_.joinable();
|
||||
}
|
||||
|
||||
WorkerState WorkerProcess::state() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return state_;
|
||||
}
|
||||
|
||||
const std::string& WorkerProcess::name() const {
|
||||
return name_;
|
||||
}
|
||||
|
||||
int WorkerProcess::exit_code() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return exit_code_;
|
||||
}
|
||||
|
||||
std::uint64_t WorkerProcess::starts() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return starts_;
|
||||
}
|
||||
|
||||
std::int64_t WorkerProcess::last_start_steady_ms() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return last_start_steady_ms_;
|
||||
}
|
||||
|
||||
std::int64_t WorkerProcess::last_exit_steady_ms() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return last_exit_steady_ms_;
|
||||
}
|
||||
|
||||
void WorkerProcess::mark_fatal() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
// Any non-running state may go fatal: Exited and Backoff after a crash,
|
||||
// Stopped after a strategy sibling stop. Live workers never do.
|
||||
if (state_ != WorkerState::Running && state_ != WorkerState::Starting &&
|
||||
state_ != WorkerState::Stopping && state_ != WorkerState::Fatal) {
|
||||
state_ = WorkerState::Fatal;
|
||||
observer_->event("supervisor.worker.fatal", name_);
|
||||
observer_->counter("molo_supervisor_worker_fatal_total", 1);
|
||||
}
|
||||
}
|
||||
|
||||
void WorkerProcess::mark_backoff() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (state_ == WorkerState::Exited) {
|
||||
state_ = WorkerState::Backoff;
|
||||
}
|
||||
}
|
||||
|
||||
void WorkerProcess::reset_for_restart() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (state_ == WorkerState::Fatal || state_ == WorkerState::Exited ||
|
||||
state_ == WorkerState::Stopped) {
|
||||
state_ = WorkerState::Stopped;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,71 @@
|
||||
#pragma once
|
||||
|
||||
#include <atomic>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/supervisor/worker_state.hpp"
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
// Work function run by a worker. Returns the process exit code (0 means
|
||||
// clean). Must poll stop_requested and return promptly; throwing counts
|
||||
// as exit code 1. Transport-agnostic: v1 runs this on a thread, v2 runs
|
||||
// the same signature in an OS process behind this interface (O7-2).
|
||||
using WorkFunc = std::function<int(const std::atomic<bool>& stop_requested)>;
|
||||
|
||||
// One supervised child (v1: in-process thread worker). Owns its lifecycle
|
||||
// only; restart decisions live in WorkerGroup. Thread-safe.
|
||||
class WorkerProcess {
|
||||
public:
|
||||
WorkerProcess(std::string name, WorkFunc func, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer);
|
||||
~WorkerProcess();
|
||||
|
||||
WorkerProcess(const WorkerProcess&) = delete;
|
||||
WorkerProcess& operator=(const WorkerProcess&) = delete;
|
||||
|
||||
// Starts the worker. Returns false when already started.
|
||||
bool start();
|
||||
void request_stop();
|
||||
void join();
|
||||
bool joinable() const;
|
||||
|
||||
WorkerState state() const;
|
||||
const std::string& name() const;
|
||||
int exit_code() const;
|
||||
std::uint64_t starts() const;
|
||||
std::int64_t last_start_steady_ms() const;
|
||||
std::int64_t last_exit_steady_ms() const;
|
||||
|
||||
// Called by the group when the intensity cap trips (S7-4).
|
||||
void mark_fatal();
|
||||
// Called by the group while a restart backoff is pending.
|
||||
void mark_backoff();
|
||||
// Clears Fatal/Exited so start() may run again (explicit restart only).
|
||||
void reset_for_restart();
|
||||
|
||||
private:
|
||||
void thread_main();
|
||||
|
||||
std::string name_;
|
||||
WorkFunc func_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
mutable std::mutex mutex_;
|
||||
std::thread thread_;
|
||||
std::atomic<bool> stop_requested_{false};
|
||||
WorkerState state_ = WorkerState::Stopped;
|
||||
int exit_code_ = 0;
|
||||
std::uint64_t starts_ = 0;
|
||||
std::int64_t last_start_steady_ms_ = 0;
|
||||
std::int64_t last_exit_steady_ms_ = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,32 @@
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
|
||||
namespace molo::supervisor {
|
||||
|
||||
// Supervised worker states (S7-4). Fatal is entered after the intensity
|
||||
// cap trips; only an explicit restart clears it.
|
||||
enum class WorkerState {
|
||||
Stopped,
|
||||
Starting,
|
||||
Running,
|
||||
Backoff,
|
||||
Stopping,
|
||||
Exited,
|
||||
Fatal,
|
||||
};
|
||||
|
||||
inline std::string to_string(WorkerState state) {
|
||||
switch (state) {
|
||||
case WorkerState::Stopped: return "stopped";
|
||||
case WorkerState::Starting: return "starting";
|
||||
case WorkerState::Running: return "running";
|
||||
case WorkerState::Backoff: return "backoff";
|
||||
case WorkerState::Stopping: return "stopping";
|
||||
case WorkerState::Exited: return "exited";
|
||||
case WorkerState::Fatal: return "fatal";
|
||||
}
|
||||
return "stopped";
|
||||
}
|
||||
|
||||
} // namespace molo::supervisor
|
||||
@@ -0,0 +1,187 @@
|
||||
#include "supervisor/tasks/cron.hpp"
|
||||
|
||||
#include <ctime>
|
||||
#include <vector>
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
namespace {
|
||||
|
||||
std::vector<std::string> split_fields(std::string_view text) {
|
||||
std::vector<std::string> fields;
|
||||
std::string current;
|
||||
for (char c : text) {
|
||||
if (c == ' ' || c == '\t') {
|
||||
if (!current.empty()) {
|
||||
fields.push_back(current);
|
||||
current.clear();
|
||||
}
|
||||
} else {
|
||||
current += c;
|
||||
}
|
||||
}
|
||||
if (!current.empty()) {
|
||||
fields.push_back(current);
|
||||
}
|
||||
return fields;
|
||||
}
|
||||
|
||||
bool parse_int(std::string_view text, int& out) {
|
||||
if (text.empty() || text.size() > 4) {
|
||||
return false;
|
||||
}
|
||||
int value = 0;
|
||||
for (char c : text) {
|
||||
if (c < '0' || c > '9') {
|
||||
return false;
|
||||
}
|
||||
value = value * 10 + (c - '0');
|
||||
}
|
||||
out = value;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Parses one field into flags[low..high]. Returns false on bad syntax.
|
||||
bool parse_field(std::string_view text, bool* flags, int low, int high, bool& restricted) {
|
||||
for (int i = low; i <= high; ++i) {
|
||||
flags[i] = false;
|
||||
}
|
||||
restricted = (text != "*");
|
||||
std::size_t pos = 0;
|
||||
bool any = false;
|
||||
while (pos <= text.size()) {
|
||||
std::size_t comma = text.find(',', pos);
|
||||
if (comma == std::string_view::npos) {
|
||||
comma = text.size();
|
||||
}
|
||||
const std::string_view item = text.substr(pos, comma - pos);
|
||||
if (item.empty()) {
|
||||
return false;
|
||||
}
|
||||
// Split off /step.
|
||||
std::string_view range = item;
|
||||
int step = 1;
|
||||
const std::size_t slash = item.find('/');
|
||||
if (slash != std::string_view::npos) {
|
||||
range = item.substr(0, slash);
|
||||
if (!parse_int(item.substr(slash + 1), step) || step < 1) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
int from = low, to = high;
|
||||
if (range == "*") {
|
||||
from = low;
|
||||
to = high;
|
||||
} else {
|
||||
const std::size_t dash = range.find('-');
|
||||
if (dash == std::string_view::npos) {
|
||||
if (!parse_int(range, from)) {
|
||||
return false;
|
||||
}
|
||||
to = (slash == std::string_view::npos) ? from : high;
|
||||
} else {
|
||||
if (!parse_int(range.substr(0, dash), from) ||
|
||||
!parse_int(range.substr(dash + 1), to)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (from < low || to > high || from > to) {
|
||||
return false;
|
||||
}
|
||||
for (int v = from; v <= to; v += step) {
|
||||
flags[v] = true;
|
||||
any = true;
|
||||
}
|
||||
if (comma == text.size()) {
|
||||
break;
|
||||
}
|
||||
pos = comma + 1;
|
||||
}
|
||||
return any;
|
||||
}
|
||||
|
||||
bool parse_dow_field(std::string_view text, bool* flags, bool& restricted) {
|
||||
// Normalize 7 to 0 before parsing.
|
||||
std::string norm(text);
|
||||
for (std::size_t i = 0; i < norm.size(); ++i) {
|
||||
if (norm[i] == '7' && (i + 1 >= norm.size() || norm[i + 1] < '0' || norm[i + 1] > '9') &&
|
||||
(i == 0 || norm[i - 1] < '0' || norm[i - 1] > '9')) {
|
||||
norm[i] = '0';
|
||||
}
|
||||
}
|
||||
return parse_field(norm, flags, 0, 6, restricted);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
bool CronExpr::parse(std::string_view text, CronExpr& out, std::string& error) {
|
||||
CronExpr expr;
|
||||
const auto fields = split_fields(text);
|
||||
if (fields.size() != 5) {
|
||||
error = "cron: need exactly 5 fields";
|
||||
return false;
|
||||
}
|
||||
bool ignored = false;
|
||||
if (!parse_field(fields[0], expr.minutes_, 0, 59, ignored)) {
|
||||
error = "cron: bad minute field";
|
||||
return false;
|
||||
}
|
||||
if (!parse_field(fields[1], expr.hours_, 0, 23, ignored)) {
|
||||
error = "cron: bad hour field";
|
||||
return false;
|
||||
}
|
||||
if (!parse_field(fields[2], expr.days_, 1, 31, expr.dom_restricted_)) {
|
||||
error = "cron: bad day-of-month field";
|
||||
return false;
|
||||
}
|
||||
if (!parse_field(fields[3], expr.months_, 1, 12, ignored)) {
|
||||
error = "cron: bad month field";
|
||||
return false;
|
||||
}
|
||||
if (!parse_dow_field(fields[4], expr.dows_, expr.dow_restricted_)) {
|
||||
error = "cron: bad day-of-week field";
|
||||
return false;
|
||||
}
|
||||
out = expr;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool CronExpr::matches(std::int64_t wall_ms) const {
|
||||
const std::time_t secs = static_cast<std::time_t>(wall_ms / 1000);
|
||||
std::tm tm_utc{};
|
||||
#if defined(_WIN32)
|
||||
gmtime_s(&tm_utc, &secs);
|
||||
#else
|
||||
gmtime_r(&secs, &tm_utc);
|
||||
#endif
|
||||
if (!minutes_[tm_utc.tm_min] || !hours_[tm_utc.tm_hour] || !months_[tm_utc.tm_mon + 1]) {
|
||||
return false;
|
||||
}
|
||||
const bool dom_match = days_[tm_utc.tm_mday];
|
||||
const bool dow_match = dows_[tm_utc.tm_wday];
|
||||
if (dom_restricted_ && dow_restricted_) {
|
||||
return dom_match || dow_match;
|
||||
}
|
||||
if (dom_restricted_) {
|
||||
return dom_match;
|
||||
}
|
||||
if (dow_restricted_) {
|
||||
return dow_match;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
std::int64_t CronExpr::next_after(std::int64_t wall_ms) const {
|
||||
// Step minute by minute, capped at 366 days of search.
|
||||
std::int64_t minute = (wall_ms / 60000) * 60000 + 60000;
|
||||
const std::int64_t limit = minute + 366LL * 86400 * 1000;
|
||||
for (; minute <= limit; minute += 60000) {
|
||||
if (matches(minute)) {
|
||||
return minute;
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,35 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
// Five-field UTC cron expression: minute hour day-of-month month day-of-week.
|
||||
// Each field accepts '*', '*/step', single values, ranges ('a-b'),
|
||||
// stepped ranges ('a-b/step'), and comma lists. Numbers only in v1
|
||||
// (no MON/JAN names, no @macros). Day matching follows Vixie semantics:
|
||||
// when both day-of-month and day-of-week are restricted, either may match.
|
||||
class CronExpr {
|
||||
public:
|
||||
static bool parse(std::string_view text, CronExpr& out, std::string& error);
|
||||
|
||||
// True when the UTC minute containing wall_ms matches the expression.
|
||||
bool matches(std::int64_t wall_ms) const;
|
||||
|
||||
// Start of the first matching minute strictly after wall_ms, or -1 when
|
||||
// no minute matches within 366 days.
|
||||
std::int64_t next_after(std::int64_t wall_ms) const;
|
||||
|
||||
private:
|
||||
bool minutes_[60] = {};
|
||||
bool hours_[24] = {};
|
||||
bool days_[32] = {}; // 1..31
|
||||
bool months_[13] = {}; // 1..12
|
||||
bool dows_[7] = {}; // 0..6, Sunday is 0 (7 folds to 0)
|
||||
bool dom_restricted_ = false;
|
||||
bool dow_restricted_ = false;
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,91 @@
|
||||
#include "supervisor/tasks/dead_letter_queue.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
DeadLetterQueue::DeadLetterQueue(common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer, int max_entries,
|
||||
long retention_days)
|
||||
: clock_(clock),
|
||||
observer_(std::move(observer)),
|
||||
max_entries_(max_entries),
|
||||
retention_days_(retention_days) {
|
||||
if (max_entries_ < 1) {
|
||||
max_entries_ = 1;
|
||||
}
|
||||
}
|
||||
|
||||
void DeadLetterQueue::push(TaskEnvelope envelope, const std::string& reason) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
while (static_cast<int>(entries_.size()) >= max_entries_) {
|
||||
entries_.pop_front();
|
||||
++evicted_total_;
|
||||
observer_->counter("molo_tasks_dead_letter_evicted_total", 1);
|
||||
observer_->event("tasks.dead_letter.evicted", "cap=oldest-first");
|
||||
}
|
||||
envelope.state = TaskState::DeadLettered;
|
||||
entries_.push_back(DeadEntry{std::move(envelope), reason, clock_.wall_ms()});
|
||||
observer_->counter("molo_tasks_dead_lettered_total", 1);
|
||||
observer_->event("tasks.dead_lettered", reason);
|
||||
}
|
||||
|
||||
std::vector<DeadEntry> DeadLetterQueue::list(std::size_t max) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<DeadEntry> out;
|
||||
for (const auto& entry : entries_) {
|
||||
if (out.size() >= max) {
|
||||
break;
|
||||
}
|
||||
out.push_back(entry);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool DeadLetterQueue::take(const std::string& id, DeadEntry& out) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (auto it = entries_.begin(); it != entries_.end(); ++it) {
|
||||
if (it->envelope.id == id) {
|
||||
out = *it;
|
||||
entries_.erase(it);
|
||||
observer_->event("tasks.dead_letter.retried", id);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool DeadLetterQueue::remove(const std::string& id) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (auto it = entries_.begin(); it != entries_.end(); ++it) {
|
||||
if (it->envelope.id == id) {
|
||||
entries_.erase(it);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
std::size_t DeadLetterQueue::size() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return entries_.size();
|
||||
}
|
||||
|
||||
std::size_t DeadLetterQueue::purge_expired() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const std::int64_t horizon = clock_.wall_ms() - retention_days_ * 86400 * 1000;
|
||||
std::size_t purged = 0;
|
||||
while (!entries_.empty() && entries_.front().dead_at_ms < horizon) {
|
||||
entries_.pop_front();
|
||||
++purged;
|
||||
}
|
||||
if (purged > 0) {
|
||||
observer_->counter("molo_tasks_dead_letter_expired_total", static_cast<long>(purged));
|
||||
}
|
||||
return purged;
|
||||
}
|
||||
|
||||
long DeadLetterQueue::evicted_total() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return evicted_total_;
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,52 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <deque>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/tasks/task_envelope.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
struct DeadEntry {
|
||||
TaskEnvelope envelope;
|
||||
std::string reason;
|
||||
std::int64_t dead_at_ms = 0;
|
||||
};
|
||||
|
||||
// Dead-letter queue (S7-2): envelopes that exhausted retries or failed
|
||||
// permanently wait here 7 days for inspection, retry, or cancel via the
|
||||
// CLI (09-cli-ops.md). Max 10000 entries; overflow evicts oldest-first
|
||||
// with an event emitted. Nothing vanishes silently.
|
||||
class DeadLetterQueue {
|
||||
public:
|
||||
DeadLetterQueue(common::Clock& clock, std::shared_ptr<common::Observer> observer,
|
||||
int max_entries = 10000, long retention_days = 7);
|
||||
|
||||
void push(TaskEnvelope envelope, const std::string& reason);
|
||||
std::vector<DeadEntry> list(std::size_t max) const;
|
||||
|
||||
// Removes and returns one entry for manual retry. Returns false when absent.
|
||||
bool take(const std::string& id, DeadEntry& out);
|
||||
bool remove(const std::string& id);
|
||||
|
||||
std::size_t size() const;
|
||||
std::size_t purge_expired(); // drops entries past retention, returns count
|
||||
long evicted_total() const;
|
||||
|
||||
private:
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
int max_entries_;
|
||||
long retention_days_;
|
||||
mutable std::mutex mutex_;
|
||||
std::deque<DeadEntry> entries_;
|
||||
long evicted_total_ = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,56 @@
|
||||
#include "supervisor/tasks/retry_policy.hpp"
|
||||
|
||||
#include <random>
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
RetryPolicy::RetryPolicy(const RetrySpec& spec) : spec_(spec) {
|
||||
if (spec_.base_ms < 0) {
|
||||
spec_.base_ms = 0;
|
||||
}
|
||||
if (spec_.cap_ms < spec_.base_ms) {
|
||||
spec_.cap_ms = spec_.base_ms;
|
||||
}
|
||||
if (spec_.jitter_ms < 0) {
|
||||
spec_.jitter_ms = 0;
|
||||
}
|
||||
if (spec_.max_retries < 0) {
|
||||
spec_.max_retries = 0;
|
||||
}
|
||||
}
|
||||
|
||||
long RetryPolicy::delay_for_attempt(int attempt, std::mt19937& rng) const {
|
||||
if (attempt < 0) {
|
||||
attempt = 0;
|
||||
}
|
||||
// base * 2^n with overflow guard: shift past 30 saturates to the cap.
|
||||
long delay = spec_.base_ms;
|
||||
if (attempt >= 31) {
|
||||
delay = spec_.cap_ms;
|
||||
} else {
|
||||
const long long grown = static_cast<long long>(spec_.base_ms) * (1LL << attempt);
|
||||
delay = grown >= spec_.cap_ms ? spec_.cap_ms : static_cast<long>(grown);
|
||||
}
|
||||
long jitter = 0;
|
||||
if (spec_.jitter_ms > 0) {
|
||||
std::uniform_int_distribution<long> dist(0, spec_.jitter_ms);
|
||||
jitter = dist(rng);
|
||||
}
|
||||
const long long total = static_cast<long long>(delay) + jitter;
|
||||
return total >= spec_.cap_ms ? spec_.cap_ms : static_cast<long>(total);
|
||||
}
|
||||
|
||||
long RetryPolicy::delay_for_attempt(int attempt) const {
|
||||
static thread_local std::mt19937 rng{std::random_device{}()};
|
||||
return delay_for_attempt(attempt, rng);
|
||||
}
|
||||
|
||||
bool RetryPolicy::exhausted(int attempts) const {
|
||||
return attempts > spec_.max_retries;
|
||||
}
|
||||
|
||||
const RetrySpec& RetryPolicy::spec() const {
|
||||
return spec_;
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
#include <random>
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
// Bounded exponential backoff with jitter (S7-2):
|
||||
// delay(n) = min(backoff_base_ms * 2^n + jitter, backoff_cap_ms)
|
||||
// where n is the zero-based retry number and jitter is uniform
|
||||
// 0..jitter_ms. Pure computation, no I/O, trivially testable.
|
||||
struct RetrySpec {
|
||||
long base_ms = 1000;
|
||||
long cap_ms = 300000;
|
||||
long jitter_ms = 1000;
|
||||
int max_retries = 5;
|
||||
};
|
||||
|
||||
class RetryPolicy {
|
||||
public:
|
||||
explicit RetryPolicy(const RetrySpec& spec = RetrySpec{});
|
||||
|
||||
long delay_for_attempt(int attempt, std::mt19937& rng) const;
|
||||
long delay_for_attempt(int attempt) const; // thread-local RNG
|
||||
bool exhausted(int attempts) const; // attempts > max_retries
|
||||
const RetrySpec& spec() const;
|
||||
|
||||
private:
|
||||
RetrySpec spec_;
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,380 @@
|
||||
#include "supervisor/tasks/scheduler.hpp"
|
||||
|
||||
#include <filesystem>
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
#include "supervisor/common/json.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
namespace {
|
||||
|
||||
// Option clamps shared by the scheduler and its LeaderElection (which has
|
||||
// no default ctor, so normalization must run in the init list).
|
||||
SchedulerOptions NormalizedOptions(SchedulerOptions o) {
|
||||
if (o.instance_id.empty()) {
|
||||
o.instance_id = "sched-" + TaskEnvelope::generate_id();
|
||||
}
|
||||
if (o.tick_ms < 10) {
|
||||
o.tick_ms = 10;
|
||||
}
|
||||
if (o.ttl_secs < 1) {
|
||||
o.ttl_secs = 1;
|
||||
}
|
||||
if (o.renew_secs < 1) {
|
||||
o.renew_secs = 1;
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
Scheduler::Scheduler(TaskQueue& queue, common::KvStore& kv, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer,
|
||||
const SchedulerOptions& options)
|
||||
: queue_(queue),
|
||||
kv_(kv),
|
||||
clock_(clock),
|
||||
observer_(std::move(observer)),
|
||||
options_(NormalizedOptions(options)),
|
||||
election_(kv_, clock_, observer_ ? *observer_ : fallback_observer_,
|
||||
options_.leader_key, options_.instance_id, options_.ttl_secs * 1000,
|
||||
options_.renew_secs * 1000) {
|
||||
if (!options_.durable_dir.empty()) {
|
||||
log_ = std::make_unique<bus::SegmentLog>(
|
||||
std::filesystem::path(options_.durable_dir) / "scheduler.log");
|
||||
replay();
|
||||
}
|
||||
}
|
||||
|
||||
void Scheduler::replay() {
|
||||
// Rebuilds schedule entries from the op log (E7-6). Last-fire records
|
||||
// live in shared KV: with the shared backend they survive too, and a
|
||||
// reboot never refires a window the old leader already fired.
|
||||
std::uint64_t seq = 1;
|
||||
while (true) {
|
||||
auto records = log_->read_from(seq, 256, 1024 * 1024);
|
||||
if (records.empty()) {
|
||||
break;
|
||||
}
|
||||
for (const auto& record : records) {
|
||||
common::JValue root;
|
||||
if (!common::json_decode(record.data, root)) {
|
||||
continue;
|
||||
}
|
||||
const common::JValue* op = root.find("op");
|
||||
if (op == nullptr) {
|
||||
continue;
|
||||
}
|
||||
if (op->as_string() == "put") {
|
||||
const common::JValue* body = root.find("entry");
|
||||
if (body == nullptr) {
|
||||
continue;
|
||||
}
|
||||
ScheduleEntry entry;
|
||||
if (decode_entry(common::json_encode(*body), entry)) {
|
||||
CronExpr cron;
|
||||
if (entry.kind == ScheduleKind::Cron) {
|
||||
std::string error;
|
||||
if (!CronExpr::parse(entry.cron, cron, error)) {
|
||||
continue;
|
||||
}
|
||||
cron_cache_[entry.key] = cron;
|
||||
}
|
||||
entries_[entry.key] = entry;
|
||||
registered_at_[entry.key] = clock_.wall_ms();
|
||||
}
|
||||
} else if (op->as_string() == "remove") {
|
||||
const common::JValue* key = root.find("key");
|
||||
if (key != nullptr) {
|
||||
entries_.erase(key->as_string());
|
||||
cron_cache_.erase(key->as_string());
|
||||
registered_at_.erase(key->as_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
seq = records.back().seq + 1;
|
||||
}
|
||||
observer_->event("tasks.schedule.replayed", "entries=" + std::to_string(entries_.size()));
|
||||
}
|
||||
|
||||
std::string Scheduler::encode_entry(const ScheduleEntry& entry) const {
|
||||
using common::JValue;
|
||||
JValue::Object obj;
|
||||
obj["key"] = JValue(entry.key);
|
||||
obj["task_name"] = JValue(entry.task_name);
|
||||
JValue payload;
|
||||
if (!common::json_decode(entry.payload, payload)) {
|
||||
payload = JValue(entry.payload);
|
||||
}
|
||||
obj["payload"] = payload;
|
||||
obj["kind"] = JValue(entry.kind == ScheduleKind::Once
|
||||
? "once"
|
||||
: (entry.kind == ScheduleKind::Interval ? "interval" : "cron"));
|
||||
obj["at_ms"] = JValue(static_cast<long long>(entry.at_ms));
|
||||
obj["interval_secs"] = JValue(static_cast<long long>(entry.interval_secs));
|
||||
obj["cron"] = JValue(entry.cron);
|
||||
obj["missed"] = JValue(entry.missed == MissedPolicy::Skip ? "skip" : "run-once");
|
||||
obj["enabled"] = JValue(entry.enabled);
|
||||
obj["max_retries"] = JValue(entry.max_retries);
|
||||
return common::json_encode(JValue(obj));
|
||||
}
|
||||
|
||||
bool Scheduler::decode_entry(const std::string& text, ScheduleEntry& out) const {
|
||||
using common::JValue;
|
||||
JValue root;
|
||||
if (!common::json_decode(text, root) || !root.is_object()) {
|
||||
return false;
|
||||
}
|
||||
const JValue* key = root.find("key");
|
||||
const JValue* task_name = root.find("task_name");
|
||||
const JValue* kind = root.find("kind");
|
||||
if (key == nullptr || task_name == nullptr || kind == nullptr || !key->is_string() ||
|
||||
!task_name->is_string() || !kind->is_string()) {
|
||||
return false;
|
||||
}
|
||||
ScheduleEntry entry;
|
||||
entry.key = key->as_string();
|
||||
entry.task_name = task_name->as_string();
|
||||
const std::string kind_text = kind->as_string();
|
||||
if (kind_text == "once") {
|
||||
entry.kind = ScheduleKind::Once;
|
||||
} else if (kind_text == "interval") {
|
||||
entry.kind = ScheduleKind::Interval;
|
||||
} else if (kind_text == "cron") {
|
||||
entry.kind = ScheduleKind::Cron;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
const JValue* payload = root.find("payload");
|
||||
if (payload != nullptr) {
|
||||
entry.payload = common::json_encode(*payload);
|
||||
}
|
||||
const JValue* at_ms = root.find("at_ms");
|
||||
const JValue* interval = root.find("interval_secs");
|
||||
if (at_ms != nullptr) {
|
||||
entry.at_ms = at_ms->as_int(0);
|
||||
}
|
||||
if (interval != nullptr) {
|
||||
entry.interval_secs = static_cast<long>(interval->as_int(60));
|
||||
}
|
||||
const JValue* cron = root.find("cron");
|
||||
if (cron != nullptr) {
|
||||
entry.cron = cron->as_string();
|
||||
}
|
||||
const JValue* missed = root.find("missed");
|
||||
if (missed != nullptr && missed->as_string() == "run-once") {
|
||||
entry.missed = MissedPolicy::RunOnce;
|
||||
}
|
||||
const JValue* enabled = root.find("enabled");
|
||||
if (enabled != nullptr) {
|
||||
entry.enabled = enabled->as_bool(true);
|
||||
}
|
||||
const JValue* retries = root.find("max_retries");
|
||||
if (retries != nullptr) {
|
||||
entry.max_retries = static_cast<int>(retries->as_int(5));
|
||||
}
|
||||
out = std::move(entry);
|
||||
return true;
|
||||
}
|
||||
|
||||
void Scheduler::put(ScheduleEntry entry) {
|
||||
if (entry.key.empty() || entry.key.size() > 256) {
|
||||
throw TaskError("Scheduler: bad entry key");
|
||||
}
|
||||
if (entry.task_name.empty()) {
|
||||
throw TaskError("Scheduler: entry needs a task name");
|
||||
}
|
||||
CronExpr cron;
|
||||
if (entry.kind == ScheduleKind::Cron) {
|
||||
std::string error;
|
||||
if (!CronExpr::parse(entry.cron, cron, error)) {
|
||||
throw TaskError("Scheduler: " + error);
|
||||
}
|
||||
}
|
||||
if (entry.kind == ScheduleKind::Interval &&
|
||||
entry.interval_secs < common::Caps::kSchedulerMinIntervalSecs) {
|
||||
entry.interval_secs = common::Caps::kSchedulerMinIntervalSecs; // S7-5 floor
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const bool is_new = entries_.find(entry.key) == entries_.end();
|
||||
entries_[entry.key] = entry;
|
||||
if (entry.kind == ScheduleKind::Cron) {
|
||||
cron_cache_[entry.key] = cron;
|
||||
} else {
|
||||
cron_cache_.erase(entry.key);
|
||||
}
|
||||
if (is_new) {
|
||||
registered_at_[entry.key] = clock_.wall_ms();
|
||||
}
|
||||
if (log_) {
|
||||
log_->append("{\"op\":\"put\",\"entry\":" + encode_entry(entry) + "}", clock_.wall_ms());
|
||||
}
|
||||
observer_->event("tasks.schedule.put", entry.key);
|
||||
}
|
||||
|
||||
bool Scheduler::remove(const std::string& key) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
cron_cache_.erase(key);
|
||||
registered_at_.erase(key);
|
||||
const bool removed = entries_.erase(key) > 0;
|
||||
if (removed && log_) {
|
||||
log_->append("{\"op\":\"remove\",\"key\":\"" + common::json_escape(key) + "\"}",
|
||||
clock_.wall_ms());
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
|
||||
std::vector<ScheduleEntry> Scheduler::entries() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<ScheduleEntry> out;
|
||||
for (const auto& [_, entry] : entries_) {
|
||||
out.push_back(entry);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool Scheduler::is_leader() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return election_.is_leader();
|
||||
}
|
||||
|
||||
std::string Scheduler::leader() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return leader_id_;
|
||||
}
|
||||
|
||||
void Scheduler::tick_once() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
const std::int64_t now = clock_.wall_ms();
|
||||
if (!elect_locked(now)) {
|
||||
return; // follower: stay passive
|
||||
}
|
||||
fire_due_locked(now);
|
||||
}
|
||||
|
||||
int Scheduler::run(const std::function<bool()>& stop_requested) {
|
||||
while (!stop_requested || !stop_requested()) {
|
||||
tick_once();
|
||||
clock_.sleep_ms(options_.tick_ms);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
bool Scheduler::elect_locked(std::int64_t now_wall) {
|
||||
(void)now_wall;
|
||||
const bool was = election_.is_leader();
|
||||
const bool now_leader = election_.tick();
|
||||
if (now_leader && !was) {
|
||||
leader_id_ = options_.instance_id;
|
||||
observer_->event("tasks.schedule.leader_won", options_.instance_id);
|
||||
} else if (!now_leader && was) {
|
||||
observer_->event("tasks.schedule.leader_lost", options_.instance_id);
|
||||
}
|
||||
if (!now_leader) {
|
||||
const std::string current = election_.leader();
|
||||
if (!current.empty()) {
|
||||
leader_id_ = current;
|
||||
}
|
||||
}
|
||||
return now_leader;
|
||||
}
|
||||
|
||||
std::int64_t Scheduler::last_fire_locked(const std::string& key, std::int64_t fallback) {
|
||||
std::string stored;
|
||||
if (kv_.get(options_.leader_key + "/fired/" + key, stored)) {
|
||||
try {
|
||||
return std::stoll(stored);
|
||||
} catch (...) {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
|
||||
void Scheduler::record_fire_locked(const std::string& key, std::int64_t now_wall, long ttl_secs) {
|
||||
kv_.set(options_.leader_key + "/fired/" + key, std::to_string(now_wall), ttl_secs);
|
||||
}
|
||||
|
||||
void Scheduler::fire_locked(const ScheduleEntry& entry, const std::string& fire_ref,
|
||||
std::int64_t now_wall) {
|
||||
TaskEnvelope env;
|
||||
env.name = entry.task_name;
|
||||
env.payload = entry.payload;
|
||||
env.max_retries = entry.max_retries;
|
||||
env.idempotency_key = "sched:" + entry.key + ":" + fire_ref;
|
||||
try {
|
||||
const std::string id = queue_.enqueue(std::move(env));
|
||||
observer_->event("tasks.schedule.fired", entry.key + " -> " + id);
|
||||
(void)now_wall;
|
||||
} catch (const TaskError& e) {
|
||||
observer_->event("tasks.schedule.fire_failed", entry.key + ": " + e.what());
|
||||
}
|
||||
}
|
||||
|
||||
void Scheduler::fire_due_locked(std::int64_t now_wall) {
|
||||
for (const auto& [key, entry] : entries_) {
|
||||
if (!entry.enabled) {
|
||||
continue;
|
||||
}
|
||||
const std::int64_t registered =
|
||||
registered_at_.count(key) ? registered_at_[key] : now_wall;
|
||||
switch (entry.kind) {
|
||||
case ScheduleKind::Once: {
|
||||
// Fired markers persist, so a reboot never refires a Once entry.
|
||||
const std::int64_t last = last_fire_locked(key, -1);
|
||||
if (last >= 0) {
|
||||
break;
|
||||
}
|
||||
if (now_wall < entry.at_ms) {
|
||||
break;
|
||||
}
|
||||
const bool missed = (now_wall - entry.at_ms > 60000);
|
||||
if (missed && entry.missed == MissedPolicy::Skip) {
|
||||
record_fire_locked(key, now_wall, 86400);
|
||||
observer_->event("tasks.schedule.missed", key);
|
||||
break;
|
||||
}
|
||||
fire_locked(entry, "once", now_wall);
|
||||
record_fire_locked(key, now_wall, 86400);
|
||||
break;
|
||||
}
|
||||
case ScheduleKind::Interval: {
|
||||
const std::int64_t span = entry.interval_secs * 1000;
|
||||
const std::int64_t last = last_fire_locked(key, registered);
|
||||
if (now_wall - last < span) {
|
||||
break;
|
||||
}
|
||||
// At most one fire per tick; last=now so followers taking over
|
||||
// never refire this window (V7-4).
|
||||
fire_locked(entry, std::to_string(now_wall / span), now_wall);
|
||||
record_fire_locked(key, now_wall, entry.interval_secs * 2 + 60);
|
||||
break;
|
||||
}
|
||||
case ScheduleKind::Cron: {
|
||||
auto cit = cron_cache_.find(key);
|
||||
if (cit == cron_cache_.end()) {
|
||||
break;
|
||||
}
|
||||
const std::int64_t minute = (now_wall / 60000) * 60000;
|
||||
const std::int64_t last = last_fire_locked(key, registered);
|
||||
if (last >= minute && cit->second.matches(now_wall)) {
|
||||
break; // already fired this minute
|
||||
}
|
||||
if (cit->second.matches(now_wall)) {
|
||||
fire_locked(entry, "m" + std::to_string(minute / 60000), now_wall);
|
||||
record_fire_locked(key, now_wall, 90000);
|
||||
} else if (entry.missed == MissedPolicy::RunOnce) {
|
||||
const std::int64_t next = cit->second.next_after(last);
|
||||
if (next >= 0 && next <= now_wall) {
|
||||
fire_locked(entry, "m" + std::to_string((next / 60000)), now_wall);
|
||||
record_fire_locked(key, now_wall, 90000);
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,114 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "operability/coord/election.h"
|
||||
#include "supervisor/bus/segment_log.hpp"
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/kv_store.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/tasks/cron.hpp"
|
||||
#include "supervisor/tasks/task_envelope.hpp"
|
||||
#include "supervisor/tasks/task_queue.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
// What to do with a fire that was missed during an outage or failover.
|
||||
enum class MissedPolicy {
|
||||
Skip, // drop the missed fire, wait for the next one
|
||||
RunOnce, // fire once now to cover the missed window
|
||||
};
|
||||
|
||||
enum class ScheduleKind {
|
||||
Once, // one delayed run at at_ms
|
||||
Interval, // fixed interval, minimum 60s (S7-5)
|
||||
Cron, // 5-field UTC cron expression
|
||||
};
|
||||
|
||||
struct ScheduleEntry {
|
||||
std::string key;
|
||||
std::string task_name;
|
||||
std::string payload = "{}";
|
||||
ScheduleKind kind = ScheduleKind::Interval;
|
||||
std::int64_t at_ms = 0; // Once
|
||||
long interval_secs = 60; // Interval
|
||||
std::string cron; // Cron
|
||||
MissedPolicy missed = MissedPolicy::Skip;
|
||||
bool enabled = true;
|
||||
int max_retries = 5;
|
||||
};
|
||||
|
||||
struct SchedulerOptions {
|
||||
std::string instance_id;
|
||||
std::string leader_key = "molo/scheduler/leader";
|
||||
std::string durable_dir; // empty means in-memory entries; set for E7-6
|
||||
long tick_ms = 1000;
|
||||
long ttl_secs = 10;
|
||||
long renew_secs = 3;
|
||||
};
|
||||
|
||||
// Delayed plus periodic scheduling (S7-5). The leader enqueues due entries
|
||||
// into the TaskQueue; followers stay passive. Leadership is a lease in the
|
||||
// shared KvStore (10s TTL, 3s renewal). Last-fire records live in the same
|
||||
// shared store so a new leader never double-fires, and every fire carries
|
||||
// an idempotency key so even a brief dual-leader overlap cannot enqueue
|
||||
// twice (the queue dedupes). Dual-run during network partitions is accepted
|
||||
// with documented bounds (O7-4): at most the overlap window, deduped by key.
|
||||
class Scheduler {
|
||||
public:
|
||||
Scheduler(TaskQueue& queue, common::KvStore& kv, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer, const SchedulerOptions& options);
|
||||
|
||||
// Adds or replaces an entry. Throws TaskError on bad cron or interval.
|
||||
void put(ScheduleEntry entry);
|
||||
bool remove(const std::string& key);
|
||||
std::vector<ScheduleEntry> entries() const;
|
||||
|
||||
bool is_leader() const;
|
||||
std::string leader() const;
|
||||
|
||||
// One tick: run leader election, then fire due entries when leader.
|
||||
// Deterministic under FrozenClock; production uses run().
|
||||
void tick_once();
|
||||
|
||||
// Ticks until stop_requested() is true. Returns 0.
|
||||
int run(const std::function<bool()>& stop_requested);
|
||||
|
||||
std::string encode_entry(const ScheduleEntry& entry) const;
|
||||
bool decode_entry(const std::string& text, ScheduleEntry& out) const;
|
||||
|
||||
private:
|
||||
void replay();
|
||||
bool elect_locked(std::int64_t now_wall);
|
||||
void fire_due_locked(std::int64_t now_wall);
|
||||
std::int64_t last_fire_locked(const std::string& key, std::int64_t fallback);
|
||||
void record_fire_locked(const std::string& key, std::int64_t now_wall, long ttl_secs);
|
||||
void fire_locked(const ScheduleEntry& entry, const std::string& fire_ref,
|
||||
std::int64_t now_wall);
|
||||
|
||||
TaskQueue& queue_;
|
||||
common::KvStore& kv_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
SchedulerOptions options_;
|
||||
std::unique_ptr<bus::SegmentLog> log_;
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::string, ScheduleEntry> entries_;
|
||||
std::map<std::string, CronExpr> cron_cache_;
|
||||
std::map<std::string, std::int64_t> registered_at_;
|
||||
std::string leader_id_;
|
||||
// Leadership moved onto molo::coord::LeaderElection (operability part);
|
||||
// the scheduler keeps its public surface and event names. Declared after
|
||||
// observer_/options_ so the init list can wire them in.
|
||||
common::NullObserver fallback_observer_;
|
||||
molo::coord::LeaderElection election_;
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,150 @@
|
||||
#include "supervisor/tasks/task_envelope.hpp"
|
||||
|
||||
#include <chrono>
|
||||
#include <iomanip>
|
||||
#include <random>
|
||||
#include <sstream>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/json.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
std::string to_string(TaskState state) {
|
||||
switch (state) {
|
||||
case TaskState::Queued: return "queued";
|
||||
case TaskState::Leased: return "leased";
|
||||
case TaskState::Running: return "running";
|
||||
case TaskState::Succeeded: return "succeeded";
|
||||
case TaskState::FailedRetryable: return "failed-retryable";
|
||||
case TaskState::FailedPermanent: return "failed-permanent";
|
||||
case TaskState::DeadLettered: return "dead-lettered";
|
||||
case TaskState::Cancelled: return "cancelled";
|
||||
}
|
||||
return "queued";
|
||||
}
|
||||
|
||||
bool parse_state(const std::string& text, TaskState& out) {
|
||||
if (text == "queued") {
|
||||
out = TaskState::Queued;
|
||||
} else if (text == "leased") {
|
||||
out = TaskState::Leased;
|
||||
} else if (text == "running") {
|
||||
out = TaskState::Running;
|
||||
} else if (text == "succeeded") {
|
||||
out = TaskState::Succeeded;
|
||||
} else if (text == "failed-retryable") {
|
||||
out = TaskState::FailedRetryable;
|
||||
} else if (text == "failed-permanent") {
|
||||
out = TaskState::FailedPermanent;
|
||||
} else if (text == "dead-lettered") {
|
||||
out = TaskState::DeadLettered;
|
||||
} else if (text == "cancelled") {
|
||||
out = TaskState::Cancelled;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string TaskEnvelope::encode() const {
|
||||
using common::JValue;
|
||||
JValue::Object obj;
|
||||
obj["id"] = JValue(id);
|
||||
obj["name"] = JValue(name);
|
||||
JValue payload_value;
|
||||
if (!common::json_decode(payload, payload_value)) {
|
||||
payload_value = JValue(payload); // non-JSON payloads ride as strings
|
||||
}
|
||||
obj["payload"] = payload_value;
|
||||
obj["attempts"] = JValue(attempts);
|
||||
obj["max_retries"] = JValue(max_retries);
|
||||
if (run_at_ms.has_value()) {
|
||||
obj["run_at"] = JValue(common::iso8601(*run_at_ms));
|
||||
} else {
|
||||
obj["run_at"] = JValue(nullptr);
|
||||
}
|
||||
obj["timeout_soft_ms"] = JValue(static_cast<long long>(timeout_soft_ms));
|
||||
obj["timeout_hard_ms"] = JValue(static_cast<long long>(timeout_hard_ms));
|
||||
obj["idempotency_key"] = JValue(idempotency_key);
|
||||
obj["enqueued_at"] = JValue(common::iso8601(enqueued_at_ms));
|
||||
obj["state"] = JValue(to_string(state));
|
||||
return common::json_encode(JValue(obj));
|
||||
}
|
||||
|
||||
bool TaskEnvelope::decode(const std::string& text, TaskEnvelope& out) {
|
||||
using common::JValue;
|
||||
JValue root;
|
||||
if (!common::json_decode(text, root) || !root.is_object()) {
|
||||
return false;
|
||||
}
|
||||
const JValue* id = root.find("id");
|
||||
const JValue* name = root.find("name");
|
||||
if (id == nullptr || name == nullptr || !id->is_string() || !name->is_string()) {
|
||||
return false;
|
||||
}
|
||||
TaskEnvelope env;
|
||||
env.id = id->as_string();
|
||||
env.name = name->as_string();
|
||||
const JValue* payload = root.find("payload");
|
||||
if (payload != nullptr) {
|
||||
env.payload = common::json_encode(*payload);
|
||||
}
|
||||
const JValue* attempts = root.find("attempts");
|
||||
const JValue* max_retries = root.find("max_retries");
|
||||
if (attempts != nullptr) {
|
||||
env.attempts = static_cast<int>(attempts->as_int(0));
|
||||
}
|
||||
if (max_retries != nullptr) {
|
||||
env.max_retries = static_cast<int>(max_retries->as_int(5));
|
||||
}
|
||||
const JValue* run_at = root.find("run_at");
|
||||
if (run_at != nullptr && run_at->is_string()) {
|
||||
const std::int64_t parsed = common::parse_iso8601(run_at->as_string());
|
||||
if (parsed < 0) {
|
||||
return false;
|
||||
}
|
||||
env.run_at_ms = parsed;
|
||||
}
|
||||
const JValue* soft = root.find("timeout_soft_ms");
|
||||
const JValue* hard = root.find("timeout_hard_ms");
|
||||
if (soft != nullptr) {
|
||||
env.timeout_soft_ms = static_cast<long>(soft->as_int(30000));
|
||||
}
|
||||
if (hard != nullptr) {
|
||||
env.timeout_hard_ms = static_cast<long>(hard->as_int(60000));
|
||||
}
|
||||
const JValue* key = root.find("idempotency_key");
|
||||
if (key != nullptr) {
|
||||
env.idempotency_key = key->as_string();
|
||||
}
|
||||
const JValue* enqueued = root.find("enqueued_at");
|
||||
if (enqueued != nullptr && enqueued->is_string()) {
|
||||
env.enqueued_at_ms = common::parse_iso8601(enqueued->as_string());
|
||||
}
|
||||
const JValue* state = root.find("state");
|
||||
if (state != nullptr && state->is_string()) {
|
||||
if (!parse_state(state->as_string(), env.state)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
out = std::move(env);
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string TaskEnvelope::generate_id() {
|
||||
// UUIDv7 shape: 48-bit unix ms plus 74 random bits, hex with dashes.
|
||||
static thread_local std::mt19937_64 rng{std::random_device{}()};
|
||||
const auto now = std::chrono::system_clock::now();
|
||||
const auto ms = static_cast<std::uint64_t>(
|
||||
std::chrono::duration_cast<std::chrono::milliseconds>(now.time_since_epoch()).count());
|
||||
const std::uint64_t hi = (ms << 16) | (rng() & 0xFFFFu);
|
||||
const std::uint64_t lo = rng();
|
||||
std::ostringstream out;
|
||||
out << std::hex << std::setfill('0') << std::setw(12) << (hi >> 16) << "-" << std::setw(4)
|
||||
<< (hi & 0xFFFFu) << "-7" << std::setw(3) << ((lo >> 48) & 0xFFFu) << "-" << std::setw(4)
|
||||
<< (((lo >> 32) & 0x3FFFu) | 0x8000u) << "-" << std::setw(12) << (lo & 0xFFFFFFFFFFFFULL);
|
||||
return out.str();
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,57 @@
|
||||
#pragma once
|
||||
|
||||
#include <cstdint>
|
||||
#include <optional>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
class TaskError : public std::runtime_error {
|
||||
public:
|
||||
explicit TaskError(const std::string& message) : std::runtime_error(message) {}
|
||||
};
|
||||
|
||||
// Task lifecycle states (S7-3).
|
||||
enum class TaskState {
|
||||
Queued,
|
||||
Leased,
|
||||
Running,
|
||||
Succeeded,
|
||||
FailedRetryable,
|
||||
FailedPermanent,
|
||||
DeadLettered,
|
||||
Cancelled,
|
||||
};
|
||||
|
||||
std::string to_string(TaskState state);
|
||||
bool parse_state(const std::string& text, TaskState& out);
|
||||
|
||||
// Immutable job record (S7-1). The only type that crosses queue boundaries.
|
||||
// JSON form doubles as the on-disk format for the durable queue:
|
||||
//
|
||||
// {"id":"...","name":"billing.charge","payload":{...},"attempts":0,
|
||||
// "max_retries":5,"run_at":"iso8601|null","timeout_soft_ms":30000,
|
||||
// "timeout_hard_ms":60000,"idempotency_key":"...","enqueued_at":"iso8601"}
|
||||
//
|
||||
// The payload is an opaque JSON object string (max 1 MiB); the queue never
|
||||
// interprets it, handlers do.
|
||||
struct TaskEnvelope {
|
||||
std::string id;
|
||||
std::string name;
|
||||
std::string payload = "{}";
|
||||
int attempts = 0;
|
||||
int max_retries = 5;
|
||||
std::optional<std::int64_t> run_at_ms; // null means run as soon as possible
|
||||
long timeout_soft_ms = 30000;
|
||||
long timeout_hard_ms = 60000;
|
||||
std::string idempotency_key;
|
||||
std::int64_t enqueued_at_ms = 0;
|
||||
TaskState state = TaskState::Queued;
|
||||
|
||||
std::string encode() const;
|
||||
static bool decode(const std::string& text, TaskEnvelope& out);
|
||||
static std::string generate_id();
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,14 @@
|
||||
#include "supervisor/tasks/task_handler.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
std::string to_string(TaskOutcome outcome) {
|
||||
switch (outcome) {
|
||||
case TaskOutcome::Success: return "success";
|
||||
case TaskOutcome::RetryableFailure: return "retryable";
|
||||
case TaskOutcome::PermanentFailure: return "permanent";
|
||||
}
|
||||
return "retryable";
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,40 @@
|
||||
#pragma once
|
||||
|
||||
#include <atomic>
|
||||
#include <string>
|
||||
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/tasks/task_envelope.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
// Handler outcome. RetryableFailure requeues with backoff; PermanentFailure
|
||||
// dead-letters immediately. Throwing from run() counts as retryable.
|
||||
enum class TaskOutcome {
|
||||
Success,
|
||||
RetryableFailure,
|
||||
PermanentFailure,
|
||||
};
|
||||
|
||||
std::string to_string(TaskOutcome outcome);
|
||||
|
||||
// Per-execution context handed to handlers.
|
||||
struct TaskContext {
|
||||
const TaskEnvelope& envelope;
|
||||
common::Clock& clock;
|
||||
std::shared_ptr<common::Observer> observer;
|
||||
std::atomic<bool>& cancel_requested; // set at the soft timeout (R7, S7-3)
|
||||
};
|
||||
|
||||
// Base class for all task handlers. Every handler MUST be idempotent:
|
||||
// delivery is at-least-once, so run() may execute more than once with the
|
||||
// same task id after worker loss or lease expiry (R7-2). Handlers run on
|
||||
// worker threads, never on HTTP event loop threads (R7-5).
|
||||
class TaskHandler {
|
||||
public:
|
||||
virtual ~TaskHandler() = default;
|
||||
virtual TaskOutcome run(TaskContext& ctx) = 0;
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,683 @@
|
||||
#include "supervisor/tasks/task_queue.hpp"
|
||||
|
||||
#include <thread>
|
||||
|
||||
#include "supervisor/common/caps.hpp"
|
||||
#include "supervisor/common/json.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
namespace {
|
||||
|
||||
std::string truncate_reason(const std::string& reason) {
|
||||
return reason.size() > 1024 ? reason.substr(0, 1024) : reason;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
TaskQueue::TaskQueue(const QueueOptions& options, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer)
|
||||
: options_(options),
|
||||
clock_(clock),
|
||||
observer_(std::move(observer)),
|
||||
retry_(options.retry),
|
||||
dead_letters_(clock_, observer_) {
|
||||
if (options_.max_depth < 1) {
|
||||
options_.max_depth = 1;
|
||||
}
|
||||
if (!options_.durable_dir.empty()) {
|
||||
log_ = std::make_unique<bus::SegmentLog>(
|
||||
std::filesystem::path(options_.durable_dir) / "task_queue.log");
|
||||
replay();
|
||||
}
|
||||
}
|
||||
|
||||
TaskQueue::~TaskQueue() {
|
||||
shutdown();
|
||||
}
|
||||
|
||||
void TaskQueue::replay() {
|
||||
// Rebuilds live state from the op log. Leases are never persisted, so a
|
||||
// task leased at crash replays as queued and is redelivered (E7-6).
|
||||
std::uint64_t seq = 1;
|
||||
while (true) {
|
||||
auto records = log_->read_from(seq, 512, 4 * 1024 * 1024);
|
||||
if (records.empty()) {
|
||||
break;
|
||||
}
|
||||
for (const auto& record : records) {
|
||||
common::JValue root;
|
||||
if (!common::json_decode(record.data, root)) {
|
||||
continue; // corrupt op skipped loudly below
|
||||
}
|
||||
const common::JValue* op = root.find("op");
|
||||
if (op == nullptr) {
|
||||
continue;
|
||||
}
|
||||
const std::string op_name = op->as_string();
|
||||
if (op_name == "enqueue") {
|
||||
const common::JValue* env = root.find("env");
|
||||
if (env == nullptr) {
|
||||
continue;
|
||||
}
|
||||
TaskEnvelope envelope;
|
||||
if (!TaskEnvelope::decode(common::json_encode(*env), envelope)) {
|
||||
continue;
|
||||
}
|
||||
envelope.state = TaskState::Queued;
|
||||
const std::int64_t run_at = envelope.run_at_ms.value_or(0);
|
||||
records_[envelope.id] = Record{envelope, ++order_};
|
||||
ready_[{run_at, order_}] = envelope.id;
|
||||
} else if (op_name == "terminal") {
|
||||
const common::JValue* id = root.find("id");
|
||||
if (id != nullptr) {
|
||||
terminal_locked(id->as_string());
|
||||
}
|
||||
} else if (op_name == "requeue") {
|
||||
const common::JValue* id = root.find("id");
|
||||
if (id == nullptr) {
|
||||
continue;
|
||||
}
|
||||
auto it = records_.find(id->as_string());
|
||||
if (it == records_.end()) {
|
||||
continue;
|
||||
}
|
||||
const common::JValue* attempts = root.find("attempts");
|
||||
const common::JValue* run_at = root.find("run_at_ms");
|
||||
if (attempts != nullptr) {
|
||||
it->second.envelope.attempts = static_cast<int>(attempts->as_int(0));
|
||||
}
|
||||
if (run_at != nullptr && !run_at->is_null()) {
|
||||
it->second.envelope.run_at_ms = run_at->as_int(0);
|
||||
} else {
|
||||
it->second.envelope.run_at_ms.reset();
|
||||
}
|
||||
it->second.envelope.state = TaskState::Queued;
|
||||
it->second.order = ++order_;
|
||||
const std::int64_t due = it->second.envelope.run_at_ms.value_or(0);
|
||||
ready_[{due, it->second.order}] = it->first;
|
||||
}
|
||||
}
|
||||
seq = records.back().seq + 1;
|
||||
}
|
||||
observer_->event("tasks.queue.replayed", "live=" + std::to_string(records_.size()));
|
||||
}
|
||||
|
||||
void TaskQueue::log_op(const std::string& op) {
|
||||
if (log_) {
|
||||
log_->append(op, clock_.wall_ms());
|
||||
}
|
||||
}
|
||||
|
||||
std::string TaskQueue::enqueue(TaskEnvelope envelope) {
|
||||
if (envelope.name.empty() || envelope.name.size() > 256) {
|
||||
throw TaskError("TaskQueue: bad task name");
|
||||
}
|
||||
if (envelope.payload.size() > common::Caps::kTaskPayloadBytes) {
|
||||
throw TaskError("TaskQueue: payload over 1 MiB cap");
|
||||
}
|
||||
if (envelope.timeout_soft_ms <= 0) {
|
||||
envelope.timeout_soft_ms = 30000;
|
||||
}
|
||||
if (envelope.timeout_hard_ms <= 0) {
|
||||
envelope.timeout_hard_ms = 60000;
|
||||
}
|
||||
if (envelope.timeout_hard_ms < envelope.timeout_soft_ms) {
|
||||
envelope.timeout_hard_ms = envelope.timeout_soft_ms;
|
||||
}
|
||||
if (envelope.max_retries < 0) {
|
||||
envelope.max_retries = 0;
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!accepting_) {
|
||||
throw TaskError("TaskQueue: not accepting new tasks (draining)");
|
||||
}
|
||||
if (!envelope.idempotency_key.empty()) {
|
||||
const std::string existing = dedupe_lookup_locked(envelope.idempotency_key);
|
||||
if (!existing.empty()) {
|
||||
return existing;
|
||||
}
|
||||
}
|
||||
if (envelope.id.empty()) {
|
||||
envelope.id = TaskEnvelope::generate_id();
|
||||
}
|
||||
if (records_.size() >= options_.max_depth) {
|
||||
throw TaskError("TaskQueue: queue depth cap reached");
|
||||
}
|
||||
envelope.enqueued_at_ms = clock_.wall_ms();
|
||||
envelope.attempts = 0;
|
||||
envelope.state = TaskState::Queued;
|
||||
const std::string id = envelope.id;
|
||||
const std::int64_t run_at = envelope.run_at_ms.value_or(0);
|
||||
records_[id] = Record{envelope, ++order_};
|
||||
ready_[{run_at, order_}] = id;
|
||||
if (!envelope.idempotency_key.empty()) {
|
||||
dedupe_store_locked(envelope.idempotency_key, id);
|
||||
}
|
||||
log_op("{\"op\":\"enqueue\",\"env\":" + envelope.encode() + "}");
|
||||
observer_->counter("molo_tasks_enqueued_total", 1);
|
||||
observer_->event("tasks.enqueued", id + " " + envelope.name);
|
||||
cv_.notify_one();
|
||||
return id;
|
||||
}
|
||||
|
||||
bool TaskQueue::cancel(const std::string& id) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = records_.find(id);
|
||||
if (it == records_.end()) {
|
||||
return false;
|
||||
}
|
||||
if (lease_by_task_.find(id) != lease_by_task_.end()) {
|
||||
return false; // leased tasks finish; cancel only affects queued ones
|
||||
}
|
||||
terminal_locked(id);
|
||||
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(id) + "\"}");
|
||||
observer_->counter("molo_tasks_cancelled_total", 1);
|
||||
observer_->event("tasks.cancelled", id);
|
||||
return true;
|
||||
}
|
||||
|
||||
std::optional<TaskQueue::Lease> TaskQueue::lease(long wait_ms) {
|
||||
std::unique_lock<std::mutex> lock(mutex_);
|
||||
const std::int64_t deadline = clock_.steady_ms() + wait_ms;
|
||||
while (true) {
|
||||
if (shutdown_) {
|
||||
return std::nullopt;
|
||||
}
|
||||
reap_expired_locked();
|
||||
if (auto got = try_lease_locked()) {
|
||||
return got;
|
||||
}
|
||||
if (wait_ms <= 0) {
|
||||
return std::nullopt;
|
||||
}
|
||||
const std::int64_t now = clock_.steady_ms();
|
||||
if (now >= deadline) {
|
||||
return std::nullopt;
|
||||
}
|
||||
// Wake early when the earliest delayed task becomes due.
|
||||
long wait_for = static_cast<long>(deadline - now);
|
||||
if (!ready_.empty()) {
|
||||
const std::int64_t due_wall = ready_.begin()->first.first;
|
||||
const std::int64_t now_wall = clock_.wall_ms();
|
||||
if (due_wall > now_wall) {
|
||||
const long until_due = static_cast<long>(due_wall - now_wall);
|
||||
if (until_due < wait_for) {
|
||||
wait_for = until_due;
|
||||
}
|
||||
}
|
||||
}
|
||||
cv_.wait_for(lock, std::chrono::milliseconds(wait_for));
|
||||
}
|
||||
}
|
||||
|
||||
std::optional<TaskQueue::Lease> TaskQueue::try_lease_locked() {
|
||||
if (ready_.empty()) {
|
||||
return std::nullopt;
|
||||
}
|
||||
const std::int64_t now_wall = clock_.wall_ms();
|
||||
auto it = ready_.begin();
|
||||
if (it->first.first > now_wall) {
|
||||
return std::nullopt; // earliest task still delayed
|
||||
}
|
||||
const std::string id = it->second;
|
||||
ready_.erase(it);
|
||||
auto record = records_.find(id);
|
||||
if (record == records_.end()) {
|
||||
return std::nullopt;
|
||||
}
|
||||
record->second.envelope.state = TaskState::Leased;
|
||||
const std::string lease_id =
|
||||
"L" + std::to_string(clock_.steady_ms()) + "-" + std::to_string(++lease_seq_);
|
||||
LeaseInfo info{id, clock_.steady_ms() + options_.lease_ms};
|
||||
leases_[lease_id] = info;
|
||||
lease_by_task_[id] = lease_id;
|
||||
observer_->counter("molo_tasks_leased_total", 1);
|
||||
return Lease{record->second.envelope, lease_id, info.deadline_steady_ms};
|
||||
}
|
||||
|
||||
void TaskQueue::reap_expired_locked() {
|
||||
const std::int64_t now = clock_.steady_ms();
|
||||
std::vector<std::string> expired;
|
||||
for (const auto& [lease_id, info] : leases_) {
|
||||
if (info.deadline_steady_ms <= now) {
|
||||
expired.push_back(lease_id);
|
||||
}
|
||||
}
|
||||
for (const auto& lease_id : expired) {
|
||||
auto lit = leases_.find(lease_id);
|
||||
if (lit == leases_.end()) {
|
||||
continue;
|
||||
}
|
||||
const std::string task_id = lit->second.task_id;
|
||||
leases_.erase(lit);
|
||||
lease_by_task_.erase(task_id);
|
||||
auto rit = records_.find(task_id);
|
||||
if (rit == records_.end()) {
|
||||
continue;
|
||||
}
|
||||
// Lease expiry without ack returns the task to queued with attempts
|
||||
// incremented (S7-3). A late ack from the original worker is ignored
|
||||
// because the lease is gone; duplicates are idempotent by contract.
|
||||
rit->second.envelope.attempts += 1;
|
||||
++expired_total_;
|
||||
observer_->counter("molo_tasks_lease_expired_total", 1);
|
||||
observer_->event("tasks.lease_expired", task_id);
|
||||
const RetryPolicy policy(RetrySpec{options_.retry.base_ms, options_.retry.cap_ms,
|
||||
options_.retry.jitter_ms,
|
||||
rit->second.envelope.max_retries});
|
||||
if (policy.exhausted(rit->second.envelope.attempts)) {
|
||||
TaskEnvelope env = rit->second.envelope;
|
||||
terminal_locked(task_id);
|
||||
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
|
||||
dead_letters_.push(std::move(env), "lease-expired-retries-exhausted");
|
||||
++failed_total_;
|
||||
} else {
|
||||
rit->second.envelope.state = TaskState::Queued;
|
||||
rit->second.envelope.run_at_ms.reset(); // redeliver immediately
|
||||
rit->second.order = ++order_;
|
||||
ready_[{0, rit->second.order}] = task_id;
|
||||
log_op("{\"op\":\"requeue\",\"id\":\"" + common::json_escape(task_id) + "\",\"attempts\":" +
|
||||
std::to_string(rit->second.envelope.attempts) + ",\"run_at_ms\":null}");
|
||||
cv_.notify_one();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bool TaskQueue::heartbeat(const std::string& lease_id) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = leases_.find(lease_id);
|
||||
if (it == leases_.end()) {
|
||||
return false;
|
||||
}
|
||||
if (it->second.deadline_steady_ms <= clock_.steady_ms()) {
|
||||
return false;
|
||||
}
|
||||
it->second.deadline_steady_ms = clock_.steady_ms() + options_.lease_ms;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool TaskQueue::ack(const std::string& lease_id) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = leases_.find(lease_id);
|
||||
if (it == leases_.end()) {
|
||||
return false;
|
||||
}
|
||||
const std::string task_id = it->second.task_id;
|
||||
leases_.erase(it);
|
||||
lease_by_task_.erase(task_id);
|
||||
terminal_locked(task_id);
|
||||
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
|
||||
++succeeded_total_;
|
||||
observer_->counter("molo_tasks_succeeded_total", 1);
|
||||
observer_->event("tasks.succeeded", task_id);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool TaskQueue::fail(const std::string& lease_id, bool retryable, const std::string& reason) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (leases_.find(lease_id) == leases_.end()) {
|
||||
return false;
|
||||
}
|
||||
fail_locked(lease_id, retryable, reason);
|
||||
return true;
|
||||
}
|
||||
|
||||
void TaskQueue::fail_locked(const std::string& lease_id, bool retryable,
|
||||
const std::string& reason) {
|
||||
auto it = leases_.find(lease_id);
|
||||
if (it == leases_.end()) {
|
||||
return;
|
||||
}
|
||||
const std::string task_id = it->second.task_id;
|
||||
leases_.erase(it);
|
||||
lease_by_task_.erase(task_id);
|
||||
auto rit = records_.find(task_id);
|
||||
if (rit == records_.end()) {
|
||||
return;
|
||||
}
|
||||
rit->second.envelope.attempts += 1;
|
||||
const int max_retries = rit->second.envelope.max_retries;
|
||||
const RetryPolicy policy(
|
||||
RetrySpec{options_.retry.base_ms, options_.retry.cap_ms, options_.retry.jitter_ms,
|
||||
max_retries});
|
||||
const bool drain = !accepting_;
|
||||
if (!retryable || policy.exhausted(rit->second.envelope.attempts) || drain) {
|
||||
TaskEnvelope env = rit->second.envelope;
|
||||
terminal_locked(task_id);
|
||||
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
|
||||
std::string why = truncate_reason(reason);
|
||||
if (drain && retryable && !policy.exhausted(rit->second.envelope.attempts)) {
|
||||
why += " (draining)";
|
||||
}
|
||||
dead_letters_.push(std::move(env), why);
|
||||
++failed_total_;
|
||||
observer_->counter("molo_tasks_failed_total", 1);
|
||||
} else {
|
||||
const long delay = policy.delay_for_attempt(rit->second.envelope.attempts - 1);
|
||||
const std::int64_t run_at = clock_.wall_ms() + delay;
|
||||
rit->second.envelope.run_at_ms = run_at;
|
||||
rit->second.envelope.state = TaskState::Queued;
|
||||
rit->second.order = ++order_;
|
||||
ready_[{run_at, rit->second.order}] = task_id;
|
||||
log_op("{\"op\":\"requeue\",\"id\":\"" + common::json_escape(task_id) + "\",\"attempts\":" +
|
||||
std::to_string(rit->second.envelope.attempts) +
|
||||
",\"run_at_ms\":" + std::to_string(run_at) + "}");
|
||||
++retried_total_;
|
||||
observer_->counter("molo_tasks_retried_total", 1);
|
||||
observer_->event("tasks.retrying", task_id + " in " + std::to_string(delay) + "ms");
|
||||
cv_.notify_one();
|
||||
}
|
||||
}
|
||||
|
||||
void TaskQueue::terminal_locked(const std::string& task_id) {
|
||||
auto rit = records_.find(task_id);
|
||||
if (rit == records_.end()) {
|
||||
return;
|
||||
}
|
||||
for (auto it = ready_.begin(); it != ready_.end(); ++it) {
|
||||
if (it->second == task_id) {
|
||||
ready_.erase(it);
|
||||
break;
|
||||
}
|
||||
}
|
||||
records_.erase(rit);
|
||||
}
|
||||
|
||||
void TaskQueue::execute(const Lease& leased, TaskRegistry& registry,
|
||||
const std::function<bool()>& stop_requested) {
|
||||
struct ExecState {
|
||||
std::atomic<bool> done{false};
|
||||
std::atomic<bool> cancel{false};
|
||||
std::mutex mutex;
|
||||
std::condition_variable cv;
|
||||
TaskOutcome outcome = TaskOutcome::RetryableFailure;
|
||||
std::string error;
|
||||
};
|
||||
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto rit = records_.find(leased.envelope.id);
|
||||
if (rit != records_.end()) {
|
||||
rit->second.envelope.state = TaskState::Running;
|
||||
}
|
||||
++running_;
|
||||
observer_->event("tasks.started", leased.envelope.id + " " + leased.envelope.name);
|
||||
}
|
||||
|
||||
std::shared_ptr<TaskHandler> handler = registry.create(leased.envelope.name);
|
||||
if (!handler) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
--running_;
|
||||
auto it = lease_by_task_.find(leased.envelope.id);
|
||||
if (it != lease_by_task_.end()) {
|
||||
fail_locked(it->second, false, "unknown-handler:" + leased.envelope.name);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
auto state = std::make_shared<ExecState>();
|
||||
TaskEnvelope env_copy = leased.envelope; // child thread owns only copies
|
||||
std::thread child([state, handler, env_copy, this]() mutable {
|
||||
TaskContext ctx{env_copy, clock_, observer_, state->cancel};
|
||||
try {
|
||||
const TaskOutcome outcome = handler->run(ctx);
|
||||
std::lock_guard<std::mutex> lock(state->mutex);
|
||||
state->outcome = outcome;
|
||||
} catch (const std::exception& e) {
|
||||
std::lock_guard<std::mutex> lock(state->mutex);
|
||||
state->outcome = TaskOutcome::RetryableFailure;
|
||||
state->error = truncate_reason(std::string("exception:") + e.what());
|
||||
} catch (...) {
|
||||
std::lock_guard<std::mutex> lock(state->mutex);
|
||||
state->outcome = TaskOutcome::RetryableFailure;
|
||||
state->error = "exception:unknown";
|
||||
}
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(state->mutex);
|
||||
state->done = true;
|
||||
}
|
||||
state->cv.notify_all();
|
||||
});
|
||||
|
||||
// Waiter: heartbeat the lease, fire the soft timeout cooperatively, and
|
||||
// enforce the hard timeout by failing the lease and recycling the worker.
|
||||
const std::int64_t start = clock_.steady_ms();
|
||||
const long soft = env_copy.timeout_soft_ms;
|
||||
const long hard = env_copy.timeout_hard_ms;
|
||||
bool soft_fired = false;
|
||||
bool hard_fired = false;
|
||||
std::int64_t last_heartbeat = start;
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state->mutex);
|
||||
while (!state->done) {
|
||||
state->cv.wait_for(lock, std::chrono::milliseconds(5));
|
||||
const std::int64_t now = clock_.steady_ms();
|
||||
if (stop_requested && stop_requested()) {
|
||||
state->cancel = true;
|
||||
}
|
||||
if (!soft_fired && now - start >= soft) {
|
||||
soft_fired = true;
|
||||
state->cancel = true;
|
||||
observer_->event("tasks.soft_timeout", env_copy.id);
|
||||
}
|
||||
if (now - last_heartbeat >= options_.heartbeat_ms / 2) {
|
||||
last_heartbeat = now;
|
||||
heartbeat(leased.lease_id);
|
||||
}
|
||||
if (now - start >= hard) {
|
||||
hard_fired = true;
|
||||
state->cancel = true;
|
||||
observer_->event("tasks.hard_timeout", env_copy.id);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!hard_fired) {
|
||||
TaskOutcome outcome;
|
||||
std::string error;
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(state->mutex);
|
||||
outcome = state->outcome;
|
||||
error = state->error;
|
||||
}
|
||||
child.join();
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
--running_;
|
||||
switch (outcome) {
|
||||
case TaskOutcome::Success: {
|
||||
auto it = leases_.find(leased.lease_id);
|
||||
if (it != leases_.end()) {
|
||||
const std::string task_id = it->second.task_id;
|
||||
leases_.erase(it);
|
||||
lease_by_task_.erase(task_id);
|
||||
terminal_locked(task_id);
|
||||
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(task_id) + "\"}");
|
||||
++succeeded_total_;
|
||||
observer_->counter("molo_tasks_succeeded_total", 1);
|
||||
observer_->event("tasks.succeeded", task_id);
|
||||
}
|
||||
break;
|
||||
}
|
||||
case TaskOutcome::RetryableFailure:
|
||||
fail_locked(leased.lease_id, true, error.empty() ? "handler-retryable" : error);
|
||||
break;
|
||||
case TaskOutcome::PermanentFailure:
|
||||
fail_locked(leased.lease_id, false, error.empty() ? "handler-permanent" : error);
|
||||
break;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Hard timeout: the lease fails retryable right away so the work is not
|
||||
// stuck; the orphaned child gets a grace period, then is detached and
|
||||
// the worker recycled (R7, S7-3/S7-4). The orphan owns only heap state
|
||||
// and copies, so detaching is memory-safe; a late finish cannot ack
|
||||
// because the lease is already gone.
|
||||
{
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
--running_;
|
||||
fail_locked(leased.lease_id, true, "hard-timeout");
|
||||
}
|
||||
{
|
||||
std::unique_lock<std::mutex> lock(state->mutex);
|
||||
state->cv.wait_for(lock, std::chrono::milliseconds(options_.orphan_grace_ms),
|
||||
[&] { return state->done.load(); });
|
||||
}
|
||||
if (state->done) {
|
||||
child.join();
|
||||
} else {
|
||||
child.detach();
|
||||
observer_->counter("molo_tasks_orphaned_total", 1);
|
||||
}
|
||||
recycle_.store(true);
|
||||
}
|
||||
|
||||
int TaskQueue::run_worker(const std::function<bool()>& stop_requested, TaskRegistry& registry) {
|
||||
recycle_.store(false);
|
||||
while (!stop_requested || !stop_requested()) {
|
||||
std::optional<Lease> got = lease(100);
|
||||
if (!got) {
|
||||
continue;
|
||||
}
|
||||
execute(*got, registry, stop_requested);
|
||||
if (recycle_.load()) {
|
||||
return 2; // hard timeout fired: recycle this worker
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
bool TaskQueue::run_one(TaskRegistry& registry, long lease_wait_ms) {
|
||||
std::optional<Lease> got = lease(lease_wait_ms);
|
||||
if (!got) {
|
||||
return false;
|
||||
}
|
||||
execute(*got, registry, {});
|
||||
return true;
|
||||
}
|
||||
|
||||
bool TaskQueue::recycle_requested() const {
|
||||
return recycle_.load();
|
||||
}
|
||||
|
||||
QueueStats TaskQueue::stats() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
QueueStats stats;
|
||||
stats.queued = ready_.size();
|
||||
stats.leased = leases_.size();
|
||||
stats.running = running_;
|
||||
stats.succeeded_total = succeeded_total_;
|
||||
stats.failed_total = failed_total_;
|
||||
stats.retried_total = retried_total_;
|
||||
stats.expired_total = expired_total_;
|
||||
return stats;
|
||||
}
|
||||
|
||||
std::optional<TaskEnvelope> TaskQueue::get(const std::string& id) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = records_.find(id);
|
||||
if (it == records_.end()) {
|
||||
return std::nullopt;
|
||||
}
|
||||
return it->second.envelope;
|
||||
}
|
||||
|
||||
std::vector<TaskEnvelope> TaskQueue::list_queued(std::size_t max) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<TaskEnvelope> out;
|
||||
for (const auto& [_, id] : ready_) {
|
||||
if (out.size() >= max) {
|
||||
break;
|
||||
}
|
||||
auto it = records_.find(id);
|
||||
if (it != records_.end()) {
|
||||
out.push_back(it->second.envelope);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
DeadLetterQueue& TaskQueue::dead_letters() {
|
||||
return dead_letters_;
|
||||
}
|
||||
|
||||
void TaskQueue::set_accepting(bool accepting) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
accepting_ = accepting;
|
||||
}
|
||||
|
||||
bool TaskQueue::accepting() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return accepting_;
|
||||
}
|
||||
|
||||
std::size_t TaskQueue::in_flight() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return leases_.size() + running_;
|
||||
}
|
||||
|
||||
void TaskQueue::abandon_all_to_dlq(const std::string& reason) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (const auto& [_, record] : records_) {
|
||||
TaskEnvelope env = record.envelope;
|
||||
log_op("{\"op\":\"terminal\",\"id\":\"" + common::json_escape(env.id) + "\"}");
|
||||
dead_letters_.push(std::move(env), reason);
|
||||
}
|
||||
records_.clear();
|
||||
ready_.clear();
|
||||
leases_.clear();
|
||||
lease_by_task_.clear();
|
||||
observer_->event("tasks.abandoned", reason);
|
||||
}
|
||||
|
||||
void TaskQueue::shutdown() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
shutdown_ = true;
|
||||
cv_.notify_all();
|
||||
}
|
||||
|
||||
void TaskQueue::compact() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (!log_) {
|
||||
return;
|
||||
}
|
||||
// Rewrite the log from live state: one enqueue op per live record.
|
||||
const std::filesystem::path file =
|
||||
std::filesystem::path(options_.durable_dir) / "task_queue.log";
|
||||
log_.reset();
|
||||
std::filesystem::remove(file);
|
||||
log_ = std::make_unique<bus::SegmentLog>(file);
|
||||
for (const auto& [_, record] : records_) {
|
||||
log_->append("{\"op\":\"enqueue\",\"env\":" + record.envelope.encode() + "}",
|
||||
clock_.wall_ms());
|
||||
}
|
||||
}
|
||||
|
||||
std::string TaskQueue::dedupe_lookup_locked(const std::string& key) {
|
||||
auto it = idempotency_.find(key);
|
||||
if (it == idempotency_.end()) {
|
||||
return "";
|
||||
}
|
||||
if (it->second.expires_wall_ms <= clock_.wall_ms()) {
|
||||
idempotency_.erase(it);
|
||||
return "";
|
||||
}
|
||||
// The key stays reserved for the full window even after the task
|
||||
// finished: duplicates return the original id without re-enqueue (S7-1).
|
||||
return it->second.task_id;
|
||||
}
|
||||
|
||||
void TaskQueue::dedupe_store_locked(const std::string& key, const std::string& task_id) {
|
||||
const std::int64_t expires =
|
||||
clock_.wall_ms() + common::Caps::kIdempotencyWindowHours * 3600 * 1000;
|
||||
idempotency_[key] = IdemEntry{task_id, expires};
|
||||
idempotency_order_.push_back(key);
|
||||
while (idempotency_order_.size() > options_.max_depth) {
|
||||
idempotency_.erase(idempotency_order_.front());
|
||||
idempotency_order_.pop_front();
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,158 @@
|
||||
#pragma once
|
||||
|
||||
#include <atomic>
|
||||
#include <condition_variable>
|
||||
#include <cstdint>
|
||||
#include <deque>
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/bus/segment_log.hpp"
|
||||
#include "supervisor/common/clock.hpp"
|
||||
#include "supervisor/common/observer.hpp"
|
||||
#include "supervisor/tasks/dead_letter_queue.hpp"
|
||||
#include "supervisor/tasks/retry_policy.hpp"
|
||||
#include "supervisor/tasks/task_envelope.hpp"
|
||||
#include "supervisor/tasks/task_registry.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
struct QueueStats {
|
||||
std::size_t queued = 0;
|
||||
std::size_t leased = 0;
|
||||
std::size_t running = 0;
|
||||
long succeeded_total = 0;
|
||||
long failed_total = 0;
|
||||
long retried_total = 0;
|
||||
long expired_total = 0;
|
||||
};
|
||||
|
||||
struct QueueOptions {
|
||||
std::string durable_dir; // empty means in-memory only (E7-2); set for E7-6
|
||||
long lease_ms = 60000;
|
||||
long heartbeat_ms = 15000;
|
||||
std::size_t max_depth = 100000;
|
||||
RetrySpec retry;
|
||||
long orphan_grace_ms = 5000; // join grace after a hard timeout before detach
|
||||
};
|
||||
|
||||
// At-least-once task queue with ack-after-success (BP-3, S7-3).
|
||||
//
|
||||
// Producer side: enqueue() validates, dedupes idempotency keys, and makes
|
||||
// the envelope leasable once its run_at arrives. Consumer side: lease()
|
||||
// hands out one envelope with an ack deadline; ack() completes it;
|
||||
// fail() retries with backoff or dead-letters; heartbeat() renews the
|
||||
// deadline. Leases that expire without ack return to queued with attempts
|
||||
// incremented. Redelivery is normal; handlers must be idempotent (R7-2).
|
||||
//
|
||||
// run_worker() is the off-event-loop consumer loop (R7-5) driven by
|
||||
// supervisor worker threads: it leases, executes with soft/hard timeouts,
|
||||
// and returns 2 when the worker should be recycled after a hard timeout.
|
||||
class TaskQueue {
|
||||
public:
|
||||
struct Lease {
|
||||
TaskEnvelope envelope;
|
||||
std::string lease_id;
|
||||
std::int64_t deadline_steady_ms = 0;
|
||||
};
|
||||
|
||||
TaskQueue(const QueueOptions& options, common::Clock& clock,
|
||||
std::shared_ptr<common::Observer> observer);
|
||||
~TaskQueue();
|
||||
|
||||
TaskQueue(const TaskQueue&) = delete;
|
||||
TaskQueue& operator=(const TaskQueue&) = delete;
|
||||
|
||||
// Returns the task id. Duplicate idempotency_key within 24h returns the
|
||||
// existing id without re-enqueue. Throws TaskError on cap violations.
|
||||
std::string enqueue(TaskEnvelope envelope);
|
||||
|
||||
bool cancel(const std::string& id);
|
||||
|
||||
// Leases one due envelope, waiting up to wait_ms (0 means non-blocking).
|
||||
// Returns nullopt on timeout or after shutdown().
|
||||
std::optional<Lease> lease(long wait_ms);
|
||||
|
||||
bool heartbeat(const std::string& lease_id);
|
||||
bool ack(const std::string& lease_id);
|
||||
bool fail(const std::string& lease_id, bool retryable, const std::string& reason);
|
||||
|
||||
// Worker entry points.
|
||||
int run_worker(const std::function<bool()>& stop_requested, TaskRegistry& registry);
|
||||
bool run_one(TaskRegistry& registry, long lease_wait_ms);
|
||||
bool recycle_requested() const;
|
||||
|
||||
QueueStats stats() const;
|
||||
std::optional<TaskEnvelope> get(const std::string& id) const;
|
||||
std::vector<TaskEnvelope> list_queued(std::size_t max) const;
|
||||
DeadLetterQueue& dead_letters();
|
||||
|
||||
// Graceful shutdown support (R7-9).
|
||||
void set_accepting(bool accepting);
|
||||
bool accepting() const;
|
||||
std::size_t in_flight() const; // leased plus running
|
||||
void abandon_all_to_dlq(const std::string& reason);
|
||||
void shutdown(); // wakes all lease waiters; run_worker loops still exit via stop
|
||||
void compact(); // rewrites the durable log from live state
|
||||
|
||||
private:
|
||||
struct Record {
|
||||
TaskEnvelope envelope;
|
||||
std::uint64_t order = 0;
|
||||
};
|
||||
|
||||
struct LeaseInfo {
|
||||
std::string task_id;
|
||||
std::int64_t deadline_steady_ms = 0;
|
||||
};
|
||||
|
||||
struct IdemEntry {
|
||||
std::string task_id;
|
||||
std::int64_t expires_wall_ms = 0;
|
||||
};
|
||||
|
||||
void replay();
|
||||
void log_op(const std::string& op);
|
||||
void reap_expired_locked();
|
||||
std::optional<Lease> try_lease_locked();
|
||||
void fail_locked(const std::string& lease_id, bool retryable, const std::string& reason);
|
||||
void terminal_locked(const std::string& task_id);
|
||||
void execute(const Lease& leased, TaskRegistry& registry,
|
||||
const std::function<bool()>& stop_requested);
|
||||
std::string dedupe_lookup_locked(const std::string& key);
|
||||
void dedupe_store_locked(const std::string& key, const std::string& task_id);
|
||||
|
||||
QueueOptions options_;
|
||||
common::Clock& clock_;
|
||||
std::shared_ptr<common::Observer> observer_;
|
||||
RetryPolicy retry_;
|
||||
DeadLetterQueue dead_letters_;
|
||||
std::unique_ptr<bus::SegmentLog> log_;
|
||||
|
||||
mutable std::mutex mutex_;
|
||||
std::condition_variable cv_;
|
||||
std::map<std::string, Record> records_; // live tasks by id
|
||||
std::map<std::pair<std::int64_t, std::uint64_t>, std::string> ready_; // (run_at, order) -> id
|
||||
std::map<std::string, LeaseInfo> leases_; // lease_id -> info
|
||||
std::map<std::string, std::string> lease_by_task_; // task_id -> lease_id
|
||||
std::map<std::string, IdemEntry> idempotency_;
|
||||
std::deque<std::string> idempotency_order_;
|
||||
std::uint64_t order_ = 0;
|
||||
std::uint64_t lease_seq_ = 0;
|
||||
std::size_t running_ = 0;
|
||||
long succeeded_total_ = 0;
|
||||
long failed_total_ = 0;
|
||||
long retried_total_ = 0;
|
||||
long expired_total_ = 0;
|
||||
bool accepting_ = true;
|
||||
bool shutdown_ = false;
|
||||
std::atomic<bool> recycle_{false};
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,36 @@
|
||||
#include "supervisor/tasks/task_registry.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
void TaskRegistry::register_handler(const std::string& name, Factory factory) {
|
||||
if (name.empty() || name.size() > 256 || !factory) {
|
||||
throw TaskError("TaskRegistry: bad handler registration");
|
||||
}
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
factories_[name] = std::move(factory);
|
||||
}
|
||||
|
||||
bool TaskRegistry::has(const std::string& name) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return factories_.find(name) != factories_.end();
|
||||
}
|
||||
|
||||
std::unique_ptr<TaskHandler> TaskRegistry::create(const std::string& name) const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto it = factories_.find(name);
|
||||
if (it == factories_.end()) {
|
||||
return nullptr;
|
||||
}
|
||||
return it->second();
|
||||
}
|
||||
|
||||
std::vector<std::string> TaskRegistry::names() const {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
std::vector<std::string> out;
|
||||
for (const auto& [name, _] : factories_) {
|
||||
out.push_back(name);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace molo::tasks
|
||||
@@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
#include <functional>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "supervisor/tasks/task_handler.hpp"
|
||||
|
||||
namespace molo::tasks {
|
||||
|
||||
// Name-to-handler factory registry. Workers resolve handler instances per
|
||||
// execution through this registry so handlers stay stateless and the queue
|
||||
// never depends on concrete handler types.
|
||||
class TaskRegistry {
|
||||
public:
|
||||
using Factory = std::function<std::unique_ptr<TaskHandler>()>;
|
||||
|
||||
void register_handler(const std::string& name, Factory factory);
|
||||
bool has(const std::string& name) const;
|
||||
std::unique_ptr<TaskHandler> create(const std::string& name) const;
|
||||
std::vector<std::string> names() const;
|
||||
|
||||
private:
|
||||
mutable std::mutex mutex_;
|
||||
std::map<std::string, Factory> factories_;
|
||||
};
|
||||
|
||||
} // namespace molo::tasks
|
||||
Reference in New Issue
Block a user