Files

T

retoor 54068f1953 docs: split plan into numbered detailed files with research

Ten sequential spec files plus index covering runtime,
observability, security, data, routing, realtime, supervisor,
clients, CLI ops, and workflow, each with best practices,
structure, rules, guidelines, specs, and cross-references.

2026-10-23 14:22:29 +02:00

15 KiB

Raw Blame History

02 - Observability Backbone

Title: Observability Backbone Scope: plan.md sections 2, 8 partial, 10 step 1 partial. Topics: logging, events, metrics, telemetry, zero-cost.

1. Purpose and philosophy

This file specifies the four observability pillars of MoloApp (logging, typed events, metrics, distributed tracing) plus the correlation plumbing that ties them together. Everything is implemented in-house in C++23 with zero third-party dependencies, but every wire format stays compatible with the surrounding ecosystem: Prometheus text exposition for metrics, W3C Trace Context for propagation, OTel-compatible span model and OTLP/HTTP JSON export for traces.

Guiding rule from plan.md: instrumentation stays everywhere and collects like a maniac, yet compiles to near-zero overhead when disabled. That means two switches for every pillar: a compile-time flag that removes code entirely, and a runtime flag that reduces a disabled call to one predictable branch. Disabled overhead is measured by benchmark, not assumed.

Correlation is the backbone inside the backbone: one request ID and one trace/span context flow through middleware, ORM, replication, tasks and all clients, and every log record, event, metric label set and span carries them.

2. Best practices (from research)

Separate API from SDK and exporters. OpenTelemetry C++ structures telemetry as an API plus SDKs plus exporters, with traces, metrics and logs all Stable, which confirms the split used in section 4 below.
Metric names use snake_case with a single-word application prefix relevant to the domain, counters end in _total, and units belong in the name (for example http_request_duration_seconds). Label sets stay low-cardinality.
Prometheus exposition is line-based text served over HTTP: # HELP and # TYPE comment lines followed by name{label="value"} value [timestamp] samples. Our /metrics renderer emits exactly this shape.
Distributed context propagates via the W3C traceparent header with version-format = trace-id "-" parent-id "-" trace-flags (trace ID 16 bytes hex, parent/span ID 8 bytes hex, 1-byte flags), plus tracestate for vendor extras. Inbound headers are parsed, outbound requests re-emit them.
HTTP server span names follow the OTel semantic convention {method} {target} where target is a low-cardinality route template (never a raw path), and {method} equals the http.request.method attribute. OTel HTTP span conventions are Stable.
Compile-time log-level removal follows the spdlog SPDLOG_ACTIVE_LEVEL pattern: calls below the active level compile to nothing, so debug and trace logging can stay in hot paths. Async logging follows the bounded-queue plus dedicated-writer-thread pattern with a drop policy and a dropped counter instead of blocking producers.

3. Structure

Directory root: src/observability/. Public headers mirror it under include/molo/observability/. One class or concern per file. Namespaces: molo::log, molo::events, molo::metrics, molo::trace, molo::corr.

Logging (src/observability/logging/):

level.h: enum class Level { Trace, Debug, Info, Warn, Error, Critical } plus to_string, from_string.
record.h: struct Record with timestamp, level, logger name, message, source location, request ID, trace/span IDs, structured fields.
logger.h/.cpp: class Logger, named loggers, per-logger level, child loggers inheriting context.
sink.h: class Sink interface (write(const Record&)); console_sink.h, file_sink.h/.cpp (size rotation), null_sink.h, ring_sink.h (in-memory tail for tests and logs inspection).
async_queue.h/.cpp: bounded MPSC queue plus writer thread, overflow policy, dropped-record counter.
json_format.h/.cpp: fixed JSON schema renderer for records.
registry.h/.cpp: class Registry holding named loggers, global level, sink list.
macros.h: MOLO_LOG_* macros with compile-time level gating and source location capture.

Events (src/observability/events/):

event.h: class Event base with type ID, timestamp, request/trace linkage.
listener.h: sync and async listener interfaces.
dispatcher.h/.cpp: class Dispatcher built on the pubsub core, listener registration, zero-listener fast path.
lifecycle_events.h: server start/stop, connection open/close, request start/finish, task enqueue/start/finish/fail.

Metrics (src/observability/metrics/):

counter.h, gauge.h, histogram.h, timer.h (RAII timer recording into a histogram).
labels.h: label set type with static cardinality caps.
registry.h/.cpp: class Registry owning all series, per-app instance with a process default.
prometheus.h/.cpp: text exposition renderer (# HELP, # TYPE, samples).

Tracing (src/observability/tracing/):

ids.h: TraceId (16 bytes), SpanId (8 bytes), TraceFlags, hex parse/format.
span_context.h: struct SpanContext (IDs plus flags plus tracestate).
span.h/.cpp: class Span with name, kind (server, client, internal), attributes, events, status, RAII end.
tracer.h/.cpp: class Tracer creating spans, null-span fast path when disabled or unsampled.
sampler.h: head-based samplers (always on/off, ratio).
propagator.h/.cpp: W3C traceparent/tracestate parse and inject for HTTP headers.
batch_processor.h/.cpp: bounded span buffer plus background batch export.
otlp_http.h/.cpp: OTLP/HTTP JSON exporter over the framework HTTP client.

Correlation (src/observability/correlation/):

request_context.h/.cpp: class RequestContext holding request ID plus current SpanContext, created per request, passed explicitly (never a global) through middleware, handlers, ORM and replication.

4. Rules (hard, numbered)

Every log call, event emission, metric update and span operation must compile to near-zero overhead when its pillar is disabled at compile time, and to at most one branch plus argument skip when disabled at runtime.
No observability call may block a request handler: logging and span export go through bounded queues with drop-on-full and a visible dropped counter.
Every log record emitted inside a request carries request_id; every record inside a trace also carries trace_id and span_id.
Metric label values must come from bounded sets only (route templates, methods, status classes, error kinds). Raw paths, user IDs, query strings and full error text are forbidden as label values.
Counters are named *_total, use snake_case, and framework metrics carry the molo_ prefix while application metrics carry the application prefix.
Histograms use cumulative buckets and always export _bucket, _sum and _count series.
Server span names are {METHOD} {route_template} per OTel HTTP conventions; raw paths in span names are forbidden.
Inbound traceparent is parsed and continued; malformed headers start a new trace and increment molo_trace_propagator_errors_total.
Outbound requests from framework clients (HTTP, WebSocket, Socket.IO, SSE, WebDAV) always inject current trace context when tracing is enabled.
Sampling decisions are head-based and immutable for the lifetime of a trace.
All buffers, series counts, label cardinalities, attribute counts and export rates have static ceilings declared in section 6; exceeding a ceiling drops data and counts the drop, never grows memory.
Secrets, passwords, tokens, session contents and request bodies are never logged, never set as span attributes, and never appear in metric labels. A shared redaction helper enforces this.

5. Development guidelines

Instrument first at every new call site: log line at the right level, one span for each cross-boundary operation, one counter or histogram for each countable outcome. Reviews reject uninstrumented handlers.
Prefer structured fields over string interpolation: MOLO_LOG_INFO(logger, "db.query", {{"table", t}, {"rows", n}}) instead of building sentences.
Levels: Trace for per-byte or per-frame detail, Debug for developer diagnostics, Info for lifecycle and business milestones, Warn for degraded but handled states, Error for failed operations needing attention, Critical for process-threatening states.
Keep hot-path instrumentation allocation-free: static format strings, integer IDs, reused label sets.
Tests assert on the ring_sink, the metrics registry snapshot and the in-memory span exporter, never on files or network.
Exporters and sinks are the only components allowed to touch I/O; pillars never call the network or filesystem directly.

6. Specifications (concrete behaviors, caps, formats)

Log record JSON schema (fixed keys, UTC RFC 3339 with millis in ts):

{"ts":"...","level":"info","logger":"molo.http","msg":"request.finished","request_id":"...","trace_id":"...","span_id":"...","fields":{...}}

Compile-time flags: MOLO_ACTIVE_LEVEL (default Debug in dev, Info in prod), MOLO_ENABLE_METRICS (default 1), MOLO_ENABLE_TRACING (default 1), MOLO_OBSERVABILITY master switch (0 removes all pillars). Runtime toggles mirror each flag and are read once per call site through a cached atomic.

Static ceilings (defaults, overridable via typed config only):

Async log queue: 8192 slots; max log line 8192 bytes (truncate plus truncated:true); file sink rotation 10 x 10 MB; ring sink 1024 records.
Metrics: max 65536 series per registry; max 64 distinct values per label key (65th maps to "other" and counts molo_metrics_cardinality_overflows_total); max 16 labels per metric.
Tracing: span buffer 4096; export batch 512 spans or 5 s interval, whichever comes first; max 64 attributes and 32 events per span; max attribute value 1024 bytes.
Default HTTP duration histogram buckets in seconds: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10.

Framework metrics shipped by default: molo_http_requests_total{method,route,code}, molo_http_request_duration_seconds{method,route}, molo_http_requests_in_flight, molo_db_queries_total{op,table}, molo_db_query_duration_seconds{op}, molo_log_records_total{level}, molo_log_dropped_total, molo_spans_total{kind}, molo_spans_dropped_total, molo_trace_propagator_errors_total, molo_events_total{type}.

Exposition endpoint contract (wired by 09-cli-ops.md): GET /metrics returns text/plain; version=0.0.4, status 200, full registry render under 100 ms for 10k series. Trace export contract: OTLP/HTTP POST of JSON ExportTraceServiceRequest to the configured collector URL with gzip, timeout 10 s, at most 3 retries with backoff.

7. Execution steps (in build order)

Correlation: RequestContext plus request-ID generation and middleware hook point (hook wired in 05-request-routing.md, context type lives here).
Logging core: levels, records, macros with compile-time gating, registry, console and null sinks, JSON format.
Logging async path: bounded queue, writer thread, file sink with rotation, ring sink, dropped counters.
Events: base event, dispatcher on pubsub core, sync/async listeners, lifecycle event set.
Metrics core: counter, gauge, histogram, timer, labels with cardinality caps, registry.
Metrics exposition: Prometheus text renderer plus render benchmarks.
Tracing core: IDs, span context, tracer, span lifecycle, samplers, null-span fast path.
Propagation: W3C parse/inject helpers plus client/server integration points (wiring in 01-runtime.md, 05-request-routing.md, 08-clients-openapi.md).
Export: batch processor plus OTLP/HTTP JSON exporter with retry and backoff.
Zero-cost pass: compile-time removal checks and disabled-overhead benchmarks for all four pillars.

8. Validation (how to verify each step)

Steps 1-2: integration test asserting every response carries X-Request-ID and every request-scoped log record contains that ID; macro test asserting calls below MOLO_ACTIVE_LEVEL emit no code (object-size or disassembly check on a probe TU).
Step 3: flood test pushing 100k records showing bounded memory, a nonzero molo_log_dropped_total under saturation, and no handler blocking beyond 1 ms p99.
Step 4: lifecycle test asserting request start/finish events fire exactly once per request; zero-listener benchmark showing dispatch under 5 ns per call.
Steps 5-6: exposition test comparing renderer output byte-shape against the Prometheus format (HELP/TYPE/sample lines, cumulative buckets); cardinality test asserting overflow maps to "other" and counts the overflow metric.
Step 7: span lifecycle test (parent/child linkage, attribute caps, truncation flags); sampler test asserting ratio sampler honors its rate within 5 points over 100k traces.
Step 8: W3C round-trip test using the spec vectors (00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 style) plus malformed-header tests; client test asserting outbound injection.
Step 9: exporter test against a stub collector asserting valid OTLP JSON, batching behavior and retry/backoff timing.
Step 10: benchmark gate failing the build if any disabled pillar adds more than 1 percent overhead versus a no-instrumentation baseline on the hot path.

9. References (sibling files this depends on or feeds)

Depends on 01-runtime.md: request lifecycle hook points, event-loop threading rules for background writer/exporter threads.
Feeds 04-data.md: per-query metrics, ORM span conventions, correlation passing into transactions.
Feeds 05-request-routing.md: middleware correlation hook, per-route label values, error-to-log mapping.
Feeds 06-realtime.md: WebSocket and Socket.IO span kinds and connection metrics.
Feeds 07-supervisor-replication.md: task lifecycle events and metrics, replication trace continuation.
Feeds 08-clients-openapi.md: client-side span creation and propagation injection, test-client observability inspection hooks.
Feeds 09-cli-ops.md: /metrics exposition renderer, health/readiness metric inputs, logs command tail source, event stream inspection.
Feeds 10-workflow-example.md: living-example dashboards and the observability demonstration checklist.

10. Open questions

OTLP/HTTP JSON is chosen over protobuf to avoid an in-house protobuf codec; revisit if a collector strictly requires protobuf.
Default trace sampling ratio in production is undecided (proposal: 0.1 with always-on errors); needs a measurement run first.
Per-app versus process-global metrics registry default needs confirmation with 05-request-routing.md once the application class shape lands.
Log file rotation count and size defaults are placeholders pending the operations profile in 09-cli-ops.md.

11. Sources (exact URLs with retrieval dates)

OpenTelemetry C++ documentation, API/SDK structure and Stable status for traces, metrics, logs: https://opentelemetry.io/docs/instrumentation/cpp/ (retrieved 2026-10-23). Main repo: https://github.com/open-telemetry/opentelemetry-cpp (retrieved 2026-10-23).
Prometheus metric and label naming (single-word application prefix): https://prometheus.io/docs/practices/naming/ (retrieved 2026-10-23).
Prometheus exposition formats (text format, HELP/TYPE/sample lines): https://prometheus.io/docs/instrumenting/exposition_formats/ (retrieved 2026-10-23).
W3C Trace Context Recommendation, traceparent version-format (REC 2021-11-23): https://www.w3.org/TR/trace-context/ (retrieved 2026-10-23).
spdlog repository, SPDLOG_ACTIVE_LEVEL compile-time gating and async logging design: https://github.com/gabime/spdlog (retrieved 2026-10-23).
OpenTelemetry semantic conventions for HTTP spans, Stable, {method} {target} naming: https://opentelemetry.io/docs/specs/semconv/http/http-spans/ (retrieved 2026-10-23).

15 KiB Raw Blame History