moloapp/plans/04-data.md

04 - Data Layer

Scope: plan.md sections 2, 5, 10 step 3. Topics: ORM dual backends, self-healing, pooling, transactions, KV, cache, pubsub.

Purpose and philosophy

One data layer serves every feature: ORM models, sessions, KV, cache, pubsub, uploads metadata, task queues. Two SQL backends (Postgres, SQLite) hide behind one declared interface so application code never branches on the driver. The framework owns both drivers dependency-free: its own Postgres wire-protocol client and its own SQLite handling, no external driver libraries.

Philosophy: parameterized always, blocking never, healing automatic, audit preserved. Every query goes through the extended protocol path with bound parameters. Every I/O operation is non-blocking on the event loop (see 01-runtime.md). Schema mismatches heal themselves at runtime and emit a migration record for review, so dev and production converge without manual DDL. Caching is explicit: reads declare their source, writes invalidate through hooks, nothing is magically stale.

Best practices (from research)

1. Postgres wire protocol: frontend/backend exchange typed messages (Startup, Parse, Bind, Describe, Execute, Sync); the extended query flow separates SQL text from parameters, which is the structural defense against SQL injection (Source 1).
2. Pooling modes matter: session pooling supports all Postgres features; transaction pooling reassigns the server connection after each transaction but breaks session state (SET, LISTEN, advisory locks, SQL-level PREPARE); protocol-level prepared plans still work under transaction pooling (Source 6). Default to transaction pooling and forbid session-state features unless a session-pinned checkout is requested.
3. MVCC: each statement sees a snapshot; readers never block writers and writers never block readers, even at serializable via SSI (Source 5). Design transactions around short snapshots, optimistic conflict handling, and explicit isolation levels per transaction.
4. SQLite WAL: readers do not block writers and writers do not block readers; I/O is more sequential with fewer fsync calls; all processes must share one host (no network filesystem); SQLITE_BUSY can still occur; checkpoints bound WAL growth (Source 2). Run WAL mode always, one writer at a time, busy timeout plus retry, scheduled checkpoints.
5. SQLite schema change is intentionally narrow: ADD COLUMN and DROP COLUMN exist, but most alterations require the documented table-rebuild procedure (create new table, copy, drop, rename, rebuild indexes) inside a transaction (Source 3). The self-healer follows exactly this procedure for SQLite widen operations.
6. Pubsub delivery is at-most-once with no persistence: a message published with no subscribers is lost; ordering holds per publisher-to-channel; pattern subscriptions and sharded channels scale fan-out (Source 4). In-process pubsub mirrors these semantics; durability is a separate opt-in owned by 07-supervisor-replication.md, never silently implied here.
7. Triangulated decision: Postgres ADD COLUMN without volatile default is metadata-only and safe to auto-apply; rewriting ALTER TABLE forms are not MVCC-safe under concurrency (Source 5 context, Postgres ALTER TABLE docs). The healer therefore prefers additive DDL and schedules rewrites behind an advisory lock with retry.

Structure

Root: molo/data/, namespace molo::data. One class or concern per file, deep over flat.

• connection.hpp: Connection abstract base (exec, query, prepare, ping, backend id, escaping off-limits).
• pool.hpp, pool_config.hpp: Pool, PoolConfig (min/max size, acquire timeout, idle eviction, health probe query).
• transaction.hpp: Transaction RAII guard (begin, commit, rollback, isolation level, nesting via savepoints).
• value.hpp, row.hpp, result.hpp: Value variant (null, int64, double, string, bytes, bool, time), Row, Result (rows, affected count, last insert id).
• error.hpp: DbError with backend code, SQLSTATE for Postgres, extended result code for SQLite, healable flag.
• drivers/pg/conn.hpp, .../message.hpp, .../auth.hpp, .../extended.hpp, .../codecs.hpp: wire client (startup, SASLprep SCRAM-SHA-256 plus cleartext fallback refused by default, simple query, Parse/Bind/Execute/Sync, binary and text codecs for core types).
• drivers/sqlite/db.hpp, .../wal.hpp, .../busy.hpp: SQLite handling (open flags, WAL activation, busy timeout, checkpoint scheduler, single-writer gate).
• orm/model.hpp: Model CRTP base (table name, primary key, field list, hooks).
• orm/field.hpp: Field<T> descriptors (type, length, null, default, unique, index, check).
• orm/query.hpp: Query<T> builder (filter, order, limit, offset, join, select columns, count, exists).
• orm/relations.hpp: BelongsTo, HasMany, ManyToMany with lazy and eager (preload) loading.
• orm/schema.hpp: Schema introspection (tables, columns, indexes) per backend.
• orm/sync.hpp: Syncer (create missing tables/columns/indexes from models).
• orm/healer.hpp: Healer (classify error, plan DDL, apply, retry once, emit migration record).
• orm/migration.hpp, orm/migrator.hpp: Migration record (id, timestamp, SQL up, reason, checksum) and applier.
• kv/store.hpp, kv/memory.hpp: KvStore interface (get, set with TTL, del, exists, scan prefix) and in-memory backend.
• cache/cache.hpp, cache/policy.hpp, cache/singleflight.hpp: Cache (get-or-compute with TTL, explicit invalidate by key/tag, LRU with cap, stampede guard).
• pubsub/broker.hpp, pubsub/channel.hpp, pubsub/subscription.hpp: Broker (publish, subscribe exact and pattern, unsubscribe), fan-out on loop, backpressure drops counted, never blocking.

Rules (hard, numbered)

1. No raw SQL string concatenation with user input anywhere. All values travel as bound parameters via the extended protocol (Postgres) or bound host parameters (SQLite). Raw SQL is allowed only as constant fragments with bound values, reviewed and flagged raw().
2. No blocking I/O on the event loop. Drivers are async over the reactor from 01-runtime.md; SQLite file I/O uses the documented off-loop worker with a bounded queue.
3. One writer at a time per SQLite database file. Enforced by an async mutex in the driver, plus busy_timeout as second line; SQLITE_BUSY after retries is a loud error with a counter, never silent.
4. Self-heal retries a failed statement at most once per statement execution. If the retry fails, the error propagates with the migration record attached.
5. The healer only widens, never narrows: add tables/columns/indexes, widen types, extend lengths, drop NOT NULL with default backfill. Narrowing, renames, and drops require an explicit human-authored migration.
6. Every heal action emits a migration record (SQL text, reason, triggering error code, timestamp, checksum) to the migration journal before retry. No journal write, no retry.
7. Transactions default to READ COMMITTED; SERIALIZABLE is opt-in per transaction. Nested scopes use savepoints with unique names. A Transaction that goes out of scope without commit rolls back.
8. Pool checkouts have timeouts and every pool has static caps (size, queue, lifetime). Exhaustion fails fast with a structured error and metric, never waits forever.
9. Cache entries always carry TTL and a size cap; invalidation is explicit through ORM write hooks and tag purges. No unbounded maps, no infinite TTL.
10. Pubsub is at-most-once and non-persistent by default. Any code needing delivery guarantees must use the durable channel from 07-supervisor-replication.md; this layer must not pretend to persist.
11. All drivers handle credentials as opaque secrets handed from 03-security.md. Connection strings are redacted in logs; auth failures emit one structured event without the secret.
12. Every public data API takes a Context carrying deadline, correlation id, and tenant/database selector from 02-observability.md. No hidden globals for the current connection.

Development guidelines

• Build the driver interface first, then Postgres wire client, then SQLite handling, then pooling, then ORM core, then sync/healer, then KV/cache/pubsub. Each step compiles and passes integration tests before the next starts.
• Types map explicitly: C++ int64, double, string, bytes, bool, chrono time map to Postgres BIGINT, DOUBLE PRECISION, TEXT/VARCHAR, BYTEA, BOOLEAN, TIMESTAMPTZ and to SQLite INTEGER, REAL, TEXT, BLOB, INTEGER 0/1, TEXT ISO-8601. The mapping table lives in orm/types.md next to the code and is the single source of truth.
• Error classification is a pure function classify(backend, code) -> HealPlan | Fatal, unit-covered through the real stack per plan.md section 8 (integration tests that trigger real 42P01, 42703, 22001 on Postgres and real "no such table/column" on SQLite).
• Migrations are append-only files migrations/NNNN_name.sql plus a schema_migrations table recording applied checksums. Generated heals land in migrations/auto/ for review.
• KV and cache share the KvStore interface; cache adds TTL enforcement, tags, and singleflight. New backends implement the interface, never fork callers.
• Pubsub channels are plain strings topic:sub with * single-segment wildcard for patterns. Payload is bytes plus content-type hint; JSON is a helper, not the wire format.
• Observe everything through 02-observability.md: per-query latency histogram, pool saturation gauge, heal counter by plan kind, cache hit ratio, pubsub publish/drop counters. All zero-cost when disabled.

Specifications (concrete behaviors, caps, formats)

• Pool defaults: Postgres min 2, max 32 connections per process; SQLite max 1 writer plus 8 readers; acquire timeout 5 s; idle eviction 60 s; max connection lifetime 30 min; max queued waiters 256, overflow fails fast. All ceilings static in PoolConfig, tunable via validated config only.
• Query limits: statement timeout 30 s default (per-Context override, hard ceiling 300 s); max bound parameters 1024; max identifier length 63 chars (Postgres limit); max row payload 64 MiB; max rows per Query::all() 10,000 with mandatory cursor/pagination beyond.
• Transaction semantics: co_await txn.commit() flushes on the owning connection; the connection stays pinned for the transaction lifetime (session-pinned checkout); savepoint names molo_sp_<depth>_<counter>; serialization failures (SQLSTATE 40001) surface as retryable DbError, the caller decides, the layer never auto-retries user transactions.
• Sync behavior: at startup (and on demand) the Syncer diffs models against introspected schema and applies additive DDL inside one transaction per table; failures abort startup in production unless data.sync_on_boot=false, in which case a loud warning plus health-check downgrade results.
• Heal matrix: undefined_table -> CREATE TABLE; undefined_column -> ADD COLUMN; string truncation (22001) -> extend VARCHAR length x4 up to TEXT promotion; type mismatch int vs bigint -> widen; SQLite equivalents via ADD COLUMN or table-rebuild procedure. Anything else -> fatal, no DDL attempted.
• SQLite specifics: PRAGMA journal_mode=WAL, synchronous=NORMAL, foreign_keys=ON, busy_timeout=5000; auto-checkpoint every 1000 pages; startup verifies same-host file (refuses network paths); checkpoint failures counted and surfaced.
• KV format: keys UTF-8, max 512 bytes, max 1,048,576 keys per store (LRU eviction beyond); values max 16 MiB; TTL seconds, max 30 days; scan(prefix) paginated, max 1000 keys per page.
• Cache: default TTL 60 s; max 131,072 entries and 512 MiB total (whichever hits first evicts LRU); tags max 16 per entry; negative caching opt-in with max 10 s TTL; singleflight coalesces concurrent computes per key.
• Pubsub: channel name max 256 chars; payload max 1 MiB; per-subscriber queue 1024 messages then oldest-drop with pubsub.dropped counter; pattern subscriptions capped at 1024 per broker; publish never awaits subscribers.
• Replication hooks: every ORM write emits an invalidation event (table, pk, op, tx id) on the internal bus consumed by cache and by 07-supervisor-replication.md; ordering key is (tx_commit_seq, statement_seq).

Execution steps (in build order)

1. Value/Row/Result/Error types plus DbError classification table skeleton.
2. Postgres wire client: startup, auth (SCRAM-SHA-256), simple query, then Parse/Bind/Execute/Sync with text codecs, then binary codecs.
3. SQLite handling: open, pragmas, WAL activation, busy handler, single-writer gate, checkpoint scheduler.
4. Pool and Transaction: pooling modes (transaction default, session-pinned), savepoints, timeouts, health probes.
5. ORM core: Model, Field, Query builder, relations, schema introspection for both backends.
6. Syncer plus migration journal and migrator (apply, checksum verify, audit list).
7. Healer: classify, plan, apply DDL, retry once, emit record; heal matrix covered per backend.
8. KV store interface plus memory backend; cache with TTL, tags, LRU, singleflight; ORM write-through invalidation hooks.
9. Pubsub broker: exact and pattern subscriptions, fan-out, drop counters, backpressure behavior.
10. Hardening pass: caps audit, timeout audit, redaction audit, metrics wiring, docs and living-example models.

Validation (how to verify each step)

• Steps 1-4: integration tests through real servers (ephemeral Postgres via loopback, temp-file SQLite): connect, auth failure redaction, 10k sequential and 128 concurrent queries, pool exhaustion fails fast, transaction commit/rollback/savepoint behavior, statement timeout fires, idle eviction observed.
• Step 5: CRUD plus relations plus eager loading against both backends with identical assertions; parameterized-query proof (injection strings stored literally, never executed); join and pagination correctness.
• Step 6: fresh database boots to full schema on both backends; checksum mismatch aborts with loud error; migration list reproducible.
• Step 7: fault-injection suite drops a column/table, shortens a VARCHAR, narrows an int on each backend, then asserts auto-heal plus one retry success plus journal record; narrowing request refused; second failure propagates with record attached.
• Step 8: TTL expiry, tag purge, LRU eviction at cap, stampede test (100 concurrent computes collapse to 1), ORM write invalidates cached read across both backends.
• Step 9: 10k messages fan-out to 100 subscribers in order; slow subscriber drops counted, publisher never blocks; pattern delivery correct; no-subscriber publish is a counted no-op.
• Step 10: caps review checklist signed, metrics present in Prometheus export (see 09-cli-ops.md), living example exercises every behavior above in its suite.

References (sibling files this depends on or feeds)

• Depends on 01-runtime.md: event loop, timeouts, off-loop worker scheduling, connection lifecycle primitives.
• Depends on 02-observability.md: log format, correlation ids, metrics and tracing macros, per-query instrumentation.
• Depends on 03-security.md: credential handling, redaction rules, query-safety policy (parameterized-only), encryption for secrets at rest.
• Feeds 05-request-routing.md: validation binding to models, DI accessors for DB/cache, error-to-status mapping for DbError.
• Feeds 06-realtime.md: pubsub transport under WebSocket/SSE fan-out.
• Feeds 07-supervisor-replication.md: invalidation event stream, durable pubsub extension point, replication conflict input (row versions, tx ids).
• Feeds 08-clients-openapi.md: model-to-schema derivation for OpenAPI, test-client DB fixtures.
• Feeds 10-workflow-example.md: integration-test templates for data behaviors, living-example models and seed data.

Open questions

1. Should the Postgres wire client support SCRAM channel binding (tls-server-end-point) in v1, or is plain SCRAM-SHA-256 acceptable with TLS 1.3 underneath?
2. Do we need logical-replication output-plugin consumption in this layer for v1, or is the ORM write-event stream sufficient input for 07-supervisor-replication.md replication?
3. SQLite BEGIN CONCURRENT (multiple writers in WAL2) is experimental upstream; do we track it or keep single-writer permanently?
4. Cache backend choice for multi-instance v1: shared SQLite file, or a networked memory protocol owned by 07-supervisor-replication.md? This layer defines the interface either way.

Sources

1. PostgreSQL 18 documentation, Chapter 54 Frontend/Backend Protocol (message framing, startup, simple and extended query flows). https://www.postgresql.org/docs/18/protocol.html (retrieved 2026-10-23).
2. SQLite documentation, Write-Ahead Logging (concurrency, checkpointing, SQLITE_BUSY, same-host requirement). https://www.sqlite.org/wal.html (retrieved 2026-10-23).
3. SQLite documentation, ALTER TABLE (ADD/DROP COLUMN, table-rebuild procedure for other changes). https://www.sqlite.org/lang_altertable.html (retrieved 2026-10-23).
4. Redis documentation, Pub/sub (at-most-once delivery, ordering, pattern subscriptions, sharded pubsub). https://redis.io/docs/latest/develop/pubsub/ (retrieved 2026-10-23).
5. PostgreSQL 18 documentation, 13.1 Concurrency Control Introduction (MVCC snapshots, readers never block writers, SSI). https://www.postgresql.org/docs/current/mvcc-intro.html (retrieved 2026-10-23).
6. PgBouncer Features (session vs transaction vs statement pooling, SQL feature compatibility map). https://www.pgbouncer.org/features.html (retrieved 2026-10-23).