moloapp/plans/12-audit-compliance.md

12 - Audit and Compliance

File: plans/12-audit-compliance.md Title: Audit and Compliance Scope: Immutable audit log (who/what/when/where, before/after), GDPR export and erasure cascade, retention policies with legal hold, field-level PII marking, log redaction, non-prod data masking.

1. Purpose and philosophy

This file owns proof: what happened, who did it, when, where, and what changed. It gives every MoloApp deployment an append-only audit log with tamper-evident chaining, a GDPR rights engine (export and erasure with cascade), retention policies with legal hold override, field-level PII marking on ORM models, centralized log redaction, and deterministic masking for non-prod data.

Philosophy:

• Accountability is a feature, not paperwork. GDPR Art. 5(2) makes the controller responsible for demonstrating compliance, so the framework generates the evidence by default.
• Append-only with teeth. Audit records are hash-chained and sealed; verification is a command, not a hope. Detection of gaps follows the signed-syslog model (sequencing plus missing-message detection).
• Redact before persistence. Sensitive values never reach a log, audit record, export, or non-prod copy. Scrubbing copies later is not a strategy.
• Erasure is a cascade with named exceptions. Art. 17 grounds drive deletion across ORM, KV, cache, audit metadata, and backups policy; Art. 17(3) exceptions (legal claims, legal obligation) plus legal holds are the only brakes, and each brake is itself audited.
• Minimize by mark. Every ORM field declares a PII class once; redaction, export, erasure, retention, and masking all read that mark. No second inventory anywhere.

2. Best practices (from research)

1. Audit records capture when, where, who and what for each event: timestamps plus interaction identifier (when), app identity plus entry point plus code location (where), source address plus user identity (who), event type plus severity plus action/object/result/reason (what). (Source: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html, retrieved 2026-10-23)
2. Log security events always: auth success and failure, authorization failure, session failures, admin actions, sensitive-data access, key use and rotation, data import and export, config change, legal opt-ins such as consent. Data changes and config modifications are at minimum considered. (Source: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html, retrieved 2026-10-23)
3. Never record directly, instead remove, mask, sanitize, hash, or encrypt: session IDs, access tokens, passwords, DB connection strings, encryption keys, bank or card data, sensitive PII (health, government identifiers, vulnerable people), data above the logging system's classification, opted-out data. Non-sensitive PII (names, phones, emails) gets special treatment; prefer de-identification (deletion, scrambling, pseudonymization). (Source: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html, retrieved 2026-10-23)
4. GDPR principles: lawfulness/fairness/transparency, purpose limitation, data minimisation, accuracy, storage limitation (identifiable form no longer than necessary), integrity and confidentiality, plus accountability (the controller must demonstrate compliance). (Source: https://gdpr-info.eu/art-5-gdpr/, retrieved 2026-10-23)
5. Erasure without undue delay when a ground applies: data no longer necessary, consent withdrawn with no other ground, successful objection, unlawful processing, legal-obligation erasure, child consent services data. Public-data controllers take reasonable steps to inform downstream processors. Exceptions: expression, legal obligation or public task, public health, archiving/research/statistics, legal claims. (Source: https://gdpr-info.eu/art-17-gdpr/, retrieved 2026-10-23)
6. Portability: the subject receives provided data in a structured, commonly used and machine-readable format and may transmit it to another controller; direct controller-to-controller transfer applies where technically feasible; the right must not adversely affect others' rights. It is without prejudice to erasure. (Source: https://gdpr-info.eu/art-20-gdpr/, retrieved 2026-10-23)
7. Rights requests: information on action taken without undue delay and within one month of receipt, extendable by two months for complex or numerous requests with reasons given inside the first month; electronic requests get electronic answers where possible; identity may be re-confirmed on reasonable doubt; manifestly unfounded or excessive requests may be charged or refused with the burden of proof on the controller. (Source: https://gdpr-info.eu/art-12-gdpr/, retrieved 2026-10-23)
8. Tamper evidence for logs: signed syslog adds origin authentication, message integrity, replay resistance, message sequencing, and detection of missing messages on top of RFC 5424. (Source: https://www.rfc-editor.org/rfc/rfc5848.txt, retrieved 2026-10-23)
9. Triangulated decisions: (a) OWASP when/where/who/what plus Art. 5 accountability define the audit record schema in section 6.1, nothing else may define it. (b) Art. 17 grounds plus Art. 17(3) exceptions plus OWASP consent logging define the erasure cascade with legal-hold brake. (c) Art. 20 machine-readable plus Art. 12 electronic-means define the JSON export format and delivery rule. (d) OWASP remove/mask/hash plus Art. 5 minimisation define redact-before-persist and PII marking. (e) RFC 5848 sequencing plus integrity define the hash chain and gap detection.

3. Structure

Root: src/molo/audit/, headers mirror under include/molo/audit/. One class or concern per file. Namespace root molo::audit, sub-namespaces log, rights, retention, pii, redact, mask.

• log/Record.h: struct Record (schema in 6.1, fixed JSON keys, no free text PII).
• log/Appender.h: class Appender (append-only write, monotonic seq, prev-hash link).
• log/Chain.h: class Chain (hash computation, seal block, verify range, gap report).
• log/Store.h: class Store interface (append, read range, seal); log/SqlStore.h (dual-backend impl over 04-data.md drivers).
• log/Query.h: class Query (filter by subject, actor, object, time range, paged).
• rights/Subject.h: struct SubjectRef (subject id type, tenant id from 11-tenancy.md).
• rights/Exporter.h: class Exporter (collect marked fields, emit JSON bundle).
• rights/Eraser.h: class Eraser (plan cascade, apply, report per-store outcome).
• rights/Request.h: class RightsRequest (id, type export/erase, state machine, due date).
• retention/Policy.h: struct Policy (per-model TTL, legal basis tag, action erase or anonymize).
• retention/Enforcer.h: class Enforcer (periodic scan driver, executed via 07-supervisor-replication.md scheduler).
• retention/Hold.h: class LegalHold (hold id, scope, reason, placed-by, release rule).
• pii/Mark.h: enum class PiiClass, struct FieldMark (see 6.4).
• pii/Registry.h: class Registry (model to field-mark map, built from ORM descriptors).
• redact/Redactor.h: class Redactor (single entry redact(record, context)).
• redact/Pattern.h: struct Pattern (named matchers for tokens, keys, card data).
• mask/Masker.h: class Masker (deterministic per-class masking, seeded).
• mask/Fixture.h: class FixtureScrubber (DB dump and fixture rewrite for non-prod).

4. Rules (hard, numbered)

1. Audit writes are append-only. No update or delete API exists on the store; correction is a new compensating record linking the original seq.
2. Every audit record carries seq, prev-hash, timestamp, actor, subject, action, object, result, before/after digests or redacted diffs, and request/trace IDs. Missing fields fail the write loudly.
3. Before/after values pass through the Redactor before persistence. Raw PII, secrets, tokens, or passwords in an audit record are a build-breaking defect.
4. All data-subject rights flow through RightsRequest with identity confirmation and due-date tracking. Ad-hoc export or delete code paths are forbidden.
5. Erasure cascades to every store holding the subject: ORM rows, KV keys, cache entries and tags, sessions, uploads metadata, task payloads, audit subject index (pseudonymized, never content-deleted). Each store reports done, held, or exempted.
6. Legal hold and Art. 17(3) exemptions block erasure and retention deletion for held records only, with reason recorded. Holds never silently expand scope.
7. Retention deletion never touches held records and never runs without a dry-run report first. Enforcer applies only the policy version that was dry-run.
8. Every ORM field is marked with exactly one PII class at model declaration. Unmarked fields fail model registration at startup.
9. Non-prod environments never hold prod PII. Seed, fixture, dump-restore, and staging-sync paths run the Masker; unmasked prod copies outside prod are forbidden.
10. Redaction is centralized in molo::audit::redact. Sibling log, error, and export code calls it; local regex scrubbers are forbidden.
11. Audit log reads require an explicit auditor role and are themselves audited (who queried what range, when). Export bundles are signed and expire.
12. All caps in section 6 are static constants. No autotuning of retention, chain sealing, export size, or masking throughput.

5. Development guidelines

• Build order follows section 7: PII marks first (everything reads them), then redaction, then the audit chain, then retention and holds, then rights, then masking.
• Extremely object-oriented: one small class per file, deep namespaces, shared behavior in exactly one base. No cross-layer imports outside declared interfaces.
• Every audit emission is a one-line call at the mutation site: AUDIT(model, action, before, after); the macro fills actor, subject, IDs, and chain link. Reviews reject manual record assembly.
• Structured over stringly: fixed JSON schemas for records, export bundles, erasure reports, and retention reports. Free-text reason fields are capped and redacted.
• Blocking work (chain sealing, export assembly, erasure cascade, retention scan, masking) runs off the event loop via 01-runtime.md blocking-work path or as 07-supervisor-replication.md tasks.
• Denials and holds emit structured logs plus metrics per 02-observability.md with reason codes, never the held content.
• Mobile-first and simplicity rules from plan.md apply to the auditor UI and rights self-service pages: responsive, plain language, one action per screen.

6. Specifications (concrete behaviors, caps, formats)

6.1 Audit record schema (JSON, fixed keys)

{"seq":n,"prev":"hex","hash":"hex","ts":"RFC3339-UTC","tenant":"id","actor":"user:42|system:scheduler|api:key-id","subject":"subject:uuid|none","action":"create|update|delete|read-sensitive|export|erase|login|grant|hold.place|hold.release|retention.run|config.change","object":"model:table:pk","result":"success|fail|defer","reason":"code","request_id":"...","trace_id":"...","before":"redacted-json|null","after":"redacted-json|null","policy":"retention-policy-version|null"}

Chain: hash = SHA256(prev || canonical-json-without-hash) using in-house hash per 03-security.md; genesis prev is 64 zeros. Seal block every 1000 records or 60 s: {"seal_seq":n,"root":"hex-chain-tip","ts":"..."} signed with the audit key epoch. Verify command replays hashes and reports first gap.

6.2 Retention and legal hold

Policy per model: {"model":"...","ttl_days":n,"basis":"contract|consent|legal-obligation|legitimate-interest","on_expiry":"erase|anonymize","grace_days":30}. Defaults: sessions 1 day, logs with PII 90 days, audit records 7 years (legal basis tag, jurisdiction-configurable), user content per-app config. Enforcer runs daily via 07-supervisor-replication.md scheduler, dry-run report first, then apply; report lists scanned, expired, held, deleted, anonymized counts. Hold: {"hold_id":"...","scope":"subject:uuid|model:table:pk|query-hash","reason":"...","by":"...","ts":"...","release":"manual|date"}; held records are skipped with counter audit_retention_held_total. Holds survive retention runs and erasure requests until released; release is audited.

6.3 Rights engine (export, erasure)

Request states: received, identity-pending, queued, running, done, refused, expired. Due date: received plus 30 days; one extension of 60 days with reason notice. Export bundle JSON: {"subject":"...","exported_at":"...","sources":[{"store":"orm:users","rows":n}],"data":{...marked fields only...},"checksum":"sha256"}; max bundle 256 MB, beyond that split into numbered parts; delivered electronically with signed URL expiry 7 days. Erasure cascade order: sessions, cache, KV, ORM rows (or anonymize where FK and audit law requires), uploads metadata, task payloads, audit subject index pseudonymized to erased:<request-id>. Report per store: done, held (hold id), exempted (Art. 17(3) limb). Refusals cite the limb and log the reason code.

6.4 PII marking

enum class PiiClass { None, Identifier, Contact, CredentialRef, Financial, Health, Location, Biometric, Child, Legal }. FieldMark { PiiClass cls; bool exportable; bool erasure; const char* basis; }. Defaults: Identifier and Contact are exportable and erasure-bound; CredentialRef never exported in raw form (presence boolean only); Financial, Health, Biometric, Child default to non-exportable raw, exportable only as category-present flags unless policy opts in. Registry is built at ORM model registration; unknown model or unmarked field aborts boot with named error.

6.5 Redaction and masking

Redactor runs on: audit before/after, log records via 02-observability.md sink hook, error bodies via 05-request-routing.md error renderer, export bundles for non-exportable classes. Patterns: session/token/key headers, *_KEY/*_SECRET/*_TOKEN, card-shaped digit runs, DB connection strings; replacement is [redacted:<class>], stable per class for grepability. Masker per class: Identifier to user-<n>, Contact to user<n>@example.invalid and +00-000-000<n>, Financial to zeros preserving length, others to class token; deterministic under seed so FK joins survive masking. FixtureScrubber rewrites dumps and seeds; max dump 2 GB, streams in 1 MB chunks, verifies zero unmasked marked values by rescan.

6.6 Caps (static)

Audit record max 64 KB (before/after each max 16 KB, overflow truncated with flag); chain seal every 1000 records or 60 s; log store retention of sealed tips forever, record bodies per policy; query page max 1000 rows; export bundle max 256 MB and max 1M rows per request; erasure cascade max 100k objects per run then continue-as-new-task; retention scan max 1M rows per run; redactor input max 1 MB per call; masker throughput floor 50 MB/s on reference host; holds max 100k active; rights requests max 10k open per tenant.

7. Execution steps (in build order)

1. PII marks: PiiClass, FieldMark, Registry wired into 04-data.md model registration with boot-time unmarked-field failure.
2. Redactor: patterns, class replacements, single redact() entry, hooks into 02-observability.md sinks and 05-request-routing.md errors.
3. Audit chain: Record, Appender, Chain, SqlStore on both backends, seal blocks, AUDIT() macro at ORM write hooks.
4. Audit query plus verify: Query with auditor-role guard, verify command with gap report, metrics per 02-observability.md.
5. Retention: Policy config, dry-run report, Enforcer as scheduled task via 07-supervisor-replication.md, per-model TTLs.
6. Legal hold: LegalHold CRUD with audit trail, hold check inside Enforcer and Eraser, release flow.
7. Rights requests: RightsRequest state machine with due dates, identity gate, export bundle builder with checksums.
8. Erasure cascade: per-store deleters (ORM, KV, cache, sessions, uploads, tasks), pseudonymization of audit subject index, report format.
9. Masking: Masker per class plus FixtureScrubber for dumps, seeds, and staging sync; rescan verifier.
10. Hardening: caps audit, auditor UI routes, CLI surface via 09-cli-ops.md, docs and living-example coverage.

8. Validation (how to verify each step)

• Step 1: model with an unmarked field refuses to boot naming the field; registry dump lists every model with its marks; integration test through a create URL.
• Step 2: planted secrets, tokens, card-shaped strings, and marked PII in logs, errors, and audit diffs all render as [redacted:<class>]; no raw value in any sink under failure injection.
• Step 3: 10k writes produce an unbroken chain on both backends; tamper-one-byte fixture fails verify naming the exact seq; seal blocks present at the configured interval.
• Step 4: non-auditor query returns 403 and is audited; auditor paged query returns exact ranges; verify on a pruned-gap fixture reports the gap range.
• Step 5: dry-run report counts match a manual query; apply deletes only expired unheld rows; TTL boundary test (day 89 kept, day 91 plus grace handled).
• Step 6: held subject survives retention and erasure with reason cited; release then re-run completes; hold place and release each emit audit records.
• Step 7: export bundle validates against schema, contains only marked exportable fields, checksum matches, second generation for identical data is byte-identical except timestamps.
• Step 8: erased subject has zero rows, keys, cache entries, sessions, or task payloads on rescan; audit records keep chain validity with pseudonymized subject; exempted-limb fixture refuses with cited reason.
• Step 9: masked dump restores and passes app boot; rescan finds zero raw marked values; FK joins still resolve; determinism check (same seed gives same output).
• Step 10: full integration suite green in the same session; caps review signed; living example demonstrates audit view, export, erasure, hold, and masked staging per 10-workflow-example.md rule 1.

9. References

• Depends on 02-observability.md: log record format, sink hook for redaction, audit metrics and trace linkage.
• Depends on 03-security.md: hash and signing primitives, key epochs, auditor auth guards, secret handling.
• Depends on 04-data.md: ORM descriptors for PII marks, dual-backend stores, KV/cache invalidation, migration journal for audit tables.
• Depends on 05-request-routing.md: error renderer hook for redaction, auditor and self-service routes, validation for rights forms.
• Depends on 07-supervisor-replication.md: scheduler for retention runs, task queue for export/erasure cascades, durable streams for audit replication.
• Depends on 11-tenancy.md: tenant selector on every record, per-tenant retention and holds isolation.
• Feeds 09-cli-ops.md: verify, export, erase, hold, retention dry-run commands and health inputs.
• Feeds 10-workflow-example.md: compliance demo routes and integration test templates.
• Defers to 13-operability.md for cross-system audit exchange formats; defers API policy surface to 14-api-governance.md; defers SBOM and provenance attestation to 15-supply-chain.md.

10. Open questions

1. Audit store engine: reuse 04-data.md SQLite WAL path for the chain, or a dedicated append-only segment file shared with 07-supervisor-replication.md streams?
2. Backup erasure: crypto-shredding via per-subject keys, or documented backup-expiry window (restores re-run erasure)? Leaning to crypto-shredding for flagged classes.
3. Default audit retention is 7 years; should the default vary per deployment region template, or stay single-default with explicit override?
4. Should audit reads require quorum (two auditor roles) for bulk export, or is single auditor plus audit-of-access enough for v1?

11. Sources

• OWASP Logging Cheat Sheet (event attributes when/where/who/what, events to log, data to exclude, de-identification): https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html (retrieved 2026-10-23)
• GDPR Art. 5, Principles relating to processing of personal data: https://gdpr-info.eu/art-5-gdpr/ (retrieved 2026-10-23)
• GDPR Art. 12, rights modalities and one-month deadline: https://gdpr-info.eu/art-12-gdpr/ (retrieved 2026-10-23)
• GDPR Art. 17, Right to erasure: https://gdpr-info.eu/art-17-gdpr/ (retrieved 2026-10-23)
• GDPR Art. 20, Right to data portability: https://gdpr-info.eu/art-20-gdpr/ (retrieved 2026-10-23)
• RFC 5848, Signed Syslog Messages (origin authentication, integrity, sequencing, missing-message detection): https://www.rfc-editor.org/rfc/rfc5848.txt (retrieved 2026-10-23)