moloapp/plans/11-tenancy.md

11 - Multi-Tenancy

Title: Multi-Tenancy Scope: plan.md sections 2, 5, 7 partial. First-class tenants: isolated data paths, per-tenant config, quotas, rate limits, feature flags, per-tenant metrics; tenant-aware ORM, auth, routing, replication. Must land before the data layer hardens.

1. Purpose and philosophy

Every request, query, log record, metric, task, and replication entry belongs to exactly one tenant or carries an explicit system exemption. Tenancy is resolved once at the edge, attached to the request and data contexts, and enforced at the lowest layer that can do it (the database), never trusted to caller discipline.

One API over a strategy spectrum: Silo (dedicated database instance or cluster per tenant), Bridge (database-per-tenant on a shared Postgres cluster, file-per-tenant on SQLite), Pool (shared tables with tenant_id plus Row-Level Security on Postgres). Strategy is chosen per tenant at provisioning time, because switching models later is costly (see Best practices).

This file must land before 04-data.md hardens: the data Context shape, pool ownership, sync and healer behavior, and the migration journal all branch on isolation strategy. Retrofitting tenancy after those freeze would fork the ORM.

2. Best practices (from research)

1. Three partitioning models: "In the silo model, each tenant has their own unique physical resource. The bridge model logically separates tenants on a physical resource, for example, a database-per-tenant on a shared physical database. In the pool model, all tenants share a single logical construct on a physical resource, for example, a single schema shared by all tenants." (Source 1)
2. Silo gives the strongest isolation ("separate database instance and separate compute instance per tenant"), motivated by noisy-neighbour, compliance, or encryption requirements; bridge exists to cut cost by sharing the instance between tenants. (Source 1)
3. On Postgres bridge, AWS recommends database-per-tenant over schema-per-tenant: schemas are logical constructs that do not separate data on disk and share one directory, which risks system limits with many tenants. (Source 1, paraphrased)
4. A Postgres client connection "can only access data in a single database, the one specified in the connection request" (Source 4), so bridge needs one pool per tenant database while pool strategy keeps a single shared pool.
5. Pool enforcement belongs in the database: "When row security is enabled on a table ... all normal access to the table for selecting rows or modifying rows must be allowed by a row security policy", and "If no policy exists for the table, a default-deny policy is used" (Source 2). Policies can target ALL commands or SELECT, INSERT, UPDATE, DELETE, specific roles, or both.
6. The tenancy model "determines how each tenant's data is mapped to storage" and "Switching to a different model later is sometimes costly" (Source 3). Single-tenancy stores one tenant per database; multi-tenancy stores several tenants per database "with mechanisms to protect data privacy" (Source 3). Hence strategy-per-tenant at provisioning plus explicit migration procedures.
7. SQLite file-per-tenant rests on ATTACH: "The ATTACH DATABASE statement adds another database file to the current database connection" (Source 5), with a static limit on attached databases per connection. Under WAL, multi-file transactions stay atomic per file but a crash mid-COMMIT can split them across files (Source 5), so bridge on SQLite uses one connection per tenant file and never spans a transaction across files.
8. Tenant isolation "is fundamental to the design and development of software as a service (SaaS) systems" and crossing a tenant boundary "would represent a significant and potentially un-recoverable event for a SaaS business" (Source 6). Hence default-deny everywhere in this file.
9. Triangulation: AWS and Microsoft agree on the silo to pool spectrum with cost versus isolation tradeoffs (Sources 1, 3). AWS and the Postgres docs agree that bridge means database-per-tenant guarded by native access controls (Sources 1, 4). Pool safety is agreed from two sides: RLS default-deny at the engine (Source 2) plus app-level tenant scoping as defense in depth (this file, Rules R1 and R2).

3. Structure

Root: src/molo/tenancy/, namespace molo::tenancy. One class or concern per file pair.

• tenant_id.h: TenantId strong type (uuid7 plus validated slug).
• tenant.h/.cpp: Tenant record (id, slug, strategy, status, timestamps).
• strategy.h: IsolationStrategy { Silo, Bridge, Pool }, strategy_from_string.
• catalog.h/.cpp: TenantCatalog (create, lookup, suspend, tombstone, purge); store/catalog_store.h interface plus store/memory_catalog.h, store/db_catalog.h.
• resolver.h/.cpp: TenantResolver (ordered chain: subdomain, header, path prefix, auth claim).
• context.h: TenantContext (id, strategy, connection selector), embedded in the 04-data.md data Context and the request scope.
• middleware.h/.cpp: TenantMiddleware (resolve, attach, reject unknown or suspended).
• provisioner.h/.cpp: TenantProvisioner (provision, strategy migration, suspend, delete with floor).
• config_overlay.h/.cpp: TenantConfig (layered typed overlay over global config, cache, invalidation).
• quota.h/.cpp: QuotaManager (usage counters, pre-allocation checks).
• tenant_buckets.h/.cpp: per-tenant rate-limit buckets built on the 03-security.md limiter core.
• feature_flags.h/.cpp: FeatureFlags (per-tenant typed flags, cache, change events).
• metrics.h/.cpp: TenantMetrics (tenant-labeled recording with allowlist cap).
• orm_scope.h/.cpp: TenantScope (automatic query filter injection, privileged escape hatch).

4. Rules (hard, numbered)

• R1. Every ORM query in pool mode carries tenant scope. Unscoped data access fails closed unless it uses the explicit privileged API, which logs tenant pair plus reason.
• R2. Pool on Postgres always enables Row-Level Security with default-deny policies on every tenant table. The app-level filter is defense in depth, never the only enforcement.
• R3. Tenant database state is set per transaction or per checkout (SET LOCAL semantics) and reset on release. Shared pooled connections never retain a tenant between checkouts.
• R4. Bridge means database-per-tenant on Postgres and file-per-tenant on SQLite. Schema-per-tenant is not offered.
• R5. Cross-tenant reads and writes are forbidden except through explicit system APIs that record both tenant ids and the reason in logs and audit.
• R6. Tenant resolution runs before auth in the middleware chain. Unknown tenant yields 404 tenant_unknown; suspended tenant yields 403 tenant_suspended.
• R7. Quotas are checked before allocation. Over-quota fails closed: 429 with Retry-After for rate quotas, 507 for storage quotas.
• R8. The tenant metric label is allowed only for tenants on the metrics allowlist (static cap); all others record as tenant="other" and count an overflow metric.
• R9. Tenant deletion is a two-phase tombstone plus purge with a retention window. Purge (data destruction) requires explicit human confirmation and is never autonomous.
• R10. Every task envelope, oplog entry, cache key namespace, pubsub channel, log record, and trace span attribute set carries the tenant id. Replay and sync filter by tenant.
• R11. Per-tenant config overlays validate against the same typed schema as global config. An invalid overlay fails the update atomically; partial application is forbidden.
• R12. Isolation strategy is fixed per tenant at provisioning. Strategy migration is an explicit procedure with pre-checks, backfill, and post-verification, never silent.

5. Development guidelines

• Build in section 7 order: resolver plus context plus catalog first, then pool scoping, then bridge pools, then quotas, flags, and metrics. Each step ends green through the real stack.
• Keep the strategy matrix in exactly one place (TenantProvisioner plus the pool factory). Callers take a TenantContext and never branch on backend or strategy.
• Fixtures always provision at least two tenants plus system scope. Every tenant-visible route test asserts the neighbor tenant sees nothing (404 or empty, never a leak).
• RLS policies are generated from the model registry, reviewed as SQL text, and applied by the same migrator as schema DDL. No hand-written policy drift.
• Quota checks sit at admission points (middleware, pool checkout, upload start, task enqueue), not sprinkled in business logic.
• Feature flags default from global config; a tenant overlay only overrides named flags. Unknown flag names fail closed at update time.
• Deletion and purge paths are dry-run first with an itemized report (databases, files, keys, streams), then confirmed execution.

6. Specifications

6.1 Identity and resolution

• TenantId: internal uuid7 plus public slug matching [a-z0-9-]{3,63}, starting and ending alphanumeric. Slug is unique per deployment; uuid is the stored key.
• Resolvers, in default order (configurable): subdomain (<slug>.example.com), header X-Tenant-ID (slug or uuid), path prefix /t/<slug>/ (stripped before routing), auth claim tid (verified token only).
• First match wins; conflicting signals (header says A, path says B) yield 400 tenant_conflict. No signal yields the configured default tenant in single-tenant dev mode, else 404 tenant_unknown.
• Statuses: provisioning, active, suspended, tombstoned. Only active serves traffic; suspended serves 403 with a fixed body; tombstoned behaves as unknown.

6.2 Catalog record (JSON, also the on-disk shape)

{"id":"uuid7","slug":"acme","strategy":"pool","status":"active","pg_database":"molo_t_acme","sqlite_file":"tenants/acme.db","quotas":{...},"flags":{...},"created_at":"iso8601","updated_at":"iso8601"}. Slug to id cached in memory (cap 65536 entries, LRU); catalog reads bypass tenant scope via the privileged API and are audit-logged.

6.3 Pool mode (shared tables)

• Every tenant model gains tenant_id UUID NOT NULL; composite indexes lead with tenant_id. The ORM injects tenant_id = current into every generated query; raw fragments must bind it explicitly.
• Postgres: ALTER TABLE <t> ENABLE ROW LEVEL SECURITY plus generated policy CREATE POLICY tenant_isolation ON <t> USING (tenant_id = current_setting('app.current_tenant')::uuid) WITH CHECK (...) for ALL commands on the app role. The tenant GUC is set with SET LOCAL inside each transaction; pooled checkouts assert the GUC is empty before use.
• SQLite pool: shared file with tenant_id column plus app filter; RLS does not exist, so integration tests additionally run a hostile-shape suite where hand-written SQL without scope must fail at the driver guard.
• Max tenants per pool deployment: 100000 (soft, operations warning at 80 percent). Beyond that, split pools or move tenants to bridge.

6.4 Bridge mode

• Postgres: one database per tenant on the shared cluster, name molo_t_<slug> (truncated to 63 chars with uuid suffix on collision). Per-tenant pool: min 0, max 8, acquire timeout 5 s. Connection strings derive from the catalog, never from request input.
• SQLite: one file per tenant at tenants/<slug>.db, same pragmas as 04-data.md (WAL, foreign keys on, busy timeout 5000). One writer gate per file; no transaction ever spans two tenant files (see Best practices 7).
• Bridge caps: max 1024 tenant databases per Postgres cluster, max 4096 tenant files per deployment (soft; shard beyond).

6.5 Silo mode

• Dedicated Postgres cluster (or instance) per tenant, or dedicated file root per tenant on SQLite. The catalog stores a connection profile name; credentials come from 03-security.md secrets per profile. All pool defaults from 04-data.md apply unchanged inside the silo. Silo tenants skip RLS (single tenant per database) but keep tenant_id columns for portability.

6.6 Sync, healer, migrations per tenant

• Sync and self-heal run per tenant target: pool runs once on shared tables (policies included); bridge and silo run once per tenant database or file. Heal DDL in pool mode never touches tenant_id columns except widening the type.
• The migration journal is per target (shared journal for pool, per-database or per-file journal otherwise). molo migrate (09-cli-ops.md) iterates tenant targets with a per-tenant report and continues past single-tenant failure, exiting nonzero with a summary.

6.7 Quotas and rate limits

• Defaults per tenant (all overridable via overlay): requests 600/min with burst 2x for 10 s; concurrent connections 256; realtime connections 1024; storage 10 GB; uploads 1 GB/day; queued tasks 10000; export rows per request 100000.
• Rate overage: 429 with Retry-After and X-Tenant-ID echo. Storage overage: 507 with a problem body naming the quota. Usage counters live in memory plus periodic catalog persist (every 60 s or 1000 deltas); restart rebuilds from stores, never trusts memory alone.
• Noisy-neighbour guard: a tenant exceeding 50 percent of shared pool checkouts over a 60 s window is throttled to its fair share and emits a warning event; sustained abuse triggers an automatic suspension proposal that a human confirms.

6.8 Per-tenant config and feature flags

• Overlay keys mirror global config paths (http.body_max, auth.session_idle, ...); unknown paths rejected. Overlay cache TTL 30 s, tag-invalidated on update; every read path takes the merged view, never global alone when a tenant is active.
• Flags: typed map (bool, int64, string; max 256 flags per tenant, name [a-z0-9_.]{1,128}, string value max 4 KB). Reads are synchronous from cache; updates emit a tenant.flags_changed event that cache invalidation and long-lived connections honor within 30 s.

6.9 Per-tenant metrics, logs, traces

• Metrics: shared registry with tenant label gated by the metrics allowlist (default cap 512 tenants, LRU admission, overflow to tenant="other" plus molo_tenant_label_overflows_total). Framework per-tenant series: requests total, request duration, db queries total, quota denials total, realtime connections gauge.
• Logs: every record in tenant scope carries tenant_id; the redaction rules of 02-observability.md apply unchanged. Traces: tenant.id span attribute on server spans; sampling stays head-based and tenant-blind.
• Cardinality audit: CI fails if any new metric uses an unbounded tenant-derived label outside this section.

6.10 Tenant-aware auth, routing, realtime, replication

• Auth (with 03-security.md): users belong to one or more tenants; sessions bind to exactly one tenant at login; switching tenant requires re-resolution plus a membership check and rotates the session id. The login URL carries the tenant signal; token tid must equal the resolved tenant.
• Routing (with 05-request-routing.md): TenantMiddleware runs immediately after error mapping and observability, before security guards that need tenant quotas. Route helpers accept a tenant parameter for URL building in path-prefix mode.
• Realtime (with 06-realtime.md): rooms and channels are namespaced tenant:<id>:<room>; cross-tenant join is rejected; connection caps enforced per tenant at upgrade; broadcasts fan out only within the tenant unless a system channel explicitly allows wider.
• Replication (with 07-supervisor-replication.md): oplog entries gain "tenant"; sync sessions declare a tenant filter (single tenant, tenant list cap 128, or privileged all); conflict policies apply per tenant key; tenant provision, suspend, and purge replicate as control entries with human-confirm gates at apply time on peers.

6.11 Provisioning and deletion

• Provision: validate slug and strategy, allocate target (database, file, or pool row), run sync plus RLS policies, seed flags and quotas, warm caches, emit tenant.provisioned. Provision timeout 120 s; partial state rolls back with an itemized cleanup log.
• Suspend: flip status, drain tenant connections within 30 s, reject new work with 403, keep data. Unsuspend reverses with a health pre-check.
• Delete: mark tombstoned, revoke access, retain data for the retention window (default 30 days), then require human-confirmed purge. Purge emits an itemized destruction report and is blocked while replicas lag on that tenant.

7. Execution steps (in build order)

• E11-1: TenantId, Tenant, IsolationStrategy, in-memory catalog, TenantContext type shared with 04-data.md. Joint milestone: data Context carries tenant before drivers harden.
• E11-2: TenantResolver (all four signals, conflict rule) plus TenantMiddleware ordering test with 05-request-routing.md.
• E11-3: Pool scoping: TenantScope filter injection, tenant_id column convention, Postgres RLS policy generator, SET LOCAL checkout discipline.
• E11-4: Bridge pools: per-tenant Postgres databases plus pools, per-tenant SQLite files plus writer gates, catalog-driven connection factory.
• E11-5: TenantProvisioner: provision, sync integration, suspend and unsuspend, tombstone plus human-gated purge.
• E11-6: Quotas plus per-tenant rate buckets at admission points; usage persistence; 429 and 507 paths.
• E11-7: Config overlay plus feature flags with cache, validation, and change events.
• E11-8: TenantMetrics allowlist plus per-tenant series; log and trace tenant fields; replication tenant filter and control entries; living-example tenant demo.

8. Validation (how to verify each step)

• E11-1: context propagation test: tenant set at edge reaches ORM, log record, and task envelope unchanged; missing tenant fails closed in pool mode.
• E11-2: resolution matrix: subdomain, header, path, claim, conflict (400), unknown (404), suspended (403), dev default; order reconfiguration honored.
• E11-3: two-tenant CRUD suite on both backends: each tenant sees only its rows; direct SQL without scope returns zero rows on Postgres (RLS proof) and is rejected by the driver guard on SQLite; pooled checkout leak test (100k checkouts, GUC never bleeds).
• E11-4: bridge suite: per-tenant databases and files created, isolated pools saturate independently, cross-file transaction attempt refused, connection factory rejects request-derived names.
• E11-5: provision plus suspend plus purge drill: partial-provision rollback leaves no residue; tombstone retains data; purge without human confirmation refuses; purge with lagging replica blocks.
• E11-6: quota matrix: each quota trips at its limit with the documented status; burst and fair-share throttle behave within 20 percent; restart preserves usage within one persist window.
• E11-7: overlay validation: unknown path rejected atomically; flag change propagates to cached readers within 30 s; event observed on the bus.
• E11-8: metrics allowlist: 513th tenant records as "other" with overflow counter; per-tenant series render in /metrics; replication filter syncs one tenant without touching others; full suite green with the living-example tenant demo.

9. References

• Depends on 01-runtime.md: connection lifecycle, checkout discipline, static caps pattern.
• Depends on 02-observability.md: log schema, metric registry and label caps, trace attributes, redaction.
• Depends on 03-security.md: auth and session binding, rate limiter core, secrets for silo profiles, upload quotas.
• Depends on 04-data.md: ORM query builder, pooling, transactions, sync and healer, migration journal, KV and cache namespaces, pubsub channels. Joint hardening milestone before either file freezes.
• Depends on 05-request-routing.md: middleware ordering, Request scope, DI providers, error shapes.
• Depends on 06-realtime.md: room and channel namespacing, upgrade-time caps, broadcast fan-out.
• Depends on 07-supervisor-replication.md: task envelope tenant field, scheduler fairness, oplog shape, sync filters, multi-instance shared stores.
• Depends on 08-clients-openapi.md: test client tenant fixtures, per-tenant OpenAPI examples.
• Depends on 09-cli-ops.md: molo tenant commands, /metrics exposition, health aggregation per tenant.
• Follows 10-workflow-example.md: spec template, integration-only tests, living example coverage.
• Feeds 12-audit-compliance.md: tenant id on every auditable record, purge reports, suspension trail.
• Feeds 14-api-governance.md: per-tenant quota and versioning surface.

10. Open questions

1. RLS plus prepared plans under transaction pooling: which pool mode stays default for pool tenants, and does SET LOCAL survive the pooler path? Needs a joint test with 04-data.md.
2. Schema-per-tenant is excluded per AWS guidance; if a reviewer demands it, which cap (max schemas) makes it safe?
3. Should tenant slug renames be supported (pointer plus async rewrite) or forbidden (new tenant plus migration)? Leaning forbidden for v1.
4. Fair-share throttle constants (50 percent, 60 s window) are placeholders pending load measurements.
5. Cross-tenant analytics for operators: privileged read API only, or a dedicated export pipeline with its own audit? Proposal: export pipeline with audit owned alongside 12-audit-compliance.md.

11. Sources

1. AWS Database Blog, Choose the right PostgreSQL data access pattern for your SaaS application (silo, bridge, pool; database-per-tenant recommendation; noisy neighbours): https://aws.amazon.com/blogs/database/choose-the-right-postgresql-data-access-pattern-for-your-saas-application/ (retrieved 2026-10-23).
2. PostgreSQL 18 documentation, 5.9 Row Security Policies (ENABLE ROW LEVEL SECURITY, per-command and per-role policies, default-deny): https://www.postgresql.org/docs/current/ddl-rowsecurity.html (retrieved 2026-10-23).
3. Microsoft Learn, Multitenant SaaS Patterns, Azure SQL Database (tenancy model definitions, switching cost): https://learn.microsoft.com/en-us/azure/azure-sql/database/saas-tenancy-app-design-patterns?view=azuresql (retrieved 2026-10-23).
4. PostgreSQL 18 documentation, 5.10 Schemas (single-database connections, CREATE SCHEMA, search path): https://www.postgresql.org/docs/current/ddl-schemas.html (retrieved 2026-10-23).
5. SQLite documentation, ATTACH DATABASE (multi-file connections, per-file atomicity under WAL, attach limit): https://www.sqlite.org/lang_attach.html (retrieved 2026-10-23).
6. AWS whitepaper, SaaS Tenant Isolation Strategies (isolation as foundational SaaS concern): https://docs.aws.amazon.com/whitepapers/latest/saas-tenant-isolation-strategies/saas-tenant-isolation-strategies.html (retrieved 2026-10-23).