Converge every account onto every policy agreement it has not declined

An instance kept production-identical for extended manual testing is
otherwise taxed forever by its own safety controls: five consents, a
versioned terms gate on every mutating request, and every account
predating the trust and safety commit reading terms_version NULL because
init_db deliberately never backfills it. AcceptanceService grants each
agreement to each account that has not declined it, so the instance stays
production byte for byte while nobody clicks the same dialog again. It is
opt-in, dry run by default, and one switch per agreement type.

The application is not allowed to know it exists. One registration line
in main.py is the only import anywhere, there is no route, schema,
template, Devii tool or environment flag, and a unit test greps the tree
and fails the suite if a second importer appears. The decline register
needs no storage: the ledger is append-only in effect, the service only
ever grants, so any withdrawn row was written by a human and that pair is
never touched again. No provenance column, nothing to observe.

Satisfaction is the gate's own expression, never a proxy, which is why
the ordering is created_at then id exactly as consent_state selects, and
why the live-account clauses are built with has_column: init_db ensures
terms_version and deletion_requested_at but not is_active, so a hardcoded
reference raises no such column on an instance where nobody was ever
suspended. Every write is one conditional statement decided on the real
rowcount, proven with sixteen processes racing one account to exactly one
ledger row and one audit row. The two existing audit keys carry it, with
actor kind service, because a service that silently mutated consent state
would be the worst possible exception to the append-only rule.

lensfl.md is the source brief accept.md records the design against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-10 00:17:47 +02:00
co-authored by Claude Opus 5
parent 1a5fc9428a
commit 7e37122f9f
20 changed files with 1834 additions and 2 deletions
+2
View File
@@ -119,6 +119,7 @@ from devplacepy.services.xmlrpc import XmlrpcService
from devplacepy.services.audit import AuditService
from devplacepy.services.moderation.service import ModerationService
from devplacepy.services.moderation.screening import ContentRefused
from devplacepy.services.acceptance.service import AcceptanceService
from devplacepy.services.audit import record as audit
from devplacepy.services.push import PushService
from devplacepy.services.telegram import TelegramService
@@ -283,6 +284,7 @@ async def lifespan(app: FastAPI):
service_manager.register(XmlrpcService())
service_manager.register(AuditService())
service_manager.register(ModerationService())
service_manager.register(AcceptanceService())
service_manager.register(PushService())
service_manager.register(TelegramService())
service_manager.register(TelegramOutboxService())
+4
View File
@@ -118,6 +118,10 @@ devplace devii reset-quota --all # Reset every quota (users and guests)
`ModerationService` is a lock-owner `BaseService` (default-enabled, hourly, floor 300s) with two jobs: it purges accounts whose deletion grace window has closed (`deletion.purge_due`, the same code path as `devplace accounts prune`), and it reports the moderation queue's service-level snapshot - logging when a report is past the published response window and exposing the queue counts, the oldest open age and the pending-purge count as `collect_metrics` stat cards. It owns no request-path work; the queue itself is entirely synchronous. Full subsystem detail in `devplacepy/services/moderation/CLAUDE.md`.
## Acceptance convergence (`services/acceptance/service.py`)
`AcceptanceService` is a lock-owner `BaseService` (**opt-in**, `default_enabled = False`, five minutes, floor 60s) that grants every policy agreement to every account which has not declined it, so a production-identical instance used for manual testing never interrupts with an acceptance dialog. It is invisible to the rest of the application by contract: one registration line in `main.py` is the only import anywhere, and there is no route, schema, template, Devii tool or environment flag. The decline register is the consent ledger itself - the service only ever writes `granted`, so any `withdrawn` row was written by a human and that pair is never touched again. Three gates stand between a fresh install and a single written row (service disabled, every agreement disabled, dry run on). Full subsystem detail in `devplacepy/services/acceptance/CLAUDE.md`; the design record is `accept.md` at the repository root.
## Multi-worker concurrency (preferred rules)
`uvicorn --workers N` = N independent processes sharing only the filesystem and SQLite DB. Module-global caches/counters are per-process, so a local `clear()` is invisible to siblings. Full reference: admin docs `Production -> Multi-worker and concurrency` (`templates/docs/production-concurrency.html`). Enforce these:
+65
View File
@@ -0,0 +1,65 @@
# CLAUDE.md
This file documents the acceptance convergence subsystem (`devplacepy/services/acceptance/`). Claude Code auto-loads it whenever a file under this directory is read or edited. The full design record is [`accept.md`](../../../accept.md) at the repository root.
## Why this subsystem exists
An operator running a production-identical instance for extended manual testing is otherwise taxed forever by the platform's own safety controls: five consents, a versioned terms gate on every mutating request, and every account predating the trust-and-safety commit reading `terms_version = NULL` because `init_db` deliberately never backfills it. This service converges each account onto the acceptance state a real population would have produced itself, so the instance stays production byte for byte while nobody has to click the same dialog again.
It is off by default and it is never appropriate on a real production host.
## The two load-bearing ideas
**The application must not know.** There is no request-path branch, no schema, no route, no template, no Jinja global, no Devii tool and no environment flag. The only import of this package anywhere is the one registration line in `main.py`, and `tests/unit/services/acceptance/isolation.py` fails the suite if a second one appears. An environment flag would be exactly the knowledge the application is not allowed to have, which is why there is none.
**The decline register needs no storage.** `user_consents` is append-only in effect, so the latest live row for a `(user, kind)` pair already is the register. The service only ever writes `granted`; it follows that any `withdrawn` row in the ledger was written by a human, and the service never touches that pair again. No provenance column, no marker, no flag, and nothing for the application to observe.
## Module map
| File | Owns |
|---|---|
| `agreements.py` | `Agreement`, `AGREEMENTS`, `setting_key`, `label_for`, `agreement_for` |
| `pending.py` | `latest_consent`, `not_withdrawn`, `satisfied_clause`, `live_account_clauses`, `current_version`, `pending` |
| `grant.py` | `converge_user` plus the two private claim shapes and the audit call |
| `service.py` | `AcceptanceService`: config fields built from the registry, `run_once`, `collect_metrics` |
`pending.py` and `grant.py` import only `devplacepy.database` and `sqlalchemy` at module top; `generate_uid` and the audit recorder are imported lazily inside the functions that use them, mirroring `services/moderation/deletion.py`.
## The registry is the completeness guarantee
`AGREEMENTS` annotates `database.CONSENT_KINDS` with two facts: which `site_settings` key holds the policy version, and which `users` column the application's own gate reads. `terms` is the only agreement with a gate column, because `needs_acceptance` reads `users.terms_version` and not the ledger; `privacy` is versioned but ledger-only, matching `VERSION_KEYS` in `routers/profile/consent.py`.
A unit test asserts `{a.kind for a in AGREEMENTS} == set(CONSENT_KINDS)`. **A sixth consent fails the suite until it is classified here**, and it then appears in the admin form with no edit to the service, because the per-agreement config fields are built from the registry rather than written out by hand.
## Rules that must not regress
- **Satisfaction is the gate's own expression, never a proxy.** `pending` compares exactly what the application compares: `users.terms_version` against `get_setting("terms_version", "1") or "1"` for `terms`, the ledger row's `version` for `privacy`, the latest state for the other three. The `or "1"` is load-bearing: an admin settings save can write `terms_version = ""`, and a bare `get_setting` would make every account pending forever.
- **Order by `created_at DESC, id DESC`, never one of the two.** That pair is what `database.consent_state` selects, so it is the expression the gate evaluates. `consent_view` on the privacy tab orders by `created_at` alone; that is the display path, not a gate. Never introduce a third ordering.
- **Build the live-account clauses with `has_column`.** `init_db` ensures `terms_version`, `terms_accepted_at` and `deletion_requested_at` on `users`, but **not `is_active`** - that column is created implicitly the first time a suspension or a deletion writes it, so a hardcoded reference raises `no such column` on an instance where nobody was ever suspended. An absent column means no account can be in that state, so omitting the clause is the correct answer.
- **Every write is one conditional statement decided on the driver's real `rowcount`**, via `db.executable.execute(text(...), params).rowcount` inside `with db:`, exactly like `deletion.claim_deletion`. Sixteen real processes racing one account produce exactly one ledger row and one audit row.
- **Never write `updated_at` on `users`.** That is why `database.atomic.conditional_update_row` cannot be reused here: it appends `updated_at` unconditionally, the table has no such column, and creating one would make the service's rows distinguishable from the route's.
- **The ledger insert must stay byte-compatible with `set_consent`**, including `withdrawn_at = ''` rather than `NULL` on the unused side. A unit test compares the two field by field. It is not `set_consent` itself only because `set_consent` cannot express a precondition or join a caller's transaction.
- **One cache bump per run, not one per account.** `clear_user_cache` propagates a global `auth` version bump that makes every worker drop its whole user cache; `run_once` bumps once at the end, and only when a gate column actually changed.
- **The audit row is deliberate.** It uses the existing `terms.accept` and `consent.grant` keys with `actor_kind="service"`, `actor_username="acceptance"`. The audit log is the operator's record and no code path reads it, so it costs nothing in invisibility and is the only trace distinguishing a converged acceptance from a human one.
## What is deliberately not an agreement
| Excluded | Why |
|---|---|
| `users.age_band` | A declaration of fact, not an agreement. Fabricating a declared age would silently unlock `restricted` content for an account that declared 13-15. Sign the test account up with an adult date of birth instead. |
| `users.mature_opt_in` | A preference gated by the age band, with no ledger row and therefore no decline register. The site setting `moderation_mature_default_hidden` already turns the interstitial off instance-wide. |
| Guest consents | Nothing writes a consent row with `owner_kind = "guest"`, and the gateway resolves a guest to owner kind `anonymous`, which `consent_denied` exempts by design. |
| Preferences (`interactions_enabled`, notifications, customization) | None of them blocks anything. |
## The terms asymmetry, and the one operator step
`POST /auth/accept-terms` grants **two** agreements: `terms` and `privacy`. The service keeps them independent on purpose, because the per-agreement switches exist for edge-case testing; enabling both reproduces the human path exactly, enabling one is a deliberate divergence.
Withdrawing the `terms` consent does not clear `users.terms_version`, so a tester who wants to exercise the gate withdraws their own `terms` consent and then has an administrator bump `terms_version` at `/admin/settings`. Every other account converges within one interval; the tester stays gated indefinitely.
## Rules for extending this
- A new consent kind: classify it in `AGREEMENTS` (the test forces this), and nothing else.
- Never add a route, a schema, a template, a Devii tool or a `docs_api` entry. The surface is the generic services admin, exactly as for `NotificationRelayService` and `AuditService`.
- Never add a second decline register, a second ordering, or an environment check.
- Never write a persisted test that enables the service and asserts convergence: the suite is serial against one seeded database, and granting `terms` to every account would poison the moderation tests that assert the gate refuses. Convergence is covered by unit tests calling `converge_user` directly.
@@ -0,0 +1 @@
# retoor <retoor@molodetz.nl>
@@ -0,0 +1,37 @@
# retoor <retoor@molodetz.nl>
from dataclasses import dataclass
from devplacepy.database import CONSENT_KINDS
@dataclass(frozen=True)
class Agreement:
kind: str
version_setting: str
gate_column: str
stamp_column: str
AGREEMENTS: tuple[Agreement, ...] = (
Agreement("terms", "terms_version", "terms_version", "terms_accepted_at"),
Agreement("privacy", "privacy_version", "", ""),
Agreement("ai_third_party", "", "", ""),
Agreement("activity_recording", "", "", ""),
Agreement("container_credentials", "", "", ""),
)
def setting_key(kind: str) -> str:
return f"acceptance_grant_{kind}"
def label_for(kind: str) -> str:
return CONSENT_KINDS.get(kind, kind)
def agreement_for(kind: str) -> Agreement | None:
for agreement in AGREEMENTS:
if agreement.kind == kind:
return agreement
return None
+130
View File
@@ -0,0 +1,130 @@
# retoor <retoor@molodetz.nl>
from sqlalchemy import text
from devplacepy.database import CONSENTS_TABLE, _now_iso, db, get_table
from .agreements import Agreement
from .pending import has_gate_column, not_withdrawn, satisfied_clause
CONSENT_COLUMNS = (
"uid",
"owner_kind",
"owner_id",
"kind",
"version",
"state",
"granted_at",
"withdrawn_at",
"created_at",
"deleted_at",
"deleted_by",
)
def _gate_set_clause(agreement: Agreement) -> str:
users = get_table("users")
assignments = [f"{agreement.gate_column} = :version"]
if agreement.stamp_column and users.has_column(agreement.stamp_column):
assignments.append(f"{agreement.stamp_column} = :now")
return ", ".join(assignments)
def _insert_consent_sql(precondition: str) -> str:
columns = ", ".join(CONSENT_COLUMNS)
sql = (
f"INSERT INTO {CONSENTS_TABLE} ({columns}) "
"SELECT :row_uid, 'user', :uid, :kind, :version, 'granted', "
":now, '', :now, NULL, NULL"
)
if precondition:
return f"{sql} WHERE {precondition}"
return sql
def _consent_params(agreement: Agreement, uid: str, version: str, now: str) -> dict:
from devplacepy.utils import generate_uid
return {
"row_uid": generate_uid(),
"uid": uid,
"kind": agreement.kind,
"version": version,
"now": now,
}
def _claim_gate_column(agreement: Agreement, uid: str, version: str, now: str) -> bool:
claim = (
f"UPDATE users SET {_gate_set_clause(agreement)} "
"WHERE uid = :uid "
f"AND COALESCE({agreement.gate_column}, '') != :version "
f"AND {not_withdrawn('users.uid')}"
)
params = _consent_params(agreement, uid, version, now)
with db:
claimed = db.executable.execute(text(claim), params).rowcount
if claimed != 1:
return False
db.executable.execute(text(_insert_consent_sql("")), params)
return True
def _claim_ledger(agreement: Agreement, uid: str, version: str, now: str) -> bool:
precondition = (
f"{not_withdrawn(':uid')} AND {satisfied_clause(agreement, ':uid', '')}"
)
params = _consent_params(agreement, uid, version, now)
with db:
written = db.executable.execute(
text(_insert_consent_sql(precondition)), params
).rowcount
return written == 1
def _record(agreement: Agreement, user: dict, version: str) -> None:
from devplacepy.services.audit import record as audit
uid = user["uid"]
username = user.get("username") or ""
links = [audit.target("user", uid, username)]
if agreement.kind == "terms":
audit.record_system(
"terms.accept",
actor_kind="service",
actor_username="acceptance",
target_type="user",
target_uid=uid,
target_label=username,
new_value=version,
summary=f"{username} accepted terms version {version}",
links=links,
)
return
audit.record_system(
"consent.grant",
actor_kind="service",
actor_username="acceptance",
target_type="user",
target_uid=uid,
target_label=username,
new_value="granted",
metadata={"kind": agreement.kind},
summary=f"granted {agreement.kind} consent for {username}",
links=links,
)
def converge_user(agreement: Agreement, user: dict, version: str) -> bool:
uid = user.get("uid")
if not uid:
return False
now = _now_iso()
if has_gate_column(agreement):
won = _claim_gate_column(agreement, uid, version, now)
else:
won = _claim_ledger(agreement, uid, version, now)
if not won:
return False
_record(agreement, user, version)
return True
+78
View File
@@ -0,0 +1,78 @@
# retoor <retoor@molodetz.nl>
from devplacepy.database import CONSENTS_TABLE, db, get_setting, get_table
from .agreements import Agreement
DEFAULT_VERSION = "1"
LIVE_ACCOUNT_CLAUSES: tuple[tuple[str, str], ...] = (
("deletion_requested_at", "COALESCE(u.deletion_requested_at, '') = ''"),
("is_active", "COALESCE(u.is_active, 1) != 0"),
("deleted_at", "u.deleted_at IS NULL"),
)
def latest_consent(column: str, user_ref: str) -> str:
return (
f"COALESCE((SELECT c.{column} FROM {CONSENTS_TABLE} c "
f"WHERE c.owner_kind = 'user' AND c.owner_id = {user_ref} "
"AND c.kind = :kind AND c.deleted_at IS NULL "
"ORDER BY c.created_at DESC, c.id DESC LIMIT 1), '')"
)
def not_withdrawn(user_ref: str) -> str:
return f"{latest_consent('state', user_ref)} != 'withdrawn'"
def has_gate_column(agreement: Agreement) -> bool:
if not agreement.gate_column:
return False
if "users" not in db.tables:
return False
return get_table("users").has_column(agreement.gate_column)
def satisfied_clause(agreement: Agreement, user_ref: str, column_prefix: str) -> str:
if has_gate_column(agreement):
return f"COALESCE({column_prefix}{agreement.gate_column}, '') != :version"
if agreement.version_setting:
return f"{latest_consent('version', user_ref)} != :version"
return f"{latest_consent('state', user_ref)} != 'granted'"
def live_account_clauses() -> list[str]:
users = get_table("users")
return [
clause for column, clause in LIVE_ACCOUNT_CLAUSES if users.has_column(column)
]
def current_version(agreement: Agreement) -> str:
if not agreement.version_setting:
return DEFAULT_VERSION
return get_setting(agreement.version_setting, DEFAULT_VERSION) or DEFAULT_VERSION
def pending(agreement: Agreement, limit: int) -> list[dict]:
if "users" not in db.tables or CONSENTS_TABLE not in db.tables:
return []
bound = max(1, int(limit))
clauses = [
*live_account_clauses(),
not_withdrawn("u.uid"),
satisfied_clause(agreement, "u.uid", "u."),
]
sql = (
"SELECT u.uid, u.username FROM users u "
f"WHERE {' AND '.join(clauses)} "
"ORDER BY u.id LIMIT :limit"
)
rows = db.query(
sql,
kind=agreement.kind,
version=current_version(agreement),
limit=bound,
)
return [dict(row) for row in rows]
+168
View File
@@ -0,0 +1,168 @@
# retoor <retoor@molodetz.nl>
import logging
from devplacepy.database import CONSENT_KINDS, bump_cache_version
from devplacepy.services.acceptance.agreements import (
AGREEMENTS,
label_for,
setting_key,
)
from devplacepy.services.acceptance.grant import converge_user
from devplacepy.services.acceptance.pending import current_version, pending
from devplacepy.services.base import BaseService, ConfigField
logger = logging.getLogger(__name__)
DRY_RUN_KEY = "acceptance_dry_run"
BATCH_SIZE_KEY = "acceptance_batch_size"
DEFAULT_BATCH_SIZE = 200
MAX_BATCH_SIZE = 5000
SAMPLE_NAMES = 20
def sample_names(candidates: list[dict]) -> str:
names = [row.get("username") or row.get("uid") or "" for row in candidates]
shown = ", ".join(names[:SAMPLE_NAMES])
remaining = len(names) - SAMPLE_NAMES
if remaining > 0:
return f"{shown} and {remaining} more"
return shown
class AcceptanceService(BaseService):
title = "Acceptance convergence"
description = (
"Grants every policy agreement to every account that has not declined it, "
"so a production-identical test instance never interrupts manual testing "
"with an acceptance dialog. Off by default and never appropriate on a real "
"production host."
)
details = (
"An account that withdrew a consent is never granted it again, with no "
"further action: the latest row in the consent ledger is the decline "
"register. The service only ever grants, so any withdrawal in the ledger "
"was written by a human. Withdraw a consent from the profile privacy tab "
"to keep an account permanently outside the convergence."
)
default_enabled = False
min_interval = 60
METRICS_SECONDS = 60
config_fields = [
ConfigField(
DRY_RUN_KEY,
"Dry run",
type="bool",
default=True,
help=(
"Log which accounts would be converged and write nothing. Switch "
"off only after the log lists what you expect."
),
group="Safety",
),
ConfigField(
BATCH_SIZE_KEY,
"Accounts per agreement per run",
type="int",
default=DEFAULT_BATCH_SIZE,
minimum=1,
maximum=MAX_BATCH_SIZE,
help="Upper bound on one sweep. The remainder converges on the next run.",
group="Safety",
),
*[
ConfigField(
setting_key(agreement.kind),
label_for(agreement.kind),
type="bool",
default=False,
help=f"Grant the {agreement.kind} agreement to every account that has not declined it.",
group="Agreements",
)
for agreement in AGREEMENTS
],
]
def __init__(self) -> None:
super().__init__("acceptance", interval_seconds=300)
self._converged = {agreement.kind: 0 for agreement in AGREEMENTS}
async def run_once(self) -> None:
config = self.get_config()
dry = config[DRY_RUN_KEY]
limit = config[BATCH_SIZE_KEY]
gate_changed = False
for agreement in AGREEMENTS:
if not config[setting_key(agreement.kind)]:
continue
version = current_version(agreement)
candidates = pending(agreement, limit)
if not candidates:
continue
if dry:
self.log(
f"[dry run] {agreement.kind}: {len(candidates)} account(s) would be "
f"converged to version {version}: {sample_names(candidates)}"
)
continue
granted = 0
for user in candidates:
if converge_user(agreement, user, version):
granted += 1
gate_changed = gate_changed or bool(agreement.gate_column)
self._converged[agreement.kind] += granted
self.log(
f"{agreement.kind}: converged {granted} of {len(candidates)} pending "
f"at version {version}"
)
if len(candidates) == limit:
self.log(
f"{agreement.kind}: batch cap of {limit} reached, more remain "
"for the next run"
)
if gate_changed:
bump_cache_version("auth")
def collect_metrics(self) -> dict:
config = self.get_config()
limit = config[BATCH_SIZE_KEY]
enabled = [
agreement
for agreement in AGREEMENTS
if config[setting_key(agreement.kind)]
]
stats = [
{"label": "Dry run", "value": 1 if config[DRY_RUN_KEY] else 0},
{"label": "Agreements enabled", "value": len(enabled)},
]
rows = []
for agreement in AGREEMENTS:
is_enabled = config[setting_key(agreement.kind)]
count = len(pending(agreement, limit)) if is_enabled else 0
display = f"{count}+" if is_enabled and count == limit else str(count)
if is_enabled:
stats.append(
{"label": f"{agreement.kind} pending", "value": display}
)
rows.append(
[
agreement.kind,
CONSENT_KINDS.get(agreement.kind, agreement.kind)[:48],
1 if is_enabled else 0,
agreement.version_setting or "",
display if is_enabled else "",
self._converged[agreement.kind],
]
)
table = {
"columns": [
"Agreement",
"Policy",
"Enabled",
"Version setting",
"Pending",
"Converged",
],
"rows": rows,
}
return {"stats": stats, "table": table}
@@ -100,4 +100,38 @@ The operator transcribes these into the store listing; every answer is a fact ab
A wrong removal is undone with the **Restore content** decision, or from `/admin/trash`, which
restores the whole deletion event under one stamp. Nothing a moderator removes is destroyed until it
is purged.
## Acceptance convergence on a test instance
An instance kept production-identical for extended manual testing otherwise stops the tester at the
same acceptance dialogs forever. The **Acceptance convergence** background service grants every
policy agreement to every account that has not declined it. It is administrator-only, **off by
default**, and **never appropriate on a real production host** - there is no environment detection
anywhere in it, deliberately, so the only control is the operator's own judgement.
**Enable it.**
1. Open `/admin/services/acceptance`, Configuration tab, and switch on only the agreements the test
run needs. There is one switch per agreement type, so a single policy can be converged while the
rest stay pending.
2. Leave **Dry run** on. Start the service, then use **Run now**. Read the Logs tab and confirm the
accounts listed are the ones expected. Nothing has been written yet.
3. Switch Dry run off. The next run converges them, within five minutes or immediately with
**Run now**.
**Test a refusal path.** Withdraw the consent from your own profile's privacy tab. The consent
ledger is the decline register: the service only ever grants, so a withdrawal is permanent and no
number of intervals will undo it. Grant it again from the same tab to rejoin the convergence.
**Test the terms gate.** Withdraw your own Terms of Service consent, then bump `terms_version` at
`/admin/settings`. Every other account converges; you stay gated and can exercise
`/auth/accept-terms`, the in-place acceptance dialog and the JSON refusal as often as you like.
**Test a partial state.** Enable Terms of Service and leave Privacy Policy off. Accounts pass the
write gate but still show the privacy policy as unaccepted on their privacy tab.
**Turn it off.** Stop the service, or switch off the individual agreement. Nothing already written
is reverted, and nothing should be: those accounts are in exactly the state a real population would
have reached. Every convergence is in the audit log under `terms.accept` and `consent.grant` with
actor kind `service`, which is how you tell a converged acceptance from a human one.
</div>