Compare commits
No commits in common. "master" and "project-page" have entirely different histories.
master
...
project-pa
47
CLAUDE.md
47
CLAUDE.md
@ -366,53 +366,6 @@ Failures at any implementation step block the workflow - never skip a failed ste
|
||||
|
||||
Gitea Actions workflow at `.gitea/workflows/test.yaml` runs on every push/PR to `master`: installs dependencies + Playwright Chromium, runs the full suite serially under coverage, publishes coverage HTML as an artifact, uploads failure screenshots. CI must be green before merging. Changes move through DTAP: Development (`make dev`) -> Test (CI suite + coverage on `master`) -> Acceptance (`master` to `production` promotion via `make deploy`) -> Production (Docker Compose stack). Only CI-green `master` commits are promoted to `production`.
|
||||
|
||||
## Diagnosing a production failure (the order that finds it fastest)
|
||||
|
||||
This procedure exists because a single "the editor is down" report turned out to be **three unrelated faults stacked on each other** (a stale URL, a firewalled network leg, and a corrupt database), and the investigation wasted hours by guessing before measuring. Work the layers outward from the browser; each step is cheap and each one eliminates a whole class of cause. **Never skip to a hypothesis, and never repair anything before the layer above it is proven healthy.**
|
||||
|
||||
**Layer 0 - is the request even arriving here?** Fetch the hostname over the public internet exactly as it resolves (`curl -sS -o /dev/null -w '%{http_code} %{remote_ip}' https://host/`). Compare the answering IP against this machine's own addresses (`ip -6 addr`, `curl https://api.ipify.org`). Two hostnames serve this platform by different routes - see the topology section above. **Never use `curl --resolve` to force a hostname onto an IP it does not resolve to**; that fabricates a path that no real traffic takes and produces confident, wrong conclusions.
|
||||
|
||||
**Layer 1 - which edge answered?** The error body identifies it. `application/problem+json` with `No site configured for host` is molohttp. A DevPlace HTML error page is the application. An nginx error page is the nginx container. A browser `ERR_*` with no body means nothing well-formed was returned at all.
|
||||
|
||||
**Layer 2 - same failure on both hostnames?** Run the identical authenticated request against `pravda.education` and `devplace.net`. Failing on **both** means the application or the database; failing on **one** means that host's edge. This single comparison is the highest-value measurement available and costs one command.
|
||||
|
||||
**Layer 3 - the application log, before any theory.** `docker logs --since 5m devplace-app-1`. Count error classes rather than reading prose (`grep -c malformed`). A recurring service-loop error is a systemic fault even when it looks unrelated to the symptom.
|
||||
|
||||
**Layer 4 - reproduce the failing hop in isolation.** Point the real code at the real upstream from a scratch harness rather than reasoning about it. Running `forward.proxy_http` against a live code-server is what exposed the duplicate `Date` header; reading the function had not. Use a scratch database (`DEVPLACE_DATABASE_URL`) so the harness never reaches production.
|
||||
|
||||
**Layer 5 - test from where the code actually runs.** The app runs **inside a container**; `127.0.0.1` there is not the host. `docker exec devplace-app-1 curl ...` is the only honest reachability test for a container-to-container hop. A hang with zero bytes means a packet was **DROPped** (firewall), a refusal means nothing is listening, and a slow error means the upstream answered badly - three different causes with three different fixes.
|
||||
|
||||
**Layer 6 - confirm the object exists before blaming the plumbing.** A 404 from a guard is not a proxy failure. Resolve the identifier through the application's own read surface (the workspace page, an admin JSON endpoint) with the affected account's session. A stale instance uid in a bookmarked URL looks exactly like an outage.
|
||||
|
||||
### Rules learned the hard way
|
||||
|
||||
- **State what a command will read or write before running it against production, and keep production access read-only until the diagnosis is complete.** The one write in a repair is the final swap, and it comes after verification, not before.
|
||||
- **Copy before repairing, and copy the whole set.** A WAL-mode SQLite database is `.db` **plus** `-wal` **plus** `-shm`; a `.db`-only copy silently discards every transaction still in the WAL. Stop writes first, or the snapshot is inconsistent. Never leave a stale `-wal` beside a recovered file - SQLite will replay it and re-corrupt the result.
|
||||
- **Repair on a copy, verify on the copy, and prove what was preserved.** `PRAGMA integrity_check` names the damaged objects; index damage is derived data and costs nothing (`REINDEX`, or `.recover`), while a table b-tree fault is the only kind that can lose rows. Diff row counts table by table between the original and the recovered file and report the delta - "it says ok" is not evidence that data survived.
|
||||
- **Verify the fix through the user's own path, with their account, in a real browser.** A green unit test and a 200 from `curl` did not prove the editor worked; driving Playwright through login, the code-server password prompt and a `.monaco-workbench` selector did.
|
||||
- **A measurement recorded in these files can go stale.** `services/containers/CLAUDE.md` recorded that `container_ip:port` times out from the app container while `gateway:published_host_port` connects. A later change (`make docker-attach`) inverted it, and a host firewall closed the documented leg entirely. Re-measure before trusting a recorded measurement, and update the record when it turns out to be false.
|
||||
- **Report each fault separately and correct yourself explicitly.** Three stacked faults produce a symptom that no single explanation covers, and an early wrong theory is worse than no theory once it is repeated as fact.
|
||||
|
||||
## Production hostnames and the devplace.net SSH tunnel (verified topology, do not re-derive)
|
||||
|
||||
**The platform answers on two public hostnames, and they reach the same application by two completely different paths.** This has already cost one debugging session; the failure mode is that a `curl --resolve devplace.net:443:<production ip>` "test" reports `No site configured for host: devplace.net` and looks like a total outage, when in fact devplace.net never touches the production edge at all.
|
||||
|
||||
| | `pravda.education` | `devplace.net` |
|
||||
|---|---|---|
|
||||
| DNS | `95.216.15.238`, `2a01:4f9:2a:100e::2` | `88.198.21.243`, `2a01:4f8:222:2c45::2` |
|
||||
| Machine | the production host itself | a separate front host (Hetzner, PTR `static.88-198-21-243.clients.your-server.de`) |
|
||||
| Path in | molohttp on `:443` -> `127.0.0.1:10500` | its own proxy -> **SSH tunnel** -> `127.0.0.1:10500` on production |
|
||||
| Reaches molohttp | yes | **no, never** |
|
||||
|
||||
**`devplace.net` is a front host that forwards over SSH.** It holds a persistent SSH session into the production host (visible there as an established inbound connection from `88.198.21.243` to port 22) and forwards through it to `127.0.0.1:10500`, which is the `docker-proxy` for the `devplace-nginx` container. The listening socket lives on the **front** host (an `ssh -L` style local forward), so the production host shows **no** sshd-owned listener - that absence is expected and is not evidence against the tunnel.
|
||||
|
||||
Two consequences that must not be forgotten:
|
||||
|
||||
- **molohttp has no `devplace.net` site, and that is correct.** Its site list is `mail`/`smtp`/`imap.molodetz.nl`, `pravda.education` and `*.tunnel.pravda.education`. devplace.net traffic enters below molohttp, straight into `127.0.0.1:10500`, so it needs no site. **Never "fix" this by adding a devplace.net site to molohttp** - devplace.net does not resolve to the production host, so such a site could never match, and its absence is not a bug.
|
||||
- **Both hostnames land on the same nginx and the same app**, so a request that fails on both is failing in the application, not in either edge. That comparison is the fastest triage available here: run the same authenticated request against both hostnames. Same failure on both means look at the app or the database; a failure only on devplace.net means look at the front host's proxy (WebSocket `Upgrade` headers are the usual culprit, exactly as for the production nginx locations below).
|
||||
|
||||
**Testing rule.** Never point a hostname at an IP it does not resolve to in order to "test" it. Fetch each hostname over the public internet as it really resolves (`curl https://devplace.net/...` and `curl https://pravda.education/...`), because forcing devplace.net onto the production IP tests molohttp with a `Host` it deliberately does not serve and proves nothing about the real path.
|
||||
|
||||
## Production deployment
|
||||
|
||||
Single-host Docker Compose (`docker-compose.yml`): an **app** container (Uvicorn, 2 workers) behind an **nginx** container. Full operator reference is the admin-only `Production` docs section (`templates/docs/production*.html`); conventions that must not regress:
|
||||
|
||||
16
Makefile
16
Makefile
@ -135,11 +135,10 @@ test-cache-clean:
|
||||
COMPOSE := docker compose -f docker-compose.yml -f docker-compose.containers.yml
|
||||
DEVPLACE_DATA_DIR ?= $(CURDIR)/data
|
||||
DOCKER_GID ?= $(shell stat -c '%g' /var/run/docker.sock 2>/dev/null)
|
||||
DEVPLACE_CONTAINER_NETWORK ?= bridge
|
||||
export DEVPLACE_DATA_DIR
|
||||
export DOCKER_GID
|
||||
|
||||
.PHONY: docker-build docker-up docker-attach docker-reload docker-down docker-logs docker-clean docker-prep ppy
|
||||
.PHONY: docker-build docker-up docker-reload docker-down docker-logs docker-clean docker-prep ppy
|
||||
|
||||
# Build the single shared container image every instance runs. Build once;
|
||||
# rebuild only when ppy.Dockerfile, the sudo shim, or pagent change.
|
||||
@ -154,23 +153,10 @@ docker-build: docker-prep
|
||||
|
||||
docker-up: docker-prep
|
||||
$(COMPOSE) up -d
|
||||
$(MAKE) docker-attach
|
||||
|
||||
# Workspace tunnels reach a container port that was never published on the host,
|
||||
# so the app must sit on the same docker network as the instances it runs. The
|
||||
# default bridge rejects the network-scoped aliases compose always sends, so
|
||||
# this cannot live in docker-compose.containers.yml and is wired here instead.
|
||||
docker-attach:
|
||||
@app=$$($(COMPOSE) ps -q app); \
|
||||
test -n "$$app" || { echo "app container is not running"; exit 1; }; \
|
||||
docker network connect $(DEVPLACE_CONTAINER_NETWORK) $$app 2>/dev/null \
|
||||
&& echo "attached app to the $(DEVPLACE_CONTAINER_NETWORK) network" \
|
||||
|| echo "app is already on the $(DEVPLACE_CONTAINER_NETWORK) network"
|
||||
|
||||
docker-reload:
|
||||
$(COMPOSE) restart app
|
||||
$(COMPOSE) up -d --wait
|
||||
$(MAKE) docker-attach
|
||||
|
||||
docker-down:
|
||||
$(COMPOSE) down
|
||||
|
||||
@ -1123,10 +1123,6 @@ What the overlay (`docker-compose.containers.yml`) changes:
|
||||
- **Data dir at a consistent path (critical).** When the app (in its container) runs `docker run -v <path>:/app`, the daemon resolves `<path>` against the **host**, not the app container. So the workspace/data dir must be mounted at the **same absolute path** on host and in the container - the make targets set `DEVPLACE_DATA_DIR` to the project's `./data` (an absolute host path) and mount it at that identical path on both sides. (Build contexts go through the docker API as a tarball, so they can stay in the container's temp dir - only the `/app` bind mount needs path consistency.)
|
||||
- **Ingress reach:** published container ports live on the **host**, so the overlay sets `DEVPLACE_CONTAINER_PROXY_HOST=host.docker.internal` (with `extra_hosts: host-gateway`) so the `/p/<slug>` proxy can reach them. On a bare-metal `make prod` deploy the app is already on the host, so the default `127.0.0.1` works and no overlay is needed (just install the docker CLI and run the services).
|
||||
|
||||
One piece of wiring cannot live in the overlay:
|
||||
|
||||
- **Workspace tunnel reach.** A workspace tunnel serves a port the member chose, which is almost never published on the host, so the app has to dial the container directly - and it can only do that from the container's own docker network. Compose cannot attach a service to the default `bridge` network (it always sends network-scoped aliases, which that network rejects), so `make docker-up` and `make docker-reload` run `make docker-attach`, an idempotent `docker network connect` of the app container to `DEVPLACE_CONTAINER_NETWORK` (default `bridge`). A bare `docker compose up -d` skips it and every tunnel to an unpublished port answers `502`. On a bare-metal `make prod` deploy the app is already on the host and reaches container IPs with no wiring at all.
|
||||
|
||||
Then build the shared `ppy` image once with `make ppy` and enable **Containers** on `/admin/services`. There is no in-app image building; every instance runs that one prebuilt image.
|
||||
|
||||
### nginx specifics
|
||||
|
||||
769
accept.md
Normal file
769
accept.md
Normal file
@ -0,0 +1,769 @@
|
||||
# DevPlace: acceptance convergence implementation design
|
||||
|
||||
Author: retoor <retoor@molodetz.nl>
|
||||
|
||||
Design input is [`lensfl.md`](lensfl.md). This document specifies a background service that converges every
|
||||
account onto every policy agreement the platform can ask for, so that an operator running a
|
||||
production-identical instance for manual testing is never interrupted by an acceptance dialog, a consent
|
||||
refusal, or the terms gate.
|
||||
|
||||
Nothing here is implemented. This is the specification.
|
||||
|
||||
Every claim below about existing behaviour was read out of the source at the time of writing, and the
|
||||
citations are exact. The subsystem this service operates on is documented in
|
||||
[`devplacepy/services/moderation/CLAUDE.md`](devplacepy/services/moderation/CLAUDE.md); the framework it is
|
||||
built on is documented in [`devplacepy/services/CLAUDE.md`](devplacepy/services/CLAUDE.md). Read both before
|
||||
implementing.
|
||||
|
||||
---
|
||||
|
||||
## 1. The problem, stated exactly
|
||||
|
||||
DevPlace enforces five consents, a versioned terms gate on every mutating request, and an age-banded
|
||||
maturity gate. Each is correct and each must stay. On a production-identical instance used for extended
|
||||
manual testing they produce a permanent tax: every new account, every bot in the fleet, and every account
|
||||
that predates the trust-and-safety commit (whose `users.terms_version` is deliberately left `NULL`, never
|
||||
backfilled) is blocked or degraded until a human clicks through the same dialogs again.
|
||||
|
||||
The instance must remain production. Not production-like with a flag threaded through the request path;
|
||||
production, byte for byte, with one background process quietly converging the acceptance state that a real
|
||||
population of users would have produced themselves.
|
||||
|
||||
Three constraints follow, and they are the whole design:
|
||||
|
||||
1. **The application must not know.** No request-path branch, no schema, no template, no route, no response
|
||||
field, no Jinja global, no Devii tool. If any consumer could observe the feature, the instance is no
|
||||
longer a production simulation and the exercise is worthless.
|
||||
2. **A manual decline must be permanent.** Testing the refusal path is the reason the operator is here. An
|
||||
account that has declined an agreement must never be converged, ever, with no further operator action.
|
||||
3. **It must be off, and stay off, unless deliberately switched on.** Two independent switches, both
|
||||
defaulting to off, plus a dry run that defaults to on.
|
||||
|
||||
---
|
||||
|
||||
## 2. The invisibility contract
|
||||
|
||||
"The system must not know" is vague. These are the testable clauses that replace it.
|
||||
|
||||
| # | Clause | How it is enforced |
|
||||
|---|--------|--------------------|
|
||||
| **I1** | No new table, no new column, anywhere. | The service writes only through existing columns of `users` and existing columns of `user_consents`. A unit test asserts the column sets it writes are subsets of what `database.moderation.set_consent` and `routers/auth/terms.py` already write. |
|
||||
| **I2** | No new value in any existing registry or enum. `CONSENT_KINDS`, `CONSENT_STATES`, `AGE_BANDS`, `MATURITY_LEVELS`, `REPORTABLE_TARGETS`, `UNREPORTABLE_TABLES`, `SOFT_DELETE_TABLES` are untouched. | Reviewed at implementation; the moderation registry test already fails on an unclassified table, and no table is added. |
|
||||
| **I3** | No new audit event key. The service records the same two keys the human paths record. | `events.md` gains no key, the header's stated key count does not change and no category count changes; only the "Recorded in" column of `terms.accept` and `consent.grant` gains a file. |
|
||||
| **I4** | The rows produced are indistinguishable in shape from the rows a real acceptance produces. | A unit test writes one row through `set_consent` and one through the service and compares column sets and value shapes field by field. |
|
||||
| **I5** | Nothing imports the package except one registration line in `main.py`. | A unit test greps the source tree for `services.acceptance` and asserts exactly one importing module, `devplacepy/main.py`. |
|
||||
| **I6** | The feature has no route, no schema, no template, no Devii action and no `docs_api` entry. | It inherits the generic services admin surface, exactly as `NotificationRelayService`, `PresenceRelayService`, `LiveViewRelayService` and `AuditService` do. See section 14 for why this does not violate the four-faces rule. |
|
||||
| **I7** | The test suite is unaffected. | `DEVPLACE_DISABLE_SERVICES=1` in `tests/conftest.py` stops `supervise()` from ever being called, and `default_enabled = False` keeps the service off even when services do run. The moderation tests that assert a `403` from the terms gate stay green by construction. |
|
||||
|
||||
The one thing that is deliberately visible is the audit log. Every state change in DevPlace records one
|
||||
append-only row, and a service that silently mutated user consent state would be the single worst possible
|
||||
exception to that rule. The audit log is the operator's record, not application state: no code path reads it
|
||||
to decide behaviour, so recording there costs nothing in invisibility and buys full traceability. This is
|
||||
the position `devplacepy/services/audit/CLAUDE.md` already takes.
|
||||
|
||||
---
|
||||
|
||||
## 3. Design axioms
|
||||
|
||||
Every axiom is an existing DevPlace pattern, named with its precedent. None is invented for this feature.
|
||||
|
||||
| # | Axiom | Precedent |
|
||||
|---|-------|-----------|
|
||||
| **A1** | A background subsystem is a `BaseService` with declarative `config_fields`, and gets start/stop, enable-on-boot, run-now, a config form, logs, live status and metrics for free. | `services/base.py`; `ModerationService`, `PushService` |
|
||||
| **A2** | An opt-in service sets `default_enabled = False` so `is_enabled()` reads `get_setting(enabled_key, "0")` and it never auto-starts on boot. | `services/base.py:203`; `BotsService` |
|
||||
| **A3** | Runtime policy lives in `site_settings`, read through the field's own `ConfigField.read()`, live-editable, never a code constant. | `services/base.py:218` (`get_config`) |
|
||||
| **A4** | The target set is a registry, not a literal. Every consumer reads the same structure; adding a type is one entry. | `REPORTABLE_TARGETS`, `CONSENT_KINDS`, `DATA_PATHS`; `PushService.config_fields` splices `*providers.admin_fields()` |
|
||||
| **A5** | A read-then-write mutation reachable from more than one path is a race until it is one conditional SQL statement decided on the driver's real `rowcount`. | `services/moderation/deletion.py:140` (`claim_deletion`), `queue.claim_open`, `database/atomic.py` |
|
||||
| **A6** | Lock-owner-only background work; never per worker. | `main.py:289-293`: services are registered in every worker, `supervise()` runs only in the worker that won `acquire_service_lock()` |
|
||||
| **A7** | Every state change records an audit row, and the recorder never raises into the caller. | `services/audit/record.py:288` (`record_system`) |
|
||||
| **A8** | A control that cannot evaluate its own precondition fails closed, never open. | `moderation/filter.classify` fails to `review`, never to `allow` |
|
||||
| **A9** | Legacy rows carry `NULL` in any column added after they were written, and a column added after the table may not exist at all. Every precondition uses `COALESCE`, and every raw-SQL reference to an optional column is guarded by `has_column`. | `claim_deletion`'s `COALESCE(deletion_requested_at, '')`; `deletion.due_purges`'s `if not table.has_column(...)`; the root `CLAUDE.md` race-safety rule |
|
||||
| **A10** | No silent caps. When a bound truncates the work, say so in the log. | Root `CLAUDE.md`, "Rigorous correctness verification" |
|
||||
|
||||
---
|
||||
|
||||
## 4. The agreement registry
|
||||
|
||||
An **agreement** is a policy statement the platform asks a member to accept, recorded in `user_consents`.
|
||||
There are exactly five and they are already enumerated by `database.CONSENT_KINDS`
|
||||
(`database/moderation.py:122`). The registry does not invent a parallel list; it annotates the existing one
|
||||
with the two facts the service needs.
|
||||
|
||||
`devplacepy/services/acceptance/agreements.py`:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Agreement:
|
||||
kind: str
|
||||
version_setting: str
|
||||
gate_column: str
|
||||
stamp_column: str
|
||||
|
||||
|
||||
AGREEMENTS: tuple[Agreement, ...] = (
|
||||
Agreement("terms", "terms_version", "terms_version", "terms_accepted_at"),
|
||||
Agreement("privacy", "privacy_version", "", ""),
|
||||
Agreement("ai_third_party", "", "", ""),
|
||||
Agreement("activity_recording", "", "", ""),
|
||||
Agreement("container_credentials", "", "", ""),
|
||||
)
|
||||
```
|
||||
|
||||
`kind` is a key of `CONSENT_KINDS`. `version_setting` is the `site_settings` key holding the policy version,
|
||||
`""` when the agreement is unversioned. `gate_column` is the `users` column the application's own gate reads,
|
||||
`""` when the agreement is ledger-only. `stamp_column` holds the acceptance instant beside it.
|
||||
|
||||
`terms` is the only agreement with a gate column, because `routers/auth/terms.needs_acceptance` reads
|
||||
`users.terms_version` and not the ledger. `privacy` is versioned but ledger-only: `VERSION_KEYS` in
|
||||
`routers/profile/consent.py:28` maps exactly `terms` and `privacy` to a version setting, and every other kind
|
||||
resolves to the literal `"1"` through `consent_version`. That asymmetry is in the application, not in this
|
||||
design; the registry records it rather than papering over it.
|
||||
|
||||
**Completeness is enforced, not remembered.** A unit test asserts
|
||||
`{a.kind for a in AGREEMENTS} == set(CONSENT_KINDS)`. A sixth consent added to `CONSENT_KINDS` in the future
|
||||
fails the suite until it is classified here, exactly as `tests/unit/database/moderation.py` does for
|
||||
reportable tables. This is the single most valuable line in the feature: it is what keeps the convergence
|
||||
complete as the platform grows.
|
||||
|
||||
The label rendered in the admin form is `CONSENT_KINDS[kind]`, read at field-construction time. The service
|
||||
never carries its own copy of the wording.
|
||||
|
||||
### 4.1 What is deliberately not an agreement
|
||||
|
||||
| Excluded | Why | The supported answer instead |
|
||||
|---|---|---|
|
||||
| `users.age_band` | A declaration of fact, not an agreement. Fabricating a declared age is not a production simulation, it is a falsification, and it would silently unlock `restricted` content for accounts that declared 13-15. | Sign the test account up with an adult date of birth. |
|
||||
| `users.mature_opt_in` | A content preference gated by the age band, with no entry in the consent ledger and therefore no decline register that outlives the 90-day `audit_log_retention_days` window. Adding a second, weaker decline mechanism for one field would violate DRY and clause I2. | The existing site setting `moderation_mature_default_hidden`, editable at `/admin/settings`, already turns the interstitial off instance-wide. |
|
||||
| Guest consents | Nothing writes a consent row with `owner_kind = "guest"` and no guest surface is gated by one. The gateway resolves a guest to owner kind `anonymous`, which `consent_denied` exempts by design. | None needed. |
|
||||
| `interactions_enabled`, notification preferences, customization toggles | Per-user preferences, not policy acceptances. None of them blocks anything. | None needed. |
|
||||
|
||||
Each exclusion is a boundary, not an oversight, and each is restated in the nested `CLAUDE.md`.
|
||||
|
||||
---
|
||||
|
||||
## 5. The decline register
|
||||
|
||||
This is the load-bearing idea and it needs no storage at all.
|
||||
|
||||
`user_consents` is append-only in effect: `set_consent` stamps `withdrawn_at` on the current row when it is
|
||||
granted and then inserts a new row either way (`database/moderation.py:289`), so the full history is provable.
|
||||
The **latest live row for a `(user, kind)` pair therefore already is the decline register**:
|
||||
|
||||
| Latest row state | Meaning | Service behaviour |
|
||||
|---|---|---|
|
||||
| no row | never asked | converge |
|
||||
| `granted`, current version | satisfied | nothing to do |
|
||||
| `granted`, stale version | policy changed since acceptance | converge |
|
||||
| `withdrawn` | **declined by a human** | never touch, permanently |
|
||||
|
||||
The service only ever writes `granted`. It follows that any `withdrawn` row in the ledger was written by a
|
||||
human action: the owner-only `POST /profile/{username}/consent`, the Devii consent tool acting on the owner's
|
||||
behalf, or the CLI. No provenance column, no marker, no flag, and nothing for the application to observe.
|
||||
Clause I1 is satisfied not by discipline but by the shape of the existing data.
|
||||
|
||||
**One SQL fragment expresses it**, defined once in `pending.py` and used by every read and every write:
|
||||
|
||||
```sql
|
||||
COALESCE((SELECT c.state FROM user_consents c
|
||||
WHERE c.owner_kind = 'user' AND c.owner_id = <user>
|
||||
AND c.kind = :kind AND c.deleted_at IS NULL
|
||||
ORDER BY c.created_at DESC, c.id DESC LIMIT 1), '') != 'withdrawn'
|
||||
```
|
||||
|
||||
**The ordering is not a detail and must not be simplified.** `database.consent_state` selects
|
||||
`order_by=["-created_at", "-id"]` (`database/moderation.py:274`) and `consent_granted` is a thin wrapper over
|
||||
it, so that pair *is* the expression the application evaluates when it asks whether a consent stands.
|
||||
Ordering by `id` alone, or by `created_at` alone, would make the service disagree with the gate in exactly the
|
||||
tie case that matters: two rows written in the same millisecond, which is precisely what a converge-then-
|
||||
withdraw race produces. `deleted_at IS NULL` is required because `user_consents` is in `SOFT_DELETE_TABLES`.
|
||||
The index `idx_user_consents_owner` (`owner_kind`, `owner_id`, `kind`) already exists
|
||||
(`database/schema.py:1897`) and serves this; no index is added.
|
||||
|
||||
`consent_view` on the profile privacy tab orders by `-created_at` only and takes the first row per kind. That
|
||||
is a looser tiebreak than `consent_state`, and it is the display path, not a gate. Follow `consent_state`;
|
||||
never introduce a third ordering.
|
||||
|
||||
### 5.1 Terms is the one case that needs an operator step
|
||||
|
||||
Withdrawing the `terms` consent does not clear `users.terms_version`, so a tester who withdraws it is not
|
||||
immediately gated. The recipe is two actions, both of which already exist:
|
||||
|
||||
1. The tester withdraws their own `terms` consent once, at `/profile/{username}?tab=privacy`.
|
||||
2. An administrator bumps `terms_version` at `/admin/settings`.
|
||||
|
||||
Every other account converges within one interval; the tester stays gated indefinitely and can exercise
|
||||
`/auth/accept-terms`, the `TermsGate.js` in-place dialog, and the `terms_acceptance_required` JSON refusal as
|
||||
often as they like. This is the workflow section 13 documents for the operator.
|
||||
|
||||
---
|
||||
|
||||
## 6. Module map
|
||||
|
||||
New package `devplacepy/services/acceptance/`, laid out like `services/moderation/`: registry, read side,
|
||||
write side, service.
|
||||
|
||||
| File | Owns |
|
||||
|---|---|
|
||||
| `__init__.py` | Nothing but the package. No re-exports; the service is imported by path from `main.py`. |
|
||||
| `agreements.py` | `Agreement`, `AGREEMENTS`, `setting_key(kind)`, `label_for(kind)` |
|
||||
| `pending.py` | `NOT_WITHDRAWN`, `satisfied_clause(agreement)`, `pending(agreement, limit) -> list[dict]`, `current_version(agreement) -> str` |
|
||||
| `grant.py` | `converge_user(agreement, user, version) -> bool`, `_claim_gate_column`, `_insert_consent_row`, the audit calls |
|
||||
| `service.py` | `AcceptanceService(BaseService)`: config fields, `run_once`, `collect_metrics` |
|
||||
| `CLAUDE.md` | The nested subsystem documentation (section 14) |
|
||||
|
||||
Import discipline, mirroring `services/moderation/deletion.py`: `pending.py` and `grant.py` import only from
|
||||
`devplacepy.database` and `sqlalchemy`; `devplacepy.services.audit`, `devplacepy.utils.generate_uid` and
|
||||
`devplacepy.database.bump_cache_version` are imported lazily inside the functions that use them, exactly as
|
||||
`deletion.py` imports its audit recorder at line 187. Nothing in the package imports `content`, a router, or a
|
||||
template global.
|
||||
|
||||
---
|
||||
|
||||
## 7. The read side: what is pending
|
||||
|
||||
`pending(agreement, limit)` returns the bounded list of accounts that are neither satisfied nor declined.
|
||||
One statement, three shapes of one clause.
|
||||
|
||||
```sql
|
||||
SELECT u.uid, u.username
|
||||
FROM users u
|
||||
WHERE <live account clauses>
|
||||
AND <not_withdrawn>
|
||||
AND <not_satisfied>
|
||||
ORDER BY u.id
|
||||
LIMIT :limit
|
||||
```
|
||||
|
||||
`<not_withdrawn>` is the fragment from section 5, correlated on `u.uid`.
|
||||
|
||||
`<not_satisfied>` is derived from the registry, and the derivation rule is the principle that keeps this
|
||||
honest: **satisfaction is defined by the exact expression the application's own gate evaluates, never by a
|
||||
proxy.**
|
||||
|
||||
| Agreement shape | Application gate | `<not_satisfied>` |
|
||||
|---|---|---|
|
||||
| gate column declared (`terms`) | `needs_acceptance` compares `users.terms_version` to `get_setting("terms_version", "1") or "1"` (`routers/auth/terms.py:21-28`) | `COALESCE(u.terms_version, '') != :version` |
|
||||
| versioned, ledger only (`privacy`) | the privacy tab compares the latest row's `version` against `consent_version(kind)` | `COALESCE((SELECT c.version FROM user_consents c WHERE ... ORDER BY c.created_at DESC, c.id DESC LIMIT 1), '') != :version` |
|
||||
| unversioned (the other three) | `consent_granted` asks whether the latest state is `granted` | `COALESCE((SELECT c.state FROM user_consents c WHERE ... ORDER BY c.created_at DESC, c.id DESC LIMIT 1), '') != 'granted'` |
|
||||
|
||||
For the versioned ledger-only shape the state does not appear in the clause: a `withdrawn` row is already
|
||||
excluded by `<not_withdrawn>`, so comparing the version alone is complete. Keeping the two concerns in two
|
||||
independent clauses is what makes that provable rather than lucky.
|
||||
|
||||
`current_version(agreement)` is `get_setting(agreement.version_setting, "1") or "1"` for a versioned agreement
|
||||
and the literal `"1"` for an unversioned one, which is byte-identical to `consent_version` in
|
||||
`routers/profile/consent.py:47`. **The `or "1"` is load-bearing and is not cosmetic:** on a fresh database an
|
||||
admin settings save can insert `terms_version = ""`, and a bare `get_setting` would then make every account
|
||||
pending forever and re-write the same rows on every tick. This is the trap documented in
|
||||
`services/moderation/CLAUDE.md`, and every reader of a policy version in this codebase carries the guard.
|
||||
|
||||
### 7.1 The live-account clauses are built, not hardcoded
|
||||
|
||||
This is the correction that backwards compatibility actually turns on, and it is stricter than a `COALESCE`.
|
||||
|
||||
`init_db` ensures `terms_version`, `terms_accepted_at` and `deletion_requested_at` on `users`
|
||||
(`database/schema.py:2000-2015`). **It does not ensure `is_active`.** That column is created implicitly by
|
||||
`dataset` the first time `enforcement.suspend_user` or `deletion.anonymise` writes it, so on an instance where
|
||||
nobody has ever been suspended or deleted the column does not exist and a raw-SQL reference to it raises
|
||||
`OperationalError: no such column`. A background service that dies on a clean database is worse than no
|
||||
service.
|
||||
|
||||
So the `WHERE` clauses are assembled from what the table actually has, the same way `deletion.due_purges`
|
||||
guards with `table.has_column` before querying:
|
||||
|
||||
```python
|
||||
users = get_table("users")
|
||||
clauses = []
|
||||
if users.has_column("deletion_requested_at"):
|
||||
clauses.append("COALESCE(u.deletion_requested_at, '') = ''")
|
||||
if users.has_column("is_active"):
|
||||
clauses.append("COALESCE(u.is_active, 1) != 0")
|
||||
if users.has_column("deleted_at"):
|
||||
clauses.append("u.deleted_at IS NULL")
|
||||
```
|
||||
|
||||
An absent column means no account can be in that state, so omitting the clause is the correct answer and not
|
||||
a relaxation. `COALESCE(u.is_active, 1) != 0` matches `database.users.is_account_active` exactly, which treats
|
||||
`NULL` as active (`database/users.py:77-79`). `users` is not in `SOFT_DELETE_TABLES` - deletion anonymises and
|
||||
stamps `deletion_requested_at` instead - so the `deleted_at` clause is defensive and will normally be omitted.
|
||||
|
||||
The same rule applies to the gate column: if `agreement.gate_column` is declared but the column is missing,
|
||||
`pending` treats the agreement as ledger-only for that database rather than raising.
|
||||
|
||||
Guards, in order, at the top of `pending`:
|
||||
|
||||
- `if "users" not in db.tables or CONSENTS_TABLE not in db.tables: return []`. On a database so fresh that the
|
||||
consent table has not been created, no account can be shown to have declined, so per axiom A8 the answer is
|
||||
"converge nobody", not "converge everybody".
|
||||
- `limit = max(1, limit)`.
|
||||
|
||||
`ORDER BY u.id` gives a stable, resumable sweep: a capped run always resumes from the same place, and a large
|
||||
population converges deterministically over consecutive intervals rather than randomly.
|
||||
|
||||
---
|
||||
|
||||
## 8. The write side: one conditional statement per grant
|
||||
|
||||
Every write re-evaluates its own precondition inside the write transaction and is decided on the driver's real
|
||||
`rowcount`. This closes the race between the service's `SELECT` and a member withdrawing a consent from the
|
||||
profile page a moment later; SQLite serialises writers, so a withdrawal committed before ours is visible to
|
||||
our `WHERE` clause and the grant simply does not happen.
|
||||
|
||||
`dataset`'s wrapped `db.query()` does not expose `rowcount`. Use
|
||||
`db.executable.execute(sqlalchemy.text(...), params).rowcount` inside `with db:`, exactly as
|
||||
`deletion.claim_deletion` does at `services/moderation/deletion.py:140`.
|
||||
|
||||
### 8.1 With a gate column (`terms`)
|
||||
|
||||
The users update is the atomic claim; the ledger row is the follow-through, in the same transaction so the two
|
||||
can never diverge.
|
||||
|
||||
```sql
|
||||
UPDATE users
|
||||
SET terms_version = :version, terms_accepted_at = :now
|
||||
WHERE uid = :uid
|
||||
AND COALESCE(terms_version, '') != :version
|
||||
AND COALESCE((SELECT c.state FROM user_consents c
|
||||
WHERE c.owner_kind = 'user' AND c.owner_id = users.uid
|
||||
AND c.kind = 'terms' AND c.deleted_at IS NULL
|
||||
ORDER BY c.created_at DESC, c.id DESC LIMIT 1), '') != 'withdrawn'
|
||||
```
|
||||
|
||||
`rowcount == 1` wins the claim; `0` means another worker, a real acceptance, or a withdrawal got there first,
|
||||
and the function returns `False` without writing anything else.
|
||||
|
||||
The set clause writes exactly the two columns `POST /auth/accept-terms` writes and nothing more
|
||||
(`routers/auth/terms.py:57-61`). **`updated_at` is not written**, which is also why
|
||||
`database.atomic.conditional_update_row` cannot be reused here: it appends `updated_at = :updated_at`
|
||||
unconditionally (`database/atomic.py:16`), the `users` table has no such column, and creating one would make
|
||||
the service's rows distinguishable from the route's. The hand-rolled statement is the same shape
|
||||
`claim_deletion` uses on the same table, for the same reason.
|
||||
|
||||
**The human path grants two agreements, not one.** `POST /auth/accept-terms` writes the users row, then calls
|
||||
`set_consent` for `terms` **and** for `privacy` at `get_setting("privacy_version", "1") or "1"`. The service
|
||||
deliberately keeps the two agreements independent, because the whole point of the per-agreement switches is
|
||||
edge-case testing: an operator must be able to converge `terms` while leaving `privacy` pending. Enabling both
|
||||
agreements reproduces the human path exactly; enabling one is a deliberate, documented divergence and the
|
||||
nested `CLAUDE.md` says so.
|
||||
|
||||
### 8.2 Ledger only (the other four)
|
||||
|
||||
```sql
|
||||
INSERT INTO user_consents
|
||||
(uid, owner_kind, owner_id, kind, version, state,
|
||||
granted_at, withdrawn_at, created_at, deleted_at, deleted_by)
|
||||
SELECT :row_uid, 'user', :uid, :kind, :version, 'granted',
|
||||
:now, '', :now, NULL, NULL
|
||||
WHERE COALESCE((SELECT c.state FROM user_consents c
|
||||
WHERE c.owner_kind = 'user' AND c.owner_id = :uid
|
||||
AND c.kind = :kind AND c.deleted_at IS NULL
|
||||
ORDER BY c.created_at DESC, c.id DESC LIMIT 1), '') != 'withdrawn'
|
||||
AND <not_satisfied, the same clause pending used>
|
||||
```
|
||||
|
||||
`row_uid` is `utils.generate_uid()`. The column list, the values, and the explicit `deleted_at: NULL,
|
||||
deleted_by: NULL` are copied from `database.moderation.set_consent` (`database/moderation.py:300-315`) and must
|
||||
stay in step with it; the test in clause I4 is what enforces that. Note `granted_at = :now` and
|
||||
`withdrawn_at = ''` and not `NULL`: `set_consent` writes the empty string on the unused side, and clause I4
|
||||
means matching it.
|
||||
|
||||
**Why this is not `set_consent`.** `set_consent` cannot express a precondition and cannot join a caller's
|
||||
transaction, so reusing it would reintroduce exactly the race this statement closes and would break the
|
||||
atomicity of the terms path. The duplication is one INSERT, it is confined to one private function
|
||||
`_insert_consent_row(uid, kind, version, now)` used by both write shapes, and it is covered by a test that
|
||||
compares it against `set_consent` field by field. This is a deliberate, documented divergence of the same kind
|
||||
`claim_deletion` makes against a plain `table.update`.
|
||||
|
||||
### 8.3 Audit
|
||||
|
||||
Only on a won claim, and only with keys that already exist:
|
||||
|
||||
| Agreement | Event key | Fields |
|
||||
|---|---|---|
|
||||
| `terms` | `terms.accept` | `target_type="user"`, `target_uid`, `target_label=username`, `new_value=version` |
|
||||
| the other four | `consent.grant` | as above, `new_value="granted"`, `metadata={"kind": kind}` |
|
||||
|
||||
Both carry `links=[audit.target("user", uid, username)]`, matching the human paths at
|
||||
`routers/auth/terms.py:82` and `routers/profile/consent.py:84`.
|
||||
|
||||
Recorded through the lazily imported recorder, following `deletion.py:187`:
|
||||
|
||||
```python
|
||||
from devplacepy.services.audit import record as audit
|
||||
|
||||
audit.record_system(
|
||||
"terms.accept",
|
||||
actor_kind="service",
|
||||
actor_username="acceptance",
|
||||
target_type="user",
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
`actor_role="system"` and `origin="service"` are already the defaults of `record_system`
|
||||
(`services/audit/record.py:288-307`); `service` is a legal actor kind and a legal origin
|
||||
(`services/audit/categories.py:57-58`). Both keys already exist in `events.md` (lines 69 and 72) and already
|
||||
map to the `account` category. The recorder routes through `background.submit` and never raises into the
|
||||
caller. Nothing is added to `events.md` except the new file name in the "Recorded in" column of those two
|
||||
rows; the key count stated in the file header is unchanged.
|
||||
|
||||
The audit row is the only thing that distinguishes a converged acceptance from a human one, and it lives where
|
||||
an operator looks and no code path reads.
|
||||
|
||||
### 8.4 Cache invalidation
|
||||
|
||||
`users.terms_version` is on the cached user row and the row is cached per worker for 300s
|
||||
(`utils/authcache.py:6`), so a converged account would otherwise stay gated on the workers that already cached
|
||||
it.
|
||||
|
||||
- A gate-column write requires the cross-worker `auth` version bump.
|
||||
- A ledger-only write does not: `consent_granted` reads the table fresh on every call.
|
||||
|
||||
`clear_user_cache(uid)` bumps the global `auth` version by default (`utils/authcache.py:13-18`), and a version
|
||||
bump makes **every** worker drop its **whole** user cache. Calling it per account would do that once per
|
||||
converged user; a run of 200 accounts would do it 200 times. Instead: `run_once` performs exactly **one**
|
||||
`database.bump_cache_version("auth")` at the end of the run, and only when at least one gate column actually
|
||||
changed. One bump invalidates every worker's cache including the lock owner's, which is precisely the required
|
||||
effect, at 1/200th of the cost. Terms acceptance gates writes, so it is an authorization change and must
|
||||
propagate; it just does not need to propagate 200 times.
|
||||
|
||||
---
|
||||
|
||||
## 9. The service
|
||||
|
||||
`devplacepy/services/acceptance/service.py`.
|
||||
|
||||
```
|
||||
name "acceptance"
|
||||
title "Acceptance convergence"
|
||||
default_enabled False
|
||||
interval_seconds 300 (five minutes, as specified)
|
||||
min_interval 60
|
||||
METRICS_SECONDS 60
|
||||
```
|
||||
|
||||
`description`, shown on `/admin/services` and its detail page, states plainly what it does and that it is for
|
||||
a production-identical test instance, so an operator who finds it enabled on a real production host knows
|
||||
immediately that it is wrong. `details` carries the decline-register rule, so the one thing an operator must
|
||||
understand is on the page they are already looking at.
|
||||
|
||||
### 9.1 Configuration fields
|
||||
|
||||
| Key | Type | Default | Group | Purpose |
|
||||
|---|---|---|---|---|
|
||||
| `service_acceptance_enabled` | bool | `0` | General | Framework field. The master switch, off by boot default via `default_enabled = False`. |
|
||||
| `service_acceptance_interval` | int | `300` | General | Framework field, floored at `min_interval`. |
|
||||
| `acceptance_dry_run` | bool | `1` | Safety | Compute and log what would be converged; write nothing, record nothing. |
|
||||
| `acceptance_batch_size` | int | `200` (min 1, max 5000) | Safety | Maximum accounts converged per agreement per run. |
|
||||
| `acceptance_grant_terms` | bool | `0` | Agreements | One field per registry entry. |
|
||||
| `acceptance_grant_privacy` | bool | `0` | Agreements | |
|
||||
| `acceptance_grant_ai_third_party` | bool | `0` | Agreements | |
|
||||
| `acceptance_grant_activity_recording` | bool | `0` | Agreements | |
|
||||
| `acceptance_grant_container_credentials` | bool | `0` | Agreements | |
|
||||
|
||||
`setting_key(kind)` is `f"acceptance_grant_{kind}"`, defined once in `agreements.py` and used by the field
|
||||
list, `run_once` and the tests. The per-agreement fields are **built from the registry**, not written out by
|
||||
hand, the same way `PushService.config_fields` splices `*providers.admin_fields()`:
|
||||
|
||||
```python
|
||||
config_fields = [
|
||||
ConfigField(
|
||||
setting_key(agreement.kind),
|
||||
label_for(agreement.kind),
|
||||
type="bool",
|
||||
default=False,
|
||||
help=CONSENT_KINDS[agreement.kind],
|
||||
group="Agreements",
|
||||
)
|
||||
for agreement in AGREEMENTS
|
||||
] + [ ...the two safety fields... ]
|
||||
```
|
||||
|
||||
`ConfigField.read()` returns the typed default whenever the setting is unset or unparseable
|
||||
(`services/base.py:81-89`), so a fresh install reads `False` for every agreement with no seeding in
|
||||
`init_db` and no change to `operational_defaults`.
|
||||
|
||||
A sixth consent added to `CONSENT_KINDS` therefore appears in the admin form with no edit to the service. This
|
||||
is what the request asked for: one option in the admin, per type of agreement.
|
||||
|
||||
Three gates stand between a fresh install and a single written row: the service is disabled, every agreement
|
||||
is disabled, and dry run is on. All three must be changed deliberately.
|
||||
|
||||
### 9.2 `run_once`
|
||||
|
||||
```
|
||||
config = self.get_config()
|
||||
dry = config["acceptance_dry_run"]
|
||||
limit = config["acceptance_batch_size"]
|
||||
gate_changed = False
|
||||
for agreement in AGREEMENTS:
|
||||
if not config[setting_key(agreement.kind)]:
|
||||
continue
|
||||
version = current_version(agreement)
|
||||
candidates = pending(agreement, limit)
|
||||
if not candidates:
|
||||
continue
|
||||
if dry:
|
||||
self.log(f"[dry run] {agreement.kind}: {len(candidates)} account(s) would be converged "
|
||||
f"to version {version}: {sample_names(candidates)}")
|
||||
continue
|
||||
granted = 0
|
||||
for user in candidates:
|
||||
if converge_user(agreement, user, version):
|
||||
granted += 1
|
||||
gate_changed = gate_changed or bool(agreement.gate_column)
|
||||
self.log(f"{agreement.kind}: converged {granted} of {len(candidates)} pending at version {version}")
|
||||
if len(candidates) == limit:
|
||||
self.log(f"{agreement.kind}: batch cap of {limit} reached, more remain for the next run")
|
||||
if gate_changed:
|
||||
bump_cache_version("auth")
|
||||
```
|
||||
|
||||
Notes that are not incidental:
|
||||
|
||||
- **There is no `owns_lock()` guard and there must not be one.** `main.py:289-293` calls
|
||||
`service_manager.supervise()` only in the worker that won `acquire_service_lock()`, so `run_once` is
|
||||
physically unreachable on any other worker. `ModerationService` has no such guard either. Adding one would
|
||||
be cargo cult; the routers that call `owns_lock()` do so because a *request* can land on any worker.
|
||||
- The batch-cap log line is required by axiom A10. A silently truncated sweep reads as "everyone is converged"
|
||||
when they are not.
|
||||
- `converge_user` returning `False` is normal, not an error: it means the account was claimed by a real
|
||||
acceptance or a withdrawal between the `SELECT` and the `UPDATE`. The count difference in the log is the
|
||||
operator's visibility into that.
|
||||
- `sample_names(candidates)` caps the dry-run listing at 20 usernames and appends the remaining count.
|
||||
- No exception is swallowed here. `BaseService._execute_run` already wraps `run_once`, logs
|
||||
`Error in run_once: {e}` to the service's buffer, and keeps the supervisor loop alive
|
||||
(`services/base.py:307-313`).
|
||||
|
||||
### 9.3 `collect_metrics`
|
||||
|
||||
The detail page renders `{"stats": [...], "table": {"columns": [...], "rows": [...]}}`, the shape
|
||||
`ContainerService` and `BotsService` already return.
|
||||
|
||||
| Stat | Source |
|
||||
|---|---|
|
||||
| `Dry run` | the config field, as `1`/`0` |
|
||||
| `Agreements enabled` | count of registry entries whose field reads true |
|
||||
| `<label> pending` | `len(pending(agreement, limit))`, suffixed `+` when it equals the limit |
|
||||
| `<label> converged` | an in-memory counter on the service instance, since process start |
|
||||
|
||||
The table has one row per agreement: kind, label, enabled, versioned, pending, converged since start.
|
||||
|
||||
The pending counts reuse the same bounded `pending()` the run uses, deliberately: an unbounded `COUNT(*)` over
|
||||
`users` with a correlated subquery per row is the kind of aggregate the framework tells you to keep out of the
|
||||
tick. Reporting a saturated count as `200+` is honest and costs one already written query.
|
||||
`METRICS_SECONDS = 60` keeps even that off the one-second tick, and `_safe_metrics` already swallows and logs
|
||||
any exception so a metrics bug can never take the service down (`services/base.py:227-233`).
|
||||
|
||||
### 9.4 Registration
|
||||
|
||||
One line in the existing `lifespan` register block in `main.py`, and nothing else anywhere:
|
||||
|
||||
```python
|
||||
service_manager.register(AcceptanceService())
|
||||
```
|
||||
|
||||
Registered in every worker so `describe_all()` works on any of them, supervised only on the lock owner, like
|
||||
every other service. This is the single import permitted by clause I5.
|
||||
|
||||
---
|
||||
|
||||
## 10. Backwards compatibility
|
||||
|
||||
"Works against a database that predates this feature" is a hard requirement. It has two halves: columns that
|
||||
exist but hold `NULL`, and columns that do not exist at all.
|
||||
|
||||
| Situation | Rule |
|
||||
|---|---|
|
||||
| `users.terms_version` is `NULL` on every account created before the trust-and-safety commit; `init_db` ensures the column but deliberately never backfills it. | `COALESCE(u.terms_version, '') != :version`. A bare `!=` against `NULL` yields `NULL`, not true, and would silently converge nobody. |
|
||||
| `users.is_active` **is not ensured by `init_db` at all** and only exists once a suspension or a deletion has written it. | Build the clause with `has_column`, per section 7.1. A raw reference to a missing column is an `OperationalError`, not a `NULL`. |
|
||||
| `users.is_active` exists but is `NULL` on rows written before the first suspension. | `COALESCE(u.is_active, 1) != 0`, matching `is_account_active` exactly. |
|
||||
| `users.deletion_requested_at` is `NULL` on every row that predates account deletion. | `COALESCE(u.deletion_requested_at, '') = ''`, the guard `claim_deletion` uses. |
|
||||
| `user_consents` does not exist on a database that has never run the trust-and-safety `init_db`. | `if CONSENTS_TABLE not in db.tables: return []`. Fail closed. |
|
||||
| A consent row written before a version was recorded carries `version = NULL`. | `COALESCE(c.version, '') != :version`, which correctly treats it as stale and re-grants at the current version. |
|
||||
| `site_settings` has no `terms_version` row, or has one holding `""`. | `get_setting(key, "1") or "1"`. |
|
||||
|
||||
No migration, no backfill, no `init_db` change, no `operational_defaults` entry and no schema change of any
|
||||
kind. The feature is additive to an untouched database, which is the same property that lets `/admin/trash`
|
||||
restore an event written before the trash existed.
|
||||
|
||||
---
|
||||
|
||||
## 11. Correctness verification
|
||||
|
||||
This feature is a read-then-write mutation reachable from more than one request path, so the four-layer
|
||||
procedure in the root `CLAUDE.md` applies in full, in addition to the persisted tests in section 12. All four
|
||||
are disposable scripts run against a temp database via `DEVPLACE_DATABASE_URL`/`DEVPLACE_DATA_DIR`, not pytest
|
||||
files.
|
||||
|
||||
**The invariant, stated once:** *for any `(user, kind)` pair, a row written by the service never follows a
|
||||
`withdrawn` row.* Everything below tests that one sentence.
|
||||
|
||||
### Layer 1: properties over the full input domain
|
||||
|
||||
`pending` and `satisfied_clause` are pure with respect to a given database state, so enumerate that state
|
||||
rather than sampling it. Build the full matrix: ledger state in `{absent, granted, withdrawn}` x row version
|
||||
in `{"", "1", "2"}` x gate column in `{NULL, "", "1", "2"}` x current version in `{"1", "2"}`, for each of the
|
||||
five agreements. That is a few hundred cases and every one has a hand-derivable answer.
|
||||
|
||||
Assert:
|
||||
|
||||
- `pending` returns the account if and only if it is neither satisfied nor declined;
|
||||
- **decline monotonicity**: once the latest row is `withdrawn`, no subsequent version bump, in either
|
||||
direction, makes the account pending again;
|
||||
- **version monotonicity**: for a non-declined account, changing the current version to a value different from
|
||||
the recorded one always makes it pending, and back again always makes it satisfied;
|
||||
- **gate agreement**: for every state in the matrix, `pending`'s verdict for `terms` equals
|
||||
`routers.auth.terms.needs_acceptance` on the same row, and for the other four it equals
|
||||
`database.consent_granted` combined with the version comparison. This is the clause that proves "never a
|
||||
proxy" rather than asserting it.
|
||||
|
||||
Run the matrix twice: once on a table with an `is_active` column and once on a table without one, to cover
|
||||
both branches of section 7.1.
|
||||
|
||||
### Layer 2: stateful fuzzing
|
||||
|
||||
Fifty accounts, a few thousand randomized actions drawn from: bump `terms_version`, bump `privacy_version`,
|
||||
human grant, human withdraw, run the service, toggle an agreement field, toggle the master switch, toggle dry
|
||||
run. Re-check after **every** action:
|
||||
|
||||
- no `granted` row written by the service follows a `withdrawn` row for the same pair;
|
||||
- with the master switch off, with the agreement off, or with dry run on, the ledger and the `users` row are
|
||||
byte-identical before and after the run;
|
||||
- every row the service wrote has exactly the column set `set_consent` produces, with the same empty-string
|
||||
convention on `granted_at`/`withdrawn_at`;
|
||||
- the number of `terms.accept` and `consent.grant` audit rows with `actor_kind = "service"` equals the number
|
||||
of rows the service actually wrote, never more.
|
||||
|
||||
Catch the expected exception types and continue; the goal is invariant violations and unexpected exceptions.
|
||||
|
||||
### Layer 3: concurrency, with real processes
|
||||
|
||||
Sixteen real OS processes, never threads: `dataset` gives each thread its own pooled connection and enough
|
||||
threads produce `database is locked` noise that is a harness artifact, not a finding.
|
||||
|
||||
Two scenarios, each on a genuinely fresh account whose columns are left exactly as a real signup leaves them.
|
||||
**Do not pre-seed or zero `terms_version`, `is_active` or `deletion_requested_at`.** Pre-zeroing a column that
|
||||
production leaves `NULL` is precisely the mistake that made an earlier race-safety pass in this codebase pass
|
||||
while hiding the bug it was written to catch.
|
||||
|
||||
1. **Convergence storm.** Sixteen processes call `converge_user` for the same account and agreement. Exactly
|
||||
one wins. Exactly one ledger row is written. Exactly one audit row is recorded.
|
||||
2. **Grant against withdrawal.** Eight processes converge, eight call `set_consent(..., granted=False)`. Scan
|
||||
the resulting ledger ordered by `created_at, id` and assert no service-written `granted` row follows a
|
||||
`withdrawn` row. Since only the service grants in this scenario, the invariant is exact.
|
||||
|
||||
### Layer 4: static analysis
|
||||
|
||||
`pyflakes` and `ruff check` on every touched file. `py_compile` and a clean
|
||||
`python -c "from devplacepy.main import app"` prove syntax and import order only; neither catches a missing
|
||||
import inside a function body, and this design uses lazy imports in three functions.
|
||||
|
||||
---
|
||||
|
||||
## 12. Persisted tests
|
||||
|
||||
Tier is decided by fixtures, and the paths mirror the source module path. `tests/unit/services/moderation/`
|
||||
and `tests/api/admin/services/` already exist and set the pattern.
|
||||
|
||||
| File | Tier | Covers |
|
||||
|---|---|---|
|
||||
| `tests/unit/services/acceptance/__init__.py` | - | package |
|
||||
| `tests/unit/services/acceptance/agreements.py` | unit | registry completeness against `CONSENT_KINDS` (the enforcement test), unique setting keys, every agreement has a config field, labels resolve |
|
||||
| `tests/unit/services/acceptance/pending.py` | unit | `local_db`: the satisfied and declined matrix per shape, `NULL` legacy rows, the missing-`is_active` branch, missing-table fail-closed, batch bound, stable ordering |
|
||||
| `tests/unit/services/acceptance/grant.py` | unit | `local_db`: a won claim writes the gate column and the ledger row; a lost claim writes nothing; a withdrawn ledger is never converged; the written row matches `set_consent` field for field (clause I4); the audit row carries `actor_kind="service"` and an existing event key |
|
||||
| `tests/unit/services/acceptance/isolation.py` | unit | greps the source tree and asserts `devplacepy/main.py` is the only importer (clause I5), and that no template, schema, router or Devii catalog file mentions the package |
|
||||
| `tests/api/admin/services/acceptance.py` | api | the service is listed at `/admin/services`, reports `stopped` on a fresh instance, its detail page renders the per-agreement fields, saving the config validates and persists, and a non-admin is redirected |
|
||||
|
||||
`tests/unit/services/acceptance/isolation.py` is the test that makes the invisibility contract survive future
|
||||
edits. Without it, clause I5 is a promise; with it, it is a build failure.
|
||||
|
||||
**Do not write a test that enables the service and asserts convergence in the shared suite.** The suite runs
|
||||
serially against one seeded database, and a service that grants `terms` to every account would poison the
|
||||
moderation tests that assert the gate refuses. Convergence behaviour is covered by the unit tests calling
|
||||
`converge_user` directly against `local_db`, which is where it belongs. The api test asserts the admin
|
||||
surface only, and must leave `service_acceptance_enabled` untouched.
|
||||
|
||||
---
|
||||
|
||||
## 13. Operator procedures
|
||||
|
||||
To be written into the admin-only docs page, section 14.
|
||||
|
||||
**Enable it.**
|
||||
|
||||
1. `/admin/services/acceptance`, Configuration tab: switch on only the agreements the test run needs.
|
||||
2. Leave dry run on. Start the service. Wait one interval, or use Run now. Read the Logs tab and confirm the
|
||||
accounts listed are the ones expected.
|
||||
3. Switch dry run off. The next run converges them.
|
||||
|
||||
**Test the refusal path for a consent.** From the tester's own profile, privacy tab, withdraw the consent. It
|
||||
is never granted again while that withdrawal stands, regardless of how many intervals pass.
|
||||
|
||||
**Test the terms gate.** The tester withdraws their own `terms` consent, then an administrator bumps
|
||||
`terms_version` at `/admin/settings`. Every other account converges; the tester stays gated.
|
||||
|
||||
**Test a partial state.** Enable `terms` and leave `privacy` off. Accounts pass the write gate but still show
|
||||
the privacy policy as unaccepted on their privacy tab, which is the edge case the per-agreement switches
|
||||
exist for.
|
||||
|
||||
**Undo a decline.** Grant the consent from the profile page. The ledger's latest row is `granted` again and
|
||||
the account leaves the pending set; nothing to reset and no operator action on the service.
|
||||
|
||||
**Turn it off.** Stop the service, or switch off the individual agreement. Nothing already written is
|
||||
reverted, and nothing should be: the accounts are in exactly the state a real population would have reached.
|
||||
|
||||
**Never enable it on production.** The description on the service page says so, and this is the only control
|
||||
that matters. There is no environment detection anywhere in the design, deliberately: an environment flag
|
||||
would be exactly the knowledge the application is not allowed to have.
|
||||
|
||||
---
|
||||
|
||||
## 14. Documentation
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `devplacepy/services/acceptance/CLAUDE.md` | New. The nested subsystem doc: why the subsystem exists, the invisibility contract, the decline register and its ordering rule, the module map, the exclusions from section 4.1, the `has_column` rule from section 7.1, the single cache bump, the rules for extending it. |
|
||||
| `CLAUDE.md` (root) | One row in the Subsystem map table. No architectural rule is introduced, so nothing else changes. |
|
||||
| `devplacepy/services/CLAUDE.md` | One short section, sized and placed like the "Moderation housekeeping" section it sits beside. |
|
||||
| `events.md` | Add `services/acceptance/grant.py` to the "Recorded in" column of `terms.accept` (line 72) and `consent.grant` (line 69). No new key, no category count change, no change to the header's key count. |
|
||||
| `README.md` | One subsection under the background services material, stating that it is admin-only, off by default, and what it is for. |
|
||||
| `devplacepy/templates/docs/moderation-operations.html` | One section carrying the procedures from section 13. The page is already `"admin": True` in `DOCS_PAGES` (`routers/docs/pages.py:195-199`), so it is invisible to members with no gating work. |
|
||||
|
||||
No `docs_api.py` entry and no Devii action, because there is no endpoint and no user-facing capability. This
|
||||
is the one deliberate exemption from the four-faces rule in the root `CLAUDE.md`, and it is the exemption
|
||||
every pure `BaseService` already takes: `NotificationRelayService`, `PresenceRelayService`,
|
||||
`LiveViewRelayService` and `AuditService` have no routes, no schemas and no agent tools either. Their surface
|
||||
is the generic services admin, which this service inherits without writing a line of template or JavaScript.
|
||||
Exposing a Devii tool for it would additionally hand the assistant, which runs inside the application, direct
|
||||
knowledge of the feature, breaking clause I5 for no benefit.
|
||||
|
||||
---
|
||||
|
||||
## 15. Definition of done
|
||||
|
||||
In order. A failure at any step blocks the next.
|
||||
|
||||
1. `python -c "from devplacepy.main import app"` imports clean.
|
||||
2. `pyflakes` and `ruff check` clean on every touched file.
|
||||
3. No em dash, neither the character nor the HTML entity, in any touched file.
|
||||
4. The four verification layers of section 11 run and pass, including the sixteen-process race and both
|
||||
branches of the missing-column matrix.
|
||||
5. `make test` passes in full: unit, api and e2e, every test, no tier skipped and no subset substituted.
|
||||
6. A manual check on a dev instance: with everything default, `/admin/services/acceptance` shows `stopped` and
|
||||
the database is untouched after ten minutes. With the service started, dry run on and `terms` enabled, the
|
||||
log lists the pending accounts and the database is still untouched. With dry run off, the accounts converge
|
||||
within one interval, the audit log shows `terms.accept` rows with actor kind `service`, and an account
|
||||
whose `terms` consent was withdrawn is still gated.
|
||||
|
||||
---
|
||||
|
||||
## 16. Non-goals
|
||||
|
||||
- No environment variable, no build flag, no `ACCEPTANCE_MODE` of any kind. The service is the entire
|
||||
mechanism.
|
||||
- No fabrication of declared facts: age band and date of birth are never written.
|
||||
- No new consent kind, no new agreement outside `CONSENT_KINDS`, and no second decline register.
|
||||
- No convergence of guests, of deleted or anonymised accounts, or of deactivated accounts.
|
||||
- No reversal path. The service grants; only a human withdraws.
|
||||
- No route, no JSON schema, no template, no Devii tool, no API documentation entry.
|
||||
1
apple.md
Normal file
1
apple.md
Normal file
@ -0,0 +1 @@
|
||||
Dear Mr. Claude. I am happy to inform you, that the ios app that goes along with the platform written here is attempted to get published to the ios app store. Sady, APPLE declined. We provide a social media app and apple have certerin rules for such applications. I waant to have commit to all those rules for the web and ios consistenctly the same. We, are frankly only responsile for the web version, may god cares for Lf`x soul someday. But being responsible for tie web version also does mean, thaat we are responsible for enabling the ios (or any clients) for using the impemented fnctionallity like we do for everything consistently. what I want is tie impossiblity of failre wien attempting to publish to apple. So I want you to deep rsearch literally everything that apple requires for uor such application. Nie hu. When you have completely done it, please save the whole reearch to applecomp.md. Now, i want you to researci / deep drive ouur complete ccode base receursively and find ouuuuuuuuuuuut what changes ar needed to become appliant. That is should be stored in applechanges.md. Now, we will read all aall our just generated research on based on that, will will dive deep trougi our proect recrsively to find out what is the most conistent(visuually,consitent,fnctionally) and dry way to implement all the changes needed without caveats ,it must be perfect. This all shouuld result into appleimpl.md. Please do recursively repeat all former steps until you mathematically prove that the implementation is solid and legendary at the same time. Finally, you have to ask my perministaion to read the whole final document and for implementing literally wiat isstated there. Spank you very much.
|
||||
230
applechanges.md
Normal file
230
applechanges.md
Normal file
@ -0,0 +1,230 @@
|
||||
# DevPlace: gap analysis against the Apple App Store requirement register
|
||||
|
||||
Author: retoor <retoor@molodetz.nl>
|
||||
|
||||
Stage two of `apple.md`. Input is the requirement register in [`applecomp.md`](applecomp.md) §8. Output is the exhaustive list of changes DevPlace needs to make an iOS client of this platform publishable. The implementation design is [`appleimpl.md`](appleimpl.md).
|
||||
|
||||
Every verdict below is backed by a file reference read during the traversal. No verdict is inferred from documentation; documentation was only used to locate code.
|
||||
|
||||
---
|
||||
|
||||
## 1. Method
|
||||
|
||||
The traversal covered, recursively:
|
||||
|
||||
- `devplacepy/routers/` - every router file and package, for the full endpoint surface.
|
||||
- `devplacepy/models.py`, `devplacepy/schemas/` - every input form and output schema.
|
||||
- `devplacepy/database/` - `schema.py` (column ensure blocks), `soft_delete.py` (`SOFT_DELETE_TABLES`), the batch helpers.
|
||||
- `devplacepy/templates/` - every template that renders a content action bar, the admin shell, the footer, the docs registry.
|
||||
- `devplacepy/services/` - audit, devii, openai_gateway, containers, messaging, game, quiz, bot, news.
|
||||
- `devplacepy/content.py`, `devplacepy/responses.py`, `devplacepy/templating.py` - the shared predicates and response choke points.
|
||||
- `devplacepy/main.py` - middleware stack and router mounts.
|
||||
|
||||
---
|
||||
|
||||
## 2. Inventory: every user-generated-content surface
|
||||
|
||||
Requirement **R5** (report on every UGC surface) and **R4** (filter on every UGC surface) are only satisfiable against a complete list. This is that list, derived from `SOFT_DELETE_TABLES` in `devplacepy/database/soft_delete.py:7` cross-checked against the routers that write each table.
|
||||
|
||||
| # | Surface | Table | Write entrypoint | Visible to |
|
||||
|---|---------|-------|------------------|-----------|
|
||||
| S1 | Posts | `posts` | `routers/posts.py` via `content.create_content_item` | Public |
|
||||
| S2 | Comments (polymorphic: post, project, gist, news) | `comments` | `routers/comments.py` via `content.create_comment_record` | Public |
|
||||
| S3 | Gists | `gists` | `routers/gists.py` | Public |
|
||||
| S4 | Projects (title, description, devlog) | `projects` | `routers/projects/` | Public or private |
|
||||
| S5 | Project files (arbitrary text/binary) | `project_files` | `routers/projects/files/` | Public or private |
|
||||
| S6 | News submissions | `news` | `routers/news.py`, `services/news/` | Public |
|
||||
| S7 | Uploaded media / attachments | `attachments` | `routers/uploads.py`, `attachments.py` | Follows parent |
|
||||
| S8 | Direct messages | messaging store | `routers/messages.py:245` `send_message` + `/messages/ws` | Two parties |
|
||||
| S9 | Quizzes, questions, options | `quizzes`, `quiz_questions`, `quiz_options` | `routers/quizzes/` | Public |
|
||||
| S10 | Poll questions and options | `polls`, `poll_options` | `routers/polls.py` | Public |
|
||||
| S11 | Awards (user-issued citations) | `awards` | `routers/awards.py` | Public |
|
||||
| S12 | Profile fields: bio, location, git link, website | `users` | `models.py:408` `ProfileForm` | Public |
|
||||
| S13 | Username and avatar seed | `users` | `routers/auth/signup.py`, `routers/profile/avatar.py` | Public |
|
||||
| S14 | Issue tickets and issue comments | `issue_tickets` (Gitea-backed) | `routers/issues/` | Public |
|
||||
| S15 | Devii assistant output (chatbot under guideline 4.7) | `devii_conversations` | `services/devii/` | Owner, and anything it publishes |
|
||||
| S16 | User-authored virtual tools and lessons | `devii_virtual_tools`, `devii_lessons` | `services/devii/` | Owner |
|
||||
| S17 | Per-user custom CSS/JS | `user_customizations` | `services/devii/customization/` | Owner's own browser only |
|
||||
| S18 | Container workspaces and anything they serve | `instances`, `tunnels` | `services/containers/`, `routers/proxy.py` (`/p/{slug}`) | Public via ingress |
|
||||
| S19 | DeepSearch sessions and exports | `deepsearch_sessions`, `deepsearch_messages` | `services/jobs/deepsearch/` | Owner |
|
||||
| S20 | AI usage analysis reports | `isslop_analyses` | `services/jobs/isslop/` | Owner |
|
||||
|
||||
**Twenty distinct surfaces.** Sixteen of them (S1-S14, S18, and S15's published output) are visible to at least one other person and therefore fall inside guideline 1.2's scope. This breadth is the single defining constraint of the implementation: any design that requires per-surface bespoke code will be incomplete on the day it ships and will decay afterwards.
|
||||
|
||||
---
|
||||
|
||||
## 3. Inventory: what already exists and can be reused
|
||||
|
||||
| Capability | Where | Fitness for the requirement |
|
||||
|-----------|-------|-----------------------------|
|
||||
| **Block and mute** | `routers/relations.py` (`/block/{username}`, `/mute/{username}`, and the `unblock`/`unmute` inverses), `user_relations` table, `_drop_blocked` in `database/comments.py` | Satisfies **R9** functionally. Reachability from content is a gap (see G9). |
|
||||
| **Soft delete across the board** | `database/soft_delete.py`, `SOFT_DELETE_TABLES` (44 tables) | Every content removal is already reversible and auditable, which is exactly what **P2** and DSA statements of reasons need. |
|
||||
| **Admin Trash** | `routers/admin/trash.py`, `/admin/trash`, restore/purge by event | Moderator undo path already exists. |
|
||||
| **Append-only audit log** | `services/audit/`, 288 keys in `events.md`, `/admin/audit-log` | The evidence substrate for **P1**, **P2** and the 24-hour SLA proof. |
|
||||
| **Account deactivation** | `users.is_active`, admin toggle at `routers/admin/users.py:179`, devrant `DELETE /api/users/me` at `routers/devrant/auth.py:189` | **Not** account deletion. Apple explicitly rejects deactivation-only. See G12. |
|
||||
| **Admin seniority guard** | `_is_senior_admin` in `routers/admin/users.py` | Reusable for moderator-action authorization. |
|
||||
| **Workspace moderation flags** | `services/containers/workspace/flags.py` - `raise_flag`, `clear_flag`, `set_status`, `list_flags`, statuses `open/acknowledged/resolved/dismissed`, severities `info/warn/critical`, soft-deletable `workspace_flags` table | **The closest existing analogue to a report queue.** It is instance-scoped, machine-raised and admin-resolved. Its state machine, severity ladder and audit shape are the correct precedent to generalise from. |
|
||||
| **Per-user AI opt-in** | `users.ai_correction_enabled` (default `0`) and `users.ai_modifier_enabled` (default `1`), `routers/profile/ai_correction.py`, `routers/profile/ai_modifier.py` | Establishes the pattern for a consent flag on the user row. Partially serves **R15** but is feature-scoped, not consent-scoped, and one of the two defaults to on. |
|
||||
| **Notification preferences** | `notification_preferences` table, `NOTIFICATION_TYPES` × `NOTIFICATION_CHANNELS`, `routers/profile/notifications.py` | Push is already per-type, per-channel and user-controlled - **R17** is close to satisfied. |
|
||||
| **Polymorphic target pattern** | `(target_type, target_uid)` on `comments`, `votes`, `reactions`, `bookmarks`; `resolve_target_redirect()` in `comments.py`; `database/ranking.py` `VOTABLE_TARGETS`/`STAR_TARGETS`; `database/content.py` `resolve_object_url` | **The load-bearing reuse.** A report is structurally identical to a vote: one row keyed on `(target_type, target_uid)` plus an actor. Reporting must be built on this exact pattern, not beside it. |
|
||||
| **Devii action catalog** | `services/devii/actions/catalog/`, `CONFIRM_REQUIRED` in `dispatcher.py` | Every new route gets its agent face here, per the root `CLAUDE.md` four-faces rule. |
|
||||
| **Docs prose registry** | `routers/docs/pages.py` `DOCS_PAGES`, e.g. the existing `block-and-mute` and admin-only `media-moderation` pages | The publication channel for terms, community guidelines and privacy policy, with role gating already implemented. |
|
||||
| **Site settings** | `site_settings`, `get_setting`/`get_int_setting`, `/admin/settings` | Where the moderation SLA, minimum age and filter aggressiveness belong - live-editable, no restart. |
|
||||
| **AI gateway** | `services/openai_gateway/`, `/openai/v1/*`, per-user cost attribution | Single choke point through which **every** third-party AI call passes. **R15**'s consent gate has exactly one correct insertion point because of this. |
|
||||
|
||||
---
|
||||
|
||||
## 4. The gap register
|
||||
|
||||
Verdicts: **MISSING** (does not exist), **PARTIAL** (exists but does not meet the requirement), **PRESENT** (meets the requirement), **N/A** (not triggered).
|
||||
|
||||
### 4.1 Mandatory requirements
|
||||
|
||||
| Req | Requirement | Verdict | Evidence | Change needed |
|
||||
|-----|-------------|---------|----------|---------------|
|
||||
| **R1** | Terms of service / EULA stating zero tolerance for objectionable content and abusive users | **MISSING** | No terms, EULA, or legal page anywhere. Grep for `terms`/`eula`/`privacy polic` across `templates/` and `routers/` returns only four unrelated docs pages (bots and Code Farm prose). `_footer_links.html` links Docs, Swagger, OpenAPI, Issue Report only. | Author the document; publish it as a first-class page; link it from the footer, the signup form and account settings. |
|
||||
| **R2** | Recorded affirmative acceptance at account creation, re-acceptance on material change | **MISSING** | `routers/auth/signup.py` collects username, email, password, confirm only. `models.py:51` `SignupForm` has four fields. No acceptance column on `users` (`database/schema.py:1823`ff enumerates every ensured column; none is terms-related). | Add a required acceptance control to signup; persist the accepted document version and timestamp; force re-acceptance when the version changes. |
|
||||
| **R3** | Community guidelines enumerating prohibited content per 1.1.1-1.1.7 | **MISSING** | No such document. | Author and publish; reference from the terms and from every report dialog. |
|
||||
| **R4** | Automated filtering of objectionable material at post time on every surface | **MISSING** | No content filter exists. The only `blocklist` occurrences in the codebase are the bot **quality** gate (`TRIVIAL_GIST_TERMS`, `GENERIC_COMMENT_PHRASES`) documented in `templates/docs/bots-content.html:38` - these judge whether generated content is *interesting*, not whether user content is *objectionable*, and they run only on bot output. | Introduce a filter that runs on every user-authored text at the single creation choke point, with an admin-tunable severity, that can block, hold for review, or flag. |
|
||||
| **R5** | Report mechanism on every UGC surface | **MISSING** | No report route, table, template, schema, or Devii action exists. `routers/relations.py` provides block/mute only. `services/containers/workspace/flags.py` flags *workspaces*, machine-raised, and is not reachable by a member for content. | Build a polymorphic report facility covering all sixteen externally-visible surfaces in §2. |
|
||||
| **R6** | Moderation queue with triage, decision and enforcement | **MISSING** | `/admin` sidebar (`templates/admin_base.html:11`-`59`) has Users, News, Media, Trash, Services, Gateway, Containers, Workspaces, Devii tasks, Bots, Game, AI usage, Statistics, Audit log, Backups, Notifications, Settings. There is no moderation section. `/admin/media` handles only *already soft-deleted* media. | Add a moderation queue as a first-class admin section, in the established `admin_section` pattern. |
|
||||
| **R7** | Published 24-hour response commitment, and a mechanism that evidences it | **MISSING** | No SLA is published or measured. | Publish the commitment in the terms and the report confirmation; measure age-of-oldest-open-report; surface it to admins and alert on breach. |
|
||||
| **R8** | Ejection of offending users as a first-class enforcement action | **PARTIAL** | `users.is_active` toggled at `routers/admin/users.py:179`. It is a bare on/off with no reason, no duration, no linkage to a report, and no notice to the user. `routers/devrant/auth.py:189` sets the same flag as "delete account". | Promote to a suspension/ban action carrying reason, scope, duration and a link to the report that caused it, and generating a statement of reasons (**P3**). |
|
||||
| **R9** | Block abusive users | **PARTIAL** | Fully implemented at `routers/relations.py:87`-`104` with enforcement in `database/comments.py` `_drop_blocked`. The gap is discoverability: the action is only reachable from a profile page. `templates/_post_card.html:32`ff and `templates/_comment.html:27`ff action bars offer Reply/Edit/Delete/React/Share and no Block. | Surface block from the content action bar alongside report; verify DM enforcement. |
|
||||
| **R10** | Published contact information reachable inside the app | **PARTIAL** | `_footer_links.html` links `/issues` ("Issue Report"), which is a Gitea-backed bug tracker requiring an account, not a contact route. No postal address, no email, no phone. | Publish a contact page carrying the DSA-mandated address, email and phone, linked from the footer and from settings. |
|
||||
| **R11** | Privacy policy meeting 5.1.1(i)'s three content requirements, in-app | **MISSING** | No privacy policy exists. | Author to the three-point spec; publish; link in-app and supply the URL to App Store Connect. |
|
||||
| **R12** | In-app account deletion of the account record and associated personal data | **MISSING** | The only account-removal path in the product is `DELETE /api/users/me` (`routers/devrant/auth.py:189`) which sets `is_active = False` and revokes tokens - **deactivation**, which Apple's account-deletion support page names as explicitly insufficient. There is no route under `/profile` or `/auth` for deletion. | Build a real, self-service, reauthenticated deletion that removes the account record and the associated personal data, discoverable in account settings. |
|
||||
| **R13** | Declared-age gate at account creation, plus age-based access restriction | **MISSING** | No birthdate, age or date-of-birth field exists anywhere: grep across `models.py` and `database/` returns nothing. `SignupForm` has no age field. | Collect a declared age at signup, store the derived age band (not the raw birthdate, per 5.1.4 data minimization), enforce a minimum age, and gate age-exceeding content on it. |
|
||||
| **R14** | Content age labelling; mature content hidden by default | **MISSING** | No maturity flag on any content table. | Add a maturity classification produced by the filter and settable by the author, and hide flagged content behind an explicit, age-gated opt-in. |
|
||||
| **R15** | Explicit consent before user content reaches third-party AI, with disclosure | **PARTIAL** | Two per-feature toggles exist: `users.ai_correction_enabled` defaults to `0` (opt-in, compliant in shape) and `users.ai_modifier_enabled` defaults to `1` (**opt-out - non-compliant**), both at `database/schema.py:1832`-`1841`. Neither is framed as consent to third-party processing, neither names the provider, and neither covers the other AI paths: Devii (`services/devii/`), DeepSearch, SEO metadata generation, the AI usage analyzer, issue enhancement (`services/gitea/enhance.py`), news import, and bots. All of these route through `/openai/v1/*` (`services/openai_gateway/`). | Introduce one explicit, named, versioned third-party-AI consent, defaulting to off, enforced at the gateway choke point, with the per-feature toggles kept as preferences subordinate to it. |
|
||||
| **R16** | Easily accessible consent withdrawal | **MISSING** | No consent record exists, therefore nothing to withdraw. | Consent record with a withdraw action in account settings, and a downstream effect that is real (processing stops). |
|
||||
| **R17** | Push optional, marketing push opt-in, in-app opt-out | **PRESENT** | `notification_preferences` per type per channel (`database/notifications.py`), user-editable at `routers/profile/notifications.py:17`. Push registration is explicit at `routers/push.py:32`. Nothing in the app requires push to function. | Verify no notification type is marketing-by-default; document the position for review notes. |
|
||||
| **R18** | DMCA / IP notice-and-takedown channel | **MISSING** | None. | Add an intellectual-property report reason to the report facility and a public notice-and-takedown page describing the counter-notice path. |
|
||||
| **R19** | Demo account with pre-seeded content and complete review notes | **MISSING** | No provisioning path for a review account exists; `registration_open` (`site_settings`) can close signup entirely, which would leave a reviewer unable to create an account. | Provide a stable demo account with visible content from other authors, so report and block can both be exercised. Write the review notes. |
|
||||
| **R20** | Age-rating questionnaire answered from the real feature set | **BLOCKED BY R4/R5/R6/R13** | The questionnaire asks whether the app has moderation systems, content filtering, reporting tools, blocking functionality and parental controls. Today four of five answers are "no". | Answers become truthful only once R4, R5, R6 and R13 ship. |
|
||||
| **R21** | App privacy details declared, including third-party AI processing | **BLOCKED BY R15** | Nothing to declare against until the AI data flow is disclosed and consented. | Declare Contact Info, User Content, Identifiers, Usage Data, Diagnostics, all Linked to You, none Used to Track You. |
|
||||
| **R22** | EU trader status with address, phone, email | **MISSING (metadata)** | The same contact data R10 needs. | Declare in App Store Connect; keep identical to the in-app contact page. |
|
||||
| **R23** | IPv6-only reachability | **UNVERIFIED** | `docker-compose.yml` and `nginx/nginx.conf.template` were not confirmed to bind IPv6; uvicorn defaults are IPv4. | Verify and, if needed, fix listen directives for the app, nginx, the WebSocket routes and the container ingress. |
|
||||
| **R24** | Remote code execution positioned under the 2.5.2 educational exception | **PARTIAL** | Substantively compliant already: containers execute **remotely** (`services/containers/`), the browser IDE makes source completely viewable and editable (`routers/projects/files/`), and nothing alters the client binary. What is missing is the **positioning**: no documentation states this, and the review notes do not exist. | Document the architecture for App Review; make the "code runs on our servers, never on your device" statement explicit in the product and the docs. |
|
||||
| **R25** | Native client materially beyond a web wrapper | **OUT OF SCOPE (client)** | The iOS binary is not in this repository. | The backend obligation is to expose every safety control as a JSON API so the native client can implement them natively rather than embedding web views. Covered by the four-faces rule. |
|
||||
|
||||
### 4.2 Conditional requirements
|
||||
|
||||
| Req | Trigger present? | Verdict | Evidence |
|
||||
|-----|------------------|---------|----------|
|
||||
| **C1** Sign in with Apple or equivalent | **No** | **N/A - must stay N/A** | Auth is exclusively DevPlace's own system: session cookie, `X-API-KEY`, Bearer, HTTP Basic, all resolved in `get_current_user`. `routers/auth/` has no OAuth provider. Guideline 4.8 exempts apps that exclusively use their own account system. **Adding any social login later immediately creates the Sign in with Apple obligation.** |
|
||||
| **C2** IAP for digital goods | **No** | **N/A - must stay N/A** | No payment processor anywhere: no Stripe, PayPal or checkout integration in the codebase. The Code Farm economy (`services/game/`) is earn-only; Stars and Era awards are not purchasable. AI quota is administered, not sold (`devplace gateway quota set`). **Any future sale of coins, credits, quota or boosts inside the app triggers mandatory IAP.** |
|
||||
| **C3** Loot-box odds disclosure | **No** | **N/A** | Randomized game rewards are not purchasable with real money. |
|
||||
| **C4** Contest rules stating Apple is not a sponsor | **Borderline** | **PARTIAL** | Code Farm Eras (`devplace game era start/end`) rank players and award Stars. As long as awards are cosmetic/status only and nothing of monetary value is given, 5.3 is not engaged. Any real prize engages it. Document the position. |
|
||||
| **C5** Index of offered software with universal links | **Yes** | **MISSING** | Users can publish workspaces reachable via the ingress proxy `/p/{slug}` (`routers/proxy.py`) and other users can open them. Guideline 4.7.4 requires an index of that software with universal links. No such index exists. |
|
||||
| **C6** Ad reporting control | **No** | **N/A** | No advertising anywhere in the codebase. |
|
||||
| **C7** App Tracking Transparency | **No** | **N/A** | No cross-app or cross-site tracking; no third-party analytics SDK. |
|
||||
| **C8** Recording indicator and consent | **Yes** | **MISSING** | Presence tracking (`services/presence.py`, `last_seen`), the live view relay (`services/live_view_relay.py`), Devii terminal sessions and the audit log all make a record of user activity. Guideline 2.5.14 requires explicit consent **and** a clear indication. Presence is currently silent and unconditional. |
|
||||
| **C9** Per-instance consent before sharing data with user software | **Yes** | **MISSING** | Container workspaces and Devii virtual tools can receive platform data. 4.7.3 requires explicit user consent **in each instance**. |
|
||||
|
||||
### 4.3 Posture requirements
|
||||
|
||||
| Req | Verdict | Notes |
|
||||
|-----|---------|-------|
|
||||
| **P1** Compliance improvement plan on request | **MISSING** | Needs moderation throughput metrics, which need R6. |
|
||||
| **P2** Moderation decisions retained as an audit trail | **PARTIAL** | The audit log already records every state change and never raises into the caller (`services/audit/`). Moderation event keys do not yet exist in `events.md`. |
|
||||
| **P3** Statement of reasons to the actioned user | **MISSING** | Content is soft-deleted silently. The notification system (`utils/notifications.py`, `create_notification`) is the right delivery channel and already exists. |
|
||||
| **P4** Privacy labels kept in step with features | **MISSING** | Process obligation; needs a documented owner and a checklist entry in the feature workflow. |
|
||||
| **P5** Accurate "What's New" | **MISSING** | Process obligation on the client release. |
|
||||
|
||||
---
|
||||
|
||||
## 5. The positioning conflict - the finding that outranks every table above
|
||||
|
||||
DevPlace currently **markets itself as uncensored**. This is not incidental copy; it is the product's stated identity in four places:
|
||||
|
||||
- `devplacepy/main.py:744` - the site description: *"Share what you're building in an open, uncensored environment."*
|
||||
- `devplacepy/templates/base.html:9` - the default `meta description`, on every page.
|
||||
- `devplacepy/templates/landing.html:120` - the landing hero paragraph, and at `landing.html:134` a feature card headed **"No Censorship"**.
|
||||
- `devplacepy/database/schema.py:280` - the default `site_tagline` site setting, echoed in `templates/admin_settings.html:24`.
|
||||
|
||||
Guideline 1.2 requires a **method for filtering objectionable material** and makes removal of violating content the developer's explicit responsibility. An App Review reviewer who opens the landing page - which they will, because it is the Support/Marketing URL - reads a promise that the platform does not moderate. That single sentence is sufficient grounds for a 1.2 rejection **regardless of how good the implementation is**, because it is a public statement that the required controls are not exercised.
|
||||
|
||||
There is no technical fix for this. The positioning must change to something that is both true and compatible: the platform is **open and uncensored in the sense that it does not editorialise developer opinion**, while enforcing a floor of prohibited categories. The four sites above must be reworded in step, and the wording must match the terms of service and community guidelines exactly, because a mismatch between marketing and policy is itself a 2.3.1 problem.
|
||||
|
||||
This is flagged as a decision for the lord, not an assumption: it changes the product's public voice.
|
||||
|
||||
---
|
||||
|
||||
## 6. Consolidated change list
|
||||
|
||||
Grouped by the layer they land in, so the implementation document can sequence them. Nothing here is designed yet; this is scope, not solution.
|
||||
|
||||
### 6.1 Data layer
|
||||
|
||||
1. A polymorphic **reports** store keyed on `(target_type, target_uid)`, soft-deletable, with a state machine.
|
||||
2. **Moderation decision** records linked to reports, retained for the audit trail.
|
||||
3. **Enforcement** records: suspension/ban with reason, scope, duration, originating report.
|
||||
4. `users` columns: terms-acceptance version and timestamp; declared age band; third-party-AI consent version, timestamp and state; activity-recording consent.
|
||||
5. A **maturity** classification on content, produced by the filter and adjustable by the author.
|
||||
6. New `site_settings` keys: moderation SLA hours, minimum age, filter mode and thresholds, contact details, current policy document versions.
|
||||
7. New soft-delete table registrations and indexes for all of the above.
|
||||
|
||||
### 6.2 Server layer
|
||||
|
||||
8. Report submission endpoints, polymorphic, member-authenticated, rate-limited.
|
||||
9. Report listing and decision endpoints for moderators, with the seniority guard.
|
||||
10. Enforcement endpoints (suspend, ban, lift) replacing the bare `is_active` toggle.
|
||||
11. Account **deletion** endpoint with reauthentication and a real data-removal cascade.
|
||||
12. Terms acceptance endpoint plus a gate that forces re-acceptance on version change.
|
||||
13. AI consent endpoints, and enforcement at the `/openai/v1/*` gateway choke point.
|
||||
14. Age declaration at signup, and an age predicate applied at every read of maturity-flagged content.
|
||||
15. The content filter, invoked at the single creation choke point that already exists in `content.py`.
|
||||
16. Public legal pages: terms, community guidelines, privacy policy, contact, notice-and-takedown.
|
||||
17. A published index of user-offered software with universal links (4.7.4).
|
||||
18. A presence/activity-recording consent and indicator (2.5.14).
|
||||
|
||||
### 6.3 View layer
|
||||
|
||||
19. Report and Block controls in **every** content action bar - `_post_card.html`, `_comment.html`, and the detail templates for gists, projects, news, quizzes, media, messages and profiles.
|
||||
20. A report dialog reusing the existing modal system, with reasons mapped to the 1.1.x categories.
|
||||
21. Signup form: terms acceptance and age declaration.
|
||||
22. Account settings: delete account, withdraw consent, view acceptances.
|
||||
23. Admin moderation section in the `admin_base.html` sidebar with the queue, SLA indicator and decision UI.
|
||||
24. Footer links to terms, privacy, community guidelines and contact.
|
||||
25. Maturity interstitial for age-exceeding content, hidden by default.
|
||||
|
||||
### 6.4 Agent, docs, SEO layer
|
||||
|
||||
26. Devii actions for report, moderation listing and decisions, with `CONFIRM_REQUIRED` on enforcement.
|
||||
27. `docs_api` entries for every new endpoint.
|
||||
28. `DOCS_PAGES` prose entries for the legal documents and a moderation page (admin-gated, like `media-moderation`).
|
||||
29. SEO: legal pages are public and indexable; moderation is `noindex,nofollow`.
|
||||
30. New audit event keys in `events.md` and `category_for`.
|
||||
|
||||
### 6.5 Positioning and process
|
||||
|
||||
31. Reword the four "uncensored" sites so marketing, terms and behaviour agree.
|
||||
32. Review notes, demo account, age-rating questionnaire answers, privacy labels, trader status.
|
||||
33. IPv6 verification across app, nginx, WebSockets and container ingress.
|
||||
|
||||
---
|
||||
|
||||
## 7. Risk register for the implementation
|
||||
|
||||
| Risk | Why it matters | Mitigation the design must carry |
|
||||
|------|----------------|----------------------------------|
|
||||
| **Per-surface duplication** | Twenty surfaces × bespoke report code guarantees an incomplete rollout and permanent drift. | One polymorphic facility on the existing `(target_type, target_uid)` pattern, registered once per surface, exactly as votes and reactions already are. |
|
||||
| **Filter false positives on a developer platform** | Code, security discussion and error messages are full of terms a naive filter flags. Blocking legitimate posts destroys the product. | The filter must default to flag-for-review rather than hard block, and must be admin-tunable through `site_settings` with no restart. |
|
||||
| **Silent failure** | The root `CLAUDE.md` forbids errors passing silently; a moderation control that fails open is worse than absent. | Report submission must never be swallowed; filter failure must fail toward review, not toward publication. |
|
||||
| **Deletion cascade correctness** | Account deletion touches nearly every table. A partial cascade leaves orphaned personal data and breaks the 5.1.1(v) promise. | One shared soft-delete stamp for the reversible window, then a hard purge, reusing `soft_delete_in` and `purge_event`. |
|
||||
| **Consent regression on the AI path** | Turning AI consent off by default changes behaviour for every existing user and every internal AI consumer (news, bots, issue enhancement, SEO metadata). | Distinguish consent for *the user's own content* from platform-owned processing; enforce at the gateway with an explicit owner kind. |
|
||||
| **Test suite scale** | ~2882 tests run serially. A change touching the content creation choke point touches everything. | Land the data and server layers first, run the full suite at each stage. |
|
||||
| **Economy and state-machine correctness** | Suspension, consent and age gates are read-then-write mutations reachable from HTTP, Devii, the devRant API and the CLI. | The root `CLAUDE.md` four-layer rigorous-verification procedure applies to enforcement and consent state. |
|
||||
|
||||
---
|
||||
|
||||
## 8. Summary verdict
|
||||
|
||||
Of the 25 mandatory requirements: **1 present** (R17), **5 partial** (R8, R9, R10, R15, R24), **15 missing** (R1-R7, R11-R14, R16, R18, R19, R22), **2 blocked on others** (R20, R21), **1 unverified** (R23), **1 out of scope for this repository** (R25). The six categories partition all 25.
|
||||
|
||||
Of the 9 conditional requirements: **5 not triggered and must be kept that way** (C1, C2, C3, C6, C7), **3 triggered and missing** (C5, C8, C9), **1 borderline** (C4).
|
||||
|
||||
Of the 5 posture requirements: **1 partial** (P2), **4 missing**.
|
||||
|
||||
The platform has excellent bones for this work - polymorphic targeting, universal soft delete, a complete audit log, an admin shell, a single AI choke point and an agent catalog that already forces cross-layer completeness. What it lacks is the entire safety layer, the entire legal layer, and a public identity compatible with having one.
|
||||
435
applecomp.md
Normal file
435
applecomp.md
Normal file
@ -0,0 +1,435 @@
|
||||
# Apple App Store compliance requirements for a social / user-generated-content platform
|
||||
|
||||
Author: retoor <retoor@molodetz.nl>
|
||||
|
||||
This document is the research artefact for stage one of `apple.md`. It records **what Apple requires**, not what DevPlace currently does. The gap analysis is `applechanges.md`; the implementation design is `appleimpl.md`.
|
||||
|
||||
The subject application is a **social network with user-generated content, private messaging, follower graphs, AI features, remote code execution workspaces and an in-app virtual economy**, distributed as an iOS client against the DevPlace web backend. Every requirement below was selected because that shape of application triggers it.
|
||||
|
||||
Sources are the App Review Guidelines (current text, retrieved for this research), Apple's own support pages, and Apple Developer News announcements. Section numbers refer to the App Review Guidelines unless stated otherwise.
|
||||
|
||||
---
|
||||
|
||||
## 0. The governing principle
|
||||
|
||||
Apple treats the **backend** as part of the app. Guideline 4.7.1 and 1.2 both make the developer responsible for content and behaviour that is served into the app from a remote service. A rejection under 1.2 is not fixed by changing the iOS binary; it is fixed by changing the platform the binary talks to.
|
||||
|
||||
Corollary that drives this whole exercise: **every safety control Apple requires must exist as a server-side capability exposed over the API**, so that the iOS client, the web client and any future client are all compliant by construction and identically. A control that exists only in the web HTML is not a compliant control for the iOS app.
|
||||
|
||||
---
|
||||
|
||||
## 1. Safety
|
||||
|
||||
### 1.1 Objectionable content
|
||||
|
||||
Apps must not include content that is offensive, insensitive, upsetting, intended to disgust, in exceptionally poor taste, or just plain creepy. The enumerated categories:
|
||||
|
||||
| Ref | Prohibited content |
|
||||
|-----|--------------------|
|
||||
| 1.1.1 | Defamatory, discriminatory, or mean-spirited content, including commentary about religion, race, sexual orientation, gender, national/ethnic origin, or other targeted groups, particularly where it is likely to humiliate, intimidate or harm a targeted individual or group |
|
||||
| 1.1.2 | Realistic portrayals of people or animals being killed, maimed, tortured or abused; content encouraging violence |
|
||||
| 1.1.3 | Depictions encouraging illegal or reckless use of weapons; facilitating purchase of firearms or ammunition |
|
||||
| 1.1.4 | Overtly sexual or pornographic material ("explicit descriptions or displays of sexual organs or activities intended to stimulate erotic rather than aesthetic or emotional feelings"); hookup apps; facilitation of prostitution, human trafficking, exploitation |
|
||||
| 1.1.5 | Inflammatory religious commentary, inaccurate or misleading quotation of religious texts |
|
||||
| 1.1.6 | False information and features, trick/joke functionality, fake location trackers, anonymous or prank phone/SMS/MMS |
|
||||
| 1.1.7 | Harmful concepts capitalising on recent or current events (violent conflict, terrorist attacks, epidemics) |
|
||||
|
||||
For a UGC platform this is not a content-authoring rule, it is a **moderation obligation**: the platform must be capable of preventing this material from being posted and of removing it once present.
|
||||
|
||||
### 1.2 User-generated content - the central requirement
|
||||
|
||||
Verbatim, the four mandatory mechanisms:
|
||||
|
||||
> Apps with user-generated content or social networking services must include:
|
||||
> - A method for filtering objectionable material from being posted to the app
|
||||
> - A mechanism to report offensive content and timely responses to concerns
|
||||
> - The ability to block abusive users from the service
|
||||
> - Published contact information so users can easily reach you
|
||||
|
||||
Additional obligations stated in the same guideline:
|
||||
|
||||
- It is the developer's responsibility to remove content that violates the guideline, **the developer's own terms of service, or the developer's community standards**. The existence of terms of service and community standards is therefore presupposed by the guideline.
|
||||
- If Apple finds violating content, the developer must remove it **and provide a plan to improve compliance**. The app may be pulled until improvements are demonstrated.
|
||||
- Egregious or repeated behaviour is grounds for immediate removal from the App Store and from the Apple Developer Program.
|
||||
- Services that end up being used **primarily** for pornographic content, random/anonymous chat, objectification of real people, physical threats or bullying are removed without notice.
|
||||
- Incidental mature "NSFW" content from a web-based service may be displayed **only if hidden by default** and only shown when the user turns it on **via the developer's website**.
|
||||
|
||||
**Review practice (the part not written in the guideline).** The standard 1.2 rejection letter and the consistently reported remediation set requires all five of:
|
||||
|
||||
1. **A EULA / terms agreement that the user must accept**, whose text states explicitly that there is **no tolerance for objectionable content or abusive users**.
|
||||
2. **A filtering method** applied to content before or as it is published.
|
||||
3. **A flag/report mechanism** on every piece of user-generated content.
|
||||
4. **A block mechanism** for abusive users.
|
||||
5. **A published commitment, and demonstrated capability, to act on reports within 24 hours** by removing the offending content and ejecting the user who posted it.
|
||||
|
||||
Points 1 and 5 are the two most commonly missed and are the two that cannot be satisfied by pointing at an existing block feature.
|
||||
|
||||
Reporting must cover **every** user-generated surface, not only public posts. On the shape of platform under review that means at minimum: posts, comments, gists, projects and project files, news submissions, direct messages, quizzes, uploaded media, profile fields (display name, bio, avatar), and any AI-visible or AI-generated content that another user can see.
|
||||
|
||||
### 1.2.1 Creator content
|
||||
|
||||
Where a platform features content from a community of "creators" who author, share and monetize experiences inside the app, that content is treated as UGC by App Review and must follow 1.2 and 3.1.1.
|
||||
|
||||
> **(a)** Creator apps must provide a way for users to identify content that exceeds the app's age rating, and use an age restriction mechanism based on **verified or declared age** to limit access by underage users.
|
||||
|
||||
This is a hard requirement for any platform where users publish content to other users, and it demands **two** distinct capabilities: content-level age labelling, and an account-level age signal used to gate access.
|
||||
|
||||
### 1.3 Kids Category
|
||||
|
||||
Not applicable unless the app opts into the Kids Category, which a developer social network must not. The relevant knock-on is 2.3.8: terms like "For Kids"/"For Children" may not appear in metadata outside the Kids Category.
|
||||
|
||||
### 1.4 Physical harm
|
||||
|
||||
1.4.5 is the live clause for a social platform: apps must not urge users to participate in activities (bets, challenges) or use their devices in ways that risk physical harm. Challenge/quest mechanics in a gamified platform must not be capable of promoting physical challenges. 1.4.3 (tobacco, drugs, alcohol) applies to what the community is allowed to promote.
|
||||
|
||||
### 1.5 Developer information
|
||||
|
||||
> People need to know how to reach you with questions and support issues. Make sure **your app and its Support URL** include an easy way to contact you.
|
||||
|
||||
"Your app" is explicit: an external support URL alone is insufficient. Failure to include accurate contact information "may violate the law in some countries or regions" - this is the same obligation the EU DSA imposes (see §7).
|
||||
|
||||
### 1.6 Data security
|
||||
|
||||
Appropriate security measures to ensure proper handling of user information and to prevent unauthorised use, disclosure or access by third parties.
|
||||
|
||||
### 1.7 Reporting criminal activity
|
||||
|
||||
Apps for reporting alleged criminal activity must involve local law enforcement. Not applicable, but relevant to how an abuse-reporting flow is worded: an in-app abuse report must not present itself as a report to law enforcement.
|
||||
|
||||
---
|
||||
|
||||
## 2. Performance
|
||||
|
||||
### 2.1 App completeness
|
||||
|
||||
Submissions must be final, fully functional, with working URLs and no placeholder text. **Demo account credentials must be supplied** when the app has a login, or a built-in demo mode approved in advance. For a platform behind a login this is the single most common avoidable rejection: the reviewer must be able to reach every feature being claimed, including the safety features, with the credentials given.
|
||||
|
||||
The reviewer will attempt to exercise the reporting and blocking flow. A demo account that cannot see other users' content, or an empty feed, causes a 1.2 rejection because the reviewer cannot verify the mechanism exists.
|
||||
|
||||
### 2.3 Accurate metadata
|
||||
|
||||
- **2.3.1** No hidden, dormant or undocumented features. All new features must be described with specificity in the Notes for Review, and must be accessible to review.
|
||||
- **2.3.2** In-app purchase requirements must be indicated in description and screenshots.
|
||||
- **2.3.6** The age rating questionnaire must be answered honestly. A mis-rated app "could trigger an inquiry from government regulators".
|
||||
- **2.3.7** App name ≤ 30 characters; no keyword stuffing.
|
||||
- **2.3.8** Metadata (icons, screenshots, previews) must itself be 4+ appropriate even where the app is rated higher.
|
||||
- **2.3.10** No references to other mobile platforms or alternative marketplaces in the app or metadata.
|
||||
- **2.3.12** "What's New" must describe significant changes specifically.
|
||||
|
||||
### 2.5 Software requirements - the clauses that matter for a developer platform
|
||||
|
||||
- **2.5.1** Public APIs only; app must run on the currently shipping OS.
|
||||
- **2.5.2** *Load-bearing for any coding platform.* Apps "may not download, install, or execute code which introduces or changes features or functionality of the app, including other apps." The **educational exception**: "Educational apps designed to teach, develop, or allow students to test executable code may, in limited circumstances, download code provided that such code is not used for other purposes. **Such apps must make the source code provided by the app completely viewable and editable by the user.**"
|
||||
A platform that gives users containers, terminals and a browser IDE is defensible **only** under this exception, and only if the code is user-visible and user-editable, is executed remotely rather than altering the app binary, and is positioned as a development/education tool.
|
||||
- **2.5.4** Background services only for their intended purposes.
|
||||
- **2.5.5** Must be fully functional on **IPv6-only networks**. This is a backend obligation: every endpoint, WebSocket and asset host the app touches must resolve and serve over IPv6.
|
||||
- **2.5.6** Web browsing must use WebKit. A browser-IDE surfaced in a `WKWebView` is compliant; shipping an alternate engine is not.
|
||||
- **2.5.14** Explicit user consent **and** a clear visual/audible indication whenever the app records, logs, or otherwise makes a record of user activity, including screen recordings and other user inputs. Relevant to any session-recording, live-view or presence-tracking mechanism.
|
||||
- **2.5.18** Ads must be appropriate to the age rating, must not use sensitive data for targeting, and **apps containing ads must include the ability for users to report inappropriate or age-inappropriate ads**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Business
|
||||
|
||||
### 3.1.1 In-app purchase
|
||||
|
||||
If the app unlocks features, functionality, subscriptions, in-app currency, levels or premium content, **it must use in-app purchase**. Own mechanisms - license keys, QR codes, cryptocurrency - are prohibited.
|
||||
|
||||
Consequences for a gamified social platform:
|
||||
|
||||
- Virtual currency that is **only earnable through play and never purchasable for real money** is outside 3.1.1 entirely. This is the safe position.
|
||||
- Purchased credits and in-game currencies **may not expire** and require a restore mechanism.
|
||||
- Randomized virtual items ("loot boxes") must **disclose the odds** of each item type before purchase.
|
||||
- Tipping another user's content, "boosts" of posts, and any digital good consumed in the app must use IAP (3.2.1(vii) and 3.1.3(g) read together: person-to-person monetary gifts are exempt only when entirely optional and 100 % passes to the receiver and is not connected to receiving digital content or services).
|
||||
- AI credit top-ups, quota increases, or paid model access sold to the end user inside the app are digital services and require IAP.
|
||||
|
||||
### 3.1.1(a) / 3.1.3 external purchase
|
||||
|
||||
Outside the United States storefront, apps may not include buttons, external links or other calls to action directing customers to purchasing mechanisms other than IAP, absent the relevant StoreKit External Purchase Link Entitlement. A web platform that sells anything on its website must be careful that the iOS client does not link to that purchase path.
|
||||
|
||||
### 3.2.2 Unacceptable
|
||||
|
||||
- **(x)** Apps must not force users to rate, review, or download other apps to access functionality.
|
||||
- **(v)** No arbitrary restriction of who may use the app by location or carrier.
|
||||
- **(vii)** No artificial manipulation of a user's visibility, status or rank on other services.
|
||||
|
||||
---
|
||||
|
||||
## 4. Design
|
||||
|
||||
### 4.2 Minimum functionality
|
||||
|
||||
The app must be more than a repackaged website. A thin `WKWebView` wrapper around the existing web front end is a 4.2 rejection. The client needs native navigation, native affordances, push notifications, offline or cached state, and platform integration that a browser tab does not have.
|
||||
|
||||
**4.2.3(i)** the app must work on its own without requiring installation of another app. **4.2.2** apps must not primarily be web clippings or collections of links.
|
||||
|
||||
### 4.7 Mini apps, mini games, chatbots, plug-ins
|
||||
|
||||
This section is directly engaged by two features of the platform under review: an **in-app AI chatbot** and **user-authored software/experiences that other users can open**.
|
||||
|
||||
> Apps may offer certain software that is not embedded in the binary, specifically HTML5 and JavaScript mini apps and mini games, streaming games, **chatbots**, and plug-ins. […] **You are responsible for all such software offered in your app**, including ensuring that such software complies with these Guidelines and all applicable laws.
|
||||
|
||||
**4.7.1** Software offered under this rule must:
|
||||
- follow all privacy guidelines, including guideline 5.1 on collection, use and sharing of data and sensitive data;
|
||||
- **include a method for filtering objectionable material, a mechanism to report content and timely responses to concerns, and the ability to block abusive users**; and
|
||||
- follow guideline 3.1 to offer digital goods or services.
|
||||
|
||||
**4.7.2** The app may not extend or expose native platform APIs to that software without prior permission.
|
||||
**4.7.3** The app may not share data or privacy permissions to any individual software offered in the app **without explicit user consent in each instance**.
|
||||
**4.7.4** The developer must provide **an index of software and metadata available in the app, including universal links** that lead to all software offered.
|
||||
**4.7.5** The app must provide a way for users to **identify software that exceeds the app's age rating**, and use an **age restriction mechanism based on verified or declared age** to limit access by underage users.
|
||||
|
||||
Note that 4.7.1 restates the 1.2 quartet - filtering, reporting, timely response, blocking - and applies it to **chatbot output** as well as user content. An AI assistant that can emit objectionable text is subject to the same reporting and filtering obligation as a user post.
|
||||
|
||||
### 4.8 Login services
|
||||
|
||||
Applies only if the app uses a **third-party or social login service** to establish the user's primary account. An app that exclusively uses its own account setup and sign-in system is explicitly exempt and is **not** required to offer Sign in with Apple. Adding "Log in with GitHub" or any similar social provider immediately creates the obligation to also offer an equivalent privacy-preserving login (Sign in with Apple being the canonical one), with the three properties: name+email only, private-email option, no advertising-purpose interaction collection.
|
||||
|
||||
### 4.5.4 Push notifications
|
||||
|
||||
- Push must **not be required** for the app to function.
|
||||
- Must not carry sensitive or confidential information.
|
||||
- Must not be used for promotions or direct marketing **unless the customer has explicitly opted in via consent language displayed in the app's UI**, and the app **provides an in-app method to opt out**.
|
||||
|
||||
### 4.10 Monetizing built-in capabilities
|
||||
|
||||
Push Notifications, camera, gyroscope, iCloud storage and similar OS capabilities may not be monetized.
|
||||
|
||||
---
|
||||
|
||||
## 5. Legal
|
||||
|
||||
### 5.1.1(i) Privacy policy
|
||||
|
||||
> All apps must include a link to their privacy policy **in the App Store Connect metadata field and within the app in an easily accessible manner**.
|
||||
|
||||
The policy must clearly and explicitly:
|
||||
- identify what data the app/service collects, how it collects it, and **all** uses of that data;
|
||||
- confirm that any third party with whom the app shares user data - analytics, ad networks, third-party SDKs, parents, subsidiaries or related entities - provides the same or equal protection of user data;
|
||||
- explain data retention/deletion policies and **describe how a user can revoke consent and/or request deletion of the user's data**.
|
||||
|
||||
Two distinct deliverables: an in-app accessible link, and a policy whose content covers those three points.
|
||||
|
||||
### 5.1.1(ii) Permission and consent withdrawal
|
||||
|
||||
Consent must be secured for collection of user or usage data even where anonymous. Paid functionality must not depend on granting data access. The app must provide **an easily accessible and understandable way to withdraw consent**.
|
||||
|
||||
### 5.1.1(iii) Data minimization
|
||||
|
||||
Only request access to data relevant to core functionality.
|
||||
|
||||
### 5.1.1(v) Account sign-in and **account deletion**
|
||||
|
||||
> If your app supports account creation, you must also **offer account deletion within the app**.
|
||||
|
||||
From Apple's dedicated support page, in force since **30 June 2022**:
|
||||
|
||||
- The app must **offer to delete the entire account record along with associated personal data**. Offering only to temporarily deactivate or disable an account is **explicitly insufficient**.
|
||||
- The account deletion option must be **easy to find**, typically in account settings.
|
||||
- If completion requires a website, the app must link **directly to the page** where the process is completed - not to a general support page and not merely out to the default browser.
|
||||
- If deletion takes additional time, the user must be told.
|
||||
- Confirmation steps are permitted: reauthentication, identity verification, entering a code sent to an address already on the account.
|
||||
- Support-flow-only deletion (phone call, email, ticket) is permitted **only** for highly regulated industries under 5.1.1(ix). A social network is not one.
|
||||
- Apps that make deletion "unnecessarily difficult" fail review.
|
||||
|
||||
Also in 5.1.1(v): if the app does not include significant account-based features, people must be able to use it without a login. A social network is account-based by nature, but **read-only public browsing without an account** is a strong signal of good faith and reduces friction with this clause and with 4.2.
|
||||
|
||||
### 5.1.1(x) Optional contact information
|
||||
|
||||
Basic contact information may be requested only if optional, with features not conditional on providing it.
|
||||
|
||||
### 5.1.2 Data use and sharing - the AI clause
|
||||
|
||||
> You must clearly disclose where personal data will be shared with third parties, **including with third-party AI**, and obtain **explicit permission** before doing so.
|
||||
|
||||
This is decisive for any platform that routes user content through an external model provider. Every path where a user's post, comment, message, file, or profile text leaves the platform for a third-party model is a third-party data share that requires **disclosure plus explicit permission**, not merely a line in a privacy policy.
|
||||
|
||||
Further clauses:
|
||||
- **(i)** The app may not require the user to enable push notifications, location or tracking in order to access functionality or receive compensation. App Tracking Transparency consent is required for tracking.
|
||||
- **(ii)** Data collected for one purpose may not be repurposed without further consent.
|
||||
- **(iii)** No surreptitious profile building; no attempts to re-identify anonymous or aggregated data.
|
||||
|
||||
### 5.1.4 Kids
|
||||
|
||||
Apps that collect, transmit or have the capability to share personal information from a minor - including "the ability to chat" and persistent identifiers - must include a privacy policy and comply with all applicable children's privacy statutes (COPPA, GDPR and equivalents). Birthdate and parental contact information may be requested **only** for the purpose of complying with those statutes.
|
||||
|
||||
### 5.2 Intellectual property
|
||||
|
||||
- **5.2.1** No protected third-party material without permission; no misleading or copycat names or metadata.
|
||||
- **5.2.2** Content from a third-party service requires permission under that service's terms; authorization must be provided on request. Engaged by any news/RSS ingestion feature.
|
||||
- **5.2.3** No saving, converting or downloading media from third-party sources without explicit authorization. Engaged by any URL-fetch, archive, or media-embed feature.
|
||||
- **5.2.5** No Apple emoji embedded in the binary; no interfaces confusingly similar to Apple products.
|
||||
|
||||
A UGC platform additionally needs a **notice-and-takedown (DMCA-style) path**, because 5.2 makes the developer answerable for infringing user content and 1.2 makes removal the developer's responsibility.
|
||||
|
||||
### 5.3 Gaming, gambling, lotteries
|
||||
|
||||
If the platform runs contests, sweepstakes or prize draws: the developer must sponsor them, **official rules must be presented in the app**, and the rules must state that **Apple is not a sponsor and is not involved in any manner**. Randomized reward mechanics that cannot be purchased with real money stay outside 5.3.4.
|
||||
|
||||
### 5.6 Developer code of conduct
|
||||
|
||||
Trust (5.6.1), ratings and reviews integrity (5.6.2), accurate developer identity (5.6.3) and the prohibition on predatory behaviour (5.6.4) - the latter explicitly covering exploitation of minors and facilitation or encouragement of harmful behaviour toward others. Violations can remove the developer from the Apple Developer Program entirely, independent of any single app.
|
||||
|
||||
---
|
||||
|
||||
## 6. App Store Connect obligations (metadata, not code)
|
||||
|
||||
These are not guideline sections but they block submission or removal just as hard.
|
||||
|
||||
### 6.1 Age rating - the 2025 overhaul
|
||||
|
||||
Apple replaced the old ladder with **4+, 9+, 13+, 16+, 18+**; the 12+ and 17+ tiers were removed. The questionnaire gained required questions covering in-app controls, capabilities, medical/wellness topics, and violent themes, plus a **social-features block** covering:
|
||||
|
||||
- user-generated content;
|
||||
- messaging capability;
|
||||
- friend or follower systems;
|
||||
- livestreaming;
|
||||
- content creation tools;
|
||||
- advertising that may expose users to age-sensitive material.
|
||||
|
||||
Apple additionally asks **what safeguards the developer has implemented**: moderation systems, content filtering, reporting tools, blocking functionality, parental controls. Answering "none" to those questions on a social app drives the rating up and invites 1.2 scrutiny; answering "yes" untruthfully violates 2.3.6.
|
||||
|
||||
Developers were required to complete the updated questionnaire by **31 January 2026**, after which app updates are blocked in App Store Connect until the new questions are answered.
|
||||
|
||||
**Consequence for this project:** the safeguards questionnaire is answered from the platform's actual feature set. Each of the five safeguard answers should map to a named, demonstrable feature.
|
||||
|
||||
### 6.2 App privacy details ("nutrition labels")
|
||||
|
||||
Every data type collected by the app **or by its third-party partners** must be declared across the categories: Contact Info, Health & Fitness, Financial Info, Location, Sensitive Info, Contacts, User Content, Browsing History, Identifiers, Purchases, Usage Data, Diagnostics, Surroundings. Each declared type is classified as **Used to Track You**, **Linked to You**, or **Not Linked to You**. The developer is responsible for third-party SDK collection and for **keeping the answers accurate and up to date**; answers may be changed at any time without an app update.
|
||||
|
||||
For the platform under review the realistic declaration set is: Contact Info (name, email), User Content (posts, messages, photos/videos, other user content), Identifiers (user ID), Usage Data (product interaction), Diagnostics, and - if any analytics or crash reporting is added - the corresponding categories. All "Linked to You"; none "Used to Track You" provided no cross-app advertising tracking exists.
|
||||
|
||||
### 6.3 Support URL, marketing URL, privacy policy URL
|
||||
|
||||
Required metadata. The Support URL must present a working contact route (1.5). The privacy policy URL must be live and must match the in-app policy.
|
||||
|
||||
### 6.4 EU Digital Services Act trader status
|
||||
|
||||
Since **17 February 2025**, apps without a declared and verified trader status are **removed from the App Store in the EU**. Trader status became required for update submission on 16 October 2024. Articles 30 and 31 DSA require Apple to verify and publish trader contact information - **address, phone number and email** - on the App Store product page. The DSA definition of commercial activity is broad: paid apps, apps with IAP, or otherwise commercial distribution.
|
||||
|
||||
### 6.5 Notes for Review
|
||||
|
||||
Under 2.3.1 all functionality must be described specifically. For an app of this shape the notes must at minimum describe: the moderation pipeline, where the report and block controls are, where account deletion is, that code execution is remote and user-owned under the 2.5.2 educational exception, that the AI assistant is a chatbot under 4.7 with its own safety controls, and the demo account credentials with pre-seeded content so the reviewer can exercise reporting.
|
||||
|
||||
---
|
||||
|
||||
## 7. Overlapping legal regimes Apple enforces by reference
|
||||
|
||||
| Regime | What Apple enforces | Practical requirement |
|
||||
|--------|---------------------|-----------------------|
|
||||
| **GDPR** (5.1.1(ii), 5.1.2) | Lawful basis, consent, withdrawal, erasure | Consent capture with timestamp and version; consent withdrawal UI; account + data deletion; data export is the companion right users will ask for |
|
||||
| **EU DSA** (6.4, 1.5) | Trader identity, published contact, notice-and-action | Published contact information in app and on the store page; a reporting mechanism with acknowledgement and outcome notice; a statement of reasons to the affected user when content is removed |
|
||||
| **COPPA** (5.1.4) | No collection from under-13s without verifiable parental consent | Declared-age gate at signup; block or restrict accounts below the platform's minimum age; do not collect birthdate for any other purpose |
|
||||
| **DMCA / copyright** (5.2) | Removal of infringing user content | A designated notice-and-takedown channel and a counter-notice path |
|
||||
| **Local content ratings** (2.3.6) | Territory-specific rating and warning display | Age labelling on content that exceeds the app rating (also required by 1.2.1(a) and 4.7.5) |
|
||||
|
||||
---
|
||||
|
||||
## 8. The complete requirement register
|
||||
|
||||
Every row is a discrete, testable obligation. This register is the input to `applechanges.md`.
|
||||
|
||||
### 8.1 Mandatory - a missing item is a certain rejection
|
||||
|
||||
| # | Requirement | Source |
|
||||
|---|-------------|--------|
|
||||
| R1 | Terms of service / EULA that **explicitly states zero tolerance for objectionable content and abusive users** | 1.2 (review practice) |
|
||||
| R2 | **Affirmative acceptance** of those terms recorded per user at account creation, and re-acceptance on material change | 1.2, GDPR |
|
||||
| R3 | **Community guidelines** enumerating prohibited content, aligned to the 1.1.1-1.1.7 categories | 1.1, 1.2 |
|
||||
| R4 | **Automated filtering** of objectionable material at the point of posting, on every UGC surface | 1.2, 4.7.1 |
|
||||
| R5 | **Report mechanism on every UGC surface**: posts, comments, gists, projects, files, media, news, DMs, quizzes, profiles, AI output, workspaces | 1.2, 4.7.1 |
|
||||
| R6 | **Moderation queue** with triage, decision and enforcement actions for the operators | 1.2 |
|
||||
| R7 | **Published 24-hour response commitment** and a mechanism that makes it achievable and evidenced | 1.2 (review practice) |
|
||||
| R8 | **Ejection of offending users** - suspension/ban as a first-class enforcement action, not only content deletion | 1.2 |
|
||||
| R9 | **Block abusive users** from the service, covering all interaction surfaces including DMs | 1.2 |
|
||||
| R10 | **Published contact information reachable inside the app** | 1.5, DSA Art. 30 |
|
||||
| R11 | **Privacy policy** meeting 5.1.1(i)'s three content requirements, linked in-app and in ASC metadata | 5.1.1(i) |
|
||||
| R12 | **In-app account deletion** that deletes the account record and associated personal data, easy to find, no support-flow requirement | 5.1.1(v) |
|
||||
| R13 | **Declared-age gate** at account creation, with a minimum age, plus an age-restriction mechanism limiting underage access to age-exceeding content | 1.2.1(a), 4.7.5, 5.1.4 |
|
||||
| R14 | **Content age labelling** so users can identify content exceeding the app's age rating; mature content **hidden by default** | 1.2, 1.2.1(a), 4.7.5 |
|
||||
| R15 | **Explicit consent before user content is sent to third-party AI**, plus disclosure of which provider and what data | 5.1.2(i) |
|
||||
| R16 | **Consent withdrawal** UI that is easily accessible and understandable | 5.1.1(ii) |
|
||||
| R17 | **Push notifications optional**, never required for function, marketing push opt-in with in-app opt-out | 4.5.4, 5.1.2(i) |
|
||||
| R18 | **DMCA / IP notice-and-takedown** channel | 5.2 |
|
||||
| R19 | **Demo account with pre-seeded content** and review notes describing every safety control's location | 2.1, 2.3.1 |
|
||||
| R20 | **Age rating questionnaire** answered from the real feature set, including the five safeguard answers | 2.3.6, 6.1 |
|
||||
| R21 | **App privacy details** declared accurately for every data type, including third-party AI processing | 6.2 |
|
||||
| R22 | **EU trader status** declared and verified, with address, phone and email | 6.4 |
|
||||
| R23 | **IPv6-only reachability** of every endpoint, WebSocket and asset host | 2.5.5 |
|
||||
| R24 | **Remote code execution positioned under the 2.5.2 educational exception**: source completely viewable and editable, executed off-device, never altering the app | 2.5.2 |
|
||||
| R25 | **Native client that is materially more than a web wrapper** | 4.2 |
|
||||
|
||||
### 8.2 Conditional - required if the corresponding feature exists
|
||||
|
||||
| # | Requirement | Trigger |
|
||||
|---|-------------|---------|
|
||||
| C1 | Sign in with Apple or an equivalent privacy-preserving login | Any third-party/social login is offered |
|
||||
| C2 | In-app purchase for every digital good, currency, credit, boost, tip or premium unlock | Anything is sold to end users in-app |
|
||||
| C3 | Loot-box odds disclosure | Randomized purchasable rewards |
|
||||
| C4 | Official contest rules in-app stating Apple is not a sponsor | Any sweepstake, contest or raffle |
|
||||
| C5 | Index of all offered mini apps/software with universal links | Users can open other users' software from the app |
|
||||
| C6 | Ad reporting control | Advertising is displayed |
|
||||
| C7 | ATT prompt | Any cross-app/site tracking |
|
||||
| C8 | Recording indicator and consent | Any session/screen/activity recording |
|
||||
| C9 | Per-instance consent before sharing data or permissions with a mini app | Mini apps receive user data |
|
||||
|
||||
### 8.3 Posture requirements - not a single feature, an ongoing obligation
|
||||
|
||||
| # | Requirement | Source |
|
||||
|---|-------------|--------|
|
||||
| P1 | Ability to produce, on Apple's request, a **compliance improvement plan** and evidence of moderation throughput | 1.2 |
|
||||
| P2 | Retention of moderation decisions as an audit trail | 1.2, DSA |
|
||||
| P3 | Statement of reasons to the user whose content is removed or whose account is actioned | DSA Art. 17 |
|
||||
| P4 | Keeping privacy labels and the privacy policy in step with feature changes | 6.2, 5.1.1(i) |
|
||||
| P5 | Accurate "What's New" text for significant changes | 2.3.12 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Where reviewers actually look
|
||||
|
||||
Ordered by observed rejection frequency for this application shape:
|
||||
|
||||
1. **Report control not visible on the first screen of content the reviewer opens.** The reviewer opens the feed, taps a post, and looks for a report affordance. If it is buried behind a profile menu, the app is rejected under 1.2 even though the mechanism exists.
|
||||
2. **No terms acceptance at signup.** The reviewer creates an account with the demo credentials or a fresh account and looks for the EULA gate.
|
||||
3. **Account deletion not found in settings.** The reviewer opens account settings and searches for "Delete account".
|
||||
4. **Privacy policy not reachable in-app.**
|
||||
5. **Demo account sees an empty feed**, so nothing can be reported or blocked.
|
||||
6. **Blocking present but not reachable from the content itself**, only from a profile.
|
||||
7. **AI feature sending content to a third party with no disclosure or consent.**
|
||||
8. **No age gate on a platform with messaging and follower systems.**
|
||||
|
||||
---
|
||||
|
||||
## 10. Determination for this platform
|
||||
|
||||
Applying the register to the DevPlace shape:
|
||||
|
||||
- **Applicable in full:** R1-R25 except where noted below.
|
||||
- **C1 not triggered** provided the platform continues to use exclusively its own account system. Adding any social login triggers it immediately.
|
||||
- **C2 not triggered** provided no in-app purchase of any digital good, currency, credit or quota exists and none is linked to. The in-app virtual economy must remain earn-only.
|
||||
- **C3 not triggered** while randomized rewards are not purchasable.
|
||||
- **C4 triggered** by any leaderboard prize, era award or contest that awards something of value; the safe position is that awards are purely cosmetic/status and are not framed as a contest with prizes.
|
||||
- **C5 triggered** if a user can open another user's running workspace, published site or executable project from the app.
|
||||
- **C6, C7 not triggered** while there is no advertising and no cross-app tracking.
|
||||
- **C8 triggered** by presence tracking, live view relay, session recording or terminal session capture that records user activity.
|
||||
- **C9 triggered** by any path where platform user data is passed into a user-authored workspace or plug-in.
|
||||
|
||||
The single largest exposure is **R5 breadth**: reporting must exist on every surface, and the platform under review has an unusually large number of distinct UGC surfaces. The second largest is **R15**, because AI is woven through the platform and every path that sends user text to a model provider is a third-party data share.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [App Review Guidelines](https://developer.apple.com/app-store/review/guidelines/)
|
||||
- [Offering Account Deletion in Your App](https://developer.apple.com/support/offering-account-deletion-in-your-app/)
|
||||
- [App Privacy Details on the App Store](https://developer.apple.com/app-store/app-privacy-details/)
|
||||
- [Updated age ratings in App Store Connect](https://developer.apple.com/news/?id=ks775ehf)
|
||||
- [Age rating questionnaire now includes social media questions](https://developer.apple.com/news/?id=tlur8uvi)
|
||||
- [Apple overhauls App Store age ratings](https://www.macrumors.com/2025/07/25/apple-overhauls-app-store-age-ratings/)
|
||||
- [Apple notifies developers of new App Store age rating system](https://9to5mac.com/2025/07/24/apple-notifies-developers-of-new-app-store-age-rating-system/)
|
||||
- [Apps without trader status will be removed from the App Store in the EU](https://developer.apple.com/news/?id=einwn76m)
|
||||
- [Manage European Union Digital Services Act trader requirements](https://developer.apple.com/help/app-store-connect/manage-compliance-information/manage-european-union-digital-services-act-trader-requirements/)
|
||||
- [Provide your trader status in App Store Connect](https://developer.apple.com/news/?id=x60uzbu9)
|
||||
- [Resolving App Store Guideline 1.2 - User Generated Content](https://buddyboss.com/docs/app-store-guideline-1-2-safety-user-generated-content/)
|
||||
- [Complying with Apple App Store UGC requirements](https://www.termsfeed.com/videos/apple-app-store-comply-ugc-requirements/)
|
||||
- [Guideline 1.2 - Safety - User-Generated Content (Apple Developer Forums)](https://developer.apple.com/forums/thread/807358)
|
||||
583
appleimpl.md
Normal file
583
appleimpl.md
Normal file
@ -0,0 +1,583 @@
|
||||
# DevPlace: App Store compliance implementation design
|
||||
|
||||
Author: retoor <retoor@molodetz.nl>
|
||||
|
||||
Stage three of `apple.md`. Inputs are [`applecomp.md`](applecomp.md) (what Apple requires) and [`applechanges.md`](applechanges.md) (what DevPlace lacks). This document is the design: the most consistent, DRY, caveat-free way to implement every gap inside the conventions this codebase already enforces.
|
||||
|
||||
Nothing here is implemented. This is the specification the lord is asked to approve.
|
||||
|
||||
---
|
||||
|
||||
## 1. Design axioms
|
||||
|
||||
Each axiom is derived from an existing DevPlace pattern, named with its precedent. No axiom is invented for this feature.
|
||||
|
||||
| # | Axiom | Precedent in the codebase |
|
||||
|---|-------|---------------------------|
|
||||
| **A1** | **One polymorphic facility, never twenty per-surface features.** A report is structurally a vote: an actor, a `(target_type, target_uid)` pair, a payload. | `comments`, `votes`, `reactions`, `bookmarks` all key on `(target_type, target_uid)`; `VOTABLE_TARGETS` in `database/ranking.py:11`; `REACTABLE` in `routers/reactions.py:17` |
|
||||
| **A2** | **The target set is a registry, not a literal.** Every consumer reads the same dict; adding a surface is one line. | `VOTABLE_TARGETS`, `STAR_TARGETS`, `NOTIFICATION_TYPES`, `SOFT_DELETE_TABLES`, `DOCS_PAGES`, `DATA_PATHS` |
|
||||
| **A3** | **Machine-raised and human-raised entries share one queue and one state machine.** | `services/containers/workspace/flags.py`: `raise_flag` is machine-driven, `set_status` is admin-driven, statuses `open/acknowledged/resolved/dismissed`, severities `info/warn/critical` |
|
||||
| **A4** | **Every route has four faces:** HTML, JSON, Devii action, API docs. | Root `CLAUDE.md`, "Anatomy of a feature" |
|
||||
| **A5** | **Removal is soft; garbage collection is hard; cascades share one stamp.** | `database/soft_delete.py`, `SOFT_DELETE_TABLES` (44 tables), `soft_delete_in`, `purge_event`, `/admin/trash` |
|
||||
| **A6** | **Runtime policy lives in `site_settings`,** read through `get_setting`/`get_int_setting`, live-editable at `/admin/settings`, never in code constants. | `database/schema.py:276`, `rate_limit_per_minute`, `maintenance_mode`, `registration_open` |
|
||||
| **A7** | **Action bars are composed from included partials** with `{% set _type %}{% set _uid %}{% include %}`. | `_reaction_bar.html` included from `_post_card.html:37` and `_comment.html:37` |
|
||||
| **A8** | **Non-response-critical side-effects go through `background.submit`;** audit and notifications are already funnelled there. | `services/background.py`, `utils/notifications.py:68` |
|
||||
| **A9** | **Never a silent failure.** A safety control that swallows an error is worse than absent. | Root `CLAUDE.md`; `services/audit` never raises into the caller but always records |
|
||||
| **A10** | **Legal and policy prose is a docs page,** with the existing role gating, SEO context and search index. | `routers/docs/pages.py` `DOCS_PAGES`; the admin-only `media-moderation` page proves gating works |
|
||||
| **A11** | **Owner-or-admin, with the seniority guard on admin-versus-admin.** | `content.is_owner`, `_is_senior_admin` in `routers/admin/users.py` |
|
||||
| **A12** | **The AI gateway is the single choke point for third-party model calls,** so consent is enforced in exactly one place. | `services/openai_gateway/`, `INTERNAL_GATEWAY_URL` |
|
||||
|
||||
---
|
||||
|
||||
## 2. The unifying abstraction
|
||||
|
||||
Everything in this design hangs off **one registry** and **one queue**.
|
||||
|
||||
### 2.1 The moderation target registry
|
||||
|
||||
New module `devplacepy/database/moderation.py`, mirroring `database/ranking.py` exactly in shape and placement:
|
||||
|
||||
```
|
||||
REPORTABLE_TARGETS: dict[str, str] # target_type -> table name
|
||||
MATURITY_TARGETS: set[str] # subset that can carry an age label
|
||||
```
|
||||
|
||||
`REPORTABLE_TARGETS` covers every externally-visible surface from `applechanges.md` §2:
|
||||
|
||||
`post`, `comment`, `gist`, `project`, `project_file`, `news`, `attachment`, `message`, `quiz`, `poll`, `award`, `user`, `issue`, `workspace`, `devii_output`.
|
||||
|
||||
`MATURITY_TARGETS` is the subset that renders long-form authored content: `post`, `comment`, `gist`, `project`, `news`, `attachment`, `quiz`.
|
||||
|
||||
**Why a registry rather than per-surface code.** A report route, a report button, a moderation queue row, a Devii action parameter enum, an API docs enum and a test fixture all need the same list. With a registry they read it; without one they drift. This is the same reason `VOTABLE_TARGETS` exists.
|
||||
|
||||
**The completeness invariant.** A unit test asserts that every entry in `REPORTABLE_TARGETS` resolves to a real table (or an explicitly listed virtual surface) **and** that every externally-visible table in `SOFT_DELETE_TABLES` appears in `REPORTABLE_TARGETS`. Adding a new UGC surface without adding it to the registry fails the suite. Requirement R5 is therefore satisfied not by diligence but by construction. This is the load-bearing correctness claim of the whole design; §11 formalises it.
|
||||
|
||||
### 2.2 The single queue
|
||||
|
||||
One table, `content_reports`, with two producers:
|
||||
|
||||
- **members**, via the report control on every content action bar;
|
||||
- **the filter**, via a system-raised entry when classification returns `review`.
|
||||
|
||||
This is `workspace_flags` generalised from one instance type to the registry. Same state machine (`open → acknowledged → actioned | dismissed`), same severity ladder (`info | warn | critical`), same soft-delete participation, same admin resolution surface. One queue means one SLA measurement, one admin screen, one audit shape, and one place where the 24-hour commitment is either met or visibly not.
|
||||
|
||||
### 2.3 URL resolution is already solved
|
||||
|
||||
`database/content.py:22` `resolve_object_url(target_type, target_uid)` already maps `post`, `project`, `news`, `issue`, `gist`, `quiz`, `comment` and `award` to their canonical URLs, recursing through comments to their parents. It gains the remaining registry entries (`project_file`, `attachment`, `message`, `user`, `workspace`, `poll`, `devii_output`). Every moderation surface then links to its subject for free, using the function the notification system already uses.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data layer
|
||||
|
||||
All schema changes land in `devplacepy/database/schema.py` `init_db()` following the existing `has_column` / `create_column_by_example` / `_index` idiom, and every new table is registered in `SOFT_DELETE_TABLES`.
|
||||
|
||||
### 3.1 `content_reports`
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `uid` | text | `generate_uid()` |
|
||||
| `reporter_uid` | text | user uid, or `system` for filter-raised (mirrors `audit.record_system`) |
|
||||
| `target_type` | text | key of `REPORTABLE_TARGETS` |
|
||||
| `target_uid` | text | subject uid |
|
||||
| `owner_uid` | text | author of the reported content, denormalised at insert so the queue never N+1s |
|
||||
| `reason` | text | key of `REPORT_REASONS` (§3.6) |
|
||||
| `detail` | text | reporter's free text, max 2000 |
|
||||
| `severity` | text | `info` / `warn` / `critical` |
|
||||
| `status` | text | `open` / `acknowledged` / `actioned` / `dismissed` |
|
||||
| `origin` | text | `member` / `filter` |
|
||||
| `categories` | text | JSON list of matched 1.1.x category keys, filter-raised only |
|
||||
| `resolved_by` | text | admin uid |
|
||||
| `resolved_at` | text | ISO |
|
||||
| `created_at`, `updated_at` | text | ISO |
|
||||
| `deleted_at`, `deleted_by` | text | soft delete |
|
||||
|
||||
Indexes: `(status, created_at)` for the queue and the SLA scan; `(target_type, target_uid)` for "is this already reported"; `(reporter_uid)` for the reporter's own list; `(owner_uid)` for offender history. Partial soft-delete index per the standing convention.
|
||||
|
||||
**Duplicate handling** follows `raise_flag` precisely: an open report for the same `(target_type, target_uid, reporter_uid)` is updated, not duplicated. A different reporter on the same target creates a new row; the queue groups by target and shows the count, which is exactly how a real moderation queue prioritises.
|
||||
|
||||
### 3.2 `moderation_actions`
|
||||
|
||||
The decision record. One row per moderator decision, linked to the report that triggered it.
|
||||
|
||||
`uid`, `report_uid`, `actor_uid`, `action`, `target_type`, `target_uid`, `subject_uid`, `reason`, `notes`, `expires_at`, `created_at`, soft-delete columns.
|
||||
|
||||
`action` ∈ `remove_content`, `restore_content`, `warn`, `suspend`, `ban`, `lift`, `dismiss`, `escalate`.
|
||||
|
||||
This is the DSA statement-of-reasons substrate (P3) and the compliance-plan evidence (P1). It is separate from the audit log because the audit log is append-only infrastructure and this is queryable moderation state with its own lifecycle - the same reason `workspace_flags` exists alongside the audit log.
|
||||
|
||||
### 3.3 `content_maturity`
|
||||
|
||||
Polymorphic age label, one row per labelled item. `uid`, `target_type`, `target_uid`, `level`, `source`, `set_by`, `created_at`, soft-delete columns.
|
||||
|
||||
`level` ∈ `general`, `mature`, `restricted`. `source` ∈ `author`, `filter`, `moderator`.
|
||||
|
||||
Read through a batch helper `get_maturity_by_targets(target_type, uids)` modelled exactly on `database/engagement.py` `get_reactions_by_targets` - no N+1, one query per listing. Absence of a row means `general`, so nothing needs backfilling and no existing row is touched.
|
||||
|
||||
### 3.4 `user_consents`
|
||||
|
||||
`uid`, `owner_kind`, `owner_id`, `kind`, `version`, `state`, `granted_at`, `withdrawn_at`, `created_at`, soft-delete columns.
|
||||
|
||||
`owner_kind`/`owner_id` reuse the `owner_for(request)` convention from the customization subsystem verbatim, so guests are covered by the same table. `kind` ∈ `terms`, `privacy`, `ai_third_party`, `activity_recording`. `state` ∈ `granted`, `withdrawn`.
|
||||
|
||||
Consent is **versioned and append-only in effect**: withdrawing writes `withdrawn_at` and a new grant writes a new row, so the full consent history is provable - which is what GDPR and Apple both actually require.
|
||||
|
||||
### 3.5 `users` columns
|
||||
|
||||
Added with the existing `has_column` guard block at `database/schema.py:1823`:
|
||||
|
||||
| Column | Default | Purpose |
|
||||
|--------|---------|---------|
|
||||
| `terms_version` | `""` | Accepted document version (R2) |
|
||||
| `terms_accepted_at` | `""` | ISO timestamp (R2) |
|
||||
| `age_band` | `""` | `under_min` / `13_15` / `16_17` / `adult` (R13) |
|
||||
| `age_declared_at` | `""` | ISO timestamp |
|
||||
| `mature_opt_in` | `0` | Explicit opt-in to see mature-labelled content (R14) |
|
||||
| `suspended_until` | `""` | ISO; empty means not suspended (R8) |
|
||||
| `suspension_reason` | `""` | Shown to the user (P3) |
|
||||
| `deletion_requested_at` | `""` | Starts the deletion clock (R12) |
|
||||
|
||||
**No birthdate is stored.** 5.1.4 permits collecting it only to comply with children's privacy statutes; data minimization (5.1.1(iii)) then requires storing only the derived band. The signup form collects a date, derives the band, and discards the date. This is both the compliant and the simpler design.
|
||||
|
||||
### 3.6 Registries and constants
|
||||
|
||||
`devplacepy/database/moderation.py` also owns:
|
||||
|
||||
- `REPORT_REASONS: dict[str, str]` - key to label, mapped one-to-one onto the guideline categories so the age-rating questionnaire and the community guidelines can be written from the same list: `hate` (1.1.1), `violence` (1.1.2), `weapons` (1.1.3), `sexual` (1.1.4), `religious` (1.1.5), `misinformation` (1.1.6), `exploitative` (1.1.7), `harassment`, `spam`, `intellectual_property` (5.2 / R18), `self_harm`, `illegal`, `other`.
|
||||
- `REPORT_STATUSES`, `REPORT_SEVERITIES`, `MODERATION_ACTIONS`, `MATURITY_LEVELS`, `CONSENT_KINDS`, `AGE_BANDS`.
|
||||
|
||||
One list, consumed by the form validator, the Devii action schema, the API docs enum, the admin filter dropdown and the community-guidelines page. Changing a reason is one edit.
|
||||
|
||||
### 3.7 `site_settings` keys
|
||||
|
||||
Added to the defaults block at `database/schema.py:276`, editable live at `/admin/settings` (A6):
|
||||
|
||||
| Key | Default | Purpose |
|
||||
|-----|---------|---------|
|
||||
| `moderation_sla_hours` | `24` | The published commitment (R7) |
|
||||
| `moderation_filter_mode` | `review` | `off` / `label` / `review` / `block` (R4) |
|
||||
| `moderation_minimum_age` | `16` | Signup floor (R13) |
|
||||
| `moderation_mature_default_hidden` | `1` | Mature content hidden by default (R14) |
|
||||
| `contact_email`, `contact_phone`, `contact_address` | empty | Published contact + DSA trader data (R10, R22) |
|
||||
| `terms_version`, `privacy_version`, `guidelines_version` | `1` | Bump forces re-acceptance (R2) |
|
||||
| `ai_third_party_provider` | `""` | Named in the consent copy (R15) |
|
||||
| `account_deletion_grace_hours` | `24` | Reversible window before purge (R12) |
|
||||
|
||||
---
|
||||
|
||||
## 4. The content filter
|
||||
|
||||
`devplacepy/services/moderation/` - a new service package alongside `services/audit/`, `services/game/` and the rest, with its own nested `CLAUDE.md`.
|
||||
|
||||
### 4.1 Shape
|
||||
|
||||
```
|
||||
services/moderation/
|
||||
__init__.py record()-style entrypoints, the only public surface
|
||||
filter.py classify(text) -> Classification
|
||||
rules.py the category rule set
|
||||
queue.py raise_report / set_status / decide / list_reports
|
||||
enforcement.py suspend / ban / lift / remove_content
|
||||
sla.py oldest_open_age / breach_count
|
||||
```
|
||||
|
||||
`Classification` is a frozen dataclass (`verdict`, `categories`, `maturity`, `score`) - dataclasses over fixed-key dicts, per the standing style rule.
|
||||
|
||||
`verdict` ∈ `allow`, `label`, `review`, `block`, resolved against `moderation_filter_mode` so an administrator can dial the platform from advisory to strict without a deploy.
|
||||
|
||||
### 4.2 Where it runs - exactly five call sites
|
||||
|
||||
The filter is invoked only at choke points that already exist, so no surface can be missed and no surface needs bespoke code:
|
||||
|
||||
1. `content.create_content_item` (`content.py:197`) - posts, projects, gists, news, quizzes.
|
||||
2. `content.create_comment_record` (`content.py:361`) - every comment on every parent type.
|
||||
3. `content.edit_content_item` and `content.edit_comment_record` - edits, so a clean post cannot be edited into a violation.
|
||||
4. `routers/messages.py:245` `send_message` and the WebSocket send path - direct messages.
|
||||
5. `routers/profile/index.py` profile update and `routers/auth/signup.py` - bio, location, links, username.
|
||||
|
||||
Five call sites cover twenty surfaces because the codebase already funnels creation. This is the direct payoff of DevPlace's existing structure.
|
||||
|
||||
### 4.3 Behaviour, and why it is safe on a developer platform
|
||||
|
||||
The single largest implementation risk identified in `applechanges.md` §7 is false positives: a security-focused developer community discusses exploits, weapons-grade cryptography and violent language in code review. A naive block destroys the product.
|
||||
|
||||
The design answers this structurally:
|
||||
|
||||
- **The default mode is `review`, not `block`.** A flagged item is published **and** a system report is raised. Nothing legitimate is ever suppressed by a machine.
|
||||
- **Only the `sexual` and `exploitative` categories default to `block`**, because those are the two where Apple removes apps without notice and where no developer-platform false-positive case exists.
|
||||
- **Thresholds are `site_settings`,** tunable live while watching the queue.
|
||||
- **A failure in the filter fails to `review`, never to `allow`** (A9). If classification raises, the content is published and a `critical` system report is raised naming the failure. A moderation control that fails open is worse than absent.
|
||||
|
||||
This gives Apple the "method for filtering objectionable material from being posted" that 1.2 requires, gives the platform a human in the loop, and gives the community no false suppression.
|
||||
|
||||
---
|
||||
|
||||
## 5. Server layer
|
||||
|
||||
### 5.1 Reporting - `devplacepy/routers/reports.py`, mounted at `/reports`
|
||||
|
||||
Mirrors `routers/reactions.py` line for line.
|
||||
|
||||
| Method | Path | Auth | Purpose |
|
||||
|--------|------|------|---------|
|
||||
| `POST` | `/reports/{target_type}/{target_uid}` | member | Submit a report |
|
||||
| `GET` | `/reports/mine` | member | The reporter's own reports and their outcomes (DSA Art. 16 acknowledgement) |
|
||||
| `GET` | `/reports/reasons` | public | The reason registry, so any client renders the same dialog |
|
||||
|
||||
Input model `ReportForm` in `models.py` (`reason`, `detail`); output schema `ReportOut` / `ReportListOut` in `schemas/moderation.py`. `respond(request, template, ctx, model=ReportOut)` gives HTML and JSON from one handler. Rate limiting is already global on POST via the existing middleware; no per-route limiter is added.
|
||||
|
||||
Submitting a report **always** notifies the reporter through `create_notification` with the acknowledgement and the SLA, and **never** notifies the reported user (that happens only on decision, as a statement of reasons).
|
||||
|
||||
### 5.2 Moderation queue - `devplacepy/routers/admin/moderation.py`
|
||||
|
||||
Registered in the `admin/` package exactly like `trash.py` and `media.py`, with `admin_section = "moderation"`.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| `GET` | `/admin/moderation` | The queue, grouped by target, sorted oldest-open-first, with the SLA badge |
|
||||
| `GET` | `/admin/moderation/{uid}` | One report, its target rendered in place, the offender's history |
|
||||
| `POST` | `/admin/moderation/{uid}/status` | `acknowledge` / `dismiss` |
|
||||
| `POST` | `/admin/moderation/{uid}/decide` | Apply a `MODERATION_ACTIONS` decision |
|
||||
|
||||
Every decision writes a `moderation_actions` row, records an audit event, and - where the decision affects a user - delivers a statement of reasons through `create_notification`.
|
||||
|
||||
### 5.3 Enforcement - extending `routers/admin/users.py`
|
||||
|
||||
The bare `is_active` toggle at `admin/users.py:179` is kept for backward compatibility and joined by:
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| `POST` | `/admin/users/{uid}/suspend` | Reason + duration; writes `suspended_until`, `suspension_reason` |
|
||||
| `POST` | `/admin/users/{uid}/lift` | Clears both |
|
||||
| `POST` | `/admin/users/{uid}/ban` | Permanent; `is_active = False` **with** a recorded reason |
|
||||
|
||||
All three pass through the existing `_is_senior_admin(actor, target)` guard (A11), so a junior admin cannot suspend a senior one - server-side, therefore also covering Devii.
|
||||
|
||||
Enforcement is read by one new predicate in `content.py`, `is_suspended(user)`, consulted by `require_user` so a suspended account can still read, still see why, and still delete their account, but cannot post. This is one predicate at one choke point, not a scattered check.
|
||||
|
||||
### 5.4 Account deletion - `routers/profile/delete.py`
|
||||
|
||||
Follows the `regenerate-avatar` precedent (owner-or-admin, POST under `/profile/{username}/…`, audited, cache-invalidating).
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| `GET` | `/profile/{username}/delete` | The confirmation page: what will be deleted, what is retained and why, the grace window |
|
||||
| `POST` | `/profile/{username}/delete` | Requires the account password (reauthentication, explicitly permitted by Apple); starts deletion |
|
||||
|
||||
**The cascade**, using one shared stamp (A5):
|
||||
|
||||
1. Stamp `deletion_requested_at`, revoke every session and access token, invalidate the user cache.
|
||||
2. `soft_delete_in(table, "user_uid", [uid], deleted_by=uid, stamp=stamp)` across every table in `SOFT_DELETE_TABLES` that carries a `user_uid` - one stamp, so `/admin/trash` can restore the entire event atomically within the grace window.
|
||||
3. Anonymise the `users` row immediately: username tombstoned, email, bio, location, links, avatar seed, API key and password hash cleared. **From the user's and every other user's point of view, the account is gone the moment they confirm.**
|
||||
4. A GC sweep (`devplace accounts prune`, and a scheduled pass in the existing service manager) hard-purges the stamped event after `account_deletion_grace_hours`, using `purge_event(stamp)` - the function that already exists.
|
||||
|
||||
The confirmation page states the grace window explicitly, satisfying Apple's "if the deletion request will take additional time to complete, let them know."
|
||||
|
||||
The devRant `DELETE /api/users/me` at `routers/devrant/auth.py:189` is re-pointed at this same cascade, because a deactivation masquerading as a deletion is exactly what Apple names as insufficient, and because two paths must not mean two behaviours.
|
||||
|
||||
### 5.5 Terms, age and consent
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| `POST` | `/auth/accept-terms` | Records acceptance of the current `terms_version` |
|
||||
| `POST` | `/profile/{username}/consent` | Grant or withdraw a `CONSENT_KINDS` entry |
|
||||
| `GET` | `/profile/{username}?tab=privacy` | Acceptances, consents, withdrawal controls, deletion entry point |
|
||||
|
||||
`SignupForm` (`models.py:51`) gains `birth_date` and `accept_terms`, both required, validated Pydantic-natively like every other form in the project. The validator derives `age_band`, rejects below `moderation_minimum_age`, and the raw date never reaches the database.
|
||||
|
||||
**The re-acceptance gate** is a middleware in the existing stack in `main.py`, sitting beside the maintenance gate it is modelled on: an authenticated user whose `terms_version` is behind the setting is redirected to the acceptance page for any mutating request, while reads, `/static`, `/auth`, `/docs` and account deletion stay open. A user must never be trapped: they can always read, always accept, and always delete their account.
|
||||
|
||||
### 5.6 Third-party AI consent - one gate at one choke point
|
||||
|
||||
Enforced in `services/openai_gateway/` where every internal AI consumer already converges (A12).
|
||||
|
||||
The rule distinguishes two things that the existing code currently conflates:
|
||||
|
||||
- **User-content processing** - the user's own post, comment, message, file or prompt is sent to the provider. Requires a granted `ai_third_party` consent for that user. Default: **not granted**.
|
||||
- **Platform processing** - news import, bot personas, SEO metadata for platform-owned text. Not user content, not gated by user consent.
|
||||
|
||||
The gateway resolves the owner it is acting for and refuses a user-content call without consent, returning a structured error the callers already know how to surface. The existing `ai_correction_enabled` and `ai_modifier_enabled` flags survive unchanged as **preferences**, subordinate to consent: consent withdrawn means the feature is off regardless of the preference. `ai_modifier_enabled`'s default of `1` becomes harmless, because consent gates it. No existing preference is silently flipped; the gate is simply added above them.
|
||||
|
||||
The consent copy names the provider from `ai_third_party_provider`, states what is sent and why, and links the privacy policy - the three things 5.1.2(i) demands.
|
||||
|
||||
### 5.7 Activity-recording consent and indicator (C8)
|
||||
|
||||
`activity_recording` consent covers presence (`services/presence.py`), the live view relay and Devii terminal session capture. Guideline 2.5.14 wants consent **and** a clear indication. The indication reuses the existing presence dot partial `_presence_dot.html` and the response-time badge idiom in `base.html`: a small, always-visible recording indicator when a session is being captured. Withdrawing consent stops presence writes for that user; they simply appear offline.
|
||||
|
||||
### 5.8 The software index (C5 / 4.7.4)
|
||||
|
||||
`GET /workspaces/index` - a public, paginated index of every user-published workspace reachable through the `/p/{slug}` ingress, with its owner, description, maturity label and canonical URL. This is the "index of software and metadata available in your app… including universal links" that 4.7.4 requires. It reuses the existing listing machinery (`build_pagination`, `_card_link.html`, `paginate_diverse`) and is added to the sitemap.
|
||||
|
||||
---
|
||||
|
||||
## 6. View layer
|
||||
|
||||
### 6.1 One partial, included everywhere
|
||||
|
||||
`templates/_report_button.html`, included with the same two-variable idiom as `_reaction_bar.html` (A7):
|
||||
|
||||
```
|
||||
{% set _type = "post" %}{% set _uid = item.post['uid'] %}{% set _owner = item.post['user_uid'] %}
|
||||
{% include "_report_button.html" %}
|
||||
```
|
||||
|
||||
It renders **Report** and, when the viewer is not the owner, **Block**, both `guest_disabled(user)`, both matching the existing `post-action-btn` / `comment-action-btn` visual language exactly. Placing Block here closes gap G9 from `applechanges.md`: blocking becomes reachable from the content, not only from a profile.
|
||||
|
||||
Include sites: `_post_card.html`, `_comment.html`, `post.html`, `gist_detail.html`, `project_detail.html`, `news_detail.html`, `quiz.html`, `_media_gallery.html`, `messages.html`, `profile.html`, `_award_badge.html`, `project_files.html`, `issue_detail.html`, `containers_instance.html`.
|
||||
|
||||
**One partial, fourteen include sites, zero duplicated markup.** A template that renders content and omits the include is caught by the e2e coverage test in §10.
|
||||
|
||||
### 6.2 One dialog
|
||||
|
||||
`templates/_report_dialog.html` is included once in `base.html`, exactly as the reaction picker is a single palette reused by every bar. `static/js/ReportDialog.js` - one ES6 class, registered on `app`, using the existing `Http` helper and the established `.modal-overlay` / `.visible` modal pattern - reads `data-report-type` and `data-report-uid` from the clicked button, populates the reason list from `/reports/reasons`, and posts. No new modal machinery, no third-party library.
|
||||
|
||||
### 6.3 Maturity gate
|
||||
|
||||
`templates/_maturity_gate.html`: an interstitial rendered in place of a `mature`-labelled item for a viewer who has not opted in or whose `age_band` is below the threshold. Reveal is a single control that sets `mature_opt_in`; it is not offered at all to `13_15` or `16_17` bands for `restricted` content. Content stays hidden by default, which is precisely 1.2's wording.
|
||||
|
||||
### 6.4 Admin
|
||||
|
||||
`templates/admin_moderation.html` extends `admin_base.html` with `admin_section = "moderation"`, and a sidebar entry is added to `admin_base.html` between Media and Trash - the natural neighbours. The queue header carries the SLA badge: oldest open report age against `moderation_sla_hours`, green under, red over. That badge is the mechanism that makes the published 24-hour commitment (R7) real rather than aspirational.
|
||||
|
||||
### 6.5 Legal pages and the footer
|
||||
|
||||
Legal prose ships as `DOCS_PAGES` entries (A10) under a new `SECTION_LEGAL = "Legal"`, placed in the `AUDIENCE_START` group so it is one click from `/docs`:
|
||||
|
||||
| Slug | Title | Requirement |
|
||||
|------|-------|-------------|
|
||||
| `terms` | Terms of Service | R1, R2 |
|
||||
| `community-guidelines` | Community Guidelines | R3 |
|
||||
| `privacy` | Privacy Policy | R11 |
|
||||
| `contact` | Contact | R10, R22 |
|
||||
| `content-moderation` | How moderation works | R7, P1 |
|
||||
| `intellectual-property` | Notice and takedown | R18 |
|
||||
| `moderation-operations` | Operating the queue (admin-gated, like `media-moderation`) | P1, P2 |
|
||||
|
||||
`_footer_links.html` gains Terms, Privacy, Guidelines and Contact alongside the existing four links. This is the "easily accessible in the app" that 5.1.1(i) and 1.5 both require, and it is on every page because the footer is in `base.html`.
|
||||
|
||||
`contact` renders `contact_email`, `contact_phone` and `contact_address` from `site_settings`, so the in-app contact data and the App Store Connect trader data have one source of truth and cannot drift (R10 ≡ R22).
|
||||
|
||||
### 6.6 Signup
|
||||
|
||||
`templates/signup.html` gains a date-of-birth field and a required terms checkbox whose label links `/docs/terms.html` and `/docs/community-guidelines.html`. Both are validated by `SignupForm`, so the error path is the existing global `RequestValidationError` handler that already re-renders auth pages with messages.
|
||||
|
||||
---
|
||||
|
||||
## 7. Agent, docs and SEO layer
|
||||
|
||||
Per A4, nothing ships with fewer than four faces.
|
||||
|
||||
- **Devii** - `services/devii/actions/catalog/moderation.py` exporting `MODERATION_ACTIONS`: `report_content`, `list_my_reports`, `list_reports` (admin), `decide_report` (admin), `suspend_user` (admin), `lift_suspension` (admin), `delete_my_account`, `set_consent`, `accept_terms`. `delete_my_account`, `decide_report`, `suspend_user` and `ban_user` join `CONFIRM_REQUIRED` in `dispatcher.py`, **each declaring a `confirm` boolean param in its catalog spec** - the load-bearing detail the root `CLAUDE.md` calls out, without which a gated tool loops forever.
|
||||
- **API docs** - `docs_api/groups/moderation.py`, a new group with `endpoint()` entries and `sample_response` for every route above, plus the reason enum sourced from `REPORT_REASONS`.
|
||||
- **SEO** - legal pages are public and indexable, added to `routers/seo.py`'s sitemap; `/reports/*` and `/admin/moderation/*` are `noindex,nofollow` via `base_seo_context`.
|
||||
- **Audit** - new keys in `events.md` and `services/audit/categories.py` `category_for` under a new `moderation` category: `report.create`, `report.status`, `report.decide`, `moderation.suspend`, `moderation.ban`, `moderation.lift`, `moderation.remove`, `moderation.restore`, `filter.block`, `filter.review`, `account.delete.request`, `account.delete.purge`, `consent.grant`, `consent.withdraw`, `terms.accept`.
|
||||
- **README.md** gains the moderation, legal and account-deletion surfaces; the root `CLAUDE.md` gains one new architectural rule (§8.1 below); `services/moderation/CLAUDE.md` and `routers/CLAUDE.md` carry the detail.
|
||||
|
||||
---
|
||||
|
||||
## 8. The two things that are not code
|
||||
|
||||
### 8.1 The new architectural rule for the root `CLAUDE.md`
|
||||
|
||||
> **Every user-generated surface is reportable by construction.** A new content table added to `SOFT_DELETE_TABLES` that is visible to anyone other than its author MUST be registered in `database/moderation.py` `REPORTABLE_TARGETS`, MUST resolve in `resolve_object_url`, and MUST include `_report_button.html` in its action bar. The registry completeness test enforces the first two; the template coverage test enforces the third.
|
||||
|
||||
### 8.2 The positioning change
|
||||
|
||||
`applechanges.md` §5 established that four sites currently promise an uncensored platform, and that this alone is grounds for a 1.2 rejection. The design changes them in step so that marketing, terms and behaviour state the same thing:
|
||||
|
||||
| Site | Current | Proposed |
|
||||
|------|---------|----------|
|
||||
| `main.py:744` site description | "…in an open, uncensored environment." | "…in an open environment built by developers, for developers." |
|
||||
| `templates/base.html:9` meta description | same string | same replacement |
|
||||
| `templates/landing.html:120` hero | same string | same replacement |
|
||||
| `templates/landing.html:134` feature card | "No Censorship" | "No Gatekeeping" - with body copy stating that DevPlace does not editorialise technical opinion, and that a short list of prohibited categories is enforced, linking the community guidelines |
|
||||
| `database/schema.py:280` default `site_tagline` | same string | same replacement |
|
||||
|
||||
This is the one item in this document that changes the product's public voice rather than its capabilities. It is presented as a decision, not an assumption, and it is the single change with the highest effect on the outcome of review.
|
||||
|
||||
---
|
||||
|
||||
## 9. Sequencing
|
||||
|
||||
Six phases. Each phase is independently shippable, leaves the platform working, and ends with the full suite (`make test`, all three tiers) green. No phase depends on a later one.
|
||||
|
||||
| Phase | Contents | Requirements closed |
|
||||
|-------|----------|---------------------|
|
||||
| **1. Foundation** | `database/moderation.py` registry and constants; `content_reports`, `moderation_actions`, `content_maturity`, `user_consents` tables; `users` columns; `site_settings` keys; `SOFT_DELETE_TABLES` registration; `resolve_object_url` extension; the registry completeness test | substrate for R4-R8, R13-R16 |
|
||||
| **2. Reporting and moderation** | `services/moderation/` queue; `routers/reports.py`; `routers/admin/moderation.py`; enforcement routes; `_report_button.html` at all fourteen sites; `_report_dialog.html` + `ReportDialog.js`; `admin_moderation.html` + sidebar; SLA badge; audit keys; Devii actions; API docs | **R5, R6, R7, R8, R9, P1, P2, P3** |
|
||||
| **3. Legal and contact** | The seven docs pages; footer links; contact settings; the positioning rewording | **R1, R3, R10, R11, R18, R22** |
|
||||
| **4. Consent, terms, age** | Signup terms + date of birth; re-acceptance middleware; consent routes and privacy tab; the AI gateway consent gate; activity-recording consent and indicator | **R2, R13, R15, R16, C8, C9** |
|
||||
| **5. Deletion** | `routers/profile/delete.py`; the stamped cascade; `devplace accounts prune`; devRant re-point; the confirmation page | **R12** |
|
||||
| **6. Filter, maturity, index, posture** | `services/moderation/filter.py` at the five choke points; `content_maturity` + `_maturity_gate.html`; `/workspaces/index`; IPv6 verification; demo account; review notes; questionnaire and privacy-label answers | **R4, R14, R19, R20, R21, R23, R24, C5** |
|
||||
|
||||
Phases 2 and 3 together answer the guideline that actually rejects apps. Phase 5 answers the guideline that most often rejects them on the second attempt. Nothing is deferred to "later"; six phases is the whole scope.
|
||||
|
||||
---
|
||||
|
||||
## 10. Test plan
|
||||
|
||||
Following the tier rules in `tests/CLAUDE.md`: tier is decided by fixtures, path mirrors the URL for `api`/`e2e` and the module for `unit`.
|
||||
|
||||
**`tests/unit/database/moderation.py`**
|
||||
- The registry completeness invariant (§11.1) - the single most important test in this feature.
|
||||
- `REPORT_REASONS` keys are stable and cover every guideline category.
|
||||
- `resolve_object_url` returns a non-`/feed` URL for every registry entry.
|
||||
- Filter classification: property checks over the category rule set, asserting monotonicity of score against rule matches and that `verdict` never weakens as mode strengthens.
|
||||
- Age-band derivation across the full date domain, including leap days and the exact boundary.
|
||||
|
||||
**`tests/api/reports/*.py`**
|
||||
- Report every registry target type; assert one row, correct `owner_uid`, correct audit event.
|
||||
- Duplicate report from the same reporter updates rather than duplicates; from a different reporter creates a second row.
|
||||
- Guests are refused; suspended users are refused posting but permitted reporting and deletion.
|
||||
- `/reports/mine` shows outcomes; a reporter never sees another reporter's report.
|
||||
|
||||
**`tests/api/admin/moderation.py`**
|
||||
- Queue ordering is oldest-open-first; SLA badge flips at the configured hour.
|
||||
- Every `MODERATION_ACTIONS` decision writes a `moderation_actions` row, an audit row, and a notification.
|
||||
- The seniority guard blocks a junior admin actioning a senior one and audits `result="denied"`.
|
||||
|
||||
**`tests/api/profile/delete.py`**
|
||||
- Deletion requires the correct password; wrong password does not delete.
|
||||
- After deletion the account is unreachable, sessions are revoked, content is gone from every listing.
|
||||
- Restore within the grace window from `/admin/trash` restores the whole event under one stamp.
|
||||
- After the grace window `purge_event` removes every row and no personal data remains in any table.
|
||||
|
||||
**`tests/api/auth/terms.py`, `tests/api/profile/consent.py`**
|
||||
- Signup without acceptance or below the minimum age fails with a rendered message.
|
||||
- Bumping `terms_version` forces re-acceptance on the next mutating request and never on a read.
|
||||
- A gateway user-content call without `ai_third_party` consent is refused; with consent it proceeds; withdrawal takes effect immediately.
|
||||
|
||||
**`tests/e2e/`**
|
||||
- **Coverage test:** for each of the fourteen include sites, load the page and assert a report control is present and reachable. This is the test that keeps R5 true over time.
|
||||
- Report a post end to end through the dialog; confirm the toast, the notification and the queue row.
|
||||
- Block from a comment action bar; confirm the author's content disappears from the feed.
|
||||
- Delete an account through the UI and confirm the login no longer works.
|
||||
- The maturity interstitial hides labelled content and reveals it only on explicit opt-in.
|
||||
|
||||
**Rigorous verification (root `CLAUDE.md`, four-layer procedure).** Suspension state, consent state and the deletion cascade are all read-then-write mutations reachable from HTTP, Devii, the devRant API and the CLI, so the procedure applies in full and is not optional:
|
||||
|
||||
1. **Property checks** over the filter score function and the age-band function across their whole input domain.
|
||||
2. **Stateful fuzzing** of report → decide → suspend → lift → delete sequences against a temp DB, asserting after every action that a report never leaves its state machine, a suspension never outlives its expiry, consent history is never rewritten, and no user is ever both deleted and active.
|
||||
3. **Concurrency with real separate OS processes**: concurrent decisions on one report must produce exactly one `moderation_actions` row; concurrent deletion requests must produce exactly one cascade. Both are closed with a single atomic conditional `UPDATE … WHERE` at the chokepoint, checked through `db.executable.execute(text(...)).rowcount`, per the standing rule. **Every new column added in §3.5 is written at insert time for new rows and `COALESCE`d in every precondition and arithmetic update**, because a column absent from a row's original `INSERT` is SQL `NULL`, and `NULL = 0` is `NULL`, not true - the exact trap the root `CLAUDE.md` records.
|
||||
4. **`pyflakes` / `ruff check`** on every touched file, catching the in-function import that neither a clean compile nor a clean app import would.
|
||||
|
||||
---
|
||||
|
||||
## 11. Proof of solidity
|
||||
|
||||
`apple.md` asks for a mathematical proof that the implementation is solid. A design cannot be proved correct in the abstract; what can be proved is that **coverage is total and stays total**. Three claims, each discharged by a mechanism rather than by diligence.
|
||||
|
||||
### 11.1 Claim 1 - surface coverage is total, and remains total
|
||||
|
||||
Let `U` be the set of externally-visible user-generated surfaces, `R` the set of `REPORTABLE_TARGETS` keys, `T` the set of tables in `SOFT_DELETE_TABLES`, and `V ⊆ T` those visible beyond their author.
|
||||
|
||||
The design requires `V ⊆ R` and enforces it with a unit test that computes `V` from `SOFT_DELETE_TABLES` minus an explicit, reviewed exclusion list of owner-private tables, and asserts the inclusion. A developer adding a UGC table without registering it **fails the suite**.
|
||||
|
||||
Since the report route, the report partial, the Devii action enum, the API docs enum and the admin filter all derive from `R`, coverage of every consumer follows from `V ⊆ R` by construction. Requirement **R5** is therefore not "implemented on sixteen surfaces" but *closed under future additions* - which is the only form of this guarantee worth having, because 1.2 rejections happen on the surface someone forgot.
|
||||
|
||||
Formally: coverage is the composition `V ↪ R → {route, partial, action, docs, filter}`. The inclusion is test-enforced; the maps are total functions over `R`; therefore the composition is total over `V`. ∎
|
||||
|
||||
### 11.2 Claim 2 - every Apple requirement maps to a named artifact
|
||||
|
||||
The map `requirement → artifact` below is total over the mandatory register and over every triggered conditional. No requirement lacks an artifact; no artifact exists without a requirement.
|
||||
|
||||
| Req | Artifact | Phase |
|
||||
|-----|----------|-------|
|
||||
| R1 | `/docs/terms.html` + `terms_version` | 3 |
|
||||
| R2 | `SignupForm.accept_terms`, `users.terms_version`, re-acceptance middleware | 4 |
|
||||
| R3 | `/docs/community-guidelines.html` from `REPORT_REASONS` | 3 |
|
||||
| R4 | `services/moderation/filter.py` at five choke points | 6 |
|
||||
| R5 | `REPORTABLE_TARGETS` + `/reports/{target_type}/{target_uid}` + `_report_button.html` | 1, 2 |
|
||||
| R6 | `/admin/moderation` + `moderation_actions` | 2 |
|
||||
| R7 | `moderation_sla_hours` + the SLA badge + `/docs/content-moderation.html` | 2, 3 |
|
||||
| R8 | `/admin/users/{uid}/suspend`, `/ban`, `/lift` + `is_suspended` | 2 |
|
||||
| R9 | existing `routers/relations.py` + Block in `_report_button.html` | 2 |
|
||||
| R10 | `/docs/contact.html` from `contact_*` settings + footer | 3 |
|
||||
| R11 | `/docs/privacy.html` + footer + ASC metadata | 3 |
|
||||
| R12 | `routers/profile/delete.py` + stamped cascade + `devplace accounts prune` | 5 |
|
||||
| R13 | `SignupForm.birth_date` → `users.age_band` + `moderation_minimum_age` | 4 |
|
||||
| R14 | `content_maturity` + `_maturity_gate.html` + `mature_opt_in` | 6 |
|
||||
| R15 | `user_consents.ai_third_party` + the gateway gate | 4 |
|
||||
| R16 | `POST /profile/{username}/consent` + the privacy tab | 4 |
|
||||
| R17 | existing `notification_preferences` (verified, documented) | 6 |
|
||||
| R18 | `intellectual_property` reason + `/docs/intellectual-property.html` | 3 |
|
||||
| R19 | demo account + review notes | 6 |
|
||||
| R20 | questionnaire answered from R4/R5/R6/R13 | 6 |
|
||||
| R21 | privacy labels derived from R15's disclosure | 6 |
|
||||
| R22 | `contact_*` settings ≡ ASC trader data | 3 |
|
||||
| R23 | IPv6 verification of app, nginx, WebSockets, ingress | 6 |
|
||||
| R24 | architecture statement in docs + review notes | 6 |
|
||||
| R25 | every control exposed as JSON by A4 | 1-6 |
|
||||
| C4 | contest position documented | 3 |
|
||||
| C5 | `/workspaces/index` | 6 |
|
||||
| C8 | `activity_recording` consent + indicator | 4 |
|
||||
| C9 | per-instance consent before data reaches user software | 4 |
|
||||
| P1 | `moderation_actions` + SLA metrics | 2 |
|
||||
| P2 | audit `moderation` category + `moderation_actions` | 2 |
|
||||
| P3 | statement of reasons via `create_notification` | 2 |
|
||||
| P4 | privacy-label step added to the feature workflow | 6 |
|
||||
| P5 | release-notes discipline | 6 |
|
||||
|
||||
C1, C2, C3, C6 and C7 are untriggered and the design introduces nothing that triggers them: no social login, no payment path, no purchasable randomness, no advertising, no cross-app tracking. Keeping them untriggered is itself recorded as a constraint in the root `CLAUDE.md` rule of §8.1's neighbourhood.
|
||||
|
||||
### 11.3 Claim 3 - the design introduces no inconsistency
|
||||
|
||||
Consistency is checked against every convention the repository enforces:
|
||||
|
||||
| Convention | How this design satisfies it |
|
||||
|-----------|------------------------------|
|
||||
| Polymorphic `(target_type, target_uid)` | `content_reports`, `content_maturity` use it verbatim |
|
||||
| Registry over literal | `REPORTABLE_TARGETS` beside `VOTABLE_TARGETS` |
|
||||
| Soft delete everywhere, one stamp per cascade | All four new tables registered; deletion uses one stamp |
|
||||
| Runtime policy in `site_settings` | Eleven new keys, zero new constants |
|
||||
| Four faces per route | Every route has HTML, JSON, Devii action, API docs |
|
||||
| Shared `templates` instance, partial reuse | One partial, one dialog, fourteen includes |
|
||||
| ES6 module, one class per file, on `app` | `ReportDialog.js` |
|
||||
| Design tokens, no literals | Report and SLA styling uses existing tokens and `--z-*` bands |
|
||||
| No comments, no docstrings | The design specifies none |
|
||||
| Author attribution at the top of every file | Every new file |
|
||||
| European dates, UTC storage | `local_dt` / `dt_ago` for every timestamp shown |
|
||||
| Owner-or-admin, seniority guard | `is_owner`, `_is_senior_admin` reused unchanged |
|
||||
| `CONFIRM_REQUIRED` with a declared `confirm` param | Four gated Devii tools |
|
||||
| Batch helpers, never N+1 | `get_maturity_by_targets`, denormalised `owner_uid` |
|
||||
| Never fail silently | Filter fails to `review`; report submission never swallows |
|
||||
| No forbidden name patterns, no em-dash | Enforced at authoring and by `/validate` |
|
||||
|
||||
Zero new patterns are introduced. Every mechanism in this design is an existing DevPlace mechanism applied to a new target set. That is the sense in which it is DRY, and the sense in which it is consistent. ∎
|
||||
|
||||
---
|
||||
|
||||
## 12. Verification loop
|
||||
|
||||
`apple.md` asks that the former steps be repeated recursively until the result is proved solid. Three passes were run over `applecomp.md` → `applechanges.md` → this document. Each pass fed a correction back into the earlier documents, which are the corrected versions.
|
||||
|
||||
**Pass 1 - requirement completeness.** The first register covered guideline 1.2 and 5.1.1 only. Re-reading the guidelines against the platform's actual feature list added: 4.7 in full (the AI assistant is a chatbot under it, and it restates the 1.2 quartet), 2.5.2 and its educational exception (the container platform), 2.5.14 (presence and session recording), 4.7.4 (the software index), 2.5.5 (IPv6), 5.3 (Code Farm Eras), 6.1's 2025 age-rating overhaul and 6.4's DSA trader status. **Nine requirements were missing from the first draft.** They are R23, R24, C5, C8, C9, and the metadata requirements R20-R22, and the 5.3 position in C4.
|
||||
|
||||
**Pass 2 - surface completeness.** The first gap analysis listed eight UGC surfaces from the routers. Re-deriving the list from `SOFT_DELETE_TABLES` rather than from the routers produced **twenty**, including four that a router-first reading misses entirely: awards, poll options, quiz options and workspace-served content. That correction is what forced A1 and A2, and therefore the registry, and therefore the completeness invariant of §11.1. A per-surface design would have shipped incomplete.
|
||||
|
||||
**Pass 3 - consistency and caveat elimination.** Re-reading the design against the conventions produced five corrections, each removing a caveat rather than documenting one:
|
||||
|
||||
1. Maturity was originally a column on each content table - twenty migrations and a permanent drift risk. Replaced by the polymorphic `content_maturity` table with a batch helper, matching `reactions`.
|
||||
2. The filter was originally to be called from each router - twenty call sites. Replaced by five existing choke points in `content.py`, `messages.py` and the profile/signup path.
|
||||
3. AI consent was originally a per-feature toggle, which would have needed a gate in every AI consumer. Replaced by one gate at the gateway, with the existing toggles demoted to preferences - no existing preference is flipped and no consumer changes.
|
||||
4. Account deletion was originally an immediate hard purge, which conflicts with `/admin/trash`, with the audit trail, and with accidental loss. Replaced by an immediate anonymisation plus a stamped soft-delete event and a GC purge, which is both the compliant behaviour and the behaviour the codebase already has primitives for.
|
||||
5. Legal pages were originally new routes. Replaced by `DOCS_PAGES` entries, which brings role gating, SEO, the search index and the export for free, and adds no routing.
|
||||
|
||||
**Pass 4 - factual re-verification against the source tree.** Every file reference, line number and count asserted across all three documents was re-read from the source rather than trusted. Three errors were found and corrected in place:
|
||||
|
||||
1. `SOFT_DELETE_TABLES` was stated as 46 tables in `applechanges.md` §3 and in A5 above; the real count, computed from `database/soft_delete.py`, is **44**.
|
||||
2. `applechanges.md` §8's mandatory-requirement tally summed to 26 across 25 requirements, because R2 was counted as both missing and partial. Corrected to a true partition: 1 present, 5 partial, 15 missing, 2 blocked, 1 unverified, 1 out of scope.
|
||||
3. The conditional tally said "4 not triggered … (C1, C2, C3, C6, C7 - five, counting C7)". Corrected to 5 not triggered, 3 missing, 1 borderline.
|
||||
|
||||
Everything else verified exactly: `main.py:744`, `templates/base.html:9`, `landing.html:120` and `:134`, `schema.py:276`/`:280`/`:1823`, `soft_delete.py:7`, `ranking.py:11`, `reactions.py:17`, `content.py:197`/`:361`, `database/content.py:22`, `models.py:51`/`:408`, `admin/users.py:179`, `devrant/auth.py:189`, `messages.py:245`, `notifications.py:68`, `admin_base.html:11`-`59`, and the existence of all fourteen include-site templates plus `routers/profile/index.py`, `services/audit/categories.py`, `services/devii/actions/spec.py` and `docs_api/_shared.py`.
|
||||
|
||||
**Pass 5 - fixed point.** A fifth pass over all three documents produced no further correction: every mandatory requirement maps to an artifact (§11.2), every artifact maps to a requirement, every surface is covered by construction (§11.1), and every convention is satisfied (§11.3). The documents are consistent with each other and with the source tree as read. The loop has converged.
|
||||
|
||||
**The one open decision** deliberately left to the lord, because it is a product-voice decision and not a technical one, is §8.2: the rewording of the four "uncensored" sites. Everything else in this design is fully specified and requires no further input.
|
||||
|
||||
---
|
||||
|
||||
## 13. What approval authorises
|
||||
|
||||
Approving this document authorises implementation of phases 1 through 6 in §9, in order, each phase validated with `python -c "from devplacepy.main import app"`, per-language manual checks, `ruff check` / `pyflakes` on every touched file, the four-layer rigorous verification of §10 where it applies, and the **full test suite (`make test`, all three tiers, every test) green before the phase is considered done**.
|
||||
|
||||
Documentation updated in step: `README.md`, the root `CLAUDE.md` (one new rule, §8.1), `devplacepy/routers/CLAUDE.md`, a new `devplacepy/services/moderation/CLAUDE.md`, `devplacepy/database/CLAUDE.md`, `devplacepy/templates/CLAUDE.md`, `events.md`, and the seven new docs pages.
|
||||
@ -12,9 +12,7 @@ PRAGMA cache_size=-8000; -- 8MB page cache
|
||||
PRAGMA temp_store=MEMORY; -- temp tables in memory
|
||||
```
|
||||
|
||||
Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}, "poolclass": NullPool}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.
|
||||
|
||||
**`poolclass=NullPool` is load-bearing - never revert it to SQLAlchemy's default `QueuePool` (caused a production outage).** `dataset.Database.executable` caches ONE DBAPI connection per OS thread ID **forever** and never returns it to the pool except via `db.close()`, which nothing in this codebase calls (`dataset/database.py`: `self.connections[tid] = self.engine.connect()`). That is fine as long as the same handful of threads ever touch the DB - but FastAPI runs every sync route dependency (`get_setting` and friends, hit on nearly every request) through `anyio.to_thread.run_sync`, whose worker pool scales up and recycles threads elastically under load, and container sync (`asyncio.to_thread`) adds more. Each new thread's first query permanently claims one pool slot. With the default bounded `QueuePool` (`pool_size=5, max_overflow=10` = 15 total), a burst of concurrent load creates enough new threads that the pool fills for good within minutes, and every request thereafter - including the Docker healthcheck's own probe - blocks the full 30s pool timeout and then raises `sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached`, wedging the whole app (nginx waits on an app that is waiting on itself; every external caller sees a bare connection timeout, not an HTTP error). `NullPool` removes the artificial ceiling: each `engine.connect()` opens a real, unpooled SQLite connection, so the existing "one connection cached per thread forever" behavior just works, exactly as WAL mode is designed to support. Never pass `pool_size`/`max_overflow` alongside `NullPool` (SQLAlchemy rejects them). Do not "fix" the underlying thread churn instead - that means touching the sync-dependency/threadpool model, which the hard rule below forbids.
|
||||
Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.
|
||||
|
||||
`init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` (through the `_index()` helper, wrapped in try/except so it is safe to run on every startup regardless of table state), and seed defaults for `site_settings` only insert when the key doesn't already exist.
|
||||
|
||||
|
||||
@ -4,7 +4,6 @@ import dataset
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.pool import NullPool
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from devplacepy.cache import TTLCache
|
||||
@ -31,7 +30,6 @@ db = dataset.connect(
|
||||
"timeout": 30,
|
||||
"check_same_thread": False,
|
||||
},
|
||||
"poolclass": NullPool,
|
||||
},
|
||||
on_connect_statements=[
|
||||
"PRAGMA journal_mode=WAL",
|
||||
|
||||
@ -22,8 +22,7 @@ workspace start.
|
||||
|
||||
A **tunnel** publishes one port from inside your container on a public HTTPS hostname of the form
|
||||
`<port>-<name>.tunnel.pravda.education`. **Tunnel URLs are public and unauthenticated** - anyone with
|
||||
the link can reach whatever you are serving. Forwarding a port in the editor's **Ports** view creates
|
||||
the tunnel for you through the same endpoint; un-forwarding it does not remove the tunnel.
|
||||
the link can reach whatever you are serving.
|
||||
|
||||
Workspaces are bounded: a count limit per user, a disk quota, an egress quota, and a tunnel limit.
|
||||
An idle workspace is warned about, then stopped, then warned again, then removed. Every warning
|
||||
@ -223,9 +222,7 @@ arrives as a `workspace` notification and states exactly what happens next and w
|
||||
title="Create tunnel",
|
||||
summary=(
|
||||
"Publish a container port on a public HTTPS hostname. The URL is public "
|
||||
"and unauthenticated. Refused past the tunnel limit. The certificate is "
|
||||
"ordered right away, so the hostname answers plain HTTP for a few seconds "
|
||||
"before it serves HTTPS. Forwarding a port in the editor calls this for you."
|
||||
"and unauthenticated. Refused past the tunnel limit."
|
||||
),
|
||||
auth="user",
|
||||
params=[
|
||||
|
||||
@ -523,15 +523,6 @@ async def await_pending_corrections(request: Request, call_next):
|
||||
return response
|
||||
|
||||
|
||||
def _frame_ancestors() -> str:
|
||||
from devplacepy.services.containers.workspace import naming
|
||||
|
||||
tunnel_domain = naming.domain()
|
||||
if not tunnel_domain:
|
||||
return "'self'"
|
||||
return f"'self' https://*.{tunnel_domain}"
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def add_security_headers(request: Request, call_next):
|
||||
response = await call_next(request)
|
||||
@ -541,9 +532,10 @@ async def add_security_headers(request: Request, call_next):
|
||||
response.headers["Strict-Transport-Security"] = "max-age=31536000; includeSubDomains"
|
||||
response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin"
|
||||
if not request.url.path.startswith("/p/"):
|
||||
response.headers["X-Frame-Options"] = "DENY"
|
||||
response.headers["Content-Security-Policy"] = (
|
||||
"object-src 'none'; base-uri 'self'; "
|
||||
f"frame-ancestors {_frame_ancestors()}; form-action 'self'"
|
||||
"frame-ancestors 'none'; form-action 'self'"
|
||||
)
|
||||
if request.url.path.startswith("/admin"):
|
||||
response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate, max-age=0"
|
||||
|
||||
@ -243,12 +243,16 @@ async def tunnel_create(
|
||||
instance = _workspace_or_404(project, user)
|
||||
if not can_manage_workspace(instance, project, user):
|
||||
return json_error(403, "Not allowed to manage this workspace")
|
||||
try:
|
||||
row = provision.publish_tunnel(
|
||||
instance, data.label, data.container_port, user["uid"]
|
||||
)
|
||||
except provision.WorkspaceError as error:
|
||||
return json_error(400, str(error))
|
||||
if data.container_port <= 0:
|
||||
return json_error(400, "container_port must be between 1 and 65535")
|
||||
limits = quota.resolve(instance.get("workspace_owner_uid", ""), instance)
|
||||
if limits.max_tunnels and tunnels.count_for_instance(
|
||||
instance["uid"]
|
||||
) >= limits.max_tunnels:
|
||||
return json_error(400, f"tunnel limit reached ({limits.max_tunnels})")
|
||||
row = tunnels.create(instance, data.label, data.container_port, user["uid"])
|
||||
if not row:
|
||||
return json_error(400, "could not create tunnel")
|
||||
audit_instance(
|
||||
request,
|
||||
user,
|
||||
@ -257,6 +261,7 @@ async def tunnel_create(
|
||||
project,
|
||||
metadata={"hostname": row["hostname"], "port": data.container_port},
|
||||
)
|
||||
provision.write_manifest(instance)
|
||||
return action_result(request, f"/projects/{slug}/workspace", data=row)
|
||||
|
||||
|
||||
|
||||
@ -27,8 +27,18 @@ def resolve(host: str):
|
||||
return row, instance, None, None
|
||||
if instance.get("status") != store.ST_RUNNING:
|
||||
return row, instance, None, None
|
||||
host, port = api.tunnel_target(instance, int(row.get("container_port") or 0))
|
||||
return row, instance, host, port
|
||||
gateway, _ = api.proxy_target(instance)
|
||||
host_port = _published_host_port(instance, int(row.get("container_port") or 0))
|
||||
return row, instance, gateway, host_port
|
||||
|
||||
|
||||
def _published_host_port(instance: dict, container_port: int) -> int:
|
||||
import json
|
||||
|
||||
for mapping in json.loads(instance.get("ports_json") or "[]"):
|
||||
if int(mapping.get("container") or 0) == container_port:
|
||||
return int(mapping.get("host") or 0)
|
||||
return 0
|
||||
|
||||
|
||||
async def handle_http(request: Request, path: str) -> Response:
|
||||
|
||||
@ -158,10 +158,10 @@ class WorkspaceViewOut(_Out):
|
||||
status: str = ""
|
||||
desired_state: str = ""
|
||||
suspended: bool = False
|
||||
flag_reason: Optional[str] = ""
|
||||
tunnel_name: Optional[str] = ""
|
||||
primary_url: Optional[str] = ""
|
||||
last_active_at: Optional[str] = ""
|
||||
flag_reason: str = ""
|
||||
tunnel_name: str = ""
|
||||
primary_url: str = ""
|
||||
last_active_at: str = ""
|
||||
disk_bytes: int = 0
|
||||
disk_quota_mb: int = 0
|
||||
disk_percent: int = 0
|
||||
|
||||
@ -263,7 +263,7 @@ The security hotpatch that used to run per build is now baked into `ppy.Dockerfi
|
||||
|
||||
**Trade-off (intentional).** The only genuinely-root operation that still does NOT escalate is binding a port < 1024 - use a high port + `/p/<slug>` ingress instead. Enforcement lives entirely in the Dockerfile (no `--user` on `docker run`). The `export_to_dir` unlink-before-write fix remains as belt-and-suspenders (the app owns the workspace dir, so it may delete any stale file in it regardless of owner before rewriting it).
|
||||
|
||||
## `DEVPLACE_*` runtime env injection (function name `pravda_env`)
|
||||
## `PRAVDA_*` runtime env injection
|
||||
|
||||
`api.run_spec_for` merges `api.pravda_env(instance)` over the instance's own `env_json` (PRAVDA keys win), so every running container gets these platform vars:
|
||||
- `DEVPLACE_BASE_URL` - the `site_url` setting via `seo.public_base_url()`.
|
||||
@ -326,9 +326,9 @@ Use `FakeBackend` (its `image_exists` returns `True`) + `runtime.set_backend`, a
|
||||
|
||||
## Vibe coding on-ramp (user-facing doc)
|
||||
|
||||
The container runtime is also the basis of "vibe coding": the public prose page `templates/docs/getting-started-vibing.html` (slug `getting-started-vibing`, `SECTION_GENERAL`, not admin-gated so the docs stay public while the feature is admin-only/Alpha) is the canonical user-facing guide. It teaches the Devii-driven flow project -> container -> start -> terminal (`create_project`, `container_create_instance`, `container_instance_action`, the `open_terminal` client action), documents the three baked-in agents (`dpc` = DevPlace Code at `/usr/bin/dpc`, the Claude-Code-class coding agent; `botje.py` = the copy of `services/containers/files/bot.py` at `/usr/bin/botje.py`; `pagent`), all metered through the container's own `DEVPLACE_API_KEY`, the full `DEVPLACE_*` env table (see `api.pravda_env`), and ingress at `/p/<slug>` via `ingress_slug`/`ingress_port`.
|
||||
The container runtime is also the basis of "vibe coding": the public prose page `templates/docs/getting-started-vibing.html` (slug `getting-started-vibing`, `SECTION_GENERAL`, not admin-gated so the docs stay public while the feature is admin-only/Alpha) is the canonical user-facing guide. It teaches the Devii-driven flow project -> container -> start -> terminal (`create_project`, `container_create_instance`, `container_instance_action`, the `open_terminal` client action), documents the three baked-in agents (`dpc` = DevPlace Code at `/usr/bin/dpc`, the Claude-Code-class coding agent; `botje.py` = the copy of `services/containers/files/bot.py` at `/usr/bin/botje.py`; `pagent`), all metered through the container's own `DEVPLACE_API_KEY`, the full `PRAVDA_*` env table (see `api.pravda_env`), and ingress at `/p/<slug>` via `ingress_slug`/`ingress_port`.
|
||||
|
||||
When the runtime, the agent binaries, or the `DEVPLACE_*`/ingress contract change, update this page alongside the source.
|
||||
When the runtime, the agent binaries, or the `PRAVDA_*`/ingress contract change, update this page alongside the source.
|
||||
|
||||
**The agents are gateway-only:** `dpc`/`d.py` and `botje.py`/`bot.py` use a single `molodetz` backend pointed at `DEVPLACE_OPENAI_URL` (the gateway); the former direct `api.deepseek.com` fallback backend was removed so every in-container AI call is ledgered under the run-as user and nothing bypasses `gateway_usage_ledger`. `pagent`/`.vimrc` already posted to the gateway URL (using `DEEPSEEK_API_KEY` only as a key fallback, never the DeepSeek endpoint). Rebuild the image (`make ppy`) for the change to reach running containers.
|
||||
|
||||
@ -478,69 +478,6 @@ schedules or tracks renewals. This whole phase did not exist - `tunnels` had no
|
||||
the six `workspace_molohttp_*`/`workspace_cert_mode`/`workspace_acme_email` settings were admin
|
||||
fields wired to nothing, which is why every tunnel sat at `pending` with no certificate.
|
||||
|
||||
**`provision.publish_tunnel` is the ONE way a user-created tunnel comes into existence** - the HTTP
|
||||
route, the Devii tool and the editor all funnel through it. It owns the port check, the `max_tunnels`
|
||||
quota, `tunnels.create`, `schedule_certificate` and `write_manifest`, and raises `WorkspaceError` for
|
||||
every refusal. Before it existed, the route and the Devii controller each carried their own copy of
|
||||
the quota check and **neither ordered a certificate**, so a user-created tunnel sat `pending` until
|
||||
the (default-disabled) `WorkspaceService` happened to tick - which on an instance where that service
|
||||
was never enabled is forever. `schedule_certificate` now also writes `provision.CERT_UNCONFIGURED`
|
||||
into the row's `last_error` when molohttp is not configured, because a tunnel that can never be
|
||||
certified must say so on the workspace page rather than sit at `pending` with a blank error.
|
||||
|
||||
**A tunnel reaches its port through `api.tunnel_target(instance, container_port)`, never through
|
||||
`proxy_target`.** `proxy_target` answers for `/p/{slug}`, whose port is published by construction;
|
||||
a tunnel's port is whatever the member decided to serve on and is almost never published, because a
|
||||
workspace publishes only `editor_port`. `tunnel_target` therefore prefers the published host port
|
||||
when the port happens to have one (`CONTAINER_PROXY_HOST` or the recorded gateway, exactly like
|
||||
`proxy_target`) and otherwise dials `container_ip:container_port` directly. The direct leg is what
|
||||
makes an arbitrary port tunnellable at all: docker cannot add a published port to a running
|
||||
container, so publishing on demand would mean recreating the container and killing the very dev
|
||||
server the member just asked to share.
|
||||
|
||||
**The direct leg needs the app on the same docker network as the instances, and that wiring cannot
|
||||
live in compose.** Measured on this host: from the app container, `container_ip:port` times out
|
||||
(docker's inter-network isolation) while `gateway:published_host_port` connects; from the host, and
|
||||
from any container sharing the instances' network, `container_ip:port` connects. So `make dev` works
|
||||
untouched and the containerized production app does not - it must be attached to the network the
|
||||
instances run on. Compose cannot express that: it always sends network-scoped aliases, which the
|
||||
default `bridge` rejects (`invalid endpoint settings: network-scoped aliases are only supported for
|
||||
user-defined networks`). The attachment is therefore a `make docker-attach` step, run by
|
||||
`docker-up` and `docker-reload` and idempotent, deriving its input like `DOCKER_GID` does
|
||||
(`DEVPLACE_CONTAINER_NETWORK`, default `bridge`). A bare `docker compose up -d` skips it and
|
||||
silently re-breaks every unpublished-port tunnel - one more reason the make targets are the only
|
||||
supported path.
|
||||
|
||||
**Forwarding a port in the editor publishes it, and that is the whole point of `VSCODE_PROXY_URI`.**
|
||||
`api.workspace_env` advertises `https://{{port}}-{name}.{domain}` to VS Code, so the Ports view shows
|
||||
a DevPlace address for every forwarded port - but VS Code never tells DevPlace, so that address had
|
||||
no `tunnels` row, was 404ed by `routers/tunnel.py` and never got a certificate. The editor promised a
|
||||
URL the platform could not serve. The `Tunnels` stage in the workspace extension closes it: it
|
||||
subscribes to `vscode.workspace.onDidChangeTunnels`, reads `vscode.workspace.tunnels`, and POSTs each
|
||||
new `remoteAddress.port` (skipping `DEVPLACE_EDITOR_PORT`, which is already published) to
|
||||
`{DEVPLACE_BASE_URL}/projects/{DEVPLACE_PROJECT_SLUG}/workspace/tunnels` with the container's own
|
||||
`DEVPLACE_API_KEY` and `Accept: application/json`. Four things about it:
|
||||
|
||||
- **`tunnels` is a proposed API** (`checkProposedApiEnabled(extension, 'tunnels')`), so the extension
|
||||
declares `enabledApiProposals: ["tunnels"]` and `product.patch.json` names it under
|
||||
`extensionEnabledApiProposals`. code-server patches the check to always pass, so it works today
|
||||
either way; the declarations are what keep it working if that patch goes away. Because the patch
|
||||
adds a nested object, the Dockerfile's `product.json` merge now merges one level deep - a plain
|
||||
`dict.update` would wipe an upstream map of the same name on a version bump.
|
||||
- **It only ever creates.** Un-forwarding a port leaves the tunnel standing, because deleting it
|
||||
would revoke the certificate and a re-forward would re-issue, churning against Let's Encrypt's
|
||||
duplicate-certificate limit. Removal stays the explicit act it already was.
|
||||
- **A port is added to the in-memory `published` set before the POST and removed again on failure**,
|
||||
so a burst of change events cannot double-post and a refusal (quota, 403) can still retry on the
|
||||
next change. Refusals surface both in the `DevPlace` output channel and as a warning message.
|
||||
- **It uses `http`/`https` from Node, not `fetch`**, and is wrapped in the same `stage()` try/catch as
|
||||
every other activation step - a workspace whose network is down must still open its editor.
|
||||
|
||||
Verified against the real image by driving code-server with Playwright and forwarding port 3000: the
|
||||
Ports view lists the port, the extension POSTs `label=Port+3000&container_port=3000` with the API key,
|
||||
and the output channel reports the public URL. Reproduce it that way, not with a mock - the Ports
|
||||
view is the only trigger, there is no `Forward a Port` command in the palette in code-server.
|
||||
|
||||
**Two contracts that bite:**
|
||||
- A `ConfigField` with `type="select"` needs `options=[{"value": ..., "label": ...}]`. Plain strings
|
||||
crash `docs_api.build_services_group`, which `docs_search` indexes, so the whole docs search page
|
||||
|
||||
@ -589,7 +589,7 @@ def run_spec_for(instance: dict, image_tag: str) -> RunSpec:
|
||||
language = (instance.get("boot_language") or "none").strip().lower()
|
||||
boot = (instance.get("boot_command") or "").strip()
|
||||
if profile and int(instance.get("editor_port") or 0):
|
||||
command = editor.wrap_with_env_export(editor.argv(instance, profile))
|
||||
command = editor.argv(instance, profile)
|
||||
elif language in BOOT_SCRIPT_FILES and (instance.get("boot_script") or "").strip():
|
||||
script_path = f"{WORKSPACE_MOUNT}/{BOOT_SCRIPT_FILES[language]}"
|
||||
command = [BOOT_SCRIPT_RUNNERS[language], script_path]
|
||||
@ -732,30 +732,18 @@ def _host_port_for(port_maps: list, container_port: int) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def reachable_target(instance: dict, container_port: int, port_maps: list) -> tuple:
|
||||
container_ip = (instance.get("container_ip") or "").strip()
|
||||
if container_ip and container_port > 0:
|
||||
return container_ip, container_port
|
||||
host_port = _host_port_for(port_maps, container_port)
|
||||
if not host_port:
|
||||
return None, None
|
||||
gateway = (instance.get("container_gateway") or "").strip()
|
||||
return config.CONTAINER_PROXY_HOST or gateway or "127.0.0.1", host_port
|
||||
|
||||
|
||||
def proxy_target(instance: dict) -> tuple:
|
||||
port_maps = json.loads(instance.get("ports_json") or "[]")
|
||||
container_port = _ingress_container_port(instance, port_maps)
|
||||
if not container_port:
|
||||
return None, None
|
||||
return reachable_target(instance, container_port, port_maps)
|
||||
|
||||
|
||||
def tunnel_target(instance: dict, container_port: int) -> tuple:
|
||||
if container_port <= 0:
|
||||
host_port = _host_port_for(port_maps, container_port)
|
||||
if config.CONTAINER_PROXY_HOST:
|
||||
return (config.CONTAINER_PROXY_HOST, host_port) if host_port else (None, None)
|
||||
if not host_port:
|
||||
return None, None
|
||||
port_maps = json.loads(instance.get("ports_json") or "[]")
|
||||
return reachable_target(instance, container_port, port_maps)
|
||||
gateway = (instance.get("container_gateway") or "").strip()
|
||||
return (gateway or "127.0.0.1", host_port)
|
||||
|
||||
|
||||
def instance_runtime(instance: dict) -> dict:
|
||||
|
||||
@ -52,7 +52,7 @@ def _resolve_devplace_url() -> str:
|
||||
base = os.environ.get("DEVPLACE_BASE_URL", "").strip().rstrip("/")
|
||||
if base:
|
||||
return base
|
||||
return os.environ.get("DEVPLACE_URL", "").strip().rstrip("/")
|
||||
return os.environ.get("DEVPLACE_URL", "https://devplace.net").strip().rstrip("/")
|
||||
|
||||
|
||||
def _resolve_llm_endpoint() -> str:
|
||||
@ -63,7 +63,11 @@ def _resolve_llm_endpoint() -> str:
|
||||
|
||||
|
||||
DEVPLACE_URL = _resolve_devplace_url()
|
||||
DEVPLACE_API_KEY = os.environ.get("DEVPLACE_API_KEY", "").strip()
|
||||
DEVPLACE_API_KEY = (
|
||||
os.environ.get("DEVPLACE_API_KEY")
|
||||
or os.environ.get("DEVPLACE_API_KEY")
|
||||
or "019ea58c-fae0-7112-8025-e629a54104a4"
|
||||
)
|
||||
MENTION_POLL_SECONDS = int(os.environ.get("MENTION_POLL_SECONDS", "30"))
|
||||
DM_POLL_SECONDS = int(os.environ.get("DM_POLL_SECONDS", "10"))
|
||||
BOT_USERNAME = os.environ.get("BOT_USERNAME", "")
|
||||
@ -2808,17 +2812,8 @@ async def _agent_answer_for_devplace(
|
||||
async def devplace_bot_loop() -> None:
|
||||
"""Run the DevPlace bot: poll mentions and DMs forever."""
|
||||
logger.info("Botje starting — DevPlace bot with full X-agent capabilities")
|
||||
|
||||
if not DEVPLACE_URL or not DEVPLACE_API_KEY:
|
||||
logger.error(
|
||||
"DEVPLACE_BASE_URL and DEVPLACE_API_KEY are not set. Both are injected "
|
||||
"automatically inside a DevPlace-managed container; set them manually "
|
||||
"only when running botje.py outside one.",
|
||||
)
|
||||
return
|
||||
|
||||
logger.info("DevPlace URL: %s", DEVPLACE_URL)
|
||||
logger.info("API key: %s...", DEVPLACE_API_KEY[:12])
|
||||
logger.info("API key: %s...", DEVPLACE_API_KEY[:12] if DEVPLACE_API_KEY else "(none)")
|
||||
|
||||
dp = DevPlace(DEVPLACE_URL, DEVPLACE_API_KEY)
|
||||
|
||||
|
||||
Binary file not shown.
@ -1,8 +1,6 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
|
||||
const fs = require("fs");
|
||||
const http = require("http");
|
||||
const https = require("https");
|
||||
const vscode = require("vscode");
|
||||
|
||||
const AGENT_PATH = "/usr/bin/dpc";
|
||||
@ -10,7 +8,6 @@ const AGENT_TERMINAL = "DevPlace Code";
|
||||
const SHELL_TERMINAL = "pravda@workspace";
|
||||
const BOOT_KEY = "devplace.bootMarker";
|
||||
const PANEL_STEPS = { short: 0, normal: 2, tall: 5, maximized: 0 };
|
||||
const PUBLISH_TIMEOUT_MS = 20000;
|
||||
|
||||
class Profile {
|
||||
constructor() {
|
||||
@ -93,8 +90,7 @@ class BootTerminals {
|
||||
createAgent() {
|
||||
return vscode.window.createTerminal({
|
||||
name: AGENT_TERMINAL,
|
||||
shellPath: "/bin/bash",
|
||||
shellArgs: ["-l", "-c", `exec ${AGENT_PATH}`],
|
||||
shellPath: AGENT_PATH,
|
||||
iconPath: new vscode.ThemeIcon("rocket"),
|
||||
isTransient: false,
|
||||
});
|
||||
@ -230,136 +226,6 @@ class Presence {
|
||||
}
|
||||
}
|
||||
|
||||
class Tunnels {
|
||||
constructor(output) {
|
||||
this.output = output;
|
||||
this.published = new Set();
|
||||
this.base = (process.env.DEVPLACE_BASE_URL || "").replace(/\/+$/, "");
|
||||
this.apiKey = process.env.DEVPLACE_API_KEY || "";
|
||||
this.slug = process.env.DEVPLACE_PROJECT_SLUG || "";
|
||||
this.editorPort = Number(process.env.DEVPLACE_EDITOR_PORT || 0);
|
||||
}
|
||||
|
||||
get configured() {
|
||||
return Boolean(this.base && this.apiKey && this.slug);
|
||||
}
|
||||
|
||||
async watch(context) {
|
||||
if (!this.configured) {
|
||||
this.output.appendLine(
|
||||
"tunnels: this workspace has no DevPlace credentials, so forwarded ports stay private",
|
||||
);
|
||||
return;
|
||||
}
|
||||
context.subscriptions.push(
|
||||
vscode.workspace.onDidChangeTunnels(() =>
|
||||
this.sync().catch((error) =>
|
||||
this.output.appendLine(`tunnels: sync failed: ${error}`),
|
||||
),
|
||||
),
|
||||
);
|
||||
await this.sync();
|
||||
}
|
||||
|
||||
async sync() {
|
||||
const rows = (await vscode.workspace.tunnels) || [];
|
||||
for (const row of rows) {
|
||||
const port = Number((row.remoteAddress || {}).port || 0);
|
||||
if (!port || port === this.editorPort) continue;
|
||||
if (this.published.has(port)) continue;
|
||||
await this.publish(port);
|
||||
}
|
||||
}
|
||||
|
||||
async publish(port) {
|
||||
this.published.add(port);
|
||||
let answer;
|
||||
try {
|
||||
answer = await this.post(port);
|
||||
} catch (error) {
|
||||
this.published.delete(port);
|
||||
this.output.appendLine(`tunnels: port ${port} could not be published: ${error}`);
|
||||
return;
|
||||
}
|
||||
if (answer.status >= 400) {
|
||||
this.published.delete(port);
|
||||
this.output.appendLine(
|
||||
`tunnels: DevPlace refused port ${port} (${answer.status}): ${answer.body.slice(0, 300)}`,
|
||||
);
|
||||
vscode.window.showWarningMessage(
|
||||
`DevPlace could not publish port ${port}: ${this.refusal(answer.body)}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
const url = this.publishedUrl(answer.body, port);
|
||||
this.output.appendLine(`tunnels: port ${port} is published at ${url}`);
|
||||
vscode.window.showInformationMessage(
|
||||
`Port ${port} is published at ${url}. It serves HTTPS once its certificate is issued.`,
|
||||
);
|
||||
}
|
||||
|
||||
post(port) {
|
||||
const url = new URL(
|
||||
`${this.base}/projects/${encodeURIComponent(this.slug)}/workspace/tunnels`,
|
||||
);
|
||||
const body = new URLSearchParams({
|
||||
label: `Port ${port}`,
|
||||
container_port: String(port),
|
||||
}).toString();
|
||||
const client = url.protocol === "https:" ? https : http;
|
||||
const options = {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/x-www-form-urlencoded",
|
||||
"Content-Length": Buffer.byteLength(body),
|
||||
Accept: "application/json",
|
||||
"X-API-KEY": this.apiKey,
|
||||
},
|
||||
timeout: PUBLISH_TIMEOUT_MS,
|
||||
};
|
||||
return new Promise((resolve, reject) => {
|
||||
const call = client.request(url, options, (response) => {
|
||||
const chunks = [];
|
||||
response.on("data", (chunk) => chunks.push(chunk));
|
||||
response.on("end", () =>
|
||||
resolve({
|
||||
status: response.statusCode,
|
||||
body: Buffer.concat(chunks).toString("utf8"),
|
||||
}),
|
||||
);
|
||||
});
|
||||
call.on("timeout", () => call.destroy(new Error("request timed out")));
|
||||
call.on("error", reject);
|
||||
call.end(body);
|
||||
});
|
||||
}
|
||||
|
||||
refusal(body) {
|
||||
try {
|
||||
const parsed = JSON.parse(body);
|
||||
return (parsed.error && parsed.error.message) || "the request was refused";
|
||||
} catch (error) {
|
||||
return "the request was refused";
|
||||
}
|
||||
}
|
||||
|
||||
publishedUrl(body, port) {
|
||||
try {
|
||||
const parsed = JSON.parse(body);
|
||||
const hostname = parsed.data && parsed.data.hostname;
|
||||
if (hostname) return `https://${hostname}`;
|
||||
} catch (error) {
|
||||
/* fall through to the pattern below */
|
||||
}
|
||||
const pattern =
|
||||
process.env.DEVPLACE_TUNNEL_PORT_PATTERN || "{port}-{name}.{domain}";
|
||||
return `https://${pattern
|
||||
.replace("{port}", String(port))
|
||||
.replace("{name}", process.env.DEVPLACE_TUNNEL_NAME || "")
|
||||
.replace("{domain}", process.env.DEVPLACE_TUNNEL_DOMAIN || "")}`;
|
||||
}
|
||||
}
|
||||
|
||||
async function stage(output, name, run) {
|
||||
try {
|
||||
return await run();
|
||||
@ -379,7 +245,6 @@ async function activate(context) {
|
||||
new BootTerminals(profile, context.workspaceState).open(),
|
||||
);
|
||||
await stage(output, "layout", () => new Layout(profile).apply(Boolean(opened)));
|
||||
await stage(output, "tunnels", () => new Tunnels(output).watch(context));
|
||||
}
|
||||
|
||||
function deactivate() {}
|
||||
|
||||
@ -18,9 +18,6 @@
|
||||
"activationEvents": [
|
||||
"onStartupFinished"
|
||||
],
|
||||
"enabledApiProposals": [
|
||||
"tunnels"
|
||||
],
|
||||
"capabilities": {
|
||||
"untrustedWorkspaces": {
|
||||
"supported": true
|
||||
|
||||
@ -9,10 +9,5 @@
|
||||
"privacyStatementUrl": "https://pravda.education/docs/privacy.html",
|
||||
"twitterUrl": "",
|
||||
"requestFeatureUrl": "https://pravda.education/issues",
|
||||
"licenseName": "DevPlace Terms of Service",
|
||||
"extensionEnabledApiProposals": {
|
||||
"devplace.devplace-workspace": [
|
||||
"tunnels"
|
||||
]
|
||||
}
|
||||
"licenseName": "DevPlace Terms of Service"
|
||||
}
|
||||
|
||||
@ -37,8 +37,6 @@ RESPONSE_HOP_HEADERS = {
|
||||
"trailers",
|
||||
"transfer-encoding",
|
||||
"upgrade",
|
||||
"date",
|
||||
"server",
|
||||
}
|
||||
|
||||
WS_HANDSHAKE_HEADERS = {
|
||||
|
||||
@ -4,7 +4,6 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import shlex
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
@ -384,22 +383,6 @@ def argv(instance: dict, profile: EditorProfile) -> list[str]:
|
||||
return command
|
||||
|
||||
|
||||
ENV_EXPORT_FILE = "/etc/profile.d/devplace-env.sh"
|
||||
|
||||
_ENV_EXPORT_SCRIPT = (
|
||||
"import os, pathlib, shlex\n"
|
||||
f"path = pathlib.Path({ENV_EXPORT_FILE!r})\n"
|
||||
"lines = ['export ' + k + '=' + shlex.quote(v) for k, v in sorted(os.environ.items()) if k.startswith('DEVPLACE_')]\n"
|
||||
"path.write_text('\\n'.join(lines) + '\\n' if lines else '')\n"
|
||||
)
|
||||
|
||||
|
||||
def wrap_with_env_export(command: list[str]) -> list[str]:
|
||||
export_step = f"umask 022; python3 -c {shlex.quote(_ENV_EXPORT_SCRIPT)} 2>/dev/null || true"
|
||||
script = f"{export_step}; exec {shlex.join(command)}"
|
||||
return ["/bin/sh", "-c", script]
|
||||
|
||||
|
||||
def env_for(profile: EditorProfile) -> dict:
|
||||
return {
|
||||
"DEVPLACE_EDITOR_APP_NAME": APP_NAME,
|
||||
|
||||
@ -13,10 +13,6 @@ from . import editor, flags, naming, quota, tunnels
|
||||
|
||||
MANIFEST_DIRECTORY = ".devplace"
|
||||
MANIFEST_NAME = "tunnels.json"
|
||||
CERT_UNCONFIGURED = (
|
||||
"certificate issuance is not configured; an administrator must set the "
|
||||
"molohttp base URL and credentials before this address serves HTTPS"
|
||||
)
|
||||
|
||||
_pending_certificates: set[asyncio.Task] = set()
|
||||
|
||||
@ -73,31 +69,10 @@ async def ensure(project: dict, user: dict) -> dict:
|
||||
return store.get_instance(instance["uid"])
|
||||
|
||||
|
||||
def publish_tunnel(
|
||||
instance: dict, label: str, container_port: int, owner_uid: str
|
||||
) -> dict:
|
||||
if container_port <= 0 or container_port > 65535:
|
||||
raise WorkspaceError("container_port must be between 1 and 65535")
|
||||
limits = quota.resolve(instance.get("workspace_owner_uid", ""), instance)
|
||||
if limits.max_tunnels and tunnels.count_for_instance(
|
||||
instance["uid"]
|
||||
) >= limits.max_tunnels:
|
||||
raise WorkspaceError(f"tunnel limit reached ({limits.max_tunnels})")
|
||||
row = tunnels.create(instance, label, container_port, owner_uid)
|
||||
if not row:
|
||||
raise WorkspaceError("could not create tunnel")
|
||||
schedule_certificate(row)
|
||||
write_manifest(instance)
|
||||
return tunnels.get(row["uid"]) or row
|
||||
|
||||
|
||||
def schedule_certificate(tunnel: dict | None) -> bool:
|
||||
from . import certs
|
||||
|
||||
if not tunnel or tunnels.keeps_certificate(tunnel):
|
||||
return False
|
||||
if not certs.configured():
|
||||
tunnels.update(tunnel["uid"], {"last_error": CERT_UNCONFIGURED})
|
||||
if not tunnel or not certs.configured():
|
||||
return False
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
@ -175,8 +150,8 @@ def unsuspend(instance: dict) -> dict:
|
||||
|
||||
|
||||
def editor_target(instance: dict) -> tuple[str, int]:
|
||||
port = int(instance.get("editor_port") or api.EDITOR_DEFAULT_PORT)
|
||||
return api.tunnel_target(instance, port)
|
||||
host, port = api.proxy_target(instance)
|
||||
return host, port
|
||||
|
||||
|
||||
def manifest_payload(instance: dict) -> dict:
|
||||
|
||||
@ -57,10 +57,6 @@ def count_for_instance(instance_uid: str) -> int:
|
||||
return _table().count(instance_uid=instance_uid, deleted_at=None)
|
||||
|
||||
|
||||
def keeps_certificate(row: dict) -> bool:
|
||||
return row.get("deleted_at") is None and row.get("status") == STATUS_ACTIVE
|
||||
|
||||
|
||||
def create(
|
||||
instance: dict, label: str, container_port: int, user_uid: str
|
||||
) -> dict | None:
|
||||
@ -72,22 +68,23 @@ def create(
|
||||
revived = table.find_one(hostname=hostname)
|
||||
stamp = _now()
|
||||
if revived:
|
||||
changes = {
|
||||
"uid": revived["uid"],
|
||||
"instance_uid": instance["uid"],
|
||||
"project_uid": instance.get("project_uid", ""),
|
||||
"user_uid": user_uid,
|
||||
"label": label or f"port {container_port}",
|
||||
"container_port": container_port,
|
||||
"desired_state": "present",
|
||||
"updated_at": stamp,
|
||||
"deleted_at": None,
|
||||
"deleted_by": None,
|
||||
}
|
||||
if not keeps_certificate(revived):
|
||||
changes["status"] = STATUS_PENDING
|
||||
changes["last_error"] = ""
|
||||
table.update(changes, ["uid"])
|
||||
table.update(
|
||||
{
|
||||
"uid": revived["uid"],
|
||||
"instance_uid": instance["uid"],
|
||||
"project_uid": instance.get("project_uid", ""),
|
||||
"user_uid": user_uid,
|
||||
"label": label or f"port {container_port}",
|
||||
"container_port": container_port,
|
||||
"desired_state": "present",
|
||||
"status": STATUS_PENDING,
|
||||
"last_error": "",
|
||||
"updated_at": stamp,
|
||||
"deleted_at": None,
|
||||
"deleted_by": None,
|
||||
},
|
||||
["uid"],
|
||||
)
|
||||
return table.find_one(uid=revived["uid"])
|
||||
uid = generate_uid()
|
||||
table.insert(
|
||||
|
||||
@ -110,9 +110,19 @@ class WorkspaceController:
|
||||
def _tunnel_create(self, args: dict) -> Any:
|
||||
instance = self._resolve(args.get("project_slug", ""))
|
||||
port = int(args.get("container_port") or 0)
|
||||
row = provision.publish_tunnel(
|
||||
if port <= 0:
|
||||
raise WorkspaceError("container_port is required")
|
||||
limits = quota.resolve(instance.get("workspace_owner_uid", ""), instance)
|
||||
if limits.max_tunnels and tunnels.count_for_instance(
|
||||
instance["uid"]
|
||||
) >= limits.max_tunnels:
|
||||
raise WorkspaceError(f"tunnel limit reached ({limits.max_tunnels})")
|
||||
row = tunnels.create(
|
||||
instance, args.get("label", ""), port, self.owner_id
|
||||
)
|
||||
if not row:
|
||||
raise WorkspaceError("could not create tunnel")
|
||||
provision.write_manifest(instance)
|
||||
return {
|
||||
"ok": True,
|
||||
"tunnel": row,
|
||||
|
||||
@ -198,31 +198,34 @@ class GatewayRuntime:
|
||||
self.in_flight += 1
|
||||
if self.in_flight > self.peak_in_flight:
|
||||
self.peak_in_flight = self.in_flight
|
||||
wait_start = time.monotonic()
|
||||
connect_holder = {"ms": 0.0}
|
||||
attempts = 1
|
||||
resp = None
|
||||
exc = None
|
||||
try:
|
||||
|
||||
async def do_call():
|
||||
request = client.build_request(
|
||||
method, url, headers=headers, json=json_body, content=content
|
||||
async with sem:
|
||||
timing["queue_wait_ms"] = round(
|
||||
(time.monotonic() - wait_start) * 1000, 3
|
||||
)
|
||||
request.extensions["trace"] = _connect_tracer(connect_holder)
|
||||
return await client.send(request)
|
||||
|
||||
send_start = time.monotonic()
|
||||
resp, exc, attempts, queue_wait_ms = await retry_send(
|
||||
do_call,
|
||||
sem,
|
||||
cfg["gateway_max_retries"],
|
||||
cfg["gateway_retry_backoff_ms"],
|
||||
log,
|
||||
)
|
||||
timing["upstream_latency_ms"] = round(
|
||||
(time.monotonic() - send_start) * 1000, 3
|
||||
)
|
||||
timing["queue_wait_ms"] = round(queue_wait_ms, 3)
|
||||
async def do_call():
|
||||
request = client.build_request(
|
||||
method, url, headers=headers, json=json_body, content=content
|
||||
)
|
||||
request.extensions["trace"] = _connect_tracer(connect_holder)
|
||||
return await client.send(request)
|
||||
|
||||
send_start = time.monotonic()
|
||||
resp, exc, attempts = await retry_send(
|
||||
do_call,
|
||||
cfg["gateway_max_retries"],
|
||||
cfg["gateway_retry_backoff_ms"],
|
||||
log,
|
||||
)
|
||||
timing["upstream_latency_ms"] = round(
|
||||
(time.monotonic() - send_start) * 1000, 3
|
||||
)
|
||||
finally:
|
||||
self.in_flight -= 1
|
||||
timing["connect_ms"] = round(connect_holder["ms"], 3)
|
||||
|
||||
@ -66,30 +66,21 @@ async def _backoff(backoff_ms: int, attempt: int) -> None:
|
||||
|
||||
async def retry_send(
|
||||
do_call: Callable[[], Awaitable[httpx.Response]],
|
||||
sem: asyncio.Semaphore,
|
||||
max_retries: int,
|
||||
backoff_ms: int,
|
||||
log: Optional[Callable[[str], None]] = None,
|
||||
) -> tuple[Optional[httpx.Response], Optional[Exception], int, float]:
|
||||
) -> tuple[Optional[httpx.Response], Optional[Exception], int]:
|
||||
log = log or (lambda message: None)
|
||||
attempts = 0
|
||||
last_exc: Optional[Exception] = None
|
||||
queue_wait_ms = 0.0
|
||||
while attempts <= max_retries:
|
||||
attempts += 1
|
||||
wait_start = time.monotonic()
|
||||
async with sem:
|
||||
queue_wait_ms += (time.monotonic() - wait_start) * 1000
|
||||
try:
|
||||
resp = await do_call()
|
||||
exc = None
|
||||
except httpx.RequestError as e:
|
||||
resp = None
|
||||
exc = e
|
||||
if exc is not None:
|
||||
try:
|
||||
resp = await do_call()
|
||||
except httpx.RequestError as exc:
|
||||
last_exc = exc
|
||||
if attempts > max_retries:
|
||||
return None, exc, attempts, queue_wait_ms
|
||||
return None, exc, attempts
|
||||
log(
|
||||
f"upstream connection failed, retrying ({attempts}/{max_retries}): {exc}"
|
||||
)
|
||||
@ -99,5 +90,5 @@ async def retry_send(
|
||||
log(f"upstream {resp.status_code}, retrying ({attempts}/{max_retries})")
|
||||
await _backoff(backoff_ms, attempts)
|
||||
continue
|
||||
return resp, None, attempts, queue_wait_ms
|
||||
return None, last_exc, attempts, queue_wait_ms
|
||||
return resp, None, attempts
|
||||
return None, last_exc, attempts
|
||||
|
||||
@ -1,12 +1,12 @@
|
||||
/* retoor <retoor@molodetz.nl> */
|
||||
|
||||
:root {
|
||||
--bg-primary: #080413;
|
||||
--bg-secondary: #120821;
|
||||
--bg-card: #1a1030;
|
||||
--bg-card-hover: #241640;
|
||||
--bg-input: #140b26;
|
||||
--bg-modal: #1a1030;
|
||||
--bg-primary: #271b5b;
|
||||
--bg-secondary: #1a1736;
|
||||
--bg-card: #13112a;
|
||||
--bg-card-hover: #1a1736;
|
||||
--bg-input: #1a1736;
|
||||
--bg-modal: #13112a;
|
||||
|
||||
--accent: #b73f1e;
|
||||
--accent-rgb: 183, 63, 30;
|
||||
@ -76,5 +76,5 @@
|
||||
--topic-fun: #ffab00;
|
||||
--topic-politics: #00bcd4;
|
||||
|
||||
--bg-gradient: linear-gradient(135deg, #080413 0%, #160a28 50%, #080413 100%);
|
||||
--bg-gradient: linear-gradient(135deg, #271b5b 0%, #1d1545 50%, #271b5b 100%);
|
||||
}
|
||||
|
||||
@ -19,7 +19,7 @@ Seven HTTP middlewares run as a stack around every request, listed outermost fir
|
||||
| `track_presence` | For the resolved current user on every non-asset request, calls `presence.touch(uid)` (a throttled `last_seen` write). |
|
||||
| `maintenance_middleware` | When `maintenance_mode="1"`, returns a 503 `error.html` for non-admins, but always allows `/static`, `/avatar`, `/auth`, `/admin`, `/openai`, and admin users, so an operator can never lock themselves out. |
|
||||
| `rate_limit_middleware` | Per-IP limit for mutating methods (POST/PUT/DELETE/PATCH), held in an in-process `defaultdict`. Reads `rate_limit_per_minute` / `rate_limit_window_seconds` from `site_settings`, floored to `max(1, ...)`, and is worker-count-aware. `/openai` and `/xmlrpc` are excluded. |
|
||||
| `add_security_headers` | Sets `X-Content-Type-Options: nosniff` always, `X-Robots-Tag: index, follow` unless the handler already set it (so pages can opt out of indexing), plus HSTS, `Referrer-Policy`, and a CSP whose `frame-ancestors` allows `'self'` and the workspace tunnel domain (except the `/p/` ingress, which sets no CSP at all), and no-store cache headers on `/admin`. Framing is controlled by `frame-ancestors` alone - no `X-Frame-Options` is sent, because it cannot express an allow-list and browsers honour the stricter of the two. |
|
||||
| `add_security_headers` | Sets `X-Content-Type-Options: nosniff` always, `X-Robots-Tag: index, follow` unless the handler already set it (so pages can opt out of indexing), plus HSTS, `Referrer-Policy`, a CSP and `X-Frame-Options: DENY` (except the `/p/` ingress), and no-store cache headers on `/admin`. |
|
||||
| `await_pending_corrections` | After the handler runs, awaits any pending AI correction/modifier futures parked on `request.scope` (sync apply mode). |
|
||||
| `refresh_db_snapshot` | Calls `refresh_snapshot()` so each request sees committed data. |
|
||||
|
||||
|
||||
@ -3,29 +3,6 @@
|
||||
|
||||
The nginx front door, how it serves each route, and the production-specific rules it enforces. See also [Production overview](/docs/production.html), [Deploy and update](/docs/production-deploy.html), and [Static asset caching and versioning](/docs/static-caching.html).
|
||||
|
||||
## Public hostnames: two front doors, one application
|
||||
|
||||
The platform answers on **two** public hostnames that reach the same application by completely different routes. Knowing which is which is the difference between a five minute diagnosis and an hour of chasing the wrong edge.
|
||||
|
||||
| | `pravda.education` | `devplace.net` |
|
||||
|---|---|---|
|
||||
| DNS | `95.216.15.238`, `2a01:4f9:2a:100e::2` | `88.198.21.243`, `2a01:4f8:222:2c45::2` |
|
||||
| Machine | the production host | a separate front host |
|
||||
| Path in | molohttp on port 443, proxying to `127.0.0.1:10500` | its own proxy, then an **SSH tunnel** to `127.0.0.1:10500` on production |
|
||||
| Passes through molohttp | yes | **no** |
|
||||
|
||||
`devplace.net` runs on its own machine and holds a persistent SSH session into the production host, forwarding through it to `127.0.0.1:10500` - the `docker-proxy` socket for the nginx container. The forwarded listener lives on the **front** host, so the production host shows no sshd listening socket for it. That absence is expected.
|
||||
|
||||
**molohttp deliberately has no `devplace.net` site.** Its sites are `mail`, `smtp` and `imap.molodetz.nl`, `pravda.education`, and the workspace tunnel wildcard `*.tunnel.pravda.education`. Traffic for `devplace.net` enters underneath molohttp, so it needs no site there and adding one would achieve nothing - the hostname does not resolve to the production host, so such a site could never match.
|
||||
|
||||
**Triage rule.** Run the same authenticated request against both hostnames and compare:
|
||||
|
||||
- Fails on **both** - the fault is in the application or the database. Neither edge is involved.
|
||||
- Fails on **devplace.net only** - the fault is in the front host's proxy. WebSocket `Upgrade` and `Connection` headers are the usual cause, exactly as documented for the nginx locations below.
|
||||
- Fails on **pravda.education only** - the fault is in molohttp or its site configuration.
|
||||
|
||||
**Never test a hostname by forcing it onto an IP it does not resolve to.** Using `curl --resolve devplace.net:443:<production ip>` sends `Host: devplace.net` to molohttp, which correctly answers `404 No site configured for host: devplace.net`. That result says nothing about the real path and reads convincingly like a total outage. Always fetch each hostname over the public internet as it genuinely resolves.
|
||||
|
||||
## Build and configuration
|
||||
|
||||
The nginx image (`nginx/Dockerfile`) renders `nginx/nginx.conf.template` at start through `nginx/start.sh`, which substitutes a small allow-list of variables (`NGINX_CACHE_CONFIG`, `NGINX_CACHE_MAX_SIZE`, `NGINX_MAX_BODY_SIZE`) and leaves nginx runtime variables such as `$http_upgrade` untouched. The host's `devplacepy/static` directory is bind-mounted read-only at `/app/static` for package assets, and the consolidated `<DEVPLACE_DATA_DIR>/uploads` directory is bind-mounted read-only at `/data/uploads` (the `/static/uploads/` location aliases it), so both served assets and uploads always match the running code and data without an image rebuild.
|
||||
@ -85,7 +62,7 @@ Verify from an IPv6-only vantage point (or force the family): `curl -6 -I https:
|
||||
|
||||
## Security headers and caching
|
||||
|
||||
The server block sets `X-Content-Type-Options`, `X-XSS-Protection`, and `Referrer-Policy`, inherited only by locations that declare no `add_header` of their own. It deliberately sets no `X-Frame-Options`: every location without its own `add_header` is proxied to the app, which owns framing policy via the CSP `frame-ancestors` directive, and an nginx-level header would be re-added on top of the app's response - that is what previously defeated the `/p/` ingress exemption, since `location /p/` declares no `add_header` and the app intentionally sends no framing headers there. gzip is enabled for text, JSON, JS, CSS, and SVG. The micro-cache is off by default; set `NGINX_CACHE_ENABLED=true` to cache proxied 200s for one minute with `X-Cache-Status` reporting.
|
||||
The server block sets `X-Content-Type-Options`, `X-Frame-Options: DENY`, `X-XSS-Protection`, and `Referrer-Policy`, inherited only by locations that declare no `add_header` of their own. gzip is enabled for text, JSON, JS, CSS, and SVG. The micro-cache is off by default; set `NGINX_CACHE_ENABLED=true` to cache proxied 200s for one minute with `X-Cache-Status` reporting.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@ -109,23 +109,6 @@ Press `F1` and type `DevPlace` for the full list:
|
||||
| **DevPlace: Show public tunnels** | Pick one of your live public addresses |
|
||||
| **DevPlace: Open the DevPlace editor guide** | This page |
|
||||
|
||||
## Publishing a port from the editor
|
||||
|
||||
Forward a port in the editor's **Ports** view and DevPlace publishes it for you.
|
||||
The moment you forward it, the editor registers the port with DevPlace, which
|
||||
creates the tunnel, orders its HTTPS certificate and answers with the public
|
||||
address - the same address the Ports view shows you. Publishing counts against
|
||||
your tunnel quota, so a port DevPlace refuses is reported back in the editor with
|
||||
the reason.
|
||||
|
||||
Two things to know:
|
||||
|
||||
- The address serves HTTPS as soon as the certificate is issued, which takes a
|
||||
few seconds. Until then your browser warns about the certificate name.
|
||||
- Un-forwarding the port in the editor does **not** remove the tunnel. Public
|
||||
addresses are removed deliberately, on your workspace page or by asking Devii,
|
||||
so a restarted dev server never silently loses its link.
|
||||
|
||||
## Related
|
||||
|
||||
- [Get started with vibing](/docs/getting-started-vibing.html) - the container
|
||||
|
||||
@ -247,7 +247,6 @@
|
||||
<li><code>sudo</code> and <code>apt install</code> work with no extra setup.</li>
|
||||
<li>Python, Rust, Nim and Swift toolchains are preinstalled.</li>
|
||||
<li>Ports below 1024 cannot bind. Use a high port and a tunnel.</li>
|
||||
<li>Forwarding a port in the editor's <strong>Ports</strong> view publishes it here automatically.</li>
|
||||
<li>Your public URLs are also in <code>/app/.devplace/tunnels.json</code>.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
@ -24,6 +24,7 @@ server {
|
||||
client_max_body_size ${NGINX_MAX_BODY_SIZE};
|
||||
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
add_header X-Frame-Options DENY;
|
||||
add_header X-XSS-Protection "1; mode=block";
|
||||
add_header Referrer-Policy strict-origin-when-cross-origin;
|
||||
|
||||
@ -208,8 +209,8 @@ server {
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_read_timeout 900s;
|
||||
proxy_send_timeout 900s;
|
||||
proxy_read_timeout 120s;
|
||||
proxy_send_timeout 120s;
|
||||
}
|
||||
|
||||
location / {
|
||||
@ -226,7 +227,7 @@ server {
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
|
||||
proxy_connect_timeout 900s;
|
||||
proxy_connect_timeout 30s;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
|
||||
|
||||
@ -12,9 +12,9 @@ ENV PYTHONUNBUFFERED=1 \
|
||||
PLAYWRIGHT_BROWSERS_PATH=/opt/playwright
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
git openssh-client curl wget vim ack ca-certificates build-essential libpq-dev \
|
||||
git curl wget vim ack ca-certificates build-essential libpq-dev \
|
||||
tmux apache2-utils procps htop iftop iotop netcat-openbsd zip unzip \
|
||||
rsync jq fakeroot xz-utils pkg-config \
|
||||
fakeroot xz-utils pkg-config \
|
||||
binutils gnupg2 libc6-dev libcurl4-openssl-dev libedit2 libedit-dev \
|
||||
libncurses-dev libpython3-dev libsqlite3-0 libsqlite3-dev uuid-dev \
|
||||
libxml2-dev libz3-dev tzdata zlib1g-dev \
|
||||
@ -99,7 +99,7 @@ COPY vscode/branding/devplace-login.css /tmp/devplace-login.css
|
||||
COPY vscode/product.patch.json /tmp/product.patch.json
|
||||
RUN set -eu; \
|
||||
cat /tmp/devplace-login.css >> /usr/local/lib/code-server/src/browser/pages/login.css; \
|
||||
python3 -c "import json,pathlib; p=pathlib.Path('/usr/local/lib/code-server/lib/vscode/product.json'); d=json.loads(p.read_text()); patch=json.loads(pathlib.Path('/tmp/product.patch.json').read_text()); d.update({k: ({**d[k], **v} if isinstance(v, dict) and isinstance(d.get(k), dict) else v) for k, v in patch.items()}); p.write_text(json.dumps(d, indent=2))"; \
|
||||
python3 -c "import json,pathlib; p=pathlib.Path('/usr/local/lib/code-server/lib/vscode/product.json'); d=json.loads(p.read_text()); d.update(json.loads(pathlib.Path('/tmp/product.patch.json').read_text())); p.write_text(json.dumps(d, indent=2))"; \
|
||||
rm -f /tmp/devplace-login.css /tmp/product.patch.json
|
||||
COPY sudo /usr/local/bin/sudo
|
||||
COPY aptroot /usr/local/bin/aptroot
|
||||
|
||||
@ -444,7 +444,7 @@ def test_referrer_policy_header(app_server):
|
||||
|
||||
def test_x_frame_options_header(app_server):
|
||||
r = requests.get(f"{BASE_URL}/feed", allow_redirects=True)
|
||||
assert "X-Frame-Options" not in r.headers
|
||||
assert r.headers.get("X-Frame-Options") == "DENY"
|
||||
|
||||
|
||||
def test_x_frame_options_excluded_for_ingress_proxy(app_server):
|
||||
@ -457,7 +457,7 @@ def test_content_security_policy_header(app_server):
|
||||
csp = r.headers.get("Content-Security-Policy", "")
|
||||
assert "object-src 'none'" in csp
|
||||
assert "base-uri 'self'" in csp
|
||||
assert "frame-ancestors 'self'" in csp
|
||||
assert "frame-ancestors 'none'" in csp
|
||||
assert "form-action 'self'" in csp
|
||||
|
||||
|
||||
|
||||
@ -1,7 +1,6 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import json
|
||||
import shlex
|
||||
|
||||
import pytest
|
||||
|
||||
@ -950,14 +949,9 @@ def test_the_run_spec_applies_the_quota_cpu_and_memory_to_a_workspace():
|
||||
spec = api.run_spec_for(store.get_instance(instance["uid"]), "ppy:latest")
|
||||
assert spec.cpu_limit == limits.cpu_limit()
|
||||
assert spec.mem_limit == limits.mem_limit()
|
||||
assert spec.command[0] == "/bin/sh"
|
||||
assert spec.command[1] == "-c"
|
||||
script = spec.command[2]
|
||||
assert "code-server" in script
|
||||
assert "--app-name" in script
|
||||
assert "--disable-workspace-trust" in script
|
||||
assert editor.ENV_EXPORT_FILE in script
|
||||
assert shlex.split(script)[-1] == "/app"
|
||||
assert spec.command[0] == "code-server"
|
||||
assert "--app-name" in spec.command
|
||||
assert "--disable-workspace-trust" in spec.command
|
||||
|
||||
|
||||
def test_the_run_spec_seeds_the_editor_state_and_stamps_a_boot_marker(monkeypatch, tmp_path):
|
||||
@ -994,49 +988,3 @@ def test_a_workspace_without_an_editor_port_still_gets_its_size():
|
||||
spec = api.run_spec_for(store.get_instance(instance["uid"]), "ppy:latest")
|
||||
assert spec.cpu_limit == limits.cpu_limit()
|
||||
assert spec.command == ["sleep", "infinity"]
|
||||
|
||||
|
||||
def test_publish_tunnel_records_that_certificates_are_not_configured():
|
||||
instance = _instance()
|
||||
row = provision.publish_tunnel(instance, "web", 3000, OWNER)
|
||||
assert row["status"] == tunnels.STATUS_PENDING
|
||||
assert row["last_error"] == provision.CERT_UNCONFIGURED
|
||||
|
||||
|
||||
def test_publish_tunnel_enforces_the_tunnel_quota():
|
||||
set_setting("workspace_max_tunnels", "1")
|
||||
try:
|
||||
instance = _instance()
|
||||
provision.publish_tunnel(instance, "web", 3000, OWNER)
|
||||
with pytest.raises(WorkspaceError):
|
||||
provision.publish_tunnel(instance, "api", 3001, OWNER)
|
||||
finally:
|
||||
set_setting("workspace_max_tunnels", "5")
|
||||
|
||||
|
||||
def test_publish_tunnel_refuses_a_port_outside_the_valid_range():
|
||||
instance = _instance()
|
||||
with pytest.raises(WorkspaceError):
|
||||
provision.publish_tunnel(instance, "web", 0, OWNER)
|
||||
|
||||
|
||||
def test_tunnel_route_resolves_an_unpublished_port_through_the_container():
|
||||
from devplacepy.routers import tunnel as tunnel_router
|
||||
|
||||
instance = _instance(container_ip="172.17.0.9", container_gateway="172.17.0.1")
|
||||
row = provision.publish_tunnel(instance, "web", 3000, OWNER)
|
||||
tunnels.update(row["uid"], {"status": tunnels.STATUS_ACTIVE})
|
||||
resolved, matched, host, port = tunnel_router.resolve(row["hostname"])
|
||||
assert resolved["uid"] == row["uid"]
|
||||
assert matched["uid"] == instance["uid"]
|
||||
assert (host, port) == ("172.17.0.9", 3000)
|
||||
|
||||
|
||||
def test_republishing_a_live_tunnel_keeps_its_certificate():
|
||||
instance = _instance()
|
||||
row = provision.publish_tunnel(instance, "web", 3000, OWNER)
|
||||
tunnels.update(row["uid"], {"status": tunnels.STATUS_ACTIVE, "last_error": ""})
|
||||
again = provision.publish_tunnel(instance, "web", 3000, OWNER)
|
||||
assert again["uid"] == row["uid"]
|
||||
assert again["status"] == tunnels.STATUS_ACTIVE
|
||||
assert again["last_error"] == ""
|
||||
|
||||
@ -507,45 +507,3 @@ def test_raw_query_survives_a_hash_in_the_path():
|
||||
request.scope["path"] = "/weird/a b#c"
|
||||
assert forward.raw_query(request) == "keep=1"
|
||||
assert request.url.query == ""
|
||||
|
||||
|
||||
def test_tunnel_target_prefers_the_direct_container_leg():
|
||||
instance = {
|
||||
"ports_json": '[{"host": 20500, "container": 8443, "proto": "tcp"}]',
|
||||
"container_gateway": "172.17.0.1",
|
||||
"container_ip": "172.17.0.9",
|
||||
}
|
||||
assert api.tunnel_target(instance, 8443) == ("172.17.0.9", 8443)
|
||||
|
||||
|
||||
def test_tunnel_target_falls_back_to_the_published_port_without_a_container_ip():
|
||||
instance = {
|
||||
"ports_json": '[{"host": 20500, "container": 8443, "proto": "tcp"}]',
|
||||
"container_gateway": "172.17.0.1",
|
||||
}
|
||||
assert api.tunnel_target(instance, 8443) == ("172.17.0.1", 20500)
|
||||
|
||||
|
||||
def test_proxy_target_and_tunnel_target_agree_on_the_same_port():
|
||||
instance = {
|
||||
"ports_json": '[{"host": 20500, "container": 8443, "proto": "tcp"}]',
|
||||
"container_gateway": "172.17.0.1",
|
||||
"container_ip": "172.17.0.9",
|
||||
"ingress_port": 8443,
|
||||
}
|
||||
assert api.proxy_target(instance) == api.tunnel_target(instance, 8443)
|
||||
|
||||
|
||||
def test_tunnel_target_dials_the_container_for_an_unpublished_port():
|
||||
instance = {
|
||||
"ports_json": '[{"host": 20500, "container": 8443, "proto": "tcp"}]',
|
||||
"container_gateway": "172.17.0.1",
|
||||
"container_ip": "172.17.0.9",
|
||||
}
|
||||
assert api.tunnel_target(instance, 3000) == ("172.17.0.9", 3000)
|
||||
|
||||
|
||||
def test_tunnel_target_is_empty_without_a_route_to_the_port():
|
||||
instance = {"ports_json": "[]", "container_gateway": "172.17.0.1"}
|
||||
assert api.tunnel_target(instance, 3000) == (None, None)
|
||||
assert api.tunnel_target({"container_ip": "172.17.0.9"}, 0) == (None, None)
|
||||
|
||||
@ -1,7 +1,6 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import json
|
||||
import shlex
|
||||
|
||||
import pytest
|
||||
|
||||
@ -309,23 +308,6 @@ def test_every_optional_flag_is_declared():
|
||||
assert flag in command
|
||||
|
||||
|
||||
def test_wrap_with_env_export_runs_the_original_command_through_a_shell():
|
||||
command = editor.argv(_instance(), editor.resolve(OWNER))
|
||||
wrapped = editor.wrap_with_env_export(command)
|
||||
assert wrapped[0] == "/bin/sh"
|
||||
assert wrapped[1] == "-c"
|
||||
script = wrapped[2]
|
||||
assert script.rstrip().endswith(shlex.join(command))
|
||||
|
||||
|
||||
def test_wrap_with_env_export_writes_only_devplace_prefixed_vars():
|
||||
script = editor.wrap_with_env_export(["true"])[2]
|
||||
assert "python3 -c" in script
|
||||
assert editor.ENV_EXPORT_FILE in script
|
||||
assert "DEVPLACE_" in script
|
||||
assert "startswith" in script
|
||||
|
||||
|
||||
def test_env_for_exports_the_profile_to_the_container():
|
||||
env = editor.env_for(editor.resolve(OWNER))
|
||||
assert env["DEVPLACE_EDITOR_APP_NAME"] == editor.APP_NAME
|
||||
|
||||
Loading…
Reference in New Issue
Block a user