This file documents `devplacepy/push/` - push notification delivery and its provider architecture. Claude Code loads it automatically whenever a file under this directory is read or edited.

## What this package is

One delivery library behind one public surface. `devplacepy.push` exports `notify_user`, `notify_registration`, `register`, `ensure_certificates`, `public_key_standard_b64` and the Web Push crypto helpers; every caller in the codebase (`main.py`, `utils/notifications.py`, `routers/push.py`) imports only those names. Everything else is internal to the package.

| Module | Role |
|---|---|
| `providers/base.py` | `PushProvider` protocol, the `Delivery` outcome and the three outcome constants |
| `providers/webpush.py` | VAPID key material, `aesgcm` payload encryption, the Web Push provider |
| `providers/apns.py` | Apple Push Notification service provider (token based, dedicated HTTP/2 client) |
| `providers/__init__.py` | `PROVIDERS` registry, `get`, `active`, `is_active`, `admin_fields`, `client_config` |
| `store.py` | Every `push_registration` read and write, including identity upsert and revive |
| `delivery.py` | `notify_user` / `notify_registration`: group by provider, one client per provider, one prepared body per provider |

The admin configuration surface lives in `devplacepy/services/push/service.py` (`PushService`), not here.

## Adding a provider

1. Write `providers/<name>.py` with a `PushProvider` subclass: `name`, `label`, `config_fields`, `is_configured`, `parse_registration`, `prepare`, `deliver`, optionally `client_config`, `stamp_registration`, `delivery_client`.
2. Add one entry to `PROVIDERS` in `providers/__init__.py`.

That is the whole change. The registration route, the delivery loop, the admin page, the audit record, the metrics and the docs are written against the protocol and need no edit. The `Enabled` toggle (`push_<name>_enabled`) comes from the base class, so a provider never declares its own.

The default `delivery_client` is `stealth_async_client`. Override it only when the destination is a first-party API that must not see Chrome impersonation, PRIORITY frames, extra browser headers, or the outbound proxy. APNs is that case.

## Invariants

- **Zero cost for the request, except the welcome probe.** Delivery of real notifications is reached only through `utils/notifications.py` `_schedule_push`, a fire-and-forget task. `POST /push.json` awaits `notify_registration` only for a newly created or revived row, so the client can see `delivered` / `error`. Never add a queue or a table to this path.
- **`deliver` never raises.** Return `Delivery(REJECTED, detail)` instead. `delivery.py` guards anyway, but a raising provider costs a log line per subscription.
- **A provider that is not configured is inert, never an error.** `is_configured()` is false, `is_active()` is false, the delivery loop skips it, and `POST /push.json` refuses a registration for it with 400. Nothing else in the platform notices.
- **`DEAD` is the only outcome that touches the database.** It soft-deletes the registration (`deleted_at`), exactly like a `404`/`410` Web Push endpoint always did - unless `Delivery.dead_before` names an instant the row was proven live again after (APNs 410 `timestamp` vs. `registered_at`, see "APNs specifics"), in which case the delete is skipped. `REJECTED` keeps the row. The Apple/Web Push reason is logged at WARNING (`Push dead via ...: <detail>`); do not drop `outcome.detail`.
- **Every insert writes `deleted_at: None`,** and every read of the live set filters `deleted_at IS NULL`. Identity lookups for upsert/revive intentionally ignore `deleted_at` so a client that re-registers a previously dead token comes back live. `push_registration` deliberately stays out of `SOFT_DELETE_TABLES` (no `deleted_by`, not restorable from Trash) - a dead device token has no owner action to undo.
- **A row without a provider is a Web Push row.** `store.provider_of` resolves `None`/`""` to `DEFAULT_PROVIDER`, so a row written by an old worker during a deploy still delivers. `init_db` backfills the column once with a single converging `UPDATE`.

## Storage

`push_registration` columns are ensured in `init_db` (`database/schema.py`) because `dataset` only creates the columns of a table's first insert, and `find(provider=...)` against a missing column matches nothing.

| Column | webpush | apns |
|---|---|---|
| `provider` | `webpush` | `apns` |
| `endpoint`, `key_auth`, `key_p256dh` | set | `NULL` |
| `token` | `NULL` | device token |
| `client_id` | unused | optional stable per-device id |
| `environment` | unused | `production` or `sandbox`, stamped server-side at register |
| `registered_at` | stamped on insert/merge | stamped on insert/merge |

`registered_at` (ISO, both providers) is stamped on insert and on every `_merge` that actually changes a row (including revival). It exists solely to arbitrate the dead-token race described below - it is not a general "last seen" field.

`store.register` returns `RegistrationWrite(record, created, revived)`. Identity, in order:

1. `user_uid` + `provider` + `client_id` (live or dead), if `client_id` is present.
2. `user_uid` + `provider` + `token` (or `endpoint` for Web Push), live or dead.
3. Exact live match on the remaining fields.
4. Insert.

A match **updates** the row (token, client_id, environment) and clears `deleted_at` if it was dead. Token rotation with the same `client_id` therefore replaces the token on one row. Sibling live rows that share the new token, the previous token, or the same `client_id` are marked dead so one device cannot accumulate duplicates. A body without `client_id` still works: the same token revives, a new token inserts. `push.update` is a real update, not a no-op of an identical POST.

## APNs specifics

- **Persistent HTTP/2 connection, not one per notification.** `apns.cached_client(timeout)` lazily creates ONE module-level `httpx.AsyncClient(http2=True, ...)` and reuses it across every `notify_user`/`notify_registration` call for the lifetime of the process, following Apple's explicit guidance to keep the connection open rather than repeatedly opening/closing (`sending-notification-requests-to-apns`). `ApnsProvider.closes_delivery_client()` returns `False` so `delivery.py` never closes it after a batch (Web Push still opens/closes per call via `stealth_async_client`, `closes_delivery_client()` defaulting `True` on the base class). Closed once, gracefully, in `main.py`'s shutdown via `push.shutdown_providers()` -> `ApnsProvider.aclose()` -> `apns.close_client()`, mirroring the identical `services/containers/forward.py` `client()`/`close_client()` pattern. Not stealth, not curl_cffi Chrome impersonation, not the outbound proxy - Apple's provider API is a first-party HTTP/2 service; browser PRIORITY frames, HPACK indexing of `:path`/`authorization`, extra `sec-ch-ua` headers, and a scraping proxy all violate that contract. This is the second documented exception to the stealth-only outbound rule (the other is container reverse-proxy forwarding). If the admin-configured delivery timeout changes, the cached client is rebuilt with the new timeout on next use and the old one is left for GC (not explicitly closed) - a deliberate, rare-path simplification.
- `POST https://{host}/3/device/{token}` over HTTP/2. Host comes from the **row's** `environment` if set, else `push_apns_environment` (`api.push.apple.com` or `api.sandbox.push.apple.com`; an unrecognised value falls back to production). Stamping environment per row lets a sandbox debug token and a production TestFlight token coexist.
- `GET /push.json` advertises `providers.apns.environment` when APNs is active so a native client can refuse to register a sandbox token against production.
- `parse_registration` requires a hexadecimal `token` (64-200 digits after stripping spaces and `<>`) and optionally `client_id` (string, max 128). A `client_id` that is not a string is a 400, not a silent drop. Server stamps `environment`; the client cannot pick the host.
- **Provider token (JWT):** `ES256`, header `kid` = key id, claims `iss` = team id and `iat`. Cached per credential fingerprint for 45 minutes (Apple refuses tokens regenerated faster than every 20 minutes, and requires a fresh one at least once an hour). **Shared cross-worker via `site_settings` key `push_apns_shared_token`** (JSON `{fingerprint, token, issued_at}`, read/written through the existing `get_setting`/`set_setting` cache-version machinery, propagating to sibling workers within ~1s like every other settings key) - a worker that finds no valid in-process cache checks the shared copy before signing a new one, so `make prod`'s multiple uvicorn workers converge on presenting Apple the SAME token instead of each independently re-signing on its own clock (which could otherwise interleave closer together than Apple's per-credential minimum spacing, worst case when workers boot near-simultaneously). Editing any credential changes the fingerprint and takes effect on the next delivery, with no restart.
- **A rejected provider token is evicted immediately, not left to expire from cache.** A 403 status, or any reason in `AUTH_REASONS` (`InvalidProviderToken`, `ExpiredProviderToken`, `BadCertificate`, `BadCertificateEnvironment`, `Forbidden`, `MissingProviderToken` - all provider-credential-shaped per Apple's reason table, never about a specific device), calls `invalidate_provider_token()`: clears the in-process cache AND the shared DB copy, so the very next delivery attempt (any worker) re-signs instead of retrying the same rejected token for up to 45 minutes.
- A `.p8` that does not parse is cached as a failure for the same window, so a misconfiguration costs one error log per window rather than one parse per notification.
- **`DEAD_REASONS` is deliberately narrow: only `BadDeviceToken` (400), `Unregistered` (410) and `ExpiredToken` (410).** These are the only reasons in Apple's documented table that describe the TOKEN itself as permanently invalid. `DeviceTokenNotForTopic` and `TopicDisallowed` are 400 errors about the **topic/provisioning matching the connection**, not the token - they fire identically for every token when `push_apns_topic` is misconfigured or the certificate/entitlements don't match, so treating them as dead would soft-delete the entire APNs subscriber base (a table deliberately kept OUT of `SOFT_DELETE_TABLES`, hence unrestorable) on the first delivery after a one-field admin typo. Never add a topic/provisioning-shaped reason to `DEAD_REASONS`.
- **A 410's `timestamp` is honored before deleting.** Apple's 410 body carries `{"reason": "Unregistered", "timestamp": <ms epoch>}` - the instant Apple last confirmed the token dead, which can trail real device state by "several days" per Apple engineering guidance (410 delivery is intentionally non-deterministic; do not use it to infer app-uninstall timing). `apns._dead_before` converts it to ISO and it rides on `Delivery.dead_before`; `store.mark_dead(id, dead_before)` compares it against the row's `registered_at` and **skips the delete** (logs at INFO instead) if the registration was re-registered/revived after that instant - closing the race where an in-flight delivery against a stale row state would otherwise undo a concurrent revival. `dead_before` is only ever set for a 410; the `BadDeviceToken`/`ExpiredToken` paths pass `None` (unconditional delete, as before), since those reasons carry no `timestamp`.
- The shared payload dict (`title`, `message`, `icon`, `url`) is translated once per batch into `aps.alert` plus the custom `url`/`icon` keys, mirroring what `service-worker.js` does for Web Push. `thread-id` mirrors the service worker's notification `tag`.
- `POST /push.json` probes a created or revived row immediately via `notify_registration`. The JSON is `{registered: true, delivered: bool, error?: string}`. `registered` stays true so existing clients keep working; `delivered`/`error` surface Apple's reason instead of killing the row silently from the client's point of view. The row is still marked dead on a `DEAD` outcome (subject to the `dead_before` guard above) so later `notify_user` calls do not keep hitting a known-bad token.
