This file documents the Backup service (`devplacepy/services/backup/`, `devplacepy/services/jobs/backup_worker.py`, `devplacepy/routers/admin/backups.py`). Claude Code loads it automatically whenever a file under `devplacepy/services/backup/` is read or edited.
## Overview
Admin-only, enterprise-grade backups built on the **same async-job pattern as zip/fork** (see `devplacepy/services/jobs/CLAUDE.md`), so nothing runs on the request path. `BackupService(JobService)` (kind `backup`) is registered in `main.py` and overrides `run_once` to call `super().run_once()` (reap/refill/sweep) and then `_fire_due_schedules()`.
## Targets and worker
- **Targets** (`store.BACKUP_TARGETS`): `database` (consistent SQLite snapshot of the main DB + `devii_tasks.db` + `devii_lessons.db` via the `sqlite3` online backup API, so it is consistent under WAL - never a raw file copy), `uploads` (`UPLOADS_DIR`), `keys` (`KEYS_DIR`), `full` (database snapshot + uploads + keys; regenerable/volatile dirs - staging, locks, zips, container workspaces, chroma - are intentionally excluded). `service._materialize(target, staging)` returns a list of `{root, path}` sources; DB targets are snapshotted into `staging/db_snapshot` first.
- **Worker** (`services/jobs/backup_worker.py`, stdlib only): reads a JSON spec `{sources:[{root,path}]}` and an output path, builds a deterministic `tar.gz` (uid/gid 0, symlinks skipped) rooted at each `root/`, and returns `{bytes_in, bytes_out, file_count, dir_count, sha256}`. Invoked via `asyncio.create_subprocess_exec` like `zip_worker`.
## Storage and data model
- **Storage:** archives go under `config.BACKUPS_DIR` (`data/backups/`, in `DATA_PATHS`) sharded with `attachments._directory_for` on the **random uuid tail** (same load-bearing reason as zips/blobs), named `{target}-{YYYYMMDD-HHMMSS}-{tail}.tar.gz`. Staging is `config.BACKUP_STAGING_DIR` (`data/backup_staging/`), removed in `process``finally`.
- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus `compute_storage_stats()` (du of every major data area + `shutil.disk_usage`, run in `asyncio.to_thread` from the route, 30s in-process TTL cache so the walk never blocks).
- **Permanent artifact:** `cleanup(job)` only removes leftover staging, NEVER the archive. Job retention prunes the `jobs` row; the archive and `backups` row persist until an admin deletes it, a schedule rotates it out (`keep_last`), or `devplace backups clear`. Deleting a backup is a HARD delete (unlink file + delete row) - correct because backups are GC artifacts, the documented exception to the soft-delete rule.
`devplacepy/services/backup/offload.py` ships completed archives to a Hetzner Storage Box over WebDAV via `rclone` (`config.RCLONE_BIN`/`config.RCLONE_CONFIG_FILE`, remote name `config.BACKUP_OFFLOAD_REMOTE`, default `storagebox:devplacepy-backups`) - deliberately **not** the `/backup` davfs2 mount, whose FUSE metadata cache lives on the root filesystem and breaks exactly when disk fills (the original outage cause). `BackupService._run_offload_cycle` (throttled to `backup_offload_interval_seconds`, default 300s, via `ConfigField`s in the `Offload` group) runs each cycle after `_fire_due_schedules`:
1.`upload_pending` - every `done` backup with `remote_uploaded_at` unset and a live `local_path` is `rclone copyto`'d to `<remote>/<target>/<filename>`, then verified by exact byte-size match (`rclone size --json`) against `size_bytes` recorded at finalize time. Only on a verified match does `store.mark_remote_uploaded` set `remote_path`/`remote_uploaded_at`. A failed or unverified upload is silently retried next cycle - `remote_uploaded_at` is the only source of truth for "is this backup actually safe off-box."
2.`enforce_local_retention` (`backup_offload_keep_local`, default 1) - per target, keeps the newest N **offloaded** local copies and unlinks the rest (`store.mark_local_purged`: clears `local_path`, sets `local_purged_at`, row and `remote_path` persist). A backup with no confirmed remote copy is never touched, no matter how old.
3.`enforce_remote_retention` (`backup_offload_keep_remote`, default 30) - per target, `rclone lsjson` the remote dir and `deletefile` anything beyond the newest N, sorted by filename (safe because the `{target}-YYYYMMDD-HHMMSS-*` name is lexicographically chronological, same property `schedule.to_iso` relies on).
**`rotate_schedule` is offload-aware:** it now skips any row with an empty `remote_uploaded_at` - a schedule's `keep_last` can never hard-delete a backup that was never confirmed off-box, even if offload is disabled entirely (rotation then simply stops happening, which is the safe failure direction).
**`prune_orphans` (`devplace backups prune`) is offload-aware for the same reason:** a `done` row with a confirmed `remote_uploaded_at` is skipped even when its `local_path` is empty/missing - that is the normal steady state after `enforce_local_retention` purges the local copy, not an orphan. Only a `done` row with no confirmed remote copy AND a missing local file counts as truly orphaned and gets hard-deleted. Without this check, running the CLI prune after offload has done its job would delete the DB record for every successfully offloaded backup, discarding the only pointer to its `remote_path`.
**Operational prerequisite (production, not automatic):** the `rclone` binary is installed in the shipped Docker image, but a working WebDAV remote still needs to exist at `config.RCLONE_CONFIG_FILE` (default `$HOME/.config/rclone/rclone.conf` inside the app container, overridable via `DEVPLACE_RCLONE_CONFIG`) with a remote named to match `config.BACKUP_OFFLOAD_REMOTE`'s prefix (default `storagebox`) pointing at the Hetzner Storage Box's WebDAV endpoint and credentials - `rclone config` (interactive) or a hand-written `rclone.conf` generates it. Until that file exists, every `upload_pending` attempt fails fast (`rclone` errors "didn't find section") and is logged and retried next cycle; local retention and rotation both stay disabled the whole time (see above), so backups simply accumulate locally with no data loss, they just never leave the box. **In Docker, `HOME=/app` (the bind-mounted repo root, `docker-compose.yml`), so the default config path resolves to `<repo>/.config/rclone/rclone.conf` on the host - `.gitignore` excludes `/.config/` precisely because this file holds live remote-storage credentials; never force-add it.**
`backup_schedules` carry `kind` (`interval`|`cron`), `every_seconds`/`cron`, `enabled`, `keep_last`, `next_run_at`, run bookkeeping. `_fire_due_schedules` (lock-owner only, so each fires once) compares `next_run_at <= to_iso(now_utc())` and enqueues a `backup` job + a `backups` record, then advances `next_run_at` via `schedule.next_run`. **Timestamp format is load-bearing:** schedule `next_run_at` uses the devii `schedule.to_iso` format (`%Y-%m-%dT%H:%M:%S`, no tz/micros) on BOTH sides of the comparison so lexicographic compare equals chronological - do not mix it with `datetime.isoformat()`.
## Routes and frontend
- **Routes** (`routers/admin/backups.py`, all `require_admin`, mounted via `admin/__init__`): `GET /admin/backups` (dashboard, `respond(..., model=BackupDashboardOut)`), `GET /admin/backups/data` (JSON dashboard for the monitor + Devii), `POST /admin/backups/run`, `GET /admin/backups/jobs/{uid}` (`BackupJobOut` for `JobPoller`), `POST /admin/backups/{uid}/delete`, `GET /admin/backups/{uid}/download` (`FileResponse`, path-guarded against `BACKUPS_DIR`), `GET /admin/backups/{uid}` (`BackupOut`), and schedule CRUD `schedules/create|{uid}/edit|toggle|run|delete`. **Route order:** every literal sub-path (`data`, `run`, `jobs/{uid}`, `schedules/...`) is declared BEFORE the `{uid}` catch-alls.
- **Frontend:** `static/js/BackupMonitor.js` (`window.BackupMonitor`, started inline from `admin_backups.html` like `AiUsageMonitor`) polls `/admin/backups/data` every 8s and renders storage/backup/schedule sections; run uses `Http.send` + `JobPoller`; delete/toggle/run/schedule actions use delegated clicks + `Http.send`; the schedule modal (create/edit) reuses the shared `.modal-overlay`/`data-modal` pattern. CSS `static/css/backups.css`. Sidebar link in `admin_base.html` (`admin_section == 'backups'`).
## Download restricted to the primary administrator (load-bearing)
A backup archive contains the whole database, uploads, and VAPID keys, so `GET /admin/backups/{uid}/download` is restricted beyond `require_admin` to the **primary administrator** - the earliest-created user who currently holds the Admin role (the founder; `utils._create_account` auto-promotes the first registered user). The keystone is `database.get_primary_admin_uid()` (`SELECT uid FROM users WHERE role='Admin' ORDER BY created_at ASC, id ASC LIMIT 1`; users have no `deleted_at`, do NOT filter it) and `utils.is_primary_admin(user)` (admin AND `uid == get_primary_admin_uid()`, also a Jinja global). The download endpoint records `security.authz.denied` and raises `403` for any other admin. The `download_url` field is gated identically at every emission point (`_download_url`/`_backup_payload`/`_job_payload`, threaded `can_download = is_primary_admin(admin)`), so `GET /admin/backups/data`, `/jobs/{uid}`, and `/{uid}` never expose the URL to a non-primary admin; `BackupDashboardOut.can_download_backups` carries the flag. **Client:**`BackupMonitor.js` reads `data.can_download_backups`; a `done` backup renders an enabled `<a>` Download only for the primary admin, otherwise a **disabled**`<button>` with `title="Not available"`. The disabled state cannot be bypassed - the URL is never sent and the endpoint 403s regardless. If the founder is demoted/removed the crown passes to the next-oldest admin. Creating/running/deleting/scheduling backups stay open to all admins.
No restore into a live server (by design - overwriting the DB/uploads while running risks corruption). Restore is a manual ops procedure documented on the `backups` docs page. The live view relay's `admin.backups` topic publishes with `can_download=False` unconditionally (see `devplacepy/services/CLAUDE.md` -> Live view relay) - `BackupMonitor` derives real download capability only from its authoritative HTTP poll.
