devplacepy/devplacepy/services/backup/CLAUDE.md

This file documents the Backup service (devplacepy/services/backup/, devplacepy/services/jobs/backup_worker.py, devplacepy/routers/admin/backups.py). Claude Code loads it automatically whenever a file under devplacepy/services/backup/ is read or edited.

Overview

Admin-only, enterprise-grade backups built on the same async-job pattern as zip/fork (see devplacepy/services/jobs/CLAUDE.md), so nothing runs on the request path. BackupService(JobService) (kind backup) is registered in main.py and overrides run_once to call super().run_once() (reap/refill/sweep) and then _fire_due_schedules().

Targets and worker

• Targets (store.BACKUP_TARGETS): database (consistent SQLite snapshot of the main DB + devii_tasks.db + devii_lessons.db via the sqlite3 online backup API, so it is consistent under WAL - never a raw file copy), uploads (UPLOADS_DIR), keys (KEYS_DIR), full (database snapshot + uploads + keys; regenerable/volatile dirs - staging, locks, zips, container workspaces, chroma - are intentionally excluded). service._materialize(target, staging) returns a list of {root, path} sources; DB targets are snapshotted into staging/db_snapshot first.
• Worker (services/jobs/backup_worker.py, stdlib only): reads a JSON spec {sources:[{root,path}]} and an output path, builds a deterministic tar.gz (uid/gid 0, symlinks skipped) rooted at each root/, and returns {bytes_in, bytes_out, file_count, dir_count, sha256}. Invoked via asyncio.create_subprocess_exec like zip_worker.

Storage and data model

• Storage: archives go under config.BACKUPS_DIR (data/backups/, in DATA_PATHS) sharded with attachments._directory_for on the random uuid tail (same load-bearing reason as zips/blobs), named {target}-{YYYYMMDD-HHMMSS}-{tail}.tar.gz. Staging is config.BACKUP_STAGING_DIR (data/backup_staging/), removed in process finally.
• Data model (store.py, ensured in database.init_db via backup_store.ensure_tables()): backups (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and backup_schedules (in SOFT_DELETE_TABLES, born-live deleted_at:None). store holds all CRUD plus compute_storage_stats() (du of every major data area + shutil.disk_usage, run in asyncio.to_thread from the route, 30s in-process TTL cache so the walk never blocks).
• Permanent artifact: cleanup(job) only removes leftover staging, NEVER the archive. Job retention prunes the jobs row; the archive and backups row persist until an admin deletes it, a schedule rotates it out (keep_last), or devplace backups clear. Deleting a backup is a HARD delete (unlink file + delete row) - correct because backups are GC artifacts, the documented exception to the soft-delete rule.

Remote offload

devplacepy/services/backup/offload.py ships completed archives to a Hetzner Storage Box over WebDAV via rclone (config.RCLONE_BIN/config.RCLONE_CONFIG_FILE, remote name config.BACKUP_OFFLOAD_REMOTE, default storagebox:devplacepy-backups) - deliberately not the /backup davfs2 mount, whose FUSE metadata cache lives on the root filesystem and breaks exactly when disk fills (the original outage cause). BackupService._run_offload_cycle (throttled to backup_offload_interval_seconds, default 300s, via ConfigFields in the Offload group) runs each cycle after _fire_due_schedules:

1. upload_pending - every done backup with remote_uploaded_at unset and a live local_path is rclone copyto'd to <remote>/<target>/<filename>, then verified by exact byte-size match (rclone size --json) against size_bytes recorded at finalize time. Only on a verified match does store.mark_remote_uploaded set remote_path/remote_uploaded_at. A failed or unverified upload is silently retried next cycle - remote_uploaded_at is the only source of truth for "is this backup actually safe off-box."
2. enforce_local_retention (backup_offload_keep_local, default 1) - per target, keeps the newest N offloaded local copies and unlinks the rest (store.mark_local_purged: clears local_path, sets local_purged_at, row and remote_path persist). A backup with no confirmed remote copy is never touched, no matter how old.
3. enforce_remote_retention (backup_offload_keep_remote, default 30) - per target, rclone lsjson the remote dir and deletefile anything beyond the newest N, sorted by filename (safe because the {target}-YYYYMMDD-HHMMSS-* name is lexicographically chronological, same property schedule.to_iso relies on).

rotate_schedule is offload-aware: it now skips any row with an empty remote_uploaded_at - a schedule's keep_last can never hard-delete a backup that was never confirmed off-box, even if offload is disabled entirely (rotation then simply stops happening, which is the safe failure direction).

prune_orphans (devplace backups prune) is offload-aware for the same reason: a done row with a confirmed remote_uploaded_at is skipped even when its local_path is empty/missing - that is the normal steady state after enforce_local_retention purges the local copy, not an orphan. Only a done row with no confirmed remote copy AND a missing local file counts as truly orphaned and gets hard-deleted. Without this check, running the CLI prune after offload has done its job would delete the DB record for every successfully offloaded backup, discarding the only pointer to its remote_path.

Operational prerequisite (production, not automatic): the rclone binary is installed in the shipped Docker image, but a working WebDAV remote still needs to exist at config.RCLONE_CONFIG_FILE (default $HOME/.config/rclone/rclone.conf inside the app container, overridable via DEVPLACE_RCLONE_CONFIG) with a remote named to match config.BACKUP_OFFLOAD_REMOTE's prefix (default storagebox) pointing at the Hetzner Storage Box's WebDAV endpoint and credentials - rclone config (interactive) or a hand-written rclone.conf generates it. Until that file exists, every upload_pending attempt fails fast (rclone errors "didn't find section") and is logged and retried next cycle; local retention and rotation both stay disabled the whole time (see above), so backups simply accumulate locally with no data loss, they just never leave the box. In Docker, HOME=/app (the bind-mounted repo root, docker-compose.yml), so the default config path resolves to <repo>/.config/rclone/rclone.conf on the host - .gitignore excludes /.config/ precisely because this file holds live remote-storage credentials; never force-add it.

Schedules

backup_schedules carry kind (interval|cron), every_seconds/cron, enabled, keep_last, next_run_at, run bookkeeping. _fire_due_schedules (lock-owner only, so each fires once) compares next_run_at <= to_iso(now_utc()) and enqueues a backup job + a backups record, then advances next_run_at via schedule.next_run. Timestamp format is load-bearing: schedule next_run_at uses the devii schedule.to_iso format (%Y-%m-%dT%H:%M:%S, no tz/micros) on BOTH sides of the comparison so lexicographic compare equals chronological - do not mix it with datetime.isoformat().

Routes and frontend

• Routes (routers/admin/backups.py, all require_admin, mounted via admin/__init__): GET /admin/backups (dashboard, respond(..., model=BackupDashboardOut)), GET /admin/backups/data (JSON dashboard for the monitor + Devii), POST /admin/backups/run, GET /admin/backups/jobs/{uid} (BackupJobOut for JobPoller), POST /admin/backups/{uid}/delete, GET /admin/backups/{uid}/download (FileResponse, path-guarded against BACKUPS_DIR), GET /admin/backups/{uid} (BackupOut), and schedule CRUD schedules/create|{uid}/edit|toggle|run|delete. Route order: every literal sub-path (data, run, jobs/{uid}, schedules/...) is declared BEFORE the {uid} catch-alls.
• Frontend: static/js/BackupMonitor.js (window.BackupMonitor, started inline from admin_backups.html like AiUsageMonitor) polls /admin/backups/data every 8s and renders storage/backup/schedule sections; run uses Http.send + JobPoller; delete/toggle/run/schedule actions use delegated clicks + Http.send; the schedule modal (create/edit) reuses the shared .modal-overlay/data-modal pattern. CSS static/css/backups.css. Sidebar link in admin_base.html (admin_section == 'backups').
• Devii (all requires_admin=True, handler="http"): backups_overview, backup_run, backup_status, backup_delete (+confirm, in CONFIRM_REQUIRED), backup_schedule_create, backup_schedule_delete (+confirm). CLI: devplace backups list|run <target>|prune|clear. Audit: job.backup.complete|failed (category backup), admin.backup.run|delete, admin.backup_schedule.create|update|toggle|delete (category admin). Docs: admin API group endpoints + admin prose page backups.

Download restricted to the primary administrator (load-bearing)

A backup archive contains the whole database, uploads, and VAPID keys, so GET /admin/backups/{uid}/download is restricted beyond require_admin to the primary administrator - the earliest-created user who currently holds the Admin role (the founder; utils._create_account auto-promotes the first registered user). The keystone is database.get_primary_admin_uid() (SELECT uid FROM users WHERE role='Admin' ORDER BY created_at ASC, id ASC LIMIT 1; users have no deleted_at, do NOT filter it) and utils.is_primary_admin(user) (admin AND uid == get_primary_admin_uid(), also a Jinja global). The download endpoint records security.authz.denied and raises 403 for any other admin. The download_url field is gated identically at every emission point (_download_url/_backup_payload/_job_payload, threaded can_download = is_primary_admin(admin)), so GET /admin/backups/data, /jobs/{uid}, and /{uid} never expose the URL to a non-primary admin; BackupDashboardOut.can_download_backups carries the flag. Client: BackupMonitor.js reads data.can_download_backups; a done backup renders an enabled <a> Download only for the primary admin, otherwise a disabled <button> with title="Not available". The disabled state cannot be bypassed - the URL is never sent and the endpoint 403s regardless. If the founder is demoted/removed the crown passes to the next-oldest admin. Creating/running/deleting/scheduling backups stay open to all admins.

No restore into a live server (by design - overwriting the DB/uploads while running risks corruption). Restore is a manual ops procedure documented on the backups docs page. The live view relay's admin.backups topic publishes with can_download=False unconditionally (see devplacepy/services/CLAUDE.md -> Live view relay) - BackupMonitor derives real download capability only from its authoritative HTTP poll.