blindxfish/devplacepy
1
0
Fork 0
forked from retoor/devplacepy
Code Pull Requests Activity

Update

Browse Source
This commit is contained in:
retoor
2026-07-07 16:09:28 +02:00
parent 32c8bbe0a9
commit 5083efb150
42 changed files with 2556 additions and 2806 deletions
Download Patch File Download Diff File Expand all files Collapse all files
devplacepy/services/backup/CLAUDE.md
+32
View File
@@ -0,0 +1,32 @@
This file documents the Backup service (`devplacepy/services/backup/`, `devplacepy/services/jobs/backup_worker.py`, `devplacepy/routers/admin/backups.py`). Claude Code loads it automatically whenever a file under `devplacepy/services/backup/` is read or edited.
## Overview
Admin-only, enterprise-grade backups built on the **same async-job pattern as zip/fork** (see `devplacepy/services/jobs/CLAUDE.md`), so nothing runs on the request path. `BackupService(JobService)` (kind `backup`) is registered in `main.py` and overrides `run_once` to call `super().run_once()` (reap/refill/sweep) and then `_fire_due_schedules()`.
## Targets and worker
- **Targets** (`store.BACKUP_TARGETS`): `database` (consistent SQLite snapshot of the main DB + `devii_tasks.db` + `devii_lessons.db` via the `sqlite3` online backup API, so it is consistent under WAL - never a raw file copy), `uploads` (`UPLOADS_DIR`), `keys` (`KEYS_DIR`), `full` (database snapshot + uploads + keys; regenerable/volatile dirs - staging, locks, zips, container workspaces, chroma - are intentionally excluded). `service._materialize(target, staging)` returns a list of `{root, path}` sources; DB targets are snapshotted into `staging/db_snapshot` first.
- **Worker** (`services/jobs/backup_worker.py`, stdlib only): reads a JSON spec `{sources:[{root,path}]}` and an output path, builds a deterministic `tar.gz` (uid/gid 0, symlinks skipped) rooted at each `root/`, and returns `{bytes_in, bytes_out, file_count, dir_count, sha256}`. Invoked via `asyncio.create_subprocess_exec` like `zip_worker`.
## Storage and data model
- **Storage:** archives go under `config.BACKUPS_DIR` (`data/backups/`, in `DATA_PATHS`) sharded with `attachments._directory_for` on the **random uuid tail** (same load-bearing reason as zips/blobs), named `{target}-{YYYYMMDD-HHMMSS}-{tail}.tar.gz`. Staging is `config.BACKUP_STAGING_DIR` (`data/backup_staging/`), removed in `process` `finally`.
- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus `compute_storage_stats()` (du of every major data area + `shutil.disk_usage`, run in `asyncio.to_thread` from the route, 30s in-process TTL cache so the walk never blocks).
- **Permanent artifact:** `cleanup(job)` only removes leftover staging, NEVER the archive. Job retention prunes the `jobs` row; the archive and `backups` row persist until an admin deletes it, a schedule rotates it out (`keep_last`), or `devplace backups clear`. Deleting a backup is a HARD delete (unlink file + delete row) - correct because backups are GC artifacts, the documented exception to the soft-delete rule.
## Schedules
`backup_schedules` carry `kind` (`interval`|`cron`), `every_seconds`/`cron`, `enabled`, `keep_last`, `next_run_at`, run bookkeeping. `_fire_due_schedules` (lock-owner only, so each fires once) compares `next_run_at <= to_iso(now_utc())` and enqueues a `backup` job + a `backups` record, then advances `next_run_at` via `schedule.next_run`. **Timestamp format is load-bearing:** schedule `next_run_at` uses the devii `schedule.to_iso` format (`%Y-%m-%dT%H:%M:%S`, no tz/micros) on BOTH sides of the comparison so lexicographic compare equals chronological - do not mix it with `datetime.isoformat()`.
## Routes and frontend
- **Routes** (`routers/admin/backups.py`, all `require_admin`, mounted via `admin/__init__`): `GET /admin/backups` (dashboard, `respond(..., model=BackupDashboardOut)`), `GET /admin/backups/data` (JSON dashboard for the monitor + Devii), `POST /admin/backups/run`, `GET /admin/backups/jobs/{uid}` (`BackupJobOut` for `JobPoller`), `POST /admin/backups/{uid}/delete`, `GET /admin/backups/{uid}/download` (`FileResponse`, path-guarded against `BACKUPS_DIR`), `GET /admin/backups/{uid}` (`BackupOut`), and schedule CRUD `schedules/create|{uid}/edit|toggle|run|delete`. **Route order:** every literal sub-path (`data`, `run`, `jobs/{uid}`, `schedules/...`) is declared BEFORE the `{uid}` catch-alls.
- **Frontend:** `static/js/BackupMonitor.js` (`window.BackupMonitor`, started inline from `admin_backups.html` like `AiUsageMonitor`) polls `/admin/backups/data` every 8s and renders storage/backup/schedule sections; run uses `Http.send` + `JobPoller`; delete/toggle/run/schedule actions use delegated clicks + `Http.send`; the schedule modal (create/edit) reuses the shared `.modal-overlay`/`data-modal` pattern. CSS `static/css/backups.css`. Sidebar link in `admin_base.html` (`admin_section == 'backups'`).
- **Devii** (all `requires_admin=True`, `handler="http"`): `backups_overview`, `backup_run`, `backup_status`, `backup_delete` (+confirm, in `CONFIRM_REQUIRED`), `backup_schedule_create`, `backup_schedule_delete` (+confirm). **CLI:** `devplace backups list|run <target>|prune|clear`. **Audit:** `job.backup.complete|failed` (category `backup`), `admin.backup.run|delete`, `admin.backup_schedule.create|update|toggle|delete` (category `admin`). **Docs:** admin API group endpoints + admin prose page `backups`.
## Download restricted to the primary administrator (load-bearing)
A backup archive contains the whole database, uploads, and VAPID keys, so `GET /admin/backups/{uid}/download` is restricted beyond `require_admin` to the **primary administrator** - the earliest-created user who currently holds the Admin role (the founder; `utils._create_account` auto-promotes the first registered user). The keystone is `database.get_primary_admin_uid()` (`SELECT uid FROM users WHERE role='Admin' ORDER BY created_at ASC, id ASC LIMIT 1`; users have no `deleted_at`, do NOT filter it) and `utils.is_primary_admin(user)` (admin AND `uid == get_primary_admin_uid()`, also a Jinja global). The download endpoint records `security.authz.denied` and raises `403` for any other admin. The `download_url` field is gated identically at every emission point (`_download_url`/`_backup_payload`/`_job_payload`, threaded `can_download = is_primary_admin(admin)`), so `GET /admin/backups/data`, `/jobs/{uid}`, and `/{uid}` never expose the URL to a non-primary admin; `BackupDashboardOut.can_download_backups` carries the flag. **Client:** `BackupMonitor.js` reads `data.can_download_backups`; a `done` backup renders an enabled `<a>` Download only for the primary admin, otherwise a **disabled** `<button>` with `title="Not available"`. The disabled state cannot be bypassed - the URL is never sent and the endpoint 403s regardless. If the founder is demoted/removed the crown passes to the next-oldest admin. Creating/running/deleting/scheduling backups stay open to all admins.
No restore into a live server (by design - overwriting the DB/uploads while running risks corruption). Restore is a manual ops procedure documented on the `backups` docs page. The live view relay's `admin.backups` topic publishes with `can_download=False` unconditionally (see `devplacepy/services/CLAUDE.md` -> Live view relay) - `BackupMonitor` derives real download capability only from its authoritative HTTP poll.