Add WebDAV backup, statistics, progress and dashboard

- WebDAV uploader: concurrency, request and bandwidth limits, retry with
  backoff, offline catch-up, encrypted manifests, durability tracking
- Automatic unique remote directory claimed with a conditional PUT
- AES-256-GCM client-side encryption with keyed blob names
- /stats, /progress, /progress/stream and a /dashboard page
- Own ctypes inotify binding with a compact watch tree (263 MB -> 76 MB RSS)
- Directory-path ignore patterns and a forget command
- Indexed range queries instead of LIKE for path prefixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

readme.md

@@ -10,10 +10,11 @@
 
 | Milestone | State |
 |---|---|
-| M1 – Core (monitor, filters, index, spool, history, diff, restore, systemd install) | **Implemented** (`src/versiond/`) |
-| M2 – WebDAV remote, unique remote directory, encryption | Not started. Versions are stored locally (`durability = local`) |
-| M3 – Retention, purge, GC | Not started (pinning exists) |
-| M4 – Agent skill, metrics, reindex/adopt | Not started |
+| M1 – Core (monitor, filters, index, spool, history, diff, restore, systemd install) | **Implemented** |
+| M2 – WebDAV remote, unique remote directory, encryption, manifests | **Implemented** |
+| Statistics, progress (incl. live stream) and web dashboard | **Implemented** |
+| M3 – Retention, thinning, remote GC | Not started (pinning and `forget` exist) |
+| M4 – Agent skill, Prometheus metrics, reindex from remote | Not started |
 
 ```bash
 pipx install -e .            # installs the `versiond` command
@@ -24,6 +25,12 @@ versiond history ~/projects/app/main.py
 versiond diff ~/projects/app/main.py            # last change
 versiond restore '~/projects/app/src/*' --as-of 2026-10-08T14:00          # dry run
 versiond restore '~/projects/app/src/*' --as-of 2026-10-08T14:00 --execute
+versiond remote set --url https://u123456.your-storagebox.de --user u123456   # prompts for the password
+versiond key export          # store the backup key somewhere safe, off this machine
+versiond progress --follow   # scan + upload progress, rate and ETA
+versiond stats               # files, versions, storage, activity
+versiond dashboard           # opens the web dashboard, already signed in
+versiond forget ~/projects/app/data --execute   # drop history of something you now ignore
 ```
 
 API docs: <http://127.0.0.1:9922/docs>. API calls need `Authorization: Bearer $(versiond token)`.
@@ -66,7 +73,7 @@ The primary use case is a safety net for workflows where files are rewritten fre
 | Web framework | FastAPI + Uvicorn |
 | Bind address | `127.0.0.1:9922` (fixed default, overridable for testing only) |
 | Metadata store | SQLite (WAL mode) |
-| File monitoring | Linux inotify via `asyncinotify` (pure ctypes, asyncio-native, no dependencies) |
+| File monitoring | Linux inotify via a built-in ctypes binding (no dependencies), read on the asyncio loop |
 | Remote storage | WebDAV (RFC 4918) |
 | API docs | Swagger UI at `/docs`, ReDoc at `/redoc`, schema at `/openapi.json` |
 
@@ -181,11 +188,14 @@ Monitored roots do not need their own remote directories: blobs are shared (cont
 
 ```
 <base_path>/<hostname-slug>-<id8>/
-  blobs/sha256/ab/cd/<full-hash>.zst      # content-addressed, zstd-compressed (optionally encrypted)
-  manifests/<yyyy>/<mm>/<dd>/<batch-id>.jsonl.zst   # append-only version records
-  meta/format.json                        # storage format version
+  blobs/<name[:2]>/<name>                 # name = HMAC-SHA256(key, sha256); zstd, then AES-256-GCM
+  manifests/<yyyy>/<mm>/<dd>/<batch-id>.jsonl.zst.enc   # append-only version + rename records, encrypted
+  meta/owner.json                         # installation id, hostname, key id (plaintext)
+  meta/format.json                        # storage format version (plaintext)
 ```
 
+One directory level under `blobs/` (256 prefixes) keeps the number of `MKCOL` requests small.
+
 Blobs are immutable and deduplicated by SHA-256 of the plaintext content. Manifests are append-only journals. Together they make it possible to fully rebuild the index (`POST /admin/reindex`) after local data loss.
 
 ### 4.2 Ingestion
@@ -198,7 +208,7 @@ The user registers one or more **roots** (`POST /roots {"path": "~/projects"}`);
 
 | Option | Verdict |
 |---|---|
-| **Raw inotify (`asyncinotify`)** | **Chosen.** One kernel watch per *directory* (not per file), costing about 1 KB of kernel memory each. Events are delivered on one file descriptor read directly by the asyncio loop: no threads, no polling, ~0 % CPU when idle. Pure ctypes, no dependencies. |
+| **Raw inotify (own ctypes binding)** | **Chosen.** One kernel watch per *directory* (not per file), costing about 1 KB of kernel memory each. Events are delivered on one file descriptor read directly by the asyncio loop (`add_reader`): no threads, no polling, ~0 % CPU when idle. Watched directories are kept as a tree of `(wd, parent, name)` nodes, so a directory rename is one pointer change and memory stays at ~100 bytes per watch. `asyncinotify` was tried first; its per-watch `Path` objects cost ~260 MB RSS at 136k watches versus ~90 MB with the tree. |
 | `watchfiles` (Rust `notify`) | Good library, but its recursive mode adds a watch to *every* directory, including `node_modules`/`.venv`; filters only drop events afterwards. It can use up the watch limit on large trees, and it pulls in a compiled dependency. |
 | `watchdog` | Heavier (emitter threads, snapshot objects), same one-watch-per-directory cost, cross-platform layer we don't need. |
 | fanotify | Unprivileged use (kernel ≥ 5.13) does not allow filesystem- or mount-wide marks, so it still needs one mark per directory, with a more complex API. No benefit without root. |
@@ -286,7 +296,7 @@ All upload behaviour is configurable (`[upload]` section and `PATCH /config/uplo
 | `batch_manifest_interval_s` | 30 | Manifest flush interval |
 | `spool_max_mb` | 1024 | When full, ingest returns `507 Insufficient Storage` |
 
-Uploads are idempotent (content-addressed; `HEAD` before `PUT`). A version counts as **durable** only after its blob and manifest entry are both stored remotely. The API shows this state per version (`pending` / `durable`).
+Uploads are idempotent (content-addressed names, so a repeated `PUT` writes identical data). No `HEAD` is sent before `PUT`; that would double the requests for the common case where the blob is new. A version counts as **durable** only after its blob and manifest entry are both stored remotely. The API shows this state per version (`pending` / `durable`).
 
 When the remote is unreachable, the service keeps working from the spool and catches up when the connection returns.
 
@@ -373,6 +383,11 @@ All endpoints are JSON, under `/api/v1` (the docs, schema and skill endpoints ar
 | POST | `/versions/{id}/pin` / `DELETE` | Pin / unpin |
 | POST | `/purge` | Run a purge (criteria + `dry_run`) |
 | DELETE | `/versions/{id}`, `/files?path=`, `/projects/{id}` | Explicit deletion |
+| GET | `/stats` | Files, versions (per source, per hour for 24 h), storage, dedup/compression, top files and projects |
+| GET | `/progress` | Active and last scans (percent, files/s, ETA), upload queue (percent, rate, ETA, errors), monitor counters |
+| GET | `/progress/stream` | Server-sent events with `/progress` every `interval` seconds |
+| GET | `/dashboard` | Web dashboard (token via `#token=` fragment or prompt) |
+| POST | `/forget` | Delete all history below a path (`dry_run` by default) |
 | POST | `/admin/gc` | Garbage-collect remote blobs |
 | POST | `/admin/reindex` | Rebuild the index from remote manifests |
 | GET | `/admin/queue` | Upload queue status |
@@ -405,7 +420,7 @@ Indexes on `(file_id, captured_at)`, `files(path)`, `versions(blob_sha256)`. Sch
 
 - **Loopback only.** The server refuses to start if configured to bind to a non-loopback address.
 - **Local authentication.** Other local users and processes (including browsers, via DNS rebinding) can reach `127.0.0.1`. Every request therefore needs a bearer token stored in `~/.config/versiond/credentials` (`0600`). The `Host` header is checked against `127.0.0.1:9922`/`localhost:9922`, and CORS is disabled.
-- **Secrets in backed-up files.** `.env` and similar files contain credentials and are sent off-machine. Client-side encryption is therefore **on by default**: blobs and manifests are encrypted with XChaCha20-Poly1305 using a key held locally (with an export/recovery procedure documented at first run). Content-addressing uses a keyed hash (HMAC-SHA-256) so the remote cannot confirm guesses of known content.
+- **Secrets in backed-up files.** `.env` and similar files contain credentials and are sent off-machine. Client-side encryption is therefore **always on**: blobs and manifests are encrypted with AES-256-GCM (random 96-bit nonce) using a 64-byte key in `~/.config/versiond/backup.key` (`0600`). Remote blob names are HMAC-SHA-256 of the content hash, so the server cannot confirm guesses of known content. **The key is the only way to read the remote copy**: `versiond key export` prints it, and it must be stored off the machine.
 - **Credential storage.** WebDAV secrets are kept in the `0600` credentials file or, when available, the Secret Service/keyring. They never appear in logs or API responses.
 - **Restore path safety.** See §4.7.
 - **Audit log.** All configuration changes, restores and purges are recorded with timestamp and client identity.