Remove encryption; user-data filters; SQLite safety; recovery, retention, scheduler

- No client-side encryption: plaintext content-addressed remote (SHA-256
  names), no backup key, no cryptography dependency (server disk encryption
  is the trust model).
- Filters back user data: images, archives, databases, PDFs accepted; 10 MiB
  cap; binary sniffing removed; temp names hardened (~$, #..#, .temp).
- SQLite zero-error policy: backup-API snapshots + integrity_check, journal
  folding, locked/corrupt loud skips, verified restores.
- Recovery: reindex from manifests, remote adopt, on-demand blob fetch,
  5-day retention + thinning, GC, date-guarded remote purge, metrics.
- Scheduler with WebDAV quota signal and 70% pressure backstop (floor kept).
- 37 tests incl. live-monitor capture safety and DB safety.

research_notes/Retention scheduling adaptation/resource-adaptation.md

@@ -0,0 +1,102 @@
+# Resource Adaptation in Backup Tools
+
+## Which knobs exist (restic --limit-upload, borg --upload-ratelimit, kopia throttling, Time Machine low-priority I/O) and what are their defaults?
+
+### Takeaway
+All Linux tools use static, opt-in, unlimited-by-default rate/priority knobs; Time Machine instead imposes mandatory kernel-level low-priority I/O (IOPOL_THROTTLE) with no per-backup user knob.
+
+### Cited Findings
+- restic exposes global `--limit-upload rate` and `--limit-download rate` in KiB/s, default unlimited (0) — [Source](https://restic.readthedocs.io/en/stable/manual_rest.html); also documented in man pages as `--limit-upload=0` / `--limit-download=0` default unlimited — [Source](https://man.archlinux.org/man/restic-options.1.en)
+- restic documents that `--limit-upload` cannot be changed mid-run without restart; users request pv-like `-R` dynamic adjustment and resort to killing/restarting or external shapers — [Source](https://forum.restic.net/t/change-limit-upload-without-aborting-restic/5958)
+- Borg 1.x exposes `--upload-ratelimit RATE` in kiByte/s, default 0=unlimited (with `--remote-ratelimit` deprecated alias) — [Source](https://manpages.debian.org/bookworm/borgbackup/borg-common.1.en.html); also `--upload-buffer` size in MiB, default no buffer — [Source](https://man.archlinux.org/man/borg-common.1.en.txt)
+- Borg 2.x removed `--remote-ratelimit`/`--upload-ratelimit`; bandwidth is limited via `BORGSTORE_BANDWIDTH` in bits/sec, default 0=unlimited, plus `BORGSTORE_LATENCY` delay per call, or via `pv -L` ProxyCommand / rclone `--bwlimit` — [Source](https://borgbackup.readthedocs.io/en/latest/faq.html)
+- Older Borg 1.x FAQ documents upload-only `--remote-ratelimit` plus `pv`-wrapper + `BORG_RSH` for download shaping with on-the-fly `pv -R $(pidof pv) -L` changes — [Source](https://borgbackup.readthedocs.io/en/stable/faq.html)
+- Borg's limiter is a token-bucket `SleepingBandwidthLimiter` with `RATELIMIT_PERIOD = 0.1`, capping burst quota at 2x period allowance — [Source](https://github.com/borgbackup/borg/blob/da3105f1/src/borg/remote.py)
+- Kopia exposes `repository throttle set` flags `--upload-bytes-per-second`, `--download-bytes-per-second`, `--concurrent-reads/writes`, `--read/write-requests-per-second`, `--list-requests-per-second` — [Source](https://kopia.io/docs/reference/command-line/common/repository-throttle-set/); readable via `repository throttle get` — [Source](https://kopia.io/docs/reference/command-line/common/repository-throttle-get/); server-side equivalent `server throttle set` — [Source](https://kopia.io/docs/reference/command-line/common/server-throttle-set/)
+- Kopia direct-connect backends accept `--max-upload-speed`/`--max-download-speed` bytes/sec at `repository connect` time (e.g. B2/S3), persisted as `maxUploadSpeedBytesPerSecond` in repository.config — [Source](https://kopia.discourse.group/t/limit-upload-speed-as-a-policy/990)
+- Kopia `repository throttle set` is rejected on server-connected repos ("operation supported only on direct repository"), and per-policy/UI global throttle was still missing as of 2024–2026 feature requests — [Source](https://github.com/kopia/kopia/issues/3051); KopiaUI throttle exposure requested — [Source](https://github.com/kopia/kopia/issues/3586)
+- Kopia upload throttling was bursty (whole-object sleeps) until PR #2682 added per-read `DuringUpload` throttling — [Source](https://github.com/kopia/kopia/pull/2682)
+- Time Machine's `backupd` runs at `IOPOL_THROTTLE`, defined as "long-running I/O intensive background work, such as backups" that "will be throttled to prevent impact on higher policy levels" — [Source](https://eclecticlight.co/2022/01/20/why-time-machine-backups-can-be-interminably-slow/)
+- Full `IOPOL` ladder is IMPORTANT (default) / STANDARD / UTILITY / THROTTLE / PASSIVE — [Source](https://eclecticlight.co/2026/03/28/explainer-i-o-throttling/)
+- Global kill-switch `sudo sysctl debug.lowpri_throttle_enabled=0` (re-enable with `=1`, lost on reboot unless persisted via `/etc/sysctl.conf` or LaunchDaemon) removes throttle for all background I/O, not just backupd — [Source](https://osxdaily.com/2016/04/17/speed-up-time-machine-by-removing-low-process-priority-throttling/); same command/LaunchDaemon recipe — [Source](https://apple.stackexchange.com/questions/181609/time-capsule-wired-backup-transfer-slow-with-fast-bursts); throttling is I/O not CPU — [Source](https://mjtsai.com/blog/2016/03/16/massively-speed-up-time-machine-backups/)
+- Measured effect of disabling throttle: copying phase 193→332 MB/s, overall backup 160→276 MB/s (>10 GB test); pre-backup 50 MB probe writes unaffected — [Source](https://eclecticlight.co/2022/02/28/does-removing-i-o-throttling-make-backups-faster/)
+- Duplicity has no native generic bandwidth-limit option (open bug #1291633); workarounds are `trickle -s -u/-d`, WonderShaper/tc, or legacy `--scp-command="scp -l N"` (kbit/s, scp backend only, option later deprecated) — [Source](https://bugs.launchpad.net/bugs/1291633); scp `-l` throttle Q&A — [Source](https://lists.libreplanet.org/archive/html/duplicity-talk/2007-09/msg00058.html); router-QoS/DSCP attempts reported ineffective, per-machine Bandwidth Limiter used instead — [Source](https://lists.libreplanet.org/archive/html/duplicity-talk/2021-09/msg00000.html)
+- Neither restic, borg, kopia, nor duplicity ships battery- or metered-network-aware auto-pause; scheduling/power-awareness is delegated to systemd timers, DAS-CTS (macOS), or external shapers (see Gaps).
+
+### Inferences
+- Rate limiting is consistently an operator-set static ceiling, not adaptive congestion control; only pv/WonderShaper/BORGSTORE allow out-of-band changes without restarting the backup.
+- Borg 2.x shifts shaping out of borg CLI into the borgstore layer, unifying ssh/sftp/rclone paths but breaking old `--upload-ratelimit` scripts.
+
+### Gaps
+- No reliable source found for a built-in metered-WiFi or on-battery auto-suspend in restic/borg/kopia/duplicity as of 2026; Time Machine power/thermal inputs are internal to DAS scoring, not user flags.
+
+## Does any tool auto-tune from measured resources (disk size, RAM, link speed) rather than static config, and how?
+
+### Takeaway
+No tool auto-tunes from measured disk/RAM/link speed; the only resource-derived defaults are CPU-count-derived worker counts, everything else is fixed static defaults.
+
+### Cited Findings
+- restic defaults: file-read concurrency 2 ("sweet spot" from HDD experiments), blob-save concurrency = `runtime.NumCPU()`, tree-save concurrency = 20x blob concurrency — [Source](https://github.com/restic/restic/blob/de9136b29f86216bd3e41397d19b25f26b578833/internal/archiver/archiver.go)
+- restic uses all available CPUs by default; `GOMAXPROCS=1` pins to one core and slightly reduces memory — [Source](https://restic.readthedocs.io/en/latest/047_tuning_parameters.html)
+- restic backend connection limit defaults to 5 (2 for local backend), tunable via `-o rest.connections=5` / `-o local.connections=2`; too-high values degrade performance — [Source](https://restic.readthedocs.io/en/latest/047_tuning_parameters.html)
+- restic `--read-concurrency` / `RESTIC_READ_CONCURRENCY` raises parallel file reads for NVMe; `--no-scan` skips the pre-backup file-count/size scan that costs extra I/O on network/FUSE mounts — [Source](https://restic.readthedocs.io/en/latest/047_tuning_parameters.html); read-concurrency flag added in 0.15.0 for fast storage — [Source](https://restic.net/blog/2023-01-12/restic-0.15.0-released/)
+- Single-large-file chunking in restic is sequential per file (~450 MB/s) with parallel hash/compress/encrypt downstream; multi-file parallelism is what scales, which is why `cores/4`-style read-concurrency guesses only hold for SSDs — [Source](https://github.com/restic/restic/issues/4477)
+- Kopia `--max-parallel-file-reads` defaults to number of logical CPU cores; lowering it lowers CPU at cost of time — [Source](https://kopia.io/docs/faqs/)
+- Kopia `--max-parallel-snapshots` controls simultaneous snapshots (server/KopiaUI); s2 `default`/`better` compressor concurrency equals logical core count, `s2-parallel-4/8` pins it — [Source](https://kopia.io/docs/advanced/compression/)
+- Time Machine scheduling via DAS-CTS scores each due background activity every few seconds against temperature, load, and priority, dispatching `com.apple.backupd-auto` via XPC only when score exceeds threshold — [Source](https://eclecticlight.co/2023/11/28/scheduling-and-dispatch-of-backups-and-other-background-activities/)
+- On Apple Silicon, background-QoS `backupd` threads are confined to Efficiency cores at reduced frequency (~972–1332 MHz, ~90% residency on E cores), capping throughput at ~300–400 items/s regardless of queue depth — [Source](https://eclecticlight.co/2022/01/20/why-time-machine-backups-can-be-interminably-slow/)
+- No evidence any tool probes link speed, disk size, or free RAM to set pack size, connections, or limits automatically; restic 16 MiB pack size, Kopia parallelism, and Borg rate defaults are all static.
+
+### Inferences
+- "Auto-tuning" in this space means CPU-count-proportional worker pools plus OS-scheduler deferral (DAS-CTS/QoS), not closed-loop adaptation to throughput or memory pressure.
+- Raising concurrency without raising memory (restic 100×1 GB test hit 300 GB RAM at connections=16/reads=16 and OOMed) shows why static defaults stay conservative — [Source](https://github.com/restic/restic/issues/4477).
+
+### Gaps
+- No source found documenting link-speed probing or disk-size-derived chunk/pack sizing in any of the five tools; if it exists it is not in public docs/CLI help.
+
+## How do they behave on constrained boxes (low RAM SQLite/index handling, single-core compression choices)?
+
+### Takeaway
+Constrained-box guidance is manual: pick cheap compression, lower parallelism/connections, enlarge packs, and accept slower runs; low-RAM index handling remains a known failure mode, not an auto-degraded mode.
+
+### Cited Findings
+- Borg compression default is lz4 (very high speed, very low compression); alternatives `zstd[,L]` (default level 3), `zlib`, `lzma`, `auto,`, `none` — [Source](https://borgbackup.readthedocs.io/en/stable/usage/help.html); usage examples recommend `zlib,6` for ratio at cost of speed — [Source](https://borgbackup.readthedocs.io/en/stable/usage/create.html)
+- Upstream Borg PR proposes moving default from `lz4` to `zstd,-4` (multithreaded, as-fast-or-faster creates, slightly better ratio; large incompressible-image corpora stay ~11% slower) with MT workers capped at 4 — [Source](https://github.com/borgbackup/borg/pull/10100)
+- Kopia compression is disabled by default and set per-policy via `kopia policy set [--global] --compression=<...>` with min/max-size gates — [Source](https://kopia.io/docs/faqs/); full option list includes `s2-default/better/parallel-4/8`, `zstd/zstd-fastest/better`, `gzip/pgzip/deflate` variants — [Source](https://kopia.io/docs/reference/command-line/common/policy-set/)
+- Kopia FAQ names compression + parallelism as the two main memory culprits; recommends disabling compression or using `s2`/`deflate`/`gzip` on small files under low memory, and lowering `--max-parallel-snapshots` / `--max-parallel-file-reads` — [Source](https://kopia.io/docs/faqs/)
+- Kopia benchmark table (466 MiB corpus): s2-default 4 GiB/s at ~375 MiB RSS vs zstd 323 MiB/s at ~238 MiB vs zstd-best 19 MiB/s; on tiny files s2 stays fastest with ~2 MiB footprint — [Source](https://kopia.io/docs/advanced/compression/)
+- Kopia zstd levels map to upstream klauspost/compress: fastest≈1, default≈3, better≈7, best≈11; higher custom levels (e.g. -22 --ultra --long) require code change — [Source](https://kopia.discourse.group/t/what-is-zstd-best-compression-and-can-i-customize/4934)
+- restic on constrained HDD/NVMe: forum-tested recipe for 2.5 TB SMR-USB run is `--read-concurrency 1 --pack-size 128 --no-cache` (pack default 16 MiB raised to reduce file count and per-pack latency), with `local.connections=1` suggested for vibration-sensitive HDDs — [Source](https://forum.restic.net/t/first-backup-2-5tb-50-hours-can-i-improve-it/8288)
+- restic large-pack tradeoff: bigger packs reduce file count and help HDD/Swift/Drive limits but need more `$TMPDIR` staging (64–384 MiB guidance) and longer single-pack uploads that wear SSDs — [Source](https://github.com/restic/restic/blob/master/doc/047_tuning_parameters.rst)
+- Borg on near-full disks: 1.5 GiB-free VM case shows Borg needs headroom for segments/cache/index; workarounds discussed are extreme compression or `--upload-ratelimit` pacing plus inotify/SIGSTOP hacks, with maintainer warning such boxes are unsuitable — [Source](https://github.com/borgbackup/borg/issues/7107)
+- Single-core guidance converges: `GOMAXPROCS=1` (restic), `-C none|laz4` (borg), `--compression=s2-default|none` + `--max-parallel-file-reads=1` (kopia) minimize CPU/RAM at cost of ratio/throughput.
+
+### Inferences
+- Low-RAM behavior is fail-slow/fail-OOM rather than graceful: operators must pre-lower parallelism and compression before the run.
+- s2-family (kopia) and lz4/zstd,-4 (borg) are the single-core-friendly choices; zlib/lzma/zstd-best are ratio-first and can dominate a weak CPU.
+
+### Gaps
+- No citable SQLite/index RAM formula (bytes-per-file or per-GB-repo) found for current restic/borg/kopia versions; vendor docs describe symptoms and knobs, not a sizing equation.
+
+## What systemd-level controls (MemoryMax, CPUQuota, IOWeight) do shipped units use, and what do docs recommend for background backup daemons?
+
+### Takeaway
+Upstream backup tools ship no restrictive resource-control units; all concrete CPU/memory/I/O caps come from downstream/community units and generic systemd resource-control docs, with `Nice=` + `CPUQuota`/`MemoryMax`/`IOWeight` as the recommended trio.
+
+### Cited Findings
+- systemd `CPUQuota=` sets a hard ceiling as % of one CPU (100%=1 core, 200%=2 cores) via `cpu.max`/`cpu.cfs_quota_us`; `CPUWeight=` (1–10000, default 100) is only relative under contention — [Source](https://manpages.debian.org/bullseye/systemd/systemd.resource-control.5.en.html); same semantics in Arch man — [Source](https://man.archlinux.org/man/systemd.resource-control.5)
+- systemd memory knobs: `MemoryHigh=` soft throttle/reclaim, `MemoryMax=` hard OOM-kill limit (K/M/G/T or % of RAM, `infinity` to disable), `MemorySwapMax=` swap cap; Red Hat recommends `MemoryHigh` as main control, `MemoryMax` as last defense — [Source](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/assembly_configuring-resource-management-using-systemd_managing-monitoring-and-updating-the-kernel)
+- Community restic timer units use only `Nice=17` (low CPU priority) plus sandboxing (`ProtectSystem=full`, `PrivateTmp=true`, `NoNewPrivileges=yes`, `RestrictAddressFamilies=`, `SystemCallFilter=`, `AmbientCapabilities=CAP_DAC_READ_SEARCH`), with `RandomizedDelaySec=300` + `Persistent=yes` on the timer — [Source](https://www.wildtechgarden.ca/onepagers/real-life-systemd-timers/)
+- Community segmented-borg systemd design sets `CPUQuota=80%` and `MemoryMax=2G` on the backup service template — [Source](https://github.com/JoZapf/segmented-borg-backup-system/blob/refs/heads/main/docs/SYSTEMD.md)
+- Modern Debian guidance: background CPU → `nice -n 19`; background disk → `ionice -c 3` (BFQ only); hard ceilings/group fairness → unit with `CPUQuota=`/`MemoryMax=`/`IOWeight=`/`IOReadBandwidthMax=`/`IOWriteBandwidthMax=` or a shared `backup.slice`; one-shots via `systemd-run --scope -p CPUQuota=50% -p MemoryMax=1G -p IOWeight=10` — [Source](https://www.bigiron.cc/guides/process-priorities-nice-ionice-cgroup-quotas)
+- Same guide warns `ionice` is silently ignored on default `mq-deadline` SSD schedulers (only BFQ honors classes); `MemoryMax` kills rather than slows (use `MemoryHigh` for pushback); `CPUQuota=100%` means one core — [Source](https://www.bigiron.cc/guides/process-priorities-nice-ionice-cgroup-quotas)
+- Example backup slice caps restic at 80 MB/s read / 30 MB/s write via `IOReadBandwidthMax=/dev/sda 80M` + `IOWriteBandwidthMax=/dev/sda 30M` with `IOWeight=10` so foreground pools keep headroom — [Source](https://www.bigiron.cc/guides/process-priorities-nice-ionice-cgroup-quotas)
+- No shipped upstream restic/borg/kopia unit found with `MemoryMax`/`CPUQuota`/`IOWeight` preset; ArchWiki/packaging ships timers/services without resource caps (report-writer: treat absence as finding, not oversight).
+
+### Inferences
+- Recommended Linux server pattern: `Nice=17–19` + optional `IOSchedulingClass=idle` for soft priority, plus a `backup.slice` with `CPUQuota=50–80%`, `MemoryHigh/Max=1–4G`, `IOWeight=10` and per-device `IOWriteBandwidthMax` for hard isolation.
+- `nice`/`ionice` alone are insufficient for noisy-neighbor backups on cgroup-v2/BFQ-mixed fleets; systemd slice quotas are the enforceable layer.
+
+### Gaps
+- Could not locate a distribution-shipped (Debian/Fedora/Arch package) backup unit that presets `MemoryMax`/`CPUQuota`/`IOWeight`; all cited values are community/blog recommendations, not vendor defaults.
+- No citable official restic/borg/kopia doc page prescribing a canonical systemd resource-control stanza as of 2026.