retoor/versioning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove encryption; user-data filters; SQLite safety; recovery, retention, scheduler

Browse Source 
- No client-side encryption: plaintext content-addressed remote (SHA-256
  names), no backup key, no cryptography dependency (server disk encryption
  is the trust model).
- Filters back user data: images, archives, databases, PDFs accepted; 10 MiB
  cap; binary sniffing removed; temp names hardened (~$, #..#, .temp).
- SQLite zero-error policy: backup-API snapshots + integrity_check, journal
  folding, locked/corrupt loud skips, verified restores.
- Recovery: reindex from manifests, remote adopt, on-demand blob fetch,
  5-day retention + thinning, GC, date-guarded remote purge, metrics.
- Scheduler with WebDAV quota signal and 70% pressure backstop (floor kept).
- 37 tests incl. live-monitor capture safety and DB safety.
This commit is contained in:
retoor
2026-10-10 03:41:36 +02:00
parent 7e05e3d26c
commit 3499f6cda0
35 changed files with 3148 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reports/Backup encryption security choices.md
+39
View File
@@ -0,0 +1,39 @@
# Versiond should encrypt locally, hide metadata
For a local Linux daemon that stages small files and syncs to a dumb WebDAV store, the winning pattern is Kopia-style client-side envelope encryption with **AES-256-GCM as default and ChaCha20-Poly1305 as portable fallback**, applied after local spool-compression in the strict order hash then compress then encrypt, with per-blob random nonces and per-content keys derived by HMAC/HKDF. Restic proves that bespoke **AES-256-CTR plus Poly1305-AES in Encrypt-then-MAC with 16-byte random IV per file** works ([Source](https://github.com/restic/restic/blob/master/doc/design.rst)), Borg 1.x proves that **AES-256-CTR plus HMAC-SHA256 with 8-byte counter IVs and reservation tracking** scales poorly to multiple writers ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)), and Borg 2 plus Kopia converge on standard AEAD precisely to escape that complexity with **AES-256-OCB or ChaCha20-Poly1305 with session keys** ([Source](https://github.com/borgbackup/borg/wiki/Borg-2.0)) and **AES256-GCM-HMAC-SHA256 default, immutable after creation** ([Source](https://github.com/kopia/kopia/blob/master/site/content/docs/FAQs/_index.md)). Duplicity delegates entirely to **GnuPG OpenPGP hybrid encryption of tar volumes** ([Source](https://man.archlinux.org/man/duplicity.1.en)) and therefore offers no daemon-friendly manifest story. Because WebDAV provides no compute, no trustworthy mtime or hash, and no append-only enforcement, versiond must do what Kopia already does for WebDAV-class stores — encrypt, pack into **20–40 MB randomly named blobs with server-opaque indexes** ([Source](https://kopia.io/docs/advanced/architecture/)), authenticate manifests with the same keys as data, and add rollback resistance through a local monotonic counter plus server-side versioning rather than cryptography alone.
## AES-GCM and ChaCha20-Poly1305 replace bespoke CTR-MAC designs
Restic encrypts every repository object except key wrappers as **IV(16) || CIPHERTEXT || MAC(16) for 32 bytes overhead** with a fresh CSPRNG nonce per encryption ([Source](https://restic.readthedocs.io/en/v0.4.0/Design), and derives its three subkeys by splitting 64 scrypt-derived bytes into a **32-byte AES-256 key plus 16-byte AES key k plus 16-byte Poly1305 key r** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)). Pack files then carry multiple independently encrypted blobs plus an encrypted header and little-endian header length, with blob type bits distinguishing plain versus compressed data and tree objects ([Source](https://restic.readthedocs.io/en/stable/100_references.html)). External review found the construction sane, quoting that **encryption is a first-class feature and the deduplication trade-off is worth it** ([Source](https://github.com/restic/restic/blob/master/doc/070_encryption.rst)), but the design is deliberately non-standard and carries Poly1305-AES masking complexity that modern AEAD avoids. Borg 1.x chose a different bespoke path with **AES-CTR-256 plus HMAC-SHA256 in Encrypt-then-MAC** ([Source](https://manpages.debian.org/bullseye/borgbackup/borg-init.1.en.html)) and a **TYPE(1) + HMAC(32) + NONCE(8) + CIPHERTEXT envelope using two different keys** ([Source](https://borgbackup.readthedocs.io/en/1.0-maint/internals.html)), where only 8 stored nonce bytes cap capacity at **2**64 times 16 bytes** ([Source](https://borgbackup.readthedocs.io/en/1.0-maint/internals.html)) and uniqueness depends on **reserving counter ranges by adding 4 GiB divided by 16 bytes and persisting through SaveFile** ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)). That reservation breaks under independent writers, with Borg documenting that **with multiple independent clients on the same repo Borg fails to provide confidentiality** ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)), which is exactly why Borg 2 abandons counters for session keys and standard AEAD. Borg 2 offers **aes256-ocb, chacha20-poly1305, authenticated-only and none modes** ([Source](https://borgbackup.readthedocs.io/en/latest/usage/repo-create.html)) to get **256-bit authenticated encryption client-side** ([Source](https://github.com/borgbackup/borg)) with goals stated as getting rid of CTR, using session keys, and moving to **Argon2 plus AES-OCB and ChaCha20-Poly1305** ([Source](https://github.com/borgbackup/borg/wiki/Borg-2.0)). Kopia made the same modern choice earlier with **DefaultAlgorithm AES256-GCM-HMAC-SHA256** ([Source](https://pkg.go.dev/github.com/kopia/kopia/repo/encryption)) and an explicit option for **CHACHA20-POLY1305-HMAC-SHA256 fixed at creation** ([Source](https://github.com/kopia/kopia/blob/master/site/content/docs/FAQs/_index.md)), where per-content keys come from **HMAC-SHA256 derivation with 28 bytes overhead** ([Source](https://github.com/kopia/kopia/blob/master/repo/encryption/aes256_gcm_hmac_sha256_encryptor.go)) and the repository format block itself is sealed with **Ke equals HKDF-SHA256 of Km over UniqueID for AES plus AD equals HKDF for CHECKSUM** ([Source](https://kopia.io/docs/advanced/encryption/)). For versiond the implication is direct: use a Go-available AEAD with random 12-byte nonces per blob and HKDF-separated data versus manifest keys, pick AES-GCM where **hardware AES favours HMAC-SHA256 paths and portable CPUs favour BLAKE2/ChaCha thinking** already documented by Borg ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)), and never invent a CTR-plus-MAC variant or shell out to GnuPG per sync like Duplicity does.
| System | Cipher and integrity | Nonce and key separation |
|---|---|---|
| restic | AES-256-CTR + Poly1305-AES Encrypt-then-MAC ([Source](https://github.com/restic/restic/blob/master/doc/design.rst)) | 16-byte random IV per file, 32 B overhead ([Source](https://restic.readthedocs.io/en/v0.4.0/Design)) |
| Borg 1.x | AES-CTR-256 + HMAC-SHA256 or keyed BLAKE2b-256 ([Source](https://manpages.debian.org/bullseye/borgbackup/borg-init.1.en.html)) | 64-bit counter with reservation commit ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)) |
| Borg 2 | AES-256-OCB or ChaCha20-Poly1305 AEAD ([Source](https://borgbackup.readthedocs.io/en/latest/usage/repo-create.html)) | Session keys to avoid global counter tracking ([Source](https://github.com/borgbackup/borg/wiki/Borg-2.0)) |
| Kopia | AES256-GCM-HMAC-SHA256 default, ChaCha20 variant optional ([Source](https://github.com/kopia/kopia/blob/master/site/content/docs/FAQs/_index.md)) | Per-content HMAC-derived keys, format block with Ke and AD separation ([Source](https://kopia.io/docs/advanced/encryption/)) |
| Duplicity | External GnuPG OpenPGP hybrid with per-recipient session keys ([Source](https://gnupg.org/ftp/blurbs/an-advanced-introduction-to-gnupg.pdf)) | Random session key s plus Enc_ri(s) per recipient, cipher negotiated by GnuPG ([Source](https://gnupg.org/ftp/blurbs/an-advanced-introduction-to-gnupg.pdf)) |
In transit the same client-side property carries over, which matters for dumb WebDAV over plain HTTP. Borg assumes **client trusted and repository untrusted with full read-write man-in-the-middle** ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)) and runs RPC **over system SSH with no own network code** ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)), while Kopia states that **all repository features are implemented client-side without any need for a custom server, thus encryption keys never leave the client** ([Source](https://github.com/kopia/repo/blob/master/README.md)). Restic promises that **everything except informational key-file metadata is encrypted and authenticated** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)) and Duplicity claims volumes **will be safe from spying and modification by the server** via librsync plus GnuPG ([Source](https://packages.debian.org/bullseye/arm64/utils/duplicity)). Versiond therefore gets confidentiality even over untrusted HTTP, but still needs TLS or SSH tunnelling to blunt traffic analysis and to stop a network attacker from deleting or replaying blobs undetected, since object MACs detect tampering but not deletion.
## Hashing before compressing before encrypting preserves deduplication
All three content-defined systems converge on **split then hash then compress then encrypt then pack**, and every deviation is documented as an anti-pattern. Restic hashes plaintext for identity in saveBlob before zstd compression and Seal, and compresses unpacked index and snapshot JSON before encryption ([Source](https://github.com/restic/restic/blob/master/internal/repository/repository.go)), explicitly noting that **unpacked files like lock, index and snapshot files are also compressed before encryption** ([Source](https://github.com/restic/restic/pull/3666)). Borg follows **id equals MAC of id_key over data, then compress, then AEAD encrypt with id and header as associated data** ([Source](https://borgbackup.readthedocs.io/en/latest/internals/security.html)) and documents that **compression is applied after deduplication, thus different methods do not influence deduplication** ([Source](https://manpages.debian.org/trixie/borgbackup2/borg2-compression.1.en.html)). Kopia pipelines **splits into chunks, hash, compare, if new compress chunk, encrypt, pack** ([Source](https://kopia.io/docs/advanced/compression/)) after deliberately rewiring from read-split-compress-hash-store to **read-split-hash-compress-store to stop compressor drift from changing IDs** ([Source](https://github.com/kopia/kopia/pull/1076)). The rationale is physical: ciphertext from randomized encryption does not compress and destroys equality, so encryption must be last while hashing must be first, and compressing whole files before chunking avalanches small edits through the compressor and shifts cut points ([Source](https://github.com/restic/restic/issues/790)). Kopia even quantifies the accepted cost of chunk-then-compress versus whole-file compression with s2 and gzip ratios differing by a few points in exchange for shift-resilient dedup ([Source](https://kopia.io/docs/advanced/compression/)). Versiond spool-compressing many small files locally should copy this exactly: chunk the spool with a content-defined splitter, hash plaintext for the dedup index, compress each new chunk independently, then AEAD-encrypt.
Codec handling differs in ways versiond can borrow selectively. Restic repository v2 supports **zstandard only for data and tree blobs** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)) negotiated per run as **off, fastest, auto default, better, max** mapped to klauspost compress levels ([Source](https://github.com/restic/restic/blob/master/internal/repository/repository.go)), recording compressed blobs with an extra **plaintext length field plus uncompressed_length in the index** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)) so that **compressed and uncompressed blobs may be mixed in one pack** without re-uploading old data ([Source](https://restic.readthedocs.io/en/stable/100_references.html)). Borg records **ctype plus clevel plus csize plus size per object** for **none, lz4, zstd, zlib and lzma** ([Source](https://borgbackup.readthedocs.io/en/latest/internals/data-structures.html)), defaults to **lz4** and explicitly allows the first writer to fix a chunk's stored compression while later runs mix freely ([Source](https://manpages.debian.org/trixie/borgbackup2/borg2-compression.1.en.html)), plus wrappers for **auto selection and obfuscate padding with about 12 percent overhead** ([Source](https://manpages.debian.org/trixie/borgbackup2/borg2-compression.1.en.html)). Kopia disables compression by default, controls it per policy path, offers a wide menu from **s2-default through pgzip to zstd-better-compression with zstd recommended** ([Source](https://kopia.io/docs/faqs/)), and since index v2 keeps **per-content compression IDs in manager bookkeeping visible via content list and stats** rather than in the ID itself ([Source](https://github.com/kopia/kopia/pull/1076)), so **dedup is unaffected because hashing precedes compression** ([Source](https://kopia.discourse.group/t/deduplication-and-compression/714)). For versiond the practical synthesis is to store one zstd default for text-like spool files plus s2 for speed-sensitive paths, record codec and level outside the content ID exactly as Borg and Kopia do, freely mix codecs across blobs, and skip compression when output grows, mirroring Kopia storing the original when compressed output is larger ([Source](https://kopia.io/docs/advanced/compression/)). Integrity then comes free from the AEAD tag plus content addressing: restic verifies **filenames as hex SHA-256 of ciphertext with sha256sum** and checks structure with optional read-data payload reads ([Source](https://restic.readthedocs.io/en/stable/100_references.html)), Borg binds **chunk ID as associated data so the server cannot swap content under an ID** ([Source](https://borgbackup.readthedocs.io/en/latest/internals/security.html)), and Kopia packs blocks with a trailing local index plus top-level index for rebuilds ([Source](https://kopia.io/docs/advanced/architecture/)).
## Envelope encryption with scrypt or Argon2id decides recoverability
Every system separates a random master secret from a password-derived wrapping key, but parameters and ergonomics decide whether versiond operators survive key loss. Restic pins its key-file example to **scrypt N equals 65536, r equals 8, p equals 1** and rejects any other KDF ([Source](https://github.com/restic/restic/blob/9e2d60e2/internal/repository/key.go)), calibrating stronger parameters only when adding keys ([Source](https://github.com/restic/restic/blob/9e2d60e2/internal/repository/key.go)). Borg 1.x wraps keys with **PBKDF2-HMAC-SHA256 plus AES-CTR with zero IV and HMAC** ([Source](https://github.com/borgbackup/borg/blob/86fd77fd/src/borg/legacy/crypto/key.py)), while Borg 2 derives a **256-bit key-encryption key with Argon2 plus random 256-bit salt then ChaCha20-Poly1305 with constant IV zero** ([Source](https://borgbackup.readthedocs.io/en/2.0.0b9/internals/security.html)) and makes **Argon2id the default with PBKDF2 kept for compatibility** ([Source](https://git.uninsane.org/shelvacu-mirrors/borg/commit/08f82ee40867f605ca6994db6dc32218d7b85cbc)). Kopia seals its format block with **Km equals PBKDF over UniqueID then Ke and AD via HKDF-SHA256 for AES and CHECKSUM domains** ([Source](https://kopia.io/docs/advanced/encryption/)), documents only **scrypt-65536-8-1 as supported at the moment** ([Source](https://kopia.io/docs/advanced/encryption/)), and in 2025 added **configurable PBKDF with 600000 iterations default for PBKDF2 and 64 MB default for scrypt** ([Source](https://github.com/kopia/kopia/pull/5145)). Ancillary systems sharpen the contrast: rclone crypt derives **80 bytes via scrypt N equals 16384, r equals 8, p equals 1** ([Source](https://rclone.org/crypt/)) and Tarsnap generates keys locally, using scrypt only to optionally wrap the key file ([Source](https://www.tarsnap.com/man-tarsnap-keygen.1.html)). Versiond should therefore follow Borg 2 on new installs with Argon2id where libargon2 is available and otherwise scrypt-65536-8-1, always with a random 32-byte UniqueID salt that doubles as HKDF info to prevent cross-repo correlation ([Source](https://kopia.io/docs/advanced/encryption/)).
Storage location and rotation complete the choice. Restic keeps **keys directory JSON in the repo with hostname, KDF params, salt and encrypted data** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)), Kopia keeps a **format blob with UniqueID, keyAlgo and encryptedBlockFormat in storage plus a local connection config without the password** ([Source](https://kopia.io/docs/reference/command-line/)), and Borg alone offers **repokey in the repo versus keyfile under home with move via key change-location** ([Source](https://borgbackup.readthedocs.io/en/2.0.0b6/usage/rcreate.html)). True rotation by re-encrypting data exists nowhere; all rotation re-wraps the same secret, whether through **restic key add and passwd managing multiple access keys sharing one master** ([Source](https://restic.readthedocs.io/en/stable/070_encryption.html)), **Borg change-passphrase only re-locking the same secrets** ([Source](https://borgbackup.readthedocs.io/en/stable/usage/key.html)), **Kopia change-password re-encrypting the format block while still connected** ([Source](https://kopia.io/docs/faqs/)), or rclone requiring full re-upload to change passwords ([Source](https://rclone.org/crypt/)). No system documents native Shamir sharing. Loss is therefore uniformly fatal by design: restic repeats that **losing your password means data is irrecoverably lost** ([Source](https://restic.readthedocs.io/en/stable/030%5Fpreparing%5Fa%5Fnew%5Frepo.html)), Kopia states **there is no way to recover a forgotten password because only you know it** ([Source](https://kopia.io/docs/faqs/)), Borg demands **both key and passphrase with offsite export** ([Source](https://borgbackup.readthedocs.io/en/stable/usage/init.html)), and Tarsnap warns that **the key file contains the only copy** ([Source](https://www.tarsnap.com/faq.html)). For versiond this means shipping multi-key wrapping from day one so a daemon key and an offline admin recovery key share the same master, exporting with **paper printouts with per-line checksums or QR plus separate passphrase storage** as Borg documents ([Source](https://borgbackup.readthedocs.io/en/master/usage/key.html)), caching the daemon password in the Linux keyring only as Kopia does transiently ([Source](https://kopia.io/docs/reference/command-line/)), and documenting that a leaked master forces new-repo migration because **revocation without re-encrypting the whole repository is impossible** ([Source](https://rustic.cli.rs/dev-docs/design/threat_model.html)).
## Random packing and padding hide far more than encryption alone
Encryption hides bytes but every system leaks sizes, counts, timing and directory shape to a curious WebDAV host, and versiond must treat that as its main residual risk. Restic filenames are **lowercase hex storage IDs verifiable by sha256sum** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)) under predictable **config, keys, locks, snapshots, index and data prefixes** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)), with the server able to **infer which packs probably contain trees via access patterns and infer backup sizes via timestamps** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)), including pre-0.18 chunk-size fingerprinting only partly fixed by **randomly assigning chunks to pack files** ([Source](https://restic.readthedocs.io/en/stable/100_references.html)). Borg is explicit that **the repository does not hide the size of chunks** and that single-chunk small files plus inode-order proximity let an observer correlate neighbours unless the operator enables the **obfuscate pseudo-compressor that pads with zero bytes** ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)). Rclone crypt is weakest here by design, stating it **does not encrypt file length within 16 bytes nor modification time used for syncing** ([Source](https://rclone.org/crypt/)) and encrypting names deterministically with **EME-AES-256 plus base32 so identical names yield identical ciphertexts** ([Source](https://rclone.org/crypt/)), while **plain WebDAV does not support modified times nor hashes** except vendor extensions ([Source](https://rclone.org/webdav/)). Kopia leaks least by default because **pack files have random names and sizes generally unrelated to content due to splitting and merging** ([Source](https://kopia.io/docs/advanced/architecture/)), exposing only **p versus q versus x prefixes for data, metadata and indices** ([Source](https://kopia.io/docs/advanced/architecture/)).
Active-server protection is where versiond needs append-only manifests plus local verification. Borg builds a Horton DAG where **every object is referenced by its parent via plaintext-MAC ID up to a TAM-signed manifest** protecting the fixed **000…000 manifest ID with HKDF-derived TAM keys** ([Source](https://borgbackup.readthedocs.io/en/stable/internals/security.html)), tracks nonces to catch replay, and ships **append-only mode that never overwrites committed segments** with transaction-log rollback ([Source](https://raw.githubusercontent.com/borgbackup/borg/1.4.5/docs/usage/notes.rst)), yet even Borg suffered **CVE-2023-36811 allowing faked archives with repo write** ([Source](https://nvd.nist.gov/vuln/detail/CVE-2023-36811)). Restic authenticates objects but concedes it is **not designed to protect against attackers deleting files and not designed to detect timestamp-grouped deletion** ([Source](https://rustic.cli.rs/dev-docs/design/threat_model.html)), warns that write access lets an attacker **create garbage snapshots covering modified files and wait for forget to prune correct ones** ([Source](https://rustic.cli.rs/dev-docs/design/threat_model.html)), and leaves management-key separation as open issue discussion ([Source](https://github.com/restic/restic/issues/5041)). Kopia relies on **snapshot verify walks plus sampled content downloads during maintenance** ([Source](https://kopia.io/docs/advanced/consistency/)) and pushes ransomware safety to **provider restricted keys plus COMPLIANCE object-lock with retention periods** ([Source](https://kopia.io/docs/advanced/ransomware-protection/)), which on WebDAV has no equivalent since **object-lock currently only supports S3 repos** ([Source](https://kopia.io/docs/advanced/ransomware-protection/)). For versiond on dumb WebDAV the synthesis is concrete: pack spool files into fixed-size-class encrypted blobs with random names, pad small manifests, jitter upload times, never trust server mtime, verify after write with GET plus AEAD open before deleting staging, keep an **encrypted local cache to prevent metadata leaks** as restic does ([Source](https://rustic.cli.rs/dev-docs/design/threat_model.html)), sign each append-only manifest with the master key and chain it to the prior manifest hash, enforce a local monotonic epoch to detect rollback, and compensate for missing server append-only with WebDAV versioning or a second prune identity that alone may delete, mirroring the split Borg recommends for serve keys.
## Conclusion
The durable lesson is that cipher negotiation matters less than pipeline order, identifier keying, and operational separation of sync versus prune credentials. A daemon that hashes with a keyed MAC, compresses per chunk with an explicitly recorded codec, seals with a standard AEAD under HKDF-separated keys, and appends chained manifests into randomly sized packs inherits the best of Borg's authentication thinking and Kopia's WebDAV-ready packing without repeating restic's bespoke crypto or Duplicity's GnuPG fragility. What remains honestly unsolved is metadata-minimal versioning on a store that willingly reveals sizes and timestamps; the next hardening step is not a stronger cipher but deterministic padding classes, falsified access ordering, and provider-enforced immutability that make deletion the only move left to an untrusted host.
reports/Retention scheduling adaptation.md
+34
View File
@@ -0,0 +1,34 @@
# Borrow Time Machine's pressure-driven thinning discipline
Leading backup systems agree more than their docs admit: declarative time policy decides what may die, cheap metadata marking runs often, expensive reclamation runs rarely, capacity only forces the issue when the target fills, and local resource politeness comes from static ceilings plus OS deferral rather than true autotuning. For a local Linux file-versioning daemon syncing to WebDAV, that means copying Kopia's built-in maintenance rhythm, Time Machine's pre-backup thinning with padding, WebDAV PROPFIND quota checks, and a systemd slice with conservative defaults — because none of the surveyed tools offers a keep-usage-below-70% knob, and the one system that deletes on full does so without any percentage at all.
## Cheap marking runs daily while reclamation waits weeks
The universal split is cheap expiration versus expensive reclamation, with very different cadences. In restic **forget only deletes snapshot metadata while prune repacks data and is very time-consuming for remote repositories**, bounded by **--max-unused default 5%** ([Source](https://restic.readthedocs.io/en/stable/060_forget.html)). Borg made the same split explicit after 1.2 where **repository disk space is not freed until you run borg compact** and docs tell operators to run it regularly but not after every command, roughly monthly or when space is needed ([Source](https://borgbackup.readthedocs.io/en/stable/usage/compact.html)). Borg's default **compact --threshold 10%** compacts only segments above that saving, with 2.x gating further to reclaimable space above threshold divided by five ([Source](https://borgbackup.readthedocs.io/en/stable/usage/compact.html)). Kopia mirrors the design with **quick maintenance hourly and full maintenance every 24h by default**, where quick work never deletes metadata without another copy existing and full work does Snapshot GC plus pack compaction over several cycles ([Source](https://kopia.io/docs/advanced/maintenance/)). Veeam applies short-term retention inline at the end of each job session while GFS deletions fall to a nightly background job, with health check verifying **only the latest restore point per chain on a monthly default** rather than whole history ([Source](https://helpcenter.veeam.com/docs/vbr/userguide/backup_health_check.html)).
Scheduling ownership falls on a spectrum from manual to fully automatic that the daemon should learn from. Restic and Borg ship **no built-in scheduler at all**, so retention becomes a wrapper convention of backup then forget plus prune or create then prune then compact, typically driven by daily cron such as **0 0 * * * with retention plus check monthly** ([Source](https://gist.github.com/perfecto25/f528f8d14e1c4b6e2a912513539a5af7)). Borgmatic warns its every-run prune plus compact plus check default suits small repos but large repos must decouple them to separate schedules ([Source](https://github.com/borgmatic-collective/borgmatic/blob/main/docs/how-to/deal-with-very-large-backups.md). Kopia inverted this by making maintenance automatic since v0.6.0, firing opportunistically whenever the client runs with **QuickCycle 1h and FullCycle 24h** tunable via maintenance set and pausable via pause flags ([Source](https://kopia.io/docs/advanced/maintenance/)). KopiaUI or server checks roughly every 10 minutes and does work only when due ([Source](https://github.com/kopia/kopia/issues/1439). Time Machine goes furthest with zero user-visible retention scheduling, keeping **hourly backups for 24 hours, daily for a month, weekly for previous months** and thinning continuously plus under pressure ([Source](https://support.apple.com/en-la/104984)). Veeam sits in the enterprise middle with a per-job scheduler plus GFS calendar and monthly health-check piggybacked on the first session of the scheduled day ([Source](https://helpcenter.veeam.com/docs/vbr/userguide/backup_health_check.html)). Safety rails are strikingly convergent and worth copying wholesale: every tool offers dry-run previews without making them default, restic refuses empty policies and requires **--unsafe-allow-remove-all plus filters** to wipe a group, Borg retains the oldest archive when no rule keeps anything and since 2.x offers **undelete until compact runs**, and Kopia elects a **single maintenance owner under exclusive lock with PackDeleteMinAge 24h** so GC takes multiple cycles unless defeated with safety none ([Source](https://kopia.io/docs/advanced/maintenance/)). Restic prune holds an exclusive lock that blocks backups while Borg and Kopia serialize through repo plus cache locks with break-lock and force overrides, and interrupted cheap passes rerun safely while expensive passes resume from checkpoints or recorded partial results ([Source](https://github.com/restic/restic/blob/master/doc/design.rst)).
## Time policy leads, capacity follows except when disk fills
Only Apple Time Machine makes capacity the primary retention driver, and even it leads with time. Backupd logs show the literal two-phase layering of **Starting pre-backup thinning: 53.57 GB requested (including padding)** then only if **No expired backups exist - deleting oldest backups to make room** ([Source](https://serverfault.com/posts/39310/revisions)). The published ladder keeps first-of-day beyond 24 hours and first-of-week beyond 30 days, deleting oldest weeklies only when space is needed ([Source](https://discussions.apple.com/thread/251224074)). There is **no published percentage watermark**, deletion is on-demand pre-backup rather than a high-water mark ([Source](https://support.apple.com/guide/mac-help/if-the-time-machine-backup-disk-is-full-mh15137/mac)). Per-destination caps come only from **tmutil setquota DESTINATION_ID QUOTA_IN_GB**, which thins to fit on the next backup ([Source](https://superuser.com/questions/445579/how-do-i-trim-time-machine-backup-history)). Local snapshots are purgeable space automatically reclaimed as needed with urgency-controlled manual relief via **thinlocalsnapshots mount_point purge_amount urgency 1-4** ([Source](https://ss64.com/mac/tmutil.html)). Every other surveyed system treats capacity as monitoring, placement, or manual pruning rather than a retention input. Veeam retention defines **the number of restore points to keep on performance and capacity extents**, purging capacity-tier blocks on the next offload session, and when extents fill guidance is simply to add a new extent ([Source](https://helpcenter.veeam.com/docs/vbr/userguide/capacity_tier_retention.html)). Veeam placement prefers the extent with fewest chains breaking ties by most free space, with **priority always to complete a backup even by violating Data-Locality** ([Source](https://veeam-best-practices-guide-v9.readthedocs.io/resource_planning/repository_sobr.html)). Forecast thresholds such as a worked **Repository Free Space 30%** example flag repositories that will run out rather than enforcing deletion ([Source](https://helpcenter.veeam.com/docs/mp/reports/capacity_planning_for_backup_repositories.html?ver=9a)). S3 Lifecycle rules use **Days, Date, and NoncurrentVersionExpiration per prefix or tag** with no capacity trigger at all ([Source](https://docs.aws.amazon.com/AmazonS3/latest/API/API_LifecycleRule.html)). Hetzner Storage Box quotas are fixed plan sizes with **10, 20, 30, or 40 manual plus automatic snapshot slots** consuming the same plan capacity alongside live data, offering no per-subaccount quota and no auto-thinning beyond setting directories read-only ([Source](https://docs.hetzner.com/storage/storage-box/snapshots/)). ZFS tooling is count-based with **-k keep NUM recent snapshots** and splits take, prune, and cron while capacity monitoring only feeds Nagios-style alerts ([Source](https://manpages.debian.org/bookworm/zfs-auto-snapshot/zfs-auto-snapshot.8.en.html)). The percentage numbers that do exist are health or performance floors, not auto-delete triggers, with Ubuntu warning that **minimum free space to preserve ZFS performance is 20%** while the pool sits at 10% ([Source](https://superuser.com/questions/1736700/how-do-i-remove-old-zfs-snapshots)) and OpenZFS advising to **keep pool free space above 10% to avoid metaslabs hitting 4%** where the allocator collapses from first-fit to best-fit ([Source](https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html)). Borg likewise has no quota-aware prune and warns that **repository disk space is not freed until you run borg compact**, urging operators to ensure always plenty of free space and to prune plus compact regularly ([Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)).
| System | Primary retention driver | Capacity role | Published percentage trigger |
|---|---|---|---|
| Time Machine | Time ladder then oldest-first on full | Pre-backup thinning with padding | None, allocation-driven |
| Veeam SOBR | Restore-point counts plus GFS | Placement plus alerting, add extent | 30% forecast example only |
| S3 Lifecycle | Days and versions | None | None |
| ZFS, sanoid, Borg, restic | Counts and time windows | Monitoring plus manual prune | 20%, 10%, 4% health floors only |
Failure at 100% full explains why a headroom design matters more than any target percentage. Time Machine cancels the run with **Stopping backup. Backup canceled. Compacting backup disk image to recover free space** then retries as a fresh standard backup, surfacing **This backup is too large for the backup disk. The backup requires XX GB but only YY GB are available** when nothing is freeable ([Source](https://serverfault.com/posts/39310/revisions)). Veeam guards production datastores with **Skip VMs when free disk is below plus a hard 2 GB floor** even when skipping is disabled ([Source](https://www.veeam.com/kb4379?ad=in-text-link)). Borg is the cautionary tale that **if you do run out of disk space, it can be hard or impossible to free space, because Borg needs free space to operate - even to delete backup archives** ([Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)). ZFS adds the snapshot-pinned trap where deleting a file present in a snapshot gains no space and can itself return unexpected ENOSPC ([Source](https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gayra/index.html)). Restic historically panicked on temp-pack writes with **no space left on device** and retried blindly on ENOSPC before rest-server fixes ([Source](https://github.com/restic/restic/issues/611)). WebDAV at least fails cleanly with **507 Insufficient Storage** on PUT, MKCOL, MOVE, or COPY ([Source](http://www.webdav.org/specs/rfc4331.html)). The robust pattern combining all three headroom elements is pre-backup estimate plus padding as Time Machine does, a reserved-space tripwire as Veeam 2 GB, Borg repo-space, and ZFS 10% do, and a recovery path that works at 100% as Time Machine compact-and-retry does and Borg notably lacks. Crucially for snapshot-capable targets, a capacity pruner must delete snapshots themselves oldest-first rather than thinning live data, because Hetzner snapshots silently consume the same plan quota ([Source](https://docs.hetzner.com/storage/storage-box/snapshots/)).
## WebDAV PROPFIND gives daemon a cheap capacity signal
WebDAV is unusually lucky among dumb backends because RFC 4331 standardizes quota discovery. Servers expose **DAV:quota-available-bytes as maximum additional storage allocatable and DAV:quota-used-bytes including sub-resources**, warning that as available bytes approaches zero further allocations may be refused ([Source](https://datatracker.ietf.org/doc/html/rfc4331)). Nextcloud implements both properties retrievable via PROPFIND ([Source](https://github.com/nextcloud/documentation/blob/master/developer_manual/client_apis/WebDAV/basic.rst)). That makes a **Depth:0 PROPFIND for quota-used-bytes plus quota-available-bytes the cheapest correct pre-backup check** for Hetzner or Nextcloud targets, avoiding any tree walk. Alternatives on other transports show why this matters. S3 has no byte-quota endpoint with accounting left to CloudWatch and Storage Lens and lifecycle evaluated by a daily asynchronous scan ([Source](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html)). Hetzner exposes usage via console or API rather than uniformly in-protocol across its FTP, SFTP, rsync, Borg, SMB, and WebDAV paths ([Source](https://docs.hetzner.com/storage/storage-box)). Veeam polls extent free space but **free space data is only retrieved when no active tasks are assigned to an extent**, so placement under continuous load uses stale data ([Source](https://bp.veeam.com/vbr/3_Build_structures/B_Veeam_Components/B_backup_repositories/scaleout.html)). Borg and restic on SSH, SFTP, or B2 do no server quota query at all, directing users to local filesystem monitoring and **borg repo-space accounting** ([Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)). Restic cache mismanagement shows the cost of local-only accounting when cache at system paths can grow to **20 to 40 GB or roughly 3 to 10% of repo size** with no built-in cap ([Source](https://github.com/restic/restic/issues/4325)). Time Machine sidesteps the whole problem by asking the local filesystem how many bytes plus padding the next backup needs and thinning until that fits rather than tracking a remote percentage, a model the daemon should emulate by combining local size estimation with a PROPFIND confirmation before each sync.
## Static ceilings plus OS deferral beat true autotuning
No surveyed tool probes link speed, disk size, or free RAM to size packs, connections, or limits. Rate knobs are consistently static, opt-in, and unlimited by default. Restic exposes **--limit-upload and --limit-download in KiB per second defaulting to 0 meaning unlimited** and cannot change them mid-run without restart ([Source](https://restic.readthedocs.io/en/stable/manual_rest.html)). Borg 1.x exposes **--upload-ratelimit in kiByte per second default 0** over a token-bucket SleepingBandwidthLimiter with 0.1 second period capping bursts at twice the allowance ([Source](https://manpages.debian.org/bookworm/borgbackup/borg-common.1.en.html)). Borg 2.x moved shaping to **BORGSTORE_BANDWIDTH in bits per second default 0 plus BORGSTORE_LATENCY** ([Source](https://borgbackup.readthedocs.io/en/latest/faq.html)). Kopia exposes **repository throttle set with upload and download bytes per second plus concurrent reads, writes, and request rates** ([Source](https://kopia.io/docs/reference/command-line/common/repository-throttle-set/)). Duplicity has no native generic limit with workarounds via trickle or tc ([Source](https://bugs.launchpad.net/bugs/1291633)). Time Machine instead imposes mandatory kernel-level low-priority I/O where **backupd runs at IOPOL_THROTTLE for long-running background work throttled to protect higher policies** ([Source](https://eclecticlight.co/2022/01/20/why-time-machine-backups-can-be-interminably-slow/)). Disabling that global throttle lifted a measured copy phase from **193 to 332 MB per second and overall backup from 160 to 276 MB per second** ([Source](https://eclecticlight.co/2022/02/28/does-removing-i-o-throttling-make-backups-faster/)). The only resource-derived defaults anywhere are CPU-count-derived worker pools. Restic defaults to **file-read concurrency 2, blob-save concurrency equal to runtime.NumCPU, tree-save at 20 times blob concurrency, and backend connections 5 or 2 for local** ([Source](https://github.com/restic/restic/blob/de9136b29f86216bd3e41397d19b25f26b578833/internal/archiver/archiver.go)). Kopia defaults **--max-parallel-file-reads to logical core count** and scales s2 compressor concurrency similarly ([Source](https://kopia.io/docs/faqs/)). Time Machine scheduling scores each due activity every few seconds against temperature and load via DAS-CTS, dispatching only above threshold, with Apple Silicon background threads confined to Efficiency cores at **972 to 1332 MHz and roughly 90% E-core residency capping throughput near 300 to 400 items per second** ([Source](https://eclecticlight.co/2023/11/28/scheduling-and-dispatch-of-backups-and-other-background-activities/)). Raising concurrency without memory awareness is dangerous, with one restic test at connections 16 and reads 16 hitting **300 GB RAM and OOM** ([Source](https://github.com/restic/restic/issues/4477)). Constrained-box guidance therefore stays manual with cheap compression, lower parallelism, and larger packs. Borg defaults to **lz4 with a proposal to move to zstd level -4 capped at 4 threads** ([Source](https://github.com/borgbackup/borg/pull/10100). Kopia disables compression by default and benchmarks **s2-default at 4 GiB per second and 375 MiB RSS versus zstd at 323 MiB per second** on a 466 MiB corpus ([Source](https://kopia.io/docs/advanced/compression/)). Restic on SMR USB disks converges on **--read-concurrency 1 plus --pack-size 128 plus --no-cache with local connections 1**, trading longer single-pack uploads and 64 to 384 MiB staging for fewer files ([Source](https://forum.restic.net/t/first-backup-2-5tb-50-hours-can-i-improve-it/8288)). Upstream tools ship no restrictive systemd units at all, leaving the enforceable layer to downstream slices. Systemd semantics give **CPUQuota as percent of one CPU, CPUWeight 1 to 10000 default 100 for contention only, MemoryHigh as soft throttle versus MemoryMax as hard OOM kill, and IOWeight plus per-device bandwidth caps** ([Source](https://manpages.debian.org/bullseye/systemd/systemd.resource-control.5.en.html)). Community practice fills the gap with **Nice 17 plus sandboxing and RandomizedDelaySec 300** on restic timers and **CPUQuota 80% plus MemoryMax 2G** on segmented Borg services ([Source](https://github.com/JoZapf/segmented-borg-backup-system/blob/refs/heads/main/docs/SYSTEMD.md)). Modern guidance recommends **Nice 19 plus ionice class idle for soft priority with a backup.slice carrying CPUQuota 50 to 80%, MemoryHigh and Max 1 to 4G, IOWeight 10, and IOWriteBandwidthMax such as 30 MB per second**, noting ionice is ignored on default mq-deadline SSD schedulers and MemoryMax kills rather than slows ([Source](https://www.bigiron.cc/guides/process-priorities-nice-ionice-cgroup-quotas)).
## Conclusion
The evidence reframes the daemon as a Kopia-style insider with Time Machine instincts rather than a restic-style script kit: own its scheduler with hourly cheap and daily full passes, enforce time policy first with capacity-triggered oldest-first eviction guarded by a minimum-retention floor, and treat PROPFIND plus local size-plus-padding as the single pre-sync gate instead of inventing a 70% watermark no vendor validates. Resource modesty then becomes configuration rather than cleverness, with static unlimited-by-default throttles, core-count workers, s2 or lz4 compression, and a systemd slice doing the throttling the backup code cannot do for itself.