Forbid em dashes everywhere; add AGENTS.md with repo rules

research_notes/Backup include exclude rules/binary-large-files.md

@@ -6,20 +6,20 @@
 Restic, Borg and Kopia all use content-defined chunking (CDC) with ~0.5–8 MiB variable chunks so small edits to large binaries only store 1–2 new chunks; without CDC (fixed blocks or full-file copies) a 1-byte insert re-stores the whole file, and Git-LFS instead punts large files to pointer+blob storage with host-enforced per-file caps.
 
 ### Cited Findings
-- Restic splits files with Rabin-fingerprint CDC over a 64-byte sliding window, cutting when low 21 bits are zero; files <512 KiB are not split, blobs are 512 KiB–8 MiB, ~1 MiB average — [Source](https://github.com/restic/restic/blob/master/doc/design.rst); background — [Source](https://restic.net/blog/2015-09-12/restic-foundation1-cdc/)
-- Restic chunker defaults aim at ~1 MiB average (`splitmask = (1<<20)-1`) with configurable Min/MaxSize — [Source](https://github.com/restic/chunker/blob/master/chunker.go)
-- Borg splits files into deduplicated chunks globally across repo (all machines/archives); chunk id is a strong hash/MAC (hmac-sha256 / keyed blake3), not the rolling-hash value — [Source](https://borgbackup.readthedocs.io/en/master/)
-- Borg default buzhash params: min 2^19 (512 KiB), max 2^23 (8 MiB), mask 21 bits (~2 MiB target), window 4095 B; `fixed` chunker option for disk/VM images — [Source](https://github.com/borgbackup/borg/blob/86fd77fd/docs/internals/data-structures.rst)
-- Borg 2.x chunkers: `fastcdc` (default, fastest), `buzhash64`/`buzhash`, keyed AES variants (`toeplitz-aes`/`rabin-aes` strongest), `fixed` for raw disk images where CDC gains little — [Source](https://borgbackup.readthedocs.io/en/latest/internals/chunker.html)
-- Borg warns fine-grained `--chunker-params=buzhash,10,23,16,4095` creates huge chunk counts and RAM/disk load; coarse default suits large volumes — [Source](https://borgbackup.readthedocs.io/en/stable/usage/notes.html)
-- Kopia calls CDC "splitters": FIXED vs DYNAMIC BUZHASH/RABINKARP, sizes 1M–8M (default DYNAMIC-4M-BUZHASH); small files = one content, large files split so metadata-only change to 10 GB video uploads only 1–2 chunks (<10 MB) — [Source](https://kopia.discourse.group/t/does-kopia-use-content-defined-chunking-cdc/1417); details — [Source](https://kopia.discourse.group/t/difference-between-the-available-splitters/894); packing — [Source](https://kopia.discourse.group/t/do-hashed-chunks-span-multiple-files/1444)
-- Kopia packs many contents into 20–40 MB pack blobs; splitter choice is set at repo creation — [Source](https://kopia.io/docs/advanced/architecture/); chunk-then-hash-then-compress-then-encrypt pipeline — [Source](https://kopia.io/docs/advanced/compression/)
-- Restic has no hard max file size but offers `--exclude-larger-than size` (suffixes k/M/G/T) to skip files over a threshold — [Source](https://restic.readthedocs.io/en/stable/040_backup.html)
-- Git-LFS has no inherent file-size limit; limits are host-enforced (GitHub: 2 GB Free/Pro, 4 GB Team, 5 GB Enterprise Cloud; >5 GB rejected); pointer file stores `version/oid sha256/size` — [Source](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage)
-- Git-LFS tracks by `.gitattributes` pattern, not by size; `--above` in `migrate import` is one-shot, no automatic by-size tracking (2025 `--min-size/autotracksize` PR still debated/unmerged) — [Source](https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-faq.adoc)
-- Git on Windows pre-2.34 could not smudge/clean files >4 GiB; workaround `GIT_LFS_SKIP_SMUDGE=1` + `git lfs pull` — [Source](https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-faq.adoc)
-- Every Git-LFS revision counts against remote storage/bandwidth quota, so per-version binaries inflate cost; clones/pulls are slow on large repos — [Source](https://get.assembla.com/blog/git-lfs/)
-- Keyed CDC (Borg/Restic/Kopia) is fingerprintable: observing chunk sizes of known data recovers keys (CCS 2025, eprint 2025/558; backup-service attacks eprint 2025/532); Restic 0.18 mitigates by random chunk-to-pack assignment — [Source](https://github.com/restic/restic/blob/master/doc/design.rst); attacks — [Source](https://eprint.iacr.org/2025/558); analysis — [Source](https://eprint.iacr.org/2025/532.pdf)
+- Restic splits files with Rabin-fingerprint CDC over a 64-byte sliding window, cutting when low 21 bits are zero; files <512 KiB are not split, blobs are 512 KiB–8 MiB, ~1 MiB average - [Source](https://github.com/restic/restic/blob/master/doc/design.rst); background - [Source](https://restic.net/blog/2015-09-12/restic-foundation1-cdc/)
+- Restic chunker defaults aim at ~1 MiB average (`splitmask = (1<<20)-1`) with configurable Min/MaxSize - [Source](https://github.com/restic/chunker/blob/master/chunker.go)
+- Borg splits files into deduplicated chunks globally across repo (all machines/archives); chunk id is a strong hash/MAC (hmac-sha256 / keyed blake3), not the rolling-hash value - [Source](https://borgbackup.readthedocs.io/en/master/)
+- Borg default buzhash params: min 2^19 (512 KiB), max 2^23 (8 MiB), mask 21 bits (~2 MiB target), window 4095 B; `fixed` chunker option for disk/VM images - [Source](https://github.com/borgbackup/borg/blob/86fd77fd/docs/internals/data-structures.rst)
+- Borg 2.x chunkers: `fastcdc` (default, fastest), `buzhash64`/`buzhash`, keyed AES variants (`toeplitz-aes`/`rabin-aes` strongest), `fixed` for raw disk images where CDC gains little - [Source](https://borgbackup.readthedocs.io/en/latest/internals/chunker.html)
+- Borg warns fine-grained `--chunker-params=buzhash,10,23,16,4095` creates huge chunk counts and RAM/disk load; coarse default suits large volumes - [Source](https://borgbackup.readthedocs.io/en/stable/usage/notes.html)
+- Kopia calls CDC "splitters": FIXED vs DYNAMIC BUZHASH/RABINKARP, sizes 1M–8M (default DYNAMIC-4M-BUZHASH); small files = one content, large files split so metadata-only change to 10 GB video uploads only 1–2 chunks (<10 MB) - [Source](https://kopia.discourse.group/t/does-kopia-use-content-defined-chunking-cdc/1417); details - [Source](https://kopia.discourse.group/t/difference-between-the-available-splitters/894); packing - [Source](https://kopia.discourse.group/t/do-hashed-chunks-span-multiple-files/1444)
+- Kopia packs many contents into 20–40 MB pack blobs; splitter choice is set at repo creation - [Source](https://kopia.io/docs/advanced/architecture/); chunk-then-hash-then-compress-then-encrypt pipeline - [Source](https://kopia.io/docs/advanced/compression/)
+- Restic has no hard max file size but offers `--exclude-larger-than size` (suffixes k/M/G/T) to skip files over a threshold - [Source](https://restic.readthedocs.io/en/stable/040_backup.html)
+- Git-LFS has no inherent file-size limit; limits are host-enforced (GitHub: 2 GB Free/Pro, 4 GB Team, 5 GB Enterprise Cloud; >5 GB rejected); pointer file stores `version/oid sha256/size` - [Source](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage)
+- Git-LFS tracks by `.gitattributes` pattern, not by size; `--above` in `migrate import` is one-shot, no automatic by-size tracking (2025 `--min-size/autotracksize` PR still debated/unmerged) - [Source](https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-faq.adoc)
+- Git on Windows pre-2.34 could not smudge/clean files >4 GiB; workaround `GIT_LFS_SKIP_SMUDGE=1` + `git lfs pull` - [Source](https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-faq.adoc)
+- Every Git-LFS revision counts against remote storage/bandwidth quota, so per-version binaries inflate cost; clones/pulls are slow on large repos - [Source](https://get.assembla.com/blog/git-lfs/)
+- Keyed CDC (Borg/Restic/Kopia) is fingerprintable: observing chunk sizes of known data recovers keys (CCS 2025, eprint 2025/558; backup-service attacks eprint 2025/532); Restic 0.18 mitigates by random chunk-to-pack assignment - [Source](https://github.com/restic/restic/blob/master/doc/design.rst); attacks - [Source](https://eprint.iacr.org/2025/558); analysis - [Source](https://eprint.iacr.org/2025/532.pdf)
 
 ### Inferences
 - A small local daemon should copy the Restic/Borg default: CDC ~1–2 MiB average, 512 KiB min, 8 MiB max; offer `--exclude-larger-than`-style cap (e.g. 50–100 MB default-off) plus extension-based excludes rather than a hard cap.
@@ -36,12 +36,12 @@ Restic, Borg and Kopia all use content-defined chunking (CDC) with ~0.5–8 MiB
 Git (and libgit2 ports) still use NUL-in-first-8000-bytes as the primary binary signal, tightened in 2008 so CRLF conversion defers to diff; it is fast and recommended as a first heuristic but known-insufficient for UTF-16 and NUL-free binaries, so explicit `.gitattributes` marking is required.
 
 ### Cited Findings
-- Git `buffer_is_binary()` checks for NUL via `memchr` in first 8000 bytes (`FIRST_FEW_BYTES`) — [Source](https://stackoverflow.com/questions/6119956/how-to-determine-if-git-handles-a-file-as-binary-or-as-text)
-- Git `convert.c` `convert_is_binary()`: binary if `lonecr` or `nul` or `(printable>>7) < nonprintable`; CRLF auto-conversion bails on binary — [Source](https://code.googlesource.com/git/+/645cc7a2a7274a92403d2848ef643a96f1589d09/convert.c)
-- libgit2 `git_blob_is_binary` uses core-git heuristic: NUL scan + printable/nonprintable ratio over first 8000 bytes — [Source](https://libgit2.org/docs/reference/main/blob/git_blob_is_binary.html)
-- 2008 patch unified heuristics: any NUL forces binary in `convert.c` so CRLF handling is stricter than diff (prior convert.c used only <1% nonprintable rule, mis-handling tar/word-processor files diff called binary) — [Source](https://public-inbox.org/git/20080116011321.GD13984@dpotapov.dyndns.org/t/)
-- Git mailing-list guidance: NUL in first 8000 bytes = binary; UTF-16 must be marked explicitly, Git does not handle it internally; short NUL-free binaries must also be marked explicitly — [Source](https://public-inbox.org/git/20151202004921.GC28197@sigill.intra.peff.net/T/)
-- `git diff --numstat` reports `-\t-` for binary (practical detector); `git check-attr` only reflects `.gitattributes`, not the heuristic — [Source](https://stackoverflow.com/questions/6119956/how-to-determine-if-git-handles-a-file-as-binary-or-as-text)
+- Git `buffer_is_binary()` checks for NUL via `memchr` in first 8000 bytes (`FIRST_FEW_BYTES`) - [Source](https://stackoverflow.com/questions/6119956/how-to-determine-if-git-handles-a-file-as-binary-or-as-text)
+- Git `convert.c` `convert_is_binary()`: binary if `lonecr` or `nul` or `(printable>>7) < nonprintable`; CRLF auto-conversion bails on binary - [Source](https://code.googlesource.com/git/+/645cc7a2a7274a92403d2848ef643a96f1589d09/convert.c)
+- libgit2 `git_blob_is_binary` uses core-git heuristic: NUL scan + printable/nonprintable ratio over first 8000 bytes - [Source](https://libgit2.org/docs/reference/main/blob/git_blob_is_binary.html)
+- 2008 patch unified heuristics: any NUL forces binary in `convert.c` so CRLF handling is stricter than diff (prior convert.c used only <1% nonprintable rule, mis-handling tar/word-processor files diff called binary) - [Source](https://public-inbox.org/git/20080116011321.GD13984@dpotapov.dyndns.org/t/)
+- Git mailing-list guidance: NUL in first 8000 bytes = binary; UTF-16 must be marked explicitly, Git does not handle it internally; short NUL-free binaries must also be marked explicitly - [Source](https://public-inbox.org/git/20151202004921.GC28197@sigill.intra.peff.net/T/)
+- `git diff --numstat` reports `-\t-` for binary (practical detector); `git check-attr` only reflects `.gitattributes`, not the heuristic - [Source](https://stackoverflow.com/questions/6119956/how-to-determine-if-git-handles-a-file-as-binary-or-as-text)
 
 ### Inferences
 - NUL-sniffing remains the recommended cheap first pass for a small daemon (scan first 8 KiB), matching Git/libgit2 behavior and user expectations.
@@ -56,20 +56,20 @@ Git (and libgit2 ports) still use NUL-in-first-8000-bytes as the primary binary
 Never `cp`/read the raw sqlite file hot: in WAL mode the consistent image spans main+`-wal`+`-shm` and byte copies tear or go stale; use the Online Backup API (`sqlite3_backup_*` / `Connection.backup()` / `.backup` CLI), `VACUUM INTO`, or a quiesced filesystem snapshot, then `PRAGMA integrity_check`.
 
 ### Cited Findings
-- Historical `cp`-under-shared-lock method is fast but blocks writers, cannot copy to/from memory DBs, and risks corruption on power/OS failure — [Source](https://sqlite.org/backup.html)
-- Backup API: source read-locked only during each `sqlite3_backup_step(nPage)`; destination write-locked throughout; incremental stepping lets writers proceed; concurrent write by another connection restarts backup automatically — [Source](https://sqlite.org/backup.html); API contract — [Source](https://sqlite.org/c3ref/backup_finish.html)
-- Python exposes as `Connection.backup(target, pages, progress, sleep)`; `pages=-1` copies all at once (holds lock), positive pages + `sleep=0.250` yields between steps; restart detected when `remaining` jumps back toward `total` — [Source](https://www.productionhardening.org/backup-recovery-data-integrity/online-backup-api-hot-copies/)
-- WAL-mode live DB is three files (main + `-wal` committed-not-checkpointed frames + `-shm` index); `cp`/`rsync`/snapshot captures them at different instants → `SQLITE_CORRUPT`/`SQLITE_NOTADB`; same hazard for `-journal` in rollback mode — [Source](https://www.productionhardening.org/backup-recovery-data-integrity/)
-- Real-world failure: `fs.copyFile()` of only `.db` in WAL mode produced 100% corrupt backups across weeks of 6-hour cron; fix was `better-sqlite3 .backup()` plus hour-granular filenames — [Source](https://scottspence.com/posts/sqlite-corruption-fs-copyfile-issue)
-- Correct one-liners: `sqlite3 app.db ".backup './db-backups/app.db'"` (general) or `VACUUM INTO` (same safety + compaction, needs SQLite 3.27+, refuses if target exists so `rm -f` first); for dedup pipelines prefer `.backup` because compaction reshuffles pages and defeats chunking — [Source](https://www.backupdata.io/resources/guides/sqlite-backups-you-can-actually-restore)
-- Borg docs: Borg just copies file as-is; if DB is written mid-read the archive may be inconsistent — use sqlite-aware method (`sqlite3 db.sqlite "VACUUM INTO 'copy.sqlite'"`) or filesystem snapshot — [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst); Borg FAQ repeats the `VACUUM INTO` advice — [Source](https://borgbackup.readthedocs.io/en/master/faq.html)
-- borgmatic best practice: dump (export) databases rather than backing internal files; streams dump directly to Borg — [Source](https://torsion.org/borgmatic/how-to/backup-your-databases/)
-- Checkpoint+lock schemes (`wal_checkpoint(TRUNCATE)` then `BEGIN IMMEDIATE` then copy main+wal) are fragile: checkpoint can fail to get writer lock, writers can interleave, autocheckpoint is per-process — use the backup API instead — [Source](https://sqlite.org/forum/forumpost/2ea989bbe9)
-- Post-backup rules: run `PRAGMA integrity_check` (expect single `ok`) on every finished image; pre-size target ~1.2× source; set `busy_timeout ≥5000ms`; write target off the source I/O queue; delete partials; on restore stop app, copy main file, delete stale `-wal`/`-shm` — [Source](https://www.productionhardening.org/backup-recovery-data-integrity/online-backup-api-hot-copies/); restore checklist — [Source](https://www.backupdata.io/resources/guides/sqlite-backups-you-can-actually-restore)
+- Historical `cp`-under-shared-lock method is fast but blocks writers, cannot copy to/from memory DBs, and risks corruption on power/OS failure - [Source](https://sqlite.org/backup.html)
+- Backup API: source read-locked only during each `sqlite3_backup_step(nPage)`; destination write-locked throughout; incremental stepping lets writers proceed; concurrent write by another connection restarts backup automatically - [Source](https://sqlite.org/backup.html); API contract - [Source](https://sqlite.org/c3ref/backup_finish.html)
+- Python exposes as `Connection.backup(target, pages, progress, sleep)`; `pages=-1` copies all at once (holds lock), positive pages + `sleep=0.250` yields between steps; restart detected when `remaining` jumps back toward `total` - [Source](https://www.productionhardening.org/backup-recovery-data-integrity/online-backup-api-hot-copies/)
+- WAL-mode live DB is three files (main + `-wal` committed-not-checkpointed frames + `-shm` index); `cp`/`rsync`/snapshot captures them at different instants → `SQLITE_CORRUPT`/`SQLITE_NOTADB`; same hazard for `-journal` in rollback mode - [Source](https://www.productionhardening.org/backup-recovery-data-integrity/)
+- Real-world failure: `fs.copyFile()` of only `.db` in WAL mode produced 100% corrupt backups across weeks of 6-hour cron; fix was `better-sqlite3 .backup()` plus hour-granular filenames - [Source](https://scottspence.com/posts/sqlite-corruption-fs-copyfile-issue)
+- Correct one-liners: `sqlite3 app.db ".backup './db-backups/app.db'"` (general) or `VACUUM INTO` (same safety + compaction, needs SQLite 3.27+, refuses if target exists so `rm -f` first); for dedup pipelines prefer `.backup` because compaction reshuffles pages and defeats chunking - [Source](https://www.backupdata.io/resources/guides/sqlite-backups-you-can-actually-restore)
+- Borg docs: Borg just copies file as-is; if DB is written mid-read the archive may be inconsistent - use sqlite-aware method (`sqlite3 db.sqlite "VACUUM INTO 'copy.sqlite'"`) or filesystem snapshot - [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst); Borg FAQ repeats the `VACUUM INTO` advice - [Source](https://borgbackup.readthedocs.io/en/master/faq.html)
+- borgmatic best practice: dump (export) databases rather than backing internal files; streams dump directly to Borg - [Source](https://torsion.org/borgmatic/how-to/backup-your-databases/)
+- Checkpoint+lock schemes (`wal_checkpoint(TRUNCATE)` then `BEGIN IMMEDIATE` then copy main+wal) are fragile: checkpoint can fail to get writer lock, writers can interleave, autocheckpoint is per-process - use the backup API instead - [Source](https://sqlite.org/forum/forumpost/2ea989bbe9)
+- Post-backup rules: run `PRAGMA integrity_check` (expect single `ok`) on every finished image; pre-size target ~1.2× source; set `busy_timeout ≥5000ms`; write target off the source I/O queue; delete partials; on restore stop app, copy main file, delete stale `-wal`/`-shm` - [Source](https://www.productionhardening.org/backup-recovery-data-integrity/online-backup-api-hot-copies/); restore checklist - [Source](https://www.backupdata.io/resources/guides/sqlite-backups-you-can-actually-restore)
 
 ### Inferences
 - Small-daemon default: if file header is `SQLite format 3`, do not copy directly; shell to `sqlite3 "$f" ".backup '$tmp'"` or `VACUUM INTO` (when compaction desired) into a temp file, then ingest the temp file; fall back to copy only when DB is quiesced and `wal_checkpoint(TRUNCATE)` shows zero log pages.
-- Smaller Borg-style chunks (~64–128 KiB target, e.g. `fastcdc,15,19,17`) cut sqlite re-storage ~3–8× vs 2 MiB defaults at the cost of chunk-count RAM — see next section.
+- Smaller Borg-style chunks (~64–128 KiB target, e.g. `fastcdc,15,19,17`) cut sqlite re-storage ~3–8× vs 2 MiB defaults at the cost of chunk-count RAM - see next section.
 
 ### Gaps
 - `.dump` (SQL text) vs `.backup` (page image) size/dedup tradeoff not quantified from primary sources in this pass; anecdotal claim that dumps dedup to KBs but take hours on 12 GB DBs is single-issue-report only.
@@ -77,20 +77,20 @@ Never `cp`/read the raw sqlite file hot: in WAL mode the consistent image spans
 ## Do images/archives dedup at all, and is per-version storage of them worth it vs plain mirroring?
 
 ### Takeaway
-Recompressed, encrypted, or gzipped-per-version artifacts barely dedup (a 1-byte change avalanches through compression; encrypted chunks are high-entropy), while uncompressed raw images and plain sqlite files dedup well under CDC — so version verbatim binaries that change little, otherwise mirror single-copy.
+Recompressed, encrypted, or gzipped-per-version artifacts barely dedup (a 1-byte change avalanches through compression; encrypted chunks are high-entropy), while uncompressed raw images and plain sqlite files dedup well under CDC - so version verbatim binaries that change little, otherwise mirror single-copy.
 
 ### Cited Findings
-- Restic maintainer: pass uncompressed data; small input change → large compressed-output change → new blob hashes → repo grows (50 GB gzipped DB dumps → 50 GB repo vs 36 GB for raw); CDC blobs identified by SHA-256 — [Source](https://github.com/restic/restic/issues/790)
-- Duplicati skips recompression/dedup for known-compressed extensions by default (saves CPU; whole-file handling) because metadata edits rewrite the stream; identical copies still dedup, moves are free — [Source](https://forum.duplicati.com/t/deduplication-for-large-files/8394)
-- Borg on 5 generations of same sqlite DB: gzipped inputs = 667 unique/667 total chunks (zero dedup, 1.6 GB); uncompressed = strong dedup (590 MB total), zstd-5 beats gzip — [Source](https://appsintheopen.com/posts/66-backing-up-sqlite-database-with-borg-and-de-duplication)
-- Borg sqlite tuning: default ~2 MiB target wastes a whole chunk per changed 4 KiB page; `fastcdc,15,19,17,2` (~128 KiB target) drastically cuts incrementals; finer `14,18,16` helps more; cost is chunk-index RAM — params apply per-run so back up DBs in a separate run — [Source](https://borgbackup.readthedocs.io/en/master/faq.html); 12.45 GB Vintage Story sqlite: 2nd backup 518 MB default → 62 MB (15,19,17) → 37 MB (10,23,16) — [Source](https://github.com/borgbackup/borg/issues/5877)
-- Kopia: 8 MB photo with metadata edit re-uploads ~8 MB under 4 MB splitter (chunk > file); fix is smaller splitter at repo-creation cost of more chunks — [Source](https://kopia.discourse.group/t/chunk-size-setting/1351)
-- Fixed-splitter warning: 10 GB video + 1 prepended byte re-uploads 10 GB; content-based (buzhash/rabinkarp) uploads 1–2 chunks — [Source](https://kopia.discourse.group/t/difference-between-the-available-splitters/894)
-- Backup pipelines must chunk-then-compress (per-chunk); encrypt-then-chunk/compress is useless since ciphertext is indistinguishable from random; encrypted dedup needs weakened convergent/MLE schemes with leakage tradeoffs — [Source](https://eprint.iacr.org/2025/532.pdf); survey — [Source](https://dl.acm.org/doi/10.1145/3685278)
-- Kopia compresses each chunk independently (s2 default, gzip optional); splitting costs little ratio (466 MB → 119 MB standalone s2 vs 133 MB via Kopia-s2) — [Source](https://kopia.io/docs/advanced/compression/)
+- Restic maintainer: pass uncompressed data; small input change → large compressed-output change → new blob hashes → repo grows (50 GB gzipped DB dumps → 50 GB repo vs 36 GB for raw); CDC blobs identified by SHA-256 - [Source](https://github.com/restic/restic/issues/790)
+- Duplicati skips recompression/dedup for known-compressed extensions by default (saves CPU; whole-file handling) because metadata edits rewrite the stream; identical copies still dedup, moves are free - [Source](https://forum.duplicati.com/t/deduplication-for-large-files/8394)
+- Borg on 5 generations of same sqlite DB: gzipped inputs = 667 unique/667 total chunks (zero dedup, 1.6 GB); uncompressed = strong dedup (590 MB total), zstd-5 beats gzip - [Source](https://appsintheopen.com/posts/66-backing-up-sqlite-database-with-borg-and-de-duplication)
+- Borg sqlite tuning: default ~2 MiB target wastes a whole chunk per changed 4 KiB page; `fastcdc,15,19,17,2` (~128 KiB target) drastically cuts incrementals; finer `14,18,16` helps more; cost is chunk-index RAM - params apply per-run so back up DBs in a separate run - [Source](https://borgbackup.readthedocs.io/en/master/faq.html); 12.45 GB Vintage Story sqlite: 2nd backup 518 MB default → 62 MB (15,19,17) → 37 MB (10,23,16) - [Source](https://github.com/borgbackup/borg/issues/5877)
+- Kopia: 8 MB photo with metadata edit re-uploads ~8 MB under 4 MB splitter (chunk > file); fix is smaller splitter at repo-creation cost of more chunks - [Source](https://kopia.discourse.group/t/chunk-size-setting/1351)
+- Fixed-splitter warning: 10 GB video + 1 prepended byte re-uploads 10 GB; content-based (buzhash/rabinkarp) uploads 1–2 chunks - [Source](https://kopia.discourse.group/t/difference-between-the-available-splitters/894)
+- Backup pipelines must chunk-then-compress (per-chunk); encrypt-then-chunk/compress is useless since ciphertext is indistinguishable from random; encrypted dedup needs weakened convergent/MLE schemes with leakage tradeoffs - [Source](https://eprint.iacr.org/2025/532.pdf); survey - [Source](https://dl.acm.org/doi/10.1145/3685278)
+- Kopia compresses each chunk independently (s2 default, gzip optional); splitting costs little ratio (466 MB → 119 MB standalone s2 vs 133 MB via Kopia-s2) - [Source](https://kopia.io/docs/advanced/compression/)
 
 ### Inferences
-- Daemon policy: (1) never store `.gz/.zip/.jpg/.mp4/.sqlite.gz` deltas expecting CDC wins — keep 1 mirror copy or thin retention; (2) ingest DBs/VM images uncompressed and let CDC + per-chunk compression do the work; (3) exclude or separately-schedule >100 MB media with tiny chunkers only if churn is low.
+- Daemon policy: (1) never store `.gz/.zip/.jpg/.mp4/.sqlite.gz` deltas expecting CDC wins - keep 1 mirror copy or thin retention; (2) ingest DBs/VM images uncompressed and let CDC + per-chunk compression do the work; (3) exclude or separately-schedule >100 MB media with tiny chunkers only if churn is low.
 - `.dump`/uncompressed SQL text is the most dedup-friendly DB form but slowest to produce; page-image `.backup` is the balanced default.
 
 ### Gaps

research_notes/Backup include exclude rules/must-include.md

@@ -3,21 +3,21 @@
 ## Which paths do guides universally say to include (~/Documents, ~/Pictures, ~/.config, project dirs, etc.)?
 
 ### Takeaway
-Guides converge on: back up all of `/home` (or whole `$HOME`) plus `/etc`, `/root`, `/var` (selectively) and `/opt`; on workstations this means XDG user dirs (`~/Documents`, `~/Pictures`, `~/Videos`, `~/Music`, `~/Downloads`), project dirs (`~/projects`, `~/src`), dotfiles/`~/.config`/`~/.local/share`, `~/.ssh`, browser profiles, mail, and package lists — never a bare `~/Documents`-only backup.
+Guides converge on: back up all of `/home` (or whole `$HOME`) plus `/etc`, `/root`, `/var` (selectively) and `/opt`; on workstations this means XDG user dirs (`~/Documents`, `~/Pictures`, `~/Videos`, `~/Music`, `~/Downloads`), project dirs (`~/projects`, `~/src`), dotfiles/`~/.config`/`~/.local/share`, `~/.ssh`, browser profiles, mail, and package lists - never a bare `~/Documents`-only backup.
 
 ### Cited Findings
-- Red Hat guide lists as must-back-up: `/etc` (system config, users/groups, networking, app configs), `/home` (all user data/downloads/documents/pictures under username), `/root` (admin scripts/notes/configs), `/var` shared/corporate data — and one dir not to back up (implied transient/cache) — [Source](https://www.redhat.com/en/blog/backup-dirs)
-- Ubuntu official help: copy personal files/settings usually in Home folder; if room, back up entire Home folder with exceptions (cache/thrash-type excludes detailed on linked page) — [Source](https://help.ubuntu.com/stable/ubuntu-help/backup-how.html.en)
-- PCWorld Linux backup roundup: as a rule a regular backup of the home directories is sufficient; `rsync -avP $HOME` to USB disk given as sufficient home backup — [Source](https://www.pcworld.com/article/2050183/best-linux-backup-tools.html)
-- OSTechNix 2026 reinstall checklist: back up more than `~/Documents`; must include personal files + `~/.local/share` (app data, game saves, Flatpak/Snap data), `~/.config` (app settings), browser profiles (Firefox `~/.mozilla` or new XDG `~/.config/mozilla` + `~/.cache/mozilla` + `~/.local/share/mozilla`, check `about:support`; Chrome/Chromium), SSH/GPG keys, dotfiles, app-specific data, installed package lists including Flatpak/Snap, system-level `/etc`, NetworkManager `system-connections`, cron/scheduled tasks; trap 1 is only backing up `~/Documents ~/Pictures ~/Downloads` and missing `~/projects`, VM images, second drives; trap 2 is forgetting hidden XDG data — [Source](https://ostechnix.com/things-to-back-up-before-reinstalling-linux)
-- OSTechNix: single `rsync` of whole `$HOME` captures dotfiles, browser profiles, SSH keys, app settings since all live under `$HOME` — [Source](https://ostechnix.com/things-to-back-up-before-reinstalling-linux)
-- Arch forum classic tar set: `tar zcvfp arch-system.gz /etc /boot /root` + per-user `/home/user1` + `/var --exclude /var/cache/pacman/pkg` — [Source](https://bbs.archlinux.org/viewtopic.php?id=83533)
-- Production borg example backs up `/home /etc /var/www /var/backups /opt /root` together with `--exclude-from` file — [Source](https://cubepath.com/docs/Backup%20Recovery/backup-with-borgbackup-deduplication)
-- Ubuntu Ask-Ubuntu full-system tar pattern excludes virtual/external `dev mnt proc sys` (and squashfs variant excludes `home media dev run mnt proc sys tmp`, then backs up `/home` separately excluding cloud mirrors like `Dropbox GoogleDrive`) — [Source](https://askubuntu.com/questions/7809/how-to-back-up-my-entire-system)
-- Linux Mint forum consensus: Timeshift = OS/system restore points, does nothing for data in `/home`; need separate file-level tool (BackInTime, FreeFileSync, Foxclone/Rescuezilla/Clonezilla for images) for Documents/Music/Pictures — [Source](https://forums.linuxmint.com/viewtopic.php?t=405449)
-- Dotfiles canon: `~/.bashrc`, `~/.bash_profile`, `~/.zshrc`, `~/.vimrc`, `~/.gitconfig`, `~/.ssh/config`, `~/.tmux.conf`, `~/.config/` XDG dir; secrets in `~/.netrc`, `~/.aws/credentials`, `~/.ssh/`, history `~/.bash_history` must be treated as sensitive / kept out of public repos — [Source](https://linuxcommandlibrary.com/man/dotfiles)
-- Dotfiles restore guides list as backup-worthy: `.ssh/config`, `.gnupg/pubring.kbx`, `.gnupg/trustdb.gpg`, plus stow-managed `zsh/git/vim/tmux` and XDG select dirs — [Source](https://github.com/RickCogley/dotfiles/blob/main/docs/how-to/backup-restore.md)
-- Keeply Linux tool scope explicitly: dotfiles, `.config` dirs, browser profiles, app settings, optionally SSH keys, emails — [Source](https://github.com/cozy533/keeply)
+- Red Hat guide lists as must-back-up: `/etc` (system config, users/groups, networking, app configs), `/home` (all user data/downloads/documents/pictures under username), `/root` (admin scripts/notes/configs), `/var` shared/corporate data - and one dir not to back up (implied transient/cache) - [Source](https://www.redhat.com/en/blog/backup-dirs)
+- Ubuntu official help: copy personal files/settings usually in Home folder; if room, back up entire Home folder with exceptions (cache/thrash-type excludes detailed on linked page) - [Source](https://help.ubuntu.com/stable/ubuntu-help/backup-how.html.en)
+- PCWorld Linux backup roundup: as a rule a regular backup of the home directories is sufficient; `rsync -avP $HOME` to USB disk given as sufficient home backup - [Source](https://www.pcworld.com/article/2050183/best-linux-backup-tools.html)
+- OSTechNix 2026 reinstall checklist: back up more than `~/Documents`; must include personal files + `~/.local/share` (app data, game saves, Flatpak/Snap data), `~/.config` (app settings), browser profiles (Firefox `~/.mozilla` or new XDG `~/.config/mozilla` + `~/.cache/mozilla` + `~/.local/share/mozilla`, check `about:support`; Chrome/Chromium), SSH/GPG keys, dotfiles, app-specific data, installed package lists including Flatpak/Snap, system-level `/etc`, NetworkManager `system-connections`, cron/scheduled tasks; trap 1 is only backing up `~/Documents ~/Pictures ~/Downloads` and missing `~/projects`, VM images, second drives; trap 2 is forgetting hidden XDG data - [Source](https://ostechnix.com/things-to-back-up-before-reinstalling-linux)
+- OSTechNix: single `rsync` of whole `$HOME` captures dotfiles, browser profiles, SSH keys, app settings since all live under `$HOME` - [Source](https://ostechnix.com/things-to-back-up-before-reinstalling-linux)
+- Arch forum classic tar set: `tar zcvfp arch-system.gz /etc /boot /root` + per-user `/home/user1` + `/var --exclude /var/cache/pacman/pkg` - [Source](https://bbs.archlinux.org/viewtopic.php?id=83533)
+- Production borg example backs up `/home /etc /var/www /var/backups /opt /root` together with `--exclude-from` file - [Source](https://cubepath.com/docs/Backup%20Recovery/backup-with-borgbackup-deduplication)
+- Ubuntu Ask-Ubuntu full-system tar pattern excludes virtual/external `dev mnt proc sys` (and squashfs variant excludes `home media dev run mnt proc sys tmp`, then backs up `/home` separately excluding cloud mirrors like `Dropbox GoogleDrive`) - [Source](https://askubuntu.com/questions/7809/how-to-back-up-my-entire-system)
+- Linux Mint forum consensus: Timeshift = OS/system restore points, does nothing for data in `/home`; need separate file-level tool (BackInTime, FreeFileSync, Foxclone/Rescuezilla/Clonezilla for images) for Documents/Music/Pictures - [Source](https://forums.linuxmint.com/viewtopic.php?t=405449)
+- Dotfiles canon: `~/.bashrc`, `~/.bash_profile`, `~/.zshrc`, `~/.vimrc`, `~/.gitconfig`, `~/.ssh/config`, `~/.tmux.conf`, `~/.config/` XDG dir; secrets in `~/.netrc`, `~/.aws/credentials`, `~/.ssh/`, history `~/.bash_history` must be treated as sensitive / kept out of public repos - [Source](https://linuxcommandlibrary.com/man/dotfiles)
+- Dotfiles restore guides list as backup-worthy: `.ssh/config`, `.gnupg/pubring.kbx`, `.gnupg/trustdb.gpg`, plus stow-managed `zsh/git/vim/tmux` and XDG select dirs - [Source](https://github.com/RickCogley/dotfiles/blob/main/docs/how-to/backup-restore.md)
+- Keeply Linux tool scope explicitly: dotfiles, `.config` dirs, browser profiles, app settings, optionally SSH keys, emails - [Source](https://github.com/cozy533/keeply)
 
 ### Inferences
 - Universal must-include = whole `/home/<user>` + `/etc` + package list; `/root` and `/var/lib`-style app data added for home-server role.
@@ -28,21 +28,21 @@ Guides converge on: back up all of `/home` (or whole `$HOME`) plus `/etc`, `/roo
 - No single 2023–2026 primary guide found quantifying mail (`~/Mail`, Thunderbird `~/.thunderbird`) include rates; only secondary tool scope mentions emails.
 - No reliable source found prescribing exact treatment of `~/.cache` vs `~/.local/share` boundary for Flatpak/Snap beyond OSTechNix summary.
 
-## How should small databases (sqlite), archives (.zip/.tar), and images be treated — raw files or dumps?
+## How should small databases (sqlite), archives (.zip/.tar), and images be treated - raw files or dumps?
 
 ### Takeaway
-Small static files (photos/images, `.zip/.tar` archives) are backed up as raw files and deduplicate well; live database files (sqlite, postgres/mysql) must be dumped (`VACUUM INTO`, `sqlite3 .backup`/backup API, `pg_dump`/`pg_dumpall`/`mysqldump`) before file backup — raw copy of a live DB is unsafe.
+Small static files (photos/images, `.zip/.tar` archives) are backed up as raw files and deduplicate well; live database files (sqlite, postgres/mysql) must be dumped (`VACUUM INTO`, `sqlite3 .backup`/backup API, `pg_dump`/`pg_dumpall`/`mysqldump`) before file backup - raw copy of a live DB is unsafe.
 
 ### Cited Findings
-- SQLite docs: safe live-copy methods are `sqlite3_rsync` (3.47.0+, 2024-10-21, bandwidth-efficient over SSH), `VACUUM INTO filename`, or backup API; plain file copy only safe with no transactions in progress, and if prior write failed must copy `-journal`/`-wal` together — [Source](https://sqlite.org/howtocorrupt.html)
-- SQLite forum (Borg user, WAL-mode pooled connections): cannot depend on perfect backup from open/changing DB because backup doesn't snapshot DB + journal in same instant; recommendation: close writers or backup API copy then back up the copy; `rm -f mydb.sqlite3.backup; sqlite3 mydb.sqlite3 "VACUUM INTO 'mydb.sqlite3.backup'"` added to backup script — [Source](https://sqlite.org/forum/forumpost/f173a78e2e?t=h)
-- Experiment on SQLite 3.50.6 WAL DB (1000 rows, 500 in `-wal`): `cp app.db backup_cp.db` yielded 500 rows, `integrity_check` still `ok` — stale copy passes checks silently; copying `app.db + app.db-wal + app.db-shm` together preserved 1000 rows; `.backup` (Online Backup API) copies page-by-page including WAL and restarts if source writes mid-copy — [Source](https://www.sqlprostudio.com/blog/68-how-to-safely-back-up-or-copy-a-live-sqlite-database)
-- `fs.copyFile()` on WAL DB copies only `.db` while `-wal/-shm` still written → instant `SQLITE_CORRUPT`; 7/7 rotating backups corrupted; fix is better-sqlite3 `.backup()` which handles all three files atomically; test restores — [Source](https://scottspence.com/posts/sqlite-corruption-fs-copyfile-issue)
-- Postgres docs: `pg_dump dbname > dumpfile` generates SQL commands to recreate DB; `pg_dumpall > dumpfile` preserves cluster-wide roles/tablespaces, restore via `psql -f dumpfile postgres` requiring superuser; file-level/WAL archiving is version-specific whereas dumps reload into newer versions — [Source](https://www.postgresql.org/docs/%EF%BC%99.6/backup-dump.html)
-- Postgres professional docs: `pg_dump` can emit text or archive formats for parallelism/fine-grained `pg_restore` control — [Source](https://postgrespro.com/docs/postgresql/17/backup-dump)
-- OSTechNix: don't copy raw data dir; use each DB's dump tool e.g. `mysqldump -u <user> -p <db> > backup.sql` — [Source](https://ostechnix.com/things-to-back-up-before-reinstalling-linux)
-- Borg quickstart warns: snapshot filesystems/volumes (LVM/ZFS useful), dump databases or stop DB servers, shut down VMs/containers before backing up disk images/volumes — [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)
-- Restic docs example: `plan.txt` change adds KiB, `archive.tar.gz` change re-adds MiB — illustrates large monolithic archives re-chunk poorly vs small files; still backed up as raw files — [Source](https://github.com/restic/restic/blob/master/doc/040_backup.rst)
+- SQLite docs: safe live-copy methods are `sqlite3_rsync` (3.47.0+, 2024-10-21, bandwidth-efficient over SSH), `VACUUM INTO filename`, or backup API; plain file copy only safe with no transactions in progress, and if prior write failed must copy `-journal`/`-wal` together - [Source](https://sqlite.org/howtocorrupt.html)
+- SQLite forum (Borg user, WAL-mode pooled connections): cannot depend on perfect backup from open/changing DB because backup doesn't snapshot DB + journal in same instant; recommendation: close writers or backup API copy then back up the copy; `rm -f mydb.sqlite3.backup; sqlite3 mydb.sqlite3 "VACUUM INTO 'mydb.sqlite3.backup'"` added to backup script - [Source](https://sqlite.org/forum/forumpost/f173a78e2e?t=h)
+- Experiment on SQLite 3.50.6 WAL DB (1000 rows, 500 in `-wal`): `cp app.db backup_cp.db` yielded 500 rows, `integrity_check` still `ok` - stale copy passes checks silently; copying `app.db + app.db-wal + app.db-shm` together preserved 1000 rows; `.backup` (Online Backup API) copies page-by-page including WAL and restarts if source writes mid-copy - [Source](https://www.sqlprostudio.com/blog/68-how-to-safely-back-up-or-copy-a-live-sqlite-database)
+- `fs.copyFile()` on WAL DB copies only `.db` while `-wal/-shm` still written → instant `SQLITE_CORRUPT`; 7/7 rotating backups corrupted; fix is better-sqlite3 `.backup()` which handles all three files atomically; test restores - [Source](https://scottspence.com/posts/sqlite-corruption-fs-copyfile-issue)
+- Postgres docs: `pg_dump dbname > dumpfile` generates SQL commands to recreate DB; `pg_dumpall > dumpfile` preserves cluster-wide roles/tablespaces, restore via `psql -f dumpfile postgres` requiring superuser; file-level/WAL archiving is version-specific whereas dumps reload into newer versions - [Source](https://www.postgresql.org/docs/%EF%BC%99.6/backup-dump.html)
+- Postgres professional docs: `pg_dump` can emit text or archive formats for parallelism/fine-grained `pg_restore` control - [Source](https://postgrespro.com/docs/postgresql/17/backup-dump)
+- OSTechNix: don't copy raw data dir; use each DB's dump tool e.g. `mysqldump -u <user> -p <db> > backup.sql` - [Source](https://ostechnix.com/things-to-back-up-before-reinstalling-linux)
+- Borg quickstart warns: snapshot filesystems/volumes (LVM/ZFS useful), dump databases or stop DB servers, shut down VMs/containers before backing up disk images/volumes - [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)
+- Restic docs example: `plan.txt` change adds KiB, `archive.tar.gz` change re-adds MiB - illustrates large monolithic archives re-chunk poorly vs small files; still backed up as raw files - [Source](https://github.com/restic/restic/blob/master/doc/040_backup.rst)
 
 ### Inferences
 - Photos/images/archives: raw-file backup is correct; no dump needed; chunked dedup tools (restic/borg/kopia) handle them but modified tar/zip re-stores large chunks.
@@ -58,17 +58,17 @@ Small static files (photos/images, `.zip/.tar` archives) are backed up as raw fi
 None auto-selects `~/Documents`/`~/.config`; all default to exactly what paths you pass (no implicit includes) and provide opt-in excludes (`--exclude*`, `--patterns-from`, `.kopiaignore`/policy, Time Machine StdExclusions + user list) for caches, build artifacts, and system pseudo-filesystems.
 
 ### Cited Findings
-- Restic `backup [FILE/DIR]...` creates snapshot of exactly given args; no default source; exclude options: `--exclude/--iexclude`, `--exclude-file`, `--exclude-caches` (CACHEDIR.TAG), `--exclude-if-present foo`, `--exclude-larger-than`, `--exclude-cloud-files` (Win/macOS OneDrive/iCloud only); excludes don't apply to explicitly passed file path itself, only contents under dirs — [Source](https://restic.readthedocs.io/en/stable/040_backup.html)
-- Restic excludes use Go `filepath.Match` against full path, gitignore-like: once dir excluded can't re-include inside; example backs up selection inside `$HOME` — [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html)
-- Restic handles `--one-file-system/-x` to not cross filesystem boundaries/subvolumes; saves/restores ACLs/xattrs; sparse holes deduped/compressed not stored explicitly — [Source](https://github.com/restic/restic/blob/master/doc/manual_rest.rst)
-- NixOS restic module example: `paths = ["/home"]` with `exclude = ["/home/*/.cache" ".git"]` pattern as conventional home selection — [Source](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/backup/restic.nix)
-- Borg `create repo::archive ~/Documents`, `~/Documents ~/src --exclude '*.pyc'`, `/home --exclude thumbnail regex`, ` / --one-file-system` are docs examples — user picks roots; no default root — [Source](https://borgbackup.readthedocs.io/en/1.0.5/usage.html)
-- Borg patterns: fnmatch default for `--exclude`, shell-style for `--pattern`; `sh:**/steamapps/common/**`, `sh:home/user/.cache/**`, trailing-slash `some/path/` keeps dir not contents vs no-slash excludes both; `--exclude-from`, `--patterns-from`, `--exclude-caches` (CACHEDIR.TAG), `--exclude-if-present`, `--keep-tag-files`, `--one-file-system`, nodump flag respected — [Source](https://man.archlinux.org/man/borg-patterns.1.txt)
-- Kopia: no default source; snapshots what you `snapshot create`; ignores via `.kopiaignore` (default file), global/per-source policy `--add-ignore/--add-dot-ignore`, `--ignore-cache-dirs true` (inherited global), `--ignore-dir-errors/--ignore-file-errors`, `--one-file-system`, never/only-compress lists; before/after folder/root actions for dumps/snapshots with timeout/modes — [Source](https://kopia.io/docs/reference/command-line/common/policy-set/)
-- Kopia ignore syntax: `#` comment, `!` negate, `*`/`**`/`?`/`[0-9a-zA-Z]`/`[abc]`, leading `/` root-only; examples `*.dat`, `/logs/*`, `tmp.db`, `**/logs/**`; must test — incomplete snapshots possible — [Source](https://kopia.io/docs/advanced/kopiaignore/)
-- Time Machine default backs up everything except system files/apps from macOS install, caches, StdExclusions (`/System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist`), backup disk itself; user adds via Options → Exclude; excluded items still in local snapshots; damaged system requires macOS reinstall + Migration Assistant — [Source](https://support.apple.com/en-ca/guide/mac-help/mh15622/mac)
-- Time Machine `tmutil isexcluded/addexclusion/removeexclusion`, sticky vs fixed-path (`-p`) exclusions semantics — [Source](https://www.unix.com/man-page/osx/8/TMUTIL)
-- ArchWiki System backup: no default set; methods are btrfs/LVM snapshots, rsync, tar, SquashFS (no ACLs); recommends 3-2-1, regular integrity + restore tests; automation via systemd timer/cron with least-privilege `CAP_DAC_READ_SEARCH` example — [Source](https://wiki.archlinux.org/title/System_backup)
+- Restic `backup [FILE/DIR]...` creates snapshot of exactly given args; no default source; exclude options: `--exclude/--iexclude`, `--exclude-file`, `--exclude-caches` (CACHEDIR.TAG), `--exclude-if-present foo`, `--exclude-larger-than`, `--exclude-cloud-files` (Win/macOS OneDrive/iCloud only); excludes don't apply to explicitly passed file path itself, only contents under dirs - [Source](https://restic.readthedocs.io/en/stable/040_backup.html)
+- Restic excludes use Go `filepath.Match` against full path, gitignore-like: once dir excluded can't re-include inside; example backs up selection inside `$HOME` - [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html)
+- Restic handles `--one-file-system/-x` to not cross filesystem boundaries/subvolumes; saves/restores ACLs/xattrs; sparse holes deduped/compressed not stored explicitly - [Source](https://github.com/restic/restic/blob/master/doc/manual_rest.rst)
+- NixOS restic module example: `paths = ["/home"]` with `exclude = ["/home/*/.cache" ".git"]` pattern as conventional home selection - [Source](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/backup/restic.nix)
+- Borg `create repo::archive ~/Documents`, `~/Documents ~/src --exclude '*.pyc'`, `/home --exclude thumbnail regex`, ` / --one-file-system` are docs examples - user picks roots; no default root - [Source](https://borgbackup.readthedocs.io/en/1.0.5/usage.html)
+- Borg patterns: fnmatch default for `--exclude`, shell-style for `--pattern`; `sh:**/steamapps/common/**`, `sh:home/user/.cache/**`, trailing-slash `some/path/` keeps dir not contents vs no-slash excludes both; `--exclude-from`, `--patterns-from`, `--exclude-caches` (CACHEDIR.TAG), `--exclude-if-present`, `--keep-tag-files`, `--one-file-system`, nodump flag respected - [Source](https://man.archlinux.org/man/borg-patterns.1.txt)
+- Kopia: no default source; snapshots what you `snapshot create`; ignores via `.kopiaignore` (default file), global/per-source policy `--add-ignore/--add-dot-ignore`, `--ignore-cache-dirs true` (inherited global), `--ignore-dir-errors/--ignore-file-errors`, `--one-file-system`, never/only-compress lists; before/after folder/root actions for dumps/snapshots with timeout/modes - [Source](https://kopia.io/docs/reference/command-line/common/policy-set/)
+- Kopia ignore syntax: `#` comment, `!` negate, `*`/`**`/`?`/`[0-9a-zA-Z]`/`[abc]`, leading `/` root-only; examples `*.dat`, `/logs/*`, `tmp.db`, `**/logs/**`; must test - incomplete snapshots possible - [Source](https://kopia.io/docs/advanced/kopiaignore/)
+- Time Machine default backs up everything except system files/apps from macOS install, caches, StdExclusions (`/System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist`), backup disk itself; user adds via Options → Exclude; excluded items still in local snapshots; damaged system requires macOS reinstall + Migration Assistant - [Source](https://support.apple.com/en-ca/guide/mac-help/mh15622/mac)
+- Time Machine `tmutil isexcluded/addexclusion/removeexclusion`, sticky vs fixed-path (`-p`) exclusions semantics - [Source](https://www.unix.com/man-page/osx/8/TMUTIL)
+- ArchWiki System backup: no default set; methods are btrfs/LVM snapshots, rsync, tar, SquashFS (no ACLs); recommends 3-2-1, regular integrity + restore tests; automation via systemd timer/cron with least-privilege `CAP_DAC_READ_SEARCH` example - [Source](https://wiki.archlinux.org/title/System_backup)
 
 ### Inferences
 - Versioning tools scope = explicit roots + exclude list; portable Linux default is effectively `/home + /etc + dumps` minus `*.cache/CACHEDIR.TAG`, `node_modules/target/.git`, VM images while running.
@@ -84,15 +84,15 @@ None auto-selects `~/Documents`/`~/.config`; all default to exactly what paths y
 Never file-copy a live DB; quiesce (close writers/stop server), filesystem/LVM/ZFS snapshot, or dump (`sqlite3 backup API/VACUUM INTO`, `pg_dumpall`, `mysqldump`); for SQLite WAL must keep `.db+-wal+-shm/-journal` together, never delete hot journals, beware POSIX `close()` dropping advisory locks and fork/link/rename hazards.
 
 ### Cited Findings
-- SQLite: backup/restore while transaction active → copy mixes old/new → corrupt; safe via `sqlite3_rsync`, `VACUUM INTO`, backup API even on live DB; idle-file copy only safe with no tx in progress — [Source](https://sqlite.org/howtocorrupt.html)
-- SQLite: never move/delete/rename hot `-journal/-wal` after crash; mispairing (swap/overwrite/move journal, copy DB without journal, overwrite DB without deleting hot journal) likely corrupts; quiescent DB has no journal, only DB file matters — [Source](https://sqlite.org/howtocorrupt.html)
-- SQLite POSIX advisory-lock quirk: any thread `open/read/close` on DB file drops locks for all threads (close cancels locks); bypassing lib for backup read can corrupt; since 3.51.0 (2025-11-04) extra WAL defenses but not cure-all — never `close()` DB file while connections open even in other threads — [Source](https://sqlite.org/howtocorrupt.html)
-- SQLite: don't use two SQLite copies linked in one app (separate lock lists), don't mix locking protocols (POSIX vs dot-file/NFS), don't unlink/rename open DB (shared journal name → cross-recovery corruption, `SQLITE_WARNING` since 3.7.17), don't multi-link/symlink same file (wrong journal lookup; canonicalization since 3.10.0), don't carry connection across `fork()` — [Source](https://sqlite.org/howtocorrupt.html)
-- SQLite WAL forgiving of out-of-order writes except during checkpoint; `COMMIT` sync failure loses durability not consistency; checkpoint infrequently as defense; QNX `mmap` + WAL needs exclusive locking/no-mmap — [Source](https://sqlite.org/howtocorrupt.html)
-- Forum: with WAL expect journal replay on open after restore, uncommitted lost; for professional frozen-moment copy don't back up while changing; full WAL checkpoint then copy main file pristine if no checkpoint during copy (disable autocheckpoint temporarily) — [Source](https://sqlite.org/forum/forumpost/f173a78e2e?t=h)
-- Borg docs: avoid programs changing files during backup; LVM/ZFS snapshot or dump/stop DBs; shutdown VMs/containers first — [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)
-- Kopia actions support `before-folder/after-folder` and `before-snapshot-root/after-snapshot-root` scripts (e.g. `zfs snapshot` + mount then snapshot mount, `zfs destroy` after; `pg_dumpall`/sqlite dump) with `essential/optional/async` modes and timeout (default 5m) — [Source](https://github.com/kopia/kopia/blob/master/site/content/docs/Advanced/Actions/_index.md)
-- SafeKeep (rdiff-backup wrapper) noted as integrating with LVM and databases for consistent backups — [Source](https://wiki.archlinux.org/title/Synchronization_and_backup_programs)
+- SQLite: backup/restore while transaction active → copy mixes old/new → corrupt; safe via `sqlite3_rsync`, `VACUUM INTO`, backup API even on live DB; idle-file copy only safe with no tx in progress - [Source](https://sqlite.org/howtocorrupt.html)
+- SQLite: never move/delete/rename hot `-journal/-wal` after crash; mispairing (swap/overwrite/move journal, copy DB without journal, overwrite DB without deleting hot journal) likely corrupts; quiescent DB has no journal, only DB file matters - [Source](https://sqlite.org/howtocorrupt.html)
+- SQLite POSIX advisory-lock quirk: any thread `open/read/close` on DB file drops locks for all threads (close cancels locks); bypassing lib for backup read can corrupt; since 3.51.0 (2025-11-04) extra WAL defenses but not cure-all - never `close()` DB file while connections open even in other threads - [Source](https://sqlite.org/howtocorrupt.html)
+- SQLite: don't use two SQLite copies linked in one app (separate lock lists), don't mix locking protocols (POSIX vs dot-file/NFS), don't unlink/rename open DB (shared journal name → cross-recovery corruption, `SQLITE_WARNING` since 3.7.17), don't multi-link/symlink same file (wrong journal lookup; canonicalization since 3.10.0), don't carry connection across `fork()` - [Source](https://sqlite.org/howtocorrupt.html)
+- SQLite WAL forgiving of out-of-order writes except during checkpoint; `COMMIT` sync failure loses durability not consistency; checkpoint infrequently as defense; QNX `mmap` + WAL needs exclusive locking/no-mmap - [Source](https://sqlite.org/howtocorrupt.html)
+- Forum: with WAL expect journal replay on open after restore, uncommitted lost; for professional frozen-moment copy don't back up while changing; full WAL checkpoint then copy main file pristine if no checkpoint during copy (disable autocheckpoint temporarily) - [Source](https://sqlite.org/forum/forumpost/f173a78e2e?t=h)
+- Borg docs: avoid programs changing files during backup; LVM/ZFS snapshot or dump/stop DBs; shutdown VMs/containers first - [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)
+- Kopia actions support `before-folder/after-folder` and `before-snapshot-root/after-snapshot-root` scripts (e.g. `zfs snapshot` + mount then snapshot mount, `zfs destroy` after; `pg_dumpall`/sqlite dump) with `essential/optional/async` modes and timeout (default 5m) - [Source](https://github.com/kopia/kopia/blob/master/site/content/docs/Advanced/Actions/_index.md)
+- SafeKeep (rdiff-backup wrapper) noted as integrating with LVM and databases for consistent backups - [Source](https://wiki.archlinux.org/title/Synchronization_and_backup_programs)
 
 ### Inferences
 - Correct pattern for single-user Linux: `pre-action: pg_dumpall/sqlite VACUUM INTO/.backup → file under /home or /var/backups` then `borg/restic/kopia snapshot` includes dump; keep live DB files too but rely on dump for restore.

research_notes/Backup include exclude rules/never-include.md

@@ -3,22 +3,22 @@
 ## What are the canonical exclude directories and file patterns across backup tools (give concrete lists)?
 
 ### Takeaway
-Canonical Linux file-backup excludes converge on: virtual/kernel filesystems (`/proc`, `/sys`, `/dev`, `/run`), ephemeral (`/tmp`, `/var/tmp`, `/var/run`, `/var/lock`), mounts/media (`/mnt`, `/media`, `/lost+found`, `/swapfile`), package/cache/logs (`/var/cache/*`, `/var/log/*`, pacman cache), per-user caches/trash/thumbnails, and repro data (build/dependency dirs, VM/container storage) — implemented as explicit exclude-files in restic/borg plus `--exclude-caches`/`--one-file-system` and Kopia policies/`.kopiaignore`.
+Canonical Linux file-backup excludes converge on: virtual/kernel filesystems (`/proc`, `/sys`, `/dev`, `/run`), ephemeral (`/tmp`, `/var/tmp`, `/var/run`, `/var/lock`), mounts/media (`/mnt`, `/media`, `/lost+found`, `/swapfile`), package/cache/logs (`/var/cache/*`, `/var/log/*`, pacman cache), per-user caches/trash/thumbnails, and repro data (build/dependency dirs, VM/container storage) - implemented as explicit exclude-files in restic/borg plus `--exclude-caches`/`--one-file-system` and Kopia policies/`.kopiaignore`.
 
 ### Cited Findings
-- restic has no built-in default excludes; user supplies `--exclude`, `--iexclude`, `--exclude-file`, `--exclude-if-present foo`, `--exclude-caches` (CACHEDIR.TAG dirs), `--exclude-larger-than`, `--exclude-cloud-files`, plus `-x/--one-file-system` to stay on one filesystem — [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html); also documented that excludes do not apply to explicitly-passed backup-source paths — [Source](https://github.com/restic/restic/blob/master/doc/040_backup.rst)
-- restic pattern syntax is Go `filepath.Match` + `**` for crossing `/`, matched on complete path components (`foo` matches `/dir1/foo/...` but not `/dir/foobar`), trailing `/` ignored, leading `/` anchors at root — [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html)
-- borg `create` supports `-e/--exclude PATTERN`, `--exclude-from FILE`, `--exclude-caches` (CACHEDIR.TAG), `--exclude-if-present NAME`, `--keep-tag-files`, `-x/--one-file-system`, and respects `nodump` flag; recommended test is `borg create --list --dry-run` — [Source](https://borgbackup.readthedocs.io/en/1.0.5/usage.html); pattern styles are `fnmatch` (default for `--exclude`), `sh:`, `re:`, path prefix/full-match, with trailing-`/` meaning “keep dir, skip contents” — [Source](https://manpages.debian.org/testing/borgbackup/borg-patterns.1.en.html)
-- borg quickstart automation example excludes `--exclude-caches --exclude 'home/*/.cache/*' --exclude 'var/tmp/*'` when backing up `/etc`-style roots — [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)
-- Kopia has no static exclude-file; it uses policy `Ignore rules` + `Read ignore rules from files` (default `.kopiaignore`) + `Ignore cache directories: true` (default true, inherits global→host→user→path) + `Scan one filesystem only: true` — [Source](https://github.com/kopia/kopia/issues/3334); `kopia policy set` flags include `--add-ignore`, `--add-dot-ignore`, `--ignore-cache-dirs [true|false|inherit]`, `--ignore-dir-errors`, `--one-file-system`-equivalent `Scan one filesystem only` — [Source](https://kopia.io/docs/reference/command-line/common/policy-set/)
-- Kopia’s CACHEDIR.TAG-equivalent (`--ignore-cache-dirs`, default true) was added to mirror restic `--exclude-caches`, defaulting to ignore caches via policy — [Source](https://github.com/kopia/kopia/issues/564)
-- ArchWiki restic full-system `/etc/restic/excludes.txt` canonical list: `/data/**`, `/dev/**`, `/home/*/**/*.pyc`, `/home/*/**/__pycache__/**`, `/home/*/**/node_modules/**`, `/home/*/.cache/**`, `/home/*/.local/lib/python*/site-packages/**`, `/home/*/.mozilla/firefox/*/Cache/**`, `/lost+found/**`, `/media/**`, `/mnt/**`, `/proc/**`, `/root/**`, `/run/**`, `/swapfile`, `/sys/**`, `/tmp/**`, `/var/cache/**`, `/var/cache/pacman/pkg/**`, `/var/lib/docker/**`, `/var/lib/libvirt/**`, `/var/lock/**`, `/var/log/**`, `/var/run/**`, noting `--one-file-system` can replace `/proc`/`/run`/`/mnt` entries while preserving mountpoints — [Source](https://wiki.archlinux.org/title/Restic)
-- Debian restic-forum practitioner system list: `/media`, `/mnt`, `/cdrom`, `/proc`, `/sys`, `/dev`, `/run`, `/tmp`, `/var/run`, `/var/lock`, `/var/tmp`, `/lost+found`, `/swapfile`, `/var/cache/restic`, Steam `.../Steam/steamapps`, plus home temp items `.gvfs`, `.local/share/gvfs-metadata`, `.local/share/Trash`, `.cache`, `.dbus`, `.xsession-errors`, `.Xauthority`, `.gksu.lock`, `.local/share/flatpak/appstream` — [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653)
-- Extended community restic list adds: `nobackup/NoBackup`, `Downloads/*`, `VirtualBox VMs/`, `Dropbox/*`, `rclone/`, `snap/`, `.var/**/cache/`, `.local/share/docker/`, `.local/share/JetBrains/`, `.vscode/extensions/`, `.config/Code/`, `.m2/repository`, `target/*`, `build/*`, `node_modules/*`, `jspm_packages/*`, `web_modules/*`, `.npm`, `.coursier/cache`, `.sbt`, `.stack`, `.sdkman`, `.jdks`, `.eclipse`, `.venv-py3`, `.wine`, `.android`, `Android/Sdk`, `.gradle`, `.adobe`, `.macromedia`, `.thumbnails`, `.thunderbird/*/Cache`, `.mozilla/firefox/*/Cache|storage|minidumps|*.sqlite*`, `.config/**/Cache|GPUCache|ShaderCache`, `.config/chromium/Default/...History|Favicons|Storage|Cache`, `.local/share/baloo|zeitgeist|akonadi`, `.gnupg/rnd|random_seed|*.lock`, `.pulse*`, `.java/deployment/cache`, `.dropbox*` — [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653)
-- ArchWiki tar full-system guidance: exclude `/opt/backup/arch-full*` (own backups), `/tmp/*`, `/var/cache/pacman/pkg/`, boot from LiveCD with plain `chroot` (not `arch-chroot`) to avoid capturing tempfs/memory — [Source](https://wiki.archlinux.org/title/Full_system_backup_with_tar_(Italiano))
-- Guide summary: when backing up `/`, explicitly exclude `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and bind/overlay mounts; when backing up only `/home`+`/etc` both borg/restic skip virtual FS by default — [Source](https://linuxjunkies.org/guides/back-up-with-borg-or-restic)
-- Kopia `.kopiaignore` syntax: one rule/line, `#` comment, `*` (any chars), `**` (any dirs), `?`, `[0-9]/[a-z]/[A-Z]/[abc]`, leading `/` roots rule, `!` negates; policy can point at alternate ignore-files — [Source](https://kopia.io/docs/advanced/kopiaignore/)
-- Time-Machine-ecosystem analogue (Arq honoring Apple TM exclusions): `.Trash`, `Library/Caches`, `Library/Logs`, `Library/Mail/.../Envelope Index*`, `Library/Safari/WebpageIcons.db`, `Library/Saved Application State`, `Library/iTunes/iPad Software Updates` — [Source](https://www.arqbackup.com/docs/arqbackup/pages/adding_folder.html)
+- restic has no built-in default excludes; user supplies `--exclude`, `--iexclude`, `--exclude-file`, `--exclude-if-present foo`, `--exclude-caches` (CACHEDIR.TAG dirs), `--exclude-larger-than`, `--exclude-cloud-files`, plus `-x/--one-file-system` to stay on one filesystem - [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html); also documented that excludes do not apply to explicitly-passed backup-source paths - [Source](https://github.com/restic/restic/blob/master/doc/040_backup.rst)
+- restic pattern syntax is Go `filepath.Match` + `**` for crossing `/`, matched on complete path components (`foo` matches `/dir1/foo/...` but not `/dir/foobar`), trailing `/` ignored, leading `/` anchors at root - [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html)
+- borg `create` supports `-e/--exclude PATTERN`, `--exclude-from FILE`, `--exclude-caches` (CACHEDIR.TAG), `--exclude-if-present NAME`, `--keep-tag-files`, `-x/--one-file-system`, and respects `nodump` flag; recommended test is `borg create --list --dry-run` - [Source](https://borgbackup.readthedocs.io/en/1.0.5/usage.html); pattern styles are `fnmatch` (default for `--exclude`), `sh:`, `re:`, path prefix/full-match, with trailing-`/` meaning “keep dir, skip contents” - [Source](https://manpages.debian.org/testing/borgbackup/borg-patterns.1.en.html)
+- borg quickstart automation example excludes `--exclude-caches --exclude 'home/*/.cache/*' --exclude 'var/tmp/*'` when backing up `/etc`-style roots - [Source](https://github.com/borgbackup/borg/blob/master/docs/quickstart.rst)
+- Kopia has no static exclude-file; it uses policy `Ignore rules` + `Read ignore rules from files` (default `.kopiaignore`) + `Ignore cache directories: true` (default true, inherits global→host→user→path) + `Scan one filesystem only: true` - [Source](https://github.com/kopia/kopia/issues/3334); `kopia policy set` flags include `--add-ignore`, `--add-dot-ignore`, `--ignore-cache-dirs [true|false|inherit]`, `--ignore-dir-errors`, `--one-file-system`-equivalent `Scan one filesystem only` - [Source](https://kopia.io/docs/reference/command-line/common/policy-set/)
+- Kopia’s CACHEDIR.TAG-equivalent (`--ignore-cache-dirs`, default true) was added to mirror restic `--exclude-caches`, defaulting to ignore caches via policy - [Source](https://github.com/kopia/kopia/issues/564)
+- ArchWiki restic full-system `/etc/restic/excludes.txt` canonical list: `/data/**`, `/dev/**`, `/home/*/**/*.pyc`, `/home/*/**/__pycache__/**`, `/home/*/**/node_modules/**`, `/home/*/.cache/**`, `/home/*/.local/lib/python*/site-packages/**`, `/home/*/.mozilla/firefox/*/Cache/**`, `/lost+found/**`, `/media/**`, `/mnt/**`, `/proc/**`, `/root/**`, `/run/**`, `/swapfile`, `/sys/**`, `/tmp/**`, `/var/cache/**`, `/var/cache/pacman/pkg/**`, `/var/lib/docker/**`, `/var/lib/libvirt/**`, `/var/lock/**`, `/var/log/**`, `/var/run/**`, noting `--one-file-system` can replace `/proc`/`/run`/`/mnt` entries while preserving mountpoints - [Source](https://wiki.archlinux.org/title/Restic)
+- Debian restic-forum practitioner system list: `/media`, `/mnt`, `/cdrom`, `/proc`, `/sys`, `/dev`, `/run`, `/tmp`, `/var/run`, `/var/lock`, `/var/tmp`, `/lost+found`, `/swapfile`, `/var/cache/restic`, Steam `.../Steam/steamapps`, plus home temp items `.gvfs`, `.local/share/gvfs-metadata`, `.local/share/Trash`, `.cache`, `.dbus`, `.xsession-errors`, `.Xauthority`, `.gksu.lock`, `.local/share/flatpak/appstream` - [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653)
+- Extended community restic list adds: `nobackup/NoBackup`, `Downloads/*`, `VirtualBox VMs/`, `Dropbox/*`, `rclone/`, `snap/`, `.var/**/cache/`, `.local/share/docker/`, `.local/share/JetBrains/`, `.vscode/extensions/`, `.config/Code/`, `.m2/repository`, `target/*`, `build/*`, `node_modules/*`, `jspm_packages/*`, `web_modules/*`, `.npm`, `.coursier/cache`, `.sbt`, `.stack`, `.sdkman`, `.jdks`, `.eclipse`, `.venv-py3`, `.wine`, `.android`, `Android/Sdk`, `.gradle`, `.adobe`, `.macromedia`, `.thumbnails`, `.thunderbird/*/Cache`, `.mozilla/firefox/*/Cache|storage|minidumps|*.sqlite*`, `.config/**/Cache|GPUCache|ShaderCache`, `.config/chromium/Default/...History|Favicons|Storage|Cache`, `.local/share/baloo|zeitgeist|akonadi`, `.gnupg/rnd|random_seed|*.lock`, `.pulse*`, `.java/deployment/cache`, `.dropbox*` - [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653)
+- ArchWiki tar full-system guidance: exclude `/opt/backup/arch-full*` (own backups), `/tmp/*`, `/var/cache/pacman/pkg/`, boot from LiveCD with plain `chroot` (not `arch-chroot`) to avoid capturing tempfs/memory - [Source](https://wiki.archlinux.org/title/Full_system_backup_with_tar_(Italiano))
+- Guide summary: when backing up `/`, explicitly exclude `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and bind/overlay mounts; when backing up only `/home`+`/etc` both borg/restic skip virtual FS by default - [Source](https://linuxjunkies.org/guides/back-up-with-borg-or-restic)
+- Kopia `.kopiaignore` syntax: one rule/line, `#` comment, `*` (any chars), `**` (any dirs), `?`, `[0-9]/[a-z]/[A-Z]/[abc]`, leading `/` roots rule, `!` negates; policy can point at alternate ignore-files - [Source](https://kopia.io/docs/advanced/kopiaignore/)
+- Time-Machine-ecosystem analogue (Arq honoring Apple TM exclusions): `.Trash`, `Library/Caches`, `Library/Logs`, `Library/Mail/.../Envelope Index*`, `Library/Safari/WebpageIcons.db`, `Library/Saved Application State`, `Library/iTunes/iPad Software Updates` - [Source](https://www.arqbackup.com/docs/arqbackup/pages/adding_folder.html)
 
 ### Inferences
 - There is no single vendor “default exclude file” for restic/borg on Linux; the de-facto standard is the ArchWiki + forum lists above plus `--exclude-caches` and `--one-file-system`.
@@ -34,16 +34,16 @@ Canonical Linux file-backup excludes converge on: virtual/kernel filesystems (`/
 Correctness excludes (virtual FS, sockets/FIFOs/devices, live DB/VM/container backing stores, cloud-online-only stubs) prevent hangs, errors, or unrestorable data; size excludes (caches, trash, thumbnails, logs, browser profiles, package caches, media/Steam) prevent bloat; reproducibility excludes (dependency/build trees) are safe to drop because lockfiles+manifests rebuild them.
 
 ### Cited Findings
-- `/proc` and `/sys` are virtual/kernel pseudo-filesystems (proc exposes live kernel/process state, `proc_sys` exposes tunable sysctls; size often reported 0, constantly changing) — backing them up captures no stable data — [Source](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/4/html/reference_guide/ch-proc); [Source](https://man7.org/linux/man-pages/man5/proc_sys.5.html)
-- restic forum guidance: use `--exclude-caches` to auto-exclude CACHEDIR.TAG-marked dirs for “non permanent system files”; full-system threads treat only a short path list as needing exclusion — [Source](https://forum.restic.net/t/linux-exclusion-of-non-permanent-system-files/6629)
-- borg by default does not open/read block/char devices or FIFOs; `--read-special` is required to force reading them as regular files (and to follow symlinks to them) — [Source](https://www.systutorials.com/linux-manual-page-1-borg)
-- restic 0.17.0+ fix “Exclude irregular files from backups” (sockets, FIFOs, devices skipped) confirms prior irregular-file handling was a correctness fix — [Source](https://github.com/restic/restic/releases)
-- Kopia policy has explicit `--ignore-dir-errors` to tolerate unreadable/locked dirs during traversal, separate from ignore-rules; issue reports show `fuse` mounts (e.g. `seafile/fuse`) still `lstat`-probed during estimation even when ignored, motivating explicit excludes for FUSE/locked paths — [Source](https://github.com/kopia/kopia/issues/3334); [Source](https://kopia.io/docs/reference/command-line/common/policy-set/)
-- Docker storage (`/var/lib/docker`: `overlay2/`, `image/`, `buildkit/`, `volumes/`, `containers/`) is large, churn-heavy, overlay-mounted and often locked/inconsistent without daemon quiesce; community Arch/restic lists therefore exclude `/var/lib/docker/**` and `/var/lib/libvirt/**`, and Proxmox `vzdump --exclude-path` examples exclude `/var/lib/docker/` alongside `/run/`, `/dev/shm`, `/dev/fuse` — [Source](https://wiki.archlinux.org/title/Restic); [Source](https://blog.devops.dev/docker-cleanup-and-relocate-81d2dcd32b8a); [Source](https://forum.proxmox.com/threads/backup-and-exclude-path.125966)
-- Docker’s own backup guidance says file-copy of the VM disk (`Docker.raw`/`docker_data.vhdx`) or `/var/lib/docker` requires Docker fully stopped; otherwise use `docker save`/`docker pull` + volume dump/restore procedures — [Source](https://docs.docker.com/desktop/settings-and-maintenance/backup-and-restore.md)
-- Veeam (VM-image backup reference) automatically excludes VM log files to cut size/time, and supports excluding swap files, deleted-block (BitLooker) data, and per-disk/template excludes for the same size/correctness reasons — [Source](https://helpcenter.veeam.com/docs/vbr/userguide/data_exclusion.html)
-- Size-category examples: pacman cache `/var/cache/pacman/pkg/`, `/var/cache/*`, `/var/log/*`, `/var/tmp/*`, browser `Cache/GPUCache/ShaderCache`, `thunderbird/*/Cache`, `~/.thumbnails`, `~/.local/share/Trash`, Steam `steamapps`, `~/Downloads/*`, Dropbox/rclone replicas — all excluded for bloat, not correctness — [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653); [Source](https://wiki.archlinux.org/title/Restic)
-- Reproducibility-category examples: `node_modules/*`, `jspm_packages/*`, `web_modules/*`, `target/*`, `build/*`, `~/.m2/repository`, `~/.ivy2`, `~/.gradle`, `~/.coursier/cache`, `~/.sbt`, `~/.stack`, `__pycache__`, `*.pyc`, `~/.local/lib/python*/site-packages/**`, `~/.npm`, `~/.pkg-cache/`, `~/.sdkman/`, `.venv-py3` — explicitly listed as rebuildable caches/toolchains — [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653); webdev backup tooling states “node_modules is never included … rebuild with yarn/npm install” while always keeping `.env`, `yarn.lock`, `package-lock.json` — [Source](https://github.com/ICJIA/backup-webdev/blob/main/README.md)
+- `/proc` and `/sys` are virtual/kernel pseudo-filesystems (proc exposes live kernel/process state, `proc_sys` exposes tunable sysctls; size often reported 0, constantly changing) - backing them up captures no stable data - [Source](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/4/html/reference_guide/ch-proc); [Source](https://man7.org/linux/man-pages/man5/proc_sys.5.html)
+- restic forum guidance: use `--exclude-caches` to auto-exclude CACHEDIR.TAG-marked dirs for “non permanent system files”; full-system threads treat only a short path list as needing exclusion - [Source](https://forum.restic.net/t/linux-exclusion-of-non-permanent-system-files/6629)
+- borg by default does not open/read block/char devices or FIFOs; `--read-special` is required to force reading them as regular files (and to follow symlinks to them) - [Source](https://www.systutorials.com/linux-manual-page-1-borg)
+- restic 0.17.0+ fix “Exclude irregular files from backups” (sockets, FIFOs, devices skipped) confirms prior irregular-file handling was a correctness fix - [Source](https://github.com/restic/restic/releases)
+- Kopia policy has explicit `--ignore-dir-errors` to tolerate unreadable/locked dirs during traversal, separate from ignore-rules; issue reports show `fuse` mounts (e.g. `seafile/fuse`) still `lstat`-probed during estimation even when ignored, motivating explicit excludes for FUSE/locked paths - [Source](https://github.com/kopia/kopia/issues/3334); [Source](https://kopia.io/docs/reference/command-line/common/policy-set/)
+- Docker storage (`/var/lib/docker`: `overlay2/`, `image/`, `buildkit/`, `volumes/`, `containers/`) is large, churn-heavy, overlay-mounted and often locked/inconsistent without daemon quiesce; community Arch/restic lists therefore exclude `/var/lib/docker/**` and `/var/lib/libvirt/**`, and Proxmox `vzdump --exclude-path` examples exclude `/var/lib/docker/` alongside `/run/`, `/dev/shm`, `/dev/fuse` - [Source](https://wiki.archlinux.org/title/Restic); [Source](https://blog.devops.dev/docker-cleanup-and-relocate-81d2dcd32b8a); [Source](https://forum.proxmox.com/threads/backup-and-exclude-path.125966)
+- Docker’s own backup guidance says file-copy of the VM disk (`Docker.raw`/`docker_data.vhdx`) or `/var/lib/docker` requires Docker fully stopped; otherwise use `docker save`/`docker pull` + volume dump/restore procedures - [Source](https://docs.docker.com/desktop/settings-and-maintenance/backup-and-restore.md)
+- Veeam (VM-image backup reference) automatically excludes VM log files to cut size/time, and supports excluding swap files, deleted-block (BitLooker) data, and per-disk/template excludes for the same size/correctness reasons - [Source](https://helpcenter.veeam.com/docs/vbr/userguide/data_exclusion.html)
+- Size-category examples: pacman cache `/var/cache/pacman/pkg/`, `/var/cache/*`, `/var/log/*`, `/var/tmp/*`, browser `Cache/GPUCache/ShaderCache`, `thunderbird/*/Cache`, `~/.thumbnails`, `~/.local/share/Trash`, Steam `steamapps`, `~/Downloads/*`, Dropbox/rclone replicas - all excluded for bloat, not correctness - [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653); [Source](https://wiki.archlinux.org/title/Restic)
+- Reproducibility-category examples: `node_modules/*`, `jspm_packages/*`, `web_modules/*`, `target/*`, `build/*`, `~/.m2/repository`, `~/.ivy2`, `~/.gradle`, `~/.coursier/cache`, `~/.sbt`, `~/.stack`, `__pycache__`, `*.pyc`, `~/.local/lib/python*/site-packages/**`, `~/.npm`, `~/.pkg-cache/`, `~/.sdkman/`, `.venv-py3` - explicitly listed as rebuildable caches/toolchains - [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653); webdev backup tooling states “node_modules is never included … rebuild with yarn/npm install” while always keeping `.env`, `yarn.lock`, `package-lock.json` - [Source](https://github.com/ICJIA/backup-webdev/blob/main/README.md)
 
 ### Inferences
 - Rule of thumb: if path is kernel-generated (`/proc`, `/sys`), socket/FIFO/device, actively memory-mapped/locked (browser `*-shm/-wal`, DB live files, running container layers, FUSE), or a mountpoint for another filesystem, exclusion is for correctness; if it is cache/log/trash/thumbnail/package-download, it is for size; if `npm install`/`pip install`/`cargo fetch`/`mvn` recreates it, it is for reproducibility.
@@ -56,18 +56,18 @@ Correctness excludes (virtual FS, sockets/FIFOs/devices, live DB/VM/container ba
 ## How do tools treat dot-directories (.cache, .git, .venv) and per-project gitignore?
 
 ### Takeaway
-Backup tools do not read `.gitignore` by default; they rely on explicit patterns, CACHEDIR.TAG/`--exclude-caches`, and `.kopiaignore`/policy rules — so `.cache` is usually globally excluded, `.git` is usually kept (small, valuable history) unless deliberately dropped, and `.venv`/`node_modules`/`target` must be explicitly excluded per-project or via `exclude-if-present` markers.
+Backup tools do not read `.gitignore` by default; they rely on explicit patterns, CACHEDIR.TAG/`--exclude-caches`, and `.kopiaignore`/policy rules - so `.cache` is usually globally excluded, `.git` is usually kept (small, valuable history) unless deliberately dropped, and `.venv`/`node_modules`/`target` must be explicitly excluded per-project or via `exclude-if-present` markers.
 
 ### Cited Findings
-- `~/.cache` is safe to drop (name indicates cached data; many tools exclude cache/trash by default); Arch forum explicitly approves excluding whole `~/.cache`, implemented via borg `--exclude-caches` + CACHEDIR.TAG spec — [Source](https://bbs.archlinux.org/viewtopic.php?id=231281)
-- restic `--exclude-caches` only skips dirs containing `CACHEDIR.TAG` (keeps the tag file); `--exclude-if-present foo` skips contents of any dir containing marker `foo` (e.g. `.nobackup`, `CACHEDIR.TAG`, custom sentinels) — [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html)
-- borg `--exclude-caches`/`--exclude-if-present`/`--keep-tag-files` semantics are identical (skip CACHEDIR.TAG dirs; optionally retain tag files) — [Source](https://borgbackup.readthedocs.io/en/1.0.5/usage.html)
-- Kopia `Ignore cache directories: true` is the default global policy (inherits unless overridden), plus per-directory `.kopiaignore` and policy ignore-rules using gitignore-like globs (`*.dat`, `/logs/*`, `tmp.db`, `**/logs/**`, `!` negation) — [Source](https://github.com/kopia/kopia/issues/3334); [Source](https://kopia.io/docs/advanced/kopiaignore/)
-- Kopia ignore-rule inheritance is surprising: defining child-path `add-ignore` can shadow global ignores (open issue: “Ignore rules not being inherited when child adds its own ignores”), so global `.cache`/`cache` rules must be repeated or merged carefully — [Source](https://github.com/kopia/kopia/issues/4155)
-- Developer-oriented backup/index defaults treat dot-dirs as noise to skip: Dank index defaults exclude `.git`, `.hg`, `.svn`, `.cache`, `.npm`, `.yarn`, `.venv`/`venv`, `.tox`, `.pytest_cache`, `__pycache__`, `.gradle`, `.m2`, `.cargo`, `.idea`, `.vscode`, `node_modules`, `target`, `dist/build/out` — [Source](https://danklinux.com/docs/1.4/danksearch/configuration); SmartBackup auto-skips `node_modules`, `venv`, `__pycache__`, `.git` for 10x speed/size — [Source](https://github.com/CodingWithMK/smartbackup_file-backup-automation)
-- Counter-practice for `.git`: default backup guidance keeps `.git` (history is irreplaceable, usually small vs `node_modules`); Backblaze-mac customization explicitly adds separate opt-in rules to drop `node_modules/` and `.git/` because neither is excluded by default — [Source](https://gist.github.com/nickcernis/bb4bd43a44efd73b87d857e29b1d5b96)
-- `tmexclude` watches the filesystem to continually re-apply Time-Machine exclusions for `node_modules`, `target`, etc., because new projects constantly recreate them — showing per-project `.gitignore` alone does not stop backup tools from capturing them — [Source](https://github.com/PhotonQuantum/tmexclude)
-- Kopia FAQ states ignored paths come from policy or `.kopiaignore` files, not from `.gitignore` — [Source](https://github.com/kopia/kopia/blob/master/site/content/docs/FAQs/_index.md)
+- `~/.cache` is safe to drop (name indicates cached data; many tools exclude cache/trash by default); Arch forum explicitly approves excluding whole `~/.cache`, implemented via borg `--exclude-caches` + CACHEDIR.TAG spec - [Source](https://bbs.archlinux.org/viewtopic.php?id=231281)
+- restic `--exclude-caches` only skips dirs containing `CACHEDIR.TAG` (keeps the tag file); `--exclude-if-present foo` skips contents of any dir containing marker `foo` (e.g. `.nobackup`, `CACHEDIR.TAG`, custom sentinels) - [Source](https://restic.readthedocs.io/en/v0.16.3/040_backup.html)
+- borg `--exclude-caches`/`--exclude-if-present`/`--keep-tag-files` semantics are identical (skip CACHEDIR.TAG dirs; optionally retain tag files) - [Source](https://borgbackup.readthedocs.io/en/1.0.5/usage.html)
+- Kopia `Ignore cache directories: true` is the default global policy (inherits unless overridden), plus per-directory `.kopiaignore` and policy ignore-rules using gitignore-like globs (`*.dat`, `/logs/*`, `tmp.db`, `**/logs/**`, `!` negation) - [Source](https://github.com/kopia/kopia/issues/3334); [Source](https://kopia.io/docs/advanced/kopiaignore/)
+- Kopia ignore-rule inheritance is surprising: defining child-path `add-ignore` can shadow global ignores (open issue: “Ignore rules not being inherited when child adds its own ignores”), so global `.cache`/`cache` rules must be repeated or merged carefully - [Source](https://github.com/kopia/kopia/issues/4155)
+- Developer-oriented backup/index defaults treat dot-dirs as noise to skip: Dank index defaults exclude `.git`, `.hg`, `.svn`, `.cache`, `.npm`, `.yarn`, `.venv`/`venv`, `.tox`, `.pytest_cache`, `__pycache__`, `.gradle`, `.m2`, `.cargo`, `.idea`, `.vscode`, `node_modules`, `target`, `dist/build/out` - [Source](https://danklinux.com/docs/1.4/danksearch/configuration); SmartBackup auto-skips `node_modules`, `venv`, `__pycache__`, `.git` for 10x speed/size - [Source](https://github.com/CodingWithMK/smartbackup_file-backup-automation)
+- Counter-practice for `.git`: default backup guidance keeps `.git` (history is irreplaceable, usually small vs `node_modules`); Backblaze-mac customization explicitly adds separate opt-in rules to drop `node_modules/` and `.git/` because neither is excluded by default - [Source](https://gist.github.com/nickcernis/bb4bd43a44efd73b87d857e29b1d5b96)
+- `tmexclude` watches the filesystem to continually re-apply Time-Machine exclusions for `node_modules`, `target`, etc., because new projects constantly recreate them - showing per-project `.gitignore` alone does not stop backup tools from capturing them - [Source](https://github.com/PhotonQuantum/tmexclude)
+- Kopia FAQ states ignored paths come from policy or `.kopiaignore` files, not from `.gitignore` - [Source](https://github.com/kopia/kopia/blob/master/site/content/docs/FAQs/_index.md)
 
 ### Inferences
 - Best practice: global-exclude `~/.cache`, `~/.npm`, `~/.cargo`, `~/.mozilla/.../Cache`, `Trash`, `thumbnails`; per-source exclude `node_modules/`, `.venv/venv/`, `__pycache__/`, `target/`, `build/dist/`, optionally via `exclude-if-present` markers (e.g. drop marker file `CACHEDIR.TAG` or `.nobackup` in roots) rather than hoping tools honor `.gitignore`.
@@ -75,7 +75,7 @@ Backup tools do not read `.gitignore` by default; they rely on explicit patterns
 
 ### Gaps
 - No evidence found that restic/borg/Kopia natively ingest `.gitignore` in 2026 builds; if any wrapper does, it was not in primary docs searched.
-- Best handling of `.venv` that contains non-recreatable local edits (pip `-e` installs, manual patches) is unresolved — exclusion assumes venv is truly disposable.
+- Best handling of `.venv` that contains non-recreatable local edits (pip `-e` installs, manual patches) is unresolved - exclusion assumes venv is truly disposable.
 
 ## What breaks when people back up dependency/build trees anyway?
 
@@ -83,15 +83,15 @@ Backup tools do not read `.gitignore` by default; they rely on explicit patterns
 Backing up `node_modules`, `venv`, `target/build`, `site-packages`, and container/VM stores inflates size, scan time, and snapshot churn, defeats deduplication (thousands of tiny files, hardlinks, changing mtimes/hashes), risks unrestorable or inconsistent restores (native bindings, symlinks, absolute paths, locked DB/pages), and can leak secrets or break privacy.
 
 ### Cited Findings
-- SmartBackup’s premise is that naive `Documents` backup “waits hours because of massive node_modules or venvs”; skipping them yields ~10x faster/smaller backups, with incremental manifest+hash tracking otherwise churning on every dependency touch — [Source](https://github.com/CodingWithMK/smartbackup_file-backup-automation)
-- Webdev backup defaults state `node_modules` excluded from full/incremental/differential/quick backups to save space; restore requires `yarn/npm install`, while lockfiles+`.env` are retained to reconstruct — [Source](https://github.com/ICJIA/backup-webdev/blob/main/README.md)
-- Rust/JS `target/` and `node_modules/` are the canonical Time-Machine-bloat examples requiring a watcher (`tmexclude`) because they reappear per `npm install`/`cargo build` and would otherwise be re-captured every snapshot — [Source](https://github.com/PhotonQuantum/tmexclude)
-- Python `site-packages` (`~/.local/lib/python*/site-packages/**`), `__pycache__`, `*.pyc`, and per-project `.venv` are listed alongside `node_modules` in Arch/restic excludes as regenerable interpreter artifacts — [Source](https://wiki.archlinux.org/title/Restic)
-- `pnpm clean/purge` exists precisely because `node_modules` contents (plus virtual-store) are disposable and safely removable via Node-aware deletion handling junctions correctly — [Source](https://pnpm.io/next/cli/clean)
-- File-level backup of `/var/lib/docker` captures `overlay2` diffs, `image/`, `buildkit` cache (often 10s of GB, e.g. 20GB build cache reclaimable via `docker system prune`) that are host-specific, layer-duplicated, and unrestorable by plain copy while daemon runs; correct path is `docker save/load`, registry push/pull, and volume dumps — [Source](https://blog.devops.dev/docker-cleanup-and-relocate-81d2dcd32b8a); [Source](https://docs.docker.com/desktop/settings-and-maintenance/backup-and-restore.md)
-- Locked/inconsistent captures produce backup errors or silent corruption: Kopia `seafile/fuse` example shows even explicitly ignored FUSE paths trigger `lstat` errors/failed snapshots unless fully avoided; `--ignore-dir-errors` merely masks valid errors — [Source](https://github.com/kopia/kopia/issues/3334)
-- Browser/IDE heavy dirs (`.config/Code/`, `.vscode/extensions/`, `.config/coc/extensions/`, `.mozilla/.../extensions`, `sonarlint/plugins`, `.coursier/cache`, `.m2/repository`) are called out as “heavy JARs/caches” that bloat snapshots while preferences should sync via cloud/accounts instead — [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653)
-- `~/.cache`-style trees and `node_modules` contain huge small-file counts that slow file-walk, inflate metadata/index, and reduce dedup/compression efficiency vs backing up manifests/lockfiles — rationale behind Dank/SmartBackup default-skips — [Source](https://danklinux.com/docs/1.4/danksearch/configuration); [Source](https://github.com/CodingWithMK/smartbackup_file-backup-automation)
+- SmartBackup’s premise is that naive `Documents` backup “waits hours because of massive node_modules or venvs”; skipping them yields ~10x faster/smaller backups, with incremental manifest+hash tracking otherwise churning on every dependency touch - [Source](https://github.com/CodingWithMK/smartbackup_file-backup-automation)
+- Webdev backup defaults state `node_modules` excluded from full/incremental/differential/quick backups to save space; restore requires `yarn/npm install`, while lockfiles+`.env` are retained to reconstruct - [Source](https://github.com/ICJIA/backup-webdev/blob/main/README.md)
+- Rust/JS `target/` and `node_modules/` are the canonical Time-Machine-bloat examples requiring a watcher (`tmexclude`) because they reappear per `npm install`/`cargo build` and would otherwise be re-captured every snapshot - [Source](https://github.com/PhotonQuantum/tmexclude)
+- Python `site-packages` (`~/.local/lib/python*/site-packages/**`), `__pycache__`, `*.pyc`, and per-project `.venv` are listed alongside `node_modules` in Arch/restic excludes as regenerable interpreter artifacts - [Source](https://wiki.archlinux.org/title/Restic)
+- `pnpm clean/purge` exists precisely because `node_modules` contents (plus virtual-store) are disposable and safely removable via Node-aware deletion handling junctions correctly - [Source](https://pnpm.io/next/cli/clean)
+- File-level backup of `/var/lib/docker` captures `overlay2` diffs, `image/`, `buildkit` cache (often 10s of GB, e.g. 20GB build cache reclaimable via `docker system prune`) that are host-specific, layer-duplicated, and unrestorable by plain copy while daemon runs; correct path is `docker save/load`, registry push/pull, and volume dumps - [Source](https://blog.devops.dev/docker-cleanup-and-relocate-81d2dcd32b8a); [Source](https://docs.docker.com/desktop/settings-and-maintenance/backup-and-restore.md)
+- Locked/inconsistent captures produce backup errors or silent corruption: Kopia `seafile/fuse` example shows even explicitly ignored FUSE paths trigger `lstat` errors/failed snapshots unless fully avoided; `--ignore-dir-errors` merely masks valid errors - [Source](https://github.com/kopia/kopia/issues/3334)
+- Browser/IDE heavy dirs (`.config/Code/`, `.vscode/extensions/`, `.config/coc/extensions/`, `.mozilla/.../extensions`, `sonarlint/plugins`, `.coursier/cache`, `.m2/repository`) are called out as “heavy JARs/caches” that bloat snapshots while preferences should sync via cloud/accounts instead - [Source](https://forum.restic.net/t/what-gnu-linux-directories-to-exclude-from-backups/6653)
+- `~/.cache`-style trees and `node_modules` contain huge small-file counts that slow file-walk, inflate metadata/index, and reduce dedup/compression efficiency vs backing up manifests/lockfiles - rationale behind Dank/SmartBackup default-skips - [Source](https://danklinux.com/docs/1.4/danksearch/configuration); [Source](https://github.com/CodingWithMK/smartbackup_file-backup-automation)
 
 ### Inferences
 - Failure modes to expect if you include them anyway: (1) backup never finishes or blows retention/bandwidth; (2) every `npm/cargo/pip` run creates a large new snapshot despite no user-data change; (3) restore on another machine/arch breaks native modules/symlinks/permissions; (4) secrets in `.npmrc`, venv activate scripts, or vendored `.env` leak into long-retained snapshots.