versioning/research_notes/Backup include exclude rules/never-include.md

Never Include in Linux Workstation/Home-Server Backups

What are the canonical exclude directories and file patterns across backup tools (give concrete lists)?

Takeaway

Canonical Linux file-backup excludes converge on: virtual/kernel filesystems (/proc, /sys, /dev, /run), ephemeral (/tmp, /var/tmp, /var/run, /var/lock), mounts/media (/mnt, /media, /lost+found, /swapfile), package/cache/logs (/var/cache/*, /var/log/*, pacman cache), per-user caches/trash/thumbnails, and repro data (build/dependency dirs, VM/container storage) - implemented as explicit exclude-files in restic/borg plus --exclude-caches/--one-file-system and Kopia policies/.kopiaignore.

Cited Findings

• restic has no built-in default excludes; user supplies --exclude, --iexclude, --exclude-file, --exclude-if-present foo, --exclude-caches (CACHEDIR.TAG dirs), --exclude-larger-than, --exclude-cloud-files, plus -x/--one-file-system to stay on one filesystem - Source; also documented that excludes do not apply to explicitly-passed backup-source paths - Source
• restic pattern syntax is Go filepath.Match + ** for crossing /, matched on complete path components (foo matches /dir1/foo/... but not /dir/foobar), trailing / ignored, leading / anchors at root - Source
• borg create supports -e/--exclude PATTERN, --exclude-from FILE, --exclude-caches (CACHEDIR.TAG), --exclude-if-present NAME, --keep-tag-files, -x/--one-file-system, and respects nodump flag; recommended test is borg create --list --dry-run - Source; pattern styles are fnmatch (default for --exclude), sh:, re:, path prefix/full-match, with trailing-/ meaning “keep dir, skip contents” - Source
• borg quickstart automation example excludes --exclude-caches --exclude 'home/*/.cache/*' --exclude 'var/tmp/*' when backing up /etc-style roots - Source
• Kopia has no static exclude-file; it uses policy Ignore rules + Read ignore rules from files (default .kopiaignore) + Ignore cache directories: true (default true, inherits global→host→user→path) + Scan one filesystem only: true - Source; kopia policy set flags include --add-ignore, --add-dot-ignore, --ignore-cache-dirs [true|false|inherit], --ignore-dir-errors, --one-file-system-equivalent Scan one filesystem only - Source
• Kopia’s CACHEDIR.TAG-equivalent (--ignore-cache-dirs, default true) was added to mirror restic --exclude-caches, defaulting to ignore caches via policy - Source
• ArchWiki restic full-system /etc/restic/excludes.txt canonical list: /data/**, /dev/**, /home/*/**/*.pyc, /home/*/**/__pycache__/**, /home/*/**/node_modules/**, /home/*/.cache/**, /home/*/.local/lib/python*/site-packages/**, /home/*/.mozilla/firefox/*/Cache/**, /lost+found/**, /media/**, /mnt/**, /proc/**, /root/**, /run/**, /swapfile, /sys/**, /tmp/**, /var/cache/**, /var/cache/pacman/pkg/**, /var/lib/docker/**, /var/lib/libvirt/**, /var/lock/**, /var/log/**, /var/run/**, noting --one-file-system can replace /proc//run//mnt entries while preserving mountpoints - Source
• Debian restic-forum practitioner system list: /media, /mnt, /cdrom, /proc, /sys, /dev, /run, /tmp, /var/run, /var/lock, /var/tmp, /lost+found, /swapfile, /var/cache/restic, Steam .../Steam/steamapps, plus home temp items .gvfs, .local/share/gvfs-metadata, .local/share/Trash, .cache, .dbus, .xsession-errors, .Xauthority, .gksu.lock, .local/share/flatpak/appstream - Source
• Extended community restic list adds: nobackup/NoBackup, Downloads/*, VirtualBox VMs/, Dropbox/*, rclone/, snap/, .var/**/cache/, .local/share/docker/, .local/share/JetBrains/, .vscode/extensions/, .config/Code/, .m2/repository, target/*, build/*, node_modules/*, jspm_packages/*, web_modules/*, .npm, .coursier/cache, .sbt, .stack, .sdkman, .jdks, .eclipse, .venv-py3, .wine, .android, Android/Sdk, .gradle, .adobe, .macromedia, .thumbnails, .thunderbird/*/Cache, .mozilla/firefox/*/Cache|storage|minidumps|*.sqlite*, .config/**/Cache|GPUCache|ShaderCache, .config/chromium/Default/...History|Favicons|Storage|Cache, .local/share/baloo|zeitgeist|akonadi, .gnupg/rnd|random_seed|*.lock, .pulse*, .java/deployment/cache, .dropbox* - Source
• ArchWiki tar full-system guidance: exclude /opt/backup/arch-full* (own backups), /tmp/*, /var/cache/pacman/pkg/, boot from LiveCD with plain chroot (not arch-chroot) to avoid capturing tempfs/memory - Source
• Guide summary: when backing up /, explicitly exclude /proc, /sys, /dev, /run, /tmp and bind/overlay mounts; when backing up only /home+/etc both borg/restic skip virtual FS by default - Source
• Kopia .kopiaignore syntax: one rule/line, # comment, * (any chars), ** (any dirs), ?, [0-9]/[a-z]/[A-Z]/[abc], leading / roots rule, ! negates; policy can point at alternate ignore-files - Source
• Time-Machine-ecosystem analogue (Arq honoring Apple TM exclusions): .Trash, Library/Caches, Library/Logs, Library/Mail/.../Envelope Index*, Library/Safari/WebpageIcons.db, Library/Saved Application State, Library/iTunes/iPad Software Updates - Source

Inferences

• There is no single vendor “default exclude file” for restic/borg on Linux; the de-facto standard is the ArchWiki + forum lists above plus --exclude-caches and --one-file-system.
• Kopia inverts the model (opt-out caches by default + per-dir .kopiaignore) vs restic/borg (opt-in excludes), so migrated exclude-lists must be translated, not copied verbatim.

Gaps

• No authoritative 2023–2026 duplicity built-in exclusion list found for Linux workstation context; duplicity relies on --exclude user patterns rather than published defaults.
• Exact current Apple StdExclusions.plist contents not verified for Linux relevance; only Arq’s documented subset was citable.

Which exclusions exist for correctness (locked files, sockets) vs size vs reproducibility?

Takeaway

Correctness excludes (virtual FS, sockets/FIFOs/devices, live DB/VM/container backing stores, cloud-online-only stubs) prevent hangs, errors, or unrestorable data; size excludes (caches, trash, thumbnails, logs, browser profiles, package caches, media/Steam) prevent bloat; reproducibility excludes (dependency/build trees) are safe to drop because lockfiles+manifests rebuild them.

Cited Findings

• /proc and /sys are virtual/kernel pseudo-filesystems (proc exposes live kernel/process state, proc_sys exposes tunable sysctls; size often reported 0, constantly changing) - backing them up captures no stable data - Source; Source
• restic forum guidance: use --exclude-caches to auto-exclude CACHEDIR.TAG-marked dirs for “non permanent system files”; full-system threads treat only a short path list as needing exclusion - Source
• borg by default does not open/read block/char devices or FIFOs; --read-special is required to force reading them as regular files (and to follow symlinks to them) - Source
• restic 0.17.0+ fix “Exclude irregular files from backups” (sockets, FIFOs, devices skipped) confirms prior irregular-file handling was a correctness fix - Source
• Kopia policy has explicit --ignore-dir-errors to tolerate unreadable/locked dirs during traversal, separate from ignore-rules; issue reports show fuse mounts (e.g. seafile/fuse) still lstat-probed during estimation even when ignored, motivating explicit excludes for FUSE/locked paths - Source; Source
• Docker storage (/var/lib/docker: overlay2/, image/, buildkit/, volumes/, containers/) is large, churn-heavy, overlay-mounted and often locked/inconsistent without daemon quiesce; community Arch/restic lists therefore exclude /var/lib/docker/** and /var/lib/libvirt/**, and Proxmox vzdump --exclude-path examples exclude /var/lib/docker/ alongside /run/, /dev/shm, /dev/fuse - Source; Source; Source
• Docker’s own backup guidance says file-copy of the VM disk (Docker.raw/docker_data.vhdx) or /var/lib/docker requires Docker fully stopped; otherwise use docker save/docker pull + volume dump/restore procedures - Source
• Veeam (VM-image backup reference) automatically excludes VM log files to cut size/time, and supports excluding swap files, deleted-block (BitLooker) data, and per-disk/template excludes for the same size/correctness reasons - Source
• Size-category examples: pacman cache /var/cache/pacman/pkg/, /var/cache/*, /var/log/*, /var/tmp/*, browser Cache/GPUCache/ShaderCache, thunderbird/*/Cache, ~/.thumbnails, ~/.local/share/Trash, Steam steamapps, ~/Downloads/*, Dropbox/rclone replicas - all excluded for bloat, not correctness - Source; Source
• Reproducibility-category examples: node_modules/*, jspm_packages/*, web_modules/*, target/*, build/*, ~/.m2/repository, ~/.ivy2, ~/.gradle, ~/.coursier/cache, ~/.sbt, ~/.stack, __pycache__, *.pyc, ~/.local/lib/python*/site-packages/**, ~/.npm, ~/.pkg-cache/, ~/.sdkman/, .venv-py3 - explicitly listed as rebuildable caches/toolchains - Source; webdev backup tooling states “node_modules is never included … rebuild with yarn/npm install” while always keeping .env, yarn.lock, package-lock.json - Source

Inferences

• Rule of thumb: if path is kernel-generated (/proc, /sys), socket/FIFO/device, actively memory-mapped/locked (browser *-shm/-wal, DB live files, running container layers, FUSE), or a mountpoint for another filesystem, exclusion is for correctness; if it is cache/log/trash/thumbnail/package-download, it is for size; if npm install/pip install/cargo fetch/mvn recreates it, it is for reproducibility.
• Live DB/VM/container volumes need application-consistent dump/snapshot (e.g. docker save, DB dump, hypervisor snapshot), not file-level copy of the running store.

Gaps

• No citable 2024–2026 benchmark found quantifying restore-corruption rate when file-copying live SQLite/WAL or overlay2 without quiesce; guidance remains consensus-based.
• restic/borg behavior on Windows/macOS cloud-online-only stubs (--exclude-cloud-files) verified in flags but Linux relevance is none; no Linux equivalent stub mechanism found.

How do tools treat dot-directories (.cache, .git, .venv) and per-project gitignore?

Takeaway

Backup tools do not read .gitignore by default; they rely on explicit patterns, CACHEDIR.TAG/--exclude-caches, and .kopiaignore/policy rules - so .cache is usually globally excluded, .git is usually kept (small, valuable history) unless deliberately dropped, and .venv/node_modules/target must be explicitly excluded per-project or via exclude-if-present markers.

Cited Findings

• ~/.cache is safe to drop (name indicates cached data; many tools exclude cache/trash by default); Arch forum explicitly approves excluding whole ~/.cache, implemented via borg --exclude-caches + CACHEDIR.TAG spec - Source
• restic --exclude-caches only skips dirs containing CACHEDIR.TAG (keeps the tag file); --exclude-if-present foo skips contents of any dir containing marker foo (e.g. .nobackup, CACHEDIR.TAG, custom sentinels) - Source
• borg --exclude-caches/--exclude-if-present/--keep-tag-files semantics are identical (skip CACHEDIR.TAG dirs; optionally retain tag files) - Source
• Kopia Ignore cache directories: true is the default global policy (inherits unless overridden), plus per-directory .kopiaignore and policy ignore-rules using gitignore-like globs (*.dat, /logs/*, tmp.db, **/logs/**, ! negation) - Source; Source
• Kopia ignore-rule inheritance is surprising: defining child-path add-ignore can shadow global ignores (open issue: “Ignore rules not being inherited when child adds its own ignores”), so global .cache/cache rules must be repeated or merged carefully - Source
• Developer-oriented backup/index defaults treat dot-dirs as noise to skip: Dank index defaults exclude .git, .hg, .svn, .cache, .npm, .yarn, .venv/venv, .tox, .pytest_cache, __pycache__, .gradle, .m2, .cargo, .idea, .vscode, node_modules, target, dist/build/out - Source; SmartBackup auto-skips node_modules, venv, __pycache__, .git for 10x speed/size - Source
• Counter-practice for .git: default backup guidance keeps .git (history is irreplaceable, usually small vs node_modules); Backblaze-mac customization explicitly adds separate opt-in rules to drop node_modules/ and .git/ because neither is excluded by default - Source
• tmexclude watches the filesystem to continually re-apply Time-Machine exclusions for node_modules, target, etc., because new projects constantly recreate them - showing per-project .gitignore alone does not stop backup tools from capturing them - Source
• Kopia FAQ states ignored paths come from policy or .kopiaignore files, not from .gitignore - Source

Inferences

• Best practice: global-exclude ~/.cache, ~/.npm, ~/.cargo, ~/.mozilla/.../Cache, Trash, thumbnails; per-source exclude node_modules/, .venv/venv/, __pycache__/, target/, build/dist/, optionally via exclude-if-present markers (e.g. drop marker file CACHEDIR.TAG or .nobackup in roots) rather than hoping tools honor .gitignore.
• Keep .git/ and lockfiles (package-lock.json, yarn.lock, Cargo.lock, requirements*.txt) while dropping the fetched trees; keep .env only deliberately (secret-sprawl risk vs rebuild need).

Gaps

• No evidence found that restic/borg/Kopia natively ingest .gitignore in 2026 builds; if any wrapper does, it was not in primary docs searched.
• Best handling of .venv that contains non-recreatable local edits (pip -e installs, manual patches) is unresolved - exclusion assumes venv is truly disposable.

What breaks when people back up dependency/build trees anyway?

Takeaway

Backing up node_modules, venv, target/build, site-packages, and container/VM stores inflates size, scan time, and snapshot churn, defeats deduplication (thousands of tiny files, hardlinks, changing mtimes/hashes), risks unrestorable or inconsistent restores (native bindings, symlinks, absolute paths, locked DB/pages), and can leak secrets or break privacy.

Cited Findings

• SmartBackup’s premise is that naive Documents backup “waits hours because of massive node_modules or venvs”; skipping them yields ~10x faster/smaller backups, with incremental manifest+hash tracking otherwise churning on every dependency touch - Source
• Webdev backup defaults state node_modules excluded from full/incremental/differential/quick backups to save space; restore requires yarn/npm install, while lockfiles+.env are retained to reconstruct - Source
• Rust/JS target/ and node_modules/ are the canonical Time-Machine-bloat examples requiring a watcher (tmexclude) because they reappear per npm install/cargo build and would otherwise be re-captured every snapshot - Source
• Python site-packages (~/.local/lib/python*/site-packages/**), __pycache__, *.pyc, and per-project .venv are listed alongside node_modules in Arch/restic excludes as regenerable interpreter artifacts - Source
• pnpm clean/purge exists precisely because node_modules contents (plus virtual-store) are disposable and safely removable via Node-aware deletion handling junctions correctly - Source
• File-level backup of /var/lib/docker captures overlay2 diffs, image/, buildkit cache (often 10s of GB, e.g. 20GB build cache reclaimable via docker system prune) that are host-specific, layer-duplicated, and unrestorable by plain copy while daemon runs; correct path is docker save/load, registry push/pull, and volume dumps - Source; Source
• Locked/inconsistent captures produce backup errors or silent corruption: Kopia seafile/fuse example shows even explicitly ignored FUSE paths trigger lstat errors/failed snapshots unless fully avoided; --ignore-dir-errors merely masks valid errors - Source
• Browser/IDE heavy dirs (.config/Code/, .vscode/extensions/, .config/coc/extensions/, .mozilla/.../extensions, sonarlint/plugins, .coursier/cache, .m2/repository) are called out as “heavy JARs/caches” that bloat snapshots while preferences should sync via cloud/accounts instead - Source
• ~/.cache-style trees and node_modules contain huge small-file counts that slow file-walk, inflate metadata/index, and reduce dedup/compression efficiency vs backing up manifests/lockfiles - rationale behind Dank/SmartBackup default-skips - Source; Source

Inferences

• Failure modes to expect if you include them anyway: (1) backup never finishes or blows retention/bandwidth; (2) every npm/cargo/pip run creates a large new snapshot despite no user-data change; (3) restore on another machine/arch breaks native modules/symlinks/permissions; (4) secrets in .npmrc, venv activate scripts, or vendored .env leak into long-retained snapshots.
• Mitigation if you must include one tree (e.g. offline/air-gapped rebuild): snapshot it to a separate low-frequency archive/tag, not the daily home backup, and document the toolchain version needed to use it.

Gaps

• No controlled measurement found of dedup-ratio collapse or file-count slowdown specifically for node_modules vs target/ vs .venv under borg/restic/Kopia 2026 versions; claims remain qualitative (“10x”, “hours”).
• Privacy impact of vendored credentials inside dependency trees (e.g. .pypirc, npm tokens in ~/.npm/_cacache) is asserted in practice guides but no incident stat was found.