versioning/research_notes/Backup include exclude rules/must-include.md

Must-Include Data for Linux Workstation / Home Server Backups

Which paths do guides universally say to include (~/Documents, ~/Pictures, ~/.config, project dirs, etc.)?

Takeaway

Guides converge on: back up all of /home (or whole $HOME) plus /etc, /root, /var (selectively) and /opt; on workstations this means XDG user dirs (~/Documents, ~/Pictures, ~/Videos, ~/Music, ~/Downloads), project dirs (~/projects, ~/src), dotfiles/~/.config/~/.local/share, ~/.ssh, browser profiles, mail, and package lists — never a bare ~/Documents-only backup.

Cited Findings

• Red Hat guide lists as must-back-up: /etc (system config, users/groups, networking, app configs), /home (all user data/downloads/documents/pictures under username), /root (admin scripts/notes/configs), /var shared/corporate data — and one dir not to back up (implied transient/cache) — Source
• Ubuntu official help: copy personal files/settings usually in Home folder; if room, back up entire Home folder with exceptions (cache/thrash-type excludes detailed on linked page) — Source
• PCWorld Linux backup roundup: as a rule a regular backup of the home directories is sufficient; rsync -avP $HOME to USB disk given as sufficient home backup — Source
• OSTechNix 2026 reinstall checklist: back up more than ~/Documents; must include personal files + ~/.local/share (app data, game saves, Flatpak/Snap data), ~/.config (app settings), browser profiles (Firefox ~/.mozilla or new XDG ~/.config/mozilla + ~/.cache/mozilla + ~/.local/share/mozilla, check about:support; Chrome/Chromium), SSH/GPG keys, dotfiles, app-specific data, installed package lists including Flatpak/Snap, system-level /etc, NetworkManager system-connections, cron/scheduled tasks; trap 1 is only backing up ~/Documents ~/Pictures ~/Downloads and missing ~/projects, VM images, second drives; trap 2 is forgetting hidden XDG data — Source
• OSTechNix: single rsync of whole $HOME captures dotfiles, browser profiles, SSH keys, app settings since all live under $HOME — Source
• Arch forum classic tar set: tar zcvfp arch-system.gz /etc /boot /root + per-user /home/user1 + /var --exclude /var/cache/pacman/pkg — Source
• Production borg example backs up /home /etc /var/www /var/backups /opt /root together with --exclude-from file — Source
• Ubuntu Ask-Ubuntu full-system tar pattern excludes virtual/external dev mnt proc sys (and squashfs variant excludes home media dev run mnt proc sys tmp, then backs up /home separately excluding cloud mirrors like Dropbox GoogleDrive) — Source
• Linux Mint forum consensus: Timeshift = OS/system restore points, does nothing for data in /home; need separate file-level tool (BackInTime, FreeFileSync, Foxclone/Rescuezilla/Clonezilla for images) for Documents/Music/Pictures — Source
• Dotfiles canon: ~/.bashrc, ~/.bash_profile, ~/.zshrc, ~/.vimrc, ~/.gitconfig, ~/.ssh/config, ~/.tmux.conf, ~/.config/ XDG dir; secrets in ~/.netrc, ~/.aws/credentials, ~/.ssh/, history ~/.bash_history must be treated as sensitive / kept out of public repos — Source
• Dotfiles restore guides list as backup-worthy: .ssh/config, .gnupg/pubring.kbx, .gnupg/trustdb.gpg, plus stow-managed zsh/git/vim/tmux and XDG select dirs — Source
• Keeply Linux tool scope explicitly: dotfiles, .config dirs, browser profiles, app settings, optionally SSH keys, emails — Source

Inferences

• Universal must-include = whole /home/<user> + /etc + package list; /root and /var/lib-style app data added for home-server role.
• ~/Documents-only backup is consistently called out as insufficient; project dirs and XDG hidden dirs hold irreplaceable state.
• SSH private keys and Wi-Fi psk= files are must-include but must be encrypted, not pushed to public dotfiles repos.

Gaps

• No single 2023–2026 primary guide found quantifying mail (~/Mail, Thunderbird ~/.thunderbird) include rates; only secondary tool scope mentions emails.
• No reliable source found prescribing exact treatment of ~/.cache vs ~/.local/share boundary for Flatpak/Snap beyond OSTechNix summary.

How should small databases (sqlite), archives (.zip/.tar), and images be treated — raw files or dumps?

Takeaway

Small static files (photos/images, .zip/.tar archives) are backed up as raw files and deduplicate well; live database files (sqlite, postgres/mysql) must be dumped (VACUUM INTO, sqlite3 .backup/backup API, pg_dump/pg_dumpall/mysqldump) before file backup — raw copy of a live DB is unsafe.

Cited Findings

• SQLite docs: safe live-copy methods are sqlite3_rsync (3.47.0+, 2024-10-21, bandwidth-efficient over SSH), VACUUM INTO filename, or backup API; plain file copy only safe with no transactions in progress, and if prior write failed must copy -journal/-wal together — Source
• SQLite forum (Borg user, WAL-mode pooled connections): cannot depend on perfect backup from open/changing DB because backup doesn't snapshot DB + journal in same instant; recommendation: close writers or backup API copy then back up the copy; rm -f mydb.sqlite3.backup; sqlite3 mydb.sqlite3 "VACUUM INTO 'mydb.sqlite3.backup'" added to backup script — Source
• Experiment on SQLite 3.50.6 WAL DB (1000 rows, 500 in -wal): cp app.db backup_cp.db yielded 500 rows, integrity_check still ok — stale copy passes checks silently; copying app.db + app.db-wal + app.db-shm together preserved 1000 rows; .backup (Online Backup API) copies page-by-page including WAL and restarts if source writes mid-copy — Source
• fs.copyFile() on WAL DB copies only .db while -wal/-shm still written → instant SQLITE_CORRUPT; 7/7 rotating backups corrupted; fix is better-sqlite3 .backup() which handles all three files atomically; test restores — Source
• Postgres docs: pg_dump dbname > dumpfile generates SQL commands to recreate DB; pg_dumpall > dumpfile preserves cluster-wide roles/tablespaces, restore via psql -f dumpfile postgres requiring superuser; file-level/WAL archiving is version-specific whereas dumps reload into newer versions — Source
• Postgres professional docs: pg_dump can emit text or archive formats for parallelism/fine-grained pg_restore control — Source
• OSTechNix: don't copy raw data dir; use each DB's dump tool e.g. mysqldump -u <user> -p <db> > backup.sql — Source
• Borg quickstart warns: snapshot filesystems/volumes (LVM/ZFS useful), dump databases or stop DB servers, shut down VMs/containers before backing up disk images/volumes — Source
• Restic docs example: plan.txt change adds KiB, archive.tar.gz change re-adds MiB — illustrates large monolithic archives re-chunk poorly vs small files; still backed up as raw files — Source

Inferences

• Photos/images/archives: raw-file backup is correct; no dump needed; chunked dedup tools (restic/borg/kopia) handle them but modified tar/zip re-stores large chunks.
• SQLite/postgres/mysql live files: dump-then-backup is mandatory; raw file backup alone is at best crash-consistent, at worst silently stale/corrupt.

Gaps

• No primary 2023–2026 benchmark found for dedup efficiency on .zip/.tar vs extracted trees under restic/borg/kopia; only illustrative restic example.
• No reliable source prescribing whether to keep both live sqlite file + dump in same backup set or dump-only.

What do tools like restic, borg, kopia, Time Machine include by default?

Takeaway

None auto-selects ~/Documents/~/.config; all default to exactly what paths you pass (no implicit includes) and provide opt-in excludes (--exclude*, --patterns-from, .kopiaignore/policy, Time Machine StdExclusions + user list) for caches, build artifacts, and system pseudo-filesystems.

Cited Findings

• Restic backup [FILE/DIR]... creates snapshot of exactly given args; no default source; exclude options: --exclude/--iexclude, --exclude-file, --exclude-caches (CACHEDIR.TAG), --exclude-if-present foo, --exclude-larger-than, --exclude-cloud-files (Win/macOS OneDrive/iCloud only); excludes don't apply to explicitly passed file path itself, only contents under dirs — Source
• Restic excludes use Go filepath.Match against full path, gitignore-like: once dir excluded can't re-include inside; example backs up selection inside $HOME — Source
• Restic handles --one-file-system/-x to not cross filesystem boundaries/subvolumes; saves/restores ACLs/xattrs; sparse holes deduped/compressed not stored explicitly — Source
• NixOS restic module example: paths = ["/home"] with exclude = ["/home/*/.cache" ".git"] pattern as conventional home selection — Source
• Borg create repo::archive ~/Documents, ~/Documents ~/src --exclude '*.pyc', /home --exclude thumbnail regex, / --one-file-system are docs examples — user picks roots; no default root — Source
• Borg patterns: fnmatch default for --exclude, shell-style for --pattern; sh:**/steamapps/common/**, sh:home/user/.cache/**, trailing-slash some/path/ keeps dir not contents vs no-slash excludes both; --exclude-from, --patterns-from, --exclude-caches (CACHEDIR.TAG), --exclude-if-present, --keep-tag-files, --one-file-system, nodump flag respected — Source
• Kopia: no default source; snapshots what you snapshot create; ignores via .kopiaignore (default file), global/per-source policy --add-ignore/--add-dot-ignore, --ignore-cache-dirs true (inherited global), --ignore-dir-errors/--ignore-file-errors, --one-file-system, never/only-compress lists; before/after folder/root actions for dumps/snapshots with timeout/modes — Source
• Kopia ignore syntax: # comment, ! negate, */**/?/[0-9a-zA-Z]/[abc], leading / root-only; examples *.dat, /logs/*, tmp.db, **/logs/**; must test — incomplete snapshots possible — Source
• Time Machine default backs up everything except system files/apps from macOS install, caches, StdExclusions (/System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist), backup disk itself; user adds via Options → Exclude; excluded items still in local snapshots; damaged system requires macOS reinstall + Migration Assistant — Source
• Time Machine tmutil isexcluded/addexclusion/removeexclusion, sticky vs fixed-path (-p) exclusions semantics — Source
• ArchWiki System backup: no default set; methods are btrfs/LVM snapshots, rsync, tar, SquashFS (no ACLs); recommends 3-2-1, regular integrity + restore tests; automation via systemd timer/cron with least-privilege CAP_DAC_READ_SEARCH example — Source

Inferences

• Versioning tools scope = explicit roots + exclude list; portable Linux default is effectively /home + /etc + dumps minus *.cache/CACHEDIR.TAG, node_modules/target/.git, VM images while running.
• Kopia ignore-cache-dirs=true and restic/borg --exclude-caches are the closest to a built-in default exclude.

Gaps

• No primary source found stating Kopia ships global ignore rules beyond ignore-cache-dirs; default policy contents not fully enumerated in fetched docs.
• Time Machine StdExclusions full list not fetched (plist path only); Linux analogue must be inferred.

Any special handling for live database files (WAL mode, locking, dump-then-backup)?

Takeaway

Never file-copy a live DB; quiesce (close writers/stop server), filesystem/LVM/ZFS snapshot, or dump (sqlite3 backup API/VACUUM INTO, pg_dumpall, mysqldump); for SQLite WAL must keep .db+-wal+-shm/-journal together, never delete hot journals, beware POSIX close() dropping advisory locks and fork/link/rename hazards.

Cited Findings

• SQLite: backup/restore while transaction active → copy mixes old/new → corrupt; safe via sqlite3_rsync, VACUUM INTO, backup API even on live DB; idle-file copy only safe with no tx in progress — Source
• SQLite: never move/delete/rename hot -journal/-wal after crash; mispairing (swap/overwrite/move journal, copy DB without journal, overwrite DB without deleting hot journal) likely corrupts; quiescent DB has no journal, only DB file matters — Source
• SQLite POSIX advisory-lock quirk: any thread open/read/close on DB file drops locks for all threads (close cancels locks); bypassing lib for backup read can corrupt; since 3.51.0 (2025-11-04) extra WAL defenses but not cure-all — never close() DB file while connections open even in other threads — Source
• SQLite: don't use two SQLite copies linked in one app (separate lock lists), don't mix locking protocols (POSIX vs dot-file/NFS), don't unlink/rename open DB (shared journal name → cross-recovery corruption, SQLITE_WARNING since 3.7.17), don't multi-link/symlink same file (wrong journal lookup; canonicalization since 3.10.0), don't carry connection across fork() — Source
• SQLite WAL forgiving of out-of-order writes except during checkpoint; COMMIT sync failure loses durability not consistency; checkpoint infrequently as defense; QNX mmap + WAL needs exclusive locking/no-mmap — Source
• Forum: with WAL expect journal replay on open after restore, uncommitted lost; for professional frozen-moment copy don't back up while changing; full WAL checkpoint then copy main file pristine if no checkpoint during copy (disable autocheckpoint temporarily) — Source
• Borg docs: avoid programs changing files during backup; LVM/ZFS snapshot or dump/stop DBs; shutdown VMs/containers first — Source
• Kopia actions support before-folder/after-folder and before-snapshot-root/after-snapshot-root scripts (e.g. zfs snapshot + mount then snapshot mount, zfs destroy after; pg_dumpall/sqlite dump) with essential/optional/async modes and timeout (default 5m) — Source
• SafeKeep (rdiff-backup wrapper) noted as integrating with LVM and databases for consistent backups — Source

Inferences

• Correct pattern for single-user Linux: pre-action: pg_dumpall/sqlite VACUUM INTO/.backup → file under /home or /var/backups then borg/restic/kopia snapshot includes dump; keep live DB files too but rely on dump for restore.
• Filesystem snapshot (btrfs/LVM/ZFS) gives crash-consistent base but DB dump still needed for logical restore across versions.

Gaps

• No 2026 primary guide found for sqlite3_rsync adoption in borg/restic/kopia hooks; only SQLite 3.47.0+ release note reference.
• No reliable latency/lock Hold-time guidance for VACUUM INTO vs backup API on large WAL DBs under active writers.