versioning/research_notes/Retention scheduling adaptation/resource-adaptation.md

Resource Adaptation in Backup Tools

Which knobs exist (restic --limit-upload, borg --upload-ratelimit, kopia throttling, Time Machine low-priority I/O) and what are their defaults?

Takeaway

All Linux tools use static, opt-in, unlimited-by-default rate/priority knobs; Time Machine instead imposes mandatory kernel-level low-priority I/O (IOPOL_THROTTLE) with no per-backup user knob.

Cited Findings

• restic exposes global --limit-upload rate and --limit-download rate in KiB/s, default unlimited (0) - Source; also documented in man pages as --limit-upload=0 / --limit-download=0 default unlimited - Source
• restic documents that --limit-upload cannot be changed mid-run without restart; users request pv-like -R dynamic adjustment and resort to killing/restarting or external shapers - Source
• Borg 1.x exposes --upload-ratelimit RATE in kiByte/s, default 0=unlimited (with --remote-ratelimit deprecated alias) - Source; also --upload-buffer size in MiB, default no buffer - Source
• Borg 2.x removed --remote-ratelimit/--upload-ratelimit; bandwidth is limited via BORGSTORE_BANDWIDTH in bits/sec, default 0=unlimited, plus BORGSTORE_LATENCY delay per call, or via pv -L ProxyCommand / rclone --bwlimit - Source
• Older Borg 1.x FAQ documents upload-only --remote-ratelimit plus pv-wrapper + BORG_RSH for download shaping with on-the-fly pv -R $(pidof pv) -L changes - Source
• Borg's limiter is a token-bucket SleepingBandwidthLimiter with RATELIMIT_PERIOD = 0.1, capping burst quota at 2x period allowance - Source
• Kopia exposes repository throttle set flags --upload-bytes-per-second, --download-bytes-per-second, --concurrent-reads/writes, --read/write-requests-per-second, --list-requests-per-second - Source; readable via repository throttle get - Source; server-side equivalent server throttle set - Source
• Kopia direct-connect backends accept --max-upload-speed/--max-download-speed bytes/sec at repository connect time (e.g. B2/S3), persisted as maxUploadSpeedBytesPerSecond in repository.config - Source
• Kopia repository throttle set is rejected on server-connected repos ("operation supported only on direct repository"), and per-policy/UI global throttle was still missing as of 2024–2026 feature requests - Source; KopiaUI throttle exposure requested - Source
• Kopia upload throttling was bursty (whole-object sleeps) until PR #2682 added per-read DuringUpload throttling - Source
• Time Machine's backupd runs at IOPOL_THROTTLE, defined as "long-running I/O intensive background work, such as backups" that "will be throttled to prevent impact on higher policy levels" - Source
• Full IOPOL ladder is IMPORTANT (default) / STANDARD / UTILITY / THROTTLE / PASSIVE - Source
• Global kill-switch sudo sysctl debug.lowpri_throttle_enabled=0 (re-enable with =1, lost on reboot unless persisted via /etc/sysctl.conf or LaunchDaemon) removes throttle for all background I/O, not just backupd - Source; same command/LaunchDaemon recipe - Source; throttling is I/O not CPU - Source
• Measured effect of disabling throttle: copying phase 193→332 MB/s, overall backup 160→276 MB/s (>10 GB test); pre-backup 50 MB probe writes unaffected - Source
• Duplicity has no native generic bandwidth-limit option (open bug #1291633); workarounds are trickle -s -u/-d, WonderShaper/tc, or legacy --scp-command="scp -l N" (kbit/s, scp backend only, option later deprecated) - Source; scp -l throttle Q&A - Source; router-QoS/DSCP attempts reported ineffective, per-machine Bandwidth Limiter used instead - Source
• Neither restic, borg, kopia, nor duplicity ships battery- or metered-network-aware auto-pause; scheduling/power-awareness is delegated to systemd timers, DAS-CTS (macOS), or external shapers (see Gaps).

Inferences

• Rate limiting is consistently an operator-set static ceiling, not adaptive congestion control; only pv/WonderShaper/BORGSTORE allow out-of-band changes without restarting the backup.
• Borg 2.x shifts shaping out of borg CLI into the borgstore layer, unifying ssh/sftp/rclone paths but breaking old --upload-ratelimit scripts.

Gaps

• No reliable source found for a built-in metered-WiFi or on-battery auto-suspend in restic/borg/kopia/duplicity as of 2026; Time Machine power/thermal inputs are internal to DAS scoring, not user flags.

Does any tool auto-tune from measured resources (disk size, RAM, link speed) rather than static config, and how?

Takeaway

No tool auto-tunes from measured disk/RAM/link speed; the only resource-derived defaults are CPU-count-derived worker counts, everything else is fixed static defaults.

Cited Findings

• restic defaults: file-read concurrency 2 ("sweet spot" from HDD experiments), blob-save concurrency = runtime.NumCPU(), tree-save concurrency = 20x blob concurrency - Source
• restic uses all available CPUs by default; GOMAXPROCS=1 pins to one core and slightly reduces memory - Source
• restic backend connection limit defaults to 5 (2 for local backend), tunable via -o rest.connections=5 / -o local.connections=2; too-high values degrade performance - Source
• restic --read-concurrency / RESTIC_READ_CONCURRENCY raises parallel file reads for NVMe; --no-scan skips the pre-backup file-count/size scan that costs extra I/O on network/FUSE mounts - Source; read-concurrency flag added in 0.15.0 for fast storage - Source
• Single-large-file chunking in restic is sequential per file (~450 MB/s) with parallel hash/compress/encrypt downstream; multi-file parallelism is what scales, which is why cores/4-style read-concurrency guesses only hold for SSDs - Source
• Kopia --max-parallel-file-reads defaults to number of logical CPU cores; lowering it lowers CPU at cost of time - Source
• Kopia --max-parallel-snapshots controls simultaneous snapshots (server/KopiaUI); s2 default/better compressor concurrency equals logical core count, s2-parallel-4/8 pins it - Source
• Time Machine scheduling via DAS-CTS scores each due background activity every few seconds against temperature, load, and priority, dispatching com.apple.backupd-auto via XPC only when score exceeds threshold - Source
• On Apple Silicon, background-QoS backupd threads are confined to Efficiency cores at reduced frequency (~972–1332 MHz, ~90% residency on E cores), capping throughput at ~300–400 items/s regardless of queue depth - Source
• No evidence any tool probes link speed, disk size, or free RAM to set pack size, connections, or limits automatically; restic 16 MiB pack size, Kopia parallelism, and Borg rate defaults are all static.

Inferences

• "Auto-tuning" in this space means CPU-count-proportional worker pools plus OS-scheduler deferral (DAS-CTS/QoS), not closed-loop adaptation to throughput or memory pressure.
• Raising concurrency without raising memory (restic 100×1 GB test hit 300 GB RAM at connections=16/reads=16 and OOMed) shows why static defaults stay conservative - Source.

Gaps

• No source found documenting link-speed probing or disk-size-derived chunk/pack sizing in any of the five tools; if it exists it is not in public docs/CLI help.

How do they behave on constrained boxes (low RAM SQLite/index handling, single-core compression choices)?

Takeaway

Constrained-box guidance is manual: pick cheap compression, lower parallelism/connections, enlarge packs, and accept slower runs; low-RAM index handling remains a known failure mode, not an auto-degraded mode.

Cited Findings

• Borg compression default is lz4 (very high speed, very low compression); alternatives zstd[,L] (default level 3), zlib, lzma, auto,, none - Source; usage examples recommend zlib,6 for ratio at cost of speed - Source
• Upstream Borg PR proposes moving default from lz4 to zstd,-4 (multithreaded, as-fast-or-faster creates, slightly better ratio; large incompressible-image corpora stay ~11% slower) with MT workers capped at 4 - Source
• Kopia compression is disabled by default and set per-policy via kopia policy set [--global] --compression=<...> with min/max-size gates - Source; full option list includes s2-default/better/parallel-4/8, zstd/zstd-fastest/better, gzip/pgzip/deflate variants - Source
• Kopia FAQ names compression + parallelism as the two main memory culprits; recommends disabling compression or using s2/deflate/gzip on small files under low memory, and lowering --max-parallel-snapshots / --max-parallel-file-reads - Source
• Kopia benchmark table (466 MiB corpus): s2-default 4 GiB/s at ~375 MiB RSS vs zstd 323 MiB/s at ~238 MiB vs zstd-best 19 MiB/s; on tiny files s2 stays fastest with ~2 MiB footprint - Source
• Kopia zstd levels map to upstream klauspost/compress: fastest≈1, default≈3, better≈7, best≈11; higher custom levels (e.g. -22 --ultra --long) require code change - Source
• restic on constrained HDD/NVMe: forum-tested recipe for 2.5 TB SMR-USB run is --read-concurrency 1 --pack-size 128 --no-cache (pack default 16 MiB raised to reduce file count and per-pack latency), with local.connections=1 suggested for vibration-sensitive HDDs - Source
• restic large-pack tradeoff: bigger packs reduce file count and help HDD/Swift/Drive limits but need more $TMPDIR staging (64–384 MiB guidance) and longer single-pack uploads that wear SSDs - Source
• Borg on near-full disks: 1.5 GiB-free VM case shows Borg needs headroom for segments/cache/index; workarounds discussed are extreme compression or --upload-ratelimit pacing plus inotify/SIGSTOP hacks, with maintainer warning such boxes are unsuitable - Source
• Single-core guidance converges: GOMAXPROCS=1 (restic), -C none|laz4 (borg), --compression=s2-default|none + --max-parallel-file-reads=1 (kopia) minimize CPU/RAM at cost of ratio/throughput.

Inferences

• Low-RAM behavior is fail-slow/fail-OOM rather than graceful: operators must pre-lower parallelism and compression before the run.
• s2-family (kopia) and lz4/zstd,-4 (borg) are the single-core-friendly choices; zlib/lzma/zstd-best are ratio-first and can dominate a weak CPU.

Gaps

• No citable SQLite/index RAM formula (bytes-per-file or per-GB-repo) found for current restic/borg/kopia versions; vendor docs describe symptoms and knobs, not a sizing equation.

What systemd-level controls (MemoryMax, CPUQuota, IOWeight) do shipped units use, and what do docs recommend for background backup daemons?

Takeaway

Upstream backup tools ship no restrictive resource-control units; all concrete CPU/memory/I/O caps come from downstream/community units and generic systemd resource-control docs, with Nice= + CPUQuota/MemoryMax/IOWeight as the recommended trio.

Cited Findings

• systemd CPUQuota= sets a hard ceiling as % of one CPU (100%=1 core, 200%=2 cores) via cpu.max/cpu.cfs_quota_us; CPUWeight= (1–10000, default 100) is only relative under contention - Source; same semantics in Arch man - Source
• systemd memory knobs: MemoryHigh= soft throttle/reclaim, MemoryMax= hard OOM-kill limit (K/M/G/T or % of RAM, infinity to disable), MemorySwapMax= swap cap; Red Hat recommends MemoryHigh as main control, MemoryMax as last defense - Source
• Community restic timer units use only Nice=17 (low CPU priority) plus sandboxing (ProtectSystem=full, PrivateTmp=true, NoNewPrivileges=yes, RestrictAddressFamilies=, SystemCallFilter=, AmbientCapabilities=CAP_DAC_READ_SEARCH), with RandomizedDelaySec=300 + Persistent=yes on the timer - Source
• Community segmented-borg systemd design sets CPUQuota=80% and MemoryMax=2G on the backup service template - Source
• Modern Debian guidance: background CPU → nice -n 19; background disk → ionice -c 3 (BFQ only); hard ceilings/group fairness → unit with CPUQuota=/MemoryMax=/IOWeight=/IOReadBandwidthMax=/IOWriteBandwidthMax= or a shared backup.slice; one-shots via systemd-run --scope -p CPUQuota=50% -p MemoryMax=1G -p IOWeight=10 - Source
• Same guide warns ionice is silently ignored on default mq-deadline SSD schedulers (only BFQ honors classes); MemoryMax kills rather than slows (use MemoryHigh for pushback); CPUQuota=100% means one core - Source
• Example backup slice caps restic at 80 MB/s read / 30 MB/s write via IOReadBandwidthMax=/dev/sda 80M + IOWriteBandwidthMax=/dev/sda 30M with IOWeight=10 so foreground pools keep headroom - Source
• No shipped upstream restic/borg/kopia unit found with MemoryMax/CPUQuota/IOWeight preset; ArchWiki/packaging ships timers/services without resource caps (report-writer: treat absence as finding, not oversight).

Inferences

• Recommended Linux server pattern: Nice=17–19 + optional IOSchedulingClass=idle for soft priority, plus a backup.slice with CPUQuota=50–80%, MemoryHigh/Max=1–4G, IOWeight=10 and per-device IOWriteBandwidthMax for hard isolation.
• nice/ionice alone are insufficient for noisy-neighbor backups on cgroup-v2/BFQ-mixed fleets; systemd slice quotas are the enforceable layer.

Gaps

• Could not locate a distribution-shipped (Debian/Fedora/Arch package) backup unit that presets MemoryMax/CPUQuota/IOWeight; all cited values are community/blog recommendations, not vendor defaults.
• No citable official restic/borg/kopia doc page prescribing a canonical systemd resource-control stanza as of 2026.