A audit for other instances of the file-index O(n^2) shape (fixed
previously via the persistent treap) found two more real issues:
1. pack_write's Section 9.2 exact-duplicate elimination was a linear scan
of every previously-seen (hash, size) pair per entry -- O(n^2) total,
invisible in the existing benchmark because its identical-content test
files made every scan match on the first comparison. Measured with
unique content instead: 80,000 entries took 1.74s, with a 20,000->80,000
step showing 15.6x for a 4x-N step, matching O(n^2)'s 16x prediction.
The same scan also trusted a (hash, size) match without ever comparing
actual bytes -- a latent correctness bug, since FNV-1a64 is explicitly
not collision-resistant. Fixed both at once with an open-addressing hash
table (load factor 1/2, linear probing) plus a memcmp verification
before ever reusing a data_off. Post-fix: 80,000 entries in 0.044s
(39.6x faster), ratio drops to 3.35x (consistent with O(n)).
Covered permanently by two new/extended tests: a white-box assertion in
test_pack_overlay.c that duplicate-content entries share one data_off
and distinct-content entries do not, and a new
tests/test_pack_write_perf.c regression tripwire against 10,000 unique
entries.
2. vfs.c's mount table uses the same full-array-copy-per-write pattern the
file index used to, confirmed O(n^2) via a new bench/bench.c category
(500/2,000/8,000 mounts, both 4x-N steps showing 15-20x). Deliberately
NOT rewritten: mount points are created by a program's own source code,
not workload-driven, so realistic mount counts never reach the scale
that made the file index's O(n^2) a real problem. Documented with full
reasoning in BENCH.md and CLAUDE.md rather than silently left as an
undocumented gap.
Also fixes a real CI gap the new tests exposed: ci.yml's sanitizer-build
steps never passed -D_GNU_SOURCE when compiling test files (only the
library .o's got it), which was harmless while no test included
internal.h and became a link failure once two did (internal.h needs
_GNU_SOURCE for pthread_rwlock_t). And documents, in CONTRIBUTING.md and
CLAUDE.md, a sandbox flake observed directly during this work's own
sanitizer runs: ASan/UBSan test binaries occasionally fail to start with
AddressSanitizer:DEADLYSIGNAL (sometimes looping rather than exiting),
non-deterministically hitting different unrelated binaries across runs --
a startup race, not a memory-safety bug, confirmed by clean passes on
retry; sanitizer runs in such an environment should be timeout-wrapped.
BENCH.md's "After" table and Appendix B are replaced with the current,
complete 54-measurement bench/bench.c run (the original 45 plus the new
mount-scaling category); the pre-fix 45-measurement "Before" table is kept
as the historical record, per this project's documentation standard.
Verified: make test (all 6 binaries, including the 2 new/changed), a clean
make all, and repeated ASan+UBSan runs (0 real findings; the DEADLYSIGNAL
flake above was observed and correctly distinguished from a real finding
by re-running until a clean pass). TSan could not be run in this sandbox
(pre-existing, documented environment limitation).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
Cross-checked every function declared in include/packfs.h against
nm -D libpackfs.so.0 as the starting point for a full documentation
pass, per the request to document literally everything rather than
just the parts already covered.
That check found a genuine bug, not just a documentation gap:
backend_pack_new was declared in the public header and named in
CLAUDE.md's architecture map, but never implemented in src/pack.c —
any caller would fail at link time. Implemented it as a standalone,
read-only `pack` Backend (every mutating call returns VFS_ERR_PERM,
consistent with concept.md Section 2.1 listing `pack` as its own
backend kind distinct from the overlay), covered it with a new test
case, and verified it under -fsanitize=undefined per CLAUDE.md's
sanitizer rule.
Added a doc comment to every previously-undocumented function and
struct field in packfs.h and internal.h (vfs_open/read/write/close/
stat/readdir/mkdir/unlink/rename, every upper_* structural/content
function, pfs_dir_*, pfs_fnv1a64, PackIndexEntry/Pack fields). Updated
README and CLAUDE.md to mention backend_pack_new and to stop gesturing
at zip/tar as though import/export exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
Implements the core design: a mount table published as an atomically-
swapped snapshot; mem/dir/pack/overlay backends; copy-on-write overlay
with copy-up and whiteout deletion; a checksummed append journal;
compaction with exact-duplicate elimination; single-writer/wait-free-
reader concurrency with a structural/content write split; openat2/
Landlock path containment for dir mounts; and load-time pack integrity
validation. Zero required third-party dependencies.
Sanitizer testing (ASan/UBSan) caught and led to fixing a genuine
heap-use-after-free in the snapshot-reclamation path: the textbook
"load pointer, then increment its refcount" pattern left a gap a
concurrent writer could free through. Closed with a small reclaim_gate
rwlock, documented in internal.h and CLAUDE.md since it's a pattern
every refcounted structure in the codebase now follows.
zip/tar import/export backends, recommended in concept.md Section 11,
will not be built — a permanent project decision recorded in CLAUDE.md
since concept.md itself is frozen and cannot be edited to reflect it.
Includes a runnable demo (examples/demo.c, `make demo`) exercising the
library end to end and proving cross-run persistence through the pack
file, plus open-source scaffolding: MIT license, README, CONTRIBUTING,
and a CI workflow running the test suite under ASan/UBSan/TSan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
retoorandClaude Sonnet 5