Fix pack_write's O(n^2) dedup + correctness bug; document mount table O(n^2)

A audit for other instances of the file-index O(n^2) shape (fixed
previously via the persistent treap) found two more real issues:

1. pack_write's Section 9.2 exact-duplicate elimination was a linear scan
   of every previously-seen (hash, size) pair per entry -- O(n^2) total,
   invisible in the existing benchmark because its identical-content test
   files made every scan match on the first comparison. Measured with
   unique content instead: 80,000 entries took 1.74s, with a 20,000->80,000
   step showing 15.6x for a 4x-N step, matching O(n^2)'s 16x prediction.
   The same scan also trusted a (hash, size) match without ever comparing
   actual bytes -- a latent correctness bug, since FNV-1a64 is explicitly
   not collision-resistant. Fixed both at once with an open-addressing hash
   table (load factor 1/2, linear probing) plus a memcmp verification
   before ever reusing a data_off. Post-fix: 80,000 entries in 0.044s
   (39.6x faster), ratio drops to 3.35x (consistent with O(n)).

   Covered permanently by two new/extended tests: a white-box assertion in
   test_pack_overlay.c that duplicate-content entries share one data_off
   and distinct-content entries do not, and a new
   tests/test_pack_write_perf.c regression tripwire against 10,000 unique
   entries.

2. vfs.c's mount table uses the same full-array-copy-per-write pattern the
   file index used to, confirmed O(n^2) via a new bench/bench.c category
   (500/2,000/8,000 mounts, both 4x-N steps showing 15-20x). Deliberately
   NOT rewritten: mount points are created by a program's own source code,
   not workload-driven, so realistic mount counts never reach the scale
   that made the file index's O(n^2) a real problem. Documented with full
   reasoning in BENCH.md and CLAUDE.md rather than silently left as an
   undocumented gap.

Also fixes a real CI gap the new tests exposed: ci.yml's sanitizer-build
steps never passed -D_GNU_SOURCE when compiling test files (only the
library .o's got it), which was harmless while no test included
internal.h and became a link failure once two did (internal.h needs
_GNU_SOURCE for pthread_rwlock_t). And documents, in CONTRIBUTING.md and
CLAUDE.md, a sandbox flake observed directly during this work's own
sanitizer runs: ASan/UBSan test binaries occasionally fail to start with
AddressSanitizer:DEADLYSIGNAL (sometimes looping rather than exiting),
non-deterministically hitting different unrelated binaries across runs --
a startup race, not a memory-safety bug, confirmed by clean passes on
retry; sanitizer runs in such an environment should be timeout-wrapped.

BENCH.md's "After" table and Appendix B are replaced with the current,
complete 54-measurement bench/bench.c run (the original 45 plus the new
mount-scaling category); the pre-fix 45-measurement "Before" table is kept
as the historical record, per this project's documentation standard.

Verified: make test (all 6 binaries, including the 2 new/changed), a clean
make all, and repeated ASan+UBSan runs (0 real findings; the DEADLYSIGNAL
flake above was observed and correctly distinguished from a real finding
by re-running until a clean pass). TSan could not be run in this sandbox
(pre-existing, documented environment limitation).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
This commit is contained in:
2026-09-14 09:55:47 +00:00
co-authored by Claude Sonnet 5
parent edbf97b3f1
commit 684c6d95de
9 changed files with 693 additions and 205 deletions
+66
View File
@@ -43,6 +43,15 @@
#define N_SMALL 20000 /* small files for the metadata-heavy suite */
#define SMALL_SIZE 128
#define N_DIRS 4000
/* vfs_mount/vfs_unmount use the identical full-snapshot-copy pattern the
* file index used to (Section 5.3's mount-table-is-part-of-the-snapshot
* unification) — this section exists to check, empirically rather than
* by assumption, whether that ever matters at a realistic mount count.
* 2,000 is deliberately far beyond any real program's mount count (a
* mount is something a program's own source code sets up once per
* distinct storage location — assets/config/tmp/per-plugin — not
* something a workload creates at file-count scale). */
#define N_MOUNTS 2000
#define N_CONC_THREADS 8
#define OPS_PER_THREAD 4000
#define LARGE_CHUNK (64 * 1024)
@@ -428,6 +437,57 @@ static void bench_concurrency(Vfs *v, const char *rawroot) {
record("concurrent create+read+unlink", "raw fs", now_sec() - t0, (double)N_CONC_THREADS * OPS_PER_THREAD * 3, 0);
}
/* ---- category 10: mount table scaling ---- */
/* Run at several N (called from main at N_MOUNTS/4, N_MOUNTS, N_MOUNTS*4)
* so the mount table's complexity class can be checked the same way
* Section "Resolution" in BENCH.md checked the file index's: if time(N)
* grows roughly as N^2 rather than N or N log N across a 4x-N step, that
* confirms (not just asserts) that vfs_mount/vfs_unmount's full-array
* copy per structural write (mirroring the file index's old design) is
* quadratic here too — expected, and, per BENCH.md, not worth fixing at
* any mount count a real program would ever reach. */
static void bench_mount_scaling(int n) {
char label[24];
Vfs *v = vfs_new();
Backend **backends = (Backend **)malloc(sizeof(Backend *) * (size_t)n);
char prefix[32];
double t0 = now_sec();
for (int i = 0; i < n; i++) {
backends[i] = backend_mem_new();
snprintf(prefix, sizeof(prefix), "/m%06d", i);
vfs_mount(v, prefix, backends[i]);
}
snprintf(label, sizeof(label), "mount %d backends", n);
record(label, "vfs", now_sec() - t0, n, 0);
/* a resolve through a full mount table, to confirm lookup itself
* (not just mount/unmount) stays fast at this N */
t0 = now_sec();
for (int i = 0; i < n; i++) {
char path[40];
snprintf(prefix, sizeof(prefix), "/m%06d", i);
snprintf(path, sizeof(path), "%s/x.txt", prefix);
VfsStat st;
vfs_stat(v, path, &st); /* NOENT expected; measures resolve cost, not the stat itself */
}
snprintf(label, sizeof(label), "resolve, %d mounts", n);
record(label, "vfs", now_sec() - t0, n, 0);
t0 = now_sec();
for (int i = 0; i < n; i++) {
snprintf(prefix, sizeof(prefix), "/m%06d", i);
vfs_unmount(v, prefix);
}
snprintf(label, sizeof(label), "unmount %d backends", n);
record(label, "vfs", now_sec() - t0, n, 0);
for (int i = 0; i < n; i++) backend_free(backends[i]);
free(backends);
vfs_free(v);
}
/* ---- main ---- */
int main(void) {
@@ -446,6 +506,7 @@ int main(void) {
printf(" small files (N): %d, %d bytes each\n", N_SMALL, SMALL_SIZE);
printf(" directories (N): %d\n", N_DIRS);
printf(" concurrency: %d threads x %d ops (create+read+unlink)\n", N_CONC_THREADS, OPS_PER_THREAD);
printf(" mounts (N): %d\n", N_MOUNTS);
printf(" large-file sizes: ");
for (size_t i = 0; i < N_LARGE_SIZES; i++) printf("%zuMB ", LARGE_SIZES[i] / (1024 * 1024));
printf("\n");
@@ -566,6 +627,11 @@ int main(void) {
section("9. concurrency (create+read+unlink)");
bench_concurrency(v, rawroot);
section("10. mount table scaling (the mount table uses the same full-snapshot-copy pattern the file index used to)");
bench_mount_scaling(N_MOUNTS / 4);
bench_mount_scaling(N_MOUNTS);
bench_mount_scaling(N_MOUNTS * 4);
vfs_unmount(v, "/");
backend_free(mem);
vfs_free(v);