packfs

6 Commits 1 Branch 1 Tag

3 Commits

Include renames

Author	SHA1	Message	Date
retoorandClaude Sonnet 5	edbf97b3f1	BENCH.md: literal, complete record -- every row, plus raw appendices The prior BENCH.md curated the benchmark output into summary tables and, in doing so, actually dropped data: the rmdir measurements were missing entirely, and the large-file read rows were never tabulated (only write was), for both the before and after runs. Not what "document this literally" asked for. Fixed: the "Before"/"After" tables now reproduce all 45 measurements each run's summary block reported, none omitted or combined. Added two appendices with the complete verbatim program output for both runs, exactly as captured, as the primary source record -- the curated tables are a presentation of that data, not a substitute for it. Cross-checked every value in both tables against the verbatim appendices; they match exactly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB	2026-09-14 08:53:04 +00:00
retoorandClaude Sonnet 5	64283d587b	Fix the O(n^2) bulk create/unlink: flat array -> persistent treap BENCH.md's earlier finding was real: every structural write (create/ unlink/mkdir) copied the entire sorted UpperSnapshot entry array before publishing the next snapshot, making bulk sequential creation O(n^2). concept.md Section 5.3 names the exact condition for reconsidering this ("a persistent structurally-shared tree structure is not required until this assumption is empirically violated") -- that condition was measured, not hypothesized, so this closes it rather than leaving it as a documented-but-open limitation. The index is now a persistent treap (src/upper.c): a structural write copies only the O(log n) nodes on the path to the change, sharing every other node (its own refcount, cascading like MutCell's and UpperSnapshot's) with whichever snapshot(s) it was built from. Chosen over a persistent AVL/red-black/weight-balanced tree because deletion in those can need O(log n) rebalancing rotations -- each a real allocation in a persistent setting -- where a treap needs only O(1) amortized rotations for insert and delete (Seidel & Aragon 1996), with expected O(log n) height regardless of insertion order, including the sorted-by-creation-order pattern that made the flat array quadratic in the first place. Liljenzin's "Confluently Persistent Sets and Maps" (arXiv:1301.3388) documents persistent treaps giving O(1) snapshots for MVCC specifically, which is this exact use case. Full reasoning and the ownership convention (functions consume one ref of their tree arguments, return one owned ref) are in upper.c's comment above struct TreapNode. Blast radius kept deliberately small: snapshot_upsert/snapshot_remove keep their exact original signatures, so upper_create/upper_mkdir/ upper_remove/upper_rename/upper_copy_up needed zero changes. upper_lookup/upper_has_children keep their exact contracts. Only upstd_readdir and overlay_readdir's manual array scans became calls to a new upper_visit_range (O(log n + r) range query, replacing an O(n) scan in both, a bonus fix beyond what was strictly necessary) since there's no flat array left to scan. Added tests/test_index_stress.c: thousands of randomized (not sequential) creates/deletes/renames across nested directories, cross-checked against an independent reference model after every round, not just "did it not crash" -- exercises exactly the code path the O(n^2) bug and this fix live in, at a scale the other tests don't reach. Verified under -fsanitize=undefined (150+ runs across this change's lifetime, 0 failures) and -fsanitize=address (100+ runs, 0 real findings; known sandbox ASan-startup flakes excluded, see prior commits) per CLAUDE.md's sanitizer rule for upper.c/overlay.c changes. Measured result (make bench, same environment as the original finding): mem create 257x faster, unlink 330x faster, mkdir 65x faster, concurrent mixed workload 131x faster -- and, the comparison that matters, mem now beats raw fs at every one of these (was losing by 6-22x before). The complexity-class change is confirmed the same way the O(n^2) was found: mkdir at N=4,000 vs create at N=20,000 now shows a 7.15x slowdown for a 5x increase in N, matching the O(n log n) prediction (5.97x) rather than the old O(n^2) one (25x). Full before/after tables in BENCH.md's new "Resolution" section, which keeps the original run as the historical record rather than overwriting it, per this project's own documentation standard. Recorded the fix in CLAUDE.md's "Known performance characteristics" (marked RESOLVED, not silently removed) and its architecture map. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB	2026-09-14 08:09:49 +00:00
retoorandClaude Sonnet 5	0b3b207bc0	Add a benchmark suite comparing PackFS against the host filesystem bench/bench.c (`make bench`) measures create/read/stat/readdir/unlink/ mkdir on mem, dir, and raw fs; large sequential I/O; random-access pack reads via mmap vs raw fs; compaction throughput; and 8-thread concurrent mixed workloads. Results from one full run, with honest analysis (what each "raw fs" vs "raw+fsync" vs "dir" label actually measures, so they aren't misread as interchangeable), are in BENCH.md. The benchmark surfaced a real, quantitatively-confirmed finding, not just favorable numbers: bulk sequential create/unlink on mem/dir is O(n^2) in file count (~9-22x slower than raw fs at N=20,000), because every structural write copies the entire snapshot entry array before publishing it (Section 5.3). concept.md itself names the exact trigger condition for reconsidering this ("a persistent structurally-shared tree structure is not required until this assumption is empirically violated") — this benchmark is that violation, measured rather than hypothesized: mkdir at N=4,000 vs create at N=20,000 (same mechanism, 5x the N) shows a 28.4x slowdown, matching the O(n^2) prediction (25x) far better than O(n) (5x). Also found: dir-backend stat() costs ~2x raw stat() (open+fstat vs one syscall, the direct cost of openat2 containment on the metadata path); mem-backed large writes lose to raw fs above ~16MB (Section 5.7's buffer-growth discipline re-copies prior writes on every capacity doubling, the price of never exposing a reader to a freed buffer). Recorded the O(n^2) finding in CLAUDE.md's new "Known performance characteristics" section, per the same pattern used for the reclaim_gate use-after-free discovery, since it's exactly the kind of fact that would otherwise have to be rediscovered by benchmarking again from scratch. Linked from README and CONTRIBUTING. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB	2026-09-14 07:40:42 +00:00

Author

SHA1

Message

Date

retoorandClaude Sonnet 5

edbf97b3f1

BENCH.md: literal, complete record -- every row, plus raw appendices

The prior BENCH.md curated the benchmark output into summary tables
and, in doing so, actually dropped data: the rmdir measurements were
missing entirely, and the large-file read rows were never tabulated
(only write was), for both the before and after runs. Not what
"document this literally" asked for.

Fixed: the "Before"/"After" tables now reproduce all 45 measurements
each run's summary block reported, none omitted or combined. Added
two appendices with the complete verbatim program output for both
runs, exactly as captured, as the primary source record -- the curated
tables are a presentation of that data, not a substitute for it.
Cross-checked every value in both tables against the verbatim
appendices; they match exactly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB

2026-09-14 08:53:04 +00:00

retoorandClaude Sonnet 5

64283d587b

Fix the O(n^2) bulk create/unlink: flat array -> persistent treap

BENCH.md's earlier finding was real: every structural write (create/
unlink/mkdir) copied the entire sorted UpperSnapshot entry array before
publishing the next snapshot, making bulk sequential creation O(n^2).
concept.md Section 5.3 names the exact condition for reconsidering this
("a persistent structurally-shared tree structure is not required
until this assumption is empirically violated") -- that condition was
measured, not hypothesized, so this closes it rather than leaving it
as a documented-but-open limitation.

The index is now a persistent treap (src/upper.c): a structural write
copies only the O(log n) nodes on the path to the change, sharing
every other node (its own refcount, cascading like MutCell's and
UpperSnapshot's) with whichever snapshot(s) it was built from. Chosen
over a persistent AVL/red-black/weight-balanced tree because deletion
in those can need O(log n) rebalancing rotations -- each a real
allocation in a persistent setting -- where a treap needs only O(1)
amortized rotations for insert and delete (Seidel & Aragon 1996), with
expected O(log n) height regardless of insertion order, including the
sorted-by-creation-order pattern that made the flat array quadratic in
the first place. Liljenzin's "Confluently Persistent Sets and Maps"
(arXiv:1301.3388) documents persistent treaps giving O(1) snapshots
for MVCC specifically, which is this exact use case. Full reasoning
and the ownership convention (functions consume one ref of their tree
arguments, return one owned ref) are in upper.c's comment above
struct TreapNode.

Blast radius kept deliberately small: snapshot_upsert/snapshot_remove
keep their exact original signatures, so upper_create/upper_mkdir/
upper_remove/upper_rename/upper_copy_up needed zero changes.
upper_lookup/upper_has_children keep their exact contracts. Only
upstd_readdir and overlay_readdir's manual array scans became calls to
a new upper_visit_range (O(log n + r) range query, replacing an O(n)
scan in both, a bonus fix beyond what was strictly necessary) since
there's no flat array left to scan.

Added tests/test_index_stress.c: thousands of randomized (not
sequential) creates/deletes/renames across nested directories,
cross-checked against an independent reference model after every
round, not just "did it not crash" -- exercises exactly the code path
the O(n^2) bug and this fix live in, at a scale the other tests don't
reach. Verified under -fsanitize=undefined (150+ runs across this
change's lifetime, 0 failures) and -fsanitize=address (100+ runs, 0
real findings; known sandbox ASan-startup flakes excluded, see prior
commits) per CLAUDE.md's sanitizer rule for upper.c/overlay.c changes.

Measured result (make bench, same environment as the original
finding): mem create 257x faster, unlink 330x faster, mkdir 65x
faster, concurrent mixed workload 131x faster -- and, the comparison
that matters, mem now beats raw fs at every one of these (was losing
by 6-22x before). The complexity-class change is confirmed the same
way the O(n^2) was found: mkdir at N=4,000 vs create at N=20,000 now
shows a 7.15x slowdown for a 5x increase in N, matching the O(n log n)
prediction (5.97x) rather than the old O(n^2) one (25x). Full
before/after tables in BENCH.md's new "Resolution" section, which
keeps the original run as the historical record rather than
overwriting it, per this project's own documentation standard.

Recorded the fix in CLAUDE.md's "Known performance characteristics"
(marked RESOLVED, not silently removed) and its architecture map.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB

2026-09-14 08:09:49 +00:00

retoorandClaude Sonnet 5

0b3b207bc0

Add a benchmark suite comparing PackFS against the host filesystem

bench/bench.c (`make bench`) measures create/read/stat/readdir/unlink/
mkdir on mem, dir, and raw fs; large sequential I/O; random-access pack
reads via mmap vs raw fs; compaction throughput; and 8-thread
concurrent mixed workloads. Results from one full run, with honest
analysis (what each "raw fs" vs "raw+fsync" vs "dir" label actually
measures, so they aren't misread as interchangeable), are in BENCH.md.

The benchmark surfaced a real, quantitatively-confirmed finding, not
just favorable numbers: bulk sequential create/unlink on mem/dir is
O(n^2) in file count (~9-22x slower than raw fs at N=20,000), because
every structural write copies the entire snapshot entry array before
publishing it (Section 5.3). concept.md itself names the exact trigger
condition for reconsidering this ("a persistent structurally-shared
tree structure is not required until this assumption is empirically
violated") — this benchmark is that violation, measured rather than
hypothesized: mkdir at N=4,000 vs create at N=20,000 (same mechanism,
5x the N) shows a 28.4x slowdown, matching the O(n^2) prediction (25x)
far better than O(n) (5x).

Also found: dir-backend stat() costs ~2x raw stat() (open+fstat vs one
syscall, the direct cost of openat2 containment on the metadata path);
mem-backed large writes lose to raw fs above ~16MB (Section 5.7's
buffer-growth discipline re-copies prior writes on every capacity
doubling, the price of never exposing a reader to a freed buffer).

Recorded the O(n^2) finding in CLAUDE.md's new "Known performance
characteristics" section, per the same pattern used for the
reclaim_gate use-after-free discovery, since it's exactly the kind of
fact that would otherwise have to be rediscovered by benchmarking
again from scratch. Linked from README and CONTRIBUTING.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB

2026-09-14 07:40:42 +00:00