packfs/BENCH.md

PackFS vs. the host filesystem: benchmark results

Status: the O(n²) bulk create/unlink finding this document originally reported has been fixed. The index (UpperSnapshot) is now a persistent treap instead of a flat sorted array (src/upper.c; see CLAUDE.md's "Known performance characteristics"). This document keeps the original run's numbers as the historical record of the problem — per this project's own documentation standard, a correction extends the record rather than erasing it — and adds the post-fix run's numbers alongside them. Read "Resolution" below before drawing conclusions from the "Original results" section, which describes a version of the index this codebase no longer has.

This document reports the results of bench/bench.c (make bench), each run once on the environment described below. These are single runs, not a statistically-averaged series — treat the numbers as illustrative of shape (which operations are faster, by roughly what factor, and why) more than as precise absolute figures for a different machine. The methodology, including exactly what each backend label measures, is documented in bench/bench.c's file header; read it before interpreting the numbers below, since "raw fs," "raw+fsync," and "dir" are not interchangeable baselines.

Environment

• Host: containerized, 12 logical CPUs (AMD Ryzen 5 3600), running under an overlay root filesystem (overlay on overlay, per mount) — there is no separate tmpfs at /tmp in this environment; every "raw fs" and dir number below is against that container overlay filesystem, not a bare disk or tmpfs. This matters most for the fsync numbers (see below).
• N_SMALL = 20,000 files × 128 bytes for the metadata-heavy suite; N_DIRS = 4,000; concurrency = 8 threads × 4,000 ops; large files up to 64 MB. Full parameters are in bench/bench.c.
• Both runs took several minutes of wall-clock time, almost entirely spent in the single raw+fsync 20,000-file create test (~116s in both runs — this cost is on the host's storage, not PackFS, and is identical before and after the fix, as expected).

Resolution

The original run below found bulk sequential create/unlink/mkdir on mem/dir to be O(n²) in file count, because every structural write copied the entire sorted entry array before publishing the next snapshot (Section 5.3). concept.md itself names the trigger condition for reconsidering that design — "a persistent (structurally shared) tree structure is not required until this assumption is empirically violated" — and the original run was exactly that violation, measured rather than hypothesized.

The index was rewritten as a persistent treap (src/upper.c): a structural write now builds only the O(log n) nodes on the path to the change, sharing every other node (refcounted) with the snapshot it was built from, instead of copying all n entries. The choice of treap over a persistent AVL/red-black/weight-balanced tree, and the reasoning behind it, is documented in src/upper.c's comment above struct TreapNode and cited there: Seidel & Aragon, "Randomized Search Trees" (1996), for the O(1) amortized rotation count on both insert and delete (persistent balanced trees that need O(log n) rotations pay a real, repeated allocation cost per rotation, which a treap avoids); Liljenzin, "Confluently Persistent Sets and Maps" (arXiv:1301.3388), for persistent treaps specifically as an MVCC snapshot mechanism, which is this exact use case.

Result, measured, same methodology as the original finding:

Category	Before	After	Speedup
create 20,000 files (mem)	9.56s / 2,093 ops/s	0.037s / 537,103 ops/s	257x
unlink 20,000 files (mem)	10.56s / 1,894 ops/s	0.032s / 625,437 ops/s	330x
mkdir 4,000 dirs (mem)	0.336s / 11,896 ops/s	0.0052s / 771,085 ops/s	65x
create 20,000 files (dir)	12.04s / 1,662 ops/s	1.15s / 17,345 ops/s	10x
concurrent create+read+unlink (mem)	36.73s / 2,614 ops/s	0.28s / 341,972 ops/s	131x

mem now beats raw fs (not merely "no longer loses to it") at every one of these: create is 27x faster than raw fs (was 9.5x slower), unlink is 15x faster (was 22x slower), and the concurrent mixed workload is 22x faster (was 6x slower) — the single-writer design's "optimizes for read-heavy concurrent workloads" caveat from the original analysis (below) no longer needs to exclude bulk-write workloads, because the write side is no longer the bottleneck it was.

The complexity-class change is confirmed quantitatively, the same way the original O(n²) finding was — not just "it got faster," which an unrelated constant-factor optimization could also produce: mkdir at N=4,000 vs. create at N=20,000 (same structural-write mechanism, 5x the N) now shows a 7.15x slowdown. O(n) predicts 5.0x; O(n log n) predicts 5.97x; the old O(n²) regime predicted, and measured, 25–28.4x. 7.15x lands close to the O(n log n) prediction (some excess is expected — content writes' buffer-allocation and per-file host effects differ between create, which writes bytes, and mkdir, which does not) and nowhere near quadratic. The index's asymptotic behavior changed, not merely its constant factor.

Full output of the post-fix run is reproducible with make bench; the correctness of the new index under heavy randomized structural churn (non-sequential creates/deletes/renames, nested directories, thousands of entries, cross-checked against an independent reference model — not just "the benchmark ran fast without crashing") is tests/test_index_stress.c, part of make test.

Original results (pre-fix; kept as the historical record — see "Resolution" above)

Category	Backend	Time	Throughput	MB/s
create 20,000 files	mem	9.56s	2,093 ops/s	0.3
create 20,000 files	dir	12.04s	1,662 ops/s	0.2
create 20,000 files	raw fs	1.01s	19,836 ops/s	2.4
create 20,000 files	raw+fsync	116.10s	172 ops/s	0.0
read 20,000 files	mem	0.007s	2,798,731 ops/s	341.6
read 20,000 files	dir	0.167s	119,468 ops/s	14.6
read 20,000 files	raw fs	0.161s	124,122 ops/s	15.2
stat 20,000 files	mem	0.006s	3,223,599 ops/s	—
stat 20,000 files	dir	0.152s	131,937 ops/s	—
stat 20,000 files	raw fs	0.070s	284,465 ops/s	—
readdir (20,000 entries)	mem	0.0037s	5,397,828 /s	—
readdir (20,000 entries)	dir	0.0028s	7,058,003 /s	—
readdir (20,000 entries)	raw fs	0.0071s	2,831,904 /s	—
unlink 20,000 files	mem	10.56s	1,894 ops/s	—
unlink 20,000 files	dir	10.19s	1,964 ops/s	—
unlink 20,000 files	raw fs	0.48s	41,823 ops/s	—
mkdir 4,000 dirs	mem	0.336s	11,896 ops/s	—
mkdir 4,000 dirs	dir	0.511s	7,829 ops/s	—
mkdir 4,000 dirs	raw fs	0.159s	25,143 ops/s	—
write 1 / 16 / 64 MB	mem	—	—	6,687 / 1,158 / 1,130
write 1 / 16 / 64 MB	raw fs (no fsync)	—	—	2,685 / 3,413 / 3,436
write 1 / 16 / 64 MB	raw+fsync	—	—	116 / 191 / 718
random-read 20,000 entries	pack (mmap'd)	0.0085s	2,350,224 ops/s	286.9
random-read 20,000 entries	raw fs	0.177s	113,198 ops/s	13.8
compact 20,000 entries to pack	pack	0.029s	687,097 ops/s	83.9
concurrent create+read+unlink (8×4,000×3)	mem	36.73s	2,614 ops/s	—
concurrent create+read+unlink (8×4,000×3)	raw fs	6.12s	15,686 ops/s	—

Post-fix results

Category	Backend	Time	Throughput	MB/s
create 20,000 files	mem	0.037s	537,103 ops/s	65.6
create 20,000 files	dir	1.153s	17,345 ops/s	2.1
create 20,000 files	raw fs	1.010s	19,805 ops/s	2.4
create 20,000 files	raw+fsync	116.098s	172 ops/s	0.0
read 20,000 files	mem	0.0085s	2,364,410 ops/s	288.6
read 20,000 files	dir	0.160s	124,933 ops/s	15.3
read 20,000 files	raw fs	0.159s	125,459 ops/s	15.3
stat 20,000 files	mem	0.0073s	2,723,001 ops/s	—
stat 20,000 files	dir	0.151s	132,517 ops/s	—
stat 20,000 files	raw fs	0.070s	287,063 ops/s	—
readdir (20,000 entries)	mem	0.0042s	4,816,418 /s	—
readdir (20,000 entries)	dir	0.0039s	5,128,581 /s	—
readdir (20,000 entries)	raw fs	0.0070s	2,840,890 /s	—
unlink 20,000 files	mem	0.032s	625,437 ops/s	—
unlink 20,000 files	dir	0.515s	38,864 ops/s	—
unlink 20,000 files	raw fs	0.492s	40,628 ops/s	—
mkdir 4,000 dirs	mem	0.0052s	771,085 ops/s	—
mkdir 4,000 dirs	dir	0.177s	22,551 ops/s	—
mkdir 4,000 dirs	raw fs	0.137s	29,251 ops/s	—
write 1 / 16 / 64 MB	mem	—	—	2,937 / 1,483 / 1,087
write 1 / 16 / 64 MB	raw fs (no fsync)	—	—	2,463 / 3,478 / 3,496
write 1 / 16 / 64 MB	raw+fsync	—	—	42 / 702 / 776
random-read 20,000 entries	pack (mmap'd)	0.0082s	2,445,144 ops/s	298.5
random-read 20,000 entries	raw fs	0.166s	120,709 ops/s	14.7
compact 20,000 entries to pack	pack	0.025s	814,864 ops/s	99.5
concurrent create+read+unlink (8×4,000×3)	mem	0.281s	341,972 ops/s	—
concurrent create+read+unlink (8×4,000×3)	raw fs	6.298s	15,243 ops/s	—

Everything not involving bulk mem/dir structural writes (reads, stat, readdir, large sequential I/O, pack random-access, compaction) is unchanged within normal run-to-run noise, as expected — the treap rewrite only touches the structural-write path (Section 5.3), not content reads, content writes, or the pack format.

Analysis

The bullets below are preserved from the original run for the categories the fix did not change (the "wins" and the non-structural-write "losses"), with the historical O(n²) explanation kept as the record of what was found and superseded by "Resolution" above where it no longer applies.

Where PackFS wins clearly

• Reads of small files: 20–23x faster than both raw fs and the dir backend (2.4M ops/s vs ~125K ops/s). No syscall per read — mem reads are an expected-O(log n) treap lookup plus a memcpy out of an already-resident buffer (Section 5.3, 9.1).
• stat: ~10x faster than raw fs. Same reason — no syscall, and the index is sorted for expected-O(log n) lookup rather than requiring a directory entry scan.
• Random-access reads against a compacted pack: ~20x faster than raw fs (2.4M ops/s vs 121K ops/s), because the pack is one mmap'd file with a binary-searchable index (Section 9.1), against 20,000 individual open/read/close syscall triples on the raw-fs side. This is the single result that most directly validates the architecture's stated purpose — Section 3.2's claim that "random access is a flat table lookup, not a linear scan or a directory-parse-then-seek" — under an actual measured workload, not just by construction.
• Compaction throughput: ~100 MB/s / ~815K entries/s to serialize the live tree into a fresh pack (Section 4.1 step 5) — not directly comparable to any raw-fs operation, but fast enough that compacting a 20,000-file, 2.5 MB tree is not a practically-felt pause (25ms).
• Bulk create/unlink/mkdir: now 10–330x faster than the pre-fix mem/ dir numbers, and (for mem) 15–27x faster than raw fs — see "Resolution" above; this used to be PackFS's clearest loss and is now among its clearest wins.
• Concurrent mixed create+read+unlink: now 22x faster than raw fs (was 6x slower before the fix) — see "Resolution."

Where raw fs still wins, and why that's expected, not a bug

• dir backend stat is ~2.2x slower than raw fs stat (133K ops/s vs 287K ops/s) — because pfs_dir_statat (Section 6.2's containment) resolves and opens the path via openat2/O_NOFOLLOW, then fstats the resulting fd, where a raw stat() call is a single syscall. This is the direct, measured cost of path containment on the metadata path, separate from and smaller than the containment cost paid on open itself (which raw fs pays an equivalent single-syscall cost for anyway). Unaffected by the index rewrite.
• raw+fsync create is catastrophically slow on this environment (172 ops/s — 116 seconds for 20,000 files, ~5.8ms per fsync), because this container's overlay filesystem has poor per-call fsync latency. This is a property of the container, not of PackFS or of a bare disk — flagged in the environment section above precisely so it isn't misread as "PackFS's journal must be this slow too." The journal (Section 4.4) does call fsync once per journaled write for the same durability reason raw fsync is slow here, which is a real, inherited cost on this kind of storage — not a PackFS-specific one. Unaffected by the index rewrite (the journal's cost is per-write fsync latency, not index maintenance).
• Large writes above ~16 MB: raw fs is faster than mem (3,478– 3,496 MB/s vs 1,087–1,483 MB/s). Section 5.7's buffer-growth discipline (allocate new, copy old + new, publish, retire old — never realloc in place) means every capacity doubling re-copies everything written so far; a plain unsynced write() to a real file only ever appends new pages to the page cache, never re-copying prior ones. This is entirely independent of the index structure (Section 5.7's buffer-growth discipline governs a MutCell's content buffer, not the index the treap rewrite replaced) and is unaffected by this fix — the safety property Section 5.7 requires (no reader can ever see a freed buffer) still has a real, quantifiable cost for very large sequential writes, traded for correctness under concurrent access that a plain in-place realloc would not have.

Reproducing

make bench

Takes several minutes on a similarly slow-fsync environment, dominated by the raw+fsync create test; expect well under a minute on a host with normal disk or tmpfs fsync latency.