BENCH.md: fix numbers left stale by the After-table replacement

Doing another literal, no-caveats audit pass (per the standing request to
verify literally everything, not just reassert it) found real, concrete
inconsistencies introduced by the previous commit: it replaced BENCH.md's
"After: complete results" table and Appendix B with a newer, more complete
bench/bench.c run (adding the mount-table-scaling category), but several
other places in the same document still quoted exact figures from the
older run those tables used to hold -- a reader cross-checking the
"Resolution" section's headline table, or the "Analysis" section's bullet
points, against the "After" table below them would have found contradictory
numbers for the same measurement.

Verified programmatically, not just by re-reading: every row's time,
throughput, and MB/s in the Before (45 rows) and After (54 rows) tables now
matches its corresponding appendix line exactly (0 mismatches out of 99
rows, checked with a script, not by eye).

Fixed:
- The "Resolution" section's headline before/after table used the older
  run's mem create/unlink/mkdir/rmdir numbers (0.0372s/0.0320s/0.0052s/
  0.0045s), which don't match the current After table (0.0400s/0.0179s/
  0.0056s/0.0040s) -- for unlink specifically, a genuine near-2x difference
  between two runs of identical, already-fixed code, not just noise. Now
  points at the single current run, with speedups recomputed (239x/590x/
  60x/85x/11x/21x/134x, mem now 25-27x faster than raw fs, complexity-class
  check now 7.14x vs the old 7.15x -- nearly identical, which is the
  reassurance that actually matters), plus an explicit note on the observed
  run-to-run variance rather than silently picking one number.
- The "Analysis" section's read/stat/random-read/compaction/dir-stat/
  large-write bullets all cited the older run's exact ops/s and MB/s
  figures (2.4M small-file reads, ~815K compaction entries/s, 133K dir
  stat, etc.); recomputed against the current After table (now 15-16x for
  small reads, ~9x for stat, ~91MB/s / ~748K entries/s for compaction, 145K
  vs 286K for dir stat, corrected large-write MB/s ranges).
- The same stale 65-330x / 15-27x / 22x range in CLAUDE.md's "Known
  performance characteristics" bulk-write bullet, updated to match and to
  explain the same run-to-run variance rather than presenting one run's
  numbers as if they were exact.

No code changes; make test still passes (all 6 binaries, unaffected by a
documentation-only change).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
This commit is contained in:
2026-09-14 10:13:57 +00:00
co-authored by Claude Sonnet 5
parent 684c6d95de
commit 529935deb4
2 changed files with 59 additions and 36 deletions
+58 -35
View File
@@ -88,21 +88,37 @@ rotation, which a treap avoids); Liljenzin, "Confluently Persistent Sets and
Maps" (arXiv:1301.3388), for persistent treaps specifically as an MVCC
snapshot mechanism, which is this exact use case.
**Headline result, measured, same methodology as the original finding:**
**Headline result, measured, same methodology as the original finding.**
The "After" figures below are drawn from the "After: complete results"
table further down this document (the current, canonical post-fix run),
not from an intermediate run kept only in this section — every number here
can be found again in that table:
| Category | Before | After | Speedup |
|---|---|---|---|
| create 20,000 files (mem) | 9.5565s / 2,093 ops/s | 0.0372s / 537,103 ops/s | **257x** |
| unlink 20,000 files (mem) | 10.5575s / 1,894 ops/s | 0.0320s / 625,437 ops/s | **330x** |
| mkdir 4,000 dirs (mem) | 0.3363s / 11,896 ops/s | 0.0052s / 771,085 ops/s | **65x** |
| rmdir 4,000 dirs (mem) | 0.3395s / 11,782 ops/s | 0.0045s / 898,169 ops/s | **75x** |
| create 20,000 files (dir) | 12.0367s / 1,662 ops/s | 1.1531s / 17,345 ops/s | **10x** |
| unlink 20,000 files (dir) | 10.1851s / 1,964 ops/s | 0.5146s / 38,864 ops/s | **20x** |
| concurrent create+read+unlink (mem) | 36.7261s / 2,614 ops/s | 0.2807s / 341,972 ops/s | **131x** |
| create 20,000 files (mem) | 9.5565s / 2,093 ops/s | 0.0400s / 499,809 ops/s | **239x** |
| unlink 20,000 files (mem) | 10.5575s / 1,894 ops/s | 0.0179s / 1,117,742 ops/s | **590x** |
| mkdir 4,000 dirs (mem) | 0.3363s / 11,896 ops/s | 0.0056s / 719,696 ops/s | **60x** |
| rmdir 4,000 dirs (mem) | 0.3395s / 11,782 ops/s | 0.0040s / 993,724 ops/s | **85x** |
| create 20,000 files (dir) | 12.0367s / 1,662 ops/s | 1.1418s / 17,517 ops/s | **11x** |
| unlink 20,000 files (dir) | 10.1851s / 1,964 ops/s | 0.4911s / 40,724 ops/s | **21x** |
| concurrent create+read+unlink (mem) | 36.7261s / 2,614 ops/s | 0.2750s / 349,127 ops/s | **134x** |
(An earlier revision of this table cited a different post-fix run — same
fixed code, an earlier `make bench` invocation — whose numbers were 65–330x
rather than 60–590x; the two runs agree on every qualitative conclusion
below, but `unlink 20,000 files (mem)` in particular differs by nearly 2x
between them, 0.0320s vs 0.0179s, both measuring an operation fast enough
(sub-20ms for 20,000 entries) that ordinary system jitter in this
containerized environment has real proportional effect. This is exactly
the kind of run-to-run variance this document's stated methodology warns
about — "illustrative of shape... more than precise absolute figures" —
and the reason the table now points at one single canonical run rather
than mixing figures from two.)
`mem` now *beats* raw fs (not merely "no longer loses to it") at every one
of these: create is 27x faster than raw fs (was 9.5x slower), unlink is 15x
faster (was 22x slower), and the concurrent mixed workload is 22x faster
of these: create is 25x faster than raw fs (was 9.5x slower), unlink is 27x
faster (was 22x slower), and the concurrent mixed workload is 21x faster
(was 6x slower) — the single-writer design's "optimizes for read-heavy
concurrent workloads" caveat from the original analysis (below) no longer
needs to exclude bulk-write workloads, because the write side is no longer
@@ -112,13 +128,17 @@ the bottleneck it was.
original O(n²) finding was** — not just "it got faster," which an unrelated
constant-factor optimization could also produce: `mkdir` at N=4,000 vs.
`create` at N=20,000 (same structural-write mechanism, 5x the N) now shows a
**7.15x** slowdown (0.0372s / 0.0052s). O(n) predicts 5.0x; O(n log n)
**7.14x** slowdown (0.0400s / 0.0056s). O(n) predicts 5.0x; O(n log n)
predicts 5.97x; the old O(n²) regime predicted, and measured, 25–28.4x.
7.15x lands close to the O(n log n) prediction (some excess is expected —
7.14x lands close to the O(n log n) prediction (some excess is expected —
`create` writes bytes into a fresh `MutCell` and `mkdir` does not, a
constant-factor difference the pure index-complexity comparison doesn't
account for) and nowhere near quadratic. The index's asymptotic behavior
changed, not merely its constant factor.
account for) and nowhere near quadratic — and is itself nearly identical to
the 7.15x this same check gave against the earlier run referenced above,
which is the reassurance that matters here: the *ratio* this check depends
on is stable across runs even where the fastest individual measurements
are not. The index's asymptotic behavior changed, not merely its constant
factor.
Full output of both runs is reproduced verbatim in the appendices below and
is reproducible again with `make bench`; the correctness of the new index
@@ -394,37 +414,39 @@ make for them; they are current-state measurements only, discussed in
## Analysis
The bullets below are preserved from the original run for the categories
the fix did not change (the "wins" and the non-structural-write "losses"),
with the historical O(n²) explanation kept as the record of what was found,
now marked where "Resolution" above supersedes it.
The bullets below are checked against the current "After" table (the
54-measurement run) rather than preserved unchanged from an earlier draft —
an earlier revision of this section cited exact figures from the prior
45-measurement after-run, which drifted out of sync with the "After" table
once that table was replaced with the newer run; the numbers below were
recomputed from the table actually in this document, not carried over.
### Where PackFS wins clearly
- **Reads of small files: 20–23x faster than both raw fs and the `dir`
backend** (2.4M ops/s vs ~125K ops/s). No syscall per read — `mem`
- **Reads of small files: 15–16x faster than both raw fs and the `dir`
backend** (2.0M ops/s vs ~125–131K ops/s). No syscall per read — `mem`
reads are an expected-O(log n) treap lookup plus a `memcpy` out of an
already-resident buffer (Section 5.3, 9.1).
- **`stat`: ~10x faster than raw fs.** Same reason — no syscall, and the
index is sorted for expected-O(log n) lookup rather than requiring a
directory entry scan.
- **`stat`: ~9x faster than raw fs** (2.6M ops/s vs 286K ops/s). Same
reason — no syscall, and the index is sorted for expected-O(log n)
lookup rather than requiring a directory entry scan.
- **Random-access reads against a compacted pack: ~20x faster than raw fs**
(2.4M ops/s vs 121K ops/s), because the pack is one `mmap`'d file with a
(2.4M ops/s vs 123K ops/s), because the pack is one `mmap`'d file with a
binary-searchable index (Section 9.1), against 20,000 individual
`open`/`read`/`close` syscall triples on the raw-fs side. This is the
single result that most directly validates the architecture's stated
purpose — Section 3.2's claim that "random access is a flat table
lookup, not a linear scan or a directory-parse-then-seek" — under an
actual measured workload, not just by construction.
- **Compaction throughput**: ~100 MB/s / ~815K entries/s to serialize the
- **Compaction throughput**: ~91 MB/s / ~748K entries/s to serialize the
live tree into a fresh pack (Section 4.1 step 5) — not directly
comparable to any raw-fs operation, but fast enough that compacting a
20,000-file, 2.5 MB tree is not a practically-felt pause (25ms).
- **Bulk create/unlink/mkdir/rmdir: RESOLVED, now 10–330x faster than the
pre-fix `mem`/`dir` numbers, and (for `mem`) 15–27x faster than raw fs** —
20,000-file, 2.5 MB tree is not a practically-felt pause (27ms).
- **Bulk create/unlink/mkdir/rmdir: RESOLVED, now 11–590x faster than the
pre-fix `mem`/`dir` numbers, and (for `mem`) 21–27x faster than raw fs** —
see "Resolution" above; this used to be PackFS's clearest loss and is now
among its clearest wins.
- **Concurrent mixed create+read+unlink: RESOLVED, now 22x faster than raw
- **Concurrent mixed create+read+unlink: RESOLVED, now 21x faster than raw
fs** (was 6x *slower* before the fix) — see "Resolution."
- **`pack_write` compaction dedup on distinct-content files: RESOLVED, 1.75–
39.6x faster depending on N, plus a latent hash-collision correctness bug
@@ -435,8 +457,8 @@ now marked where "Resolution" above supersedes it.
### Where raw fs still wins, and why that's expected, not a bug
- **`dir` backend `stat` is ~2.2x *slower* than raw fs `stat`** (133K ops/s
vs 287K ops/s) — because `pfs_dir_statat` (Section 6.2's containment)
- **`dir` backend `stat` is ~2.0x *slower* than raw fs `stat`** (145K ops/s
vs 286K ops/s) — because `pfs_dir_statat` (Section 6.2's containment)
resolves and opens the path via `openat2`/`O_NOFOLLOW`, then `fstat`s
the resulting fd, where a raw `stat()` call is a single syscall. This is
the direct, measured cost of path containment on the metadata path,
@@ -444,8 +466,9 @@ now marked where "Resolution" above supersedes it.
itself (which raw fs pays an equivalent single-syscall cost for anyway).
Unaffected by the index rewrite.
- **`raw+fsync` create is catastrophically slow on this environment**
(172 ops/s — 116 seconds for 20,000 files, ~5.8ms per `fsync`, identical
in both runs to three decimal places), because this container's overlay
(172 ops/s — 116 seconds for 20,000 files, ~5.8ms per `fsync`, within
measurement noise across the "Before" and "After" runs — 116.1030s vs
116.2147s, a 0.1% difference), because this container's overlay
filesystem has poor per-call `fsync` latency. This is a property of the
container, not of PackFS or of a bare disk — flagged in the environment
section above precisely so it isn't misread as "PackFS's journal must be
@@ -454,8 +477,8 @@ now marked where "Resolution" above supersedes it.
which is a real, inherited cost on this kind of storage — not a
PackFS-specific one. Unaffected by the index rewrite (the journal's cost
is per-write `fsync` latency, not index maintenance).
- **Large writes above ~16 MB: raw fs is faster than `mem`** (3,478–
3,496 MB/s vs 1,087–1,483 MB/s). Section 5.7's buffer-growth discipline
- **Large writes above ~16 MB: raw fs is faster than `mem`** (3,450–
3,656 MB/s vs 1,091–1,458 MB/s). Section 5.7's buffer-growth discipline
(allocate new, copy old + new, publish, retire old — never realloc in
place) means every capacity doubling re-copies everything written so
far; a plain unsynced `write()` to a real file only ever appends new