packfs/BENCH.md

# PackFS vs. the host filesystem: benchmark results

**Status: the O(n²) bulk create/unlink finding this document originally
reported has been fixed.** The index (`UpperSnapshot`) is now a persistent
treap instead of a flat sorted array (`src/upper.c`; see CLAUDE.md's "Known
performance characteristics"). This document keeps the original run's
numbers as the historical record of the problem — per this project's own
documentation standard, a correction extends the record rather than erasing
it — and adds the post-fix run's numbers alongside them. **Read "Resolution"
below before drawing conclusions from the "Original results" section**,
which describes a version of the index this codebase no longer has.

This document reports the results of `bench/bench.c` (`make bench`), each
run once on the environment described below. These are single runs, not a
statistically-averaged series — treat the numbers as illustrative of *shape*
(which operations are faster, by roughly what factor, and why) more than as
precise absolute figures for a different machine. The methodology, including
exactly what each backend label measures, is documented in `bench/bench.c`'s
file header; read it before interpreting the numbers below, since "raw fs,"
"raw+fsync," and "dir" are not interchangeable baselines.

## Environment

- Host: containerized, 12 logical CPUs (AMD Ryzen 5 3600), running under an
  `overlay` root filesystem (`overlay` on `overlay`, per `mount`) — there is
  no separate tmpfs at `/tmp` in this environment; every "raw fs" and `dir`
  number below is against that container overlay filesystem, not a bare
  disk or tmpfs. This matters most for the `fsync` numbers (see below).
- `N_SMALL` = 20,000 files × 128 bytes for the metadata-heavy suite;
  `N_DIRS` = 4,000; concurrency = 8 threads × 4,000 ops; large files up to
  64 MB. Full parameters are in `bench/bench.c`.
- Both runs took several minutes of wall-clock time, almost entirely spent
  in the single `raw+fsync` 20,000-file create test (~116s in both runs —
  this cost is on the host's storage, not PackFS, and is identical before
  and after the fix, as expected).

## Resolution

The original run below found bulk sequential `create`/`unlink`/`mkdir` on
`mem`/`dir` to be O(n²) in file count, because every structural write copied
the entire sorted entry array before publishing the next snapshot
(Section 5.3). `concept.md` itself names the trigger condition for
reconsidering that design — "a persistent (structurally shared) tree
structure is not required until this assumption is empirically violated" —
and the original run was exactly that violation, measured rather than
hypothesized.

The index was rewritten as a **persistent treap** (`src/upper.c`): a
structural write now builds only the O(log n) nodes on the path to the
change, sharing every other node (refcounted) with the snapshot it was
built from, instead of copying all n entries. The choice of treap over a
persistent AVL/red-black/weight-balanced tree, and the reasoning behind it,
is documented in `src/upper.c`'s comment above `struct TreapNode` and cited
there: Seidel & Aragon, "Randomized Search Trees" (1996), for the O(1)
amortized rotation count on both insert and delete (persistent balanced
trees that need O(log n) rotations pay a real, repeated allocation cost per
rotation, which a treap avoids); Liljenzin, "Confluently Persistent Sets and
Maps" (arXiv:1301.3388), for persistent treaps specifically as an MVCC
snapshot mechanism, which is this exact use case.

**Result, measured, same methodology as the original finding:**

| Category | Before | After | Speedup |
|---|---|---|---|
| create 20,000 files (mem) | 9.56s / 2,093 ops/s | 0.037s / 537,103 ops/s | **257x** |
| unlink 20,000 files (mem) | 10.56s / 1,894 ops/s | 0.032s / 625,437 ops/s | **330x** |
| mkdir 4,000 dirs (mem) | 0.336s / 11,896 ops/s | 0.0052s / 771,085 ops/s | **65x** |
| create 20,000 files (dir) | 12.04s / 1,662 ops/s | 1.15s / 17,345 ops/s | **10x** |
| concurrent create+read+unlink (mem) | 36.73s / 2,614 ops/s | 0.28s / 341,972 ops/s | **131x** |

`mem` now *beats* raw fs (not merely "no longer loses to it") at every one
of these: create is 27x faster than raw fs (was 9.5x slower), unlink is 15x
faster (was 22x slower), and the concurrent mixed workload is 22x faster
(was 6x slower) — the single-writer design's "optimizes for read-heavy
concurrent workloads" caveat from the original analysis (below) no longer
needs to exclude bulk-write workloads, because the write side is no longer
the bottleneck it was.

**The complexity-class change is confirmed quantitatively, the same way the
original O(n²) finding was** — not just "it got faster," which an unrelated
constant-factor optimization could also produce: `mkdir` at N=4,000 vs.
`create` at N=20,000 (same structural-write mechanism, 5x the N) now shows a
**7.15x** slowdown. O(n) predicts 5.0x; O(n log n) predicts 5.97x; the old
O(n²) regime predicted, and measured, 25–28.4x. 7.15x lands close to the
O(n log n) prediction (some excess is expected — content writes'
buffer-allocation and per-file host effects differ between `create`, which
writes bytes, and `mkdir`, which does not) and nowhere near quadratic. The
index's asymptotic behavior changed, not merely its constant factor.

Full output of the post-fix run is reproducible with `make bench`; the
correctness of the new index under heavy randomized structural churn
(non-sequential creates/deletes/renames, nested directories, thousands of
entries, cross-checked against an independent reference model — not just
"the benchmark ran fast without crashing") is `tests/test_index_stress.c`,
part of `make test`.

## Original results (pre-fix; kept as the historical record — see "Resolution" above)

| Category | Backend | Time | Throughput | MB/s |
|---|---|---|---|---|
| create 20,000 files | mem | 9.56s | 2,093 ops/s | 0.3 |
| create 20,000 files | dir | 12.04s | 1,662 ops/s | 0.2 |
| create 20,000 files | raw fs | 1.01s | 19,836 ops/s | 2.4 |
| create 20,000 files | raw+fsync | 116.10s | 172 ops/s | 0.0 |
| read 20,000 files | mem | 0.007s | 2,798,731 ops/s | 341.6 |
| read 20,000 files | dir | 0.167s | 119,468 ops/s | 14.6 |
| read 20,000 files | raw fs | 0.161s | 124,122 ops/s | 15.2 |
| stat 20,000 files | mem | 0.006s | 3,223,599 ops/s | — |
| stat 20,000 files | dir | 0.152s | 131,937 ops/s | — |
| stat 20,000 files | raw fs | 0.070s | 284,465 ops/s | — |
| readdir (20,000 entries) | mem | 0.0037s | 5,397,828 /s | — |
| readdir (20,000 entries) | dir | 0.0028s | 7,058,003 /s | — |
| readdir (20,000 entries) | raw fs | 0.0071s | 2,831,904 /s | — |
| unlink 20,000 files | mem | 10.56s | 1,894 ops/s | — |
| unlink 20,000 files | dir | 10.19s | 1,964 ops/s | — |
| unlink 20,000 files | raw fs | 0.48s | 41,823 ops/s | — |
| mkdir 4,000 dirs | mem | 0.336s | 11,896 ops/s | — |
| mkdir 4,000 dirs | dir | 0.511s | 7,829 ops/s | — |
| mkdir 4,000 dirs | raw fs | 0.159s | 25,143 ops/s | — |
| write 1 / 16 / 64 MB | mem | — | — | 6,687 / 1,158 / 1,130 |
| write 1 / 16 / 64 MB | raw fs (no fsync) | — | — | 2,685 / 3,413 / 3,436 |
| write 1 / 16 / 64 MB | raw+fsync | — | — | 116 / 191 / 718 |
| random-read 20,000 entries | pack (mmap'd) | 0.0085s | 2,350,224 ops/s | 286.9 |
| random-read 20,000 entries | raw fs | 0.177s | 113,198 ops/s | 13.8 |
| compact 20,000 entries to pack | pack | 0.029s | 687,097 ops/s | 83.9 |
| concurrent create+read+unlink (8×4,000×3) | mem | 36.73s | 2,614 ops/s | — |
| concurrent create+read+unlink (8×4,000×3) | raw fs | 6.12s | 15,686 ops/s | — |

## Post-fix results

| Category | Backend | Time | Throughput | MB/s |
|---|---|---|---|---|
| create 20,000 files | mem | 0.037s | 537,103 ops/s | 65.6 |
| create 20,000 files | dir | 1.153s | 17,345 ops/s | 2.1 |
| create 20,000 files | raw fs | 1.010s | 19,805 ops/s | 2.4 |
| create 20,000 files | raw+fsync | 116.098s | 172 ops/s | 0.0 |
| read 20,000 files | mem | 0.0085s | 2,364,410 ops/s | 288.6 |
| read 20,000 files | dir | 0.160s | 124,933 ops/s | 15.3 |
| read 20,000 files | raw fs | 0.159s | 125,459 ops/s | 15.3 |
| stat 20,000 files | mem | 0.0073s | 2,723,001 ops/s | — |
| stat 20,000 files | dir | 0.151s | 132,517 ops/s | — |
| stat 20,000 files | raw fs | 0.070s | 287,063 ops/s | — |
| readdir (20,000 entries) | mem | 0.0042s | 4,816,418 /s | — |
| readdir (20,000 entries) | dir | 0.0039s | 5,128,581 /s | — |
| readdir (20,000 entries) | raw fs | 0.0070s | 2,840,890 /s | — |
| unlink 20,000 files | mem | 0.032s | 625,437 ops/s | — |
| unlink 20,000 files | dir | 0.515s | 38,864 ops/s | — |
| unlink 20,000 files | raw fs | 0.492s | 40,628 ops/s | — |
| mkdir 4,000 dirs | mem | 0.0052s | 771,085 ops/s | — |
| mkdir 4,000 dirs | dir | 0.177s | 22,551 ops/s | — |
| mkdir 4,000 dirs | raw fs | 0.137s | 29,251 ops/s | — |
| write 1 / 16 / 64 MB | mem | — | — | 2,937 / 1,483 / 1,087 |
| write 1 / 16 / 64 MB | raw fs (no fsync) | — | — | 2,463 / 3,478 / 3,496 |
| write 1 / 16 / 64 MB | raw+fsync | — | — | 42 / 702 / 776 |
| random-read 20,000 entries | pack (mmap'd) | 0.0082s | 2,445,144 ops/s | 298.5 |
| random-read 20,000 entries | raw fs | 0.166s | 120,709 ops/s | 14.7 |
| compact 20,000 entries to pack | pack | 0.025s | 814,864 ops/s | 99.5 |
| concurrent create+read+unlink (8×4,000×3) | mem | 0.281s | 341,972 ops/s | — |
| concurrent create+read+unlink (8×4,000×3) | raw fs | 6.298s | 15,243 ops/s | — |

Everything not involving bulk `mem`/`dir` structural writes (reads, `stat`,
`readdir`, large sequential I/O, pack random-access, compaction) is
unchanged within normal run-to-run noise, as expected — the treap rewrite
only touches the structural-write path (Section 5.3), not content reads,
content writes, or the pack format.

## Analysis

The bullets below are preserved from the original run for the categories
the fix did not change (the "wins" and the non-structural-write "losses"),
with the historical O(n²) explanation kept as the record of what was found
and superseded by "Resolution" above where it no longer applies.

### Where PackFS wins clearly

- **Reads of small files: 20–23x faster than both raw fs and the `dir`
  backend** (2.4M ops/s vs ~125K ops/s). No syscall per read — `mem`
  reads are an expected-O(log n) treap lookup plus a `memcpy` out of an
  already-resident buffer (Section 5.3, 9.1).
- **`stat`: ~10x faster than raw fs.** Same reason — no syscall, and the
  index is sorted for expected-O(log n) lookup rather than requiring a
  directory entry scan.
- **Random-access reads against a compacted pack: ~20x faster than raw fs**
  (2.4M ops/s vs 121K ops/s), because the pack is one `mmap`'d file with a
  binary-searchable index (Section 9.1), against 20,000 individual
  `open`/`read`/`close` syscall triples on the raw-fs side. This is the
  single result that most directly validates the architecture's stated
  purpose — Section 3.2's claim that "random access is a flat table
  lookup, not a linear scan or a directory-parse-then-seek" — under an
  actual measured workload, not just by construction.
- **Compaction throughput**: ~100 MB/s / ~815K entries/s to serialize the
  live tree into a fresh pack (Section 4.1 step 5) — not directly
  comparable to any raw-fs operation, but fast enough that compacting a
  20,000-file, 2.5 MB tree is not a practically-felt pause (25ms).
- **Bulk create/unlink/mkdir: now 10–330x faster than the pre-fix `mem`/
  `dir` numbers, and (for `mem`) 15–27x faster than raw fs** — see
  "Resolution" above; this used to be PackFS's clearest loss and is now
  among its clearest wins.
- **Concurrent mixed create+read+unlink: now 22x faster than raw fs**
  (was 6x *slower* before the fix) — see "Resolution."

### Where raw fs still wins, and why that's expected, not a bug

- **`dir` backend `stat` is ~2.2x *slower* than raw fs `stat`** (133K ops/s
  vs 287K ops/s) — because `pfs_dir_statat` (Section 6.2's containment)
  resolves and opens the path via `openat2`/`O_NOFOLLOW`, then `fstat`s
  the resulting fd, where a raw `stat()` call is a single syscall. This is
  the direct, measured cost of path containment on the metadata path,
  separate from and smaller than the containment cost paid on `open`
  itself (which raw fs pays an equivalent single-syscall cost for anyway).
  Unaffected by the index rewrite.
- **`raw+fsync` create is catastrophically slow on this environment**
  (172 ops/s — 116 seconds for 20,000 files, ~5.8ms per `fsync`), because
  this container's overlay filesystem has poor per-call `fsync` latency.
  This is a property of the container, not of PackFS or of a bare disk —
  flagged in the environment section above precisely so it isn't
  misread as "PackFS's journal must be this slow too." The journal
  (Section 4.4) does call `fsync` once per journaled write for the same
  durability reason raw `fsync` is slow here, which is a real, inherited
  cost on this kind of storage — not a PackFS-specific one. Unaffected by
  the index rewrite (the journal's cost is per-write `fsync` latency, not
  index maintenance).
- **Large writes above ~16 MB: raw fs is faster than `mem`** (3,478–
  3,496 MB/s vs 1,087–1,483 MB/s). Section 5.7's buffer-growth discipline
  (allocate new, copy old + new, publish, retire old — never realloc in
  place) means every capacity doubling re-copies everything written so
  far; a plain unsynced `write()` to a real file only ever appends new
  pages to the page cache, never re-copying prior ones. This is entirely
  independent of the index structure (Section 5.7's buffer-growth
  discipline governs a `MutCell`'s content buffer, not the index the
  treap rewrite replaced) and is unaffected by this fix — the safety
  property Section 5.7 requires (no reader can ever see a freed buffer)
  still has a real, quantifiable cost for very large sequential writes,
  traded for correctness under concurrent access that a plain in-place
  realloc would not have.

## Reproducing

```sh
make bench
```

Takes several minutes on a similarly slow-`fsync` environment, dominated
by the `raw+fsync` create test; expect well under a minute on a host with
normal disk or tmpfs `fsync` latency.