retoorandClaude Sonnet 5
0b3b207bc0
bench/bench.c (`make bench`) measures create/read/stat/readdir/unlink/
mkdir on mem, dir, and raw fs; large sequential I/O; random-access pack
reads via mmap vs raw fs; compaction throughput; and 8-thread
concurrent mixed workloads. Results from one full run, with honest
analysis (what each "raw fs" vs "raw+fsync" vs "dir" label actually
measures, so they aren't misread as interchangeable), are in BENCH.md.
The benchmark surfaced a real, quantitatively-confirmed finding, not
just favorable numbers: bulk sequential create/unlink on mem/dir is
O(n^2) in file count (~9-22x slower than raw fs at N=20,000), because
every structural write copies the entire snapshot entry array before
publishing it (Section 5.3). concept.md itself names the exact trigger
condition for reconsidering this ("a persistent structurally-shared
tree structure is not required until this assumption is empirically
violated") — this benchmark is that violation, measured rather than
hypothesized: mkdir at N=4,000 vs create at N=20,000 (same mechanism,
5x the N) shows a 28.4x slowdown, matching the O(n^2) prediction (25x)
far better than O(n) (5x).
Also found: dir-backend stat() costs ~2x raw stat() (open+fstat vs one
syscall, the direct cost of openat2 containment on the metadata path);
mem-backed large writes lose to raw fs above ~16MB (Section 5.7's
buffer-growth discipline re-copies prior writes on every capacity
doubling, the price of never exposing a reader to a freed buffer).
Recorded the O(n^2) finding in CLAUDE.md's new "Known performance
characteristics" section, per the same pattern used for the
reclaim_gate use-after-free discovery, since it's exactly the kind of
fact that would otherwise have to be rediscovered by benchmarking
again from scratch. Linked from README and CONTRIBUTING.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
8.5 KiB
PackFS
A statically linked, in-process virtual file system for C. PackFS treats a
shipped file tree as an immutable pack image and derives writability
from a mem or dir upper layer through a copy-on-write overlay. zip and
tar are not part of this project and never will be — see "Status" below.
The full design rationale — why this shape, what alternatives were rejected,
and the concurrency, path-containment, and integrity models this
implementation follows — is specified in concept.md, which is
frozen (see CLAUDE.md) and is the authoritative source for
every design decision below. This README documents the implementation that
followed from it, not a restatement of the rationale.
Status
This is an initial, partial implementation of the spec, not a complete one.
Implemented and tested: mount table, mem/dir/pack/overlay backends
(including the standalone read-only pack backend, backend_pack_new —
every mutating call against it returns VFS_ERR_PERM), copy-up, whiteouts,
compaction, an append journal, path containment, and pack integrity
validation.
Will never be built, by explicit project decision: the zip (miniz) and
tar (USTAR) import/export backends. concept.md Section 11 recommends
them, but that recommendation is superseded — see CLAUDE.md, "Project
decisions that supersede concept.md." backend_overlay_new reads and writes
this project's own pack format exclusively; there is no zip/tar support and
none is planned. Do not open an issue or PR adding one.
Deliberately out of scope for v0 (concept.md Section 10, not gaps):
full POSIX semantics, enforced permissions/symlinks/hard links,
cross-process concurrency (design specified in Section 5.6, unimplemented),
and content-defined chunking/delta compression.
Tested on Linux only, in one environment. The dir-mount containment
fallback path for kernels without openat2 (Section 6.3) is implemented but
has not been exercised on such a kernel, nor on macOS or Windows.
Building
Zero required third-party dependencies — only a C11 compiler, make, and
pthread (Section 11.1 of concept.md makes this a hard constraint, not a
preference).
make # builds libpackfs.a and libpackfs.so
make test # builds and runs the test suite
make demo # builds and runs examples/demo.c — see "Try it" below
make bench # builds and runs bench/bench.c — see "Benchmarks" below
make install # installs to $PREFIX (default /usr/local)
Try it
examples/demo.c is a small, runnable, human-readable program — not another
automated test — that exercises the library end to end and prints what it
did at each step: a pack-backed overlay (write, read, mkdir, readdir,
stat, a copy-up-then-whiteout delete, compaction via vfs_sync), and a
sandboxed dir mount that demonstrates a ../../../etc/passwd escape
attempt being rejected. Run make demo twice in a row: the second run's
first readdir shows the first run's files, proving that compaction and
reload actually persist data through the pack file, not just within one
process's lifetime.
Benchmarks
bench/bench.c (make bench) measures PackFS against the host filesystem
across metadata operations (create/read/stat/readdir/unlink/mkdir), large
sequential I/O, random-access pack reads, and concurrent mixed workloads.
Full results from one run, with honest analysis of both the wins and the
losses — including a real, quantitatively-confirmed O(n²) cost in bulk
sequential file creation that concept.md Section 5.3 explicitly
anticipated and named the trigger condition for — are in
BENCH.md. Read it before quoting a number from it: what
raw fs vs raw+fsync vs dir each actually measure is not
interchangeable, and the file explains why.
openat2/Landlock support (Section 6) is detected automatically at compile
time via <sys/syscall.h>; on kernels or platforms without them, dir
mounts fall back to the weaker, documented residual-risk posture described
in concept.md Section 6.3 rather than failing to build.
Quick example
#include <packfs.h>
Vfs *v = vfs_new();
/* a plain in-memory writable tree */
Backend *mem = backend_mem_new();
vfs_mount(v, "/", mem);
int err = 0;
VfsFile *f = vfs_open(v, "/hello.txt", VFS_O_WRONLY | VFS_O_CREAT, &err);
vfs_write(f, "hello", 5);
vfs_close(f);
vfs_free(v);
backend_free(mem);
A shipped pack with a writable overlay on top:
Backend *mem = backend_mem_new();
int err = 0;
Backend *ov = backend_overlay_new("assets.pack", mem, &err); /* loads assets.pack if it exists */
vfs_mount(v, "/", ov);
/* ... reads served straight from the pack; writes copy-up into mem ... */
vfs_sync(v, "/"); /* compacts the overlay into a fresh assets.pack (Section 4.1, 5.3) */
A sandboxed host directory:
int derr = 0;
Backend *dir = backend_dir_new("/var/lib/myapp/data", &derr);
vfs_mount(v, "/data", dir);
/* every lookup under /data is contained to that directory (Section 6.2);
* ".." and symlink escapes are rejected, not merely discouraged. */
A shipped pack mounted read-only, no writable layer at all:
int perr = 0;
Backend *ro = backend_pack_new("assets.pack", &perr);
vfs_mount(v, "/assets", ro);
/* vfs_write, vfs_mkdir, vfs_unlink, vfs_rename, vfs_sync against anything
* under /assets all return VFS_ERR_PERM; there is no upper layer to
* absorb a write into. */
API
The public API is include/packfs.h; every function and
struct is documented there with a pointer to the concept.md section that
specifies its behavior. In outline:
vfs_new/vfs_free— aVfsowns a mount table, nothing else.backend_mem_new/backend_dir_new/backend_pack_new/backend_overlay_new— construct a backend;vfs_mount/vfs_unmountattach or detach it at a path prefix.backend_pack_newmounts a pack read-only, with no writable upper layer at all — every mutating call against it returnsVFS_ERR_PERM; wrap the same pack inbackend_overlay_newinstead when writability is wanted.vfs_open/vfs_read/vfs_write/vfs_close— file I/O.vfs_stat/vfs_readdir/vfs_mkdir/vfs_unlink/vfs_rename— metadata and namespace operations.vfs_sync— compacts an overlay mount into a fresh pack.vfs_harden_process_with_landlock— optional, opt-in, process-wide Landlock confinement to the process's currentdirmounts. Deliberately not applied automatically byvfs_mount(Section 6.2 explains why: Landlock restrictions are irreversible and process-wide, which would be a surprising side effect for an embeddable library to trigger on its own).
Concurrency
Single-writer, wait-free-reader (Section 5): readers never take a lock and
never observe a write in progress; structural changes (create/unlink/rename/
mkdir/mount/unmount) publish a new immutable snapshot via one atomic pointer
swap; ordinary content writes to an already-existing file update a per-entry
cell directly and never touch the snapshot. mem-backed buffer growth
never mutates a buffer address a reader might be reading (Section 5.7):
growth always allocates a new buffer and publishes it, never reallocates in
place. tests/test_concurrency.c exercises this under concurrent reader and
writer threads, and the suite is regularly run under ThreadSanitizer and
AddressSanitizer (see .github/workflows/ci.yml).
Security
dir mounts are capability-scoped (Section 6.2): a mount holds an already-open
directory file descriptor, not a path string, and every lookup beneath it is
resolved with openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS) on Linux 5.6+,
which atomically rejects .. and symlink escapes in one kernel call. Where
that syscall is unavailable, containment falls back to per-component
O_NOFOLLOW resolution — weaker, and documented as such (Section 6.3), not
silently assumed equivalent. Pack files are treated as untrusted input unless
they came from this process's own compaction: every on-disk offset is
bounds-checked and the pack's checksum is verified before any of it is
trusted (Section 7). See tests/test_dir.c for a containment regression test
and tests/test_pack_overlay.c for a corrupted-pack rejection test.
Contributing
See CONTRIBUTING.md. Read concept.md and CLAUDE.md
first — they are the project's actual specification and its enforced
documentation standard, respectively, and every design decision in the code
traces back to one of them.
License
MIT — see LICENSE.