packfs/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository status

This repository contains a working v0 implementation of the concept.md specification: include/packfs.h (public API), src/*.c (implementation), tests/test_*.c (test suite), and open-source project scaffolding (README.md, LICENSE, CONTRIBUTING.md, CHANGELOG.md, SECURITY.md, packfs.pc.in, .gitea/workflows/ci.yml — this project is hosted on Gitea, not GitHub), alongside the frozen concept.md and this file. Every file under src/ and include/ carries an SPDX-License-Identifier: MIT tag; include/packfs.h's PACKFS_VERSION_* macros are the single source of truth for the project's version — pfs_version() (runtime) and packfs.pc (generated by make install) are both derived from them, never maintained separately.

Build, test, and lint commands

make            # builds libpackfs.a and libpackfs.so (zero required third-party deps, Section 11.1)
make test       # builds and runs every tests/test_*.c
make clean
make install PREFIX=/some/prefix

A single test: make build/test_mem && ./build/test_mem (substitute any test_* basename). There is no separate lint step — the build itself uses -Wall -Wextra -Wpedantic, and a warning introduced by a change is a build failure, not something to leave in place.

Sanitizer builds are not wired into make test (they need per-file compilation with sanitizer flags plus -D_GNU_SOURCE -Iinclude -Isrc); see .gitea/workflows/ci.yml (Gitea Actions) for the exact invocation, which CI runs on every push. Any change to upper.c, overlay.c, or vfs.c must be verified under -fsanitize=address,undefined and -fsanitize=thread before being considered done — this is not a formality: exactly this process caught a real use-after-free in the snapshot-reclamation logic during initial development (see the reclaim_gate note below), which the plain build and even repeated plain test runs never surfaced.

If your environment cannot run ThreadSanitizer at all, say so rather than skipping it silently. TSan needs personality(ADDR_NO_RANDOMIZE) to disable ASLR for itself; some sandboxes block that syscall outright, in which case every TSan build fails identically (FATAL: ThreadSanitizer: unexpected memory mapping), including a trivial unrelated pthread program — that is the confirming test, not a PackFS-specific symptom. In that situation, ASan/UBSan are still run and still matter (they are what actually caught the reclaim_gate bug above), but they do not perform TSan's happens-before race analysis and are not a substitute for it; a change is TSan-verified only once it has passed on a machine or CI run that can actually execute it, not merely because ASan/UBSan passed.

A related but separate flake affects ASan/UBSan themselves in that kind of sandbox, not just TSan: a sanitizer-built test binary can non-deterministically fail to start with AddressSanitizer:DEADLYSIGNAL, occasionally as an unbounded repeating loop rather than a single line — a sandbox startup race, confirmed by it hitting different, unrelated binaries across repeated runs, each of which then passes cleanly on retry. See CONTRIBUTING.md's workflow section for the full description and the required mitigation (timeout-wrap sanitizer runs in such an environment; treat DEADLYSIGNAL alone, without an actual ERROR: AddressSanitizer or runtime error: string, as inconclusive and re-run rather than as a finding).

Code architecture

• include/packfs.h — the entire public API.
• src/internal.h — every internal type shared across .c files; read this first when touching implementation code.
• src/vfs.c — the Vfs mount table itself (MountSnapshot, refcounted, atomically swapped), the public API's dispatch-by-longest-prefix-match, and pfs_version() (trivially returns PACKFS_VERSION_STRING, kept next to vfs_new/vfs_free as the other whole-library-lifecycle entry points).
• src/upper.c — the writable layer shared by mem, dir, and the overlay's upper side: UpperSnapshot/UpperEntry/MutCell, the persistent-treap index (TreapNode and treap_*/node_* — see "Known performance characteristics" below for why it's a treap and not a flat array), the structural-write functions (upper_create/upper_mkdir/upper_remove/upper_rename/upper_copy_up), the content-write fast path (upper_cell_read/upper_cell_write), the range-query API (upper_visit_range, used by readdir in both this file and overlay.c), and the standalone mem/dir Backend (backend_mem_new/backend_dir_new).
• src/overlay.c — composes a Pack (lower, read-only) with an UpperStore (upper): copy-up, whiteouts, the merged view (overlay_stat/overlay_readdir), the append journal, and vfs_sync's compaction.
• src/pack.c — the on-disk pack format: load-time integrity validation, binary-search lookup/range-query, the compaction writer (atomic rename, exact-duplicate elimination via a DedupSlot open-addressing hash table over (hash, size) plus a memcmp verification before ever reusing a data_off — see "Known performance characteristics" below for why it isn't a linear scan), and the standalone read-only pack Backend (backend_pack_new) for mounting a pack with no writable upper layer at all.
• src/containment.c — dir-mount path containment (openat2/O_NOFOLLOW fallback) and opt-in Landlock hardening.
• src/path.c — virtual-namespace ./.. canonicalization.
• src/hash.c — FNV-1a64, used for pack checksums and compaction dedup.
• tests/ — one binary per concern (test_mem, test_dir, test_pack_overlay, test_concurrency, test_index_stress — the treap's correctness under thousands of randomized structural operations, cross-checked against an independent reference model); test_harness.h is a small assertion-macro header, not a framework, consistent with the zero-dependency constraint.

One architectural fact spans every mutable structure and is easy to miss reading any single file in isolation: MountSnapshot (vfs.c), UpperSnapshot (upper.c), and a mem-backed MutCell's buffer (upper.c, Section 5.7) are all retired through the same two-step pattern — publish the replacement via atomic_store_explicit(..., memory_order_release), then free the superseded object only under a dedicated reclaim_gate rwlock's write side, while every acquirer takes that same gate's read side around its own load-then-increment. This exists because a plain "load a pointer, then atomically increment its refcount" leaves a real gap between those two steps in which a concurrent writer can free the very object being acquired — confirmed by ASan as an actual heap-use-after-free during development, not a theoretical concern. See the comment on UpperStore.reclaim_gate in internal.h for the full reasoning. Any new refcounted, concurrently-reclaimed structure added to this codebase needs the same gate, not just an atomic pointer and a naive refcount. The one exception, and the reason it's an exception rather than a hole: TreapNode's own per-node refcounting (upper.c) does not need its own reclaim_gate, because nothing ever acquires a TreapNode* independently the way upper_acquire acquires an UpperSnapshot* — a reader only ever reaches tree nodes by walking s->root after already holding a valid UpperSnapshot reference, which transitively keeps the whole tree alive for the reader's purposes; node_ref/node_unref are called only by the single writer (under writer_lock) and by upper_release's cascade (already inside reclaim_gate's protection at the snapshot level), never by a reader racing a writer the way the snapshot pointer itself is raced.

What this project is

PackFS is the design for a statically linked, in-process virtual file system (VFS) written in C, intended for embedding in sandboxed, agent-driven, TUI, or game-shaped programs. concept.md is the full specification (architecture, write model, concurrency model, path-containment model, pack-integrity validation, C object-model sketch, on-disk pack format, explicit v0 exclusions, and a self-evaluation) — read it in full before doing any implementation work here, rather than relying on the summary below.

Documentation standard (enforced)

Every document in this repository — this file included — is written and revised under the following rule. It was established after concept.md was rewritten into its current form, and that rewrite is the reference example: when in doubt about whether a piece of prose conforms, compare it against concept.md.

• Register. Formal, precise, third-person or directly-stated-constraint prose. No colloquialisms, no rhetorical asides, no filler sections ("Tips," "Support," "Common Tasks") invented for their own sake.
• Structure scales to content, not the other way around. A specification-length document (concept.md) earns full academic structure — Abstract, Problem Statement, numbered body sections, Evaluation, Conclusion, References. A short instructions file (this one) states its rules directly without manufacturing sections it doesn't need.
• Constraints are labeled by strength. A requirement an implementation must satisfy is stated as such explicitly ("hard constraint," "must," "required"). A recommendation that could reasonably be revisited is stated as such ("recommended," "a legitimate simpler fallback"). The two are never left ambiguous relative to each other.
• Revisions never delete information. An error, once caught, is corrected and the correction is explained in place; prior content is extended or superseded, not silently dropped. This applies to every living document in this repository; concept.md is the one exception, having been declared frozen rather than living — see "concept.md is immutable" below.
• External claims are cited, not asserted. Where a decision leans on prior art or an established system (e.g., LMDB, OverlayFS), it is attributed with a references entry rather than presented as self-evident.
• Every specification-length document carries its own self-evaluation. A short methodology note, an assessment table scored per relevant category, and one final overall grade with justification — see concept.md Section 12 for the pattern. This file's own self-evaluation is at the bottom.

concept.md is immutable

As of this revision, concept.md is frozen. It is not to be edited, rewritten, appended to, or otherwise modified by any future session, for any reason — including to fix a typo, to close a newly-found gap, or to keep it "in sync" with a later decision. It stands as the fixed record of the design as reasoned through its own research and review cycles; the self-evaluation and grade inside it describe that fixed text, and would themselves become inaccurate if the text under them kept moving.

This is a deliberate exception to the "revisions never delete information" pattern above, not a stricter version of it: that pattern describes how a living document is corrected in place; a frozen document is not corrected in place at all. The correct response to a newly-found gap, error, or improvement in the design is to open a new, separate document (for example, an addendum or a versioned successor) that amends or supersedes concept.md, leaving its text untouched. If such a document is ever created, this file should be updated to point to it alongside concept.md, and to state plainly which of the two is authoritative where they overlap.

The load-bearing constraints summarized below are therefore also frozen in substance: they describe what concept.md says, not a moving target. A future change to the actual design changes what is authoritative — a new document — not what concept.md says happened.

Project decisions that supersede concept.md

concept.md cannot be edited (above), but the project's actual scope has diverged from it in one place, by an explicit, permanent decision — recorded here per this file's own protocol for citing an amendment alongside the frozen text it overrides:

• The zip and tar import/export backends will never be built. concept.md Section 3.1 lists them as candidate backends and Section 11 recommends them ("Zip via miniz as import/export. Tar as snapshot/export"), with Section 11.1 describing a compile-time switch at the zip/tar backend boundary for miniz/libarchive. That recommendation is superseded: this project supports the pack format only. Do not propose, scaffold, stub, or partially implement a zip or tar backend, and do not add miniz or libarchive as a dependency for any reason. backend_overlay_new reads and writes this project's own pack format exclusively. The rest of Section 11's recommendation (core + mem + dir + custom pack + overlay + compaction) stands as written and is what this codebase implements.

Load-bearing constraints from concept.md

These are hard requirements stated in the spec, not stylistic suggestions — any implementation work must respect them:

• Static linking is enforced, not aspirational. The core (vfs_*, mem, dir, pack, overlay, compaction) must build with zero required third-party libraries and zero dynamic loading. concept.md describes miniz/libarchive sitting behind a compile-time switch at the zip/tar backend boundary as an optional addition that must never be required — but per "Project decisions that supersede concept.md" above, that boundary will never actually be built, so this constraint is satisfied trivially: there is no optional dependency at all, required or otherwise.
• Archive formats (zip, tar) are, and will remain, entirely absent from this codebase — not merely import/export skins. The system of record is, and is the only supported format, a custom indexed pack file (immutable image: header + index + blobs + strings). See "Project decisions that supersede concept.md" above.
• Writability comes from a copy-on-write overlay, not from mutating the pack. Pack mounted read-only at a path; mem or dir mounted as the writable upper layer; open(O_RDWR) triggers copy-up. Deleting a lower-layer (pack) entry requires writing a whiteout tombstone in the upper layer — there is no way to delete a pack entry directly, and skipping the whiteout leaves deletion of pack-originated files undefined.
• Compaction and the journal must be crash-safe by construction: compaction writes pack.img.tmp, fsyncs, then atomically rename()s over pack.img (never in place); journal records are length/checksum self-describing so replay stops cleanly at the first torn record; journal "truncation" after compaction means writing the surviving tail to a new file and renaming it over the journal, since a file's front cannot be truncated in place.
• Concurrency model is single-writer, wait-free-reader (LMDB-style MVCC), not a global lock. Live state — upper index, whiteout set, and the mount table — is an immutable, refcounted VfsSnapshot reached through one atomic pointer; readers load it once (wait-free) and never observe an in-progress write; exactly one writer at a time builds the next snapshot copy-on-write and publishes it with a single atomic pointer swap using release/acquire ordering (getting this ordering wrong is a real C11 data race). vfs_mount/vfs_unmount are structural writes like any other — the mount table is not separate global state exempt from this model. Structural writes (create/unlink/rename/mkdir/whiteout/mount/unmount) publish a new snapshot; content writes (bytes into an already-existing file) do not — they update a per-entry size/mtime cell under a per-entry lock instead, so an ordinary vfs_write never requires copying and republishing the whole index. A mem-backed write that grows a file's buffer past its current allocation must allocate a new buffer and publish it with a release store, never realloc the existing address in place — a concurrent reader may otherwise copy out of freed memory, not just a stale value; the old buffer is retired under the same reclamation discipline as a retired snapshot. Compaction is just another writer against a frozen snapshot plus a recorded journal watermark; if writing pack.img.tmp or its fsync fails, compaction aborts and the existing pack + journal are untouched. Known, explicitly-accepted gaps: the per-entry lock covers dir-upper only within this process, not against a second uncoordinated process writing the same host directory; rename-over-a-mapped-pack is safe on POSIX but not guaranteed on Windows; naive refcounting contends under high read concurrency (hazard pointers are the recommended upgrade path, not epoch-based reclamation, because this system's workloads can plausibly stall a reader thread); cross-process concurrency has a specified LMDB-style reader-table design but is not implemented in v0.
• Path containment for dir mounts is a hard requirement, not best-effort. A dir mount is an already-open directory file descriptor (capability-scoped, WASI-style), not a re-resolved path string. On Linux 5.6+, every lookup under it uses openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS) to atomically block .. escapes, symlink escapes, and the canonicalize-then-open TOCTOU window in one kernel call; Landlock (5.13+) is a recommended additional layer. On other platforms, no equivalent atomic primitive is assumed — the fallback (O_NOFOLLOW per component + canonicalize-and-verify) is an accepted, explicitly weaker residual-risk posture, not a claim of parity. The VFS core additionally canonicalizes ./.. in the virtual namespace itself, before any backend is reached, so a crafted path cannot escape a mount's prefix even when no host directory is involved.
• A pack file is untrusted input unless it came from this run's own compaction. Every index entry's data_off + size and name_off must be bounds-checked against the actual file/strings-region length before being trusted by vfs_open/vfs_stat/vfs_readdir; a failed check invalidates the whole pack load, it is not skipped silently. An externally-sourced pack additionally needs a checksum verified at load, the same self-checking-record principle already required of the journal.
• The on-disk and in-memory indexes are kept sorted by full path, not insertion order, so exact lookups are a binary search and vfs_readdir is a bounded range query (two binary searches), not a linear scan.
• An empty directory needs an explicit index entry. vfs_mkdir writes a zero-size entry with a reserved directory-marker bit in mode, otherwise a directory with no files in it has nothing referencing its path and vanishes across compaction. The marker is superseded (not deleted) once any file exists under that path.
• mode is stored/restored but not enforced. Permission bits, symlinks, and hard links have no access-control or link semantics in v0 — mode round-trips through compaction (including the directory-marker bit above) but nothing in the VFS core interprets it as a security boundary.
• Explicitly out of scope for v0: FUSE as the first backend, invoking libarchive per-read(), full POSIX semantics/locks/sockets/mmap of virtual files, any single format trying to double as initrd + game pak + user home, implementing the cross-process reader-table design (specified, not built), content-defined chunking/delta compression in the pack format (only exact-duplicate elimination during compaction is in scope), and enforced permissions/symlinks/hard links.

concept.md will not be updated again (see "concept.md is immutable" above), so the constraints above will not drift out from under it. If a future document amends or supersedes part of the design, update this section to cite that document alongside concept.md — without editing concept.md itself.

Known performance characteristics

bench/bench.c (make bench) measures PackFS against the host filesystem; full results and analysis are in BENCH.md. The one finding here that future work on this codebase needs to know without re-running the benchmark:

• RESOLVED: bulk sequential create/unlink/mkdir on mem/dir was O(n²) in the number of entries — confirmed empirically (BENCH.md), then fixed, not just worked around. The index (UpperSnapshot) was a flat sorted array; every structural write copied it in full before publishing the next snapshot (Section 5.3's single-writer model), and at 20,000 sequential creates this measured ~9–22x slower than the raw host filesystem. concept.md Section 5.3 names the exact trigger for reconsidering that design — "a persistent (structurally shared) tree structure is not required until this assumption is empirically violated" — and that trigger was hit. The index is now a persistent treap (src/upper.c, struct TreapNode and the functions around it — see the citations in that comment, Seidel & Aragon 1996 and Liljenzin arXiv:1301.3388, for why a treap and not a persistent AVL/red-black/weight-balanced tree): a structural write now copies only the O(log n) nodes on the path to the change. Post-fix, the same benchmark shows mem create/unlink/mkdir/rmdir 11–590x faster than before (the range is wide because at these post-fix speeds — single-digit milliseconds for 20,000 entries — ordinary run-to-run system jitter has real proportional effect; see BENCH.md's "Resolution" for the two-run comparison that makes this explicit rather than hiding it behind one cherry-picked number), and — the more important, and more stable, comparison — mem create and unlink are now 25–27x faster than raw fs where they used to be 9.5–22x slower; the concurrent mixed workload separately went from 6x slower than raw fs to 21x faster. BENCH.md's "Resolution" section has the full before/after table and the complexity-class re-confirmation (the same mkdir-at-N=4,000-vs-create-at-N=20,000 methodology that found O(n²) now shows a ratio consistent with O(n log n), not O(n²)). A new test, tests/test_index_stress.c, cross-checks the treap's correctness under thousands of randomized (non-sequential) creates/deletes/renames against an independent reference model — read it, not just the benchmark, before touching snapshot_upsert/snapshot_remove/the treap functions again.
• dir-backend stat costs roughly 2x a raw stat() call, because pfs_dir_statat (containment, Section 6.2) resolves and opens the path, then fstats the fd, where raw stat() is one syscall. Expected, and the same containment cost open already pays — recorded so it isn't mistaken for a regression if someone benchmarks it again later.
• mem-backed large sequential writes lose to a plain unsynced host write() above roughly 16 MB, because Section 5.7's buffer-growth discipline (allocate new, copy everything, publish, retire old) re-copies previously-written bytes on every capacity doubling, where the host page cache only ever appends new pages. This is the measured cost of the safety property Section 5.7 requires (no reader ever sees a freed buffer), not an accidental inefficiency.
• RESOLVED: pack_write's compaction-time exact-duplicate elimination (Section 9.2) was O(n²) in entry count, plus a latent correctness bug, both fixed together. The dedup check was a linear scan of every previously-seen (hash, size) pair per entry, comparing only the hash and size — never the actual bytes — before trusting a match, so two different-content entries that happened to collide on (hash, size) under FNV-1a64 (explicitly not collision-resistant, src/hash.c) would have been silently merged, corrupting one of them; this was never observed in practice but was true of the code as written. bench/bench.c's own compaction benchmark never surfaced either problem because its test files are byte-identical, which made every scan match on the first comparison (the scan's best case, not its worst case). Measuring with unique content instead (pack_write called directly, bypassing the journal/fsync path) showed 80,000 entries taking 1.74s, with a 20,000→80,000 (4x N) step showing 15.6x — matching O(n²)'s 16x prediction. Fixed with an open-addressing hash table (DedupSlot, load factor 1/2, linear probing) plus a memcmp verification before ever reusing a data_off, closing both the complexity issue and the correctness gap at once; post-fix, the same 80,000-entry case takes 0.044s (39.6x faster) and the 20,000→80,000 ratio drops to 3.35x, consistent with O(n). BENCH.md's "Resolution #2" has the full before/after numbers. Correctness is covered permanently by tests/test_pack_overlay.c (asserts duplicate-content entries share one data_off and distinct-content entries do not); the performance fix is covered permanently by tests/test_pack_write_perf.c (a regression tripwire against 10,000 unique entries, run as part of make test).
• NOT FIXED, by deliberate decision: vfs.c's mount table (MountSnapshot) is O(n²) in mount count, using the same full-array-copy-per-write pattern the file index used to. Confirmed via bench/bench.c category 10 (500/2,000/8,000 mounts, both 4x-N steps showing 15–20x, matching O(n²)'s 16x prediction, for mount, resolve, and unmount alike). Unlike the file index, this is not being converted to a persistent tree: mount points are created by calls written into a program's own source code, not driven by workload or user data, so realistic mount counts do not reach the scale that made the file index's O(n²) an actual problem — concept.md Section 5.3's own stated trigger for needing a persistent structure ("not required until this assumption is empirically violated") has not been violated here the way it was for the file index. BENCH.md's "Finding: mount table scaling" has the full numbers and reasoning; revisit only if a real use case needs thousands of runtime mount points (e.g. one mount per tenant).

Self-evaluation

Methodology. This file was checked against the repository's actual state (make test passing, including two new tests, test_pack_write_perf and the dedup-verification block added to test_pack_overlay; include/, src/, tests/, bench/ present and matching the description below; confirmed by directory listing and a live build), against concept.md as frozen (for factual consistency of the constraints and architecture summarized above), and against the documentation standard stated in this file. This revision followed a deliberate audit for other instances of the file index's O(n²) shape elsewhere in the codebase — prompted by a request to leave no caveats undocumented — which found two more real issues, not zero: pack_write's compaction dedup (O(n²), plus a latent hash-collision correctness bug, both now fixed) and the mount table (O(n²), confirmed and deliberately left unfixed, with the reasoning recorded). The audit also found and fixed a CI gap the new tests exposed — the CI config's (now .gitea/workflows/ci.yml; at the time of this finding it lived at .github/workflows/ci.yml, before this project's scaffolding was set up for Gitea hosting instead of the GitHub-Actions convention it started from — this repository was never actually hosted on GitHub) sanitizer-build steps never passed -D_GNU_SOURCE when compiling test files, which was harmless while no test included internal.h and became a real link failure once two did (internal.h needs it for pthread_rwlock_t) — and documented a sandbox flake (ASan/UBSan's own DEADLYSIGNAL startup race, distinct from and in addition to the already-documented TSan limitation) observed directly during this audit's own sanitizer runs, in CONTRIBUTING.md and cross-referenced here. Every finding below was confirmed empirically (direct pack_write timing, bench/bench.c's new mount-scaling category, repeated sanitizer runs with timeout), not asserted.

Category	Grade	Notes
Factual accuracy	A	Build/test commands and the code map were verified against a live make test run and the actual file layout, not written from memory of intent; both new performance-characteristics entries were confirmed by direct measurement (pack_write timing, bench/bench.c category 10), not assumed from reading the code.
Adherence to the documentation standard	A	Direct, constraint-labeled prose; the mount-table entry is explicitly labeled "not fixed, by deliberate decision" rather than left ambiguous with the "resolved" entries beside it — constraints and decisions are labeled by strength here exactly as the standard above requires.
Completeness for its purpose	A	Covers repository status, build/test/lint commands, a per-file code map, every hard constraint from concept.md relevant to implementation work, the permanent zip/tar exclusion, the cross-cutting reclaim_gate pattern, both sandbox flakes (TSan's outright failure and ASan/UBSan's intermittent DEADLYSIGNAL), and now three performance findings (one historical-and-resolved, one newly-resolved, one confirmed-and-deliberately-not-resolved) instead of one.
Avoidance of generic or invented content	A	No fabricated sections; the new mount-table entry could have been written as a vague "may not scale to many mounts" hedge but instead states the measured ratios and the specific reason it isn't being fixed, which is the standard the other entries in this section already set.

Overall grade: A. The file states only what is verifiably true of the repository, the spec, and the implementation; labels constraints and decisions by strength; documents the architectural pattern (snapshot reclamation via reclaim_gate) that spans multiple files; and, per this revision's audit, now also documents two further real findings (one fixed, one consciously not) that a less thorough pass would have missed, plus a real CI gap and a real environment flake, both found by direct observation during the same audit rather than left for a future session to rediscover.

A real gap this audit found and fixed, not just documented: backend_pack_new was declared in include/packfs.h and referenced in this file's own architecture map, but was never implemented in src/pack.c — any program calling it would fail at link time. It is now implemented (a standalone, read-only pack Backend; every mutating operation returns VFS_ERR_PERM), covered by a new case in tests/test_pack_overlay.c, and verified under -fsanitize=undefined per this file's own testing rule. This is recorded here because it is exactly the kind of error "document literally all" is supposed to catch: a documentation pass that describes an unimplemented function accurately is still wrong in a way that matters.