Add PackFS v0: statically linked in-process VFS implementing concept.md
Implements the core design: a mount table published as an atomically- swapped snapshot; mem/dir/pack/overlay backends; copy-on-write overlay with copy-up and whiteout deletion; a checksummed append journal; compaction with exact-duplicate elimination; single-writer/wait-free- reader concurrency with a structural/content write split; openat2/ Landlock path containment for dir mounts; and load-time pack integrity validation. Zero required third-party dependencies. Sanitizer testing (ASan/UBSan) caught and led to fixing a genuine heap-use-after-free in the snapshot-reclamation path: the textbook "load pointer, then increment its refcount" pattern left a gap a concurrent writer could free through. Closed with a small reclaim_gate rwlock, documented in internal.h and CLAUDE.md since it's a pattern every refcounted structure in the codebase now follows. zip/tar import/export backends, recommended in concept.md Section 11, will not be built — a permanent project decision recorded in CLAUDE.md since concept.md itself is frozen and cannot be edited to reflect it. Includes a runnable demo (examples/demo.c, `make demo`) exercising the library end to end and proving cross-run persistence through the pack file, plus open-source scaffolding: MIT license, README, CONTRIBUTING, and a CI workflow running the test suite under ASan/UBSan/TSan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,168 @@
|
||||
# PackFS
|
||||
|
||||
A statically linked, in-process virtual file system for C. PackFS treats a
|
||||
shipped file tree as an immutable **pack** image, derives writability from a
|
||||
`mem` or `dir` upper layer through a copy-on-write overlay, and confines
|
||||
`zip`/`tar` to import/export roles rather than treating either as a live,
|
||||
writable store.
|
||||
|
||||
The full design rationale — why this shape, what alternatives were rejected,
|
||||
and the concurrency, path-containment, and integrity models this
|
||||
implementation follows — is specified in [`concept.md`](concept.md), which is
|
||||
frozen (see [`CLAUDE.md`](CLAUDE.md)) and is the authoritative source for
|
||||
every design decision below. This README documents the implementation that
|
||||
followed from it, not a restatement of the rationale.
|
||||
|
||||
## Status
|
||||
|
||||
This is an initial, partial implementation of the spec, not a complete one.
|
||||
|
||||
**Implemented and tested:** mount table, `mem`/`dir`/`pack`/overlay backends,
|
||||
copy-up, whiteouts, compaction, an append journal, path containment, and
|
||||
pack integrity validation.
|
||||
|
||||
**Will never be built, by explicit project decision:** the `zip` (miniz) and
|
||||
`tar` (USTAR) import/export backends. `concept.md` Section 11 recommends
|
||||
them, but that recommendation is superseded — see `CLAUDE.md`, "Project
|
||||
decisions that supersede concept.md." `backend_overlay_new` reads and writes
|
||||
this project's own pack format exclusively; there is no zip/tar support and
|
||||
none is planned. Do not open an issue or PR adding one.
|
||||
|
||||
**Deliberately out of scope for v0** (`concept.md` Section 10, not gaps):
|
||||
full POSIX semantics, enforced permissions/symlinks/hard links,
|
||||
cross-process concurrency (design specified in Section 5.6, unimplemented),
|
||||
and content-defined chunking/delta compression.
|
||||
|
||||
**Tested on Linux only**, in one environment. The `dir`-mount containment
|
||||
fallback path for kernels without `openat2` (Section 6.3) is implemented but
|
||||
has not been exercised on such a kernel, nor on macOS or Windows.
|
||||
|
||||
## Building
|
||||
|
||||
Zero required third-party dependencies — only a C11 compiler, `make`, and
|
||||
`pthread` (Section 11.1 of `concept.md` makes this a hard constraint, not a
|
||||
preference).
|
||||
|
||||
```sh
|
||||
make # builds libpackfs.a and libpackfs.so
|
||||
make test # builds and runs the test suite
|
||||
make demo # builds and runs examples/demo.c — see "Try it" below
|
||||
make install # installs to $PREFIX (default /usr/local)
|
||||
```
|
||||
|
||||
## Try it
|
||||
|
||||
`examples/demo.c` is a small, runnable, human-readable program — not another
|
||||
automated test — that exercises the library end to end and prints what it
|
||||
did at each step: a pack-backed overlay (write, read, `mkdir`, `readdir`,
|
||||
`stat`, a copy-up-then-whiteout delete, compaction via `vfs_sync`), and a
|
||||
sandboxed `dir` mount that demonstrates a `../../../etc/passwd` escape
|
||||
attempt being rejected. Run `make demo` twice in a row: the second run's
|
||||
first `readdir` shows the first run's files, proving that compaction and
|
||||
reload actually persist data through the pack file, not just within one
|
||||
process's lifetime.
|
||||
|
||||
`openat2`/Landlock support (Section 6) is detected automatically at compile
|
||||
time via `<sys/syscall.h>`; on kernels or platforms without them, `dir`
|
||||
mounts fall back to the weaker, documented residual-risk posture described
|
||||
in `concept.md` Section 6.3 rather than failing to build.
|
||||
|
||||
## Quick example
|
||||
|
||||
```c
|
||||
#include <packfs.h>
|
||||
|
||||
Vfs *v = vfs_new();
|
||||
|
||||
/* a plain in-memory writable tree */
|
||||
Backend *mem = backend_mem_new();
|
||||
vfs_mount(v, "/", mem);
|
||||
|
||||
int err = 0;
|
||||
VfsFile *f = vfs_open(v, "/hello.txt", VFS_O_WRONLY | VFS_O_CREAT, &err);
|
||||
vfs_write(f, "hello", 5);
|
||||
vfs_close(f);
|
||||
|
||||
vfs_free(v);
|
||||
backend_free(mem);
|
||||
```
|
||||
|
||||
A shipped pack with a writable overlay on top:
|
||||
|
||||
```c
|
||||
Backend *mem = backend_mem_new();
|
||||
int err = 0;
|
||||
Backend *ov = backend_overlay_new("assets.pack", mem, &err); /* loads assets.pack if it exists */
|
||||
vfs_mount(v, "/", ov);
|
||||
|
||||
/* ... reads served straight from the pack; writes copy-up into mem ... */
|
||||
|
||||
vfs_sync(v, "/"); /* compacts the overlay into a fresh assets.pack (Section 4.1, 5.3) */
|
||||
```
|
||||
|
||||
A sandboxed host directory:
|
||||
|
||||
```c
|
||||
int derr = 0;
|
||||
Backend *dir = backend_dir_new("/var/lib/myapp/data", &derr);
|
||||
vfs_mount(v, "/data", dir);
|
||||
/* every lookup under /data is contained to that directory (Section 6.2);
|
||||
* ".." and symlink escapes are rejected, not merely discouraged. */
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
The public API is [`include/packfs.h`](include/packfs.h); every function and
|
||||
struct is documented there with a pointer to the `concept.md` section that
|
||||
specifies its behavior. In outline:
|
||||
|
||||
- `vfs_new` / `vfs_free` — a `Vfs` owns a mount table, nothing else.
|
||||
- `backend_mem_new` / `backend_dir_new` / `backend_overlay_new` — construct a
|
||||
backend; `vfs_mount`/`vfs_unmount` attach or detach it at a path prefix.
|
||||
- `vfs_open` / `vfs_read` / `vfs_write` / `vfs_close` — file I/O.
|
||||
- `vfs_stat` / `vfs_readdir` / `vfs_mkdir` / `vfs_unlink` / `vfs_rename` —
|
||||
metadata and namespace operations.
|
||||
- `vfs_sync` — compacts an overlay mount into a fresh pack.
|
||||
- `vfs_harden_process_with_landlock` — optional, opt-in, process-wide
|
||||
Landlock confinement to the process's current `dir` mounts. Deliberately
|
||||
**not** applied automatically by `vfs_mount` (Section 6.2 explains why:
|
||||
Landlock restrictions are irreversible and process-wide, which would be a
|
||||
surprising side effect for an embeddable library to trigger on its own).
|
||||
|
||||
## Concurrency
|
||||
|
||||
Single-writer, wait-free-reader (Section 5): readers never take a lock and
|
||||
never observe a write in progress; structural changes (create/unlink/rename/
|
||||
mkdir/mount/unmount) publish a new immutable snapshot via one atomic pointer
|
||||
swap; ordinary content writes to an already-existing file update a per-entry
|
||||
cell directly and never touch the snapshot. `mem`-backed buffer growth
|
||||
never mutates a buffer address a reader might be reading (Section 5.7):
|
||||
growth always allocates a new buffer and publishes it, never reallocates in
|
||||
place. `tests/test_concurrency.c` exercises this under concurrent reader and
|
||||
writer threads, and the suite is regularly run under ThreadSanitizer and
|
||||
AddressSanitizer (see `.github/workflows/ci.yml`).
|
||||
|
||||
## Security
|
||||
|
||||
`dir` mounts are capability-scoped (Section 6.2): a mount holds an already-open
|
||||
directory file descriptor, not a path string, and every lookup beneath it is
|
||||
resolved with `openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS)` on Linux 5.6+,
|
||||
which atomically rejects `..` and symlink escapes in one kernel call. Where
|
||||
that syscall is unavailable, containment falls back to per-component
|
||||
`O_NOFOLLOW` resolution — weaker, and documented as such (Section 6.3), not
|
||||
silently assumed equivalent. Pack files are treated as untrusted input unless
|
||||
they came from this process's own compaction: every on-disk offset is
|
||||
bounds-checked and the pack's checksum is verified before any of it is
|
||||
trusted (Section 7). See `tests/test_dir.c` for a containment regression test
|
||||
and `tests/test_pack_overlay.c` for a corrupted-pack rejection test.
|
||||
|
||||
## Contributing
|
||||
|
||||
See [`CONTRIBUTING.md`](CONTRIBUTING.md). Read `concept.md` and `CLAUDE.md`
|
||||
first — they are the project's actual specification and its enforced
|
||||
documentation standard, respectively, and every design decision in the code
|
||||
traces back to one of them.
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [`LICENSE`](LICENSE).
|
||||
Reference in New Issue
Block a user