packfs/README.md

# PackFS

A statically linked, in-process virtual file system for C. PackFS treats a
shipped file tree as an immutable **pack** image and derives writability
from a `mem` or `dir` upper layer through a copy-on-write overlay. `zip` and
`tar` are not part of this project and never will be — see "Status" below.

The full design rationale — why this shape, what alternatives were rejected,
and the concurrency, path-containment, and integrity models this
implementation follows — is specified in [`concept.md`](concept.md), which is
frozen (see [`CLAUDE.md`](CLAUDE.md)) and is the authoritative source for
every design decision below. This README documents the implementation that
followed from it, not a restatement of the rationale.

## Status

This is an initial, partial implementation of the spec, not a complete one.

**Implemented and tested:** mount table, `mem`/`dir`/`pack`/overlay backends
(including the standalone read-only `pack` backend, `backend_pack_new` —
every mutating call against it returns `VFS_ERR_PERM`), copy-up, whiteouts,
compaction, an append journal, path containment, and pack integrity
validation.

**Will never be built, by explicit project decision:** the `zip` (miniz) and
`tar` (USTAR) import/export backends. `concept.md` Section 11 recommends
them, but that recommendation is superseded — see `CLAUDE.md`, "Project
decisions that supersede concept.md." `backend_overlay_new` reads and writes
this project's own pack format exclusively; there is no zip/tar support and
none is planned. Do not open an issue or PR adding one.

**Deliberately out of scope for v0** (`concept.md` Section 10, not gaps):
full POSIX semantics, enforced permissions/symlinks/hard links,
cross-process concurrency (design specified in Section 5.6, unimplemented),
and content-defined chunking/delta compression.

**Tested on Linux only**, in one environment. The `dir`-mount containment
fallback path for kernels without `openat2` (Section 6.3) is implemented but
has not been exercised on such a kernel, nor on macOS or Windows.

## Building

Zero required third-party dependencies — only a C11 compiler, `make`, and
`pthread` (Section 11.1 of `concept.md` makes this a hard constraint, not a
preference).

```sh
make            # builds libpackfs.a and libpackfs.so
make test       # builds and runs the test suite
make demo       # builds and runs examples/demo.c — see "Try it" below
make bench      # builds and runs bench/bench.c — see "Benchmarks" below
make install    # installs to $PREFIX (default /usr/local)
```

## Try it

`examples/demo.c` is a small, runnable, human-readable program — not another
automated test — that exercises the library end to end and prints what it
did at each step: a pack-backed overlay (write, read, `mkdir`, `readdir`,
`stat`, a copy-up-then-whiteout delete, compaction via `vfs_sync`), and a
sandboxed `dir` mount that demonstrates a `../../../etc/passwd` escape
attempt being rejected. Run `make demo` twice in a row: the second run's
first `readdir` shows the first run's files, proving that compaction and
reload actually persist data through the pack file, not just within one
process's lifetime.

## Benchmarks

`bench/bench.c` (`make bench`) measures PackFS against the host filesystem
across metadata operations (create/read/stat/readdir/unlink/mkdir), large
sequential I/O, random-access pack reads, mount-table scaling, and
concurrent mixed workloads. [`BENCH.md`](BENCH.md) has the full results and
honest analysis of both the wins and the losses, including the story of
three real O(n²) findings this project's own benchmarking turned up — not
just the wins. Two are fixed: bulk sequential file creation (`src/upper.c`'s
index is a persistent treap now, not a flat array — see "Resolution") and
`pack_write`'s compaction-time duplicate-content elimination, which also had
a latent correctness bug now closed alongside it (see "Resolution #2"). One
is confirmed and *deliberately not* fixed: the mount table scales O(n²) in
mount count, the same way the file index used to, but mount counts are
bounded by a program's own source code rather than workload-driven, so it
isn't worth the added complexity — see "Finding: mount table scaling" for
the reasoning and the numbers behind that call. Read the whole file before
quoting a number from it: what `raw fs` vs `raw+fsync` vs `dir` each
actually measure is not interchangeable, and it explains why.

`openat2`/Landlock support (Section 6) is detected automatically at compile
time via `<sys/syscall.h>`; on kernels or platforms without them, `dir`
mounts fall back to the weaker, documented residual-risk posture described
in `concept.md` Section 6.3 rather than failing to build.

## Quick example

```c
#include <packfs.h>

Vfs *v = vfs_new();

/* a plain in-memory writable tree */
Backend *mem = backend_mem_new();
vfs_mount(v, "/", mem);

int err = 0;
VfsFile *f = vfs_open(v, "/hello.txt", VFS_O_WRONLY | VFS_O_CREAT, &err);
vfs_write(f, "hello", 5);
vfs_close(f);

vfs_free(v);
backend_free(mem);
```

A shipped pack with a writable overlay on top:

```c
Backend *mem = backend_mem_new();
int err = 0;
Backend *ov = backend_overlay_new("assets.pack", mem, &err); /* loads assets.pack if it exists */
vfs_mount(v, "/", ov);

/* ... reads served straight from the pack; writes copy-up into mem ... */

vfs_sync(v, "/"); /* compacts the overlay into a fresh assets.pack (Section 4.1, 5.3) */
```

A sandboxed host directory:

```c
int derr = 0;
Backend *dir = backend_dir_new("/var/lib/myapp/data", &derr);
vfs_mount(v, "/data", dir);
/* every lookup under /data is contained to that directory (Section 6.2);
 * ".." and symlink escapes are rejected, not merely discouraged. */
```

A shipped pack mounted read-only, no writable layer at all:

```c
int perr = 0;
Backend *ro = backend_pack_new("assets.pack", &perr);
vfs_mount(v, "/assets", ro);
/* vfs_write, vfs_mkdir, vfs_unlink, vfs_rename, vfs_sync against anything
 * under /assets all return VFS_ERR_PERM; there is no upper layer to
 * absorb a write into. */
```

## API

The public API is [`include/packfs.h`](include/packfs.h); every function and
struct is documented there with a pointer to the `concept.md` section that
specifies its behavior. In outline:

- `vfs_new` / `vfs_free` — a `Vfs` owns a mount table, nothing else.
- `backend_mem_new` / `backend_dir_new` / `backend_pack_new` /
  `backend_overlay_new` — construct a backend; `vfs_mount`/`vfs_unmount`
  attach or detach it at a path prefix. `backend_pack_new` mounts a pack
  read-only, with no writable upper layer at all — every mutating call
  against it returns `VFS_ERR_PERM`; wrap the same pack in
  `backend_overlay_new` instead when writability is wanted.
- `vfs_open` / `vfs_read` / `vfs_write` / `vfs_close` — file I/O.
- `vfs_stat` / `vfs_readdir` / `vfs_mkdir` / `vfs_unlink` / `vfs_rename` —
  metadata and namespace operations.
- `vfs_sync` — compacts an overlay mount into a fresh pack.
- `vfs_harden_process_with_landlock` — optional, opt-in, process-wide
  Landlock confinement to the process's current `dir` mounts. Deliberately
  **not** applied automatically by `vfs_mount` (Section 6.2 explains why:
  Landlock restrictions are irreversible and process-wide, which would be a
  surprising side effect for an embeddable library to trigger on its own).

## Concurrency

Single-writer, wait-free-reader (Section 5): readers never take a lock and
never observe a write in progress; structural changes (create/unlink/rename/
mkdir/mount/unmount) publish a new immutable snapshot via one atomic pointer
swap; ordinary content writes to an already-existing file update a per-entry
cell directly and never touch the snapshot. `mem`-backed buffer growth
never mutates a buffer address a reader might be reading (Section 5.7):
growth always allocates a new buffer and publishes it, never reallocates in
place. `tests/test_concurrency.c` exercises this under concurrent reader and
writer threads, and the suite is regularly run under ThreadSanitizer and
AddressSanitizer (see `.github/workflows/ci.yml`).

## Security

`dir` mounts are capability-scoped (Section 6.2): a mount holds an already-open
directory file descriptor, not a path string, and every lookup beneath it is
resolved with `openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS)` on Linux 5.6+,
which atomically rejects `..` and symlink escapes in one kernel call. Where
that syscall is unavailable, containment falls back to per-component
`O_NOFOLLOW` resolution — weaker, and documented as such (Section 6.3), not
silently assumed equivalent. Pack files are treated as untrusted input unless
they came from this process's own compaction: every on-disk offset is
bounds-checked and the pack's checksum is verified before any of it is
trusted (Section 7). See `tests/test_dir.c` for a containment regression test
and `tests/test_pack_overlay.c` for a corrupted-pack rejection test.

## Contributing

See [`CONTRIBUTING.md`](CONTRIBUTING.md). Read `concept.md` and `CLAUDE.md`
first — they are the project's actual specification and its enforced
documentation standard, respectively, and every design decision in the code
traces back to one of them.

## License

MIT — see [`LICENSE`](LICENSE).