retoor/packfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
153d44ee3c50d6daa94884847684a6b5b1c35931
packfs/CHANGELOG.md
T
retoorandClaude Sonnet 5 153d44ee3c
CI / build-and-test (push) Failing after 33s
Details
Fix silent journal-write-failure data loss; confirm TSan blocked twice over 
Prompted by "fix literally everything still open" after the previous
session's data-integrity work. Went through each open item in turn:

1. TSan: tried a genuinely different execution environment (a remote
   cloud sandbox, via a dedicated agent) rather than re-stating the local
   sandbox's limitation. Result: identical block there too --
   personality(ADDR_NO_RANDOMIZE) returns EPERM, a trivial pthread
   program fails TSan identically, and all 6 PackFS test binaries fail
   with the same FATAL: ThreadSanitizer: unexpected memory mapping
   signature. This is now confirmed in two independent environments, not
   one -- strong evidence it's a real infrastructure restriction, not a
   one-off fluke worth chasing further with the tools available here.

2. While investigating the "disk-full mid-write" gap flagged as untested
   last session, found two real, previously-unknown bugs by reading the
   journal code (not by a test catching them unprompted):

   - journal_append_record and everything that called it were void, and
     none of the fwrite/fflush/fsync calls inside had their return values
     checked. A real write failure (disk full, quota, I/O error) was
     silently reported as success to vfs_write/vfs_mkdir/vfs_unlink/
     vfs_rename -- directly contradicting Section 4.4's premise that
     success means durable.

   - Fixing that alone was not enough, confirmed by direct reproduction:
     a partial write leaves a torn record in the journal, and
     journal_replay correctly stops at the first record it can't fully
     read (Section 4.3) -- which means every record appended *after* the
     torn one, including ones that themselves wrote perfectly fine later,
     became silently unreachable on reopen. Reproduced directly before
     fixing: a forced-failed write followed by a genuinely successful one
     was unrecoverable. Fixed by rolling the journal file back to its
     exact pre-record length on any failed write.

   Both closed in src/overlay.c (journal_append_record/_put/_delete/
   _mkdir/journal_put_current now return and propagate success/failure;
   overlay_write/_mkdir/_unlink/_rename return VFS_ERR_IO on a durability
   failure without rolling back the already-applied in-memory change,
   the same asymmetry a real write()-then-failed-fsync() has). Covered
   permanently by the new tests/test_journal_failure.c, which forces a
   real failure via RLIMIT_FSIZE + ignored SIGXFSZ, not a mock.

   Also fixed in the same pass, found by inspection while touching this
   code: journal_put_current used to pass a NULL buffer into a memcpy of
   a nonzero size when malloc(size) failed (an OOM-triggered NULL-pointer
   dereference) -- closed with an explicit allocation-failure check.
   Not test-triggered (forcing malloc() failure portably isn't practical
   here); verified by code inspection instead, stated as such rather than
   claimed as tested.

3. The remaining "journal-truncation-specific crash window" gap from last
   session was investigated, not silently dropped: reliably targeting
   that narrow a window would need real concurrency (a second writer
   thread racing the kill) for benefit the existing compaction-crash test
   already gets probabilistically -- a poor trade, so left as a stated,
   deliberate non-goal (CLAUDE.md) rather than built.

4. Cross-process contention is NOT addressed here and should not be read
   as an oversight: it is concept.md's own explicit, permanent "not
   implemented in v0" scope boundary (a specified-but-unbuilt LMDB-style
   reader-table design), not a bug -- building it would be a large,
   unrequested feature addition outside this session's actual scope.

Verified: clean make all + make test (all 8 binaries), make bench and
make demo still build and the demo runs correctly end to end, and a full
ASan/UBSan sweep of all 8 binaries with zero real findings (some retries
needed for the already-documented DEADLYSIGNAL flake, which
test_crash_consistency hits more often than other tests simply because it
forks 60+ subprocesses per run -- noted in CONTRIBUTING.md so this isn't
mistaken for a regression later).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
2026-09-14 19:40:23 +00:00

6.9 KiB
Raw Blame History

Changelog

All notable changes to this project are documented in this file, in the style of Keep a Changelog. This project follows Semantic Versioning; see PACKFS_VERSION_STRING in include/packfs.h for the authoritative current version (packfs.pc, generated by make install, is derived from it, not maintained separately).

This project is pre-release (0.x, an initial, partial implementation of concept.md, not a completed one; see README.md's "Status"). v0.1.0 is tagged locally in this repository's own git history but has not been pushed to any remote — there is no public release yet, only a local one.

[Unreleased]

Fixed

  • A failed journal write (disk full, quota, an I/O error) used to be silently reported as success. journal_append_record and its callers were void and never checked fwrite/fflush/fsync's return values, so vfs_write/vfs_mkdir/vfs_unlink/vfs_rename on an overlay-backed file all reported success even when the durable journal append actually failed. Fixed by propagating success/failure through the whole call chain; the affected call now returns VFS_ERR_IO.
  • A second bug, found only by reproducing the first fix's edge case directly: a partial write left a torn record in the journal that made every later, individually-successful write unrecoverable on reopen too, not just the failed one — journal_replay stops at the first record it can't fully read, regardless of what valid records follow it. Fixed by rolling the journal file back to its exact pre-record length whenever a record fails partway. Both bugs covered permanently by the new tests/test_journal_failure.c, which forces a real write failure via RLIMIT_FSIZE rather than a mock.
  • Neither bug was caught by an existing test — both were found by code review while investigating a question about data-integrity trustworthiness, then confirmed by direct reproduction before and after each fix, not merely inspected and assumed correct.

Added

  • tests/test_crash_consistency.c: real fork()+SIGKILL fault injection (not a hand-truncated file) against journaled writes and compaction, with an independent progress side-channel so the test's own oracle can distinguish "never attempted before the kill" from "attempted and lost" — a distinction its own first draft got wrong, reporting 128 false failures before that side-channel was added. Zero real corruption found across dozens of genuine interruptions per run, both scenarios, on repeated runs including under ASan/UBSan.

Documented

  • The pack integrity checksum's (FNV-1a64) non-cryptographic nature — already documented for the compaction-dedup bug — was never explicitly connected to its other job, load-time pack integrity validation, where it matters more (SECURITY.md).
  • TSan's inability to run is now confirmed in a second, independent sandboxed environment (a separate remote cloud sandbox, tried specifically to check whether the restriction was one environment's fluke), not just this project's own — same root cause (personality(ADDR_NO_RANDOMIZE) blocked by seccomp) confirmed both generically and against this project's actual test suite.

[0.1.0] - 2026-09-14

Added

  • Initial implementation of concept.md: vfs_* core, mem/dir/pack backends, the copy-on-write overlay (copy-up, whiteouts, compaction, an append journal), path containment (openat2/Landlock on Linux 5.6+/5.13+, a documented weaker fallback elsewhere), and pack integrity validation.
  • tests/test_*.c covering each backend, concurrency, and randomized structural-index stress testing against an independent reference model.
  • bench/bench.c (make bench) and BENCH.md, comparing PackFS against the host filesystem across metadata operations, large sequential I/O, random-access pack reads, mount-table scaling, and concurrent workloads.
  • PACKFS_VERSION_MAJOR/MINOR/PATCH/STRING and pfs_version() for compile-time and runtime version/ABI checks.
  • packfs.pc (pkg-config), generated by make install from packfs.pc.in, version always derived from include/packfs.h.
  • SPDX-License-Identifier: MIT on every file under src/ and include/.

Fixed

  • Bulk sequential create/unlink/mkdir on mem/dir was O(n²) in file count. The index (UpperSnapshot) was a flat sorted array copied in full on every structural write; rewritten as a persistent treap (src/upper.c), reducing a structural write to the O(log n) nodes on the path to the change. See BENCH.md's "Resolution" for the full before/after measurement and complexity-class confirmation.
  • pack_write's compaction-time exact-duplicate elimination (Section 9.2) was O(n²) in entry count, and trusted a hash match without comparing actual bytes (a latent correctness bug: FNV-1a64 is explicitly not collision-resistant). Rewritten as an open-addressing hash table with a memcmp verification before ever reusing a data_off, fixing both at once. See BENCH.md's "Resolution #2".
  • backend_pack_new was declared in include/packfs.h but never implemented — any program calling it failed at link time. Implemented as a standalone, read-only pack Backend.
  • The CI config's (.gitea/workflows/ci.yml) sanitizer-build steps never passed -D_GNU_SOURCE when compiling test files (only the library object files got it), which was harmless until a test included internal.h (needed for pthread_rwlock_t) and became a link failure.

Changed

  • CI moved from a GitHub-Actions-convention config (.github/workflows/) to Gitea Actions (.gitea/workflows/) — this project is hosted on Gitea, not GitHub, and was never actually hosted on GitHub; only the scaffolding's starting convention changed. SECURITY.md, CLAUDE.md, CONTRIBUTING.md, and README.md updated to match.

Documented

  • vfs.c's mount table (MountSnapshot) is O(n²) in mount count, the same pattern the file index used to have — confirmed, and deliberately left unfixed. Mount counts are bounded by a program's own source code, not workload-driven, so they do not reach the scale that made the file index's O(n²) a real problem. See BENCH.md's "Finding: mount table scaling" for the numbers and reasoning.
  • A ThreadSanitizer environment limitation (some sandboxes block the personality(ADDR_NO_RANDOMIZE) syscall TSan needs) and a related ASan/UBSan sandbox startup flake (AddressSanitizer:DEADLYSIGNAL), both in CONTRIBUTING.md and CLAUDE.md, with the confirming tests and the required mitigation (timeout-wrapped sanitizer runs).
Reference in New Issue View Git Blame Copy Permalink