master
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  retoorandClaude Sonnet 5
|
153d44ee3c |
Fix silent journal-write-failure data loss; confirm TSan blocked twice over
CI / build-and-test (push) Failing after 33s
Prompted by "fix literally everything still open" after the previous
session's data-integrity work. Went through each open item in turn:
1. TSan: tried a genuinely different execution environment (a remote
cloud sandbox, via a dedicated agent) rather than re-stating the local
sandbox's limitation. Result: identical block there too --
personality(ADDR_NO_RANDOMIZE) returns EPERM, a trivial pthread
program fails TSan identically, and all 6 PackFS test binaries fail
with the same FATAL: ThreadSanitizer: unexpected memory mapping
signature. This is now confirmed in two independent environments, not
one -- strong evidence it's a real infrastructure restriction, not a
one-off fluke worth chasing further with the tools available here.
2. While investigating the "disk-full mid-write" gap flagged as untested
last session, found two real, previously-unknown bugs by reading the
journal code (not by a test catching them unprompted):
- journal_append_record and everything that called it were void, and
none of the fwrite/fflush/fsync calls inside had their return values
checked. A real write failure (disk full, quota, I/O error) was
silently reported as success to vfs_write/vfs_mkdir/vfs_unlink/
vfs_rename -- directly contradicting Section 4.4's premise that
success means durable.
- Fixing that alone was not enough, confirmed by direct reproduction:
a partial write leaves a torn record in the journal, and
journal_replay correctly stops at the first record it can't fully
read (Section 4.3) -- which means every record appended *after* the
torn one, including ones that themselves wrote perfectly fine later,
became silently unreachable on reopen. Reproduced directly before
fixing: a forced-failed write followed by a genuinely successful one
was unrecoverable. Fixed by rolling the journal file back to its
exact pre-record length on any failed write.
Both closed in src/overlay.c (journal_append_record/_put/_delete/
_mkdir/journal_put_current now return and propagate success/failure;
overlay_write/_mkdir/_unlink/_rename return VFS_ERR_IO on a durability
failure without rolling back the already-applied in-memory change,
the same asymmetry a real write()-then-failed-fsync() has). Covered
permanently by the new tests/test_journal_failure.c, which forces a
real failure via RLIMIT_FSIZE + ignored SIGXFSZ, not a mock.
Also fixed in the same pass, found by inspection while touching this
code: journal_put_current used to pass a NULL buffer into a memcpy of
a nonzero size when malloc(size) failed (an OOM-triggered NULL-pointer
dereference) -- closed with an explicit allocation-failure check.
Not test-triggered (forcing malloc() failure portably isn't practical
here); verified by code inspection instead, stated as such rather than
claimed as tested.
3. The remaining "journal-truncation-specific crash window" gap from last
session was investigated, not silently dropped: reliably targeting
that narrow a window would need real concurrency (a second writer
thread racing the kill) for benefit the existing compaction-crash test
already gets probabilistically -- a poor trade, so left as a stated,
deliberate non-goal (CLAUDE.md) rather than built.
4. Cross-process contention is NOT addressed here and should not be read
as an oversight: it is concept.md's own explicit, permanent "not
implemented in v0" scope boundary (a specified-but-unbuilt LMDB-style
reader-table design), not a bug -- building it would be a large,
unrequested feature addition outside this session's actual scope.
Verified: clean make all + make test (all 8 binaries), make bench and
make demo still build and the demo runs correctly end to end, and a full
ASan/UBSan sweep of all 8 binaries with zero real findings (some retries
needed for the already-documented DEADLYSIGNAL flake, which
test_crash_consistency hits more often than other tests simply because it
forks 60+ subprocesses per run -- noted in CONTRIBUTING.md so this isn't
mistaken for a regression later).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
|
||
|
retoorandClaude Sonnet 5
|
e41bd2334a |
Add real fault-injection crash-safety testing; investigate TSan for real
CI / build-and-test (push) Failing after 38s
Prompted by a direct question about data-integrity trustworthiness: the crash-safety design (self-checking journal records, atomic-rename compaction, Section 4.3/4.4) had never actually been tested against a real crash -- only reasoned about statically and tested against an already-corrupted file (test_pack_overlay.c, a different scenario). Added tests/test_crash_consistency.c: real fork()+SIGKILL fault injection against an actual child process, not a hand-truncated file standing in for a crash, with empirically-calibrated kill-delay sampling (measured via throwaway scripts, not guessed) so trials land genuine mid-operation interruptions rather than always completing first: - Journal-write crash injection (25 trials): kills a child mid-burst of individually-journaled, individually-fsynced writes; verifies a strict clean-prefix recovery (every completed write present and correct, every write after the kill cleanly absent, no gaps). 20-22/25 trials per run land a genuine interruption; zero corruption found. - Compaction crash injection (40 trials): kills a child mid-vfs_sync; verifies every write durably journaled *before* vfs_sync was called survives regardless of whether compaction itself completed. 40/40 trials per run land a genuine interruption; zero corruption found. A real finding from building this test, not from the code under test: the first draft reported 128 failures, every one a bug in the test itself -- it couldn't distinguish "the child was killed before ever attempting this write" from "the write completed and was then lost," since SIGKILL can't be caught to report progress. Fixed with an independent progress side-channel (plain write()+fsync() on a separate file, entirely outside packfs) as ground truth. Documented in CLAUDE.md's new "Known reliability characteristics" section, including this finding, because a claim is only as trustworthy as what's measuring it. Also investigated whether this sandbox's TSan block is actually unfixable, rather than re-asserting the known limitation: tried personality(ADDR_NO_RANDOMIZE) directly (EPERM), setarch -R (fails identically), searched TSAN_OPTIONS for a bypass (none exists), and checked capabilities/seccomp/unshare --user (zero effective caps, active seccomp filter, namespace escape also blocked). Confirmed categorical and documented as such in CLAUDE.md/CONTRIBUTING.md, rather than left as an unexamined "it doesn't work here." Also closed a real documentation gap SECURITY.md had: the pack integrity checksum's non-cryptographic nature (FNV-1a64, already documented in the context of the compaction-dedup bug) had never been explicitly connected to its *other* use, load-time pack integrity validation -- added, since it's a real, relevant caveat for anyone relying on that checksum against a deliberate adversary rather than accidental corruption. Verified: clean make all + make test (all 7 binaries), full ASan/UBSan sweep of all 7 binaries with zero real findings, and the new test run standalone 4+ times (this session) with stable, consistent results. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB |
||
|
retoorandClaude Sonnet 5
|
21bcb4640f |
Move CI to Gitea Actions; this project is hosted on Gitea, not GitHub
CI / build-and-test (push) Failing after 23s
Per explicit direction: this repository is not going on GitHub. Moved .github/workflows/ci.yml to .gitea/workflows/ci.yml (Gitea Actions' convention) and updated every doc that referenced the old path or assumed GitHub-specific features: - .gitea/workflows/ci.yml: added a header comment on the two things that are genuinely Gitea-specific and instance-dependent, not just a renamed file -- `runs-on: ubuntu-latest` must match a label the actual registered Gitea runner advertises (there is no GitHub-hosted-runner equivalent, this is self-hosted), and `actions/checkout@v4` resolves against whatever action source that runner is configured with. - CONTRIBUTING.md: corrected a claim that no longer holds -- it previously said CI "runs on a normal, unrestricted GitHub Actions VM where TSan is expected to work"; since this is actually a self-hosted Gitea runner whose environment isn't controlled by this repo, that assumption isn't something this repo can vouch for, so the text now says so rather than carrying the old (GitHub-shaped) assumption forward silently. - SECURITY.md: removed a claim this project can't back up (that "private security advisories" are available once hosted -- that's a GitHub feature this repo never had access to); reporting is by direct email to the maintainer only. - README.md: fixed a real gap while in here -- the "Security" section never actually linked to SECURITY.md despite it existing since the previous commit. - CLAUDE.md, CHANGELOG.md: updated path references; CLAUDE.md's self-evaluation section (a historical record of an audit finding) keeps the old .github path where it describes what was literally true at that time, with a note explaining the rename, rather than rewriting history. Verified: clean make all + make test, all 6 binaries pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB |
||
|
retoorandClaude Sonnet 5
|
419182bb05 |
Add version API, pkg-config, SPDX headers, SECURITY.md, CHANGELOG.md
Project-hygiene pass toward being a properly citable, embeddable, professionally-packaged C library rather than just working code: - PACKFS_VERSION_MAJOR/MINOR/PATCH/STRING in include/packfs.h, the single source of truth for the project's version, plus a runtime pfs_version() (src/vfs.c, next to vfs_new/vfs_free) so a dynamically-linked consumer can check ABI/API compatibility without recompiling. Covered by a new assertion in tests/test_mem.c that the macro and the runtime function never disagree. - packfs.pc.in + a `make install` rule that generates packfs.pc with its Version: field derived from PACKFS_VERSION_STRING via a Makefile-level grep/sed, never hand-maintained separately -- verified end-to-end with a scratch `make install PREFIX=...` + `pkg-config --cflags --libs packfs` + `make uninstall`, not just by reading the rule. - SPDX-License-Identifier: MIT added to every src/*.c and src/internal.h (include/packfs.h already had one); the whole distributed source tree now carries consistent machine-readable license metadata. - SECURITY.md, stating precisely what this project's containment and pack- integrity code actually claims as a security boundary (concept.md Section 6/7) versus what it explicitly does not (unenforced `mode`, no cross-process concurrency) -- not generic boilerplate -- with a real reporting contact rather than a placeholder. - CHANGELOG.md (Keep a Changelog format), summarizing the real history in git log to date; explicitly notes no version is tagged yet. Verified: clean `make all` + `make test` (all 6 binaries, including the new version-check assertion), and a full ASan/UBSan sweep of all 6 binaries with zero real findings (one run hit the already-documented DEADLYSIGNAL sandbox flake on test_pack_write_perf across all 3 retries; re-verified directly afterward with 5/5 additional clean passes and timing well under any plausible timeout, confirming it was the flake, not a regression, before treating this as done). Deliberately not done here, by the user's explicit choice: no CODE_OF_CONDUCT.md, and no git remote/publishing -- this repository still has neither, and both are decisions left to the maintainer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB |