v0.1.0
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  retoorandClaude Sonnet 5
|
21bcb4640f |
Move CI to Gitea Actions; this project is hosted on Gitea, not GitHub
CI / build-and-test (push) Failing after 23s
Per explicit direction: this repository is not going on GitHub. Moved .github/workflows/ci.yml to .gitea/workflows/ci.yml (Gitea Actions' convention) and updated every doc that referenced the old path or assumed GitHub-specific features: - .gitea/workflows/ci.yml: added a header comment on the two things that are genuinely Gitea-specific and instance-dependent, not just a renamed file -- `runs-on: ubuntu-latest` must match a label the actual registered Gitea runner advertises (there is no GitHub-hosted-runner equivalent, this is self-hosted), and `actions/checkout@v4` resolves against whatever action source that runner is configured with. - CONTRIBUTING.md: corrected a claim that no longer holds -- it previously said CI "runs on a normal, unrestricted GitHub Actions VM where TSan is expected to work"; since this is actually a self-hosted Gitea runner whose environment isn't controlled by this repo, that assumption isn't something this repo can vouch for, so the text now says so rather than carrying the old (GitHub-shaped) assumption forward silently. - SECURITY.md: removed a claim this project can't back up (that "private security advisories" are available once hosted -- that's a GitHub feature this repo never had access to); reporting is by direct email to the maintainer only. - README.md: fixed a real gap while in here -- the "Security" section never actually linked to SECURITY.md despite it existing since the previous commit. - CLAUDE.md, CHANGELOG.md: updated path references; CLAUDE.md's self-evaluation section (a historical record of an audit finding) keeps the old .github path where it describes what was literally true at that time, with a note explaining the rename, rather than rewriting history. Verified: clean make all + make test, all 6 binaries pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB |
||
|
retoorandClaude Sonnet 5
|
684c6d95de |
Fix pack_write's O(n^2) dedup + correctness bug; document mount table O(n^2)
A audit for other instances of the file-index O(n^2) shape (fixed previously via the persistent treap) found two more real issues: 1. pack_write's Section 9.2 exact-duplicate elimination was a linear scan of every previously-seen (hash, size) pair per entry -- O(n^2) total, invisible in the existing benchmark because its identical-content test files made every scan match on the first comparison. Measured with unique content instead: 80,000 entries took 1.74s, with a 20,000->80,000 step showing 15.6x for a 4x-N step, matching O(n^2)'s 16x prediction. The same scan also trusted a (hash, size) match without ever comparing actual bytes -- a latent correctness bug, since FNV-1a64 is explicitly not collision-resistant. Fixed both at once with an open-addressing hash table (load factor 1/2, linear probing) plus a memcmp verification before ever reusing a data_off. Post-fix: 80,000 entries in 0.044s (39.6x faster), ratio drops to 3.35x (consistent with O(n)). Covered permanently by two new/extended tests: a white-box assertion in test_pack_overlay.c that duplicate-content entries share one data_off and distinct-content entries do not, and a new tests/test_pack_write_perf.c regression tripwire against 10,000 unique entries. 2. vfs.c's mount table uses the same full-array-copy-per-write pattern the file index used to, confirmed O(n^2) via a new bench/bench.c category (500/2,000/8,000 mounts, both 4x-N steps showing 15-20x). Deliberately NOT rewritten: mount points are created by a program's own source code, not workload-driven, so realistic mount counts never reach the scale that made the file index's O(n^2) a real problem. Documented with full reasoning in BENCH.md and CLAUDE.md rather than silently left as an undocumented gap. Also fixes a real CI gap the new tests exposed: ci.yml's sanitizer-build steps never passed -D_GNU_SOURCE when compiling test files (only the library .o's got it), which was harmless while no test included internal.h and became a link failure once two did (internal.h needs _GNU_SOURCE for pthread_rwlock_t). And documents, in CONTRIBUTING.md and CLAUDE.md, a sandbox flake observed directly during this work's own sanitizer runs: ASan/UBSan test binaries occasionally fail to start with AddressSanitizer:DEADLYSIGNAL (sometimes looping rather than exiting), non-deterministically hitting different unrelated binaries across runs -- a startup race, not a memory-safety bug, confirmed by clean passes on retry; sanitizer runs in such an environment should be timeout-wrapped. BENCH.md's "After" table and Appendix B are replaced with the current, complete 54-measurement bench/bench.c run (the original 45 plus the new mount-scaling category); the pre-fix 45-measurement "Before" table is kept as the historical record, per this project's documentation standard. Verified: make test (all 6 binaries, including the 2 new/changed), a clean make all, and repeated ASan+UBSan runs (0 real findings; the DEADLYSIGNAL flake above was observed and correctly distinguished from a real finding by re-running until a clean pass). TSan could not be run in this sandbox (pre-existing, documented environment limitation). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB |
||
|
retoorandClaude Sonnet 5
|
64283d587b |
Fix the O(n^2) bulk create/unlink: flat array -> persistent treap
BENCH.md's earlier finding was real: every structural write (create/
unlink/mkdir) copied the entire sorted UpperSnapshot entry array before
publishing the next snapshot, making bulk sequential creation O(n^2).
concept.md Section 5.3 names the exact condition for reconsidering this
("a persistent structurally-shared tree structure is not required
until this assumption is empirically violated") -- that condition was
measured, not hypothesized, so this closes it rather than leaving it
as a documented-but-open limitation.
The index is now a persistent treap (src/upper.c): a structural write
copies only the O(log n) nodes on the path to the change, sharing
every other node (its own refcount, cascading like MutCell's and
UpperSnapshot's) with whichever snapshot(s) it was built from. Chosen
over a persistent AVL/red-black/weight-balanced tree because deletion
in those can need O(log n) rebalancing rotations -- each a real
allocation in a persistent setting -- where a treap needs only O(1)
amortized rotations for insert and delete (Seidel & Aragon 1996), with
expected O(log n) height regardless of insertion order, including the
sorted-by-creation-order pattern that made the flat array quadratic in
the first place. Liljenzin's "Confluently Persistent Sets and Maps"
(arXiv:1301.3388) documents persistent treaps giving O(1) snapshots
for MVCC specifically, which is this exact use case. Full reasoning
and the ownership convention (functions consume one ref of their tree
arguments, return one owned ref) are in upper.c's comment above
struct TreapNode.
Blast radius kept deliberately small: snapshot_upsert/snapshot_remove
keep their exact original signatures, so upper_create/upper_mkdir/
upper_remove/upper_rename/upper_copy_up needed zero changes.
upper_lookup/upper_has_children keep their exact contracts. Only
upstd_readdir and overlay_readdir's manual array scans became calls to
a new upper_visit_range (O(log n + r) range query, replacing an O(n)
scan in both, a bonus fix beyond what was strictly necessary) since
there's no flat array left to scan.
Added tests/test_index_stress.c: thousands of randomized (not
sequential) creates/deletes/renames across nested directories,
cross-checked against an independent reference model after every
round, not just "did it not crash" -- exercises exactly the code path
the O(n^2) bug and this fix live in, at a scale the other tests don't
reach. Verified under -fsanitize=undefined (150+ runs across this
change's lifetime, 0 failures) and -fsanitize=address (100+ runs, 0
real findings; known sandbox ASan-startup flakes excluded, see prior
commits) per CLAUDE.md's sanitizer rule for upper.c/overlay.c changes.
Measured result (make bench, same environment as the original
finding): mem create 257x faster, unlink 330x faster, mkdir 65x
faster, concurrent mixed workload 131x faster -- and, the comparison
that matters, mem now beats raw fs at every one of these (was losing
by 6-22x before). The complexity-class change is confirmed the same
way the O(n^2) was found: mkdir at N=4,000 vs create at N=20,000 now
shows a 7.15x slowdown for a 5x increase in N, matching the O(n log n)
prediction (5.97x) rather than the old O(n^2) one (25x). Full
before/after tables in BENCH.md's new "Resolution" section, which
keeps the original run as the historical record rather than
overwriting it, per this project's own documentation standard.
Recorded the fix in CLAUDE.md's "Known performance characteristics"
(marked RESOLVED, not silently removed) and its architecture map.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
|
||
|
retoorandClaude Sonnet 5
|
0b3b207bc0 |
Add a benchmark suite comparing PackFS against the host filesystem
bench/bench.c (`make bench`) measures create/read/stat/readdir/unlink/
mkdir on mem, dir, and raw fs; large sequential I/O; random-access pack
reads via mmap vs raw fs; compaction throughput; and 8-thread
concurrent mixed workloads. Results from one full run, with honest
analysis (what each "raw fs" vs "raw+fsync" vs "dir" label actually
measures, so they aren't misread as interchangeable), are in BENCH.md.
The benchmark surfaced a real, quantitatively-confirmed finding, not
just favorable numbers: bulk sequential create/unlink on mem/dir is
O(n^2) in file count (~9-22x slower than raw fs at N=20,000), because
every structural write copies the entire snapshot entry array before
publishing it (Section 5.3). concept.md itself names the exact trigger
condition for reconsidering this ("a persistent structurally-shared
tree structure is not required until this assumption is empirically
violated") — this benchmark is that violation, measured rather than
hypothesized: mkdir at N=4,000 vs create at N=20,000 (same mechanism,
5x the N) shows a 28.4x slowdown, matching the O(n^2) prediction (25x)
far better than O(n) (5x).
Also found: dir-backend stat() costs ~2x raw stat() (open+fstat vs one
syscall, the direct cost of openat2 containment on the metadata path);
mem-backed large writes lose to raw fs above ~16MB (Section 5.7's
buffer-growth discipline re-copies prior writes on every capacity
doubling, the price of never exposing a reader to a freed buffer).
Recorded the O(n^2) finding in CLAUDE.md's new "Known performance
characteristics" section, per the same pattern used for the
reclaim_gate use-after-free discovery, since it's exactly the kind of
fact that would otherwise have to be rediscovered by benchmarking
again from scratch. Linked from README and CONTRIBUTING.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
|
||
|
retoorandClaude Sonnet 5
|
72e3c900f2 |
Add PackFS v0: statically linked in-process VFS implementing concept.md
Implements the core design: a mount table published as an atomically- swapped snapshot; mem/dir/pack/overlay backends; copy-on-write overlay with copy-up and whiteout deletion; a checksummed append journal; compaction with exact-duplicate elimination; single-writer/wait-free- reader concurrency with a structural/content write split; openat2/ Landlock path containment for dir mounts; and load-time pack integrity validation. Zero required third-party dependencies. Sanitizer testing (ASan/UBSan) caught and led to fixing a genuine heap-use-after-free in the snapshot-reclamation path: the textbook "load pointer, then increment its refcount" pattern left a gap a concurrent writer could free through. Closed with a small reclaim_gate rwlock, documented in internal.h and CLAUDE.md since it's a pattern every refcounted structure in the codebase now follows. zip/tar import/export backends, recommended in concept.md Section 11, will not be built — a permanent project decision recorded in CLAUDE.md since concept.md itself is frozen and cannot be edited to reflect it. Includes a runnable demo (examples/demo.c, `make demo`) exercising the library end to end and proving cross-run persistence through the pack file, plus open-source scaffolding: MIT license, README, CONTRIBUTING, and a CI workflow running the test suite under ASan/UBSan/TSan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB |