Files
retoorandClaude Sonnet 5 684c6d95de Fix pack_write's O(n^2) dedup + correctness bug; document mount table O(n^2)
A audit for other instances of the file-index O(n^2) shape (fixed
previously via the persistent treap) found two more real issues:

1. pack_write's Section 9.2 exact-duplicate elimination was a linear scan
   of every previously-seen (hash, size) pair per entry -- O(n^2) total,
   invisible in the existing benchmark because its identical-content test
   files made every scan match on the first comparison. Measured with
   unique content instead: 80,000 entries took 1.74s, with a 20,000->80,000
   step showing 15.6x for a 4x-N step, matching O(n^2)'s 16x prediction.
   The same scan also trusted a (hash, size) match without ever comparing
   actual bytes -- a latent correctness bug, since FNV-1a64 is explicitly
   not collision-resistant. Fixed both at once with an open-addressing hash
   table (load factor 1/2, linear probing) plus a memcmp verification
   before ever reusing a data_off. Post-fix: 80,000 entries in 0.044s
   (39.6x faster), ratio drops to 3.35x (consistent with O(n)).

   Covered permanently by two new/extended tests: a white-box assertion in
   test_pack_overlay.c that duplicate-content entries share one data_off
   and distinct-content entries do not, and a new
   tests/test_pack_write_perf.c regression tripwire against 10,000 unique
   entries.

2. vfs.c's mount table uses the same full-array-copy-per-write pattern the
   file index used to, confirmed O(n^2) via a new bench/bench.c category
   (500/2,000/8,000 mounts, both 4x-N steps showing 15-20x). Deliberately
   NOT rewritten: mount points are created by a program's own source code,
   not workload-driven, so realistic mount counts never reach the scale
   that made the file index's O(n^2) a real problem. Documented with full
   reasoning in BENCH.md and CLAUDE.md rather than silently left as an
   undocumented gap.

Also fixes a real CI gap the new tests exposed: ci.yml's sanitizer-build
steps never passed -D_GNU_SOURCE when compiling test files (only the
library .o's got it), which was harmless while no test included
internal.h and became a link failure once two did (internal.h needs
_GNU_SOURCE for pthread_rwlock_t). And documents, in CONTRIBUTING.md and
CLAUDE.md, a sandbox flake observed directly during this work's own
sanitizer runs: ASan/UBSan test binaries occasionally fail to start with
AddressSanitizer:DEADLYSIGNAL (sometimes looping rather than exiting),
non-deterministically hitting different unrelated binaries across runs --
a startup race, not a memory-safety bug, confirmed by clean passes on
retry; sanitizer runs in such an environment should be timeout-wrapped.

BENCH.md's "After" table and Appendix B are replaced with the current,
complete 54-measurement bench/bench.c run (the original 45 plus the new
mount-scaling category); the pre-fix 45-measurement "Before" table is kept
as the historical record, per this project's documentation standard.

Verified: make test (all 6 binaries, including the 2 new/changed), a clean
make all, and repeated ASan+UBSan runs (0 real findings; the DEADLYSIGNAL
flake above was observed and correctly distinguished from a real finding
by re-running until a clean pass). TSan could not be run in this sandbox
(pre-existing, documented environment limitation).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqJpkdJ6Njnt1pw3CbghzB
2026-09-14 09:55:47 +00:00

657 lines
24 KiB
C

/*
* bench.c — PackFS vs. the host filesystem, across as many operations as
* the public API exposes.
*
* This is a benchmark, not a test: it makes no pass/fail assertions, and
* its numbers are specific to whatever machine and filesystem it runs
* on — see the environment banner it prints before any numbers, and read
* it before drawing conclusions from the results. In particular:
*
* - "raw fs (no fsync)" writes go through plain POSIX open/write/close
* with no fsync, so they land in the page cache, exactly like the
* `mem` backend lands in a malloc'd buffer — this is the fair,
* apples-to-apples comparison for "how much does PackFS's own
* bookkeeping cost, independent of durability."
* - "raw fs (fsync)" adds an fsync per file, which is what it actually
* costs to make a write durable on this host — this is the fair
* comparison against nothing in PackFS, since no backend here
* fsyncs a `mem`-backed write (there is nothing on disk to sync);
* the closest durable analogue is the `dir` backend, benchmarked
* separately, and even that does not fsync per write (Section 5.4
* does not require it — only compaction and the journal do,
* Section 4.3).
* - The `dir` backend's numbers include the openat2/O_NOFOLLOW
* containment cost (Section 6.2) on every single lookup, which raw
* fs access has no equivalent of paying — that gap *is* the
* measurement, not noise to explain away.
*/
#include <dirent.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <time.h>
#include <unistd.h>
#include "packfs.h"
/* ---- tunables: this is the "extreme" in "make it extreme" ---- */
#define N_SMALL 20000 /* small files for the metadata-heavy suite */
#define SMALL_SIZE 128
#define N_DIRS 4000
/* vfs_mount/vfs_unmount use the identical full-snapshot-copy pattern the
* file index used to (Section 5.3's mount-table-is-part-of-the-snapshot
* unification) — this section exists to check, empirically rather than
* by assumption, whether that ever matters at a realistic mount count.
* 2,000 is deliberately far beyond any real program's mount count (a
* mount is something a program's own source code sets up once per
* distinct storage location — assets/config/tmp/per-plugin — not
* something a workload creates at file-count scale). */
#define N_MOUNTS 2000
#define N_CONC_THREADS 8
#define OPS_PER_THREAD 4000
#define LARGE_CHUNK (64 * 1024)
static const size_t LARGE_SIZES[] = { 1u << 20, 16u << 20, 64u << 20 };
#define N_LARGE_SIZES (sizeof(LARGE_SIZES) / sizeof(LARGE_SIZES[0]))
/* ---- timing + results table ---- */
static double now_sec(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (double)ts.tv_sec + (double)ts.tv_nsec / 1e9;
}
typedef struct Row {
char category[48];
char backend[24];
double seconds;
double ops_per_sec;
double mb_per_sec; /* 0 if not applicable */
} Row;
static Row g_rows[256];
static int g_row_count = 0;
static void record(const char *category, const char *backend, double seconds, double count, double bytes) {
Row *r = &g_rows[g_row_count++];
snprintf(r->category, sizeof(r->category), "%s", category);
snprintf(r->backend, sizeof(r->backend), "%s", backend);
r->seconds = seconds;
r->ops_per_sec = seconds > 0 ? count / seconds : 0;
r->mb_per_sec = (bytes > 0 && seconds > 0) ? (bytes / (1024.0 * 1024.0)) / seconds : 0;
printf(" %-28s %-10s %9.4fs %14.0f ops/s", category, backend, seconds, r->ops_per_sec);
if (r->mb_per_sec > 0) printf(" %10.1f MB/s", r->mb_per_sec);
printf("\n");
}
static void section(const char *title) {
printf("\n== %s ==\n", title);
}
static void fmt_name(char *buf, size_t cap, const char *prefix, int i) {
snprintf(buf, cap, "%s%06d.dat", prefix, i);
}
static char g_payload[SMALL_SIZE];
/* ---- category 1: small-file create (write from scratch) ---- */
static void bench_create_mem(Vfs *v) {
char path[64];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
fmt_name(path, sizeof(path), "/f", i);
int err = 0;
VfsFile *f = vfs_open(v, path, VFS_O_WRONLY | VFS_O_CREAT, &err);
vfs_write(f, g_payload, SMALL_SIZE);
vfs_close(f);
}
record("create N small files", "mem", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
}
static void bench_create_dirbackend(Vfs *v) {
char path[64];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
fmt_name(path, sizeof(path), "/f", i);
int err = 0;
VfsFile *f = vfs_open(v, path, VFS_O_WRONLY | VFS_O_CREAT, &err);
vfs_write(f, g_payload, SMALL_SIZE);
vfs_close(f);
}
record("create N small files", "dir", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
}
static void bench_create_rawfs(const char *root, int fsync_each) {
char path[256];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
char name[64];
fmt_name(name, sizeof(name), "/f", i);
snprintf(path, sizeof(path), "%s%s", root, name);
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, g_payload, SMALL_SIZE);
if (fsync_each) fsync(fd);
close(fd);
}
record("create N small files", fsync_each ? "raw+fsync" : "raw fs", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
}
/* ---- category 2: small-file read ---- */
static void bench_read_mem(Vfs *v) {
char path[64], buf[SMALL_SIZE];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
fmt_name(path, sizeof(path), "/f", i);
int err = 0;
VfsFile *f = vfs_open(v, path, VFS_O_RDONLY, &err);
vfs_read(f, buf, sizeof(buf));
vfs_close(f);
}
record("read N small files", "mem", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
}
static void bench_read_dirbackend(Vfs *v) {
char path[64], buf[SMALL_SIZE];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
fmt_name(path, sizeof(path), "/f", i);
int err = 0;
VfsFile *f = vfs_open(v, path, VFS_O_RDONLY, &err);
vfs_read(f, buf, sizeof(buf));
vfs_close(f);
}
record("read N small files", "dir", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
}
static void bench_read_rawfs(const char *root) {
char path[256], buf[SMALL_SIZE];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
char name[64];
fmt_name(name, sizeof(name), "/f", i);
snprintf(path, sizeof(path), "%s%s", root, name);
int fd = open(path, O_RDONLY);
read(fd, buf, sizeof(buf));
close(fd);
}
record("read N small files", "raw fs", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
}
/* ---- category 3: stat ---- */
static void bench_stat_mem(Vfs *v) {
char path[64]; VfsStat st;
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) { fmt_name(path, sizeof(path), "/f", i); vfs_stat(v, path, &st); }
record("stat N files", "mem", now_sec() - t0, N_SMALL, 0);
}
static void bench_stat_dirbackend(Vfs *v) {
char path[64]; VfsStat st;
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) { fmt_name(path, sizeof(path), "/f", i); vfs_stat(v, path, &st); }
record("stat N files", "dir", now_sec() - t0, N_SMALL, 0);
}
static void bench_stat_rawfs(const char *root) {
char path[256]; struct stat st;
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
char name[64]; fmt_name(name, sizeof(name), "/f", i);
snprintf(path, sizeof(path), "%s%s", root, name);
stat(path, &st);
}
record("stat N files", "raw fs", now_sec() - t0, N_SMALL, 0);
}
/* ---- category 4: readdir ---- */
static void bench_readdir_mem(Vfs *v) {
double t0 = now_sec();
VfsDir dir;
vfs_readdir(v, "/", &dir);
double dt = now_sec() - t0;
record("readdir (N entries)", "mem", dt, dir.count, 0);
vfs_dir_free(&dir);
}
static void bench_readdir_dirbackend(Vfs *v) {
double t0 = now_sec();
VfsDir dir;
vfs_readdir(v, "/", &dir);
double dt = now_sec() - t0;
record("readdir (N entries)", "dir", dt, dir.count, 0);
vfs_dir_free(&dir);
}
static void bench_readdir_rawfs(const char *root) {
double t0 = now_sec();
DIR *d = opendir(root);
long count = 0;
struct dirent *de;
while ((de = readdir(d)) != NULL) if (de->d_name[0] != '.') count++;
closedir(d);
double dt = now_sec() - t0;
record("readdir (N entries)", "raw fs", dt, (double)count, 0);
}
/* ---- category 5: unlink ---- */
static void bench_unlink_mem(Vfs *v) {
char path[64];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) { fmt_name(path, sizeof(path), "/f", i); vfs_unlink(v, path); }
record("unlink N files", "mem", now_sec() - t0, N_SMALL, 0);
}
static void bench_unlink_dirbackend(Vfs *v) {
char path[64];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) { fmt_name(path, sizeof(path), "/f", i); vfs_unlink(v, path); }
record("unlink N files", "dir", now_sec() - t0, N_SMALL, 0);
}
static void bench_unlink_rawfs(const char *root) {
char path[256];
double t0 = now_sec();
for (int i = 0; i < N_SMALL; i++) {
char name[64]; fmt_name(name, sizeof(name), "/f", i);
snprintf(path, sizeof(path), "%s%s", root, name);
unlink(path);
}
record("unlink N files", "raw fs", now_sec() - t0, N_SMALL, 0);
}
/* ---- category 6: mkdir/rmdir ---- */
static void bench_mkdir_mem(Vfs *v) {
char path[64];
double t0 = now_sec();
for (int i = 0; i < N_DIRS; i++) { snprintf(path, sizeof(path), "/d%06d", i); vfs_mkdir(v, path); }
double dt = now_sec() - t0;
record("mkdir N dirs", "mem", dt, N_DIRS, 0);
t0 = now_sec();
for (int i = 0; i < N_DIRS; i++) { snprintf(path, sizeof(path), "/d%06d", i); vfs_unlink(v, path); }
record("rmdir N dirs", "mem", now_sec() - t0, N_DIRS, 0);
}
static void bench_mkdir_dirbackend(Vfs *v) {
char path[64];
double t0 = now_sec();
for (int i = 0; i < N_DIRS; i++) { snprintf(path, sizeof(path), "/d%06d", i); vfs_mkdir(v, path); }
double dt = now_sec() - t0;
record("mkdir N dirs", "dir", dt, N_DIRS, 0);
t0 = now_sec();
for (int i = 0; i < N_DIRS; i++) { snprintf(path, sizeof(path), "/d%06d", i); vfs_unlink(v, path); }
record("rmdir N dirs", "dir", now_sec() - t0, N_DIRS, 0);
}
static void bench_mkdir_rawfs(const char *root) {
char path[256];
double t0 = now_sec();
for (int i = 0; i < N_DIRS; i++) { snprintf(path, sizeof(path), "%s/d%06d", root, i); mkdir(path, 0755); }
double dt = now_sec() - t0;
record("mkdir N dirs", "raw fs", dt, N_DIRS, 0);
t0 = now_sec();
for (int i = 0; i < N_DIRS; i++) { snprintf(path, sizeof(path), "%s/d%06d", root, i); rmdir(path); }
record("rmdir N dirs", "raw fs", now_sec() - t0, N_DIRS, 0);
}
/* ---- category 7: large sequential write/read ---- */
static void bench_large_mem(Vfs *v, size_t size) {
char *chunk = malloc(LARGE_CHUNK);
memset(chunk, 0x5a, LARGE_CHUNK);
char label[32]; snprintf(label, sizeof(label), "write %zuMB", size / (1024 * 1024));
int err = 0;
double t0 = now_sec();
VfsFile *f = vfs_open(v, "/large.bin", VFS_O_WRONLY | VFS_O_CREAT | VFS_O_TRUNC, &err);
for (size_t off = 0; off < size; off += LARGE_CHUNK) vfs_write(f, chunk, LARGE_CHUNK);
vfs_close(f);
record(label, "mem", now_sec() - t0, 1, (double)size);
snprintf(label, sizeof(label), "read %zuMB", size / (1024 * 1024));
t0 = now_sec();
f = vfs_open(v, "/large.bin", VFS_O_RDONLY, &err);
while (vfs_read(f, chunk, LARGE_CHUNK) > 0) {}
vfs_close(f);
record(label, "mem", now_sec() - t0, 1, (double)size);
vfs_unlink(v, "/large.bin");
free(chunk);
}
static void bench_large_rawfs(const char *root, size_t size, int fsync_it) {
char *chunk = malloc(LARGE_CHUNK);
memset(chunk, 0x5a, LARGE_CHUNK);
char path[256]; snprintf(path, sizeof(path), "%s/large.bin", root);
char label[32]; snprintf(label, sizeof(label), "write %zuMB", size / (1024 * 1024));
double t0 = now_sec();
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
for (size_t off = 0; off < size; off += LARGE_CHUNK) write(fd, chunk, LARGE_CHUNK);
if (fsync_it) fsync(fd);
close(fd);
record(label, fsync_it ? "raw+fsync" : "raw fs", now_sec() - t0, 1, (double)size);
snprintf(label, sizeof(label), "read %zuMB", size / (1024 * 1024));
t0 = now_sec();
fd = open(path, O_RDONLY);
while (read(fd, chunk, LARGE_CHUNK) > 0) {}
close(fd);
record(label, fsync_it ? "raw+fsync" : "raw fs", now_sec() - t0, 1, (double)size);
unlink(path);
free(chunk);
}
/* ---- category 8: pack (mmap'd, read-only) random access vs raw fs ---- */
static void bench_pack_random(const char *pack_path, int *order, Vfs *v) {
char buf[SMALL_SIZE], path[64];
double t0 = now_sec();
for (int k = 0; k < N_SMALL; k++) {
fmt_name(path, sizeof(path), "/f", order[k]);
int err = 0;
VfsFile *f = vfs_open(v, path, VFS_O_RDONLY, &err);
if (f) { vfs_read(f, buf, sizeof(buf)); vfs_close(f); }
}
record("random-read N entries", "pack", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
(void)pack_path;
}
static void bench_rawfs_random(const char *root, int *order) {
char buf[SMALL_SIZE], path[256];
double t0 = now_sec();
for (int k = 0; k < N_SMALL; k++) {
char name[64]; fmt_name(name, sizeof(name), "/f", order[k]);
snprintf(path, sizeof(path), "%s%s", root, name);
int fd = open(path, O_RDONLY);
if (fd >= 0) { read(fd, buf, sizeof(buf)); close(fd); }
}
record("random-read N entries", "raw fs", now_sec() - t0, N_SMALL, (double)N_SMALL * SMALL_SIZE);
}
/* ---- category 9: concurrency ---- */
typedef struct ThreadArgs {
Vfs *v;
const char *rawroot;
int thread_id;
} ThreadArgs;
static void *conc_worker_mem(void *arg) {
ThreadArgs *ta = (ThreadArgs *)arg;
char path[64];
for (int i = 0; i < OPS_PER_THREAD; i++) {
snprintf(path, sizeof(path), "/t%d_%06d.dat", ta->thread_id, i);
int err = 0;
VfsFile *f = vfs_open(ta->v, path, VFS_O_WRONLY | VFS_O_CREAT, &err);
if (f) { vfs_write(f, g_payload, SMALL_SIZE); vfs_close(f); }
f = vfs_open(ta->v, path, VFS_O_RDONLY, &err);
if (f) { char buf[SMALL_SIZE]; vfs_read(f, buf, sizeof(buf)); vfs_close(f); }
vfs_unlink(ta->v, path);
}
return NULL;
}
static void *conc_worker_rawfs(void *arg) {
ThreadArgs *ta = (ThreadArgs *)arg;
char path[256];
for (int i = 0; i < OPS_PER_THREAD; i++) {
snprintf(path, sizeof(path), "%s/t%d_%06d.dat", ta->rawroot, ta->thread_id, i);
int fd = open(path, O_WRONLY | O_CREAT, 0644);
if (fd >= 0) { write(fd, g_payload, SMALL_SIZE); close(fd); }
fd = open(path, O_RDONLY);
if (fd >= 0) { char buf[SMALL_SIZE]; read(fd, buf, sizeof(buf)); close(fd); }
unlink(path);
}
return NULL;
}
static void bench_concurrency(Vfs *v, const char *rawroot) {
pthread_t threads[N_CONC_THREADS];
ThreadArgs args[N_CONC_THREADS];
double t0 = now_sec();
for (int i = 0; i < N_CONC_THREADS; i++) {
args[i] = (ThreadArgs){ v, NULL, i };
pthread_create(&threads[i], NULL, conc_worker_mem, &args[i]);
}
for (int i = 0; i < N_CONC_THREADS; i++) pthread_join(threads[i], NULL);
record("concurrent create+read+unlink", "mem", now_sec() - t0, (double)N_CONC_THREADS * OPS_PER_THREAD * 3, 0);
t0 = now_sec();
for (int i = 0; i < N_CONC_THREADS; i++) {
args[i] = (ThreadArgs){ NULL, rawroot, i };
pthread_create(&threads[i], NULL, conc_worker_rawfs, &args[i]);
}
for (int i = 0; i < N_CONC_THREADS; i++) pthread_join(threads[i], NULL);
record("concurrent create+read+unlink", "raw fs", now_sec() - t0, (double)N_CONC_THREADS * OPS_PER_THREAD * 3, 0);
}
/* ---- category 10: mount table scaling ---- */
/* Run at several N (called from main at N_MOUNTS/4, N_MOUNTS, N_MOUNTS*4)
* so the mount table's complexity class can be checked the same way
* Section "Resolution" in BENCH.md checked the file index's: if time(N)
* grows roughly as N^2 rather than N or N log N across a 4x-N step, that
* confirms (not just asserts) that vfs_mount/vfs_unmount's full-array
* copy per structural write (mirroring the file index's old design) is
* quadratic here too — expected, and, per BENCH.md, not worth fixing at
* any mount count a real program would ever reach. */
static void bench_mount_scaling(int n) {
char label[24];
Vfs *v = vfs_new();
Backend **backends = (Backend **)malloc(sizeof(Backend *) * (size_t)n);
char prefix[32];
double t0 = now_sec();
for (int i = 0; i < n; i++) {
backends[i] = backend_mem_new();
snprintf(prefix, sizeof(prefix), "/m%06d", i);
vfs_mount(v, prefix, backends[i]);
}
snprintf(label, sizeof(label), "mount %d backends", n);
record(label, "vfs", now_sec() - t0, n, 0);
/* a resolve through a full mount table, to confirm lookup itself
* (not just mount/unmount) stays fast at this N */
t0 = now_sec();
for (int i = 0; i < n; i++) {
char path[40];
snprintf(prefix, sizeof(prefix), "/m%06d", i);
snprintf(path, sizeof(path), "%s/x.txt", prefix);
VfsStat st;
vfs_stat(v, path, &st); /* NOENT expected; measures resolve cost, not the stat itself */
}
snprintf(label, sizeof(label), "resolve, %d mounts", n);
record(label, "vfs", now_sec() - t0, n, 0);
t0 = now_sec();
for (int i = 0; i < n; i++) {
snprintf(prefix, sizeof(prefix), "/m%06d", i);
vfs_unmount(v, prefix);
}
snprintf(label, sizeof(label), "unmount %d backends", n);
record(label, "vfs", now_sec() - t0, n, 0);
for (int i = 0; i < n; i++) backend_free(backends[i]);
free(backends);
vfs_free(v);
}
/* ---- main ---- */
int main(void) {
char rawroot[] = "/tmp/packfs_bench_raw_XXXXXX";
char dirroot[] = "/tmp/packfs_bench_dir_XXXXXX";
if (!mkdtemp(rawroot) || !mkdtemp(dirroot)) { fprintf(stderr, "mkdtemp failed\n"); return 1; }
char pack_path[256];
snprintf(pack_path, sizeof(pack_path), "/tmp/packfs_bench_%d.pack", (int)getpid());
unlink(pack_path);
memset(g_payload, 0x42, sizeof(g_payload));
printf("=== PackFS vs. host filesystem: benchmark ===\n\n");
printf("environment:\n");
printf(" raw fs test root: %s\n", rawroot);
printf(" dir backend root: %s\n", dirroot);
printf(" small files (N): %d, %d bytes each\n", N_SMALL, SMALL_SIZE);
printf(" directories (N): %d\n", N_DIRS);
printf(" concurrency: %d threads x %d ops (create+read+unlink)\n", N_CONC_THREADS, OPS_PER_THREAD);
printf(" mounts (N): %d\n", N_MOUNTS);
printf(" large-file sizes: ");
for (size_t i = 0; i < N_LARGE_SIZES; i++) printf("%zuMB ", LARGE_SIZES[i] / (1024 * 1024));
printf("\n");
printf(" NOTE: see the file header for what \"raw fs\" vs \"raw+fsync\" vs\n");
printf(" \"dir\" actually measure — they are not interchangeable.\n");
Vfs *v = vfs_new();
Backend *mem = backend_mem_new();
vfs_mount(v, "/", mem);
section("1. create N small files");
bench_create_mem(v);
bench_create_rawfs(rawroot, 0);
bench_create_rawfs(rawroot, 1);
section("2. read N small files");
bench_read_mem(v);
bench_read_rawfs(rawroot);
section("3. stat N files");
bench_stat_mem(v);
bench_stat_rawfs(rawroot);
section("4. readdir");
bench_readdir_mem(v);
bench_readdir_rawfs(rawroot);
/* ---- dir backend, same operations, same file count, its own root ---- */
Vfs *vd = vfs_new();
int derr = 0;
Backend *dir = backend_dir_new(dirroot, &derr);
if (!dir) { fprintf(stderr, "backend_dir_new failed: %d\n", derr); return 1; }
vfs_mount(vd, "/", dir);
section("1b. create N small files (dir backend vs. what it wraps)");
bench_create_dirbackend(vd);
section("2b. read N small files (dir backend)");
bench_read_dirbackend(vd);
section("3b. stat N files (dir backend)");
bench_stat_dirbackend(vd);
section("4b. readdir (dir backend)");
bench_readdir_dirbackend(vd);
section("5. unlink N files");
bench_unlink_mem(v);
bench_unlink_dirbackend(vd);
bench_unlink_rawfs(rawroot);
section("6. mkdir/rmdir N directories");
bench_mkdir_mem(v);
bench_mkdir_dirbackend(vd);
bench_mkdir_rawfs(rawroot);
section("7. large sequential write/read");
for (size_t i = 0; i < N_LARGE_SIZES; i++) bench_large_mem(v, LARGE_SIZES[i]);
for (size_t i = 0; i < N_LARGE_SIZES; i++) bench_large_rawfs(rawroot, LARGE_SIZES[i], 0);
for (size_t i = 0; i < N_LARGE_SIZES; i++) bench_large_rawfs(rawroot, LARGE_SIZES[i], 1);
/* ---- pack: build one, then random-access read it ---- */
section("8. random-access read: mmap'd pack vs. raw fs");
{
/* re-populate mem with N_SMALL files, then compact into a pack */
for (int i = 0; i < N_SMALL; i++) {
char path[64]; fmt_name(path, sizeof(path), "/f", i);
int err = 0;
VfsFile *f = vfs_open(v, path, VFS_O_WRONLY | VFS_O_CREAT, &err);
vfs_write(f, g_payload, SMALL_SIZE);
vfs_close(f);
}
Backend *ovmem = backend_mem_new();
int oerr = 0;
Vfs *vc = vfs_new();
Backend *ov = backend_overlay_new(pack_path, ovmem, &oerr);
vfs_mount(vc, "/", ov);
for (int i = 0; i < N_SMALL; i++) {
char path[64]; fmt_name(path, sizeof(path), "/f", i);
int err = 0;
VfsFile *f = vfs_open(vc, path, VFS_O_WRONLY | VFS_O_CREAT, &err);
vfs_write(f, g_payload, SMALL_SIZE);
vfs_close(f);
}
double t0 = now_sec();
vfs_sync(vc, "/");
double compact_dt = now_sec() - t0;
record("compact N entries to pack", "pack", compact_dt, N_SMALL, (double)N_SMALL * SMALL_SIZE);
vfs_unmount(vc, "/");
backend_free(ov);
backend_free(ovmem);
vfs_free(vc);
/* same N files as plain files on the raw fs, for the random-read comparison */
for (int i = 0; i < N_SMALL; i++) {
char name[64], path[256];
fmt_name(name, sizeof(name), "/f", i);
snprintf(path, sizeof(path), "%s%s", rawroot, name);
int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, g_payload, SMALL_SIZE);
close(fd);
}
int *order = malloc(sizeof(int) * N_SMALL);
srand(12345);
for (int i = 0; i < N_SMALL; i++) order[i] = i;
for (int i = N_SMALL - 1; i > 0; i--) { int j = rand() % (i + 1); int t = order[i]; order[i] = order[j]; order[j] = t; }
Vfs *vp = vfs_new();
int perr = 0;
Backend *ro = backend_pack_new(pack_path, &perr);
vfs_mount(vp, "/", ro);
bench_pack_random(pack_path, order, vp);
bench_rawfs_random(rawroot, order);
vfs_unmount(vp, "/");
backend_free(ro);
vfs_free(vp);
free(order);
}
section("9. concurrency (create+read+unlink)");
bench_concurrency(v, rawroot);
section("10. mount table scaling (the mount table uses the same full-snapshot-copy pattern the file index used to)");
bench_mount_scaling(N_MOUNTS / 4);
bench_mount_scaling(N_MOUNTS);
bench_mount_scaling(N_MOUNTS * 4);
vfs_unmount(v, "/");
backend_free(mem);
vfs_free(v);
vfs_unmount(vd, "/");
backend_free(dir);
vfs_free(vd);
/* cleanup */
char cmd[1200];
snprintf(cmd, sizeof(cmd), "rm -rf '%s' '%s' '%s' '%s.jnl' '%s.tmp'", rawroot, dirroot, pack_path, pack_path, pack_path);
if (system(cmd) != 0) { /* best-effort cleanup; nothing to do if it fails */ }
printf("\n=== summary (%d measurements) ===\n", g_row_count);
printf(" %-28s %-10s %11s %16s %12s\n", "category", "backend", "seconds", "throughput", "MB/s");
for (int i = 0; i < g_row_count; i++) {
Row *r = &g_rows[i];
printf(" %-28s %-10s %10.4fs %14.0f ops/s", r->category, r->backend, r->seconds, r->ops_per_sec);
if (r->mb_per_sec > 0) printf(" %10.1f MB/s", r->mb_per_sec); else printf("%15s", "");
printf("\n");
}
return 0;
}