Profiled a real search with Massif and found the memory-per-input-byte
multiplier at 13.4x, dominated by two avoidable costs:
- build_matbuf widened every BINARY/ASCII byte into a uint32_t before
matching, a 4x copy that mode never needed (a byte never exceeds
255). Removed it: MCtx/MatBuf now carry an optional text8 (borrowed,
unwidened) alongside the existing UTF8 text (owned, decoded code
points), reconciled per-read through text_at()/buf_at(). BINARY/
ASCII mode now points directly into the Input's own buffer.
- Input_from_file always copied the whole file into a malloc'd buffer.
It now mmaps regular files read-only (MAP_PRIVATE) instead, so pages
stay clean and reclaimable under memory pressure and the file is
never copied. Non-seekable sources (pipes, FIFOs, process
substitution, stdin) and mmap failures fall back to the previous
incremental-read behavior via a separate read_fd_incrementally.
Together these bring the multiplier to 8.4x and let a 200MB file that
previously OOM-crashed complete a full non-matching search in about 6
seconds at roughly 1.69GB peak RSS.
Also tried, measured, and reverted: capping compute_maxrun/
compute_next_prevmatch's table size with a plain-scan fallback above
the cap. A real 200MB non-matching search against this fallback hung
for minutes instead of failing fast, because disabling either table
reintroduces the O(n^2) behavior they exist to prevent, and O(n^2) at
n in the hundreds of millions is not practically finite. A fast,
diagnosable allocation failure is a better failure mode than a silent,
unbounded hang, so the tables are allocated unconditionally again; the
finding is recorded in code comments, concept.md 7.6, and README's
"Memory footprint" section so it is not retried blindly later.
Separately audited every allocation on an input-proportional path
(da_push, build_matbuf's UTF-8 decode loop, all three Input_from_file
sites) and made each fail cleanly through PatternError instead of
crashing on an unchecked NULL dereference.
A 1GB file still exceeds available memory in the current environment;
this is a property of the machine it was measured on, not a defect,
and is documented as such (practical ceiling: available memory / 8.4
for search-family operations, pending the streaming automaton design
in concept.md 7.2).
Verified with four clean `make test` passes (3252/3252) and a clean
ASan/UBSan pass after the change; rxgrep's mmap-backed paths (--sub
with a regular file, with a non-seekable process-substitution source,
and BINARY-mode embedded NUL handling) re-checked directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjuMk8kY9SDus1wWe2K9xY
retoorandClaude Sonnet 5
b24d63e5fd