JestDotty
7c87f28599
Implement a character-level muncher parser as an alternative to the existing for-loop approach for text statistics. The muncher processes text by tracking whitespace and dot states to count words, sentences, capitalized words, and numeric characters. Benchmark results show the muncher completes in 504ms versus 5033ms for for-loops, though with different counts for capitalized words (16% vs 2%) and forbidden words (0% vs 2%) due to implementation differences. The README now includes both benchmark comparisons and data integrity notes documenting the statistical discrepancies between the two methods.
for https://retoor.molodetz.nl/retoor/isspam
https://snek.molodetz.nl/terminal.html ubuntu running thing instructions:
mkdir /project
cd /project
git clone https://retoor.molodetz.nl/retoor/isspam.git
apt install valgrind curl
export RUSTUP_HOME=/project/.rustup
export CARGO_HOME=/project/.cargo
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "/project/.cargo/env"
cd isspam
rustup install nightly
rustup default nightly
make
make benchmark
python3 bench.py
clone: git clone https://gitlab.com/jestdotty-group/draft/jisspam.git jest_rust
edit make: vi makefile and add build:
build_jest:
@echo "compiling jest_rust project"
cd jest_rust && cargo build --release && cp target/release/jisspam ..
append to all script:
all: build run valgrind build_risspam run_risspam build_cpp build_borded_cpp build_py build_jest
add to bench: vi bench.py
time_start = time.time()
subprocess.check_output('./jisspam books/*.txt', shell=True)
print("Time Jest Rust:", time.time() - time_start)
run: python3 bench.py
output looks something like this:
***benchmarking***
Time C: 31.315868377685547
Time Rust: 41.232205867767334
Time CPP: 20.1683189868927
Time Borded CPP: 15.468477964401245
Time Jest Rust: 54.74523115158081
Time Retoor Python: 287.63036131858826
***end benchmark***
add /jisspam to .gitignore to not commit the executable accidentally
local machine benchmarks
single threaded:
***benchmarking***
Time C: 2.4082751274108887
Time Rust: 2.865687847137451
Time CPP: 1.1568822860717773
Time Borded CPP: 1.9657189846038818
Time Jest Rust: 33.63373279571533
Time Retoor Python: 133.92413425445557
***end benchmark***
rayon:
***benchmarking***
Time C: 2.457853317260742
Time Rust: 3.0170154571533203
Time CPP: 1.1482579708099365
Time Borded CPP: 2.002591371536255
Time Jest Rust: 4.294418811798096
Time Retoor Python: 201.2997748851776
***end benchmark***
tokio:
***benchmarking***
Time C: 2.448648452758789
Time Rust: 3.095592737197876
Time CPP: 1.1662013530731201
Time Borded CPP: 1.9207634925842285
Time Jest Rust: 4.717588901519775
Time Retoor Python: 139.8203284740448
***end benchmark***
lto not thin: Time Jest Rust: 5.306957483291626 slower
lto fat: Time Jest Rust: 5.413678407669067 slower
codegen-units 1: Time Jest Rust: 4.451631546020508 faster
opt-level z: Time Jest Rust: 7.045313119888306 slower
strip true: Time Jest Rust: 4.337219476699829 faster
lto true: Time Jest Rust: 4.703521728515625 slower
lto none: Time Jest Rust: 4.817203998565674
lto thin: Time Jest Rust: 4.429729223251343 faster
data integrity
(this isn't tested, just guessed, and I don't have data to compare it with) for loops:
file count: 904
failed file count: 0
sentence count: 5602301
word count: 81701260
capitalized count: 1753639
numeric count: 14981248
forbidden count: 1237059
words per sentence average: 14.6
forbidden word percentage: 2%
capitalized word percentage: 2%
benchmark: 5033ms
muncher:
file count: 904
failed file count: 0
sentence count: 5338705
word count: 86765116
capitalized count: 13640820
numeric count: 10902254
forbidden count: 0
words per sentence average: 16.3
forbidden word percentage: 0%
capitalized word percentage: 16%
benchmark: 504ms