local machine benchmarks

single threaded: Time Jest Rust: 33.63373279571533

rayon: Time Jest Rust: 4.294418811798096

tokio: Time Jest Rust: 4.717588901519775

compile options benchmarks

lto not thin: Time Jest Rust: 5.306957483291626 slower

lto fat: Time Jest Rust: 5.413678407669067 slower

codegen-units 1: Time Jest Rust: 4.451631546020508 faster

opt-level z: Time Jest Rust: 7.045313119888306 slower

strip true: Time Jest Rust: 4.337219476699829 faster

lto true: Time Jest Rust: 4.703521728515625 slower

lto none: Time Jest Rust: 4.817203998565674

lto thin: Time Jest Rust: 4.429729223251343 faster

data integrity

(this isn't tested, just guessed, and I don't have data to compare it with)

for loops:

file count: 904
failed file count: 0
sentence count: 5602301
word count: 81701260
capitalized count: 1753639
numeric count: 14981248
forbidden count: 1237059
words per sentence average: 14.6
forbidden word percentage: 2%
capitalized word percentage: 2%

benchmark: 5033ms

muncher:

file count: 904
failed file count: 0
sentence count: 5338705
word count: 86765116
capitalized count: 13640820
numeric count: 10902254
forbidden count: 0
words per sentence average: 16.3
forbidden word percentage: 0%
capitalized word percentage: 16%

benchmark: 504ms

with forbidden words:

file count: 904
failed file count: 0
sentence count: 5338705
word count: 86765116
capitalized count: 13640820
numeric count: 10902254
forbidden count: 279717
words per sentence average: 16.3
forbidden word percentage: 0%
capitalized word percentage: 16%

benchmark: 6078ms

forbidden words benchmarks

seems they take up about 4000ms to churn through in the original version

for loops count forbidden word once only:

file count: 904
failed file count: 0
sentence count: 5602301
word count: 81701260
capitalized count: 1753639
numeric count: 14981248
forbidden count: 1143234
words per sentence average: 14.6
forbidden word percentage: 1%
capitalized word percentage: 2%

benchmark: 4737ms

for loops with trie:

file count: 904
failed file count: 0
sentence count: 5602301
word count: 81701260
capitalized count: 1753639
numeric count: 14981248
forbidden count: 176528
words per sentence average: 14.6
forbidden word percentage: 0%
capitalized word percentage: 2%

benchmark: 1588ms

muncher with trie is 2600ms

for loops with fxhash trie: 1200ms

ubuntu terminal running

https://snek.molodetz.nl/terminal.html ubuntu running thing instructions:

mkdir /project
cd /project
git clone https://retoor.molodetz.nl/retoor/isspam.git
apt install valgrind curl
export RUSTUP_HOME=/project/.rustup
export CARGO_HOME=/project/.cargo
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "/project/.cargo/env"
cd isspam
rustup install nightly
rustup default nightly
make
make benchmark
python3 bench.py

clone: git clone https://gitlab.com/jestdotty-group/draft/jisspam.git jest_rust

edit make: vi makefile and add build:

build_jest:
	@echo "compiling jest_rust project"
	cd jest_rust && cargo build --release && cp target/release/jisspam ..

append to all script:

all: build run valgrind build_risspam run_risspam build_cpp build_borded_cpp build_py build_jest

add to bench: vi bench.py

time_start = time.time()
subprocess.check_output('./jisspam books/*.txt', shell=True)
print("Time Jest Rust:", time.time() - time_start)

run: python3 bench.py output looks something like this:

***benchmarking***
Time C: 31.315868377685547
Time Rust: 41.232205867767334
Time CPP: 20.1683189868927
Time Borded CPP: 15.468477964401245
Time Jest Rust: 54.74523115158081
Time Retoor Python: 287.63036131858826
***end benchmark***

add /jisspam to .gitignore to not commit the executable accidentally

..
src
.gitignore
Cargo.toml
README.md