retoor/isspam
2
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: retoor/isspam:3cd96f670b1bb12de84320a2dbd5f33ea0fb1736
Branches Tags
retoor/isspam:main
..
compare: retoor/isspam:main
Branches Tags
retoor/isspam:main
93 Commits
3cd96f670b .. main
Author SHA1 Message Date
retoor 903ff442fc feat: add profile-guided optimization build step and aggressive compiler flags to Makefile
isspam build / build (push) Successful in 2m58s
Details 
- Replace basic -O3 with -Ofast and add PGO instrumentation (fprofile-generate) and usage (fprofile-use) stages in the default 'build' target
- Introduce new 'build_quick' target for compilation without PGO
- Add numerous performance-oriented flags: fipa-pta, fipa-cp-clone, fdevirtualize-speculatively, fno-plt, fno-semantic-interposition, fomit-frame-pointer, fmerge-all-constants, fno-stack-protector, DNDEBUG, and floating-point relaxation flags
- Update linker flags to use flto=auto and fuse-linker-plugin
- Include training runs on spam, not_spam, and optional books directories during PGO instrumentation phase
 2026-02-03 09:55:16 +00:00
retoor 2f0d088beb feat: add Swift implementation of isspam (sisspam) with CI build and benchmark integration 
- Add swift_isspam package with optimized main.swift using unsafe memory flags
- Extend CI workflow to install Swift 5.9.2 and include sisspam in build_all and benchmark
- Update Makefile with build_swift target and publish step for sisspam binary
- Add sisspam to .gitignore and README with author credit
- Enable Swift timing in bench.py alongside existing language implementations
 2026-02-03 01:24:34 +00:00
retoor 39c9bfb36f chore: remove Rust-based trie spam detection and restore pure C implementation 2025-12-14 20:09:30 +00:00
retoor d86aab450c feat: replace naive text analysis with SIMD-optimized trie-based multi-threaded spam detector 
Rewrite the isspam.c program to use memory-mapped files, a compact trie data structure for forbidden word lookup, and pthread-based parallel processing across up to 16 threads. Remove the old fixed-size buffer and linear forbidden word array in favor of a SIMD-accelerated scanning approach with AVX2 instructions. Update the Makefile to enable aggressive x86-64 optimization flags including -march=native, -flto, -ffast-math, and -mavx2 for maximum performance on supported hardware.
 2025-10-07 18:20:58 +00:00
12bitfloat 9118d88eab fix: remove commented-out debug timing and file-reading code from work function 2025-10-06 23:03:56 +00:00
12bitfloat dc3533c8f6 chore: add isspam.py to gitignore to prevent accidental tracking 2025-10-06 22:57:51 +00:00
12bitfloat 5c50a45101 fix: comment out slow python3 benchmark section in bench.py 
The Python benchmark using isspam.py was removed from the benchmark
run because its execution time was excessively slow compared to the
C++ and Rust implementations, skewing overall timing results.
 2025-10-06 22:56:44 +00:00
12bitfloat e516f5e869 feat: add initial SIMD analysis and io_uring module, drop core_affinity and memmap2 deps 
Introduce a new `analyze_simd` function in main.rs using AVX2 256-bit registers for parallel text scanning, replacing the previous sequential analysis path. Add a stub `uring` module for future async I/O integration. Remove `core_affinity` and `memmap2` crate dependencies from Cargo.toml and Cargo.lock, along with their associated `libc`, `num_cpus`, `hermit-abi`, and `winapi` transitive dependencies. Comment out `mimalloc` and `io-uring` crate references. Uncomment file reading timing instrumentation and add a `bench_rust_only` Makefile target for direct benchmark execution.
 2025-10-06 22:55:06 +00:00
12bitfloat 83f3729670 fix: remove cargo env sourcing and prefix build steps with cargo bin path 2025-10-06 22:32:25 +00:00
12bitfloat 7e0bb8ea88 fix: source cargo environment before build steps in gitea workflow 
The CI workflow was failing because the Rust toolchain installed via rustup
was not available in subsequent steps. Adding `. "$HOME/.cargo/env"` ensures
the cargo and rustc binaries are on PATH before the build commands execute.
 2025-10-06 22:18:30 +00:00
12bitfloat 76aaf79365 chore: remove individual build/run steps and replace with combined build_all target 2025-10-06 22:14:21 +00:00
JestDotty 11a4974ff3 chore: remove stale performance comments from for_loops and muncher parsers 2025-10-06 15:58:01 +00:00
12bitfloat 46282684b3 fix: correct .gitignore trailing slashes and add missing risspam rust source files 2025-10-06 15:43:44 +00:00
12bitfloat bb97bff828 docs: add build_all target and benchmark_only shortcut to Makefile, update README with version credits and build instructions 2025-10-06 11:52:54 +00:00
12bitfloat 12d83c4ae1 fix: suppress dead_code and unused import warnings in risspam main.rs 
- Added `#![allow(dead_code)]` attribute to suppress warnings for unused items
- Removed unused imports: `Mmap`, `CoreId`, `OnceCell`, `OsString`, `File`, `assert_unchecked`, `MaybeUninit`, `Deref`, `stat`, `FileExt`, `OpenOptionsExt`, `fs`, `mem`, `thread`
- Commented out unused variable `do_parallel` to silence unused variable warning
- Added `#[allow(unused_mut)]` attribute on `stats` variable with safety justification comment
- Fixed variable name from `num_cores` to `num_threads` in `ThreadPoolBuilder::num_threads()` call
 2025-10-06 11:45:18 +00:00
12bitfloat 2468b85451 perf: add core_affinity and memmap2 dependencies, switch to edition 2024, and rewrite main.rs with mmap-based parallel file processing 2025-10-06 11:39:29 +00:00
JestDotty 1cc1df976c docs: add benchmark observation that FxHashMap outperforms BTreeMap in README 2025-10-04 14:03:15 +00:00
JestDotty 8c5938786e chore: add for_loops_forbidden_only parser and switch to fat LTO in release profile 2025-10-04 13:58:40 +00:00
JestDotty 93cf77cb96 refactor: extract parser module with muncher and for_loops submodules from main.rs 
Move Stats processing methods into dedicated parser module with separate muncher and for_loops implementations, update .gitignore to exclude test_books directory, and simplify README to remove outdated benchmark instructions
 2025-10-04 13:18:03 +00:00
JestDotty 305d2d5243 perf: move file reading out of async tasks to reduce sequential I/O overhead 
The change shifts `fs::read_to_string` from inside each spawned async task into the main synchronous loop before spawning. This eliminates redundant sequential file reads across concurrent tasks, yielding a ~30ms improvement on NVMe SSD workloads. The `Stats::process` method now accepts a pre-read `&str` instead of a file path, and failed file counts are tracked directly in the main loop.
 2025-03-24 20:03:36 +00:00
JestDotty 8bfd58b613 docs: add compile options benchmarks and restructure forbidden words benchmarks in README 2025-03-24 04:48:04 +00:00
JestDotty 62310e09a5 feat: replace thread-local LazyCell with static LazyLock and add fxhash-backed trie children for 24% speedup 2025-03-24 04:20:15 +00:00
JestDotty 40cb4f8cff feat: replace static forbidden word list with trie-based lookup for 3x speedup 
Extract Stats struct into its own module with public fields and AddAssign/Display impls; introduce Trie data structure with insert and contains methods; refactor main.rs to use thread-local LazyCell<Trie> populated from the same forbidden word set; update README with trie benchmark results showing 1588ms vs 4737ms original.
 2025-03-24 03:58:34 +00:00
JestDotty 12a95bb8d8 fix: add break to forbidden word loops to count each word only once in for_loops method 2025-03-24 03:23:40 +00:00
JestDotty 7c87f28599 feat: add muncher-based text analysis with benchmark data and README results 
Implement a character-level muncher parser as an alternative to the existing for-loop approach for text statistics. The muncher processes text by tracking whitespace and dot states to count words, sentences, capitalized words, and numeric characters. Benchmark results show the muncher completes in 504ms versus 5033ms for for-loops, though with different counts for capitalized words (16% vs 2%) and forbidden words (0% vs 2%) due to implementation differences. The README now includes both benchmark comparisons and data integrity notes documenting the statistical discrepancies between the two methods.
 2025-03-24 03:14:30 +00:00
JestDotty 4d42726854 chore: remove jest_rust Cargo.lock from version control to stop tracking generated dependencies 2025-03-24 02:31:14 +00:00
JestDotty 32fda51edd feat: replace rayon parallel iteration with tokio async mpsc channels for file processing 2025-03-24 02:29:54 +00:00
JestDotty 144fc9b2ac chore: add rayon parallel dependency and release profile optimizations to jest_rust 
Add rayon 1.10.0 as a dependency in Cargo.toml and refactor Stats processing into a dedicated method that uses `par_iter()` for parallel file processing. Configure release profile with thin LTO, disabled debug symbols, and abort-on-panic for maximum runtime performance. Update README with build and benchmark integration instructions for the jest_rust project.
 2025-03-24 01:39:12 +00:00
retoor 114e596105 chore: add isspam.py to gitignore and refactor forbidden_words list to set with ThreadPoolExecutor 2025-03-24 01:31:28 +00:00
Jest Dotty 5efa63f648 chore: flatten jest_rust directory by removing nested jisspam subdirectory 
Remove the intermediate jisspam subdirectory within jest_rust, moving Cargo.lock, .gitignore, and test_files (spam1.txt, spam2.txt, spam3.txt, not_spam.txt) up one level. Update Makefile build_jest target to compile directly from jest_rust instead of jest_rust/jisspam and copy binary to parent directory.
 2025-03-24 00:23:38 +00:00
Jest Dotty d1630e5a9c feat: add jest_rust jisspam spam detector with build target and benchmark integration 
Add new Rust-based spam detection project under jest_rust/jisspam with Cargo.toml, main.rs implementing forbidden word analysis, and test files for spam/not-spam classification. Include build_jest target in Makefile, add jisspam binary to .gitignore, and integrate its execution into bench.py benchmark suite alongside existing C, CPP, and Python implementations.
 2025-03-24 00:03:09 +00:00
BordedDev dee74d4ddb fix: remove TBB library link from borded C++ build target in Makefile 
The `-ltbb` flag was removed from the `build_borded_cpp` target's g++ command
in the Makefile, eliminating the dependency on Intel Threading Building Blocks
for the borded C++ executable compilation.
 2025-03-23 22:37:31 +00:00
BordedDev ae2d5a7b9d fix: swap benchmark order to measure Rust after C instead of before 2025-03-23 21:40:43 +00:00
BordedDev b3f6056e9c chore: reorder benchmark execution order and add TBB-linked main3.cpp variant 
- Swapped benchmark order in bench.py to run Rust before C for consistent measurement
- Added new main3.cpp source with parallel TBB-based implementation using async I/O and CRC32
- Updated Makefile to link borded_cpp_exec against libtbb and compile from main3.cpp
- Modified CMakeLists.txt to add main3.cpp target with TBB linkage and disable -Werror
 2025-03-23 21:06:48 +00:00
BordedDev 4729cb0a7a feat: add main2.cpp without struct and refactor check_word to exact match 
Replace substring-based bad word detection with exact match in check_word, and create main2.cpp as a variant that removes the AnalysisResult struct entirely while preserving all other logic.
 2025-03-23 02:36:29 +00:00
retoor 219650f4f1 feat: add build_py target and benchmark for Python isspam implementation 
Add a new Makefile target to copy the Python isspam script into the project root, and extend the benchmark script to measure execution time for the Python version alongside existing C and C++ implementations.
 2025-03-23 02:25:51 +00:00
BordedDev 49a2136030 fix: replace std::vector with constexpr std::array for BAD_WORDS and add utf8-aware print fallback in main.cpp 2025-03-23 01:13:38 +00:00
retoor 2848d1b80c feat: add python-based spam detection script with file analysis and forbidden word checking 
Implement a new Python script `isspam.py` that analyzes text files for spam indicators by counting total words, capitalized words, sentences, numbers, and occurrences of a predefined list of 40 forbidden words. The script supports multi-file input via command-line arguments, reads files in 1024-byte chunks, and uses threading for concurrent analysis.
 2025-03-22 22:55:48 +00:00
BordedDev ed54602bf4 feat: add borded_cpp benchmark target and switch Dockerfile to gcc 
Add borded_cpp_exec to .gitignore, include build_borded_cpp in Makefile all target, and extend bench.py with Borded CPP timing. Change borded_cpp Dockerfile base from alpine to gcc:latest with apt-based dependencies.
 2025-03-20 22:32:24 +00:00
BordedDev 1dfc4a495d feat: add build target for borded_cpp and lower cmake minimum version to 3.25 
Add a new `build_borded_cpp` make target that compiles the borded_cpp
version of isspam using C++23 with `-Ofast` optimization. Also reduce
the CMake minimum required version from 3.30 to 3.25 in the borded_cpp
CMakeLists.txt to improve compatibility with older CMake installations.
 2025-03-20 22:21:33 +00:00
retoor e17945027b feat: add Dockerfile, compose.yml, and doit.sh for C++ build environment 
Introduce a Docker-based development setup for the borded_cpp project. The Dockerfile uses Alpine Linux with build-base, SQLite, Jansson, and CMake. The compose.yml mounts the project directory and a shared books volume, runs doit.sh which cleans, configures, and builds the project via CMake.
 2025-03-20 21:52:53 +00:00
retoor be7dda9daf fix: replace C++23 modules and print with C++17 headers and iostream in borded_cpp 
Replace the `import std;` module directive with explicit C++17 standard library includes
and substitute `std::println` calls with `std::cout` equivalents to ensure compatibility
with older compiler toolchains. Also add an overloaded `operator<<` for `AnalysisResult`
to enable formatted output via iostream, and convert `std::wifstream::open` to accept a
`std::string` instead of `std::string_view` for broader C++17 support.
 2025-03-20 21:23:41 +00:00
BordedDev 67640f034f feat: add initial C++ project structure with CMake build and spam word parser 
Implement the first version of the borded spam parser as a C++26 project. The commit introduces a complete project skeleton including a `.gitignore` file for CMake and IDE artifacts, a `CMakeLists.txt` with strict warning flags and MSVC compatibility, and the core `src/main.cpp` containing the spam detection logic. The parser reads text files, tokenizes words, counts sentences and capitalized words, and flags occurrences from a predefined list of 35 forbidden spam-related terms (e.g., "crypto", "bitcoin", "recovery", "hack"). Results are aggregated via an `AnalysisResult` struct with a custom `operator+` for combining multiple file analyses.
 2025-03-20 20:44:22 +00:00
retoor 3aa6ac2451 chore: delete all isspam_v1 through v4 Rust spam analysis files 2025-03-20 14:40:04 +00:00
retoor ca42ad8abc feat: add C++ port of isspam with updated tokenizer and benchmark integration 
Add a new C++ implementation of the isspam text analysis tool in retoor_c/isspam.cpp, mirroring the C version's functionality with thread-based file analysis. Update the Makefile with a build_cpp target and add the new binary to .gitignore. Modify bench.py to include timing for the C++ executable. Fix the C tokenizer's delimiter string from punctuation-only to include all standard whitespace characters (\f\v\r\n\t).
 2025-03-20 14:27:02 +00:00
retoor 4b44ad85ec chore: add .r_history to gitignore and strip strict compiler flags from Makefile 2025-03-20 00:18:41 +00:00
retoor d4867f571e fix: add explicit exit(0) call before return in isspam.c main function 2025-03-19 20:07:22 +00:00
retoor 0d6afe12f2 chore: add target, isspam, and risspam entries to .gitignore 2024-12-04 22:39:05 +00:00
retoor 6042864d56 chore: remove entire target directory with compiled artifacts and cached metadata 2024-12-04 22:38:26 +00:00
retoor f3a93b6cbc feat: replace forked child processes with pthreads for concurrent file analysis in main 
Replace the fork-based parallel execution in main() with POSIX threads, adding a new analyze_file_thread wrapper function. The change removes per-file printf output and eliminates the separate child process for each argument, instead creating a thread per file, joining all threads, and freeing the allocated thread array. This improves resource sharing and avoids process overhead for concurrent file analysis.
 2024-12-04 22:10:45 +00:00
retoor 79965cfbb1 fix: suppress command echoing in publish target by prefixing with @ 2024-12-02 14:30:51 +00:00
retoor 03fafdf101 fix: correct typo in wget flag for publish download command 
The diff shows a single character change in the Makefile's publish target, where the wget command for downloading the publish script had a space between the double dash and the quiet flag (`-- quiet`), which has been corrected to the proper format (`--quiet`). This fix ensures the wget command executes correctly without flag parsing errors.
 2024-12-02 14:30:10 +00:00
retoor 7a7c28543b chore: update build trigger timestamp and add quiet flag to wget in Makefile 
- Bump .build-trigger timestamp from 2014-12-02 15:22 to 15:26 to force rebuild
- Add --quiet flag to both wget calls in publish target to suppress download progress output
 2024-12-02 14:27:08 +00:00
retoor fb71c4a7b6 chore: add .build-trigger-2014-12-02 15:22 marker to .gitignore for build automation 2024-12-02 14:22:52 +00:00
retoor 6f6e78940e chore: suppress command echoing in benchmark target by adding @ prefix to all commands 2024-12-02 14:16:31 +00:00
retoor eec8569e08 chore: remove redundant cargo env source and reorder build steps in CI workflow 
- Remove duplicate `source $HOME/.cargo/env && make` step that was redundant with later build commands
- Move cargo environment sourcing to `build_risspam` step where it is actually needed for compilation
- Keep `make build` and `make run` steps without explicit cargo env to rely on default PATH setup
 2024-12-02 14:13:03 +00:00
retoor 3de0edd36a chore: add __pycache__ to .gitignore and remove tracked pyc file 
The __pycache__ directory was being tracked by git, causing compiled Python bytecode files like env.cpython-312.pyc to appear in the repository. This change adds __pycache__ to .gitignore and removes the existing pyc file from version control.
 2024-12-02 14:10:58 +00:00
retoor b27cc00f58 chore: add build, run, benchmark and publish steps to CI workflow and introduce isspam_v4 spam analysis module 2024-12-02 14:10:18 +00:00
retoor 05998da4a0 chore: update compiled risspam binary in 12bitfloat_rust and root directories 2024-12-01 22:24:22 +00:00
retoor 524ef8343c feat: refactor sentence and word analysis to use counts instead of collections 
Rewrite `get_sentences` to return a `usize` count instead of `Vec<&str>`, removing the trailing-dot cleanup logic. Replace `get_words` and `get_capitalized_words` with a single `get_words` function that mutates output counters for total words, capitalized words, and forbidden words, iterating directly over the content. Add release profile optimizations with thin LTO and abort-on-panic in Cargo.toml.
 2024-12-01 22:23:32 +00:00
retoor 6ce4c8b9cd feat: add forking per-file processing in C version and enable parallel flag in Rust benchmark 
Add fork() call in retoor_c/isspam.c main loop to spawn child process for each input file, allowing parallel analysis. Update bench.py to pass -p flag to Rust binary risspam for enabling parallel execution mode. Rebuild isspam binary to reflect the C source changes.
 2024-12-01 21:32:23 +00:00
retoor a5fb2b9c3c chore: add books directory to gitignore and commit crossbeam fingerprint artifacts 
Add 'books' to .gitignore to exclude it from version control, and include generated Rust build fingerprint files for crossbeam-deque, crossbeam-epoch, crossbeam-utils, either, and rayon dependencies under 12bitfloat_rust/risspam/target/release/.fingerprint/
 2024-12-01 21:03:07 +00:00
retoor ef3c9b5c5b feat: add benchmark target to Makefile and optimize isspam.c with stricmp 
- Add 'benchmark' target to Makefile that extracts books archive and runs bench.py
- Reorder includes in isspam.c (rstr.h before rstring_list.h)
- Replace file_exists() with case-insensitive stricmp() for forbidden word matching
- Remove unused flags (show_capitalized, show_sentences, etc.) and stripws() function
- Add memory usage comment and fix trailing newline in rstr.h header guard
 2024-12-01 21:02:32 +00:00
retoor 8b7f05b129 refactor: remove redundant sentence splitting and word extraction in isspam analysis functions 
Refactor `get_capitalized_words` and `get_forbidden_words` to accept pre-extracted word lists instead of raw content strings, eliminating duplicate calls to `get_words` and `get_sentences`. Move word extraction into the caller `analyze` to compute word count once and reuse the list across all analysis steps, reducing memory allocations and freeing overhead.
 2024-12-01 18:40:01 +00:00
retoor 074a5f8779 chore: bump Cargo.lock format to v4 and add rayon dependency for risspam 
- Update Cargo.lock version from 3 to 4
- Add rayon 1.10.0 with core, crossbeam-deque, crossbeam-epoch, crossbeam-utils, and either dependencies
- Register rayon as a dependency in the risspam package
- Update compiled release binary for risspam and top-level risspam binary
 2024-11-30 22:48:02 +00:00
retoor 08c9e97298 feat: add rayon parallel processing and refactor get_numbers to accept clean_content slice 
- Add rayon dependency to Cargo.toml and import parallel prelude in both isspam_v3.rs and risspam/src/main.rs
- Change get_numbers signature to take &str instead of &str and return Vec<&str> instead of Vec<String>, removing unnecessary allocation
- Guard word_count_per_sentence calculation against empty sentences list to prevent division by zero
- Enable file reading from command-line arguments in isspam_v1.rs by uncommenting and fixing the arg loop with skip(1)
- Remove embedded SPAM1 static test data from isspam_v1.rs and create new isspam_v2.rs with additional analysis functions
 2024-11-30 22:46:08 +00:00
retoor 9a67649df6 docs: add blank lines between sections and fix Valgrind status formatting in README 
Improve readability of the README by inserting blank lines between major sections
and reformatting the Valgrind status paragraph to separate the Rust variant comment
from the date line, ensuring consistent markdown structure throughout the document.
 2024-11-30 21:13:27 +00:00
retoor 5100a36421 fix: move cargo env sourcing into make step and fix build echo target name 2024-11-30 21:11:41 +00:00
retoor 6fb8e59311 docs: add version overview and clarify build/run instructions in readme 
- Document that repository contains both Rust (risspam) and C (isspam) implementations
- Update build and run examples to distinguish between the two versions
- Clarify stdin usage works only for isspam and valgrind status applies to isspam only
 2024-11-30 21:08:28 +00:00
retoor e7725601b2 feat: move source files into retoor_c directory and add build echo messages 
- Add echo messages for both build targets to indicate which project is being compiled
- Update isspam.c path to reflect new retoor_c/ subdirectory structure
- Keep binary output location unchanged at project root
 2024-11-30 21:05:06 +00:00
retoor 17ff82c199 feat: add compressed archive of book files to repository 
The archive books.tar.gz has been added, containing the initial set of book data files for the project.
 2024-11-30 20:39:14 +00:00
retoor 04c1cf45cb fix: correct cargo environment source path in CI workflow 
The CI workflow was sourcing the cargo binary directory ($HOME/.cargo/bin) instead of the environment setup script ($HOME/.cargo/env), which prevented proper initialization of Rust toolchain paths during build steps.
 2024-11-30 20:32:18 +00:00
retoor d053dac468 fix: source cargo bin before checkout in build workflow to fix PATH 2024-11-30 20:30:48 +00:00
retoor bab72cd97e fix: source cargo environment before running make in build workflow 2024-11-30 20:28:30 +00:00
retoor ca71cc5177 ci: remove --profile minimal flag from rustup install in build workflow 2024-11-30 20:25:18 +00:00
retoor 9967ea9a6c feat: add rust toolchain installation and build step for risspam in CI 
Add curl to apt install list and install Rust nightly via rustup in the CI workflow. Update Makefile all target to include build_risspam before run_risspam, ensuring the Rust-based risspam binary is compiled before execution.
 2024-11-30 20:10:24 +00:00
retoor 857be54db5 chore: remove valgrind target from build workflow and simplify CI step 2024-11-30 20:04:32 +00:00
retoor 3e1d5e3619 fix: correct target name from rispam to risspam in Makefile all and run_risspam dependencies 2024-11-30 20:04:01 +00:00
retoor e9adda3804 chore: move risspam binary to new location in repository structure 2024-11-30 19:59:27 +00:00
retoor c6311b10fe feat: add initial Rust spam detection module with isspam and risspam variants 
Implement core spam analysis functions including content cleaning, sentence splitting, capitalized word detection, and forbidden word filtering across three Rust source files with nightly let_chains feature
 2024-11-30 19:58:19 +00:00
retoor 79e2dc4819 feat: add rust implementation with build and run targets in Makefile 
Add build_risspam target that compiles the Rust risspam binary from
12bitfloat_rust/risspam using cargo run --release, and add run_risspam
target with spam and not_spam test execution via run_spam_risspam and
run_not_spam_risspam recipes.
 2024-11-30 19:58:00 +00:00
retoor c8abc02688 fix: update valgrind status date to 2024-11-30 with refreshed heap summary 2024-11-30 19:34:39 +00:00
retoor b607207647 feat: add per-file analysis counter and percentage outputs to isspam, plus totals.py summarizer 
Introduce a global `total` counter incremented on each `analyze()` call, printing the file sequence number. Extend the analysis block to compute and display capitalized and forbidden word percentages when forbidden words exist. Add a new Python script `totals.py` that reads the application output and computes average percentages across all processed files.
 2024-11-30 19:32:55 +00:00
retoor 48e59e2bd8 fix: prevent division by zero in word count per sentence calculation 
The commit updates the README example output to reflect a spam file analysis with
detailed forbidden word breakdown and increased memory usage. It also fixes a
potential division by zero in `analyze()` by guarding `sentences->count` with a
ternary operator that defaults to 1 when zero, ensuring stable output for edge
cases. The binary `isspam` is rebuilt to include this fix.
 2024-11-29 06:54:49 +00:00
retoor baa1ffe9c2 fix: change word_count_per_sentence type from uint to ulonglong in analyze function to prevent overflow on large text inputs 2024-11-29 06:25:09 +00:00
retoor 0582fd8667 fix: migrate loop counters and count fields from unsigned int to unsigned long long across isspam.c and rstring_list.h 2024-11-29 06:24:30 +00:00
retoor 284544880d fix: remove locale-specific formatting and increase stats buffer size 
- Remove the apostrophe flag from sprintf format specifier in rmalloc_lld_format to avoid locale-dependent thousands separator, which caused unexpected commas in output
- Increase static buffer size in rmalloc_stats from 200 to 300 bytes to prevent potential buffer overflow when formatting memory statistics with larger values
 2024-11-29 05:49:48 +00:00
retoor 9ed1c43bc0 feat: add conditional display flags and file_exists helper to isspam.c 
Introduce four boolean flags (show_capitalized, show_sentences, show_words, show_numbers) that control whether their respective analysis results are printed via sld(), replacing the previously commented-out calls. Add a file_exists() utility function that checks if a given path points to an existing readable file. Also append "publish" to .gitignore.
 2024-11-29 05:45:06 +00:00
retoor a9d1c175fe docs: add valgrind build instructions to README for memory checking 
Add a new code block in the README under the "Build" section showing how to run the build with valgrind for memory checking. The block includes a comment that valgrind must be installed and shows the `make valgrind` command. This helps developers easily find and use the memory check build target.
 2024-11-28 18:05:34 +00:00
retoor 00760067b8 docs: add example output section to README with sample isspam run 
Add a new "Example output" section to the README that demonstrates the expected output format when running the isspam tool on a non-spam file, including metrics for capitalized words, sentences, word counts, and memory usage statistics.
 2024-11-28 18:04:06 +00:00
retoor 46594bf5a3 docs: add README with build, run, and valgrind usage instructions for isspam project 2024-11-28 18:02:57 +00:00
retoor 22911a7c8b fix: correct typographical errors and ensure consistent spelling across documentation 2024-11-28 17:41:36 +00:00
retoor bbe721b620 feat: add initial project structure with spam detection tool and supporting libraries 
Add the complete source tree for the isspam spam detection utility, including the main C implementation (isspam.c), custom memory allocator (rmalloc.h), string builder (rstr.h), dynamic string list (rstring_list.h), build configuration (Makefile, .clang-format), CI workflow (.gitea/workflows/build.yaml), and sample test data (spam/examole_spam3.txt, not_spam/correct1.txt). The core logic defines a list of forbidden words and provides functions for tokenizing input, removing preserved characters, and checking content against the spam keyword set.
 2024-11-28 17:39:34 +00:00
65 changed files with 7514 additions and 580 deletions
Expand all files Collapse all files
.gitea/workflows/build.yaml
+11 -2
View File
@@ -7,10 +7,19 @@ jobs:
runs-on: ubuntu-latest
steps:
- run: apt update
- run: apt install build-essential valgrind make -y
- run: apt install build-essential valgrind make curl wget libcurl4-openssl-dev libxml2-dev binutils libc6-dev libgcc-s1 libstdc++6 zlib1g-dev -y
- run: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain nightly -y
- name: Install Swift
run: |
cd /opt
wget -q https://download.swift.org/swift-5.9.2-release/ubuntu2204/swift-5.9.2-RELEASE/swift-5.9.2-RELEASE-ubuntu22.04.tar.gz
tar xzf swift-5.9.2-RELEASE-ubuntu22.04.tar.gz
ln -s /opt/swift-5.9.2-RELEASE-ubuntu22.04/usr/bin/swift /usr/local/bin/swift
- name: Check out repository code
uses: actions/checkout@v4
- name: List files in the repository
run: |
ls ${{ gitea.workspace }}
- run: make valgrind
- run: export PATH=$HOME/.cargo/bin:/opt/swift-5.9.2-RELEASE-ubuntu22.04/usr/bin:$PATH && make build_all
- run: export PATH=$HOME/.cargo/bin:/opt/swift-5.9.2-RELEASE-ubuntu22.04/usr/bin:$PATH && make benchmark
- run: make publish
.gitignore
+16 -2
View File
@@ -1,3 +1,17 @@
.r_history
.history
.vscode
publish
.vscode/
publish/
books/
__pycache__/
target/
./isspam.py
/isspam
/risspam
/jisspam
/sisspam
/isspam_cpp
/borded_cpp_exec
/isspam.py
swift_isspam/.build/
.build-trigger-2014-12-02 15:26
12bitfloat_rust/isspam.rs
-121
View File
@@ -1,121 +0,0 @@
#![feature(let_chains)]
use std::{env, fs};
fn clean_content(content: &str) -> String {
let alloed_ichars = "01234567891abcdefghijklmnopqrstuvwxyz \n.,!?";
let clean_content = content.chars()
.filter(|&c| alloed_ichars.contains(c))
.collect::<String>();
clean_content
}
fn get_sentences(content: &str) -> Vec<&str> {
let mut sentences = content.split('.')
.map(|s| s.trim_start()) // Remove leading whitespace
.collect::<Vec<_>>();
// Remove last "sentence" if didn't end with a dot
if let Some(last) = sentences.last() && !last.ends_with('.') {
sentences.pop();
}
sentences
}
fn get_words(sentences: &str) -> impl Iterator<Item = &str> + Clone {
sentences.split_whitespace()
}
fn is_fully_capitalized_word(word: &str) -> bool {
word.chars()
.all(|c| !c.is_ascii_alphanumeric() || c.is_ascii_uppercase())
}
fn get_capitalized_words(content: &str) -> Vec<&str> {
let sentences = get_sentences(content);
let mut cap_words = vec![];
for sentence in sentences {
// Always skip the first word since sentences start with
for word in get_words(sentence).skip(1) {
if is_fully_capitalized_word(word) {
cap_words.push(word);
}
}
}
cap_words
}
fn get_numbers(content: &str) -> Vec<String> {
let clean = clean_content(content);
clean.split(|c: char| c.is_ascii_digit())
.map(|n| n.to_string())
.collect()
}
fn get_forbidden_words(content: &str) -> Vec<&str> {
fn check_forbidden(w: &str) -> bool {
FORBIDDEN_WORDS.iter()
.find(|fw| str::eq_ignore_ascii_case(w, fw))
.is_some()
}
get_words(content)
.filter(|w| check_forbidden(w))
.collect()
}
fn analyze(data: &str) {
let clean_data = clean_content(data);
drop(clean_data); // You aren't actually using clean_data :O
// All capitalized words
let cap_words = get_capitalized_words(data);
println!("All capitalized words: {}", cap_words.len());
// All sentences
let sentences = get_sentences(data);
println!("Sentences: {}", sentences.len());
// All words
let words = get_words(data);
println!("Words: {}", words.clone().count());
// Numbers
let numbers = get_numbers(data);
println!("Numbers: {}", numbers.len());
// Forbidden words
let fw = get_forbidden_words(data);
println!("Forbidden words: {}", fw.len());
let word_count_per_sentence = words.count() / sentences.len();
println!("Word count per sentence: {}", word_count_per_sentence);
}
fn main() {
// Read in files from args
for arg in env::args().skip(1) { // skip program arg
let Ok(text) = fs::read_to_string(&arg) else {
eprintln!("{arg} isn't a valid file or couldn't be read");
continue;
};
analyze(&text);
}
// analyze(&SPAM1);
}
static FORBIDDEN_WORDS: &'static [&'static str] = &[
"recovery", "techie", "http", "https", "digital", "hack", "::", "//", "com",
"@", "crypto", "bitcoin", "wallet", "hacker", "welcome", "whatsapp", "email", "cryptocurrency",
"stolen", "freeze", "quick", "crucial", "tracing", "scammers", "expers", "hire", "century",
"transaction", "essential", "managing", "contact", "contacting", "understanding", "assets", "funds"
];
12bitfloat_rust/isspam_v1.rs
+1 -140
View File
@@ -1,140 +1 @@
#!+[feature(let_chains)]
fn clean_content(content: &str) -> String {
let alloed_ichars = "01234567891abcdefghijklmnopqrstuvwxyz \n.,!?";
let clean_content = content.chars()
.filter(|&c| alloed_ichars.contains(c))
.collect::<String>();
clean_content
}
fn get_sentences(content: &str) -> Vec<&str> {
let mut sentences = content.split('.')
.map(|s| s.trim_start()) // Remove leading whitespace
.collect::<Vec<_>>();
// Remove last "sentence" if didn't end with a dot
if let Some(last) = sentences.last() && !last.ends_with('.') {
sentences.pop();
}
sentences
}
fn get_words(sentences: &str) -> impl Iterator<Item = &str> + Clone {
sentences.split_whitespace()
}
fn is_fully_capitalized_word(word: &str) -> bool {
word.chars()
.all(|c| !c.is_ascii_alphanumeric() || c.is_ascii_uppercase())
}
fn get_capitalized_words(content: &str) -> Vec<&str> {
let sentences = get_sentences(content);
let mut cap_words = vec![];
for sentence in sentences {
// Always skip the first word since sentences start with
for word in get_words(sentence).skip(1) {
if is_fully_capitalized_word(word) {
cap_words.push(word);
}
}
}
cap_words
}
fn get_numbers(content: &str) -> Vec<String> {
let clean = clean_content(content);
clean.split(|c: char| c.is_ascii_digit())
.map(|n| n.to_string())
.collect()
}
fn get_forbidden_words(content: &str) -> Vec<&str> {
fn check_forbidden(w: &str) -> bool {
FORBIDDEN_WORDS.iter()
.find(|fw| str::eq_ignore_ascii_case(w, fw))
.is_some()
}
get_words(content)
.filter(|w| check_forbidden(w))
.collect()
}
fn analyze(data: &str) {
let clean_data = clean_content(data);
drop(clean_data); // You aren't actually using clean_data :O
// All capitalized words
let cap_words = get_capitalized_words(data);
println!("All capitalized words: {}", cap_words.len());
// All sentences
let sentences = get_sentences(data);
println!("Sentences: {}", sentences.len());
// All words
let words = get_words(data);
println!("Words: {}", words.clone().count());
// Numbers
let numbers = get_numbers(data);
println!("Numbers: {}", numbers.len());
// Forbidden words
let fw = get_forbidden_words(data);
println!("Forbidden words: {}", fw.len());
let word_count_per_sentence = words.count() / sentences.len();
println!("Word count per sentence: {}", word_count_per_sentence);
}
fn main() {
// // Read in files from args
// for arg in env::args() {
// let Ok(text) = fs::read_to_string(arg) else {
// eprintln!("{arg} isn't a valid file or couldn't be read");
// continue;
// };
//
// analyze(&text);
// }
analyze(&SPAM1);
}
static FORBIDDEN_WORDS: &'static [&'static str] = &[
"recovery", "techie", "http", "https", "digital", "hack", "::", "//", "com",
"@", "crypto", "bitcoin", "wallet", "hacker", "welcome", "whatsapp", "email", "cryptocurrency",
"stolen", "freeze", "quick", "crucial", "tracing", "scammers", "expers", "hire", "century",
"transaction", "essential", "managing", "contact", "contacting", "understanding", "assets", "funds"
];
static SPAM1: &'static str = "HIRE Century Web Recovery TO RECOVER YOUR LOST BITCOIN
If you’ve lost your Bitcoin to an online scam, hiring a professional recovery service can significantly improve your chances of getting your funds back. Century Web Recovery specializes in Bitcoin recovery, helping victims reclaim their stolen assets. Here’s what you need to know:
Understanding the Recovery Process
The recovery process begins with contacting Century Web Recovery. Their team will guide you through the steps necessary to initiate an investigation into your case. Understanding the process is key to managing your expectations.
Documenting Your Case
To facilitate recovery, it’s essential to document all relevant information regarding the scam. This includes transaction records, wallet addresses, and any communications with the scammer. Century Web Recovery will help you gather this information to build a strong case.
Investigation and Tracking
Once you hire Century Web Recovery, their experts will begin investigating your case. They use sophisticated tools to track the stolen Bitcoin, identifying the paths taken by the scammers. This tracing is crucial for successful recovery.
Freezing Stolen Assets
Quick action is vital in recovering stolen Bitcoin.Century Web Recovery works directly with cryptocurrency exchanges to freeze any stolen assets, preventing the scammers from cashing out your funds. This collaboration is essential for a successful recovery.
Legal Support and Guidance
If necessary, Century Web Recovery can provide legal support. They will guide you on reporting the scam to law enforcement and assist in filing any legal claims. Their expertise in crypto-related cases ensures you receive the best advice on how to proceed.
If you’ve lost Bitcoin to an online scam, don’t hesitate. Hire Century Web Recovery to recover your lost assets and regain your financial security.";
#![feature(let_chains)]\n\nuse std::{env, fs};\n\nfn clean_content(content: &str) -> String {\n\tlet alloed_ichars = \"01234567891abcdefghijklmnopqrstuvwxyz \\n.,!?\";\n\t\n\tlet clean_content = content.chars()\n\t\t.filter(|&c| alloed_ichars.contains(c))\n\t\t.collect::<String>();\n\t\n\tclean_content\n}\n\nfn get_sentences(content: &str) -> Vec<&str> {\n\tlet mut sentences = content.split('.')\n\t\t.map(|s| s.trim_start()) // Remove leading whitespace\n\t\t.collect::<Vec<_>>();\n\t\n\t// Remove last \"sentence\" if didn't end with a dot\n\tif let Some(last) = sentences.last() && !last.ends_with('.') {\n\t\tsentences.pop();\n\t}\n\t\n\tsentences\n}\n\nfn get_words(sentences: &str) -> impl Iterator<Item = &str> + Clone {\n\tsentences.split_whitespace()\n}\n\nfn is_fully_capitalized_word(word: &str) -> bool {\n\tword.chars()\n\t\t.all(|c| !c.is_ascii_alphanumeric() || c.is_ascii_uppercase())\n}\n\nfn get_capitalized_words(content: &str) -> Vec<&str> {\n\tlet sentences = get_sentences(content);\n\tlet mut cap_words = vec![];\n\t\n\tfor sentence in sentences {\n\t\t// Always skip the first word since sentences start with\n\t\tfor word in get_words(sentence).skip(1) {\n\t\t\tif is_fully_capitalized_word(word) {\n\t\t\t\tcap_words.push(word);\n\t\t\t}\n\t\t}\n\t}\n\t\n\tcap_words\n}\n\nfn get_numbers(content: &str) -> Vec<String> {\n\tlet clean = clean_content(content);\n\t\n\tclean.split(|c: char| c.is_ascii_digit())\n\t\t.map(|n| n.to_string())\n\t\t.collect()\n}\n\nfn get_forbidden_words(content: &str) -> Vec<&str> {\n\tfn check_forbidden(w: &str) -> bool {\n\t\tFORBIDDEN_WORDS.iter()\n\t\t\t.find(|fw| str::eq_ignore_ascii_case(w, fw))\n\t\t\t.is_some()\n\t}\n\t\n\tget_words(content)\n\t\t.filter(|w| check_forbidden(w))\n\t\t.collect()\n}\n\nfn analyze(data: &str) {\n\tlet clean_data = clean_content(data);\n\tdrop(clean_data); // You aren't actually using clean_data :O\n\t\n\t// All capitalized words\n\tlet cap_words = get_capitalized_words(data);\n\tprintln!(\"All capitalized words: {}\", cap_words.len());\n\t\n\t// All sentences\n\tlet sentences = get_sentences(data);\n\tprintln!(\"Sentences: {}\", sentences.len());\n\t\n\t// All words\n\tlet words = get_words(data);\n\tprintln!(\"Words: {}\", words.clone().count());\n\t\n\t// Numbers\n\tlet numbers = get_numbers(data);\n\tprintln!(\"Numbers: {}\", numbers.len());\n\t\n\t// Forbidden words\n\tlet fw = get_forbidden_words(data);\n\tprintln!(\"Forbidden words: {}\", fw.len());\n\t\n\tlet word_count_per_sentence = words.count() / sentences.len();\n\tprintln!(\"Word count per sentence: {}\", word_count_per_sentence);\n}\n\nfn main() {\n // Read in files from args\n for arg in env::args().skip(1) { // skip program arg\n \tlet Ok(text) = fs::read_to_string(&arg) else {\n \t\teprintln!(\"{arg} isn't a valid file or couldn't be read\");\n \t\tcontinue;\n \t};\n \t\n \tanalyze(&text);\n }\n \n//\tanalyze(&SPAM1);\n}\n\nstatic FORBIDDEN_WORDS: &'static [&'static str] = &[\n \"recovery\", \"techie\", \"http\", \"https\", \"digital\", \"hack\", \"::\", \"//\", \"com\",\n \"@\", \"crypto\", \"bitcoin\", \"wallet\", \"hacker\", \"welcome\", \"whatsapp\", \"email\", \"cryptocurrency\",\n \"stolen\", \"freeze\", \"quick\", \"crucial\", \"tracing\", \"scammers\", \"expers\", \"hire\", \"century\",\n \"transaction\", \"essential\", \"managing\", \"contact\", \"contacting\", \"understanding\", \"assets\", \"funds\"\n];\n
12bitfloat_rust/isspam_v2.rs
+1
View File
@@ -0,0 +1 @@
#!+[feature(let_chains)]\n\n\nfn clean_content(content: &str) -> String {\n\tlet alloed_ichars = \"01234567891abcdefghijklmnopqrstuvwxyz \\n.,!?\";\n\t\n\tlet clean_content = content.chars()\n\t\t.filter(|&c| alloed_ichars.contains(c))\n\t\t.collect::<String>();\n\t\n\tclean_content\n}\n\nfn get_sentences(content: &str) -> Vec<&str> {\n\tlet mut sentences = content.split('.')\n\t\t.map(|s| s.trim_start()) // Remove leading whitespace\n\t\t.collect::<Vec<_>>();\n\t\n\t// Remove last \"sentence\" if didn't end with a dot\n\tif let Some(last) = sentences.last() && !last.ends_with('.') {\n\t\tsentences.pop();\n\t}\n\t\n\tsentences\n}\n\nfn get_words(sentences: &str) -> impl Iterator<Item = &str> + Clone {\n\tsentences.split_whitespace()\n}\n\nfn is_fully_capitalized_word(word: &str) -> bool {\n\tword.chars()\n\t\t.all(|c| !c.is_ascii_alphanumeric() || c.is_ascii_uppercase())\n}\n\nfn get_capitalized_words(content: &str) -> Vec<&str> {\n\tlet sentences = get_sentences(content);\n\tlet mut cap_words = vec![];\n\t\n\tfor sentence in sentences {\n\t\t// Always skip the first word since sentences start with\n\t\tfor word in get_words(sentence).skip(1) {\n\t\t\tif is_fully_capitalized_word(word) {\n\t\t\t\tcap_words.push(word);\n\t\t\t}\n\t\t}\n\t}\n\t\n\tcap_words\n}\n\nfn get_numbers(content: &str) -> Vec<String> {\n\tlet clean = clean_content(content);\n\t\n\tclean.split(|c: char| c.is_ascii_digit())\n\t\t.map(|n| n.to_string())\n\t\t.collect()\n}\n\nfn get_forbidden_words(content: &str) -> Vec<&str> {\n\tfn check_forbidden(w: &str) -> bool {\n\t\tFORBIDDEN_WORDS.iter()\n\t\t\t.find(|fw| str::eq_ignore_ascii_case(w, fw))\n\t\t\t.is_some()\n\t}\n\t\n\tget_words(content)\n\t\t.filter(|w| check_forbidden(w))\n\t\t.collect()\n}\n\nfn analyze(data: &str) {\n\tlet clean_data = clean_content(data);\n\tdrop(clean_data); // You aren't actually using clean_data :O\n\t\n\t// All capitalized words\n\tlet cap_words = get_capitalized_words(data);\n\tprintln!(\"All capitalized words: {}\", cap_words.len());\n\t\n\t// All sentences\n\tlet sentences = get_sentences(data);\n\tprintln!(\"Sentences: {}\", sentences.len());\n\t\n\t// All words\n\tlet words = get_words(data);\n\tprintln!(\"Words: {}\", words.clone().count());\n\t\n\t// Numbers\n\tlet numbers = get_numbers(data);\n\tprintln!(\"Numbers: {}\", numbers.len());\n\t\n\t// Forbidden words\n\tlet fw = get_forbidden_words(data);\n\tprintln!(\"Forbidden words: {}\", fw.len());\n\t\n\tlet word_count_per_sentence = words.count() / sentences.len();\n\tprintln!(\"Word count per sentence: {}\", word_count_per_sentence);\n}\n\nfn main() {\n//\t// Read in files from args\n//\tfor arg in env::args() {\n//\t\tlet Ok(text) = fs::read_to_string(arg) else {\n//\t\t\teprintln!(\"{arg} isn't a valid file or couldn't be read\");\n//\t\t\tcontinue;\n//\t\t};\n//\t\t\n//\t\t
12bitfloat_rust/isspam_v3.rs
+1
View File
@@ -0,0 +1 @@
#![feature(let_chains)]\\n\\nuse rayon::prelude::*;\\n//use rayon::prelude::*;\\nuse std::{env, fs};\\n\\nfn clean_content(content: &str) -> String {\\n\\tlet alloed_ichars = \\\"01234567891abcdefghijklmnopqrstuvwxyz \\\\n.,!?\\\";\\n\\t\\n\\tlet clean_content = content.chars()\\n\\t\\t.filter(|&c| alloed_ichars.contains(c))\\n\\t\\t.collect::<String>();\\n\\t\\n\\tclean_content\\n}\\n\\nfn get_sentences(content: &str) -> usize {\\n\\tlet sentences = content.split('.')\\n\\t\\t.map(|s| s.trim_start()) // Remove leading whitespace\\n\\t\\t.count();\\n\\t\\n//\\t// Remove last \\\"sentence\\\" if didn't end with a dot\\n//\\tif let Some(last) = sentences.last() && !last.ends_with('.') {\\n//\\t\\tsentences.pop();\\n//\\t}\\n\\t\\n\\tsentences\\n}\\n\\nfn get_words(content: &str, words: &mut usize, caps: &mut usize, fw: &mut usize) {\\n\\tfn check_forbidden(w: &str) -> bool {\\n\\t\\tFORBIDDEN_WORDS.iter()\\n\\t\\t\\t.find(|fw| str::eq_ignore_ascii_case(w, fw))\\n\\t\\t\\t.is_some()\\n\\t}\\n\\t\\n\\tfor word in content.split_whitespace() {\\n\\t\\t*words += 1;\\n\\t\\t\\n\\t\\tif is_fully_capitalized_word(word) {\\n\\t\\t\\t*caps += 1;\\n\\t\\t}\\n\\t\\tif check_forbidden(word) {\\n\\t\\t\\t*fw += 1;\\n\\t\\t}\\n\\t}\\n}\\n\\nfn is_fully_capitalized_word(word: &str) -> bool {\\n\\tword.chars()\\n\\t\\t.all(|c| !c.is_ascii_alphanumeric() || c.is_ascii_uppercase())\\n}\\n\\nfn get_numbers(clean_content: &str) -> usize {\\n\\tclean_content.split(|c: char| !c.is_ascii_digit())\\n\\t\\t.count()\\n}\\n\\nfn analyze(data: &str) {\\n\\tlet clean_data = clean_content(data);\\n//\\tdrop(clean_data); // You aren't actually using clean_data :O\\n\\t\\n\\t// All capitalized words\\n\\tlet mut words = 0;\\n\\tlet mut fw = 0;\\n\\tlet mut cap_words = 0;\\n\\tget_words(&clean_data, &mut words, &mut fw, &mut cap_words);\\n\\t\\n\\tprintln!(\\\"All capitalized words: {}\\\", cap_words);\\n\\t\\n\\t// All sentences\\n\\tlet sentences = get_sentences(data);\\n\\tprintln!(\\\"Sentences: {}\\\", sentences);\\n\\t\\n\\t// All words\\n\\tprintln!(\\\"Words: {}\\\", words);\\n\\t\\n\\t// Numbers\\n\\tlet numbers = get_numbers(&clean_data);\\n\\tprintln!(\\\"Numbers: {}\\\", numbers);\\n\\t\\n\\t// Forbidden words\\n\\tprintln!(\\\"Forbidden words: {}\\\", fw);\\n\\t\\n\\tif sentences > 0 {\\n\\t\\tlet word_count_per_sentence = words / sentences;\\n\\t\\tprintln!(\\\"Word count per sentence: {}\\\", word_count_per_sentence);\\n\\t}\\n}\\n\\nfn main() {\\n // Read in files from args\\n\\tlet mut files = Vec::with_capacity(env::args().len());\\n\\tlet mut do_parallel = false;\\n\\t\\n\\tfor arg in env::args().skip(1) { // skip program arg\\n\\t\\tif arg == \\\"-p\\\" {\\n\\t\\t\\tdo_parallel = true;\\n\\t\\t} else {\\n\\t\\t\\tfiles.push(arg);\\n\\t\\t}\\n\\t}\\n\\t\\n\\t// Do the work\\n\\tlet work = |file| {\\n\\t let Ok(text) = fs::read_to_string(&file) else {\\n\\t\\t\\teprintln!(\\\"{file} isn't a valid file or couldn't be read\\\");\\n\\t\\t\\treturn;\\n\\t };\\n\\t \\tanalyze(&text);\\n\\t};\\n\\t\\n\\tif !do_parallel {\\n\\t\\tfiles.iter().for_each(work);\\n\\t} else {\\n\\t\\tfiles.par_iter().for_each(work)\\n\\t}\\n}\\n\\nstatic FORBIDDEN_WORDS: &'static [&'static str] = &[\\n \\\"recovery\\\", \\\"techie\\\", \\\"http\\\", \\\"https\\\", \\\"digital\\\", \\\"hack\\\", \\\"::\\\", \\\"//\\\", \\\"com\\\",\\n \\\"@\\\", \\\"crypto\\\", \\\"bitcoin\\\", \\\"wallet\\\", \\\"hacker\\\", \\\"welcome\\\", \\\"whatsapp\\\", \\\"email\\\", \\\"cryptocurrency\\\",\\n \\\"stolen\\\", \\\"freeze\\\", \\\"quick\\\", \\\"crucial\\\", \\\"tracing\\\", \\\"scammers\\\", \\\"expers\\\", \\\"hire\\\", \\\"century\\\",\\n \\\"transaction\\\", \\\"essential\\\", \\\"managing\\\", \\\"contact\\\", \\\"contacting\\\", \\\"understanding\\\", \\\"assets\\\", \\\"funds\\\"\\n];\\n
12bitfloat_rust/isspam_v4.rs
+1
View File
@@ -0,0 +1 @@
#![feature(let_chains)]\\n\\nuse rayon::prelude::*;\\n//use rayon::prelude::*;\\nuse std::{env, fs};\\n\\nfn clean_content(content: &str) -> String {\\n\\tlet alloed_ichars = \\\"01234567891abcdefghijklmnopqrstuvwxyz \\\\n.,!?\\\";\\n\\t\\n\\tlet clean_content = content.chars()\\n\\t\\t.filter(|&c| alloed_ichars.contains(c))\\n\\t\\t.collect::<String>();\\n\\t\\n\\tclean_content\\n}\\n\\nfn get_sentences(content: &str) -> usize {\\n\\tlet sentences = content.split('.')\\n\\t\\t.map(|s| s.trim_start()) // Remove leading whitespace\\n\\t\\t.count();\\n\\t\\n//\\t// Remove last \\\"sentence\\\" if didn't end with a dot\\n//\\tif let Some(last) = sentences.last() && !last.ends_with('.') {\\n//\\t\\tsentences.pop();\\n//\\t}\\n\\t\\n\\tsentences\\n}\\n\\nfn get_words(content: &str, words: &mut usize, caps: &mut usize, fw: &mut usize) {\\n\\tfn check_forbidden(w: &str) -> bool {\\n\\t\\tFORBIDDEN_WORDS.iter()\\n\\t\\t\\t.find(|fw| str::eq_ignore_ascii_case(w, fw))\\n\\t\\t\\t.is_some()\\n\\t}\\n\\t\\n\\tfor word in content.split_whitespace() {\\n\\t\\t*words += 1;\\n\\t\\t\\n\\t\\tif is_fully_capitalized_word(word) {\\n\\t\\t\\t*caps += 1;\\n\\t\\t}\\n\\t\\tif check_forbidden(word) {\\n\\t\\t\\t*fw += 1;\\n\\t\\t}\\n\\t}\\n}\\n\\nfn is_fully_capitalized_word(word: &str) -> bool {\\n\\tword.chars()\\n\\t\\t.all(|c| !c.is_ascii_alphanumeric() || c.is_ascii_uppercase())\\n}\\n\\nfn get_numbers(clean_content: &str) -> usize {\\n\\tclean_content.split(|c: char| !c.is_ascii
12bitfloat_rust/risspam/.cargo/config.toml
+9
View File
@@ -0,0 +1,9 @@
[build]
rustflags = [
"-Ztls-model=initial-exec",
"-Ctarget-cpu=native"
]
#[unstable]
#build-std = ["compiler_builtins", "alloc", "std", "panic_abort"] # choose only what you need
#build-std-features = ["compiler-builtins-mem"]
12bitfloat_rust/risspam/Cargo.lock
Generated
+55 -1
View File
@@ -1,7 +1,61 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
version = 4
[[package]]
name = "crossbeam-deque"
version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "613f8cc01fe9cf1a3eb3d7f488fd2fa8388403e97039e2f73692932e291a770d"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
]
[[package]]
name = "crossbeam-epoch"
version = "0.9.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22ec99545bb0ed0ea7bb9b8e1e9122ea386ff8a48c0922e43f36d45ab09e0e80"
[[package]]
name = "either"
version = "1.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60b1af1c220855b6ceac025d3f6ecdd2b7c4894bfe9cd9bda4fbb4bc7c0d4cf0"
[[package]]
name = "rayon"
version = "1.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b418a60154510ca1a002a752ca9714984e21e4241e804d32555251faf8b78ffa"
dependencies = [
"either",
"rayon-core",
]
[[package]]
name = "rayon-core"
version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1465873a3dfdaa8ae7cb14b4383657caab0b3e8a0aa9ae8e04b044854c8dfce2"
dependencies = [
"crossbeam-deque",
"crossbeam-utils",
]
[[package]]
name = "risspam"
version = "0.1.0"
dependencies = [
"rayon",
]
12bitfloat_rust/risspam/Cargo.toml
+16 -1
View File
@@ -1,6 +1,21 @@
[package]
name = "risspam"
version = "0.1.0"
edition = "2021"
edition = "2024"
[profile.release]
lto = "thin"
panic = "abort"
codegen-units = 1
debug = "line-tables-only"
[dependencies]
rayon = "1.10.0"
#mimalloc = "0.1.48"
#io-uring = "0.7"
#memmap2 = "0.9.8"
#core_affinity = "0.8.3"
#libc = "0.2.176"
#glommio = "0.9.0"
#monoio = "0.2.4"
#phf = { version = "0.13.1", features = ["macros"] }
12bitfloat_rust/risspam/src/books.rs
+907
View File
@@ -0,0 +1,907 @@
pub static FULL_BOOK_PATHS: &[&'static str] = &[
"books/0192806807.pdf - Unknown.txt",
"books/0_Deep Learning Cookbook - Practical Recipes to Get Started Quickly.txt",
"books/0_Deep Learning for Search.txt",
"books/0_Deep Learning with Python.txt",
"books/10Algorithms-08.txt",
"books/1407.7502v3.txt",
"books/1491912766_Advanced.txt",
"books/18374.txt",
"books/2014-data-science-salary-survey.txt",
"books/21 Recipes for Mining Twitter.txt",
"books/240415.txt",
"books/3dprinting.txt",
"books/9780077418182.pdf - W. Schiff.txt",
"books/A Developer’s Guide to the Semantic Web.txt",
"books/Advanced Analytics with Spark - Patterns for Learning from Data at Scale.txt",
"books/Advanced Analytics with Spark - Sandy Ryza, Uri Laserson, Sean Owen.txt",
"books/AdvancedBashScripting.txt",
"books/advanced-microservices.txt",
"books/Advanced Techniques in Web Intelligence – Part II.txt",
"books/Advanced Techniques in Web Intelligence – Part I.txt",
"books/Agile Business Intelligence.txt",
"books/Agile Data Science.txt",
"books/Agile Estimating and Planning.txt",
"books/Agile for Everybody - Creating Fast, Flexible, and Customer First Organizations.txt",
"books/Agile Methods - Large-Scale Development, Refactoring, Testing, and Estimation.txt",
"books/Agile Retrospectives - Making Good Teams Great.txt",
"books/Agile_Software_Development.txt",
"books/Agile Testing - A Practical Guide for Testers and Agile Teams.txt",
"books/Algorithmic Graph Theory and Sage.txt",
"books/Algorithms for Interviews.txt",
"books/algoritmos-programacion-Python.txt",
"books/a-little-book-of-r-for-time-series.txt",
"books/Amazon Web Services in Action.txt",
"books/[Andreas_M._Antonopoulos]_Mastering_Bitcoin_Unloc(BookZZ.org).txt",
"books/android9developmentcookbook.txt",
"books/AndroidForensics.txt",
"books/androidprogrammingforbeginners.txt",
"books/AndroidProgrammingPushingTheLimits.txt",
"books/AndroidSensorProgramming.txt",
"books/AndroidUIDesign.txt",
"books/AngualrJS Fundamentals.txt",
"books/angular-2-test-driven-development-2nd.txt",
"books/Angular2.txt",
"books/angular6forenterprise-readywebapplications.txt",
"books/Angular in Action.txt",
"books/AngularJS by Example.txt",
"books/AngularJS by Example - Unknown.txt",
"books/AngularJsNoviceToNinja.txt",
"books/AngularJS.txt",
"books/Angular.txt",
"books/angular_upandrunning.txt",
"books/AnIntroductionToGCC.txt",
"books/AnIntroductionToGNUMakeTool.txt",
"books/An Introduction to Information Retreival.txt",
"books/An Introduction to Machine Learning Interpretability.txt",
"books/antitextbookGo.txt",
"books/Anything You Want - 40 Lessons for a New Kind of Entrepreneur.txt",
"books/Apache Kafka Cookbook.txt",
"books/Apache Mesos Cookbook.txt",
"books/Apache Sqoop Cookbook.txt",
"books/ApacheTomcatCookbook.txt",
"books/API Design Cookbook.txt",
"books/api-driven-devops.txt",
"books/APIs A Strategy Guide.txt",
"books/Applied Text Analysis with Python - Enabling Language Aware Data Products with Machine Learning.txt",
"books/AprendiendoJavaScript(spanish).txt",
"books/Architecting Modern Data Platforms - A Guide To Enterprise Hadoop At Scale.txt",
"books/architectingmodernjavaeeapplications.txt",
"books/Arduino_Succinctly.txt",
"books/artificialintelligenceinthe21stcentury.txt",
"books/Art of Agile Development.txt",
"books/artofdatascience.txt",
"books/aspectos_avanzados_en_seguridad_en_redes_modulos.txt",
"books/aspnetcore2andangular5.txt",
"books/ASPNetCore.txt",
"books/aspnetmvc4_Succinctly.txt",
"books/ASP.NET_MVC_Succinctly.txt",
"books/Atomic Habits - An Easy & Proven Way to Build Good Habits & Break Bad Ones.txt",
"books/autocad2019beginningandintermediate.txt",
"books/autocad20203dmodeling.txt",
"books/autodeskrevit2020architecture.txt",
"books/Automate the Boring Stuff with Python.txt",
"books/A Workflow Approach to Stream Processing.txt",
"books/Bad Data Handbook - Cleaning Up The Data So You Can Get Back To Work.txt",
"books/bashcookbook.txt",
"books/Bash Guide for Beginners.txt",
"books/BasicsProgrammableLogicControllerPrinciples.txt",
"books/Bayesian_computation_with_R-libre.txt",
"books/Bayesian Networks and Influence Diagrams A Guide to Construction and Analysis.txt",
"books/Bayesian Reasoning and Machine Learning .txt",
"books/bdd-in-action.txt",
"books/BDD.txt",
"books/Beautiful Code.txt",
"books/Beautiful_Code.txt",
"books/Beautiful Data.txt",
"books/Beautiful Visualization.txt",
"books/become-ninja-angular2.txt",
"books/Beginning Amazon Web Services with Node.js.txt",
"books/BeginningAndroidGames.txt",
"books/BeginningJSON.txt",
"books/Big_Data_Analytics_with_R.txt",
"books/Big Data Analytics with Spark - A Practitioner's Guide to Using Spark for Large Scale Data Analysis.txt",
"books/Big Data, Data Mining and Machine Learning.txt",
"books/Big Data For Dummies.txt",
"books/Big Data Glossary.txt",
"books/Blockchain.txt",
"books/bookL.txt",
"books/book-no-solutions-aug-21-2014.txt",
"books/book.txt",
"books/BootstrapCookbook.txt",
"books/build-apis-you-wont-hate.txt",
"books/Building Adaptable Software with Microservices.txt",
"books/Building Evolutionary Architectures.txt",
"books/Building Hypermedia APIs with HTML5 and Node.txt",
"books/Building Hypermedia APIs with HTML5 and No - Mike Amundsen.txt",
"books/Building Machine Learning Projects with TensorFlow.txt",
"books/Building Machine Learning Systems with Python.txt",
"books/building-microservices-designing-fine-grained-systems.txt",
"books/Building Microservices.txt",
"books/buildingrestfulpythonwebservices - Unknown.txt",
"books/Building-web-apps-with-Node.js.txt",
"books/BuildingWebAppsWithNode.js.txt",
"books/Business Adventures - Twelve Classic Tales from the World of Wall Street.txt",
"books/Business Intelligence Data Mining and Optimization for Decision Making.txt",
"books/business-models-for-data-economy.txt",
"books/Can I Freeze It_ _ How to Use the Most Ver - Susie Theodorou.txt",
"books/Category Theory for Computer Science.txt",
"books/Category Theory for Computer Science - Unknown.txt",
"books/C++_CreatingGamesStepByStepGUIDE.txt",
"books/Chapter-13-Association-Rules.txt",
"books/Chapter-14-Cluster-Analysis.txt",
"books/Chapter-16-Regression-Based-Forecasting.txt",
"books/Chapter-1-Introduction.txt",
"books/Chapter-2-Overview-of-the-Data-Mining-Process.txt",
"books/Chapter-3-Data-Visualization.txt",
"books/Chapter 4_Dimension Reduction (Data Mining - Nitin R. Patel.txt",
"books/Chapter-4-Dimension-Reduction.txt",
"books/Chapter-5-Evaluating-Classification-and-Predictive-Performance.txt",
"books/Chapter-6-Multiple-Linear-Regression.txt",
"books/Chapter-7-k-Nearest-Neighbors--k-NN-.txt",
"books/Chapter-8-Naive-Bayes.txt",
"books/Chapter-9-Classification-and-Regression-Trees.txt",
"books/Christian Rudder-Dataclysm_ who we are (when we think no one's looking)-Crown (2014).txt",
"books/ciml-v0_9-all.txt",
"books/Classic Computer Science Problems in Python.txt",
"books/classicgamedesign.txt",
"books/classicshellscripting.txt",
"books/Clean Code - A Handbook of Agile Software Craftsmanship.txt",
"books/clean-coder-conduct-professional-programmers.txt",
"books/Clean Code.txt",
"books/cloudcomputingbasics_aselfteachingintroduction.txt",
"books/Cloud Native DevOps with Kubernetes.txt",
"books/Cloud Native Patterns - Designing change tolerant software.txt",
"books/Code Complete - A Practical Handbook of Software Construction.txt",
"books/Code Complete.txt",
"books/Collaborative filtering.txt",
"books/Collective Intelligence in Action.txt",
"books/Collective Intelligence.txt",
"books/Competing Against Luck - The Story of Innovation and Customer Choice.txt",
"books/Compilers-Principles-TechniquesAndTools2ndEdition.txt",
"books/Computational Intelligence.txt",
"books/Concurrency in Go_ Tools and Techniques fo - Katherine Cox-Buday.txt",
"books/Concurrency in Go - Tools and Techniques for Developers.txt",
"books/ConsumersurplusatUber_PR.txt",
"books/Contagious - Why Things Catch On.txt",
"books/Conversion_Optimization.txt",
"books/cover - Jolcia.txt",
"books/Cplusplus_Today.txt",
"books/cprogramming_aselfteachingintroduction.txt",
"books/C Programming - PhD Rajiv Chopra.txt",
"books/Create a Data Driven Organization.txt",
"books/Creating a Data-Driven Organization - Carl Anderson.txt",
"books/Crossing the Chasm - Marketing and Selling Disruptive Products to Mainstream Customers.txt",
"books/Crypto101.txt",
"books/CSharpProfesional.txt",
"books/C_Sharp_Succinctly.txt",
"books/CSS3_Succinctly.txt",
"books/CSS3.txt",
"books/CSS FlexBox.txt",
"books/CssGridLayout.txt",
"books/CSS in Depth.txt",
"books/CSSMaster2ndEdition.txt",
"books/csspocketreference.txt",
"books/CSS-Programming-Cookbook.txt",
"books/CSS_Secrets.txt",
"books/css_thedefinitiveguide.txt",
"books/CursoHTML5.txt",
"books/D3-Tips-and-Tricks.txt",
"books/Daily Rituals - How Great Minds Make Time, Find Inspiration, and Get to Work.txt",
"books/Dark Web Exploring and Data Mining the Dark Side of the Web.txt",
"books/Dart-A-Modern-Web-Language.txt",
"books/dart_in_action.txt",
"books/dart_programming_tutorial.txt",
"books/Data_Algorithms (1).txt",
"books/Data_Algorithms.txt",
"books/Data Analysis with Open Source Tools.txt",
"books/Data_Analytics_in_Sports.txt",
"books/Data_Analytics_with_Hadoop (1).txt",
"books/datacleaning_pocketprimer.txt",
"books/data-driven.txt",
"books/datalog2011-dedalus.txt",
"books/Data Mashups in R.txt",
"books/Data Mining and Statistics for Decision Making.txt",
"books/DataMining-ch1.txt",
"books/DataMining-ch2.txt",
"books/DataMining-ch3.txt",
"books/DataMining-ch4.txt",
"books/DataMining-ch4 - Unknown.txt",
"books/DataMining-ch5.txt",
"books/DataMining-ch6.txt",
"books/DataMining-ch7.txt",
"books/Datamining-ch8.txt",
"books/Data Mining Concepts and Techniques.txt",
"books/Data Mining - Concepts, Models, Methods, and Algorithms.txt",
"books/Data Mining Methods for Recommender Systems.txt",
"books/Data Mining - Practical Machine Learning Tools and Techniques.txt",
"books/datamining.txt",
"books/Data_Science_from_Scratch (1).txt",
"books/Data Science from Scratch - Joel Grus.txt",
"books/Data Science from Scratch.txt",
"books/Data_Science_from_Scratch.txt",
"books/Data Source Handbook.txt",
"books/Data Stream Mining - A Practical Approach.txt",
"books/Data Structures and Algorithms.txt",
"books/Data Structures and Algorithms with JavaScript.txt",
"books/datastyle.txt",
"books/Data Visualization with D3.js Cookbook.txt",
"books/datavisualizationwithpythonandjavascript.txt",
"books/Data_Visualization_with_Python_and_JavaScript.txt",
"books/data-wrangling-cheatsheet.txt",
"books/Data Wrangling with JavaScript.txt",
"books/Data_Wrangling_with_Python (1).txt",
"books/dbSecurityBook.txt",
"books/DE_0_PHYTON -.txt",
"books/Dealing with China - An Insider Unmasks the New Economic Superpower.txt",
"books/DebianHandBookSpanish.txt",
"books/Debugging Teams - Better Productivity through Collaboration.txt",
"books/Decision Support Systems For Business Intelligence.txt",
"books/Deep Work - Cal Newport.txt",
"books/Deep Work - Rules for Focused Success in a Distracted World.txt",
"books/Dependency Injection Principles, Practices, and Patterns.txt",
"books/Design Driven Testing.txt",
"books/Design for How People Think - Using Brain Science to Build Better Products.txt",
"books/Designing Data-Intensive Applications - The Big Ideas Behind Reliable, Scalable and Maintainable Systems.txt",
"books/Designing_Data_Intensive_Applications.txt",
"books/Designing Data-Intensive Web Applications.txt",
"books/Designing Data Visualizations.txt",
"books/Designing Interfaces - Patterns for Effective Interaction Design.txt",
"books/DesigningUXForms.txt",
"books/designingwebapis.txt",
"books/Designing with the Mind in Mind Simple Guide to Understanding User Interface Design Rules.txt",
"books/Designing with the Mind in Mind Simple Gui - Unknown.txt",
"books/DesignPatterns.txt",
"books/developer-testing.txt",
"books/Developing Large Web Applications.txt",
"books/developing-microservices-node-js.txt",
"books/Developing Microservices with Node.js.txt",
"books/devops-2-0-toolkit.txt",
"books/devops-2-1-toolkit-deploying-monitoring.txt",
"books/DevOps Automation Cookbook.txt",
"books/devops-web-development.txt",
"books/DiveIntoPython.txt",
"books/django2webdevelopmentcookbook.txt",
"books/docker-bootcamp.txt",
"books/DockerContainerizationCookbook.txt",
"books/docker-cookbook-solutions-examples.txt",
"books/docker-in-action.txt",
"books/Docker in Action.txt",
"books/docker-in-practice.txt",
"books/Docker in Practice.txt",
"books/docker-orchestration.txt",
"books/Docker_ Up and Running - Matthias, Karl.txt",
"books/Docker Up and Running.txt",
"books/Docker_Up_and_Running.txt",
"books/domain-driven-design-distilled.txt",
"books/DotNETCore.txt",
"books/ECMAScript_6.txt",
"books/effectiveawk.txt",
"books/Effective_DevOps.txt",
"books/EF JS sonsivri.txt",
"books/Elasticsearch Blueprints - A practical project-based guide to generating compelling search solutions using the dynamic and powerful features of Elasticsearch.txt",
"books/Elasticsearch Cookbook.txt",
"books/Elasticsearch in Action.txt",
"books/Elasticsearch Indexing - Improve search experiences with Elasticsearch's powerful indexing functionality.txt",
"books/Elasticsearch Server.txt",
"books/Elasticsearch - The Definitive Guide.txt",
"books/ElasticSearchTutorial.txt",
"books/elasticsearch.txt",
"books/ElectronGettingStarted.txt",
"books/Electron in Action.txt",
"books/ElectronQuickIntro.txt",
"books/Elegant_SciPy.txt",
"books/Elixir in Action.txt",
"books/Elm Accelerated - James Porter.txt",
"books/Eloquent_JavaScript.txt",
"books/ELS2015.txt",
"books/embeddedvision.txt",
"books/Emergent Web Intelligence Advanced Information Retrieval.txt",
"books/Emergent Web Intelligence Advanced Semantic Technologies.txt",
"books/Enterprise_Big_Data_Lake (1).txt",
"books/EntityFrameworkCodeFirst.txt",
"books/EntityFrameworkCore.txt",
"books/entrepreneur revolution.txt",
"books/Eric Ries - The Lean Startup.txt",
"books/ESLII_print10.txt",
"books/Essential JavaScript Design Patterns.txt",
"books/eurosys10-boom 2.txt",
"books/eurosys10-boom.txt",
"books/expert-javascript.txt",
"books/expertpythonprogramming - Unknown.txt",
"books/Exploring Design Pattern For Dummies.txt",
"books/express-in-action.txt",
"books/Facebook - A Focus on Efficieny.txt",
"books/fashioning-data.txt",
"books/fcdae.txt",
"books/Feature Engineering for Machine Learning - Principles and Techniques for Data Scientists.txt",
"books/FlaskReleaseMarch03-2017.txt",
"books/flaskwebdevelopment.txt",
"books/Fluent_Python.txt",
"books/Foundations_for_Analytics_with_Python.txt",
"books/Foundations for Architecting Data Solutions - Managing Successful Data Projects.txt",
"books/Framing-Analytics-Requirements-v5.13.txt",
"books/FRIED_Jason_-_Rework.txt",
"books/FullStackJsDevelopmentWithMEAN.txt",
"books/Fundamentals of Data Visualization - A Primer on Making Informative and Compelling Figures.txt",
"books/gamedevelopmentusingpython.txt",
"books/GameProgrammingForKids.txt",
"books/Gaussian Processes for Machine Learning - Carl Edward Rasmussen.txt",
"books/Getting_Data_Right_Ch04_PE_Tamr.txt",
"books/getting-started-kubernetes-2nd.txt",
"books/GettingStartedWithASP.Net4.5WebForms.txt",
"books/Getting Started with Kubernetes.txt",
"books/Getting Started with Kudu - Jean-Marc Spaggiari.txt",
"books/GettingStartedWithLINQPad.txt",
"books/GettingStartedWithReactJs.txt",
"books/Getting Started with RStudio.txt",
"books/Getting Started with Storm.txt",
"books/Getting Started with TensorFlow.txt",
"books/Git - Giant Undo Button.txt",
"books/Git Internals.txt",
"books/GitInternals.txt",
"books/Git Internals - Unknown.txt",
"books/Global UX Design and Research in a Connected World.txt",
"books/GNU_C_LibraryReferenceManual.txt",
"books/GNULinuxAdvancedAdminstration.txt",
"books/go-building-web-applications.txt",
"books/go-design-patterns.txt",
"books/go-in-action.txt",
"books/go-in-practice.txt",
"books/go-programming-blueprints-2nd.txt",
"books/Go Recipes.txt",
"books/go.txt",
"books/go-web-programming.txt",
"books/Graph Algorithms - Practical Examples in Apache Spark and Neo4j.txt",
"books/Graph Databases - Ian Robinson, Jim Webber.txt",
"books/Graph Databases.txt",
"books/GraphDatabases.txt",
"books/Graphics of Large Datasets.txt",
"books/greppocketref.txt",
"books/Grokking Deep Learning.txt",
"books/GrowthHacking.txt",
"books/gsl_stats.txt",
"books/Hackers and Painters.txt",
"books/Hadoop in the Enterprise - Architecture - A Guide to Successful Integration.txt",
"books/Hadoop_Security.txt",
"books/Hadoop_ The Definitive Guide - Tom White.txt",
"books/Hadoop - The Definitive Guide.txt",
"books/hadoop-what-you-need-to-know.txt",
"books/hadoop-with-python.txt",
"books/HAL.txt",
"books/Handbook_Pt1.txt",
"books/Handbook_Pt2.txt",
"books/Handbook_Pt3.txt",
"books/Handbook_Pt4.txt",
"books/Hands-On Design Patterns with React Native - Mateusz Grzesiukiewicz.txt",
"books/hands-onfullstackdevelopmentwithspringboot20andreact.txt",
"books/hands-onfullstackwebdevelopmentwithangular6andlaravel5.txt",
"books/Hands-on Machine Learning with Scikit-Lear - Aurelien Geron.txt",
"books/Hands-On Machine Learning with Scikit Learn and TensorFlow - Concepts, Tools, and Techniques to Build Intelligent Systems.txt",
"books/Hands_On_Machine_Learning_with_Scikit_Learn_and_TensorFlow.txt",
"books/Hands On Machine Learning with Scikit Learn, Keras, and Tensorflow - Concepts, Tools, and Techniques to Build Intelligent Systems (Updated Release).txt",
"books/hdlwithdigitaldesign.txt",
"books/HeadFirstC.txt",
"books/Healing With Herbs and Spices_ Heal Your B - Simone McGrath.txt",
"books/HelloAndroid.txt",
"books/highperformanceimages.txt",
"books/High Performance JavaScript.txt",
"books/High_Performance_Mobile_Web.txt",
"books/HowToBuildAndScaleWithMicroServices.txt",
"books/HowToBuildAndScaleWithMicroServices - Unknown.txt",
"books/How to Live Forever - Alok Jha.txt",
"books/How to Pass Exams - Dominic O'Brien.txt",
"books/HTML5 and JavaScript Web Apps.txt",
"books/HTML5 Architecture.txt",
"books/HTML5CanvasReference.txt",
"books/HTML5 Canvas.txt",
"books/HTML5 Cookbook.txt",
"books/HTML5 & CSS3 FOR THE REAL WORLD.txt",
"books/HTML5-Programming-Cookbook.txt",
"books/HTML5SecurityCheatSheet.txt",
"books/HTML5_Vulnerabilities.txt",
"books/htmlcss2sample.txt",
"books/HTTP - 2 in Action.txt",
"books/human javascript - Henrik Joreteg.txt",
"books/Human JavaScript.txt",
"books/Identity and Data Security for Web Development Best Practices.txt",
"books/I Heart Logs Event Data, Stream Processing, and Data Integration.txt",
"books/Information Architecture For the Web and Beyond.txt",
"books/Information_Architecture_Fourth_Edition.txt",
"books/Information Theory, Inference, and Learning Algorithms .txt",
"books/Innovations in Classification, Data Science, and Information Systems.txt",
"books/Interactive Data Visualization for the Web.txt",
"books/Interactive_Data_Visualization_for_the_Web.txt",
"books/Interactive Data Visualization for the Web - Unknown.txt",
"books/Interview Preparations Kit - Software Engineer.txt",
"books/IntouchScriptingAndLogicGuide.txt",
"books/Introducing-Go.txt",
"books/introducingregularexpressions.txt",
"books/introduction-machine-learning-python.txt",
"books/introductionto3dgameprogrammingwithdirectx12.txt",
"books/Introduction to Docker.txt",
"books/IntroductionToDocker.txt",
"books/IntroductionToLinux.txt",
"books/IntroductionToNginx.txt",
"books/IntroToCrypto.txt",
"books/InventYourOwnGamesWithPython.txt",
"books/IPSUR.txt",
"books/ISLR Fourth Printing.txt",
"books/Java2.txt",
"books/JavaDesignPatterns.txt",
"books/JavaDevelopmentOnLnx.txt",
"books/JavaFXCookBook.txt",
"books/Java-JDBC.txt",
"books/JavaMultithreadingAndConcurrency.txt",
"books/JavaNIOCookbook.txt",
"books/JavaPersistenceAPI.txt",
"books/JavaScript A Beginners Guide .txt",
"books/JavaScript Cookbook.txt",
"books/JavaScriptInterviewQuestions.txt",
"books/JavaScript Patterns.txt",
"books/JavaScript_Succinctly.txt",
"books/JavaScript The Definitive Guide.txt",
"books/javascript_the_good_parts.txt",
"books/JavaScript The Good Parts.txt",
"books/JavaScript Web Applications.txt",
"books/JavaStartingIntoHibernate.txt",
"books/Java-ThinkJava.txt",
"books/Jenkins 2 - Up and Running - Evolve Your Deployment Pipeline for Next Generation Automation.txt",
"books/jenkins-the-definitive-guide.txt",
"books/JQueryHost.txt",
"books/JQueryNoviceToNinja.txt",
"books/jQuery_Succinctly.txt",
"books/JsNoviceToNinja2ndEdition.txt",
"books/JsNoviceToNinja.txt",
"books/Jurans Quality Handbook.txt",
"books/Kafka Streams in Action - Real time apps and microservices with the Kafka Streaming API.txt",
"books/Kafka - The Definitive Guide - Real Time Data and Stream Processing at Scale.txt",
"books/Kafka - The Definitive Guide.txt",
"books/Kubernetes Cookbook.txt",
"books/Kubernetes in Action.txt",
"books/Kubernetes Management Design Patterns With Docker, CoreOS Linux, and Other Platforms.txt",
"books/Kubernetes Microservices with Docker.txt",
"books/Kubernetes-Microservices with Docker.txt",
"books/Kuhn_Johnson_Applied_Predictive_Modeling.txt",
"books/LaBibliaDeMySQL.pdf.txt",
"books/lazy-analysts-guide-to-faster-sql.txt",
"books/Lean_Analytics.txt",
"books/Lean Customer Development.txt",
"books/Lean Enterprise.txt",
"books/Lean UX.txt",
"books/Learn Functional Programming by Implementing SQL with Underscore.js Presentation.txt",
"books/Learning_Agile.txt",
"books/Learning Apache Kafka.txt",
"books/Learning Apache Kafka - Unknown.txt",
"books/Learning Chaos Engineering - Russ Miles.txt",
"books/learningconcurrencyinpython - Unknown.txt",
"books/Learning Docker.txt",
"books/Learning ELK Stack - Build mesmerizing visualizations, analytics, and logs from your data using Elasticsearch, Logstash, and Kibana.txt",
"books/learninggnuemacs_3rdedition.txt",
"books/learninggraphql.txt",
"books/LearningJavaByBuildingAndroidGames.txt",
"books/Learning.Java_Oreilly_4th.Edition_Jun.2013.txt",
"books/Learning Java - Patrick Niemeyer.txt",
"books/Learning JavaScript Design Patterns.txt",
"books/learningjavascript.txt",
"books/Learning Java.txt",
"books/learningjquery3.txt",
"books/learningnodejsdevelopment.txt",
"books/learningphpmysqlandjavascript.txt",
"books/Learning Python, 5th Edition.txt",
"books/Learning Python - Mark Lutz.txt",
"books/Learning Python - Powerful Object-Oriented Programming.txt",
"books/LearningPython.txt",
"books/learningreact1.txt",
"books/learningroboticsusingpython - Unknown.txt",
"books/Learning Single-page Web Application Development.txt",
"books/Learning Spark.txt",
"books/Learning_Spark.txt",
"books/Learning_Swift.txt",
"books/learningthebashshell_3rdedition.txt",
"books/learningtheviandvimeditors_7thedition.txt",
"books/Learning Website Development with Django.txt",
"books/learnqt5.txt",
"books/lecture-22.txt",
"books/Linear Algebra Explained In Four Pages.txt",
"books/Linear Algebra.txt",
"books/Linked Data - Evolving The Web Into A Global Data Space.txt",
"books/Linked Open Data - The Essentials.txt",
"books/Linux Bible.txt",
"books/Linux ColecciĂłn completa (2004).txt",
"books/LinuxCommandLineSheet.txt",
"books/LinuxCookBook.pdf - Pankaj Kumar.txt",
"books/linuxdevicedrivers.txt",
"books/LinuxEmbeddedDevelopment.txt",
"books/Linux From Scratch.txt",
"books/LinuxFromScratch.txt",
"books/linuxinanutshell.txt",
"books/Linux Internals_ Como funciona - Daniel Ezquerra.txt",
"books/LinuxKali.txt",
"books/LinuxNetworkingCookbook.txt",
"books/linuxpocketguide_3rdedition.txt",
"books/Linux Pocket.txt",
"books/LinuxPracticalSecurityCookBook.txt",
"books/LinuxShellScripting.txt",
"books/linuxsystemprogramming.txt",
"books/LittleInferenceBook.txt",
"books/Machine Learning Cheat Sheet.txt",
"books/Machine learning for hackers.txt",
"books/Machine Learning for Hackers.txt",
"books/Machine_Learning_with_R_Second_Edition.txt",
"books/Machine Learning with Spark.txt",
"books/Machine Learning with TensorFlow.txt",
"books/Maintainable JavaScript.txt",
"books/Making Isometric Social Real-Time Games with HTML5 CSS3 and JavaScript.txt",
"books/Management 3.0; Leading Agile Developers, - Jurgen Appelo.txt",
"books/ManualDePowerBuilder.txt",
"books/ManualDeSEO.txt",
"books/mapping-big-data.txt",
"books/MapReduce Design Patterns - Building Effective Algorithms and Analytics for Hadoop and Other Systems.txt",
"books/Mastering-Advanced-Analytics-With-Apache-Spark.txt",
"books/Mastering_Dart__Master_the_art_of.txt",
"books/Mastering ElasticSearch - Extend your knowledge on ElasticSearch, and querying and data handling, along with its internal workings.txt",
"books/masteringios12programming.txt",
"books/Mastering Kubernetes.txt",
"books/masteringmodularjavascript.txt",
"books/Mastering Modular JavaScript.txt",
"books/masteringpythonnetworking - Unknown.txt",
"books/masteringpython - Unknown.txt",
"books/masteringregularexpressions.txt",
"books/Mastering Regular Expressions.txt",
"books/Mastering Web Application Development with Express.txt",
"books/masteringxamarinuidevelopment.txt",
"books/mesos-in-action.txt",
"books/microservices-building-scalable-software.txt",
"books/microservices-deployment-cookbook.txt",
"books/Microservices Designing Deploying.txt",
"books/microservices-docker-microsoft-azure.txt",
"books/microservices-flexible-software-architecture.txt",
"books/microservices-from-day-one.txt",
"books/Microservices Patterns - With examples in Java.txt",
"books/microsoftaccess2019programmingwithvbaxmlandasp.txt",
"books/microsoftexcel2019programmingwithvbaxmlandasp.txt",
"books/microsoftexcelfunctionsandformulas_5e.txt",
"books/microsoftoffice2013_365andbeyond.txt",
"books/Mining Business Databases.txt",
"books/Mining of Data with Complex Structures.txt",
"books/Mining of Massive Datasets.txt",
"books/Mining Text Data.txt",
"books/Mining_the_Social_Web__Second_Edition (1).txt",
"books/Mining_the_Social_Web__Second_Edition.txt",
"books/Mining the Social Web.txt",
"books/Modeling With Data.txt",
"books/Modern Java in Action - Lambda, streams, functional and reactive programming.txt",
"books/ModernJs.txt",
"books/modernpythoncookbook - Unknown.txt",
"books/MongoDB3.txt",
"books/MongoDB - Applied Design Patterns, Practical Use Cases with the Leading NoSQL Database.txt",
"books/MongoDB Applied Design Patterns - Rick Copeland.txt",
"books/MongoDB Cookbook.txt",
"books/MongoDB - The Definitive Guide.txt",
"books/MongoDBTheDefinitiveGuide.txt",
"books/Monitoring with Graphite - Jason Dixon.txt",
"books/msexcel2016.txt",
"books/multimediawebdesignanddevelopment.txt",
"books/MySQLPluginDevelopmen.txt",
"books/native-docker-clustering-swarm.txt",
"books/Natural Language Annotation for Machine Learning.txt",
"books/Natural_Language_Annotation_for_Machine_Learning.txt",
"books/Natural Language Processing in Action - Understanding, analyzing, and generating text with Python.txt",
"books/Natural Language Processing with PyTorch - Build Intelligent Language Applications Using Deep Learning.txt",
"books/negron-muntaner-jennifers-butt.txt",
"books/NetworkProgrammingIndotNET.txt",
"books/Network_Security_Through_Data_Analysis.txt",
"books/New Trends in Computational Collective Intelligence.txt",
"books/Nodedotjs_Web_Development_Third_Edition.txt",
"books/Node for Front-End Developers.txt",
"books/NodeJsAdvancedGuide.txt",
"books/Node.js By Example.txt",
"books/Node.js Design Patterns.txt",
"books/Node.js in Action.txt",
"books/Node.js Recipes.txt",
"books/Node.js the Right Way.txt",
"books/NodeJs.txt",
"books/Node Up and Running.txt",
"books/Node- Up and Running.txt",
"books/NoSQLArchitectsGuide.txt",
"books/NoSQL Database Technology - A Survey and Comparison of Systems.txt",
"books/OraclePL-SQL3Edition.txt",
"books/OraclePL-SQL.txt",
"books/Oreilly.Beautiful.Data.Jul.2009.txt",
"books/O'Reilly Media -- Template for Microsoft W - na na.txt",
"books/OReilly.REST.API.Design.Rulebook.Oct.2011.ISBN.1449310508.txt",
"books/OReilly Twisted Network Programming Essentials 2nd Edition 2013.txt",
"books/Organizational_Profiles.txt",
"books/out-of-the-tar-pit.txt",
"books/PatternDesignInC++WithQt4.txt",
"books/PHP-And-MySql-NoviceToNinja.txt",
"books/Postgres.txt",
"books/Practical Cloud Security - A Guide for Secure Design and Deployment.txt",
"books/practicaldatacleaning.txt",
"books/Practical Machine Learning Tools and Techniques.txt",
"books/Practical Machine Learning.txt",
"books/Practical Node.js.txt",
"books/Practical Recommender Systems.txt",
"books/Practical Semantic Web and Linked Data Applications.txt",
"books/Practical_Statistics_for_Data_Scientists.txt",
"books/PrincipiosDeCompiladores1EraEdicion.txt",
"books/Principles of Data Quality.txt",
"books/Privacy and Big Data.txt",
"books/ProbStatBook.txt",
"books/pro-continuous-delivery-jenkins-2.txt",
"books/pro-docker.txt",
"books/Production Ready Microservices.txt",
"books/Pro Express.js.txt",
"books/Professional Node.js.txt",
"books/proGit.txt",
"books/Pro GIT.txt",
"books/ProgramacionEnC.txt",
"books/Programmable Logic Controller - Basic Prin - Lab-Volt.txt",
"books/Programming Hive - Edward Capriolo, Dean Wampler.txt",
"books/Programming HTML5 Applications.txt",
"books/Programming Kubernetes - michael Hausenblas.txt",
"books/Programming_Pig_Second_Edition.txt",
"books/Programming_Scala_Second_Edition.txt",
"books/Programming The Semantic Web.txt",
"books/Pro HTML5 Programming.txt",
"books/Pro JavaScript Design Patterns.txt",
"books/Pro .NET 2.0 Graphics Programming.txt",
"books/Pro Node.js for Developers.txt",
"books/Pro React.txt",
"books/Pro REST API Development with Node.js.txt",
"books/pro-vim-2014.txt",
"books/pynput.txt",
"books/py-quant-econ.txt",
"books/Python3CookBook.txt",
"books/python3_pocketprimer.txt",
"books/PythonBeginnerCheatSheet.txt",
"books/Python Cookbook, 2nd Edition.txt",
"books/Python Cookbook, 3rd Edition.txt",
"books/python-crash-course.txt",
"books/pythondataanalysiscookbook - Unknown.txt",
"books/Python Data Science Essentials.txt",
"books/pythondatascienceessentials - Unknown.txt",
"books/pythondatastructuresandalgorithms - Unknown.txt",
"books/Python Essential Reference.txt",
"books/PythonEssentialsCheatSheet.txt",
"books/Python for Data Analysis.txt",
"books/Python_for_Finance.txt",
"books/PythonGamesDevelopmentForBeginners.txt",
"books/Python GUI Programming Cookbook - Second Edition.txt",
"books/pythonguiprogrammingcookbook - Unknown.txt",
"books/Python GUI programming with Tkinter ( PDFDrive.com ) (2).txt",
"books/pythonhighperformance - Unknown.txt",
"books/Python_introduction.txt",
"books/Python Machine Learning Blueprints.txt",
"books/Python Machine Learning.txt",
"books/Python_Machine_Learning.txt",
"books/pythonmachinelearning - Unknown.txt",
"books/PythonMakingGamesWithPygame.txt",
"books/pythonmicroservicesdevelopment - Unknown.txt",
"books/Python-NetworkHacks.txt",
"books/python-pocket-reference-5th-edition.txt",
"books/pythonprogrammingwithraspberrypi - Unknown.txt",
"books/PythonTestingBeginnerGuide.txt",
"books/Python.Tkinter.Programming.txt",
"books/python-tricks.txt",
"books/PyWebScrapingBook.txt",
"books/Qt5 Python GUI Programming Cookbook_ Building responsive and powerful cross-platform applications with PyQt ( PDFDrive.com ).txt",
"books/quality-code-software-testing-principles-practices-and-patterns.txt",
"books/radziwill_statisticseasierwithr_preview.txt",
"books/randomforest2001.txt",
"books/R Cookbook - JD Long.txt",
"books/R_Cookbook.txt",
"books/R Data Structures and Algorithms.txt",
"books/R Deep Learning Cookbook.txt",
"books/reactandreactnative.txt",
"books/reactdesignpatternsandbestpractices.txt",
"books/Reactive Applications with Akka.Net.txt",
"books/Reactive Design Patterns.txt",
"books/ReactJs.txt",
"books/reactnativecookbook_ward.txt",
"books/React Native in Action.txt",
"books/REACT.txt",
"books/Real_Time_Big_Data_Analytics.txt",
"books/Real-World_Hadoop_MapR.txt",
"books/Redis Essentials.txt",
"books/Redis Essentials - Unknown.txt",
"books/Redis in Action.txt",
"books/Refactoring Improving the Design of Existing Code.txt",
"books/Regular Expression Pocket Reference.txt",
"books/Regular Expression Pocket Reference - Unknown.txt",
"books/Regular Expressions Cookbook.txt",
"books/RegularExpressions_Succinctly.txt",
"books/Relevant Search_ With applications for Sol - Doug Turnbull John Berryman.txt",
"books/Relevant Search - With applications for Solr and Elasticsearch.txt",
"books/ResponsiveDesign.txt",
"books/Responsive Web Design.txt",
"books/Responsive Web Design with HTML5 and CSS3.txt",
"books/rest-advanced-research-topics-and-practical-applications.txt",
"books/RESTful Java Patterns and Best Practices.txt",
"books/RESTful Java Web Services Security.txt",
"books/RESTful Java with JAX-RS 2.0, 2nd Edition.txt",
"books/RESTful Web API Design with Node.js.txt",
"books/RESTful Web APIs.txt",
"books/RESTful Web Clients - Enabling Reuse Through Hypermedia.txt",
"books/RESTful_Web_Services.txt",
"books/RESTful Web Services with Dropwizard.txt",
"books/Rexer_Analytics_2013_Data_Miner_Survey_Summary_Report.txt",
"books/RFP Proyecto CRM - Herve Cayard.txt",
"books/R_in_Action_Second__v15_MEAP.txt",
"books/R in a Nutshell, 2nd Edition.txt",
"books/Roy Cohn Part 01 of 01.txt",
"books/R_Packages.txt",
"books/R_ProgrammingSuccinctly.txt",
"books/rprogramming.txt",
"books/running-lean-iterate-from-plan-a-to-a-plan-that-works-lean-series.txt",
"books/RW.txt",
"books/Rxjs in Action.txt",
"books/Scala_Cookbook.txt",
"books/scala-test-driven-development.txt",
"books/Schema Matching and Mapping.txt",
"books/Secrets of the JavaScript Ninja.txt",
"books/Securing Devops - Safe Services in the Cloud.txt",
"books/sedandawk.txt",
"books/Semantic Web for the Working Ontologist.txt",
"books/Semantic Web for the Working Ontologist - Unknown.txt",
"books/Semantic Web Programming.txt",
"books/Semantic Web Services For Web Databases.txt",
"books/Semantic Web Services.txt",
"books/Semantic Web Technologies for Business Intelligence.txt",
"books/Site Reliability Engineering - How Google Runs Production Systems.txt",
"books/Slides - Communicating to Company.txt",
"books/Slides - How to Market.txt",
"books/Slides - How to Turn Feature Ideas Into User Stories.txt",
"books/Slides - Talking to Customers.txt",
"books/Slides - User Stories to Actual Features.txt",
"books/Slides - What do Product Managers Do.txt",
"books/Slides - What I Did As a Product Manager.txt",
"books/Slides - What Is Agile Development.txt",
"books/Slides - WhosOnTheTeam.txt",
"books/Slides - Working With Developers.txt",
"books/SLS_corrected_1.4.16.txt",
"books/SmashingNodeJs.txt",
"books/socc2012_bloom_lattices.txt",
"books/Social Data Mining.txt",
"books/softwarearchitecturewithpython - Unknown.txt",
"books/SoftwareDesignPatterns.txt",
"books/software-paradox.txt",
"books/software takes command.txt",
"books/softwaretestingprinciplesandpractices.txt",
"books/SoftwareTesting.txt",
"books/SolidPrinciples.txt",
"books/S.O.L.I.D_Principles.txt",
"books/Spark in Action.txt",
"books/Spark - The Definitive Guide - Big Data Processing Made Simple.txt",
"books/spatialEpiBook.txt",
"books/Speed Reading for Professionals - Mantesh.txt",
"books/spring5designpatterns.txt",
"books/SQL.Cookbook.2005.txt",
"books/StartingIntoAzure.txt",
"books/StartingIntoCouchDB.txt",
"books/StartingIntoGIT.txt",
"books/StartingIntoHTML5.txt",
"books/StartingIntoIonic.txt",
"books/StartingIntoMySQL.txt",
"books/StartingIntoNodeJs.txt",
"books/StartingIntoPHPEnvironment.txt",
"books/StartingIntoPLC_Programming.txt",
"books/StartingIntoXamarinForms.txt",
"books/StartUpBestPractices.txt",
"books/steve_jobs_walter_isaacson.txt",
"books/Streaming Data - Understanding the Real Time Pipeline.txt",
"books/Stunning CSS3 A project-based guide to the latest in CSS.txt",
"books/Swift.txt",
"books/t._cormen_-_introduction_to_algorithms_3rd_edition.txt",
"books/tdd-ebook-sample.txt",
"books/tensorflow2.txt",
"books/TensorFlow for Machine Intelligence - A Hands-On Introduction to Learning Algorithms.txt",
"books/TensorFlow Machine Learning Cookbook.txt",
"books/Testable JavaScript.txt",
"books/Test-Driven JavaScript Development 2.txt",
"books/Test Driven.txt",
"books/Testing Angular Applications.txt",
"books/Text Mining Classification, Clustering, and Applications.txt",
"books/TextMiningO.txt",
"books/TheArt&ScienceOfJS.txt",
"books/The Bastard Operator From Hell.txt",
"books/TheBeginnersGuideToNoSQL.txt",
"books/The Clean Coder - A Code of Conduct for Professional Programmers.txt",
"books/The CSS3 Anthology.txt",
"books/The Dart Programming Language.txt",
"books/The Data Analytics Handbook.txt",
"books/The Data Science Book.txt",
"books/The Design of Everyday Things.txt",
"books/The DevOps 2.0 Toolkit - Automating the Continuous Deployment Pipeline with Containerized Microservices.txt",
"books/The DevOps Adoption Playbook - A Guide to Adopting DevOps in a Multi-Speed IT Enterprise.txt",
"books/the-docker-book.txt",
"books/The Elements of Statistical Learning - Data Mining, Inference, and Prediction.txt",
"books/The Elements of Statistical Learning.txt",
"books/The Enterprise Big Data Lake - Delivering the Promise of Big Data and Data Science.txt",
"books/The Essential Guide to User Interface Design.txt",
"books/The_GNU_Debbuger.txt",
"books/The.Go.Programming.Language.txt",
"books/The Grammar of Graphics.txt",
"books/TheGuideToWireFraming.txt",
"books/The Laws of Simplicity.txt",
"books/the-lean-mindset-ask-the-right-questions.txt",
"books/The Lean Startup - How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.txt",
"books/TheLinuxDevelopmentPlatform.txt",
"books/TheLinuxKernelModuleProgrammingGuid.txt",
"books/TheLinuxProgrammingInterface.txt",
"books/The Lion Way - Machine Learning plus Intelligent Optimization.txt",
"books/The Meaning of Tingo_ And Other Extraordin - Adam Jacot De Boinod.txt",
"books/The Minto Pyramid Principle - Logic in Writing, Thinking, & Problem Solving.txt",
"books/The Pragmatic Programmer From Journeyman to Master.txt",
"books/The Pragmatic Programmer.txt",
"books/The Principles of Beautiful Web Design.txt",
"books/The Site Reliability Workbook - Practical Ways to Implement SRE.txt",
"books/The Startup Owner s Manual_ The Step-by-Step Guide for Building a Great Company - Blank, Steve.txt",
"books/TheUltimateGuideToPrototyping.txt",
"books/Think Bayes - Bayesian Statistics Made Simple.txt",
"books/thinkbayes.txt",
"books/thinkcomplexity.txt",
"books/Think Like a Data Scientist. Tackle the data science process step by step.txt",
"books/Think Like a Programmer - An Intro. to Creative Problem Solving - V. Spraul (No Starch, 2012) BBS.txt",
"books/thinkpython.txt",
"books/Think Python.txt",
"books/thinkstats2.txt",
"books/Think Stats - Allen B. Downey.txt",
"books/Think Stats - Exploratory Data Analysis in Python.txt",
"books/thinkstats.txt",
"books/Think Stats.txt",
"books/Third-Party JavaScript.txt",
"books/tmux-taster-2014.txt",
"books/Transactions on Computational Collective I - Ngoc Thanh Nguyen (Editor).txt",
"books/Transactions on Computational Collective Intelligence III.txt",
"books/Transactions on Computational Collective Intelligence II.txt",
"books/Transactions on Computational Collective Intelligence I.txt",
"books/Transactions on Computational Collective Intelligence V.txt",
"books/Twitter_Bootstrap3_Succinctly.txt",
"books/TypeScript Design Patterns.txt",
"books/TypeScript.txt",
"books/UbuntuServerGuide.txt",
"books/understanding-chief-data-officer.txt",
"books/Understanding Computation - From Simple Machines to Impossible Programs.txt",
"books/UnderstandingDocker.txt",
"books/UnderstandingLinuxKernel3erEdition.txt",
"books/Understanding the Chief Data Officer - Unknown.txt",
"books/UnityGameDevelopment.txt",
"books/university-startups-and-spin-offs-guide-for-entrepreneurs-in-academia.txt",
"books/unixpowertools.txt",
"books/Unknown - Unknown.txt",
"books/User Interface Design for Programmers.txt",
"books/User Story Mapping - Discover the Whole Story, Build the Right Product.txt",
"books/using-asyncio-python-understanding-asynchronous.txt",
"books/Using AWS Lambda and Claudia.js.txt",
"books/using-docker.txt",
"books/Using Node.js for UI Testing.txt",
"books/usingsvgwithcss3andhtml5.txt",
"books/UX for Leaan Startups.txt",
"books/UX_Strategy.txt",
"books/VBAProfessionalTipsSecrets.txt",
"books/Version Control by Example.txt",
"books/Visualizing Data.txt",
"books/vuejs2designpatternsandbestpractices.txt",
"books/VueJs2.txt",
"books/vuejs_upandrunning.txt",
"books/Web Crawling and Data Mining with Apache Nutch.txt",
"books/Web Data Mining.txt",
"books/Web Development Recipes.txt",
"books/webdevelopmentwithdjangocookbook - Unknown.txt",
"books/Web Development with Node and Express.txt",
"books/Web Information Retrieval.txt",
"books/Web Mining and Social Networking Techniques and Applications.txt",
"books/Web Scraping with Python - Collecting More Data from the Modern Web.txt",
"books/why-startups-fail-and-how-yours-can-succeed.txt",
"books/Wiley - Pairs Trading - Quantitative Methods and Analysis.txt",
"books/wordpress5complete.txt",
"books/WPF.txt",
"books/youdontknowjs_es6andbeyond.txt",
"books/youdontknowjs_scopeandclosures.txt",
"books/youdontknowjs_upandgoing.txt",
"books/zero-one.txt",
"books/ZooKeeper - Distributed process coordination.txt",
];
12bitfloat_rust/risspam/src/main.rs
+1169 -90
View File
File diff suppressed because it is too large Load Diff
12bitfloat_rust/risspam/src/main_old.rs
+155
View File
@@ -0,0 +1,155 @@
#![feature(let_chains)]
use rayon::prelude::*;
//use rayon::prelude::*;
use std::{env, fs};
fn clean_content(content: &str) -> String {
let alloed_ichars = "01234567891abcdefghijklmnopqrstuvwxyz \n.,!?";
let clean_content = content.chars()
.filter(|&c| alloed_ichars.contains(c))
.collect::<String>();
clean_content
}
fn get_sentences(content: &str) -> usize {
let sentences = content.split('.')
.map(|s| s.trim_start()) // Remove leading whitespace
.count();
// // Remove last "sentence" if didn't end with a dot
// if let Some(last) = sentences.last() && !last.ends_with('.') {
// sentences.pop();
// }
sentences
}
fn get_words(content: &str, words: &mut usize, caps: &mut usize, fw: &mut usize) {
fn check_forbidden(w: &str) -> bool {
FORBIDDEN_WORDS.iter()
.find(|fw| str::eq_ignore_ascii_case(w, fw))
.is_some()
}
for word in content.split_whitespace() {
*words += 1;
if is_fully_capitalized_word(word) {
*caps += 1;
}
if check_forbidden(word) {
*fw += 1;
}
}
}
fn is_fully_capitalized_word(word: &str) -> bool {
word.chars()
.all(|c| !c.is_ascii_alphanumeric() || c.is_ascii_uppercase())
}
//fn get_capitalized_words(content: &str) -> usize {
// let sentences = get_sentences(content);
//// let mut cap_words = vec![];
// let mut count = 0;
//
// for sentence in sentences {
// // Always skip the first word since sentences start with
// for word in get_words(sentence).skip(1) {
// if is_fully_capitalized_word(word) {
// count += 1;
// }
// }
// }
//
// count
//}
fn get_numbers(clean_content: &str) -> usize {
clean_content.split(|c: char| !c.is_ascii_digit())
.count()
}
//fn get_forbidden_words(content: &str) -> usize {
// fn check_forbidden(w: &str) -> bool {
// FORBIDDEN_WORDS.iter()
// .find(|fw| str::eq_ignore_ascii_case(w, fw))
// .is_some()
// }
//
// get_words(content)
// .filter(|w| check_forbidden(w))
// .collect()
//}
fn analyze(data: &str) {
let clean_data = clean_content(data);
// drop(clean_data); // You aren't actually using clean_data :O
// All capitalized words
let mut words = 0;
let mut fw = 0;
let mut cap_words = 0;
get_words(&clean_data, &mut words, &mut fw, &mut cap_words);
println!("All capitalized words: {}", cap_words);
// All sentences
let sentences = get_sentences(data);
println!("Sentences: {}", sentences);
// All words
println!("Words: {}", words);
// Numbers
let numbers = get_numbers(&clean_data);
println!("Numbers: {}", numbers);
// Forbidden words
println!("Forbidden words: {}", fw);
if sentences > 0 {
let word_count_per_sentence = words / sentences;
println!("Word count per sentence: {}", word_count_per_sentence);
}
}
fn main() {
// Read in files from args
let mut files = Vec::with_capacity(env::args().len());
let mut do_parallel = false;
for arg in env::args().skip(1) { // skip program arg
if arg == "-p" {
do_parallel = true;
} else {
files.push(arg);
}
}
// Do the work
let work = |file| {
let Ok(text) = fs::read_to_string(&file) else {
eprintln!("{file} isn't a valid file or couldn't be read");
return;
};
analyze(&text);
};
if !do_parallel {
files.iter().for_each(work);
} else {
files.par_iter().for_each(work)
}
}
static FORBIDDEN_WORDS: &'static [&'static str] = &[
"recovery", "techie", "http", "https", "digital", "hack", "::", "//", "com",
"@", "crypto", "bitcoin", "wallet", "hacker", "welcome", "whatsapp", "email", "cryptocurrency",
"stolen", "freeze", "quick", "crucial", "tracing", "scammers", "expers", "hire", "century",
"transaction", "essential", "managing", "contact", "contacting", "understanding", "assets", "funds"
];
12bitfloat_rust/risspam/src/main_pre_monoio.rs
+828
View File
@@ -0,0 +1,828 @@
#![feature(likely_unlikely)]
mod books;
use crate::books::FULL_BOOK_PATHS;
use core_affinity::CoreId;
use memmap2::Mmap;
use rayon::prelude::*;
use std::cell::OnceCell;
use std::cell::RefCell;
use std::ffi::{OsStr, OsString};
use std::fs::File;
use std::mem::MaybeUninit;
use std::ops::Deref;
use std::os::linux::raw::stat;
use std::sync::Mutex;
use std::thread::available_parallelism;
use std::time::{Duration, Instant};
use std::{array, env, fs, hint, mem, process, thread};
use std::io::Read;
use libc::{aio_read, aiocb};
#[inline]
fn is_ascii_whitespace(b: u8) -> bool {
matches!(b, b'\t' | b'\n' | b'\x0C' | b'\r' | b' ')
}
#[inline]
fn is_ascii_upper(b: u8) -> bool {
matches!(b, b'A'..=b'Z')
}
#[inline]
fn is_ascii_digit(b: u8) -> bool {
matches!(b, b'0'..=b'9')
}
#[repr(align(128))]
#[derive(Copy, Clone)]
struct Stats {
pub sentences: u32,
pub words: u32,
pub capitalizeds: u32,
pub numbers: u32,
pub forbiddens: u32,
}
static TIME_SPENT_READING_FILES: Mutex<Duration> = Mutex::new(Duration::from_secs(0));
const TEMP_MEM_SIZE: usize = 6 * 1024 * 1024;
thread_local! {
static WORK_STATE: RefCell<WorkState> = RefCell::new(WorkState::new());
}
pub struct WorkState {
pub work_mem: Box<[u8]>,
// pub io_mem: Box<[u8]>,
// pub curr_read: Option<aiocb>,
// pub had_first_load: bool,
}
impl WorkState {
pub fn new() -> Self {
Self {
work_mem: vec![0; TEMP_MEM_SIZE].into_boxed_slice(),
// io_mem: vec![0; TEMP_MEM_SIZE].into_boxed_slice(),
// curr_read: None,
// had_first_load: false,
}
}
}
fn work(file_path: &OsStr, stats: &mut Stats) {
WORK_STATE.with_borrow_mut(|state: &mut WorkState| {
// // Load file
// let start_time = Instant::now();
// let Ok(text) = fs::read(file_path) else {
// eprintln!("invalid file!");
// process::abort();
// };
let mut file = File::open(file_path).unwrap();
let file_len = file.metadata().unwrap().len() as usize;
file.read_exact(&mut state.work_mem[..file_len]).unwrap();
let text = &state.work_mem[..file_len];
unsafe {
let mut cb = mem::zeroed();
aio_read(&raw mut cb);
}
// let text = include_bytes!("../../../books/Advanced Techniques in Web Intelligence – Part II.txt").as_slice();
// let time_reading = start_time.elapsed();
// {
// let mut guard = TIME_SPENT_READING_FILES.lock().unwrap();
// *guard += time_reading;
// }
analyze(&text, stats);
});
}
fn analyze(text: &[u8], stats: &mut Stats) {
// // NOTE: mmap is quite a bit slower
// // Load file
// let Ok(file) = File::open(file_path) else {
// eprintln!("invalid file!");
// std::process::abort();
// };
// let mmap = unsafe {
// Mmap::map(&file).unwrap()
// };
// mem::forget(file);
// let text = &*mmap;
// // Load file
// let start_time = Instant::now();
// let Ok(text) = fs::read(file_path) else {
// eprintln!("invalid file!");
// process::abort();
// };
// let time_reading = start_time.elapsed();
// {
// let mut guard = TIME_SPENT_READING_FILES.lock().unwrap();
// *guard += time_reading;
// }
let mut sentences = 0;
let mut words = 0;
let mut capitalizeds = 0;
let mut numbers = 0;
let mut forbiddens = 0;
let mut idx = 0;
'full_loop: loop {
// Skip whitespace
while is_ascii_whitespace(text[idx]) {
idx += 1;
if idx >= text.len() {
break 'full_loop;
}
}
// Find end of word
let word_start = idx;
let mut has_non_upper = false;
'find_word_end: while let b = text[idx] && !is_ascii_whitespace(b) {
idx += 1;
if idx >= text.len() {
break 'find_word_end;
}
// Per-char logic
if b == b'.' {
sentences += 1;
}
if !is_ascii_upper(b) {
has_non_upper = true;
}
if is_ascii_digit(b) {
numbers += 1;
}
}
let word = &text[word_start..idx];
// dbg!(str::from_utf8(word).unwrap());
words += 1;
if !has_non_upper {
capitalizeds += 1;
}
// Check forbidden
if unsafe { FW_TAB.lookup(word) } {
forbiddens += 1;
}
}
/*
for token in text.split(|&b| is_ascii_whitespace(b)) {
if token.is_empty() {
continue;
}
words += 1;
// Sentence count, folded into this loop
// instead of another loop (better cache usage)
for &b in token {
if b == b'.' {
sentences += 1;
}
}
// Check if upper
if token.iter().all(|&b| is_ascii_upper(b)) {
capitalizeds += 1;
}
// Check digits
for &b in token {
if is_ascii_digit(b) {
numbers += 1;
}
}
// Check if words
// if FORBIDDEN_WORDS.contains(&token) {
// if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, token) } {
if unsafe { FW_TAB.lookup(token) } {
forbiddens += 1;
}
}
*/
/*
// NOTE: This is pretty slow:
let mut idx = 0;
let mut word_start = 0;
let mut is_in_word = false;
let mut has_non_upper = false;
loop {
let b = unsafe { *text.get_unchecked(idx) };
let mut process_word = false;
if is_ascii_whitespace(b) {
if is_in_word {
process_word = true;
// Reset state for next word
is_in_word = false;
has_non_upper = false;
}
} else {
if !is_in_word {
word_start = idx;
is_in_word = true;
}
has_non_upper |= !is_ascii_upper(b);
}
// Check digits
if is_ascii_digit(b) {
numbers += 1;
}
// Check sentences
if b == b'.' {
sentences += 1;
}
let word = &text[word_start..idx];
idx += 1;
if process_word || idx >= text.len() {
words += 1;
if !has_non_upper {
capitalizeds += 1;
}
// // DEBUG:
// println!("'{}'", str::from_utf8(word).unwrap());
if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, word) } {
forbiddens += 1;
}
}
if idx >= text.len() {
break;
}
}
*/
stats.sentences = sentences;
stats.words = words;
stats.capitalizeds = capitalizeds;
stats.numbers = numbers;
stats.forbiddens = forbiddens;
}
/*
fn analyze_old(file_path: &OsStr, stats: &mut Stats) {
// Load file
let Ok(text) = fs::read(file_path) else {
eprintln!("invalid file!");
std::process::abort();
};
let mut sentences = 0;
let mut words = 0;
let mut capitalizeds = 0;
let mut numbers = 0;
let mut forbiddens = 0;
for token in text.split(|&b| is_ascii_whitespace(b)) {
if token.is_empty() {
continue;
}
words += 1;
// Sentence count, folded into this loop
// instead of another loop (better cache usage)
for &b in token {
if b == b'.' {
sentences += 1;
}
}
// Check if upper
if token.iter().all(|&b| is_ascii_upper(b)) {
capitalizeds += 1;
}
// Check digits
for &b in token {
if is_ascii_digit(b) {
numbers += 1;
}
}
// Check if words
// if FORBIDDEN_WORDS.contains(&token) {
if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, token) } {
forbiddens += 1;
}
}
stats.sentences = sentences;
stats.words = words;
stats.capitalizeds = capitalizeds;
stats.numbers = numbers;
stats.forbiddens = forbiddens;
}
*/
fn main() {
// Read in files from args
let mut files = Vec::with_capacity(env::args().len());
let mut do_parallel = false;
let start_time = Instant::now();
for arg in env::args_os().skip(1) {
// skip program arg
if arg == "-p" {
do_parallel = true;
} else {
files.push(arg);
}
}
println!("[PROFILE] taking args took {:?}", start_time.elapsed());
// env::args_os().
// let files = FULL_BOOK_PATHS;
// // Build table
// let tab = FwTab::build();
// tab.compile();
// Do the work
let mut stats = vec![Stats {
sentences: 0,
words: 0,
capitalizeds: 0,
numbers: 0,
forbiddens: 0,
}; files.len()];
let start_time = Instant::now();
let num_cores = available_parallelism().unwrap().get();
let num_threads = num_cores * 1;
// DEBUG:
dbg!(num_threads);
dbg!(num_cores);
rayon::ThreadPoolBuilder::new()
.num_threads(num_threads)
.build_global()
.unwrap();
files.par_iter()
.enumerate()
.for_each(|(idx, p)| {
let s = unsafe {
&mut *stats.as_ptr()
.offset(idx as isize)
.cast_mut()
};
// let mut path = OsString::from("../../");
// path.push(p);
let path = p;
work(path, s);
});
// thread::scope(|scope| {
// let files_per_thread = files.len() / num_threads;
//
// for thread_idx in 0..num_threads {
// let capture_files = &files;
// let capture_stats = &stats;
// thread::Builder::new().spawn_scoped(scope, move || {
// let files = capture_files;
// let stats = capture_stats;
//
// // Set thread affinity
// assert!(core_affinity::set_for_current(CoreId { id: thread_idx % num_cores }));
//
// // Do work
// let thread_start = thread_idx * files_per_thread;
// for i in 0..files_per_thread {
// let real_idx = thread_start + i;
// let file_path = &files[real_idx];
// let st = unsafe {
// &mut *stats.as_ptr()
// .offset(real_idx as isize)
// .cast_mut()
// };
//
// work(&file_path, st);
// }
// }).unwrap();
// }
// });
println!("[PROFILE] processing text took {:?}", start_time.elapsed());
// Accumulate stats
let start_time = Instant::now();
let mut total_words = 0;
let mut total_capitalizeds = 0;
let mut total_sentences = 0;
let mut total_numbers = 0;
let mut total_forbiddens = 0;
for stat in &stats {
total_words += stat.words;
total_capitalizeds += stat.capitalizeds;
total_sentences += stat.sentences;
total_numbers += stat.numbers;
total_forbiddens += stat.forbiddens;
}
let capitalized_percentage = (total_capitalizeds as f32 / total_words as f32) * 100.0;
let forbidden_percentage = (total_forbiddens as f32 / total_words as f32) * 100.0;
let word_count_per_sentence = total_words as f32 / total_sentences as f32;
println!();
println!("Total Words: {total_words}");
println!("Total Capitalized words: {total_capitalizeds}");
println!("Total Sentences: {total_sentences}");
println!("Total Numbers: {total_numbers}");
println!("Total Forbidden words: {total_forbiddens}");
println!("Capitalized percentage: {capitalized_percentage:.6}");
println!("Forbidden percentage: {forbidden_percentage:.6}");
println!("Word count per sentence: {word_count_per_sentence:.6}");
println!("Total files read: {}", files.len());
println!("[PROFILE] accumulating stats took {:?}", start_time.elapsed());
println!("[PROFILE] total file reading took {:?}", &*TIME_SPENT_READING_FILES.lock().unwrap());
// Exit process to avoid running drops
process::exit(0);
}
#[repr(C)]
struct FwTab {
// pub dir_and_len_bits: [u32; 256],
pub dir_len_bits: [u16; 256],
pub dir: [u8; 256],
pub strs: [u8; 256],
}
impl FwTab {
pub fn build() -> Self {
// Sort fws by first char
let mut sorted_fws: Vec<Vec<&'static [u8]>> = vec![vec![]; 256];
for word in FORBIDDEN_WORDS {
sorted_fws[word[0] as usize].push(&word);
}
for i in 0..256 {
sorted_fws[i].sort()
}
// // DEBUG:
// println!("{:#?}", sorted_fws[b'@' as usize].iter().map(|s| str::from_utf8(s).unwrap()).collect::<Vec<_>>());
// Build str tab
let mut fw_dir = [0u8; 256];
// let mut fw_dir_len_bits = [0u32; 256];
let mut fw_dir_len_bits = [0u16; 256];
let mut fw_strs: Vec<u8> = vec![];
fw_strs.push(b'\0'); // push dummy value so that 0 in the dir means no-entries
for c in 0..256 {
for fw in FORBIDDEN_WORDS {
if c == fw[0] as usize {
fw_dir_len_bits[c] |= 0x1 << fw.len();
}
}
if !sorted_fws[c].is_empty() {
let sublist_start_offset = fw_strs.len().try_into().unwrap();
fw_dir[c] = sublist_start_offset;
// DEBUG:
println!("{c} start offset: {}", sublist_start_offset);
println!("{:#?}", sorted_fws[c].iter().map(|s| str::from_utf8(s).unwrap()).collect::<Vec<_>>());
// Push strings
for fw in &sorted_fws[c] {
fw_strs.push(fw.len().try_into().unwrap());
for &c in &fw[1..] {
fw_strs.push(c);
}
}
// Mark end of per-char word sublist
fw_strs.push(b'\0');
}
}
// DEBUG:
println!("strs len: {}", fw_strs.len());
assert_eq!(fw_dir.len(), 256);
assert!(fw_strs.len() <= 256);
fw_strs.resize(256, 0);
let tab = FwTab {
dir: fw_dir,
dir_len_bits: fw_dir_len_bits,
// dir_and_len_bits: array::from_fn(|idx| {
// (fw_dir_len_bits[idx] & 0xff_ff_ff) | ((fw_dir[idx] as u32) << 24)
// }),
strs: fw_strs.try_into().unwrap(),
};
// DEBUG: Test some strings
unsafe {
dbg!(tab.lookup(b"cpm"));
dbg!(tab.lookup(b"com"));
dbg!(tab.lookup(b"coma"));
dbg!(tab.lookup(b"co"));
dbg!(tab.lookup(b"cam"));
dbg!(tab.lookup(b"crypto"));
dbg!(tab.lookup(b"@"));
dbg!(tab.lookup(b""));
dbg!(tab.lookup(b" "));
dbg!(tab.lookup(b"test"));
dbg!(tab.lookup(b"expers"));
}
tab
}
pub fn compile(&self) {
println!("static FW_TAB: FwTab = FwTab {{");
// println!("\tdir_and_len_bits: [");
// for chunk in self.dir_and_len_bits.chunks(16) {
// print!("\t\t");
// for &b in chunk {
// print!("0x{b:08x}, ");
// }
// println!();
// }
// println!("\t],");
println!("\tdir: [");
for chunk in self.dir.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:02x}, ");
}
println!();
}
println!("\t],");
println!("\tdir_len_bits: [");
for chunk in self.dir_len_bits.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:04x}, ");
}
println!();
}
println!("\t],");
println!("\tstrs: [");
for chunk in self.strs.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:02x}, ");
}
println!();
}
println!("\t],");
println!("}};");
}
#[inline]
pub unsafe fn lookup(&self, word: &[u8]) -> bool {
let &[first_char, ..] = word else {
return false;
};
// let dir_and_len_bits = unsafe {
// *self.dir_and_len_bits.get_unchecked(first_char as usize)
// };
// if word.len() < 23 && ((dir_and_len_bits >> word.len()) & 0x1) == 0 {
// return false;
// }
let len_bits = unsafe {
*self.dir_len_bits.get_unchecked(first_char as usize)
};
if word.len() < 16 && ((len_bits >> word.len()) & 0x1) == 0 {
return false;
}
// let mut str_offset = (dir_and_len_bits >> 24) as usize;
let mut str_offset = unsafe {
*self.dir.get_unchecked(first_char as usize) as usize
};
// Char doesn't have any strings in the table
if str_offset == 0 {
return false;
}
// Iterate over strs
loop {
// let fw_len = u16::from_le_bytes([
// self.strs[str_offset],
// self.strs[str_offset+1]
// ]);
let fw_len: u8 = unsafe {
*self.strs.get_unchecked(str_offset)
};
if fw_len == 0 {
// We've reached the end of the word sublist
return false;
}
// Only compare words if they are the same length
if word.len() == fw_len as usize {
// Compare strs
let mut char_offset = 1usize;
loop {
// Found the word!
if char_offset == word.len() {
return true;
}
let fw_char = unsafe { *self.strs.get_unchecked(str_offset + char_offset) };
let word_char = unsafe { *word.get_unchecked(char_offset) };
if fw_char > word_char {
// Word can't possible be in the sorted list, return
return false;
}
if fw_char < word_char {
// Try next word
break;
}
char_offset += 1;
}
}
// Advance to next word
// let str_len_bytes = 2;
let str_len_bytes = 1;
str_offset += (fw_len as usize - 1) + str_len_bytes;
}
}
}
const FORBIDDEN_WORDS: [&'static [u8]; 35] = [
b"recovery",
b"techie",
b"http",
b"https",
b"digital",
b"hack",
b"::",
b"//",
b"com",
b"@",
b"crypto",
b"bitcoin",
b"wallet",
b"hacker",
b"welcome",
b"whatsapp",
b"email",
b"cryptocurrency",
b"stolen",
b"freeze",
b"quick",
b"crucial",
b"tracing",
b"scammers",
b"expers",
b"hire",
b"century",
b"transaction",
b"essential",
b"managing",
b"contact",
b"contacting",
b"understanding",
b"assets",
b"funds",
];
static FW_TAB: FwTab = FwTab {
dir: [
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00,
0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x09, 0x10, 0x18, 0x4f, 0x57, 0x6c, 0x00, 0x78, 0x00, 0x00, 0x00, 0x00, 0x90, 0x00, 0x00,
0x00, 0x99, 0x9f, 0xa8, 0xb7, 0xd0, 0x00, 0xde, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
],
dir_len_bits: [
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0004,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0004, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0002, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0040, 0x0080, 0x44c8, 0x0080, 0x0260, 0x0060, 0x0000, 0x0070, 0x0000, 0x0000, 0x0000, 0x0000, 0x0100, 0x0000, 0x0000,
0x0000, 0x0020, 0x0100, 0x0140, 0x08c0, 0x2000, 0x0000, 0x01c0, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
],
strs: [
0x00, 0x02, 0x2f, 0x00, 0x02, 0x3a, 0x00, 0x01, 0x00, 0x06, 0x73, 0x73, 0x65, 0x74, 0x73, 0x00,
0x07, 0x69, 0x74, 0x63, 0x6f, 0x69, 0x6e, 0x00, 0x07, 0x65, 0x6e, 0x74, 0x75, 0x72, 0x79, 0x03,
0x6f, 0x6d, 0x07, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x0a, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74,
0x69, 0x6e, 0x67, 0x07, 0x72, 0x75, 0x63, 0x69, 0x61, 0x6c, 0x06, 0x72, 0x79, 0x70, 0x74, 0x6f,
0x0e, 0x72, 0x79, 0x70, 0x74, 0x6f, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x00, 0x07,
0x69, 0x67, 0x69, 0x74, 0x61, 0x6c, 0x00, 0x05, 0x6d, 0x61, 0x69, 0x6c, 0x09, 0x73, 0x73, 0x65,
0x6e, 0x74, 0x69, 0x61, 0x6c, 0x06, 0x78, 0x70, 0x65, 0x72, 0x73, 0x00, 0x06, 0x72, 0x65, 0x65,
0x7a, 0x65, 0x05, 0x75, 0x6e, 0x64, 0x73, 0x00, 0x04, 0x61, 0x63, 0x6b, 0x06, 0x61, 0x63, 0x6b,
0x65, 0x72, 0x04, 0x69, 0x72, 0x65, 0x04, 0x74, 0x74, 0x70, 0x05, 0x74, 0x74, 0x70, 0x73, 0x00,
0x08, 0x61, 0x6e, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x00, 0x05, 0x75, 0x69, 0x63, 0x6b, 0x00, 0x08,
0x65, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x79, 0x00, 0x08, 0x63, 0x61, 0x6d, 0x6d, 0x65, 0x72, 0x73,
0x06, 0x74, 0x6f, 0x6c, 0x65, 0x6e, 0x00, 0x06, 0x65, 0x63, 0x68, 0x69, 0x65, 0x07, 0x72, 0x61,
0x63, 0x69, 0x6e, 0x67, 0x0b, 0x72, 0x61, 0x6e, 0x73, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x00,
0x0d, 0x6e, 0x64, 0x65, 0x72, 0x73, 0x74, 0x61, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x00, 0x06, 0x61,
0x6c, 0x6c, 0x65, 0x74, 0x07, 0x65, 0x6c, 0x63, 0x6f, 0x6d, 0x65, 0x08, 0x68, 0x61, 0x74, 0x73,
0x61, 0x70, 0x70, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
],
};
//static FW_TAB_DIR: [u8; 256] = [
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x09, 0x10, 0x18, 0x4f, 0x57, 0x6c, 0x00, 0x78, 0x00, 0x00, 0x00, 0x00, 0x90, 0x00, 0x00,
// 0x00, 0x99, 0x9f, 0xa8, 0xb7, 0xd0, 0x00, 0xde, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
//];
//static FW_TAB_STRS: [u8; 244] = [
// 0x00, 0x02, 0x2f, 0x00, 0x02, 0x3a, 0x00, 0x01, 0x00, 0x06, 0x73, 0x73, 0x65, 0x74, 0x73, 0x00,
// 0x07, 0x69, 0x74, 0x63, 0x6f, 0x69, 0x6e, 0x00, 0x07, 0x65, 0x6e, 0x74, 0x75, 0x72, 0x79, 0x03,
// 0x6f, 0x6d, 0x07, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x0a, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74,
// 0x69, 0x6e, 0x67, 0x07, 0x72, 0x75, 0x63, 0x69, 0x61, 0x6c, 0x06, 0x72, 0x79, 0x70, 0x74, 0x6f,
// 0x0e, 0x72, 0x79, 0x70, 0x74, 0x6f, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x00, 0x07,
// 0x69, 0x67, 0x69, 0x74, 0x61, 0x6c, 0x00, 0x05, 0x6d, 0x61, 0x69, 0x6c, 0x09, 0x73, 0x73, 0x65,
// 0x6e, 0x74, 0x69, 0x61, 0x6c, 0x06, 0x78, 0x70, 0x65, 0x72, 0x73, 0x00, 0x06, 0x72, 0x65, 0x65,
// 0x7a, 0x65, 0x05, 0x75, 0x6e, 0x64, 0x73, 0x00, 0x04, 0x61, 0x63, 0x6b, 0x06, 0x61, 0x63, 0x6b,
// 0x65, 0x72, 0x04, 0x69, 0x72, 0x65, 0x04, 0x74, 0x74, 0x70, 0x05, 0x74, 0x74, 0x70, 0x73, 0x00,
// 0x08, 0x61, 0x6e, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x00, 0x05, 0x75, 0x69, 0x63, 0x6b, 0x00, 0x08,
// 0x65, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x79, 0x00, 0x08, 0x63, 0x61, 0x6d, 0x6d, 0x65, 0x72, 0x73,
// 0x06, 0x74, 0x6f, 0x6c, 0x65, 0x6e, 0x00, 0x06, 0x65, 0x63, 0x68, 0x69, 0x65, 0x07, 0x72, 0x61,
// 0x63, 0x69, 0x6e, 0x67, 0x0b, 0x72, 0x61, 0x6e, 0x73, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x00,
// 0x0d, 0x6e, 0x64, 0x65, 0x72, 0x73, 0x74, 0x61, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x00, 0x06, 0x61,
// 0x6c, 0x6c, 0x65, 0x74, 0x07, 0x65, 0x6c, 0x63, 0x6f, 0x6d, 0x65, 0x08, 0x68, 0x61, 0x74, 0x73,
// 0x61, 0x70, 0x70, 0x00,
//];
12bitfloat_rust/risspam/src/main_pre_simd_cmp.rs
+891
View File
@@ -0,0 +1,891 @@
#![feature(likely_unlikely)]
#![feature(rust_cold_cc)]
mod books;
use crate::books::FULL_BOOK_PATHS;
use core_affinity::CoreId;
use libc::{aio_read, aiocb, read};
use memmap2::Mmap;
use rayon::prelude::*;
use std::cell::OnceCell;
use std::cell::RefCell;
use std::ffi::{OsStr, OsString};
use std::fs::{File, OpenOptions};
use std::io::Read;
use std::mem::MaybeUninit;
use std::ops::Deref;
use std::os::linux::raw::stat;
use std::sync::Mutex;
use std::thread::available_parallelism;
use std::time::{Duration, Instant};
use std::{array, env, fs, hint, mem, process, thread};
use std::hint::assert_unchecked;
use std::os::unix::fs::{FileExt, OpenOptionsExt};
#[inline]
fn is_ascii_whitespace(b: u8) -> bool {
matches!(b, b'\t' | b'\n' | b'\x0C' | b'\r' | b' ')
}
#[inline]
fn is_ascii_upper(b: u8) -> bool {
matches!(b, b'A'..=b'Z')
}
#[inline]
fn is_ascii_digit(b: u8) -> bool {
matches!(b, b'0'..=b'9')
}
#[repr(align(128))]
#[derive(Copy, Clone)]
struct Stats {
pub sentences: u32,
pub words: u32,
pub capitalizeds: u32,
pub numbers: u32,
pub forbiddens: u32,
}
static TIME_SPENT_READING_FILES: Mutex<Duration> = Mutex::new(Duration::from_secs(0));
const TEMP_MEM_SIZE: usize = 6 * 1024 * 1024;
thread_local! {
static WORK_STATE: RefCell<WorkState> = RefCell::new(WorkState::new());
}
pub struct WorkState {
pub work_mem: Box<[u8]>,
// pub io_mem: Box<[u8]>,
// pub curr_read: Option<aiocb>,
// pub had_first_load: bool,
}
impl WorkState {
pub fn new() -> Self {
Self {
work_mem: vec![0; TEMP_MEM_SIZE].into_boxed_slice(),
// io_mem: vec![0; TEMP_MEM_SIZE].into_boxed_slice(),
// curr_read: None,
// had_first_load: false,
}
}
}
#[cold]
#[inline(never)]
extern "rust-cold" fn die() -> ! {
println!("Something went wrong! I'm going to die now");
process::abort()
}
fn work(file_path: &OsStr, stats: &mut Stats) {
WORK_STATE.with_borrow_mut(|state: &mut WorkState| {
// // Load file
// let start_time = Instant::now();
// let Ok(text) = fs::read(file_path) else {
// eprintln!("invalid file!");
// process::abort();
// };
// NOTE: Reading the file like this is noticeably faster!
let mut file = OpenOptions::new()
.read(true)
// .custom_flags(libc::O_DIRECT) // O_DIRECT is A LOT slower!!
.open(file_path)
.unwrap_or_else(|_| die());
let mut read_offset = 0;
loop {
// let rb = file.read_at(&mut state.work_mem[read_offset..], read_offset as u64)
let rb = file.read(&mut state.work_mem[read_offset..])
.unwrap_or_else(|_| die());
if hint::unlikely(rb == 0) {
break;
}
read_offset += rb;
}
let text = &state.work_mem[..read_offset];
// file.read_exact(&mut state.work_mem[..file_len]).unwrap();
// let text = include_bytes!("../../../books/Advanced Techniques in Web Intelligence – Part II.txt").as_slice();
// let time_reading = start_time.elapsed();
// {
// let mut guard = TIME_SPENT_READING_FILES.lock().unwrap();
// *guard += time_reading;
// }
analyze(&text, stats);
});
}
fn analyze(text: &[u8], stats: &mut Stats) {
// // NOTE: mmap is quite a bit slower
// // Load file
// let Ok(file) = File::open(file_path) else {
// eprintln!("invalid file!");
// std::process::abort();
// };
// let mmap = unsafe {
// Mmap::map(&file).unwrap()
// };
// mem::forget(file);
// let text = &*mmap;
// // Load file
// let start_time = Instant::now();
// let Ok(text) = fs::read(file_path) else {
// eprintln!("invalid file!");
// process::abort();
// };
// let time_reading = start_time.elapsed();
// {
// let mut guard = TIME_SPENT_READING_FILES.lock().unwrap();
// *guard += time_reading;
// }
let mut sentences = 0;
let mut words = 0;
let mut capitalizeds = 0;
let mut numbers = 0;
let mut forbiddens = 0;
let mut idx = 0;
'full_loop: loop {
// Skip whitespace
while is_ascii_whitespace(text[idx]) {
idx += 1;
if hint::unlikely(idx >= text.len()) {
break 'full_loop;
}
}
// Find end of word
let word_start = idx;
let mut has_non_upper = false;
'find_word_end: while let b = text[idx] && !is_ascii_whitespace(b) {
idx += 1;
if hint::unlikely(idx >= text.len()) {
break 'find_word_end;
}
// Per-char logic
if !is_ascii_upper(b) {
has_non_upper = true;
}
if b == b'.' {
sentences += 1;
}
if is_ascii_digit(b) {
numbers += 1;
}
// sentences += (b == b'.') as u32;
// numbers += is_ascii_digit(b) as u32;
}
let word = &text[word_start..idx];
// dbg!(str::from_utf8(word).unwrap());
words += 1;
if !has_non_upper {
capitalizeds += 1;
}
// Check forbidden
if unsafe { FW_TAB.lookup(word) } {
// if FW_PHF.contains(word) { // phf is a lot slower than my FwTab
forbiddens += 1;
}
}
/*
for token in text.split(|&b| is_ascii_whitespace(b)) {
if token.is_empty() {
continue;
}
words += 1;
// Sentence count, folded into this loop
// instead of another loop (better cache usage)
for &b in token {
if b == b'.' {
sentences += 1;
}
}
// Check if upper
if token.iter().all(|&b| is_ascii_upper(b)) {
capitalizeds += 1;
}
// Check digits
for &b in token {
if is_ascii_digit(b) {
numbers += 1;
}
}
// Check if words
// if FORBIDDEN_WORDS.contains(&token) {
// if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, token) } {
if unsafe { FW_TAB.lookup(token) } {
forbiddens += 1;
}
}
*/
/*
// NOTE: This is pretty slow:
let mut idx = 0;
let mut word_start = 0;
let mut is_in_word = false;
let mut has_non_upper = false;
loop {
let b = unsafe { *text.get_unchecked(idx) };
let mut process_word = false;
if is_ascii_whitespace(b) {
if is_in_word {
process_word = true;
// Reset state for next word
is_in_word = false;
has_non_upper = false;
}
} else {
if !is_in_word {
word_start = idx;
is_in_word = true;
}
has_non_upper |= !is_ascii_upper(b);
}
// Check digits
if is_ascii_digit(b) {
numbers += 1;
}
// Check sentences
if b == b'.' {
sentences += 1;
}
let word = &text[word_start..idx];
idx += 1;
if process_word || idx >= text.len() {
words += 1;
if !has_non_upper {
capitalizeds += 1;
}
// // DEBUG:
// println!("'{}'", str::from_utf8(word).unwrap());
if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, word) } {
forbiddens += 1;
}
}
if idx >= text.len() {
break;
}
}
*/
stats.sentences = sentences;
stats.words = words;
stats.capitalizeds = capitalizeds;
stats.numbers = numbers;
stats.forbiddens = forbiddens;
}
/*
fn analyze_old(file_path: &OsStr, stats: &mut Stats) {
// Load file
let Ok(text) = fs::read(file_path) else {
eprintln!("invalid file!");
std::process::abort();
};
let mut sentences = 0;
let mut words = 0;
let mut capitalizeds = 0;
let mut numbers = 0;
let mut forbiddens = 0;
for token in text.split(|&b| is_ascii_whitespace(b)) {
if token.is_empty() {
continue;
}
words += 1;
// Sentence count, folded into this loop
// instead of another loop (better cache usage)
for &b in token {
if b == b'.' {
sentences += 1;
}
}
// Check if upper
if token.iter().all(|&b| is_ascii_upper(b)) {
capitalizeds += 1;
}
// Check digits
for &b in token {
if is_ascii_digit(b) {
numbers += 1;
}
}
// Check if words
// if FORBIDDEN_WORDS.contains(&token) {
if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, token) } {
forbiddens += 1;
}
}
stats.sentences = sentences;
stats.words = words;
stats.capitalizeds = capitalizeds;
stats.numbers = numbers;
stats.forbiddens = forbiddens;
}
*/
fn main() {
// Read in files from args
let mut files = Vec::with_capacity(env::args().len());
let mut do_parallel = false;
let start_time = Instant::now();
for arg in env::args_os().skip(1) {
// skip program arg
if arg == "-p" {
do_parallel = true;
} else {
files.push(arg);
}
}
println!("[PROFILE] taking args took {:?}", start_time.elapsed());
// env::args_os().
// let files = FULL_BOOK_PATHS;
// // Build table
// let tab = FwTab::build();
// tab.compile();
// Do the work
let mut stats = vec![Stats {
sentences: 0,
words: 0,
capitalizeds: 0,
numbers: 0,
forbiddens: 0,
}; files.len()];
let start_time = Instant::now();
let num_cores = available_parallelism().unwrap().get();
let num_threads = num_cores * 1;
// // DEBUG:
// dbg!(num_threads);
// dbg!(num_cores);
rayon::ThreadPoolBuilder::new()
.num_threads(num_threads)
.build_global()
.unwrap();
files.par_iter()
.enumerate()
.for_each(|(idx, p)| {
let s = unsafe {
&mut *stats.as_ptr()
.offset(idx as isize)
.cast_mut()
};
// let mut path = OsString::from("../../");
// path.push(p);
let path = p;
work(path, s);
});
// thread::scope(|scope| {
// let files_per_thread = files.len() / num_threads;
//
// for thread_idx in 0..num_threads {
// let capture_files = &files;
// let capture_stats = &stats;
// thread::Builder::new().spawn_scoped(scope, move || {
// let files = capture_files;
// let stats = capture_stats;
//
// // Set thread affinity
// assert!(core_affinity::set_for_current(CoreId { id: thread_idx % num_cores }));
//
// // Do work
// let thread_start = thread_idx * files_per_thread;
// for i in 0..files_per_thread {
// let real_idx = thread_start + i;
// let file_path = &files[real_idx];
// let st = unsafe {
// &mut *stats.as_ptr()
// .offset(real_idx as isize)
// .cast_mut()
// };
//
// work(&file_path, st);
// }
// }).unwrap();
// }
// });
println!("[PROFILE] processing text took {:?}", start_time.elapsed());
// Accumulate stats
let start_time = Instant::now();
let mut total_words = 0;
let mut total_capitalizeds = 0;
let mut total_sentences = 0;
let mut total_numbers = 0;
let mut total_forbiddens = 0;
for stat in &stats {
total_words += stat.words;
total_capitalizeds += stat.capitalizeds;
total_sentences += stat.sentences;
total_numbers += stat.numbers;
total_forbiddens += stat.forbiddens;
}
let capitalized_percentage = (total_capitalizeds as f32 / total_words as f32) * 100.0;
let forbidden_percentage = (total_forbiddens as f32 / total_words as f32) * 100.0;
let word_count_per_sentence = total_words as f32 / total_sentences as f32;
println!();
println!("Total Words: {total_words}");
println!("Total Capitalized words: {total_capitalizeds}");
println!("Total Sentences: {total_sentences}");
println!("Total Numbers: {total_numbers}");
println!("Total Forbidden words: {total_forbiddens}");
println!("Capitalized percentage: {capitalized_percentage:.6}");
println!("Forbidden percentage: {forbidden_percentage:.6}");
println!("Word count per sentence: {word_count_per_sentence:.6}");
println!("Total files read: {}", files.len());
println!("[PROFILE] accumulating stats took {:?}", start_time.elapsed());
println!("[PROFILE] total file reading took {:?}", &*TIME_SPENT_READING_FILES.lock().unwrap());
// Exit process to avoid running drops
process::exit(0);
}
#[repr(C)]
struct FwTab {
// pub dir_and_len_bits: [u32; 256],
pub dir_len_bits: [u16; 256],
pub dir: [u8; 256],
pub strs: [u8; 256],
}
impl FwTab {
pub fn build() -> Self {
// Sort fws by first char
let mut sorted_fws: Vec<Vec<&'static [u8]>> = vec![vec![]; 256];
for word in FORBIDDEN_WORDS {
sorted_fws[word[0] as usize].push(&word);
}
for i in 0..256 {
sorted_fws[i].sort()
}
// // DEBUG:
// println!("{:#?}", sorted_fws[b'@' as usize].iter().map(|s| str::from_utf8(s).unwrap()).collect::<Vec<_>>());
// Build str tab
let mut fw_dir = [0u8; 256];
// let mut fw_dir_len_bits = [0u32; 256];
let mut fw_dir_len_bits = [0u16; 256];
let mut fw_strs: Vec<u8> = vec![];
fw_strs.push(b'\0'); // push dummy value so that 0 in the dir means no-entries
for c in 0..256 {
for fw in FORBIDDEN_WORDS {
if c == fw[0] as usize {
fw_dir_len_bits[c] |= 0x1 << fw.len();
}
}
if !sorted_fws[c].is_empty() {
let sublist_start_offset = fw_strs.len().try_into().unwrap();
fw_dir[c] = sublist_start_offset;
// DEBUG:
println!("{c} start offset: {}", sublist_start_offset);
println!("{:#?}", sorted_fws[c].iter().map(|s| str::from_utf8(s).unwrap()).collect::<Vec<_>>());
// Push strings
for fw in &sorted_fws[c] {
fw_strs.push(fw.len().try_into().unwrap());
for &c in &fw[1..] {
fw_strs.push(c);
}
}
// Mark end of per-char word sublist
fw_strs.push(b'\0');
}
}
// DEBUG:
println!("strs len: {}", fw_strs.len());
assert_eq!(fw_dir.len(), 256);
assert!(fw_strs.len() <= 256);
fw_strs.resize(256, 0);
let tab = FwTab {
dir: fw_dir,
dir_len_bits: fw_dir_len_bits,
// dir_and_len_bits: array::from_fn(|idx| {
// (fw_dir_len_bits[idx] & 0xff_ff_ff) | ((fw_dir[idx] as u32) << 24)
// }),
strs: fw_strs.try_into().unwrap(),
};
// DEBUG: Test some strings
unsafe {
dbg!(tab.lookup(b"cpm"));
dbg!(tab.lookup(b"com"));
dbg!(tab.lookup(b"coma"));
dbg!(tab.lookup(b"co"));
dbg!(tab.lookup(b"cam"));
dbg!(tab.lookup(b"crypto"));
dbg!(tab.lookup(b"@"));
dbg!(tab.lookup(b""));
dbg!(tab.lookup(b" "));
dbg!(tab.lookup(b"test"));
dbg!(tab.lookup(b"expers"));
}
tab
}
pub fn compile(&self) {
println!("static FW_TAB: FwTab = FwTab {{");
// println!("\tdir_and_len_bits: [");
// for chunk in self.dir_and_len_bits.chunks(16) {
// print!("\t\t");
// for &b in chunk {
// print!("0x{b:08x}, ");
// }
// println!();
// }
// println!("\t],");
println!("\tdir: [");
for chunk in self.dir.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:02x}, ");
}
println!();
}
println!("\t],");
println!("\tdir_len_bits: [");
for chunk in self.dir_len_bits.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:04x}, ");
}
println!();
}
println!("\t],");
println!("\tstrs: [");
for chunk in self.strs.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:02x}, ");
}
println!();
}
println!("\t],");
println!("}};");
}
#[inline]
pub unsafe fn lookup(&self, word: &[u8]) -> bool {
// let &[first_char, ..] = word else {
// return false;
// };
let first_char = unsafe { *word.get_unchecked(0) };
// let dir_and_len_bits = unsafe {
// *self.dir_and_len_bits.get_unchecked(first_char as usize)
// };
// if word.len() < 23 && ((dir_and_len_bits >> word.len()) & 0x1) == 0 {
// return false;
// }
let len_bits = unsafe {
*self.dir_len_bits.get_unchecked(first_char as usize)
};
if hint::likely(word.len() < 16 && ((len_bits >> word.len()) & 0x1) == 0) {
return false;
}
// let mut str_offset = (dir_and_len_bits >> 24) as usize;
let mut str_offset = unsafe {
*self.dir.get_unchecked(first_char as usize) as usize
};
// Char doesn't have any strings in the table
if str_offset == 0 {
return false;
}
// Iterate over strs
loop {
// let fw_len = u16::from_le_bytes([
// self.strs[str_offset],
// self.strs[str_offset+1]
// ]);
let fw_len: u8 = unsafe {
*self.strs.get_unchecked(str_offset)
};
if fw_len == 0 {
// We've reached the end of the word sublist
return false;
}
// Only compare words if they are the same length
if hint::unlikely(word.len() == fw_len as usize) {
// Compare strs
let mut char_offset = 1usize;
loop {
// Found the word!
if char_offset == word.len() {
return true;
}
let fw_char = unsafe { *self.strs.get_unchecked(str_offset + char_offset) };
let word_char = unsafe { *word.get_unchecked(char_offset) };
if fw_char > word_char {
// Word can't possible be in the sorted list, return
return false;
}
if fw_char < word_char {
// Try next word
break;
}
char_offset += 1;
}
}
// Advance to next word
// let str_len_bytes = 2;
let str_len_bytes = 1;
str_offset += (fw_len as usize - 1) + str_len_bytes;
}
}
}
const FORBIDDEN_WORDS: [&'static [u8]; 35] = [
b"recovery",
b"techie",
b"http",
b"https",
b"digital",
b"hack",
b"::",
b"//",
b"com",
b"@",
b"crypto",
b"bitcoin",
b"wallet",
b"hacker",
b"welcome",
b"whatsapp",
b"email",
b"cryptocurrency",
b"stolen",
b"freeze",
b"quick",
b"crucial",
b"tracing",
b"scammers",
b"expers",
b"hire",
b"century",
b"transaction",
b"essential",
b"managing",
b"contact",
b"contacting",
b"understanding",
b"assets",
b"funds",
];
static FW_TAB: FwTab = FwTab {
dir: [
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00,
0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x09, 0x10, 0x18, 0x4f, 0x57, 0x6c, 0x00, 0x78, 0x00, 0x00, 0x00, 0x00, 0x90, 0x00, 0x00,
0x00, 0x99, 0x9f, 0xa8, 0xb7, 0xd0, 0x00, 0xde, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
],
dir_len_bits: [
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0004,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0004, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0002, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0040, 0x0080, 0x44c8, 0x0080, 0x0260, 0x0060, 0x0000, 0x0070, 0x0000, 0x0000, 0x0000, 0x0000, 0x0100, 0x0000, 0x0000,
0x0000, 0x0020, 0x0100, 0x0140, 0x08c0, 0x2000, 0x0000, 0x01c0, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
],
strs: [
0x00, 0x02, 0x2f, 0x00, 0x02, 0x3a, 0x00, 0x01, 0x00, 0x06, 0x73, 0x73, 0x65, 0x74, 0x73, 0x00,
0x07, 0x69, 0x74, 0x63, 0x6f, 0x69, 0x6e, 0x00, 0x07, 0x65, 0x6e, 0x74, 0x75, 0x72, 0x79, 0x03,
0x6f, 0x6d, 0x07, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x0a, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74,
0x69, 0x6e, 0x67, 0x07, 0x72, 0x75, 0x63, 0x69, 0x61, 0x6c, 0x06, 0x72, 0x79, 0x70, 0x74, 0x6f,
0x0e, 0x72, 0x79, 0x70, 0x74, 0x6f, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x00, 0x07,
0x69, 0x67, 0x69, 0x74, 0x61, 0x6c, 0x00, 0x05, 0x6d, 0x61, 0x69, 0x6c, 0x09, 0x73, 0x73, 0x65,
0x6e, 0x74, 0x69, 0x61, 0x6c, 0x06, 0x78, 0x70, 0x65, 0x72, 0x73, 0x00, 0x06, 0x72, 0x65, 0x65,
0x7a, 0x65, 0x05, 0x75, 0x6e, 0x64, 0x73, 0x00, 0x04, 0x61, 0x63, 0x6b, 0x06, 0x61, 0x63, 0x6b,
0x65, 0x72, 0x04, 0x69, 0x72, 0x65, 0x04, 0x74, 0x74, 0x70, 0x05, 0x74, 0x74, 0x70, 0x73, 0x00,
0x08, 0x61, 0x6e, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x00, 0x05, 0x75, 0x69, 0x63, 0x6b, 0x00, 0x08,
0x65, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x79, 0x00, 0x08, 0x63, 0x61, 0x6d, 0x6d, 0x65, 0x72, 0x73,
0x06, 0x74, 0x6f, 0x6c, 0x65, 0x6e, 0x00, 0x06, 0x65, 0x63, 0x68, 0x69, 0x65, 0x07, 0x72, 0x61,
0x63, 0x69, 0x6e, 0x67, 0x0b, 0x72, 0x61, 0x6e, 0x73, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x00,
0x0d, 0x6e, 0x64, 0x65, 0x72, 0x73, 0x74, 0x61, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x00, 0x06, 0x61,
0x6c, 0x6c, 0x65, 0x74, 0x07, 0x65, 0x6c, 0x63, 0x6f, 0x6d, 0x65, 0x08, 0x68, 0x61, 0x74, 0x73,
0x61, 0x70, 0x70, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
],
};
static FW_PHF: phf::Set<&'static [u8]> = phf::phf_set! {
b"recovery",
b"techie",
b"http",
b"https",
b"digital",
b"hack",
b"::",
b"//",
b"com",
b"@",
b"crypto",
b"bitcoin",
b"wallet",
b"hacker",
b"welcome",
b"whatsapp",
b"email",
b"cryptocurrency",
b"stolen",
b"freeze",
b"quick",
b"crucial",
b"tracing",
b"scammers",
b"expers",
b"hire",
b"century",
b"transaction",
b"essential",
b"managing",
b"contact",
b"contacting",
b"understanding",
b"assets",
b"funds",
};
//static FW_TAB_DIR: [u8; 256] = [
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x09, 0x10, 0x18, 0x4f, 0x57, 0x6c, 0x00, 0x78, 0x00, 0x00, 0x00, 0x00, 0x90, 0x00, 0x00,
// 0x00, 0x99, 0x9f, 0xa8, 0xb7, 0xd0, 0x00, 0xde, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
//];
//static FW_TAB_STRS: [u8; 244] = [
// 0x00, 0x02, 0x2f, 0x00, 0x02, 0x3a, 0x00, 0x01, 0x00, 0x06, 0x73, 0x73, 0x65, 0x74, 0x73, 0x00,
// 0x07, 0x69, 0x74, 0x63, 0x6f, 0x69, 0x6e, 0x00, 0x07, 0x65, 0x6e, 0x74, 0x75, 0x72, 0x79, 0x03,
// 0x6f, 0x6d, 0x07, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x0a, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74,
// 0x69, 0x6e, 0x67, 0x07, 0x72, 0x75, 0x63, 0x69, 0x61, 0x6c, 0x06, 0x72, 0x79, 0x70, 0x74, 0x6f,
// 0x0e, 0x72, 0x79, 0x70, 0x74, 0x6f, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x00, 0x07,
// 0x69, 0x67, 0x69, 0x74, 0x61, 0x6c, 0x00, 0x05, 0x6d, 0x61, 0x69, 0x6c, 0x09, 0x73, 0x73, 0x65,
// 0x6e, 0x74, 0x69, 0x61, 0x6c, 0x06, 0x78, 0x70, 0x65, 0x72, 0x73, 0x00, 0x06, 0x72, 0x65, 0x65,
// 0x7a, 0x65, 0x05, 0x75, 0x6e, 0x64, 0x73, 0x00, 0x04, 0x61, 0x63, 0x6b, 0x06, 0x61, 0x63, 0x6b,
// 0x65, 0x72, 0x04, 0x69, 0x72, 0x65, 0x04, 0x74, 0x74, 0x70, 0x05, 0x74, 0x74, 0x70, 0x73, 0x00,
// 0x08, 0x61, 0x6e, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x00, 0x05, 0x75, 0x69, 0x63, 0x6b, 0x00, 0x08,
// 0x65, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x79, 0x00, 0x08, 0x63, 0x61, 0x6d, 0x6d, 0x65, 0x72, 0x73,
// 0x06, 0x74, 0x6f, 0x6c, 0x65, 0x6e, 0x00, 0x06, 0x65, 0x63, 0x68, 0x69, 0x65, 0x07, 0x72, 0x61,
// 0x63, 0x69, 0x6e, 0x67, 0x0b, 0x72, 0x61, 0x6e, 0x73, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x00,
// 0x0d, 0x6e, 0x64, 0x65, 0x72, 0x73, 0x74, 0x61, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x00, 0x06, 0x61,
// 0x6c, 0x6c, 0x65, 0x74, 0x07, 0x65, 0x6c, 0x63, 0x6f, 0x6d, 0x65, 0x08, 0x68, 0x61, 0x74, 0x73,
// 0x61, 0x70, 0x70, 0x00,
//];
12bitfloat_rust/risspam/src/main_slow_monoio.rs
+925
View File
@@ -0,0 +1,925 @@
#![feature(likely_unlikely)]
mod books;
use crate::books::FULL_BOOK_PATHS;
use core_affinity::CoreId;
use libc::{aio_read, aiocb};
use memmap2::Mmap;
use rayon::prelude::*;
use std::cell::OnceCell;
use std::cell::RefCell;
use std::ffi::{OsStr, OsString};
use std::fs::File;
use std::io::Read;
use std::mem::MaybeUninit;
use std::ops::Deref;
use std::os::linux::raw::stat;
use std::sync::atomic::{AtomicU8, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::thread::available_parallelism;
use std::time::{Duration, Instant};
use std::{array, env, fs, hint, mem, process, thread};
use monoio::IoUringDriver;
#[inline]
fn is_ascii_whitespace(b: u8) -> bool {
matches!(b, b'\t' | b'\n' | b'\x0C' | b'\r' | b' ')
}
#[inline]
fn is_ascii_upper(b: u8) -> bool {
matches!(b, b'A'..=b'Z')
}
#[inline]
fn is_ascii_digit(b: u8) -> bool {
matches!(b, b'0'..=b'9')
}
#[repr(align(128))]
#[derive(Copy, Clone)]
struct Stats {
pub sentences: u32,
pub words: u32,
pub capitalizeds: u32,
pub numbers: u32,
pub forbiddens: u32,
}
static TIME_SPENT_READING_FILES: Mutex<Duration> = Mutex::new(Duration::from_secs(0));
const TEMP_MEM_SIZE: usize = 6 * 1024 * 1024;
thread_local! {
static WORK_STATE: RefCell<WorkState> = RefCell::new(WorkState::new());
}
pub struct WorkState {
pub work_mem: Box<[u8]>,
pub empty_vec: Box<[u8]>,
// pub io_mem: Box<[u8]>,
// pub curr_read: Option<aiocb>,
// pub had_first_load: bool,
}
impl WorkState {
pub fn new() -> Self {
Self {
work_mem: vec![0; TEMP_MEM_SIZE].into_boxed_slice(),
empty_vec: vec![].into_boxed_slice(),
// io_mem: vec![0; TEMP_MEM_SIZE].into_boxed_slice(),
// curr_read: None,
// had_first_load: false,
}
}
}
fn work(file_path: &OsStr, stats: &mut Stats) {
WORK_STATE.with_borrow_mut(|state: &mut WorkState| {
// // Load file
// let start_time = Instant::now();
// let Ok(text) = fs::read(file_path) else {
// eprintln!("invalid file!");
// process::abort();
// };
let mut file = File::open(file_path).unwrap();
let file_len = file.metadata().unwrap().len() as usize;
file.read_exact(&mut state.work_mem[..file_len]).unwrap();
let text = &state.work_mem[..file_len];
unsafe {
let mut cb = mem::zeroed();
aio_read(&raw mut cb);
}
// let text = include_bytes!("../../../books/Advanced Techniques in Web Intelligence – Part II.txt").as_slice();
// let time_reading = start_time.elapsed();
// {
// let mut guard = TIME_SPENT_READING_FILES.lock().unwrap();
// *guard += time_reading;
// }
analyze(&text, stats);
});
}
fn analyze(text: &[u8], stats: &mut Stats) {
// // NOTE: mmap is quite a bit slower
// // Load file
// let Ok(file) = File::open(file_path) else {
// eprintln!("invalid file!");
// std::process::abort();
// };
// let mmap = unsafe {
// Mmap::map(&file).unwrap()
// };
// mem::forget(file);
// let text = &*mmap;
// // Load file
// let start_time = Instant::now();
// let Ok(text) = fs::read(file_path) else {
// eprintln!("invalid file!");
// process::abort();
// };
// let time_reading = start_time.elapsed();
// {
// let mut guard = TIME_SPENT_READING_FILES.lock().unwrap();
// *guard += time_reading;
// }
let mut sentences = 0;
let mut words = 0;
let mut capitalizeds = 0;
let mut numbers = 0;
let mut forbiddens = 0;
let mut idx = 0;
'full_loop: loop {
// TODO: Necessary for now
if idx >= text.len() {
break 'full_loop;
}
// Skip whitespace
while is_ascii_whitespace(text[idx]) {
idx += 1;
if idx >= text.len() {
break 'full_loop;
}
}
// Find end of word
let word_start = idx;
let mut has_non_upper = false;
'find_word_end: while let b = text[idx] && !is_ascii_whitespace(b) {
idx += 1;
if idx >= text.len() {
break 'find_word_end;
}
// Per-char logic
if b == b'.' {
sentences += 1;
}
if !is_ascii_upper(b) {
has_non_upper = true;
}
if is_ascii_digit(b) {
numbers += 1;
}
}
let word = &text[word_start..idx];
// dbg!(str::from_utf8(word).unwrap());
words += 1;
if !has_non_upper {
capitalizeds += 1;
}
// Check forbidden
if unsafe { FW_TAB.lookup(word) } {
forbiddens += 1;
}
}
/*
for token in text.split(|&b| is_ascii_whitespace(b)) {
if token.is_empty() {
continue;
}
words += 1;
// Sentence count, folded into this loop
// instead of another loop (better cache usage)
for &b in token {
if b == b'.' {
sentences += 1;
}
}
// Check if upper
if token.iter().all(|&b| is_ascii_upper(b)) {
capitalizeds += 1;
}
// Check digits
for &b in token {
if is_ascii_digit(b) {
numbers += 1;
}
}
// Check if words
// if FORBIDDEN_WORDS.contains(&token) {
// if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, token) } {
if unsafe { FW_TAB.lookup(token) } {
forbiddens += 1;
}
}
*/
/*
// NOTE: This is pretty slow:
let mut idx = 0;
let mut word_start = 0;
let mut is_in_word = false;
let mut has_non_upper = false;
loop {
let b = unsafe { *text.get_unchecked(idx) };
let mut process_word = false;
if is_ascii_whitespace(b) {
if is_in_word {
process_word = true;
// Reset state for next word
is_in_word = false;
has_non_upper = false;
}
} else {
if !is_in_word {
word_start = idx;
is_in_word = true;
}
has_non_upper |= !is_ascii_upper(b);
}
// Check digits
if is_ascii_digit(b) {
numbers += 1;
}
// Check sentences
if b == b'.' {
sentences += 1;
}
let word = &text[word_start..idx];
idx += 1;
if process_word || idx >= text.len() {
words += 1;
if !has_non_upper {
capitalizeds += 1;
}
// // DEBUG:
// println!("'{}'", str::from_utf8(word).unwrap());
if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, word) } {
forbiddens += 1;
}
}
if idx >= text.len() {
break;
}
}
*/
stats.sentences = sentences;
stats.words = words;
stats.capitalizeds = capitalizeds;
stats.numbers = numbers;
stats.forbiddens = forbiddens;
}
/*
fn analyze_old(file_path: &OsStr, stats: &mut Stats) {
// Load file
let Ok(text) = fs::read(file_path) else {
eprintln!("invalid file!");
std::process::abort();
};
let mut sentences = 0;
let mut words = 0;
let mut capitalizeds = 0;
let mut numbers = 0;
let mut forbiddens = 0;
for token in text.split(|&b| is_ascii_whitespace(b)) {
if token.is_empty() {
continue;
}
words += 1;
// Sentence count, folded into this loop
// instead of another loop (better cache usage)
for &b in token {
if b == b'.' {
sentences += 1;
}
}
// Check if upper
if token.iter().all(|&b| is_ascii_upper(b)) {
capitalizeds += 1;
}
// Check digits
for &b in token {
if is_ascii_digit(b) {
numbers += 1;
}
}
// Check if words
// if FORBIDDEN_WORDS.contains(&token) {
if unsafe { FwTab::lookup_raw(&FW_TAB_DIR, &FW_TAB_STRS, token) } {
forbiddens += 1;
}
}
stats.sentences = sentences;
stats.words = words;
stats.capitalizeds = capitalizeds;
stats.numbers = numbers;
stats.forbiddens = forbiddens;
}
*/
fn main() {
// Read in files from args
let mut files = Vec::with_capacity(env::args().len());
let mut do_parallel = false;
let start_time = Instant::now();
for arg in env::args_os().skip(1) {
// skip program arg
if arg == "-p" {
do_parallel = true;
} else {
files.push(arg);
}
}
println!("[PROFILE] taking args took {:?}", start_time.elapsed());
// env::args_os().
// let files = FULL_BOOK_PATHS;
// // Build table
// let tab = FwTab::build();
// tab.compile();
// Do the work
let mut stats = vec![Stats {
sentences: 0,
words: 0,
capitalizeds: 0,
numbers: 0,
forbiddens: 0,
}; files.len()];
let start_time = Instant::now();
let num_cores = available_parallelism().unwrap().get();
let num_threads = num_cores * 1;
// DEBUG:
dbg!(num_threads);
dbg!(num_cores);
let next_file_idx = &*Box::leak(Box::new(AtomicUsize::new(0)));
thread::scope(|scope| {
for thread_idx in 0..num_threads {
// Set thread affinity
assert!(core_affinity::set_for_current(CoreId { id: thread_idx % num_cores }));
let cap_next_file_idx = &next_file_idx;
let cap_stats_ptr = stats.as_ptr() as usize;
let cap_files = &files;
thread::Builder::new().spawn_scoped(scope, move || {
let files = cap_files;
// let exec = glommio::LocalExecutorBuilder::new(Placement::Unbound).make().unwrap();
// exec.run(async {
// println!("Running in glommio thread {core_idx}");
// });
let mut rt = monoio::RuntimeBuilder::<IoUringDriver>::new()
.build()
.unwrap();
let mut work_mem = vec![0; TEMP_MEM_SIZE].into_boxed_slice();
let files_per_thread = files.len() / num_threads;
rt.block_on(async {
// // Claim next file id
// loop {
// let work_idx = cap_next_file_idx.fetch_add(1, Ordering::Relaxed);
// if work_idx >= files.len() {
// return;
// }
// Do work
let thread_start = thread_idx * files_per_thread;
for i in 0..files_per_thread {
let work_idx = thread_start + i;
let path = &files[work_idx];
let stat = unsafe {
&mut *(cap_stats_ptr as *mut Stats)
.offset(work_idx as isize)
};
let file = monoio::fs::File::open(path)
.await
.unwrap();
struct CappedReadBuf(pub Box<[u8]>, usize);
unsafe impl monoio::buf::IoBufMut for CappedReadBuf {
fn write_ptr(&mut self) -> *mut u8 {
monoio::buf::IoBufMut::write_ptr(&mut self.0)
}
fn bytes_total(&mut self) -> usize {
self.1
}
unsafe fn set_init(&mut self, pos: usize) {
monoio::buf::IoBufMut::set_init(&mut self.0, pos)
}
}
let meta = file.metadata().await.unwrap();
let io_mem = mem::take(&mut work_mem);
let (res, buf) = file.read_exact_at(CappedReadBuf(io_mem, meta.len() as usize), 0).await;
res.unwrap();
work_mem = buf.0;
analyze(&work_mem, stat);
}
})
// exec.run(async {
// // Claim next file id
// let work_idx = cap_next_file_idx.fetch_add(1, Ordering::Relaxed);
//
// let path = &files[work_idx];
// let stat = unsafe {
// &mut *(stats_ptr as *mut Stats)
// .offset(work_idx as isize)
// };
//
// work(path, stat);
// });
}).unwrap();
}
});
// rayon::ThreadPoolBuilder::new()
// .num_threads(num_threads)
// .build_global()
// .unwrap();
//
// files.par_iter()
// .enumerate()
// .for_each(|(idx, p)| {
// let s = unsafe {
// &mut *stats.as_ptr()
// .offset(idx as isize)
// .cast_mut()
// };
//
//// let mut path = OsString::from("../../");
//// path.push(p);
// let path = p;
// work(path, s);
// });
// thread::scope(|scope| {
// let files_per_thread = files.len() / num_threads;
//
// for thread_idx in 0..num_threads {
// let capture_files = &files;
// let capture_stats = &stats;
// thread::Builder::new().spawn_scoped(scope, move || {
// let files = capture_files;
// let stats = capture_stats;
//
// // Set thread affinity
// assert!(core_affinity::set_for_current(CoreId { id: thread_idx % num_cores }));
//
// // Do work
// let thread_start = thread_idx * files_per_thread;
// for i in 0..files_per_thread {
// let real_idx = thread_start + i;
// let file_path = &files[real_idx];
// let st = unsafe {
// &mut *stats.as_ptr()
// .offset(real_idx as isize)
// .cast_mut()
// };
//
// work(&file_path, st);
// }
// }).unwrap();
// }
// });
println!("[PROFILE] processing text took {:?}", start_time.elapsed());
// Accumulate stats
let start_time = Instant::now();
let mut total_words = 0;
let mut total_capitalizeds = 0;
let mut total_sentences = 0;
let mut total_numbers = 0;
let mut total_forbiddens = 0;
for stat in &stats {
total_words += stat.words;
total_capitalizeds += stat.capitalizeds;
total_sentences += stat.sentences;
total_numbers += stat.numbers;
total_forbiddens += stat.forbiddens;
}
let capitalized_percentage = (total_capitalizeds as f32 / total_words as f32) * 100.0;
let forbidden_percentage = (total_forbiddens as f32 / total_words as f32) * 100.0;
let word_count_per_sentence = total_words as f32 / total_sentences as f32;
println!();
println!("Total Words: {total_words}");
println!("Total Capitalized words: {total_capitalizeds}");
println!("Total Sentences: {total_sentences}");
println!("Total Numbers: {total_numbers}");
println!("Total Forbidden words: {total_forbiddens}");
println!("Capitalized percentage: {capitalized_percentage:.6}");
println!("Forbidden percentage: {forbidden_percentage:.6}");
println!("Word count per sentence: {word_count_per_sentence:.6}");
println!("Total files read: {}", files.len());
println!("[PROFILE] accumulating stats took {:?}", start_time.elapsed());
println!("[PROFILE] total file reading took {:?}", &*TIME_SPENT_READING_FILES.lock().unwrap());
// Exit process to avoid running drops
process::exit(0);
}
#[repr(C)]
struct FwTab {
// pub dir_and_len_bits: [u32; 256],
pub dir_len_bits: [u16; 256],
pub dir: [u8; 256],
pub strs: [u8; 256],
}
impl FwTab {
pub fn build() -> Self {
// Sort fws by first char
let mut sorted_fws: Vec<Vec<&'static [u8]>> = vec![vec![]; 256];
for word in FORBIDDEN_WORDS {
sorted_fws[word[0] as usize].push(&word);
}
for i in 0..256 {
sorted_fws[i].sort()
}
// // DEBUG:
// println!("{:#?}", sorted_fws[b'@' as usize].iter().map(|s| str::from_utf8(s).unwrap()).collect::<Vec<_>>());
// Build str tab
let mut fw_dir = [0u8; 256];
// let mut fw_dir_len_bits = [0u32; 256];
let mut fw_dir_len_bits = [0u16; 256];
let mut fw_strs: Vec<u8> = vec![];
fw_strs.push(b'\0'); // push dummy value so that 0 in the dir means no-entries
for c in 0..256 {
for fw in FORBIDDEN_WORDS {
if c == fw[0] as usize {
fw_dir_len_bits[c] |= 0x1 << fw.len();
}
}
if !sorted_fws[c].is_empty() {
let sublist_start_offset = fw_strs.len().try_into().unwrap();
fw_dir[c] = sublist_start_offset;
// DEBUG:
println!("{c} start offset: {}", sublist_start_offset);
println!("{:#?}", sorted_fws[c].iter().map(|s| str::from_utf8(s).unwrap()).collect::<Vec<_>>());
// Push strings
for fw in &sorted_fws[c] {
fw_strs.push(fw.len().try_into().unwrap());
for &c in &fw[1..] {
fw_strs.push(c);
}
}
// Mark end of per-char word sublist
fw_strs.push(b'\0');
}
}
// DEBUG:
println!("strs len: {}", fw_strs.len());
assert_eq!(fw_dir.len(), 256);
assert!(fw_strs.len() <= 256);
fw_strs.resize(256, 0);
let tab = FwTab {
dir: fw_dir,
dir_len_bits: fw_dir_len_bits,
// dir_and_len_bits: array::from_fn(|idx| {
// (fw_dir_len_bits[idx] & 0xff_ff_ff) | ((fw_dir[idx] as u32) << 24)
// }),
strs: fw_strs.try_into().unwrap(),
};
// DEBUG: Test some strings
unsafe {
dbg!(tab.lookup(b"cpm"));
dbg!(tab.lookup(b"com"));
dbg!(tab.lookup(b"coma"));
dbg!(tab.lookup(b"co"));
dbg!(tab.lookup(b"cam"));
dbg!(tab.lookup(b"crypto"));
dbg!(tab.lookup(b"@"));
dbg!(tab.lookup(b""));
dbg!(tab.lookup(b" "));
dbg!(tab.lookup(b"test"));
dbg!(tab.lookup(b"expers"));
}
tab
}
pub fn compile(&self) {
println!("static FW_TAB: FwTab = FwTab {{");
// println!("\tdir_and_len_bits: [");
// for chunk in self.dir_and_len_bits.chunks(16) {
// print!("\t\t");
// for &b in chunk {
// print!("0x{b:08x}, ");
// }
// println!();
// }
// println!("\t],");
println!("\tdir: [");
for chunk in self.dir.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:02x}, ");
}
println!();
}
println!("\t],");
println!("\tdir_len_bits: [");
for chunk in self.dir_len_bits.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:04x}, ");
}
println!();
}
println!("\t],");
println!("\tstrs: [");
for chunk in self.strs.chunks(16) {
print!("\t\t");
for &b in chunk {
print!("0x{b:02x}, ");
}
println!();
}
println!("\t],");
println!("}};");
}
#[inline]
pub unsafe fn lookup(&self, word: &[u8]) -> bool {
let &[first_char, ..] = word else {
return false;
};
// let dir_and_len_bits = unsafe {
// *self.dir_and_len_bits.get_unchecked(first_char as usize)
// };
// if word.len() < 23 && ((dir_and_len_bits >> word.len()) & 0x1) == 0 {
// return false;
// }
let len_bits = unsafe {
*self.dir_len_bits.get_unchecked(first_char as usize)
};
if word.len() < 16 && ((len_bits >> word.len()) & 0x1) == 0 {
return false;
}
// let mut str_offset = (dir_and_len_bits >> 24) as usize;
let mut str_offset = unsafe {
*self.dir.get_unchecked(first_char as usize) as usize
};
// Char doesn't have any strings in the table
if str_offset == 0 {
return false;
}
// Iterate over strs
loop {
// let fw_len = u16::from_le_bytes([
// self.strs[str_offset],
// self.strs[str_offset+1]
// ]);
let fw_len: u8 = unsafe {
*self.strs.get_unchecked(str_offset)
};
if fw_len == 0 {
// We've reached the end of the word sublist
return false;
}
// Only compare words if they are the same length
if word.len() == fw_len as usize {
// Compare strs
let mut char_offset = 1usize;
loop {
// Found the word!
if char_offset == word.len() {
return true;
}
let fw_char = unsafe { *self.strs.get_unchecked(str_offset + char_offset) };
let word_char = unsafe { *word.get_unchecked(char_offset) };
if fw_char > word_char {
// Word can't possible be in the sorted list, return
return false;
}
if fw_char < word_char {
// Try next word
break;
}
char_offset += 1;
}
}
// Advance to next word
// let str_len_bytes = 2;
let str_len_bytes = 1;
str_offset += (fw_len as usize - 1) + str_len_bytes;
}
}
}
const FORBIDDEN_WORDS: [&'static [u8]; 35] = [
b"recovery",
b"techie",
b"http",
b"https",
b"digital",
b"hack",
b"::",
b"//",
b"com",
b"@",
b"crypto",
b"bitcoin",
b"wallet",
b"hacker",
b"welcome",
b"whatsapp",
b"email",
b"cryptocurrency",
b"stolen",
b"freeze",
b"quick",
b"crucial",
b"tracing",
b"scammers",
b"expers",
b"hire",
b"century",
b"transaction",
b"essential",
b"managing",
b"contact",
b"contacting",
b"understanding",
b"assets",
b"funds",
];
static FW_TAB: FwTab = FwTab {
dir: [
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00,
0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x09, 0x10, 0x18, 0x4f, 0x57, 0x6c, 0x00, 0x78, 0x00, 0x00, 0x00, 0x00, 0x90, 0x00, 0x00,
0x00, 0x99, 0x9f, 0xa8, 0xb7, 0xd0, 0x00, 0xde, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
],
dir_len_bits: [
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0004,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0004, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0002, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0040, 0x0080, 0x44c8, 0x0080, 0x0260, 0x0060, 0x0000, 0x0070, 0x0000, 0x0000, 0x0000, 0x0000, 0x0100, 0x0000, 0x0000,
0x0000, 0x0020, 0x0100, 0x0140, 0x08c0, 0x2000, 0x0000, 0x01c0, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
],
strs: [
0x00, 0x02, 0x2f, 0x00, 0x02, 0x3a, 0x00, 0x01, 0x00, 0x06, 0x73, 0x73, 0x65, 0x74, 0x73, 0x00,
0x07, 0x69, 0x74, 0x63, 0x6f, 0x69, 0x6e, 0x00, 0x07, 0x65, 0x6e, 0x74, 0x75, 0x72, 0x79, 0x03,
0x6f, 0x6d, 0x07, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x0a, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74,
0x69, 0x6e, 0x67, 0x07, 0x72, 0x75, 0x63, 0x69, 0x61, 0x6c, 0x06, 0x72, 0x79, 0x70, 0x74, 0x6f,
0x0e, 0x72, 0x79, 0x70, 0x74, 0x6f, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x00, 0x07,
0x69, 0x67, 0x69, 0x74, 0x61, 0x6c, 0x00, 0x05, 0x6d, 0x61, 0x69, 0x6c, 0x09, 0x73, 0x73, 0x65,
0x6e, 0x74, 0x69, 0x61, 0x6c, 0x06, 0x78, 0x70, 0x65, 0x72, 0x73, 0x00, 0x06, 0x72, 0x65, 0x65,
0x7a, 0x65, 0x05, 0x75, 0x6e, 0x64, 0x73, 0x00, 0x04, 0x61, 0x63, 0x6b, 0x06, 0x61, 0x63, 0x6b,
0x65, 0x72, 0x04, 0x69, 0x72, 0x65, 0x04, 0x74, 0x74, 0x70, 0x05, 0x74, 0x74, 0x70, 0x73, 0x00,
0x08, 0x61, 0x6e, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x00, 0x05, 0x75, 0x69, 0x63, 0x6b, 0x00, 0x08,
0x65, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x79, 0x00, 0x08, 0x63, 0x61, 0x6d, 0x6d, 0x65, 0x72, 0x73,
0x06, 0x74, 0x6f, 0x6c, 0x65, 0x6e, 0x00, 0x06, 0x65, 0x63, 0x68, 0x69, 0x65, 0x07, 0x72, 0x61,
0x63, 0x69, 0x6e, 0x67, 0x0b, 0x72, 0x61, 0x6e, 0x73, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x00,
0x0d, 0x6e, 0x64, 0x65, 0x72, 0x73, 0x74, 0x61, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x00, 0x06, 0x61,
0x6c, 0x6c, 0x65, 0x74, 0x07, 0x65, 0x6c, 0x63, 0x6f, 0x6d, 0x65, 0x08, 0x68, 0x61, 0x74, 0x73,
0x61, 0x70, 0x70, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
],
};
//static FW_TAB_DIR: [u8; 256] = [
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x07, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x09, 0x10, 0x18, 0x4f, 0x57, 0x6c, 0x00, 0x78, 0x00, 0x00, 0x00, 0x00, 0x90, 0x00, 0x00,
// 0x00, 0x99, 0x9f, 0xa8, 0xb7, 0xd0, 0x00, 0xde, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
// 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
//];
//static FW_TAB_STRS: [u8; 244] = [
// 0x00, 0x02, 0x2f, 0x00, 0x02, 0x3a, 0x00, 0x01, 0x00, 0x06, 0x73, 0x73, 0x65, 0x74, 0x73, 0x00,
// 0x07, 0x69, 0x74, 0x63, 0x6f, 0x69, 0x6e, 0x00, 0x07, 0x65, 0x6e, 0x74, 0x75, 0x72, 0x79, 0x03,
// 0x6f, 0x6d, 0x07, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x0a, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74,
// 0x69, 0x6e, 0x67, 0x07, 0x72, 0x75, 0x63, 0x69, 0x61, 0x6c, 0x06, 0x72, 0x79, 0x70, 0x74, 0x6f,
// 0x0e, 0x72, 0x79, 0x70, 0x74, 0x6f, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x00, 0x07,
// 0x69, 0x67, 0x69, 0x74, 0x61, 0x6c, 0x00, 0x05, 0x6d, 0x61, 0x69, 0x6c, 0x09, 0x73, 0x73, 0x65,
// 0x6e, 0x74, 0x69, 0x61, 0x6c, 0x06, 0x78, 0x70, 0x65, 0x72, 0x73, 0x00, 0x06, 0x72, 0x65, 0x65,
// 0x7a, 0x65, 0x05, 0x75, 0x6e, 0x64, 0x73, 0x00, 0x04, 0x61, 0x63, 0x6b, 0x06, 0x61, 0x63, 0x6b,
// 0x65, 0x72, 0x04, 0x69, 0x72, 0x65, 0x04, 0x74, 0x74, 0x70, 0x05, 0x74, 0x74, 0x70, 0x73, 0x00,
// 0x08, 0x61, 0x6e, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x00, 0x05, 0x75, 0x69, 0x63, 0x6b, 0x00, 0x08,
// 0x65, 0x63, 0x6f, 0x76, 0x65, 0x72, 0x79, 0x00, 0x08, 0x63, 0x61, 0x6d, 0x6d, 0x65, 0x72, 0x73,
// 0x06, 0x74, 0x6f, 0x6c, 0x65, 0x6e, 0x00, 0x06, 0x65, 0x63, 0x68, 0x69, 0x65, 0x07, 0x72, 0x61,
// 0x63, 0x69, 0x6e, 0x67, 0x0b, 0x72, 0x61, 0x6e, 0x73, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x00,
// 0x0d, 0x6e, 0x64, 0x65, 0x72, 0x73, 0x74, 0x61, 0x6e, 0x64, 0x69, 0x6e, 0x67, 0x00, 0x06, 0x61,
// 0x6c, 0x6c, 0x65, 0x74, 0x07, 0x65, 0x6c, 0x63, 0x6f, 0x6d, 0x65, 0x08, 0x68, 0x61, 0x74, 0x73,
// 0x61, 0x70, 0x70, 0x00,
//];
12bitfloat_rust/risspam/src/uring.rs
+8
View File
@@ -0,0 +1,8 @@
pub fn test() {
// let ring = io_uring::Builder::<io_uring::squeue::Entry, io_uring::cqueue::Entry>::default()
// .build(128)
// .unwrap();
//
// ring.
}
12bitfloat_rust/risspam/target/.rustc_info.json
-1
View File
@@ -1 +0,0 @@
{"rustc_fingerprint":10200614701319076238,"outputs":{"4614504638168534921":{"success":true,"status":"","code":0,"stdout":"rustc 1.85.0-nightly (d10a6823f 2024-11-29)\nbinary: rustc\ncommit-hash: d10a6823f4c3176be7a05a2454e5c33b861cb05f\ncommit-date: 2024-11-29\nhost: x86_64-unknown-linux-gnu\nrelease: 1.85.0-nightly\nLLVM version: 19.1.4\n","stderr":""},"15729799797837862367":{"success":true,"status":"","code":0,"stdout":"___\nlib___.rlib\nlib___.so\nlib___.so\nlib___.a\nlib___.so\n/home/retoor/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu\noff\npacked\nunpacked\n___\ndebug_assertions\nfmt_debug=\"full\"\noverflow_checks\npanic=\"unwind\"\nproc_macro\nrelocation_model=\"pic\"\ntarget_abi=\"\"\ntarget_arch=\"x86_64\"\ntarget_endian=\"little\"\ntarget_env=\"gnu\"\ntarget_family=\"unix\"\ntarget_feature=\"fxsr\"\ntarget_feature=\"sse\"\ntarget_feature=\"sse2\"\ntarget_has_atomic\ntarget_has_atomic=\"16\"\ntarget_has_atomic=\"32\"\ntarget_has_atomic=\"64\"\ntarget_has_atomic=\"8\"\ntarget_has_atomic=\"ptr\"\ntarget_has_atomic_equal_alignment=\"16\"\ntarget_has_atomic_equal_alignment=\"32\"\ntarget_has_atomic_equal_alignment=\"64\"\ntarget_has_atomic_equal_alignment=\"8\"\ntarget_has_atomic_equal_alignment=\"ptr\"\ntarget_has_atomic_load_store\ntarget_has_atomic_load_store=\"16\"\ntarget_has_atomic_load_store=\"32\"\ntarget_has_atomic_load_store=\"64\"\ntarget_has_atomic_load_store=\"8\"\ntarget_has_atomic_load_store=\"ptr\"\ntarget_os=\"linux\"\ntarget_pointer_width=\"64\"\ntarget_thread_local\ntarget_vendor=\"unknown\"\nub_checks\nunix\n","stderr":""}},"successes":{}}
12bitfloat_rust/risspam/target/CACHEDIR.TAG
-3
View File
@@ -1,3 +0,0 @@
Signature: 8a477f597d28d172789f06886806bc55
# This file is a cache directory tag created by cargo.
# For information about cache directory tags see https://bford.info/cachedir/
12bitfloat_rust/risspam/target/release/.cargo-lock
View File
12bitfloat_rust/risspam/target/release/.fingerprint/risspam-588b7c16f8fda172/bin-risspam
-1
View File
@@ -1 +0,0 @@
bcbff30c781130f6
12bitfloat_rust/risspam/target/release/.fingerprint/risspam-588b7c16f8fda172/bin-risspam.json
-1
View File
@@ -1 +0,0 @@
{"rustc":7959095874983568062,"features":"[]","declared_features":"[]","target":10519780086885261595,"profile":18277820415669657429,"path":10602529704205407992,"deps":[],"local":[{"CheckDepInfo":{"dep_info":"release/.fingerprint/risspam-588b7c16f8fda172/dep-bin-risspam","checksum":false}}],"rustflags":[],"metadata":7797948686568424061,"config":2202906307356721367,"compile_kind":0}
12bitfloat_rust/risspam/target/release/.fingerprint/risspam-588b7c16f8fda172/dep-bin-risspam
BIN
View File
Binary file not shown.
12bitfloat_rust/risspam/target/release/.fingerprint/risspam-588b7c16f8fda172/invoked.timestamp
-1
View File
@@ -1 +0,0 @@
This file has an mtime of when this was started.
12bitfloat_rust/risspam/target/release/.fingerprint/risspam-edd96cae17e7d87c/invoked.timestamp
-1
View File
@@ -1 +0,0 @@
This file has an mtime of when this was started.
12bitfloat_rust/risspam/target/release/.fingerprint/risspam-edd96cae17e7d87c/output-bin-risspam
-3
View File
@@ -1,3 +0,0 @@
{"$message_type":"diagnostic","message":"`#![feature]` may not be used on the stable release channel","code":{"code":"E0554","explanation":"Feature attributes are only allowed on the nightly release channel. Stable or\nbeta compilers will not comply.\n\nErroneous code example:\n\n```ignore (depends on release channel)\n#![feature(lang_items)] // error: `#![feature]` may not be used on the\n // stable release channel\n```\n\nIf you need the feature, make sure to use a nightly release of the compiler\n(but be warned that the feature may be removed or altered in the future).\n"},"level":"error","spans":[{"file_name":"src/main.rs","byte_start":0,"byte_end":23,"line_start":1,"line_end":1,"column_start":1,"column_end":24,"is_primary":true,"text":[{"text":"#![feature(let_chains)]","highlight_start":1,"highlight_end":24}],"label":null,"suggested_replacement":null,"suggestion_applicability":null,"expansion":null}],"children":[],"rendered":"\u001b[0m\u001b[1m\u001b[38;5;9merror[E0554]\u001b[0m\u001b[0m\u001b[1m: `#![feature]` may not be used on the stable release channel\u001b[0m\n\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m--> \u001b[0m\u001b[0msrc/main.rs:1:1\u001b[0m\n\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m|\u001b[0m\n\u001b[0m\u001b[1m\u001b[38;5;12m1\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m#![feature(let_chains)]\u001b[0m\n\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;12m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[1m\u001b[38;5;9m^^^^^^^^^^^^^^^^^^^^^^^\u001b[0m\n\n"}
{"$message_type":"diagnostic","message":"aborting due to 1 previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"\u001b[0m\u001b[1m\u001b[38;5;9merror\u001b[0m\u001b[0m\u001b[1m: aborting due to 1 previous error\u001b[0m\n\n"}
{"$message_type":"diagnostic","message":"For more information about this error, try `rustc --explain E0554`.","code":null,"level":"failure-note","spans":[],"children":[],"rendered":"\u001b[0m\u001b[1mFor more information about this error, try `rustc --explain E0554`.\u001b[0m\n"}
12bitfloat_rust/risspam/target/release/deps/risspam-588b7c16f8fda172
BIN
View File
Binary file not shown.
12bitfloat_rust/risspam/target/release/deps/risspam-588b7c16f8fda172.d
-5
View File
@@ -1,5 +0,0 @@
/home/retoor/projects/spam/rust/risspam/target/release/deps/risspam-588b7c16f8fda172: src/main.rs
/home/retoor/projects/spam/rust/risspam/target/release/deps/risspam-588b7c16f8fda172.d: src/main.rs
src/main.rs:
12bitfloat_rust/risspam/target/release/deps/risspam-edd96cae17e7d87c.d
-5
View File
@@ -1,5 +0,0 @@
/home/retoor/projects/spam/rust/risspam/target/release/deps/risspam-edd96cae17e7d87c: src/main.rs
/home/retoor/projects/spam/rust/risspam/target/release/deps/risspam-edd96cae17e7d87c.d: src/main.rs
src/main.rs:
12bitfloat_rust/risspam/target/release/risspam
BIN
View File
Binary file not shown.
12bitfloat_rust/risspam/target/release/risspam.d
-1
View File
@@ -1 +0,0 @@
/home/retoor/projects/spam/rust/risspam/target/release/risspam: /home/retoor/projects/spam/rust/risspam/src/main.rs
Makefile
+74 -9
View File
@@ -1,20 +1,65 @@
CC = gcc
CFLAGS = -Wall -Werror -Wextra -Ofast -std=c2x
CC = gcc
CFLAGS = -Ofast -march=native -mtune=native -flto=auto -fuse-linker-plugin \
-funroll-all-loops -finline-functions -finline-limit=10000 \
-fprefetch-loop-arrays -ftracer -fmodulo-sched \
-fmodulo-sched-allow-regmoves -fgcse-sm -fgcse-las \
-ftree-loop-distribution -ftree-loop-im -ftree-loop-ivcanon \
-fivopts -fvariable-expansion-in-unroller -fvect-cost-model=unlimited \
-fipa-pta -fipa-cp-clone -fdevirtualize-speculatively \
-fno-plt -fno-semantic-interposition -fomit-frame-pointer \
-fmerge-all-constants -fno-stack-protector -DNDEBUG \
-fno-trapping-math -fno-signed-zeros -freciprocal-math \
-mavx2 -mfma -mbmi2 -mlzcnt -mpopcnt \
-pthread
all: build run run_rust
all: build run valgrind build_risspam run_risspam build_cpp build_borded_cpp build_py build_jest
build:
@# removed -pedantic flag because it doesn't accept ' for formatting numbers
@# using printf
@$(CC) $(CFLAGS) isspam.c -o isspam
@echo "Building retoor_c with PGO..."
@rm -rf pgo_data && mkdir -p pgo_data
@$(CC) $(CFLAGS) -fprofile-dir=pgo_data -fprofile-generate retoor_c/isspam.c -o isspam
@./isspam ./spam/*.txt > /dev/null 2>&1 || true
@./isspam ./not_spam/*.txt > /dev/null 2>&1 || true
@[ -d books ] && ./isspam ./books/*.txt > /dev/null 2>&1 || true
@$(CC) $(CFLAGS) -fprofile-dir=pgo_data -fprofile-use -fprofile-correction retoor_c/isspam.c -o isspam
@rm -rf pgo_data
build_quick:
@echo "Compiling retoor_c project (no PGO)."
@$(CC) $(CFLAGS) retoor_c/isspam.c -o isspam
build_py:
@echo "Copying py file"
@cp retoor_c/isspam.py isspam.py
build_cpp:
@echo "Compiling C++ version of isspam."
@g++ -Ofast retoor_c/isspam.cpp -o isspam_cpp
build_borded_cpp:
@echo "Compiling Borded C++ version of isspam."
@g++ -std=c++23 -Ofast borded_cpp/src/main3.cpp -o borded_cpp_exec
build_risspam:
cd 12bitfloat_rust/risspam && cargo run --release && cp target/release/risspam ../../
@echo "Compiling 12bitfloat_risspam project."
cd 12bitfloat_rust/risspam && cargo build --release && cp target/release/risspam ../../
build_jest:
@echo "compiling jest_rust project"
cd jest_rust && cargo build --release && cp target/release/jisspam ..
build_swift:
@echo "Compiling Swift version of isspam."
cd swift_isspam && swift build -c release && cp .build/release/sisspam ../
build_all: build build_py build_cpp build_borded_cpp build_risspam build_jest build_swift
run: run_spam wl run_not_spam
run_risspam: run_spam_rispam run_not_spam_rispam
run_risspam: run_spam_risspam run_not_spam_risspam
bench_rust: build_risspam benchmark_only
bench_rust_only: build_risspam
cd 12bitfloat_rust/risspam && time ./target/release/risspam ../../books/*.txt
format:
clang-format *.c *.h -i
@@ -34,6 +79,26 @@ run_spam_risspam:
run_not_spam_risspam:
@./risspam ./not_spam/*.txt
valgrind: build
valgrind ./isspam ./spam/*.txt
publish:
@wget https://retoor.molodetz.nl/api/packages/retoor/generic/env.py/1.0.0/env.py --quiet
@wget https://retoor.molodetz.nl/api/packages/retoor/generic/publish/1.0.0/publish --quiet
@chmod +x publish
@./publish isspam
@./publish risspam
@./publish sisspam
@rm publish
@rm env.py
benchmark:
-@rm -rf books
@echo "Extracting books."
@tar -xzf books.tar.gz books/
@echo "Extracted books."
@python bench.py
# Skip extracting books over and over
benchmark_only:
@python bench.py
README.md
+36 -7
View File
@@ -1,26 +1,50 @@
# Isspam
Fast as light evaluator for text files to summarize specific details about the text files.
# isspam
Fast as light evaluator for text files to summarize specific details about the text files.
This repository contains multiple versions of the same(-ish) algorithm.
## Versions
- C (isspam) written by **@retoor**
- Rust (risspam) written by **@12bitfloat**
- C++ (isspam_cpp) written by **@BordedDev**
- Rust (jisspam) written by **@jestdotty**
- Swift (sisspam) written by **@retoor**
## Building
Build all versions to the repo root:
```
make build
make build_all
```
Build with memory check (requires valgrind to be installed):
Build isspam (C) with memory check (requires valgrind to be installed):
```
make valgrind
```
## Benchmarking
After all binaries have been build to the repo root, you can benchmark them like this:
```
make benchmark
```
or without extracting books again:
```
make benchmark_only
```
## Running
### Using files as parameter
```
./isspam ./spam/*.txt
./isspam ./not_spam/*.txt
./(r)isspam ./spam/*.txt
./(r)isspam ./not_spam/*.txt
```
### Using stdin
Useful for automation.
Useful for automation. Works only on the isspam version.
```
cat ./spam/example_spam1.txt | ./isspam
```
## Example output
Output example made by isspam.
```
File: ./spam/example_spam3.txt
Capitalized words: 39
@@ -47,7 +71,12 @@ Word count per sentence: 21
Memory usage: 1 MB, 6.460 (re)allocated, 4.222 unqiue free'd, 0 in use.
```
## Valgrind status
Valgrind output for isspam version.
Rust variant thinks it's too cool for memory checks afterwards.
Date: 2024-11-30
```
==58062==
==58062== HEAP SUMMARY:
bench.py
+23
View File
@@ -0,0 +1,23 @@
import subprocess
import time
print("***benchmarking***")
time_start = time.time()
subprocess.check_output('./isspam books/*.txt', shell=True)
print("Time C:",time.time() - time_start)
time_start = time.time()
subprocess.check_output('./risspam -p books/*.txt', shell=True)
print("Time Rust:",time.time() - time_start)
time_start = time.time()
subprocess.check_output('./isspam_cpp books/*.txt', shell=True)
print("Time CPP:",time.time() - time_start)
time_start = time.time()
subprocess.check_output('./borded_cpp_exec books/*.txt', shell=True)
print("Time Borded CPP:",time.time() - time_start)
time_start = time.time()
subprocess.check_output('./jisspam books/*.txt', shell=True)
print("Time Jest Rust:", time.time() - time_start)
time_start = time.time()
subprocess.check_output('./sisspam books/*.txt', shell=True)
print("Time Swift:", time.time() - time_start)
print("***end benchmark***")
books.tar.gz
BIN
View File
Binary file not shown.
borded_cpp/.gitignore
+97
View File
@@ -0,0 +1,97 @@
*.d
*.slo
*.lo
*.o
*.obj
*.gch
*.pch
*.so
*.dylib
*.dll
*.mod
*.smod
*.lai
*.la
*.a
*.lib
*.exe
*.out
*.app
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
.idea/**/aws.xml
.idea/**/contentModel.xml
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
.idea/**/gradle.xml
.idea/**/libraries
.idea
cmake-build-*/
.idea/**/mongoSettings.xml
*.iws
out/
.idea_modules/
atlassian-ide-plugin.xml
.idea/replstate.xml
.idea/sonarlint/
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
.idea/httpRequests
.idea/caches/build_file_checksums.ser
*~
.fuse_hidden*
.directory
.Trash-*
.nfs*
CMakeLists.txt.user
CMakeCache.txt
CMakeFiles
CMakeScripts
Testing
Makefile
cmake_install.cmake
install_manifest.txt
compile_commands.json
CTestTestfile.cmake
_deps
CMakeUserPresets.json
Thumbs.db
Thumbs.db:encryptable
ehthumbs.db
ehthumbs_vista.db
*.stackdump
[Dd]esktop.ini
$RECYCLE.BIN/
*.cab
*.msi
*.msix
*.msm
*.msp
*.lnk
.DS_Store
.AppleDouble
.LSOverride
Icon
._*
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
borded_cpp/CMakeLists.txt
+27
View File
@@ -0,0 +1,27 @@
cmake_minimum_required(VERSION 3.25)
project(isspam)
set(CMAKE_CXX_STANDARD 26)
if (MSVC)
add_compile_options(/W4)
add_compile_options(/WX)
add_compile_options(/external:anglebrackets)
add_compile_options(/external:W0)
add_compile_options(/wd4100)
add_compile_options(/wd5050)
add_definitions(-DWIN32_LEAN_AND_MEAN -DVC_EXTRALEAN)
add_compile_definitions(WIN32_LEAN_AND_MEAN NOMINMAX)
else ()
add_compile_options(-Wall)
add_compile_options(-Wextra)
add_compile_options(-Wpedantic)
# add_compile_options(-Werror)
endif ()
add_executable(${PROJECT_NAME} src/main.cpp)
add_executable(${PROJECT_NAME}3 src/main3.cpp)
if (LINUX)
target_link_libraries(${PROJECT_NAME} tbb)
target_link_libraries(${PROJECT_NAME}3 tbb)
endif ()
borded_cpp/Dockerfile
+3
View File
@@ -0,0 +1,3 @@
FROM gcc:latest
RUN apt update && apt install -y cmake gdb
WORKDIR /home
borded_cpp/compose.yml
+9
View File
@@ -0,0 +1,9 @@
services:
cpp:
build: .
command: ["sh","doit.sh"]
tty: true
stdin_open: true
volumes:
- ./:/home
- ../books:/books
borded_cpp/doit.sh
Executable
+2
View File
@@ -0,0 +1,2 @@
rm -rf build | true
mkdir build && cd build && cmake .. && make
borded_cpp/src/main.cpp
+221
View File
@@ -0,0 +1,221 @@
#include <string>
#include <string_view>
#include <fstream>
#include <algorithm>
#include <iostream>
#include <numeric>
#include <execution>
#include <format>
#include <codecvt>
#include <ranges>
#ifdef __cpp_lib_print
#include <print>
#else
namespace std {
template <typename T, typename... Args>
inline void print(T format, Args &&... args) {
auto f = std::vformat(format, std::make_format_args(args...));
std::cout << f;
}
template <typename T, typename... Args>
inline void println(T format, Args &&... args) {
auto f = std::vformat(format, std::make_format_args(args...));
std::cout << f << std::endl;
}
}
#endif
constexpr std::array<std::wstring_view, 35> BAD_WORDS = {
L"recovery",
L"techie",
L"http",
L"https",
L"digital",
L"hack",
L"::",
L"//",
L"com",
L"@",
L"crypto",
L"bitcoin",
L"wallet",
L"hacker",
L"welcome",
L"whatsapp",
L"email",
L"cryptocurrency",
L"stolen",
L"freeze",
L"quick",
L"crucial",
L"tracing",
L"scammers",
L"expers",
L"hire",
L"century",
L"transaction",
L"essential",
L"managing",
L"contact",
L"contacting",
L"understanding",
L"assets",
L"funds",
};
constexpr auto SHORTEST_BAD_WORD = std::ranges::fold_left(BAD_WORDS, std::numeric_limits<std::size_t>::max(),
[](std::size_t current, const std::wstring_view &word) {
return std::min(current, word.size());
}
);
constexpr auto LONGEST_BAD_WORD = std::ranges::fold_left(BAD_WORDS, std::numeric_limits<std::size_t>::min(),
[](std::size_t current, const std::wstring_view &word) {
return std::max(current, word.size());
}
);
struct AnalysisResult {
std::size_t totalWordCount = 0;
std::size_t totalCapitalizedCount = 0;
std::size_t totalSentenceCount = 0;
std::size_t totalNumberCount = 0;
std::size_t totalForbiddenCount = 0;
std::size_t fileCount = 1;
std::size_t failCount = 0;
operator std::string() const {
return std::format(
"Word Count: {}\nCapitalized Count: {}\nSentence Count: {}\nNumber Count: {}\nForbidden Count: {}\nFile Count: {}\nFail Count: {}",
totalWordCount, totalCapitalizedCount, totalSentenceCount, totalNumberCount, totalForbiddenCount, fileCount, failCount
);
}
friend AnalysisResult operator+(const AnalysisResult &lhs, const AnalysisResult &rhs) {
return {
lhs.totalWordCount + rhs.totalWordCount,
lhs.totalCapitalizedCount + rhs.totalCapitalizedCount,
lhs.totalSentenceCount + rhs.totalSentenceCount,
lhs.totalNumberCount + rhs.totalNumberCount,
lhs.totalForbiddenCount + rhs.totalForbiddenCount,
lhs.fileCount + rhs.fileCount,
lhs.failCount + rhs.failCount
};
};
};
void check_word(std::wstring &word, std::size_t &forbiddenCount) {
if (word.size() < SHORTEST_BAD_WORD || word.size() > LONGEST_BAD_WORD) {
return;
}
std::ranges::transform(word, word.begin(), ::towlower);
if (std::ranges::find(BAD_WORDS, word) != BAD_WORDS.end()) {
forbiddenCount++;
}
// if (std::ranges::find_if(BAD_WORDS, [&word](const std::wstring_view &badWord) {
// return word.contains(badWord);
// }
// ) != BAD_WORDS.end()) {
// forbiddenCount++;
// }
}
AnalysisResult parseFile(const std::string_view &filename) {
std::wifstream file;
// surpress warning of deprecation
#pragma warning(push)
#pragma warning(suppress : 4996)
file.imbue(std::locale(std::locale(), new std::codecvt_utf8<wchar_t>));
#pragma warning(pop)
file.open(std::string(filename));
if (!file.is_open()) {
std::println("File doesn't exist: {}", filename);
return { };
}
AnalysisResult result{ };
bool inWord = false;
bool isDigit = false;
wchar_t c;
std::wstring word;
while (file.get(c)) {
if (c == '.') {
result.totalSentenceCount++;
}
if (std::isspace(c)) {
inWord = false;
isDigit = false;
if (!word.empty()) {
check_word(word, result.totalForbiddenCount);
word.clear();
}
continue;
} else {
if (!inWord) {
result.totalWordCount++;
if (std::isupper(c)) {
result.totalCapitalizedCount++;
}
}
inWord = true;
if (std::isdigit(c) && !isDigit) {
result.totalNumberCount++;
isDigit = true;
}
word.push_back(c);
}
};
// std::cout << "File state: " << file.rdstate() << " EOF" << file.eof() << " Fail" << file.fail() << " Bad" << file.bad() << std::endl;
if (!word.empty()) {
check_word(word, result.totalForbiddenCount);
}
file.close();
if (file.fail() && !file.eof()) {
result.failCount++;
}
return result;
}
int main(const int argc, char *argv[]) {
if (argc < 2) {
std::println("Usage: {} <file1> <file2> ... <fileN>", argv[0]);
return 1;
}
const AnalysisResult result = std::transform_reduce(std::execution::par_unseq, std::next(argv), argv + argc,
AnalysisResult{.fileCount = 0},
std::plus{ },
parseFile
);
double capitalizedPercentage = (result.totalWordCount > 0)
? static_cast<double>(result.totalCapitalizedCount) / result.totalWordCount * 100.0
: 0;
double forbiddenPercentage = (result.totalWordCount > 0)
? static_cast<double>(result.totalForbiddenCount) / result.totalWordCount * 100.0
: 0;
double wordCountPerSentence = (result.totalSentenceCount > 0)
? static_cast<double>(result.totalWordCount) / result.totalSentenceCount
: 0;
std::println("{}\nCapitalized Percentage: {}%\nForbidden Percentage: {}%\nWord Count Per Sentence: {}", std::string(result),
capitalizedPercentage, forbiddenPercentage, wordCountPerSentence
);
return 0;
}
borded_cpp/src/main2.cpp
+195
View File
@@ -0,0 +1,195 @@
#include <string>
#include <string_view>
#include <fstream>
#include <algorithm>
#include <iostream>
#include <numeric>
#include <execution>
#include <format>
#include <codecvt>
#include <ranges>
#ifdef __cpp_lib_print
#include <print>
#else
namespace std {
template <typename T, typename... Args>
inline void print(T format, Args &&... args) {
auto f = std::vformat(format, std::make_format_args(args...));
std::cout << f;
}
template <typename T, typename... Args>
inline void println(T format, Args &&... args) {
auto f = std::vformat(format, std::make_format_args(args...));
std::cout << f << std::endl;
}
}
#endif
constexpr std::array<std::wstring_view, 35> BAD_WORDS = {
L"recovery",
L"techie",
L"http",
L"https",
L"digital",
L"hack",
L"::",
L"//",
L"com",
L"@",
L"crypto",
L"bitcoin",
L"wallet",
L"hacker",
L"welcome",
L"whatsapp",
L"email",
L"cryptocurrency",
L"stolen",
L"freeze",
L"quick",
L"crucial",
L"tracing",
L"scammers",
L"expers",
L"hire",
L"century",
L"transaction",
L"essential",
L"managing",
L"contact",
L"contacting",
L"understanding",
L"assets",
L"funds",
};
constexpr auto SHORTEST_BAD_WORD = std::ranges::fold_left(BAD_WORDS, std::numeric_limits<std::size_t>::max(),
[](std::size_t current, const std::wstring_view &word) {
return std::min(current, word.size());
}
);
constexpr auto LONGEST_BAD_WORD = std::ranges::fold_left(BAD_WORDS, std::numeric_limits<std::size_t>::min(),
[](std::size_t current, const std::wstring_view &word) {
return std::max(current, word.size());
}
);
std::size_t totalWordCount = 0;
std::size_t totalCapitalizedCount = 0;
std::size_t totalSentenceCount = 0;
std::size_t totalNumberCount = 0;
std::size_t totalForbiddenCount = 0;
std::size_t fileCount = 1;
std::size_t failCount = 0;
void check_word(std::wstring &word, std::size_t &forbiddenCount) {
if (word.size() < SHORTEST_BAD_WORD || word.size() > LONGEST_BAD_WORD) {
return;
}
std::ranges::transform(word, word.begin(), ::towlower);
if (std::ranges::find(BAD_WORDS, word) != BAD_WORDS.end()) {
forbiddenCount++;
}
// if (std::ranges::find_if(BAD_WORDS, [&word](const std::wstring_view &badWord) {
// return word.contains(badWord);
// }
// ) != BAD_WORDS.end()) {
// forbiddenCount++;
// }
}
void parseFile(const std::string_view &filename) {
std::wifstream file;
// surpress warning of deprecation
#pragma warning(push)
#pragma warning(suppress : 4996)
file.imbue(std::locale(std::locale(), new std::codecvt_utf8<wchar_t>));
#pragma warning(pop)
file.open(std::string(filename));
if (!file.is_open()) {
std::println("File doesn't exist: {}", filename);
return;
}
bool inWord = false;
bool isDigit = false;
wchar_t c;
std::wstring word;
while (file.get(c)) {
if (c == '.') {
totalSentenceCount++;
}
if (std::isspace(c)) {
inWord = false;
isDigit = false;
if (!word.empty()) {
check_word(word, totalForbiddenCount);
word.clear();
}
continue;
} else {
if (!inWord) {
totalWordCount++;
if (std::isupper(c)) {
totalCapitalizedCount++;
}
}
inWord = true;
if (std::isdigit(c) && !isDigit) {
totalNumberCount++;
isDigit = true;
}
word.push_back(c);
}
};
// std::cout << "File state: " << file.rdstate() << " EOF" << file.eof() << " Fail" << file.fail() << " Bad" << file.bad() << std::endl;
if (!word.empty()) {
check_word(word, totalForbiddenCount);
}
file.close();
if (file.fail() && !file.eof()) {
failCount++;
}
}
int main(const int argc, char *argv[]) {
if (argc < 2) {
std::println("Usage: {} <file1> <file2> ... <fileN>", argv[0]);
return 1;
}
std::for_each(std::execution::par_unseq, std::next(argv), argv + argc, parseFile);
double capitalizedPercentage = (totalWordCount > 0)
? static_cast<double>(totalCapitalizedCount) / totalWordCount * 100.0
: 0;
double forbiddenPercentage = (totalWordCount > 0)
? static_cast<double>(totalForbiddenCount) / totalWordCount * 100.0
: 0;
double wordCountPerSentence = (totalSentenceCount > 0)
? static_cast<double>(totalWordCount) / totalSentenceCount
: 0;
std::println(
"Word Count: {}\nCapitalized Count: {}\nSentence Count: {}\nNumber Count: {}\nForbidden Count: {}\nFile Count: {}\nFail Count: {}\nCapitalized Percentage: {}%\nForbidden Percentage: {}%\nWord Count Per Sentence: {}",
totalWordCount, totalCapitalizedCount, totalSentenceCount, totalNumberCount, totalForbiddenCount, fileCount, failCount,
capitalizedPercentage, forbiddenPercentage, wordCountPerSentence
);
return 0;
}
borded_cpp/src/main3.cpp
+576
View File
@@ -0,0 +1,576 @@
#include <string>
#include <string_view>
#include <fstream>
#include <algorithm>
#include <iostream>
#include <execution>
#include <format>
#include <cstdio>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <aio.h>
#include <condition_variable>
#include <unordered_set>
#include <sys/signal.h>
#ifdef __cpp_lib_print
#include <print>
#else
namespace std {
template <typename T, typename... Args>
inline void print(T format, Args &&... args) {
auto f = std::vformat(format, std::make_format_args(args...));
std::cout << f;
}
template <typename T, typename... Args>
inline void println(T format, Args &&... args) {
auto f = std::vformat(format, std::make_format_args(args...));
std::cout << f << std::endl;
}
}
#endif
constexpr std::array<std::string_view, 35> BAD_WORDS = {
"recovery",
"techie",
"http",
"https",
"digital",
"hack",
"::",
"//",
"com",
"@",
"crypto",
"bitcoin",
"wallet",
"hacker",
"welcome",
"whatsapp",
"email",
"cryptocurrency",
"stolen",
"freeze",
"quick",
"crucial",
"tracing",
"scammers",
"expers",
"hire",
"century",
"transaction",
"essential",
"managing",
"contact",
"contacting",
"understanding",
"assets",
"funds",
};
const std::unordered_set<std::string_view> BAD_WORDS_SET(BAD_WORDS.begin(), BAD_WORDS.end());
static constexpr unsigned int crc_table[256] = {
0x00000000,
0x77073096,
0xee0e612c,
0x990951ba,
0x076dc419,
0x706af48f,
0xe963a535,
0x9e6495a3,
0x0edb8832,
0x79dcb8a4,
0xe0d5e91e,
0x97d2d988,
0x09b64c2b,
0x7eb17cbd,
0xe7b82d07,
0x90bf1d91,
0x1db71064,
0x6ab020f2,
0xf3b97148,
0x84be41de,
0x1adad47d,
0x6ddde4eb,
0xf4d4b551,
0x83d385c7,
0x136c9856,
0x646ba8c0,
0xfd62f97a,
0x8a65c9ec,
0x14015c4f,
0x63066cd9,
0xfa0f3d63,
0x8d080df5,
0x3b6e20c8,
0x4c69105e,
0xd56041e4,
0xa2677172,
0x3c03e4d1,
0x4b04d447,
0xd20d85fd,
0xa50ab56b,
0x35b5a8fa,
0x42b2986c,
0xdbbbc9d6,
0xacbcf940,
0x32d86ce3,
0x45df5c75,
0xdcd60dcf,
0xabd13d59,
0x26d930ac,
0x51de003a,
0xc8d75180,
0xbfd06116,
0x21b4f4b5,
0x56b3c423,
0xcfba9599,
0xb8bda50f,
0x2802b89e,
0x5f058808,
0xc60cd9b2,
0xb10be924,
0x2f6f7c87,
0x58684c11,
0xc1611dab,
0xb6662d3d,
0x76dc4190,
0x01db7106,
0x98d220bc,
0xefd5102a,
0x71b18589,
0x06b6b51f,
0x9fbfe4a5,
0xe8b8d433,
0x7807c9a2,
0x0f00f934,
0x9609a88e,
0xe10e9818,
0x7f6a0dbb,
0x086d3d2d,
0x91646c97,
0xe6635c01,
0x6b6b51f4,
0x1c6c6162,
0x856530d8,
0xf262004e,
0x6c0695ed,
0x1b01a57b,
0x8208f4c1,
0xf50fc457,
0x65b0d9c6,
0x12b7e950,
0x8bbeb8ea,
0xfcb9887c,
0x62dd1ddf,
0x15da2d49,
0x8cd37cf3,
0xfbd44c65,
0x4db26158,
0x3ab551ce,
0xa3bc0074,
0xd4bb30e2,
0x4adfa541,
0x3dd895d7,
0xa4d1c46d,
0xd3d6f4fb,
0x4369e96a,
0x346ed9fc,
0xad678846,
0xda60b8d0,
0x44042d73,
0x33031de5,
0xaa0a4c5f,
0xdd0d7cc9,
0x5005713c,
0x270241aa,
0xbe0b1010,
0xc90c2086,
0x5768b525,
0x206f85b3,
0xb966d409,
0xce61e49f,
0x5edef90e,
0x29d9c998,
0xb0d09822,
0xc7d7a8b4,
0x59b33d17,
0x2eb40d81,
0xb7bd5c3b,
0xc0ba6cad,
0xedb88320,
0x9abfb3b6,
0x03b6e20c,
0x74b1d29a,
0xead54739,
0x9dd277af,
0x04db2615,
0x73dc1683,
0xe3630b12,
0x94643b84,
0x0d6d6a3e,
0x7a6a5aa8,
0xe40ecf0b,
0x9309ff9d,
0x0a00ae27,
0x7d079eb1,
0xf00f9344,
0x8708a3d2,
0x1e01f268,
0x6906c2fe,
0xf762575d,
0x806567cb,
0x196c3671,
0x6e6b06e7,
0xfed41b76,
0x89d32be0,
0x10da7a5a,
0x67dd4acc,
0xf9b9df6f,
0x8ebeeff9,
0x17b7be43,
0x60b08ed5,
0xd6d6a3e8,
0xa1d1937e,
0x38d8c2c4,
0x4fdff252,
0xd1bb67f1,
0xa6bc5767,
0x3fb506dd,
0x48b2364b,
0xd80d2bda,
0xaf0a1b4c,
0x36034af6,
0x41047a60,
0xdf60efc3,
0xa867df55,
0x316e8eef,
0x4669be79,
0xcb61b38c,
0xbc66831a,
0x256fd2a0,
0x5268e236,
0xcc0c7795,
0xbb0b4703,
0x220216b9,
0x5505262f,
0xc5ba3bbe,
0xb2bd0b28,
0x2bb45a92,
0x5cb36a04,
0xc2d7ffa7,
0xb5d0cf31,
0x2cd99e8b,
0x5bdeae1d,
0x9b64c2b0,
0xec63f226,
0x756aa39c,
0x026d930a,
0x9c0906a9,
0xeb0e363f,
0x72076785,
0x05005713,
0x95bf4a82,
0xe2b87a14,
0x7bb12bae,
0x0cb61b38,
0x92d28e9b,
0xe5d5be0d,
0x7cdcefb7,
0x0bdbdf21,
0x86d3d2d4,
0xf1d4e242,
0x68ddb3f8,
0x1fda836e,
0x81be16cd,
0xf6b9265b,
0x6fb077e1,
0x18b74777,
0x88085ae6,
0xff0f6a70,
0x66063bca,
0x11010b5c,
0x8f659eff,
0xf862ae69,
0x616bffd3,
0x166ccf45,
0xa00ae278,
0xd70dd2ee,
0x4e048354,
0x3903b3c2,
0xa7672661,
0xd06016f7,
0x4969474d,
0x3e6e77db,
0xaed16a4a,
0xd9d65adc,
0x40df0b66,
0x37d83bf0,
0xa9bcae53,
0xdebb9ec5,
0x47b2cf7f,
0x30b5ffe9,
0xbdbdf21c,
0xcabac28a,
0x53b39330,
0x24b4a3a6,
0xbad03605,
0xcdd70693,
0x54de5729,
0x23d967bf,
0xb3667a2e,
0xc4614ab8,
0x5d681b02,
0x2a6f2b94,
0xb40bbe37,
0xc30c8ea1,
0x5a05df1b,
0x2d02ef8d
};
constexpr uint32_t crc32(std::string_view str) {
uint32_t crc = 0xffffffff;
for (auto c : str)
crc = (crc >> 8) ^ crc_table[(crc ^ c) & 0xff];
return crc ^ 0xffffffff;
}
constexpr uint32_t crc32(char const *str, const size_t size) {
uint32_t crc = 0xffffffff;
for (size_t i = 0; i < size; ++i)
crc = (crc >> 8) ^ crc_table[(crc ^ str[i]) & 0xff];
return crc ^ 0xffffffff;
}
constexpr std::array<uint32_t, 35> BAD_WORDS_HASH = {
crc32("recovery"),
crc32("techie"),
crc32("http"),
crc32("https"),
crc32("digital"),
crc32("hack"),
crc32("::"),
crc32("//"),
crc32("com"),
crc32("@"),
crc32("crypto"),
crc32("bitcoin"),
crc32("wallet"),
crc32("hacker"),
crc32("welcome"),
crc32("whatsapp"),
crc32("email"),
crc32("cryptocurrency"),
crc32("stolen"),
crc32("freeze"),
crc32("quick"),
crc32("crucial"),
crc32("tracing"),
crc32("scammers"),
crc32("expers"),
crc32("hire"),
crc32("century"),
crc32("transaction"),
crc32("essential"),
crc32("managing"),
crc32("contact"),
crc32("contacting"),
crc32("understanding"),
crc32("assets"),
crc32("funds")
};
const std::unordered_set BAD_WORDS_STR(BAD_WORDS.begin(), BAD_WORDS.end());
constexpr auto SHORTEST_BAD_WORD = std::ranges::fold_left(BAD_WORDS, std::numeric_limits<std::size_t>::max(),
[](std::size_t current, const std::string_view &word) {
return std::min(current, word.size());
}
);
constexpr auto LONGEST_BAD_WORD = std::ranges::fold_left(BAD_WORDS, std::numeric_limits<std::size_t>::min(),
[](std::size_t current, const std::string_view &word) {
return std::max(current, word.size());
}
);
int totalWordCount = 0;
int totalCapitalizedCount = 0;
int totalSentenceCount = 0;
int totalNumberCount = 0;
int totalForbiddenCount = 0;
int fileCount = 1;
int failCount = 0;
int done = 0;
struct info {
std::string_view name;
aiocb *cb;
const std::chrono::time_point<std::chrono::steady_clock> start = std::chrono::steady_clock::now();
};
constexpr void check_word_simple(const char *word, const ssize_t size) {
if (size < SHORTEST_BAD_WORD || size > LONGEST_BAD_WORD) {
return;
}
// if (BAD_WORDS_SET.contains(word)) {
// totalForbiddenCount++;
// }
const auto hs = crc32(word, size);
for (int i = 0; i < BAD_WORDS_HASH.size(); ++i) {
if (BAD_WORDS_HASH[i] == hs) {
totalForbiddenCount++;
return;
}
}
}
void read_str(char *str, ssize_t size) {
int mark = -1;
int fileWords = 0;
for (int pos = 0; pos <= size; ++pos) {
char *c = str + pos;
if (*c == '.') {
totalSentenceCount++;
}
if (*c == ' ' || *c == '\n' || *c == '\r' || *c == '\t') {
if (mark != -1) {
check_word_simple(str + mark, pos - mark);
mark = -1;
}
} else if (mark == -1) {
++fileWords;
if (*c >= 'A' && *c <= 'Z') {
totalCapitalizedCount++;
}
mark = pos;
} else if (*c >= '0' && *c <= '9') {
totalNumberCount++;
for (; pos <= size; ++pos) {
c = str + pos;
if (*c == '.') {
totalSentenceCount++;
}
if (*c == ' ' || *c == '\n' || *c == '\r' || *c == '\t') {
break;
}
}
mark = -1;
}
}
if (mark != -1) {
check_word_simple(str + mark, size - mark);
}
totalWordCount += fileWords;
}
void aio_completion_handler(sigval_t sigval) {
fileCount++;
info *data = (info *)sigval.sival_ptr;
auto req = data->cb;
// auto req = (struct aiocb *)sigval.sival_ptr;
/* Did the request complete? */
auto error = aio_error(req);
if (error == 0) {
/* Request completed successfully, get the return status */
// const auto start{std::chrono::steady_clock::now()};
// const std::chrono::duration<double> start_seconds{start - (data->start)};
// std::println("File started {} in {}", data->name, start_seconds.count());
read_str((char *)req->aio_buf, aio_return(req));
// const auto finish{std::chrono::steady_clock::now()};
// const std::chrono::duration<double> elapsed_seconds{finish - (data->start)};
// std::println("File read {} in {}", data->name, elapsed_seconds.count());
} else {
std::println("Error at aio_error ({}): ", error);
failCount++;
}
--done;
}
int main(const int argc, char *argv[]) {
if (argc < 2) {
std::println("Usage: {} <file1> <file2> ... <fileN>", argv[0]);
return 1;
}
done = argc - 1;
// lio_listio
auto aiocb_list = (struct aiocb *)malloc(sizeof(struct aiocb) * (argc - 1));
auto aiocb_list_ptr = (struct aiocb **)malloc(sizeof(struct aiocb *) * (argc - 1));
// char *memchnk = (char *)malloc(5 * 1024 * 1024 * (argc - 1));
for (std::size_t i = 0; i < argc - 1; i++) {
aiocb_list[i].aio_fildes = open(argv[i + 1], O_RDONLY);
aiocb_list[i].aio_offset = 0;
// 5mb
aiocb_list[i].aio_buf = malloc(5 * 1024 * 1024);
aiocb_list[i].aio_nbytes = (5 * 1024 * 1024);;
aiocb_list[i].aio_sigevent.sigev_notify = SIGEV_THREAD;
aiocb_list[i].aio_sigevent.sigev_notify_function = aio_completion_handler;
aiocb_list[i].aio_sigevent.sigev_notify_attributes = nullptr;
// aiocb_list[i].aio_sigevent.sigev_value.sival_ptr = &aiocb_list[i];
aiocb_list[i].aio_sigevent.sigev_value.sival_ptr = new info{
argv[i + 1],
&aiocb_list[i]};
// aiocb_list[i].aio_reqprio = SIGRTMIN;
aiocb_list_ptr[i] = &aiocb_list[i];
}
lio_listio(LIO_WAIT, aiocb_list_ptr, argc - 1, nullptr);
while (done > 0) {
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
std::println("Done reading files, {} done", done);
double capitalizedPercentage = (totalWordCount > 0)
? static_cast<double>(totalCapitalizedCount) / totalWordCount * 100.0
: 0;
double forbiddenPercentage = (totalWordCount > 0)
? static_cast<double>(totalForbiddenCount) / totalWordCount * 100.0
: 0;
double wordCountPerSentence = (totalSentenceCount > 0)
? static_cast<double>(totalWordCount) / totalSentenceCount
: 0;
std::println(
"Word Count: {}\nCapitalized Count: {}\nSentence Count: {}\nNumber Count: {}\nForbidden Count: {}\nFile Count: {}\nFail Count: {}\nCapitalized Percentage: {}%\nForbidden Percentage: {}%\nWord Count Per Sentence: {}",
totalWordCount, totalCapitalizedCount, totalSentenceCount, totalNumberCount, totalForbiddenCount, fileCount, failCount,
capitalizedPercentage, forbiddenPercentage, wordCountPerSentence
);
for (std::size_t i = 0; i < argc - 1; i++) {
close(aiocb_list[i].aio_fildes);
free((void *)aiocb_list[i].aio_buf);
}
free(aiocb_list);
free(aiocb_list_ptr);
// free(memchnk);
if (failCount > 0) {
return 1;
}
}
isspam
BIN
View File
Binary file not shown.
jest_rust/.gitignore
+3
View File
@@ -0,0 +1,3 @@
/target
/Cargo.lock
/test_books
jest_rust/Cargo.toml
+15
View File
@@ -0,0 +1,15 @@
[package]
name = "jisspam"
version = "0.1.0"
edition = "2024"
[dependencies]
fxhash = "0.2.1"
tokio = { version = "1.44.1", features = ["full"] }
[profile.release]
codegen-units = 1 # less means more compile work but better optimized
lto = "fat" # thin has best performance. fat the worst
strip = true
# opt-level = "z" # slows down
panic = "abort"
jest_rust/README.md
+84
View File
@@ -0,0 +1,84 @@
for https://retoor.molodetz.nl/retoor/isspam
extract `../books.tar.gz`
# local machine benchmarks
single threaded: `33.63373279571533`
rayon: `4.294418811798096`
tokio: `4.717588901519775`
tokio:
muncher: `2486ms`
for_loops: `1227ms`
for_loops_forbidden_only: `987ms`
trie creation and stats accumulation take 0ms
FxHashMap faster than BTreeMap
## compile options benchmarks
`lto` thin, fat doesn't change much
`codegen-units` 0, 1 doesn't change much
`opt-level = "z"` slow things down
# ubuntu terminal running
https://snek.molodetz.nl/terminal.html ubuntu running thing instructions:
```
mkdir /project
cd /project
git clone https://retoor.molodetz.nl/retoor/isspam.git
apt install valgrind curl
export RUSTUP_HOME=/project/.rustup
export CARGO_HOME=/project/.cargo
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "/project/.cargo/env"
cd isspam
rustup install nightly
rustup default nightly
make
make benchmark
python3 bench.py
```
clone: `git clone https://gitlab.com/jestdotty-group/draft/jisspam.git jest_rust`
edit make: `vi makefile` and add build:
```
build_jest:
@echo "compiling jest_rust project"
cd jest_rust && cargo build --release && cp target/release/jisspam ..
```
append to all script:
```
all: build run valgrind build_risspam run_risspam build_cpp build_borded_cpp build_py build_jest
```
add to bench: `vi bench.py`
```py
time_start = time.time()
subprocess.check_output('./jisspam books/*.txt', shell=True)
print("Time Jest Rust:", time.time() - time_start)
```
run: `python3 bench.py`
output looks something like this:
```
***benchmarking***
Time C: 31.315868377685547
Time Rust: 41.232205867767334
Time CPP: 20.1683189868927
Time Borded CPP: 15.468477964401245
Time Jest Rust: 54.74523115158081
Time Retoor Python: 287.63036131858826
***end benchmark***
```
add `/jisspam` to `.gitignore` to not commit the executable accidentally
jest_rust/src/main.rs
+122
View File
@@ -0,0 +1,122 @@
mod parser;
mod stats;
mod trie;
use stats::Stats;
use std::{env, fs, sync::LazyLock};
use tokio::sync::mpsc;
use trie::Trie;
static FORBIDDEN_WORDS: LazyLock<Trie> = LazyLock::new(|| {
let mut trie = Trie::default();
for word in [
"recovery",
"techie",
"http",
"https",
"digital",
"hack",
"::",
"//",
"@",
"com",
"crypto",
"bitcoin",
"wallet",
"hacker",
"welcome",
"whatsapp",
"email",
"cryptocurrency",
"stolen",
"freeze",
"quick",
"crucial",
"tracing",
"scammers",
"expers",
"hire",
"century",
"transaction",
"essential",
"managing",
"contact",
"contacting",
"understanding",
"assets",
"funds",
] {
trie.insert(word);
}
trie
});
#[tokio::main]
async fn main() {
let files = env::args().skip(1);
let mut rx = {
let (tx, rx) = mpsc::unbounded_channel();
for file in files {
let tx = tx.clone();
tokio::spawn(async move {
let mut stats = Stats::default();
//reading files in threads doesn't change speed of any sort but oh well
if let Ok(text) = fs::read_to_string(&file) {
stats.file_count += 1;
parser::for_loops::parse(&mut stats, &text);
} else {
stats.failed_file_count += 1;
}
let _ = tx.send(stats);
});
}
rx
};
let mut stats = Stats::default();
while let Some(file_stat) = rx.recv().await {
stats += file_stat;
}
println!("{stats}");
}
/// needs ../books.tar.gz to be extracted into ../books
#[test]
fn test() {
use std::{env, fs, process::Command, time::Instant};
println!("cwd: {}", env::current_dir().unwrap().display());
//compile
let mut compile = Command::new("cargo");
let compile_arged = compile.arg("build").arg("--release");
match compile_arged.output() {
Ok(output) => println!("compiled {}", String::from_utf8_lossy(&output.stdout)),
Err(err) => eprintln!("compile failed: {err}"),
}
//get test files
let files = fs::read_dir("../books")
.unwrap()
.map(|f| {
f.unwrap()
.path()
.canonicalize()
.unwrap()
.to_str()
.unwrap()
.to_string()
})
.collect::<Vec<_>>();
println!("test files found: {}", files.len());
println!();
//benchmark run
let benchmark = Instant::now();
let mut run = Command::new("target/release/jisspam");
let run_arged = run.args(files);
match run_arged.output() {
Ok(output) => println!("{}", String::from_utf8_lossy(&output.stdout)),
Err(err) => eprintln!("run failed: {err}"),
}
println!("benchmark: {}ms", benchmark.elapsed().as_millis());
}
jest_rust/src/parser/for_loops.rs
+35
View File
@@ -0,0 +1,35 @@
use crate::{FORBIDDEN_WORDS, stats::Stats};
#[allow(dead_code)]
pub fn parse(stats: &mut Stats, text: &str) {
for sentence in text
.split('.')
.map(|s| s.trim())
.filter(|s| !s.is_empty())
{
stats.sentence_count += 1;
for word in sentence
.split_ascii_whitespace()
.map(|s| s.trim())
.filter(|s| !s.is_empty())
{
stats.word_count += 1;
//get all numbers counted
let mut all_capitalized = true;
for char in word.chars() {
if char.is_numeric() {
stats.numeric_count += 1;
all_capitalized = false;
} else if !char.is_ascii_uppercase() {
all_capitalized = false;
}
}
if all_capitalized {
stats.capitalized_count += 1;
}
if FORBIDDEN_WORDS.contains(&word.to_lowercase()) {
stats.forbidden_count += 1;
}
}
}
}
jest_rust/src/parser/for_loops_forbidden_only.rs
+14
View File
@@ -0,0 +1,14 @@
use crate::{FORBIDDEN_WORDS, stats::Stats};
#[allow(dead_code)]
pub fn parse(stats: &mut Stats, text: &str) {
for word in text
.split_ascii_whitespace()
.map(|s| s.trim())
.filter(|s| !s.is_empty())
{
if FORBIDDEN_WORDS.contains(&word.to_lowercase()) {
stats.forbidden_count += 1;
}
}
}
jest_rust/src/parser/mod.rs
+3
View File
@@ -0,0 +1,3 @@
pub mod for_loops;
pub mod for_loops_forbidden_only;
pub mod muncher;
jest_rust/src/parser/muncher.rs
+63
View File
@@ -0,0 +1,63 @@
use crate::{FORBIDDEN_WORDS, stats::Stats};
#[allow(dead_code)]
/// probably buggy. for example, are new lines sentences? what if the text has no last period?
pub fn parse(stats: &mut Stats, text: &str) {
let mut capitalized = true;
let mut whitespaced = false;
let mut dotted = false;
let mut word = String::new();
for char in text.chars() {
if whitespaced {
if !char.is_whitespace() {
whitespaced = false; //end whiteness
}
continue;
} else if char.is_whitespace() {
whitespaced = true;
stats.word_count += 1; //end of word
if capitalized {
stats.capitalized_count += 1;
} else {
//reset capitalized word
capitalized = true;
}
let lowercase_word = word.to_lowercase();
if FORBIDDEN_WORDS.contains(&lowercase_word) {
stats.forbidden_count += 1;
}
word = String::new();
continue;
}
if dotted {
if char != '.' {
dotted = false; //end sentencing
}
continue;
} else if char == '.' {
dotted = true;
stats.sentence_count += 1;
stats.word_count += 1; //end of word
if capitalized {
stats.capitalized_count += 1;
} else {
//reset capitalized word
capitalized = true;
}
let lowercase_word = word.to_lowercase();
if FORBIDDEN_WORDS.contains(&lowercase_word) {
stats.forbidden_count += 1;
}
word = String::new();
continue;
}
word += &char.to_string();
if char.is_numeric() {
stats.numeric_count += 1;
capitalized = false;
}
if !char.is_ascii_uppercase() {
capitalized = false;
}
}
}
jest_rust/src/stats.rs
+58
View File
@@ -0,0 +1,58 @@
use std::{fmt::Display, ops::AddAssign};
#[derive(Debug, Default)]
pub struct Stats {
pub file_count: u32,
pub failed_file_count: u32,
pub sentence_count: u32,
pub word_count: u32,
pub capitalized_count: u32,
pub numeric_count: u32,
pub forbidden_count: u32,
}
impl AddAssign for Stats {
fn add_assign(&mut self, rhs: Self) {
self.file_count += rhs.file_count;
self.failed_file_count += rhs.failed_file_count;
self.sentence_count += rhs.sentence_count;
self.word_count += rhs.word_count;
self.capitalized_count += rhs.capitalized_count;
self.numeric_count += rhs.numeric_count;
self.forbidden_count += rhs.forbidden_count;
}
}
impl Display for Stats {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
writeln!(f, "file count: {}", self.file_count)?;
writeln!(f, "failed file count: {}", self.failed_file_count)?;
writeln!(f, "sentence count: {}", self.sentence_count)?;
writeln!(f, "word count: {}", self.word_count)?;
writeln!(f, "capitalized count: {}", self.capitalized_count)?;
writeln!(f, "numeric count: {}", self.numeric_count)?;
writeln!(f, "forbidden count: {}", self.forbidden_count)?;
let word_count = self.word_count as f32;
writeln!(
f,
"words per sentence average: {:.1}",
word_count / self.sentence_count as f32
)?;
writeln!(
f,
"forbidden word percentage: {:.2}%",
(self.forbidden_count as f32 / word_count) * 100.0,
)?;
write!(
f,
"capitalized word percentage: {:.2}%",
(self.capitalized_count as f32 / word_count) * 100.0,
)
}
}
jest_rust/src/trie.rs
+33
View File
@@ -0,0 +1,33 @@
use fxhash::FxBuildHasher;
use std::collections::HashMap;
type FxHashMap<K, V> = HashMap<K, V, FxBuildHasher>; //simpler, slightly faster
#[derive(Default, Debug, Clone)]
struct Node {
end: bool,
children: FxHashMap<char, Node>,
}
#[derive(Default, Debug, Clone)]
pub struct Trie {
root: Node,
}
impl Trie {
pub fn insert(&mut self, word: &str) {
let mut node = &mut self.root;
for char in word.chars() {
node = node.children.entry(char).or_default();
}
node.end = true;
}
pub fn contains(&self, word: &str) -> bool {
let mut current_node = &self.root;
for char in word.chars() {
match current_node.children.get(&char) {
Some(node) => current_node = node,
None => return false,
}
}
current_node.end
}
}
retoor_c/isspam.c
+245
View File
@@ -0,0 +1,245 @@
// retoor <retoor@molodetz.nl>
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <pthread.h>
#include <unistd.h>
#include <immintrin.h>
#include <stdint.h>
#define MAX_THREADS 16
typedef struct {
uint64_t wc, cc, sc, nc, fc;
} Stats;
typedef struct {
char* path;
Stats result;
} FileTask;
typedef struct {
FileTask* tasks;
int next_task;
int total_tasks;
pthread_mutex_t mutex;
} WorkQueue;
static uint8_t is_ws[256];
static uint8_t is_upper[256];
static uint16_t fw_len_bits[256];
static __uint128_t fw_words[256][8];
static uint8_t fw_counts[256];
static void init_tables(void) {
static const char* forbidden[] = {
"recovery", "techie", "http", "https", "digital", "hack", "::", "//", "com",
"@", "crypto", "bitcoin", "wallet", "hacker", "welcome", "whatsapp", "email",
"cryptocurrency", "stolen", "freeze", "quick", "crucial", "tracing", "scammers",
"expers", "hire", "century", "transaction", "essential", "managing", "contact",
"contacting", "understanding", "assets", "funds"
};
for (int i = 0; i < 256; i++) {
is_ws[i] = (i == ' ' || i == '\t' || i == '\n' || i == '\r' || i == '\f');
is_upper[i] = (i >= 'A' && i <= 'Z');
fw_len_bits[i] = 0;
fw_counts[i] = 0;
}
for (int i = 0; i < 35; i++) {
int len = strlen(forbidden[i]);
uint8_t first = (uint8_t)forbidden[i][0];
fw_len_bits[first] |= (1u << len);
__uint128_t val = 0;
memcpy(&val, forbidden[i], len);
fw_words[first][fw_counts[first]++] = val;
}
}
static const __uint128_t len_masks[17] = {
0, 0xFFULL, 0xFFFFULL, 0xFFFFFFULL, 0xFFFFFFFFULL,
0xFFFFFFFFFFULL, 0xFFFFFFFFFFFFULL, 0xFFFFFFFFFFFFFFULL, 0xFFFFFFFFFFFFFFFFULL,
((__uint128_t)0xFF << 64) | 0xFFFFFFFFFFFFFFFFULL,
((__uint128_t)0xFFFF << 64) | 0xFFFFFFFFFFFFFFFFULL,
((__uint128_t)0xFFFFFF << 64) | 0xFFFFFFFFFFFFFFFFULL,
((__uint128_t)0xFFFFFFFF << 64) | 0xFFFFFFFFFFFFFFFFULL,
((__uint128_t)0xFFFFFFFFFF << 64) | 0xFFFFFFFFFFFFFFFFULL,
((__uint128_t)0xFFFFFFFFFFFF << 64) | 0xFFFFFFFFFFFFFFFFULL,
((__uint128_t)0xFFFFFFFFFFFFFF << 64) | 0xFFFFFFFFFFFFFFFFULL,
(__uint128_t)-1
};
static inline __attribute__((always_inline, hot)) int is_forbidden(const uint8_t* word, size_t len) {
if (__builtin_expect(len > 14, 0)) return 0;
uint8_t first = word[0];
uint16_t bits = fw_len_bits[first];
if (__builtin_expect((bits & (1u << len)) == 0, 1)) return 0;
__uint128_t w = 0;
memcpy(&w, word, len);
w &= len_masks[len];
__uint128_t* fwords = fw_words[first];
int cnt = fw_counts[first];
for (int i = 0; i < cnt; i++) {
if (fwords[i] == w) return 1;
}
return 0;
}
static void process_file(FileTask* task) {
int fd = open(task->path, O_RDONLY);
if (fd < 0) return;
struct stat st;
if (fstat(fd, &st) < 0 || st.st_size == 0) {
close(fd);
return;
}
size_t size = st.st_size;
uint8_t* data = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
close(fd);
if (data == MAP_FAILED) return;
madvise(data, size, MADV_SEQUENTIAL | MADV_WILLNEED);
uint64_t wc = 0, cc = 0, sc = 0, nc = 0, fc = 0;
const __m256i dot_vec = _mm256_set1_epi8('.');
const __m256i zero_vec = _mm256_set1_epi8('0' - 1);
const __m256i nine_vec = _mm256_set1_epi8('9' + 1);
size_t i = 0;
size_t simd_end = size & ~127ULL;
while (i < simd_end) {
__builtin_prefetch(data + i + 512, 0, 0);
__m256i v0 = _mm256_loadu_si256((__m256i*)(data + i));
__m256i v1 = _mm256_loadu_si256((__m256i*)(data + i + 32));
__m256i v2 = _mm256_loadu_si256((__m256i*)(data + i + 64));
__m256i v3 = _mm256_loadu_si256((__m256i*)(data + i + 96));
sc += __builtin_popcount(_mm256_movemask_epi8(_mm256_cmpeq_epi8(v0, dot_vec)));
sc += __builtin_popcount(_mm256_movemask_epi8(_mm256_cmpeq_epi8(v1, dot_vec)));
sc += __builtin_popcount(_mm256_movemask_epi8(_mm256_cmpeq_epi8(v2, dot_vec)));
sc += __builtin_popcount(_mm256_movemask_epi8(_mm256_cmpeq_epi8(v3, dot_vec)));
nc += __builtin_popcount(_mm256_movemask_epi8(_mm256_and_si256(_mm256_cmpgt_epi8(v0, zero_vec), _mm256_cmpgt_epi8(nine_vec, v0))));
nc += __builtin_popcount(_mm256_movemask_epi8(_mm256_and_si256(_mm256_cmpgt_epi8(v1, zero_vec), _mm256_cmpgt_epi8(nine_vec, v1))));
nc += __builtin_popcount(_mm256_movemask_epi8(_mm256_and_si256(_mm256_cmpgt_epi8(v2, zero_vec), _mm256_cmpgt_epi8(nine_vec, v2))));
nc += __builtin_popcount(_mm256_movemask_epi8(_mm256_and_si256(_mm256_cmpgt_epi8(v3, zero_vec), _mm256_cmpgt_epi8(nine_vec, v3))));
i += 128;
}
while (i < size) {
uint8_t c = data[i];
sc += (c == '.');
nc += (c >= '0' && c <= '9');
i++;
}
i = 0;
while (i < size) {
while (i < size && is_ws[data[i]]) i++;
if (i >= size) break;
size_t word_start = i;
int all_upper = 1;
while (i < size && !is_ws[data[i]]) {
all_upper &= is_upper[data[i]];
i++;
}
wc++;
cc += all_upper;
size_t wlen = i - word_start;
if (wlen <= 14) {
fc += is_forbidden(data + word_start, wlen);
}
}
task->result.wc = wc;
task->result.cc = cc;
task->result.sc = sc;
task->result.nc = nc;
task->result.fc = fc;
munmap(data, size);
}
static void* worker(void* arg) {
WorkQueue* q = (WorkQueue*)arg;
while (1) {
pthread_mutex_lock(&q->mutex);
int task_id = q->next_task++;
pthread_mutex_unlock(&q->mutex);
if (task_id >= q->total_tasks) break;
process_file(&q->tasks[task_id]);
}
return NULL;
}
int main(int argc, char* argv[]) {
if (argc < 2) {
fprintf(stderr, "Usage: %s <file1> [file2] ...\n", argv[0]);
return 1;
}
init_tables();
int num_files = argc - 1;
FileTask* tasks = calloc(num_files, sizeof(FileTask));
if (!tasks) return 1;
for (int i = 0; i < num_files; i++) {
tasks[i].path = argv[i + 1];
}
WorkQueue queue = {
.tasks = tasks,
.next_task = 0,
.total_tasks = num_files,
.mutex = PTHREAD_MUTEX_INITIALIZER
};
int nthreads = num_files < MAX_THREADS ? num_files : MAX_THREADS;
pthread_t threads[MAX_THREADS];
for (int i = 0; i < nthreads; i++) {
pthread_create(&threads[i], NULL, worker, &queue);
}
for (int i = 0; i < nthreads; i++) {
pthread_join(threads[i], NULL);
}
unsigned long long twc = 0, tcc = 0, tsc = 0, tnc = 0, tfc = 0;
for (int i = 0; i < num_files; i++) {
twc += tasks[i].result.wc;
tcc += tasks[i].result.cc;
tsc += tasks[i].result.sc;
tnc += tasks[i].result.nc;
tfc += tasks[i].result.fc;
}
double cc_pct = twc > 0 ? (double)tcc / twc * 100.0 : 0;
double fc_pct = twc > 0 ? (double)tfc / twc * 100.0 : 0;
double wps = tsc > 0 ? (double)twc / tsc : 0;
printf("\nTotal Words: %llu\n", twc);
printf("Total Capitalized words: %llu\n", tcc);
printf("Total Sentences: %llu\n", tsc);
printf("Total Numbers: %llu\n", tnc);
printf("Total Forbidden words: %llu\n", tfc);
printf("Capitalized percentage: %.6f%%\n", cc_pct);
printf("Forbidden percentage: %.6f%%\n", fc_pct);
printf("Word count per sentence: %.6f\n", wps);
printf("Total files read: %d\n", num_files);
free(tasks);
pthread_mutex_destroy(&queue.mutex);
return 0;
}
isspam.c → retoor_c/isspam.c.bak
+124 -182
View File
@@ -4,8 +4,8 @@
#include <string.h>
#include <unistd.h>
#include "rstring_list.h"
#include "rstr.h"
#include "rstring_list.h"
#include <ctype.h>
#define sl rstring_list_t
@@ -25,24 +25,16 @@ char *forbidden_words[] = {
"transaction", "essential", "managing", "contact", "contacting", "understanding", "assets", "funds", NULL};
bool show_capitalized = false;
bool show_sentences = false;
bool show_words = false;
bool show_numbers = false;
bool show_forbidden_words = true;
bool file_exists(char * path){
FILE * f = fopen(path, "r");
bool result = f != NULL;
if(f){
fclose(f);
bool stricmp(char *word1, char *word2) {
while (*word1 && tolower(*word1) == tolower(*word2)) {
word1++;
word2++;
}
return result;
return *word1 == *word2;
}
void sld(sl *lst) {
for (ulonglong i = 0; i < lst->count; i++) {
printf("<%llu:%s>\n", i, lst->strings[i]);
@@ -65,6 +57,7 @@ char *remove_preserved_chars(char *content) {
}
return cc;
}
//Memory usage: 29 TB, 213.322.618 (re)allocated, 106.670.251 unqiue free'd, 0 in use.
char *slds(sl *lst) {
str_t *buffer = strn(1337);
@@ -81,20 +74,6 @@ char *slds(sl *lst) {
bool isws(char c) { return c == '\t' || c == '\n' || c == ' ' || c == ','; }
char *stripws(char *content) {
char *cc = (char *)malloc(strlen(content) + 1);
*cc = 0;
char *ccp = cc;
while (*content) {
if (!isws(*content)) {
*ccp = *content;
ccp++;
*ccp = 0;
}
content++;
}
return cc;
}
char *fread_till_eof(FILE *f) {
char c;
@@ -106,12 +85,10 @@ char *fread_till_eof(FILE *f) {
return content;
}
rstring_list_t *get_sentences(char *content) {
rstring_list_t *sentences = rstring_list_new();
int get_sentences(char *content) {
int count = 0;
char *sentence_buffer = (char *)malloc(strlen(content) + 1);
char *sentence_buffer_p = sentence_buffer;
// rbuffer_t * buffer = rbuffer_new(NULL,0);
bool in_line = false;
while (*content) {
if ((*content == ' ' || *content == '\t' || *content == '\n') && !in_line) {
@@ -124,7 +101,7 @@ rstring_list_t *get_sentences(char *content) {
*sentence_buffer_p = *content;
sentence_buffer_p++;
*sentence_buffer_p = 0;
rstring_list_add(sentences, sentence_buffer);
count++;
sentence_buffer_p = sentence_buffer;
*sentence_buffer = 0;
content++;
@@ -137,32 +114,55 @@ rstring_list_t *get_sentences(char *content) {
content++;
}
free(sentence_buffer);
return sentences;
return count;
}
rstring_list_t *get_words(char *content) {
rstring_list_t *words = rstring_list_new();
bool is_forbidden_word(char *word) {
for (int j = 0; forbidden_words[j] != NULL; j++) {
if (stricmp(word, forbidden_words[j])) {
return true;
}
}
return false;
}
int get_words(char *content, int * count_caps, int *fw_count) {
int count = 0;
char *word_buffer = (char *)malloc(strlen(content) + 1);
char *word_buffer_p = word_buffer;
*word_buffer_p = 0;
bool has_lcase = false;
// rbuffer_t * buffer = rbuffer_new(NULL,0);
while (*content) {
if (*content == ' ' || *content == '\t' || *content == '\n') {
if (word_buffer_p != word_buffer) {
rstring_list_add(words, word_buffer);
if(!has_lcase)
{
(*count_caps)++;
}
count++;
if(is_forbidden_word(word_buffer)){
(*fw_count)++;
}
word_buffer_p = word_buffer;
*word_buffer = 0;
}
has_lcase = false;
content++;
continue;
}
*word_buffer_p = *content;
if(islower(*content) == *content)
has_lcase = true;
word_buffer_p++;
*word_buffer_p = 0;
content++;
}
free(word_buffer);
return words;
return count;
}
bool is_fully_capitalized_word(char *word) {
@@ -174,31 +174,24 @@ bool is_fully_capitalized_word(char *word) {
return true;
}
sl *get_capitalized_words(char *content) {
sl *capitalized_words = sln();
sl *sentences = get_sentences(content);
for (uint j = 0; j < sentences->count; j++) {
char *sentence = sentences->strings[j];
sl *all_words = get_words(sentence);
// Always skip the first word since sentences start with
for (uint i = 0; i < all_words->count; i++) {
if (is_fully_capitalized_word(all_words->strings[i])) {
rstring_list_add(capitalized_words, all_words->strings[i]);
}
int get_capitalized_words(sl *all_words) {
int count = 0;
for (uint i = 0; i < all_words->count; i++) {
if (is_fully_capitalized_word(all_words->strings[i])) {
count++;
}
slf(all_words);
}
slf(sentences);
return capitalized_words;
return count;
}
char *clean_content(char *content) {
char *allowed_ichars = "01234567891abcdefghijklmnopqrstuvwxyz \n.,!?";
char *allowed_ichars = "01234567891abcdefghijklmnopqrstuvwxyz.,!?";
char *clean_content = (char *)malloc(strlen(content) + 1);
char *clean_content_p = clean_content;
*clean_content_p = 0;
while (*content) {
if (strchr(allowed_ichars, tolower(*content))) {
*clean_content_p = *content;
clean_content_p++;
@@ -209,182 +202,131 @@ char *clean_content(char *content) {
return clean_content;
}
sl *get_numbers(char *content) {
char *cc = clean_content(content);
char *ccc = stripws(cc);
int get_numbers(char *cc) {
int count = 0;
char *ccc = cc;
char *cccp = ccc;
free(cc);
char *number_buffer = (char *)malloc(strlen(ccc) + 1);
*number_buffer = 0;
char *number_buffer_p = number_buffer;
sl *numbers = sln();
while (*cccp) {
if (isdigit((*cccp))) {
*number_buffer_p = *cccp;
number_buffer_p++;
*number_buffer_p = 0;
} else if (number_buffer != number_buffer_p) {
sla(numbers, number_buffer);
count++;
*number_buffer = 0;
number_buffer_p = number_buffer;
}
cccp++;
}
free(number_buffer);
free(ccc);
return numbers;
return count;
}
bool stricmp(char *word1, char *word2) {
while (*word1 && tolower(*word1) == tolower(*word2)) {
word1++;
word2++;
}
return *word1 == *word2;
}
bool containswordi(sl *words, char *word) {
for (uint i = 0; i < words->count; i++) {
if (stricmp(words->strings[i], word))
return true;
}
return false;
}
sl *get_forbidden_words(char *content) {
sl *words = get_words(content);
sl *found = sln();
for (int j = 0; forbidden_words[j] != NULL; j++) {
if (containswordi(words, forbidden_words[j])) {
rstring_list_add(found, forbidden_words[j]);
}
}
slf(words);
return found;
}
unsigned int total = 0;
char *readall(FILE *f) {
if (fseek(f, 0, SEEK_END) != 0) {
fclose(f);
return NULL;
}
size_t file_size = ftell(f);
if (file_size == (size_t)-1L) {
fclose(f);
return NULL;
}
if (fseek(f, 0, SEEK_SET) != 0) {
fclose(f);
return NULL;
}
char *buffer = (char *)malloc(file_size + 1);
if (!buffer) {
fclose(f);
return NULL;
}
size_t bytes_read = fread(buffer, 1, file_size, f);
buffer[bytes_read] = 0;
return buffer;
}
void analyze(FILE *f) {
if(!f){
// File doesn't exist
return;
}
total = total + 1;
printf("#%u\n", total);
char *data = fread_till_eof(f);
str_t *all = strn(1337);
char *sbuf = NULL;
char *data = readall(f);
if(!data)
return;
char *clean_data = clean_content(data);
free(clean_data);
// All capitalized words
sl *capitalized_words = get_capitalized_words(data);
ulonglong capitalized_words_count = capitalized_words->count;
printf("Capitalized words: %llu\n", capitalized_words_count);
if(show_capitalized)
sld(capitalized_words);
sbuf = slds(capitalized_words);
stra(all, sbuf);
free(sbuf);
sl *sentences = get_sentences(data);
// All sentences
printf("Sentences: %llu\n", sentences->count);
if(show_sentences)
sld(sentences);
sbuf = slds(sentences);
stra(all, sbuf);
free(sbuf);
sl *words = get_words(data);
int capitalized_words = 0;
int fw = 0;
int words = get_words(data,&capitalized_words,&fw);
int sentences = get_sentences(data);
int numbers = get_numbers(clean_data);
// All words
printf("Words: %llu\n", words->count);
if(show_words)
sld(words);
sbuf = slds(words);
stra(all, sbuf);
free(sbuf);
printf("Words: %d\n", words);
// All capitalized words
printf("Capitalized words: %d\n", capitalized_words);
// All sentences
printf("Sentences: %i\n", sentences);
// Numbers
sl *numbers = get_numbers(data);
printf("Numbers: %llu\n", numbers->count);
if(show_numbers)
sld(numbers);
sbuf = slds(numbers);
stra(all, sbuf);
free(sbuf);
printf("Numbers: %d\n", numbers);
// Forbidden words
sl *fw = get_forbidden_words(data);
printf("Forbidden words: %llu\n", fw->count);
if(show_forbidden_words)
sld(fw);
sbuf = slds(fw);
stra(all, sbuf);
free(sbuf);
strd(all);
if(words->count){
double capitalized_word_percentage = 100 * ((double)capitalized_words->count / (double)words->count);
printf("Capitalized percentage: %f%%\n",capitalized_word_percentage);
double forbidden_word_percentage = 100 * ((double)fw->count / (double)words->count);
printf("Forbidden percentage: %f%%\n",forbidden_word_percentage);
ulonglong word_count_per_sentence = words->count / (sentences->count ? sentences->count : 1);
printf("Word count per sentence: %llu\n", word_count_per_sentence);
}
slf(capitalized_words);
slf(sentences);
slf(words);
slf(numbers);
slf(fw);
printf("Forbidden words: %d\n", fw);
if (words) {
double capitalized_word_percentage = 100 * ((double)capitalized_words / (double)words);
printf("Capitalized percentage: %f%%\n", capitalized_word_percentage);
double forbidden_word_percentage = 100 * ((double)fw / (double)words);
printf("Forbidden percentage: %f%%\n", forbidden_word_percentage);
ulonglong word_count_per_sentence = words / (sentences ? sentences : 1);
printf("Word count per sentence: %llu\n", word_count_per_sentence);
}
free(clean_data);
free(data);
}
void analyze_file(char *path) {
FILE *f = fopen(path, "r");
if(f){
analyze(f);
fclose(f);
}else{
printf("File doesn't exist: %s\n",path);
}
}
void * analyze_file_thread(void *path){
analyze_file((char *)path);
return NULL;
}
int main(int argc, char *argv[]) {
if (argc > 1) {
pthread_t *threads = (pthread_t *)malloc(argc * sizeof(pthread_t));
for (int i = 1; i < argc; i++) {
if(!strcmp(argv[1],"--hide-capitalized")){
show_capitalized=false;
}else if(!strcmp(argv[1],"--show-sentences")){
show_sentences=true;
}else if(!strcmp(argv[1],"--show-words")){
show_words=true;
}else if(!strcmp(argv[1],"--show-numbers")){
show_words=true;
}else if(!strcmp(argv[1],"--hide-forbidden-words")){
show_forbidden_words=false;
}else if(!strcmp(argv[1],"help") || !strcmp(argv[1],"--help")){
printf("%s",
"Usage: spam [file] [file] [file]\n"
"Flag defaults:\n"
" hide-capitalized = true\n"
" show-sentences = false\n"
" show-words = false\n"
" show-numbers = false\n"
" hide-forbidden-words = false\n");
return 0;
}
printf("File: %s\n", argv[i]);
analyze_file(argv[i]);
printf("%s\n", rmalloc_stats());
printf("\n");
pthread_create(&threads[i-1],NULL,analyze_file_thread,(void *)argv[i]);
}
for(int i = 1; i < argc; i++){
pthread_join(threads[i-1],NULL);
}
free(threads);
return 0;
}
analyze(stdin);
printf("%s\n", rmalloc_stats());
exit(0);
return 0;
}
retoor_c/isspam.cpp
+129
View File
@@ -0,0 +1,129 @@
// Author: retoor@molodetz.nl
#include <iostream>
#include <fstream>
#include <string>
#include <vector>
#include <thread>
#include <unordered_set>
#include <algorithm>
#include <sstream>
#define FORBIDDEN_WORDS_COUNT 40
const std::unordered_set<std::string> forbidden_words = {
"recovery", "techie", "http", "https", "digital", "hack", "::", "//", "com",
"@", "crypto", "bitcoin", "wallet", "hacker", "welcome", "whatsapp", "email", "cryptocurrency",
"stolen", "freeze", "quick", "crucial", "tracing", "scammers", "expers", "hire", "century",
"transaction", "essential", "managing", "contact", "contacting", "understanding", "assets", "funds",
};
struct AnalysisResult {
std::string filename;
long long total_word_count = 0;
long long total_capitalized_count = 0;
long long total_sentence_count = 0;
long long total_number_count = 0;
long long total_forbidden_count = 0;
};
std::string read_file(const std::string& filename) {
std::ifstream file(filename);
if (!file) {
std::cerr << "File doesn't exist: " << filename << std::endl;
return "";
}
std::ostringstream content;
content << file.rdbuf(); // Read the entire file into a string
return content.str();
}
void analyze_file(AnalysisResult& result) {
std::string text = read_file(result.filename);
if (!text.empty()) {
long long word_count = 0;
long long capitalized_count = 0;
long long sentence_count = 0;
long long number_count = 0;
long long forbidden_count = 0;
for (char c : text) {
if (c == '.') {
sentence_count++;
}
}
std::istringstream stream(text);
std::string token;
while (stream >> token) {
word_count++;
if (std::isupper(token[0])) {
capitalized_count++;
}
if (std::any_of(token.begin(), token.end(), ::isdigit)) {
number_count++;
}
if (forbidden_words.find(token) != forbidden_words.end()) {
forbidden_count++;
}
}
result.total_word_count = word_count;
result.total_capitalized_count = capitalized_count;
result.total_sentence_count = sentence_count;
result.total_number_count = number_count;
result.total_forbidden_count = forbidden_count;
}
}
int main(int argc, char *argv[]) {
if (argc < 2) {
std::cerr << "Usage: " << argv[0] << " <file1> <file2> ... <fileN>" << std::endl;
return 1;
}
std::vector<std::thread> threads;
std::vector<AnalysisResult> results(argc - 1);
for (int i = 1; i < argc; i++) {
results[i - 1].filename = argv[i];
threads.emplace_back(analyze_file, std::ref(results[i - 1]));
}
for (auto& thread : threads) {
thread.join();
}
long long total_word_count = 0;
long long total_capitalized_count = 0;
long long total_sentence_count = 0;
long long total_number_count = 0;
long long total_forbidden_count = 0;
for (const auto& result : results) {
total_word_count += result.total_word_count;
total_capitalized_count += result.total_capitalized_count;
total_sentence_count += result.total_sentence_count;
total_number_count += result.total_number_count;
total_forbidden_count += result.total_forbidden_count;
}
double capitalized_percentage = (total_word_count > 0) ? (static_cast<double>(total_capitalized_count) / total_word_count * 100.0) : 0;
double forbidden_percentage = (total_word_count > 0) ? (static_cast<double>(total_forbidden_count) / total_word_count * 100.0) : 0;
double word_count_per_sentence = (total_sentence_count > 0) ? (static_cast<double>(total_word_count) / total_sentence_count) : 0;
std::cout << "\nTotal Words: " << total_word_count << std::endl;
std::cout << "Total Capitalized words: " << total_capitalized_count << std::endl;
std::cout << "Total Sentences: " << total_sentence_count << std::endl;
std::cout << "Total Numbers: " << total_number_count << std::endl;
std::cout << "Total Forbidden words: " << total_forbidden_count << std::endl;
std::cout << "Capitalized percentage: " << capitalized_percentage << "%" << std::endl;
std::cout << "Forbidden percentage: " << forbidden_percentage << "%" << std::endl;
std::cout << "Word count per sentence: " << word_count_per_sentence << std::endl;
std::cout << "Total files read: " << (argc - 1) << std::endl;
return 0;
}
retoor_c/isspam.py
+85
View File
@@ -0,0 +1,85 @@
import os
import sys
import threading
from concurrent.futures import ThreadPoolExecutor
MAX_TEXT_LENGTH = 1024
FORBIDDEN_WORDS_COUNT = 40
forbidden_words = set([
"recovery", "techie", "http", "https", "digital", "hack", "::", "//", "com",
"@", "crypto", "bitcoin", "wallet", "hacker", "welcome", "whatsapp", "email", "cryptocurrency",
"stolen", "freeze", "quick", "crucial", "tracing", "scammers", "expers", "hire", "century",
"transaction", "essential", "managing", "contact", "contacting", "understanding", "assets", "funds",
])
class AnalysisResult:
def __init__(self, filename):
self.filename = filename
self.total_word_count = 0
self.total_capitalized_count = 0
self.total_sentence_count = 0
self.total_number_count = 0
self.total_forbidden_count = 0
def is_forbidden(word):
return word in forbidden_words
def read_file(filename):
if not os.path.exists(filename):
print(f"File doesn't exist: {filename}")
return None
with open(filename, 'r') as file:
return file.read()
def analyze_file(result):
text = read_file(result.filename)
if text:
result.total_sentence_count = text.count('.')
tokens = text.split()
result.total_word_count = len(tokens)
result.total_capitalized_count = sum(1 for token in tokens if token[0].isupper())
result.total_number_count = sum(1 for token in tokens if any(char.isdigit() for char in token))
result.total_forbidden_count = sum(1 for token in tokens if is_forbidden(token))
def main():
if len(sys.argv) < 2:
print(f"Usage: {sys.argv[0]} <file1> <file2> ... <fileN>")
return
results = []
with ThreadPoolExecutor() as executor:
futures = []
for filename in sys.argv[1:]:
result = AnalysisResult(filename)
results.append(result)
futures.append(executor.submit(analyze_file, result))
for future in futures:
future.result()
total_word_count = sum(result.total_word_count for result in results)
total_capitalized_count = sum(result.total_capitalized_count for result in results)
total_sentence_count = sum(result.total_sentence_count for result in results)
total_number_count = sum(result.total_number_count for result in results)
total_forbidden_count = sum(result.total_forbidden_count for result in results)
capitalized_percentage = (total_word_count > 0) * (total_capitalized_count / total_word_count * 100.0)
forbidden_percentage = (total_word_count > 0) * (total_forbidden_count / total_word_count * 100.0)
word_count_per_sentence = (total_sentence_count > 0) * (total_word_count / total_sentence_count)
print(f"\nTotal Words: {total_word_count}")
print(f"Total Capitalized words: {total_capitalized_count}")
print(f"Total Sentences: {total_sentence_count}")
print(f"Total Numbers: {total_number_count}")
print(f"Total Forbidden words: {total_forbidden_count}")
print(f"Capitalized percentage: {capitalized_percentage:.6f}%")
print(f"Forbidden percentage: {forbidden_percentage:.6f}%")
print(f"Word count per sentence: {word_count_per_sentence:.6f}")
print(f"Total files read: {len(sys.argv) - 1}")
if __name__ == "__main__":
main()
rmalloc.h → retoor_c/rmalloc.h
View File
rstr.h → retoor_c/rstr.h
+1 -3
View File
@@ -26,8 +26,6 @@ void stra(str_t *str, const char *to_append) {
if (required_new_length > str->size) {
str->size += required_new_length + str->buffer_size;
str->content = (char *)realloc(str->content, str->size + 1);
} else {
// printf("NO NDEED\n");
}
strcat(str->content, to_append);
str->content[str->length] = 0;
@@ -49,4 +47,4 @@ char *strc(str_t *str) {
return content;
}
#endif
#endif
rstring_list.h → retoor_c/rstring_list.h
View File
totals.py → retoor_c/totals.py
View File
risspam
BIN
View File
Binary file not shown.
swift_isspam/Package.swift
+16
View File
@@ -0,0 +1,16 @@
// swift-tools-version: 5.9
// retoor <retoor@molodetz.nl>
import PackageDescription
let package = Package(
name: "sisspam",
targets: [
.executableTarget(
name: "sisspam",
swiftSettings: [
.unsafeFlags(["-O", "-whole-module-optimization"])
]
)
]
)
swift_isspam/Sources/sisspam/main.swift
+227
View File
@@ -0,0 +1,227 @@
// retoor <retoor@molodetz.nl>
import Foundation
import Dispatch
struct Stats {
var words: UInt64 = 0
var capitalized: UInt64 = 0
var sentences: UInt64 = 0
var numbers: UInt64 = 0
var forbidden: UInt64 = 0
}
let charFlags: UnsafeMutablePointer<UInt8> = {
let t = UnsafeMutablePointer<UInt8>.allocate(capacity: 256)
t.initialize(repeating: 0, count: 256)
t[0x09] = 1; t[0x0A] = 1; t[0x0B] = 1; t[0x0C] = 1; t[0x0D] = 1; t[0x20] = 1
for i in 0x30...0x39 { t[i] |= 2 }
for i in 0x41...0x5A { t[i] |= 4 }
t[0x2E] = 8
return t
}()
let forbiddenWords: [String] = [
"recovery", "techie", "http", "https", "digital", "hack", "::", "//", "com",
"@", "crypto", "bitcoin", "wallet", "hacker", "welcome", "whatsapp", "email", "cryptocurrency",
"stolen", "freeze", "quick", "crucial", "tracing", "scammers", "expers", "hire", "century",
"transaction", "essential", "managing", "contact", "contacting", "understanding", "assets", "funds"
]
struct ForbiddenLookup {
let lengthBits: UnsafeMutablePointer<UInt16>
let wordData: UnsafeMutablePointer<UInt64>
let wordCount: UnsafeMutablePointer<UInt8>
init() {
lengthBits = UnsafeMutablePointer<UInt16>.allocate(capacity: 256)
wordData = UnsafeMutablePointer<UInt64>.allocate(capacity: 256 * 16)
wordCount = UnsafeMutablePointer<UInt8>.allocate(capacity: 256)
lengthBits.initialize(repeating: 0, count: 256)
wordData.initialize(repeating: 0, count: 256 * 16)
wordCount.initialize(repeating: 0, count: 256)
for word in forbiddenWords {
let bytes = Array(word.utf8)
guard !bytes.isEmpty && bytes.count <= 14 else { continue }
let firstChar = Int(bytes[0])
let len = bytes.count
lengthBits[firstChar] |= UInt16(1 << len)
var lo: UInt64 = 0
var hi: UInt64 = 0
for (i, b) in bytes.enumerated() {
if i < 8 {
lo |= UInt64(b) << (i * 8)
} else {
hi |= UInt64(b) << ((i - 8) * 8)
}
}
let idx = Int(wordCount[firstChar])
if idx < 8 {
wordData[firstChar * 16 + idx * 2] = lo
wordData[firstChar * 16 + idx * 2 + 1] = hi
wordCount[firstChar] = UInt8(idx + 1)
}
}
}
@inline(__always)
func check(_ ptr: UnsafePointer<UInt8>, _ len: Int) -> UInt64 {
let firstChar = Int(ptr[0])
if (lengthBits[firstChar] & UInt16(1 << len)) == 0 { return 0 }
let raw = UnsafeRawPointer(ptr)
var lo: UInt64
var hi: UInt64 = 0
if len <= 8 {
switch len {
case 1: lo = UInt64(ptr[0])
case 2: lo = UInt64(raw.loadUnaligned(as: UInt16.self))
case 3: lo = UInt64(raw.loadUnaligned(as: UInt16.self)) | (UInt64(ptr[2]) << 16)
case 4: lo = UInt64(raw.loadUnaligned(as: UInt32.self))
case 5: lo = UInt64(raw.loadUnaligned(as: UInt32.self)) | (UInt64(ptr[4]) << 32)
case 6: lo = UInt64(raw.loadUnaligned(as: UInt32.self)) | (UInt64(raw.advanced(by: 4).loadUnaligned(as: UInt16.self)) << 32)
case 7: lo = UInt64(raw.loadUnaligned(as: UInt32.self)) | (UInt64(raw.advanced(by: 4).loadUnaligned(as: UInt16.self)) << 32) | (UInt64(ptr[6]) << 48)
default: lo = raw.loadUnaligned(as: UInt64.self)
}
} else {
lo = raw.loadUnaligned(as: UInt64.self)
switch len {
case 9: hi = UInt64(ptr[8])
case 10: hi = UInt64(raw.advanced(by: 8).loadUnaligned(as: UInt16.self))
case 11: hi = UInt64(raw.advanced(by: 8).loadUnaligned(as: UInt16.self)) | (UInt64(ptr[10]) << 16)
case 12: hi = UInt64(raw.advanced(by: 8).loadUnaligned(as: UInt32.self))
case 13: hi = UInt64(raw.advanced(by: 8).loadUnaligned(as: UInt32.self)) | (UInt64(ptr[12]) << 32)
default: hi = UInt64(raw.advanced(by: 8).loadUnaligned(as: UInt32.self)) | (UInt64(raw.advanced(by: 12).loadUnaligned(as: UInt16.self)) << 32)
}
}
let base = firstChar * 16
let count = Int(wordCount[firstChar])
for j in 0..<count {
if wordData[base + j * 2] == lo && wordData[base + j * 2 + 1] == hi {
return 1
}
}
return 0
}
}
let lookup = ForbiddenLookup()
@inline(__always)
func analyzeBuffer(_ ptr: UnsafePointer<UInt8>, _ length: Int) -> Stats {
var words: UInt64 = 0
var capitalized: UInt64 = 0
var sentences: UInt64 = 0
var numbers: UInt64 = 0
var forbidden: UInt64 = 0
var i = 0
let end = length
while i < end {
let f0 = charFlags[Int(ptr[i])]
sentences &+= UInt64((f0 >> 3) & 1)
if (f0 & 1) != 0 {
i += 1
continue
}
let wordStart = i
capitalized &+= UInt64((f0 >> 2) & 1)
var hasDigit = UInt64((f0 >> 1) & 1)
i += 1
while i < end {
let f = charFlags[Int(ptr[i])]
if (f & 1) != 0 { break }
sentences &+= UInt64((f >> 3) & 1)
hasDigit |= UInt64((f >> 1) & 1)
i += 1
}
let wordLen = i - wordStart
words &+= 1
numbers &+= hasDigit
if wordLen <= 14 {
forbidden &+= lookup.check(ptr + wordStart, wordLen)
}
}
return Stats(words: words, capitalized: capitalized, sentences: sentences, numbers: numbers, forbidden: forbidden)
}
func processFile(_ path: String) -> Stats {
let fd = open(path, O_RDONLY)
guard fd >= 0 else { return Stats() }
defer { close(fd) }
var st = stat()
guard fstat(fd, &st) == 0 else { return Stats() }
let size = Int(st.st_size)
guard size > 0 else { return Stats() }
guard let mapped = mmap(nil, size, PROT_READ, MAP_PRIVATE, fd, 0), mapped != MAP_FAILED else {
return Stats()
}
defer { munmap(mapped, size) }
let ptr = mapped.assumingMemoryBound(to: UInt8.self)
return analyzeBuffer(ptr, size)
}
func main() {
let args = CommandLine.arguments
guard args.count > 1 else {
print("Usage: \(args[0]) <file1> <file2> ... <fileN>")
return
}
let files = Array(args.dropFirst())
let fileCount = files.count
let resultsPtr = UnsafeMutablePointer<Stats>.allocate(capacity: fileCount)
resultsPtr.initialize(repeating: Stats(), count: fileCount)
defer { resultsPtr.deallocate() }
DispatchQueue.concurrentPerform(iterations: fileCount) { idx in
resultsPtr[idx] = processFile(files[idx])
}
var total = Stats()
for i in 0..<fileCount {
let r = resultsPtr[i]
total.words &+= r.words
total.capitalized &+= r.capitalized
total.sentences &+= r.sentences
total.numbers &+= r.numbers
total.forbidden &+= r.forbidden
}
let capitalizedPct = total.words > 0 ? Double(total.capitalized) / Double(total.words) * 100.0 : 0.0
let forbiddenPct = total.words > 0 ? Double(total.forbidden) / Double(total.words) * 100.0 : 0.0
let wordsPerSentence = total.sentences > 0 ? Double(total.words) / Double(total.sentences) : 0.0
print("")
print("Total Words: \(total.words)")
print("Total Capitalized words: \(total.capitalized)")
print("Total Sentences: \(total.sentences)")
print("Total Numbers: \(total.numbers)")
print("Total Forbidden words: \(total.forbidden)")
print(String(format: "Capitalized percentage: %.6f%%", capitalizedPct))
print(String(format: "Forbidden percentage: %.6f%%", forbiddenPct))
print(String(format: "Word count per sentence: %.6f", wordsPerSentence))
print("Total files read: \(fileCount)")
}
main()