109 Commits 1 Branch 0 Tags 1.3 MiB
a6760cda17
Commit Graph

3 Commits

Author SHA1 Message Date
retoor
4c12097369 chore: add chroot container support and refactor tokenizer variable naming 
- Add CHANGELOG entry for version 0.5.0 documenting new chroot container feature
- Rename ambiguous variables in tokenizer.py (word→current_word, word_start→word_start_position, etc.)
- Standardize string quotes from single to double in argparse arguments
- Normalize whitespace in comments and remove trailing spaces
 2025-11-05 13:12:23 +00:00
retoor
fb02bc786d fix: replace while-true-break pattern with walrus operator in tokenizer file read loop 2025-01-05 15:00:38 +00:00
retoor
a4fb287b54 feat: add tokenizer module with whitespace splitting, punctuation handling, and SQLite-backed vocabulary builder 2025-01-05 14:57:33 +00:00