82 Commits 1 Branch 0 Tags 1.3 MiB
6b22dea5c8
Commit Graph

2 Commits

Author SHA1 Message Date
retoor
fb02bc786d fix: replace while-true-break pattern with walrus operator in tokenizer file read loop 2025-01-05 15:00:38 +00:00
retoor
a4fb287b54 feat: add tokenizer module with whitespace splitting, punctuation handling, and SQLite-backed vocabulary builder 2025-01-05 14:57:33 +00:00