retoor b2ad31f5f6 feat: add tokenizer.c with utf-8 byte pair encoding implementation
Implement tokenizer.c containing is_valid_char helper and main function for reading text files, extracting alphanumeric words with dash/underscore support, and tracking word positions, line numbers, and paragraph counts across the input stream.
2025-01-05 15:20:06 +00:00

Examples / research

This is a collection of source code examples and research that I share with others online. Not all is written by me. It's also used for just sharing resources on discussion forums. The contents are mainly destined to be read by a software developer.

Do you see something you want to know more about? Contact retoor@molodetz.nl.

S
Description
Information I want to share with other people.
Readme
1.3 MiB
Languages
Python 86.7%
C 8.7%
HTML 2.8%
JavaScript 1.5%
Shell 0.3%