Commit Graph

1 Commits

Author SHA1 Message Date
e8a993062c feat: implement web crawler with url queue and html parsing engine
Add a new Crawler class in crawler.wren that manages a URL queue, tracks visited pages, throttles concurrent requests to a maximum of 10 in-flight connections, and processes responses with error handling and status code validation. The crawler uses a fiber-based event loop that yields to the C host until all crawling is complete, then signals completion with duration and page count statistics.
2025-07-29 13:14:51 +00:00