retoor
e8a993062c
Add a new Crawler class in crawler.wren that manages a URL queue, tracks visited pages, throttles concurrent requests to a maximum of 10 in-flight connections, and processes responses with error handling and status code validation. The crawler uses a fiber-based event loop that yields to the C host until all crawling is complete, then signals completion with duration and page count statistics.