Example Crawler Project

This is a simple example crawler project. Follow the instructions below to set up and run the crawler.

Setup

Open a terminal in the project directory.
Run make to set up the environment, install dependencies, and start the crawler:

make

This will create a virtual environment, install the package in editable mode from the parent directory, and run the main script.

To remove the virtual environment, run:

make clean

The project installs the package with -e ../../. to include the parent package devranta in editable mode.
Ensure that the parent package is correctly set up in the directory structure.

Happy crawling!