This is a modified version of James Mills' web crawler from http://code.activestate.com/recipes/576551-simple-web-crawler/ .
This is a completely overahuled python3 crawler.
Built using Python 3.10.9
To install, download via zip, or clone onto your machine
In your terminal of the current working directory python-webcrawler
type bash install.sh
This will install all the dependencies that are needed as well as a separate virtual environment.