What's Changed
- Created a simple management Web UI. Try it on localhost:4001
- Added the possibility of creating spiders with the help of the YML format. Read more here: https://github.com/elixir-crawly/crawly/blob/master/documentation/spiders_in_yml.md
- Added the possibility to run Crawly (and your scraping projects) without Elixir. Read more here: https://github.com/elixir-crawly/crawly/blob/master/documentation/standalone_crawly.md
- Added generators for Crawly spiders and configuration files to reduce boilerplate
- Improved UniqueRequest middleware so that it can store hashes instead of complete URLs (special thanks to @serpent213)
- Added SameDomainFilter middleware, my favorite, which will probably deprecate the need to rely on the base_url in the future. Again thanks to @serpent213!