Skip to content

Release 0.13.0

Compare
Choose a tag to compare
@oltarasenko oltarasenko released this 05 Feb 20:22
· 73 commits to master since this release
  1. Bugfix for start_urls size (now it's possible to have very large start URLs)
  2. Split business logs from other logs. Per spider logging
  3. Send logs to CrawlyUI (optional)
  4. Allow to override more spider options:
    • closespider_itemcount
    • closespider_timeout
    • concurrent_requests_per_domain (number of started workers)
  5. Change on_spider_log_callback (now it also gets the crawl_id)
  6. Parse pipelines