Log headers when hitting Github API rate limit

[njh]

https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api

[80avin]

I'll take this.
I already have more changes locally, like

• parallel download of multiple repo/commits/users to speed up download
• save as CSV files for faster processing.
  • saving large json on every download is slow as serialization is required.
  • json takes too much space.
  • CSV can be easily appended, making the save faster as we only add new rows instead of saving entire json object.
• track etag/last-modified headers to download only if resource has changed
  • This saves the rate limit quota because unchanged resources are not downloaded.
• if rate limit is reached, automatically wait for appropriate time.
• Error handling

will push after some testing and cleanup

[njh]

That will be awesome, thanks!

I had thought that putting everything in a SQLite database would be desirable - but it would require quite a lot of re-engineering

[80avin]

I also thought of using the SQLite database. While it had a few advantages, they weren't significant enough to counter the human-readable factor of CSV.

NVM, I have pushed my changes in #43
You can review and give your suggestions there.