parallelize fetch-data (#106) #107
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reworks the output for fetch-data so each line clearly specifies which crash id it's related to. Further, it removes a lot of the coloring. It also removes start/end indicators. The end result is a lot less output where all the important information is in the line and it can be read left-to-right and sorted and make sense. While doing this, I adjusted fetch-data to use "crash_id" instead of both "crashid" and "crash_id".
This adds a `--workers` option that lets you specify the number of workers to use for downloading crash data in parallel. This vastly speeds up downloading.
willkg
commented
Oct 27, 2023
| @@ -232,38 +248,42 @@ def fetch_data( | |||
| + "information.[/yellow]" | |||
| ) | |||
|
|
|||
| if workers > 1 and not api_token: | |||
There was a problem hiding this comment.
We should validate workers to be a number between 1 and 10 and print the number of workers being used.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This vastly improves
fetch-dataspeed as it allows us to do something like this:It defaults to 1 worker (serial). It requires a
CRASHSTATS_API_TOKENfor multiple workers. The output is a lot leaner and easier to read.Fixes #106.