[BFCL] Adds support for parallel inference and batching #498
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Parallel Inference Support for berkeley-function-call-leaderboard
This PR adds support for running
berkeley-function-call-leaderboard
inference in parallel, reducing running time by 4x or more depending on--batch-size
.Changes
Modifies
berkeley-function-call-leaderboard/model_handler/handler.py
write
function to make it async usingaiofiles
sort_results
function to sort the results based onidx
after each individual test_cate is oversort_results
function returns the indices after sorting, supporting resuming functionalityModifies
berkeley-function-call-leaderboard/openfunctions_evaluation.py
--batch-size
arg, defaults to1
-> controls number of parallel requestsfetch_and_process
functionmake_async
function to wrap sync functions as async (used for handler.inference)main
functionnum_existing_lines
Resume Support
Improved resuming functionality in async code:
None
as a placeholder, which becomes the conditional for skipping test casesscreenshot
is also uploaded bottom of this PR to confirm that it works as intendedTesting
Tested on a custom OpenAI-compatible model running on vllm:
Benchmark Results
Debug Logs for new
Resume System