Berkeley Function Calling Leaderboard Updates (v1.0)

[ShishirPatil]

Highlights

🏆 We are thrilled to announce the stable v1.0 release of the Berkeley Function Calling Leaderboard data-set and eval-pipeline! A heartfelt thank you to all our contributors and users for your enthusiastic engagement and support throughout v1. We are just getting started! Buckle-up for v2 🚀 🚀 🚀

What's Changed

• better handle float value comparison by @vandyxiaowei in better handle float value comparison #407
• Bump pymysql from 1.1.0 to 1.1.1 in /goex by @dependabot in Bump pymysql from 1.1.0 to 1.1.1 in /goex #453
• Fixes For NexusHandler by @VenkatKS in Fixes For NexusHandler #437
• [BFCL] PR#407 Evaluation Pipeline Robustness Patch by @HuanzhiMao in [BFCL] PR#407 Evaluation Pipeline Robustness Patch #462
• Add firefunction-v2 to the leaderboard by @pgarbacki in Add firefunction-v2 to the leaderboard #470
• [BFCL] Add Claude 3.5 Sonnet Function Calling Infernece Inference by @Fanjia-Yan in [BFCL] Add Claude 3.5 Sonnet Function Calling Infernece Inference  #480
• [BFCL] Standardize Model Name Among handler_map and eval_runner_helper by @HuanzhiMao in [BFCL] Standardize Model Name Among handler_map and eval_runner_helper #439
• Remove redundant tokens from GPT-handler by @hellovai in Remove redundant tokens from GPT-handler #490
• [GoEx] Undo Minor Bug Fix + README Minor Improvement by @royh02 in [GoEx] Undo Minor Bug Fix + README Minor Improvement #468
• [BFCL] Add ability to evaluate Nemotron-4-340B-Instruct by @Fanjia-Yan in [BFCL] Add ability to evaluate Nemotron-4-340B-Instruct #489
• fix some data issues in parallel/parallel multiple answers by @vandyxiaowei in fix some data issues in parallel/parallel multiple answers #423
• [BFCL] Add Support for GLM-4-9B function calling inference by @Fanjia-Yan in [BFCL] Add Support for GLM-4-9B function calling inference #474
• [BFCL] Sanity check is now optional by @ShishirPatil in [BFCL] Sanity check is now optional #496
• [BFCL] Improved tree-sitter java, javascript installation by @CharlieJCJ in [BFCL] Improved tree-sitter java, javascript installation #505
• [BFCL] Fix Possible Answer for AST Parallel and Parallel_Multiple Category by @HuanzhiMao in [BFCL] Fix Possible Answer for AST Parallel and Parallel_Multiple Category #503
• [BFCL] Add Test Dataset to Repository by @HuanzhiMao in [BFCL] Add Test Dataset to Repository #504
• [BFCL] Support Category-Specific Generation for OSS Model, Remove eval_data_compilation Step by @HuanzhiMao in [BFCL] Support Category-Specific Generation for OSS Model, Remove eval_data_compilation Step #512
• [BFCL] Fix Double-Casting Issue in model_handler for Java and JS category. by @HuanzhiMao in [BFCL] Fix Double-Casting Issue in model_handler for Java and JS category.  #516
• [BFCL] Fix Dataset Issue for executable_parallel_multiple Category by @HuanzhiMao in [BFCL] Fix Dataset Issue for executable_parallel_multiple Category #522
• [BFCL] add ibm-granite-20b-functioncallling model by @MayankAgarwal in [BFCL] add ibm-granite-20b-functioncallling model #525
• [BFCL] Overhaul apply_function_credential_config.py for Enhanced Usability by @HuanzhiMao in [BFCL] Overhaul apply_function_credential_config.py for Enhanced Usability #508
• Fixed the warning message "Setting pad_token_id to eos_token_id:1… by @dineshkumarsarangapani in Fixed the warning message "Setting pad_token_id to eos_token_id:1… #110
• [BFCL] Specify package version in requirements.txt by @HuanzhiMao in [BFCL] Specify package version in requirements.txt #515
• [BFCL] Standardize TEST_CATEGORY Among eval_runner.py and openfunctions_evaluation.py by @HuanzhiMao in [BFCL] Standardize TEST_CATEGORY Among eval_runner.py and openfunctions_evaluation.py #506
• fix line return by @fantasist in fix line return #531
• [BFCL] Apply Fix to Newly Introduced Model Handler Missed in Previous PR Merge by @HuanzhiMao in [BFCL] Apply Fix to Newly Introduced Model Handler Missed in Previous PR Merge #536
• [RAFT] Fix Datapoint Field in Formatter for Data Generation by @HuanzhiMao in [RAFT] Fix Datapoint Field in Formatter for Data Generation #535
• [BFCL] Fix language_specific_pre_processing for Java and JavaScript Test Category by @HuanzhiMao in [BFCL] Fix language_specific_pre_processing for Java and JavaScript Test Category #538
• [BFCL] Patch Generation Script for Locally Hosted OSS model by @HuanzhiMao in [BFCL] Patch Generation Script for Locally Hosted OSS model #537
• [BFCL] Support Multi-Model Multi-Category Generation; Add Index to Dataset; Handle vLLM Benign Error by @HuanzhiMao in [BFCL] Support Multi-Model Multi-Category Generation; Add Index to Dataset; Handle vLLM Benign Error #540
• Add NousResearch/{Hermes-2-Pro-Llama-3-8B,Hermes-2-Theta-Llama-3-8B} models by @alonsosilvaallende in Add NousResearch/{Hermes-2-Pro-Llama-3-8B,Hermes-2-Theta-Llama-3-8B} models #542
• [BFCL] Fix Dataset Pre-Processing for Java and JavaScript Test Category, Part 2 by @HuanzhiMao in [BFCL] Fix Dataset Pre-Processing for Java and JavaScript Test Category, Part 2 #545
• Add Salesforce xLAM handler and fix minor issues by @zuxin666 in Add Salesforce xLAM handler and fix minor issues #532
• Add NousResearch/Hermes-2-{Pro-Llama-3-80B,Theta-Llama-3-80B} by @alonsosilvaallende in Add NousResearch/Hermes-2-{Pro-Llama-3-80B,Theta-Llama-3-80B} #556
• Add Yi Handler by @fantasist in Add Yi Handler #543
• Add more descriptive error message in eval_runner.py by @alonsosilvaallende in Add more descriptive error message in eval_runner.py #552
• [BFCL] Fix JS type converter to handle dictionaries with array values by @CharlieJCJ in [BFCL] Fix JS type converter to handle dictionaries with array values #549
• [BFCL] Handling rate limits by @ShishirPatil in [BFCL] Handling rate limits #559
• [BFCL] Fix Dataset and Possible Answer Issue by @HuanzhiMao in [BFCL] Fix Dataset and Possible Answer Issue #557
• [BFCL] Dataset Question Fix for Executable Parallel Category by @HuanzhiMao in [BFCL] Dataset Question Fix for Executable Parallel Category #568
• [BFCL] Add New Model gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18 by @HuanzhiMao in [BFCL] Add New Model gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18 #569
• [BFCL] Add New Model open-mistral-nemo-2407, open-mixtral-8x22b, open-mixtral-8x7b by @HuanzhiMao in [BFCL] Add New Model open-mistral-nemo-2407, open-mixtral-8x22b, open-mixtral-8x7b #570
• [BFCL] Improve Warning Message when Aggregating Results by @HuanzhiMao in [BFCL] Improve Warning Message when Aggregating Results #517
• [BFCL] Add New Model functionary-small-v3.1, functionary-small-v3.2, functionary-medium-v3.1; Update Token Price by @HuanzhiMao in [BFCL] Add New Model functionary-small-v3.1, functionary-small-v3.2, functionary-medium-v3.1; Update Token Price #573
• [BFCL] Set Model Temperature to 0.001 for All Models by @HuanzhiMao in [BFCL] Set Model Temperature to 0.001 for All Models #574
• [BFCL] Support Parallel Inference for Hosted Models by @HuanzhiMao in [BFCL] Support Parallel Inference for Hosted Models #571
• [BFCL Chore] Fix Functionary Medium 3.1 model name & add readme parallel inference by @CharlieJCJ in [BFCL Chore] Fix Functionary Medium 3.1 model name & add readme parallel inference #577

New Contributors

• @dependabot made their first contribution in Bump pymysql from 1.1.0 to 1.1.1 in /goex #453
• @VenkatKS made their first contribution in Fixes For NexusHandler #437
• @pgarbacki made their first contribution in Add firefunction-v2 to the leaderboard #470
• @hellovai made their first contribution in Remove redundant tokens from GPT-handler #490
• @MayankAgarwal made their first contribution in [BFCL] add ibm-granite-20b-functioncallling model #525
• @dineshkumarsarangapani made their first contribution in Fixed the warning message "Setting pad_token_id to eos_token_id:1… #110
• @fantasist made their first contribution in fix line return #531
• @alonsosilvaallende made their first contribution in Add NousResearch/{Hermes-2-Pro-Llama-3-8B,Hermes-2-Theta-Llama-3-8B} models #542

Full Changelog: v0.3...v1.0

---

This discussion was created from the release Berkeley Function Calling Leaderboard Updates (v1.0).