Skip to content

Releases: open-compass/opencompass

OpenCompass v0.2.3

12 Mar 03:53
@Leymore Leymore
0.2.3
ab6cdb2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.3

The OpenCompass team is thrilled to announce the release of OpenCompass v0.2.3! This version is packed with new features, crucial fixes, and documentation updates to improve your experience. We're continuously working to enhance OpenCompass, making it more robust and versatile for all users.

🌟 Highlights:

  • Enhanced Model Support: Introduction of new models and configurations, including support for the LightllmApi, lmdeploy pytorch engine, and more.
  • New Datasets and Benchmarks: Expanding our dataset repository with additions like OpenFinData, lveval benchmark, and an upgrade to Needlebench.
  • Documentation and Sync Improvements: Updated dataset pack URLs, fixed documentation errors, and synchronized with internal codes for consistency.

Explore the key updates in this release:

🌟 New Features:

  • 📦 Dataset and Benchmark Expansion:

    • Support for new datasets like OpenFinData and an upgrade to Needlebench, offering broader evaluation capabilities (#896, #913).
    • Introduction of the lveval benchmark to enrich the evaluation landscape (#914).

  • 🛠 Model and API Integrations:

    • Enhanced functionality with support for LightllmApi input_format and prompt templates, alongside the introduction of get_ppl for TurbomindModel (#888, #878).
    • New model configurations added, including support for gemini and deepseek-coder, further broadening the tools available for users (#931, #943).

  • 📖 Documentation and Sync Updates:

    • Updated dataset pack URLs and rank link in README to ensure users have access to the latest resources (#922, #911).
    • Several syncs with internal codes and GitHub blacklist update to maintain consistency and integrity (#929, #953).

🐛 Bug Fixes:

  • Addressed various configuration and template issues to ensure smoother operation across different models and benchmarks (#894, #893).
  • Fixed issues related to IFEval, including type hints and config bugs, enhancing evaluation accuracy and functionality (#906, #915).

🎉 Welcome New Contributors:

🔗 Full Changelog

For a detailed overview of all changes, check out our Full Changelog.

Contributors

x22x22, tonysy, and 10 other contributors
Assets 2
Loading

OpenCompass v0.2.2

08 Feb 06:04
@Leymore Leymore
0.2.2
08133e0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.2

Welcome to OpenCompass v0.2.2, a release brimming with new features, essential fixes, and significant improvements across the board. With a focus on enhancing functionality and expanding dataset support, this update underscores our commitment to providing a robust platform for our users.

🌟 Highlights:

  • Broadened Dataset Support: Introduction of diverse datasets like T-Eval, CIBench, IFEval, and NPHardEval, and more, broadening the horizons for research and evaluation.
  • API Integrations and Updates: New support for APIs like Nanbeige and updates to existing ones such as Zhipu and Sensetime, enhancing model interaction capabilities.
  • Dataset Collection Release: Integrated dataset collection is availabe in 0.2.2.rc1. Dataset used in OpenCompass 2.0 leaderboard is NOT included in this collection.

Dive into what's new and improved:

🌟 New Features:

  • 📦 Datasets Expansion:

    • Addition of multiple new datasets and evaluations, including T-Eval, CIBench, IFEval, and NPHardEval, offering more versatility for users (#813, #829, #809, #835).

  • 🛠 API and Model Enhancements:

    • Support for new APIs like Nanbeige and updates to enhance the functionality of existing ones (#786, #847, #834).
    • Configurations and support for models and evaluators have been improved and expanded (#791, #812, #845).

  • 📖 Documentation and CI Enhancements:

    • Updated FAQs, contribution guides, and added new test runners to improve CI/CD processes (#830, #751, #874).

🐛 Bug Fixes:

  • Various fixes have been applied to address issues across datasets, evaluators, and configurations, ensuring a smoother experience for all users (#787, #788, #789).

🎉 Welcome New Contributors:

🔗 Full Changelog

For a full list of updates, visit our Full Changelog.

Thank you to every contributor, old and new. Your dedication is shaping OpenCompass into a more robust and versatile tool. 🙌 🎉

Remember to star 🌟 our GitHub repository if OpenCompass aids your research and development! Your support and feedback are crucial for our continuous improvement.

Contributors

tonysy, QipengGuo, and 17 other contributors
Assets 2
Loading

OpenCompass v0.2.2.rc1

06 Feb 17:33
@Leymore Leymore
0.2.2.rc1
e257254
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.2.rc1 Pre-release
Pre-release

Provide with more parsed datasets:

OpenCompassData-core-20240207.zip
OpenCompassData-complete-20240207.zip

Important updates compared to previous version are as follow:

  • Subjective: Add AlignBench, MTBench
  • Agent: Add T-Eval
  • Medicine: Add MedBench
  • Code: Add HumanEval-X, DS-1000
  • Finance: Add FinanceIQ
  • Law: Update LawBench Evaluation Assets

OpenCompassData-core-20240207.zip
AGIEval ARC BBH ceval CLUE cmmlu
commonsenseqa drop FewCLUE flores_first100 GAOKAO-BENCH gsm8k
hellaswag humaneval lambada LCSTS math mbpp
mmlu nq openbookqa piqa race siqa
strategyqa summedits SuperGLUE TheoremQA triviaqa tydiqa
winogrande xstory_cloze Xsum

OpenCompassData-complete-20240207.zip
AGIEval anli ARC BBH CDME ceval
cibench_dataset cleva clozeTest-maxmin CLUE CMB cmmlu
commonsenseqa commonsenseqa_cn crowspairs_cn drop ds1000_data FewCLUE
FinanceIQ flores200_dataset flores_first100 FunctionalMT game24 GAOKAO-BENCH
gpqa gsm8k hellaswag humaneval humaneval_cn humaneval_multipl-e
humanevalx HungarianExamMath InfiniteBench lambada lanQ lawbench
LCSTS math math401 mbpp mbpp_cn mbpp_plus
MedBench mmlu MNIST NPHardEval nq nq_cn
nq-open openbookqa piqa py150 qabench race
scibench siqa SQuAD2.0 strategyqa alignment_bench mtbench
summedits SuperGLUE svamp teval TheoremQA triviaqa
tydiqa winogrande xiezhi xlsum xstory_cloze Xsum
Assets 4
Loading

OpenCompass v0.2.1

08 Jan 14:57
@bittersweet1999 bittersweet1999
0.2.1
a74e4c1
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.1

We're thrilled to announce OpenCompass v0.2.1, loaded with new datasets, features, and vital fixes. This release is a testament to our ongoing commitment to enhancing user experience and broadening research capabilities.

🌟 Highlights:

  • Add Agent and Code datasets: Diverse new datasets like GPQA, mastermath2024v1, and more, significantly expanding the scope of OpenCompass.
  • Support Different JudgeLLM Subjective Evaluation: Providing more choice when choose judgellms.
  • Support Needle in Haystack: Support Needle in Haystack for longtext evaluation.
  • Add VLLM Evaluation: We support VLLM inference and evaluation.

Here's what's new:

🚀 New Features:

  • 📦 Dataset Expansion:

    • Added rwkv-5-3b model (#666)
    • Integration of diverse datasets including GPQA, Creationbench, and more.
    • Support for new datasets like mastermath2024v1, mbpp_plus, and sanitized_mbpp (#744, #770, #745)

  • 🛠 Functional Enhancements:

    • Subjective evaluation improvements (#692, #724)
    • Updated python action, slurm, and docker docs (#694, #718)
    • Turbomind API support and Qwen API integration (#693, #735)

  • 📖 Documentation Updates:

    • Updated contamination, alignmentbench, and other docs for better clarity (#698, #707)
    • Fixed dead links and typos in various documents (#455, #773, #774)

🐛 Bug Fixes:

  • Addressed various issues including those in alignmentbench, configs, and postprocess scripts.
  • Fixed bugs concerning subjective evaluation and EOS string detection.
  • Quick fixes for improved performance and reliability.

🎉 Welcome New Contributors:

🔗 Full Changelog

For a full list of updates, visit our Full Changelog.

Thank you to every contributor, old and new. Your dedication is shaping OpenCompass into a more robust and versatile tool. 🙌 🎉

Remember to star 🌟 our GitHub repository if OpenCompass aids your research and development! Your support and feedback are crucial for our continuous improvement.

Contributors

hzhwcmhf, tpoisonooo, and 19 other contributors
Assets 2
Loading

OpenCompass v0.2.0

12 Dec 06:42
@yingfhu yingfhu
0.2.0
4780b39
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.2.0

🌟 Highlights

  • 🛠 Data Contamination Analysis: A novel feature for analyzing and ensuring the integrity of dataset inputs.
  • 🧠 Enhanced Subjective Evaluation: Implementation of a new subjective judgement system, providing more nuanced and accurate evaluations.
  • 🚀 Chat Style Inferencer Support: Introduction of a new chat style inferencer, enhancing interactive capabilities.
  • 🌐 Multilingual Features: Expansion to support Chinese versions of commonsenseqa, crowspairs, and nq datasets.
  • 📊 New Datasets Integration: Addition of wikibench, rolebench, and updated versions of gsm8k and MathBench datasets for broader research applications.
  • 🛠 Enhancements and Bug Fixes: Numerous improvements including a new subjective judgement system and updates in MathBench CodeInterpreter.
  • 📝 Documentation and API Updates: Comprehensive updates to README and API interfaces for better user guidance and experience.

🚀 New Features & Enhancements

  • Support for chat style inferencer, offering a more dynamic interaction model (#643).
  • Addition of Chinese versions for key datasets: commonsenseqa, crowspairs, and nq (#144).
  • Introduction of the wikibench dataset, providing a new benchmark for knowledge-based tasks (#655).
  • Updated gsm8k and MathBench configurations for enhanced performance and accuracy (#652, #657).
  • Addition of rolebench dataset, expanding the range of evaluative scenarios (#633).
  • Implementation of new subjective judgement criteria for improved assessment accuracy (#660).
  • Integration of advanced models like qwen-1.8b/72b and deepseek-7b/67b in the platform's configuration (#672).
  • Launch of Data Contamination Analysis as a new feature, enhancing data integrity checks (#639).

🛠 Improvements & Fixes

  • Removal of colossalai dependency to streamline operations (#645).
  • Resolution of various bugs including hellaswag_ppl_47bff9 and standard deviation summarizer issues (#648, #675).
  • Update and fix of the MathBench CodeInterpreter and related bugs (#657).
  • Enhancement of API interface for improved functionality and user experience (#681).

📚 Documentation Updates

  • Updated README for clearer guidance and information (#682).
  • Documentation and docstring updates for accuracy and comprehensiveness (#684).

🎊 New Contributors

🔗 Full Changelog

Thank you to all contributors for your hard work and dedication. OpenCompass v0.2.0 marks another step forward in our journey, bringing enhanced features and capabilities to the community. Let's continue to innovate and expand the horizons of OpenCompass! 🎉🌐💡

Contributors

tonysy, Leymore, and 9 other contributors
Assets 2
Loading

OpenCompass v0.1.9

28 Nov 03:53
@Leymore Leymore
0.1.9
e20d654
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.1.9

🌟 Highlights

  • 🚀 New API Integrations: A leap forward with the addition of multiple new APIs, including Baidu, Moonshot, Sensetime, and more, broadening the scope and capabilities of OpenCompass.
  • 🔵 Circular Evaluation Feature: Introducing Circular Eval, an enhancement for comprehensive and dynamic evaluations within the platform.
  • 🤖 Turbomind Inference Integration: Integration of Turbomind inference through its RPC API, enhancing the platform's inferencing capabilities.

🚀 New Features & Enhancements

  • Model & API Development: Explore new capabilities with DataCanvas Alaya LM, Lightllm API, 360API, and enhanced Turbomind Python API integration (#612, #613, #601, #484).
  • Circular Evaluation Implementation: Elevate your evaluation methods with the newly added Circular Eval feature, offering a more nuanced and detailed analysis capability (#610).
  • Rich Dataset Additions: Enrich your research with new datasets - FinanceIQ, SVAMP, GSM_Hard, and updated Mathbench for diverse applications (#596, #604, #619, #580, #607).

🛠 Improvements & Fixes

  • Subjective Evaluation Bug Fixes: Improved accuracy in subjective evaluations (#589).
  • Dataset and Feature Fixes: Resolving issues in CMB dataset, various feature enhancements, and fixes (#587, #592, #615, #632).

📚 Documentation Updates

  • README & FAQ Enhancements: Updated for better clarity and assistance (#582, #622, #628, #629).
  • Typo and Spelling Corrections: Ensuring accuracy and professionalism in documentation (#594, #637).

🎊 New Contributors

Welcoming new contributors to the OpenCompass family!

What's Changed

Explore the detailed changes in the full changelog.

Thank you to all the contributors for this release. Your dedication and hard work continue to enhance OpenCompass, making it an ever-evolving and dynamic tool for the community. Let's dive into the new possibilities with OpenCompass v0.1.9! 🎉🧮💻

Contributors

lvhan028, tonysy, and 12 other contributors
Assets 2
Loading

OpenCompass v0.1.8

13 Nov 08:43
@Leymore Leymore
0.1.8
1ea88d5
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.1.8

🔥 Highlights

  • 🌐 New Dataset Integrations: Expanding our dataset collection with Tabmwp, py150, maxmin, and more.
  • 💡 Compatibility and API Support: Enhancements with MiniGPT-4 and MiniMax API, and support for Xunfei API.
  • 🛠️ Local Environment and Debugging Improvements: Streamlined local debugging and usage of datasets from local paths.

🚀 New Features & Enhancements

  • Datasets Galore: Unleash the power of new datasets including Tabmwp, py150, maxmin, and updates to existing ones like Mathbench for broader research scope (#505, #546, #562).
  • MiniGPT-4 & MiniMax API Compatibility: Stay up-to-date with the latest versions and extended API support (#539, #548).
  • Xunfei API Model & Update: Explore new possibilities with the integration and update of Xunfei API (#547, #572).

🛠 Improvements & Fixes

  • Local Debug Mode Restriction: Enhanced resource management in local debug mode (#522 by @yingfhu).
  • Various Fixes and Updates: Addressing typos, import issues, and log redirections for smoother operation (#520, #549, #551, #555, #564).

📚 Documentation Updates

  • Enhanced README and FAQs: Get all your queries answered and understand OpenCompass better with updated documentation (#523, #531, #535, #540, #567).
  • Typo Corrections: Ensuring clarity and accuracy in our documentation (#530, #533).

🎊 New Contributors

A warm welcome to the new members of the OpenCompass community!

Changelog

Explore the detailed changes in the full changelog.

Thank you to everyone who contributed to this release. Your efforts are immensely appreciated and are helping to make OpenCompass a more robust and versatile tool. Let's continue to push the boundaries with OpenCompass v0.1.8! 🚀🌐🛠️

Contributors

Sanster, tonysy, and 10 other contributors
Assets 2
Loading

OpenCompass v0.1.8.rc1

10 Nov 11:05
@Leymore Leymore
0.1.8.rc1
7f77e8d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.1.8.rc1 Pre-release
Pre-release

Provide with more parsed datasets:

OpenCompassData-core.zip

AGIEval ARC BBH ceval CLUE cmmlu
commonsenseqa drop FewCLUE flores_first100 GAOKAO-BENCH gsm8k
hellaswag humaneval lambada LCSTS math mbpp
mmlu nq openbookqa piqa race siqa
strategyqa summedits SuperGLUE TheoremQA triviaqa tydiqa
winogrande xstory_cloze Xsum

OpenCompassData-complete.zip

AGIEval anli ARC BBH ceval cleva
CLUE CMB cmmlu commonsenseqa drop ds1000
FewCLUE flores200_dataset flores_first100 game24 GAOKAO-BENCH govrep
gsm8k hellaswag humaneval jigsawmultilingual lambada lawbench
LCSTS math mbpp mmlu narrativeqa nq
openbookqa piqa QASPER race realtoxicprompts scibench
siqa SQuAD2.0 strategyqa summedits SummScreen SuperGLUE
TheoremQA triviaqa triviaqa-rc tydiqa winogrande xiezhi
xlsum xstory_cloze Xsum FinanceIQ
Assets 4
Loading

OpenCompass v0.1.7

27 Oct 15:49
@Leymore Leymore
0.1.7
6a398d1
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.1.7

🌟 Highlights

  • Sampling Control: Enforce do_sample=False for precise control over sampling behavior in HF model.
  • Subjective Evaluation Guidance: Enhanced evaluation mechanisms for a more comprehensive understanding and analysis of models.
  • Eval Details Dump: Now, evaluation details for certain datasets are available for a deeper insight and analysis.

🚀 New Features

  • Eval Details Dump for a deeper insight on each test cases. (#517 by @Leymore).
  • MathBench Dataset and Circular Evaluator to bolster mathematical benchmarking capabilities (#408 by @liushz).
  • Support for Math/GMS8k Agent Config providing new avenues for configuration (#494 by @yingfhu).
  • Default Example Summarizer making summarization tasks more accessible (#508 by @Leymore).
  • Model Keyword Arguments Setting for HF Model enhancing customization (#507 by @Leymore).

🛠 Improvements & Refactorings

  • Local API Speed Up with fixed concurrent users for better performance (#497 by @yingfhu).
  • Local Runner Support for Windows expanding the platform support (#515 by @yingfhu).
  • Sync with Internal Implements for updated and refined functionalities (#488 by @Leymore).

🐛 Bug Fixes

  • Summary Default Fix for accurate summarization (#483 by @Leymore).
  • Enforce do_sample=False in HF Model for correct sampling behavior (#506 by @Leymore).
  • Invalid Link Fix in documentation for better navigation (#499 by @yingfhu).

📚 Documentation & Maintenance

  • Subjective Comparison Introduction for organized documentation (#510 by @frankweijue).
  • README Update for better project understanding (#496 by @saakshii12).
  • Owner Update for correct ownership information (#504 by @Leymore).

🎊 New Contributors

We're delighted to welcome new contributors to the OpenCompass community!

Changelog

The full list of changes is available in the changelog. A massive thank you to all the community members who contributed to this release. Your efforts are propelling OpenCompass further! 🙌

Embark on new explorations with OpenCompass v0.1.7!

Contributors

Leymore, frankweijue, and 3 other contributors
Assets 2
Loading

OpenCompass v0.1.6

13 Oct 12:16
@Leymore Leymore
0.1.6
6317da0
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
OpenCompass v0.1.6

Welcome to the newest version of OpenCompass! v0.1.6 brings forth exciting dataset additions, crucial fixes, and enhanced documentation. We're confident that this release will provide a better and smoother experience for all users.

🆕 Highlights:

  • Dataset Enrichment: Multiple additions, especially from the GLUE suite, to provide more versatility and better testing capabilities.
  • Documentation Revamp: Fixed dead links and updated the 'get_started' section to assist our users in navigating OpenCompass effortlessly.
  • Introducing New Faces: A warm welcome to our newest contributors. Your dedication and contributions are pivotal to our progress!

Dive into the details:

🌟 New Features:

  • 📦 Datasets Galore:

    • Introduced WikiText-2&103 dataset (#397)
    • GLUE dataset additions:
    • Lawbench dataset addition (#460)

  • 🛠 Utilities and Enhancements:

    • Re-implementation of ceval load dataset (#446)
    • Integrated turbomind inference through its RPC API (#414)
    • Moved fix_id_list to Retriever for better code organization (#442)

  • 📖 Documentation and Syncs:

    • Updated dataset list and get_started section (#437, #435)
    • Resolved dead links in the readme (#455)
    • Enhancements to LongEval and subjective evaluation (#443, #475)

🐛 Bug Fixes:

  • Addressed issues related to clp errors and support for bs>1 (#439)
  • Resolved issues concerning jieba rouge (#459, #467)
  • Enhanced EOS string detection for splitting (#477)
  • Various other fixes for optimal performance.

🎉 Welcome New Contributors:

Huge thanks to all contributors! Your constant efforts make OpenCompass better with each release. 🙌 🎉

Changelog

For a detailed overview, check out our Full Changelog.

If you find OpenCompass beneficial, kindly star 🌟 our GitHub repository! We value your feedback, reviews, and continued support.

Contributors

lvhan028, Leymore, and 4 other contributors
Assets 2
Loading