Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leaderboard: Add zero-shot filter to leaderboard #1372

Closed
KennethEnevoldsen opened this issue Nov 1, 2024 · 2 comments
Closed

Leaderboard: Add zero-shot filter to leaderboard #1372

KennethEnevoldsen opened this issue Nov 1, 2024 · 2 comments
Labels
leaderboard issues related to the leaderboard

Comments

@KennethEnevoldsen
Copy link
Contributor

Add zero-shot generalization to model metadata. This would allow us to filter MTEB models.

@Muennighoff
Copy link
Contributor

Cc'ing @jhyuklee & Xiaoqi Ren (can't find the GitHub handle) as you were very interested in this! Hopefully will be there with a new leaderboard soon, any thoughts from you are ofc very welcome

I recall that you mentioned that https://huggingface.co/BAAI/bge-en-icl has the ArguAna test set in their training dataset they released and relatedly oddly high ArguAna Retrieval scores, so that's one to remember to flag in our metadata for the new leaderboard

@isaac-chung isaac-chung added the leaderboard issues related to the leaderboard label Nov 9, 2024
@KennethEnevoldsen
Copy link
Contributor Author

Closing in favor of #1556

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
leaderboard issues related to the leaderboard
Projects
None yet
Development

No branches or pull requests

3 participants