Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leaderboard: Add an Overview leaderboard #1432

Open
KennethEnevoldsen opened this issue Nov 11, 2024 · 2 comments
Open

Leaderboard: Add an Overview leaderboard #1432

KennethEnevoldsen opened this issue Nov 11, 2024 · 2 comments
Labels
leaderboard issues related to the leaderboard

Comments

@KennethEnevoldsen
Copy link
Contributor

Instead of having either English or Multilingual as the default why not have "overview" as the default?

Something like:

Model Name Multilingual English LongEmbed ...
M1 ...
M2

(this does not need to be added for the first version of the benchmark)

@isaac-chung
Copy link
Collaborator

That's an interesting idea! When we think of a leaderboard, we think of rankings. Would this be sort of a "meta" leaderboard, where the models have a rank based on all the benchmarks, e.g. a borda count over all benchmarks? Or based on one of multilingual or English?

If it's the former, one probable scenario I'm not sure how to deal with is with missing values, i.e. if that model does not have values for a particular benchmark(s).

Otherwise, we could enable sorting models based on each benchmark available. And users can click into each benchmark's breakdown.

@x-tabdeveloping
Copy link
Collaborator

Hmm I'm not sure whether this is something we actually want to have. If people are agnostic and just want to get a reasonable overview, wouldn't they just choose the multilingual or the English benchmarks? Like why would anyone be equally interested in a somewhat arbitrary selection of benchmarks that are sometimes subsets of each other? (European, Scandinavian, French, German, etc.)

@x-tabdeveloping x-tabdeveloping added the leaderboard issues related to the leaderboard label Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
leaderboard issues related to the leaderboard
Projects
None yet
Development

No branches or pull requests

3 participants