Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Leaderboard UI improvements #1370

Merged
merged 5 commits into from
Nov 4, 2024
Merged

fix: Leaderboard UI improvements #1370

merged 5 commits into from
Nov 4, 2024

Conversation

x-tabdeveloping
Copy link
Collaborator

Improvements:

Implemented the following things:

  • Added RegEx-sensitive search bar
  • Reorganized model and benchmark criteria in a way that's more sensible
  • Table improvements:
    • Better column names
    • Reorganized columns
    • Added clickable model links
    • Adjusted column widths

Screenshot:

image

@KennethEnevoldsen
Copy link
Contributor

Would change:
"should be sentenceTransformer compatible" -> "SentenceTransformer": from the context it is clear

Discussion points:
"Availability": Why not have it be "Open", or "Proprietary" and just check off both by default? Same for instructions.

Copy link
Collaborator

@isaac-chung isaac-chung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the table improvements!
In general, I don't see why we are changing some of the components unless they are based on collected feedback.

  1. Search: Is it possible to replicate the current leaderboard search i.e. separate multiple queries with ;? If not, I'd prefer to retain that instead of replacing it with regex. Ref: Leaderboard 2.0: Re-add model searchbar from 1.0 #1347
  2. Defaults: Current leaderboard has both "open" and "proprietary" selected, and both with/ without instructions selected.
  3. Benchmark: What is the expected use of the filters underneath "Benchmarks"? Are users expected to select a benchmark, then add more filters if desired?

@x-tabdeveloping
Copy link
Collaborator Author

  1. Using the ; syntax in the search bar is technically equivalent to using | in RegEx. And the current leaderboard technically also uses regex, so I think it's quite redundant.
  2. I thought the way it was organized in the old leaderboard was a bit unclear, and I wanted to spell it out a bit more to communicate the intentions of these buttons clearer. I'm open to changing it up if people see fit.
  3. The idea is that people can customize a benchmark to their liking. For instance if you are interested in retrieval and reranking in Danish but nothing else, you can select these and rank the models based on that. I think this is a rather useful thing to have.

@isaac-chung
Copy link
Collaborator

isaac-chung commented Nov 1, 2024

  1. Great! Let's spell that out then.
  2. Sounds good.
  3. Great! Definitely useful to have.

Copy link
Collaborator

@isaac-chung isaac-chung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! A few small suggestions.

mteb/leaderboard/app.py Outdated Show resolved Hide resolved
mteb/leaderboard/app.py Outdated Show resolved Hide resolved
@x-tabdeveloping x-tabdeveloping enabled auto-merge (squash) November 4, 2024 13:59
@x-tabdeveloping x-tabdeveloping merged commit 92fe9cb into main Nov 4, 2024
10 checks passed
@x-tabdeveloping x-tabdeveloping deleted the searchbar branch November 4, 2024 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants