Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

leaderboard v2 - add autoupdate #1605

Open
KennethEnevoldsen opened this issue Dec 16, 2024 · 3 comments
Open

leaderboard v2 - add autoupdate #1605

KennethEnevoldsen opened this issue Dec 16, 2024 · 3 comments
Labels
leaderboard issues related to the leaderboard

Comments

@KennethEnevoldsen
Copy link
Contributor

Seems like we currently need to manually update the MTEB leaderboard (@orionw I believe you fixed the old one. I believe we really only need to restart the space daily)

@isaac-chung isaac-chung added the leaderboard issues related to the leaderboard label Dec 18, 2024
@orionw
Copy link
Contributor

orionw commented Dec 18, 2024

Yes, definitely! I'm trying to figure out how the leaderboard gets its data, seems like there's a function in MTEB now that provides the results, is that right? (also cc'ing @x-tabdeveloping). The way I restarted it daily before was by updating the results file in the repo daily and then the results repo would re-start.

If I understand correctly, would the best way to do this be to auto-restart the leaderboard on MTEB merges? Or perhaps MTEB results github merges? I don't see the connection to the results github, but I haven't traced it all the way back yet.

@x-tabdeveloping
Copy link
Collaborator

So the way it currently happens is that I download the results and cache them everytime the Docker container is rebuilt.
The results live in the specific Docker image, so the app needs a rebuild every time we update.
I know this isn't optimal, but we can look into ways of making this more seamless

@orionw
Copy link
Contributor

orionw commented Dec 19, 2024

Totally works great :) Do you mind pointing me to that step? I can work out how to get the leaderboard to refresh after it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
leaderboard issues related to the leaderboard
Projects
None yet
Development

No branches or pull requests

4 participants