Leaderboard 2.0: Missing results

[x-tabdeveloping]

Some pretty essential results seem to be missing from the new leaderboard.
Here's a list of things that we should probably fix before releasing the leaderboard:

MTEB(Multilingual)

I have only looked into models we promised to run in the review response, the problems might be more widespread

• openai/text-embedding-3-large
  • SprintDuplicateQuestions
• openai/text-embedding-3-small
  • OpusparcusPC
  • SprintDuplicateQuestions
• gte-Qwen2-7B-instruct
  • Core17InstructionRetrieval
  • HagridRetrieval
  • MIRACLRetrievalHardNegatives
  • News21InstructionRetrieval
  • Robust04InstructionRetrieval
  • WebLINXCandidatesReranking
• BAAI/bge-large-en-v1.5
• gritlm-8x7B
  • Core17InstructionRetrieval
  • HagridRetrieval
  • MIRACLRetrievalHardNegatives
  • Robust04InstructionRetrieval
  • WebLINXCandidatesReranking
• salesforce/SFR-Embeddings-2R
  • Robust04InstructionRetrieval
  • News21InstructionRetrieval
  • Core17InstructionRetrieval
• snowflake/arctic-embed-m-v1.5:
  • missing almost all tasks: This is a problem with all snowflake models
• WhereIsAI/UAE-Large-V1:
  • missing almost all tasks
• stella_en_1.5B_v5:
  • Core17InstructionRetrieval
  • News21InstructionRetrieval
  • Robust04InstructionRetrieval
• stella_en_400M_v5
  • Core17InstructionRetrieval
  • News21InstructionRetrieval
  • Robust04InstructionRetrieval
• intfloat/e5-base-v2
  • Core17InstructionRetrieval
  • News21InstructionRetrieval
  • Robust04InstructionRetrieval
• intfloat/e5-large-v2
  • Core17InstructionRetrieval
• intfloat/e5-small-v2
  • Core17InstructionRetrieval
  • News21InstructionRetrieval
  • Robust04InstructionRetrieval
• jina/jina-embeddings-v3
• mixedbread-ai/mxbai-embed-v1
  • BrazilianToxicTweetsClassification
  • CEDRClassification
  • Core17InstructionRetrieval
  • KorHateSpeechMLClassification
  • MalteseNewsClassification
  • MultiEURLEXMultilabelClassification
  • News21InstructionRetrieval
  • Robust04InstructionRetrieval
• nomic-ai/nomic-embed-text-v1.5
  • BUCC.v2
  • BibleNLPBitextMining
  • BornholmBitextMining
  • BrazilianToxicTweetsClassification
  • CEDRClassification
  • Core17InstructionRetrieval
  • DiaBlaBitextMining
  • FloresBitextMining
  • IN22GenBitextMining
  • IndicGenBenchFloresBitextMining
  • KorHateSpeechMLClassification
  • MalteseNewsClassification
  • MultiEURLEXMultilabelClassification
  • NTREXBitextMining
  • News21InstructionRetrieval
  • NollySentiBitextMining
  • NorwegianCourtsBitextMining
  • NusaTranslationBitextMining
  • NusaXBitextMining
  • Robust04InstructionRetrieval
  • Tatoeba
• NV-embed-v2 missing metadata Add new models nvidia, gte, linq #1436

MTEB(eng, classic)

Problematic tasks:

• CQADupstackProgrammersRetrieval (almost all models have nans)

Problematic models:

• openai/text-embedding-3-large:
  • SprintDuplicateQuestions
  • SummEval
• openai/text-embedding-3-small:
  • SprintDuplicateQuestions
  • SummEval
• e5-mistral-7b-instruct:
  • STS22
• multilingual-e5-large-instruct:
  - [ ] ArxivClusteringP2P
  - [ ] BiorxivClusteringP2P
  - [ ] BiorxivClusteringS2S
  - [ ] DBPedia
  - [ ] HotpotQA
  - [ ] MedrxivClusteringP2P
  - [ ] MedrxivClusteringS2S
• snowflake/arctic-embed-m-v1.5:
  • AmazonCounterfactualClassification
  • ArguAna
  • MassiveIntentClassification
  • NFCorpus
  • SCIDOCS
  • SICK-R
  • STS12
  • STS13
  • STS14
  • STS15
  • STS16
  • STS17
  • SciFact
  • SprintDuplicateQuestions
  • TRECCOVID
  • ToxicConversationsClassification

[x-tabdeveloping]

@Muennighoff Anything pops out to you as something that has been run before but is missing? Also, can you run some or all of these?

[x-tabdeveloping]

There are also some problematic tasks in MTEB(deu) and MTEB(fra), I will look into those too

[x-tabdeveloping]

MTEB(fra)

Quite a few individual results are missing for a lot of models, maybe something wrong with the scraping or data loading.

Problematic tasks:

• MLSUMClusteringP2P almost all models missing

MTEB(deu)

Problematic tasks:

• TenKGnadClusteringP2P **almost all models missing **

In both benchmarks we're missing some tasks from 7B models.

[x-tabdeveloping]

MTEB(eng, beta)

Problematic tasks:

• HotpotQAHardNegatives majority of models missing
• MedrxivClusteringS2S.v2 most notably: stella, gte, SFR, GritLM8x7b, bge, UAE
• StackExchangeClusteringP2P.v2 almost all missing
• SummEvalSummarization.v2 almost all missing
• TwentyNewsgroupsClustering.v2 almost all missing

[x-tabdeveloping]

@Samoed @Muennighoff @KennethEnevoldsen I would really appreciate your help investigating and fixing these issues

[Muennighoff]

Great overview! Ofc can run anything that is missing if the model is loadable via mteb and ideally it's a python list of task & model pairs like here

mteb/scripts/running_model/create_slurm_jobs.py

Line 76 in e605c7b

model_names = [

[x-tabdeveloping]

I'll compile you a list

[KennethEnevoldsen]

we might also include this in the list: embeddings-benchmark/results#65 (review)

As a side note, we would like to run these models [list in issue] on the remaining tasks in the MTEB(Medical) benchmark. However, we initially held off due to API cost constraints. Do you have access to credits with these providers that we could use for this purpose? Alternatively, would it be possible for you to run them on your side?

(won't have too much time to look into this as I am at the neurips conference, but I will take a closer look once I get back)

[x-tabdeveloping]

Well, I have compiled a list of all task results that are missing for all benchmarks in the new leaderboard + all models that already show up in the leaderboard (have metadata and don't miss all results on a benchmark).
The list is very long, but luckily the most important models usually only miss one or two things. There also seem to be patterns, I'm wondering if it's got something to do with our versioning scheme for tasks.

I used the following script to get the missing results:

import json
from pathlib import Path

import pandas as pd
from tqdm import tqdm

import mteb
from mteb.leaderboard.table import scores_to_tables

benchmarks = mteb.get_benchmarks()

all_results = mteb.load_results()

results = {
    benchmark.name: benchmark.load_results(base_results=all_results)
    .join_revisions()
    .filter_models()
    for benchmark in tqdm(benchmarks, desc="Loading all benchmark results")
}


def to_pandas(gr_df) -> pd.DataFrame:
    cols = gr_df.value["headers"]
    data = gr_df.value["data"]
    return pd.DataFrame(data, columns=cols)


all_task_tables = {
    name: to_pandas(scores_to_tables(res.get_scores(format="long"))[1]).set_index(
        "Model"
    )
    for (name, res) in results.items()
}

missing_results = {}
for bench_name, table in all_task_tables.items():
    missing_results[bench_name] = {}
    for model_name, model_res in table.iterrows():
        nas = model_res.loc[model_res.isna()].index.to_list()
        if nas:
            missing_results[bench_name][model_name] = nas

And got this file:
missing_results.json

In the following format:

{
    "<benchmark_name>": {"<model_name>": ["task_name1", "task_name2", ...]}
}

We'll probably have to prioritize some models over others

[x-tabdeveloping]

Here it is for only the top 50 models, this should be a bit more reasonable to run:
missing_important.json

And here it is as a list of model-task pairs, as requested @Muennighoff :
missing_model_task_list_important.json

[KennethEnevoldsen]

(see embeddings-benchmark/results#80)