fix: Add Korean AutoRAGRetrieval

[yjoonjang]

Adding datasets checklist

Reason for dataset addition:
This is a Korean Retrieval benchmark dataset covering 5 domains: Finance, Public, Medicine, Law, Commerce

• I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • {"ndcg_at_1": 0.29825, "ndcg_at_5": 0.3973, "ndcg_at_10": 0.42296}
  • intfloat/multilingual-e5-small
    • {"ndcg_at_1": 0.59649, "ndcg_at_5": 0.73205, "ndcg_at_10": 0.75466}
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[yjoonjang]

Hello, I tried to contribute this AutoRAGRetrieval task, but I got some errors while testing. Can I please get some help?
My codes are in mteb/tasks/Retrieval/kor/AutoRAGRetrieval.py

Thank you.

[KennethEnevoldsen]

If the metadata is not filled out you will get an error in the tests. Let me know if there are any problems with these.

[KennethEnevoldsen]

@yjoonjang I think we are almost there. Will you complete the checklist for adding a new dataset?

I also want to double check that annotations are human and not LLM generated?

[yjoonjang]

Hello, @KennethEnevoldsen

• I'm sorry, but I when I try to test with the dataset AutoRAGRetrieval, I still get this error: "KeyError: 'AutoRAGRetrieval' not found. Did you mean: DuRetrieval?"

  • My command was this: mteb run -m sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 -t AutoRAGRetrieval
  • I added my dataset name to Retrieval/__init__.py, but I don't know why I get this issue. Cand I get some help please?

• And yes, this dataset is not LLM-generated.

[Samoed]

I tried to run your code and maybe you need to reinstall your package to fix your problems, because I can run task like this mteb run -m intfloat/multilingual-e5-small -t AutoRAGRetrieval

[yjoonjang]

I tried to run your code and maybe you need to reinstall your package to fix your problems, because I can run task like this mteb run -m intfloat/multilingual-e5-small -t AutoRAGRetrieval

I still had the same error, but I ran the test by python code.
I updated the scores on the checklist !!
It would be great if you could check it.

Thank you, @KennethEnevoldsen @Samoed

[yjoonjang]

I have deleted the descriptive_stats in TaskMetadata field and made it to a json file with running calculated_metadata_metrics.

[Samoed]

I got these results
sentence-transformers__paraphrase-multilingual-MiniLM-L12-v2 AutoRAGRetrieval.json
intfloat__multilingual-e5-small AutoRAGRetrieval.json

[yjoonjang]

I got these results sentence-transformers__paraphrase-multilingual-MiniLM-L12-v2 AutoRAGRetrieval.json intfloat__multilingual-e5-small AutoRAGRetrieval.json

This corresponds with the results I attatched on the checklist above!

[Samoed]

Results for the e5-small model are slightly different. I suspect this is because I overlooked that the model should be initialized with model = mteb.get_model(model_name) rather than directly with SentenceTransformer. Using SentenceTransformer directly would skip the prompts.

[yjoonjang]

Results for the e5-small model are slightly different. I suspect this is because I overlooked that the model should be initialized with model = mteb.get_model(model_name) rather than directly with SentenceTransformer. Using SentenceTransformer directly would skip the prompts.

Oh yes. I tested with model = mteb.get_model(model_name) and got the same reslult as yours.
I don't really know why. Because e5-small doesn't need prompts.
Do you have some opinions?
And also, is this the end of contribution, or do I have to do anything else ?

[KennethEnevoldsen]

I don't really know why. Because e5-small doesn't need prompts.

e5 does use prompts though they are minimal (e.g. "query: " ).

[KennethEnevoldsen]

Things looks good on my end - @Samoed feel free to merge this in if you feel the same

[yjoonjang]

Thank you so much for your help.
Is my point automatically updated, or do I have to add my contribution to points folder ?

[KennethEnevoldsen]

@yjoonjang we no longer take point for the MMTEB contribution (it has already been submitted). However you will appear as a contributor on MTEB.