Skip to content

Commit

Permalink
fix: change "group" to "tag" for mmlu_branch task config
Browse files Browse the repository at this point in the history
"group" is being deprecated in the task config as
of lm-eval version 0.4.4. "tag" should be used instead.

For more information see "Backwards Incompatibilities" at:
https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.4

Signed-off-by: Ali Maredia <[email protected]>
(cherry picked from commit e87062c)
  • Loading branch information
alimaredia authored and mergify[bot] committed Oct 14, 2024
1 parent 4ca3fd6 commit aa08ac8
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/instructlab/sdg/eval_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ def _create_mmlu_evaluation_task(task_name, eval_data_file_path, yaml_file_path)
}
],
"dataset_kwargs": {"data_files": {"test": eval_data_file_path}},
"group": "mmlu_pr",
"tag": "mmlu_pr",
}
with open(yaml_file_path, "w", encoding="utf-8") as yaml_file:
yaml.dump(task_yaml, yaml_file, default_flow_style=False)
Expand Down

0 comments on commit aa08ac8

Please sign in to comment.