what means reworded in the prompt directory?

[Connor-Shen]

it seems that there exists reworded dataset in prompt, but are not explained in the hf dataset

[arjunguha]

The "reworded" prompts are the MultiPL-E prompts that everyone uses. If you're benchmarking a model for comparison, use the reworded prompts and not the other variations. (Or use the bigcode-evaluation-harness, which uses reworded by default.)

Here is a quote from the MultiPL-E paper (Section 6A):

Our compilers target multiple distinct regions of the prompt for each problem. We explore the impact of each component in an ablation study of our MultiPL-HumanEvalbenchmark suite with Codex. We ran four versions of the MultiPL-HumanEval prompts, with some or all regions translated:

Original Prompt: does not translate doctests or natural language terminology (e.g. prompts as in HumanEval);

Test-only Translation: translates doctests but not Python-specific terminology;

Full Translation: translates unit tests, doctests, and Python-terminology in the prompt; and

No Doctests: removes doctests and does not translate natural language terminology.

The reworded prompts are the "Full Translation".