decoding parameters (e.g., temperature) for Gemma-2?

[iseesaw]

Hello, How should I set the decoding parameters (e.g., temperature) for Gemma-2? My result is about ~50.0, far from the benchmark of 76.

[xiamengzhou]

Hi, please refer to the parameters in this script: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/models_configs/gemma-2-9b-it-SimPO/configs.yaml

[MaoXinn]

Hi, i also met this problem. I only got WR/LC as follows:
54.47204968944099,59.969975205397596

here is my evaluation config:

Gemma-2-Aligned-simpo:
completions_kwargs:
batch_size: 900
max_new_tokens: 4096
model_kwargs:
dtype: bfloat16
model_name: princeton-nlp/gemma-2-9b-it-SimPO
stop_token_ids:
- 1
- 107
temperature: 0.5
top_p: 1.0
fn_completions: vllm_local_completions
pretty_name: gemma-2-9b-it-SimPO
prompt_template: ./eval_config/gemma2_prompt.txt

The only different is that i remove "do_sample: true".

I reviewed your config and your conversation with the AE author on GitHub, and now I’m quite confused.
Even after downgrading AE2 to 0.62, I still couldn’t run it based on the configuration you provided. The main problem seems to lie with beam search. Should I enable beam search? If so, the temperature must be set to 0, but I don’t know what the beam size should be.

Thank you~

[LotuSrc]

Maybe you use alpaca_eval_gpt4_turbo_fn. In this setting, the result is close to the result you reported.

[xiamengzhou]

@MaoXinn It’s a bit tricky to interpret what happened based on the information you provided. How about we troubleshoot it step by step? You could begin by running the evaluations with the outputs we provided on AlpacaEval and check if you can get a similar score first.