FastEval uses model-specific prompt templates to prompt the model.
This is different from other tools that prompt all models the same way.
Using model-specific prompt templates makes the evaluation closer to how the models will actually be used in practice.
However, using model-specific prompt templates also means that the corresponding template must be implemented for the model that you want to evaluate.
Luckily, many models share the same prompt template and FastEval implements various of them.
The following is a list of supported templates & other model types in FastEval.
The model type can be specified with the -t flag, e.g. -t alpaca-with-prefix.
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
[user_input]
### Response:
[assistant_output]
Example models that make use of this prompt template:
### Instruction:
[user_input]
### Response:
[assistant_output]
Example models that make use of this prompt template:
<|im_start>system
[system_message]<|im_end|>
<|im_start|>user
[user_input]<|im_end|>
<|im_start|>assistant
[assistant_output]<|im_end|>
Example models that make use of this prompt template:
SYSTEM: [system_message]
USER: [user_input]
ASSISTANT: [assistant_output]
Example models that make use of this prompt template:
User: [user_input]
Assistant: [assistant_output]
Example models that make use of this prompt template:
A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
### Human: [user_input]
### Assistant: [assistant_output]
Example models that make use of this prompt template:
[INST] <<SYS>>
[system_message]
<</SYS>>
[user_input] [/INST] [assistant_output]
Example models that make use of this prompt template:
<|prompter|>[user_input]<|endoftext|><|assistant|>[assistant_output]<|endoftext|>
For llama models, the <|endoftext|> is replaced with </s> instead.
Example models that make use of this prompt template:
- OpenAssistant/oasst-sft-1-pythia-12b
- OpenAssistant/pythia-12b-sft-v8-7k-steps
- OpenAssistant/oasst-sft-7-llama-30b
- OpenAssistant/falcon-40b-sft-top1-560
- OpenAssistant/falcon-40b-sft-mix-1226
User: [user_input]<|end_of_turn|>Assistant: [assistant_output]<|end_of_turn|>
Example models that make use of this prompt template:
### System:
[system_message]
### User:
[user_input]
### Assistant:
[assistant_output]
Example models that make use of this prompt template:
<|system|>
[system_message]<|end|>
<user>
[user_input]<|end|>
<|assistant|>
[assistant_output]<|end|>
FastEval can also use fastchat as a backend for inference. In this case, the conversation is directly sent to fastchat which will then actually convert it to a prompt according to its own logic and run it through the model.
Note that while using this model type is easy, it will have cons:
- Reproducability will be worse because fastchat changes some prompt templates from time to time.
- No data parallel evaluation is currently implemented for this backend.
- Only vLLM can be used for fast evaluation. Using text-generation-inference is not supported.
Example models that make use of this model type:
If you choose this model type, then the model name will refer to an OpenAI model instead of a huggingface model or a local path.
Example models that make use of this model type:
- gpt-3.5-turbo-0301
- gpt-3.5-turbo-0613
- gpt-4-0613