Is there a way to Train/Finetune LLM using determined

[ASH1998]

Hi, I am new to determined, and I like the all-in-one solution which determined is building towards.
I wanted to know more about if it is possible to train/finetune a LLM model using determined core API. I have seen the blogs on inferencing using gpt2 and openLLAMA by @coreystatendet, but essentially it was using HF library to inference and monitor using determined.

So basically I have the below questions :

1. Is there a doc on training of LLM using determined or is it just monitoring for now ?
2. Is there a plan to support quantised models ?
3. What about inferencing on scale for a cluster of servers using determined master-agent nodes.

thanks!

[aciborowska]

Hello, and thanks for joining us here!

1. For a tutorial on training/fine-tuning LLMs in Core API take a look at the hf_language_modeling example.
   It uses HF Trainer for training/fine-tuning, while reporting metrics and checkpoints to Determined. It also supports hyperparameter search and gives a few deepspeed configs in case you need model parallel training.
   Feel free to build on top of this example, by e.g., experimenting with different trainers.
   Determined allows you to train an LLM using Core API with any framework of your choice, however, in such scenarios, you are responsible for reporting metrics, checkpoints, and low-level synchronization between workers. The example above takes care of all of that by integrating with HF Trainer through Callback mechanism, so that as long as you add the callback to the HF Trainer, you don't need to make any additional changes to your code and can run it in Determined.

2. Determined allows you to work with any arbitrary model (quantized or not), however, you will need to decide on the framework you want to use to work with it and load it correctly. I would start by looking into loading quantized models to HF.

3. We don't have a general built-in inference support just yet. We support offline batch inference (example 1, example 2, video) if that's what are you interested in.
   To set up more general inference on your own, you can follow Corey's tutorial on how to use HF and DeepSpeed Inference.