workflow for adjusting training parameters makes experimentation difficult

[rusheb]

it is difficult to train different models because of the steps required to update configs. this involves editing the configs in config.py by modifying or adding items to _GPT_CONFIGS_LIST or _TRAINING_CONFIGS_LIST. changing python files affects version control which adds further inconvenience

Not sure on the best way to fix this. Some options:

• setting training parameters from the command line. Though, there may be so many parameters that also makes this inconvenient
• use notebooks?
• can we use wandb to make this easier? e.g. by leveraging artifacts?
• config management libraries e.g. hydra?

Requires more research

[rusheb]

@cmathw added this ticket based on what you were telling me today. Please feel free to add/clarify details

[mivanit]

I think this is mostly resolved via #177 and the ability to modify config params via CLI