Add automatic tracking of validation loss and an optional recommendation

[EssamWisam]

I can access training_losses from report(mach) but I was surprised that there is no validation_losses (although these are certainly computed with a holdout or resampling strategy).

I understand that I can use callbacks for this but it felt asymmetric (and less friendly) to only find training_losses but no num_epochs_finished (although it can be derived from the log anyway) or validation_losses.

As for the optional recommendation, currently the argument controls allows two types of functionality:

• Specifying conditions to stop training
• Specifying callbacks to call while training

I think it's more intuitive to have two separate arguments. Maybe stop_conditions=[...] for the former and callbacks=[...] for the latter.

[EssamWisam]

Thanks for the great functionality in this package. Such a key distintion of MLJ to be this flexible for iterated models.

[ablaom]

Great to get this feedback, thanks.

I can access training_losses from report(mach) but I was surprised that there is no validation_losses (although these are certainly computed with a holdout or resampling strategy).

I understand that I can use callbacks for this but it felt asymmetric (and less friendly) to only find training_losses but no num_epochs_finished (although it can be derived from the log anyway) or validation_losses.

I agree adding final loss and n_iterations (instead of the proposed num_epochs_finished for consistency with this API) to the report makes sense. Maybe there was some reason we didn't do this, but it escapes me at the moment.

(although it can be derived from the log anyway)

I actually don't find it that convenient to extract it from the log and there is an existing issue related to this.

As for the optional recommendation, currently the argument controls allows two types of functionality:

Specifying conditions to stop training
Specifying callbacks to call while training

I think it's more intuitive to have two separate arguments. Maybe stop_conditions=[...] for the former and callbacks=[...] for the latter

Not quite true. What about the Step() control? That's not either really.

I do see how calling a purely reporting-type callback creates some cognitive dissonance, because it's not actually interfering with the iteration. However, some callbacks can be used to change the iteration materially, because it can mutate a model learning rate, for example.

For my part, then , I'm not sure a breaking change is warranted here.

[EssamWisam]

Thank you for the comprehensive response. Highly valued. Yes, I agree it's not worth a breaking change and callbacks can interfere with training it was just a thought; users can anyway split them in their variables if they want.

That said, I would be quite happy to see both training and validation losses being recorded in the log. It would make basic examples with iterated models look much more friendly.