Caching

[pat-s]

After dealing with the idea of caching in mlr recently, I think this is an important topic for mlr3.
It would be a core base feature and should be integrated right from the start.

While in mlr I just did it for caching filter values for now, we should think of implementing it as a pkg option and make it available for all calls (resample, train, tuning, filtering, etc).

Most calls (dataset, learner, hyperpars) are unique and caching won't have that much of an effect as for filtering (for which the call to generate the filter values is always the same and the subsetting happens afterwards).

However, it can also have a positive effect on "normal" train/test calls:

• If a run (resample, tuneParams, benchmark) errors and a seed is set, the user can just rerun and profit from the cached calls
• For tuning methods like grid search settings might be redundant more often and the user can profit from caching
• Most often it will apply for simple train/test calls without tuning.

I've added a function delete_cache() and get_cache_dir() in my mlr PR to make the cache handling more convenient. We could think about a own class cache for such things.

Please share your opinions.

[mllg]

I think caching would be best implemented in mlr3pipelines. This way we could cache any preprocessing step.

[berndbischl]

After dealing with the idea of caching in mlr recently, I think this is an important topic for mlr3.

agreed

[berndbischl]

I think caching would be best implemented in mlr3pipelines. This way we could cache any preprocessing step.

i think so too. and this should really be properly defined in a design doc,as this is pretty hard to get right. for this we need a little bit more progress on mlr3pipelines.

[berndbischl]

shall we move the issue?

[berndbischl]

• If a run (resample, tuneParams, benchmark) errors and a seed is set, the user can just rerun and profit from the cached calls

do you mean for degugging purposes? seems potentially nice then, but i really would hold that separate as usecase and regarding technical solutions

[berndbischl]

like i said above to move forward we need a formal definition / proposal how caching would work in mlr3

[mb706]

Some Thoughts:

• mlr3 makes caching potentially easier because it provides hashes for us, although see Hashing might not work reliably for caching in pipelines mlr3#132
• Caching has larger benefit for some PipeOps (e.g. filtering) than others (e.g. scaling); important is the trade-off between how much memory the cached state takes vs. how long the computation takes. There should probably be a way for the user to change caching behaviour depending on his use case.
• In some cases (e.g. Filtering) it would be easiest to just cache the $state of a PipeOp; if a known input task is seen on $train(), one can then just call $predict() with the cached $state instead. This doesn't work for all PipeOps; it doesn't work for PipeOpLearnerCV.

[pfistfl]

tackled in #382