Is it possible to tune threshold for binary classifiers?

[cjm715]

.... or set the threshold to a fixed value in a tuning instance?

[berndbischl]

this is currently not supported directly in mlr3tuning, but definitely in scope as a next feature
@mllg @mb706 what do we currently have in mlr3 here and in mlr3pipelines?

[mllg]

There is another issue about this somewhere. We need to arrange a call to decide what to do with other predict-time hyperparameters.

[mllg]

There is is now a prototype to tune a single prediction in the threshold branch.

[berndbischl]

is it documented how to set thresholds simply with models? can you link it here for reference pls?
also my feeling is that threshold optimization should be a PipeOp in mlr3pipelines, so we must dicuss this with @mb706

[mb706]

This should be a PipeOp and would work relatively efficiently when we implement #212. There is a small problem here: This would not be as efficient as it could be if the Learner that produces the predictions has a slow predict() function. Ideally we would want some functionality that notices that only parameters of the PipeOp after the Learner have changed, and that evaluates the PipeOp$predict() multiple times, but the Learner$train() only once.

[pfistfl]

Started implementation in mlr-org/mlr3pipelines#356.

[sebffischer]

this is implemented via po("tunethreshold")