added-ICI-measure

[Lee-xli]

Hi mlr3proba team:

Just a couple of notes and thoughts (My apologies that I realised these after committing/pushing the document):

• Line 30: pred = t(prediction$distr$cdf(tm))

I noticed that using pred = t(1 - prediction$distr$survival(tm)) will yield same results. The $survival prediction is perhaps preferred.

• ICI plots:

I'm not sure if this is the right place to generate these plots. At the moment, I'm unable to add a proper title. The code generates the plots but terminates with an error. It worked for me for the time being, but it may be argued that these plots are not necessary for benchmarking comparisons, and thus should be made available only in a more general setting.

Thank you!

[bblodfon]

Thanks @Lee-xli! Just a few comments before I take over the implementation:

• Note that the measure should be a scalar/numeric value, not a plot. The latter we can add separately in the autoplot function.
• I took a brief look at Austin's paper, I think the ICI is what you want to implement right? (E50 and E90 as well?)
  

The idea is: from the survival distr prediction of a survival learner, construct the smoothed calibration curves (same code in autoplot function pretty much) => calculate the score(s) as in the paper above, right?

[Lee-xli]

Thanks @bblodfon. The current code should be able to calculate ICI (with default plot=F) or generate plots (with no ICI calculated plot=T). I didn't implement E50 and E90 here as ICI alone (for our use case) is enough as an additional measure to existing measures such as IBS and others (such as alpha and beta). ICI is calculated with no errors/problems, only needs time specified.

"The idea is: from the survival distr prediction of a survival learner, construct the smoothed calibration curves (same code in autoplot function pretty much) => calculate the score(s) as in the paper above, right?"

Yes, that is correct.

Hope this answers your questions. Let me know if I can help further.

[Lee-xli]

Incidentally, we initially explored this method primarily for its plot functionality. The BMJ paper, along with many other references on clinical prediction models (such as those by Harrell and Steyerberg), places great emphasis on the calibration plot during the validation process. The ICI measure was a secondary consideration. We found the ICI plot worth pursuing because very few calibration plots claim to: 1. construct a plot without binning (e.g., dCalib can be plotted, but this method relies on grouping subjects into deciles), and 2. be applicable in machine learning. The Austin paper suggested that this could potentially be a versatile method, with an example applied in random forests.