What exactly does the ELPD difference mean and show in az.plot_compare?

[jankaWIS]

Hello,

My question may be silly and simple but I'm struggling to understand what the reason is behind having the plot_ic_diff turned on by default in az.plot_compare, what exactly this 'triangle' shows and how to interpret it. My understanding is that it is simply a difference between ELPD between each model and the top-ranked model (see az.compare docs), although the docs of plot_compare says it's "standard error of the difference in ELPD between each model and the top-ranked model". But why would I want to plot this?

My understanding was that when I want to compare two models and say they're statistically significantly different, it'd be enough to check whether their ELPD distributions overlap, i.e. whether the SD in the plot overlap. For instance, if the mean of one model's ELPD is in SD of the other, they're not significantly different, such as in the example of arviz docs:

On the other hand, if the distributions of the different models don't overlap/contain mean of the other distribution in its SD, they're significantly different, like in the Bambi docs example (maybe model 3 and model 2 are not, hard to say from the plot but definitely model 1 is different from the other two):

So why would I need to plot or show the ELPD difference? What does it mean and bring?

Thank you for your help and advice.

[aloctavodia]

We implemented plot_compare from the first edition of Statistical Rethinking. Here you have a screenshot

Nevertheless, we are working on a new version of ArviZ. We have simplified the plot in the new version, and we follow more closely the recommendations by Vehtari et all (see for example https://mc-stan.org/loo/articles/online-only/faq.html#se_diff). This is the plot_compare in the new version https://arviz-plots.readthedocs.io/en/latest/gallery/plot_compare.html. The grey band shows a difference of 4 units mentioned by Vehtari et al

Notice that the "new ArviZ" is still under development and functionality is still not on par with the current stable version of ArviZ.

[jankaWIS]

Hi @aloctavodia and thank you for your answer. I'm, however, not sure I understand. What the snapshot suggest is that to compare the two models, I cannot simply look at the 'black lines', i.e. the SE of the WAIC, because those might be correlated. Firstly, I'm not sure if this applies to ELPD estimated by PSIS-LOO as well. Secondly, what exactly then determines if the models are or are not significantly different? What condition has to be satisfied for the triangles/grey lines/black lines?

Lastly, maybe I'm reading it wrong but from the reference you provided:

tl;dr When the difference (elpd_diff) is larger than 4, the number of observations is larger than 100 and the model is not badly misspecified then normal approximation and SE are quite reliable description of the uncertainty in the difference. Differences smaller than 4 are small and then the models have very similar predictive performance and it doesn’t matter if the normal approximation fails or SE is underestimated (Sivula, Magnusson and Vehtari, 2020).

these 4 units are one of the conditions that need to be checked in order to know whether the SD estimate can be trusted, it does not say anything about a meaningful difference between the models (except that they have a very similar predictive performance).

Thank you.

[aloctavodia]

I agree that the plot should be simpler and the interpretation should be focused on the elpd point estimates and their SE, that's the reason we make that the default in the new ArviZ. Maybe we should also make that the default on the stable version. On a personal note, that the way I have been using that plot for a while. About the 4 units difference, I think the point is that the difference is smaller than 4 units then the models have very similar performance, so it is of little importance if the SE estimates are reliable or not. Maybe this wording is more clear https://mc-stan.org/loo/articles/online-only/faq.html#modelselection (last bullet point)