Potential fix for numerical instabilities for corr and partial_corr: z-normalize

[wolfgang-ganglberger]

Hi Raphael,

Big fan of your work and of this package, thanks so much. I was wondering if you have thoughts on numerical stability. The correlation and partial correlation can run into numerical instabilities when scales of variables are very different (factor 1e5 difference approximately). The functions do throw an error and the p-value is set to nan, however one fix could be to z-normalize the data before performing the matrix computations. Correlation and partial correlation functions are independent of scale.

See the screenshot for an example.

I think the pro is that it makes computations usually more robust, perhaps preventing people to use wrong results when numerical instabilities occur.
A con might be it also does not solve all instabilities, e.g. if you have a variable with 0 standard deviation.

[raphaelvallat]

Hi @wolfgang-ganglberger,

Thanks for your kind words and the detailed proposal!

I think that sounds like a good approach to me, we just need to make sure not to overwrite the input data with the z-scored version of the variables.

Please feel free to submit a PR and we can continue the conversation there!

Thanks
Raphael

[raphaelvallat]

Revisiting this — after second thought, I would perhaps prefer to raise a warning if the scale of the input variables differ by more than 1e5, and ask the user to manually rescale or standardize their data before using Pingouin.

[wolfgang-ganglberger]

Right, I agree, that works too. It could also be some flag argument with default no scaling and warning. This issue probably does not occur that often anyway so raising a warning sounds fine and hands the issue back to the user to check and scale their data.