in Jupyter, Mapply has crashed because of insufficient RAM but jupyter/mapply keeps acting like it is still working.

[thoughtfuldata]

What you were trying to do (and why)

I am trying to use mapply to process dataframe rows. Each row requires a very memory intensive operations( i.e Tree Edit Distance).

What happened (including reproducible example)

Even Though, the operation has failed because of insufficient RAM. Jupyter notebook and cell keeps stating it is still working for hours. No error message anywhere.

I will be checking if this issue happens in a regular python script and creating a reproducible example soon.

What you expected to happen

I expect mapply to fail and print out the reason it failed. I know that it is failing because I am monitoring with htop

Step-by-step reproduction instructions

Run mapply on a memory intensive operation that causes the usage of more than available RAM or in my case WSL RAM ( currently at 58.9GB).

[ddelange]

Hi 👋

I have a feeling that's an upstream bug @ pathos, or at least we as end users would expect our multiprocessing_imap to raise an exception when one of the (forked) subprocesses dies.

Can you try to create a minimal reproducible example that we can post there?

[ddelange]

Closing in favour of uqfoundation/pathos#265