Multiprocessed feature extraction is not supported on Windows

[jvdd]

Currently we do not support multiprocessed feature extraction on Windows, and we do not plan (in the near future) to work on this.

What do we use for multiprocessing?
For our multiprocessing functionality we use multiprocess, a fork of multiprocessing that uses under the hood dill for serialization. In contrast to multiprocessing, this package allows multiprocessing lambdas, locally defined functions, imported functions and furthermore sharing of (simple data) in memory.
We believe that users should be able to conveniently define their functions as imports of packages, inline defined functions, or even lambdas. Therefore we will not switch to multiprocessing.

What is the problem with Windows?
Serializing in Windows happens differently than on Linux (and MacOS).

So why will we not set dill.settings["recursive"] = True if the OS is Windows?
Although this works in most cases, we do not believe that the outcome is what we want;

1. This tends to be slower.
2. This fails when multiprocessing a feature collection that contains nested functions (see: closured function fails with RecursionError uqfoundation/dill#211). For example tsfresh its feature functions contain such nested functions.

Some related issues (I think);

• RecursionError for local functions, referencing themselves. uqfoundation/dill#399
• Recursive functions with recurse=True not serializing uqfoundation/dill#344

NOTE: multiprocessing is still supported on Linux & MacOS 😄

[jvdd]

If the community has a proper fix for this, feel free to create a PR! 😃

(This is currently a low priority feature for the core devs as we are all working on linux.)

[jvdd]

After revisiting this open issue with the latest version of dill supporting nested function pickling, we can now support multiprocessing on Windows. However, I noticed in PR #91 that multiprocessing on Windows can be quite slow compared to sequential feature extraction, at best showing only a 5% improvement. As a result, I've decided to close PR #91 and leave the implementation as is.