Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop using fork method to spawn processes #1600

Open
progval opened this issue Oct 19, 2024 · 0 comments
Open

Stop using fork method to spawn processes #1600

progval opened this issue Oct 19, 2024 · 0 comments

Comments

@progval
Copy link
Owner

progval commented Oct 19, 2024

Python 3.14 switches multiprocessing's default start method from fork to forkserver, which requires target functions to be picklable. This is because fork() is unsafe in threaded programs, like Limnoria.

However, Limnoria currently strongly depends on the fork() method (and explicitly opts into it in order to support Python >= 3.14), so that plugins can use commands.process() to call functions defined in plugins, despite plugins not being in Python's import path through:

https://github.com/progval/Limnoria/blob/54c09809786db7a6468c48dedc788287fbcded72/src/plugin.py

With the forkserver method, this wouldn't work because when a plugin calls commands.process(target=f) where f is a function defined in the plugin, the forkee would try to import the plugin (because it is needed to unpickle f), and fail.

Even if we could somehow make the forkee import all plugins and unpickle of Limnoria's state, this would be very wasteful, especially considering some plugins run commands.process() very often.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant