[BUG] Parallel states crash when context contains unpicklable objects on spawning platforms

[lkubb]

Description
If the context dictionary contains unpicklable objects when a parallel state is launched on a spawning platform, the execution fails with a TypeError.

The __context__ is often used as a cache for database connections or - in the case of salt.modules.match - a LazyLoader instance.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• onedir packaging

On bare-metal macOS with official 3007.1 package.

Note that Windows is a spawning platform as well.

Steps to Reproduce the behavior

# /opt/srv/salt/fail.sls

{%- do salt["match.glob"]("foo") %}

foo:
  test.nop:
    - parallel: true

salt-call state.apply fail

Expected behavior
Works.

Screenshots

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/opt/salt/lib/python3.10/site-packages/salt/state.py", line 2424, in call
    ret = self.call_parallel(cdata, low)
  File "/opt/salt/lib/python3.10/site-packages/salt/state.py", line 2280, in call_parallel
    proc.start()
  File "/opt/salt/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/salt/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/salt/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/opt/salt/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/salt/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/salt/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/salt/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

local:
----------
          ID: foo
    Function: test.nop
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/salt/lib/python3.10/site-packages/salt/state.py", line 2424, in call
                  ret = self.call_parallel(cdata, low)
                File "/opt/salt/lib/python3.10/site-packages/salt/state.py", line 2280, in call_parallel
                  proc.start()
                File "/opt/salt/lib/python3.10/multiprocessing/process.py", line 121, in start
                  self._popen = self._Popen(self)
                File "/opt/salt/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
                  return _default_context.get_context().Process._Popen(process_obj)
                File "/opt/salt/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
                  return Popen(process_obj)
                File "/opt/salt/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
                  super().__init__(process_obj)
                File "/opt/salt/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
                  self._launch(process_obj)
                File "/opt/salt/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
                  reduction.dump(process_obj, fp)
                File "/opt/salt/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
                  ForkingPickler(file, protocol).dump(obj)
              TypeError: cannot pickle '_thread.RLock' object
     Started: 23:02:14.890844
    Duration: 2.649 ms
     Changes:

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:27:26) [Clang 15.0.0 (clang-1500.0.40.1)]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: darwin 23.6.0
        locale: utf-8
       machine: arm64
       release: 23.6.0
        system: Darwin
       version: 14.7 arm64

Additional context
While arbitrary data in the context dict is a tricky situation in general, I think implementing a workaround by just catching the error and filtering any unpicklable objects would be less bad than just outright crashing.