[BUG] regression in 3005.x - salt hangs after highstate for windows minion

[TheBigBear]

Description
Since the upgrade to the latest version 3005.x on the master the windows minions no longer send highstate output back to linux master.

Setup
upgrade master to 3005.x from 3004.2 and any previously working test only
salt mywinminion state.highstate test=true or an actual salt mywinminion1 state.highstate times out and only results in a generic:

mywinminion1:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20221114161118227117

But this job output never gets any data even hours after having been run.

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

on linux salt master output is:
salt --versions
Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Sep 13 2022, 07:19:15)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 22.0.3
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: almalinux 8.7 Stone Smilodon
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-425.3.1.el8.x86_64
        system: Linux
       version: AlmaLinux 8.7 Stone Smilodon

and windows minion versions output is:

salt-call --versions
C:\Program Files\Salt Project\Salt\bin\lib\site-packages\_distutils_hack\__init_
_.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.8.15 (tags/v3.8.15:44adf8a, Nov  8 2022, 17:20:07) [MSC v.1929
 64 bit (AMD64)]
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 22.0.3
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 2016Server
        system: Windows
       version: 2016Server 10.0.14393 SP0 Multiprocessor Free

Additional context
Add any other context about the problem here.

[OrangeDog]

Please fill in the salt --versions-report above.

[TheBigBear]

Please fill in the salt --versions-report above.

ok, done.

[OrangeDog]

This appears to be the same issue as #62937, but you already have the update with PyZMQ 22.0.3.

Has the minion service definitely been restarted after the upgrade?

[TheBigBear]

This appears to be the same issue as #62937, but you already have the update with PyZMQ 22.0.3.

Has the minion service definitely been restarted after the upgrade?

Yes, the minion service has definitely been restarted.

Yes, I agree it looks or feels like the same, but I am told that the locally run salt-call state.highstate on windows minion and the master run salt state.highstate on linux are not really related.

And strangely I have to agree, as I have seen myself that the manual direct upgrading of salt-call pip.install pyzmq==22.0.3 back on 3005.1-1 and 3005.1-2 on the windows minion had resolved the issue of having to kill it using Ctrl-C, but that local upgrade on a classic install had no benefit on the same highstate being run from linux master remote side.

( as per salt-users group email thread here: https://groups.google.com/g/salt-users/c/O5mAujEzjmc/m/XvCw0T-QBgAJ )

[TheBigBear]

@twangboy do you have any insights or suggestions on how I could debug this better to find the blocker of this highstate output?

[twangboy]

A similar stacktrace here, but with a syndic: #62577 (comment)

[twangboy]

@TheBigBear Do you happen to be on a Multi-master set up? I'm wondering about your order_masters setting. (#62599 (comment))

[TheBigBear]

@TheBigBear Do you happen to be on a Multi-master set up? I'm wondering about your order_masters setting. (#62599 (comment))

@twangboy, no we have a 'simple' single master setup.

[twangboy]

Do you have an example of the states you are running? In #62550 it was exposed by anything that communicated with the master fileserver, for example file.managed. Perhaps one of the states you're running are exposing another issue.

So far I have been unable to reproduce it.

[TheBigBear]

@twangboyI finally did what I should have done at the beginning. ;-) I let it display my "top.sls" for the windows minion and then went through all of the 6 states one by one to see which one actually has any issues.
And all but one worked just fine.
We are making use of the 'salt-formula' from 'https://github.com/saltstack-formulas/salt-formula.git'

salt mywinminion1 state.apply salt.minion test=true

produces following error output

mywinminion1:
    Passed invalid arguments to state.apply: can not serialize 'CommandExecutionError' object

        .. versionadded:: 2015.5.0

        This function will call :mod:`state.highstate
        <salt.modules.state.highstate>` or :mod:`state.sls
        <salt.modules.state.sls>` based on the arguments passed to this function.
        It exists as a more intuitive way of applying states.

        .. rubric:: APPLYING ALL STATES CONFIGURED IN TOP.SLS (A.K.A. :ref:`HIGHSTATE <running-highstate>`)

        To apply all configured states, simply run ``state.apply``:

        .. code-block:: bash

            salt '*' state.apply

        The following additional arguments are also accepted when applying all
        states configured in top.sls:

        test
            Run states in test-only (dry-run) mode

        mock
            The mock option allows for the state run to execute without actually
            calling any states. This then returns a mocked return which will show
            the requisite ordering as well as fully validate the state run.

            .. versionadded:: 2015.8.4

        pillar
            Custom Pillar values, passed as a dictionary of key-value pairs

            .. code-block:: bash

                salt '*' state.apply stuff pillar='{"foo": "bar"}'

            .. note::
                Values passed this way will override Pillar values set via
                ``pillar_roots`` or an external Pillar source.

        exclude
            Exclude specific states from execution. Accepts a list of sls names, a
            comma-separated string of sls names, or a list of dictionaries
            containing ``sls`` or ``id`` keys. Glob-patterns may be used to match
            multiple states.

            .. code-block:: bash

                salt '*' state.apply exclude=bar,baz
                salt '*' state.apply exclude=foo*
                salt '*' state.apply exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls_to_exclude'}]"

        queue : False
            Instead of failing immediately when another state run is in progress,
            queue the new state run to begin running once the other has finished.

            This option starts a new thread for each queued state run, so use this
            option sparingly.

        localconfig
            Optionally, instead of using the minion config, load minion opts from
            the file specified by this argument, and then merge them with the
            options from the minion config. This functionality allows for specific
            states to be run with their own custom minion configuration, including
            different pillars, file_roots, etc.

            .. code-block:: bash

                salt '*' state.apply localconfig=/path/to/minion.yml


        .. rubric:: APPLYING INDIVIDUAL SLS FILES (A.K.A. :py:func:`STATE.SLS <salt.modules.state.sls>`)

        To apply individual SLS files, pass them as a comma-separated list:

        .. code-block:: bash

            # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
            salt '*' state.apply stuff

            # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
            # and salt://pkgs.sls (or salt://pkgs/init.sls).
            salt '*' state.apply stuff,pkgs

            # Run the states configured in a more deeply nested directory such as salt://my/organized/stuff.sls (or salt://my/organized/stuff/init.sls)
            salt '*' state.apply my.organized.stuff

        The following additional arguments are also accepted when applying
        individual SLS files:

        test
            Run states in test-only (dry-run) mode

        mock
            The mock option allows for the state run to execute without actually
            calling any states. This then returns a mocked return which will show
            the requisite ordering as well as fully validate the state run.

            .. versionadded:: 2015.8.4

        pillar
            Custom Pillar values, passed as a dictionary of key-value pairs

            .. code-block:: bash

                salt '*' state.apply stuff pillar='{"foo": "bar"}'

            .. note::
                Values passed this way will override Pillar values set via
                ``pillar_roots`` or an external Pillar source.

        queue : False
            Instead of failing immediately when another state run is in progress,
            queue the new state run to begin running once the other has finished.

            This option starts a new thread for each queued state run, so use this
            option sparingly.

        concurrent : False
            Execute state runs concurrently instead of serially

            .. warning::

                This flag is potentially dangerous. It is designed for use when
                multiple state runs can safely be run at the same time. Do *not*
                use this flag for performance optimization.

        saltenv
            Specify a salt fileserver environment to be used when applying states

            .. versionchanged:: 0.17.0
                Argument name changed from ``env`` to ``saltenv``

            .. versionchanged:: 2014.7.0
                If no saltenv is specified, the minion config will be checked for an
                ``environment`` parameter and if found, it will be used. If none is
                found, ``base`` will be used. In prior releases, the minion config
                was not checked and ``base`` would always be assumed when the
                saltenv was not explicitly set.

        pillarenv
            Specify a Pillar environment to be used when applying states. This
            can also be set in the minion config file using the
            :conf_minion:`pillarenv` option. When neither the
            :conf_minion:`pillarenv` minion config option nor this CLI argument is
            used, all Pillar environments will be merged together.

        localconfig
            Optionally, instead of using the minion config, load minion opts from
            the file specified by this argument, and then merge them with the
            options from the minion config. This functionality allows for specific
            states to be run with their own custom minion configuration, including
            different pillars, file_roots, etc.

            .. code-block:: bash

                salt '*' state.apply stuff localconfig=/path/to/minion.yml

        sync_mods
            If specified, the desired custom module types will be synced prior to
            running the SLS files:

            .. code-block:: bash

                salt '*' state.apply stuff sync_mods=states,modules
                salt '*' state.apply stuff sync_mods=all

            .. note::
                This option is ignored when no SLS files are specified, as a
                :ref:`highstate <running-highstate>` automatically syncs all custom
                module types.

            .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

And the windows minion rendered salt.minion sls is:

salt mywinminion1 state.show_sls salt.minion

mywinminion1:
    ----------
    permissions-minion-config:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            - managed
            |_
              ----------
              order:
                  10004
    permissions-minion.pem:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pem
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        salt-minion-pki-dir
            - managed
            |_
              ----------
              order:
                  10006
    permissions-minion.pub:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pub
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        salt-minion-pki-dir
            - managed
            |_
              ----------
              order:
                  10007
    remove-old-minion-conf-file:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion.d\_defaults.conf
            - absent
            |_
              ----------
              order:
                  10003
    salt-minion:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        cmd:
            |_
              ----------
              name:
                  salt-call.bat --local service.restart salt-minion
            |_
              ----------
              bg:
                  True
            |_
              ----------
              onchanges:
                  |_
                    ----------
                    pkg:
                        salt-minion
                  |_
                    ----------
                    file:
                        salt-minion
                  |_
                    ----------
                    file:
                        remove-old-minion-conf-file
            - run
            |_
              ----------
              order:
                  10002
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion.d
            |_
              ----------
              template:
                  jinja
            |_
              ----------
              source:
                  salt://salt/files/minion.d
            |_
              ----------
              context:
                  ----------
                  standalone:
                      False
            |_
              ----------
              clean:
                  True
            |_
              ----------
              exclude_pat:
                  _*
            - recurse
            |_
              ----------
              order:
                  10001
        pkg:
            |_
              ----------
              name:
                  salt-minion-py3
            |_
              ----------
              require_in:
                  |_
                    ----------
                    service:
                        salt-minion
            - installed
            |_
              ----------
              order:
                  10000
        service:
            |_
              ----------
              enable:
                  True
            |_
              ----------
              name:
                  salt-minion
            |_
              ----------
              watch:
                  |_
                    ----------
                    file:
                        remove-old-minion-conf-file
            |_
              ----------
              order:
                  last
            - running
    salt-minion-pki-dir:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              makedirs:
                  True
            - directory
            |_
              ----------
              order:
                  10005

and for reference IF I run the relevant state directly on the windows minion itself using salt-call this is the output it produces:

salt-call state.apply salt.minion test=true

C:\Program Files\Salt Project\Salt\bin\lib\site-packages\_distutils_hack\__init_
_.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
[WARNING ] The group argument for C:\ProgramData\Salt Project\Salt\conf\minion h
as been ignored as this is a Windows system. Please use the `win_*` parameters t
o set permissions in Windows.
[ERROR   ] Invalid user/group or sid: root
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is
invalid.')
[ERROR   ] An exception occurred in this state: Traceback (most recent call last
):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is
invalid.')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\state.py",
 line 2276, in call
    ret = self.states[cdata["full"]](
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs
)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1276, in wrapper
    return f(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\states\fil
e.py", line 3025, in managed
    u_check = _check_user(user, group)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\states\fil
e.py", line 375, in _check_user
    uid = __salt__["file.user_to_uid"](user)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs
)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\modules\wi
n_file.py", line 521, in user_to_uid
    return __utils__["dacl.get_sid_string"](user)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs
)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1186, in get_sid_string
    principal = get_sid(principal)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1149, in get_sid
    raise CommandExecutionError("Invalid user/group or sid: {}".format(principal
))
salt.exceptions.CommandExecutionError: Invalid user/group or sid: root

[WARNING ] The group argument for C:\ProgramData\Salt Project\Salt\conf\pki\mini
on has been ignored as this is a Windows system. Please use the `win_*` paramete
rs to set permissions in Windows.
[ERROR   ] Invalid user/group or sid: root
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\utils\win_
dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is
invalid.')
[ERROR   ] Invalid user/group or sid: root

Passed invalid arguments: can not serialize 'CommandExecutionError' object.

Usage:

    .. versionadded:: 2015.5.0

    This function will call :mod:`state.highstate
    <salt.modules.state.highstate>` or :mod:`state.sls
    <salt.modules.state.sls>` based on the arguments passed to this function.
    It exists as a more intuitive way of applying states.

    .. rubric:: APPLYING ALL STATES CONFIGURED IN TOP.SLS (A.K.A. :ref:`HIGHSTAT
E <running-highstate>`)

    To apply all configured states, simply run ``state.apply``:

    .. code-block:: bash

        salt '*' state.apply

    The following additional arguments are also accepted when applying all
    states configured in top.sls:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    exclude
        Exclude specific states from execution. Accepts a list of sls names, a
        comma-separated string of sls names, or a list of dictionaries
        containing ``sls`` or ``id`` keys. Glob-patterns may be used to match
        multiple states.

        .. code-block:: bash

            salt '*' state.apply exclude=bar,baz
            salt '*' state.apply exclude=foo*
            salt '*' state.apply exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls
_to_exclude'}]"

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply localconfig=/path/to/minion.yml


    .. rubric:: APPLYING INDIVIDUAL SLS FILES (A.K.A. :py:func:`STATE.SLS <salt.
modules.state.sls>`)

    To apply individual SLS files, pass them as a comma-separated list:

    .. code-block:: bash

        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sl
s)
        salt '*' state.apply stuff

        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sl
s)
        # and salt://pkgs.sls (or salt://pkgs/init.sls).
        salt '*' state.apply stuff,pkgs

        # Run the states configured in a more deeply nested directory such as sa
lt://my/organized/stuff.sls (or salt://my/organized/stuff/init.sls)
        salt '*' state.apply my.organized.stuff

    The following additional arguments are also accepted when applying
    individual SLS files:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    concurrent : False
        Execute state runs concurrently instead of serially

        .. warning::

            This flag is potentially dangerous. It is designed for use when
            multiple state runs can safely be run at the same time. Do *not*
            use this flag for performance optimization.

    saltenv
        Specify a salt fileserver environment to be used when applying states

        .. versionchanged:: 0.17.0
            Argument name changed from ``env`` to ``saltenv``

        .. versionchanged:: 2014.7.0
            If no saltenv is specified, the minion config will be checked for an

            ``environment`` parameter and if found, it will be used. If none is
            found, ``base`` will be used. In prior releases, the minion config
            was not checked and ``base`` would always be assumed when the
            saltenv was not explicitly set.

    pillarenv
        Specify a Pillar environment to be used when applying states. This
        can also be set in the minion config file using the
        :conf_minion:`pillarenv` option. When neither the
        :conf_minion:`pillarenv` minion config option nor this CLI argument is
        used, all Pillar environments will be merged together.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply stuff localconfig=/path/to/minion.yml

    sync_mods
        If specified, the desired custom module types will be synced prior to
        running the SLS files:

        .. code-block:: bash

            salt '*' state.apply stuff sync_mods=states,modules
            salt '*' state.apply stuff sync_mods=all

        .. note::
            This option is ignored when no SLS files are specified, as a
            :ref:`highstate <running-highstate>` automatically syncs all custom
            module types.

        .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

and the same without the 'test=true' shows the same basically ( I think):

salt-call state.apply salt.minion

C:\Program Files\Salt Project\Salt\bin\lib\site-packages\_distutils_hack\__init_
_.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
[WARNING ] The group argument for C:\ProgramData\Salt Project\Salt\conf\minion h
as been ignored as this is a Windows system. Please use the `win_*` parameters t
o set permissions in Windows.
[ERROR   ] Invalid user/group or sid: root
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is
invalid.')
[ERROR   ] An exception occurred in this state: Traceback (most recent call last
):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is
invalid.')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\state.py",
 line 2276, in call
    ret = self.states[cdata["full"]](
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs
)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1276, in wrapper
    return f(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\states\fil
e.py", line 3025, in managed
    u_check = _check_user(user, group)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\states\fil
e.py", line 375, in _check_user
    uid = __salt__["file.user_to_uid"](user)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs
)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\modules\wi
n_file.py", line 521, in user_to_uid
    return __utils__["dacl.get_sid_string"](user)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs
)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\loader\laz
y.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1186, in get_sid_string
    principal = get_sid(principal)
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1149, in get_sid
    raise CommandExecutionError("Invalid user/group or sid: {}".format(principal
))
salt.exceptions.CommandExecutionError: Invalid user/group or sid: root

[WARNING ] The group argument for C:\ProgramData\Salt Project\Salt\conf\pki\mini
on has been ignored as this is a Windows system. Please use the `win_*` paramete
rs to set permissions in Windows.
[ERROR   ] Invalid user/group or sid: root
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt\utils\win_
dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is
invalid.')
[ERROR   ] Invalid user/group or sid: root
C:\Program Files\Salt Project\Salt\bin\lib\site-packages\_distutils_hack\__init_
_.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

Passed invalid arguments: can not serialize 'CommandExecutionError' object.

Usage:

    .. versionadded:: 2015.5.0

    This function will call :mod:`state.highstate
    <salt.modules.state.highstate>` or :mod:`state.sls
    <salt.modules.state.sls>` based on the arguments passed to this function.
    It exists as a more intuitive way of applying states.

    .. rubric:: APPLYING ALL STATES CONFIGURED IN TOP.SLS (A.K.A. :ref:`HIGHSTAT
E <running-highstate>`)

    To apply all configured states, simply run ``state.apply``:

    .. code-block:: bash

        salt '*' state.apply

    The following additional arguments are also accepted when applying all
    states configured in top.sls:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    exclude
        Exclude specific states from execution. Accepts a list of sls names, a
        comma-separated string of sls names, or a list of dictionaries
        containing ``sls`` or ``id`` keys. Glob-patterns may be used to match
        multiple states.

        .. code-block:: bash

            salt '*' state.apply exclude=bar,baz
            salt '*' state.apply exclude=foo*
            salt '*' state.apply exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls
_to_exclude'}]"

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply localconfig=/path/to/minion.yml


    .. rubric:: APPLYING INDIVIDUAL SLS FILES (A.K.A. :py:func:`STATE.SLS <salt.
modules.state.sls>`)

    To apply individual SLS files, pass them as a comma-separated list:

    .. code-block:: bash

        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sl
s)
        salt '*' state.apply stuff

        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sl
s)
        # and salt://pkgs.sls (or salt://pkgs/init.sls).
        salt '*' state.apply stuff,pkgs

        # Run the states configured in a more deeply nested directory such as sa
lt://my/organized/stuff.sls (or salt://my/organized/stuff/init.sls)
        salt '*' state.apply my.organized.stuff

    The following additional arguments are also accepted when applying
    individual SLS files:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    concurrent : False
        Execute state runs concurrently instead of serially

        .. warning::

            This flag is potentially dangerous. It is designed for use when
            multiple state runs can safely be run at the same time. Do *not*
            use this flag for performance optimization.

    saltenv
        Specify a salt fileserver environment to be used when applying states

        .. versionchanged:: 0.17.0
            Argument name changed from ``env`` to ``saltenv``

        .. versionchanged:: 2014.7.0
            If no saltenv is specified, the minion config will be checked for an

            ``environment`` parameter and if found, it will be used. If none is
            found, ``base`` will be used. In prior releases, the minion config
            was not checked and ``base`` would always be assumed when the
            saltenv was not explicitly set.

    pillarenv
        Specify a Pillar environment to be used when applying states. This
        can also be set in the minion config file using the
        :conf_minion:`pillarenv` option. When neither the
        :conf_minion:`pillarenv` minion config option nor this CLI argument is
        used, all Pillar environments will be merged together.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply stuff localconfig=/path/to/minion.yml

    sync_mods
        If specified, the desired custom module types will be synced prior to
        running the SLS files:

        .. code-block:: bash

            salt '*' state.apply stuff sync_mods=states,modules
            salt '*' state.apply stuff sync_mods=all

        .. note::
            This option is ignored when no SLS files are specified, as a
            :ref:`highstate <running-highstate>` automatically syncs all custom
            module types.

        .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

local:
    True

I hope and trust this helps you reproduce this?
And looking at it this way, I am now really wondering if the salt-formula is even trying to do the right thing here?

I am a bit surprised by it trying to use linux rights and permissions mechanisms and linux users like user 'root' and group 'root' to set permissions on a bunch of windows minion installation files and directories?

So @twangboyt is this actually two separate issues in the end?
One some sort of salt regression in 3005.x and the second an issue in the other salt-formula repo?

permissions-minion-config
    permissions-minion.pem
    permissions-minion.pub
salt-minion-pki-dir

[twangboy]

These may be the issue:

Passed invalid arguments to state.apply: can not serialize 'CommandExecutionError' object
Passed invalid arguments: can not serialize 'CommandExecutionError' object

[TheBigBear]

OK, I have reverted back to 3004.2-1 on a home/test/lab setup and it works, again.
So it is a 3005.x regression after-all.

Note: now it properly reports the user as 'system' under the windows minion. remember under the 3005.x it reported it as 'root' instead.

Here is the output of salt INT-UR-TEST-1 state.show_sls salt.minion

INT-UR-TEST-1:
    ----------
    permissions-minion-config:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion
            |_
              ----------
              user:
                  system
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            - managed
            |_
              ----------
              order:
                  10004
    permissions-minion.pem:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pem
            |_
              ----------
              user:
                  system
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        salt-minion-pki-dir
            - managed
            |_
              ----------
              order:
                  10006
    permissions-minion.pub:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pub
            |_
              ----------
              user:
                  system
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        salt-minion-pki-dir
            - managed
            |_
              ----------
              order:
                  10007
    remove-old-minion-conf-file:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion.d\_defaults.conf
            - absent
            |_
              ----------
              order:
                  10003
    salt-minion:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        cmd:
            |_
              ----------
              name:
                  salt-call.bat --local service.restart salt-minion
            |_
              ----------
              bg:
                  True
            |_
              ----------
              onchanges:
                  |_
                    ----------
                    pkg:
                        salt-minion
                  |_
                    ----------
                    file:
                        salt-minion
                  |_
                    ----------
                    file:
                        remove-old-minion-conf-file
            - run
            |_
              ----------
              order:
                  10002
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion.d
            |_
              ----------
              template:
                  jinja
            |_
              ----------
              source:
                  salt://salt/files/minion.d
            |_
              ----------
              context:
                  ----------
                  standalone:
                      False
            |_
              ----------
              clean:
                  True
            |_
              ----------
              exclude_pat:
                  _*
            - recurse
            |_
              ----------
              order:
                  10001
        pkg:
            |_
              ----------
              name:
                  salt-minion-py3
            |_
              ----------
              require_in:
                  |_
                    ----------
                    service:
                        salt-minion
            - installed
            |_
              ----------
              order:
                  10000
        service:
            |_
              ----------
              enable:
                  True
            |_
              ----------
              name:
                  salt-minion
            |_
              ----------
              watch:
                  |_
                    ----------
                    file:
                        remove-old-minion-conf-file
            |_
              ----------
              order:
                  last
            - running
    salt-minion-pki-dir:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion
            |_
              ----------
              user:
                  system
            |_
              ----------
              group:
                  root
            |_
              ----------
              makedirs:
                  True
            - directory
            |_
              ----------
              order:
                  10005

[TheBigBear]

@dafyddj or @twangboy what else do I have to capture first while I am still on 3004.2-1 ?
And before I upgrade to 3005.x where it will presumably break again?
Because once I upgraded I can't really easily downgrade the home/test/lab master down to 3004.2-1 again to get more logs ...

[dafyddj]

I would like to see grains.items and pillar.items

[TheBigBear]

@dafyddj your wish is my command ;-)

salt INT-UR-TEST-1 grains.items

INT-UR-TEST-1:
    ----------
    biosversion:
        Default System BIOS
    cpu_model:
        Intel(R) Core(TM) i3 CPU       M 350  @ 2.27GHz
    cpuarch:
        AMD64
    cwd:
        C:\Program Files\Salt Project\Salt\bin
    disks:
        - \\.\PhysicalDrive0
    domain:
    efi:
        False
    efi-secure-boot:
        False
    fqdn:
        INT-UR-TEST-1
    fqdn_ip4:
        - 10.<redacted>
    fqdn_ip6:
        - fe80::a3db:5baf:3bdf:8971
    fqdns:
    gpus:
    groupname:
    host:
        INT-UR-TEST-1
    hwaddr_interfaces:
        ----------
        Ralink RT5390 802.11b/g/n WiFi Adapter:
            90:00:4E:08:0F:23
        Software Loopback Interface 1:
            :::::
    id:
        INT-UR-TEST-1
    init:
        Windows
    ip4_interfaces:
        ----------
        Ralink RT5390 802.11b/g/n WiFi Adapter:
            - 10.<redacted>
        Software Loopback Interface 1:
            - 127.0.0.1
    ip6_interfaces:
        ----------
        Ralink RT5390 802.11b/g/n WiFi Adapter:
            - fe80::a3db:5baf:3bdf:8971
        Software Loopback Interface 1:
            - ::1
    ip_interfaces:
        ----------
        Ralink RT5390 802.11b/g/n WiFi Adapter:
            - 10.<redacted>
            - fe80::a3db:5baf:3bdf:8971
        Software Loopback Interface 1:
            - 127.0.0.1
            - ::1
    ipv4:
        - 10.<redacted>
        - 127.0.0.1
    ipv6:
        - ::1
        - fe80::a3db:5baf:3bdf:8971
    kernel:
        Windows
    kernelrelease:
        10.0.19045
    kernelversion:
        10.0.19041
    locale_info:
        ----------
        defaultencoding:
            cp1252
        defaultlanguage:
            en_GB
        detectedencoding:
            cp1252
        timezone:
            GMT Standard Time
    localhost:
        INT-UR-TEST-1
    manufacturer:
        Hewlett-Packard
    master:
        ur-salt
    mem_total:
        2933
    motherboard:
        ----------
        productname:
            1425
        serialnumber:
            PX11M011ZZX2SF
    nodename:
        INT-UR-TEST-1
    num_cpus:
        4
    num_gpus:
        0
    os:
        Windows
    os_family:
        Windows
    osfinger:
        Windows-10
    osfullname:
        Microsoft Windows 10 Home
    osmanufacturer:
        Microsoft Corporation
    osrelease:
        10
    osrelease_info:
        - 10
    osservicepack:
        None
    osversion:
        10.0.19045
    path:
        C:\ProgramData\Boxstarter;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\tools;C:\Program Files\Salt Project\Salt;C:\Program Files\PowerShell\7\;C:\tools\BCURRAN3;;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps
    pending_reboot:
        False
    pid:
        3020
    productname:
        HP G62 Notebook PC
    ps:
        tasklist.exe
    pythonexecutable:
        C:\Program Files\Salt Project\Salt\bin\python.exe
    pythonpath:
        - C:\Program Files\Salt Project\Salt\bin\Scripts
        - C:\Program Files\Salt Project\Salt\bin\lib\site-packages\git\ext\gitdb
        - C:\Program Files\Salt Project\Salt\bin\python38.zip
        - C:\Program Files\Salt Project\Salt\bin\DLLs
        - C:\Program Files\Salt Project\Salt\bin\lib
        - C:\Program Files\Salt Project\Salt\bin
        - C:\Program Files\Salt Project\Salt\bin\lib\site-packages
        - C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.2-py3.8.egg
        - C:\Program Files\Salt Project\Salt\bin\lib\site-packages\win32
        - C:\Program Files\Salt Project\Salt\bin\lib\site-packages\win32\lib
        - C:\Windows\[Microsoft.NET](http://microsoft.net/)\Framework64\v4.0.30319\
        - C:\Program Files\Salt Project\Salt\bin\lib\site-packages\gitdb\ext\smmap
    pythonversion:
        - 3
        - 8
        - 8
        - final
        - 0
    saltpath:
        C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt-3004.2-py3.8.egg\salt
    saltversion:
        3004.2
    saltversioninfo:
        - 3004
        - 2
    serialnumber:
        CNF10235DM
    server_id:
        1504829806
    shell:
        C:\Windows\system32\cmd.exe
    ssds:
    systempath:
        - C:\ProgramData\Boxstarter
        - C:\Windows\system32
        - C:\Windows
        - C:\Windows\System32\Wbem
        - C:\Windows\System32\WindowsPowerShell\v1.0\
        - C:\Windows\System32\OpenSSH\
        - C:\ProgramData\chocolatey\bin
        - C:\Program Files\Git\cmd
        - C:\tools
        - C:\Program Files\Salt Project\Salt
        - C:\Program Files\PowerShell\7\
        - C:\tools\BCURRAN3
        - C:\Program Files (x86)\Gpg4win\..\GnuPG\bin
        - C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps
    timezone:
        (UTC+00:00) Dublin, Edinburgh, Lisbon, London
    transactional:
        False
    username:
        SYSTEM
    uuid:
        31464e43-3230-3533-444d-984be1936d78
    virtual:
        physical
    windowsdomain:
        WORKGROUP
    windowsdomaintype:
        Workgroup

salt INT-UR-TEST-1 pillar.items

INT-UR-TEST-1:
    ----------
    salt:
        ----------
        config_path:
            C:\ProgramData\Salt Project\Salt\conf
        master:
            ----------
            file_recv:
                True
            file_roots:
                ----------
                base:
                    - /srv/salt
                    - /srv/formulas/salt-formula
            hash_type:
                sha256
            pillar_roots:
                ----------
                base:
                    - /srv/pillar
            show_timeout:
                True
            worker_threads:
                20
        minion:
            ----------
            master:
                ur-salt
            mine_interval:
                60
            state_output:
                changes
            use_superseded:
                - module.run
        py_ver:
            py3
    salt_minion:
        ----------
        salt_minion_id:
            INT-UR-TEST-1
    users:
        ----------
        ansibleur:
            ----------
            fullname:
                ansible service user ur
    users-formula:
        ----------
        lookup:
            ----------
            root_group:
                root
        use_vim_formula:
            True

salt master versions:

salt --versions

[root@int-ur-it-salt salt]# salt --versions
Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Sep 13 2022, 07:19:15)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: almalinux 8.6 Sky Tiger
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-372.26.1.el8_6.x86_64
        system: Linux
       version: AlmaLinux 8.6 Sky Tiger

salt windows minion versions:

salt-call --versions

C:\Windows\system32>salt-call --versions
Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: 3.1.18
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)]
  python-gnupg: 0.4.7
        PyYAML: 5.4.1
         PyZMQ: 19.0.0
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 10
        system: Windows
       version: 10 10.0.19041 SP0 Multiprocessor Free

and state.apply salt.minion and state.apply salt.minion test=true outputs

salt INT-UR-TEST-1 state.apply salt.minion test=true

INT-UR-TEST-1:
----------
          ID: salt-minion
    Function: pkg.installed
        Name: salt-minion-py3
      Result: True
     Comment: All specified packages are already installed
     Started: 18:48:30.956247
    Duration: 1312.489 ms
     Changes:
----------
          ID: salt-minion
    Function: file.recurse
        Name: C:\ProgramData\Salt Project\Salt\conf\minion.d
      Result: None
     Comment: #### C:\ProgramData\Salt Project\Salt\conf\minion.d ####
              The directory "C:\ProgramData\Salt Project\Salt\conf\minion.d" will be changed

              #### C:\ProgramData\Salt Project\Salt\conf\minion.d\reactor.conf ####
              The file C:\ProgramData\Salt Project\Salt\conf\minion.d\reactor.conf is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.

              #### C:\ProgramData\Salt Project\Salt\conf\minion.d\engine.conf ####
              The file C:\ProgramData\Salt Project\Salt\conf\minion.d\engine.conf is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.

              #### C:\ProgramData\Salt Project\Salt\conf\minion.d\beacons.conf ####
              The file C:\ProgramData\Salt Project\Salt\conf\minion.d\beacons.conf is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.

              #### C:\ProgramData\Salt Project\Salt\conf\minion.d\f_defaults.conf ####
              The file C:\ProgramData\Salt Project\Salt\conf\minion.d\f_defaults.conf is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.
     Started: 18:48:32.268736
    Duration: 1015.616 ms
     Changes:
              ----------
              C:\ProgramData\Salt Project\Salt\conf\minion.d:
                  ----------
                  owner:
                      SYSTEM
              C:\ProgramData\Salt Project\Salt\conf\minion.d\beacons.conf:
                  ----------
                  newfile:
                      C:\ProgramData\Salt Project\Salt\conf\minion.d\beacons.conf
              C:\ProgramData\Salt Project\Salt\conf\minion.d\engine.conf:
                  ----------
                  newfile:
                      C:\ProgramData\Salt Project\Salt\conf\minion.d\engine.conf
              C:\ProgramData\Salt Project\Salt\conf\minion.d\f_defaults.conf:
                  ----------
                  newfile:
                      C:\ProgramData\Salt Project\Salt\conf\minion.d\f_defaults.conf
              C:\ProgramData\Salt Project\Salt\conf\minion.d\reactor.conf:
                  ----------
                  newfile:
                      C:\ProgramData\Salt Project\Salt\conf\minion.d\reactor.conf
----------
          ID: remove-old-minion-conf-file
    Function: file.absent
        Name: C:\ProgramData\Salt Project\Salt\conf\minion.d\_defaults.conf
      Result: True
     Comment: File C:\ProgramData\Salt Project\Salt\conf\minion.d\_defaults.conf is not present
     Started: 18:48:33.299975
    Duration: 0.0 ms
     Changes:
----------
          ID: salt-minion
    Function: cmd.run
        Name: salt-call.bat --local service.restart salt-minion
      Result: None
     Comment: Command "salt-call.bat --local service.restart salt-minion" would have been executed
     Started: 18:48:33.299975
    Duration: 0.0 ms
     Changes:
----------
          ID: permissions-minion-config
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\conf\minion
      Result: None
     Comment: File C:\ProgramData\Salt Project\Salt\conf\minion not updated
     Started: 18:48:33.299975
    Duration: 109.374 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: salt-minion-pki-dir
    Function: file.directory
        Name: C:\ProgramData\Salt Project\Salt\conf\pki\minion
      Result: None
     Comment: The directory "C:\ProgramData\Salt Project\Salt\conf\pki\minion" will be changed
     Started: 18:48:33.409349
    Duration: 0.0 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: permissions-minion.pem
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pem
      Result: None
     Comment: File C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pem not updated
     Started: 18:48:33.409349
    Duration: 15.625 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: permissions-minion.pub
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pub
      Result: None
     Comment: File C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pub not updated
     Started: 18:48:33.424974
    Duration: 0.0 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: salt-minion
    Function: service.running
      Result: True
     Comment: The service salt-minion is already running
     Started: 18:48:33.565602
    Duration: 8249.913 ms
     Changes:

Summary for INT-UR-TEST-1
------------
Succeeded: 9 (unchanged=6, changed=5)
Failed:    0
------------
Total states run:     9
Total run time:  10.703 s

salt INT-UR-TEST-1 state.apply salt.minion

[root@int-ur-it-salt salt]# 
INT-UR-TEST-1:
----------
          ID: salt-minion
    Function: pkg.installed
        Name: salt-minion-py3
      Result: True
     Comment: All specified packages are already installed
     Started: 18:50:13.657307
    Duration: 1296.863 ms
     Changes:
----------
          ID: salt-minion
    Function: file.recurse
        Name: C:\ProgramData\Salt Project\Salt\conf\minion.d
      Result: True
     Comment: Recursively updated C:\ProgramData\Salt Project\Salt\conf\minion.d
     Started: 18:50:14.969795
    Duration: 1171.861 ms
     Changes:
              ----------
              C:\ProgramData\Salt Project\Salt\conf\minion.d:
                  ----------
                  owner:
                      SYSTEM
              C:\ProgramData\Salt Project\Salt\conf\minion.d\beacons.conf:
                  ----------
                  diff:
                      New file
              C:\ProgramData\Salt Project\Salt\conf\minion.d\engine.conf:
                  ----------
                  diff:
                      New file
              C:\ProgramData\Salt Project\Salt\conf\minion.d\f_defaults.conf:
                  ----------
                  diff:
                      New file
              C:\ProgramData\Salt Project\Salt\conf\minion.d\reactor.conf:
                  ----------
                  diff:
                      New file
----------
          ID: remove-old-minion-conf-file
    Function: file.absent
        Name: C:\ProgramData\Salt Project\Salt\conf\minion.d\_defaults.conf
      Result: True
     Comment: File C:\ProgramData\Salt Project\Salt\conf\minion.d\_defaults.conf is not present
     Started: 18:50:16.157287
    Duration: 0.0 ms
     Changes:
----------
          ID: salt-minion
    Function: cmd.run
        Name: salt-call.bat --local service.restart salt-minion
      Result: True
     Comment: Command "salt-call.bat --local service.restart salt-minion" run
     Started: 18:50:16.157287
    Duration: 46.871 ms
     Changes:
              ----------
              pid:
                  7152
              retcode:
                  None
              stderr:
              stdout:
----------
          ID: permissions-minion-config
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\conf\minion
      Result: True
     Comment:
     Started: 18:50:16.204158
    Duration: 140.622 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: salt-minion-pki-dir
    Function: file.directory
        Name: C:\ProgramData\Salt Project\Salt\conf\pki\minion
      Result: True
     Comment: Directory C:\ProgramData\Salt Project\Salt\conf\pki\minion updated
     Started: 18:50:16.344780
    Duration: 15.626 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: permissions-minion.pem
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pem
      Result: True
     Comment:
     Started: 18:50:16.360406
    Duration: 15.624 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: permissions-minion.pub
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pub
      Result: True
     Comment:
     Started: 18:50:16.376030
    Duration: 0.0 ms
     Changes:
              ----------
              owner:
                  SYSTEM
----------
          ID: salt-minion
    Function: service.running
      Result: True
     Comment: The service salt-minion is already running
     Started: 18:50:16.532286
    Duration: 9421.776 ms
     Changes:

Summary for INT-UR-TEST-1
------------
Succeeded: 9 (changed=6)
Failed:    0
------------
Total states run:     9
Total run time:  12.109 s

Will that do? Or do I need to capture a higher log-level or diagnostic level output as well do you think?

[TheBigBear]

These may be the issue:

Passed invalid arguments to state.apply: can not serialize 'CommandExecutionError' object
Passed invalid arguments: can not serialize 'CommandExecutionError' object

Hey @twangboy, yes, I think this is the likely culprit, salt should be able to handle a mis-firing state and not break under its output, I think.

I am setting up a home/test/lab to test the regression of going from 3004.2-1 to 3005.x and will follow-up on the parts that broke in the salt formulas community repo where it no longer properly detects the rootuser under windows and ver >= 3005.x.

But my python mojo is not up to scratch to find why there is a serialization error in 3005.x, can you follow that one up internally, please? Thanks. Much appreciated.

[twangboy]

Would you mind verifying this is still a problem with 3006.0?

I'm wondering if it's unable to resolve a SID:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

[twangboy]

@TheBigBear Could you share the contents of salt://salt/minion.sls?

[TheBigBear]

@twangboy I am using the salt formula and a pillar in the middle so what I can supply is the state.show_sls of a windows salt.minion. Yes I have retested and it is still the same issue on latest 3006.1 linux master and 3006.1 windows server 2022 minon. ;-(

So this has remained broken ever since I upgraded from 3004.2 to 3005.x and remained broken even now on 3006.1

Here is my rendered above salt.minion state

salt INT-EU-MIM state.show_sls salt.minion
INT-EU-MIM:
    ----------
    permissions-minion-config:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            - managed
            |_
              ----------
              order:
                  10004
    permissions-minion.pem:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pem
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        salt-minion-pki-dir
            - managed
            |_
              ----------
              order:
                  10006
    permissions-minion.pub:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion\minion.pub
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              replace:
                  False
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        salt-minion-pki-dir
            - managed
            |_
              ----------
              order:
                  10007
    remove-old-minion-conf-file:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion.d\_defaults.conf
            - absent
            |_
              ----------
              order:
                  10003
    salt-minion:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        cmd:
            |_
              ----------
              name:
                  salt-call.bat --local service.restart salt-minion
            |_
              ----------
              bg:
                  True
            |_
              ----------
              onchanges:
                  |_
                    ----------
                    pkg:
                        salt-minion
                  |_
                    ----------
                    file:
                        salt-minion
                  |_
                    ----------
                    file:
                        remove-old-minion-conf-file
            - run
            |_
              ----------
              order:
                  10002
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\minion.d
            |_
              ----------
              template:
                  jinja
            |_
              ----------
              source:
                  salt://salt/files/minion.d
            |_
              ----------
              context:
                  ----------
                  standalone:
                      False
            |_
              ----------
              clean:
                  True
            |_
              ----------
              exclude_pat:
                  _*
            - recurse
            |_
              ----------
              order:
                  10001
        pkg:
            |_
              ----------
              name:
                  salt-minion-py3
            |_
              ----------
              require_in:
                  |_
                    ----------
                    service:
                        salt-minion
            - installed
            |_
              ----------
              order:
                  10000
        service:
            |_
              ----------
              enable:
                  True
            |_
              ----------
              name:
                  salt-minion
            |_
              ----------
              watch:
                  |_
                    ----------
                    file:
                        remove-old-minion-conf-file
            |_
              ----------
              order:
                  last
            - running
    salt-minion-pki-dir:
        ----------
        __env__:
            base
        __sls__:
            salt.minion
        file:
            |_
              ----------
              name:
                  C:\ProgramData\Salt Project\Salt\conf\pki\minion
            |_
              ----------
              user:
                  root
            |_
              ----------
              group:
                  root
            |_
              ----------
              makedirs:
                  True
            - directory
            |_
              ----------
              order:
                  10005

So here is a step by step re-run of how to duplicate the issue under 3006.1 on the linux master it looks as follows:

salt INT-EU-MIM state.show_top
INT-EU-MIM:
    ----------
    base:
        - salt_minion_id
        - salt.minion
        - windows.check_mk
        - windows.std-packages

And now I am running those 4 states one by one ( spoiler - only the salt.minion state has an issue )

salt INT-EU-MIM state.apply salt_minion_id test=true
INT-EU-MIM:
----------
          ID: salt_minion_id
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\conf\minion_id
      Result: True
     Comment: The file C:\ProgramData\Salt Project\Salt\conf\minion_id is in the correct state
     Started: 13:57:31.127401
    Duration: 93.75 ms
     Changes:

Summary for INT-EU-MIM
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:  93.750 ms

So that ran just fine.

But now comes the culprit.

salt INT-EU-MIM state.apply salt.minion test=true
INT-EU-MIM:
    Passed invalid arguments to state.apply: can not serialize 'CommandExecutionError' object

        .. versionadded:: 2015.5.0

        This function will call :mod:`state.highstate
        <salt.modules.state.highstate>` or :mod:`state.sls
        <salt.modules.state.sls>` based on the arguments passed to this function.
        It exists as a more intuitive way of applying states.

        .. rubric:: APPLYING ALL STATES CONFIGURED IN TOP.SLS (A.K.A. :ref:`HIGHSTATE <running-highstate>`)

        To apply all configured states, simply run ``state.apply``:

        .. code-block:: bash

            salt '*' state.apply

        The following additional arguments are also accepted when applying all
        states configured in top.sls:

        test
            Run states in test-only (dry-run) mode

        mock
            The mock option allows for the state run to execute without actually
            calling any states. This then returns a mocked return which will show
            the requisite ordering as well as fully validate the state run.

            .. versionadded:: 2015.8.4

        pillar
            Custom Pillar values, passed as a dictionary of key-value pairs

            .. code-block:: bash

                salt '*' state.apply stuff pillar='{"foo": "bar"}'

            .. note::
                Values passed this way will override Pillar values set via
                ``pillar_roots`` or an external Pillar source.

        exclude
            Exclude specific states from execution. Accepts a list of sls names, a
            comma-separated string of sls names, or a list of dictionaries
            containing ``sls`` or ``id`` keys. Glob-patterns may be used to match
            multiple states.

            .. code-block:: bash

                salt '*' state.apply exclude=bar,baz
                salt '*' state.apply exclude=foo*
                salt '*' state.apply exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls_to_exclude'}]"

        queue : False
            Instead of failing immediately when another state run is in progress,
            a value of ``True`` will queue the new state run to begin running once
            the other has finished.

            This option starts a new thread for each queued state run, so use this
            option sparingly.

            .. versionchanged:: 3006.0
                This parameter can also be set via the ``state_queue`` configuration
                option. Additionally, it can now be set to an integer representing
                the maximum queue size which can be attained before the state runs
                will fail to be queued. This can prevent runaway conditions where
                new threads are started until system performance is hampered.

        localconfig
            Optionally, instead of using the minion config, load minion opts from
            the file specified by this argument, and then merge them with the
            options from the minion config. This functionality allows for specific
            states to be run with their own custom minion configuration, including
            different pillars, file_roots, etc.

            .. code-block:: bash

                salt '*' state.apply localconfig=/path/to/minion.yml

        state_events
            The state_events option sends progress events as each function in
            a state run completes execution.

            .. versionadded:: 3006.0


        .. rubric:: APPLYING INDIVIDUAL SLS FILES (A.K.A. :py:func:`STATE.SLS <salt.modules.state.sls>`)

        To apply individual SLS files, pass them as a comma-separated list:

        .. code-block:: bash

            # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
            salt '*' state.apply stuff

            # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
            # and salt://pkgs.sls (or salt://pkgs/init.sls).
            salt '*' state.apply stuff,pkgs

            # Run the states configured in a more deeply nested directory such as salt://my/organized/stuff.sls (or salt://my/organized/stuff/init.sls)
            salt '*' state.apply my.organized.stuff

        The following additional arguments are also accepted when applying
        individual SLS files:

        test
            Run states in test-only (dry-run) mode

        mock
            The mock option allows for the state run to execute without actually
            calling any states. This then returns a mocked return which will show
            the requisite ordering as well as fully validate the state run.

            .. versionadded:: 2015.8.4

        pillar
            Custom Pillar values, passed as a dictionary of key-value pairs

            .. code-block:: bash

                salt '*' state.apply stuff pillar='{"foo": "bar"}'

            .. note::
                Values passed this way will override Pillar values set via
                ``pillar_roots`` or an external Pillar source.

        queue : False
            Instead of failing immediately when another state run is in progress,
            a value of ``True`` will queue the new state run to begin running once
            the other has finished.

            This option starts a new thread for each queued state run, so use this
            option sparingly.

            .. versionchanged:: 3006.0
                This parameter can also be set via the ``state_queue`` configuration
                option. Additionally, it can now be set to an integer representing
                the maximum queue size which can be attained before the state runs
                will fail to be queued. This can prevent runaway conditions where
                new threads are started until system performance is hampered.

        concurrent : False
            Execute state runs concurrently instead of serially

            .. warning::

                This flag is potentially dangerous. It is designed for use when
                multiple state runs can safely be run at the same time. Do *not*
                use this flag for performance optimization.

        saltenv
            Specify a salt fileserver environment to be used when applying states

            .. versionchanged:: 0.17.0
                Argument name changed from ``env`` to ``saltenv``

            .. versionchanged:: 2014.7.0
                If no saltenv is specified, the minion config will be checked for an
                ``environment`` parameter and if found, it will be used. If none is
                found, ``base`` will be used. In prior releases, the minion config
                was not checked and ``base`` would always be assumed when the
                saltenv was not explicitly set.

        pillarenv
            Specify a Pillar environment to be used when applying states. This
            can also be set in the minion config file using the
            :conf_minion:`pillarenv` option. When neither the
            :conf_minion:`pillarenv` minion config option nor this CLI argument is
            used, all Pillar environments will be merged together.

        localconfig
            Optionally, instead of using the minion config, load minion opts from
            the file specified by this argument, and then merge them with the
            options from the minion config. This functionality allows for specific
            states to be run with their own custom minion configuration, including
            different pillars, file_roots, etc.

            .. code-block:: bash

                salt '*' state.apply stuff localconfig=/path/to/minion.yml

        sync_mods
            If specified, the desired custom module types will be synced prior to
            running the SLS files:

            .. code-block:: bash

                salt '*' state.apply stuff sync_mods=states,modules
                salt '*' state.apply stuff sync_mods=all

            .. note::
                This option is ignored when no SLS files are specified, as a
                :ref:`highstate <running-highstate>` automatically syncs all custom
                module types.

            .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

        state_events
            The state_events option sends progress events as each function in
            a state run completes execution.

            .. versionadded:: 3006.0

ERROR: Minions returned with non-zero exit code

And just to show the last 2 states, they also run without any issues.

salt INT-EU-MIM state.apply windows.check_mk test=true
INT-EU-MIM:
----------
          ID: check-mk-agent-pkg
    Function: pkg.latest
        Name: check_mk_agent
      Result: True
     Comment: Package check_mk_agent is already up-to-date
     Started: 14:00:13.814625
    Duration: 640.626 ms
     Changes:
----------
          ID: check_mk_users.yml
    Function: file.managed
        Name: C:\ProgramData\checkmk\agent\check_mk.user.yml
      Result: True
     Comment: The file C:\ProgramData\checkmk\agent\check_mk.user.yml is in the correct state
     Started: 14:00:14.455251
    Duration: 109.374 ms
     Changes:
----------
          ID: register-check-mk-agent-receiver
    Function: cmd.run
        Name: C:\Progra~2\checkmk\service\cmk-agent-ctl.exe register --trust-cert --hostname INT-EU-MIM --server <redacted>
      Result: True
     Comment: onlyif condition is false
     Started: 14:00:14.564625
    Duration: 1562.505 ms
     Changes:
----------
          ID: C:\ProgramData\checkmk\agent\plugins\\windows_updates.vbs
    Function: file.managed
      Result: True
     Comment: The file C:\ProgramData\checkmk\agent\plugins\\windows_updates.vbs is in the correct state
     Started: 14:00:16.127130
    Duration: 62.494 ms
     Changes:
----------
          ID: C:\ProgramData\checkmk\agent\plugins\\mk_inventory.vbs
    Function: file.managed
      Result: True
     Comment: The file C:\ProgramData\checkmk\agent\plugins\\mk_inventory.vbs is in the correct state
     Started: 14:00:16.189624
    Duration: 46.878 ms
     Changes:
----------
          ID: C:\ProgramData\checkmk\agent\plugins\\win_license.bat
    Function: file.managed
      Result: True
     Comment: The file C:\ProgramData\checkmk\agent\plugins\\win_license.bat is in the correct state
     Started: 14:00:16.236502
    Duration: 46.875 ms
     Changes:
----------
          ID: checkmk_windows_service
    Function: service.running
        Name: CheckMKService
      Result: True
     Comment: The service CheckMKService is already running
     Started: 14:00:16.283377
    Duration: 31.252 ms
     Changes:

Summary for INT-EU-MIM
------------
Succeeded: 7
Failed:    0
------------
Total states run:     7
Total run time:   2.500 s

and the last one is fine too

salt INT-EU-MIM state.show_top
INT-EU-MIM:
    ----------
    base:
        - salt_minion_id
        - salt.minion
        - windows.check_mk
        - windows.std-packages
root@salt salt_minion (master *=) $ salt INT-EU-MIM state.apply windows.std-packages test=true
INT-EU-MIM:
----------
          ID: telnet-client
    Function: win_servermanager.installed
      Result: None
     Comment:
     Started: 14:11:30.204388
    Duration: 2062.526 ms
     Changes:
              ----------
              telnet-client:
                  Will be installed recurse=True
----------
          ID: FS-SMB1
    Function: win_servermanager.removed
      Result: True
     Comment: The following features are not installed:
              - FS-SMB1
     Started: 14:11:32.266914
    Duration: 1203.121 ms
     Changes:
----------
          ID: npp-pkg
    Function: pkg.latest
        Name: npp
      Result: True
     Comment: Package npp is already up-to-date
     Started: 14:11:35.860662
    Duration: 66593.568 ms
     Changes:
----------
          ID: salt-minion-pkg
    Function: pkg.latest
        Name: salt-minion-py3
      Result: True
     Comment: Package salt-minion-py3 is already up-to-date
     Started: 14:12:42.454230
    Duration: 62.575 ms
     Changes:
----------
          ID: 7-zip-pkg
    Function: pkg.latest
        Name: 7zip
      Result: True
     Comment: Package 7zip is already up-to-date
     Started: 14:12:42.516805
    Duration: 62.424 ms
     Changes:
----------
          ID: IIS-Crypto
    Function: cmd.script
        Name: salt://windows/std-packages/files/IISCryptoCli.exe
      Result: True
     Comment: C:\ProgramData\Salt Project\Salt\var\iiscrypto.3.3.txt exists
     Started: 14:12:42.579229
    Duration: 1234.373 ms
     Changes:
----------
          ID: IIS-Crypto
    Function: file.managed
        Name: C:\ProgramData\Salt Project\Salt\var\iiscrypto.3.3.txt
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 14:12:43.813602
    Duration: 0.0 ms
     Changes:
----------
          ID: SMB-Firewall-cmd
    Function: cmd.script
        Name: salt://windows/std-packages/files/firewall.ps1
      Result: None
     Comment:
     Started: 14:12:43.813602
    Duration: 15.625 ms
     Changes:
----------
          ID: ICMP-Firewall-cmd
    Function: cmd.script
        Name: salt://windows/std-packages/files/firewall.ps1
      Result: None
     Comment:
     Started: 14:12:43.829227
    Duration: 0.0 ms
     Changes:
----------
          ID: RDP-Firewall-cmd
    Function: cmd.script
        Name: salt://windows/std-packages/files/firewall.ps1
      Result: None
     Comment:
     Started: 14:12:43.829227
    Duration: 0.0 ms
     Changes:
----------
          ID: spectreFeatureSettingsOverride
    Function: reg.present
        Name: HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
      Result: True
     Comment: FeatureSettingsOverride in HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management is already present
     Started: 14:12:43.829227
    Duration: 0.0 ms
     Changes:
----------
          ID: spectreFeatureSettingsOverrideMask
    Function: reg.present
        Name: HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
      Result: True
     Comment: FeatureSettingsOverrideMask in HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management is already present
     Started: 14:12:43.829227
    Duration: 0.0 ms
     Changes:

Summary for INT-EU-MIM
------------
Succeeded: 12 (unchanged=4, changed=1)
Failed:    0
------------
Total states run:    12
Total run time:  71.234 s

BUT a state.highstate fails with zero helpful output on the linux master and on the windows minon shows at least the errros so one is not totally blind on the minion side.

salt INT-EU-MIM state.highstate test=true
INT-EU-MIM:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20230630141521214350
ERROR: Minions returned with non-zero exit code

And this salt job never gets any feedback at all.

However on the windows minion side using salt-call instead I can glean more error feedback:

First this is how the salt.minon state runs and what it shows:

salt-call state.apply salt.minion test=true
[WARNING ] The group argument for C:\ProgramData\Salt Project\Salt\conf\minion has been ignored as this is a Windows system. Please use the `win_*` parameters to set permissions in Windows.
[ERROR   ] Invalid user/group or sid: root
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\win_dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\win_dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is invalid.')
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\win_dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\win_dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is invalid.')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\state.py", line 2385, in call
    ret = self.states[cdata["full"]](
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 1232, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 1247, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 1280, in wrapper
    return f(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\states\file.py", line 3111, in managed
    u_check = _check_user(user, group)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\states\file.py", line 386, in _check_user
    uid = __salt__["file.user_to_uid"](user)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 1232, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 1247, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\modules\win_file.py", line 523, in user_to_uid
    return __utils__["dacl.get_sid_string"](user)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 1232, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\loader\lazy.py", line 1247, in _run_as
    return _func_or_method(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\win_dacl.py", line 1186, in get_sid_string
    principal = get_sid(principal)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\win_dacl.py", line 1149, in get_sid
    raise CommandExecutionError("Invalid user/group or sid: {}".format(principal))
salt.exceptions.CommandExecutionError: Invalid user/group or sid: root

[WARNING ] The group argument for C:\ProgramData\Salt Project\Salt\conf\pki\minion has been ignored as this is a Windows system. Please use the `win_*` parameters to set permissions in Windows.
[ERROR   ] Invalid user/group or sid: root
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\utils\win_dacl.py", line 1146, in get_sid
    sid = win32security.ConvertStringSidToSid(sid)
pywintypes.error: (1337, 'ConvertStringSidToSid', 'The security ID structure is invalid.')
[ERROR   ] Invalid user/group or sid: root

Passed invalid arguments: can not serialize 'CommandExecutionError' object.

Usage:

[TheBigBear]

I think a big part of this regression is the fact that the salt-formula is not being maintained very well. I had created a report in the salt-formula repo back at end of 22 and it is still broken today as per this issue here. saltstack-formulas/salt-formula#541
@dafyddj or @twangboy is saltstack not maintaining it's own salt-formula?

[twangboy]

Those formulas are trying to set permissions for the user root, which, as you know, doesn't exist on Windows. That is the error we're seeing:

[ERROR   ] Invalid user/group or sid: root
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\Lib\site-packages\salt\utils\win_
dacl.py", line 1183, in get_sid_string
    return win32security.ConvertSidToStringSid(principal)
TypeError: The object is not a PySID object

The salt-formula needs to be updated to provide valid users on Windows. These may not have been applied in earlier versions of Salt and that is why you don't see the error in 3004.2.

[TheBigBear]

@twangboy how is this ‘completed’?
The bug saltstack-formulas/salt-formula#541 (comment) has not been touched or fixed.
Does salt not maintain its own formula?
So the article about yes salt can salt itself , is a blatant lie nowadays?

[OrangeDog]

Does salt not maintain its own formula?

No. They're completely independent.