Cannot use gitfs with salt-ssh

[davidkarlsen]

Related thread in salt user mailing list.
Same problem with newer version than I reported on the mailinglist:
salt-ssh --version
salt-ssh 2015.5.0 (Lithium)

Thread:
https://groups.google.com/forum/#!searchin/salt-users/gitfs$20salt-ssh/salt-users/yFt-k_Qh0QU/7t6lhOP_SW0J

Debug log output in http://www.davidkarlsen.com/debuglog

I can see the git checkouts landing in the var directory, but they are not found by git-ssh when executing.

[jfindlay]

@davidkarlsen, thanks for the report.

[rallytime]

@davidkarlsen I am able to successfully use salt-ssh with gitfs, both with 2014.7.5 as well as with 2015.5.0. Perhaps something is a bit off in your configuration? Here's what I have working:

salt-ssh is running on an Ubuntu 14.04 master with the following (gitfs-related) configurations:

fileserver_backend:
  - git
  - roots
gitfs_provider: GitPython
gitfs_remotes:
  - git://github.com/rallytime/salt-pkg-install-tests.git

The repo that I have gitfs pointing to is a test repo that I use to test new packages of salt (without having salt already installed on the VM). The states are able to run and execute successfully via salt-ssh:

# salt-ssh nt-cent7 state.sls test_install
nt-cent7:
----------
          ID: epel
    Function: cmd.run
        Name: yum -y install epel-release
      Result: True
     Comment: Command "yum -y install epel-release" run
     Started: 21:43:17.199054
    Duration: 15620.64 ms
     Changes:
              ----------
              pid:
                  2229
              retcode:
                  0
              stderr:
              stdout:
                  Loaded plugins: fastestmirror
                  Determining fastest mirrors
                   * base: mirrors.linode.com
                   * extras: mirrors.linode.com
                   * updates: mirrors.linode.com
                  Resolving Dependencies
                  --> Running transaction check
                  ---> Package epel-release.noarch 0:7-5 will be installed
                  --> Finished Dependency Resolution

                  Dependencies Resolved

<snipped>

                 Total download size: 12 k
                  Installed size: 1.4 k
                  Downloading packages:
                  Running transaction check
                  Running transaction test
                  Transaction test succeeded
                  Running transaction
                    Installing : salt-syndic-2015.5.0-1.el7.noarch                            1/1
                    Verifying  : salt-syndic-2015.5.0-1.el7.noarch                            1/1

                  Installed:
                    salt-syndic.noarch 0:2015.5.0-1.el7

                  Complete!

Summary
------------
Succeeded: 7 (changed=7)
Failed:    0
------------
Total states run:     7

Here's my versions report:

# salt-ssh --versions
           Salt: 2015.5.0
         Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
         Jinja2: 2.7.3
       M2Crypto: 0.21.1
 msgpack-python: 0.4.4
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.0.1
           RAET: Not Installed
            ZMQ: 4.0.4
           Mako: 0.9.1

[rallytime]

Do the states you're referencing with gitfs work with "regular" salt, instead of salt-ssh? (i.e. Could something be off in your referenced states?)

I also tried changing my git reference from git://github.com/rallytime/salt-pkg-install-tests.git to [email protected]:rallytime/salt-pkg-install-tests.git and that worked fine as well.

[davidkarlsen]

Yes - the formulas work with "regular salt". I could add that I run salt-ssh as a non-root user - does that influence in any way?
Regarding the configuration: do you see anything off with my config in the mail-thread I referenced?

[rallytime]

@davidkarlsen Okay, those are good details. I run my salt-ssh commands as root, so that could be it. I also run my states as state.sls instead of state.highstate (this is where I am thinking the culprit might be, but I have not confirmed this) so that could be something to look into as well. If you try running the salt-ssh command as root on a test box, do you see the same errors? (Just to rule out this possibility.)

I don't see anything particularly off on your config files. The only thing I see missing is defining the gitfs_provider. Try defining that and seeing if anything clears up.

I have seen people encounter problems when they define many gitfs remotes with varying top files and then run state.highstate. Sometimes the top-files step on each other and cause some problems when they don't know where to look for includes. etc. The thing that is strange about that though, is you say this exact setup is working with "regular salt" commands. I believe @ssgward was the one experiencing this, so he might have some good thoughts/input on that.

[epcim]

This is still valid, fails with salt-ssh. I have my formula under salt/env/apealive/states/users, loaded properly but fails to render mam.jinja in there

salt-ssh $(hostname) state.apply test=true --roster-file salt/env/apealive/roster 
cmp1-nuc11:
    - Rendering SLS 'apealive:users' failed: Jinja error: users/map.jinja
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
          output = template.render(**decoded_context)
        File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
          return original_render(self, *args, **kwargs)
        File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
          return self.environment.handle_exception(exc_info, True)
        File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
          reraise(exc_type, exc_value, tb)
        File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
          raise value.with_traceback(tb)
        File "<template>", line 2, in <module>
        File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 198, in get_source
          raise TemplateNotFound(template)
      jinja2.exceptions.TemplateNotFound: users/map.jinja
      
      ; line 2
      
      ---
      # vim: sts=2 ts=2 sw=2 et ai
      {% from "users/map.jinja" import users with context %}    <======================

[OrangeDog]

@epcim that's a completely different error, and a duplicate of #61174.

[OrangeDog]

I don't know if this has been reproduced recently, but there seems to be the opposite problem now: #59839

[baby-gnu]

@OrangeDog yes \o/

I was so used for gitfs to not work that I didn't even try recently but it's working on my

current version

Salt Version:
          Salt: 3004
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.3
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.11.0
        pygit2: 1.6.1
        Python: 3.9.10 (main, Feb 22 2022, 13:54:07)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.3.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian unstable sid
        locale: utf-8
       machine: x86_64
       release: 5.16.0-4-amd64
        system: Linux
       version: Debian GNU/Linux unstable sid

[baby-gnu]

debug logs to run salt-ssh testmachine2 state.apply openssh.auth_map

   2068 2022-03-14 11:17:44,965 [salt.fileclient  :1155][DEBUG   ][41791] In saltenv 'base', looking at rel_path 'openssh/auth_map.sls' to resolve 'salt://openssh/auth_map.sls'
   2069 2022-03-14 11:17:44,965 [salt.fileclient  :1164][DEBUG   ][41791] In saltenv 'base', ** considering ** path '/var/tmp/.root_8dccac_salt/running_data/var/cache/salt/minion/files/
   2069 base/openssh/auth_map.sls' to resolve 'salt://openssh/auth_map.sls'
   2070 2022-03-14 11:17:44,965 [salt.fileclient  :1185][DEBUG   ][41791] Fetching file from saltenv 'base', ** attempting ** 'salt://openssh/auth_map.sls'
   2071 2022-03-14 11:17:44,965 [salt.fileclient  :1214][DEBUG   ][41791] No dest file found
   2072 2022-03-14 11:17:44,965 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2073 2022-03-14 11:17:44,966 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2074 2022-03-14 11:17:44,966 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2075 2022-03-14 11:17:44,967 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2076 2022-03-14 11:17:44,967 [salt.fileclient  :1301][INFO    ][41791] Fetching file from saltenv 'base', ** done ** 'openssh/auth_map.sls'
   2077 2022-03-14 11:17:44,967 [salt.template    :53  ][DEBUG   ][41791] compile template: /var/tmp/.root_8dccac_salt/running_data/var/cache/salt/minion/files/base/openssh/auth_map.sls
   2078 2022-03-14 11:17:44,968 [salt.utils.jinja :86  ][DEBUG   ][41791] Jinja search path: ['/var/tmp/.root_8dccac_salt/running_data/var/cache/salt/minion/files/base']
   2079 2022-03-14 11:17:44,973 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2080 2022-03-14 11:17:44,974 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2081 2022-03-14 11:17:44,974 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2082 2022-03-14 11:17:44,975 [salt.fileclient  :1155][DEBUG   ][41791] In saltenv 'base', looking at rel_path 'openssh/map.jinja' to resolve 'salt://openssh/map.jinja'
   2083 2022-03-14 11:17:44,975 [salt.fileclient  :1164][DEBUG   ][41791] In saltenv 'base', ** considering ** path '/var/tmp/.root_8dccac_salt/running_data/var/cache/salt/minion/files/   2083 base/openssh/map.jinja' to resolve 'salt://openssh/map.jinja'
   2084 2022-03-14 11:17:44,975 [salt.fileclient  :1185][DEBUG   ][41791] Fetching file from saltenv 'base', ** attempting ** 'salt://openssh/map.jinja'
   2085 2022-03-14 11:17:44,975 [salt.fileclient  :1214][DEBUG   ][41791] No dest file found
   2086 2022-03-14 11:17:44,975 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2087 2022-03-14 11:17:44,976 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2088 2022-03-14 11:17:44,976 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2089 2022-03-14 11:17:44,977 [salt.utils.gitfs :2886][DEBUG   ][41791] Re-using gitfs object for process 41791
   2090 2022-03-14 11:17:44,977 [salt.fileclient  :1301][INFO    ][41791] Fetching file from saltenv 'base', ** done ** 'openssh/map.jinja'

[OrangeDog]

I'll close this then, as it does work in general, and there are already other issues for when it doesn't.