Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt.minion fails when applied with salt-ssh #172

Open
TimJones opened this issue Aug 22, 2015 · 20 comments
Open

salt.minion fails when applied with salt-ssh #172

TimJones opened this issue Aug 22, 2015 · 20 comments

Comments

@TimJones
Copy link

I just posted this issue with SaltStack as I believe the bug exists in salt-ssh rather than in this formula directly. This is really just for visibility.

Brief summary:
When calling salt.minion via salt-ssh, the formulas.jinja file is not included and so the minion.d/f_defaults.conf file fails to render.

Workaround is to simply modify the following lines in salt/files/minion.d/f_defaults.conf:

@@ -12,7 +12,6 @@
 #{{ configname }}: {{ default_value }}
 {%- endif -%}
 {%- endmacro -%}
-{%- from 'salt/formulas.jinja' import file_roots, formulas with context -%}
 ##### Primary configuration settings #####
 ##########################################

@@ -235,8 +234,6 @@ file_client: local
 {{ file_roots(cfg_minion['file_roots']) }}
 {%- elif 'file_roots' in cfg_salt -%}
 {{ file_roots(cfg_salt['file_roots']) }}
-{%- elif formulas|length -%}
-{{ file_roots({'base': ['/srv/salt']}) }}
 {%- endif %}

 # limit fileserver traversal
@tiger-seo
Copy link
Contributor

+1

@TimJones
Copy link
Author

As mentioned in the SaltStack issue, another work-around with salt-ssh is to mark the file as an external resource:

salt-ssh 'minion-id' state.sls salt.minion --extra-filerefs salt://salt/formulas.jinja

@tiger-seo
Copy link
Contributor

workaround doesn't work for me

    - Rendering SLS 'base:salt.ssh' failed: Jinja error: salt/map.jinja
    - Traceback (most recent call last):
    -   File "/usr/lib/python2.6/site-packages/salt/utils/templates.py", line 306, in render_jinja_tmpl
    -     output = template.render(**decoded_context)
    -   File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 969, in render
    -     return self.environment.handle_exception(exc_info, True)
    -   File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 742, in handle_exception
    -     reraise(exc_type, exc_value, tb)
    -   File "<template>", line 1, in top-level template code
    -   File "/usr/lib/python2.6/site-packages/salt/utils/jinja.py", line 143, in get_source
    -     raise TemplateNotFound(template)
    - TemplateNotFound: salt/map.jinja
    -
    - ; line 1
    -
    - ---
    - {% from "salt/map.jinja" import salt_settings with context %}    <======================
    -
    - {% if salt_settings.install_packages %}
    - ensure-salt-ssh-is-installed:
    -   pkg.installed:
    -     - name: {{ salt_settings.salt_ssh }}
    - [...]
    - ---

@iggy
Copy link
Contributor

iggy commented Aug 25, 2015

Think yours might be a different problem... that's map.jinja (vs formulas.jinja)

@tiger-seo
Copy link
Contributor

@iggy
Copy link
Contributor

iggy commented Aug 26, 2015

So you'd have to add map.jinja to the extra filerefs

@tiger-seo
Copy link
Contributor

doesn't help :(

@zenitraM
Copy link

I can confirm trying to use salt-ssh to provision salt.minion using this formula from gitfs still fails with a jinja2.exceptions.TemplateNotFound: salt/map.jinja.

Adding both salt://salt/formulas.jinja and salt://salt/map.jinja to extra-filerefs still doesn't seem to fix it when loading the formula from gitfs - apparently due to saltstack/salt#19564.

@myii
Copy link
Member

myii commented Jan 21, 2020

@zenitraM This is almost definitely due to defaults.merge in map.jinja, as we had with the template-formula:

If I can provide you with an updated map.jinja, will you be able to test it in your environment to confirm that it works or not? If it does, we'll get that merged into the formula.

@zenitraM
Copy link

@myii sure thing!

@myii
Copy link
Member

myii commented Jan 22, 2020

@zenitraM This should do the trick. Use this as your map.jinja instead:

# -*- coding: utf-8 -*-
# vim: ft=jinja

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{#- Start imports as #}
{%- import_yaml tplroot ~ "/defaults.yaml" as default_settings %}
{%- import_yaml tplroot ~ "/osfamilymap.yaml" as osfamilymap %}
{%- import_yaml tplroot ~ "/osmap.yaml" as osmap %}

{#- Retrieve the config dict only once #}
{%- set _config = salt['config.get'](tplroot, default={}) %}

{%- set defaults = salt['grains.filter_by'](
      default_settings,
      default=tplroot,
      merge=salt['grains.filter_by'](
        osfamilymap,
        grain='os_family',
        merge=salt['grains.filter_by'](
          osmap,
          grain='os',
          merge=salt['grains.filter_by'](
            _config,
            default='lookup'
          )
        )
      )
    )
%}

{%- set config = salt['grains.filter_by'](
      {'defaults': defaults},
      default='defaults',
      merge=_config
    )
%}

{%- set salt_settings = config %}
{%- set formulas_settings = salt['pillar.get']('salt_formulas',default=default_settings['salt_formulas'], merge=True) %}

@windowsrefund
Copy link

This updated map.jinja produces the same TemplateNotFound: salt/map.jinja Traceback for me using a fresh install of Salt 3000

@myii
Copy link
Member

myii commented Mar 11, 2020

@windowsrefund It's difficult to troubleshoot without further details of your setup.

@baby-gnu If you get a chance, can you try this out on salt-ssh with the updated map.jinja I've proposed above: #172 (comment)?

@baby-gnu
Copy link
Contributor

Hello @myii, I tried with the following pillar:

# -*- coding: utf-8 -*-
# vim: ft=yaml
---
salt:
  master:
    fileserver_backend:
      - rootfs
    file_roots:
      base:
        - /srv/salt
    pillar_roots:
      base:
        - /srv/pillar

it's failing with the following error:

salt-ssh testmachine state.apply salt test=True
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 392, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 6, in <module>
TypeError: get() missing 1 required positional argument: 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 167, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 438, in render_jinja_tmpl
    raise SaltRenderError('Jinja error: {0}{1}'.format(exc, out),
salt.exceptions.SaltRenderError: Jinja error: get() missing 1 required positional argument: 'default'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 392, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 6, in <module>
TypeError: get() missing 1 required positional argument: 'default'

; line 6

---
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
  - salt.pkgrepo
      {%- if salt.config.get('salt_formulas:list') %}    <======================
  - salt.formulas
      {%- endif %}
      {%- if salt.config.get('salt:master')|length > 1 %}
  - salt.master
      {%- endif %}
[...]
---
[CRITICAL] Rendering SLS 'base:salt' failed: Jinja error: get() missing 1 required positional argument: 'default'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 392, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 6, in <module>
TypeError: get() missing 1 required positional argument: 'default'

; line 6

---
# -*- coding: utf-8 -*-
# vim: ft=sls

include:
  - salt.pkgrepo
      {%- if salt.config.get('salt_formulas:list') %}    <======================
  - salt.formulas
      {%- endif %}
      {%- if salt.config.get('salt:master')|length > 1 %}
  - salt.master
      {%- endif %}
[...]
---
testmachine:
    - Rendering SLS 'base:salt' failed: Jinja error: get() missing 1 required positional argument: 'default'
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 392, in render_jinja_tmpl
          output = template.render(**decoded_context)
        File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
          return original_render(self, *args, **kwargs)
        File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
          return self.environment.handle_exception(exc_info, True)
        File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
          reraise(exc_type, exc_value, tb)
        File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
          raise value.with_traceback(tb)
        File "<template>", line 6, in <module>
      TypeError: get() missing 1 required positional argument: 'default'
      
      ; line 6
      
      ---
      # -*- coding: utf-8 -*-
      # vim: ft=sls
      
      include:
        - salt.pkgrepo
            {%- if salt.config.get('salt_formulas:list') %}    <======================
        - salt.formulas
            {%- endif %}
            {%- if salt.config.get('salt:master')|length > 1 %}
        - salt.master
            {%- endif %}
      [...]
      ---
Salt Version:
           Salt: 3000
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10.1
        libgit2: 0.28.3
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 1.0.3
         Python: 3.8.2 (default, Feb 25 2020, 13:04:52)
   python-gnupg: Not Installed
         PyYAML: 5.3
          PyZMQ: 18.1.1
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2
 
System Versions:
           dist: debian unstable sid
         locale: utf-8
        machine: x86_64
        release: 5.4.0-4-amd64
         system: Linux
        version: Debian GNU/Linux unstable sid

@baby-gnu
Copy link
Contributor

I'm not using gitfs.

It looks like the import from a Jinja2 template is not working as for sls files.

As an example, here is a test:

  • test/init.sls
    {%- from "test/test.jinja" import testvar %}
    
    test-a-file:
      file.managed:
        - name: /tmp/test.txt
        - source: salt://test/test-file
        - template: jinja
        - context:
            foo: {{ testvar }}
    
  • test/test.jinja
    {%- set testvar = "bar" %}
  • test/test-file
    From context '{{ foo }}'
    
    {%- from "test/test.jinja" import testvar %}
    From import '{{ testvar }}'

When running:

salt-ssh testmachine state.apply test test=True
testmachine:
----------
          ID: test-a-file
    Function: file.managed
        Name: /tmp/test.txt
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/var/tmp/.root_482cf9_salt/pyall/salt/utils/templates.py", line 392, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/asyncsupport.py", line 76, in render
                  return original_render(self, *args, **kwargs)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/environment.py", line 1008, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/environment.py", line 780, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/_compat.py", line 37, in reraise
                  raise value.with_traceback(tb)
                File "<template>", line 3, in top-level template code
                File "/var/tmp/.root_482cf9_salt/pyall/salt/utils/jinja.py", line 196, in get_source
                  raise TemplateNotFound(template)
              jinja2.exceptions.TemplateNotFound: test/test.jinja
              
              During handling of the above exception, another exception occurred:
              
              Traceback (most recent call last):
                File "/var/tmp/.root_482cf9_salt/pyall/salt/state.py", line 1981, in call
                  **cdata['kwargs'])
                File "/var/tmp/.root_482cf9_salt/pyall/salt/loader.py", line 1977, in wrapper
                  return f(*args, **kwargs)
                File "/var/tmp/.root_482cf9_salt/pyall/salt/states/file.py", line 3041, in managed
                  **kwargs
                File "/var/tmp/.root_482cf9_salt/pyall/salt/modules/file.py", line 4850, in check_managed_changes
                  **kwargs)
                File "/var/tmp/.root_482cf9_salt/pyall/salt/modules/file.py", line 4291, in get_managed
                  **kwargs)
                File "/var/tmp/.root_482cf9_salt/pyall/salt/utils/templates.py", line 167, in render_tmpl
                  output = render_str(tmplstr, context, tmplpath)
                File "/var/tmp/.root_482cf9_salt/pyall/salt/utils/templates.py", line 441, in render_jinja_tmpl
                  trace=tracestr)
              salt.exceptions.SaltRenderError: Jinja error: test/test.jinja
              Traceback (most recent call last):
                File "/var/tmp/.root_482cf9_salt/pyall/salt/utils/templates.py", line 392, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/asyncsupport.py", line 76, in render
                  return original_render(self, *args, **kwargs)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/environment.py", line 1008, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/environment.py", line 780, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "/var/tmp/.root_482cf9_salt/pyall/jinja2/_compat.py", line 37, in reraise
                  raise value.with_traceback(tb)
                File "<template>", line 3, in top-level template code
                File "/var/tmp/.root_482cf9_salt/pyall/salt/utils/jinja.py", line 196, in get_source
                  raise TemplateNotFound(template)
              jinja2.exceptions.TemplateNotFound: test/test.jinja
              
              ; line 3
              
              ---
              From context '{{ foo }}'
              
              {%- from "test/test.jinja" import testvar %}    <======================
              From import '{{ testvar }}'
              
              Traceback (most recent call last):
                File "/var/tmp/.root_482cf9_salt/pyall/salt/utils/templates.py", line 392, in render_jinja_tmpl
                  output = template.render(**decoded_context)
              [...]
              ---
     Started: 11:23:35.764784
    Duration: 111.883 ms
     Changes:   

Summary for testmachine
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time: 111.883 ms

When I remove all the import from test-file

salt-ssh testmachine state.apply test test=True
testmachine:
----------
          ID: test-a-file
    Function: file.managed
        Name: /tmp/test.txt
      Result: None
     Comment: The file /tmp/test.txt is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.
     Started: 11:24:51.484942
    Duration: 73.155 ms
     Changes:   
              ----------
              newfile:
                  /tmp/test.txt

Summary for testmachine
------------
Succeeded: 1 (unchanged=1, changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  73.155 ms

I found several references in salt bug tracker:

Another argument for passing context ? 😉

@baby-gnu
Copy link
Contributor

So, there is 2 issues:

  1. the map.jinja should not use defaults.merge
  2. the configuration files should not import or include

@myii
Copy link
Member

myii commented Mar 11, 2020

@baby-gnu Thanks for digging down into this. So didn't the suggested map.jinja make a difference, since it removes defaults.merge? Or is that where you got stuck with {%- if salt.config.get('salt_formulas:list') %}?

@baby-gnu
Copy link
Contributor

@myii no difference since the TypeError: get() missing 1 required positional argument: 'default' fails before trying to import map.jinja.

@myii
Copy link
Member

myii commented Mar 11, 2020

@baby-gnu Looks like we'll have our work cut out when we try to introduce the new map.jinja to this formula...

@baby-gnu
Copy link
Contributor

For the TemplateNotFound issue I may have an idea to solve it cleanly saltstack/salt#58762.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants