feature request: template with on-the-fly consul-kv / vault-kv backend support

[dmclf]

Proposal

template block now supports two source definitions

• data as file-as-string-representation
• source , to source an artifact
  • which then supports downloading http, https, git, hg and S3 artifacts

request:

• would it be possible to also do some on-the-fly saving of a file to consul-kv
  • to keep the nomad jobspec tidier / smaller
  • ease templating

Use-cases

example..

      driver = "docker"
      kill_timeout = "5s"
      leader = "false"
      template {
        data-to-save-to-consul-or-vault = path/to/file
        destination = "secrets/my-file"
        env         = false
}

this would make templating more 'native' to the nomad cli and keep the jobspec cleaner

Attempted Solutions

data = <<EOF
large-blob
EOF

can get out of hand as docs state for larger files

and the data -> source <-> artifact path also is not the most convenient to setup.

example, if Nomad can upload these files on-the-fly, then from a gitlab CI one would then easily be able to regenerate the files.
however, if one depends on artifacts and http, yes, it can work from gitlab-ci but one would need long-living tokens, so no deploy-specific tokens, to make that work longer term.

example: you deploy your job on the first day of the week, but next week the job restarts and tokens expired, job fails to fetch artifacts and falls over.

and with artifact, these tokens (or ssh-keys) then will be part of the nomad-jobspec , which does not seem very secure?

security concerns:

(reference)[https://developer.hashicorp.com/nomad/docs/job-specification/artifact#download-file]

1. To set HTTP headers in the request for the source the optional headers field can be configured.

• plaintext part of nomad jobspec. (for example, a gitlab token would need to be unsecurely exposed)

  headers {
    User-Agent    = "nomad-[${NOMAD_JOB_ID}]-[${NOMAD_GROUP_NAME}]-[${NOMAD_TASK_NAME}]"
    X-Nomad-Alloc = "${NOMAD_ALLOC_ID}"
  }

2. To use HTTP basic authentication, preprend the username and password to the hostname in the URL.

• plaintext part of nomad jobspec, user pass exposed.

artifact {
  source = "https://exampleUser:pass%2Fword%[email protected]/file.txt"
}

3. To download from a private repo, sshkey needs to be set.

• plaintext part of nomad jobspec, private key would be insecurely exposed in plain base64 encoded string.

  options {
    # Make sure that the system known hosts file is populated:
    # ssh-keyscan github.com | sudo tee -a /etc/ssh/ssh_known_hosts
    # https://github.com/hashicorp/go-getter/issues/55
    sshkey = "${base64encode(file("/path/to/private-key"))}"
  }

so writing to a (secure path) of consul-kv (or even vault-kv), natively managed by nomad,
might be a lot more hashicorp-native, cleaner and safer way?

[tgross]

Hi @dmclf! The workflow you're describing is a little unclear to me. I understand that you want the template block to write data to some path in Consul or Vault KV? (Or, presumably, Nomad Variables.) But your example doesn't have a source for that data. Are you suggesting that we'd render the template and then write it back to a KV afterwards? How does writing it back to a KV help in failure scenarios -- the template's source data can't be the same KV without hitting a chicken-and-the-egg problem.

Can you provide a full proposed template block with a motivating example to help us understand what you're looking to do here?

[dmclf]

hi @tgross
Sure, let me try to be a bit clearer, imagine you have a gitlab CI where you have levant based job-spec rendering

and then in the jobspec now roughly you have something like (dry-jobs-levant

      template {
                data = <<EOH
[[ fileContents "nomad/template/secretfile" ]]
EOH
        destination = "secrets/secretfile"
        env         = false
}

then that may work fine for small files, although when having bigger jobs that span several groups with a good amount of tasks per group that all have a common used secret, the jobspec itself can grow rapidly.

and yes artifacts can work, but may have several security issues too.

so I was wondering if Nomad more 'natively' , example..

• instead of levant's fileContents
• there would be some kind of logic where the fileContents get pushed to a nomad-connected kv (vault/consul)
• and instead of the full fileContents in the nomad jobspec, the jobspec, like levant, would only contain a reference to that kv to include.

lets say it turns it to

      template {
        file_to_upload_and_use_as_source = nomad/template/secretfile
        destination = "secrets/secretfile"
        env         = false
}

where nomad/template/secretfile gets transformed to , ex, a consul-kv entry.

on final nomad jobspec, you would only see this special reference

and practically, nomad would use that consul-kv in a same way as the 'artifact' (ie, download/render)

where in the end the following would be achieved

• smaller jobspecs
• easier adoption for end-user
• safe(r) storage of (secret) files (compared to artifacts with plaintext passwords/sshkeys/tokens on jobspec for private repo's)

but if this would only make things simpler for the end-user, but requires a big amount of changes on Nomad side that do not keep things simple, then I guess this may likely not happen.

(I guess it logically should be more part of the 'templating / abstraction' phase, before any 'nomad job' cmd's?)

ok, maybe will just close it as I guess this likely won't be accepted, seeing Nomad also wants to keep things simple (on Nomad side)

just my setup now is generic usable levant based templating with json-files through gitlab-ci steps that everyone seems to easily understand
sadly just we have some jobs that are... 'special', and may benefit from some optimisations, but I suppose i'll have to figure out other steps to reduce job size..

[tgross]

Ok to close, but just for anyone who comes along later looking for something like this...

Just to clarify, you're looking for a workflow where:

1. The initial contents of the template data are from a file on disk (which you can do today with HCL's file function)
2. After the allocation is placed, the template is rendered normally, pulling data from Consul/Vault/Nomad as needed.
3. And then the rendered content is written back to KV.
4. If the allocation is rescheduled, the template skips normal rendering and instead is read back from KV.

As I've noted, the new feature here would be just steps 2-4. This logic would be mostly contained in the consul-template project. The cluster admin would likely have some challenging work to do to set up the appropriate ACL policies for minting Consul/Vault tokens from Workload Identity as well.

But while the implementation of this would be relatively complex, I think it has more fundamental design problems:

• There can be multiple copies of the task in the cluster and there's no guarantee they have the same rendered contents. Ex. suppose the template body includes {{ env "NOMAD_ALLOC_ID" }}; it'd be last-write-wins among all the allocations, but only on initial deployment because replacement allocations and updates would use the contents from KV.
• If you have to pull the rendered content from KV, then there's no way to update the content from Consul/Vault/Nomad if it changes. If you don't have data that changes, you might as well just use the file function and then you're done.

[tgross]

Addendum: if it weren't for the "write back to KV" aspect of this, I could imagine a feature where we have something like this:

template {
  consul_kv_source = "foo/bar"
  destination = "/secret/mysecret.txt"
}

that gets automatically transformed behind the scenes into:

template {
  data = "{{ key: \"foo/bar\"}}"
  destination = "/secret/mysecret.txt" 
}

But that only works for Consul's API. For Vault KV and Nomad Variables you need to dereference the items in the value by name as well.

[dmclf]

In the situation I envisioned I'd indeed see the KV as the 'template' source, per your Addendum, and not rendered/write back 👌

I suppose, file also is a bit hard to work with if you have a fairly sized cluster.. (pre-copy file to all possible clients on each deploy, prefer more of a 'single point of entry' for such a source, and shared storage may work.. but can introduce other issues / bottlenecks )

maybe i'll just have to reconsider the artifact section , accept that its security is not great..
and have fun trying to modify the current templating workflow to upload and reference artifacts..
or, look at other options.
or just accept the situation.

[tgross]

Ok, but if you're not writing back, then you can just do:

template {
  data = "{{ key: \"foo/bar\"}}"
  destination = "/secret/mysecret.txt" 
}

And then you're done, for Consul.

For Vault the equivalent would be:

template {
  data = "{{with secret \"foo/bar\"}}{{index .Data \"myfield\"}}{{end}}"
  destination = "/secret/mysecret.txt"
}

For Nomad the equivalent would be:

template {
  data = "{{ with nomadVar \"foo/bar\" }}{{ .myfield }}{{ end }}"
  destination = "/secret/mysecret.txt"
}

}

This wouldn't let you render a template nested in that template, but that's an open issue for CT here: hashicorp/consul-template#1981

[dmclf]

Hi @tgross
to clarify, but yes, i'm currently heavily using this template-block with data = <<EOH [..] EOH syntax like

      template {
                data = <<EOH
[[ fileContents "nomad/template/secretfile" ]]
EOH
                destination = "/secret/mysecret.txt" 

but for optimizations I was wondering about other approaches to not make the actual file part of the jobspec (and which wouldnt be 'artifact' based)

anyway, thank you for your input,

I will take a few moments to consider possible next steps to improve on our end, and I may keep an eye on that CT issue you mentioned and see if that possibly can come in handy in the future.