Providing the Puppet hieradata encryption keys as a terraform variable

[ocaisa]

Currently, one needs to bring a cluster up and manually extract/use (or replace) the Puppet hieradata encryption keys to create the profile::slurm::controller::tfe_token for cluster scaling. If one could provide the keys as a terraform variable (and from that deploy them on the puppet server) then you can set profile::slurm::controller::tfe_token in advance since you know the key, and get scaling working without the need for additional steps after the cluster comes up.

[ocaisa]

As a terraform variable, you can store/maintain the keys on Terraform Cloud.

[ocaisa]

This does mean the keys would become part of the terraform state, but I think this is ok since the use case is a cluster managed by Terraform Cloud.

The variable might look like:

puppet_pkcs7 = { public = file("public_key.pkcs7.pem"), private = file("private_key.pkcs7.pem") } 

and then you could have a conditional deploy step for the keys in https://github.com/ComputeCanada/magic_castle/blob/main/common/provision/main.tf

[cmd-ntrf]

I have started working on this.

• I think it is possible to generate the public key using the private key with openssl. Therefore, the user would only have to provide the private key, i.e:

  puppet_pkcs7 = file("private_key.pkcs7.pem")
  

  I was able to generate a file that eyaml seems to accept as a public key to encrypt secret using openssl and the eyaml generated private key like this:

  openssl req -new -x509 -key /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem -out p2.cer -days 100000 -subj '/'
  

  My mains source of inspiration for this was eyaml source code
• I think if we make the provision module responsible for deploying the private encrypting key, we will avoid having private key in the Terraform state. I would make it part of the cloud-init yaml only if there are no SSH jump host to copy the file, like we do for terraform_data.

[cmd-ntrf]

After thinking about this, one thing bugged me really bad: why do hiera-yaml need the public key to decrypt the secret? Public key is normally used to encrypt and private key to decrypt, why needing both when decrypting?

After reading PKCS7 RFC, I figured that what is called a "public key" in hiera-eyaml context, is actually a signing certificate that in the general context of PKCS7 can be used to validate the signature of the encrypted value while the private key is used to decrypt the encrypted text. The RFC mentions a situation where there no signing certificate. So I dug some more on the Ruby side of thing and eventually found out why hiera-eyaml requires the "public key" to decrypt: there was a bug in ruby-openssl. PKCS7#decrypt function did not support being provided nil for the signing certificate](rhenium/ruby-openssl@769b557).

This bug was fixed in ruby-openssl 2.2.1. The author of the decrypt function in hiera-eyaml mentionned that he noted that both the public and the private key had to be provided when implementing it.

Puppet 7 ships with ruby 2.7 and ruby-openssl 2.1.4, so no dice to get the decrypt function with public key optional. However, Puppet 8.6 ships with ruby 3.2.3 and ruby-openssl 3.1.0.

I wanted to test this, so I installed Puppet 8, hiera-eyaml, and patched:

/opt/puppetlabs/puppet/lib/ruby/gems/3.2.0/gems/hiera-eyaml-3.4.0/lib/hiera/backend/eyaml/encryptors/pkcs7.rb

replacing

pkcs7.decrypt(private_key_rsa, public_key_x509)

by

pkcs7.decrypt(private_key_rsa, nil)

and it works just fine!

[cmd-ntrf]

I submitted a PR to hiera-eyaml - voxpupuli/hiera-eyaml#364.

Since openssl is a default Ruby gem, it is bundled with a specific Ruby version and therefore depends on Puppet version. Puppet 7 uses Ruby 2.7, which highest openssl version is 2.1, see https://stdgems.org/default_gems.json. This unfortunately is too old. So to move forward with this, we will have to move to Puppet 8, which comes bundled with Ruby 3.2.

Steps to move forward with this:

• hiera-yaml PR 364 is merged and released or the other hiera-eyaml PR that drop X509 as the default public key format and no longer require the public key to decrypt.
• Define variable to allow user to provide its private RSA 2048 bits encryption key string.
• Make the provision module copy the key on the Puppet server
• Document how to generate an RSA 2048 bits encryption key with OpenSSL CLI and Terraform

In case none of my PR make it into hiera-eyaml, it is possible to generate a certificate with the private key with the serial number set to 1 as hiera-eyaml would do it. So we don't have to transmit the public key, just the private one.

openssl genrsa -out private_key.pkcs7.pem 2048
openssl req -new -key private_key.pkcs7.pem -out public_key.pkcs7.pem -set_serial 1 -batch