Secrets rotation improvements

[romanfurst]

Hello,

in our enterprise company, we are facing challenges with secrets rotation. Since we have dozens of independent development teams (each with its own Kubernetes cluster) and, as a result, dozens or even hundreds of CyberArk accounts, we require a stable and user-friendly solution that also takes into consideration the utilization of Kubernetes resources.

Additionally, we have determined that the rotation process should be performed without requiring a restart of the microservice containers. To achieve a seamless password rotation, we have implemented a dual-account principle, although it is beyond the scope of this text.

Current password rotation design with the push-to-file concept has some downsides for us:

• Every POD needs its own sidecar. Since we already have istio envoy running in each POD, adding another sidecar container would be excessive for the Kubernetes cluster.
• If the deployment is running in multiple instances, each POD's conjur-provider sidecar duplicates the same call. The same data is retrieved for every instance.
• In some cases, we might prefer using secret values as environment variables rather than consuming them from a file.

Alternatively, we could deploy the secret-provider as a Kubernetes scheduled Job, but it doesn't fully meet our requirements.

So, we have come up with a custom design solution as follows:

• The secrets-provider runs as a standalone Deployment POD in Kubernetes (k8s)
• The secrets-provider can operate with all k8s Secrets within a given namespace
• The secrets-provider manipulates only labeled Secrets, eliminating the need for hardcoded Secret names in the conjur-provider configuration
• The secrets-provider periodically performs secrets provisioning based on a configured time interval
• The secrets-provider has registered a k8s watcher on labeled k8s Secrets. This ensures that whenever a new k8s Secret is deployed or an existing one is modified, secrets provisioning is immediately triggered for it
• The configuration is compatible with the conjur.org/conjur-secrets.{secret-group} and conjur.org/secret-file-template.{secret-group} configuration annotations. However, instead of pushing the rendered template to a file, it is written into the k8s Secrets, where {secret-group} is used as the secret's item key

Advantages:

• Only one instance of the secrets-provider needs to be deployed in the cluster
• Easy addition or removal of interested k8s Secrets via labeling
• Since the provider periodically checks and provides Conjur secrets, it is suitable for password rotation scenarios
• If an application POD is running in multiple instances and consumes data from a single k8s Secret, there is no need to run conjur-provider separately for each instance
• If the secret's data items are mounted into the container's filesystem as a file, any changes in the k8s Secret are automatically reflected in the mounted file within the container (this depends on the application framework's ability to reload changed configuration files on-the-fly during runtime, eliminating the need to restart the application container)

Here is high level scheme:

Here is detailed architecture:

0… conjur-secrets-provider is deployed and watch all labeled k8s Secrets
1… microservice with its k8s Secret is deployed with mounted Secret data item as a file
2… conjur-secrets-provider detect the new labeled k8s Secret
3… conjur-secrets-provider run provision process and retrieve accounts from Conjur
4… conjur-secrets-provider inject retrieved accounts values into the k8s Secret
5… new k8s Secret values are projected to container mounted config file
6… microservice runtime framework read the new values from the config file
7… after time period conjur-secret-provider run provision process and retrieve accounts from Conjur for all labeled k8s Secrets
8… if there are account changes it is injected into appropriete k8s Secrets
9… ad 5)
10… ad 6)

I wonder if it looks interesting to you. Does it make sense to adopt the solution or some of its parts into the current solution? If you are interested, I can provide a merge request with the described implementation changes above and we can tune up details.