Option to have ArgoCD helm charts with meta.helm.sh annotations

[hanseltime]

Summary

Allow an argocd configuration option that will add meta.helm.sh annotations when provided.

In the argocd-cm.yaml, we could do:

helm.addMetaHelmAnnotations: true

This would then add the annotations:

meta.helm.sh/release-name: <release>
meta.helm.sh/release-namespace: <namespace>

Motivation

We use ArgoCD as our main visibility for status and configuration for our developers. However, we have to set up certain things before Argo is available (i.e. logging, cni, etc.). To do this, we need to use helm to deploy charts with the same values that we map into our ArgoCD deployment of the same chart. This allows us to have something like fluentbit up early (in case argocd has a failure) and then have ArgoCD take over managing that. In a reverse of that, we use ArgoCD as an upgrade testing UI for us to try tweaking particular settings on a helm chart and then backport them into IAC as needed.

The ArgoCD testing case is the bigger problem if it creates a new resource (ex: we created a new storageclass by updating a values.yaml), because helm will refuse to install since the created resource does not have the meta.helm.sh annotations. The only way to solve this is to either manually find the new resources and add the correct annotations or to delete the resource and then helm install - both of which make it much harder to keep an automatic CI process going

Since ArgoCD just uses the helm go pkg and it's template function, it would be ideal if helm supported this, however, due to this discussion, it seems clear that helm will not be providing this functionality.

Proposal

As the summary above describes, there would be a new configuration value for argocd called: helm.addMetaHelmAnnotations. This would be an opt-in feature so that existing users see no change in functionality (in case they have set up more complex workflows to handle the above described problem).

If the value is provided, when Applications or ApplicationSets create their helm charts with the release name and namespaces, we would update each manifest that was output with the annotation since we have those values at the moment of template rendering anyway.

[crenshaw-dev]

According to the linked thread, Helm uses those annotations to communicate to itself its ownership of those resources. It seems unwise for Argo CD to "trick" Helm. Neither Argo CD nor Helm are really designed for shared ownership of entire resources. I'm sympathetic to the comment "it seems to work, so why not ship it?" I'm just not sure that's enough to justify a first class feature.

Workarounds include writing a CMP or wrapping the chart with Kustomize and using commonAnnotations.

[hanseltime]

My understanding of ArgoCD's "render the template and then kubectl apply it" paradigm for helm was that it was more geared to tolerating shared ownership. There's no mechanism in ArgoCD that respects those ownership annotations so it's already very happy to overwrite something that helm has previously created (unless I'm missing some critical configuration property).

I wouldn't promote this as a default feature by any means, but since Argo is:

1. Not available for the earliest deployments (i.e. a late actor in manifest management during a new deploy)
2. The one trying to share a resource via a "helm implementation that doesn't enforce all helm semantics"

This feels like a natural feature set to stem the problems that Argo's loose ownership model creates.

I have considered writing a separate CMP for this as well, but that feels like a last-ditch effort since that would then require keeping parity with ArgoCD's helm rendering logic since any helm chart can reveal the need for this CMP during a later upgrade and most engineers will want to have the assurance that the helm functionality is the exact same as before, just with annotations.

As for wrapping in kustomize or even just specifying annotations at the application level, I'd like to avoid the burden on the user of ArgoCD to be certain that a large tooling change presents the same outputs as normal.

If the ArgoCD project definitively does not want to support this, I can work on a CMP. If this is something that Argo would support as opt-in, I would be willing to contribute it (to ease the burden).

[crenshaw-dev]

I'd like to avoid the burden on the user of ArgoCD to be certain that a large tooling change presents the same outputs as normal.

Can you rephrase? I'm not sure I follow.

If the ArgoCD project definitively does not want to support this, I can work on a CMP.

So far I personally lean "no," but we should probably discuss in the contributors call.

[hanseltime]

Can you rephrase?

I'll definitely try!

I'd like to avoid the burden on the user of ArgoCD to be certain that a large tooling change presents the same outputs as normal.

To try and clarify what I meant, I was thinking along the lines of:

• Let's say I'm the maintainer of a company's large Argo app of apps
• I run into this issue where someone updated an ArgoCD helm application that adds a new resource and now I realize that our CI/CD practices/processes for early deployment will all fail as soon as ArgoCD gets to deploy a manifest first for any helm chart - So I realize I need to categorically solve this for all helm charts that could ever be synced via helm externally.
• In the case of a CMP, I now need to vet the CMP plugin to see if it generates the same manifests as the previous argo native methods + the annotations I expect and then run regressions on every helm chart that I cut over
• In the case of kustomize, since I was just using helm before, I find myself in the same regression QA process and then additionally have to write up tooling/processes to ensure new engineers don't forget to include that same pattern.

I think what I was trying to get at was that in many organizations, the CMP/kustomize wrapper would come with the overhead of a migration since it is a new technology that may or may not behave the same as the base ArgoCD helm that we have come to trust and write tooling around. This is in contrast to the idea of opting-in to a flag that means my manifests use the same rendering mechanism but with 2 additional annotations.

I hope that clarifies my thoughts (whether they were any good or not). I'll keep an eye out for any resolution and appreciate your time!

[crenshaw-dev]

Gotcha, I think i follow now. Basically the features behavior is super simple and predictable, so you don't have to worry much about using it to solve this problem.

You bet! Will see what people think on the contributors call and report back.