[APM Onboarding] Disable APM Instrumentation on Datadog namespace #1252

liliyadd · 2023-11-20T19:35:54Z

What this PR does / why we need it:

This PR adds the namespace where the datadog-agent is deployed to the exclude-list of APM Single Step Instrumentation. It's important to exclude cluster-agent deployment and agent daemonset by default to make sure sure that cluster-agent pods creation is not blocked if the admission-controller is not available (if cluster-agent pods are not ready/available).

Special notes for your reviewer:

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

Chart Version bumped
Documentation has been updated with helm-docs (run: .github/helm-docs.sh)
CHANGELOG.md has been updated
Variables are documented in the README.md
For Datadog Operator chart or value changes update the test baselines (run: make update-test-baselines)

…-ssi-on-dd-agent

charts/datadog/templates/_helpers.tpl

Kyle-Verhoog

We should be sure to document in the public manual instructions when not using the helm chart/operator to exclude the Agent namespace as well. I don't think we have any public docs on the setting yet so this would be future work.

Kyle-Verhoog · 2023-11-20T21:33:08Z

charts/datadog/templates/cluster-agent-deployment.yaml

          - name: DD_APM_INSTRUMENTATION_ENABLED
            value: "true"
          {{- end }}
          {{- if .Values.datadog.apm.instrumentation.enabledNamespaces }}
          - name: DD_APM_INSTRUMENTATION_ENABLED_NAMESPACES
            value: {{ .Values.datadog.apm.instrumentation.enabledNamespaces | toJson | quote }}
          {{- end }}
-          {{- if .Values.datadog.apm.instrumentation.disabledNamespaces }}
+          {{- if ne (include "apmInstrumentation.disabledNamespaces" .) "" }}


Should we default apmInstrumentation.disabledNamespaces to an empty list when not defined to simplify this check?

apmInstrumentation.disabledNamespaces is a string; it's "" by default.

https://github.com/DataDog/helm-charts/blob/main/charts/datadog/templates/_helpers.tpl#L618-L631 has similar pattern and usage https://github.com/DataDog/helm-charts/blob/main/charts/datadog/templates/_containers-common-env.yaml#L27-L30

…-ssi-on-dd-agent

clamoriniere · 2023-11-21T08:39:26Z

charts/datadog/templates/cluster-agent-deployment.yaml

+          {{- if .Values.datadog.apm.instrumentation.enabled }}
          - name: DD_APM_INSTRUMENTATION_ENABLED
            value: "true"
          {{- end }}


why the {{- if .Values.datadog.apm.instrumentation.enabled }} doesn't inclose the other apmInstrumentation options?
We usually use this patern (have a global setting like .Values.datadog.apm.instrumentation.enabled to enable or disable a full feature). And I suppose it doesn't make sense to not set DD_APM_INSTRUMENTATION_ENABLED but we still inject the DD_APM_INSTRUMENTATION_DISABLED_NAMESPACES or DD_APM_INSTRUMENTATION_ENABLED_NAMESPACES

As per RFC:
To allow more granular control of the injection process, Single Step Instrumentation introduces a set of new configuration parameters on the Cluster Agent:
datadog.apm.instrumentation.enabled - boolean indicating if automatic injection should be enabled in the whole cluster. Default value - false.
datadog.apm.instrumentation.enabledNamespaces - comma-separated list of namespaces where automatic injection is enabled. Default value - empty list. This setting applies only when cluster-level configuration datadog.apm.instrumentation.enabled is set to false.
datadog.apm.instrumentation.disabledNamespaces - comma-separated list of namespaces where automatic injection needs to be disabled. Applies only when datadog.apm.instrumentation.enabled is set to true. Default value - empty list. Single Step Instrumentation will always skip injections in Kubernetes system namespace kube-system.

This question was raised in Operator PR as well here.

Hi @liliyadd

I think the behavior should be changed, because all the other features in the agent don't use the same pattern, and IMO the current behavior is error-prone.

What I think can make more sense is:

datadog.apm.instrumentation.enabled - boolean indicating if automatic injection should be enabled

datadog.apm.instrumentation.enabledNamespaces - comma-separated list of namespaces where automatic injection is enabled. (Default value - empty string). If value is empty, automatic injection is enabled on the whole cluster*. (*except kube-systems, and the "agent install" namespace).

datadog.apm.instrumentation.disabledNamespaces - comma-separated list of namespaces where automatic injection needs to be disabled. (Default value - empty string). If disabledNamespaces automatic injection is active on every namespace except the one present in disabledNamespaces and kube-systems, and the "agent install" namespace.

I think this change can still be done, and make more sense for cross agent configuration genericity.

clamoriniere · 2023-11-21T08:43:42Z

charts/datadog/templates/_helpers.tpl

+{{/*
+Return all namespaces with disabled Single Step Instrumentation
+*/}}
+{{- define "apmInstrumentation.disabledNamespaces" -}}


General behaviour question what happens if disabledNamespaces is never empty?

Based on the other PR #1211 and specifically this part of the code https://github.com/DataDog/helm-charts/pull/1211/files#diff-5dec0964fafbdc987c2b122b6f38a16f03c7129501b7cb14fd5bbe37afb2cd0dR168
I think the user will not be able to set the enabledNamespaces option.

Is it the expected behaviour? should have a precedence between the option? do we have a reason to not allow both options at the same time?

The main reason not to allow disabledNamespaces and enabledNamespaces set at the same time is to prevent unspecified behavior when a namespace is present in both lists.

disabledNamespaces shouldn't be empty only when SSI is on for the cluster and we just want disable it in the few namespaces.

If users set enabledNamespaces, generally it means that SSI is disable in the whole cluster.

liliyadd · 2023-11-21T18:02:53Z

Test cases:

Disabled namespaces set, datadog running in ns default
values.yaml

    # APM Single Step Instrumentation
    # This feature is in beta. It requires Cluster Agent 7.49+.
    instrumentation:
      # datadog.apm.instrumentation.enabled -- Enable injecting the Datadog APM libraries into all pods in the cluster (beta).
      enabled: true

      # datadog.apm.instrumentation.enabledNamespaces -- Enable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      enabledNamespaces: []

      # datadog.apm.instrumentation.disabledNamespaces -- Disable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      disabledNamespaces:
      - test1
      - test2

      # datadog.apm.instrumentation.libVersions -- Inject specific version of tracing libraries with Single Step Instrumentation (beta).
      libVersions: {}

Result env variables on cluster-agent:

          - name: DD_APM_INSTRUMENTATION_ENABLED
            value: "true"
          - name: DD_APM_INSTRUMENTATION_DISABLED_NAMESPACES
            value: "[\"test1\",\"test2\",\"default\"]"

Disabled namespaces not set, atadog running in ns default
values.yaml

    # APM Single Step Instrumentation
    # This feature is in beta. It requires Cluster Agent 7.49+.
    instrumentation:
      # datadog.apm.instrumentation.enabled -- Enable injecting the Datadog APM libraries into all pods in the cluster (beta).
      enabled: true

      # datadog.apm.instrumentation.enabledNamespaces -- Enable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      enabledNamespaces: []

      # datadog.apm.instrumentation.disabledNamespaces -- Disable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      disabledNamespaces: []

      # datadog.apm.instrumentation.libVersions -- Inject specific version of tracing libraries with Single Step Instrumentation (beta).
      libVersions: {}

Result env variables on cluster-agent:

          - name: DD_APM_INSTRUMENTATION_ENABLED
            value: "true"
          - name: DD_APM_INSTRUMENTATION_DISABLED_NAMESPACES
            value: "[\"default\"]"

Enabled namespaces has default namespace
values.yaml

    # APM Single Step Instrumentation
    # This feature is in beta. It requires Cluster Agent 7.49+.
    instrumentation:
      # datadog.apm.instrumentation.enabledNamespaces -- Enable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      enabledNamespaces: [test1, default]

      # datadog.apm.instrumentation.disabledNamespaces -- Disable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      disabledNamespaces: []

      # datadog.apm.instrumentation.libVersions -- Inject specific version of tracing libraries with Single Step Instrumentation (beta).
      libVersions: {}

Result env variables on cluster-agent:

          - name: DD_APM_INSTRUMENTATION_ENABLED_NAMESPACES
            value: "[\"test1\"]"

Enabled namespaces has default namespace, with instrumentation.enabled='true'
values.yaml

    instrumentation:
      # datadog.apm.instrumentation.enabled -- Enable injecting the Datadog APM libraries into all pods in the cluster (beta).
      enabled: true

      # datadog.apm.instrumentation.enabledNamespaces -- Enable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      enabledNamespaces: [test1, default]

      # datadog.apm.instrumentation.disabledNamespaces -- Disable injecting the Datadog APM libraries into pods in specific namespaces (beta).
      disabledNamespaces: []

      # datadog.apm.instrumentation.libVersions -- Inject specific version of tracing libraries with Single Step Instrumentation (beta).
      libVersions: {}

Result env variables on cluster-agent:

          - name: DD_APM_INSTRUMENTATION_ENABLED
            value: "true"
          - name: DD_APM_INSTRUMENTATION_DISABLED_NAMESPACES
            value: "[\"default\"]"

levan-m · 2023-11-21T23:29:16Z

Looks good to me.

I tried various combinations of enabled, enabledNamespace disabledNamespaces and chart renders correct env variables.

Disable APM Instrumentation on Datadog namespace

cc582e2

liliyadd requested review from clamoriniere and a team November 20, 2023 19:35

liliyadd requested a review from a team as a code owner November 20, 2023 19:35

liliyadd added 2 commits November 20, 2023 14:36

Add empty line to .helpers

769090f

Merge remote-tracking branch 'origin/main' into liliya.belaus/disable…

ee33230

…-ssi-on-dd-agent

Kyle-Verhoog reviewed Nov 20, 2023

View reviewed changes

charts/datadog/templates/_helpers.tpl Show resolved Hide resolved

Normalize disabledNamespaces to list

aa89c38

Kyle-Verhoog approved these changes Nov 20, 2023

View reviewed changes

liliyadd requested a review from levan-m November 20, 2023 22:46

Merge remote-tracking branch 'origin/main' into liliya.belaus/disable…

2f7bb74

…-ssi-on-dd-agent

clamoriniere reviewed Nov 21, 2023

View reviewed changes

liliyadd added 2 commits November 21, 2023 12:13

Remove datadog namespace from enabledNamespaces list

8e4cebe

Fix usage of .Release.Namespace

6e5b351

liliyadd added 2 commits November 21, 2023 13:03

Fix enabledNamespaces option

c9ef11f

Revert values.yaml to correct default values

9decb90

liliyadd requested a review from a team November 21, 2023 22:45

levan-m approved these changes Nov 21, 2023

View reviewed changes

levan-m merged commit e18b2b7 into main Nov 21, 2023
14 checks passed

levan-m deleted the liliya.belaus/disable-ssi-on-dd-agent branch November 21, 2023 23:35

levan-m mentioned this pull request Nov 22, 2023

Add APM Single Step Instrumentation DataDog/datadog-operator#981

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM Onboarding] Disable APM Instrumentation on Datadog namespace #1252

[APM Onboarding] Disable APM Instrumentation on Datadog namespace #1252

liliyadd commented Nov 20, 2023

Kyle-Verhoog left a comment

Kyle-Verhoog Nov 20, 2023

liliyadd Nov 20, 2023

clamoriniere Nov 21, 2023

liliyadd Nov 21, 2023

levan-m Nov 21, 2023

clamoriniere Nov 22, 2023

clamoriniere Nov 21, 2023

liliyadd Nov 21, 2023

liliyadd commented Nov 21, 2023

levan-m commented Nov 21, 2023

[APM Onboarding] Disable APM Instrumentation on Datadog namespace #1252

[APM Onboarding] Disable APM Instrumentation on Datadog namespace #1252

Conversation

liliyadd commented Nov 20, 2023

What this PR does / why we need it:

Special notes for your reviewer:

Checklist

Kyle-Verhoog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liliyadd commented Nov 21, 2023

Test cases:

levan-m commented Nov 21, 2023