Authentication failures for Datadog scaler

[karolorlamowski]

Report

Getting 403 error when I try to use Datadog scaler following this steps https://keda.sh/docs/2.7/scalers/datadog/

There is no problems when I try invoke API from local machine via CURL using same apiKey and appToken.

Expected Behavior

Activation of Keda Datadog scaler

Actual Behavior

"error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"

Steps to Reproduce the Problem

1. Get your API and APP token from Datadog
2. Deploy Keda autoscaler on your cluster following docs

Logs from KEDA operator

1.6547744315708387e+09	DEBUG	events	Warning	{"object": {"kind":"ScaledObject","namespace":"b","name":"core-fr-celery-priority-worker","uid":"7cb8bfc9-9c62-419d-8b25-07c2ab670718","apiVersion":"keda.sh/v1alpha1","resourceVersion":"405447786"}, "reason": "KEDAScalerFailed", "message": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"}
1.6547744315709755e+09	DEBUG	events	Warning	{"object": {"kind":"ScaledObject","namespace":"b","name":"core-fr-celery-priority-worker","uid":"7cb8bfc9-9c62-419d-8b25-07c2ab670718","apiVersion":"keda.sh/v1alpha1","resourceVersion":"405447786"}, "reason": "ScaledObjectCheckFailed", "message": "Failed to ensure HPA is correctly created for ScaledObject"}
1.654774431622108e+09	ERROR	controller.scaledobject	Reconciler error	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "core-fr-celery-priority-worker", "namespace": "b", "error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"}

KEDA Version

2.7.1

Kubernetes Version

1.22

Platform

Google Cloud

Scaler Details

Datadog

Anything else?

No response

[JorTurFer]

hey
Could you share your scaleobject? (without secrets ofc)

[karolorlamowski]

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: core-fr-celery-priority-worker
  namespace: b
spec:
  scaleTargetRef:
    name: core-fr-celery-priority-worker
  triggers:
  - type: datadog
    metricType: "Value"
    metadata:
      query: "sum:queue.by_app.celery_regular_worker.count{*}.as_count()"
      queryValue: "1"
      age: "120"
      metricUnavailableValue: "0"
    authenticationRef:
      kind: ClusterTriggerAuthentication
      name: keda-trigger-auth-datadog-secret

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-trigger-auth-datadog-secret
spec:
  secretTargetRef:
  - key: apiKey
    name: datadog-secrets
    parameter: apiKey
  - key: appKey
    name: datadog-secrets
    parameter: appKey
  - key: datadogSite
    name: datadog-secrets
    parameter: datadogSite

apiVersion: v1
data:
  apiKey: <apiKey>
  appKey: <appKey>
  datadogSite: <datadogSite>
kind: Secret
metadata:
  name: datadog-secrets
  namespace: keda
type: Opaque

[tomkerkhove]

@arapulido Are you able to help?

[arapulido]

@karolorlamowski what's your datadogSite?

[karolorlamowski]

it's datadoghq.eu

3m34s       Warning   KEDAScalerFailed          scaledobject/nginx-scaled                               error establishing Datadog connection: error connecting to Datadog API endpoint: The variable site in the server URL has invalid value datadoghq.eu...

[arapulido]

@karolorlamowski thanks! I will try to reproduce on my end and will submit a patch

[arapulido]

@karolorlamowski I tried to reproduce the issue on my side, with a datadoghq.eu account and it works fine for me. Can you check that the secret was created correctly? (without leading spaces, leading dots, etc.) ?

[arapulido]

You can check it by getting the secret and decoding the site:

k get secret datadog-secrets -o jsonpath='{.data.datadogSite}' | base64 -d
datadoghq.eu%

[arapulido]

(you can see that there are not leading spaces, or carrier returns)

[arapulido]

@karolorlamowski any updates?

[Eduxz]

@arapulido
My issue is exactly with the same information, but my datadogSite is datadoghq.com.
For example, this is the url that I am using https://app.datadoghq.com
I am using this source: https://keda.sh/docs/2.14/scalers/datadog/

[Eduxz]

Information:

ScaledObject:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-monitor-scaledobject
  namespace: datadog
spec:
  scaleTargetRef:
    name: your-deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: datadog
    metricType: "AverageValue"
    metadata:
      # Comma-seperated querys count as a single API call:
      query: "sum:trace.express.request.hits{*}.as_rate()"
      queryValue: "100"
      age: "90"
      metricUnavailableValue: "0"
      # Optional: The delayed time window offset (in seconds) to wait for the metric to be available. The values of some queries might be not available at now and need a small delay to become available, try to adjust it if you encounter this issue. Default: 0
      timeWindowOffset: "30"
      # Optional: The offset to retrieve the X to last data point. The value of last data point of some queries might be inaccurate, try to adjust to 1 if you encounter this issue. Default: 0
      lastAvailablePointOffset: "1"
    authenticationRef:
      kind: ClusterTriggerAuthentication
      name: keda-trigger-auth-datadog-secret
---
apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  namespace: datadog
type: Opaque
stringData:
  appKey: ...
  apiKey: ...
  datadogSite: datadoghq.com
---
apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-trigger-auth-datadog-secret
  namespace: datadog
spec:
  secretTargetRef:
  - parameter: apiKey
    name: datadog-secret
    key: apiKey
  - parameter: appKey
    name: datadog-secret
    key: appKey
  # - parameter: datadogSite
  #   name: datadog-secret
  #   key: datadogSite

[Eduxz]

Helm Chart information
keda keda 1 2024-06-17 15:30:08.411388 -0400 -0400 deployed keda-2.14.2 2.14.0

[Eduxz]

My problem was because I didn't put the secret and the ClusterTriggerAuthentication in the same namespace as keda.
thank you @arapulido

[arapulido]

We debugged @Eduxz issue on slack, and @karolorlamowski never replied again. @JorTurFer I think we can close this one.