Getting "error parsing cloudwatch metadata: awsAccessKeyID not found" even though I'm using a TriggerAuthentication

[pmmalinov01]

Hello I am deploying KEDA 2.15.1 and seeing the following error
2024-10-09T13:29:06Z ERROR Failed to get TriggerAuthentication {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"udbm-service","namespace":"qa"}, "namespace": "qa", "name": "udbm-service", "reconcileID": "dc837a36-4325-48f9-96f3-1cf0557a18ad", "error": "TriggerAuthentication.keda.sh "keda-aws-credentials" not found"}
github.com/kedacore/keda/v2/pkg/status.updateTriggerAuthenticationStatus
/workspace/pkg/status/status.go:103
github.com/kedacore/keda/v2/pkg/status.UpdateTriggerAuthenticationStatusFromTriggers
/workspace/pkg/status/status.go:139
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateTriggerAuthenticationStatus
/workspace/controllers/keda/scaledobject_controller.go:605
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
/workspace/controllers/keda/scaledobject_controller.go:212
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-10-09T13:29:06Z ERROR Reconciler error {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"udbm-service","namespace":"qa"}, "namespace": "qa", "name": "udbm-service", "reconcileID": "dc837a36-4325-48f9-96f3-1cf0557a18ad", "error": "error parsing cloudwatch metadata: awsAccessKeyID not found"}

The serviceaccount is
apiVersion: v1 automountServiceAccountToken: true kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::RED:role/redacted meta.helm.sh/release-name: keda meta.helm.sh/release-namespace: keda creationTimestamp: "2024-10-09T07:47:13Z" labels: app.kubernetes.io/component: operator app.kubernetes.io/instance: keda app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: keda-operator app.kubernetes.io/part-of: keda-operator app.kubernetes.io/version: 2.15.1 helm.sh/chart: keda-2.15.1 name: keda-operator namespace: keda resourceVersion: "32156157" uid: b1084887-e71b-4c54-9fd8-ada2486b1b5e

and the role looks fine the trust policy is

    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::RED:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/49A22671ED67D6F64C61F800057F2097"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/49A22671ED67D6F64C61F800057F2097:sub": "system:serviceaccount:qa:keda_operator",
                    "oidc.eks.us-east-1.amazonaws.com/id/49A22671ED67D6F64C61F800057F2097:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}```
the scaledObject and trigger auth I am using are:
`apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: udbm-service
  namespace: qa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: udbm-service
  pollingInterval: 30
  cooldownPeriod: 180
  minReplicaCount: 3
  maxReplicaCount: 9
  triggers:
  - type: aws-cloudwatch
    metadata:
      awsRegion: "us-east-1"
      identityOwner: "operator"
      awsEndpoint: "[vpce-070c379132f8b2905-f9hn3z82.sts.us-east-1.vpce.amazonaws.com](http://vpce-070c379132f8b2905-f9hn3z82.sts.us-east-1.vpce.amazonaws.com/)"
      targetMetricValue: "5"
      namespace: "RED-RED"
      metricName: "Queue Size"
      dimensionName: "Environment"
      dimensionValue: "QA"
      statistic: "Maximum"
      targetValue: "50"
      minMetricValue: "0"
      metricCollectionTime: "300"
      metricStat: "Average"
    authenticationRef:
      name: keda-aws-credentials
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
          policies:
          - type: Pods
            value: 3
            periodSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 180
          policies:
          - type: Percent
            value: 100
            periodSeconds: 180
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-aws-credentials
spec:
  podIdentity:
    provider: aws-eks`
    
  any guide suggestion will be helpful where to look or poke.
the cluster is using only vpc endpoints and no internet access.

[pmmalinov01]

The first issue I am having is with the conditon of the role. the namespace was incorrect and the keda_operator must be keda-operator.
But even fixing this has not helped me.
I have started a pod with that service account and and managed to run aws sts get-caller-identity
[cloudshell-user@ip-10-74-30-94 keda]$ kubectl run -ti --image=654654361737.dkr.ecr.us-east-1.amazonaws.com/bitnami/aws-cli:v2 --overrides='{ "spec": { "serviceAccount": "keda-operator" } }' -n keda foo -- sts get-caller-identity If you don't see a command prompt, try pressing enter. { "UserId": "AROAZQ3DQ2CEYG6HXVGT7:botocore-session-1728554533", "Account": "RED", "Arn": "arn:aws:sts::RED:assumed-role/RED-qa-psn_qa_keda_hpa_cloudwatch_role/botocore-session-1728554533" }

which shows me that there is no miss configured AWS ROLE<>K8S serviceaccount.
I also managed to run KEDA with k8s secret (token and access key) and it run fine. it created the ScaledObject and HPA.