Keda TriggerAuth Aws SecretManager assumes wrong pod identity

[sreeboppana]

Report

Keda version: keda:2.14.0
ghcr.io/kedacore/keda:2.14.0

Goal : Configure Keda KAfka Trigger for scaling our service. But to authenticate with Kafka the client needs api-key, api-secret. So used TriggerAuthentication with awsSecretManager resource as shown below.

Note:

1. TriggerAuth, ScaledObject are deployed within my-microservice-namespace namespace.
2. KEda operator has been deployed within **keda** namespace.

STEP-1
TriggerAuth.yaml

apiVersion: v1
items:
  kind: TriggerAuthentication
  metadata:
    **name: test-triggerauth**
    namespace: my-microservice-namespace
   spec:
    awsSecretManager:
      podIdentity:
        provider: aws
        roleArn: arn:aws:iam::<accountId>:test-role/<test-role-ID>
      region: us-east-1
      secrets:
      - name: service/test-svc/svc-api-key-path
        parameter: api-key
      - name: service/test-svc/svc-api-key-path
        parameter: api-secret

STEp-2
ScaledObject.yaml

Name:         my-microservice
Namespace:    my-microservice-namespace
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Spec:
  Triggers:
    Metadata:
      Value:      10
    Metric Type:  Utilization
    Type:         cpu
    Metadata:
      Value:      10
    Metric Type:  Utilization
    Type:         memory
    Authentication Ref:
      **Name:  test-triggerauth**
    Metadata:
      Bootstrap Servers:  serverHostNames:9092
      Consumer Group:     xyz
      Lag Threshold:      30
      Topic:              event.xyz
    Type:                 kafka

STEP-3
Deployed the resources and then keda operator logs are shown below:

2024-09-26T20:27:13Z	ERROR	**scale_handler	error trying to read secret from Aws Secret Manager**	
{
	"type": "ScaledObject",
	 "namespace": "my-microservice-namespace", 
	 "name": "my-microservice", 
	 "triggerAuthRef.Name": "test-triggerauth", 
	 "secret.Name": "service/test-svc/svc-api-key-path", 
	 "secret.Version": "", 
	 "secret.VersionStage": "", 

	 "error": "operation error Secrets Manager: GetSecretValue, 
	 	https response error StatusCode: 400, 
	 	RequestID: 4314b83a-0192-4bf1-9a7f-2e7f7950abe0, 
	 		api error AccessDeniedException: 
	 			User: **arn:aws:sts::<awsAccountId>:assumed-role/keda-operator-role/123456**
	 				is not authorized to perform: 
	 					secretsmanager:GetSecretValue on resource: service/test-svc/svc-api-key-path
	 						because no identity-based policy allows the secretsmanager:GetSecretValue action"
}

Expected Behavior

The expected behavior was the keda operator should have assumed the IAM role which TriggerAuth had been configured with arn:aws:iam::<accountId>:test-role/<test-role-ID> (bolded below) and not it's own IAM role (arn:aws:sts:::assumed-role/keda-operator-role/123456).

Note:

1. TriggerAuth, ScaledObject are deployed within my-microservice-namespace namespace.
2. KEda operator has been deployed within **keda** namespace.

TriggerAuth.yaml

apiVersion: v1
items:
  kind: TriggerAuthentication
  metadata:
    **name: test-triggerauth**
    namespace: my-namespace
   spec:
    awsSecretManager:
      podIdentity:
        provider: aws
        **roleArn: arn:aws:iam::<accountId>:test-role/<test-role-ID>**
      region: us-east-1
      secrets:
      - name: service/test-svc/svc-api-key-path
        parameter: api-key
      - name: service/test-svc/svc-api-key-path
        parameter: api-secret

Actual Behavior

Keda operator assumes wrong IAM role while trying to Get Secret from AWS SecretsManager service.

Note: Keda operator had been deployed within keda namespace.

2024-09-26T20:27:13Z	ERROR	**scale_handler	error trying to read secret from Aws Secret Manager**	
{
	"type": "ScaledObject",
	 "namespace": "my-microservice-namespace", 
	 "name": "my-microservice", 
	 "triggerAuthRef.Name": "test-triggerauth", 
	 "secret.Name": "service/test-svc/svc-api-key-path", 
	 "secret.Version": "", 
	 "secret.VersionStage": "", 

	 "error": "operation error Secrets Manager: GetSecretValue, 
	 	https response error StatusCode: 400, 
	 	RequestID: 4314b83a-0192-4bf1-9a7f-2e7f7950abe0, 
	 		api error AccessDeniedException: 
	 			User: **arn:aws:sts::<awsAccountId>:assumed-role/keda-operator-role/123456**
	 				is not authorized to perform: 
	 					secretsmanager:GetSecretValue on resource: service/test-svc/svc-api-key-path
	 						because no identity-based policy allows the secretsmanager:GetSecretValue action"
}

Steps to Reproduce the Problem

Keda version: keda:2.14.0
ghcr.io/kedacore/keda:2.14.0

Goal : Configure Keda KAfka Trigger for scaling our service. But to authenticate with Kafka the client needs api-key, api-secret. So used TriggerAuthentication with awsSecretManager resource as shown below.

Note:

1. TriggerAuth, ScaledObject are deployed within my-microservice-namespace namespace.
2. KEda operator has been deployed within **keda** namespace.

STEP-1
TriggerAuth.yaml

apiVersion: v1
items:
  kind: TriggerAuthentication
  metadata:
    **name: test-triggerauth**
    namespace: my-microservice-namespace
   spec:
    awsSecretManager:
      podIdentity:
        provider: aws
        roleArn: arn:aws:iam::<accountId>:test-role/<test-role-ID>
      region: us-east-1
      secrets:
      - name: service/test-svc/svc-api-key-path
        parameter: api-key
      - name: service/test-svc/svc-api-key-path
        parameter: api-secret

STEp-2
ScaledObject.yaml

Name:         my-microservice
Namespace:    my-microservice-namespace
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Spec:
  Triggers:
    Metadata:
      Value:      10
    Metric Type:  Utilization
    Type:         cpu
    Metadata:
      Value:      10
    Metric Type:  Utilization
    Type:         memory
    Authentication Ref:
      **Name:  test-triggerauth**
    Metadata:
      Bootstrap Servers:  serverHostNames:9092
      Consumer Group:     xyz
      Lag Threshold:      30
      Topic:              event.xyz
    Type:                 kafka

STEP-3
Deployed the resources and then keda operator logs are shown below:

2024-09-26T20:27:13Z	ERROR	**scale_handler	error trying to read secret from Aws Secret Manager**	
{
	"type": "ScaledObject",
	 "namespace": "my-microservice-namespace", 
	 "name": "my-microservice", 
	 "triggerAuthRef.Name": "test-triggerauth", 
	 "secret.Name": "service/test-svc/svc-api-key-path", 
	 "secret.Version": "", 
	 "secret.VersionStage": "", 

	 "error": "operation error Secrets Manager: GetSecretValue, 
	 	https response error StatusCode: 400, 
	 	RequestID: 4314b83a-0192-4bf1-9a7f-2e7f7950abe0, 
	 		api error AccessDeniedException: 
	 			User: **arn:aws:sts::<awsAccountId>:assumed-role/keda-operator-role/123456**
	 				is not authorized to perform: 
	 					secretsmanager:GetSecretValue on resource: service/test-svc/svc-api-key-path
	 						because no identity-based policy allows the secretsmanager:GetSecretValue action"
}

Logs from KEDA operator

keda operator log:

2024-09-26T20:27:13Z	ERROR	**scale_handler	error trying to read secret from Aws Secret Manager**	
{
	"type": "ScaledObject",
	 "namespace": "my-microservice-namespace", 
	 "name": "my-microservice", 
	 "triggerAuthRef.Name": "test-triggerauth", 
	 "secret.Name": "service/test-svc/svc-api-key-path", 
	 "secret.Version": "", 
	 "secret.VersionStage": "", 

	 "error": "operation error Secrets Manager: GetSecretValue, 
	 	https response error StatusCode: 400, 
	 	RequestID: 4314b83a-0192-4bf1-9a7f-2e7f7950abe0, 
	 		api error AccessDeniedException: 
	 			User: **arn:aws:sts::<awsAccountId>:assumed-role/keda-operator-role/123456**
	 				is not authorized to perform: 
	 					secretsmanager:GetSecretValue on resource: service/test-svc/svc-api-key-path
	 						because no identity-based policy allows the secretsmanager:GetSecretValue action"
}

KEDA Version

2.14.0

Kubernetes Version

1.29

Platform

Amazon Web Services

Scaler Details

kafka

Anything else?

No response

[JorTurFer]

Thanks for reporting the issue. I've found the root cause and I've created this PR to fix it for the next version -> #6197