Keda - Scaling based on Azure Service bus message along with pod identity

[sai25590]

I followed the instructions here to setup a autoscaling of pods based on number of messages in the service bus queue. I also set the pod binding in KEDA deployment so that authentication occurs via managed identity. But I get the following error in the keda operator pod:

kubectl logs keda-operator-5b7b47c48f-khq8w --namespace keda -f
{"level":"error","ts":1599627991.2951298,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"scaledobject-controller","request":"default/autofab-autoscale-scaledobject","error":"error getting scaler for trigger #0: error parsing azure service bus metadata: No subscription name provided with topic name","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/zroubali/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/Users/zroubali/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/Users/zroubali/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/Users/zroubali/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/Users/zroubali/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/Users/zroubali/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/Users/zroubali/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}
My important configuration:
Install KEDA operator and bind the aadpodidbinding :

# Install keda autoscaler using helm
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
kubectl create namespace keda
helm upgrade --install keda kedacore/keda --namespace keda --set aadPodIdentity=azureresourceaccess

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: autofab-autoscale-deployment
  labels:
    app: autofab-autoscale
    aadpodidbinding: azureresourceaccess
spec:
  template:
    metadata:
      name: autofab-autoscale
      labels:
        app: autofab-autoscale
        aadpodidbinding: azureresourceaccess
    spec:
      containers:
        - name: autofab-autoscale
          image: ${acrlogin}/${autofabautoscaleimage}:${version}
          imagePullPolicy: Always
          resources:
            requests:
              cpu: 1000m
          volumeMounts:
            - mountPath: /mnt/azure
              name: volume
          env:
            - name: SERVICEBUS_NAMESPACE
              value: ${sbnamespace}
            - name: AUTOSCALE_QUEUE_NAME
              value: "autofab-autoscaleq"
      restartPolicy: Always
      volumes:
        - name: volume
          persistentVolumeClaim:
            claimName: azurefile
  selector:
    matchLabels:
      app: autofab-autoscale

ScaledObject:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
namespace: default # must be same namespace as the Triggerauth
metadata:
  name: autofab-autoscale-scaledobject
spec:
  scaleTargetRef:
    deploymentName: autofab-autoscale-deployment # must be in the same namespace as the ScaledObject
  pollingInterval: 30  # Optional. Default: 30 seconds
  #cooldownPeriod:  300 # Optional. Default: 300 seconds. Matters only on scaling to 0
  minReplicaCount: 3   # Optional. Default: 0
  maxReplicaCount: 100 # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      queueName: autofab-autoscaleq
      subscriptionName: ${subscriptionname}
      queueLength: "1" # Optional. Subscription length target for HPA. Default: 5 messages
    authenticationRef:
      name: autoscale-servicebus-triggerauth

TriggerAuthentication:

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: autoscale-servicebus-triggerauth
  namespace: default # must be same namespace as the ScaledObject
spec:
  podIdentity:
      provider: azure # Optional. Default: none

[tomkerkhove]

The problem lies in your trigger definition:

- type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      queueName: autofab-autoscaleq
      subscriptionName: ${subscriptionname}
      queueLength: "1" # Optional. Subscription length target for HPA. Default: 5 messages

You are using a queue, but yet still provide the subscriptionName which is used for topics. Given that one is specified, we block because the name of the topic is not there.

[tomkerkhove]

To your trigger metadata:

- type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      queueName: autofab-autoscaleq
      namespace: your-service-bus-namespace
      queueLength: "1" # Optional. Subscription length target for HPA. Default: 5 messages

Feel free to let me know if it works.

[sai25590]

Thanks edited it. Will do.

[sai25590]

Seemed to fix the error 👍 I see the pods autoscaling. But not the way I expected. I have a dummy receiver that receives messages from service bus (receive and delete) waits a minute and then goes back to next message. I have 80 messages in the queue. I have 1 to 1 scaling. Expected ~80 odd pods, but only see 52 max (sometimes 16)

root@autofab-test-hopbox-vm:/home/sasubra# kubectl get pods -l app=autofab-autoscale | wc -l
51
root@autofab-test-hopbox-vm:/home/sasubra# kubectl get pods -l app=autofab-autoscale | wc -l
16
root@autofab-test-hopbox-vm:/home/sasubra# kubectl get pods -l app=autofab-autoscale | wc -l
14
root@autofab-test-hopbox-vm:/home/sasubra# kubectl get pods -l app=autofab-autoscale | wc -l
14

Second noob question is that should I define a HPA for this deployment on top of KEDA autoscaling?

Third noob question is that I have the KEDA operator's MI permissions only to read the service bus. But should I not give it permissions to scale another pod (may be something like access to AKS)?

Fourth question is that the deployment's aad binding and KEDA operator's AAD binding can be different right?

[tomkerkhove]

So you want to have 1 instance process 1 message, is that correct? In that case, you'd rather want to use a job instead of deployment.

Second noob question is that should I define a HPA for this deployment on top of KEDA autoscaling?

No you don't need to create an HPA, that's what KEDA does

Third noob question is that I have the KEDA operator's MI permissions only to read the service bus. But should I not give it permissions to scale another pod (may be something like access to AKS)?

Not sure what you mean here? You can clearly see that KEDA is already autoscalign your workload, no?

[sai25590]

>Not sure what you mean here? You can clearly see that KEDA is already autoscalign your workload, no?
Agreed. My pod identity for my deployment has stopped working now hence asked if I had to change something. I found the problem was that I named both the azure pod identies the same (for keda and other pods). Once I changed it things started working fine. Thanks for the help @tomkerkhove !

>In that case, you'd rather want to use a job instead of deployment.
Point taken. Will look into it. Hoping it's not too much of a change as this has been a production service for long (without autoscaling).