If pendingPodConditions isn't set, KEDA never detects any pending jobs

[Makeshift]

Report

If pendingPodConditions is not set on a ScaledJob with strategy accurate, KEDA appears to incorrectly set the number of pending jobs as always being 0. This means that if a job takes longer to start up than the poll time of the trigger, you end up with duplicate jobs.

Expected Behavior

When using scalingStrategy.strategy: accurate, I'd expect KEDA to correctly count the number of pods that had been scheduled but are not yet running, and calculate the quantity to scale up by as QueueLength-RunningJobs-PendingJobs.

Actual Behavior

If pendingPodConditions is not set in your ScaledJob, KEDA will always calculate the number of pending jobs as being 0, resulting in duplicate jobs if your job has a long startup time.

Steps to Reproduce the Problem

1. Define a ScaledJob with:

jobTargetRef:
  pollingInterval: 5
  scalingStrategy:
    strategy: accurate

• a pod spec that will spend at least 10 seconds in the pending state

2. Add 3 items to the trigger such that KEDA will scale up to 3 replicas. These 3 replicas should stay in the 'pending' state, however the KEDA operator will log "Number of pending Jobs": 0

3. After 5 seconds (the next poll), KEDA will attempt to launch 3 additional replicas (bringing the total to 6) because it does not see the pending jobs.

4. Reset and modify your ScaledJob to add all possible pendingPodConditions:

jobTargetRef:
  pollingInterval: 5
  scalingStrategy:
    strategy: accurate
    pendingPodConditions:
      - Ready
      - PodReadyToStartContainers
      - ContainersReady
      - Initialized
      - PodScheduled

5. Add 3 items to the trigger such that KEDA will scale up to 3 replicas. These 3 replicas should stay in the 'pending' state, and KEDA should log

"Number of pending Jobs": 4
No need to create jobs - all requested jobs already exist

Logs from KEDA operator

15s poll time with pendingPodConditions unset, comments mine:

2024-09-12T14:48:13Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of running Jobs": 0}
2024-09-12T14:48:13Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 0}
# 3 items submitted to queue - 3 jobs created and pending
2024-09-12T14:48:13Z	INFO	scaleexecutor	Creating jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Effective number of max jobs": 3}
2024-09-12T14:48:13Z	INFO	scaleexecutor	Creating jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
2024-09-12T14:48:13Z	INFO	scaleexecutor	Created jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
# KEDA claims 0 pending jobs
2024-09-12T14:48:28Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of running Jobs": 0}
2024-09-12T14:48:28Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 0}
# No new items added, previous 3 jobs still pending, KEDA creates 3 more jobs
2024-09-12T14:48:28Z	INFO	scaleexecutor	Creating jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Effective number of max jobs": 6}
2024-09-12T14:48:28Z	INFO	scaleexecutor	Creating jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
2024-09-12T14:48:28Z	INFO	scaleexecutor	Created jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of jobs": 3}
# The first 3 jobs are finally now running, KEDA still can't see the most recent 3 jobs that are pending
2024-09-12T14:48:43Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of running Jobs": 3}
2024-09-12T14:48:43Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 0}

15s poll time with pendingPodConditions set to all conditions, only part of the log but shows it's working correctly:

2024-09-12T14:42:13Z	INFO	scaleexecutor	Scaling Jobs	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "Number of pending Jobs": 4}
2024-09-12T14:42:13Z	INFO	scaleexecutor	No need to create jobs - all requested jobs already exist	{"scaledJob.Name": "qj-editionsendconsumer-staging", "scaledJob.Namespace": "staging", "jobs": 0}

KEDA Version

2.15.1

Kubernetes Version

1.28

Platform

Amazon Web Services

Scaler Details

SQS FIFO queue

Anything else?

I dug into this a tiny bit and I think the culprit is here. It looks like this function might be returning that a pod is running or complete when it's actually pending.

My conclusion that this is a bug is based on how the default behaviour is described here:

Default behavior - Job that have not finished yet and the underlying pod is either not running or has not been completed yet

[quiqueg]

I noticed the same behavior and can confirm that adding pendingPodConditions fixed it for us.

[andretibolaintelipost]

Adding pendingPodConditions also fixed for me, but it took me a while to get this fixed. I resorted to a large polling time at first, but that was affecting end users. Only after coming across this issue that i finally got to scale without duplicates and with a short polling period