Scaled Jobs always leave behind extraneous jobs after work is finished. (AWS SQS, jobs deleted after completed)

[ZTGallagher]

We're not sure why. It's not reliably reproducible. Keda simply ends up creating more jobs than we need, and they hang around for hours until we delete them. They do not create pods anymore after the 6 retries, but the jobs linger.

The use case is we want a single container to spin up, handle a single message, and then die. Each node supports one container right now (gpu bound). So we've arbitrarily chosen 6 max nodes, thus 6 max containers.

Do our settings now support that? We thought they would.

So code looks like:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: scaled-job-scaler
  namespace: default
  labels:
    app: {{ template "name" $ }}
spec:
  pollingInterval: 5
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 5
  maxReplicaCount: 6
  rolloutStrategy: gradual
  scalingStrategy:
    strategy: "default"
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: {{ .Values.keda.sqs_queue }}
      queueLength: 1
      awsRegion: {{ .Values.env.AWS_REGION }}
      identityOwner: Operator
  jobTargetRef:
    parallelism: 1
    completions: 1
    backoffLimit: 6
    template:
      metadata:
        labels:
          app: {{ template "name" $ }}
          release: {{ $.Release.Name }}
      spec:
        (truncated for brevity)