Understanding the behavior of duration in scale up triggers #601

Puneeth-n · 2021-06-03T19:24:26Z

Puneeth-n
Jun 3, 2021

Hi,

What is the expected behavior of "duration" of scale up triggers? I have the below configuration and I expected that the runners would be brought down if they are more than 90 mins old. However, I am seeing them run even after 4 hours.

manifest

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: comtravo-github-actions-core-runner-deployment
  namespace: ${kubernetes_namespace.ci.metadata[0].name}
spec:
  template:
    spec:
      imagePullSecrets:
        - name: ${kubernetes_secret.actions_docker_hub_regcred.metadata[0].name}
      nodeSelector:
        ${var.eks_node_labels.core_runners.key}: ${var.eks_node_labels.core_runners.value}
      image: harbor.infra.comtravo.com/cache/comtravo/actions-runner:v2.277.1
      repository: ${local.actions.git_repository}
      labels:
        - core-runner
      serviceAccountName: ${local.actions.service_account_name}
      securityContext:
        fsGroup: 1447
      resources:
        limits:
          cpu: "6.0"
          memory: "12Gi"
        requests:
          cpu: "6.0"
          memory: "12Gi"
      volumes:
        - name: runner-var-lib-docker
          hostPath:
            path: /mnt/runner-var-lib-docker
            type: DirectoryOrCreate
      dockerVolumeMounts:
        - mountPath: /var/lib/docker
          name: runner-var-lib-docker
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: comtravo-github-actions-core-runner-deployment-autoscaler
  namespace: ${kubernetes_namespace.ci.metadata[0].name}
spec:
  scaleTargetRef:
    name: comtravo-github-actions-core-runner-deployment
  minReplicas: 1
  maxReplicas: 85
  scaleUpTriggers:
  - githubEvent:
      checkRun:
        types: ["created"]
        status: "queued"
    amount: 1
    duration: "90m"

Running runners

k -n ci get pods --sort-by=.status.startTime | grep core-runner
comtravo-github-actions-core-runner-deployment-nswk4-wpt6q        2/2     Running   0          4h37m
comtravo-github-actions-core-runner-deployment-nswk4-w6lmv        2/2     Running   0          4h37m
comtravo-github-actions-core-runner-deployment-nswk4-qvxgf        2/2     Running   0          4h36m
comtravo-github-actions-core-runner-deployment-nswk4-njfwb        2/2     Running   0          4h36m
comtravo-github-actions-core-runner-deployment-nswk4-m7tdd        2/2     Running   0          4h36m
comtravo-github-actions-core-runner-deployment-nswk4-4g5lb        2/2     Running   0          4h36m
comtravo-github-actions-core-runner-deployment-nswk4-v4sgd        2/2     Running   0          4h36m
comtravo-github-actions-core-runner-deployment-nswk4-mzx8k        2/2     Running   0          4h35m
comtravo-github-actions-core-runner-deployment-nswk4-g9wnc        2/2     Running   0          4h35m
comtravo-github-actions-core-runner-deployment-nswk4-7xbmn        2/2     Running   0          4h35m
comtravo-github-actions-core-runner-deployment-nswk4-m86r9        2/2     Running   0          4h35m
comtravo-github-actions-core-runner-deployment-nswk4-8mvvs        2/2     Running   0          4h34m
comtravo-github-actions-core-runner-deployment-nswk4-76w7f        2/2     Running   0          4h34m
comtravo-github-actions-core-runner-deployment-nswk4-6h8wq        2/2     Running   0          4h32m
comtravo-github-actions-core-runner-deployment-nswk4-p74hp        2/2     Running   0          4h29m
comtravo-github-actions-core-runner-deployment-nswk4-wnxfs        2/2     Running   0          4h28m
comtravo-github-actions-core-runner-deployment-nswk4-f7qkt        2/2     Running   0          4h28m
comtravo-github-actions-core-runner-deployment-nswk4-q2rwh        2/2     Running   0          111m
comtravo-github-actions-core-runner-deployment-nswk4-bkwsk        2/2     Running   0          111m
comtravo-github-actions-core-runner-deployment-nswk4-pnvlv        2/2     Running   0          111m
comtravo-github-actions-core-runner-deployment-nswk4-m2447        2/2     Running   0          111m
comtravo-github-actions-core-runner-deployment-nswk4-dhs78        2/2     Running   0          110m
comtravo-github-actions-core-runner-deployment-nswk4-f8zdq        2/2     Running   0          97m
comtravo-github-actions-core-runner-deployment-nswk4-7zbdm        2/2     Running   0          97m
comtravo-github-actions-core-runner-deployment-nswk4-jtnnz        2/2     Running   0          73m
comtravo-github-actions-core-runner-deployment-nswk4-fvpff        2/2     Running   0          73m
comtravo-github-actions-core-runner-deployment-nswk4-9cx5m        2/2     Running   0          73m
comtravo-github-actions-core-runner-deployment-nswk4-fzj57        2/2     Running   0          68m
comtravo-github-actions-core-runner-deployment-nswk4-jtqkp        2/2     Running   0          67m
comtravo-github-actions-core-runner-deployment-nswk4-tg97l        2/2     Running   0          67m
comtravo-github-actions-core-runner-deployment-nswk4-x562f        2/2     Running   0          67m
comtravo-github-actions-core-runner-deployment-nswk4-wh4v7        2/2     Running   0          67m
comtravo-github-actions-core-runner-deployment-nswk4-g69fs        2/2     Running   0          67m
comtravo-github-actions-core-runner-deployment-nswk4-w7qqn        2/2     Running   0          66m
comtravo-github-actions-core-runner-deployment-nswk4-7tlqs        2/2     Running   0          66m
comtravo-github-actions-core-runner-deployment-nswk4-mpmgc        2/2     Running   0          66m
comtravo-github-actions-core-runner-deployment-nswk4-8n722        2/2     Running   0          66m
comtravo-github-actions-core-runner-deployment-nswk4-4zpg9        2/2     Running   0          65m
comtravo-github-actions-core-runner-deployment-nswk4-mtmz8        2/2     Running   0          63m
comtravo-github-actions-core-runner-deployment-nswk4-mpnhb        2/2     Running   0          63m
comtravo-github-actions-core-runner-deployment-nswk4-cxqc7        2/2     Running   0          62m
comtravo-github-actions-core-runner-deployment-nswk4-jbwzc        2/2     Running   0          61m
comtravo-github-actions-core-runner-deployment-nswk4-mzftg        2/2     Running   0          59m
comtravo-github-actions-core-runner-deployment-nswk4-x8b6q        2/2     Running   0          59m
comtravo-github-actions-core-runner-deployment-nswk4-xrxj9        2/2     Running   0          59m
comtravo-github-actions-core-runner-deployment-nswk4-s5gd9        2/2     Running   0          59m
comtravo-github-actions-core-runner-deployment-nswk4-vl6m9        2/2     Running   0          59m
comtravo-github-actions-core-runner-deployment-nswk4-9cdxv        2/2     Running   0          59m
comtravo-github-actions-core-runner-deployment-nswk4-98r5z        2/2     Running   0          58m
comtravo-github-actions-core-runner-deployment-nswk4-6zjmv        2/2     Running   0          58m
comtravo-github-actions-core-runner-deployment-nswk4-56j7z        2/2     Running   0          58m
comtravo-github-actions-core-runner-deployment-nswk4-hqhtx        2/2     Running   0          58m
comtravo-github-actions-core-runner-deployment-nswk4-cxtsp        2/2     Running   0          58m
comtravo-github-actions-core-runner-deployment-nswk4-n55qn        2/2     Running   0          57m
comtravo-github-actions-core-runner-deployment-nswk4-thgh7        2/2     Running   0          57m
comtravo-github-actions-core-runner-deployment-nswk4-jhnz4        2/2     Running   0          57m
comtravo-github-actions-core-runner-deployment-nswk4-78n7l        2/2     Running   0          57m
comtravo-github-actions-core-runner-deployment-nswk4-hsgz6        2/2     Running   0          56m
comtravo-github-actions-core-runner-deployment-nswk4-lq57g        2/2     Running   0          56m
comtravo-github-actions-core-runner-deployment-nswk4-7nwhz        2/2     Running   0          52m
comtravo-github-actions-core-runner-deployment-nswk4-np957        2/2     Running   0          52m
comtravo-github-actions-core-runner-deployment-nswk4-6kcwm        2/2     Running   0          51m
comtravo-github-actions-core-runner-deployment-nswk4-ng8n7        2/2     Running   0          51m
comtravo-github-actions-core-runner-deployment-nswk4-np677        2/2     Running   0          50m
comtravo-github-actions-core-runner-deployment-nswk4-m5x8t        2/2     Running   0          47m
comtravo-github-actions-core-runner-deployment-nswk4-8dr8m        2/2     Running   0          47m
comtravo-github-actions-core-runner-deployment-nswk4-7clch        2/2     Running   0          27m
comtravo-github-actions-core-runner-deployment-nswk4-2spbc        2/2     Running   0          27m
comtravo-github-actions-core-runner-deployment-nswk4-gmxh4        2/2     Running   0          27m
comtravo-github-actions-core-runner-deployment-nswk4-l5brw        2/2     Running   0          27m
comtravo-github-actions-core-runner-deployment-nswk4-4tcn8        2/2     Running   0          27m
comtravo-github-actions-core-runner-deployment-nswk4-4rvkd        2/2     Running   0          17m
comtravo-github-actions-core-runner-deployment-nswk4-jf2jl        2/2     Running   0          17m
comtravo-github-actions-core-runner-deployment-nswk4-98b8g        2/2     Running   0          17m
comtravo-github-actions-core-runner-deployment-nswk4-d7d8l        2/2     Running   0          17m
comtravo-github-actions-core-runner-deployment-nswk4-sngnl        2/2     Running   0          16m

Answered by mumoshu

Jun 7, 2021

@Puneeth-n It should be scale-down after 90m since the addition of the capacity.

What's in your HRA.Spec.CapacityReservations? Try running kubectl get on it to see. Each scale-up trigger results in adding an entry under that, and only "unexpired" capacity reservations count into the final desired replicas number.

View full answer

toast-gear · 2021-06-04T10:35:51Z

toast-gear
Jun 4, 2021
Maintainer

https://github.com/actions-runner-controller/actions-runner-controller#runnerdeployments

Important!!! We no longer include the attribute replicas in our RunnerDeployment if we are configuring autoscaling!

As pointed out in the docs, you should not be using both the replicas attribute with the minReplicas and maxReplicas attributes. Remove the replicas key first and test again.

1 reply

Puneeth-n Jun 4, 2021
Author

@toast-gear I updated my code snippet above. I do not have replicas. Just min and max

mumoshu · 2021-06-07T08:39:27Z

mumoshu
Jun 7, 2021
Maintainer

@Puneeth-n It should be scale-down after 90m since the addition of the capacity.

What's in your HRA.Spec.CapacityReservations? Try running kubectl get on it to see. Each scale-up trigger results in adding an entry under that, and only "unexpired" capacity reservations count into the final desired replicas number.

3 replies

Puneeth-n Jun 7, 2021
Author

k -n ci describe horizontalrunnerautoscalers.actions.summerwind.dev comtravo-github-actions-core-runner-deployment-autoscaler
Name:         comtravo-github-actions-core-runner-deployment-autoscaler
Namespace:    ci
Labels:       <none>
Annotations:  <none>
API Version:  actions.summerwind.dev/v1alpha1
Kind:         HorizontalRunnerAutoscaler
Metadata:
  Creation Timestamp:  2021-05-14T21:26:02Z
  Generation:          53103
  Managed Fields:
    API Version:  actions.summerwind.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
      f:spec:
        .:
        f:minReplicas:
        f:scaleTargetRef:
          .:
          f:name:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2021-05-19T18:47:32Z
    API Version:  actions.summerwind.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:maxReplicas:
        f:scaleUpTriggers:
    Manager:      kubectl
    Operation:    Update
    Time:         2021-06-02T19:15:26Z
    API Version:  actions.summerwind.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:cacheEntries:
        f:desiredReplicas:
        f:lastSuccessfulScaleOutTime:
    Manager:      manager
    Operation:    Update
    Time:         2021-06-07T14:29:54Z
    API Version:  actions.summerwind.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:capacityReservations:
    Manager:         github-webhook-server
    Operation:       Update
    Time:            2021-06-07T14:31:02Z
  Resource Version:  105760197
  Self Link:         /apis/actions.summerwind.dev/v1alpha1/namespaces/ci/horizontalrunnerautoscalers/comtravo-github-actions-core-runner-deployment-autoscaler
  UID:               d5b39912-d0f8-4b44-8a13-d7d16664fb70
Spec:
  Capacity Reservations:
    Expiration Time:  2021-06-07T14:31:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:31:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:42:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:43:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:44:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:45:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:13Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:14Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:14Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:20Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:24Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:46:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:23Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:47:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:20Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:21Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:48:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:49:00Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:49:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:49:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:49:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:49:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:49:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:13Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:21Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:24Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:50:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:51:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:52:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:52:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:52:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:14Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:53:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:13Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:23Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:54:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:55:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:56:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:56:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:56:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T14:59:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:01:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:02:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:00Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:13Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:21Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:24Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:03:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:04:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:05:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:06:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:06:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:06:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:06:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:06:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:06:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:06:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:07:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:07:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:07:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:13Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:14Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:09:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:17:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:14Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:18:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:00Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:19:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:24Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:20:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:23Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:21:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:22:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:22:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:26:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:27:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:27:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:27:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:30:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:30:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:31:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:33:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:13Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:14Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:34:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:35:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:20Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:36:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:37:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:37:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:37:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:37:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:37:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:39:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:39:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:40:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:40:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:40:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:40:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:40:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:40:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:13Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:26Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:41:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:19Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:20Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:21Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:23Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:36Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:43Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:44Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:42:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:00Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:43:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:00Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:25Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:28Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:44:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:11Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:17Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:27Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:29Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:45:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:02Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:14Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:15Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:16Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:18Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:20Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:21Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:23Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:32Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:33Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:46:34Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:08Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:09Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:22Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:37Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:38Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:47:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:48:07Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:48:35Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:48:39Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:48:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:00Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:01Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:03Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:04Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:05Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:06Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:21Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:45Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:46Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:47Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:48Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:49Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:51Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:52Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:53Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:54Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:55Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:56Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:57Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:58Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:49:59Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:00Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:30Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:31Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:40Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:41Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:50:42Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:55:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:57:10Z
    Replicas:         1
    Expiration Time:  2021-06-07T15:57:12Z
    Replicas:         1
    Expiration Time:  2021-06-07T16:00:50Z
    Replicas:         1
    Expiration Time:  2021-06-07T16:01:02Z
    Replicas:         1
  Max Replicas:       85
  Min Replicas:       1
  Scale Target Ref:
    Name:  comtravo-github-actions-core-runner-deployment
  Scale Up Triggers:
    Amount:    1
    Duration:  90m
    Github Event:
      Check Run:
        Status:  queued
        Types:
          created
Status:
  Cache Entries:
    Expiration Time:               2021-06-07T14:39:44Z
    Key:                           desiredReplicas
    Value:                         1
  Desired Replicas:                85
  Last Successful Scale Out Time:  2021-06-07T06:56:18Z
Events:                            <none>

mumoshu Jun 16, 2021
Maintainer

@Puneeth-n Thanks! Okay, so the capacity reservations/timed scaling seems to be working as expected. So you should have no "real" issues.

The point is that actions-runner-controller doesn't prioritize older runner pods to be deleted earlier than newer ons, on scale down. And each capacity reservation doesn't map 1:1 to runner pod. It's just that all the non-expiring capacity reservations are summed up to compute the desired replicas number.

The fact that you still have 85 runners is due to that you had enough GitHub Actions job runs to keep that number. The fact you're seeing pods older than 90m is due to that the controller doesn't prioritize older pods for deletion on scale down.

Puneeth-n Jun 16, 2021
Author

@mumoshu good to know :) Thanks

Puneeth-n · 2021-06-16T06:35:13Z

Puneeth-n
Jun 16, 2021
Author

Thanks for the clarification @mumoshu

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Understanding the behavior of duration in scale up triggers #601

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Understanding the behavior of duration in scale up triggers #601

Puneeth-n Jun 3, 2021

Replies: 3 comments · 4 replies

toast-gear Jun 4, 2021 Maintainer

Puneeth-n Jun 4, 2021 Author

mumoshu Jun 7, 2021 Maintainer

Puneeth-n Jun 7, 2021 Author

mumoshu Jun 16, 2021 Maintainer

Puneeth-n Jun 16, 2021 Author

Puneeth-n Jun 16, 2021 Author

Puneeth-n
Jun 3, 2021

Replies: 3 comments 4 replies

toast-gear
Jun 4, 2021
Maintainer

Puneeth-n Jun 4, 2021
Author

mumoshu
Jun 7, 2021
Maintainer

Puneeth-n Jun 7, 2021
Author

mumoshu Jun 16, 2021
Maintainer

Puneeth-n Jun 16, 2021
Author

Puneeth-n
Jun 16, 2021
Author