[Bug] Mutate fails if the pod is waiting to be scheduled on a node

[vijaygupta18]

Kyverno Version

1.7

Kubernetes Version

1.20

Kubernetes Platform

EKS

Description

Hi
I’m using the following Kyverno policy to add the node’s zone label to each Pod’s labels. However, I’m encountering an issue where it fails sometimes if the Pod has not been scheduled to a node yet. The failure occurs because the nodeName field is not populated, and the API call to retrieve the node’s zone label doesn’t return any data.

Issue

The policy fails with the following error:

failed to mutate existing resource, rule project-foo, response error: variable substitution failed: failed to resolve zoneLabel at path /mutate/patchStrategicMerge/metadata/labels/topology.kubernetes.io\\/zone: failed to store data for APICall: failed to apply JMESPath metadata.labels.[\"topology.kubernetes.io/zone\"][0] for context entry zoneLabel: Unknown key \"labels\" in path" logger=background name=ur-tv7v2 policy=add-node-labels-pod resource=v1/Pod/atlas/test-pod

Here’s the policy I’m using:

apiVersion: kyverno.io/v2beta1
kind: ClusterPolicy
metadata:
  name: add-node-labels-pod
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
    policies.kyverno.io/title: Add scheduled Node's zone label to a Pod
    policies.kyverno.io/description: >
      This policy retrieves the scheduled node's topology zone label
      and injects it into the pod's metadata.labels.
spec:
  rules:
    - name: project-foo
      skipBackgroundRequests: false
      match:
        any:
        - resources:
            kinds:
            - Pod
            operations:
            - CREATE
            - UPDATE
      context:
        - name: node
          variable:
            jmesPath: request.object.spec.nodeName
            default: ""
        - name: zoneLabel
          apiCall:
            urlPath: "/api/v1/nodes/{{node}}"
            jmesPath: metadata.labels.["topology.kubernetes.io/zone"][0]
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              topology.kubernetes.io/zone: "{{ zoneLabel }}"

Is there a way to ensure this mutation only happens when the nodeName field is available (i.e., after the Pod is scheduled)? It seems the current setup doesn’t handle cases where the node information isn’t yet accessible, causing the mutation to fail.

Steps to reproduce

1. create a pod in zone where nodes needs to be upscaled

Expected behavior

It should ideally mutate once the pod is scheduled on a node and node information is there

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.