Flakey behavior of on.create handler not reacting to CRO creation event.

[androiddrew]

Long story short

If I submit 10 of my RuntimeService objects to my cluster's Kube API approximately 1 of them will fail to create pretty consistently. The only thing I see in my operator's log about the CRO is kopf debug statements adding it's finalizer and annotations. Theses are submitted at a rate of 1 request per second.

All other CRO's submitted triggered the on.create handler and created a deployment, service, and ingress record as was defined by the handler. I can clearly see with kubectl describe rts rts-356ed03f-69bd-4d9a-b73b-f98d11f4d51f That the CRD was received by the KubeAPI as expected and indeed the kopf annotations are there.

This has occurred on two separate Kubernetes clusters.

Kopf version

1.36.2

Kubernetes version

1.25

Python version

3.11

Code

# RuntimeServiceBuilder translates the CRD to K8s objects. 

@kopf.on.create("runtime-services")
def rts_on_create(body, spec, name, namespace, logger, **kwargs):
    """Creates Kubernetes objects in response to a RuntimeService custom resource creation event."""
    logger.debug(f"The created handler is called with body: {body}")
    rts_builder = RuntimeServiceBuilder(body=body)
    logger.info("Received create request for %s", rts_builder.name)
    deployment = rts_builder.build_deployment()
    service = rts_builder.build_service()
    ingress = rts_builder.build_ingress()

    kopf.adopt(deployment)
    kopf.adopt(service)
    kopf.adopt(ingress)

    # Service
    k8s_core_v1 = kubernetes.client.CoreV1Api()
    logger.info("Creating Service: %s", name)
    k8s_core_v1.create_namespaced_service(namespace=namespace, body=service)
    logger.info("Service %s submitted", name)

    # Ingress
    k8s_networking_v1 = kubernetes.client.NetworkingV1Api()
    logger.info("Creating Ingress: %s", name)
    k8s_networking_v1.create_namespaced_ingress(namespace=namespace, body=ingress)
    logger.info("Ingress %s submitted", name)

    # Deployment
    k8s_apps_v1 = kubernetes.client.AppsV1Api()
    logger.info("Creating Deployment: %s", name)
    k8s_apps_v1.create_namespaced_deployment(namespace=namespace, body=deployment)
    logger.info("Deployment %s submitted", name)

    return {"rts-create": "submitted"}

Logs

[2023-12-14 20:53:28,051] kopf.objects         [DEBUG   ] [compute-runtime-api/rts-356ed03f-69bd-4d9a-b73b-f98d11f4d51f] Adding the finalizer, thus preventing the actual deletion.
[2023-12-14 20:53:28,051] kopf.objects         [DEBUG   ] [compute-runtime-api/rts-356ed03f-69bd-4d9a-b73b-f98d11f4d51f] Patching with: {'metadata': {'finalizers': ['kopf.zalando.org/KopfFinalizerMarker']}}
[2023-12-14 20:53:28,275] kopf.objects         [DEBUG   ] [compute-runtime-api/rts-356ed03f-69bd-4d9a-b73b-f98d11f4d51f] Something has changed, but we are not interested (the essence is the same).
[2023-12-14 20:53:28,276] kopf.objects         [DEBUG   ] [compute-runtime-api/rts-356ed03f-69bd-4d9a-b73b-f98d11f4d51f] Handling cycle is finished, waiting for new changes.

Additional information

No response

[eslam-gomaa-careem]

We're facing the same issue