controllers.Gateway: Reconciler error "service ref kong/kong-gateway-proxy did not match controller manager ref"

[mikyll]

I'm sorry to be always bothering you all, but I'm stuck and I cannot debug this error:

2023-12-12T11:09:14Z    error   controllers.Gateway     Reconciler error        {"reconcileID": "0b9efaca-3efe-4740-90d0-cc45c018719b", "error": "service ref kong/kong-gateway-proxy did not match controller manager ref kong/kong-1702379291-kong-proxy or /"}

• What does it mean?
• How should I procede to solve it?

Context

System: Windows with WSL2 and Docker Desktop
Kubernetes Cluster: minikube

I'm trying to deploy KIC DB-backed, to test Oauth2 plugin on a service. I'm using kong/kong helm chart to deploy it:

helm install kong/kong --generate-name --create-namespace -n kong -f values.yaml

And the values.yaml file is the following (the username and password are set like this for testing purposes), which I edited from minimal-k4k8s-with-kong-enterprise.yaml:

image:
  repository: kong/kong-gateway
  tag: "3.5"

admin:
  enabled: true
  http:
    enabled: true
    servicePort: 8001
    containerPort: 8001

enterprise:
  enabled: false
  vitals:
    enabled: false
  portal:
    enabled: false
  rbac:
    enabled: false
  smtp:
    enabled: false

portal:
  enabled: false

portalapi:
  enabled: false

env:
  prefix: /kong_prefix/
  database: postgres
  pg_user: kong-db-user
  pg_password: kong-db-password-test

postgresql:
  enabled: true
  auth:
    database: kong
    username: kong-db-user
    password: kong-db-password-test

ingressController:
  enabled: true

Pods:

$ kubectl get pods -n kong
NAME                                         READY   STATUS      RESTARTS   AGE
kong-1702379291-kong-6f6f65cb9f-qtsph        2/2     Running     0          48s
kong-1702379291-kong-init-migrations-6q8np   0/1     Completed   0          48s
kong-1702379291-postgresql-0                 1/1     Running     0          47s

Is there anyone that could be so kind to help me out and tell how to solve it or just address me to some resource/documentation to that could be useful to understand it?
Should I use kong/ingress instead? How could I configure the DB with that chart?

[pmalek]

Using --generate-name has the effect of having the helm release name generated which in turn is used in several crucial places in our helm chart, e.g. as a publish service (see https://docs.konghq.com/kubernetes-ingress-controller/latest/reference/cli-arguments/)

This is being used in the chart in
https://github.com/Kong/charts/blob/940bcdc52030b8cdad607710561b054003c1146e/charts/kong/templates/_helpers.tpl#L457-L463.

The log that you attached indicates that you installed a Gateway API Gateway with probably the annotation indicating to use kong-gateway-proxy Service as the publish service, i.e. the konghq.com/publish-service.

Is that the case? Can you share the Gateway manifest?

[pmalek]

I assume then that you have the Gateway already installed in the kong namespace (prior to the helm install above).

Can you share the output of: kubectl get gateways -n kong?

[mikyll]

You might be right, I totally forgot to delete the Gateway CRDs instances after uninstalling the chart 🤔

That would also explain why if I add a Service + HTTPRoute, my requests get forwarded correctly 😅

[mikyll]

Yup, it was already installed.

Here's the output of the command kubectl get gateways -n kong:

NAME   CLASS   ADDRESS     PROGRAMMED   AGE
kong   kong    127.0.0.1   True         21d

[pmalek]

You can either remove that Gateway to get rid of the error or make it use a different GatewayClass which would specify a different controllerName (this way your installation from the original message here would not interfere with that older Gateway).

See an example which does that here:

kubernetes-ingress-controller/examples/gateway-httproute.yaml

Line 84 in 628fe6a

controllerName: konghq.com/kic-gateway-controller

[pmalek]

Just for wider reference, there's an issue that I created to make the error message clearer and document the internal usage of mentioned annotation on the Gateway: #5328