Unable to create Host resource due to tls cert error from emissary-apiext

[fs185143]

Describe the bug
Cannot create Host resource after upgrading emissary-apiext and emissary-ingress

To Reproduce
Steps to reproduce the behavior:

1. Create Host resource on an environment running ambassador 2.0.4
2. Upgrade from 2.0.4 to 3.9.1
3. Observe emissary-apiext start as expected
4. Try create Host resource again
5. Observe error in emissary-apiext logs:

Host/emissary/ingress-host dry-run failed, error: conversion webhook for getambassador.io/v3alpha1, Kind=Host failed: Post "https://emissary-apiext.emissary-system.svc:443/?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority

6. emissary/emissary-ingress deployment fails indefinitely as it includes this Host
7. Restart emissary-apiext pod in emissary-system namespace
8. Host applies fine and emissary-ingress deployment can proceed

Expected behavior
A clear and concise description of what you expected to happen.

Should be able to apply Host without getting above webhook error from emissary-system/emissary-apiext

Versions (please complete the following information):

• Ambassador: 3.9.1
• Kubernetes environment: GKE
• Version: v1.27.14-gke.1042001

Additional context
Wondering if some sort of race condition is occurring

[fs185143]

Am I correct in thinking that the certificate in question is the value of caBundle here?

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  generation: 2
  labels:
    app.kubernetes.io/instance: emissary-apiext
    app.kubernetes.io/managed-by: kubectl_apply_-f_emissary-apiext.yaml
    app.kubernetes.io/name: emissary-apiext
    app.kubernetes.io/part-of: emissary-apiext
  name: hosts.getambassador.io
  resourceVersion: "244433"
  uid: ef8bf370-0ca4-485c-859d-2a083a67db40
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        caBundle: LS0t...0tLQo=

[fs185143]

something i noticed after running the b64 decoded caBundle value through openssl x509 is that the validity is

        Validity
            Not Before: Jul 25 07:40:24 2024 GMT
            Not After : Jul 25 07:40:24 2025 GMT

whereas the CRD's status shows

status:
  acceptedNames:
    categories:
    - ambassador-crds
    kind: Host
    listKind: HostList
    plural: hosts
    singular: host
  conditions:
  - lastTransitionTime: "2024-07-25T12:09:21Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2024-07-25T12:09:21Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v2

and metadata.creationTimestamp of creationTimestamp: "2024-07-25T12:09:21Z"

the cert error from above was at 07:42:19.561

[fs185143]

suspect these logs from emissary-system/emissary-apiext may be relevant

time="2024-07-25 11:57:06.6676" level=info msg="Configuring conversion for \"authservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.6901" level=info msg="Configuring conversion for \"consulresolvers.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.6995" level=info msg="Configuring conversion for \"devportals.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.7135" level=info msg="Configuring conversion for \"hosts.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.7348" level=info msg="Configuring conversion for \"kubernetesendpointresolvers.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:06.7446" level=info msg="Configuring conversion for \"kubernetesserviceresolvers.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:07.0713" level=info msg="Configuring conversion for \"logservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:07.4710" level=info msg="Configuring conversion for \"mappings.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:07.8720" level=info msg="Configuring conversion for \"modules.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:08.2707" level=info msg="Configuring conversion for \"ratelimitservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:08.6711" level=info msg="Configuring conversion for \"tcpmappings.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.0709" level=info msg="Configuring conversion for \"tlscontexts.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.4713" level=info msg="Configuring conversion for \"tracingservices.getambassador.io\"" func=github.com/emissary-ingress/emissary/v3/pkg/apiext/internal.updateCRD file="/go/pkg/apiext/internal/inject.go:137" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.8716" level=error msg="goroutine \"/configure-crds\" exited with error: 13 errors:\n 1. customresourcedefinitions.apiextensions.k8s.io \"authservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 2. customresourcedefinitions.apiextensions.k8s.io \"consulresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 3. customresourcedefinitions.apiextensions.k8s.io \"devportals.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 4. customresourcedefinitions.apiextensions.k8s.io \"hosts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 5. customresourcedefinitions.apiextensions.k8s.io \"kubernetesendpointresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 6. customresourcedefinitions.apiextensions.k8s.io \"kubernetesserviceresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 7. customresourcedefinitions.apiextensions.k8s.io \"logservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 8. customresourcedefinitions.apiextensions.k8s.io \"mappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 9. customresourcedefinitions.apiextensions.k8s.io \"modules.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 10. customresourcedefinitions.apiextensions.k8s.io \"ratelimitservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 11. customresourcedefinitions.apiextensions.k8s.io \"tcpmappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 12. customresourcedefinitions.apiextensions.k8s.io \"tlscontexts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 13. customresourcedefinitions.apiextensions.k8s.io \"tracingservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=apiext PID=1 THREAD=/configure-crds
time="2024-07-25 11:57:09.8717" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238" CMD=apiext PID=1 THREAD=":shutdown_logger"
time="2024-07-25 11:57:09.8720" level=info msg="  final goroutine statuses:" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:84" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8720" level=info msg="    /configure-crds: exited with error" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8721" level=info msg="    /serve-http    : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8721" level=info msg="    /serve-https   : exited" func=github.com/datawire/dlib/dgroup.logGoroutineStatuses file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:95" CMD=apiext PID=1 THREAD=":shutdown_status"
time="2024-07-25 11:57:09.8722" level=error msg="shut down with error error: 13 errors:\n 1. customresourcedefinitions.apiextensions.k8s.io \"authservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 2. customresourcedefinitions.apiextensions.k8s.io \"consulresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 3. customresourcedefinitions.apiextensions.k8s.io \"devportals.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 4. customresourcedefinitions.apiextensions.k8s.io \"hosts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 5. customresourcedefinitions.apiextensions.k8s.io \"kubernetesendpointresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 6. customresourcedefinitions.apiextensions.k8s.io \"kubernetesserviceresolvers.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 7. customresourcedefinitions.apiextensions.k8s.io \"logservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 8. customresourcedefinitions.apiextensions.k8s.io \"mappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 9. customresourcedefinitions.apiextensions.k8s.io \"modules.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 10. customresourcedefinitions.apiextensions.k8s.io \"ratelimitservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 11. customresourcedefinitions.apiextensions.k8s.io \"tcpmappings.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 12. customresourcedefinitions.apiextensions.k8s.io \"tlscontexts.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope\n 13. customresourcedefinitions.apiextensions.k8s.io \"tracingservices.getambassador.io\" is forbidden: User \"system:serviceaccount:emissary-system:emissary-apiext\" cannot update resource \"customresourcedefinitions/status\" in API group \"apiextensions.k8s.io\" at the cluster scope" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=apiext PID=1

[fs185143]

seems related to #5468 which afaik is not included in v3.9.1

[cindymullins-dw]

Yes, at a glance it does seem that fix could be related and was merged after the v.3.9.1 release. Future releases of Emissary will depend on the maintainers collectively so that schedule is currently undecided. It's possible to build Emissary from source to capture PRs that have been merged but not yet released in a new version, if you wanted to test that.

[fs185143]

Yes, at a glance it does seem that fix could be related and was merged after the v.3.9.1 release. Future releases of Emissary will depend on the maintainers collectively so that schedule is currently undecided. It's possible to build Emissary from source to capture PRs that have been merged but not yet released in a new version, if you wanted to test that.

Is there a discussion or anything that relates to recent news regarding the future of emissary releases? We would only be able to target stable releases/patch fixes

[fs185143]

also after investigating patches, i think this is likely to be the fix for our particular issue c8edb16 - once it gets released anyway

[kflynn]

@fs185143 We're looking into an Emissary 3.10 that will include that fix -- watch this space. 🙂