Whether or not to support "Unmanaged" Gateway resources

[shaneutt]

We're coming up on the end of Gateway API Milestone 3 which we're currently intending to result in upgrading our Gateway API integration from alpha to beta which has the following implications:

1. v1beta1 APIs will become officially supported, at minimum until the end of the v2.x.x KIC release cycle
2. the feature gate for Gateway will switch to on by default

Presently we have support for Gateway resources in an "unmanaged" mode available in the KIC: you can only effectively have 1 Gateway resource and it's a reflection of a Kong Gateway (or proxy if you prefer) who's lifecycle is managed elsewhere (often manually by a human operator using helm). This unmanaged mode is an alternative to relying on Gateway resources managed by the Kong Gateway Operator (KGO) and has some benefits and drawbacks:

Benefits:

• existing deployments using the helm chart can take advantage of *Routes functionality in a backwards compatible manner
• we can get Gateway support into the hands of users faster than if we had to wait the months required to get a beta/GA release of the KGO ready

Drawbacks:

• we have to commit to maintaining an implementation-specific Gateway mode which is not defined by the upstream specification
• we can not provide the full expected Gateway featureset using this mode, as the API was designed for operator-style management in mind

When we started the Gateway API integration we intentionally created --feature-gates and put this integration behind that to avoid being committed, now before we call our integration with Gateway beta we have the option to decide what we're actually committing to as we currently are uncommitted to the API at all.

I want to get some feedback from other maintainers about what they think: please answer the poll, but ideally also comment on this discussion with your thoughts in depth on what path we should take.

[rainest]

Yes, barring some as yet unknown issue that makes it much harder. Reviewing what's in Gateway, the single Gateway limitation is basically fine--it's the same set of limitations (no segmentation of configuration by listen address, mainly) that you'd have with a single Kong instance anyway--so I'm fine imposing that limitation.

The code to scan Kong-side listener configuration and report if it cannot satisfy the Gateway requested is already in place, and I wouldn't expect that we need more than that. Are we aware of anything else we need to add?

Allowing this provides a more gradual migration path for existing Ingress-based configurations, since you can migrate your route-level configuration to HTTPRoute and such without migrating your Deployment management.

[shaneutt]

It's pretty much impossible to measure the future so we'll never really know the maintenance costs we'll incur by committing to two implementations until we're paying it, but the point about more gradual migration path is well taken and a strong point for committing. I still have a slight bias towards "no" personally (to be transparent), but I don't have a strong argument against your points because I think they're good points and the migration point in particular is persuasive. Suffice to say reducing the amount of maintenance and complexity we have overall in the projects we maintain is more ideal in general, but on the flip side requiring KGO to use GW routes will reduce the amount of usage and participation we get in the short term certainly. I'll continue to think about this and wait for some other viewpoints, but I can see myself being persuaded as I'm kinda on the line atm 🤔

[rainest]

Does the controller meaningfully distinguish between the two? It can't create Kong instances no matter what we do, since that functionality lives in the operator. The check to see if Kong configuration matches should be irrelevant for managed, but it's fine to run it still since the outcome should always be "yes, the Kong and Gateway configuration matches" in managed mode, and it won't do anything other than consume a negligible amount of CPU.

We may have more trouble with handling the standard Gateway conditions, e.g. adding a Condition and avoiding some routes if a new TCP Listener tries to request a port already occupied by a TLS Listener. That logic currently lives in the controller, but arguably needs to exist in the operator also (with the additional effect that it does not modify Kong listen configuration to include it). Unmanaged mode can't exclude problem routes or certificates unless the controller has the ability to perform those checks. That said I think it should be easy enough to make the Listener status checker a shareable package and add a controller flag to disable it if necessary (I'm not sure if the duplicate effort would be a problem, but if so the operator manages the controller config and can add the flag to disable the controller version).

[pmalek]

I can just comment from my general experience (not experience with Gateway API and/or k8s controllers):

Remember that no in Open Source is temporary and yes is forever :)

I do see the benefits of having earlier adoption but I'd consider twice (if not more than that) committing for (even short-term) support of features that we foresee dropping at some point. Actually I believe I'm not sure about this so let me ask this explicitly: do we foresee dropping support for unmanaged Gateway support in KIC at some point (in favor of support in Kong Gateway Operator)?

[shaneutt]

I definitely see a point where the KIC becomes a subsequent part of the KGO, and so "yes". This is part of the reason I'm leaning towards not releasing it in the first place.

[pmalek]

KIC becomes a subsequent part of the KGO

Can you elaborate? I don't think I understand what this mean.

[shaneutt]

It seems like in the future we could definitely deprecate our helm chart in favor of the KGO. If we did this and we supported migrations from helm to KGO, I could see us no longer needing the "unmanaged mode".

[mflendrich]

Supporting unmanaged Gateways is virtually equivalent to supporting the existing Helm chart for Gateway API features (because it's the only currently endorsed production-grade deployment method of KIC).

Then the big question can be paraphrased: do we want to force users to rip their existing Helm installs apart when willing to adopt Gateway API features? Some food for thought:

• We'd end up in the "everything is either deprecated or not ready yet" territory - this would effectively deprecate the Helm chart while the KGO is not GA,
• We'd make Gateway API incompatible with Kong deployment modes unsupported by operator at the given time,
• Adoption of Gateway API would practically mean redoing API management in one's org from scratch.

That said, I think that we should:

• recommend KGO as the installation method of choice but present it alongside the Helm chart. Perhaps define a feature comparison table between Helm-based deployments and Operator-based deployments so that the new user can make a choice depending on their requirements (with a strong recommendation from our side to use KGO).
• both now and in the future (when the operator is GA), offer a gradual adoption path where the move from Ingress to Gateway API can be a separate step from migrating Helm->KGO

I agree with @rainest's answer in full, and I believe that the cost of Drawbacks in @shaneutt's original message is worth paying for the reasons described above. That said, I empathize with the spirit of dropping unmanaged Gateways; I just believe that the degree of disruption would be difficult to justify as a net positive.

To @pmalek's point: once KGO is GA, we can (at some point) deprecate the Helm chart (and therefore the unmanaged mode); however the process of userbase migration from Helm to KGO will stretch well beyond that. So I think that we should expect the unmanaged mode to remain (possibly in the deprecated fashion, though) for 2 years or so.

[pmalek]

We'd end up in the "everything is either deprecated or not ready yet" territory - this would effectively deprecate the Helm chart while the KGO is not GA,

I have to - sadly - agree.

👍 on the table comparing the deployments via Gateway operator and helm chart.