Issue with upgrading to terraform-aws-service-catalog to v.100.0 for eks-core-components through pipeline - upgrade from k8s 1.23 to 1.24.

[sewmiuraj]

Hello,

I am starting to upgrade our eks cluster from k8s 1.23 to k8s 1.24. We are currently in the version v0.95.1 for repository - terraform-aws-service-catalog. My understanding is that i need to upgrade eks-core-components first and then upgrade eks-cluster.

I was able to upgrade the core components from my local pc without any issues. I also upgraded my local kubergrunt version to v.10.0 to support the upgrade.

This is how the plan looked like (locally):

 # module.fargate_fluent_bit["enable"].kubernetes_config_map.logging will be updated in-place
  ~ resource "kubernetes_config_map" "logging" {
      ~ data        = {
          + "filters.conf" = <<-EOT
                [FILTER]
                  Name kubernetes
                  Match *
                  Merge_Log Off
                  Buffer_Size 0
                  Kube_Meta_Cache_TTL 300s
            EOT
            # (1 unchanged element hidden)
        }
        id          = "aws-observability/aws-logging"
        # (2 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.k8s_external_dns["enable"].helm_release.k8s_external_dns will be updated in-place
  ~ resource "helm_release" "k8s_external_dns" {
        id                         = "external-dns"
        name                       = "external-dns"
      ~ version                    = "6.2.4" -> "6.12.2"
        # (26 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

This was applied locally successfully, but I never could run even the plan from the pipeline. It would throw the error:

Error: Get "https://a219bdea5fe77ddb324de47fa9153a05.yl4.eu-central-1.eks.amazonaws.com/api/v1/namespaces/aws-observability": getting credentials: decoding stdout: no kind "ExecCredential" is registered for version "client.authentication.k8s.io/v1alpha1" in scheme "pkg/runtime/scheme.go:100"

Am I correct to understand that this is an issue with kubergrunt versions, or is this something else?
(I can see the ability to add the url of the kubergrunt version is introduced only to the eks-cluster and not eks-core-components).

---

Tracked in ticket #110238

[ryehowell]

Hi @sewmiuraj, support for EKS 1.24 isn't available until version v0.100.0 of the terraform-aws-service-catalog. Trying to upgrade to EKS 1.24 with any version older than v0.100.0 will be problematic.

The reason for this is that EKS/K8s 1.24 came with multiple backward incompatible/breaking changes including some key API removals; which is the primary reason for the kubergrunt version dependency/upgrade requirement.

I suspect the error you are seeing is noted below (ExecCredential removal).

Here are a couple snippets from the K8s 1.24 Release Notes that are an overview of the major breaking changes:

• The client.authentication.k8s.io/v1alpha1 ExecCredential has been removed. If you are using a client-go credential plugin that relies on the v1alpha1 API please contact the distributor of your plugin for instructions on how to migrate to the v1 API
• Docker runtime support using dockershim in the kubelet is now completely removed in 1.24. The kubelet used to have a module called dockershim, which implements CRI support for Docker, and it has seen maintenance issues in the Kubernetes community. From 1.24 onwards, please move to a container runtime that is a full-fledged implementation of CRI (v1alpha1 or v1 compliant) as they become available.

Here are some additional links with additional information on the EKS/K8s 1.24 release:

• AWS EKS 1.24 Info
• Kubernetes 1.24 Official Announcement

To upgrade to 1.24, the suggested path is to incrementally upgrade the service-catalog as there are other backward incompatible changes that may need to be accounted for and worked thru, until you are to the version you would like to be at.

[sewmiuraj]

Thank you @ryehowell. My issue is a little different however. I can upgrade eks-core-services without issues to v.1.24 using service-catalog version v.100.0 in my local machine. But this error is thrown in our pipeline - we are using Github actions with ecs-deploy-runner. I maybe wrong, but I believe this is because I can't specify the kubergrunt version in the pipeline, but be more flexible with the version in my local machine.

I can see that the variable "kubergrunt_download_url" exposed in the eks-cluster module,
https://github.com/gruntwork-io/terraform-aws-service-catalog/blob/master/modules/services/eks-cluster/variables.tf

but in the eks-core-services module, I can't see a way to specify the version:
https://github.com/gruntwork-io/terraform-aws-service-catalog/blob/master/modules/services/eks-core-services/variables.tf

[ryehowell]

Hi @sewmiuraj, what version of the service catalog is the ecs-deploy-runner deployed from? Assuming that it is the same version of the service catalog previously mentioned v0.95.1, then the kubergrunt version likely being used is v0.8.0 which doesn't support EKS 1.24. As you mentioned, you can easily switch versions of the underlying tooling on a local dev machine. With the CI system, that tooling is a bit more integrated. You could potentially modify the version of kubergrunt that is being used via the ecs-deploy-runner, but I suspect you will still run into issues even with a supported version of kubergrunt due to other dependent resources and incompatible changes within the underlying modules of the service catalog, for example:

• Helm Provider Configuration (which is configured for the client.authentication.k8s.io/v1alpha1 API version, which is the error you are seeing).

I think the suggested path forward is to update the infrastructure to the latest needed version of the service catalog (doing this will also update the ecs-deploy-runner) so that all of the supporting and dependent resources are at a compatible version.

[ryehowell]

Hi @sewmiuraj, I wanted to follow up regarding the last comment. Are you still facing the original issue posted?

[sewmiuraj]

Hi @ryehowell ,

I hope you are doing well.

Thank you very much for reaching out. Unfortunately, the issue is still there. Based on your instructions, I have tried the following:

1. Upgraded the ecs-deploy-runner from v.0.95.1 to v.100.0 - but there weren't any changes introduced during the update.
2. Updated the versions in our pipelines.yml that runs terraform plan and deploy to have the following versions:
   GRUNTWORK_INSTALLER_VERSION: v0.0.39
   MODULE_CI_VERSION: v0.50.12
   MODULE_SECURITY_VERSION: v0.67.1
3. I had removed the workaround you have explained in https://github.com/orgs/gruntwork-io/discussions/671 assuming it was what might have caused the issue.

# add a new generate block in the file where eks is configured, such as in the: eks-cluster.hcl file
generate "k8s_provider_version" {
  path      = "k8s_version_override.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "< 2.17"
    }
  }
}
EOF
}

I have added it back, and the error now in the pipeline looks like this:

What I am working on now at the moment?
As the only other place that could be updated, I am now working to re-create our image in ECR for kaniko and ecs-deploy-runner based on the instructions provided in https://github.com/gruntwork-io/terraform-aws-ci/releases?page=2 for releasev0.50.12.

Your thoughts in this is highly appreciated.

[ryehowell]

Hi @sewmiuraj,

You are correct with rebuilding the Docker images as you described, and I think I see the root cause of the issue. If the ecs-deploy-runner image was built with the default configuration (no build args changed), then an incompatible version of kubergrunt will be used (and thus the client.authentication.k8s.io/v1alpha1 API error).

Here is the offending line.

The dependency (kubergrunt version) looks like it wasn't updated to v0.10.0 which is where support for EKS 1.24 was added. Updating to v0.100.0 of the Service Catalog should have provided the proper support by default, but this looks like it was missed. So version 0.8.0 of kubergrunt is still being used by default which does not support EKS 1.24.

Can you try rebuilding the ecs-deploy-runner Docker image as you were planning to do, and provide a build arg to override the default kubergrunt version and instead use v0.10.0?

Something like: --build-arg kubergrunt_version=v0.10.0

[sewmiuraj]

Hi @ryehowell ,

Thank you for sharing your insights on this. After building the ecs-deployer image by setting the kubergrunt version to v.10.0, I was able to upgrade our cluster using the pipeline. :)

Thank you for your time...

[ryehowell]

That is great to hear! Very glad we were able to get you unblocked. Thank you for opening up the discussion and for your participation in getting to the resolution 😃