AWS EKS - Optimize EC2 node kernel

[logo306142054]

It doesn't optimize EC2 node kernel by default when we create EKS cluster on the AWS by eksctl tool according official docs.

To optimize the EC2 node kernel, we must set the EC2 startup script when creating EKS cluster. The K8S has been using containerd as the container runtime since version 1.25. The kernel parameters in POD are affected by the startup parameters of containterd. Therefore, we also need to modify the startup configuration of containerd at the same time.

Since there are many parameter configurations involved, we use configuration file instead of command line parameters. For specific usage, please refer to EKSCTL official documentation. A configuration file example is as follows:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster-name
  region: us-west-2
  version: "1.26"

nodeGroups:
  - name: ng-1
    instanceType: c5.2xlarge
    desiredCapacity: 3
    volumeSize: 8
    #availabilityZones: ["us-west-2b"]
    amiFamily: AmazonLinux2
    ami: ami-05403c688b627ad3a
    iam:
      withAddonPolicies:
        ebs: true
    overrideBootstrapCommand: |
      #!/bin/bash
      echo "
      net.bridge.bridge-nf-call-ip6tables=1
      net.bridge.bridge-nf-call-iptables=1
      net.ipv4.ip_forward=1
      net.ipv4.conf.all.forwarding=1
      net.ipv4.neigh.default.gc_thresh1=4096
      net.ipv4.neigh.default.gc_thresh2=6144
      net.ipv4.neigh.default.gc_thresh3=8192
      net.ipv4.neigh.default.gc_interval=60
      net.ipv4.neigh.default.gc_stale_time=120
      kernel.perf_event_paranoid=-1
      #sysctls for k8s node config
      net.ipv4.tcp_slow_start_after_idle=0
      net.core.rmem_max=16777216
      fs.inotify.max_user_watches=524288
      kernel.softlockup_all_cpu_backtrace=1
      kernel.softlockup_panic=0
      kernel.watchdog_thresh=30
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.ip_local_port_range="1024 65535"
      fs.inotify.max_user_instances=8192
      fs.inotify.max_queued_events=16384
      vm.max_map_count=262144
      fs.may_detach_mounts=1
      net.core.netdev_max_backlog=16384
      net.ipv4.tcp_wmem=4096 12582912 16777216
      net.core.wmem_max=16777216
      net.core.somaxconn=32768
      net.ipv4.ip_forward=1
      net.ipv4.tcp_max_syn_backlog=8096
      net.ipv4.tcp_rmem=4096 12582912 16777216
      net.ipv6.conf.all.disable_ipv6=1
      net.ipv6.conf.default.disable_ipv6=1
      net.ipv6.conf.lo.disable_ipv6=1
      kernel.yama.ptrace_scope=0
      vm.swappiness=0
      kernel.core_uses_pid=1
      # Do not accept source routing
      net.ipv4.conf.default.accept_source_route=0
      net.ipv4.conf.all.accept_source_route=0
      # Promote secondary addresses when the primary address is removed
      net.ipv4.conf.default.promote_secondaries=1
      net.ipv4.conf.all.promote_secondaries=1
      # Enable hard and soft link protection
      fs.protected_hardlinks=1
      fs.protected_symlinks=1
      net.ipv4.conf.all.rp_filter=0
      net.ipv4.conf.default.rp_filter=0
      net.ipv4.conf.default.arp_announce = 2
      net.ipv4.conf.lo.arp_announce=2
      net.ipv4.conf.all.arp_announce=2
      net.ipv4.tcp_max_tw_buckets=5000
      net.ipv4.tcp_syncookies=1
      net.ipv4.tcp_fin_timeout=5
      net.ipv4.tcp_synack_retries=2
      kernel.sysrq=1
      fs.file-max=30000000
      fs.nr_open=30000000
      " >> /etc/sysctl.conf
      sysctl -p

      cat >> /etc/security/limits.conf <<EOF
      * soft nofile 30000000
      * hard nofile 30000000
      EOF

      sed -ri "s/LimitNPROC=infinity/LimitNPROC=30000000/g" /usr/lib/systemd/system/containerd.service
      sed -ri "s/LimitNOFILE=infinity/LimitNOFILE=30000000/g" /usr/lib/systemd/system/containerd.service
      systemctl daemon-reload
      
      cat >> /etc/systemd/system.conf <<EOF
      DefaultLimitNOFILE=30000000
      EOF
      systemctl daemon-reexec
    
      systemctl restart containerd
      
      source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh
      # Note: the first param is your cluster name that you defined in metadata above!!!
      /etc/eks/bootstrap.sh cluster-name --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS}"

Saving the above content as the file config.yaml and create EKS cluster using the following command:

eksctl create cluster -f config.yaml

For the created EKS cluster, the AWS will save the EC2 node startup script as a "launch template", which can be found in the EC2 console "Instances" -> "Launch Templates". 

It should be noted that the overrideBootstrapCommand in the above configuration file is saved in the launch template in the form of "user data" and encoded using base64. When manually editing "user data", you needs to decode it first.

The launch template has the concept of version management. After updating the launch template, the initial version is still used by default. We need to manually point the default version to the latest version. Only then can the latest launch template be automatically enabled when EKS node scaling. Another way is to enter the EKS control panel, go to the "Compute" tab of the cluster, and set the "Launch template" of the Node Group.

After updating the default version, you need to apply the latest launch template through EKS node scaling. The strategy is to first expand nodes, and then scale down nodes after the new nodes are created (to be verified). It is recommended that after manually updating the default version, use the second method to switch the Node Group's launch template. At this time, EKS will automatically roll update EC2 nodes without manual node scaling.