Start here to participate in the Amazon EKS preview program for EC2 ARM instances. Using the instructions and code in this repository you can run containers using EC2 A1 or M6 instances as part of a Kubernetes cluster that is managed by Amazon EKS.
EC2 ARM instances deliver significant cost savings for scale-out and Arm-based applications such as web servers, containerized microservices, caching fleets, and distributed data stores.
- A1 instances deliver significant cost savings for scale-out and Arm-based applications. These are the first EC2 instances powered by AWS Graviton Processors that feature 64-bit Arm Neoverse cores and custom silicon designed by AWS.
- M6g instances are powered by Arm-based AWS Graviton2 processors. They deliver up to 40% better price performance over current generation M5 instances and offer a balance of compute, memory, and networking resources for a broad set of workloads.
Note: The assets and instructions in this repository folder are offered as part of a public preview program administered by AWS.
Using the instructions and assets in this repository folder is governed as a preview program under the AWS Service Terms.
- The assets and instructions in this repository are offered on an as-is basis as part of a public preview program for new AWS service functionality.
- Please open an issue for comments, questions, or to report a bug.
- For issues with the Amazon EKS service (creating, modifying, deleting a cluster) or with your AWS account, please contact AWS support using the AWS console.
- As always, if you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions here or email AWS security directly.
- Make sure you have an active and valid AWS account. If you don't, you can create one here.
- If you haven't used Kubernetes before, familiarize yourself with the basics of Kubernetes
- If you haven't used Amazon EKS before, familiarize yourself with the EKS user guide. We also have a tutorial that is a good starting point for new users.
The specific resources you need to run containers on EC2 ARM instances with Amazon EKS are within this repository folder. All other resources needed to successfully start and manage an EKS cluster can be found within the EKS user guide.
- EKS currently supports the ability to run all nodes on ARM instances with Kubernetes version 1.15 (the default), 1.14 and 1.13.
Follow these instructions to create a Kubernetes cluster with Amazon EKS and start a service on EC2 ARM nodes.
Note: This guide requires that you create a new EKS cluster. Please ensure you complete all steps to avoid issues.
To create our cluster, we will use eksctl, the command line tool for EKS.
- Ensure you have the latest version of Homebrew installed.
If you don't have Homebrew, you can install it with the command:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" - Install the Weaveworks Homebrew tap:
brew tap weaveworks/tap - Install ekstctl:
brew install weaveworks/tap/eksctl - Test that your installation was successful:
eksctl --help
If you used the Homebrew instructions above to install eksctl on macOS, then kubectl and the aws-iam-authenticator have already been installed on your system. Otherwise, you can refer to the Amazon EKS getting started guide prerequisites.
Create an EKS cluster without provisioning worker nodes using the following eksctl command (change the --version if you don't want Kubernetes 1.15):
eksctl create cluster \
--name arm-preview \
--version 1.15 \
--region us-west-2 \
--without-nodegroup
Launching an EKS cluster using eksctl creates a CloudFormation stack. The launch process for this stack typically takes 10-15 minutes. You can monitor the progress in the EKS console.
Once the launch process has completed, we will want to review the CloudFormation stack to record the IDs of the Control Plane security group as well as the VPC ID. Navigate to the CloudFormation console. You will see a stack named eksctl-<cluster name>-cluster. Select this stack, and on the right-hand side panel, click the tab for Outputs. Record the values of the items for SecurityGroup and VPC.
Test that your cluster is running using kubectl get svc. It should return information such as the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP ww.xx.yy.zz <none> 443/TCP 20m
In order to support having only ARM nodes on our EKS cluster, we need to update some of the Kubernetes components. Follow the steps below to update CoreDNS, Kube-Proxy, and install the AWS ARM64 CNI plugin.
Run one of the below commands based upon the version of Kubernetes you are using to install an updated version of CoreDNS:
Kubernetes 1.15
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/dns-arm-1.15.yamlKubernetes 1.14
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/dns-arm-1.14.yamlKubernetes 1.13
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/dns-arm-1.13.yamlRun the below command based upon the version of Kubernetes you are using to install an updated version of kube-proxy:
Kubernetes 1.15
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/kube-proxy-arm-1.15.yamlKubernetes 1.14
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/kube-proxy-arm-1.14.yamlKubernetes 1.13
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/kube-proxy-arm-1.13.yamlRun the below command to install the AWS ARM64 CNI Plugin (this config works on all Kubernetes versions):
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/aws-k8s-cni-arm64.yaml- Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation. Ensure that you are in the AWS region that you created your EKS cluster in.
- Choose Create stack.
- For Choose a template, select Specify an Amazon S3 template URL.
- Paste the following URL into the text area and choose Next
https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-11-15/amazon-eks-arm-nodegroup.yaml - On the Specify Details page, fill out the following parameters accordingly, and choose Next.
- Stack name: Choose a stack name for your AWS CloudFormation stack. For example, you can call it -worker-nodes.
- KubernetesVersion Select the version of Kubernetes you chose when launching your EKS cluster. Important This version must match the version you used in Step 1: Create Your Amazon EKS Cluster; otherwise, your worker nodes cannot join the cluster.
- ClusterName: Enter the name that you used when you created your Amazon EKS cluster. Important This name must exactly match the name you used in Step 1: Create Your Amazon EKS Cluster; otherwise, your worker nodes cannot join the cluster.
- ClusterControlPlaneSecurityGroup: You will be presented with a drop-down list of security groups. Choose the value from the AWS CloudFormation output that you captured in the Create your Amazon EKS Cluster VPC step. (e.g. eksctl-<cluster name>-cluster-ControlPlaneSecurityGroup-XXXXXXXXXXXXX)
- NodeGroupName: Enter a name for your node group. This name can be used later to identify the Auto Scaling node group that is created for your worker nodes.
- NodeAutoScalingGroupMinSize: Enter the minimum number of nodes that your worker node Auto Scaling group can scale in to.
- NodeAutoScalingGroupDesiredCapacity: Enter the desired number of nodes to scale to when your stack is created.
- NodeAutoScalingGroupMaxSize: Enter the maximum number of nodes that your worker node Auto Scaling group can scale out to.
- NodeInstanceType: Choose one of the ARM instance types for your worker nodes (e.g.:
a1.large). - NodeVolumeSize: Enter node volume size. The default of 20 is fine.
- KeyName: Enter the name of an Amazon EC2 key pair that you can use to decrypt administrator password while RDP into your worker nodes after they launch. If you don't already have an Amazon EC2 key pair, you can create one in the AWS Management Console. For more information, see Amazon EC2 Key Pairs in the Amazon EC2 User Guide for Linux Instances.
Note: If you do not provide a key pair here, the AWS CloudFormation stack creation will fail.
-
VpcId: Choose the value from the AWS CloudFormation output that you captured in the Create your Amazon EKS Cluster VPC step. (e.g. eksctl-<cluster name>-cluster/VPC)
-
Subnets: Choose the subnets that you created in Create your Amazon EKS Cluster VPC.
-
NodeImageAMI11X: The Amazon EC2 Systems Manager parameter for the AMI image ID. You should not make any changes to this parameter.
- On the Options page, you can choose to tag your stack resources. Choose Next.
- On the Review page, review your information, acknowledge that the stack might create IAM resources, and then choose Create.
- After the ARM worker nodes stack has finished creating, select it in the console and choose the Outputs tab.
- Record the value of NodeInstanceRole for the node group that was created. You need this when you configure your Amazon EKS worker nodes in step 10.
Step 9. Configure the AWS authenticator configuration map to enable worker nodes to join your cluster
-
Download the configuration map
wget https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/aws-auth-cm-arm64.yaml -
Open the file with your favorite text editor. Replace the <ARN of instance role (not instance profile) of arm64 nodes (see step 9)> snippet with the NodeInstanceRole values that you recorded from step 10 above, and save the file.
Important: Do not modify any other lines in this file.
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: <ARN of instance role (not instance profile) of arm64 nodes (see step 10)>
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- Apply the configuration. This command may take a few minutes to finish:
kubectl apply -f aws-auth-cm-arm64.yaml
Note: If you receive the error "aws-iam-authenticator": executable file not found in PATH, then kubectl on your machine is not configured correctly for Amazon EKS. For more information, see Installing aws-iam-authenticator.
If you receive any other authorization or resource type errors, see Unauthorized or Access Denied (kubectl).
- Watch the status of your nodes and wait for them to reach the Ready status:
kubectl get nodes --watch
Launch the metrics server to test that you can schedule pods.
kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/eks-arm-preview/cni-metrics-helper-arm64.yaml
- Output:
clusterrole.rbac.authorization.k8s.io/cni-metrics-helper created
serviceaccount/cni-metrics-helper created
clusterrolebinding.rbac.authorization.k8s.io/cni-metrics-helper created
deployment.extensions/cni-metrics-helper created
Check the scheduled pods:
kubectl -n kube-system get pods -o wide
-
Run your own containers on your new EKS cluster.
-
Leave comments or questions on our GitHub issue.
-
This is an evolving project. As we roll out new features and functionality, we will update this repository and the roadmap issue.