Lab 04 - Ingress with ALB
-In this lab you are going to learn how to route traffic to your applications running inside EKS using Application Load Balancer offered by AWS.
+In this lab you are going to learn how to route traffic to your applications running inside EKS using Application Load Balancer offered by AWS.
Pre Requisites
-
-
- [ ] EKS Cluster +
- [ ] EKS Cluster
- [ ] Two Subnets in two AZs
- [ ] Public/Private Subnets should have relevant tags (Already done if created with Cloudformation Template)
Install AWS Load Balancer Cont
⠀ The LBC listens to the Ingress resources in the Kubernetes cluster and dynamically updates the ALB configuration in AWS to match the defined routing rules, ensuring seamless traffic management.
-Download IAM Policy
+Download IAM Policy
curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/docs/install/iam_policy.json
Create IAM Policy
+Create IAM Policy
aws iam create-policy \
--policy-name AWSLoadBalancerControllerIAMPolicy \
--policy-document file://iam_policy.json
Note the ARN which you will use in the next command.
+Note the ARN which you will use in the next command.
Create IAM Role using eksctl
Replace eks-cluster-01 with the name of your cluster, 111122223333 with your account ID, and then run the command. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.
eksctl create iamserviceaccount \
@@ -326,9 +326,9 @@ Install AWS Load Balancer Cont
--name=aws-load-balancer-controller \
--role-name AmazonEKSLoadBalancerControllerRole \
--attach-policy-arn=arn:aws:iam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \
- --approve
+ --approve
validate
+validate
eksctl get iamserviceaccount --cluster eks-cluster-01
Install AWS Load Balancer Controller using Helm V3
@@ -347,7 +347,7 @@Install AWS Load Bal --set serviceAccount.name=aws-load-balancer-controller -
validate
+validate
helm list -A
kubectl get deployment,pods -n kube-system -l "app.kubernetes.io/instance=aws-load-balancer-controller"
@@ -387,28 +387,28 @@ Create Ingress Rules
port:
number: 80
kubectl apply -f vote-ing.yaml
+kubectl apply -f vote-ing.yaml
-kubectl get ing
+kubectl get ing
-You could further get the details of the ALB using AWS CLI as
+You could further get the details of the ALB using AWS CLI as
aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-instavot-vote`) == `true`]'
At this time, you shall see a Application Load Balancer(ALB) created with the rules autoamtically added to route traffic to vote and result apps. 
Add Local DNS
-You have created the ingress rules based on hostnames e.g. vote.example.com and result.example.com. In order for you to be able to access those, there has to be a dns entry pointing to your ALB.
+You have created the ingress rules based on hostnames e.g. vote.example.com and result.example.com. In order for you to be able to access those, there has to be a dns entry pointing to your ALB.
vote.example.com -------+ +----- vote:80
| +-------------+ |
| | ingress | |
+===> | node:80 | ===+
| +-------------+ |
| |
- results.example.com -------+ +----- result:80
+ result.example.com -------+ +----- result:80
To achieve this you need to either,
-- Find out the IP Address that your ALB is pointing to
+- Find out the IP Address that your ALB is pointing to
- Create a DNS entry, provided you own the domain and have access to the dns management console.
- Create a local hosts file entry. On unix systems its in
/etc/hosts file. On windows its at C:\Windows\System32\drivers\etc\hosts. You need admin access to edit this file.
@@ -417,13 +417,13 @@ Add Local DNS
sudo vim /etc/hosts
And add an entry such as ,
-xxx.xxx.xxx.xxx vote.example.com results.example.com kube-ops-view.example.org
+xxx.xxx.xxx.xxx vote.example.com result.example.com
where,
- xxx.xxx.xxx.xxx is one of the actual IP addresss of ALB.
-You could find the IP address by using the following command
+You could find the IP address by using the following command
nslookup k8s-instavot-vote-e764f4b4a3-2050376139.ap-southeast-1.elb.amazonaws.com
where DNS name is copied from ALB’s DNS name
diff --git a/index.html b/index.html
index be9b3148..ca290668 100644
--- a/index.html
+++ b/index.html
@@ -313,5 +313,5 @@
Team
diff --git a/search/search_index.json b/search/search_index.json
index 3eb50cde..5f0d3707 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Kubernete Tutorials with CKA and CKAD Prep Guide Welcome to the Kubernetes Tutorial with CKA and CKAD Prep Guide by School of Devops This document was originally created as a Lab Guide to go along with the Kubernetes Course by School of Devops . However, you could also use it on its own now to practice learning kubernetes topics. If you still believe you could learn better and faster by audio visual means, definitely consider subscribing to our course at schoolofdevops.com . Team Gourav Shah Vijayboopathy Venkat Abhijeet Nirmal","title":"Home"},{"location":"#kubernete-tutorials-with-cka-and-ckad-prep-guide","text":"Welcome to the Kubernetes Tutorial with CKA and CKAD Prep Guide by School of Devops This document was originally created as a Lab Guide to go along with the Kubernetes Course by School of Devops . However, you could also use it on its own now to practice learning kubernetes topics. If you still believe you could learn better and faster by audio visual means, definitely consider subscribing to our course at schoolofdevops.com .","title":"Kubernete Tutorials with CKA and CKAD Prep Guide"},{"location":"#team","text":"Gourav Shah Vijayboopathy Venkat Abhijeet Nirmal","title":"Team"},{"location":"10_kubernetes_autoscaling/","text":"Autoscaling Pods with HPA With Horizontal Pod Autoscaling, Kubernetes automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization (or, with alpha support, on some other, application-provided metrics). The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behavior of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment to match the observed average CPU utilization to the target specified by user Prerequisites Metrics Server . This needs to be setup if you are using kubeadm etc. and replaces heapster starting with kubernetes version 1.8. Resource Requests and Limits. Defining CPU as well as Memory requirements for containers in Pod Spec is a must Deploying Metrics Server Kubernetes Horizontal Pod Autoscaler along with kubectl top command depends on the core monitoring data such as cpu and memory utilization which is scraped and provided by kubelet, which comes with in built cadvisor component. Earlier, you would have to install a additional component called heapster in order to collect this data and feed it to the hpa controller. Starting with 1.8 version of Kubernetes, this behavior is changed, and now metrics-server would provide this data. Metric server is being included as a essential component for kubernetes cluster, and being incroporated into kubernetes to be included out of box. It stores the core monitoring information using in-memory data store. If you try to pull monitoring information using the following commands kubectl top pod kubectl top node it does not show it, rather gives you a error message similar to [output] Error from server (NotFound): the server could not find the requested resource (get services http:heapster:) Even though the error mentions heapster, its replaced with metrics server by default now. Deploy metric server with the following commands, cd ~ git clone https://github.com/schoolofdevops/metrics-server.git kubectl apply -k metrics-server/manifests/overlays/release Validate kubectl get deploy,pods -n kube-system --selector='k8s-app=metrics-server' You could validate again with kubectl top pod kubectl top node where expected output should be similar to, kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% kind-control-plane 123m 6% 688Mi 17% kind-worker 39m 1% 498Mi 12% kind-worker2 31m 1% 422Mi 10% If you see a similar output, monitoring is now been setup. Create a HPA To demonstrate Horizontal Pod Autoscaler we will use a custom docker image based on the php-apache image file: vote-hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vote spec: minReplicas: 2 maxReplicas: 10 metrics: - type: ContainerResource containerResource: name: cpu container: vote # change this as per actual container name target: type: Utilization averageUtilization: 50 scaleTargetRef: apiVersion: apps/v1 kind: Deployment # change it to Deployment if have created a deployment already name: vote behavior: scaleDown: policies: - type: Pods value: 2 periodSeconds: 120 - type: Percent value: 25 periodSeconds: 120 stabilizationWindowSeconds: 60 scaleUp: stabilizationWindowSeconds: 45 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 2 periodSeconds: 15 selectPolicy: Max apply kubectl apply -f vote-hpa.yaml Validate kubectl get hpa you should see [sample output] NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE vote Deployment/vote 2%/50% 2 10 2 12m If you see unknown/50% under TARGETS , there is a underlying issue that you may need to resolve. Check if you have defined the resources (rquests, limits) for the containers in the pod. kubectl describe hpa vote kubectl get pod,deploy If you have a monitoring system such as grafana, you could also view the graphs for vote deployment. Launch a Load Test If you do not have the service for vote app to receive the traffic, create one as cd k8s-code/projects/instavote/dev kubectl apply -f vote-svc.yaml Prepare to monitor the autoscaling by opening a new window connected to your cluster and by launching, watch 'kubectl top pods;kubectl get pods; kubectl get hpa ; kubectl get deploy' Create a Load Test Job as, file: loadtest-job.yaml apiVersion: batch/v1 kind: Job metadata: generateName: loadtest spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: [\"siege\", \"--concurrent=1\", \"--benchmark\", \"--time=4m\", \"http://vote\"] restartPolicy: Never backoffLimit: 4 and launch it as kubectl create -f loadtest-job.yaml This will launch a one off Job which would run for 4 minutes. To get information about the job kubectl get jobs kubectl describe job loadtest-xxx replace loadtest-xxx with actual job name. To check the load test output kubectl logs -f loadtest-xxxx [replace loadtest-xxxx with the actual pod id.] [Sample Output] ** SIEGE 3.0.8 ** Preparing 15 concurrent users for battle. root@kube-01:~# kubectl logs vote-loadtest-tv6r2 -f ** SIEGE 3.0.8 ** Preparing 15 concurrent users for battle. ..... Lifting the server siege... done. Transactions: 41618 hits Availability: 99.98 % Elapsed time: 299.13 secs Data transferred: 127.05 MB Response time: 0.11 secs Transaction rate: 139.13 trans/sec Throughput: 0.42 MB/sec Concurrency: 14.98 Successful transactions: 41618 Failed transactions: 8 Longest transaction: 3.70 Shortest transaction: 0.00 FILE: /var/log/siege.log You can disable this annoying message by editing the .siegerc file in your home directory; change the directive 'show-logfile' to false. Now check the job status again, kubectl get jobs NAME DESIRED SUCCESSFUL AGE vote-loadtest 1 1 10m While it is running, Keep monitoring for the load on the pod as the job progresses. You should see hpa in action as it scales out/in the vote deployment with the increasing/decreasing load. Summary In this lab, you have successful configured and demonstrated dynamic scaling ability of kubernetes using HorizontalPodAutoscaler. You have also learnt about a new jobs controller type for running one off or batch jobs. Reading List Kubernetes Monitoring Architecture Core Metrics Pipeline Metrics Server Assigning Resources to Containers and Pods Horizontal Pod Autoscaler","title":"Lab K301 - Auto Scaling with HPA"},{"location":"10_kubernetes_autoscaling/#autoscaling-pods-with-hpa","text":"With Horizontal Pod Autoscaling, Kubernetes automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization (or, with alpha support, on some other, application-provided metrics). The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behavior of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment to match the observed average CPU utilization to the target specified by user","title":"Autoscaling Pods with HPA"},{"location":"10_kubernetes_autoscaling/#prerequisites","text":"Metrics Server . This needs to be setup if you are using kubeadm etc. and replaces heapster starting with kubernetes version 1.8. Resource Requests and Limits. Defining CPU as well as Memory requirements for containers in Pod Spec is a must","title":"Prerequisites"},{"location":"10_kubernetes_autoscaling/#deploying-metrics-server","text":"Kubernetes Horizontal Pod Autoscaler along with kubectl top command depends on the core monitoring data such as cpu and memory utilization which is scraped and provided by kubelet, which comes with in built cadvisor component. Earlier, you would have to install a additional component called heapster in order to collect this data and feed it to the hpa controller. Starting with 1.8 version of Kubernetes, this behavior is changed, and now metrics-server would provide this data. Metric server is being included as a essential component for kubernetes cluster, and being incroporated into kubernetes to be included out of box. It stores the core monitoring information using in-memory data store. If you try to pull monitoring information using the following commands kubectl top pod kubectl top node it does not show it, rather gives you a error message similar to [output] Error from server (NotFound): the server could not find the requested resource (get services http:heapster:) Even though the error mentions heapster, its replaced with metrics server by default now. Deploy metric server with the following commands, cd ~ git clone https://github.com/schoolofdevops/metrics-server.git kubectl apply -k metrics-server/manifests/overlays/release Validate kubectl get deploy,pods -n kube-system --selector='k8s-app=metrics-server' You could validate again with kubectl top pod kubectl top node where expected output should be similar to, kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% kind-control-plane 123m 6% 688Mi 17% kind-worker 39m 1% 498Mi 12% kind-worker2 31m 1% 422Mi 10% If you see a similar output, monitoring is now been setup.","title":"Deploying Metrics Server"},{"location":"10_kubernetes_autoscaling/#create-a-hpa","text":"To demonstrate Horizontal Pod Autoscaler we will use a custom docker image based on the php-apache image file: vote-hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vote spec: minReplicas: 2 maxReplicas: 10 metrics: - type: ContainerResource containerResource: name: cpu container: vote # change this as per actual container name target: type: Utilization averageUtilization: 50 scaleTargetRef: apiVersion: apps/v1 kind: Deployment # change it to Deployment if have created a deployment already name: vote behavior: scaleDown: policies: - type: Pods value: 2 periodSeconds: 120 - type: Percent value: 25 periodSeconds: 120 stabilizationWindowSeconds: 60 scaleUp: stabilizationWindowSeconds: 45 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 2 periodSeconds: 15 selectPolicy: Max apply kubectl apply -f vote-hpa.yaml Validate kubectl get hpa you should see [sample output] NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE vote Deployment/vote 2%/50% 2 10 2 12m If you see unknown/50% under TARGETS , there is a underlying issue that you may need to resolve. Check if you have defined the resources (rquests, limits) for the containers in the pod. kubectl describe hpa vote kubectl get pod,deploy If you have a monitoring system such as grafana, you could also view the graphs for vote deployment.","title":"Create a HPA"},{"location":"10_kubernetes_autoscaling/#launch-a-load-test","text":"If you do not have the service for vote app to receive the traffic, create one as cd k8s-code/projects/instavote/dev kubectl apply -f vote-svc.yaml Prepare to monitor the autoscaling by opening a new window connected to your cluster and by launching, watch 'kubectl top pods;kubectl get pods; kubectl get hpa ; kubectl get deploy' Create a Load Test Job as, file: loadtest-job.yaml apiVersion: batch/v1 kind: Job metadata: generateName: loadtest spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: [\"siege\", \"--concurrent=1\", \"--benchmark\", \"--time=4m\", \"http://vote\"] restartPolicy: Never backoffLimit: 4 and launch it as kubectl create -f loadtest-job.yaml This will launch a one off Job which would run for 4 minutes. To get information about the job kubectl get jobs kubectl describe job loadtest-xxx replace loadtest-xxx with actual job name. To check the load test output kubectl logs -f loadtest-xxxx [replace loadtest-xxxx with the actual pod id.] [Sample Output] ** SIEGE 3.0.8 ** Preparing 15 concurrent users for battle. root@kube-01:~# kubectl logs vote-loadtest-tv6r2 -f ** SIEGE 3.0.8 ** Preparing 15 concurrent users for battle. ..... Lifting the server siege... done. Transactions: 41618 hits Availability: 99.98 % Elapsed time: 299.13 secs Data transferred: 127.05 MB Response time: 0.11 secs Transaction rate: 139.13 trans/sec Throughput: 0.42 MB/sec Concurrency: 14.98 Successful transactions: 41618 Failed transactions: 8 Longest transaction: 3.70 Shortest transaction: 0.00 FILE: /var/log/siege.log You can disable this annoying message by editing the .siegerc file in your home directory; change the directive 'show-logfile' to false. Now check the job status again, kubectl get jobs NAME DESIRED SUCCESSFUL AGE vote-loadtest 1 1 10m While it is running, Keep monitoring for the load on the pod as the job progresses. You should see hpa in action as it scales out/in the vote deployment with the increasing/decreasing load.","title":"Launch a Load Test"},{"location":"10_kubernetes_autoscaling/#summary","text":"In this lab, you have successful configured and demonstrated dynamic scaling ability of kubernetes using HorizontalPodAutoscaler. You have also learnt about a new jobs controller type for running one off or batch jobs.","title":"Summary"},{"location":"10_kubernetes_autoscaling/#reading-list","text":"Kubernetes Monitoring Architecture Core Metrics Pipeline Metrics Server Assigning Resources to Containers and Pods Horizontal Pod Autoscaler","title":"Reading List"},{"location":"12_troubleshooting/","text":"Troubleshooting the Kubernetes cluster In this chapter we will learn about how to trouble shoot our Kubernetes cluster at control plane level and at application level . Troubleshooting the control plane Listing the nodes in a cluster First thing to check if your cluster is working fine or not is to list the nodes associated with your cluster. kubectl get nodes Make sure that all nodes are in Ready state. List the control plane pods If your nodes are up and running, next thing to check is the status of Kubernetes components. Run, kubectl get pods -n kube-system If any of the pod is restarting or crashing , look in to the issue. This can be done by getting the pod's description. For example, in my cluster kube-dns is crashing. In order to fix this first check the deployment for errors. kubectl describe deployment -n kube-system kube-dns Log files Master If your deployment is good, the next thing to look for is log files. The locations of log files are given below... /var/log/kube-apiserver.log - For API Server logs /var/log/kube-scheduler.log - For Scheduler logs /var/log/kube-controller-manager.log - For Replication Controller logs If your Kubernetes components are running as pods, then you can get their logs by following the steps given below, Keep in mind that the actual pod's name may differ from cluster to cluster... kubectl logs -n kube-system -f kube-apiserver-node1 kubectl logs -n kube-system -f kube-scheduler-node1 kubectl logs -n kube-system -f kube-controller-manager-node1 Worker Nodes In your worker, you will need to check for errors in kubelet's log... sudo journalctl -u kubelet Troubleshooting the application Sometimes your application(pod) may fail to start because of various reasons. Let's see how to troubleshoot. Getting detailed status of an object (pods, deployments) object.status shows a detailed information about whats the status of an object ( e.g. pod) and why its in that condition. This can be very useful to identify the issues. Example kubectl get pod vote -o yaml example output snippet when a wrong image was used to create a pod. status: ... containerStatuses: .... state: waiting: message: 'rpc error: code = Unknown desc = Error response from daemon: manifest for schoolofdevops/vote:latst not found' reason: ErrImagePull hostIP: 139.59.232.248 Checking the status of Deployment For this example I have a sample deployment called nginx. FILE: nginx-deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: ngnix:latest ports: - containerPort: 80 List the deployment to check for the availability of pods kubectl get deployment nginx NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx 1 1 1 0 20h It is clear that my pod is unavailable. Lets dig further. Check the events of your deployment. kubectl describe deployment nginx List the pods to check for any registry related error kubectl get pods NAME READY STATUS RESTARTS AGE nginx-57c88d7bb8-c6kpc 0/1 ImagePullBackOff 0 7m As we can see, we are not able to pull the image( ImagePullBackOff ). Let's investigate further. kubectl describe pod nginx-57c88d7bb8-c6kpc Check the events of the pod's description. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m default-scheduler Successfully assigned nginx-57c88d7bb8-c6kpc to ip-11-0-1-111.us-west-2.compute.internal Normal SuccessfulMountVolume 9m kubelet, ip-11-0-1-111.us-west-2.compute.internal MountVolume.SetUp succeeded for volume \"default-token-8cwn4\" Normal Pulling 8m (x4 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal pulling image \"ngnix\" Warning Failed 8m (x4 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal Failed to pull image \"ngnix\": rpc error: code = Unknown desc = Error response from daemon: repository ngnix not found: does not exist or no pull access Normal BackOff 7m (x6 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal Back-off pulling image \"ngnix\" Warning FailedSync 4m (x24 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal Error syncing pod Bingo! The name of the image is ngnix instead of nginx . So fix the typo in your deployment file and redo the deployment . Sometimes, your application(pod) may fail to start because of some configuration issues. For those errors, we can follow the logs of the pod. kubectl logs -f nginx-57c88d7bb8-c6kpc If you have any errors it will get populated in your logs.","title":"Troubleshooting Tips"},{"location":"12_troubleshooting/#troubleshooting-the-kubernetes-cluster","text":"In this chapter we will learn about how to trouble shoot our Kubernetes cluster at control plane level and at application level .","title":"Troubleshooting the Kubernetes cluster"},{"location":"12_troubleshooting/#troubleshooting-the-control-plane","text":"","title":"Troubleshooting the control plane"},{"location":"12_troubleshooting/#listing-the-nodes-in-a-cluster","text":"First thing to check if your cluster is working fine or not is to list the nodes associated with your cluster. kubectl get nodes Make sure that all nodes are in Ready state.","title":"Listing the nodes in a cluster"},{"location":"12_troubleshooting/#list-the-control-plane-pods","text":"If your nodes are up and running, next thing to check is the status of Kubernetes components. Run, kubectl get pods -n kube-system If any of the pod is restarting or crashing , look in to the issue. This can be done by getting the pod's description. For example, in my cluster kube-dns is crashing. In order to fix this first check the deployment for errors. kubectl describe deployment -n kube-system kube-dns","title":"List the control plane pods"},{"location":"12_troubleshooting/#log-files","text":"","title":"Log files"},{"location":"12_troubleshooting/#master","text":"If your deployment is good, the next thing to look for is log files. The locations of log files are given below... /var/log/kube-apiserver.log - For API Server logs /var/log/kube-scheduler.log - For Scheduler logs /var/log/kube-controller-manager.log - For Replication Controller logs If your Kubernetes components are running as pods, then you can get their logs by following the steps given below, Keep in mind that the actual pod's name may differ from cluster to cluster... kubectl logs -n kube-system -f kube-apiserver-node1 kubectl logs -n kube-system -f kube-scheduler-node1 kubectl logs -n kube-system -f kube-controller-manager-node1","title":"Master"},{"location":"12_troubleshooting/#worker-nodes","text":"In your worker, you will need to check for errors in kubelet's log... sudo journalctl -u kubelet","title":"Worker Nodes"},{"location":"12_troubleshooting/#troubleshooting-the-application","text":"Sometimes your application(pod) may fail to start because of various reasons. Let's see how to troubleshoot.","title":"Troubleshooting the application"},{"location":"12_troubleshooting/#getting-detailed-status-of-an-object-pods-deployments","text":"object.status shows a detailed information about whats the status of an object ( e.g. pod) and why its in that condition. This can be very useful to identify the issues. Example kubectl get pod vote -o yaml example output snippet when a wrong image was used to create a pod. status: ... containerStatuses: .... state: waiting: message: 'rpc error: code = Unknown desc = Error response from daemon: manifest for schoolofdevops/vote:latst not found' reason: ErrImagePull hostIP: 139.59.232.248","title":"Getting detailed status of an object (pods, deployments)"},{"location":"12_troubleshooting/#checking-the-status-of-deployment","text":"For this example I have a sample deployment called nginx. FILE: nginx-deployment.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 1 template: metadata: labels: app: nginx spec: containers: - name: nginx image: ngnix:latest ports: - containerPort: 80 List the deployment to check for the availability of pods kubectl get deployment nginx NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx 1 1 1 0 20h It is clear that my pod is unavailable. Lets dig further. Check the events of your deployment. kubectl describe deployment nginx List the pods to check for any registry related error kubectl get pods NAME READY STATUS RESTARTS AGE nginx-57c88d7bb8-c6kpc 0/1 ImagePullBackOff 0 7m As we can see, we are not able to pull the image( ImagePullBackOff ). Let's investigate further. kubectl describe pod nginx-57c88d7bb8-c6kpc Check the events of the pod's description. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m default-scheduler Successfully assigned nginx-57c88d7bb8-c6kpc to ip-11-0-1-111.us-west-2.compute.internal Normal SuccessfulMountVolume 9m kubelet, ip-11-0-1-111.us-west-2.compute.internal MountVolume.SetUp succeeded for volume \"default-token-8cwn4\" Normal Pulling 8m (x4 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal pulling image \"ngnix\" Warning Failed 8m (x4 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal Failed to pull image \"ngnix\": rpc error: code = Unknown desc = Error response from daemon: repository ngnix not found: does not exist or no pull access Normal BackOff 7m (x6 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal Back-off pulling image \"ngnix\" Warning FailedSync 4m (x24 over 9m) kubelet, ip-11-0-1-111.us-west-2.compute.internal Error syncing pod Bingo! The name of the image is ngnix instead of nginx . So fix the typo in your deployment file and redo the deployment . Sometimes, your application(pod) may fail to start because of some configuration issues. For those errors, we can follow the logs of the pod. kubectl logs -f nginx-57c88d7bb8-c6kpc If you have any errors it will get populated in your logs.","title":"Checking the status of Deployment"},{"location":"13_redis_statefulset/","text":"Deploying Redis Cluster with StatefulSets What will you learn Statefulsets initContainers Creating a headless service We will use Redis as Statefulsets for our instavote application stack. It is similar to Deployment, but Statefulsets requires a Service Name . Lets being by cleaning up the existing redis installation kubectl delete svc redis kubectl delete deploy redis So we will create a headless service for redis first. A headless service is the one which will be created with no ClusterIP of its own, but will return the actual IP address of its endpoints (e.g. pods), which we would create later in this case. file: dev/redis-sts/redis-svc.yml apiVersion: v1 kind: Service metadata: name: redis labels: app: redis spec: type: ClusterIP ports: - name: redis port: 6379 targetPort: redis clusterIP: None selector: app: redis statefulset.kubernetes.io/pod-name: redis-0 Observe clusterIP value is set to None selector has been updated to send traffic only to the master now apply this and validate cd k8s-code/projects/instavote/dev/redis-sts kubectl apply -f redis-svc.yml kubectl get svc kubectl describe svc redis Adding Redis configurations with ConfigMap Lets now add the redis configuration with configmap. Redis ConfigMap has two keys * master.conf - to provide Redis master configs * slave.conf - to provide Redis slave configs file: redis-cm.yml apiVersion: v1 kind: ConfigMap metadata: name: redis data: master.conf: | bind 0.0.0.0 protected-mode yes port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize no supervised no pidfile /var/run/redis_6379.pid loglevel notice logfile \"\" slave.conf: | slaveof redis-0.redis 6379 apply and validate kubectl apply -f redis-cm.yml kubectl get cm kubectl describe cm redis Using initContainers to configure redis replication We have to deploy redis master/slave set up from one statefulset cluster. This requires two different redis cofigurations , which needs to be described in one Pod template. This complexity can be resolved by using init containers. These init containers copy the appropriate redis configuration by analysing the hostname of the pod. If the Pod's (host)name has 0 as Ordinal number , then it is choosen as the master and master.conf is copied to /etc/ directory. Every other pod will be configured as a replica with slave.conf as configuration. file: redis-sts.yml [...] initContainers: - name: init-redis image: redis:4.0.9 command: - bash - \"-c\" - | set -ex # Generate mysql server-id from pod ordinal index. [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} # Copy appropriate conf.d files from config-map to emptyDir. if [[ $ordinal -eq 0 ]]; then cp /mnt/config-map/master.conf /etc/redis.conf else cp /mnt/config-map/slave.conf /etc/redis.conf fi volumeMounts: - name: conf mountPath: /etc subPath: redis.conf - name: config-map mountPath: /mnt/config-map Deploying Redis Master Slaves with Statefulsets These redis containers are started after initContainers are successfully run and exit. One thing to note here, these containers mount the same volume, conf , from the initContainers which has the proper Redis configuration. file: redis-sts.yaml [...] containers: - name: redis image: redis:4.0.9 command: [\"redis-server\"] args: [\"/etc/redis.conf\"] env: - name: ALLOW_EMPTY_PASSWORD value: \"yes\" ports: - name: redis containerPort: 6379 volumeMounts: - name: redis-data mountPath: /data - name: conf mountPath: /etc/ subPath: redis.conf To apply kubectl apply -f redis-sts.yml Validate the MASTER-SLAVE configuration kubectl exec redis-0 redis-cli ROLE kubectl exec redis-1 redis-cli ROLE redis-0 should have been configured as master, redis-1 as slave. You should also see that redis-1 is been configured as the slave of redis-0.redis as follows, kubectl exec redis-1 redis-cli ROLE slave redis-0.redis 6379 connected 28 This validated the redis master slave configuration. Nano Project: Configuring persistent volume claim per instance of redis Similar to databases, each redis instance needs its own data store, which should also be persistent. Current code that you have applied uses emptyDir as the volume type. This is a problem as emptyDir gets deleted when the pod is deleted, and is not persistent. Your task is to update the YAML file for the statefulset with the following changes, use volumeClaimTemplate instead of volumes for volume redis-data . This will ensure a persistentVolumeClaim per replica/pod. All other volumes remain unchanged. Provide the storageClass as NFS, a provisioner for which is already been configured Size of the volume could be 200Mi as its just a key value store used by the instavote app to store the votes. accessModes should be ReadWriteOnce as the volumes for redis should not be shared between multiple instances. Update the statefulSet, apply and validate that persistentVolumeClaim and persistentVolume are created for each instance/pod of redis application. Reading List Redis Replication Run Replicated Statefulsets Applications Init Containers Search Keywords init containers kubernetes statefulsets redis replication","title":"Lab K206 - Statefulsets"},{"location":"13_redis_statefulset/#deploying-redis-cluster-with-statefulsets","text":"What will you learn Statefulsets initContainers","title":"Deploying Redis Cluster with StatefulSets"},{"location":"13_redis_statefulset/#creating-a-headless-service","text":"We will use Redis as Statefulsets for our instavote application stack. It is similar to Deployment, but Statefulsets requires a Service Name . Lets being by cleaning up the existing redis installation kubectl delete svc redis kubectl delete deploy redis So we will create a headless service for redis first. A headless service is the one which will be created with no ClusterIP of its own, but will return the actual IP address of its endpoints (e.g. pods), which we would create later in this case. file: dev/redis-sts/redis-svc.yml apiVersion: v1 kind: Service metadata: name: redis labels: app: redis spec: type: ClusterIP ports: - name: redis port: 6379 targetPort: redis clusterIP: None selector: app: redis statefulset.kubernetes.io/pod-name: redis-0 Observe clusterIP value is set to None selector has been updated to send traffic only to the master now apply this and validate cd k8s-code/projects/instavote/dev/redis-sts kubectl apply -f redis-svc.yml kubectl get svc kubectl describe svc redis","title":"Creating a headless service"},{"location":"13_redis_statefulset/#adding-redis-configurations-with-configmap","text":"Lets now add the redis configuration with configmap. Redis ConfigMap has two keys * master.conf - to provide Redis master configs * slave.conf - to provide Redis slave configs file: redis-cm.yml apiVersion: v1 kind: ConfigMap metadata: name: redis data: master.conf: | bind 0.0.0.0 protected-mode yes port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize no supervised no pidfile /var/run/redis_6379.pid loglevel notice logfile \"\" slave.conf: | slaveof redis-0.redis 6379 apply and validate kubectl apply -f redis-cm.yml kubectl get cm kubectl describe cm redis","title":"Adding Redis configurations with ConfigMap"},{"location":"13_redis_statefulset/#using-initcontainers-to-configure-redis-replication","text":"We have to deploy redis master/slave set up from one statefulset cluster. This requires two different redis cofigurations , which needs to be described in one Pod template. This complexity can be resolved by using init containers. These init containers copy the appropriate redis configuration by analysing the hostname of the pod. If the Pod's (host)name has 0 as Ordinal number , then it is choosen as the master and master.conf is copied to /etc/ directory. Every other pod will be configured as a replica with slave.conf as configuration. file: redis-sts.yml [...] initContainers: - name: init-redis image: redis:4.0.9 command: - bash - \"-c\" - | set -ex # Generate mysql server-id from pod ordinal index. [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 ordinal=${BASH_REMATCH[1]} # Copy appropriate conf.d files from config-map to emptyDir. if [[ $ordinal -eq 0 ]]; then cp /mnt/config-map/master.conf /etc/redis.conf else cp /mnt/config-map/slave.conf /etc/redis.conf fi volumeMounts: - name: conf mountPath: /etc subPath: redis.conf - name: config-map mountPath: /mnt/config-map","title":"Using initContainers to configure redis replication"},{"location":"13_redis_statefulset/#deploying-redis-master-slaves-with-statefulsets","text":"These redis containers are started after initContainers are successfully run and exit. One thing to note here, these containers mount the same volume, conf , from the initContainers which has the proper Redis configuration. file: redis-sts.yaml [...] containers: - name: redis image: redis:4.0.9 command: [\"redis-server\"] args: [\"/etc/redis.conf\"] env: - name: ALLOW_EMPTY_PASSWORD value: \"yes\" ports: - name: redis containerPort: 6379 volumeMounts: - name: redis-data mountPath: /data - name: conf mountPath: /etc/ subPath: redis.conf To apply kubectl apply -f redis-sts.yml Validate the MASTER-SLAVE configuration kubectl exec redis-0 redis-cli ROLE kubectl exec redis-1 redis-cli ROLE redis-0 should have been configured as master, redis-1 as slave. You should also see that redis-1 is been configured as the slave of redis-0.redis as follows, kubectl exec redis-1 redis-cli ROLE slave redis-0.redis 6379 connected 28 This validated the redis master slave configuration.","title":"Deploying Redis Master Slaves with Statefulsets"},{"location":"13_redis_statefulset/#nano-project-configuring-persistent-volume-claim-per-instance-of-redis","text":"Similar to databases, each redis instance needs its own data store, which should also be persistent. Current code that you have applied uses emptyDir as the volume type. This is a problem as emptyDir gets deleted when the pod is deleted, and is not persistent. Your task is to update the YAML file for the statefulset with the following changes, use volumeClaimTemplate instead of volumes for volume redis-data . This will ensure a persistentVolumeClaim per replica/pod. All other volumes remain unchanged. Provide the storageClass as NFS, a provisioner for which is already been configured Size of the volume could be 200Mi as its just a key value store used by the instavote app to store the votes. accessModes should be ReadWriteOnce as the volumes for redis should not be shared between multiple instances. Update the statefulSet, apply and validate that persistentVolumeClaim and persistentVolume are created for each instance/pod of redis application.","title":"Nano Project: Configuring persistent volume claim per instance of redis"},{"location":"13_redis_statefulset/#reading-list","text":"Redis Replication Run Replicated Statefulsets Applications Init Containers Search Keywords init containers kubernetes statefulsets redis replication","title":"Reading List"},{"location":"9-vote-configmaps_and_secrets/","text":"","title":"9 vote configmaps and secrets"},{"location":"adv-k8s-project/","text":"Mini Project: Deploying Multi Tier Application Stack In this project , you would write definitions for deploying the instavote application stack with all components/tiers which include, vote redis worker db result Project Specs Apply the deployment and service code for the applications marked as ready Complete the code for deployments and services marked as TODO Apply the definitions that you have completed Validate the application workflow is operational by loading vote and result applications from the browser Set up ingress controller and add ingress rules so that you are able to access the apps using domain names e.g. vote.example.com and result.example.com Following table depicts the state of readiness of the above services. App Deployment Service vote TODO ready redis ready ready worker TODO n/a db ready ready result TODO TODO Phase I - Apply existing code Create a namespace and switch to it kubectl create namespace instavote kubectl config set-context --current --namespace=instavote kubectl config get-contexts Apply the existing manifests cd k8s-code/projects/instavote/dev/ kubectl apply -f vote-svc.yaml kubectl apply -f redis-deploy.yaml -f redis-svc.yaml kubectl apply -f db-deploy.yaml -f db-svc.yaml validate kubectl get all Where you should see, deplyoment and services for redis and db created nodeport service for vote app If you see the above objects, proceed with the next task. Phase II - Create Deployments and Services for Remaining Services You may find the files available in the same directory as above i.e. k8s-code/projects/instavote/dev/ with either partial or blank code. Your job is to complete the deployment and service yaml specs and apply those. While writing the specs, you could refer to the following specification. vote image: schoolofdevops/vote:v1 application port: 80 service type: NodePort nodePort : 30000 worker image: schoolofdevops/worker:latest results image: schoolofdevops/vote-result application port: 80 service type: NodePort nodePort : 30100 To Validate: kubectl get all The above command should show, * five deployments and four services created * services for vote and result app should have been exposed with NodePort Find out the NodePort for vote and service apps and load those from your browser. This is how the vote application should look like. You could also load the Result page on the 30100 port of your Node IP/Address as, Above is how the result app should look like. Final validation is, if you submit the vote, you should see the results changing accordingly. You would see this behavior only if all deployments and services are created and configured properly. Phase III - Set up host based traffic routing with Ingress Use the documentation here Nginx Ingress Controller with KIND to set up ingress controller. Launch Nginx Ingress controller as : kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml Check the pod for Nginx Ingress, if its running kubectl get pods -n ingress-nginx You may see the pod in pending state. Check why its pending and fix it. Thats part of your project work. Once you have the ingress controller working, port this ingress rule for vote app and also add one for result app to make it work with nginx ingress controller that you have. Also add the host file configuration as per the same document and validate you are able to use http://vote.example.com/ and http://result.example.com/ to access your services respectively.","title":"XER003 - Deploy Instavote Stack"},{"location":"adv-k8s-project/#mini-project-deploying-multi-tier-application-stack","text":"In this project , you would write definitions for deploying the instavote application stack with all components/tiers which include, vote redis worker db result","title":"Mini Project: Deploying Multi Tier Application Stack"},{"location":"adv-k8s-project/#project-specs","text":"Apply the deployment and service code for the applications marked as ready Complete the code for deployments and services marked as TODO Apply the definitions that you have completed Validate the application workflow is operational by loading vote and result applications from the browser Set up ingress controller and add ingress rules so that you are able to access the apps using domain names e.g. vote.example.com and result.example.com Following table depicts the state of readiness of the above services. App Deployment Service vote TODO ready redis ready ready worker TODO n/a db ready ready result TODO TODO","title":"Project Specs"},{"location":"adv-k8s-project/#phase-i-apply-existing-code","text":"Create a namespace and switch to it kubectl create namespace instavote kubectl config set-context --current --namespace=instavote kubectl config get-contexts Apply the existing manifests cd k8s-code/projects/instavote/dev/ kubectl apply -f vote-svc.yaml kubectl apply -f redis-deploy.yaml -f redis-svc.yaml kubectl apply -f db-deploy.yaml -f db-svc.yaml validate kubectl get all Where you should see, deplyoment and services for redis and db created nodeport service for vote app If you see the above objects, proceed with the next task.","title":"Phase I - Apply existing code"},{"location":"adv-k8s-project/#phase-ii-create-deployments-and-services-for-remaining-services","text":"You may find the files available in the same directory as above i.e. k8s-code/projects/instavote/dev/ with either partial or blank code. Your job is to complete the deployment and service yaml specs and apply those. While writing the specs, you could refer to the following specification. vote image: schoolofdevops/vote:v1 application port: 80 service type: NodePort nodePort : 30000 worker image: schoolofdevops/worker:latest results image: schoolofdevops/vote-result application port: 80 service type: NodePort nodePort : 30100","title":"Phase II - Create Deployments and Services for Remaining Services"},{"location":"adv-k8s-project/#to-validate","text":"kubectl get all The above command should show, * five deployments and four services created * services for vote and result app should have been exposed with NodePort Find out the NodePort for vote and service apps and load those from your browser. This is how the vote application should look like. You could also load the Result page on the 30100 port of your Node IP/Address as, Above is how the result app should look like. Final validation is, if you submit the vote, you should see the results changing accordingly. You would see this behavior only if all deployments and services are created and configured properly.","title":"To Validate:"},{"location":"adv-k8s-project/#phase-iii-set-up-host-based-traffic-routing-with-ingress","text":"Use the documentation here Nginx Ingress Controller with KIND to set up ingress controller. Launch Nginx Ingress controller as : kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml Check the pod for Nginx Ingress, if its running kubectl get pods -n ingress-nginx You may see the pod in pending state. Check why its pending and fix it. Thats part of your project work. Once you have the ingress controller working, port this ingress rule for vote app and also add one for result app to make it work with nginx ingress controller that you have. Also add the host file configuration as per the same document and validate you are able to use http://vote.example.com/ and http://result.example.com/ to access your services respectively.","title":"Phase III - Set up host based traffic routing with Ingress"},{"location":"advanced_deployment/","text":"Deployments Deployments provide reliability and scalability to our application stack. Deployment makes sure that desired number of pods, which is specified declaratively in the deployment file (ex. catalogue-deploy.yml), are always up and running. If a pod fails to run, deployment will remove that pod and replace it with a new one. Deployments vs Replication Controllers Replication Controllers are the older version of Replica Sets. In some ways RC are superior than RS and in some cases it is inferior. But, how does replication work when compared to deployments? Replication Controllers are, * Older method of deploying application * Updates to the application is done with kubectl rolling-update command * Does not support roll back feature like in Deployments. * It has no backends, interacts with pods directly. Deployments are, * The newer method of deploying application. * Provides both rolling update and rollback * Replica Sets are the backend of Deployments. Deployment API Specs A typical deployment file looks like the following, apiVersion: apps/v1beta1 kind: Deployment metadata: name: label: spec: replicas: template: metadata: label: spec: containers: Defining Deployment Strategies We have two types of deployment strategies in Kubernetes, which are, * Rolling Update * Recreate. Rolling Update Let us look at our catalogue deployment file, File: catalogue-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: catalogue namespace: instavote labels: app: catalogue env: dev spec: replicas: 1 template: metadata: labels: app: catalogue env: dev spec: tolerations: - key: \"dedicate\" operator: \"Equal\" value: \"catalogue\" effect: \"NoExecute\" containers: - name: catalogue image: schoolofdevops/catalogue imagePullPolicy: Always ports: - containerPort: 80 We have not defined any deployment strategies here. Let us apply this deployment file. kubectl apply -f catalogue-deploy.yml [output] deployment \"catalogue\" created kubectl get deployment catalogue --export -o yaml [...] strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate As we can see, deployment strategy of rolling update is applied by default . When we use rolling update, * We can avoid down time since old version of application is still receiving traffic. * Only a few pod is updated at a time. Along with the type RollingUpdate, we see two fields. * maxSurge * maxUnavailable MaxSurge The maximum batch size of the available pods will get updated at once. For example, if we have 4 pods running with max surge value of 25%, then one pod will be updated at once. MaxUnavailable As the title implies, how many pods can be unavailable during the update process. Rolling updates are the preferred way to update an application. It is slower than recreate type, but it provides us with zero downtime deployments. When you create a deployment, a replica set for that deployment is also created. kubectl get rs --selector app=catalogue [output] NAME DESIRED CURRENT READY AGE catalogue-6bc67654d5 1 1 1 17m When you do an update, a new replica set is created. But the old replica set is kept for roll back purposes. Let us change the deployment to use a new version. File: catalogue-deploy.yml [...] containers: - name: catalogue image: schoolofdevops/catalogue:v1 imagePullPolicy: Always ports: - containerPort: 80 Apply the changes kubectl apply -f catalogue-deploy.yml kubectl get pods --selector app=catalogue [output] NAME READY STATUS RESTARTS AGE catalogue-6bc67654d5-tbn9h 0/1 ContainerCreating 0 7s catalogue-6c4f7b49d8-bdtgh 1/1 Running 0 8m kubectl get rs --selector app=catalogue [output] NAME DESIRED CURRENT READY AGE catalogue-6bc67654d5 0 0 0 21m catalogue-6c4f7b49d8 1 1 1 1m kubectl rollout status deployment catalogue [output] deployment \"catalogue\" successfully rolled kubectl rollout history deployment catalogue [output] deployments \"catalogue\" REVISION CHANGE-CAUSE 1 2 As we can see, we have two revisions of our deployment. This revisions are used for roll backs about which we will learn later in this chapter. Recreate When the Recreate deployment strategy is used, * The old pods will be deleted * Then the new pods will be created. This will create some downtime in our stack. Hence, this strategy is only recommended for development use cases. Let us change the deployment strategy to recreate and image tag to latest . File: catalogue-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: catalogue namespace: instavote labels: app: catalogue env: dev spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: catalogue env: dev spec: tolerations: - key: \"dedicate\" operator: \"Equal\" value: \"catalogue\" effect: \"NoExecute\" containers: - name: catalogue image: schoolofdevops/catalogue:latest imagePullPolicy: Always ports: - containerPort: 80 Rollbacks As we discussed earlier, the major advantage of using deployments over replication controller is its roll back feature. Let us see how it is done. First check the image being used by the deployment. kubectl describe deployment catalogue [...] Containers: catalogue: Image: schoolofdevops/catalogue:v1 Port: 80/TCP Environment: Mounts: Volumes: The current deployment has the image with the tag v1 . kubectl rollout history deployment catalogue [output] deployments \"catalogue\" REVISION CHANGE-CAUSE 1 2 We have two revisions. Revision 1 is the first ever deployment we have done while revision 2 is the one in which we changed the image tag to v1. Let us rollback to revision 1. Let us rollback to revision 1 which has the image tag of none . kubectl rollout undo deployment catalogue --to-revision=1 [output] deployment \"catalogue\" kubectl describe deployment catalogue [...] Containers: catalogue: Image: schoolofdevops/catalogue Port: 80/TCP Environment: Mounts: Volumes: Pause/Unpause When you are in the middle of a new update for your application and you found out that the application is behaving as intended. In those situations, 1. we can pause the update, 2. fix the issue, 3. resume the update. Let us change the image tag to V2 in pod spec. File: catalogue-deploy.yml containers: - name: catalogue image: schoolofdevops/catalogue:V2 imagePullPolicy: Always ports: - containerPort: 80 Apply the changes. kubectl apply -f catalogue-deploy.yml kubectl get pods [Output] NAME READY STATUS RESTARTS AGE catalogue-6c4f7b49d8-g5dgc 1/1 Running 0 16m catalogue-765554cc7-xsbhs 0/1 ErrImagePull 0 9s Our deployment is failing. From some debugging, we can conclude that we are using a wrong image tag. Now pause the update kubectl rollout pause Set the deployment to use v2 version of the image. kubectl set image deployment catalogue catalogue=schoolofdevops/catalogue:v2 [output] deployment \"catalogue\" image updated Now resume the update kubectl rollout resume deployment catalogue kubectl rollout status deployment catalogue [Ouput] deployment \"catalogue\" successfully rolled out kubectl get pods [Output] NAME READY STATUS RESTARTS AGE catalogue-6875c8df8f-k4hls 1/1 Running 0 1m When we do this, we skip the need of creating a new rolling update altogether. Additional Release Strategies Along with rolling update and recreate strategies, we can also do, Canary releases Blue/Green deployments. Canary Releases Blue/Green Deployments Create two more catalogue images with different products. Recreate type gives an error. Need to be fixed. Add Use-cases for recreate strategy type. Add additional details of why we skip creating a replica set / rolling update in pause/unpause section.","title":"Advanced deployment"},{"location":"advanced_deployment/#deployments","text":"Deployments provide reliability and scalability to our application stack. Deployment makes sure that desired number of pods, which is specified declaratively in the deployment file (ex. catalogue-deploy.yml), are always up and running. If a pod fails to run, deployment will remove that pod and replace it with a new one.","title":"Deployments"},{"location":"advanced_deployment/#deployments-vs-replication-controllers","text":"Replication Controllers are the older version of Replica Sets. In some ways RC are superior than RS and in some cases it is inferior. But, how does replication work when compared to deployments? Replication Controllers are, * Older method of deploying application * Updates to the application is done with kubectl rolling-update command * Does not support roll back feature like in Deployments. * It has no backends, interacts with pods directly. Deployments are, * The newer method of deploying application. * Provides both rolling update and rollback * Replica Sets are the backend of Deployments.","title":"Deployments vs Replication Controllers"},{"location":"advanced_deployment/#deployment-api-specs","text":"A typical deployment file looks like the following, apiVersion: apps/v1beta1 kind: Deployment metadata: name: label: spec: replicas: template: metadata: label: spec: containers: ","title":"Deployment API Specs"},{"location":"advanced_deployment/#defining-deployment-strategies","text":"We have two types of deployment strategies in Kubernetes, which are, * Rolling Update * Recreate.","title":"Defining Deployment Strategies"},{"location":"advanced_deployment/#rolling-update","text":"Let us look at our catalogue deployment file, File: catalogue-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: catalogue namespace: instavote labels: app: catalogue env: dev spec: replicas: 1 template: metadata: labels: app: catalogue env: dev spec: tolerations: - key: \"dedicate\" operator: \"Equal\" value: \"catalogue\" effect: \"NoExecute\" containers: - name: catalogue image: schoolofdevops/catalogue imagePullPolicy: Always ports: - containerPort: 80 We have not defined any deployment strategies here. Let us apply this deployment file. kubectl apply -f catalogue-deploy.yml [output] deployment \"catalogue\" created kubectl get deployment catalogue --export -o yaml [...] strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate As we can see, deployment strategy of rolling update is applied by default . When we use rolling update, * We can avoid down time since old version of application is still receiving traffic. * Only a few pod is updated at a time. Along with the type RollingUpdate, we see two fields. * maxSurge * maxUnavailable","title":"Rolling Update"},{"location":"advanced_deployment/#maxsurge","text":"The maximum batch size of the available pods will get updated at once. For example, if we have 4 pods running with max surge value of 25%, then one pod will be updated at once.","title":"MaxSurge"},{"location":"advanced_deployment/#maxunavailable","text":"As the title implies, how many pods can be unavailable during the update process. Rolling updates are the preferred way to update an application. It is slower than recreate type, but it provides us with zero downtime deployments. When you create a deployment, a replica set for that deployment is also created. kubectl get rs --selector app=catalogue [output] NAME DESIRED CURRENT READY AGE catalogue-6bc67654d5 1 1 1 17m When you do an update, a new replica set is created. But the old replica set is kept for roll back purposes. Let us change the deployment to use a new version. File: catalogue-deploy.yml [...] containers: - name: catalogue image: schoolofdevops/catalogue:v1 imagePullPolicy: Always ports: - containerPort: 80 Apply the changes kubectl apply -f catalogue-deploy.yml kubectl get pods --selector app=catalogue [output] NAME READY STATUS RESTARTS AGE catalogue-6bc67654d5-tbn9h 0/1 ContainerCreating 0 7s catalogue-6c4f7b49d8-bdtgh 1/1 Running 0 8m kubectl get rs --selector app=catalogue [output] NAME DESIRED CURRENT READY AGE catalogue-6bc67654d5 0 0 0 21m catalogue-6c4f7b49d8 1 1 1 1m kubectl rollout status deployment catalogue [output] deployment \"catalogue\" successfully rolled kubectl rollout history deployment catalogue [output] deployments \"catalogue\" REVISION CHANGE-CAUSE 1 2 As we can see, we have two revisions of our deployment. This revisions are used for roll backs about which we will learn later in this chapter.","title":"MaxUnavailable"},{"location":"advanced_deployment/#recreate","text":"When the Recreate deployment strategy is used, * The old pods will be deleted * Then the new pods will be created. This will create some downtime in our stack. Hence, this strategy is only recommended for development use cases. Let us change the deployment strategy to recreate and image tag to latest . File: catalogue-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: catalogue namespace: instavote labels: app: catalogue env: dev spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: catalogue env: dev spec: tolerations: - key: \"dedicate\" operator: \"Equal\" value: \"catalogue\" effect: \"NoExecute\" containers: - name: catalogue image: schoolofdevops/catalogue:latest imagePullPolicy: Always ports: - containerPort: 80","title":"Recreate"},{"location":"advanced_deployment/#rollbacks","text":"As we discussed earlier, the major advantage of using deployments over replication controller is its roll back feature. Let us see how it is done. First check the image being used by the deployment. kubectl describe deployment catalogue [...] Containers: catalogue: Image: schoolofdevops/catalogue:v1 Port: 80/TCP Environment: Mounts: Volumes: The current deployment has the image with the tag v1 . kubectl rollout history deployment catalogue [output] deployments \"catalogue\" REVISION CHANGE-CAUSE 1 2 We have two revisions. Revision 1 is the first ever deployment we have done while revision 2 is the one in which we changed the image tag to v1. Let us rollback to revision 1. Let us rollback to revision 1 which has the image tag of none . kubectl rollout undo deployment catalogue --to-revision=1 [output] deployment \"catalogue\" kubectl describe deployment catalogue [...] Containers: catalogue: Image: schoolofdevops/catalogue Port: 80/TCP Environment: Mounts: Volumes: ","title":"Rollbacks"},{"location":"advanced_deployment/#pauseunpause","text":"When you are in the middle of a new update for your application and you found out that the application is behaving as intended. In those situations, 1. we can pause the update, 2. fix the issue, 3. resume the update. Let us change the image tag to V2 in pod spec. File: catalogue-deploy.yml containers: - name: catalogue image: schoolofdevops/catalogue:V2 imagePullPolicy: Always ports: - containerPort: 80 Apply the changes. kubectl apply -f catalogue-deploy.yml kubectl get pods [Output] NAME READY STATUS RESTARTS AGE catalogue-6c4f7b49d8-g5dgc 1/1 Running 0 16m catalogue-765554cc7-xsbhs 0/1 ErrImagePull 0 9s Our deployment is failing. From some debugging, we can conclude that we are using a wrong image tag. Now pause the update kubectl rollout pause Set the deployment to use v2 version of the image. kubectl set image deployment catalogue catalogue=schoolofdevops/catalogue:v2 [output] deployment \"catalogue\" image updated Now resume the update kubectl rollout resume deployment catalogue kubectl rollout status deployment catalogue [Ouput] deployment \"catalogue\" successfully rolled out kubectl get pods [Output] NAME READY STATUS RESTARTS AGE catalogue-6875c8df8f-k4hls 1/1 Running 0 1m When we do this, we skip the need of creating a new rolling update altogether.","title":"Pause/Unpause"},{"location":"advanced_deployment/#additional-release-strategies","text":"Along with rolling update and recreate strategies, we can also do, Canary releases Blue/Green deployments.","title":"Additional Release Strategies"},{"location":"advanced_deployment/#canary-releases","text":"","title":"Canary Releases"},{"location":"advanced_deployment/#bluegreen-deployments","text":"Create two more catalogue images with different products. Recreate type gives an error. Need to be fixed. Add Use-cases for recreate strategy type. Add additional details of why we skip creating a replica set / rolling update in pause/unpause section.","title":"Blue/Green Deployments"},{"location":"advanced_pod_design/","text":"","title":"Lab K201 - Advanced Pod Design"},{"location":"advanced_pod_scheduling/","text":"Lab K203 - Advanced Pod Scheduling In the Kubernetes bootcamp training, we have seen how to create a pod and and some basic pod configurations to go with it. But this chapter explains some advanced topics related to pod scheduling. From the api document for version 1.11 following are the pod specs which are relevant from scheduling perspective. nodeSelector nodeName affinity schedulerName tolerations nodeName You could bind a pod to a specific node with a name using nodeName spec. Lets take an example where you want to run the deployment for result service on a specific node. Lets look at how you would do it, Begin by listing the nodes kubectl get nodes [sample output] NAME STATUS ROLES AGE VERSION kind-control-plane Ready control-plane 35h v1.29.2 kind-worker Ready 35h v1.29.2 kind-worker2 Ready 35h v1.29.2 Now, bind your pod to one node e.g. kind-worker by modyfying the deployment spec as: File : result-deploy.yaml apiVersion: apps/v1 kind: Deployment .... ..... spec: containers: - image: schoolofdevops/vote-result name: vote-result nodeName: kind-worker apply and validate kubectl apply -f result-deploy.yaml kubectl get pods -o wide nodeSelector Using nodeSelector instead of directly specifying nodeName in Kubernetes offers greater flexibility and resilience in scheduling pods. While nodeName forces a pod to schedule on a specific node, effectively bypassing Kubernetes' scheduler, nodeSelector allows for more dynamic placement by specifying a set of criteria that nodes must meet for the pod to be scheduled there. This approach utilizes Kubernetes' intelligent scheduling capabilities, enabling the system to consider multiple suitable nodes that meet the specified labels. This not only improves fault tolerance by avoiding dependencies on a single node but also facilitates more efficient resource utilization across the cluster. Additionally, nodeSelector supports scenarios where the environment might change, such as when nodes are added or removed, or their labels are updated, ensuring that the pods can still be scheduled according to the current state of the cluster. To use nodeSelector, begin by labeling your nodes as: kubectl get nodes --show-labels kubectl label nodes zone=aaa kubectl get nodes --show-labels e.g. kubectl label nodes kind-worker zone=aaa kubectl label nodes kind-worker2 zone=bbb kubectl get nodes --show-labels where, replace kind-worker and kind-worker2 can be the the actual nodes in your cluster. Now update one of the deployments and add the nodeSelector spec to the pod e.g. File : result-deploy.yaml spec: containers: - image: schoolofdevops/vote-result name: vote-result nodeSelector: zone: bbb Note: ensure you have removed nodeName if present. apply and validate kubectl apply -f result-deploy.yaml kubectl get pods -o wide You shall see the pod being recreated now on the node matching the label selected using nodeSelector. Affinity and Anti-Affinity We have discussed about scheduling a pod on a particular node using NodeSelector . Using affinity and anti-affinity in Kubernetes offers a more sophisticated and granular level of control compared to nodeSelector, enabling not just simple label matching but also complex rules that govern pod placement. Affinity rules allow you to specify preferences that attract pods to certain nodes, either based on the node's properties or other pods that are already running on those nodes. Conversely, anti-affinity rules are used to ensure pods are spread across different nodes or node groups, enhancing high availability and reducing the risk of simultaneous failures. This is particularly useful in large-scale deployments where maintaining workload balance and resilience is crucial. For example, you can ensure that multiple instances of a service run in different racks or availability zones, minimizing potential impact from physical infrastructure failures. These features allow Kubernetes to more effectively manage the distribution and redundancy of workloads, which is vital for maintaining robust, scalable applications. More over, using nodeSelector wolud mean defining a strict condition which must be met. If the condition is not met, the pod cannot be scheduled. Node/Pod affinity and anti-affinity solves this issue by introducing soft and hard conditions which are flexible based on when they are applied. This is controlled using the following properties required preferred DuringScheduling DuringExecution and using theese operators In NotIn Exists DoesNotExist Gt Lt Lets take up some examples and understand this. nodeAffinity Examine the current pod distribution kubectl get pods -o wide --selector=\"role=vote\" and node labels kubectl get nodes --show-labels Lets create node affinity criteria as Pods for vote app must not run on the master nodes Pods for vote app preferably run on a node in zone bbb First is a hard affinity versus second being soft affinity. file: vote-deploy-nodeaffinity.yaml .... template: .... spec: containers: - name: app image: schoolofdevops/vote:v1 ports: - containerPort: 80 protocol: TCP affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: zone operator: In values: - bbb clearn up previous deployment and apply this code as kubectl delete deploy vote kubectl apply -f vote-deploy-nodeaffinity.yaml kubectl get pods -o wide podAffinity and podAntiAffinity Lets define pod affinity criteria as, Pods for vote and redis should be co located as much as possible (preferred) No two pods with redis app should be running on the same node (required) kubectl get pods -o wide --selector=\"role in (vote,redis)\" file: vote-deploy-podaffinity.yaml ... template: ... spec: containers: - name: app image: schoolofdevops/vote:v1 ports: - containerPort: 80 protocol: TCP affinity: ... podAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 podAffinityTerm: labelSelector: matchExpressions: - key: role operator: In values: - redis topologyKey: kubernetes.io/hostname file: redis-deploy-podaffinity.yaml .... template: ... spec: containers: - image: schoolofdevops/redis:latest imagePullPolicy: Always name: redis ports: - containerPort: 6379 protocol: TCP restartPolicy: Always affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: role operator: In values: - redis topologyKey: \"kubernetes.io/hostname\" clean up the previous deployments and apply as kubectl delete deploy vote kubectl delete deploy,sts redis kubectl apply -f redis-deploy-podaffinity.yaml kubectl apply -f vote-deploy-podaffinity.yaml check the pods distribution kubectl get pods -o wide --selector=\"role in (vote,redis)\" Observations from the above output, Since redis has a hard constraint not to be on the same node, you would observe redis pods being on differnt nodes (node2 and node4) since vote app has a soft constraint, you see some of the pods running on node4 (same node running redis), others continue to run on node 3 If you kill the pods on node3, at the time of scheduling new ones, scheduler meets all affinity rules Now try scaling up redis instances kubectl scale deploy/redis --replicas=4 kubectl get pods -o wide Are all redis pods runnning ? Why? When you are done experimenting, revert to original configurations kubectl delete deploy vote kubectl delete deploy redis kubectl apply -f vote-deploy.yaml -f redis-deploy.yaml Taints and Tolerations Affinity is defined for pods Taints are defined for nodes You could add the taints with criteria and effects. Effetcs can be Taint Specs : effect NoSchedule PreferNoSchedule NoExecute key value timeAdded (only written for NoExecute taints) Observe the pods distribution kubectl get pods -o wide Lets taint a node. kubectl taint node kind-worker2 dedicated=worker:NoExecute kubectl describe node kind-worker2 after tainting the node kubectl get pods -o wide All pods running on node2 just got evicted. Add toleration in the Deployment for worker. File: worker-deploy.yml apiVersion: apps/v1 ..... template: .... spec: containers: - name: app image: schoolofdevops/vote-worker:latest tolerations: - key: \"dedicated\" operator: \"Equal\" value: \"worker\" effect: \"NoExecute\" apply kubectl apply -f worker-deploy.yml Observe the pod distribution now. $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE db-66496667c9-qggzd 1/1 Running 0 4h 10.233.74.74 node4 redis-5bf748dbcf-ckn65 1/1 Running 0 3m 10.233.71.26 node3 redis-5bf748dbcf-vxppx 1/1 Running 0 31m 10.233.74.79 node4 result-5c7569bcb7-4fptr 1/1 Running 0 4h 10.233.71.18 node3 result-5c7569bcb7-s4rdx 1/1 Running 0 4h 10.233.74.75 node4 vote-56bf599b9c-22lpw 1/1 Running 0 30m 10.233.74.80 node4 vote-56bf599b9c-4l6bc 1/1 Running 0 12m 10.233.74.83 node4 vote-56bf599b9c-bqsrq 1/1 Running 0 12m 10.233.74.82 node4 vote-56bf599b9c-xw7zc 1/1 Running 0 12m 10.233.74.81 node4 worker-6cc8dbd4f8-6bkfg 1/1 Running 0 1m 10.233.75.15 node2 You should see worker being scheduled on kind-worker2 To remove the taint created above kubectl taint node kind-worker2 dedicated=worker:NoExecute- Exercise Master node is unschedulable because of a taint. Find the taint on the master node and remove it. See if new pods get scheduled on it after that.","title":"Lab K303 - Advanced Pod Scheduling"},{"location":"advanced_pod_scheduling/#lab-k203-advanced-pod-scheduling","text":"In the Kubernetes bootcamp training, we have seen how to create a pod and and some basic pod configurations to go with it. But this chapter explains some advanced topics related to pod scheduling. From the api document for version 1.11 following are the pod specs which are relevant from scheduling perspective. nodeSelector nodeName affinity schedulerName tolerations","title":"Lab K203 - Advanced Pod Scheduling"},{"location":"advanced_pod_scheduling/#nodename","text":"You could bind a pod to a specific node with a name using nodeName spec. Lets take an example where you want to run the deployment for result service on a specific node. Lets look at how you would do it, Begin by listing the nodes kubectl get nodes [sample output] NAME STATUS ROLES AGE VERSION kind-control-plane Ready control-plane 35h v1.29.2 kind-worker Ready 35h v1.29.2 kind-worker2 Ready 35h v1.29.2 Now, bind your pod to one node e.g. kind-worker by modyfying the deployment spec as: File : result-deploy.yaml apiVersion: apps/v1 kind: Deployment .... ..... spec: containers: - image: schoolofdevops/vote-result name: vote-result nodeName: kind-worker apply and validate kubectl apply -f result-deploy.yaml kubectl get pods -o wide","title":"nodeName"},{"location":"advanced_pod_scheduling/#nodeselector","text":"Using nodeSelector instead of directly specifying nodeName in Kubernetes offers greater flexibility and resilience in scheduling pods. While nodeName forces a pod to schedule on a specific node, effectively bypassing Kubernetes' scheduler, nodeSelector allows for more dynamic placement by specifying a set of criteria that nodes must meet for the pod to be scheduled there. This approach utilizes Kubernetes' intelligent scheduling capabilities, enabling the system to consider multiple suitable nodes that meet the specified labels. This not only improves fault tolerance by avoiding dependencies on a single node but also facilitates more efficient resource utilization across the cluster. Additionally, nodeSelector supports scenarios where the environment might change, such as when nodes are added or removed, or their labels are updated, ensuring that the pods can still be scheduled according to the current state of the cluster. To use nodeSelector, begin by labeling your nodes as: kubectl get nodes --show-labels kubectl label nodes zone=aaa kubectl get nodes --show-labels e.g. kubectl label nodes kind-worker zone=aaa kubectl label nodes kind-worker2 zone=bbb kubectl get nodes --show-labels where, replace kind-worker and kind-worker2 can be the the actual nodes in your cluster. Now update one of the deployments and add the nodeSelector spec to the pod e.g. File : result-deploy.yaml spec: containers: - image: schoolofdevops/vote-result name: vote-result nodeSelector: zone: bbb Note: ensure you have removed nodeName if present. apply and validate kubectl apply -f result-deploy.yaml kubectl get pods -o wide You shall see the pod being recreated now on the node matching the label selected using nodeSelector.","title":"nodeSelector"},{"location":"advanced_pod_scheduling/#affinity-and-anti-affinity","text":"We have discussed about scheduling a pod on a particular node using NodeSelector . Using affinity and anti-affinity in Kubernetes offers a more sophisticated and granular level of control compared to nodeSelector, enabling not just simple label matching but also complex rules that govern pod placement. Affinity rules allow you to specify preferences that attract pods to certain nodes, either based on the node's properties or other pods that are already running on those nodes. Conversely, anti-affinity rules are used to ensure pods are spread across different nodes or node groups, enhancing high availability and reducing the risk of simultaneous failures. This is particularly useful in large-scale deployments where maintaining workload balance and resilience is crucial. For example, you can ensure that multiple instances of a service run in different racks or availability zones, minimizing potential impact from physical infrastructure failures. These features allow Kubernetes to more effectively manage the distribution and redundancy of workloads, which is vital for maintaining robust, scalable applications. More over, using nodeSelector wolud mean defining a strict condition which must be met. If the condition is not met, the pod cannot be scheduled. Node/Pod affinity and anti-affinity solves this issue by introducing soft and hard conditions which are flexible based on when they are applied. This is controlled using the following properties required preferred DuringScheduling DuringExecution and using theese operators In NotIn Exists DoesNotExist Gt Lt Lets take up some examples and understand this.","title":"Affinity and Anti-Affinity"},{"location":"advanced_pod_scheduling/#nodeaffinity","text":"Examine the current pod distribution kubectl get pods -o wide --selector=\"role=vote\" and node labels kubectl get nodes --show-labels Lets create node affinity criteria as Pods for vote app must not run on the master nodes Pods for vote app preferably run on a node in zone bbb First is a hard affinity versus second being soft affinity. file: vote-deploy-nodeaffinity.yaml .... template: .... spec: containers: - name: app image: schoolofdevops/vote:v1 ports: - containerPort: 80 protocol: TCP affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: zone operator: In values: - bbb clearn up previous deployment and apply this code as kubectl delete deploy vote kubectl apply -f vote-deploy-nodeaffinity.yaml kubectl get pods -o wide","title":"nodeAffinity"},{"location":"advanced_pod_scheduling/#podaffinity-and-podantiaffinity","text":"Lets define pod affinity criteria as, Pods for vote and redis should be co located as much as possible (preferred) No two pods with redis app should be running on the same node (required) kubectl get pods -o wide --selector=\"role in (vote,redis)\" file: vote-deploy-podaffinity.yaml ... template: ... spec: containers: - name: app image: schoolofdevops/vote:v1 ports: - containerPort: 80 protocol: TCP affinity: ... podAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 podAffinityTerm: labelSelector: matchExpressions: - key: role operator: In values: - redis topologyKey: kubernetes.io/hostname file: redis-deploy-podaffinity.yaml .... template: ... spec: containers: - image: schoolofdevops/redis:latest imagePullPolicy: Always name: redis ports: - containerPort: 6379 protocol: TCP restartPolicy: Always affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: role operator: In values: - redis topologyKey: \"kubernetes.io/hostname\" clean up the previous deployments and apply as kubectl delete deploy vote kubectl delete deploy,sts redis kubectl apply -f redis-deploy-podaffinity.yaml kubectl apply -f vote-deploy-podaffinity.yaml check the pods distribution kubectl get pods -o wide --selector=\"role in (vote,redis)\" Observations from the above output, Since redis has a hard constraint not to be on the same node, you would observe redis pods being on differnt nodes (node2 and node4) since vote app has a soft constraint, you see some of the pods running on node4 (same node running redis), others continue to run on node 3 If you kill the pods on node3, at the time of scheduling new ones, scheduler meets all affinity rules Now try scaling up redis instances kubectl scale deploy/redis --replicas=4 kubectl get pods -o wide Are all redis pods runnning ? Why? When you are done experimenting, revert to original configurations kubectl delete deploy vote kubectl delete deploy redis kubectl apply -f vote-deploy.yaml -f redis-deploy.yaml","title":"podAffinity and podAntiAffinity"},{"location":"advanced_pod_scheduling/#taints-and-tolerations","text":"Affinity is defined for pods Taints are defined for nodes You could add the taints with criteria and effects. Effetcs can be Taint Specs : effect NoSchedule PreferNoSchedule NoExecute key value timeAdded (only written for NoExecute taints) Observe the pods distribution kubectl get pods -o wide Lets taint a node. kubectl taint node kind-worker2 dedicated=worker:NoExecute kubectl describe node kind-worker2 after tainting the node kubectl get pods -o wide All pods running on node2 just got evicted. Add toleration in the Deployment for worker. File: worker-deploy.yml apiVersion: apps/v1 ..... template: .... spec: containers: - name: app image: schoolofdevops/vote-worker:latest tolerations: - key: \"dedicated\" operator: \"Equal\" value: \"worker\" effect: \"NoExecute\" apply kubectl apply -f worker-deploy.yml Observe the pod distribution now. $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE db-66496667c9-qggzd 1/1 Running 0 4h 10.233.74.74 node4 redis-5bf748dbcf-ckn65 1/1 Running 0 3m 10.233.71.26 node3 redis-5bf748dbcf-vxppx 1/1 Running 0 31m 10.233.74.79 node4 result-5c7569bcb7-4fptr 1/1 Running 0 4h 10.233.71.18 node3 result-5c7569bcb7-s4rdx 1/1 Running 0 4h 10.233.74.75 node4 vote-56bf599b9c-22lpw 1/1 Running 0 30m 10.233.74.80 node4 vote-56bf599b9c-4l6bc 1/1 Running 0 12m 10.233.74.83 node4 vote-56bf599b9c-bqsrq 1/1 Running 0 12m 10.233.74.82 node4 vote-56bf599b9c-xw7zc 1/1 Running 0 12m 10.233.74.81 node4 worker-6cc8dbd4f8-6bkfg 1/1 Running 0 1m 10.233.75.15 node2 You should see worker being scheduled on kind-worker2 To remove the taint created above kubectl taint node kind-worker2 dedicated=worker:NoExecute-","title":"Taints and Tolerations"},{"location":"advanced_pod_scheduling/#exercise","text":"Master node is unschedulable because of a taint. Find the taint on the master node and remove it. See if new pods get scheduled on it after that.","title":"Exercise"},{"location":"advanced_services/","text":"Services Services are a way to expose your backend pods to outside world. Services can also be used for internal networks. Service Discovery Whenever we create a service, the kubernetes api server will create a route for that service within the cluster. So you will be able to access the service using the dns entry for that service. Usually the service dns will look like Let us create a service for frontend File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 kubectl apply -f frontend-svc.yml kubectl exec -it front-end-5cbc986f44-dqws2 sh nslookup front-end.instavote Name: front-end.instavote Address 1: 10.233.6.236 front-end.instavote.svc.cluster.local This is achieved by the cluster dns add-on . This is how backend of service A consumes service B. Labels and Selectors When we create a service, it automatically adds appropriate pods to its backend. How does this happen? This is achieved by a mechanism called labels and selectors . We label our pods with keys and values. Our service template has selectors with these keys and values. So when a service is created, it will check for pods running with the same label. If it finds a pod with that label, the pod will be added to endpoint of the service. kubectl describe svc front-end Name: front-end Namespace: instavote Labels: app=front-end env=dev Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"front-end\",\"env\":\"dev\"},\"name\":\"front-end\",\"namespace\":\"instavote\"},\"... Selector: app=front-end Type: ClusterIP IP: 10.233.6.236 Port: 8079/TCP TargetPort: 8079/TCP Endpoints: 10.233.71.13:8079 Session Affinity: None Events: Since we have only one pod running with the label app=front-end , it is added to the service endpoint list. Picking up the right type of a service We have four type of services to choose from. Each has a unique use case for which it can be used. Let us see about the types of services below. Types: ClusterIp Internal network. Can not accessed from outside world. Best suited for backend services which should not be exposed (ex: DB). NodePort Can be accessed from outside world. NodePort is best suited for development environment(debugging). Not suited for production use. LoadBalancer Normal cloud load balancer. Exposes service to outside world. ExternalName Used for entities which are outside the cluster. CNAME record for an external service. No ports are defined. We have one more service configuration, which is not exactly a service type. That is ., * ExternalIP Cluster IP and DNS ClusterIP is internal to the cluster. ClusterIP is provided by default. If you do not define the type of the service, then the service is configured with a clusterIP. A typical service type of clusterIP looks like, File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: ClusterIP NodePort Exposes a port on node within 30000 to 32767 range. If a service is defined as nodePort, we can access the service on . Let us change the frontend service type to NodePort and see what happens. File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: NodePort Lets get the node port created for the front-end service. kubectl get svc front-end Visit this node port on any of node's IP. In my case, ExternalIP ExternalIP is not exactly a service type. It can be coupled with any other service type. For example a service with NodePort can be used with ExternalIP as well. Only cache here is, the external IP has to one of the node/master's IP. It is used to route the traffic to one or more cluster nodes. File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: NodePort externalIPs: - 10.40.1.26 - 10.40.1.11 Lets check how externalIPs works, kubectl describe svc front-end curl 10.40.1.26 curl 10.40.1.11 LoadBalancer Service type LoadBalancer is only supported on AWS,GCE,Azure and Openstack. If you have you cluster running on any of these clouds, give it a try. It will create a load balancer for us with a ephemeral IP. We can also specify a loadBalancerIP. Mostly not recommended to use. Using service type LoadBalancer will rise your cloud provider spendings. Think about launching 10 load balancers for 10 different services. Thats where ingress come into picture (explained in the later part). apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: LoadBalancer Headless services with Endpoints/External Names Sometimes you may to decouple the services from backend pods. Sometimes you may want to use some external services outside the cluster and want to integrate those external services with the cluster. For these kind of use cases we can use Headless services . Let us see an example. Let us consider a scenario where you have your redis cluster hosted outside the cluster. Your backend in cluster need to use this redis cluster. In this case, you will create a service with ExternalName, which is nothing but a CNAME record of your redis cluster. This type of services will neither have any ports defined nor have any selectors. But, how do you make sure the backend pods uses this service? For that, you will create a custom endpoint, in which map your service to specific endpoints. Ex: External Name service kind: Service apiVersion: v1 metadata: name: redis-aws namespace: instavote spec: type: ExternalName externalName: redis.aws.schoolofdevops.com Ex: Endpoints kind: Endpoints apiVersion: v1 metadata: name: redis-aws subsets: - addresses: - ip: 10.40.30.123 - ip: 10.40.30.324 - ip: 10.40.30.478 ports: - port: 9376 These IPs have to be manually managed by the cluster administrator. Ingress Ingress controller is an Kubernetes object that manages external access to the backend services. This is the preferred way of exposing your services. Ingress Controller To use an ingress resource, an ingress controller (ex. nginx, traefik) must be running. This controller has to be deployed manually. To create an ingress controller in our cluster, run the following commands, kubectl apply -f traefik-rbac.yaml kubectl apply -f traefik-deployment.yaml Check whether the ingress controller is running or not. kubectl get pods -n kube-system [...] kubernetes-dashboard-75c5ff6844-r9sk8 1/1 Running 6 3d nginx-proxy-node3 1/1 Running 8 27d nginx-proxy-node4 1/1 Running 7 27d traefik-ingress-controller-568dd77566-vk72b 1/1 Running 0 5m Purpose of an Ingress Ingress can be used for, * Load balancing, * SSL Termination, * Name based virtual hosting. Types of Ingress Let us see about different implementation of ingress. Keep or delete the following(Need to come up with use cases). ### Single Service Ingress ### Simple fanout Ingress ### Name based virtual hosting ### TLS ### Load Balancer Services Vs Ingress Service type LoadBalancer Service type Nodeport Creating service resources for applications and backing services Need a demo set up for types: load balancer and external name Need input on this: Creating service resources for applications and backing services, Service API specs, Traffic Policy Can add some points about scaling of ingress controller (running it as a daemonset on all the nodes)","title":"Services"},{"location":"advanced_services/#services","text":"Services are a way to expose your backend pods to outside world. Services can also be used for internal networks.","title":"Services"},{"location":"advanced_services/#service-discovery","text":"Whenever we create a service, the kubernetes api server will create a route for that service within the cluster. So you will be able to access the service using the dns entry for that service. Usually the service dns will look like Let us create a service for frontend File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 kubectl apply -f frontend-svc.yml kubectl exec -it front-end-5cbc986f44-dqws2 sh nslookup front-end.instavote Name: front-end.instavote Address 1: 10.233.6.236 front-end.instavote.svc.cluster.local This is achieved by the cluster dns add-on . This is how backend of service A consumes service B.","title":"Service Discovery"},{"location":"advanced_services/#labels-and-selectors","text":"When we create a service, it automatically adds appropriate pods to its backend. How does this happen? This is achieved by a mechanism called labels and selectors . We label our pods with keys and values. Our service template has selectors with these keys and values. So when a service is created, it will check for pods running with the same label. If it finds a pod with that label, the pod will be added to endpoint of the service. kubectl describe svc front-end Name: front-end Namespace: instavote Labels: app=front-end env=dev Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"front-end\",\"env\":\"dev\"},\"name\":\"front-end\",\"namespace\":\"instavote\"},\"... Selector: app=front-end Type: ClusterIP IP: 10.233.6.236 Port: 8079/TCP TargetPort: 8079/TCP Endpoints: 10.233.71.13:8079 Session Affinity: None Events: Since we have only one pod running with the label app=front-end , it is added to the service endpoint list.","title":"Labels and Selectors"},{"location":"advanced_services/#picking-up-the-right-type-of-a-service","text":"We have four type of services to choose from. Each has a unique use case for which it can be used. Let us see about the types of services below.","title":"Picking up the right type of a service"},{"location":"advanced_services/#types","text":"ClusterIp Internal network. Can not accessed from outside world. Best suited for backend services which should not be exposed (ex: DB). NodePort Can be accessed from outside world. NodePort is best suited for development environment(debugging). Not suited for production use. LoadBalancer Normal cloud load balancer. Exposes service to outside world. ExternalName Used for entities which are outside the cluster. CNAME record for an external service. No ports are defined. We have one more service configuration, which is not exactly a service type. That is ., * ExternalIP","title":"Types:"},{"location":"advanced_services/#cluster-ip-and-dns","text":"ClusterIP is internal to the cluster. ClusterIP is provided by default. If you do not define the type of the service, then the service is configured with a clusterIP. A typical service type of clusterIP looks like, File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: ClusterIP","title":"Cluster IP and DNS"},{"location":"advanced_services/#nodeport","text":"Exposes a port on node within 30000 to 32767 range. If a service is defined as nodePort, we can access the service on . Let us change the frontend service type to NodePort and see what happens. File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: NodePort Lets get the node port created for the front-end service. kubectl get svc front-end Visit this node port on any of node's IP. In my case,","title":"NodePort"},{"location":"advanced_services/#externalip","text":"ExternalIP is not exactly a service type. It can be coupled with any other service type. For example a service with NodePort can be used with ExternalIP as well. Only cache here is, the external IP has to one of the node/master's IP. It is used to route the traffic to one or more cluster nodes. File: frontend-svc.yml apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: NodePort externalIPs: - 10.40.1.26 - 10.40.1.11 Lets check how externalIPs works, kubectl describe svc front-end curl 10.40.1.26 curl 10.40.1.11","title":"ExternalIP"},{"location":"advanced_services/#loadbalancer","text":"Service type LoadBalancer is only supported on AWS,GCE,Azure and Openstack. If you have you cluster running on any of these clouds, give it a try. It will create a load balancer for us with a ephemeral IP. We can also specify a loadBalancerIP. Mostly not recommended to use. Using service type LoadBalancer will rise your cloud provider spendings. Think about launching 10 load balancers for 10 different services. Thats where ingress come into picture (explained in the later part). apiVersion: v1 kind: Service metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: selector: app: front-end ports: - protocol: TCP port: 8079 type: LoadBalancer","title":"LoadBalancer"},{"location":"advanced_services/#headless-services-with-endpointsexternal-names","text":"Sometimes you may to decouple the services from backend pods. Sometimes you may want to use some external services outside the cluster and want to integrate those external services with the cluster. For these kind of use cases we can use Headless services . Let us see an example. Let us consider a scenario where you have your redis cluster hosted outside the cluster. Your backend in cluster need to use this redis cluster. In this case, you will create a service with ExternalName, which is nothing but a CNAME record of your redis cluster. This type of services will neither have any ports defined nor have any selectors. But, how do you make sure the backend pods uses this service? For that, you will create a custom endpoint, in which map your service to specific endpoints. Ex: External Name service kind: Service apiVersion: v1 metadata: name: redis-aws namespace: instavote spec: type: ExternalName externalName: redis.aws.schoolofdevops.com Ex: Endpoints kind: Endpoints apiVersion: v1 metadata: name: redis-aws subsets: - addresses: - ip: 10.40.30.123 - ip: 10.40.30.324 - ip: 10.40.30.478 ports: - port: 9376 These IPs have to be manually managed by the cluster administrator.","title":"Headless services with Endpoints/External Names"},{"location":"advanced_services/#ingress","text":"Ingress controller is an Kubernetes object that manages external access to the backend services. This is the preferred way of exposing your services.","title":"Ingress"},{"location":"advanced_services/#ingress-controller","text":"To use an ingress resource, an ingress controller (ex. nginx, traefik) must be running. This controller has to be deployed manually. To create an ingress controller in our cluster, run the following commands, kubectl apply -f traefik-rbac.yaml kubectl apply -f traefik-deployment.yaml Check whether the ingress controller is running or not. kubectl get pods -n kube-system [...] kubernetes-dashboard-75c5ff6844-r9sk8 1/1 Running 6 3d nginx-proxy-node3 1/1 Running 8 27d nginx-proxy-node4 1/1 Running 7 27d traefik-ingress-controller-568dd77566-vk72b 1/1 Running 0 5m","title":"Ingress Controller"},{"location":"advanced_services/#purpose-of-an-ingress","text":"Ingress can be used for, * Load balancing, * SSL Termination, * Name based virtual hosting.","title":"Purpose of an Ingress"},{"location":"advanced_services/#types-of-ingress","text":"Let us see about different implementation of ingress. Keep or delete the following(Need to come up with use cases). ### Single Service Ingress ### Simple fanout Ingress ### Name based virtual hosting ### TLS ### Load Balancer","title":"Types of Ingress"},{"location":"advanced_services/#services-vs-ingress","text":"Service type LoadBalancer Service type Nodeport","title":"Services Vs Ingress"},{"location":"advanced_services/#creating-service-resources-for-applications-and-backing-services","text":"Need a demo set up for types: load balancer and external name Need input on this: Creating service resources for applications and backing services, Service API specs, Traffic Policy Can add some points about scaling of ingress controller (running it as a daemonset on all the nodes)","title":"Creating service resources for applications and backing services"},{"location":"argo_events/","text":"Argo Events Author: Gourav Shah Publisher: School of Devops Version : v2024.06.02.01 Project: Trigger CI Pipeline on GitHub Changes Set up Argo Events Launch Argo Workflow Environment with Killercoda , set up server UI and have it open. Install Argo Events kubectl create namespace argo-events kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml # Install with a validating admission controller kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install-validating-webhook.yaml kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml Create RBAC Policies kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/sensor-rbac.yaml kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/workflow-rbac.yaml Setup Components to Trigger CI Pipeline You will now set up the components required for the Argo Events to trigger the CI workflow based on changes to GitHub. For this you are going to need Event Source - To listen to GitHub change events Sensor - Which activates on updated to event source and triggers the workflow Workflow Template - To create an instance of the CI Pipeline Workflow Container Registry Credentials - Used by the workflow to publish images with Create an Argo Events EventSource and Sensor to handle the events sent by your polling job. File: webook-eventsource.yaml apiVersion: argoproj.io/v1alpha1 kind: EventSource metadata: name: webhook namespace: argo-events spec: service: ports: - port: 12000 targetPort: 12000 webhook: example: port: \"12000\" endpoint: /example method: POST github: port: \"12000\" endpoint: /github method: POST apply kubectl apply -f webook-eventsource.yaml Check the Argo Workflow for Vote CI Pipeline converted into a Workflow Template . Create workflow template using this code as kubectl apply -f https://gist.githubusercontent.com/initcron/c1704b560909f424e66062d86af9ff5c/raw/f7c5f73605a732d358a93854bc2da652113de494/vote-ci-template.yaml validate kubectl get workflowtemplate -A argo template list -A add registry credentials to argo-events namespace again with kubectl create secret -n argo-events docker-registry docker-registry-creds \\ --docker-server=https://index.docker.io/v1/ \\ --docker-username=xxxx --docker-password=yyyy where replace, xxxx with registry username yyyy with registry access token Add sensor which will listen to updates to github webhook created with event source and then trigger the argo workflow as, File : sensor.yaml apiVersion: argoproj.io/v1alpha1 kind: Sensor metadata: name: polling-sensor namespace: argo-events spec: template: serviceAccountName: operate-workflow-sa dependencies: - name: poll-github eventSourceName: webhook eventName: github triggers: - template: name: launch-vote-ci argoWorkflow: operation: submit source: resource: apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: ci-pipeline- spec: workflowTemplateRef: name: vote-ci-template arguments: parameters: - name: repo-url value: \"https://github.com/xxxxxx/vote.git\" - name: branch value: \"main\" - name: image value: \"xxxxxx/vote\" - name: dockerfile value: \"Dockerfile\" where, replace repo-url with your code repository url update image tag by replacing xxxxxx with your docker hub user name apply kubectl apply -f sensor.yaml validate kubectl get pods -n argo-events you could also check the logs for sensor controller (useful for troubleshooting) using kubectl logs -n argo-events -l \"controller=sensor-controller\" you should see a new pod launched to run this sensor. At this time, if you have Argo Workflow set up, you should see a event flow such as this Event Flow : Deploy GitHub Poller After setting up the event flow, you also need to set up something which will trigger the event source on changes to GitHub. You could do this in two ways Using Webhooks : You could expose the event source service to outside and let GitHub trigger a webhook whenever there is a push event. This is useful if your event source can be publically available (GitHub can connect to it). In-cluster Polling : You could alternately set up in cluster system to periodically poll GitHub for changes, and trigger the event source. This is useful when you can not expose event source service pubically, and are running your cluster in a private network. Since we do not assume your cluster is public, we will employ the second approach. To achisve in-cluster polling, create the following Kubernetes CronJob that periodically polls GitHub for new commits. If new commits are found, the job can trigger the event source webhook created earlier. File: poller-cronjob.yaml --- apiVersion: batch/v1 kind: CronJob metadata: name: github-polling-job spec: schedule: \"* * * * *\" # Poll every minute jobTemplate: spec: template: spec: containers: - name: poller image: schoolofdevops/github-poller:latest env: - name: GITHUB_API_URL value: \"https://api.github.com/repos/yourusername/yourrepo/commits\" - name: GITHUB_TOKEN valueFrom: secretKeyRef: name: github-token-secret key: token - name: LAST_COMMIT_FILE value: \"/data/last_commit.txt\" - name: ARGO_EVENT_SOURCE_URL value: \"http://webhook-eventsource-svc.argo-events.svc.cluster.local:12000/github\" volumeMounts: - name: commit-storage mountPath: /data restartPolicy: OnFailure volumes: - name: commit-storage persistentVolumeClaim: claimName: poller-pvc # Use a PVC to persist the last commit file --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: poller-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Mi where, replace value of GITHUB_API_URL to match your user name and repo components e.g. https://api.github.com/repos/devops-0001/vote/commits verify values for all env variables to be correct Also create secret with github token with access to the repository, which is used in the cronjob above. kubectl create secret generic github-token-secret --from-literal=token=xxxx replace xxxx with actul GitHub access token. create this cronjob as kubectl apply -f poller-cronjob.yaml now start watching the cronjobs as well as event flow from Argo Workflow dashboard. Event Flow : Workflows:","title":"Lab K606 - Argo Events"},{"location":"argo_events/#argo-events","text":"Author: Gourav Shah Publisher: School of Devops Version : v2024.06.02.01 Project: Trigger CI Pipeline on GitHub Changes","title":"Argo Events"},{"location":"argo_events/#set-up-argo-events","text":"Launch Argo Workflow Environment with Killercoda , set up server UI and have it open. Install Argo Events kubectl create namespace argo-events kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml # Install with a validating admission controller kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install-validating-webhook.yaml kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml Create RBAC Policies kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/sensor-rbac.yaml kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/workflow-rbac.yaml","title":"Set up Argo Events"},{"location":"argo_events/#setup-components-to-trigger-ci-pipeline","text":"You will now set up the components required for the Argo Events to trigger the CI workflow based on changes to GitHub. For this you are going to need Event Source - To listen to GitHub change events Sensor - Which activates on updated to event source and triggers the workflow Workflow Template - To create an instance of the CI Pipeline Workflow Container Registry Credentials - Used by the workflow to publish images with Create an Argo Events EventSource and Sensor to handle the events sent by your polling job. File: webook-eventsource.yaml apiVersion: argoproj.io/v1alpha1 kind: EventSource metadata: name: webhook namespace: argo-events spec: service: ports: - port: 12000 targetPort: 12000 webhook: example: port: \"12000\" endpoint: /example method: POST github: port: \"12000\" endpoint: /github method: POST apply kubectl apply -f webook-eventsource.yaml Check the Argo Workflow for Vote CI Pipeline converted into a Workflow Template . Create workflow template using this code as kubectl apply -f https://gist.githubusercontent.com/initcron/c1704b560909f424e66062d86af9ff5c/raw/f7c5f73605a732d358a93854bc2da652113de494/vote-ci-template.yaml validate kubectl get workflowtemplate -A argo template list -A add registry credentials to argo-events namespace again with kubectl create secret -n argo-events docker-registry docker-registry-creds \\ --docker-server=https://index.docker.io/v1/ \\ --docker-username=xxxx --docker-password=yyyy where replace, xxxx with registry username yyyy with registry access token Add sensor which will listen to updates to github webhook created with event source and then trigger the argo workflow as, File : sensor.yaml apiVersion: argoproj.io/v1alpha1 kind: Sensor metadata: name: polling-sensor namespace: argo-events spec: template: serviceAccountName: operate-workflow-sa dependencies: - name: poll-github eventSourceName: webhook eventName: github triggers: - template: name: launch-vote-ci argoWorkflow: operation: submit source: resource: apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: ci-pipeline- spec: workflowTemplateRef: name: vote-ci-template arguments: parameters: - name: repo-url value: \"https://github.com/xxxxxx/vote.git\" - name: branch value: \"main\" - name: image value: \"xxxxxx/vote\" - name: dockerfile value: \"Dockerfile\" where, replace repo-url with your code repository url update image tag by replacing xxxxxx with your docker hub user name apply kubectl apply -f sensor.yaml validate kubectl get pods -n argo-events you could also check the logs for sensor controller (useful for troubleshooting) using kubectl logs -n argo-events -l \"controller=sensor-controller\" you should see a new pod launched to run this sensor. At this time, if you have Argo Workflow set up, you should see a event flow such as this Event Flow :","title":"Setup Components to Trigger CI Pipeline"},{"location":"argo_events/#deploy-github-poller","text":"After setting up the event flow, you also need to set up something which will trigger the event source on changes to GitHub. You could do this in two ways Using Webhooks : You could expose the event source service to outside and let GitHub trigger a webhook whenever there is a push event. This is useful if your event source can be publically available (GitHub can connect to it). In-cluster Polling : You could alternately set up in cluster system to periodically poll GitHub for changes, and trigger the event source. This is useful when you can not expose event source service pubically, and are running your cluster in a private network. Since we do not assume your cluster is public, we will employ the second approach. To achisve in-cluster polling, create the following Kubernetes CronJob that periodically polls GitHub for new commits. If new commits are found, the job can trigger the event source webhook created earlier. File: poller-cronjob.yaml --- apiVersion: batch/v1 kind: CronJob metadata: name: github-polling-job spec: schedule: \"* * * * *\" # Poll every minute jobTemplate: spec: template: spec: containers: - name: poller image: schoolofdevops/github-poller:latest env: - name: GITHUB_API_URL value: \"https://api.github.com/repos/yourusername/yourrepo/commits\" - name: GITHUB_TOKEN valueFrom: secretKeyRef: name: github-token-secret key: token - name: LAST_COMMIT_FILE value: \"/data/last_commit.txt\" - name: ARGO_EVENT_SOURCE_URL value: \"http://webhook-eventsource-svc.argo-events.svc.cluster.local:12000/github\" volumeMounts: - name: commit-storage mountPath: /data restartPolicy: OnFailure volumes: - name: commit-storage persistentVolumeClaim: claimName: poller-pvc # Use a PVC to persist the last commit file --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: poller-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Mi where, replace value of GITHUB_API_URL to match your user name and repo components e.g. https://api.github.com/repos/devops-0001/vote/commits verify values for all env variables to be correct Also create secret with github token with access to the repository, which is used in the cronjob above. kubectl create secret generic github-token-secret --from-literal=token=xxxx replace xxxx with actul GitHub access token. create this cronjob as kubectl apply -f poller-cronjob.yaml now start watching the cronjobs as well as event flow from Argo Workflow dashboard. Event Flow : Workflows:","title":"Deploy GitHub Poller"},{"location":"argo_experiments_analysis/","text":"Analysis and Experiments Setup Metrics Server If you try to pull monitoring information using the following commands kubectl top pod kubectl top node it does not show it, rather gives you a error message similar to [output] Error from server (NotFound): the server could not find the requested resource (get services http:heapster:) Even though the error mentions heapster, its replaced with metrics server by default now. Deploy metric server with the following commands, cd ~ git clone https://github.com/schoolofdevops/metrics-server.git kubectl apply -k metrics-server/manifests/overlays/release Validate kubectl get deploy,pods -n kube-system --selector='k8s-app=metrics-server' You could validate again with kubectl top pod kubectl top node where expected output should be similar to, kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% kind-control-plane 123m 6% 688Mi 17% kind-worker 39m 1% 498Mi 12% kind-worker2 31m 1% 422Mi 10% If you see a similar output, monitoring is now been setup. Deploy Prometheus and Grafana helm upgrade --install prom -n monitoring \\ prometheus-community/kube-prometheus-stack \\ --set grafana.service.type=NodePort \\ --set grafana.service.nodePort=30400 \\ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false Redeploy Nginx Ingress Controller Re deploy nginx ingress controller with helm, this time enabling the exposing the metrics which can then be scraped/collected by prometheus. helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true --set \\ controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" \\ --set controller.hostPort.enabled=true \\ --set controller.hostPort.ports.http=80 \\ --set controller.hostPort.ports.https=443 \\ --set-string controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\ --set-string controller.nodeSelector.ingress-ready=\"true\" Setup Grafana Dashboard for Nginx Ingress Controller Now, login to grafana and import custom dashboard for Nginx Ingress as Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" \u2800 It may look similar to this, with possibly less data initially However, if you see some metric coming in, your setup with Nginx Ingress and Promethus Integration is working ! You may pat your back at this time :) Updated Rollout Configuration with Experiment and Analysis File: /prod/rollout.yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: vote spec: replicas: 5 strategy: blueGreen: null canary: canaryService: vote-preview stableService: vote steps: - setCanaryScale: replicas: 2 - experiment: duration: 3m templates: - name: canary specRef: canary service: name: experiment analyses: - name: fitness-test templateName: canary-fitness-test - setWeight: 20 - pause: duration: 10s - setWeight: 40 - pause: duration: 10s - setWeight: 60 - analysis: templates: - templateName: loadtest - templateName: latency - setWeight: 80 - pause: duration: 10s - setWeight: 100 trafficRouting: nginx: stableIngress: vote additionalIngressAnnotations: canary-by-header: X-Canary canary-by-header-value: siege Explanation Rollout Configuration : The rollout strategy includes canary steps with set weights and pauses. Each canary step includes an experiment with a specified duration (e.g., 3 minutes). The experiment step runs a experimental replicaset and launches a fitness test to validate if the new version looks okay. After 60% traffic is shifted to canary, a load test is lauched along with analysis from prometheus to check if the new version will perform okay with the load. Analysis Templates : Defines a templates for running various tests and analyses. The loadtest container runs the load testing script against the canary service ( vote-preview ). The fitness-test job runs a test to validate if the new version is fit for deployment. the latency analysis fetches latency metrics from Prometheus and checks if the application is responding in acceptable time frame even with load conditions. \u2800 How it Works At each setWeight step, traffic is gradually shifted to the canary version. The analysis step includes both the load test and the metric analysis. The experiment runs for 3 minutes, during which the fitness test is conducted. Simultaneously with load test , the analysis template checks Prometheus metrics to ensure the canary is performing correctly. If the analysis detects errors beyond the acceptable threshold, the rollout will trigger a rollback. If the canary passes the load test and analysis, the rollout proceeds to the next step. \u2800 By configuring the experiment and analysis to run in parallel, you can ensure comprehensive testing and validation of the canary version, enabling automatic rollback if any issues are detected. Template for Load Testing File prod/loadtest-analysistemplate.yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: loadtest spec: metrics: - name: loadtest-vote provider: job: spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: - siege - \"--concurrent=2\" - \"--benchmark\" - \"--time=5m\" - \"--header='X-Canary: siege'\" - \"http://vote.example.com\" restartPolicy: Never hostAliases: - ip: \"xx.xx.xx.xx\" hostnames: - \"vote.example.com\" backoffLimit: 4 where, * replace xx.xx.xx.xx with internal IP Address of worker node. Find out by using kubectl get nodes -o wide [sample output] NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kind-control-plane Ready control-plane 2d23h v1.30.0 172.18.0.2 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.15 kind-worker Ready 2d23h v1.30.0 172.18.0.4 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.15 kind-worker2 Ready 2d23h v1.30.0 172.18.0.3 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.15 From this output, you are going to use 172.18.0.4 in the configuration above. AnalysisTemplate for Prometheus Metrics File : prod/latency-analysistemplate.yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: latency spec: metrics: - name: nginx-latency-ms initialDelay: 1m interval: 1m failureLimit: 2 count: 4 successCondition: result < 50.0 failureCondition: result >= 50.0 provider: prometheus: address: http://prom-kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090 query: | scalar( 1000 * histogram_quantile(0.99, sum( rate( nginx_ingress_controller_request_duration_seconds_bucket{ingress=\"vote\", exported_namespace=\"prod\"}[1m] ) ) by (le) ) ) Fitness Test for Canary File: prod/fitness-analysistemplate.yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: canary-fitness-test spec: metrics: - name: canary-fitness interval: 30s count: 3 successCondition: result == \"true\" failureLimit: 1 provider: job: spec: template: spec: containers: - name: fitness-test image: curlimages/curl command: [\"/bin/sh\", \"-c\"] args: - | FITNESS_RESULT=\"false\" CANARY_SERVICE_URL=\"http://vote-preview\" # Perform the fitness test RESPONSE=$(curl -s $CANARY_SERVICE_URL) # Check if the response contains the expected string if [[ \"$RESPONSE\" == *\"Processed by container ID\"* ]]; then FITNESS_RESULT=\"true\" fi # Return the fitness test result echo $FITNESS_RESULT restartPolicy: Never backoffLimit: 1 Update Kustomization for Prod File : prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base - ingress.yaml - fitness-analysistemplate.yaml - latency-analysistemplate.yaml - loadtest-analysistemplate.yaml apply kustomize build prod kubectl apply -k prod watch the rollout using kubectl argo rollouts get rollout vote","title":"Lab K608 - Experiments and Analysis"},{"location":"argo_experiments_analysis/#analysis-and-experiments","text":"","title":"Analysis and Experiments"},{"location":"argo_experiments_analysis/#setup-metrics-server","text":"If you try to pull monitoring information using the following commands kubectl top pod kubectl top node it does not show it, rather gives you a error message similar to [output] Error from server (NotFound): the server could not find the requested resource (get services http:heapster:) Even though the error mentions heapster, its replaced with metrics server by default now. Deploy metric server with the following commands, cd ~ git clone https://github.com/schoolofdevops/metrics-server.git kubectl apply -k metrics-server/manifests/overlays/release Validate kubectl get deploy,pods -n kube-system --selector='k8s-app=metrics-server' You could validate again with kubectl top pod kubectl top node where expected output should be similar to, kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% kind-control-plane 123m 6% 688Mi 17% kind-worker 39m 1% 498Mi 12% kind-worker2 31m 1% 422Mi 10% If you see a similar output, monitoring is now been setup.","title":"Setup Metrics Server"},{"location":"argo_experiments_analysis/#deploy-prometheus-and-grafana","text":"helm upgrade --install prom -n monitoring \\ prometheus-community/kube-prometheus-stack \\ --set grafana.service.type=NodePort \\ --set grafana.service.nodePort=30400 \\ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false","title":"Deploy Prometheus and Grafana"},{"location":"argo_experiments_analysis/#redeploy-nginx-ingress-controller","text":"Re deploy nginx ingress controller with helm, this time enabling the exposing the metrics which can then be scraped/collected by prometheus. helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true --set \\ controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" \\ --set controller.hostPort.enabled=true \\ --set controller.hostPort.ports.http=80 \\ --set controller.hostPort.ports.https=443 \\ --set-string controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\ --set-string controller.nodeSelector.ingress-ready=\"true\"","title":"Redeploy Nginx Ingress Controller"},{"location":"argo_experiments_analysis/#setup-grafana-dashboard-for-nginx-ingress-controller","text":"Now, login to grafana and import custom dashboard for Nginx Ingress as Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" \u2800 It may look similar to this, with possibly less data initially However, if you see some metric coming in, your setup with Nginx Ingress and Promethus Integration is working ! You may pat your back at this time :)","title":"Setup Grafana Dashboard for Nginx Ingress Controller"},{"location":"argo_experiments_analysis/#updated-rollout-configuration-with-experiment-and-analysis","text":"File: /prod/rollout.yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: vote spec: replicas: 5 strategy: blueGreen: null canary: canaryService: vote-preview stableService: vote steps: - setCanaryScale: replicas: 2 - experiment: duration: 3m templates: - name: canary specRef: canary service: name: experiment analyses: - name: fitness-test templateName: canary-fitness-test - setWeight: 20 - pause: duration: 10s - setWeight: 40 - pause: duration: 10s - setWeight: 60 - analysis: templates: - templateName: loadtest - templateName: latency - setWeight: 80 - pause: duration: 10s - setWeight: 100 trafficRouting: nginx: stableIngress: vote additionalIngressAnnotations: canary-by-header: X-Canary canary-by-header-value: siege","title":"Updated Rollout Configuration with Experiment and Analysis"},{"location":"argo_experiments_analysis/#explanation","text":"Rollout Configuration : The rollout strategy includes canary steps with set weights and pauses. Each canary step includes an experiment with a specified duration (e.g., 3 minutes). The experiment step runs a experimental replicaset and launches a fitness test to validate if the new version looks okay. After 60% traffic is shifted to canary, a load test is lauched along with analysis from prometheus to check if the new version will perform okay with the load. Analysis Templates : Defines a templates for running various tests and analyses. The loadtest container runs the load testing script against the canary service ( vote-preview ). The fitness-test job runs a test to validate if the new version is fit for deployment. the latency analysis fetches latency metrics from Prometheus and checks if the application is responding in acceptable time frame even with load conditions. \u2800","title":"Explanation"},{"location":"argo_experiments_analysis/#how-it-works","text":"At each setWeight step, traffic is gradually shifted to the canary version. The analysis step includes both the load test and the metric analysis. The experiment runs for 3 minutes, during which the fitness test is conducted. Simultaneously with load test , the analysis template checks Prometheus metrics to ensure the canary is performing correctly. If the analysis detects errors beyond the acceptable threshold, the rollout will trigger a rollback. If the canary passes the load test and analysis, the rollout proceeds to the next step. \u2800 By configuring the experiment and analysis to run in parallel, you can ensure comprehensive testing and validation of the canary version, enabling automatic rollback if any issues are detected.","title":"How it Works"},{"location":"argo_experiments_analysis/#template-for-load-testing","text":"File prod/loadtest-analysistemplate.yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: loadtest spec: metrics: - name: loadtest-vote provider: job: spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: - siege - \"--concurrent=2\" - \"--benchmark\" - \"--time=5m\" - \"--header='X-Canary: siege'\" - \"http://vote.example.com\" restartPolicy: Never hostAliases: - ip: \"xx.xx.xx.xx\" hostnames: - \"vote.example.com\" backoffLimit: 4 where, * replace xx.xx.xx.xx with internal IP Address of worker node. Find out by using kubectl get nodes -o wide [sample output] NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kind-control-plane Ready control-plane 2d23h v1.30.0 172.18.0.2 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.15 kind-worker Ready 2d23h v1.30.0 172.18.0.4 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.15 kind-worker2 Ready 2d23h v1.30.0 172.18.0.3 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.15 From this output, you are going to use 172.18.0.4 in the configuration above.","title":"Template for Load Testing"},{"location":"argo_experiments_analysis/#analysistemplate-for-prometheus-metrics","text":"File : prod/latency-analysistemplate.yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: latency spec: metrics: - name: nginx-latency-ms initialDelay: 1m interval: 1m failureLimit: 2 count: 4 successCondition: result < 50.0 failureCondition: result >= 50.0 provider: prometheus: address: http://prom-kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090 query: | scalar( 1000 * histogram_quantile(0.99, sum( rate( nginx_ingress_controller_request_duration_seconds_bucket{ingress=\"vote\", exported_namespace=\"prod\"}[1m] ) ) by (le) ) )","title":"AnalysisTemplate for Prometheus Metrics"},{"location":"argo_experiments_analysis/#fitness-test-for-canary","text":"File: prod/fitness-analysistemplate.yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: canary-fitness-test spec: metrics: - name: canary-fitness interval: 30s count: 3 successCondition: result == \"true\" failureLimit: 1 provider: job: spec: template: spec: containers: - name: fitness-test image: curlimages/curl command: [\"/bin/sh\", \"-c\"] args: - | FITNESS_RESULT=\"false\" CANARY_SERVICE_URL=\"http://vote-preview\" # Perform the fitness test RESPONSE=$(curl -s $CANARY_SERVICE_URL) # Check if the response contains the expected string if [[ \"$RESPONSE\" == *\"Processed by container ID\"* ]]; then FITNESS_RESULT=\"true\" fi # Return the fitness test result echo $FITNESS_RESULT restartPolicy: Never backoffLimit: 1","title":"Fitness Test for Canary"},{"location":"argo_experiments_analysis/#update-kustomization-for-prod","text":"File : prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base - ingress.yaml - fitness-analysistemplate.yaml - latency-analysistemplate.yaml - loadtest-analysistemplate.yaml apply kustomize build prod kubectl apply -k prod watch the rollout using kubectl argo rollouts get rollout vote","title":"Update Kustomization for Prod"},{"location":"argo_experiments_analysis/#_1","text":"","title":""},{"location":"argo_iamge_updater/","text":"Argo Image Updater Author: Gourav Shah Publisher: School of Devops Version : v2024.06.04.01 Project : Connect CI Pipeline set up with Argo Workflow and Argo Events with the CD Pipeline created with ArgoCD and Argo Rollouts. Set up a workflow where Whenever there is change to the main branch of the application code, it triggers the CI Pipeline with Argo Events + Argo Workflow. The result of this is a new image being published to the container registery. Set up a watcher which is monitor the container registry, this is where argo image updater comes in. Whenever a new tag with a certain pattern is available in the registry, update the image tag in the main branch of the deployment repo with it. Auto commit to GitHub. Change to the image tag should be automatically picked up by Argo CD and it should trigger the deployment to staging. Since its been integrated with argo rollouts, it implments the release strategy (e.g. Blue/Green) configured with the rollout spec. Deployment to prod is just one pull request away. Whenever the change (e.g. new image tag) is merged from master to release branch, prod deployment should get triggered based on another application deployment flow set up with Argo CD. And that GitOps in action for you... Set up automatic Image Updater with ArgoCD Install Argo CD Image Updater as, kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml Create a Secret with your GitHub credentials so that the image updater can discover and commit the image tags to git repository. kubectl -n argocd create secret generic git-creds \\ --from-literal=username=xxxxxx \\ --from-literal=password=ghp_yyyyyy Note : replace xxxxxx with your GitHub Username and ghp_yyyyyy with GitHub Access Token with write access to repository. If you do not already have token, create it from GitHub Marking Staging Application for Auto Image Updates When you set up staging deployment, ArgoCD has created a application object for you. You would need to add a few annotations to it so that the Image Updater knows that this application is been marked for image auto update. Observe the existing application created from ArgoCD UI as, kubectl get application -n argocd kubectl describe application -n argocd vote-staging specifically observe the annotations Annotations: To update this application with appropriate annotations create a patch file with the name and content given below, File : argo_applications_vote-staging_patch.yaml metadata: annotations: argocd-image-updater.argoproj.io/git-branch: main argocd-image-updater.argoproj.io/image-list: myimage=xxxxxx/vote argocd-image-updater.argoproj.io/myimage.allow-tags: regexp:^[0-9a-f]{7}$ argocd-image-updater.argoproj.io/myimage.ignore-tags: latest, dev argocd-image-updater.argoproj.io/myimage.update-strategy: latest argocd-image-updater.argoproj.io/myimage.kustomize.image-name: schoolofdevops/vote argocd-image-updater.argoproj.io/myimage.force-update: \"true\" argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/git-creds argocd-image-updater.argoproj.io/write-back-target: \"kustomization:../base\" Source: Mark Staging App for Automatic Image Updates from Argo Where, Replace xxxxxx/vote with your own repo in the argocd-image-updater.argoproj.io/image-list annotation. Apply the above patch as, kubectl patch application --type=merge -n argocd vote-staging --patch-file argo_applications_vote-staging_patch.yaml Validate annotations are added, kubectl describe application -n argocd vote-staging [sample output] ... Labels: Annotations: argocd-image-updater.argoproj.io/git-branch: main argocd-image-updater.argoproj.io/image-list: myimage=initcron/argovote argocd-image-updater.argoproj.io/myimage.allow-tags: regexp:^[0-9a-f]{7}$ argocd-image-updater.argoproj.io/myimage.force-update: true argocd-image-updater.argoproj.io/myimage.ignore-tags: latest, dev argocd-image-updater.argoproj.io/myimage.kustomize.image-name: schoolofdevops/vote argocd-image-updater.argoproj.io/myimage.update-strategy: latest argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/git-creds argocd-image-updater.argoproj.io/write-back-target: kustomization:../base API Version: argoproj.io/v1alpha1 Kind: Application ... If everything goes well, within a few minutes, you should see a commit to the main branch of the vote-deploy repository that you have. And a few minutes after that, you should see the staging deployment on ArgoCD pick up the newly updated image tag and deploy it. You could tally it from Rollout Dashboard that it has picked up the new image with commit hash as tag and also validate with kubectl describe application -n argocd vote-staging where you should see the following in the output status Summary: Images: initcron/vote:52eea4d Sync: Compared To: Destination: Namespace: staging Server: https://kubernetes.default.svc Source: From now, its just matter of creating a pull request and merging it to release branch to deploy to prod. You could check the logs for the image updater which is running in argocd namespace by using a command similar to kubectl logs -f -l \"app.kubernetes.io/name=argocd-image-updater\" -n argocd Thats all ! If you have gotten till here, congratulate yourself as you have just built a simplistic but completely working modern CI/CD Pipeline ! Hurray !!","title":"Lab K607 - Argo Image Updater"},{"location":"argo_iamge_updater/#argo-image-updater","text":"Author: Gourav Shah Publisher: School of Devops Version : v2024.06.04.01 Project : Connect CI Pipeline set up with Argo Workflow and Argo Events with the CD Pipeline created with ArgoCD and Argo Rollouts. Set up a workflow where Whenever there is change to the main branch of the application code, it triggers the CI Pipeline with Argo Events + Argo Workflow. The result of this is a new image being published to the container registery. Set up a watcher which is monitor the container registry, this is where argo image updater comes in. Whenever a new tag with a certain pattern is available in the registry, update the image tag in the main branch of the deployment repo with it. Auto commit to GitHub. Change to the image tag should be automatically picked up by Argo CD and it should trigger the deployment to staging. Since its been integrated with argo rollouts, it implments the release strategy (e.g. Blue/Green) configured with the rollout spec. Deployment to prod is just one pull request away. Whenever the change (e.g. new image tag) is merged from master to release branch, prod deployment should get triggered based on another application deployment flow set up with Argo CD. And that GitOps in action for you...","title":"Argo Image Updater"},{"location":"argo_iamge_updater/#set-up-automatic-image-updater-with-argocd","text":"Install Argo CD Image Updater as, kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml Create a Secret with your GitHub credentials so that the image updater can discover and commit the image tags to git repository. kubectl -n argocd create secret generic git-creds \\ --from-literal=username=xxxxxx \\ --from-literal=password=ghp_yyyyyy Note : replace xxxxxx with your GitHub Username and ghp_yyyyyy with GitHub Access Token with write access to repository. If you do not already have token, create it from GitHub","title":"Set up automatic Image Updater with ArgoCD"},{"location":"argo_iamge_updater/#marking-staging-application-for-auto-image-updates","text":"When you set up staging deployment, ArgoCD has created a application object for you. You would need to add a few annotations to it so that the Image Updater knows that this application is been marked for image auto update. Observe the existing application created from ArgoCD UI as, kubectl get application -n argocd kubectl describe application -n argocd vote-staging specifically observe the annotations Annotations: To update this application with appropriate annotations create a patch file with the name and content given below, File : argo_applications_vote-staging_patch.yaml metadata: annotations: argocd-image-updater.argoproj.io/git-branch: main argocd-image-updater.argoproj.io/image-list: myimage=xxxxxx/vote argocd-image-updater.argoproj.io/myimage.allow-tags: regexp:^[0-9a-f]{7}$ argocd-image-updater.argoproj.io/myimage.ignore-tags: latest, dev argocd-image-updater.argoproj.io/myimage.update-strategy: latest argocd-image-updater.argoproj.io/myimage.kustomize.image-name: schoolofdevops/vote argocd-image-updater.argoproj.io/myimage.force-update: \"true\" argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/git-creds argocd-image-updater.argoproj.io/write-back-target: \"kustomization:../base\" Source: Mark Staging App for Automatic Image Updates from Argo Where, Replace xxxxxx/vote with your own repo in the argocd-image-updater.argoproj.io/image-list annotation. Apply the above patch as, kubectl patch application --type=merge -n argocd vote-staging --patch-file argo_applications_vote-staging_patch.yaml Validate annotations are added, kubectl describe application -n argocd vote-staging [sample output] ... Labels: Annotations: argocd-image-updater.argoproj.io/git-branch: main argocd-image-updater.argoproj.io/image-list: myimage=initcron/argovote argocd-image-updater.argoproj.io/myimage.allow-tags: regexp:^[0-9a-f]{7}$ argocd-image-updater.argoproj.io/myimage.force-update: true argocd-image-updater.argoproj.io/myimage.ignore-tags: latest, dev argocd-image-updater.argoproj.io/myimage.kustomize.image-name: schoolofdevops/vote argocd-image-updater.argoproj.io/myimage.update-strategy: latest argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/git-creds argocd-image-updater.argoproj.io/write-back-target: kustomization:../base API Version: argoproj.io/v1alpha1 Kind: Application ... If everything goes well, within a few minutes, you should see a commit to the main branch of the vote-deploy repository that you have. And a few minutes after that, you should see the staging deployment on ArgoCD pick up the newly updated image tag and deploy it. You could tally it from Rollout Dashboard that it has picked up the new image with commit hash as tag and also validate with kubectl describe application -n argocd vote-staging where you should see the following in the output status Summary: Images: initcron/vote:52eea4d Sync: Compared To: Destination: Namespace: staging Server: https://kubernetes.default.svc Source: From now, its just matter of creating a pull request and merging it to release branch to deploy to prod. You could check the logs for the image updater which is running in argocd namespace by using a command similar to kubectl logs -f -l \"app.kubernetes.io/name=argocd-image-updater\" -n argocd Thats all ! If you have gotten till here, congratulate yourself as you have just built a simplistic but completely working modern CI/CD Pipeline ! Hurray !!","title":"Marking Staging Application for Auto Image Updates"},{"location":"argo_multi_env_deploy/","text":"Setting up Automated Deployments with ArgoCD Author: Gourav Shah Publisher: School of Devops Version : v2024.06.02.01 Project: : Setup Automated Deployment to Staging and Prod with Argo CD. Setup ArgoCD Install ArgoCD kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml Reset admin password to password # bcrypt(password)=$2a$10$rRyBsGSHK6.uc8fntPwVIuLVHgsAhAX7TcdrqW/RADU0uh7CaChLa kubectl -n argocd patch secret argocd-secret \\ -p '{\"stringData\": { \"admin.password\": \"$2a$10$rRyBsGSHK6.uc8fntPwVIuLVHgsAhAX7TcdrqW/RADU0uh7CaChLa\", \"admin.passwordMtime\": \"'$(date +%FT%T%Z)'\" }}' Source: reset-argo-password.sh Reference: argo-cd/faq.md at master \u00b7 argoproj/argo-cd \u00b7 GitHub kubectl get all -n argocd kubectl patch svc argocd-server -n argocd --patch \\ '{\"spec\": { \"type\": \"NodePort\", \"ports\": [ { \"nodePort\": 32100, \"port\": 443, \"protocol\": \"TCP\", \"targetPort\": 8080 } ] } }' source: patch_argo_svc.sh kubectl get svc -n argocd Find out the IP address for one of the nodes. One way to do so is to run the following command, kubectl get nodes -o wide Note IP address for one of the nodes and browse to https://NODEIP:32100 where, replace NODEIP with the actual. You should be presented with the login page for ArgoCD as follows username =. admin password = password Configuring Repository and Project Ensure that you have checked in all the code from earlier labs and pushed it to your repository. Once logged in to ArgoCD, select settings from left menu and browse to Projects Click on New Project -> Create and provide Project Name and Description as Proceed to create the project. From Project Configuration page that you are redirected to, edit DESTINATIONS Select default cluster name from dropdown Select in-cluster as Name Add two entries, one for staging and another for prod Namespace Save From settings from left menu and browse to Repositories Select Connet Repo and provide the following configuration Via: HTTPS Type: git Project: instavote Repository URL: https://github.com/xxxx/argo-labs.git (replace with actual) Username: GitHub Username (If Private Repo) Password: GitHub Password or Token (If Private Repo) Finally click on Connect to add the repo. Setup Staging and Prod Deployments with ArgoCD Clean up resources in the namespaces for staging and prod environments if you haven't already, cd argo-labs kubectl delete -k staging/ -n staging kubectl delete -k prod/ -n prod Browse to ArgoCD web console and click on Create Application From General , Application Name : vote-staging Project : instavote Sync Policy : Automatic Prune Resources: Checked From Source, Repository URL : Your Repo URL (https) Revision : main / HEAD Path : staging From Destination, Cluster URL : https://kubernetes.default.svc (default) Namespace : staging Click on CREATE button on the top validate with kubectl get all -n staging Set up Deploy to Prod Configuration You will deploy to prod based on a specific git branch e.g. release . Create a release branch to deploy the application to prod with, cd argo-labs/ git pull origin main git checkout -b release git push origin release You will see a new branch created on GitHub for this repository. Alternately, you could also create the branch from GitHub Web UI. Create another application, repeat the same configurations with the following changes, Application Name: vote-prod Project Name: instavote Sync Policy: Automatic Prune Resources: Checked Repository: Same Repo Revision: release (Branches) Path: prod Cluster URL: default from dropdown. Namespace : prod Create and Sync manually. Once synced, you should see two applications configured on ArgoCD tracking two environments. You could also check the applications using kubectl as kubectl get applications -n argocd kubectl describe application vote-prod -n argocd Deployments in Action Open two terminals and start watching for changes in the staging namespace Terminal 1 watch kubectl get ro,all -n staging Terminal 2 watch kubectl get ro,all -n prod Watch for the changes in the console as well as on Argo. You shall see the application synced from the Git Repo to the Kubernetes Cluster in a few seconds. Staging Prod Validate by accessing the vote apps on Staging : http://NODEIP:30000 Prod : http://NODEIP:30200 where, replace NODEIP with actual IP or Hostname of your cluster node. e.g. Exercises Set up branch protection rule to lock down release branch and allow changes via pull requests. You can experiment by adding additional policies as well. Try modifying YAML manifests in Deploy Repo in Git in main branch by changing the image tag in staging/kustomization.yaml and wait for the staging deployment. Then raise the pull request to merge it into release and see it deployed to prod. References Getting Started with Argo Getting Started - Argo CD - Declarative GitOps CD for Kubernetes Reset admin password argo-cd/faq.md at master \u00b7 argoproj/argo-cd \u00b7 GitHub","title":"Lab K604 - ArgoCD"},{"location":"argo_multi_env_deploy/#setting-up-automated-deployments-with-argocd","text":"Author: Gourav Shah Publisher: School of Devops Version : v2024.06.02.01 Project: : Setup Automated Deployment to Staging and Prod with Argo CD.","title":"Setting up Automated Deployments with ArgoCD"},{"location":"argo_multi_env_deploy/#setup-argocd","text":"Install ArgoCD kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml Reset admin password to password # bcrypt(password)=$2a$10$rRyBsGSHK6.uc8fntPwVIuLVHgsAhAX7TcdrqW/RADU0uh7CaChLa kubectl -n argocd patch secret argocd-secret \\ -p '{\"stringData\": { \"admin.password\": \"$2a$10$rRyBsGSHK6.uc8fntPwVIuLVHgsAhAX7TcdrqW/RADU0uh7CaChLa\", \"admin.passwordMtime\": \"'$(date +%FT%T%Z)'\" }}' Source: reset-argo-password.sh Reference: argo-cd/faq.md at master \u00b7 argoproj/argo-cd \u00b7 GitHub kubectl get all -n argocd kubectl patch svc argocd-server -n argocd --patch \\ '{\"spec\": { \"type\": \"NodePort\", \"ports\": [ { \"nodePort\": 32100, \"port\": 443, \"protocol\": \"TCP\", \"targetPort\": 8080 } ] } }' source: patch_argo_svc.sh kubectl get svc -n argocd Find out the IP address for one of the nodes. One way to do so is to run the following command, kubectl get nodes -o wide Note IP address for one of the nodes and browse to https://NODEIP:32100 where, replace NODEIP with the actual. You should be presented with the login page for ArgoCD as follows username =. admin password = password","title":"Setup ArgoCD"},{"location":"argo_multi_env_deploy/#configuring-repository-and-project","text":"Ensure that you have checked in all the code from earlier labs and pushed it to your repository. Once logged in to ArgoCD, select settings from left menu and browse to Projects Click on New Project -> Create and provide Project Name and Description as Proceed to create the project. From Project Configuration page that you are redirected to, edit DESTINATIONS Select default cluster name from dropdown Select in-cluster as Name Add two entries, one for staging and another for prod Namespace Save From settings from left menu and browse to Repositories Select Connet Repo and provide the following configuration Via: HTTPS Type: git Project: instavote Repository URL: https://github.com/xxxx/argo-labs.git (replace with actual) Username: GitHub Username (If Private Repo) Password: GitHub Password or Token (If Private Repo) Finally click on Connect to add the repo.","title":"Configuring Repository and Project"},{"location":"argo_multi_env_deploy/#setup-staging-and-prod-deployments-with-argocd","text":"Clean up resources in the namespaces for staging and prod environments if you haven't already, cd argo-labs kubectl delete -k staging/ -n staging kubectl delete -k prod/ -n prod Browse to ArgoCD web console and click on Create Application From General , Application Name : vote-staging Project : instavote Sync Policy : Automatic Prune Resources: Checked From Source, Repository URL : Your Repo URL (https) Revision : main / HEAD Path : staging From Destination, Cluster URL : https://kubernetes.default.svc (default) Namespace : staging Click on CREATE button on the top validate with kubectl get all -n staging","title":"Setup Staging and Prod Deployments with ArgoCD"},{"location":"argo_multi_env_deploy/#set-up-deploy-to-prod-configuration","text":"You will deploy to prod based on a specific git branch e.g. release . Create a release branch to deploy the application to prod with, cd argo-labs/ git pull origin main git checkout -b release git push origin release You will see a new branch created on GitHub for this repository. Alternately, you could also create the branch from GitHub Web UI. Create another application, repeat the same configurations with the following changes, Application Name: vote-prod Project Name: instavote Sync Policy: Automatic Prune Resources: Checked Repository: Same Repo Revision: release (Branches) Path: prod Cluster URL: default from dropdown. Namespace : prod Create and Sync manually. Once synced, you should see two applications configured on ArgoCD tracking two environments. You could also check the applications using kubectl as kubectl get applications -n argocd kubectl describe application vote-prod -n argocd","title":"Set up Deploy to Prod Configuration"},{"location":"argo_multi_env_deploy/#deployments-in-action","text":"Open two terminals and start watching for changes in the staging namespace Terminal 1 watch kubectl get ro,all -n staging Terminal 2 watch kubectl get ro,all -n prod Watch for the changes in the console as well as on Argo. You shall see the application synced from the Git Repo to the Kubernetes Cluster in a few seconds. Staging Prod Validate by accessing the vote apps on Staging : http://NODEIP:30000 Prod : http://NODEIP:30200 where, replace NODEIP with actual IP or Hostname of your cluster node. e.g.","title":"Deployments in Action"},{"location":"argo_multi_env_deploy/#exercises","text":"Set up branch protection rule to lock down release branch and allow changes via pull requests. You can experiment by adding additional policies as well. Try modifying YAML manifests in Deploy Repo in Git in main branch by changing the image tag in staging/kustomization.yaml and wait for the staging deployment. Then raise the pull request to merge it into release and see it deployed to prod.","title":"Exercises"},{"location":"argo_multi_env_deploy/#references","text":"Getting Started with Argo Getting Started - Argo CD - Declarative GitOps CD for Kubernetes Reset admin password argo-cd/faq.md at master \u00b7 argoproj/argo-cd \u00b7 GitHub","title":"References"},{"location":"argo_rollout_blue_green/","text":"Blue Green Releases with Argo Rollouts Author: Gourav Shah Publisher: School of Devops Version : v2024.06.03.01 Launch Vote App with Deployment Lets begin by first deploying the vote service with the deployment code available. Create a fork of Kubernetes Deployment Code for Vote App Review the code created with kustomization overlay configured for staging environment in additional to the base manifests. Create namespaces for staging environments as, kubectl create ns staging kubectl get ns kubectl config set-context --current --namespace=staging validate kubectl config get-contexts [sample output] CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kind-kind kind-kind kind-kind staging Now clone the forked repo and switch to it git clone https://github.com/xxxx/argo-labs.git replace xxxx with your user name change it to argo-labs and examine the code for base as well as staging as cd argo-labs kustomize build based kustomize build staging then deploy vote service to staging as kubectl apply -k staging where, -k option applies the kustomization spec. validate kubectl get all You should see the deployment and service for vote app. And you should be able to access the staging deployment with nodeport 30000 . Install Argo Rollouts Install Argo Rollouts Controller and CRDs with , kubectl create namespace argo-rollouts kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml Validate with, kubectl api-resources | grep -i argo Optionally, you could install argo plugin for kubectl On linux cd ~ curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64 chmod +x ./kubectl-argo-rollouts-linux-amd64 sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts On Mac. curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64 chmod +x ./kubectl-argo-rollouts-darwin-amd64 sudo mv ./kubectl-argo-rollouts-darwin-amd64 /usr/local/bin/kubectl-argo-rollouts And validate as, kubectl argo rollouts version Also install Kustomize by following the instructions in official documentation here. Create a Preview Service Create a preview service during blue-green analysis File: base/preview-service.yaml --- apiVersion: v1 kind: Service metadata: name: vote-preview labels: role: vote spec: selector: app: vote ports: - port: 80 targetPort: 80 protocol: TCP nodePort: 30100 type: NodePort Update base/kustomization.yaml with the following apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml - preview-service.yaml Create rollout along with preview service with kubectl apply -k staging Validate kubectl get all kubectl describe svc vote kubectl describe svc vote-preview Both services should be pointing to the same set of pods. Migrate Deployment to Argo Rollout From argo-labs/base , create a copy of existing deployment spec as, git mv deployment.yaml rollout.yaml also update kustomization.yaml to replace deployment.yaml with rollout.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - rollout.yaml - service.yaml - preview-service.yaml Now edit base/rollout.yaml to add Blue/Green Release spec as per the documentation here . Update the following properties apiVersion: argoproj.io/v1alpha1 kind: Rollout replicas: 4 Remove spec.template.metadata.name if present. Replace the strategy with blue green as, strategy: blueGreen: autoPromotionEnabled: true autoPromotionSeconds: 30 activeService: vote previewService: vote-preview Also update argo-labs/staging/kustomization.yaml to remove the following file: staging/kustomization.yaml replicas: - count: 2 name: vote Delete the deployment kubectl delete deploy vote And then create the rollout from argo-labs path as, kubectl apply -k staging Validate kubectl get ro,all kubectl describe ro vote Deploy a Blue/Green Release Open a new terminal and start watching with watch kubectl get ro,all --show-labels Also open two more terminal windows to watch for vote and vote-preview services respectively as watch kubectl describe svc vote watch kubectl describe svc vote-preview If you had installed the argo rollout plugin for kubectl , you could also launch the Rollout UI with kubectl argo rollouts dashboard -p 3100 and then start watching for the rollout using http://localhost:3100/rollouts . Replace localhost with actul IP address of the host if you are running kubectl on a remote host. Now, trigger a rollout by updating the image by updating the image tag in base/rollout.yaml spec: containers: - image: schoolofdevops/vote:v2 and then by applying it as kubectl apply -k staging In the same terminal you could watch the status of the rollout with, kubectl argo rollouts status vote You could also watch the status using Argo Rollouts Dashboard as You would notice that, A new replica set is created with new version Service vote-bg (Preview service) starts pointing to the new version, while the active service vote still pointing to previous version After all pods with new version are ready, it will pause for about 30 seconds as configured in the rollout configuration. Once the wait interval is over, active service starts pointing to the new version pods created with the newly launched replica set. This is where there maybe just a blip of a nominal downtime. Eventually the replicase with older version will scale down to zero, completing the rollout. You could try rolling out (by updating the image version) a few times to learn how it works. Publish Changes to Repo Scale down the staging replicas so that you could accommodate more replicas created for prod environment, which will be yet another namespace in the same cluster. edit base/rollout.yaml and set the replicas count to 1 spec: replicas: 1 also in the same file, set the image tag back to v1 spec: containers: - image: schoolofdevops/vote:v1 apply kubectl apply -k staging/ validate kubectl get all Before committing the changes, make sure you have created a GitHub Access Token with repo access. To do so, Login to GiHub Click on your User Profile on top right From Settings -> Developer Settings -> Personal access tokens -> Tokens(classic) Select Generate new token(classic) , provide authentication as needed and proceed to create token. From token creation page, provide a token name and select repo configurations Generate Token and copy it somewhere safe. You will need this token multiple times throughout this course to be added as kubernetes secret, so keep it handy. Commit all the changes that you have made so far to the repo git status git add base/*.yaml git status git commit -am \"updated staging deployment code with blue green release\" git push origin main Cleaning Up Once you are done with it, clean up the environment with kubectl delete -k staging/","title":"Lab K602 - Blue/Green with Argo Rollouts"},{"location":"argo_rollout_blue_green/#blue-green-releases-with-argo-rollouts","text":"Author: Gourav Shah Publisher: School of Devops Version : v2024.06.03.01","title":"Blue Green Releases with Argo Rollouts"},{"location":"argo_rollout_blue_green/#launch-vote-app-with-deployment","text":"Lets begin by first deploying the vote service with the deployment code available. Create a fork of Kubernetes Deployment Code for Vote App Review the code created with kustomization overlay configured for staging environment in additional to the base manifests. Create namespaces for staging environments as, kubectl create ns staging kubectl get ns kubectl config set-context --current --namespace=staging validate kubectl config get-contexts [sample output] CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kind-kind kind-kind kind-kind staging Now clone the forked repo and switch to it git clone https://github.com/xxxx/argo-labs.git replace xxxx with your user name change it to argo-labs and examine the code for base as well as staging as cd argo-labs kustomize build based kustomize build staging then deploy vote service to staging as kubectl apply -k staging where, -k option applies the kustomization spec. validate kubectl get all You should see the deployment and service for vote app. And you should be able to access the staging deployment with nodeport 30000 .","title":"Launch Vote App with Deployment"},{"location":"argo_rollout_blue_green/#install-argo-rollouts","text":"Install Argo Rollouts Controller and CRDs with , kubectl create namespace argo-rollouts kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml Validate with, kubectl api-resources | grep -i argo Optionally, you could install argo plugin for kubectl On linux cd ~ curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64 chmod +x ./kubectl-argo-rollouts-linux-amd64 sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts On Mac. curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64 chmod +x ./kubectl-argo-rollouts-darwin-amd64 sudo mv ./kubectl-argo-rollouts-darwin-amd64 /usr/local/bin/kubectl-argo-rollouts And validate as, kubectl argo rollouts version Also install Kustomize by following the instructions in official documentation here.","title":"Install Argo Rollouts"},{"location":"argo_rollout_blue_green/#create-a-preview-service","text":"Create a preview service during blue-green analysis File: base/preview-service.yaml --- apiVersion: v1 kind: Service metadata: name: vote-preview labels: role: vote spec: selector: app: vote ports: - port: 80 targetPort: 80 protocol: TCP nodePort: 30100 type: NodePort Update base/kustomization.yaml with the following apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml - preview-service.yaml Create rollout along with preview service with kubectl apply -k staging Validate kubectl get all kubectl describe svc vote kubectl describe svc vote-preview Both services should be pointing to the same set of pods.","title":"Create a Preview Service"},{"location":"argo_rollout_blue_green/#migrate-deployment-to-argo-rollout","text":"From argo-labs/base , create a copy of existing deployment spec as, git mv deployment.yaml rollout.yaml also update kustomization.yaml to replace deployment.yaml with rollout.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - rollout.yaml - service.yaml - preview-service.yaml Now edit base/rollout.yaml to add Blue/Green Release spec as per the documentation here . Update the following properties apiVersion: argoproj.io/v1alpha1 kind: Rollout replicas: 4 Remove spec.template.metadata.name if present. Replace the strategy with blue green as, strategy: blueGreen: autoPromotionEnabled: true autoPromotionSeconds: 30 activeService: vote previewService: vote-preview Also update argo-labs/staging/kustomization.yaml to remove the following file: staging/kustomization.yaml replicas: - count: 2 name: vote Delete the deployment kubectl delete deploy vote And then create the rollout from argo-labs path as, kubectl apply -k staging Validate kubectl get ro,all kubectl describe ro vote","title":"Migrate Deployment to Argo Rollout"},{"location":"argo_rollout_blue_green/#deploy-a-bluegreen-release","text":"Open a new terminal and start watching with watch kubectl get ro,all --show-labels Also open two more terminal windows to watch for vote and vote-preview services respectively as watch kubectl describe svc vote watch kubectl describe svc vote-preview If you had installed the argo rollout plugin for kubectl , you could also launch the Rollout UI with kubectl argo rollouts dashboard -p 3100 and then start watching for the rollout using http://localhost:3100/rollouts . Replace localhost with actul IP address of the host if you are running kubectl on a remote host. Now, trigger a rollout by updating the image by updating the image tag in base/rollout.yaml spec: containers: - image: schoolofdevops/vote:v2 and then by applying it as kubectl apply -k staging In the same terminal you could watch the status of the rollout with, kubectl argo rollouts status vote You could also watch the status using Argo Rollouts Dashboard as You would notice that, A new replica set is created with new version Service vote-bg (Preview service) starts pointing to the new version, while the active service vote still pointing to previous version After all pods with new version are ready, it will pause for about 30 seconds as configured in the rollout configuration. Once the wait interval is over, active service starts pointing to the new version pods created with the newly launched replica set. This is where there maybe just a blip of a nominal downtime. Eventually the replicase with older version will scale down to zero, completing the rollout. You could try rolling out (by updating the image version) a few times to learn how it works.","title":"Deploy a Blue/Green Release"},{"location":"argo_rollout_blue_green/#publish-changes-to-repo","text":"Scale down the staging replicas so that you could accommodate more replicas created for prod environment, which will be yet another namespace in the same cluster. edit base/rollout.yaml and set the replicas count to 1 spec: replicas: 1 also in the same file, set the image tag back to v1 spec: containers: - image: schoolofdevops/vote:v1 apply kubectl apply -k staging/ validate kubectl get all Before committing the changes, make sure you have created a GitHub Access Token with repo access. To do so, Login to GiHub Click on your User Profile on top right From Settings -> Developer Settings -> Personal access tokens -> Tokens(classic) Select Generate new token(classic) , provide authentication as needed and proceed to create token. From token creation page, provide a token name and select repo configurations Generate Token and copy it somewhere safe. You will need this token multiple times throughout this course to be added as kubernetes secret, so keep it handy. Commit all the changes that you have made so far to the repo git status git add base/*.yaml git status git commit -am \"updated staging deployment code with blue green release\" git push origin main","title":"Publish Changes to Repo"},{"location":"argo_rollout_blue_green/#cleaning-up","text":"Once you are done with it, clean up the environment with kubectl delete -k staging/","title":"Cleaning Up"},{"location":"argo_rollout_canary/","text":"Implementing Canary Release for Prod Author: Gourav Shah Publisher: School of Devops Version : v2024.06.03.01 Prepare Prod Environment Create and switch to prod namespace kubectl create ns prod kubectl get ns kubectl config set-context --current --namespace=prod validate kubectl config get-contexts [sample output] CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kind-kind kind-kind kind-kind prod Create a copy of the kustomization code to set up prod with: cd argo-labs cp -r staging prod update NodePort for prod File : prod/service.yaml apiVersion: v1 kind: Service metadata: name: vote spec: ports: - name: \"80\" nodePort: 30200 port: 80 protocol: TCP targetPort: 80 type: NodePort create previw service kustomization overlay File: prod/preview-service.yaml apiVersion: v1 kind: Service metadata: name: vote-preview spec: ports: - name: \"80\" nodePort: 30300 port: 80 protocol: TCP targetPort: 80 type: NodePort update kustomization with namespace set to prod path for preview-service.yaml added to patches section File: prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base namespace: prod commonAnnotations: supported-by: sre@example.com labels: - includeSelectors: false pairs: project: instavote patches: - path: service.yaml - path: preview-service.yaml check kustomize build prod apply with kubectl apply -k prod/ validate as kubectl get all Create Canary Release Create prod/rollout.yaml with the patch configurations to update Replicas Count Strategy as Filename: prod/rollout.yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: vote spec: replicas: 5 strategy: blueGreen: null canary: steps: - setWeight: 20 - pause: duration: 10s - setWeight: 40 - pause: duration: 10s - setWeight: 60 - pause: duration: 10s - setWeight: 80 - pause: duration: 10s - setWeight: 100 add this rollout overlay spec to prod/kustomization.yaml in patches section as: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base namespace: prod commonAnnotations: supported-by: sre@example.com labels: - includeSelectors: false pairs: project: instavote patches: - path: service.yaml - path: preview-service.yaml - path: rollout.yaml If you have kustomize installed, verify the configs from argo-labs as kustomize build prod apply kubectl apply -k prod/ validate kubectl get all,ro If you have the Argo Rollouts Dashboard, switch the namespace from top right corner to prod and check the Canary Release as Before starting rollout start wathing for the following in 3 different terminals [Termina 1] watch kubectl get ro,all --show-labels [Terminal 2] watch kubectl describe svc vote [Terminal 3] watch kubectl describe svc vote-preview You could also keep monitoring the Argo Rollouts Dashboard. Launch it if required as kubectl argo rollouts dashboard -p 3100 Trigger a new rollout by modifying base/rollouts.yaml file with new image tag as spec: containers: - image: schoolofdevops/vote:v2 and apply kubectl apply -k prod kubectl argo rollouts status vote Here you could see the progressive canary in action, implmenting it step by step, ultimately rolling out the new version. A new replicaset is created to maintain the canary deployment during the rollout. Based on the weightage set in the strategy, proportionate number of pods are maintained for each replicaSet. Gradually, all pods are replaced with new version, shifting 100% traffic to it. here is the output of the rollout status command above [sample output] Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Progressing - updated replicas are still becoming available Progressing - waiting for all steps to complete Healthy Getting Ready to add Traffic Management - Set up Nginx Ingress Controller Install helm to setup Nginx Ingress Controller. To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version Launch Nginx Ingress controller using helm as : helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace \\ --set controller.hostPort.enabled=true \\ --set controller.hostPort.ports.http=80 \\ --set-string controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\ --set-string controller.nodeSelector.ingress-ready=\"true\" Check the pod for Nginx Ingress, if its running kubectl get pods -n ingress-nginx You may see the pod in pending state. Check why its pending by describing it. Once you descibe, you will see that its pending because it has a nodeSelector defined which is looking for a node with label set to ingress-ready=\"true\" . Check the label on the current nodes kubectl get nodes --show-labels Add this lable to first of the worker nodes as kubectl label node kind-worker ingress-ready=\"true\" validate kubectl get nodes --show-labels This time you should see the label added, and nginx ingress controller running, which can be validated using kubectl get pods -n ingress-nginx --watch Wait for the container for nginx ingress controller to be up. You could also validate by connecting to the IPADDRESS of your node where the cluster is beeng setup on port 80, where you should see **404 Not Found** error. This is the sign that nginx is set up as a ingress controller and looking for a request with hostname/path defined. Add Ingress Rule with Host based Routing Once you have the ingress controller working, add the following ingress rule File : prod/ingress.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote spec: ingressClassName: nginx rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 add this new manifest resources section of kustomization.yaml so that it gets applied as File: prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base - ingress.yaml and apply with kubectl apply -k prod/ At this time, you would have ingress rule applied. You could validate it using kubectl get ing kubectl describe ing vote Also add the host file configuration as per this lab guide and validate you are able to use http://vote.example.com/ to access vote service via ingress. If you browse to the nginx ingress URI, you should see the app as With this you have successfully set up Nginx Ingress Controller in front of your prod app and are ready to use Traffic Management features of Argo Rollouts. Canary with Traffic Routing Read this document to understand the need for traffic routing. You could set up the traffic routing rules with Nginx by modifying the rollout spec as File : prod/rollout.yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: vote spec: replicas: 5 strategy: blueGreen: null canary: canaryService: vote-preview stableService: vote trafficRouting: nginx: stableIngress: vote steps: - setWeight: 20 - pause: duration: 10s - setWeight: 40 - pause: duration: 10s - setWeight: 60 - pause: duration: 10s - setWeight: 80 - pause: duration: 10s - setWeight: 100 You could refer to Nginx Ingress Controller for Traffic Routing document to understand this spec. and apply as kubectl apply -k prod/ Once the new configuration is applied, you could try rolling out a few times by updating the image tag in base/rollout.yaml . You could watch using the same commands as earlier as well as using Argo Dashboard. You could also watch for a new ingress created for canary service created during rollout as kubectl describe ing vote-vote-canary where you will see the weight changing as the release progresses. e.g. when weight is set to 20% Every 2.0s: kubectl describe ing vote-vote-canary argo-01: Tue Jun 4 08:08:10 2024 Name: vote-vote-canary Labels: Namespace: prod Address: Ingress Class: nginx Default backend: Rules: Host Path Backends ---- ---- -------- vote.example.com / vote-preview:80 (10.244.1.18:80) Annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-weight: 20 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 20s (x2 over 2m) nginx-ingress-controller Scheduled for sync after weight changed to 40% Annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-weight: 40 While you are rolling our a Canary with traffic routing this time, you will observe that While the release is in progress, unlike earlier, the stable/main replicaSet does not reduce proportionate to step capacity percentage. Ingress Controller/Service Mesh, in this case Nginx, does the job of routing between stable and canary versions, not tied to the proportationte number of pods. This is to make sure that, any time there is a need to abort and roll back, 100% capacity is available with the stable version. Try rolling out a few times to understand the nuances of how canary works with nginx ingress controller and traffic routing rules. Publish Changes to Repo Commit all the changes that you have made so far to the repo as: git status git add base/*.yaml git add prod/*.yaml git status git commit -am \"added canary releases for prod\" git push origin main Cleaning Up Once you are done with this lab, clean up the environment with kubectl delete -k prod/","title":"Lab K603 - Canary with Argo Rollouts"},{"location":"argo_rollout_canary/#implementing-canary-release-for-prod","text":"Author: Gourav Shah Publisher: School of Devops Version : v2024.06.03.01","title":"Implementing Canary Release for Prod"},{"location":"argo_rollout_canary/#prepare-prod-environment","text":"Create and switch to prod namespace kubectl create ns prod kubectl get ns kubectl config set-context --current --namespace=prod validate kubectl config get-contexts [sample output] CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kind-kind kind-kind kind-kind prod Create a copy of the kustomization code to set up prod with: cd argo-labs cp -r staging prod update NodePort for prod File : prod/service.yaml apiVersion: v1 kind: Service metadata: name: vote spec: ports: - name: \"80\" nodePort: 30200 port: 80 protocol: TCP targetPort: 80 type: NodePort create previw service kustomization overlay File: prod/preview-service.yaml apiVersion: v1 kind: Service metadata: name: vote-preview spec: ports: - name: \"80\" nodePort: 30300 port: 80 protocol: TCP targetPort: 80 type: NodePort update kustomization with namespace set to prod path for preview-service.yaml added to patches section File: prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base namespace: prod commonAnnotations: supported-by: sre@example.com labels: - includeSelectors: false pairs: project: instavote patches: - path: service.yaml - path: preview-service.yaml check kustomize build prod apply with kubectl apply -k prod/ validate as kubectl get all","title":"Prepare Prod Environment"},{"location":"argo_rollout_canary/#create-canary-release","text":"Create prod/rollout.yaml with the patch configurations to update Replicas Count Strategy as Filename: prod/rollout.yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: vote spec: replicas: 5 strategy: blueGreen: null canary: steps: - setWeight: 20 - pause: duration: 10s - setWeight: 40 - pause: duration: 10s - setWeight: 60 - pause: duration: 10s - setWeight: 80 - pause: duration: 10s - setWeight: 100 add this rollout overlay spec to prod/kustomization.yaml in patches section as: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base namespace: prod commonAnnotations: supported-by: sre@example.com labels: - includeSelectors: false pairs: project: instavote patches: - path: service.yaml - path: preview-service.yaml - path: rollout.yaml If you have kustomize installed, verify the configs from argo-labs as kustomize build prod apply kubectl apply -k prod/ validate kubectl get all,ro If you have the Argo Rollouts Dashboard, switch the namespace from top right corner to prod and check the Canary Release as Before starting rollout start wathing for the following in 3 different terminals [Termina 1] watch kubectl get ro,all --show-labels [Terminal 2] watch kubectl describe svc vote [Terminal 3] watch kubectl describe svc vote-preview You could also keep monitoring the Argo Rollouts Dashboard. Launch it if required as kubectl argo rollouts dashboard -p 3100 Trigger a new rollout by modifying base/rollouts.yaml file with new image tag as spec: containers: - image: schoolofdevops/vote:v2 and apply kubectl apply -k prod kubectl argo rollouts status vote Here you could see the progressive canary in action, implmenting it step by step, ultimately rolling out the new version. A new replicaset is created to maintain the canary deployment during the rollout. Based on the weightage set in the strategy, proportionate number of pods are maintained for each replicaSet. Gradually, all pods are replaced with new version, shifting 100% traffic to it. here is the output of the rollout status command above [sample output] Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Paused - CanaryPauseStep Progressing - more replicas need to be updated Progressing - updated replicas are still becoming available Progressing - waiting for all steps to complete Healthy","title":"Create Canary Release"},{"location":"argo_rollout_canary/#getting-ready-to-add-traffic-management-set-up-nginx-ingress-controller","text":"Install helm to setup Nginx Ingress Controller. To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version Launch Nginx Ingress controller using helm as : helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace \\ --set controller.hostPort.enabled=true \\ --set controller.hostPort.ports.http=80 \\ --set-string controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\ --set-string controller.nodeSelector.ingress-ready=\"true\" Check the pod for Nginx Ingress, if its running kubectl get pods -n ingress-nginx You may see the pod in pending state. Check why its pending by describing it. Once you descibe, you will see that its pending because it has a nodeSelector defined which is looking for a node with label set to ingress-ready=\"true\" . Check the label on the current nodes kubectl get nodes --show-labels Add this lable to first of the worker nodes as kubectl label node kind-worker ingress-ready=\"true\" validate kubectl get nodes --show-labels This time you should see the label added, and nginx ingress controller running, which can be validated using kubectl get pods -n ingress-nginx --watch Wait for the container for nginx ingress controller to be up. You could also validate by connecting to the IPADDRESS of your node where the cluster is beeng setup on port 80, where you should see **404 Not Found** error. This is the sign that nginx is set up as a ingress controller and looking for a request with hostname/path defined.","title":"Getting Ready to add Traffic Management - Set up Nginx Ingress Controller"},{"location":"argo_rollout_canary/#add-ingress-rule-with-host-based-routing","text":"Once you have the ingress controller working, add the following ingress rule File : prod/ingress.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote spec: ingressClassName: nginx rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 add this new manifest resources section of kustomization.yaml so that it gets applied as File: prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base - ingress.yaml and apply with kubectl apply -k prod/ At this time, you would have ingress rule applied. You could validate it using kubectl get ing kubectl describe ing vote Also add the host file configuration as per this lab guide and validate you are able to use http://vote.example.com/ to access vote service via ingress. If you browse to the nginx ingress URI, you should see the app as With this you have successfully set up Nginx Ingress Controller in front of your prod app and are ready to use Traffic Management features of Argo Rollouts.","title":"Add Ingress Rule with Host based Routing"},{"location":"argo_rollout_canary/#canary-with-traffic-routing","text":"Read this document to understand the need for traffic routing. You could set up the traffic routing rules with Nginx by modifying the rollout spec as File : prod/rollout.yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: vote spec: replicas: 5 strategy: blueGreen: null canary: canaryService: vote-preview stableService: vote trafficRouting: nginx: stableIngress: vote steps: - setWeight: 20 - pause: duration: 10s - setWeight: 40 - pause: duration: 10s - setWeight: 60 - pause: duration: 10s - setWeight: 80 - pause: duration: 10s - setWeight: 100 You could refer to Nginx Ingress Controller for Traffic Routing document to understand this spec. and apply as kubectl apply -k prod/ Once the new configuration is applied, you could try rolling out a few times by updating the image tag in base/rollout.yaml . You could watch using the same commands as earlier as well as using Argo Dashboard. You could also watch for a new ingress created for canary service created during rollout as kubectl describe ing vote-vote-canary where you will see the weight changing as the release progresses. e.g. when weight is set to 20% Every 2.0s: kubectl describe ing vote-vote-canary argo-01: Tue Jun 4 08:08:10 2024 Name: vote-vote-canary Labels: Namespace: prod Address: Ingress Class: nginx Default backend: Rules: Host Path Backends ---- ---- -------- vote.example.com / vote-preview:80 (10.244.1.18:80) Annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-weight: 20 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 20s (x2 over 2m) nginx-ingress-controller Scheduled for sync after weight changed to 40% Annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-weight: 40 While you are rolling our a Canary with traffic routing this time, you will observe that While the release is in progress, unlike earlier, the stable/main replicaSet does not reduce proportionate to step capacity percentage. Ingress Controller/Service Mesh, in this case Nginx, does the job of routing between stable and canary versions, not tied to the proportationte number of pods. This is to make sure that, any time there is a need to abort and roll back, 100% capacity is available with the stable version. Try rolling out a few times to understand the nuances of how canary works with nginx ingress controller and traffic routing rules.","title":"Canary with Traffic Routing"},{"location":"argo_rollout_canary/#publish-changes-to-repo","text":"Commit all the changes that you have made so far to the repo as: git status git add base/*.yaml git add prod/*.yaml git status git commit -am \"added canary releases for prod\" git push origin main","title":"Publish Changes to Repo"},{"location":"argo_rollout_canary/#cleaning-up","text":"Once you are done with this lab, clean up the environment with kubectl delete -k prod/","title":"Cleaning Up"},{"location":"argo_workflow_ci/","text":"Argo Workflows Author: Gourav Shah Publisher: School of Devops Version : v2024.06.03.01 Project: : Set up a Continuous Integration Pipeline with Argo Workflow. Setting up Argo Workflow Signup and Login to Killercoda From Argo Workflows by argoproj launch Workflow Examples scenario This will take a few minutes for the scenario to be ready with Argo Workflow installed Once set up, click on START and then run the command under View the server UI Select Click here to access the UI to see th Workflow Dashboard as follows Building CI Pipeline with Argo Workflow Before you begin, fork the repository containing source code for vote service on to your account. You are going to use this repository to set up the CI Pipeline with. You are going to set up Argo Workflow which will build a CI Pipeline for you. This workflow will have the following stpes/stages clone - Clones the source code from Git and store it in a volume which is available to all subsequent steps. build - Build the application. In case of this python flask app, its just about checking if the dependencies are being built/instlled with python-pip. test - Run unit tests with python nose testing framework. imagebuild - Uses kaniko to build and publish container image. This step will require you to provide credentials to container registry. Create a secret with your container registry credentials which is then used in imagebuild step of the workflow as described above with: kubectl create secret -n argo docker-registry docker-registry-creds \\ --docker-server=https://index.docker.io/v1/ \\ --docker-username=xxxx --docker-password=yyyy where replace, xxxx with registry username yyyy with registry access token Instead of providing your password for --docker-password , it is recommended that you create a access token. For Docker Hub, you could do that as follows: Sign in to Docker Hub From top right corner where you see your profile picture, select Account settings From Security -> Access Tokens select New Access Token Add a Description/Name and set access permissions to Read & Write Proceed to Generate the token, and copy it to a safe location. Keep it handy as you are going to need this a few times throughout this course. Once you have the token, proceed to create the secret with the command provided above. Once you create the secret, create the Argo Workflow with all the necessary steps as described earlier. File : vote-ci-workflow.yaml apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: vote-ci- spec: entrypoint: main arguments: parameters: - name: repo-url value: \"https://github.com/xxxxxx/vote.git\" - name: branch value: \"main\" - name: image value: \"yyyyyy/vote\" - name: dockerfile value: \"Dockerfile\" volumeClaimTemplates: - metadata: name: workspace spec: accessModes: [\"ReadWriteOnce\"] resources: requests: storage: 100Mi volumes: - name: docker-config secret: secretName: docker-registry-creds items: - key: .dockerconfigjson path: config.json templates: - name: main inputs: parameters: - name: repo-url - name: branch - name: image - name: dockerfile steps: - - name: clone template: clone arguments: parameters: - name: repo-url value: \"{{inputs.parameters.repo-url}}\" - name: branch value: \"{{inputs.parameters.branch}}\" - - name: build template: build - - name: test template: test - - name: imagebuild template: imagebuild arguments: parameters: - name: commit-sha value: \"{{steps.clone.outputs.parameters.commit-sha}}\" - name: image value: \"{{inputs.parameters.image}}\" - name: dockerfile value: \"{{inputs.parameters.dockerfile}}\" # Clone task - name: clone inputs: parameters: - name: repo-url - name: branch script: image: alpine/git command: [sh] source: | #!/bin/sh git clone --branch {{inputs.parameters.branch}} {{inputs.parameters.repo-url}} /workspace cd /workspace COMMIT_SHA=$(git rev-parse --short HEAD) echo $COMMIT_SHA > /workspace/commit-sha.txt volumeMounts: - name: workspace mountPath: /workspace outputs: parameters: - name: commit-sha valueFrom: path: /workspace/commit-sha.txt # Build task - name: build script: image: python:3.9 command: [\"sh\"] source: | #!/bin/sh cd /workspace pip install -r requirements.txt volumeMounts: - name: workspace mountPath: /workspace # Test task - name: test script: image: python:3.9 command: [\"sh\"] source: | #!/bin/sh cd /workspace pip install nose nosetests volumeMounts: - name: workspace mountPath: /workspace # Image build and publish task using Kaniko - name: imagebuild inputs: parameters: - name: commit-sha - name: image - name: dockerfile container: image: gcr.io/kaniko-project/executor:latest command: [\"/kaniko/executor\"] args: - --dockerfile=/workspace/{{inputs.parameters.dockerfile}} - --context=/workspace - --destination={{inputs.parameters.image}}:{{inputs.parameters.commit-sha}} - --force volumeMounts: - name: workspace mountPath: /workspace - name: docker-config mountPath: /kaniko/.docker env: - name: DOCKER_CONFIG value: /kaniko/.docker create a workflow by providing your own repo and image tag and start watching it using the following command: argo submit -n argo --watch vote-ci-workflow.yaml \\ -p repo-url=https://github.com/xxxxxx/vote.git \\ -p branch=main \\ -p image=yyyyyy/vote \\ -p dockerfile=Dockerfile where, Replace xxxxxx with approapriate repo URL Replace yyyyyy with your docker hub user id. Update the repo name as necessary. you could also watch the pods watch kubectl get pods -n argo and using dashboard as If you were watching the workflow here is the sample output Name: vote-ci-x5hzc Namespace: argo ServiceAccount: argo Status: Succeeded Conditions: PodRunning False Completed True Created: Tue Jun 04 09:01:06 +0000 (2 minutes ago) Started: Tue Jun 04 09:01:06 +0000 (2 minutes ago) Finished: Tue Jun 04 09:03:50 +0000 (now) Duration: 2 minutes 44 seconds Progress: 4/4 ResourcesDuration: 12s*(1 cpu),2m33s*(100Mi memory) Parameters: repo-url: https://github.com/devops-0001/vote.git branch: master image: initcron/flask-app dockerfile: Dockerfile STEP TEMPLATE PODNAME DURATION MESSAGE \u2714 vote-ci-x5hzc main \u251c\u2500\u2500\u2500\u2714 clone clone vote-ci-x5hzc-clone-2858201196 34s \u251c\u2500\u2500\u2500\u2714 build build vote-ci-x5hzc-build-959094096 47s \u251c\u2500\u2500\u2500\u2714 test test vote-ci-x5hzc-test-1680485113 10s \u2514\u2500\u2500\u2500\u2714 imagebuild imagebuild vote-ci-x5hzc-imagebuild-1986147349 43s if you broese to DockerHub account, you should see a new image tag published as a result of the argo workflow. Summary With this lab you learnt how to set up a simple Continuous Integration Pipeline using Argo Workflows. This pipeline runs a sequence of jobs including build, test and container image build and publish. The result of this pipeline is a new container image available on the registry. This can be further iterated over to create conditionl logic, parallel execution of steps etc. with Argo Workflow.","title":"Lab K605 - Argo Workflows"},{"location":"argo_workflow_ci/#argo-workflows","text":"Author: Gourav Shah Publisher: School of Devops Version : v2024.06.03.01 Project: : Set up a Continuous Integration Pipeline with Argo Workflow.","title":"Argo Workflows"},{"location":"argo_workflow_ci/#setting-up-argo-workflow","text":"Signup and Login to Killercoda From Argo Workflows by argoproj launch Workflow Examples scenario This will take a few minutes for the scenario to be ready with Argo Workflow installed Once set up, click on START and then run the command under View the server UI Select Click here to access the UI to see th Workflow Dashboard as follows","title":"Setting up Argo Workflow"},{"location":"argo_workflow_ci/#building-ci-pipeline-with-argo-workflow","text":"Before you begin, fork the repository containing source code for vote service on to your account. You are going to use this repository to set up the CI Pipeline with. You are going to set up Argo Workflow which will build a CI Pipeline for you. This workflow will have the following stpes/stages clone - Clones the source code from Git and store it in a volume which is available to all subsequent steps. build - Build the application. In case of this python flask app, its just about checking if the dependencies are being built/instlled with python-pip. test - Run unit tests with python nose testing framework. imagebuild - Uses kaniko to build and publish container image. This step will require you to provide credentials to container registry. Create a secret with your container registry credentials which is then used in imagebuild step of the workflow as described above with: kubectl create secret -n argo docker-registry docker-registry-creds \\ --docker-server=https://index.docker.io/v1/ \\ --docker-username=xxxx --docker-password=yyyy where replace, xxxx with registry username yyyy with registry access token Instead of providing your password for --docker-password , it is recommended that you create a access token. For Docker Hub, you could do that as follows: Sign in to Docker Hub From top right corner where you see your profile picture, select Account settings From Security -> Access Tokens select New Access Token Add a Description/Name and set access permissions to Read & Write Proceed to Generate the token, and copy it to a safe location. Keep it handy as you are going to need this a few times throughout this course. Once you have the token, proceed to create the secret with the command provided above. Once you create the secret, create the Argo Workflow with all the necessary steps as described earlier. File : vote-ci-workflow.yaml apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: vote-ci- spec: entrypoint: main arguments: parameters: - name: repo-url value: \"https://github.com/xxxxxx/vote.git\" - name: branch value: \"main\" - name: image value: \"yyyyyy/vote\" - name: dockerfile value: \"Dockerfile\" volumeClaimTemplates: - metadata: name: workspace spec: accessModes: [\"ReadWriteOnce\"] resources: requests: storage: 100Mi volumes: - name: docker-config secret: secretName: docker-registry-creds items: - key: .dockerconfigjson path: config.json templates: - name: main inputs: parameters: - name: repo-url - name: branch - name: image - name: dockerfile steps: - - name: clone template: clone arguments: parameters: - name: repo-url value: \"{{inputs.parameters.repo-url}}\" - name: branch value: \"{{inputs.parameters.branch}}\" - - name: build template: build - - name: test template: test - - name: imagebuild template: imagebuild arguments: parameters: - name: commit-sha value: \"{{steps.clone.outputs.parameters.commit-sha}}\" - name: image value: \"{{inputs.parameters.image}}\" - name: dockerfile value: \"{{inputs.parameters.dockerfile}}\" # Clone task - name: clone inputs: parameters: - name: repo-url - name: branch script: image: alpine/git command: [sh] source: | #!/bin/sh git clone --branch {{inputs.parameters.branch}} {{inputs.parameters.repo-url}} /workspace cd /workspace COMMIT_SHA=$(git rev-parse --short HEAD) echo $COMMIT_SHA > /workspace/commit-sha.txt volumeMounts: - name: workspace mountPath: /workspace outputs: parameters: - name: commit-sha valueFrom: path: /workspace/commit-sha.txt # Build task - name: build script: image: python:3.9 command: [\"sh\"] source: | #!/bin/sh cd /workspace pip install -r requirements.txt volumeMounts: - name: workspace mountPath: /workspace # Test task - name: test script: image: python:3.9 command: [\"sh\"] source: | #!/bin/sh cd /workspace pip install nose nosetests volumeMounts: - name: workspace mountPath: /workspace # Image build and publish task using Kaniko - name: imagebuild inputs: parameters: - name: commit-sha - name: image - name: dockerfile container: image: gcr.io/kaniko-project/executor:latest command: [\"/kaniko/executor\"] args: - --dockerfile=/workspace/{{inputs.parameters.dockerfile}} - --context=/workspace - --destination={{inputs.parameters.image}}:{{inputs.parameters.commit-sha}} - --force volumeMounts: - name: workspace mountPath: /workspace - name: docker-config mountPath: /kaniko/.docker env: - name: DOCKER_CONFIG value: /kaniko/.docker create a workflow by providing your own repo and image tag and start watching it using the following command: argo submit -n argo --watch vote-ci-workflow.yaml \\ -p repo-url=https://github.com/xxxxxx/vote.git \\ -p branch=main \\ -p image=yyyyyy/vote \\ -p dockerfile=Dockerfile where, Replace xxxxxx with approapriate repo URL Replace yyyyyy with your docker hub user id. Update the repo name as necessary. you could also watch the pods watch kubectl get pods -n argo and using dashboard as If you were watching the workflow here is the sample output Name: vote-ci-x5hzc Namespace: argo ServiceAccount: argo Status: Succeeded Conditions: PodRunning False Completed True Created: Tue Jun 04 09:01:06 +0000 (2 minutes ago) Started: Tue Jun 04 09:01:06 +0000 (2 minutes ago) Finished: Tue Jun 04 09:03:50 +0000 (now) Duration: 2 minutes 44 seconds Progress: 4/4 ResourcesDuration: 12s*(1 cpu),2m33s*(100Mi memory) Parameters: repo-url: https://github.com/devops-0001/vote.git branch: master image: initcron/flask-app dockerfile: Dockerfile STEP TEMPLATE PODNAME DURATION MESSAGE \u2714 vote-ci-x5hzc main \u251c\u2500\u2500\u2500\u2714 clone clone vote-ci-x5hzc-clone-2858201196 34s \u251c\u2500\u2500\u2500\u2714 build build vote-ci-x5hzc-build-959094096 47s \u251c\u2500\u2500\u2500\u2714 test test vote-ci-x5hzc-test-1680485113 10s \u2514\u2500\u2500\u2500\u2714 imagebuild imagebuild vote-ci-x5hzc-imagebuild-1986147349 43s if you broese to DockerHub account, you should see a new image tag published as a result of the argo workflow.","title":"Building CI Pipeline with Argo Workflow"},{"location":"argo_workflow_ci/#summary","text":"With this lab you learnt how to set up a simple Continuous Integration Pipeline using Argo Workflows. This pipeline runs a sequence of jobs including build, test and container image build and publish. The result of this pipeline is a new container image available on the registry. This can be further iterated over to create conditionl logic, parallel execution of steps etc. with Argo Workflow.","title":"Summary"},{"location":"aws_ack_demo/","text":"AWS Controller for Kubernetes (ACK ) Demo From AWS Console, create user with FullS3Access. Note down the Access/Secret keys. Add these credentials as kubernetes secret in a new namespace as kubectl create namespace ack-system kubectl create secret generic aws-creds --from-literal=key=XXXX --from-literal=secret=YYYY --namespace ack-system Setup S3 Controller for ACK You could find the ACK Conrtrollers from ECR Public Gallery Example is S# Cotroller : https://gallery.ecr.aws/aws-controllers-k8s/s3-controller Code for this is available at https://github.com/aws-controllers-k8s/s3-controlle Clone the git repo which contains the helm chart for the controller as git clone https://github.com/aws-controllers-k8s/s3-controller Add the configuration cd s3-controller/helm/ edit values.yaml and approx at line 62 add the following config extraEnvVars: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: aws-creds key: key - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: aws-creds key: secret also at line 98 add the region as aws: # If specified, use the AWS region for AWS API calls region: \"us-east-1\" install helm chart as helm install -n ack-system ack-s3 . and validate kubectl get all -n ack-system Create S3 Bucket from Kubernetes List the CRDs installed by the controller kubectl get crds | grep s3 kubectl get buckets kubectl explain buckets Write a Custom Resource to create a bucket with as apiVersion: s3.services.k8s.aws/v1alpha1 kind: Bucket metadata: name: my-ack-s3-bucket spec: name: my-ack-s3-bucket-xxxxxx # S3 bucket names need to be globally unique where replace xxxxxx with some unique number and apply kubectl apply -f my-bucket.yaml and like a magic, you shall see a s3 bucket created on AWS. you could also explore kubectl get buckets kubectl describe bucket my-ack-s3-bucket and finally kubectl delete bucket my-ack-s3-bucket to see it gone from AWS as well \u2026. poof !","title":"Lab K405 - AWS Controller for Kubernetes"},{"location":"aws_ack_demo/#aws-controller-for-kubernetes-ack-demo","text":"From AWS Console, create user with FullS3Access. Note down the Access/Secret keys. Add these credentials as kubernetes secret in a new namespace as kubectl create namespace ack-system kubectl create secret generic aws-creds --from-literal=key=XXXX --from-literal=secret=YYYY --namespace ack-system","title":"AWS Controller for Kubernetes (ACK ) Demo"},{"location":"aws_ack_demo/#setup-s3-controller-for-ack","text":"You could find the ACK Conrtrollers from ECR Public Gallery Example is S# Cotroller : https://gallery.ecr.aws/aws-controllers-k8s/s3-controller Code for this is available at https://github.com/aws-controllers-k8s/s3-controlle Clone the git repo which contains the helm chart for the controller as git clone https://github.com/aws-controllers-k8s/s3-controller Add the configuration cd s3-controller/helm/ edit values.yaml and approx at line 62 add the following config extraEnvVars: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: aws-creds key: key - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: aws-creds key: secret also at line 98 add the region as aws: # If specified, use the AWS region for AWS API calls region: \"us-east-1\" install helm chart as helm install -n ack-system ack-s3 . and validate kubectl get all -n ack-system","title":"Setup S3 Controller for ACK"},{"location":"aws_ack_demo/#create-s3-bucket-from-kubernetes","text":"List the CRDs installed by the controller kubectl get crds | grep s3 kubectl get buckets kubectl explain buckets Write a Custom Resource to create a bucket with as apiVersion: s3.services.k8s.aws/v1alpha1 kind: Bucket metadata: name: my-ack-s3-bucket spec: name: my-ack-s3-bucket-xxxxxx # S3 bucket names need to be globally unique where replace xxxxxx with some unique number and apply kubectl apply -f my-bucket.yaml and like a magic, you shall see a s3 bucket created on AWS. you could also explore kubectl get buckets kubectl describe bucket my-ack-s3-bucket and finally kubectl delete bucket my-ack-s3-bucket to see it gone from AWS as well \u2026. poof !","title":"Create S3 Bucket from Kubernetes"},{"location":"base_setup/","text":"Base Setup Skip this step if using a pre configured lab environment Pick Ubuntu nodes which would be part of this cluster. Then download and run this script to set up the nodes and prepare those to install kubernetes.","title":"Base Setup"},{"location":"base_setup/#base-setup","text":"Skip this step if using a pre configured lab environment Pick Ubuntu nodes which would be part of this cluster. Then download and run this script to set up the nodes and prepare those to install kubernetes.","title":"Base Setup"},{"location":"building-publishing-docker-images/","text":"Lab : Build a docker image for Instavote frontend vote app Voteapp is a app written in python. Its a simple, web based application which serves as a frontend for Instavote project. As a devops engineer, you have been tasked with building an image for vote app and publish it to docker hub registry. Approach 1: Building docker image for voteapp manually on the host git clone https://github.com/schoolofdevops/vote docker container run -idt --name dev -p 8000:80 python:alpine3.17 sh cd vote docker cp . dev:/app docker exec -it dev sh inside the container cd /app pip install -r requirements.txt gunicorn app:app -b 0.0.0.0:80 Validate by accessing http://IPADDRESS:8000 on the host docker diff dev docker container commit dev docker.io//vote:v1 [where should be replaced with your registry username ] docker login docker image push docker.io//vote:v1 Approach 2: Building image with Dockerfile Change into vote directory which containts the source code. This assumes you have already cloned the repo. If not, clone it from https://github.com/schoolofdevops/vote cd vote ls app.py requirements.txt static templates Add/create Dockerfile the the same directory (vote) with the following content, FROM python:alpine3.17 WORKDIR /app COPY . . RUN pip install -r requirements.txt EXPOSE 80 CMD gunicorn app:app -b 0.0.0.0:80 Build image using, docker build -t docker.io//vote:v2 . where, : your docker registry user/namespace. Replace this with the actual user validate docker image ls docker image history docker.io//vote:v2 docker image history docker.io//vote:v1 docker container run -idt -P docker.io//vote:v2 docker ps Check by connecting to your host:port to validate if vote web application shows up. Once validated, tag and push docker image tag docker.io//vote:v2 docker.io//vote:latest docker login docker push docker.io//vote docker push docker.io//vote:v2","title":"Lab D102 - Building and Publishing Docker Images"},{"location":"building-publishing-docker-images/#lab-build-a-docker-image-for-instavote-frontend-vote-app","text":"Voteapp is a app written in python. Its a simple, web based application which serves as a frontend for Instavote project. As a devops engineer, you have been tasked with building an image for vote app and publish it to docker hub registry.","title":"Lab : Build a docker image for Instavote frontend vote app"},{"location":"building-publishing-docker-images/#approach-1-building-docker-image-for-voteapp-manually","text":"on the host git clone https://github.com/schoolofdevops/vote docker container run -idt --name dev -p 8000:80 python:alpine3.17 sh cd vote docker cp . dev:/app docker exec -it dev sh inside the container cd /app pip install -r requirements.txt gunicorn app:app -b 0.0.0.0:80 Validate by accessing http://IPADDRESS:8000 on the host docker diff dev docker container commit dev docker.io//vote:v1 [where should be replaced with your registry username ] docker login docker image push docker.io//vote:v1","title":"Approach 1: Building docker image for voteapp manually"},{"location":"building-publishing-docker-images/#approach-2-building-image-with-dockerfile","text":"Change into vote directory which containts the source code. This assumes you have already cloned the repo. If not, clone it from https://github.com/schoolofdevops/vote cd vote ls app.py requirements.txt static templates Add/create Dockerfile the the same directory (vote) with the following content, FROM python:alpine3.17 WORKDIR /app COPY . . RUN pip install -r requirements.txt EXPOSE 80 CMD gunicorn app:app -b 0.0.0.0:80 Build image using, docker build -t docker.io//vote:v2 . where, : your docker registry user/namespace. Replace this with the actual user validate docker image ls docker image history docker.io//vote:v2 docker image history docker.io//vote:v1 docker container run -idt -P docker.io//vote:v2 docker ps Check by connecting to your host:port to validate if vote web application shows up. Once validated, tag and push docker image tag docker.io//vote:v2 docker.io//vote:latest docker login docker push docker.io//vote docker push docker.io//vote:v2","title":"Approach 2: Building image with Dockerfile"},{"location":"cluster-administration/","text":"Lab K208 - Kubernetes Cluster Administration Defining Quotas Create and switch to a new staging namespace. config get-contexts kubectl create namespace staging kubectl config set-context --current --namespace=staging config get-contexts Define quota file: staging-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: staging namespace: staging spec: hard: requests.cpu: \"0.5\" requests.memory: 500Mi limits.cpu: \"2\" limits.memory: 2Gi count/deployments.apps: 1 kubectl get quota -n staging kubectl apply -f staging-quota.yaml kubectl get quota -n staging kubectl describe quota file: nginx-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: staging spec: replicas: 2 selector: matchLabels: app: web template: metadata: name: nginx labels: app: web spec: containers: - name: nginx image: nginx resources: limits: memory: \"500Mi\" cpu: \"500m\" requests: memory: \"200Mi\" cpu: \"200m\" kubectl apply -f nginx-deploy.yaml kubectl describe quota -n staging Lets now try to scale up the deployment and observe. kubectl scale deploy nginx --replicas=4 kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE nginx 2/4 2 2 2m55s What happened ? Even though deployment updated the number of desired replicas, only 2 are available Deployment calls replicaset to launch new replicas. If you describe the replicaset it throws an error related to quota being exceeded. e.g. # kubectl get rs NAME DESIRED CURRENT READY AGE nginx-56c479cd4f 4 2 2 5m4s # kubectl describe rs nginx-56c479cd4f Warning FailedCreate 34s (x5 over 73s) replicaset-controller (combined from similar events): Error creating: pods \"nginx-56c479cd4f-kwf9h\" is forbidden: exceeded quota: staging, requested: requests.cpu=200m,requests.memory=200Mi, used: requests.cpu=400m,requests.memory=400Mi, limited: requests.cpu=500m,requests.memory=500Mi You just configured resource quota based on a namespace. Now, switch back your namespace to instavote or the the one you were using before the beginning of this lab. kubectl config set-context --current --namespace=instavote kubectl config get-contexts Nodes Maintenance You could isolate a problematic node for further troubleshooting by cordonning it off. You could also drain it while preparing for maintenance. Cordon a Node kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE db-66496667c9-qggzd 1/1 Running 0 5h 10.233.74.74 node4 redis-5bf748dbcf-ckn65 1/1 Running 0 42m 10.233.71.26 node3 redis-5bf748dbcf-vxppx 1/1 Running 0 1h 10.233.74.79 node4 result-5c7569bcb7-4fptr 1/1 Running 0 5h 10.233.71.18 node3 result-5c7569bcb7-s4rdx 1/1 Running 0 5h 10.233.74.75 node4 vote-56bf599b9c-22lpw 1/1 Running 0 1h 10.233.74.80 node4 vote-56bf599b9c-4l6bc 1/1 Running 0 50m 10.233.74.83 node4 vote-56bf599b9c-bqsrq 1/1 Running 0 50m 10.233.74.82 node4 vote-56bf599b9c-xw7zc 1/1 Running 0 50m 10.233.74.81 node4 worker-6cc8dbd4f8-6bkfg 1/1 Running 0 39m 10.233.75.15 node2 Lets cordon one of the nodes and observe. kubectl cordon node4 node/node4 cordoned Observe the changes $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE db-66496667c9-qggzd 1/1 Running 0 5h 10.233.74.74 node4 redis-5bf748dbcf-ckn65 1/1 Running 0 43m 10.233.71.26 node3 redis-5bf748dbcf-vxppx 1/1 Running 0 1h 10.233.74.79 node4 result-5c7569bcb7-4fptr 1/1 Running 0 5h 10.233.71.18 node3 result-5c7569bcb7-s4rdx 1/1 Running 0 5h 10.233.74.75 node4 vote-56bf599b9c-22lpw 1/1 Running 0 1h 10.233.74.80 node4 vote-56bf599b9c-4l6bc 1/1 Running 0 51m 10.233.74.83 node4 vote-56bf599b9c-bqsrq 1/1 Running 0 51m 10.233.74.82 node4 vote-56bf599b9c-xw7zc 1/1 Running 0 51m 10.233.74.81 node4 worker-6cc8dbd4f8-6bkfg 1/1 Running 0 40m 10.233.75.15 node2 $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node2 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 node3 Ready node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node4 Ready,SchedulingDisabled node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 Now launch a new deployment and scale it. kubectl create deployment cordontest --image=busybox --replicas=5 kubectl scale deploy cordontest --replicas=5 kubectl get pods -o wide what happened ? New pods scheduled due the deployment above, do not get launched on the node which is been cordoned off. $ kubectl uncordon node4 node/node4 uncordoned $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node2 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 node3 Ready node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node4 Ready node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 delete the test deployment kubectl delete deploy cordontest Drain a Node Draining a node will not only mark it unschedulable but also will evict existing pods running on it. Use it with care. $ kubectl drain node3 node/node3 cordoned error: unable to drain node \"node3\", aborting command... There are pending nodes to be drained: node3 error: pods with local storage (use --delete-local-data to override): kubernetes-dashboard-55fdfd74b4-jdgch; DaemonSet-managed pods (use --ignore-daemonsets to ignore): calico-node-4f8xc Drain with options kubectl drain node3 --delete-local-data --ignore-daemonsets Observe the effect, kubectl get pods -o wide kubectl get nodes -o wide To add the node back to the available schedulable node pool, kubectl uncordon node4 node/node4 uncordoned Summary In this lab, we learnt about limiting resource by defining per namespace quota, as well as learnt how to prepare nodes for maintenance by cordoning and draining it.","title":"Lab K304 - Cluster Administration"},{"location":"cluster-administration/#lab-k208-kubernetes-cluster-administration","text":"","title":"Lab K208 - Kubernetes Cluster Administration"},{"location":"cluster-administration/#defining-quotas","text":"Create and switch to a new staging namespace. config get-contexts kubectl create namespace staging kubectl config set-context --current --namespace=staging config get-contexts Define quota file: staging-quota.yaml apiVersion: v1 kind: ResourceQuota metadata: name: staging namespace: staging spec: hard: requests.cpu: \"0.5\" requests.memory: 500Mi limits.cpu: \"2\" limits.memory: 2Gi count/deployments.apps: 1 kubectl get quota -n staging kubectl apply -f staging-quota.yaml kubectl get quota -n staging kubectl describe quota file: nginx-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: staging spec: replicas: 2 selector: matchLabels: app: web template: metadata: name: nginx labels: app: web spec: containers: - name: nginx image: nginx resources: limits: memory: \"500Mi\" cpu: \"500m\" requests: memory: \"200Mi\" cpu: \"200m\" kubectl apply -f nginx-deploy.yaml kubectl describe quota -n staging Lets now try to scale up the deployment and observe. kubectl scale deploy nginx --replicas=4 kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE nginx 2/4 2 2 2m55s What happened ? Even though deployment updated the number of desired replicas, only 2 are available Deployment calls replicaset to launch new replicas. If you describe the replicaset it throws an error related to quota being exceeded. e.g. # kubectl get rs NAME DESIRED CURRENT READY AGE nginx-56c479cd4f 4 2 2 5m4s # kubectl describe rs nginx-56c479cd4f Warning FailedCreate 34s (x5 over 73s) replicaset-controller (combined from similar events): Error creating: pods \"nginx-56c479cd4f-kwf9h\" is forbidden: exceeded quota: staging, requested: requests.cpu=200m,requests.memory=200Mi, used: requests.cpu=400m,requests.memory=400Mi, limited: requests.cpu=500m,requests.memory=500Mi You just configured resource quota based on a namespace. Now, switch back your namespace to instavote or the the one you were using before the beginning of this lab. kubectl config set-context --current --namespace=instavote kubectl config get-contexts","title":"Defining Quotas"},{"location":"cluster-administration/#nodes-maintenance","text":"You could isolate a problematic node for further troubleshooting by cordonning it off. You could also drain it while preparing for maintenance.","title":"Nodes Maintenance"},{"location":"cluster-administration/#cordon-a-node","text":"kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE db-66496667c9-qggzd 1/1 Running 0 5h 10.233.74.74 node4 redis-5bf748dbcf-ckn65 1/1 Running 0 42m 10.233.71.26 node3 redis-5bf748dbcf-vxppx 1/1 Running 0 1h 10.233.74.79 node4 result-5c7569bcb7-4fptr 1/1 Running 0 5h 10.233.71.18 node3 result-5c7569bcb7-s4rdx 1/1 Running 0 5h 10.233.74.75 node4 vote-56bf599b9c-22lpw 1/1 Running 0 1h 10.233.74.80 node4 vote-56bf599b9c-4l6bc 1/1 Running 0 50m 10.233.74.83 node4 vote-56bf599b9c-bqsrq 1/1 Running 0 50m 10.233.74.82 node4 vote-56bf599b9c-xw7zc 1/1 Running 0 50m 10.233.74.81 node4 worker-6cc8dbd4f8-6bkfg 1/1 Running 0 39m 10.233.75.15 node2 Lets cordon one of the nodes and observe. kubectl cordon node4 node/node4 cordoned Observe the changes $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE db-66496667c9-qggzd 1/1 Running 0 5h 10.233.74.74 node4 redis-5bf748dbcf-ckn65 1/1 Running 0 43m 10.233.71.26 node3 redis-5bf748dbcf-vxppx 1/1 Running 0 1h 10.233.74.79 node4 result-5c7569bcb7-4fptr 1/1 Running 0 5h 10.233.71.18 node3 result-5c7569bcb7-s4rdx 1/1 Running 0 5h 10.233.74.75 node4 vote-56bf599b9c-22lpw 1/1 Running 0 1h 10.233.74.80 node4 vote-56bf599b9c-4l6bc 1/1 Running 0 51m 10.233.74.83 node4 vote-56bf599b9c-bqsrq 1/1 Running 0 51m 10.233.74.82 node4 vote-56bf599b9c-xw7zc 1/1 Running 0 51m 10.233.74.81 node4 worker-6cc8dbd4f8-6bkfg 1/1 Running 0 40m 10.233.75.15 node2 $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node2 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 node3 Ready node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node4 Ready,SchedulingDisabled node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 Now launch a new deployment and scale it. kubectl create deployment cordontest --image=busybox --replicas=5 kubectl scale deploy cordontest --replicas=5 kubectl get pods -o wide what happened ? New pods scheduled due the deployment above, do not get launched on the node which is been cordoned off. $ kubectl uncordon node4 node/node4 uncordoned $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node2 Ready master,node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 node3 Ready node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-130-generic docker://17.3.2 node4 Ready node 1d v1.10.4 Ubuntu 16.04.4 LTS 4.4.0-124-generic docker://17.3.2 delete the test deployment kubectl delete deploy cordontest","title":"Cordon a Node"},{"location":"cluster-administration/#drain-a-node","text":"Draining a node will not only mark it unschedulable but also will evict existing pods running on it. Use it with care. $ kubectl drain node3 node/node3 cordoned error: unable to drain node \"node3\", aborting command... There are pending nodes to be drained: node3 error: pods with local storage (use --delete-local-data to override): kubernetes-dashboard-55fdfd74b4-jdgch; DaemonSet-managed pods (use --ignore-daemonsets to ignore): calico-node-4f8xc Drain with options kubectl drain node3 --delete-local-data --ignore-daemonsets Observe the effect, kubectl get pods -o wide kubectl get nodes -o wide To add the node back to the available schedulable node pool, kubectl uncordon node4 node/node4 uncordoned","title":"Drain a Node"},{"location":"cluster-administration/#summary","text":"In this lab, we learnt about limiting resource by defining per namespace quota, as well as learnt how to prepare nodes for maintenance by cordoning and draining it.","title":"Summary"},{"location":"cluster-troubleshooting/","text":"","title":"Lab K305 - Cluster Troubleshooting"},{"location":"cluster_setup_kubespray/","text":"High Available Kubernetes Cluster Setup using Kubespray Kubespray is an Ansible based kubernetes provisioner. It helps us to setup a production grade, highly available and highly scalable Kubernetes cluster. Prerequisites Hardware Pre requisites 4 Nodes: Virtual/Physical Machines Memory: 2GB CPU: 1 Core Hard disk: 20GB available Software Pre Requisites On All Nodes Ubuntu 16.04 OS Python SSH Server Privileged user On Ansible Control Node Ansible version 2.4 or greater Jinja Networking Pre Requisites Internet access to download docker images and install softwares IPv4 Forwarding should be enabled Firewall should allow ssh access as well as ports required by Kubernetes. Internally open all the ports between node. Architecture of a high available kubernetes cluster Preparing the nodes Run instructions in the section On all nodes in the cluster. This includes Ansible controller too. Install Python Ansible needs python to be installed on all the machines. sudo apt update sudo apt install python Enable IPv4 Forwarding On all nodes Enalbe IPv4 forwarding by uncommenting the following line echo \"net.ipv4.ip_forward=1\" >> /etc/sysctl.conf Disable Swap swapoff -a Setup passwordless SSH between ansible controller and kubernetes nodes On control node Ansible uses passwordless ssh 1 to create the cluster. Let us see how to set it up from your control node . Generate ssh keypair if not present already using the following command. ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ubuntu/.ssh/id_rsa. Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub. The key fingerprint is: SHA256:yC4Tl6RYc+saTPcLKFdGlTLOWOIuDgO1my/NrMBnRxA ubuntu@node1 The key's randomart image is: +---[RSA 2048]----+ | E .. | | . o +.. | | . +o*+o | |. .o+Bo+ | |. .++.X S | |+ +ooX . | |.=.OB.+ . | | .=o*= . . | | .o. . | +----[SHA256]-----+ Just leave the fields to defaults. This command will generate a public key and private key for you. Copy over the public key to all nodes. Example, assuming ubuntu as the user which has a privileged access on the node with ip address 10.10.1.101 , ssh-copy-id ubuntu@10.10.1.101 This will copy our newly generated public key to the remote machine. After running this command you will be able to SSH into the machine directly without using a password. Replace 10.40.1.26 with your respective machine's IP. e.g. ssh ubuntu@10.10.1.101 Make sure to copy the public key to all kubernetes nodes. Replace username with the actual user on your system . If the above mentioned command fails, then copy your public key and paste it in the remote machine's ~/.ssh/authorized_keys file. e.g. (Only if ssh-copy-id fails) cat ~/.ssh/id_rsa.pub ssh ubunut@10.10.1.101 vim ~/.ssh/authorized_keys # Paste the public key Setup Ansible Control node and Kubespray On control node Set Locale export LC_ALL=\"en_US.UTF-8\" export LC_CTYPE=\"en_US.UTF-8\" sudo dpkg-reconfigure locales Do no select any other locale in the menu. Just press ( OK ) in the next two screens. Setup kubespray Kubespray is hosted on GitHub. Let us the clone the official repository . git clone https://github.com/kubernetes-incubator/kubespray.git cd kubespray Install Prerequisites Install the python dependencies. This step installs Ansible as well. You do not need to install Ansible separately . sudo apt install python-pip -y sudo pip install -r requirements.txt Set Remote User for Ansible Add the following section in ansible.cfg file remote_user=ubuntu If the user you are going to connect is differnt, use that instead. Your ansible.cfg file should look like this. [ssh_connection] pipelining=True ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null #control_path = ~/.ssh/ansible-%%r@%%h:%%p [defaults] host_key_checking=False gathering = smart fact_caching = jsonfile fact_caching_connection = /tmp stdout_callback = skippy library = ./library callback_whitelist = profile_tasks roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles deprecation_warnings=False remote_user=ubuntu Create Inventory cp -rfp inventory/sample inventory/prod where prod is the custom configuration name. Replace is with whatever name you would like to assign to the current cluster. To build the inventory file, execute the inventory script along with the IP addresses of our cluster as arguments CONFIG_FILE=inventory/prod/hosts.ini python3 contrib/inventory_builder/inventory.py 10.10.1.101 10.10.1.102 10.10.1.103 10.10.1.104 Where replace the IP addresses (e.g. 10.10.1.101) with the actual IPs of your nodes Once its run, you should see an inventory file generated which may look similar to below file: inventory/prod/hosts.ini [all] node1 ansible_host=10.10.1.101 ip=10.10.1.101 node2 ansible_host=10.10.1.102 ip=10.10.1.102 node3 ansible_host=10.10.1.103 ip=10.10.1.103 node4 ansible_host=10.10.1.104 ip=10.10.1.104 [kube-master] node1 node2 [kube-node] node1 node2 node3 node4 [etcd] node1 node2 node3 [k8s-cluster:children] kube-node kube-master [calico-rr] [vault] node1 node2 node3 Customise Kubernetes Cluster Configs There are two configs files in your inventroy directory's group_vars (e.g. inventory/prod/group_vars) viz. all.yml k8s-cluster.yml Ansible is data driven, and most of the configurations of the cluster can be tweaked by changing the variable values from the above files. Few of the configurations you may want to modify file: inventory/prod/group_vars/k8s-cluster.yml kubelet_max_pods: 100 cluster_name: prod helm_enabled: true Provisioning kubernetes cluster with kubespray On control node We are set to provision the cluster. Run the following ansible-playbook command to provision our Kubernetes cluster. ansible-playbook -b -v -i inventory/prod/hosts.ini cluster.yml Option -i = Inventory file path Option -b = Become as root user Option -v = Give verbose output If you face this following error, while running ansible-playbook command, you can fix it by running following instructions ERROR : ERROR! Unexpected Exception, this is probably a bug: (cryptography 1.2.3 (/usr/lib/python3/dist-packages), Requirement.parse('cryptography>=1.5'), {'paramiko'}) FIX : sudo pip install --upgrade pip sudo pip uninstall cryptography sudo pip install cryptography ansible-playbook -b -v -i inventory/prod/hosts.ini cluster.yml This Ansible run will take around 30 mins to complete. Kubectl Configs On kube master node Once the cluster setup is done, copy the configuration and setup the permissions. mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Check the State of the Cluster On the node where kubectl is setup Let us check the state of the cluster by running, kubectl cluster-info Kubernetes master is running at https://10.10.1.101:6443 KubeDNS is running at https://10.10.1.101:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready master,node 21h v1.9.0+coreos.0 node2 Ready master,node 21h v1.9.0+coreos.0 node3 Ready node 21h v1.9.0+coreos.0 node4 Ready node 21h v1.9.0+coreos.0 If you are able to see this, your cluster has been set up successfully. 1 You can use private key / password instead of passwordless ssh. But it requires additional knowledge in using Ansible. Access Kubernetes Cluster Remotely (Optional) On your local machine You could also install kubectl on your laptop/workstation. To learn how to install it for your OS, refer to the procedure here . e.g. To install kubectl on Ubuntu, sudo apt-get update && sudo apt-get install -y apt-transport-https curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - sudo touch /etc/apt/sources.list.d/kubernetes.list echo \"deb http://apt.kubernetes.io/ kubernetes-xenial main\" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubectl Copy kubernetes config to your local machine Copy kubeconfig file to your local machine mkdir ~/.kube scp -r ubuntu@MASTER_HOST_IP:/etc/kubernetes/admin.conf ~/.kube/config kubectl get nodes Deploy Kubernetes Objects Since its a new cluster, which is differnt than what you have created with kubeadm earlier, or if this is the first time you are creating a kubernetes cluster with kubespray as part of Advanced Workshop , you need to deploy services which have been covered as part of the previous topics. In order to do that, use the following commands on the node where you have configured kubectl git clone https://github.com/schoolofdevops/k8s-code.git cd k8s-code/projects/instavote kubectl apply -f instavote-ns.yaml kubectl apply -f prod/ Switch to instavote namespace and validate, kubectl config set-context $(kubectl config current-context) --namespace=instavote kubectl get pods,deploy,svc where, --cluster=prod : prod is the cluter name you created earlier. If not, use the correct name of the cluster ( kubectl config view) --user=admin-prod: is the admin user created by default while installing with kubespray --namespace=instavote : the namespace you just created to deploy instavote app stack [sample output] $ kubectl get pods,deploy,svc NAME READY STATUS RESTARTS AGE pod/db-66496667c9-qggzd 1/1 Running 0 7m pod/redis-6555998885-4k5cr 1/1 Running 0 7m pod/redis-6555998885-fb8rk 1/1 Running 0 7m pod/result-5c7569bcb7-4fptr 1/1 Running 0 7m pod/result-5c7569bcb7-s4rdx 1/1 Running 0 7m pod/vote-5d88d47fc8-gbzbq 1/1 Running 0 7m pod/vote-5d88d47fc8-q4vj6 1/1 Running 0 7m pod/worker-7c98c96fb4-7tzzw 1/1 Running 0 7m NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.extensions/db 1 1 1 1 7m deployment.extensions/redis 2 2 2 2 7m deployment.extensions/result 2 2 2 2 7m deployment.extensions/vote 2 2 2 2 7m deployment.extensions/worker 1 1 1 1 7m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/db ClusterIP 10.233.16.207 5432/TCP 7m service/redis ClusterIP 10.233.14.61 6379/TCP 7m service/result NodePort 10.233.22.10 80:30100/TCP 7m service/vote NodePort 10.233.19.111 80:30000/TCP 7m References Installing Kubernetes On Premises/On Cloud with Kubespray Kubespray on Github","title":"High Available Kubernetes Cluster Setup using Kubespray"},{"location":"cluster_setup_kubespray/#high-available-kubernetes-cluster-setup-using-kubespray","text":"Kubespray is an Ansible based kubernetes provisioner. It helps us to setup a production grade, highly available and highly scalable Kubernetes cluster.","title":"High Available Kubernetes Cluster Setup using Kubespray"},{"location":"cluster_setup_kubespray/#prerequisites","text":"","title":"Prerequisites"},{"location":"cluster_setup_kubespray/#hardware-pre-requisites","text":"4 Nodes: Virtual/Physical Machines Memory: 2GB CPU: 1 Core Hard disk: 20GB available","title":"Hardware Pre requisites"},{"location":"cluster_setup_kubespray/#software-pre-requisites","text":"On All Nodes Ubuntu 16.04 OS Python SSH Server Privileged user On Ansible Control Node Ansible version 2.4 or greater Jinja","title":"Software Pre Requisites"},{"location":"cluster_setup_kubespray/#networking-pre-requisites","text":"Internet access to download docker images and install softwares IPv4 Forwarding should be enabled Firewall should allow ssh access as well as ports required by Kubernetes. Internally open all the ports between node.","title":"Networking Pre Requisites"},{"location":"cluster_setup_kubespray/#architecture-of-a-high-available-kubernetes-cluster","text":"","title":"Architecture of a high available kubernetes cluster"},{"location":"cluster_setup_kubespray/#preparing-the-nodes","text":"Run instructions in the section On all nodes in the cluster. This includes Ansible controller too.","title":"Preparing the nodes"},{"location":"cluster_setup_kubespray/#install-python","text":"Ansible needs python to be installed on all the machines. sudo apt update sudo apt install python","title":"Install Python"},{"location":"cluster_setup_kubespray/#enable-ipv4-forwarding","text":"On all nodes Enalbe IPv4 forwarding by uncommenting the following line echo \"net.ipv4.ip_forward=1\" >> /etc/sysctl.conf Disable Swap swapoff -a","title":"Enable IPv4 Forwarding"},{"location":"cluster_setup_kubespray/#setup-passwordless-ssh-between-ansible-controller-and-kubernetes-nodes","text":"On control node Ansible uses passwordless ssh 1 to create the cluster. Let us see how to set it up from your control node . Generate ssh keypair if not present already using the following command. ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ubuntu/.ssh/id_rsa. Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub. The key fingerprint is: SHA256:yC4Tl6RYc+saTPcLKFdGlTLOWOIuDgO1my/NrMBnRxA ubuntu@node1 The key's randomart image is: +---[RSA 2048]----+ | E .. | | . o +.. | | . +o*+o | |. .o+Bo+ | |. .++.X S | |+ +ooX . | |.=.OB.+ . | | .=o*= . . | | .o. . | +----[SHA256]-----+ Just leave the fields to defaults. This command will generate a public key and private key for you. Copy over the public key to all nodes. Example, assuming ubuntu as the user which has a privileged access on the node with ip address 10.10.1.101 , ssh-copy-id ubuntu@10.10.1.101 This will copy our newly generated public key to the remote machine. After running this command you will be able to SSH into the machine directly without using a password. Replace 10.40.1.26 with your respective machine's IP. e.g. ssh ubuntu@10.10.1.101 Make sure to copy the public key to all kubernetes nodes. Replace username with the actual user on your system . If the above mentioned command fails, then copy your public key and paste it in the remote machine's ~/.ssh/authorized_keys file. e.g. (Only if ssh-copy-id fails) cat ~/.ssh/id_rsa.pub ssh ubunut@10.10.1.101 vim ~/.ssh/authorized_keys # Paste the public key","title":"Setup passwordless SSH between ansible controller and kubernetes nodes"},{"location":"cluster_setup_kubespray/#setup-ansible-control-node-and-kubespray","text":"On control node","title":"Setup Ansible Control node and Kubespray"},{"location":"cluster_setup_kubespray/#set-locale","text":"export LC_ALL=\"en_US.UTF-8\" export LC_CTYPE=\"en_US.UTF-8\" sudo dpkg-reconfigure locales Do no select any other locale in the menu. Just press ( OK ) in the next two screens.","title":"Set Locale"},{"location":"cluster_setup_kubespray/#setup-kubespray","text":"Kubespray is hosted on GitHub. Let us the clone the official repository . git clone https://github.com/kubernetes-incubator/kubespray.git cd kubespray","title":"Setup kubespray"},{"location":"cluster_setup_kubespray/#install-prerequisites","text":"Install the python dependencies. This step installs Ansible as well. You do not need to install Ansible separately . sudo apt install python-pip -y sudo pip install -r requirements.txt","title":"Install Prerequisites"},{"location":"cluster_setup_kubespray/#set-remote-user-for-ansible","text":"Add the following section in ansible.cfg file remote_user=ubuntu If the user you are going to connect is differnt, use that instead. Your ansible.cfg file should look like this. [ssh_connection] pipelining=True ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null #control_path = ~/.ssh/ansible-%%r@%%h:%%p [defaults] host_key_checking=False gathering = smart fact_caching = jsonfile fact_caching_connection = /tmp stdout_callback = skippy library = ./library callback_whitelist = profile_tasks roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles deprecation_warnings=False remote_user=ubuntu","title":"Set Remote User for Ansible"},{"location":"cluster_setup_kubespray/#create-inventory","text":"cp -rfp inventory/sample inventory/prod where prod is the custom configuration name. Replace is with whatever name you would like to assign to the current cluster. To build the inventory file, execute the inventory script along with the IP addresses of our cluster as arguments CONFIG_FILE=inventory/prod/hosts.ini python3 contrib/inventory_builder/inventory.py 10.10.1.101 10.10.1.102 10.10.1.103 10.10.1.104 Where replace the IP addresses (e.g. 10.10.1.101) with the actual IPs of your nodes Once its run, you should see an inventory file generated which may look similar to below file: inventory/prod/hosts.ini [all] node1 ansible_host=10.10.1.101 ip=10.10.1.101 node2 ansible_host=10.10.1.102 ip=10.10.1.102 node3 ansible_host=10.10.1.103 ip=10.10.1.103 node4 ansible_host=10.10.1.104 ip=10.10.1.104 [kube-master] node1 node2 [kube-node] node1 node2 node3 node4 [etcd] node1 node2 node3 [k8s-cluster:children] kube-node kube-master [calico-rr] [vault] node1 node2 node3","title":"Create Inventory"},{"location":"cluster_setup_kubespray/#customise-kubernetes-cluster-configs","text":"There are two configs files in your inventroy directory's group_vars (e.g. inventory/prod/group_vars) viz. all.yml k8s-cluster.yml Ansible is data driven, and most of the configurations of the cluster can be tweaked by changing the variable values from the above files. Few of the configurations you may want to modify file: inventory/prod/group_vars/k8s-cluster.yml kubelet_max_pods: 100 cluster_name: prod helm_enabled: true","title":"Customise Kubernetes Cluster Configs"},{"location":"cluster_setup_kubespray/#provisioning-kubernetes-cluster-with-kubespray","text":"On control node We are set to provision the cluster. Run the following ansible-playbook command to provision our Kubernetes cluster. ansible-playbook -b -v -i inventory/prod/hosts.ini cluster.yml Option -i = Inventory file path Option -b = Become as root user Option -v = Give verbose output If you face this following error, while running ansible-playbook command, you can fix it by running following instructions ERROR : ERROR! Unexpected Exception, this is probably a bug: (cryptography 1.2.3 (/usr/lib/python3/dist-packages), Requirement.parse('cryptography>=1.5'), {'paramiko'}) FIX : sudo pip install --upgrade pip sudo pip uninstall cryptography sudo pip install cryptography ansible-playbook -b -v -i inventory/prod/hosts.ini cluster.yml This Ansible run will take around 30 mins to complete.","title":"Provisioning kubernetes cluster with kubespray"},{"location":"cluster_setup_kubespray/#kubectl-configs","text":"On kube master node Once the cluster setup is done, copy the configuration and setup the permissions. mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config","title":"Kubectl Configs"},{"location":"cluster_setup_kubespray/#check-the-state-of-the-cluster","text":"On the node where kubectl is setup Let us check the state of the cluster by running, kubectl cluster-info Kubernetes master is running at https://10.10.1.101:6443 KubeDNS is running at https://10.10.1.101:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready master,node 21h v1.9.0+coreos.0 node2 Ready master,node 21h v1.9.0+coreos.0 node3 Ready node 21h v1.9.0+coreos.0 node4 Ready node 21h v1.9.0+coreos.0 If you are able to see this, your cluster has been set up successfully. 1 You can use private key / password instead of passwordless ssh. But it requires additional knowledge in using Ansible.","title":"Check the State of the Cluster"},{"location":"cluster_setup_kubespray/#access-kubernetes-cluster-remotely-optional","text":"On your local machine You could also install kubectl on your laptop/workstation. To learn how to install it for your OS, refer to the procedure here . e.g. To install kubectl on Ubuntu, sudo apt-get update && sudo apt-get install -y apt-transport-https curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - sudo touch /etc/apt/sources.list.d/kubernetes.list echo \"deb http://apt.kubernetes.io/ kubernetes-xenial main\" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubectl","title":"Access Kubernetes Cluster Remotely (Optional)"},{"location":"cluster_setup_kubespray/#copy-kubernetes-config-to-your-local-machine","text":"Copy kubeconfig file to your local machine mkdir ~/.kube scp -r ubuntu@MASTER_HOST_IP:/etc/kubernetes/admin.conf ~/.kube/config kubectl get nodes","title":"Copy kubernetes config to your local machine"},{"location":"cluster_setup_kubespray/#deploy-kubernetes-objects","text":"Since its a new cluster, which is differnt than what you have created with kubeadm earlier, or if this is the first time you are creating a kubernetes cluster with kubespray as part of Advanced Workshop , you need to deploy services which have been covered as part of the previous topics. In order to do that, use the following commands on the node where you have configured kubectl git clone https://github.com/schoolofdevops/k8s-code.git cd k8s-code/projects/instavote kubectl apply -f instavote-ns.yaml kubectl apply -f prod/ Switch to instavote namespace and validate, kubectl config set-context $(kubectl config current-context) --namespace=instavote kubectl get pods,deploy,svc where, --cluster=prod : prod is the cluter name you created earlier. If not, use the correct name of the cluster ( kubectl config view) --user=admin-prod: is the admin user created by default while installing with kubespray --namespace=instavote : the namespace you just created to deploy instavote app stack [sample output] $ kubectl get pods,deploy,svc NAME READY STATUS RESTARTS AGE pod/db-66496667c9-qggzd 1/1 Running 0 7m pod/redis-6555998885-4k5cr 1/1 Running 0 7m pod/redis-6555998885-fb8rk 1/1 Running 0 7m pod/result-5c7569bcb7-4fptr 1/1 Running 0 7m pod/result-5c7569bcb7-s4rdx 1/1 Running 0 7m pod/vote-5d88d47fc8-gbzbq 1/1 Running 0 7m pod/vote-5d88d47fc8-q4vj6 1/1 Running 0 7m pod/worker-7c98c96fb4-7tzzw 1/1 Running 0 7m NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.extensions/db 1 1 1 1 7m deployment.extensions/redis 2 2 2 2 7m deployment.extensions/result 2 2 2 2 7m deployment.extensions/vote 2 2 2 2 7m deployment.extensions/worker 1 1 1 1 7m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/db ClusterIP 10.233.16.207 5432/TCP 7m service/redis ClusterIP 10.233.14.61 6379/TCP 7m service/result NodePort 10.233.22.10 80:30100/TCP 7m service/vote NodePort 10.233.19.111 80:30000/TCP 7m","title":"Deploy Kubernetes Objects"},{"location":"cluster_setup_kubespray/#references","text":"Installing Kubernetes On Premises/On Cloud with Kubespray Kubespray on Github","title":"References"},{"location":"cni/","text":"","title":"Lab K404 - CNI"},{"location":"configuring_authentication_and_authorization/","text":"Lab K202 - Kubernetes Access Control: Authentication and Authorization In this lab you are going to, Create users and groups and setup certs based authentication Create service accounts for applications Create Roles and ClusterRoles to define authorizations Map Roles and ClusterRoles to subjects i.e. users, groups and service accounts using RoleBingings and ClusterRoleBindings. How one can access the Kubernetes API? The Kubernetes API can be accessed by three ways. Kubectl - A command line utility of Kubernetes Client libraries - Go, Python, etc., REST requests Who can access the Kubernetes API? Kubernetes API can be accessed by, Human Users Service Accounts Each of these topics will be discussed in detail in the later part of this chapter. Stages of a Request When a request tries to contact the API , it goes through various stages as illustrated in the image given below. source: official kubernetes site api groups and resources apiGroup Resources apps daemonsets, deployments, deployments/rollback, deployments/scale, replicasets, replicasets/scale, statefulsets, statefulsets/scale core configmaps, endpoints, persistentvolumeclaims, replicationcontrollers, replicationcontrollers/scale, secrets, serviceaccounts, services,services/proxy autoscaling horizontalpodautoscalers batch cronjobs, jobs policy poddisruptionbudgets networking.k8s.io networkpolicies authorization.k8s.io localsubjectaccessreviews rbac.authorization.k8s.io rolebindings,roles extensions deprecated (read notes) Notes In addition to the above apiGroups, you may see extensions being used in some example code snippets. Please note that extensions was initially created as a experiement and is been deprecated, by moving most of the matured apis to one of the groups mentioned above. You could read this comment and the thread to get clarity on this. Role Based Access Control (RBAC) Group User Namespaces Resources Access Type (verbs) ops maya all all get, list, watch, update, patch, create, delete, deletecollection dev kim instavote deployments, statefulsets, services, pods, configmaps, secrets, replicasets, ingresses, endpoints, cronjobs, jobs, persistentvolumeclaims get, list , watch, update, patch, create interns yono instavote readonly get, list, watch Service Accounts Namespace Resources Access Type (verbs) monitoring all all readonly Creating Kubernetes Users and Groups Generate the user's private key mkdir -p ~/.kube/users cd ~/.kube/users openssl genrsa -out kim.key 2048 [sample Output] openssl genrsa -out kim.key 2048 Generating RSA private key, 2048 bit long modulus .............................................................+++ .........................+++ e is 65537 (0x10001) Lets now create a Certification Signing Request (CSR) for each of the users. When you generate the csr make sure you also provide CN: This will be set as username O: Org name. This is actually used as a group by kubernetes while authenticating/authorizing users. You could add as many as you need e.g. openssl req -new -key kim.key -out kim.csr -subj \"/CN=kim/O=dev/O=example.org\" In order to be deemed authentic, these CSRs need to be signed by the Certification Authority (CA) which in this case is Kubernetes Master. You need access to the folllwing files on kubernetes master. Certificate : ca.crt (kubeadm) or ca.key (kubespray) Pricate Key : ca.key (kubeadm) or ca-key.pem (kubespray) You would typically find it the following paths /etc/kubernetes/pki To verify which one is your cert and which one is key, use the following command, $ file /etc/kubernetes/pki/ca.crt ca.pem: PEM certificate $ file /etc/kubernetes/pki/ca.key ca-key.pem: PEM RSA private key Once signed, .csr files with added signatures become the certificates that could be used to authenticate. You could either move the crt files to k8s master, sign and download copy over the CA certs and keys to your management node and use it to sign. Make sure to keep your CA related files secure. In the example here, I have already downloaded ca.pem and ca-key.pem to my management workstation, which are used to sign the CSRs. Assuming all the files are in the same directory, sign the CSR as, openssl x509 -req -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -days 730 -in kim.csr -out kim.crt Setting up User configs with kubectl In order to configure the users that you created above, following steps need to be performed with kubectl Add credentials in the configurations Set context to login as a user to a cluster Switch context in order to assume the user's identity while working with the cluster to add credentials, kubectl config set-credentials kim --client-certificate=/root/.kube/users/kim.crt --client-key=/root/.kube/users/kim.key where, /root/.kube/users/kim.crt : Absolute path to the users' certificate /root/.kube/users/kim.key: Absolute path to the users' key And proceed to set/create contexts (user@cluster). If you are not sure whats the cluster name, use the following command to find, kubectl config get-contexts [sample output] CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kubernetes-admin@kubernetes kubernetes kubernetes-admin instavote where, kubernetes is the cluster name. To set context for kubernetes cluster, kubectl config set-context kim-kubernetes --cluster=kubernetes --user=kim --namespace=instavote Where, kim-kubernetes : name of the context kubernetes : name of the cluster you set while creating it kim : user you created and configured above to connect to the cluster You could verify the configs with kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE kim-kubernetes kubernetes kim instavote * kubernetes-admin@kubernetes kubernetes kubernetes-admin instavote and kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://178.128.109.8:6443 name: kubernetes contexts: - context: cluster: kubernetes namespace: instavote user: kim name: kim-kubernetes - context: cluster: kubernetes namespace: instavote user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kim user: client-certificate: users/kim.crt client-key: users/kim.key - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED Where, you should see the configurations for the new user you created have been added. You could assume the identity of user kim and connect to the kubernetes cluster as, kubectl config use-context kim-kubernetes kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kim-kubernetes kubernetes kim instavote kubernetes-admin@kubernetes kubernetes kubernetes-admin instavote This time * appears on the line which lists context kim-kubernetes that you just created. And then try running any command as, kubectl get pods Alternately, if you are a admin user, you could impersonate a user and run a command with that literally using --as option kubectl config use-context admin-prod kubectl get pods --as yono [Sample Output] No resources found. Error from server (Forbidden): pods is forbidden: User \"yono\" cannot list pods in the namespace \"instavote\" Either ways, since there are authorization rules set, the user can not make any api calls. Thats when you would create some roles and bind it to the users in the next section. Define authorisation rules with Roles and ClusterRoles Whats the difference between Roles and ClusterRoles ?? Role is limited to a namespace (Projects/Orgs/Env) ClusterRole is Global Lets say you want to provide read only access to instavote , a project specific namespace to all users in the example.org kubectl config use-context kubernetes-admin@kubernetes cd projects/instavote/dev file: readonly-role.yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: namespace: instavote name: readonly rules: - apiGroups: [\"*\"] resources: [\"*\"] verbs: [\"get\", \"list\", \"watch\"] In order to map it to all users in example.org , create a RoleBinding as file: readonly-rolebinding.yml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: readonly namespace: instavote subjects: - kind: Group name: dev apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: readonly apiGroup: rbac.authorization.k8s.io kubectl apply -f readonly-role.yaml kubectl apply -f readonly-rolebinding.yml To get information about the objects created above, kubectl get roles,rolebindings -n instavote kubectl describe role readonly kubectl describe rolebinding readonly To validate the access, kubectl config get-contexts kubectl config use-context kim-kubernetes kubectl get pods To switch back to admin, kubectl config use-context kubernetes-admin@kubernetes Exercise Create a Role and Rolebinding for dev group with the authorizations defined in the table above. Once applied, test it","title":"Lab K401 - Creating Users, Groups and Authorization"},{"location":"configuring_authentication_and_authorization/#lab-k202-kubernetes-access-control-authentication-and-authorization","text":"In this lab you are going to, Create users and groups and setup certs based authentication Create service accounts for applications Create Roles and ClusterRoles to define authorizations Map Roles and ClusterRoles to subjects i.e. users, groups and service accounts using RoleBingings and ClusterRoleBindings.","title":"Lab K202 - Kubernetes Access Control: Authentication and Authorization"},{"location":"configuring_authentication_and_authorization/#how-one-can-access-the-kubernetes-api","text":"The Kubernetes API can be accessed by three ways. Kubectl - A command line utility of Kubernetes Client libraries - Go, Python, etc., REST requests","title":"How one can access the Kubernetes API?"},{"location":"configuring_authentication_and_authorization/#who-can-access-the-kubernetes-api","text":"Kubernetes API can be accessed by, Human Users Service Accounts Each of these topics will be discussed in detail in the later part of this chapter.","title":"Who can access the Kubernetes API?"},{"location":"configuring_authentication_and_authorization/#stages-of-a-request","text":"When a request tries to contact the API , it goes through various stages as illustrated in the image given below. source: official kubernetes site","title":"Stages of a Request"},{"location":"configuring_authentication_and_authorization/#api-groups-and-resources","text":"apiGroup Resources apps daemonsets, deployments, deployments/rollback, deployments/scale, replicasets, replicasets/scale, statefulsets, statefulsets/scale core configmaps, endpoints, persistentvolumeclaims, replicationcontrollers, replicationcontrollers/scale, secrets, serviceaccounts, services,services/proxy autoscaling horizontalpodautoscalers batch cronjobs, jobs policy poddisruptionbudgets networking.k8s.io networkpolicies authorization.k8s.io localsubjectaccessreviews rbac.authorization.k8s.io rolebindings,roles extensions deprecated (read notes)","title":"api groups and resources"},{"location":"configuring_authentication_and_authorization/#notes","text":"In addition to the above apiGroups, you may see extensions being used in some example code snippets. Please note that extensions was initially created as a experiement and is been deprecated, by moving most of the matured apis to one of the groups mentioned above. You could read this comment and the thread to get clarity on this.","title":"Notes"},{"location":"configuring_authentication_and_authorization/#role-based-access-control-rbac","text":"Group User Namespaces Resources Access Type (verbs) ops maya all all get, list, watch, update, patch, create, delete, deletecollection dev kim instavote deployments, statefulsets, services, pods, configmaps, secrets, replicasets, ingresses, endpoints, cronjobs, jobs, persistentvolumeclaims get, list , watch, update, patch, create interns yono instavote readonly get, list, watch Service Accounts Namespace Resources Access Type (verbs) monitoring all all readonly","title":"Role Based Access Control (RBAC)"},{"location":"configuring_authentication_and_authorization/#creating-kubernetes-users-and-groups","text":"Generate the user's private key mkdir -p ~/.kube/users cd ~/.kube/users openssl genrsa -out kim.key 2048 [sample Output] openssl genrsa -out kim.key 2048 Generating RSA private key, 2048 bit long modulus .............................................................+++ .........................+++ e is 65537 (0x10001) Lets now create a Certification Signing Request (CSR) for each of the users. When you generate the csr make sure you also provide CN: This will be set as username O: Org name. This is actually used as a group by kubernetes while authenticating/authorizing users. You could add as many as you need e.g. openssl req -new -key kim.key -out kim.csr -subj \"/CN=kim/O=dev/O=example.org\" In order to be deemed authentic, these CSRs need to be signed by the Certification Authority (CA) which in this case is Kubernetes Master. You need access to the folllwing files on kubernetes master. Certificate : ca.crt (kubeadm) or ca.key (kubespray) Pricate Key : ca.key (kubeadm) or ca-key.pem (kubespray) You would typically find it the following paths /etc/kubernetes/pki To verify which one is your cert and which one is key, use the following command, $ file /etc/kubernetes/pki/ca.crt ca.pem: PEM certificate $ file /etc/kubernetes/pki/ca.key ca-key.pem: PEM RSA private key Once signed, .csr files with added signatures become the certificates that could be used to authenticate. You could either move the crt files to k8s master, sign and download copy over the CA certs and keys to your management node and use it to sign. Make sure to keep your CA related files secure. In the example here, I have already downloaded ca.pem and ca-key.pem to my management workstation, which are used to sign the CSRs. Assuming all the files are in the same directory, sign the CSR as, openssl x509 -req -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -days 730 -in kim.csr -out kim.crt","title":"Creating Kubernetes Users and Groups"},{"location":"configuring_authentication_and_authorization/#setting-up-user-configs-with-kubectl","text":"In order to configure the users that you created above, following steps need to be performed with kubectl Add credentials in the configurations Set context to login as a user to a cluster Switch context in order to assume the user's identity while working with the cluster to add credentials, kubectl config set-credentials kim --client-certificate=/root/.kube/users/kim.crt --client-key=/root/.kube/users/kim.key where, /root/.kube/users/kim.crt : Absolute path to the users' certificate /root/.kube/users/kim.key: Absolute path to the users' key And proceed to set/create contexts (user@cluster). If you are not sure whats the cluster name, use the following command to find, kubectl config get-contexts [sample output] CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kubernetes-admin@kubernetes kubernetes kubernetes-admin instavote where, kubernetes is the cluster name. To set context for kubernetes cluster, kubectl config set-context kim-kubernetes --cluster=kubernetes --user=kim --namespace=instavote Where, kim-kubernetes : name of the context kubernetes : name of the cluster you set while creating it kim : user you created and configured above to connect to the cluster You could verify the configs with kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE kim-kubernetes kubernetes kim instavote * kubernetes-admin@kubernetes kubernetes kubernetes-admin instavote and kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://178.128.109.8:6443 name: kubernetes contexts: - context: cluster: kubernetes namespace: instavote user: kim name: kim-kubernetes - context: cluster: kubernetes namespace: instavote user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kim user: client-certificate: users/kim.crt client-key: users/kim.key - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED Where, you should see the configurations for the new user you created have been added. You could assume the identity of user kim and connect to the kubernetes cluster as, kubectl config use-context kim-kubernetes kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kim-kubernetes kubernetes kim instavote kubernetes-admin@kubernetes kubernetes kubernetes-admin instavote This time * appears on the line which lists context kim-kubernetes that you just created. And then try running any command as, kubectl get pods Alternately, if you are a admin user, you could impersonate a user and run a command with that literally using --as option kubectl config use-context admin-prod kubectl get pods --as yono [Sample Output] No resources found. Error from server (Forbidden): pods is forbidden: User \"yono\" cannot list pods in the namespace \"instavote\" Either ways, since there are authorization rules set, the user can not make any api calls. Thats when you would create some roles and bind it to the users in the next section.","title":"Setting up User configs with kubectl"},{"location":"configuring_authentication_and_authorization/#define-authorisation-rules-with-roles-and-clusterroles","text":"Whats the difference between Roles and ClusterRoles ?? Role is limited to a namespace (Projects/Orgs/Env) ClusterRole is Global Lets say you want to provide read only access to instavote , a project specific namespace to all users in the example.org kubectl config use-context kubernetes-admin@kubernetes cd projects/instavote/dev file: readonly-role.yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: namespace: instavote name: readonly rules: - apiGroups: [\"*\"] resources: [\"*\"] verbs: [\"get\", \"list\", \"watch\"] In order to map it to all users in example.org , create a RoleBinding as file: readonly-rolebinding.yml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: readonly namespace: instavote subjects: - kind: Group name: dev apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: readonly apiGroup: rbac.authorization.k8s.io kubectl apply -f readonly-role.yaml kubectl apply -f readonly-rolebinding.yml To get information about the objects created above, kubectl get roles,rolebindings -n instavote kubectl describe role readonly kubectl describe rolebinding readonly To validate the access, kubectl config get-contexts kubectl config use-context kim-kubernetes kubectl get pods To switch back to admin, kubectl config use-context kubernetes-admin@kubernetes","title":"Define authorisation rules with Roles and ClusterRoles"},{"location":"configuring_authentication_and_authorization/#exercise","text":"Create a Role and Rolebinding for dev group with the authorizations defined in the table above. Once applied, test it","title":"Exercise"},{"location":"create-and-deploy-kubernetes-pods/","text":"Launching Pods with Kubernetes In this lab, you would learn how to launch applications using the basic deployment unit of kubernetes i.e. pods . This time, you are going to do it by writing declarative configs with yaml syntax. Launching Pods Manually You could use kubectl run to launch a pod by specifying just the image. For example, if you would like to launch a pod for redis, with image redis:alpine, the following command would work, kubectl run redis --image=redis Kubernetes Resources and writing YAML specs Each entity created with kubernetes is a resource including pod, service, deployments, replication controller etc. Resources can be defined as YAML or JSON. Here is the syntax to create a YAML specification. AKMS => Resource Configs Specs apiVersion: v1 kind: metadata: spec: Use this Kubernetes API Reference Document while writing the API specs. This is the most important reference while working with kubernetes, so its highly advisible that you bookmark it for the version of kubernetes that you use. To find the version of kubernetes use the following command, kubectl version -o yaml To list the running pods, kubectl get pods To list API objects, use the following commands, kubectl api-resources LAB PART I Writing Pod Spec Lets now create the Pod config by adding the kind and specs to schma given in the file vote-pod.yaml as follows. Filename: k8s-code/pods/vote-pod.yaml apiVersion: kind: Pod metadata: spec: Problem Statement: Create a YAML spec to launch a pod with one container to run vote application, which matches the following specs. pod: metadata: name: vote labels: app: python role: vote version: v1 containers: name: app image: schoolofdevops/vote:v1 Refer to the Pod Spec find out the relevant properties and add it to the vote-pod.yaml provided in the supporting code repo. Filename: k8s-code/pods/vote-pod.yaml apiVersion: v1 kind: Pod metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 Use this example link to refer to pod spec Launching and operating Pods To launch a monitoring screen to see whats being launched, use the following command in a new terminal window where kubectl is configured. watch kubectl get all kubectl Syntax: kubectl kubectl apply --help kubectl apply -f FILE to do a dry run before applying use the following command, kubectl apply -f vote-pod.yaml --dry-run=client to finally launch pod using configs above, remove dry run options and run as kubectl apply -f vote-pod.yaml To view pods kubectl get pods kubectl get po kubectl get pods -o wide kubectl get pods --show-labels kubectl get pods -l \"role=vote,version=v1\" kubectl get pods vote To get detailed info kubectl describe pods vote Commands to operate the pod kubectl logs vote kubectl exec -it vote sh Run the following commands inside the container in a pod after running exec command as above ifconfig cat /etc/issue hostname cat /proc/cpuinfo ps aux use ^d or exit to log out LAB PART II Adding a Volume for data persistence Lets create a pod for database and attach a volume to it. To achieve this we will need to create a volumes definition attach volume to container using VolumeMounts property Local host volumes are of two types: emptyDir hostPath We will pick hostPath. Refer to this doc to read more about hostPath. File: db-pod.yaml apiVersion: v1 kind: Pod metadata: name: db labels: app: postgres role: database tier: back spec: containers: - name: db image: postgres:9.4 ports: - containerPort: 5432 volumeMounts: - name: db-data mountPath: /var/lib/postgresql/data volumes: - name: db-data hostPath: path: /var/lib/pgdata type: DirectoryOrCreate To create this pod, kubectl apply -f db-pod.yaml kubectl get pods At this time, you may see the pod in CrashLoopBackOff state. Try debugging the issue by checking the logs and resolve it before proceeding. Once resolved, check the pod is in running state kubectl get pods kubectl describe pod db kubectl get events Exercise : Examine /var/lib/pgdata on the node on which its scheduled (use kubectl get pods -o wide to find the node) to check if the directory is been created and if the data is present. Creating Multi Container Pods - Sidecar Example file: multi_container_pod.yaml apiVersion: v1 kind: Pod metadata: name: web labels: tier: front app: nginx role: ui spec: containers: - name: nginx image: nginx:stable-alpine ports: - containerPort: 80 protocol: TCP volumeMounts: - name: data mountPath: /var/www/html-sample-app - name: sync image: schoolofdevops/sync:v2 volumeMounts: - name: data mountPath: /var/www/app volumes: - name: data emptyDir: {} To create this pod kubectl apply -f multi_container_pod.yaml Check Status root@kube-01:~# kubectl get pods NAME READY STATUS RESTARTS AGE web 0/2 ContainerCreating 0 7s vote 1/1 Running 0 3m Checking logs, logging in kubectl logs web -c sync kubectl logs web -c nginx kubectl exec -it web sh -c nginx kubectl exec -it web sh -c sync Observe whats common and whats isolated in two containers running inside the same pod using the following commands, shared hostname ifconfig isolated cat /etc/issue ps aux df -h Adding Resource requests and limits We can control the amount of resource requested and also put a limit on the maximum a container in a pod could take up. This can be done by adding to the existing pod spec as below. Refer to official document on resource management here. Filename: vote-pod.yaml apiVersion: v1 kind: Pod metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" Lets apply the changes now kubectl apply -f vote-pod.yaml If you already have vote pod running, you may see an output similar to below, [sample output] The Pod \"vote\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations) {\"Volumes\":[{\"Name\":\"default-token-snbj4\",\"HostPath\":null,\"EmptyDir\":null,\"GCEPersistentDisk\":null,\"AWSElasticBlockStore\":null,\"GitRepo\":null,\"Secret\":{\"SecretName\":\"default-token-snbj4\",\"Items\":null,\"DefaultMode\":420,\"Optional\":null},\"NFS\":null,\"ISCSI\":null,\"Glusterfs\":null,\"PersistentVolumeClaim\":null,\"RBD\":null,\"Quobyte\":null,\"FlexVolume\":null,\"Cinder\":null,\"CephFS\":null,\"Flocker\":null,\"DownwardAPI\":null,\"FC\":null,\"AzureFile\":null,\"ConfigMap\":null,\"VsphereVolume\":null,\"AzureDisk\":null,\"PhotonPersistentDisk\":null,\"Projected\":null,\"PortworxVolume\":null,\"ScaleIO\":null,\"StorageOS\":null}],\"InitContainers\":null,\"Containers\":[{\"Name\":\"app\",\"Image\":\"schoolofdevops/vote:v1\",\"Command\":null,\"Args\":null,\"WorkingDir\":\"\",\"Ports\":null,\"EnvFrom\":null,\"Env\":null,\"Resources\":{\"Limits\": .... ... From the above output, its clear that not all the fields are mutable(except for a few e.g labels). Container based deployments primarily follow concept of immutable deployments . So to bring your change into effect, you need to re create the pod as, kubectl delete pod vote kubectl apply -f vote-pod.yaml kubectl describe pod vote From the output of the describe command above, you could confirm the resource constraints you added are in place. Exercise * Define the value of **cpu.request** > **cpu.limit** Try to apply and observe. * Define the values for **memory.request** and **memory.limit** higher than the total system memory. Apply and observe the deployment and pods. Deleting Pods Now that you are done experimenting with pod, delete it with the following command, kubectl delete pod vote web db kubectl get pods Reading List PodSpec: Managing Volumes with Kubernetes Node Selectors, Affinity","title":"Lab K103 - Pods"},{"location":"create-and-deploy-kubernetes-pods/#launching-pods-with-kubernetes","text":"In this lab, you would learn how to launch applications using the basic deployment unit of kubernetes i.e. pods . This time, you are going to do it by writing declarative configs with yaml syntax.","title":"Launching Pods with Kubernetes"},{"location":"create-and-deploy-kubernetes-pods/#launching-pods-manually","text":"You could use kubectl run to launch a pod by specifying just the image. For example, if you would like to launch a pod for redis, with image redis:alpine, the following command would work, kubectl run redis --image=redis","title":"Launching Pods Manually"},{"location":"create-and-deploy-kubernetes-pods/#kubernetes-resources-and-writing-yaml-specs","text":"Each entity created with kubernetes is a resource including pod, service, deployments, replication controller etc. Resources can be defined as YAML or JSON. Here is the syntax to create a YAML specification. AKMS => Resource Configs Specs apiVersion: v1 kind: metadata: spec: Use this Kubernetes API Reference Document while writing the API specs. This is the most important reference while working with kubernetes, so its highly advisible that you bookmark it for the version of kubernetes that you use. To find the version of kubernetes use the following command, kubectl version -o yaml To list the running pods, kubectl get pods To list API objects, use the following commands, kubectl api-resources","title":"Kubernetes Resources and writing YAML specs"},{"location":"create-and-deploy-kubernetes-pods/#lab-part-i","text":"","title":"LAB PART I"},{"location":"create-and-deploy-kubernetes-pods/#writing-pod-spec","text":"Lets now create the Pod config by adding the kind and specs to schma given in the file vote-pod.yaml as follows. Filename: k8s-code/pods/vote-pod.yaml apiVersion: kind: Pod metadata: spec: Problem Statement: Create a YAML spec to launch a pod with one container to run vote application, which matches the following specs. pod: metadata: name: vote labels: app: python role: vote version: v1 containers: name: app image: schoolofdevops/vote:v1 Refer to the Pod Spec find out the relevant properties and add it to the vote-pod.yaml provided in the supporting code repo. Filename: k8s-code/pods/vote-pod.yaml apiVersion: v1 kind: Pod metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 Use this example link to refer to pod spec","title":"Writing Pod Spec"},{"location":"create-and-deploy-kubernetes-pods/#launching-and-operating-pods","text":"To launch a monitoring screen to see whats being launched, use the following command in a new terminal window where kubectl is configured. watch kubectl get all kubectl Syntax: kubectl kubectl apply --help kubectl apply -f FILE to do a dry run before applying use the following command, kubectl apply -f vote-pod.yaml --dry-run=client to finally launch pod using configs above, remove dry run options and run as kubectl apply -f vote-pod.yaml To view pods kubectl get pods kubectl get po kubectl get pods -o wide kubectl get pods --show-labels kubectl get pods -l \"role=vote,version=v1\" kubectl get pods vote To get detailed info kubectl describe pods vote Commands to operate the pod kubectl logs vote kubectl exec -it vote sh Run the following commands inside the container in a pod after running exec command as above ifconfig cat /etc/issue hostname cat /proc/cpuinfo ps aux use ^d or exit to log out","title":"Launching and operating Pods"},{"location":"create-and-deploy-kubernetes-pods/#lab-part-ii","text":"","title":"LAB PART II"},{"location":"create-and-deploy-kubernetes-pods/#adding-a-volume-for-data-persistence","text":"Lets create a pod for database and attach a volume to it. To achieve this we will need to create a volumes definition attach volume to container using VolumeMounts property Local host volumes are of two types: emptyDir hostPath We will pick hostPath. Refer to this doc to read more about hostPath. File: db-pod.yaml apiVersion: v1 kind: Pod metadata: name: db labels: app: postgres role: database tier: back spec: containers: - name: db image: postgres:9.4 ports: - containerPort: 5432 volumeMounts: - name: db-data mountPath: /var/lib/postgresql/data volumes: - name: db-data hostPath: path: /var/lib/pgdata type: DirectoryOrCreate To create this pod, kubectl apply -f db-pod.yaml kubectl get pods At this time, you may see the pod in CrashLoopBackOff state. Try debugging the issue by checking the logs and resolve it before proceeding. Once resolved, check the pod is in running state kubectl get pods kubectl describe pod db kubectl get events Exercise : Examine /var/lib/pgdata on the node on which its scheduled (use kubectl get pods -o wide to find the node) to check if the directory is been created and if the data is present.","title":"Adding a Volume for data persistence"},{"location":"create-and-deploy-kubernetes-pods/#creating-multi-container-pods-sidecar-example","text":"file: multi_container_pod.yaml apiVersion: v1 kind: Pod metadata: name: web labels: tier: front app: nginx role: ui spec: containers: - name: nginx image: nginx:stable-alpine ports: - containerPort: 80 protocol: TCP volumeMounts: - name: data mountPath: /var/www/html-sample-app - name: sync image: schoolofdevops/sync:v2 volumeMounts: - name: data mountPath: /var/www/app volumes: - name: data emptyDir: {} To create this pod kubectl apply -f multi_container_pod.yaml Check Status root@kube-01:~# kubectl get pods NAME READY STATUS RESTARTS AGE web 0/2 ContainerCreating 0 7s vote 1/1 Running 0 3m Checking logs, logging in kubectl logs web -c sync kubectl logs web -c nginx kubectl exec -it web sh -c nginx kubectl exec -it web sh -c sync Observe whats common and whats isolated in two containers running inside the same pod using the following commands, shared hostname ifconfig isolated cat /etc/issue ps aux df -h","title":"Creating Multi Container Pods - Sidecar Example"},{"location":"create-and-deploy-kubernetes-pods/#adding-resource-requests-and-limits","text":"We can control the amount of resource requested and also put a limit on the maximum a container in a pod could take up. This can be done by adding to the existing pod spec as below. Refer to official document on resource management here. Filename: vote-pod.yaml apiVersion: v1 kind: Pod metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" Lets apply the changes now kubectl apply -f vote-pod.yaml If you already have vote pod running, you may see an output similar to below, [sample output] The Pod \"vote\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations) {\"Volumes\":[{\"Name\":\"default-token-snbj4\",\"HostPath\":null,\"EmptyDir\":null,\"GCEPersistentDisk\":null,\"AWSElasticBlockStore\":null,\"GitRepo\":null,\"Secret\":{\"SecretName\":\"default-token-snbj4\",\"Items\":null,\"DefaultMode\":420,\"Optional\":null},\"NFS\":null,\"ISCSI\":null,\"Glusterfs\":null,\"PersistentVolumeClaim\":null,\"RBD\":null,\"Quobyte\":null,\"FlexVolume\":null,\"Cinder\":null,\"CephFS\":null,\"Flocker\":null,\"DownwardAPI\":null,\"FC\":null,\"AzureFile\":null,\"ConfigMap\":null,\"VsphereVolume\":null,\"AzureDisk\":null,\"PhotonPersistentDisk\":null,\"Projected\":null,\"PortworxVolume\":null,\"ScaleIO\":null,\"StorageOS\":null}],\"InitContainers\":null,\"Containers\":[{\"Name\":\"app\",\"Image\":\"schoolofdevops/vote:v1\",\"Command\":null,\"Args\":null,\"WorkingDir\":\"\",\"Ports\":null,\"EnvFrom\":null,\"Env\":null,\"Resources\":{\"Limits\": .... ... From the above output, its clear that not all the fields are mutable(except for a few e.g labels). Container based deployments primarily follow concept of immutable deployments . So to bring your change into effect, you need to re create the pod as, kubectl delete pod vote kubectl apply -f vote-pod.yaml kubectl describe pod vote From the output of the describe command above, you could confirm the resource constraints you added are in place.","title":"Adding Resource requests and limits"},{"location":"create-and-deploy-kubernetes-pods/#exercise","text":"* Define the value of **cpu.request** > **cpu.limit** Try to apply and observe. * Define the values for **memory.request** and **memory.limit** higher than the total system memory. Apply and observe the deployment and pods.","title":"Exercise"},{"location":"create-and-deploy-kubernetes-pods/#deleting-pods","text":"Now that you are done experimenting with pod, delete it with the following command, kubectl delete pod vote web db kubectl get pods","title":"Deleting Pods"},{"location":"create-and-deploy-kubernetes-pods/#reading-list","text":"PodSpec: Managing Volumes with Kubernetes Node Selectors, Affinity","title":"Reading List"},{"location":"custom_autoscaling/","text":"Auto Scaling on Custom Metric For this lab, you must have configured Prometheus and Grafana using help already. Refer to Lab K113 - HELM Package Manager - Kubernetes Tutorial with CKA/CKAD Prep to set it up. Setup Nginx Ingress with Metrics Delete existing nginx ingress controller setup with KIND kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml Re deploy nginx ingress controller with helm, this time enabling the exposing the metrics which can then be scraped/collected by prometheus. helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true --set \\ controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" \\ --set controller.hostPort.enabled=true \\ --set controller.hostPort.ports.http=80 \\ --set controller.hostPort.ports.https=443 \\ --set-string controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\ --set-string controller.nodeSelector.ingress-ready=\"true\" Configure Prometheus to monitor Nginx Ingress Update the existing Prometheus installation to support scrpaing services in all namespaces assuming you deployed prometheus stack from kube-prometheus-stack path, switch to cd ~/kube-prometheus-stack/ and upgrade helm upgrade prom -n monitoring --values my.values.yaml . --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false What this does is starts collecting metrics from services from all the namespaces not just the one in which prometheus is deployed. And it reads the annotations from the pods to figure out where to collect the metrics from. Now, login to grafana and import custom dashboard for Nginx Ingress as Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" It may look similar to this, with possibly less data initially However, if you see some metric coming in, your setup with Nginx Ingress and Promethus Integration is working ! You may pat your back at this time :) Setup Prometheus Adapter Prometheus adapter converts the prometheus metrics and makes it available in kubernetes under custom.metrics.k8s.io api which can then read by the HPA and used as a input for autoscaling. Begin by creating the configuration for this adapter to pick a specific metric from prometheus and expose it under custom API as, File : prometheus-adapter-values.yaml prometheus: url: http://prom-kube-prometheus-stack-prometheus # Ensure this is correctly pointing to your Prometheus port: 9090 rules: default: false custom: - seriesQuery: 'nginx_ingress_controller_requests' resources: overrides: namespace: {resource: \"namespace\"} ingress: {resource: \"ingress\"} name: matches: \"^nginx_ingress_controller_requests$\" as: \"nginx_ingress_controller_requests_per_second\" metricsQuery: 'sum(rate(nginx_ingress_controller_requests{status=~\"2..\"}[2m])) by (namespace, ingress)' - seriesQuery: 'nginx_ingress_controller_request_duration_seconds_bucket' resources: overrides: namespace: {resource: \"namespace\"} ingress: {resource: \"ingress\"} name: matches: \"^nginx_ingress_controller_request_duration_seconds_bucket$\" as: \"nginx_ingress_controller_latency\" metricsQuery: 'sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{}[2m])) by (namespace, ingress, le) / 1000' install the adapter as, helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter -f prometheus-adapter-values.yaml -n monitoring validate its running kubectl get pods -n monitoring validate that custom apis are working with kubectl get --raw \"/apis/custom.metrics.k8s.io\" | jq . kubectl get --raw \"/apis/custom.metrics.k8s.io/v1beta1/namespaces/instavote/ingresses/vote/nginx_ingress_controller_requests_per_second\" | jq . kubectl get --raw \"/apis/custom.metrics.k8s.io/v1beta1/namespaces/instavote/ingresses/vote/nginx_ingress_controller_latency\" | jq . with the second command, you are expected to see [sample output] { \"kind\": \"MetricValueList\", \"apiVersion\": \"custom.metrics.k8s.io/v1beta1\", \"metadata\": {}, \"items\": [ { \"describedObject\": { \"kind\": \"Ingress\", \"namespace\": \"instavote\", \"name\": \"vote\", \"apiVersion\": \"networking.k8s.io/v1\" }, \"metricName\": \"nginx_ingress_controller_requests_per_second\", \"timestamp\": \"2024-05-10T02:41:54Z\", \"value\": \"0\", \"selector\": null } ] } If you see this, your adapter configuration integratd with prometheus is set up and ready to be consumed by the hpa. Configure Custom Autoscaling Policies Add autoscaling policies based on the custom metric as File : vote-custom-hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vote-custom namespace: instavote spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: vote minReplicas: 2 maxReplicas: 10 metrics: - type: Object object: metric: name: nginx_ingress_controller_requests_per_second describedObject: apiVersion: networking.k8s.io/v1 kind: Ingress name: vote target: type: Value value: 100 # Target 200 requests per second per replica - type: Object object: metric: name: nginx_ingress_controller_latency describedObject: apiVersion: networking.k8s.io/v1 kind: Ingress name: vote target: type: Value value: 50 # Target 50 milliseconds behavior: scaleDown: policies: - type: Pods value: 2 periodSeconds: 120 - type: Percent value: 25 periodSeconds: 120 stabilizationWindowSeconds: 60 selectPolicy: Min scaleUp: stabilizationWindowSeconds: 45 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 2 periodSeconds: 15 selectPolicy: Max Apply it as kubectl delete hpa vote kubectl apply -f vote-custom-hpa.yaml validate kubectl get hpa [sample output] NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE vote-custom Deployment/vote 0/100, 0/50 2 10 2 10h now create a new load test config File: loadtest-ing-job.yaml apiVersion: batch/v1 kind: Job metadata: generateName: loadtest-ing spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: [\"siege\", \"--concurrent=2\", \"--benchmark\", \"--time=4m\", \"http://vote.staging.example.com\"] restartPolicy: Never hostAliases: - ip: \"ww.xx.yy.zz\" hostnames: - \"vote.example.com\" backoffLimit: 4 where, replace ww.xx.yy.zz with the INTERNAL-IP address of the node which runs ingress which you can find using kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-control-plane Ready control-plane 47h v1.29.2 172.18.0.2 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.13 kube-worker Ready 47h v1.29.2 172.18.0.3 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.13 kube-worker2 Ready 47h v1.29.2 172.18.0.4 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.13 in the above example, kube-worker is the node where ingress is running which has and ip address of 172.18.0.3 . Thats what should be used in the load test config. Before launching the load test, ensure that the http auth for the ingress is disabled. Remove the annotations block similar to follows from the ingress configuration for vote if you see it.... e.g. change the ingress spec from apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - Vote App' to apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote Finally create an instance of this job as, kubectl create -f loadtest-ing-job.yaml and now watch the scaling watch 'kubectl top pods; kubectl get hpa; kubectl get pods'","title":"Lab K302 - Autoscaling on Custom Metric"},{"location":"custom_autoscaling/#auto-scaling-on-custom-metric","text":"For this lab, you must have configured Prometheus and Grafana using help already. Refer to Lab K113 - HELM Package Manager - Kubernetes Tutorial with CKA/CKAD Prep to set it up.","title":"Auto Scaling on Custom Metric"},{"location":"custom_autoscaling/#setup-nginx-ingress-with-metrics","text":"Delete existing nginx ingress controller setup with KIND kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml Re deploy nginx ingress controller with helm, this time enabling the exposing the metrics which can then be scraped/collected by prometheus. helm upgrade --install ingress-nginx ingress-nginx \\ --repo https://kubernetes.github.io/ingress-nginx \\ --namespace ingress-nginx --create-namespace \\ --set controller.metrics.enabled=true \\ --set controller.metrics.serviceMonitor.enabled=true --set \\ controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" \\ --set controller.hostPort.enabled=true \\ --set controller.hostPort.ports.http=80 \\ --set controller.hostPort.ports.https=443 \\ --set-string controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\ --set-string controller.nodeSelector.ingress-ready=\"true\"","title":"Setup Nginx Ingress with Metrics"},{"location":"custom_autoscaling/#configure-prometheus-to-monitor-nginx-ingress","text":"Update the existing Prometheus installation to support scrpaing services in all namespaces assuming you deployed prometheus stack from kube-prometheus-stack path, switch to cd ~/kube-prometheus-stack/ and upgrade helm upgrade prom -n monitoring --values my.values.yaml . --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false What this does is starts collecting metrics from services from all the namespaces not just the one in which prometheus is deployed. And it reads the annotations from the pods to figure out where to collect the metrics from. Now, login to grafana and import custom dashboard for Nginx Ingress as Left menu (hover over +) -> Dashboard Click \"Import\" Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/grafana/dashboards/nginx.json Click Import JSON Select the Prometheus data source Click \"Import\" It may look similar to this, with possibly less data initially However, if you see some metric coming in, your setup with Nginx Ingress and Promethus Integration is working ! You may pat your back at this time :)","title":"Configure Prometheus to monitor Nginx Ingress"},{"location":"custom_autoscaling/#setup-prometheus-adapter","text":"Prometheus adapter converts the prometheus metrics and makes it available in kubernetes under custom.metrics.k8s.io api which can then read by the HPA and used as a input for autoscaling. Begin by creating the configuration for this adapter to pick a specific metric from prometheus and expose it under custom API as, File : prometheus-adapter-values.yaml prometheus: url: http://prom-kube-prometheus-stack-prometheus # Ensure this is correctly pointing to your Prometheus port: 9090 rules: default: false custom: - seriesQuery: 'nginx_ingress_controller_requests' resources: overrides: namespace: {resource: \"namespace\"} ingress: {resource: \"ingress\"} name: matches: \"^nginx_ingress_controller_requests$\" as: \"nginx_ingress_controller_requests_per_second\" metricsQuery: 'sum(rate(nginx_ingress_controller_requests{status=~\"2..\"}[2m])) by (namespace, ingress)' - seriesQuery: 'nginx_ingress_controller_request_duration_seconds_bucket' resources: overrides: namespace: {resource: \"namespace\"} ingress: {resource: \"ingress\"} name: matches: \"^nginx_ingress_controller_request_duration_seconds_bucket$\" as: \"nginx_ingress_controller_latency\" metricsQuery: 'sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{}[2m])) by (namespace, ingress, le) / 1000' install the adapter as, helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter -f prometheus-adapter-values.yaml -n monitoring validate its running kubectl get pods -n monitoring validate that custom apis are working with kubectl get --raw \"/apis/custom.metrics.k8s.io\" | jq . kubectl get --raw \"/apis/custom.metrics.k8s.io/v1beta1/namespaces/instavote/ingresses/vote/nginx_ingress_controller_requests_per_second\" | jq . kubectl get --raw \"/apis/custom.metrics.k8s.io/v1beta1/namespaces/instavote/ingresses/vote/nginx_ingress_controller_latency\" | jq . with the second command, you are expected to see [sample output] { \"kind\": \"MetricValueList\", \"apiVersion\": \"custom.metrics.k8s.io/v1beta1\", \"metadata\": {}, \"items\": [ { \"describedObject\": { \"kind\": \"Ingress\", \"namespace\": \"instavote\", \"name\": \"vote\", \"apiVersion\": \"networking.k8s.io/v1\" }, \"metricName\": \"nginx_ingress_controller_requests_per_second\", \"timestamp\": \"2024-05-10T02:41:54Z\", \"value\": \"0\", \"selector\": null } ] } If you see this, your adapter configuration integratd with prometheus is set up and ready to be consumed by the hpa.","title":"Setup Prometheus Adapter"},{"location":"custom_autoscaling/#configure-custom-autoscaling-policies","text":"Add autoscaling policies based on the custom metric as File : vote-custom-hpa.yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vote-custom namespace: instavote spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: vote minReplicas: 2 maxReplicas: 10 metrics: - type: Object object: metric: name: nginx_ingress_controller_requests_per_second describedObject: apiVersion: networking.k8s.io/v1 kind: Ingress name: vote target: type: Value value: 100 # Target 200 requests per second per replica - type: Object object: metric: name: nginx_ingress_controller_latency describedObject: apiVersion: networking.k8s.io/v1 kind: Ingress name: vote target: type: Value value: 50 # Target 50 milliseconds behavior: scaleDown: policies: - type: Pods value: 2 periodSeconds: 120 - type: Percent value: 25 periodSeconds: 120 stabilizationWindowSeconds: 60 selectPolicy: Min scaleUp: stabilizationWindowSeconds: 45 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 2 periodSeconds: 15 selectPolicy: Max Apply it as kubectl delete hpa vote kubectl apply -f vote-custom-hpa.yaml validate kubectl get hpa [sample output] NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE vote-custom Deployment/vote 0/100, 0/50 2 10 2 10h now create a new load test config File: loadtest-ing-job.yaml apiVersion: batch/v1 kind: Job metadata: generateName: loadtest-ing spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: [\"siege\", \"--concurrent=2\", \"--benchmark\", \"--time=4m\", \"http://vote.staging.example.com\"] restartPolicy: Never hostAliases: - ip: \"ww.xx.yy.zz\" hostnames: - \"vote.example.com\" backoffLimit: 4 where, replace ww.xx.yy.zz with the INTERNAL-IP address of the node which runs ingress which you can find using kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-control-plane Ready control-plane 47h v1.29.2 172.18.0.2 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.13 kube-worker Ready 47h v1.29.2 172.18.0.3 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.13 kube-worker2 Ready 47h v1.29.2 172.18.0.4 Debian GNU/Linux 12 (bookworm) 6.8.0-31-generic containerd://1.7.13 in the above example, kube-worker is the node where ingress is running which has and ip address of 172.18.0.3 . Thats what should be used in the load test config. Before launching the load test, ensure that the http auth for the ingress is disabled. Remove the annotations block similar to follows from the ingress configuration for vote if you see it.... e.g. change the ingress spec from apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - Vote App' to apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote Finally create an instance of this job as, kubectl create -f loadtest-ing-job.yaml and now watch the scaling watch 'kubectl top pods; kubectl get hpa; kubectl get pods'","title":"Configure Custom Autoscaling Policies"},{"location":"daemonsets_cronsjobs/","text":"DaemonSets & CronJobs In this lab you will explore about additional controllers such as DaemonSets, Jobs and CronJobs. Create a new Namespace and switch to it kubectl create namespace demo kubectl config set-context --current --namespace=demo kubectl config get-contexts Converting Vote Deployment to a DaemonSet cp vote-deploy.yaml vote-ds.yaml Edit vote-ds.yaml Change Kind to DaemonSet Remove spec.replicas Remove strategy Save file and apply as, kubectl apply -f vote-ds.yaml validate kubectl get ds,pods -o wide You shall notice that vote-xxx-yyy pods are running now via a DaemonSet which schedules one and only one instance of it on every node currently available for scheduling (Control Plane nodes are not schedulable by default). Once done, you could delete it as kubectl delete ds vote Creating a CronJob Jobs and CronJobs are similar, the one difference between the two is Cron Jobs as the name suggest runs on a schedule, versus Jobs are run once and run to completion. What that means is once the jobs is completed, there is no need keep running it unlike the pods which are created by the deployment etc. which keep running all the time. filename: every-minute-job.yaml apiVersion: batch/v1 kind: CronJob metadata: name: every-minute-job spec: schedule: \"* * * * *\" # This schedule runs at every minute jobTemplate: spec: template: spec: containers: - name: hello image: busybox args: - /bin/sh - -c - date; echo Hello from the Kubernetes cluster restartPolicy: OnFailure Here\u2019s a breakdown of the key components in this CronJob spec: apiVersion: batch/v1: Specifies the API version for the CronJob resource. kind: CronJob: Defines the kind of Kubernetes resource, which in this case is a CronJob. metadata: Metadata about the CronJob, such as its name. spec.schedule: The schedule on which the job should be run. The * * * * * pattern specifies that the job should run every minute. jobTemplate: The template for the job to be created. This specifies what the job will do when it runs. containers: A list of containers to run within the pod created by the job. The container uses the busybox image and executes a command to print the current date and time along with a custom message. restartPolicy: OnFailure: Specifies that the container should only be restarted if it fails. apply this code as, kubectl apply -f every-minute-job.yaml validate and watch the job run every minute with watch kubectl get pods,cronjobs [Sample Output after 2/3 minutes] Every 2.0s: kubectl get pods,cronjobs ci-01: Sun May 5 11:46:02 2024 NAME READY STATUS RESTARTS AGE pod/every-minute-job-28581825-mpxl6 0/1 Completed 0 62s pod/every-minute-job-28581826-528v4 0/1 Completed 0 2s NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob.batch/every-minute-job * * * * * False 1 2s 92s Observe how many instances of the pod are retained and figure out why. Once done, clean up with kubectl delete cronjob every-minute-job Clean up and Switch the Namespace back to Instavote kubectl delete namespace demo kubectl config set-context --current --namespace=instavote kubectl config get-contexts","title":"Lab K205 - DaemonsSets & CronJobs"},{"location":"daemonsets_cronsjobs/#daemonsets-cronjobs","text":"In this lab you will explore about additional controllers such as DaemonSets, Jobs and CronJobs. Create a new Namespace and switch to it kubectl create namespace demo kubectl config set-context --current --namespace=demo kubectl config get-contexts","title":"DaemonSets & CronJobs"},{"location":"daemonsets_cronsjobs/#converting-vote-deployment-to-a-daemonset","text":"cp vote-deploy.yaml vote-ds.yaml Edit vote-ds.yaml Change Kind to DaemonSet Remove spec.replicas Remove strategy Save file and apply as, kubectl apply -f vote-ds.yaml validate kubectl get ds,pods -o wide You shall notice that vote-xxx-yyy pods are running now via a DaemonSet which schedules one and only one instance of it on every node currently available for scheduling (Control Plane nodes are not schedulable by default). Once done, you could delete it as kubectl delete ds vote","title":"Converting Vote Deployment to a DaemonSet"},{"location":"daemonsets_cronsjobs/#creating-a-cronjob","text":"Jobs and CronJobs are similar, the one difference between the two is Cron Jobs as the name suggest runs on a schedule, versus Jobs are run once and run to completion. What that means is once the jobs is completed, there is no need keep running it unlike the pods which are created by the deployment etc. which keep running all the time. filename: every-minute-job.yaml apiVersion: batch/v1 kind: CronJob metadata: name: every-minute-job spec: schedule: \"* * * * *\" # This schedule runs at every minute jobTemplate: spec: template: spec: containers: - name: hello image: busybox args: - /bin/sh - -c - date; echo Hello from the Kubernetes cluster restartPolicy: OnFailure Here\u2019s a breakdown of the key components in this CronJob spec: apiVersion: batch/v1: Specifies the API version for the CronJob resource. kind: CronJob: Defines the kind of Kubernetes resource, which in this case is a CronJob. metadata: Metadata about the CronJob, such as its name. spec.schedule: The schedule on which the job should be run. The * * * * * pattern specifies that the job should run every minute. jobTemplate: The template for the job to be created. This specifies what the job will do when it runs. containers: A list of containers to run within the pod created by the job. The container uses the busybox image and executes a command to print the current date and time along with a custom message. restartPolicy: OnFailure: Specifies that the container should only be restarted if it fails. apply this code as, kubectl apply -f every-minute-job.yaml validate and watch the job run every minute with watch kubectl get pods,cronjobs [Sample Output after 2/3 minutes] Every 2.0s: kubectl get pods,cronjobs ci-01: Sun May 5 11:46:02 2024 NAME READY STATUS RESTARTS AGE pod/every-minute-job-28581825-mpxl6 0/1 Completed 0 62s pod/every-minute-job-28581826-528v4 0/1 Completed 0 2s NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob.batch/every-minute-job * * * * * False 1 2s 92s Observe how many instances of the pod are retained and figure out why. Once done, clean up with kubectl delete cronjob every-minute-job","title":"Creating a CronJob"},{"location":"daemonsets_cronsjobs/#clean-up-and-switch-the-namespace-back-to-instavote","text":"kubectl delete namespace demo kubectl config set-context --current --namespace=instavote kubectl config get-contexts","title":"Clean up and Switch the Namespace back to Instavote"},{"location":"docker-networking/","text":"Lab: Docker Networking Host Networking bridge host peer none Examine the existing network docker network ls NETWORK ID NAME DRIVER SCOPE b3d405dd37e4 bridge bridge local 7527c821537c host host local 773bea4ca095 none null local Creating new network docker network create -d bridge mynet validate docker network ls NETWORK ID NAME DRIVER SCOPE b3d405dd37e4 bridge bridge local 7527c821537c host host local 4e0d9b1a39f8 mynet bridge local 773bea4ca095 none null local docker network inspect mynet [ { \"Name\": \"mynet\", \"Id\": \"4e0d9b1a39f859af4811986534c91527146bc9d2ce178e5de02473c0f8ce62d5\", \"Created\": \"2018-05-03T04:44:19.187296148Z\", \"Scope\": \"local\", \"Driver\": \"bridge\", \"EnableIPv6\": false, \"IPAM\": { \"Driver\": \"default\", \"Options\": {}, \"Config\": [ { \"Subnet\": \"172.18.0.0/16\", \"Gateway\": \"172.18.0.1\" } ] }, \"Internal\": false, \"Attachable\": false, \"Ingress\": false, \"ConfigFrom\": { \"Network\": \"\" }, \"ConfigOnly\": false, \"Containers\": {}, \"Options\": {}, \"Labels\": {} } ] Launching containers in different bridges Launch two containers nt01 and nt02 in default bridge network docker container run -idt --name nt01 alpine sh docker container run -idt --name nt02 alpine sh Launch two containers nt03 and nt04 in mynet bridge network docker container run -idt --name nt03 --net mynet alpine sh docker container run -idt --name nt04 --net mynet alpine sh Now, lets examine if they can interconnect, docker exec nt01 ifconfig eth0 docker exec nt02 ifconfig eth0 docker exec nt03 ifconfig eth0 docker exec nt04 ifconfig eth0 This is what I see nt01 : 172.17.0.18 nt02 : 172.17.0.19 nt03 : 172.18.0.2 nt04 : 172.18.0.3 Create a table with the ips on your host. Once you do that, Try to, ping from nt01 to nt02 ping from nt01 to nt03 ping from nt03 to nt04 ping from nt03 to nt02 e.g. [replace ip addresses as per your setup] docker exec nt01 ping 172.17.0.19 docker exec nt01 ping 172.18.0.2 docker exec nt03 ping 172.17.0.19 docker exec nt03 ping 172.18.0.2 Clearly, these two are two differnt subnets/networks even though running on the same host. nt01 and nt02 can connect with each other, whereas nt03 and nt04 can connect. But connection between containers attached to two different subnets is not possible. Using None Network Driver docker container run -idt --name nt05 --net none alpine sh docker exec -it nt05 sh ifconfig Using Host Network Driver docker container run -idt --name nt05 --net host alpine sh docker exec -it nt05 sh ifconfig Observe docker bridge, routing and port mapping Exercise: Read about netshoot utility here Launch netshoot and connect to the host network docker run -it --net host --privileged nicolaka/netshoot Examine port mapping, iptables -nvL -t nat Traverse host port to container ip and port. Observe docker bridge and routing with the following command, brctl show ip route show","title":"Lab D103 - Docker Networking"},{"location":"docker-networking/#lab-docker-networking","text":"","title":"Lab: Docker Networking"},{"location":"docker-networking/#host-networking","text":"bridge host peer none Examine the existing network docker network ls NETWORK ID NAME DRIVER SCOPE b3d405dd37e4 bridge bridge local 7527c821537c host host local 773bea4ca095 none null local Creating new network docker network create -d bridge mynet validate docker network ls NETWORK ID NAME DRIVER SCOPE b3d405dd37e4 bridge bridge local 7527c821537c host host local 4e0d9b1a39f8 mynet bridge local 773bea4ca095 none null local docker network inspect mynet [ { \"Name\": \"mynet\", \"Id\": \"4e0d9b1a39f859af4811986534c91527146bc9d2ce178e5de02473c0f8ce62d5\", \"Created\": \"2018-05-03T04:44:19.187296148Z\", \"Scope\": \"local\", \"Driver\": \"bridge\", \"EnableIPv6\": false, \"IPAM\": { \"Driver\": \"default\", \"Options\": {}, \"Config\": [ { \"Subnet\": \"172.18.0.0/16\", \"Gateway\": \"172.18.0.1\" } ] }, \"Internal\": false, \"Attachable\": false, \"Ingress\": false, \"ConfigFrom\": { \"Network\": \"\" }, \"ConfigOnly\": false, \"Containers\": {}, \"Options\": {}, \"Labels\": {} } ]","title":"Host Networking"},{"location":"docker-networking/#launching-containers-in-different-bridges","text":"Launch two containers nt01 and nt02 in default bridge network docker container run -idt --name nt01 alpine sh docker container run -idt --name nt02 alpine sh Launch two containers nt03 and nt04 in mynet bridge network docker container run -idt --name nt03 --net mynet alpine sh docker container run -idt --name nt04 --net mynet alpine sh Now, lets examine if they can interconnect, docker exec nt01 ifconfig eth0 docker exec nt02 ifconfig eth0 docker exec nt03 ifconfig eth0 docker exec nt04 ifconfig eth0 This is what I see nt01 : 172.17.0.18 nt02 : 172.17.0.19 nt03 : 172.18.0.2 nt04 : 172.18.0.3 Create a table with the ips on your host. Once you do that, Try to, ping from nt01 to nt02 ping from nt01 to nt03 ping from nt03 to nt04 ping from nt03 to nt02 e.g. [replace ip addresses as per your setup] docker exec nt01 ping 172.17.0.19 docker exec nt01 ping 172.18.0.2 docker exec nt03 ping 172.17.0.19 docker exec nt03 ping 172.18.0.2 Clearly, these two are two differnt subnets/networks even though running on the same host. nt01 and nt02 can connect with each other, whereas nt03 and nt04 can connect. But connection between containers attached to two different subnets is not possible.","title":"Launching containers in different bridges"},{"location":"docker-networking/#using-none-network-driver","text":"docker container run -idt --name nt05 --net none alpine sh docker exec -it nt05 sh ifconfig","title":"Using None Network Driver"},{"location":"docker-networking/#using-host-network-driver","text":"docker container run -idt --name nt05 --net host alpine sh docker exec -it nt05 sh ifconfig","title":"Using Host Network Driver"},{"location":"docker-networking/#observe-docker-bridge-routing-and-port-mapping","text":"Exercise: Read about netshoot utility here Launch netshoot and connect to the host network docker run -it --net host --privileged nicolaka/netshoot Examine port mapping, iptables -nvL -t nat Traverse host port to container ip and port. Observe docker bridge and routing with the following command, brctl show ip route show","title":"Observe docker bridge, routing and port mapping"},{"location":"docker-volumes/","text":"Lab: Persistent Volumes with Docker Types of volumes automatic volumes named volumes volume binding Automatic Volumes docker container run -idt --name vt01 -v /var/lib/mysql alpine sh docker inspect vt01 | grep -i mounts -A 10 Named volumes docker container run -idt --name vt02 -v db-data:/var/lib/mysql alpine sh docker inspect vt02 | grep -i mounts -A 10 Volume binding mkdir /root/sysfoo docker container run -idt --name vt03 -v /root/sysfoo:/var/lib/mysql alpine sh docker inspect vt03 | grep -i mounts -A 10 Sharing files between host and the container ls /root/sysfoo/ touch /root/sysfoo/file1 docker exec -it vt03 sh ls sysfoo/","title":"Lab D104 - Docker Volumes"},{"location":"docker-volumes/#lab-persistent-volumes-with-docker","text":"","title":"Lab: Persistent Volumes with Docker"},{"location":"docker-volumes/#types-of-volumes","text":"automatic volumes named volumes volume binding Automatic Volumes docker container run -idt --name vt01 -v /var/lib/mysql alpine sh docker inspect vt01 | grep -i mounts -A 10 Named volumes docker container run -idt --name vt02 -v db-data:/var/lib/mysql alpine sh docker inspect vt02 | grep -i mounts -A 10 Volume binding mkdir /root/sysfoo docker container run -idt --name vt03 -v /root/sysfoo:/var/lib/mysql alpine sh docker inspect vt03 | grep -i mounts -A 10 Sharing files between host and the container ls /root/sysfoo/ touch /root/sysfoo/file1 docker exec -it vt03 sh ls sysfoo/","title":"Types of volumes"},{"location":"eks_cleanup/","text":"Cleanup Checklist [x] Delete EKS Cluster eksctl delete cluster --force -f cluster.yaml [x] Delete Cloudformation Stacks - check for reaming Cloudformation Stacks if any and delete those. [x] Delete S3 Bucket [x] Delete EC2 Volumes [x] Delete Load Balancers + Target Groups","title":"Lab K599 - EKS Cleanup"},{"location":"eks_cleanup/#cleanup-checklist","text":"[x] Delete EKS Cluster eksctl delete cluster --force -f cluster.yaml [x] Delete Cloudformation Stacks - check for reaming Cloudformation Stacks if any and delete those. [x] Delete S3 Bucket [x] Delete EC2 Volumes [x] Delete Load Balancers + Target Groups","title":"Cleanup Checklist"},{"location":"eks_cluster_autoscaler/","text":"Lab 06 - Scaling EKS Nodes When cluster nodes are exhausted of capacity, and when you have more pods to schedule, you would also want the underlying nodes to scale. This can be achieved in two differnt ways as follows Using Cluster Autoscale With Karpenter In this lab, you will set up Cluster Autoscaler to scale the node groups. Scale Down the NodeGroup Before starting to scale, you may want to scale down your node group to minimum, so that you could quickly see the autoscaler in action eksctl scale nodegroup --cluster eks-cluster-01 ng-2-workers --nodes 1 validate eksctl get nodegroups -c eks-cluster-01 kubectl get nodes wait for a few minutes for the addiontal nodes to be drained and removed, leaving only one node in the node group. Also make sure all the pods are rescheduled and are running kubectl get pods -A Set up Cluster Autoscaler Create IAM Policy for Cluster Autoscaler From IAM -> Policies -> Create Policy. Select JSON tab and add the following policy { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"autoscaling:DescribeAutoScalingGroups\", \"autoscaling:DescribeAutoScalingInstances\", \"autoscaling:DescribeLaunchConfigurations\", \"autoscaling:DescribeTags\", \"autoscaling:SetDesiredCapacity\", \"autoscaling:TerminateInstanceInAutoScalingGroup\", \"ec2:DescribeLaunchTemplateVersions\" ], \"Resource\": \"*\" } ] } Proceed to Next, add the policy name eg. ClusterAutoscaler along with a description. Proceed to create the policy. From IAM -> Roles , search for node which select the role associated with the nodegroup From Permissions -> Add Permissions -> Attach policies Select the policies created above e.g. ClusterAutoscaler and Add permissions. Now Install Cluster Autoscaler as kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml kubectl patch deployment cluster-autoscaler -n kube-system \\ --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/template/spec/containers/0/command/6\", \"value\": \"--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/eks-cluster-01\"}]' where, replace eks-cluster-01 with actual cluster name. Also ensure that this pod can not be evicted with kubectl -n kube-system annotate deployment.apps/cluster-autoscaler \\ cluster-autoscaler.kubernetes.io/safe-to-evict=\"false\" validate with kubectl get pods -n kube-system -l \"app=cluster-autoscaler\" kubectl describe pods -n kube-system -l \"app=cluster-autoscaler\" Now start watching the cluster autoscaler logs in one dedicated window so that you can see the autoscaling in action kubectl logs -f -n kube-system -l \"app=cluster-autoscaler\" Cluster Auotscaler in Action Ensure that the deployment spec for vote app has resources defined e.g. vote-deploy.yaml spec: containers: - image: 665496447754.dkr.ecr.ap-southeast-1.amazonaws.com/demo:v1 name: vote resources: requests: cpu: \"50m\" memory: \"64Mi\" limits: cpu: \"250m\" memory: \"128Mi\" without resources being defined properly, the scheduler and the autoscaler will not work optimally. Now, start scaling up in steps kubectl scale deploy vote --replicas=7 kubectl scale deploy vote --replicas=15 If you are watching the cluster autoscaler logs, you may see lines such as this: [sample autoscaler log] I0523 11:37:09.853039 1 klogx.go:87] Pod instavote/vote-6b7c9f85c5-2fnws is unschedulable I0523 11:37:09.853045 1 klogx.go:87] Pod instavote/vote-6b7c9f85c5-2b4cs is unschedulable I0523 11:37:09.853097 1 scale_up.go:194] Upcoming 0 nodes I0523 11:37:09.853458 1 waste.go:55] Expanding Node Group eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd would waste 95.00% CPU, 93.35% Memory, 94.17% Blended I0523 11:37:09.853476 1 scale_up.go:282] Best option to resize: eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd I0523 11:37:09.853487 1 scale_up.go:286] Estimated 1 nodes needed in eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd I0523 11:37:09.853511 1 scale_up.go:405] Final scale-up plan: [{eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd 2->3 (max: 5)}] I0523 11:37:09.853531 1 scale_up.go:608] Scale-up: setting group eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd size to 3 I0523 11:37:09.853560 1 auto_scaling_groups.go:248] Setting asg eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd size to 3 I0523 11:37:09.854328 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"kube-system\", Name:\"cluster-autoscaler-status\", UID:\"a52ab2f7-d7c4-4c82-bbed-058e1069e2b8\", APIVersion:\"v1\", ResourceVersion:\"273910\", FieldPath:\"\"}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd size to 3 instead of 2 (max: 5) this indicated the scaling activity. From EC2 => Auto Scaler Groups => Group Name => Activity, you could observe the node autoscaling as : After a few minutes, scale down to 2 kubectl scale deploy vote --replicas=2 In a few minutes time, you should start seeing cluster autoscaler trying to optimize and scale down the nodes I0523 11:45:32.175261 1 eligibility.go:102] Scale-down calculation: ignoring 1 nodes unremovable in the last 5m0s I0523 11:45:32.175281 1 cluster.go:153] ip-192-168-127-100.ap-southeast-1.compute.internal for removal I0523 11:45:32.175374 1 hinting_simulator.go:77] Pod instavote/vote-6b7c9f85c5-2fnws can be moved to ip-192-168-208-12.ap-southeast-1.compute.internal I0523 11:45:32.175390 1 cluster.go:176] node ip-192-168-127-100.ap-southeast-1.compute.internal may be removed I0523 11:45:32.175399 1 nodes.go:84] ip-192-168-127-100.ap-southeast-1.compute.internal is unneeded since 2024-05-23 11:43:21.695771883 +0000 UTC m=+769.391776888 duration 2m10.478714846s I0523 11:45:32.175434 1 static_autoscaler.go:589] Scale down status: lastScaleUpTime=2024-05-23 11:37:09.852038741 +0000 UTC m=+397.548043742 lastScaleDownDeleteTime=2024-05-23 10:30:38.231213092 +0000 UTC m=-3594.072781890 lastScaleDownFailTime=2024-05-23 10:30:38.231213092 +0000 UTC m=-3594.072781890 scaleDownForbidden=false scaleDownInCooldown=true I0523 11:47:12.590771 1 delete.go:103] Successfully added DeletionCandidateTaint on node ip-192-168-127-100.ap-southeast-1.compute.internal You could also corraborate from the EC2 Autoscaling Group to see the cluster nodes are scaling down.","title":"Lab K507 - EKS Cluster Autoscaler"},{"location":"eks_cluster_autoscaler/#lab-06-scaling-eks-nodes","text":"When cluster nodes are exhausted of capacity, and when you have more pods to schedule, you would also want the underlying nodes to scale. This can be achieved in two differnt ways as follows Using Cluster Autoscale With Karpenter In this lab, you will set up Cluster Autoscaler to scale the node groups.","title":"Lab 06 - Scaling EKS Nodes"},{"location":"eks_cluster_autoscaler/#scale-down-the-nodegroup","text":"Before starting to scale, you may want to scale down your node group to minimum, so that you could quickly see the autoscaler in action eksctl scale nodegroup --cluster eks-cluster-01 ng-2-workers --nodes 1 validate eksctl get nodegroups -c eks-cluster-01 kubectl get nodes wait for a few minutes for the addiontal nodes to be drained and removed, leaving only one node in the node group. Also make sure all the pods are rescheduled and are running kubectl get pods -A","title":"Scale Down the NodeGroup"},{"location":"eks_cluster_autoscaler/#set-up-cluster-autoscaler","text":"","title":"Set up Cluster Autoscaler"},{"location":"eks_cluster_autoscaler/#create-iam-policy-for-cluster-autoscaler","text":"From IAM -> Policies -> Create Policy. Select JSON tab and add the following policy { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"autoscaling:DescribeAutoScalingGroups\", \"autoscaling:DescribeAutoScalingInstances\", \"autoscaling:DescribeLaunchConfigurations\", \"autoscaling:DescribeTags\", \"autoscaling:SetDesiredCapacity\", \"autoscaling:TerminateInstanceInAutoScalingGroup\", \"ec2:DescribeLaunchTemplateVersions\" ], \"Resource\": \"*\" } ] } Proceed to Next, add the policy name eg. ClusterAutoscaler along with a description. Proceed to create the policy. From IAM -> Roles , search for node which select the role associated with the nodegroup From Permissions -> Add Permissions -> Attach policies Select the policies created above e.g. ClusterAutoscaler and Add permissions. Now Install Cluster Autoscaler as kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml kubectl patch deployment cluster-autoscaler -n kube-system \\ --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/template/spec/containers/0/command/6\", \"value\": \"--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/eks-cluster-01\"}]' where, replace eks-cluster-01 with actual cluster name. Also ensure that this pod can not be evicted with kubectl -n kube-system annotate deployment.apps/cluster-autoscaler \\ cluster-autoscaler.kubernetes.io/safe-to-evict=\"false\" validate with kubectl get pods -n kube-system -l \"app=cluster-autoscaler\" kubectl describe pods -n kube-system -l \"app=cluster-autoscaler\" Now start watching the cluster autoscaler logs in one dedicated window so that you can see the autoscaling in action kubectl logs -f -n kube-system -l \"app=cluster-autoscaler\"","title":"Create IAM Policy for Cluster Autoscaler"},{"location":"eks_cluster_autoscaler/#cluster-auotscaler-in-action","text":"Ensure that the deployment spec for vote app has resources defined e.g. vote-deploy.yaml spec: containers: - image: 665496447754.dkr.ecr.ap-southeast-1.amazonaws.com/demo:v1 name: vote resources: requests: cpu: \"50m\" memory: \"64Mi\" limits: cpu: \"250m\" memory: \"128Mi\" without resources being defined properly, the scheduler and the autoscaler will not work optimally. Now, start scaling up in steps kubectl scale deploy vote --replicas=7 kubectl scale deploy vote --replicas=15 If you are watching the cluster autoscaler logs, you may see lines such as this: [sample autoscaler log] I0523 11:37:09.853039 1 klogx.go:87] Pod instavote/vote-6b7c9f85c5-2fnws is unschedulable I0523 11:37:09.853045 1 klogx.go:87] Pod instavote/vote-6b7c9f85c5-2b4cs is unschedulable I0523 11:37:09.853097 1 scale_up.go:194] Upcoming 0 nodes I0523 11:37:09.853458 1 waste.go:55] Expanding Node Group eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd would waste 95.00% CPU, 93.35% Memory, 94.17% Blended I0523 11:37:09.853476 1 scale_up.go:282] Best option to resize: eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd I0523 11:37:09.853487 1 scale_up.go:286] Estimated 1 nodes needed in eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd I0523 11:37:09.853511 1 scale_up.go:405] Final scale-up plan: [{eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd 2->3 (max: 5)}] I0523 11:37:09.853531 1 scale_up.go:608] Scale-up: setting group eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd size to 3 I0523 11:37:09.853560 1 auto_scaling_groups.go:248] Setting asg eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd size to 3 I0523 11:37:09.854328 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:\"ConfigMap\", Namespace:\"kube-system\", Name:\"cluster-autoscaler-status\", UID:\"a52ab2f7-d7c4-4c82-bbed-058e1069e2b8\", APIVersion:\"v1\", ResourceVersion:\"273910\", FieldPath:\"\"}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group eks-ng-1-workers-eec7d297-f093-b47e-bb83-2cd15d8505fd size to 3 instead of 2 (max: 5) this indicated the scaling activity. From EC2 => Auto Scaler Groups => Group Name => Activity, you could observe the node autoscaling as : After a few minutes, scale down to 2 kubectl scale deploy vote --replicas=2 In a few minutes time, you should start seeing cluster autoscaler trying to optimize and scale down the nodes I0523 11:45:32.175261 1 eligibility.go:102] Scale-down calculation: ignoring 1 nodes unremovable in the last 5m0s I0523 11:45:32.175281 1 cluster.go:153] ip-192-168-127-100.ap-southeast-1.compute.internal for removal I0523 11:45:32.175374 1 hinting_simulator.go:77] Pod instavote/vote-6b7c9f85c5-2fnws can be moved to ip-192-168-208-12.ap-southeast-1.compute.internal I0523 11:45:32.175390 1 cluster.go:176] node ip-192-168-127-100.ap-southeast-1.compute.internal may be removed I0523 11:45:32.175399 1 nodes.go:84] ip-192-168-127-100.ap-southeast-1.compute.internal is unneeded since 2024-05-23 11:43:21.695771883 +0000 UTC m=+769.391776888 duration 2m10.478714846s I0523 11:45:32.175434 1 static_autoscaler.go:589] Scale down status: lastScaleUpTime=2024-05-23 11:37:09.852038741 +0000 UTC m=+397.548043742 lastScaleDownDeleteTime=2024-05-23 10:30:38.231213092 +0000 UTC m=-3594.072781890 lastScaleDownFailTime=2024-05-23 10:30:38.231213092 +0000 UTC m=-3594.072781890 scaleDownForbidden=false scaleDownInCooldown=true I0523 11:47:12.590771 1 delete.go:103] Successfully added DeletionCandidateTaint on node ip-192-168-127-100.ap-southeast-1.compute.internal You could also corraborate from the EC2 Autoscaling Group to see the cluster nodes are scaling down.","title":"Cluster Auotscaler in Action"},{"location":"eks_costs/","text":"Costs Monitoring with Kubecost Visit Kubecost from AWS Marketplace. Select Continue to Subscribe It will take a few minutes to process your request Once its configured, continue to Configure the software and Launch it on EKS Console. Fulfillment Options : Amazon EKS Add-on Kubecost - Amazon EKS Cost Monitoring - EKS Add On Software Version - Keep Auto Selected Once its configured, head over to EKS CLuster , go to Add-ons -> Get more add-ons. Search for kubecost Select the add-on, click on next and have it be installed. After its installed, validate with kubectl get all -n kubecost set the service type to NodePort as kubectl patch svc cost-analyzer -n kubecost --type='json' -p '[{\"op\":\"replace\",\"path\":\"/spec/type\",\"value\":\"NodePort\"}]' validate kubectl get svc -n kubecost [sample output] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cost-analyzer NodePort 10.100.149.159 9003:32367/TCP,9090:32751/TCP 93m pick up the port mapped with 9090 e.g. 32751 in above case and use that to access kubecost UI. e.g. From here on you should be explore kubecost insights.","title":"Lab K511 - EKS Costs"},{"location":"eks_costs/#costs-monitoring-with-kubecost","text":"Visit Kubecost from AWS Marketplace. Select Continue to Subscribe It will take a few minutes to process your request Once its configured, continue to Configure the software and Launch it on EKS Console. Fulfillment Options : Amazon EKS Add-on Kubecost - Amazon EKS Cost Monitoring - EKS Add On Software Version - Keep Auto Selected Once its configured, head over to EKS CLuster , go to Add-ons -> Get more add-ons. Search for kubecost Select the add-on, click on next and have it be installed. After its installed, validate with kubectl get all -n kubecost set the service type to NodePort as kubectl patch svc cost-analyzer -n kubecost --type='json' -p '[{\"op\":\"replace\",\"path\":\"/spec/type\",\"value\":\"NodePort\"}]' validate kubectl get svc -n kubecost [sample output] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cost-analyzer NodePort 10.100.149.159 9003:32367/TCP,9090:32751/TCP 93m pick up the port mapped with 9090 e.g. 32751 in above case and use that to access kubecost UI. e.g. From here on you should be explore kubecost insights.","title":"Costs Monitoring with Kubecost"},{"location":"eks_deploy_apps/","text":"Lab 03 - Deploy Apps on EKS In this project , you would write definitions for deploying the instavote application stack with all components/tiers which include, vote redis worker db result Project Specs Clone the code with git clone https://github.com/initcron/k8s-code.git and switch to k8s-code/projects/instavote/dev path. Apply the deployment and service code for the applications marked as ready Complete the code for deployments and services marked as TODO Apply the definitions that you have completed Validate the application workflow is operational by loading vote and result applications from the browser \u2800 Following table depicts the state of readiness of the above services. App Deployment Service vote TODO ready redis ready ready worker TODO n/a db ready ready result TODO TODO Phase I - Apply existing code Create a namespace and switch to it kubectl create namespace instavote kubectl config set-context --current --namespace=instavote kubectl config get-contexts Apply the existing manifests cd k8s-code/projects/instavote/dev/ kubectl apply -f vote-svc.yaml kubectl apply -f redis-deploy.yaml -f redis-svc.yaml kubectl apply -f db-deploy.yaml -f db-svc.yaml validate kubectl get all Where you should see, deplyoment and services for redis and db created nodeport service for vote app \u2800 If you see the above objects, proceed with the next task. Phase II - Migrate to a New Node Group You may notice that your pods are not running or stuck in containerCreating stage. This could be due to the insufficient capacity due to the t2.micro instance types. You could mitigate this by migrating to a new Node Group. This could be done by adding a new node group e.g. .... managedNodeGroups: - name: ng-1-workers labels: { role: workers } instanceType: t3.micro desiredCapacity: 1 minSize: 1 maxSize: 5 maxPodsPerNode: 17 ssh: allow: true publicKeyName: eks-spore tags: k8s.io/cluster-autoscaler/enabled: \"true\" k8s.io/cluster-autoscaler/eks-cluster-01: \"owned\" updateConfig: maxUnavailable: 1 - name: ng-2-workers labels: { role: workers } instanceType: t3.small desiredCapacity: 2 minSize: 1 maxSize: 4 maxPodsPerNode: 17 ssh: allow: true publicKeyName: eks-spore tags: k8s.io/cluster-autoscaler/enabled: \"true\" k8s.io/cluster-autoscaler/eks-cluster-01: \"owned\" updateConfig: maxUnavailable: 1 Notice the ng-2-workers node group definition above. It will create a new set of nodes using t3.medium instance type. To launch the new node group: eksctl create nodegroup -f cluster.yaml --include=ng-2-workers and to delete the previous one: eksctl delete nodegroup -f cluster.yaml --include=ng-1-workers --approve This should help you migrate the workloads to the new node group and help it run with sufficient resources. You could also scale it up with eksctl scale nodegroup --cluster eks-cluster-01 ng-2-workers --nodes 2 Phase III - Create Deployments and Services for Remaining Services You may find the files available in the same directory as above i.e. k8s-code/projects/instavote/dev/ with either partial or blank code. Your job is to complete the deployment and service yaml specs and apply those. While writing the specs, you could refer to the following specification. vote image: schoolofdevops/vote:v1 application port: 80 replicas: 2 service type: NodePort nodePort : 30000 worker image: schoolofdevops/worker:latest results image: schoolofdevops/vote-result application port: 80 replicas: 1 service type: NodePort nodePort : 30100 \u2800 To Validate: kubectl get all The above command should show, five deployments and four services created services for vote and result app should have been exposed with NodePort Find out the NodePort for vote and service apps and load those from your browser. This is how the vote application should look like. You could also load the Result page on the 30100 port of your Node IP/Address as, Above is how the result app should look like. Final validation is, if you submit the vote, you should see the results changing accordingly. You would see this behavior only if all deployments and services are created and configured properly.","title":"Lab K503 - Deploy Apps on EKS"},{"location":"eks_deploy_apps/#lab-03-deploy-apps-on-eks","text":"In this project , you would write definitions for deploying the instavote application stack with all components/tiers which include, vote redis worker db result","title":"Lab 03 - Deploy Apps on EKS"},{"location":"eks_deploy_apps/#project-specs","text":"Clone the code with git clone https://github.com/initcron/k8s-code.git and switch to k8s-code/projects/instavote/dev path. Apply the deployment and service code for the applications marked as ready Complete the code for deployments and services marked as TODO Apply the definitions that you have completed Validate the application workflow is operational by loading vote and result applications from the browser \u2800 Following table depicts the state of readiness of the above services. App Deployment Service vote TODO ready redis ready ready worker TODO n/a db ready ready result TODO TODO","title":"Project Specs"},{"location":"eks_deploy_apps/#phase-i-apply-existing-code","text":"Create a namespace and switch to it kubectl create namespace instavote kubectl config set-context --current --namespace=instavote kubectl config get-contexts Apply the existing manifests cd k8s-code/projects/instavote/dev/ kubectl apply -f vote-svc.yaml kubectl apply -f redis-deploy.yaml -f redis-svc.yaml kubectl apply -f db-deploy.yaml -f db-svc.yaml validate kubectl get all Where you should see, deplyoment and services for redis and db created nodeport service for vote app \u2800 If you see the above objects, proceed with the next task.","title":"Phase I - Apply existing code"},{"location":"eks_deploy_apps/#phase-ii-migrate-to-a-new-node-group","text":"You may notice that your pods are not running or stuck in containerCreating stage. This could be due to the insufficient capacity due to the t2.micro instance types. You could mitigate this by migrating to a new Node Group. This could be done by adding a new node group e.g. .... managedNodeGroups: - name: ng-1-workers labels: { role: workers } instanceType: t3.micro desiredCapacity: 1 minSize: 1 maxSize: 5 maxPodsPerNode: 17 ssh: allow: true publicKeyName: eks-spore tags: k8s.io/cluster-autoscaler/enabled: \"true\" k8s.io/cluster-autoscaler/eks-cluster-01: \"owned\" updateConfig: maxUnavailable: 1 - name: ng-2-workers labels: { role: workers } instanceType: t3.small desiredCapacity: 2 minSize: 1 maxSize: 4 maxPodsPerNode: 17 ssh: allow: true publicKeyName: eks-spore tags: k8s.io/cluster-autoscaler/enabled: \"true\" k8s.io/cluster-autoscaler/eks-cluster-01: \"owned\" updateConfig: maxUnavailable: 1 Notice the ng-2-workers node group definition above. It will create a new set of nodes using t3.medium instance type. To launch the new node group: eksctl create nodegroup -f cluster.yaml --include=ng-2-workers and to delete the previous one: eksctl delete nodegroup -f cluster.yaml --include=ng-1-workers --approve This should help you migrate the workloads to the new node group and help it run with sufficient resources. You could also scale it up with eksctl scale nodegroup --cluster eks-cluster-01 ng-2-workers --nodes 2","title":"Phase II - Migrate to a New Node Group"},{"location":"eks_deploy_apps/#phase-iii-create-deployments-and-services-for-remaining-services","text":"You may find the files available in the same directory as above i.e. k8s-code/projects/instavote/dev/ with either partial or blank code. Your job is to complete the deployment and service yaml specs and apply those. While writing the specs, you could refer to the following specification. vote image: schoolofdevops/vote:v1 application port: 80 replicas: 2 service type: NodePort nodePort : 30000 worker image: schoolofdevops/worker:latest results image: schoolofdevops/vote-result application port: 80 replicas: 1 service type: NodePort nodePort : 30100 \u2800","title":"Phase III - Create Deployments and Services for Remaining Services"},{"location":"eks_deploy_apps/#to-validate","text":"kubectl get all The above command should show, five deployments and four services created services for vote and result app should have been exposed with NodePort Find out the NodePort for vote and service apps and load those from your browser. This is how the vote application should look like. You could also load the Result page on the 30100 port of your Node IP/Address as, Above is how the result app should look like. Final validation is, if you submit the vote, you should see the results changing accordingly. You would see this behavior only if all deployments and services are created and configured properly.","title":"To Validate:"},{"location":"eks_ingress_alb/","text":"Lab 04 - Ingress with ALB In this lab you are going to learn how to route traffic to your applications running inside EKS using Application Load Balancer offered by AWS. Pre Requisites [ ] EKS Cluster [ ] Two Subnets in two AZs [ ] Public/Private Subnets should have relevant tags (Already done if created with Cloudformation Template) Install AWS Load Balancer Controller (LBC) The Load Balancer Controller (LBC) is a Kubernetes controller that manages Elastic Load Balancers (ELBs) for a Kubernetes cluster. It automatically provisions and configures AWS Application Load Balancers (ALBs) or Network Load Balancers (NLBs) to expose Kubernetes services to external traffic, ensuring scalability, high availability, and secure ingress traffic management. The LBC monitors Kubernetes Ingress resources and dynamically adjusts the configuration of the load balancers to reflect changes in the cluster, thereby simplifying the deployment of applications that require external access and integrating seamlessly with AWS networking and security features. Here is a diagram that explains how the Load Balancer Controller (LBC) works by listening to Ingress rules and controlling Application Load Balancers (ALBs): +---------------------+ +-----------------------+ | Kubernetes Cluster | | AWS | | | | | | | | | | +--------------+ | | +---------------+ | | | | | | | | | | | Ingress | | | | ALB | | | | Resource | | | | (Application | | | +------+-------+ | | | Load Balancer)| | | | | | +-------+-------+ | | | | | | | | +------+-------+ | | | | | | | | Listens to | | | | | LBC (Load +------------------------------->| | | | Balancer | | Ingress | Controls | | | | Controller) | | Rules | ALBs | | | +------+-------+ | | | | | | | | | | | | | | | | | +------+-------+ | | +-------+-------+ | | | | | | | | | | | Services/ +---------------------->| Target Groups | | | | Pods | | | | | | | | | | | +---------------+ | +---------------------+ +-----------------------+ Kubernetes Cluster : Represents your Kubernetes environment where applications are deployed. Ingress Resource : Defines rules for how inbound traffic should be routed to services within the cluster. Load Balancer Controller (LBC) : Watches for changes in Ingress resources and automatically manages ALBs in AWS. AWS : Represents the AWS cloud where the Application Load Balancer (ALB) and Target Groups are hosted. ALB (Application Load Balancer) : A type of load balancer that LBC configures to route external traffic based on the Ingress rules. Services/Pods : The backend endpoints within the Kubernetes cluster that the ALB routes traffic to, organized in target groups. \u2800 The LBC listens to the Ingress resources in the Kubernetes cluster and dynamically updates the ALB configuration in AWS to match the defined routing rules, ensuring seamless traffic management. Download IAM Policy curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/docs/install/iam_policy.json Create IAM Policy aws iam create-policy \\ --policy-name AWSLoadBalancerControllerIAMPolicy \\ --policy-document file://iam_policy.json Note the ARN which you will use in the next command. Create IAM Role using eksctl Replace eks-cluster-01 with the name of your cluster, 111122223333 with your account ID, and then run the command. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.\u2028 eksctl create iamserviceaccount \\ --cluster=eks-cluster-01 \\ --namespace=kube-system \\ --name=aws-load-balancer-controller \\ --role-name AmazonEKSLoadBalancerControllerRole \\ --attach-policy-arn=arn:aws:iam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \\ --approve validate eksctl get iamserviceaccount --cluster eks-cluster-01 Install AWS Load Balancer Controller using Helm V3 Add and update eks-charts Helm chart repository. AWS maintains this repository on GitHub. helm repo add eks https://aws.github.io/eks-charts helm repo update eks Install the AWS Load Balancer Controller. Replace eks-cluster-01 with the name of your cluster. In the following command, aws-load-balancer-controller is the Kubernetes service account that you created in a previous step. helm install aws-load-balancer-controller eks/aws-load-balancer-controller \\ -n kube-system \\ --set clusterName=eks-cluster-01 \\ --set serviceAccount.create=false \\ --set replicaCount=1 \\ --set serviceAccount.name=aws-load-balancer-controller validate helm list -A kubectl get deployment,pods -n kube-system -l \"app.kubernetes.io/instance=aws-load-balancer-controller\" Create Ingress Rules Create ingress rules to route traffic to the existing vote and result apps. File : vote-ing.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: namespace: instavote name: vote annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip spec: ingressClassName: alb rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 - host: result.example.com http: paths: - path: / pathType: Prefix backend: service: name: result port: number: 80 kubectl apply -f vote-ing.yaml kubectl get ing You could further get the details of the ALB using AWS CLI as aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-instavot-vote`) == `true`]' At this time, you shall see a Application Load Balancer(ALB) created with the rules autoamtically added to route traffic to vote and result apps. Add Local DNS You have created the ingress rules based on hostnames e.g. vote.example.com and result.example.com . In order for you to be able to access those, there has to be a dns entry pointing to your ALB. vote.example.com -------+ +----- vote:80 | +-------------+ | | | ingress | | +===> | node:80 | ===+ | +-------------+ | | | results.example.com -------+ +----- result:80 To achieve this you need to either, Find out the IP Address that your ALB is pointing to Create a DNS entry, provided you own the domain and have access to the dns management console. Create a local hosts file entry. On unix systems its in /etc/hosts file. On windows its at C:\\Windows\\System32\\drivers\\etc\\hosts . You need admin access to edit this file. \u2800 For example, on a linux or osx, you could edit it as, sudo vim /etc/hosts And add an entry such as , xxx.xxx.xxx.xxx vote.example.com results.example.com kube-ops-view.example.org where, xxx.xxx.xxx.xxx is one of the actual IP addresss of ALB. You could find the IP address by using the following command nslookup k8s-instavot-vote-e764f4b4a3-2050376139.ap-southeast-1.elb.amazonaws.com where DNS name is copied from ALB\u2019s DNS name And then access the app urls using http://vote.example.com or http://result.example.com Reference AWS Load Balancer Controller : What is the AWS Load Balancer Controller? Application load balancing on Amazon EKS Application load balancing on Amazon EKS Install ALB Ingress Controller using HELM : Install the AWS Load Balancer Controller using Helm Load balancers in Instance Mode vs IP Mode: IP mode Grouping Ingress Rules onto one ALB: Multiple Ingress pattern","title":"Lab K504 - Ingress with ALB"},{"location":"eks_ingress_alb/#lab-04-ingress-with-alb","text":"In this lab you are going to learn how to route traffic to your applications running inside EKS using Application Load Balancer offered by AWS.","title":"Lab 04 - Ingress with ALB"},{"location":"eks_ingress_alb/#pre-requisites","text":"[ ] EKS Cluster [ ] Two Subnets in two AZs [ ] Public/Private Subnets should have relevant tags (Already done if created with Cloudformation Template)","title":"Pre Requisites"},{"location":"eks_ingress_alb/#install-aws-load-balancer-controller-lbc","text":"The Load Balancer Controller (LBC) is a Kubernetes controller that manages Elastic Load Balancers (ELBs) for a Kubernetes cluster. It automatically provisions and configures AWS Application Load Balancers (ALBs) or Network Load Balancers (NLBs) to expose Kubernetes services to external traffic, ensuring scalability, high availability, and secure ingress traffic management. The LBC monitors Kubernetes Ingress resources and dynamically adjusts the configuration of the load balancers to reflect changes in the cluster, thereby simplifying the deployment of applications that require external access and integrating seamlessly with AWS networking and security features. Here is a diagram that explains how the Load Balancer Controller (LBC) works by listening to Ingress rules and controlling Application Load Balancers (ALBs): +---------------------+ +-----------------------+ | Kubernetes Cluster | | AWS | | | | | | | | | | +--------------+ | | +---------------+ | | | | | | | | | | | Ingress | | | | ALB | | | | Resource | | | | (Application | | | +------+-------+ | | | Load Balancer)| | | | | | +-------+-------+ | | | | | | | | +------+-------+ | | | | | | | | Listens to | | | | | LBC (Load +------------------------------->| | | | Balancer | | Ingress | Controls | | | | Controller) | | Rules | ALBs | | | +------+-------+ | | | | | | | | | | | | | | | | | +------+-------+ | | +-------+-------+ | | | | | | | | | | | Services/ +---------------------->| Target Groups | | | | Pods | | | | | | | | | | | +---------------+ | +---------------------+ +-----------------------+ Kubernetes Cluster : Represents your Kubernetes environment where applications are deployed. Ingress Resource : Defines rules for how inbound traffic should be routed to services within the cluster. Load Balancer Controller (LBC) : Watches for changes in Ingress resources and automatically manages ALBs in AWS. AWS : Represents the AWS cloud where the Application Load Balancer (ALB) and Target Groups are hosted. ALB (Application Load Balancer) : A type of load balancer that LBC configures to route external traffic based on the Ingress rules. Services/Pods : The backend endpoints within the Kubernetes cluster that the ALB routes traffic to, organized in target groups. \u2800 The LBC listens to the Ingress resources in the Kubernetes cluster and dynamically updates the ALB configuration in AWS to match the defined routing rules, ensuring seamless traffic management. Download IAM Policy curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/docs/install/iam_policy.json Create IAM Policy aws iam create-policy \\ --policy-name AWSLoadBalancerControllerIAMPolicy \\ --policy-document file://iam_policy.json Note the ARN which you will use in the next command. Create IAM Role using eksctl Replace eks-cluster-01 with the name of your cluster, 111122223333 with your account ID, and then run the command. If your cluster is in the AWS GovCloud (US-East) or AWS GovCloud (US-West) AWS Regions, then replace arn:aws: with arn:aws-us-gov:.\u2028 eksctl create iamserviceaccount \\ --cluster=eks-cluster-01 \\ --namespace=kube-system \\ --name=aws-load-balancer-controller \\ --role-name AmazonEKSLoadBalancerControllerRole \\ --attach-policy-arn=arn:aws:iam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \\ --approve validate eksctl get iamserviceaccount --cluster eks-cluster-01","title":"Install AWS Load Balancer Controller (LBC)"},{"location":"eks_ingress_alb/#install-aws-load-balancer-controller-using-helm-v3","text":"Add and update eks-charts Helm chart repository. AWS maintains this repository on GitHub. helm repo add eks https://aws.github.io/eks-charts helm repo update eks Install the AWS Load Balancer Controller. Replace eks-cluster-01 with the name of your cluster. In the following command, aws-load-balancer-controller is the Kubernetes service account that you created in a previous step. helm install aws-load-balancer-controller eks/aws-load-balancer-controller \\ -n kube-system \\ --set clusterName=eks-cluster-01 \\ --set serviceAccount.create=false \\ --set replicaCount=1 \\ --set serviceAccount.name=aws-load-balancer-controller validate helm list -A kubectl get deployment,pods -n kube-system -l \"app.kubernetes.io/instance=aws-load-balancer-controller\"","title":"Install AWS Load Balancer Controller using Helm V3"},{"location":"eks_ingress_alb/#create-ingress-rules","text":"Create ingress rules to route traffic to the existing vote and result apps. File : vote-ing.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: namespace: instavote name: vote annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip spec: ingressClassName: alb rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 - host: result.example.com http: paths: - path: / pathType: Prefix backend: service: name: result port: number: 80 kubectl apply -f vote-ing.yaml kubectl get ing You could further get the details of the ALB using AWS CLI as aws elbv2 describe-load-balancers --query 'LoadBalancers[?contains(LoadBalancerName, `k8s-instavot-vote`) == `true`]' At this time, you shall see a Application Load Balancer(ALB) created with the rules autoamtically added to route traffic to vote and result apps.","title":"Create Ingress Rules"},{"location":"eks_ingress_alb/#add-local-dns","text":"You have created the ingress rules based on hostnames e.g. vote.example.com and result.example.com . In order for you to be able to access those, there has to be a dns entry pointing to your ALB. vote.example.com -------+ +----- vote:80 | +-------------+ | | | ingress | | +===> | node:80 | ===+ | +-------------+ | | | results.example.com -------+ +----- result:80 To achieve this you need to either, Find out the IP Address that your ALB is pointing to Create a DNS entry, provided you own the domain and have access to the dns management console. Create a local hosts file entry. On unix systems its in /etc/hosts file. On windows its at C:\\Windows\\System32\\drivers\\etc\\hosts . You need admin access to edit this file. \u2800 For example, on a linux or osx, you could edit it as, sudo vim /etc/hosts And add an entry such as , xxx.xxx.xxx.xxx vote.example.com results.example.com kube-ops-view.example.org where, xxx.xxx.xxx.xxx is one of the actual IP addresss of ALB. You could find the IP address by using the following command nslookup k8s-instavot-vote-e764f4b4a3-2050376139.ap-southeast-1.elb.amazonaws.com where DNS name is copied from ALB\u2019s DNS name And then access the app urls using http://vote.example.com or http://result.example.com","title":"Add Local DNS"},{"location":"eks_ingress_alb/#reference","text":"AWS Load Balancer Controller : What is the AWS Load Balancer Controller? Application load balancing on Amazon EKS Application load balancing on Amazon EKS Install ALB Ingress Controller using HELM : Install the AWS Load Balancer Controller using Helm Load balancers in Instance Mode vs IP Mode: IP mode Grouping Ingress Rules onto one ALB: Multiple Ingress pattern","title":"Reference"},{"location":"eks_irsa/","text":"RBAC and IRSA(IAM Roles for Service Accounts) Overview In this lab, you'll learn how to deploy a Flask application on Amazon EKS using IAM Roles for Service Accounts (IRSA) to securely manage permissions for accessing AWS resources like S3. The Flask application ( schoolofdevops/s3checker:latest ) will check access to an S3 bucket and create a file in it to verify permissions. Why Use IRSA? IAM Roles for Service Accounts (IRSA) provides a way to securely assign AWS IAM roles to Kubernetes pods running on Amazon EKS. This allows your application to assume IAM roles and access AWS resources without needing to manage AWS credentials directly. This improves security by leveraging temporary credentials managed by AWS and reduces the risk of exposing sensitive information. +---------------------+ | AWS IAM | | +----------------+ | | | IAM Role | | | | (S3AccessRole) | | | +-------+--------+ | | | | | | | +----------|----------+ | | Assumes role v +---------------------+ | Amazon EKS Cluster | | +----------------+ | | | Service Account| | | | (s3-access-sa) | | | +-------+--------+ | | | | | | | +----------|----------+ | | Bound to v +----------------------------+ | EKS Pod | | +------------------------+ | | | Flask App Container | | | | (schoolofdevops/ | | | | s3checker:latest) | | | +---------+--------------+ | | | | | | | +-----------|----------------+ | | Uses IRSA v +----------------------------+ | AWS S3 Bucket | | (your-s3-bucket-name) | | | | - ListBucket | | - PutObject | | - GetObject | +----------------------------+ AWS IAM Role Creation : An IAM role ( S3AccessRole ) is created with a policy that allows access to the S3 bucket ( your-s3-bucket-name ). IAM Role Association with EKS Service Account : The IAM role is associated with a Kubernetes service account ( s3-access-sa ) in the EKS cluster. This is done using eksctl which sets up the necessary trust relationship. EKS Pod Deployment : A Kubernetes pod running the Flask application ( schoolofdevops/s3checker:latest ) is deployed in the EKS cluster. This pod uses the service account ( s3-access-sa ) which is annotated with the IAM role. IRSA Mechanism : The pod assumes the IAM role ( S3AccessRole ) through the service account ( s3-access-sa ). This is facilitated by the IRSA mechanism, which uses Kubernetes service account tokens and AWS STS (Security Token Service) to issue temporary credentials to the pod. Access S3 Bucket : The Flask application running inside the pod uses the temporary credentials obtained via IRSA to access the S3 bucket ( your-s3-bucket-name ). The application can list, put, and get objects in the bucket as per the IAM policy. \u2800 By using IRSA, the application running in the EKS pod can securely access AWS resources without embedding long-term AWS credentials in the application code or environment variables. Instead, it leverages temporary credentials that are managed and rotated by AWS. Prerequisites An AWS account An EKS cluster set up and running AWS CLI installed and configured kubectl installed and configured to access your EKS cluster eksctl installed for easy EKS management Deploy S3 Checker Service Create a S3 bucket in the same region, to test this service with. Note down the name of the bucket. Create a Kubernetes deployment + service YAML file File: s3checker-all.yaml apiVersion: apps/v1 kind: Deployment metadata: name: s3checker namespace: default spec: replicas: 1 selector: matchLabels: app: s3checker template: metadata: labels: app: s3checker spec: containers: - name: s3checker image: schoolofdevops/s3checker:latest ports: - containerPort: 5000 env: - name: BUCKET_NAME value: xxxxxxxx --- apiVersion: v1 kind: Service metadata: name: s3checker namespace: default spec: type: NodePort selector: app: s3checker ports: - protocol: TCP port: 80 targetPort: 5000 nodePort: 30500 where, * replace xxxxxxxx with bucket name. Apply the deployment using kubectl : kubectl apply -f s3checker-all.yaml validate kubectl get pods,svc -n default Access the application using the external IP of any Node and 30500 port. Configure IRSA Create an IAM policy that grants the necessary permissions to access your S3 bucket. { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"s3:ListBucket\", \"s3:PutObject\", \"s3:GetObject\" ], \"Resource\": [ \"arn:aws:s3:::your-s3-bucket-name\", \"arn:aws:s3:::your-s3-bucket-name/*\" ] } ] } where, replace your-s3-bucket-name with the name of the bucket you created earlier. Save this policy as s3-access-policy.json and create the policy using the AWS CLI: aws iam create-policy --policy-name S3AccessPolicy --policy-document file://s3-access-policy.json Note the ARN of the created policy. Create an IAM Role for the Service Account Use eksctl to create an IAM role and associate it with a Kubernetes service account: eksctl utils associate-iam-oidc-provider --region ap-southeast-1 --cluster eks-cluster-01 eksctl create iamserviceaccount \\ --name s3-access-sa \\ --namespace default \\ --cluster eks-cluster-01 \\ --attach-policy-arn arn:aws:iam:::policy/S3AccessPolicy \\ --approve \\ --override-existing-serviceaccounts validate eksctl get iamserviceaccount --cluster eks-cluster-01 kubectl get sa -n default Update Your Application Deployment to Use the Service Account File: s3checker-all.yaml ..... template: metadata: labels: app: s3checker spec: serviceAccountName: s3-access-sa containers: - name: s3checker Apply the deployment using kubectl : kubectl apply -f s3checker-all.yaml Now check the application to see if its able to connect to s3 bucket and check the s3 bucket if the application has created a file. Clean up Empty the s3 bucket and delete it. Delete deployment as kubectl delete -f s3checker-all.yaml Summary In this tutorial, you learned how to: Create an IAM policy and role for accessing S3. Associate an IAM role with a Kubernetes service account using IRSA. Deploy a Flask application on EKS using the configured service account. \u2800 By following these steps, you have securely deployed a Flask application on EKS with the necessary permissions to access AWS resources using IRSA. This method enhances security by managing permissions through IAM roles instead of static credentials.","title":"Lab K506 - IRSA"},{"location":"eks_irsa/#rbac-and-irsaiam-roles-for-service-accounts","text":"","title":"RBAC and IRSA(IAM Roles for Service Accounts)"},{"location":"eks_irsa/#overview","text":"In this lab, you'll learn how to deploy a Flask application on Amazon EKS using IAM Roles for Service Accounts (IRSA) to securely manage permissions for accessing AWS resources like S3. The Flask application ( schoolofdevops/s3checker:latest ) will check access to an S3 bucket and create a file in it to verify permissions.","title":"Overview"},{"location":"eks_irsa/#why-use-irsa","text":"IAM Roles for Service Accounts (IRSA) provides a way to securely assign AWS IAM roles to Kubernetes pods running on Amazon EKS. This allows your application to assume IAM roles and access AWS resources without needing to manage AWS credentials directly. This improves security by leveraging temporary credentials managed by AWS and reduces the risk of exposing sensitive information. +---------------------+ | AWS IAM | | +----------------+ | | | IAM Role | | | | (S3AccessRole) | | | +-------+--------+ | | | | | | | +----------|----------+ | | Assumes role v +---------------------+ | Amazon EKS Cluster | | +----------------+ | | | Service Account| | | | (s3-access-sa) | | | +-------+--------+ | | | | | | | +----------|----------+ | | Bound to v +----------------------------+ | EKS Pod | | +------------------------+ | | | Flask App Container | | | | (schoolofdevops/ | | | | s3checker:latest) | | | +---------+--------------+ | | | | | | | +-----------|----------------+ | | Uses IRSA v +----------------------------+ | AWS S3 Bucket | | (your-s3-bucket-name) | | | | - ListBucket | | - PutObject | | - GetObject | +----------------------------+ AWS IAM Role Creation : An IAM role ( S3AccessRole ) is created with a policy that allows access to the S3 bucket ( your-s3-bucket-name ). IAM Role Association with EKS Service Account : The IAM role is associated with a Kubernetes service account ( s3-access-sa ) in the EKS cluster. This is done using eksctl which sets up the necessary trust relationship. EKS Pod Deployment : A Kubernetes pod running the Flask application ( schoolofdevops/s3checker:latest ) is deployed in the EKS cluster. This pod uses the service account ( s3-access-sa ) which is annotated with the IAM role. IRSA Mechanism : The pod assumes the IAM role ( S3AccessRole ) through the service account ( s3-access-sa ). This is facilitated by the IRSA mechanism, which uses Kubernetes service account tokens and AWS STS (Security Token Service) to issue temporary credentials to the pod. Access S3 Bucket : The Flask application running inside the pod uses the temporary credentials obtained via IRSA to access the S3 bucket ( your-s3-bucket-name ). The application can list, put, and get objects in the bucket as per the IAM policy. \u2800 By using IRSA, the application running in the EKS pod can securely access AWS resources without embedding long-term AWS credentials in the application code or environment variables. Instead, it leverages temporary credentials that are managed and rotated by AWS.","title":"Why Use IRSA?"},{"location":"eks_irsa/#prerequisites","text":"An AWS account An EKS cluster set up and running AWS CLI installed and configured kubectl installed and configured to access your EKS cluster eksctl installed for easy EKS management","title":"Prerequisites"},{"location":"eks_irsa/#deploy-s3-checker-service","text":"Create a S3 bucket in the same region, to test this service with. Note down the name of the bucket. Create a Kubernetes deployment + service YAML file File: s3checker-all.yaml apiVersion: apps/v1 kind: Deployment metadata: name: s3checker namespace: default spec: replicas: 1 selector: matchLabels: app: s3checker template: metadata: labels: app: s3checker spec: containers: - name: s3checker image: schoolofdevops/s3checker:latest ports: - containerPort: 5000 env: - name: BUCKET_NAME value: xxxxxxxx --- apiVersion: v1 kind: Service metadata: name: s3checker namespace: default spec: type: NodePort selector: app: s3checker ports: - protocol: TCP port: 80 targetPort: 5000 nodePort: 30500 where, * replace xxxxxxxx with bucket name. Apply the deployment using kubectl : kubectl apply -f s3checker-all.yaml validate kubectl get pods,svc -n default Access the application using the external IP of any Node and 30500 port.","title":"Deploy S3 Checker Service"},{"location":"eks_irsa/#configure-irsa","text":"Create an IAM policy that grants the necessary permissions to access your S3 bucket. { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"s3:ListBucket\", \"s3:PutObject\", \"s3:GetObject\" ], \"Resource\": [ \"arn:aws:s3:::your-s3-bucket-name\", \"arn:aws:s3:::your-s3-bucket-name/*\" ] } ] } where, replace your-s3-bucket-name with the name of the bucket you created earlier. Save this policy as s3-access-policy.json and create the policy using the AWS CLI: aws iam create-policy --policy-name S3AccessPolicy --policy-document file://s3-access-policy.json Note the ARN of the created policy.","title":"Configure IRSA"},{"location":"eks_irsa/#create-an-iam-role-for-the-service-account","text":"Use eksctl to create an IAM role and associate it with a Kubernetes service account: eksctl utils associate-iam-oidc-provider --region ap-southeast-1 --cluster eks-cluster-01 eksctl create iamserviceaccount \\ --name s3-access-sa \\ --namespace default \\ --cluster eks-cluster-01 \\ --attach-policy-arn arn:aws:iam:::policy/S3AccessPolicy \\ --approve \\ --override-existing-serviceaccounts validate eksctl get iamserviceaccount --cluster eks-cluster-01 kubectl get sa -n default Update Your Application Deployment to Use the Service Account File: s3checker-all.yaml ..... template: metadata: labels: app: s3checker spec: serviceAccountName: s3-access-sa containers: - name: s3checker Apply the deployment using kubectl : kubectl apply -f s3checker-all.yaml Now check the application to see if its able to connect to s3 bucket and check the s3 bucket if the application has created a file.","title":"Create an IAM Role for the Service Account"},{"location":"eks_irsa/#clean-up","text":"Empty the s3 bucket and delete it. Delete deployment as kubectl delete -f s3checker-all.yaml","title":"Clean up"},{"location":"eks_irsa/#summary","text":"In this tutorial, you learned how to: Create an IAM policy and role for accessing S3. Associate an IAM role with a Kubernetes service account using IRSA. Deploy a Flask application on EKS using the configured service account. \u2800 By following these steps, you have securely deployed a Flask application on EKS with the necessary permissions to access AWS resources using IRSA. This method enhances security by managing permissions through IAM roles instead of static credentials.","title":"Summary"},{"location":"eks_monitoring/","text":"Lab K205 - Monitoring setup with HELM In this lab, you are going to install and configure helm, and in turns, use it to configure a monitoring system for kubernetes using prometheus and grafana stack. Installing Helm (version 3) To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version Deploy Prometheus Stack with HELM Read about kube-prometheus-stack 33.1.0 \u00b7 prometheus/prometheus-community chart at artifacthub.io Add helm repository using , helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update Download the chart as, cd ~ helm fetch --untar prometheus-community/kube-prometheus-stack Change into the charts directory cd kube-prometheus-stack/ ls Deploy prometheus stack as, kubectl create ns monitoring helm install prom -n monitoring prometheus-community/kube-prometheus-stack Validate helm list -A kubectl get all -n monitoring kubectl get pods,svc -n monitoring Customising Prometheus Configurations To update the Grafana service to use NodePort configurations, upgrade the helm release by setting up customized properties as, helm upgrade prom -n monitoring prometheus-community/kube-prometheus-stack \\ --set grafana.service.type=NodePort \\ --set grafana.service.nodePort=30200 Validate helm list -A kubectl get svc -n monitoring You should see new revision of monitoring stack deployed, and Grafana service changed to NodePort. Note down the node port and access Grafana with http://IPADDRESS:30200 remember to replace node name/ip address and node port as actual. Login using User : admin Pass: prom-operator Once logged in, you should be able to browse and view Grafana dashboards from Dashboards menu. An example dashboard is as follows, You could further explore various Grafana dashboards and configurations. Import EKS Dashboard You could import EKS Specific Dashboard onto grafana using the following instructions. From Dashboards -> New -> Import Provide Dashboard ID as 17119 , select Load Select Prometheus Data Source instance Import Once imported, go to Dashboard Settings -> Variables select node on configuration page,Query options -> Query select Metric as kube_pod_info Ensure Label is set to node Save Dashboard When you switch back to the dashboard you should see cluster health You could try installing additional dashboards e.g. Node Exporter Cluster Autoscaler Node Exporter for Prometheus Uninstalling the App with HELM Once you are done experiementing and learning, you could uninstall the application stack that you earlier installed with helm easily. To uninstall prometheus and grafana stack, begin by listing it helm list -A helm uninstall -n monitoring prom This should clean up everything that was deployed with helm earlier. Summary In this lab, we not only learnt about HELM, a kubernetes package manager, but also have setup a sophisticated health monitoring system with prometheus and grafana.","title":"Lab K510 - EKS Monitoring"},{"location":"eks_monitoring/#lab-k205-monitoring-setup-with-helm","text":"In this lab, you are going to install and configure helm, and in turns, use it to configure a monitoring system for kubernetes using prometheus and grafana stack.","title":"Lab K205 - Monitoring setup with HELM"},{"location":"eks_monitoring/#installing-helm-version-3","text":"To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version","title":"Installing Helm (version 3)"},{"location":"eks_monitoring/#deploy-prometheus-stack-with-helm","text":"Read about kube-prometheus-stack 33.1.0 \u00b7 prometheus/prometheus-community chart at artifacthub.io Add helm repository using , helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update Download the chart as, cd ~ helm fetch --untar prometheus-community/kube-prometheus-stack Change into the charts directory cd kube-prometheus-stack/ ls Deploy prometheus stack as, kubectl create ns monitoring helm install prom -n monitoring prometheus-community/kube-prometheus-stack Validate helm list -A kubectl get all -n monitoring kubectl get pods,svc -n monitoring","title":"Deploy Prometheus Stack with HELM"},{"location":"eks_monitoring/#customising-prometheus-configurations","text":"To update the Grafana service to use NodePort configurations, upgrade the helm release by setting up customized properties as, helm upgrade prom -n monitoring prometheus-community/kube-prometheus-stack \\ --set grafana.service.type=NodePort \\ --set grafana.service.nodePort=30200 Validate helm list -A kubectl get svc -n monitoring You should see new revision of monitoring stack deployed, and Grafana service changed to NodePort. Note down the node port and access Grafana with http://IPADDRESS:30200 remember to replace node name/ip address and node port as actual. Login using User : admin Pass: prom-operator Once logged in, you should be able to browse and view Grafana dashboards from Dashboards menu. An example dashboard is as follows, You could further explore various Grafana dashboards and configurations.","title":"Customising Prometheus Configurations"},{"location":"eks_monitoring/#import-eks-dashboard","text":"You could import EKS Specific Dashboard onto grafana using the following instructions. From Dashboards -> New -> Import Provide Dashboard ID as 17119 , select Load Select Prometheus Data Source instance Import Once imported, go to Dashboard Settings -> Variables select node on configuration page,Query options -> Query select Metric as kube_pod_info Ensure Label is set to node Save Dashboard When you switch back to the dashboard you should see cluster health You could try installing additional dashboards e.g. Node Exporter Cluster Autoscaler Node Exporter for Prometheus","title":"Import EKS Dashboard"},{"location":"eks_monitoring/#uninstalling-the-app-with-helm","text":"Once you are done experiementing and learning, you could uninstall the application stack that you earlier installed with helm easily. To uninstall prometheus and grafana stack, begin by listing it helm list -A helm uninstall -n monitoring prom This should clean up everything that was deployed with helm earlier.","title":"Uninstalling the App with HELM"},{"location":"eks_monitoring/#summary","text":"In this lab, we not only learnt about HELM, a kubernetes package manager, but also have setup a sophisticated health monitoring system with prometheus and grafana.","title":"Summary"},{"location":"eks_prep/","text":"Lab 01 - EKS Preparatory Setup Author: Gourav Shah Publisher: School of Devops Version : 22.05.2024.01 This lab is about getting ready to setup a EKS Cluster on AWS. Before you begin, make sure you have a operational AWS Account with access to create IAM Users and access to all relevant services including IAM VPC EKS S3 * EC2 Installing Pre-requisite Tools Install eksctl If you haven't installed eksctl yet, you can do so by following these instructions: macOS/Linux: curl --silent --location \"https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz\" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin Windows: Download the latest eksctl binary from the releases page and add it to your system path. Refer to the official documentation here Installation to get the exact and up to date instructions to install eksctl as well as auto shell completions. Install kubectl macOS/Linux: curl -LO \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/$(uname -s | tr '[:upper:]' '[:lower:]')/amd64/kubectl\" chmod +x kubectl sudo mv kubectl /usr/local/bin Windows: Download the latest release of kubectl from the official Kubernetes releases page . Rename the downloaded file to kubectl.exe . Move the file to a directory included in your system's PATH . \u2800 You can also install kubectl on Windows using chocolatey or scoop : Using Chocolatey: choco install kubernetes-cli Using Scoop: scoop install kubectl Verify Installation After installing kubectl , verify the installation by checking the version: kubectl version --client This command should display the client version of kubectl , indicating that it has been installed successfully. Install aws CLI Refer to the official documentation to install aws CLI with os specific instructions from Install or update to the latest version of the AWS CLI - AWS Command Line Interface Install Helm To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version Configuring EKS Admin User When configuring the AWS CLI to create and manage an EKS cluster using eksctl , the IAM user (or role) whose credentials you are using needs to have sufficient permissions to perform all the necessary actions. This includes permissions to create and manage EKS clusters, EC2 instances, VPCs, IAM roles, and other related resources. Here\u2019s a list of the permissions required: Required IAM Permissions Amazon EKS Full Access Amazon EC2 Full Access AWS CloudFormation Full Access IAM Permissions to Create Roles and Policies Amazon S3 Full Access (if you need to use S3 for storing logs or other purposes) Systems Manager (SSM) - Optional if you use Parameter Store AWS Key Management Service (KMS) - Optional if you encrypt Secrets using KMS Here\u2019s a detailed policy document that grants the necessary permissions. You can create a custom IAM policy with these permissions and attach it to the IAM user or role. Refer to Installation documentation to understand the authorization required for the user which will run eksctl JSON Policy (use it in next step) { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"eks:*\", \"ec2:*\", \"cloudformation:*\", \"iam:*\", \"ecr:*\", \"autoscaling:*\", \"cloudwatch:*\", \"elasticloadbalancing:*\", \"s3:*\", \"ssm:GetParameter\", \"ssm:DescribeParameters\", \"kms:ListAliases\", \"kms:DescribeKey\", \"kms:CreateGrant\", \"kms:RetireGrant\", \"kms:RevokeGrant\", \"sts:AssumeRole\" ], \"Resource\": \"*\" } ] } Steps to Attach the Policy Create a Custom Policy: Go to the IAM console in AWS. Click on Policies in the left-hand menu. Click on Create policy . Select the JSON tab. Copy and paste the above JSON policy document. Click on Next button to Review policy . Name the policy (e.g., EKSFullAccessPolicy ) and provide a description. Click on Create policy . Attach the Policy to a User: Go to Users in the IAM console. Create a new user eks-admin who will be creating the EKS cluster. Do not select Provide user access to the AWS Management Console. Select Next . Select Attach policies directly and search for EKSFullAccessPolicy as created above. Select Next . Select the policy and click on Next: Review , then click on Create user . Create Access Keys for eks-admin User: Go to Users in the IAM console. Select eks-admin user From Security credentials select Create access key Select use case as Command Line Interface (CLI) , provide confirmation and select Next . Add a description value as something e.g. AWS CLI Utility and proceed to Create access key . Note the Access Key and Secret key which you will use in the next step to configure with AWS CLI. Configure AWS CLI with EKS Admin User Once you have created the user, proceed to create Security Credentials and generate the AWS Access Key + AWS Secret Key pair. Then make sure your AWS CLI is configured with the necessary permissions to create EKS clusters. You can configure it using: aws configure You'll need your AWS Access Key ID, Secret Access Key, region, and output format. The available output formats are: json : The default output format. Provides the output in JSON format. text : Provides the output in plain text format. This format is useful for simple parsing and readability. table : Provides the output in a readable table format. This format is useful for human readability but less suitable for parsing. yaml : Provides the output in YAML format. This is useful for configurations and other uses where YAML is preferred. yaml-stream : Provides the output in a streaming YAML format, where each document is separated by --- . This format is useful for continuous data processing. Select region as ap-southeast-1 to be consistent with the labs created as part of this course. [sample output] \u279c ~ aws configure AWS Access Key ID [****************QSJX]: XXXX AWS Secret Access Key [****************P01d]: YYYY Default region name [ap-southeast-1]: ap-southeast-1 Default output format [None]: table \u279c ~ cat ~/.aws/config [default] region = ap-southeast-1 output = table Create SSH key Pair To create a SSH key pair that you would use to log into the nodes created with EKS, * Switch to EC2 Service from AWS Console, and make sure you have selected the same region you are going to launch EKS cluster inside e.g. Singapore * Select Key Pairs from left side menu header Network & Security . * Create a key pair of type RSA and download a .pem or .ppk file based on how and from where you are going to connect. Move the downloaded private key to a safe location and change its permissions. e.g. mv ~/Downloads/eks-spore.pem ~/.ssh ls -al ~/.ssh/eks-spore.pem chmod 600 ~/.ssh/eks-spore.pem ls -al ~/.ssh/eks-spore.pem Now that the preparatory setup is done, you could proceed to create a EKS Cluster.","title":"Lab K501 - EKS Preparatory Setup"},{"location":"eks_prep/#lab-01-eks-preparatory-setup","text":"Author: Gourav Shah Publisher: School of Devops Version : 22.05.2024.01 This lab is about getting ready to setup a EKS Cluster on AWS. Before you begin, make sure you have a operational AWS Account with access to create IAM Users and access to all relevant services including IAM VPC EKS S3 * EC2","title":"Lab 01 - EKS Preparatory Setup"},{"location":"eks_prep/#installing-pre-requisite-tools","text":"","title":"Installing Pre-requisite Tools"},{"location":"eks_prep/#install-eksctl","text":"If you haven't installed eksctl yet, you can do so by following these instructions: macOS/Linux: curl --silent --location \"https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz\" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin Windows: Download the latest eksctl binary from the releases page and add it to your system path. Refer to the official documentation here Installation to get the exact and up to date instructions to install eksctl as well as auto shell completions.","title":"Install eksctl"},{"location":"eks_prep/#install-kubectl","text":"macOS/Linux: curl -LO \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/$(uname -s | tr '[:upper:]' '[:lower:]')/amd64/kubectl\" chmod +x kubectl sudo mv kubectl /usr/local/bin Windows: Download the latest release of kubectl from the official Kubernetes releases page . Rename the downloaded file to kubectl.exe . Move the file to a directory included in your system's PATH . \u2800 You can also install kubectl on Windows using chocolatey or scoop : Using Chocolatey: choco install kubernetes-cli Using Scoop: scoop install kubectl","title":"Install kubectl"},{"location":"eks_prep/#verify-installation","text":"After installing kubectl , verify the installation by checking the version: kubectl version --client This command should display the client version of kubectl , indicating that it has been installed successfully.","title":"Verify Installation"},{"location":"eks_prep/#install-aws-cli","text":"Refer to the official documentation to install aws CLI with os specific instructions from Install or update to the latest version of the AWS CLI - AWS Command Line Interface","title":"Install aws CLI"},{"location":"eks_prep/#install-helm","text":"To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version","title":"Install Helm"},{"location":"eks_prep/#configuring-eks-admin-user","text":"When configuring the AWS CLI to create and manage an EKS cluster using eksctl , the IAM user (or role) whose credentials you are using needs to have sufficient permissions to perform all the necessary actions. This includes permissions to create and manage EKS clusters, EC2 instances, VPCs, IAM roles, and other related resources. Here\u2019s a list of the permissions required:","title":"Configuring EKS Admin User"},{"location":"eks_prep/#required-iam-permissions","text":"Amazon EKS Full Access Amazon EC2 Full Access AWS CloudFormation Full Access IAM Permissions to Create Roles and Policies Amazon S3 Full Access (if you need to use S3 for storing logs or other purposes) Systems Manager (SSM) - Optional if you use Parameter Store AWS Key Management Service (KMS) - Optional if you encrypt Secrets using KMS Here\u2019s a detailed policy document that grants the necessary permissions. You can create a custom IAM policy with these permissions and attach it to the IAM user or role. Refer to Installation documentation to understand the authorization required for the user which will run eksctl JSON Policy (use it in next step) { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": [ \"eks:*\", \"ec2:*\", \"cloudformation:*\", \"iam:*\", \"ecr:*\", \"autoscaling:*\", \"cloudwatch:*\", \"elasticloadbalancing:*\", \"s3:*\", \"ssm:GetParameter\", \"ssm:DescribeParameters\", \"kms:ListAliases\", \"kms:DescribeKey\", \"kms:CreateGrant\", \"kms:RetireGrant\", \"kms:RevokeGrant\", \"sts:AssumeRole\" ], \"Resource\": \"*\" } ] }","title":"Required IAM Permissions"},{"location":"eks_prep/#steps-to-attach-the-policy","text":"Create a Custom Policy: Go to the IAM console in AWS. Click on Policies in the left-hand menu. Click on Create policy . Select the JSON tab. Copy and paste the above JSON policy document. Click on Next button to Review policy . Name the policy (e.g., EKSFullAccessPolicy ) and provide a description. Click on Create policy . Attach the Policy to a User: Go to Users in the IAM console. Create a new user eks-admin who will be creating the EKS cluster. Do not select Provide user access to the AWS Management Console. Select Next . Select Attach policies directly and search for EKSFullAccessPolicy as created above. Select Next . Select the policy and click on Next: Review , then click on Create user . Create Access Keys for eks-admin User: Go to Users in the IAM console. Select eks-admin user From Security credentials select Create access key Select use case as Command Line Interface (CLI) , provide confirmation and select Next . Add a description value as something e.g. AWS CLI Utility and proceed to Create access key . Note the Access Key and Secret key which you will use in the next step to configure with AWS CLI.","title":"Steps to Attach the Policy"},{"location":"eks_prep/#configure-aws-cli-with-eks-admin-user","text":"Once you have created the user, proceed to create Security Credentials and generate the AWS Access Key + AWS Secret Key pair. Then make sure your AWS CLI is configured with the necessary permissions to create EKS clusters. You can configure it using: aws configure You'll need your AWS Access Key ID, Secret Access Key, region, and output format. The available output formats are: json : The default output format. Provides the output in JSON format. text : Provides the output in plain text format. This format is useful for simple parsing and readability. table : Provides the output in a readable table format. This format is useful for human readability but less suitable for parsing. yaml : Provides the output in YAML format. This is useful for configurations and other uses where YAML is preferred. yaml-stream : Provides the output in a streaming YAML format, where each document is separated by --- . This format is useful for continuous data processing. Select region as ap-southeast-1 to be consistent with the labs created as part of this course. [sample output] \u279c ~ aws configure AWS Access Key ID [****************QSJX]: XXXX AWS Secret Access Key [****************P01d]: YYYY Default region name [ap-southeast-1]: ap-southeast-1 Default output format [None]: table \u279c ~ cat ~/.aws/config [default] region = ap-southeast-1 output = table","title":"Configure AWS CLI with EKS Admin User"},{"location":"eks_prep/#create-ssh-key-pair","text":"To create a SSH key pair that you would use to log into the nodes created with EKS, * Switch to EC2 Service from AWS Console, and make sure you have selected the same region you are going to launch EKS cluster inside e.g. Singapore * Select Key Pairs from left side menu header Network & Security . * Create a key pair of type RSA and download a .pem or .ppk file based on how and from where you are going to connect. Move the downloaded private key to a safe location and change its permissions. e.g. mv ~/Downloads/eks-spore.pem ~/.ssh ls -al ~/.ssh/eks-spore.pem chmod 600 ~/.ssh/eks-spore.pem ls -al ~/.ssh/eks-spore.pem Now that the preparatory setup is done, you could proceed to create a EKS Cluster.","title":"Create SSH key Pair"},{"location":"eks_setup/","text":"Lab 02 - Setting up EKS Cluster with eksctl Author: Gourav Shah Publisher: School of Devops Version : 22.05.2024.01 While setting up EKS Cluster on AWS, there are two main options that you could consider Using eksctl which is a dedicated EKS Cluster Management Tool Using aws CLI along with kubectl which is a more complex approach, but comes with more control in your hands Both eksctl and the combination of aws cli with kubectl have their advantages and use cases when creating and managing AWS EKS clusters. Here's a comparison to help you decide which approach might be best for your course: eksctl Pros: 1. Ease of Use: eksctl is specifically designed to simplify the creation and management of EKS clusters. It abstracts many of the complexities involved. 2. Quick Setup: With a single command, you can create a fully functioning EKS cluster. 3. Default Best Practices: It follows AWS best practices for cluster creation and configuration. 4. Less Code: Requires fewer commands and less code to manage the cluster. \u2800 Cons: 1. Less Granular Control: While it simplifies many tasks, it might abstract away some of the details, giving you less control over specific configurations. 2. Dependency: Adds an additional dependency that students need to install and manage. \u2800 aws cli with kubectl Pros: 1. Granular Control: Provides more control over the EKS cluster configuration and setup, allowing for fine-tuning and customization. 2. Learning Opportunity: Teaches students more about the underlying AWS services and Kubernetes configurations. 3. Versatility: aws cli and kubectl are versatile tools that can be used for a wide range of AWS and Kubernetes operations beyond just cluster creation. Cons: 1. Complexity: Requires more steps and a deeper understanding of AWS and Kubernetes concepts, which might be overwhelming for beginners. 2. More Commands: Involves writing more commands and scripts, which can be error-prone. As part of this lab, you will take a simpler approach and use eksctl to quickly create a cluster. Create VPC with Public Subnets EKS needs a VPC to launch itself in. There are three different options while creating the VPCs as follows VPC with Public and Private Subnets VPC with only Public Subnets VPC with only Private Subnets While you would ideally create VPC with public and private subnets so that you could host your applications in private subnets and set up load balancers and ingress controllers in public, as part of this lab guide, you are going to set up VPC with public subnets only. This is to avoid creation of two NAT Gateways, which would incur additional costs. The VPC that you are going to create has three public subnets that are deployed into different Availability Zones in an AWS Region. All nodes are automatically assigned public IPv4 addresses and can send and receive internet traffic through an internet gateway . A security group is deployed that denies all inbound traffic and allows all outbound traffic. The subnets are tagged so that Kubernetes can deploy load balancers to them. If you choose to create a VPC with a different configuration, please pick a relevant template from Creating a VPC for your Amazon EKS cluster . \u2800 To create your VPC Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation . From the navigation bar, select an AWS Region that supports EKS e.g. Singapore . Choose Create stack , With new resources (standard) . From Prepare Temaplte, select Choose and existing template and Under Template source , select Amazon S3 URL . Paste the following URL into the text area under Amazon S3 URL and choose Next : https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-vpc-sample.yaml On the Specify Details page, enter the stack name e.g. eks-vpc-01 and update parameters if you want to change any, and then choose Next and create VPC stack. Proceed to Review and create and submit the stack. It takes a few minutes to have the status change from to once VPC is created, you could verify everything from VPC console. Also note down the VpcId and SubnetIds from the Outputs . Configure IAM Role for EKS Step 1: Create the IAM Role Open the IAM Console : Go to the IAM console . Create a Role : Click on Roles and then Create role . Choose EKS as the service that will use this role. Select the EKS - Cluster use case. From Add Permissions page you will see the following policy already attached: AmazonEKSClusterPolicy Select Next From Name, review and create page , provide a Role name e.g. EKSClusterServiceRole and proceed to Create role . From IAM Console -> Roles search for EKSClusterServiceRole and select it. From Permissions tab, Add Permissions dropdown, Choose Attach policies. Search for AmazonEKSVPCResourceController and attach it to the role. Set the Trust Relationship : Ensure the trust relationship for the role allows EKS to assume the role. The trust policy should look like this: { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Principal\": { \"Service\": \"eks.amazonaws.com\" }, \"Action\": \"sts:AssumeRole\" } ] } This should already be there. Verify it from the Trust relationships tab. \u2800 Note down the role ARN e.g. arn:aws:iam::665496447754:role/EKSClusterServiceRole which you would add to cluster configuration later. Also note the new policy being added to the role. Launch EKS Cluster with eksctl Start with eksctl eksctl info Create cluster configurarion as follows. You could refer to Config File Schema to explore more options. File: cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: eks-cluster-01 region: ap-southeast-1 vpc: id: \"vpc-aaaa\" subnets: public: apsoutheast1a: id: subnet-xxxx apsoutheast1b: id: subnet-yyyy apsoutheast1c: id: subnet-zzzz managedNodeGroups: - name: ng-1-workers labels: { role: workers } instanceType: t2.micro desiredCapacity: 2 maxPodsPerNode: 100 minSize: 1 maxSize: 4 ssh: allow: true publicKeyName: xxxx tags: k8s.io/cluster-autoscaler/enabled: \"true\" k8s.io/cluster-autoscaler/eks-cluster-01: \"owned\" iam: withOIDC: true serviceRoleARN: arn:aws:iam::xxxxxxxxx:role/EKSClusterServiceRole \u2800 Edit this file and replace the following with actual values * vpc id * subnets * public key name (listed ssh key on ec2 that you would like to ssh in with) * ServiceRoleARN eksctl create cluster -f cluster.yaml [sample output] 2024-05-22 14:53:47 [\u2714] all EKS cluster resources for \"eks-cluster-01\" have been created 2024-05-22 14:53:48 [\u2139] nodegroup \"ng-1-workers\" has 1 node(s) 2024-05-22 14:53:48 [\u2139] node \"ip-192-168-240-31.ap-southeast-1.compute.internal\" is ready 2024-05-22 14:53:48 [\u2139] waiting for at least 1 node(s) to become ready in \"ng-1-workers\" 2024-05-22 14:53:48 [\u2139] nodegroup \"ng-1-workers\" has 1 node(s) 2024-05-22 14:53:48 [\u2139] node \"ip-192-168-240-31.ap-southeast-1.compute.internal\" is ready 2024-05-22 14:53:48 [\u2714] created 1 nodegroup(s) in cluster \"eks-cluster-01\" 2024-05-22 14:53:48 [\u2714] created 0 managed nodegroup(s) in cluster \"eks-cluster-01\" 2024-05-22 14:53:49 [\u2139] kubectl command should work with \"/Users/gshah/.kube/config\", try 'kubectl get nodes' 2024-05-22 14:53:49 [\u2714] EKS cluster \"eks-cluster-01\" in \"ap-southeast-1\" region is ready validate eksctl get cluster eksctl get nodegroup --cluster eks-cluster-01 kubectl get nodes kubectl config get-contexts check the max number of pods on each node kubectl get nodes -o jsonpath='{.items[*].status.capacity.pods}' kubectl get nodes -o jsonpath='{.items[*].status.allocatable.pods}' Also observe the following resources created * EKS Cluster * Node Group ( EKS Cluster => Compute) * EC2 Instances * VPC, Subnets etc. Setup Visualizer cd ~ git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ To check whether visualiser has come up, use the following commands, kubectl get pods,services [sample output ] \u279c ~ kubectl get pods,services NAME READY STATUS RESTARTS AGE pod/kube-ops-view-75fddd9cc4-dg9cr 1/1 Running 0 92s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-ops-view NodePort 10.100.38.21 80:32000/TCP 92s To access the visualizer and other apps exposed with nodePort, create a new security group and attach it to the instances created with From EC2 => Network & Security => Security Groups => Create Security Group Provide security group name e.g. NodePort Select VPC created for EKS Add Inbound Rule (do not touch outbound) Custom TCP Port range : 30000-32767 Source : Anywhere IPv4 Proceed to create security group and note down the id once created e.g. sg-033ad52b6dcc79277 Add this security group to all nodes (ec2 instances) in the cluster from EC2 => Actions => Security => Change security groups Search and Add security group, Save . Now select the External IP of any of the nodes e.g. kubectl get nodes -o wide and access the visualizer on port 32000 e.g. http://xxxx:32000/#scale=2.0 e.g. Reference Creating Cluster with eksctl Creating and managing clusters Config File Schema: Config File Schema","title":"Lab K502 - EKS Setup"},{"location":"eks_setup/#lab-02-setting-up-eks-cluster-with-eksctl","text":"Author: Gourav Shah Publisher: School of Devops Version : 22.05.2024.01 While setting up EKS Cluster on AWS, there are two main options that you could consider Using eksctl which is a dedicated EKS Cluster Management Tool Using aws CLI along with kubectl which is a more complex approach, but comes with more control in your hands Both eksctl and the combination of aws cli with kubectl have their advantages and use cases when creating and managing AWS EKS clusters. Here's a comparison to help you decide which approach might be best for your course:","title":"Lab 02 - Setting up EKS Cluster with eksctl"},{"location":"eks_setup/#eksctl","text":"Pros: 1. Ease of Use: eksctl is specifically designed to simplify the creation and management of EKS clusters. It abstracts many of the complexities involved. 2. Quick Setup: With a single command, you can create a fully functioning EKS cluster. 3. Default Best Practices: It follows AWS best practices for cluster creation and configuration. 4. Less Code: Requires fewer commands and less code to manage the cluster. \u2800 Cons: 1. Less Granular Control: While it simplifies many tasks, it might abstract away some of the details, giving you less control over specific configurations. 2. Dependency: Adds an additional dependency that students need to install and manage. \u2800","title":"eksctl"},{"location":"eks_setup/#aws-cli-with-kubectl","text":"Pros: 1. Granular Control: Provides more control over the EKS cluster configuration and setup, allowing for fine-tuning and customization. 2. Learning Opportunity: Teaches students more about the underlying AWS services and Kubernetes configurations. 3. Versatility: aws cli and kubectl are versatile tools that can be used for a wide range of AWS and Kubernetes operations beyond just cluster creation. Cons: 1. Complexity: Requires more steps and a deeper understanding of AWS and Kubernetes concepts, which might be overwhelming for beginners. 2. More Commands: Involves writing more commands and scripts, which can be error-prone. As part of this lab, you will take a simpler approach and use eksctl to quickly create a cluster.","title":"aws cli with kubectl"},{"location":"eks_setup/#create-vpc-with-public-subnets","text":"EKS needs a VPC to launch itself in. There are three different options while creating the VPCs as follows VPC with Public and Private Subnets VPC with only Public Subnets VPC with only Private Subnets While you would ideally create VPC with public and private subnets so that you could host your applications in private subnets and set up load balancers and ingress controllers in public, as part of this lab guide, you are going to set up VPC with public subnets only. This is to avoid creation of two NAT Gateways, which would incur additional costs. The VPC that you are going to create has three public subnets that are deployed into different Availability Zones in an AWS Region. All nodes are automatically assigned public IPv4 addresses and can send and receive internet traffic through an internet gateway . A security group is deployed that denies all inbound traffic and allows all outbound traffic. The subnets are tagged so that Kubernetes can deploy load balancers to them. If you choose to create a VPC with a different configuration, please pick a relevant template from Creating a VPC for your Amazon EKS cluster . \u2800","title":"Create VPC with Public Subnets"},{"location":"eks_setup/#to-create-your-vpc","text":"Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation . From the navigation bar, select an AWS Region that supports EKS e.g. Singapore . Choose Create stack , With new resources (standard) . From Prepare Temaplte, select Choose and existing template and Under Template source , select Amazon S3 URL . Paste the following URL into the text area under Amazon S3 URL and choose Next : https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-vpc-sample.yaml On the Specify Details page, enter the stack name e.g. eks-vpc-01 and update parameters if you want to change any, and then choose Next and create VPC stack. Proceed to Review and create and submit the stack. It takes a few minutes to have the status change from to once VPC is created, you could verify everything from VPC console. Also note down the VpcId and SubnetIds from the Outputs .","title":"To create your VPC"},{"location":"eks_setup/#configure-iam-role-for-eks","text":"","title":"Configure IAM Role for EKS"},{"location":"eks_setup/#step-1-create-the-iam-role","text":"Open the IAM Console : Go to the IAM console . Create a Role : Click on Roles and then Create role . Choose EKS as the service that will use this role. Select the EKS - Cluster use case. From Add Permissions page you will see the following policy already attached: AmazonEKSClusterPolicy Select Next From Name, review and create page , provide a Role name e.g. EKSClusterServiceRole and proceed to Create role . From IAM Console -> Roles search for EKSClusterServiceRole and select it. From Permissions tab, Add Permissions dropdown, Choose Attach policies. Search for AmazonEKSVPCResourceController and attach it to the role. Set the Trust Relationship : Ensure the trust relationship for the role allows EKS to assume the role. The trust policy should look like this: { \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Principal\": { \"Service\": \"eks.amazonaws.com\" }, \"Action\": \"sts:AssumeRole\" } ] } This should already be there. Verify it from the Trust relationships tab. \u2800 Note down the role ARN e.g. arn:aws:iam::665496447754:role/EKSClusterServiceRole which you would add to cluster configuration later. Also note the new policy being added to the role.","title":"Step 1: Create the IAM Role"},{"location":"eks_setup/#launch-eks-cluster-with-eksctl","text":"Start with eksctl eksctl info Create cluster configurarion as follows. You could refer to Config File Schema to explore more options. File: cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: eks-cluster-01 region: ap-southeast-1 vpc: id: \"vpc-aaaa\" subnets: public: apsoutheast1a: id: subnet-xxxx apsoutheast1b: id: subnet-yyyy apsoutheast1c: id: subnet-zzzz managedNodeGroups: - name: ng-1-workers labels: { role: workers } instanceType: t2.micro desiredCapacity: 2 maxPodsPerNode: 100 minSize: 1 maxSize: 4 ssh: allow: true publicKeyName: xxxx tags: k8s.io/cluster-autoscaler/enabled: \"true\" k8s.io/cluster-autoscaler/eks-cluster-01: \"owned\" iam: withOIDC: true serviceRoleARN: arn:aws:iam::xxxxxxxxx:role/EKSClusterServiceRole \u2800 Edit this file and replace the following with actual values * vpc id * subnets * public key name (listed ssh key on ec2 that you would like to ssh in with) * ServiceRoleARN eksctl create cluster -f cluster.yaml [sample output] 2024-05-22 14:53:47 [\u2714] all EKS cluster resources for \"eks-cluster-01\" have been created 2024-05-22 14:53:48 [\u2139] nodegroup \"ng-1-workers\" has 1 node(s) 2024-05-22 14:53:48 [\u2139] node \"ip-192-168-240-31.ap-southeast-1.compute.internal\" is ready 2024-05-22 14:53:48 [\u2139] waiting for at least 1 node(s) to become ready in \"ng-1-workers\" 2024-05-22 14:53:48 [\u2139] nodegroup \"ng-1-workers\" has 1 node(s) 2024-05-22 14:53:48 [\u2139] node \"ip-192-168-240-31.ap-southeast-1.compute.internal\" is ready 2024-05-22 14:53:48 [\u2714] created 1 nodegroup(s) in cluster \"eks-cluster-01\" 2024-05-22 14:53:48 [\u2714] created 0 managed nodegroup(s) in cluster \"eks-cluster-01\" 2024-05-22 14:53:49 [\u2139] kubectl command should work with \"/Users/gshah/.kube/config\", try 'kubectl get nodes' 2024-05-22 14:53:49 [\u2714] EKS cluster \"eks-cluster-01\" in \"ap-southeast-1\" region is ready validate eksctl get cluster eksctl get nodegroup --cluster eks-cluster-01 kubectl get nodes kubectl config get-contexts check the max number of pods on each node kubectl get nodes -o jsonpath='{.items[*].status.capacity.pods}' kubectl get nodes -o jsonpath='{.items[*].status.allocatable.pods}' Also observe the following resources created * EKS Cluster * Node Group ( EKS Cluster => Compute) * EC2 Instances * VPC, Subnets etc.","title":"Launch EKS Cluster with eksctl"},{"location":"eks_setup/#setup-visualizer","text":"cd ~ git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ To check whether visualiser has come up, use the following commands, kubectl get pods,services [sample output ] \u279c ~ kubectl get pods,services NAME READY STATUS RESTARTS AGE pod/kube-ops-view-75fddd9cc4-dg9cr 1/1 Running 0 92s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-ops-view NodePort 10.100.38.21 80:32000/TCP 92s To access the visualizer and other apps exposed with nodePort, create a new security group and attach it to the instances created with From EC2 => Network & Security => Security Groups => Create Security Group Provide security group name e.g. NodePort Select VPC created for EKS Add Inbound Rule (do not touch outbound) Custom TCP Port range : 30000-32767 Source : Anywhere IPv4 Proceed to create security group and note down the id once created e.g. sg-033ad52b6dcc79277 Add this security group to all nodes (ec2 instances) in the cluster from EC2 => Actions => Security => Change security groups Search and Add security group, Save . Now select the External IP of any of the nodes e.g. kubectl get nodes -o wide and access the visualizer on port 32000 e.g. http://xxxx:32000/#scale=2.0 e.g.","title":"Setup Visualizer"},{"location":"eks_setup/#reference","text":"Creating Cluster with eksctl Creating and managing clusters Config File Schema: Config File Schema","title":"Reference"},{"location":"eks_storage/","text":"Lab 05 - Persistent Storage with EBS In this lab you are going to re deploy the database, this time with persistent storage, using EBS volumes offered by AWS. Following diagram depicts this dynamic provisining workflow: User | | v +---------------------+ | | | Persistent Volume | | Claim | | (PVC) | | | +---------+-----------+ | | Provision PV | +---------v-----------+ | | | CSI Driver | | (AWS EBS) | | | +---------+-----------+ | | Provision PV | +---------v-----------+ | | | Persistent Volume | | (PV) | | | +---------------------+ ^ | +---------+-----------+ | | | Storage Class | | (gp2) | | | +---------------------+ User : The user (developer/you) creates a PVC (PersistentVolumeClaim) to request storage resources for their application. The PVC specifies the desired characteristics of the storage, such as size and access mode. \u2800 Storage Class (gp2) : The StorageClass defines the properties for dynamically provisioning PVs. It includes parameters such as volume type, size, and the provisioner (CSI driver) responsible for fulfilling PVC requests. \u2800 CSI Driver (AWS EBS) : When the user creates the PVC, the Kubernetes control plane interprets the request and uses the StorageClass (gp2) to determine which provisioner (CSI driver) is responsible for provisioning the PV. \u2800 Persistent Volume (PV) : The CSI driver provisions a PV in response to the PVC creation request, based on the specifications defined in the PVC and the associated StorageClass (gp2). \u2800 In summary, the StorageClass defines the policies and provisioner used for dynamically provisioning PVs. When a user creates a PVC, the Kubernetes control plane uses the specified StorageClass to determine how to provision the PV, ultimately triggering the CSI driver to fulfill the PVC request. Lets see this in action now. Add Amazon EBS CSI Driver Create a IAM Role for CSI Driver using eksctl as : eksctl create iamserviceaccount \\ --name ebs-csi-controller-sa \\ --namespace kube-system \\ --cluster eks-cluster-01 \\ --role-name AmazonEKS_EBS_CSI_DriverRole \\ --role-only \\ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \\ --approve validate: eksctl get iamserviceaccount --cluster eks-cluster-01 From EKS Console, * Select cluster * From Add-ons select Get more add-ons Select Amazon EBS CSI Drive and go Next Select AmazonEKS_EBS_CSI_DriverRole created above. Next From Review and add screen proceed with Create . Validate the add-on is available from EKS Cluster. validate the CSI Components are created with kubectl get pods -n kube-system -l \"app.kubernetes.io/component=csi-driver\" [sample output] NAME READY STATUS RESTARTS AGE ebs-csi-controller-c95fcc9fb-qjk9h 6/6 Running 0 116s ebs-csi-controller-c95fcc9fb-rd64g 6/6 Running 0 116s ebs-csi-node-57n2m 3/3 Running 0 116s ebs-csi-node-dmp49 3/3 Running 0 116s Recreating DB with EBS Volume List the storage class available kubectl get sc [sample output] NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 16h This is the default storage class created by EKS. When you invoke this storage class with a PVC, the CSI driver will call the AWS APIs to create a actual EBS Volume and configure it as a PV. To watch this in action, start monitoring the following in a separate console: watch kubectl get pods,pvc,pv,sc Now redeploy the database, this time with a PVC configuration added as , kubectl delete -f db-deploy.yaml kubectl apply -f db-deploy-pvc.yaml At this time, you shall see the db pod in pending state e.g. NAME READY STATUS RESTARTS AGE pod/db-58b4db7665-rh6mv 0/1 Pending 0 39s Describe the pod as: kubectl describe pod -l \"role=db\" to see the issue in the events as: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 93s (x2 over 108s) default-scheduler 0/2 nodes are available: persistentvolumeclaim \"db-pvc\" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. This is because of the PVC missing. Now create a claim spec as File : db-pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: db-pvc spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 1Gi storageClassName: gp2 and apply kubectl apply -f db-pvc.yaml at this time, you should see the PV created and bound with pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOL UMEATTRIBUTESCLASS AGE persistentvolumeclaim/db-pvc Bound pvc-94165721-83f9-4110-b37c-4e3c9eb0c951 1Gi RWO gp2 4m52s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-94165721-83f9-4110-b37c-4e3c9eb0c951 1Gi RWO Delete Bound instavote/db-pv c gp2 4m47s Your pod at this time progresses further from pending state, however goes in a CrashLoopBackOff state. NAME READY STATUS RESTARTS AGE pod/db-58b4db7665-rh6mv 0/1 CrashLoopBackOff 6 (37s ago) 11m To find why, check the logs kubectl logs -l \"role=db\" [sample output] initdb: directory \"/var/lib/postgresql/data\" exists but is not empty It contains a lost+found directory, perhaps due to it being a mount point. Using a mount point directly as the data directory is not recommended. Create a subdirectory under the mount point. The database cluster will be initialized with locale \"en_US.utf8\". The default database encoding has accordingly been set to \"UTF8\". The default text search configuration will be set to \"english\". Data page checksums are disabled. This is happening because the EBS volume created comes with lost+found directory, which needs to be cleaned up before starting the database. To do that, you could add a init container to existing deployment as File : db-deploy-pvc.yaml .... spec: initContainers: - name: init-pg-data image: busybox:latest command: ['sh', '-c', 'rm -rf /var/lib/postgresql/data/*'] volumeMounts: - name: db-vol mountPath: /var/lib/postgresql/data containers: - image: postgres:9.4 imagePullPolicy: Always name: db env: - name: POSTGRES_HOST_AUTH_METHOD value: trust You need to add the initContainer block inside the spec, at the same level as existing container block. apply the changes kubectl apply -f db-deploy-pvc.yaml and you should now see the pod running, this time with a EBS volume NAME READY STATUS RESTARTS AGE pod/db-7b7fd4bcc7-z7zdr 1/1 Running 0 47s You should also see the EBS volume been provisionined and attached on the same EC2 instance that is running the db pod. Summary In this lab you learnt how to provision persistent storage, mount it on a pod and how to do this dynamically using the EBS as storage backend, gp2 storage class creted by EKS and by adding a CSI Driver ( Provisioner) for EBS.","title":"Lab K505 - Persistent Storage with EBS"},{"location":"eks_storage/#lab-05-persistent-storage-with-ebs","text":"In this lab you are going to re deploy the database, this time with persistent storage, using EBS volumes offered by AWS. Following diagram depicts this dynamic provisining workflow: User | | v +---------------------+ | | | Persistent Volume | | Claim | | (PVC) | | | +---------+-----------+ | | Provision PV | +---------v-----------+ | | | CSI Driver | | (AWS EBS) | | | +---------+-----------+ | | Provision PV | +---------v-----------+ | | | Persistent Volume | | (PV) | | | +---------------------+ ^ | +---------+-----------+ | | | Storage Class | | (gp2) | | | +---------------------+ User : The user (developer/you) creates a PVC (PersistentVolumeClaim) to request storage resources for their application. The PVC specifies the desired characteristics of the storage, such as size and access mode. \u2800 Storage Class (gp2) : The StorageClass defines the properties for dynamically provisioning PVs. It includes parameters such as volume type, size, and the provisioner (CSI driver) responsible for fulfilling PVC requests. \u2800 CSI Driver (AWS EBS) : When the user creates the PVC, the Kubernetes control plane interprets the request and uses the StorageClass (gp2) to determine which provisioner (CSI driver) is responsible for provisioning the PV. \u2800 Persistent Volume (PV) : The CSI driver provisions a PV in response to the PVC creation request, based on the specifications defined in the PVC and the associated StorageClass (gp2). \u2800 In summary, the StorageClass defines the policies and provisioner used for dynamically provisioning PVs. When a user creates a PVC, the Kubernetes control plane uses the specified StorageClass to determine how to provision the PV, ultimately triggering the CSI driver to fulfill the PVC request. Lets see this in action now.","title":"Lab 05 - Persistent Storage with EBS"},{"location":"eks_storage/#add-amazon-ebs-csi-driver","text":"Create a IAM Role for CSI Driver using eksctl as : eksctl create iamserviceaccount \\ --name ebs-csi-controller-sa \\ --namespace kube-system \\ --cluster eks-cluster-01 \\ --role-name AmazonEKS_EBS_CSI_DriverRole \\ --role-only \\ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \\ --approve validate: eksctl get iamserviceaccount --cluster eks-cluster-01 From EKS Console, * Select cluster * From Add-ons select Get more add-ons Select Amazon EBS CSI Drive and go Next Select AmazonEKS_EBS_CSI_DriverRole created above. Next From Review and add screen proceed with Create . Validate the add-on is available from EKS Cluster. validate the CSI Components are created with kubectl get pods -n kube-system -l \"app.kubernetes.io/component=csi-driver\" [sample output] NAME READY STATUS RESTARTS AGE ebs-csi-controller-c95fcc9fb-qjk9h 6/6 Running 0 116s ebs-csi-controller-c95fcc9fb-rd64g 6/6 Running 0 116s ebs-csi-node-57n2m 3/3 Running 0 116s ebs-csi-node-dmp49 3/3 Running 0 116s","title":"Add Amazon EBS CSI Driver"},{"location":"eks_storage/#recreating-db-with-ebs-volume","text":"List the storage class available kubectl get sc [sample output] NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 16h This is the default storage class created by EKS. When you invoke this storage class with a PVC, the CSI driver will call the AWS APIs to create a actual EBS Volume and configure it as a PV. To watch this in action, start monitoring the following in a separate console: watch kubectl get pods,pvc,pv,sc Now redeploy the database, this time with a PVC configuration added as , kubectl delete -f db-deploy.yaml kubectl apply -f db-deploy-pvc.yaml At this time, you shall see the db pod in pending state e.g. NAME READY STATUS RESTARTS AGE pod/db-58b4db7665-rh6mv 0/1 Pending 0 39s Describe the pod as: kubectl describe pod -l \"role=db\" to see the issue in the events as: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 93s (x2 over 108s) default-scheduler 0/2 nodes are available: persistentvolumeclaim \"db-pvc\" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. This is because of the PVC missing. Now create a claim spec as File : db-pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: db-pvc spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 1Gi storageClassName: gp2 and apply kubectl apply -f db-pvc.yaml at this time, you should see the PV created and bound with pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOL UMEATTRIBUTESCLASS AGE persistentvolumeclaim/db-pvc Bound pvc-94165721-83f9-4110-b37c-4e3c9eb0c951 1Gi RWO gp2 4m52s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-94165721-83f9-4110-b37c-4e3c9eb0c951 1Gi RWO Delete Bound instavote/db-pv c gp2 4m47s Your pod at this time progresses further from pending state, however goes in a CrashLoopBackOff state. NAME READY STATUS RESTARTS AGE pod/db-58b4db7665-rh6mv 0/1 CrashLoopBackOff 6 (37s ago) 11m To find why, check the logs kubectl logs -l \"role=db\" [sample output] initdb: directory \"/var/lib/postgresql/data\" exists but is not empty It contains a lost+found directory, perhaps due to it being a mount point. Using a mount point directly as the data directory is not recommended. Create a subdirectory under the mount point. The database cluster will be initialized with locale \"en_US.utf8\". The default database encoding has accordingly been set to \"UTF8\". The default text search configuration will be set to \"english\". Data page checksums are disabled. This is happening because the EBS volume created comes with lost+found directory, which needs to be cleaned up before starting the database. To do that, you could add a init container to existing deployment as File : db-deploy-pvc.yaml .... spec: initContainers: - name: init-pg-data image: busybox:latest command: ['sh', '-c', 'rm -rf /var/lib/postgresql/data/*'] volumeMounts: - name: db-vol mountPath: /var/lib/postgresql/data containers: - image: postgres:9.4 imagePullPolicy: Always name: db env: - name: POSTGRES_HOST_AUTH_METHOD value: trust You need to add the initContainer block inside the spec, at the same level as existing container block. apply the changes kubectl apply -f db-deploy-pvc.yaml and you should now see the pod running, this time with a EBS volume NAME READY STATUS RESTARTS AGE pod/db-7b7fd4bcc7-z7zdr 1/1 Running 0 47s You should also see the EBS volume been provisionined and attached on the same EC2 instance that is running the db pod.","title":"Recreating DB with EBS Volume"},{"location":"eks_storage/#summary","text":"In this lab you learnt how to provision persistent storage, mount it on a pod and how to do this dynamically using the EBS as storage backend, gp2 storage class creted by EKS and by adding a CSI Driver ( Provisioner) for EBS.","title":"Summary"},{"location":"eks_vpa/","text":"Lab 07 - Verticle Pod Autoscaler Before you begin, ensure OpenSSL is installed and is up to date on your system. Clone the kubernetes/autoscaler GitHub repository and switch to path where VPA manifests are, git clone https://github.com/kubernetes/autoscaler.git cd autoscaler/vertical-pod-autoscaler/ deploy VPA as ./hack/vpa-up.sh Create a VPA Policy File: vote-vpa.yaml --- apiVersion: \"autoscaling.k8s.io/v1\" kind: VerticalPodAutoscaler metadata: name: vote spec: # recommenders field can be unset when using the default recommender. # When using an alternative recommender, the alternative recommender's name # can be specified as the following in a list. # recommenders: # - name: 'alternative' targetRef: apiVersion: \"apps/v1\" kind: Deployment name: vote resourcePolicy: containerPolicies: - containerName: '*' minAllowed: cpu: 50m memory: 64Mi maxAllowed: cpu: 500 memory: 512Mi controlledResources: [\"cpu\", \"memory\"] apply kubectl apply -f vote-vpa.yaml watch kubectl get vpa vote --watch [sample output] NAME MODE CPU MEM PROVIDED AGE vote Auto 5s vote Auto 50m 262144k True 30s Create a Load Test Job if not already present as, file: loadtest-job.yaml apiVersion: batch/v1 kind: Job metadata: generateName: loadtest spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: [\"siege\", \"--concurrent=1\", \"--benchmark\", \"--time=4m\", \"http://vote\"] restartPolicy: Never backoffLimit: 4 and launch it as kubectl create -f loadtest-job.yaml This will launch a one off Job which would run for 4 minutes. keep watching the following in differnt windows terminal 1 kubectl get vpa vote --watch terminal 2 kubectl get hpa vote --watch you should see, both hpa and vpa doing complimentary jobs where, * HPA is launching new pods to ensure more requests are being served * VPA is updating the requests spec based on actual metrics You could check the modified request spec by VPA by describing the one of the pods runnnig vote app kubectl describe pod vote-xxxx-yyyy Restart Count: 0 Limits: cpu: 1235m memory: 500Mi Requests: cpu: 247m memory: 262144k Environment: Mounts:","title":"Lab K509 - Verticle Pod Autoscaler"},{"location":"eks_vpa/#lab-07-verticle-pod-autoscaler","text":"Before you begin, ensure OpenSSL is installed and is up to date on your system. Clone the kubernetes/autoscaler GitHub repository and switch to path where VPA manifests are, git clone https://github.com/kubernetes/autoscaler.git cd autoscaler/vertical-pod-autoscaler/ deploy VPA as ./hack/vpa-up.sh Create a VPA Policy File: vote-vpa.yaml --- apiVersion: \"autoscaling.k8s.io/v1\" kind: VerticalPodAutoscaler metadata: name: vote spec: # recommenders field can be unset when using the default recommender. # When using an alternative recommender, the alternative recommender's name # can be specified as the following in a list. # recommenders: # - name: 'alternative' targetRef: apiVersion: \"apps/v1\" kind: Deployment name: vote resourcePolicy: containerPolicies: - containerName: '*' minAllowed: cpu: 50m memory: 64Mi maxAllowed: cpu: 500 memory: 512Mi controlledResources: [\"cpu\", \"memory\"] apply kubectl apply -f vote-vpa.yaml watch kubectl get vpa vote --watch [sample output] NAME MODE CPU MEM PROVIDED AGE vote Auto 5s vote Auto 50m 262144k True 30s Create a Load Test Job if not already present as, file: loadtest-job.yaml apiVersion: batch/v1 kind: Job metadata: generateName: loadtest spec: template: spec: containers: - name: siege image: schoolofdevops/loadtest:v1 command: [\"siege\", \"--concurrent=1\", \"--benchmark\", \"--time=4m\", \"http://vote\"] restartPolicy: Never backoffLimit: 4 and launch it as kubectl create -f loadtest-job.yaml This will launch a one off Job which would run for 4 minutes. keep watching the following in differnt windows terminal 1 kubectl get vpa vote --watch terminal 2 kubectl get hpa vote --watch you should see, both hpa and vpa doing complimentary jobs where, * HPA is launching new pods to ensure more requests are being served * VPA is updating the requests spec based on actual metrics You could check the modified request spec by VPA by describing the one of the pods runnnig vote app kubectl describe pod vote-xxxx-yyyy Restart Count: 0 Limits: cpu: 1235m memory: 500Mi Requests: cpu: 247m memory: 262144k Environment: Mounts:","title":"Lab 07 - Verticle Pod Autoscaler"},{"location":"elk_monitoring/","text":"","title":"Elk monitoring"},{"location":"helm/","text":"Lab K205 - Monitoring setup with HELM In this lab, you are going to install and configure helm, and in turns, use it to configure a monitoring system for kubernetes using prometheus and grafana stack. Installing Helm (version 3) To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version Deploy Prometheus Stack with HELM Read about kube-prometheus-stack 33.1.0 \u00b7 prometheus/prometheus-community chart at artifacthub.io Add helm repository using , helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update Download the chart as, cd ~ helm fetch --untar prometheus-community/kube-prometheus-stack Change into the charts directory cd kube-prometheus-stack/ ls Deploy prometheus stack as, kubectl create ns monitoring helm install prom -n monitoring prometheus-community/kube-prometheus-stack Validate helm list -A kubectl get all -n monitoring kubectl get pods,svc -n monitoring Customising Prometheus Configurations To update the Grafana service to use NodePort configurations, upgrade the helm release by setting up customized properties as, helm upgrade prom -n monitoring prometheus-community/kube-prometheus-stack \\ --set grafana.service.type=NodePort \\ --set grafana.service.nodePort=30200 Validate helm list -A kubectl get svc -n monitoring You should see new revision of monitoring stack deployed, and Grafana service changed to NodePort. Note down the node port and access Grafana with http://IPADDRESS:30200 remember to replace node name/ip address and node port as actual. Login using User : admin Pass: prom-operator Once logged in, you should be able to browse and view Grafana dashboards from Dashboards menu. An example dashboard is as follows, You could further explore various Grafana dashboards and configurations. Uninstalling the App with HELM Once you are done experiementing and learning, you could uninstall the application stack that you earlier installed with helm easily. To uninstall prometheus and grafana stack, begin by listing it helm list -A helm uninstall -n monitoring prom This should clean up everything that was deployed with helm earlier. Summary In this lab, we not only learnt about HELM, a kubernetes package manager, but also have setup a sophisticated health monitoring system with prometheus and grafana.","title":"Lab K113 - HELM Package Manager"},{"location":"helm/#lab-k205-monitoring-setup-with-helm","text":"In this lab, you are going to install and configure helm, and in turns, use it to configure a monitoring system for kubernetes using prometheus and grafana stack.","title":"Lab K205 - Monitoring setup with HELM"},{"location":"helm/#installing-helm-version-3","text":"To install helm version 3 on Linux or MacOS, you can follow following instructions. curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash You could further refer to Official HELM Install Instructions for alternative options. Verify the installtion is successful, helm --help helm version","title":"Installing Helm (version 3)"},{"location":"helm/#deploy-prometheus-stack-with-helm","text":"Read about kube-prometheus-stack 33.1.0 \u00b7 prometheus/prometheus-community chart at artifacthub.io Add helm repository using , helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update Download the chart as, cd ~ helm fetch --untar prometheus-community/kube-prometheus-stack Change into the charts directory cd kube-prometheus-stack/ ls Deploy prometheus stack as, kubectl create ns monitoring helm install prom -n monitoring prometheus-community/kube-prometheus-stack Validate helm list -A kubectl get all -n monitoring kubectl get pods,svc -n monitoring","title":"Deploy Prometheus Stack with HELM"},{"location":"helm/#customising-prometheus-configurations","text":"To update the Grafana service to use NodePort configurations, upgrade the helm release by setting up customized properties as, helm upgrade prom -n monitoring prometheus-community/kube-prometheus-stack \\ --set grafana.service.type=NodePort \\ --set grafana.service.nodePort=30200 Validate helm list -A kubectl get svc -n monitoring You should see new revision of monitoring stack deployed, and Grafana service changed to NodePort. Note down the node port and access Grafana with http://IPADDRESS:30200 remember to replace node name/ip address and node port as actual. Login using User : admin Pass: prom-operator Once logged in, you should be able to browse and view Grafana dashboards from Dashboards menu. An example dashboard is as follows, You could further explore various Grafana dashboards and configurations.","title":"Customising Prometheus Configurations"},{"location":"helm/#uninstalling-the-app-with-helm","text":"Once you are done experiementing and learning, you could uninstall the application stack that you earlier installed with helm easily. To uninstall prometheus and grafana stack, begin by listing it helm list -A helm uninstall -n monitoring prom This should clean up everything that was deployed with helm earlier.","title":"Uninstalling the App with HELM"},{"location":"helm/#summary","text":"In this lab, we not only learnt about HELM, a kubernetes package manager, but also have setup a sophisticated health monitoring system with prometheus and grafana.","title":"Summary"},{"location":"helm_charts/","text":"Building your own Helm Charts If you have not installed helm yet, do so by using curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash helm Part I - Generating a Chart with One Service Create a namespace and switch to it kubectl create namespace dev kubectl config set-context --current --namespace=dev kubectl config get-contexts Generate a helm chart scaffold and change into the directory created to create a copy of values.yaml cd ~ helm create instavote cd instavote/ cp values.yaml values.dev.yaml Edit values.dev.yaml to update replicaCount, image and tag as replicaCount: 4 image: repository: schoolofdevops/vote pullPolicy: IfNotPresent # Overrides the image tag whose default is the chart appVersion. tag: \"v5\" Update service type to Node Port service: type: NodePort port: 80 nodePort: 30300 Also update the service.yaml template with the additional property for nodePort defined as , File : service.yaml apiVersion: v1 kind: Service metadata: name: vote labels: {{- include \"instavote.labels\" . | nindent 4 }} spec: type: {{ .Values.service.type }} ports: - port: {{ .Values.service.port }} targetPort: http protocol: TCP name: http nodePort: {{ .Values.service.nodePort }} selector: {{- include \"instavote.selectorLabels\" . | nindent 4 }} Install this chart with helm to deploy the vote service as: helm install instavote -n dev --values=values.dev.yaml . --dry-run helm install instavote -n dev --values=values.dev.yaml . Validate with helm list -A kubectl get all Part II - Adding support for More Microservices Create copy of the templates so that you could add support for one more service e.g. redis cd templates/ mkdir vote redis mv deployment.yaml hpa.yaml ingress.yaml service.yaml vote/ cp vote/*.yaml redis/ You will start editing the templates so that template variables such as Values.replicaCount will be replaced with Values.vote.replicaCount . Thats the theme you will have to follow from here on. e.g. this is the exisinng code {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} which should be changes to spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.vote.replicaCount }} {{- end }} To make it easier, you could use text replacement feature of sed cd templates/vote/ sed -i 's/Values./Values.vote./g' deployment.yaml sed -i 's/Values./Values.vote./g' service.yaml and then cd templates/redis/ sed -i 's/Values./Values.redis./g' deployment.yaml sed -i 's/Values./Values.redis./g' service.yaml Also change the name in the metadata field for the following files templates/vote/deployment.yaml templates/vote/service.yaml from metadata: name: {{ include \"instavote.fullname\" . }} to metadata: name: vote And similarly in the following files templates/redis/deployment.yaml templates/redis/service.yaml from metadata: name: {{ include \"instavote.fullname\" . }} to metadata: name: redis Now update instavote/values.yaml to support multiple services add vote as top level key indent all properties so that those become sub properties of key added above i.e. vote copy over and create blocks for redis Use this file as a reference values.yaml Now go ahead and deploy by Creating your own copy of values.yaml Providing values specific to those services You could download this file with sample values values.dev.yaml using the following command : wget -c https://gist.githubusercontent.com/initcron/67e104f3a2949f20c2da06e19c854faa/raw/90c7b0c330f133efae43ea71efb1beba314ea451/values.dev.yaml And apply helm uninstall instavote helm install instavote -n dev --values=values.dev.yaml . helm list -A kubectl get all Validate that vote and redis are running with the values that you provided. Part III : Overriding Values, Rollbacks Lets learn how to override values from the command line. To do so, lets add one property File : templates/vote/deployment.yaml containers: - name: vote env: - name: OPTION_A value: {{ .Values.vote.options.A }} - name: OPTION_B value: {{ .Values.vote.options.B }} This will read from values file and set those as environmnt variables. Lets set the default values. File: values.dev.yaml vote: replicaCount: 2 options: A: MacOS B: Windows now upgrade the release as helm upgrade instavote -n dev --values=values.dev.yaml . Check the application to see the default values being visible. You could override the values from the command line as helm upgrade instavote --values values.dev.yaml --set vote.options.A=Green --set vote.options.B=Yellow . check the web app now try one more time helm upgrade instavote --values values.dev.yaml --set vote.options.A=Orange --set vote.options.B=Blue . you should see the values change every time. Check the release now helm list -A [sample output] NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION instavote dev 7 2024-05-14 03:31:34.717912894 +0000 UTC deployed instavote-0.1.0 1.16.0 try rolling back a version e.g. helm rollback instavote you could also go back to a specific version using the revision number as helm rollback instavote xx where replace xx with the revision number you wish to roll back to. Exercise Now that you have deployed vote and redis, go ahead and add the code to deploy worker, db and result as well.","title":"Lab K114 - Writing HELM Charts"},{"location":"helm_charts/#building-your-own-helm-charts","text":"If you have not installed helm yet, do so by using curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash helm","title":"Building your own Helm Charts"},{"location":"helm_charts/#part-i-generating-a-chart-with-one-service","text":"Create a namespace and switch to it kubectl create namespace dev kubectl config set-context --current --namespace=dev kubectl config get-contexts Generate a helm chart scaffold and change into the directory created to create a copy of values.yaml cd ~ helm create instavote cd instavote/ cp values.yaml values.dev.yaml Edit values.dev.yaml to update replicaCount, image and tag as replicaCount: 4 image: repository: schoolofdevops/vote pullPolicy: IfNotPresent # Overrides the image tag whose default is the chart appVersion. tag: \"v5\" Update service type to Node Port service: type: NodePort port: 80 nodePort: 30300 Also update the service.yaml template with the additional property for nodePort defined as , File : service.yaml apiVersion: v1 kind: Service metadata: name: vote labels: {{- include \"instavote.labels\" . | nindent 4 }} spec: type: {{ .Values.service.type }} ports: - port: {{ .Values.service.port }} targetPort: http protocol: TCP name: http nodePort: {{ .Values.service.nodePort }} selector: {{- include \"instavote.selectorLabels\" . | nindent 4 }} Install this chart with helm to deploy the vote service as: helm install instavote -n dev --values=values.dev.yaml . --dry-run helm install instavote -n dev --values=values.dev.yaml . Validate with helm list -A kubectl get all","title":"Part I - Generating a Chart with One Service"},{"location":"helm_charts/#part-ii-adding-support-for-more-microservices","text":"Create copy of the templates so that you could add support for one more service e.g. redis cd templates/ mkdir vote redis mv deployment.yaml hpa.yaml ingress.yaml service.yaml vote/ cp vote/*.yaml redis/ You will start editing the templates so that template variables such as Values.replicaCount will be replaced with Values.vote.replicaCount . Thats the theme you will have to follow from here on. e.g. this is the exisinng code {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} which should be changes to spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.vote.replicaCount }} {{- end }} To make it easier, you could use text replacement feature of sed cd templates/vote/ sed -i 's/Values./Values.vote./g' deployment.yaml sed -i 's/Values./Values.vote./g' service.yaml and then cd templates/redis/ sed -i 's/Values./Values.redis./g' deployment.yaml sed -i 's/Values./Values.redis./g' service.yaml Also change the name in the metadata field for the following files templates/vote/deployment.yaml templates/vote/service.yaml from metadata: name: {{ include \"instavote.fullname\" . }} to metadata: name: vote And similarly in the following files templates/redis/deployment.yaml templates/redis/service.yaml from metadata: name: {{ include \"instavote.fullname\" . }} to metadata: name: redis Now update instavote/values.yaml to support multiple services add vote as top level key indent all properties so that those become sub properties of key added above i.e. vote copy over and create blocks for redis Use this file as a reference values.yaml Now go ahead and deploy by Creating your own copy of values.yaml Providing values specific to those services You could download this file with sample values values.dev.yaml using the following command : wget -c https://gist.githubusercontent.com/initcron/67e104f3a2949f20c2da06e19c854faa/raw/90c7b0c330f133efae43ea71efb1beba314ea451/values.dev.yaml And apply helm uninstall instavote helm install instavote -n dev --values=values.dev.yaml . helm list -A kubectl get all Validate that vote and redis are running with the values that you provided.","title":"Part II - Adding support for More Microservices"},{"location":"helm_charts/#part-iii-overriding-values-rollbacks","text":"Lets learn how to override values from the command line. To do so, lets add one property File : templates/vote/deployment.yaml containers: - name: vote env: - name: OPTION_A value: {{ .Values.vote.options.A }} - name: OPTION_B value: {{ .Values.vote.options.B }} This will read from values file and set those as environmnt variables. Lets set the default values. File: values.dev.yaml vote: replicaCount: 2 options: A: MacOS B: Windows now upgrade the release as helm upgrade instavote -n dev --values=values.dev.yaml . Check the application to see the default values being visible. You could override the values from the command line as helm upgrade instavote --values values.dev.yaml --set vote.options.A=Green --set vote.options.B=Yellow . check the web app now try one more time helm upgrade instavote --values values.dev.yaml --set vote.options.A=Orange --set vote.options.B=Blue . you should see the values change every time. Check the release now helm list -A [sample output] NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION instavote dev 7 2024-05-14 03:31:34.717912894 +0000 UTC deployed instavote-0.1.0 1.16.0 try rolling back a version e.g. helm rollback instavote you could also go back to a specific version using the revision number as helm rollback instavote xx where replace xx with the revision number you wish to roll back to.","title":"Part III : Overriding Values, Rollbacks"},{"location":"helm_charts/#exercise","text":"Now that you have deployed vote and redis, go ahead and add the code to deploy worker, db and result as well.","title":"Exercise"},{"location":"helm_charts_publish/","text":"Publishing Charts To publish your Helm charts using GitHub Pages, you'll need a GitHub account and a new or existing repository where you can store and serve your charts. Here's a detailed step-by-step process to get your Helm charts published using GitHub Pages: Step 1: Prepare Your GitHub Repository Create a New Repository : If you don't already have one, create a new GitHub repository. It can be named anything, but something like helm-charts is descriptive and common. Step 2: Clone Your Repository Locally cd~ git clone https://github.com/xxxxxx/helm-charts.git cd helm-charts Replace with your GitHub username and with the name of your repository. Step 3: Package Your Helm Chart Assuming you have your chart directory for instavote ready, move that inside the helm-charts e.g. mv instavote helm-charts/ cd helm-charts git add instavote/ git commit -am \"added instavote chart\" git push origin main git log helm package instavote ls This command will create a .tgz file of your chart. Step 4: Create or Update the Helm Chart Repository Index After packaging your chart, create an index file that Helm can use to fetch chart metadata: helm repo index . --url https://xxxxxx.github.io/helm-charts/ This command generates or updates an index.yaml file with information about charts stored in the directory. Step 5: Switch to the gh-pages Branch If the gh-pages branch does not exist, create it and switch to it: git checkout --orphan gh-pages git status git rm -rf instavote README.md Step 6: Commit and Push Your Changes git add *.tgz index.yaml git commit -m \"Publish Helm charts\" git push origin gh-pages Step 7: Ensure GitHub Pages is Set Up Correctly Go to your repository's settings. Find the \"Pages\" section. Set the source to the gh-pages branch (you may need to create this branch in your repository if it doesn't exist yet). Step 8: Add Your Repository to Helm Now that your chart is published, you can add your repository to Helm: helm repo add instavote https://xxxxxx.github.io/helm-charts/ Step 9: Update Helm Repositories helm repo list helm repo update This will update your local Helm client with the latest charts from your newly published repository. Step 10: Installing Charts You and others can now install charts from your repository: helm search repo instavote helm install instavote instavote/instavote Replace with a name for your Helm deployment and with the name of the chart in your repository. This process sets up a straightforward, cost-effective, and robust method for hosting and managing your Helm charts using GitHub and GitHub Pages.","title":"Lab K115 - Publishing HELM Charts"},{"location":"helm_charts_publish/#publishing-charts","text":"To publish your Helm charts using GitHub Pages, you'll need a GitHub account and a new or existing repository where you can store and serve your charts. Here's a detailed step-by-step process to get your Helm charts published using GitHub Pages:","title":"Publishing Charts"},{"location":"helm_charts_publish/#step-1-prepare-your-github-repository","text":"Create a New Repository : If you don't already have one, create a new GitHub repository. It can be named anything, but something like helm-charts is descriptive and common.","title":"Step 1: Prepare Your GitHub Repository"},{"location":"helm_charts_publish/#step-2-clone-your-repository-locally","text":"cd~ git clone https://github.com/xxxxxx/helm-charts.git cd helm-charts Replace with your GitHub username and with the name of your repository.","title":"Step 2: Clone Your Repository Locally"},{"location":"helm_charts_publish/#step-3-package-your-helm-chart","text":"Assuming you have your chart directory for instavote ready, move that inside the helm-charts e.g. mv instavote helm-charts/ cd helm-charts git add instavote/ git commit -am \"added instavote chart\" git push origin main git log helm package instavote ls This command will create a .tgz file of your chart.","title":"Step 3: Package Your Helm Chart"},{"location":"helm_charts_publish/#step-4-create-or-update-the-helm-chart-repository-index","text":"After packaging your chart, create an index file that Helm can use to fetch chart metadata: helm repo index . --url https://xxxxxx.github.io/helm-charts/ This command generates or updates an index.yaml file with information about charts stored in the directory.","title":"Step 4: Create or Update the Helm Chart Repository Index"},{"location":"helm_charts_publish/#step-5-switch-to-the-gh-pages-branch","text":"If the gh-pages branch does not exist, create it and switch to it: git checkout --orphan gh-pages git status git rm -rf instavote README.md","title":"Step 5: Switch to the gh-pages Branch"},{"location":"helm_charts_publish/#step-6-commit-and-push-your-changes","text":"git add *.tgz index.yaml git commit -m \"Publish Helm charts\" git push origin gh-pages","title":"Step 6: Commit and Push Your Changes"},{"location":"helm_charts_publish/#step-7-ensure-github-pages-is-set-up-correctly","text":"Go to your repository's settings. Find the \"Pages\" section. Set the source to the gh-pages branch (you may need to create this branch in your repository if it doesn't exist yet).","title":"Step 7: Ensure GitHub Pages is Set Up Correctly"},{"location":"helm_charts_publish/#step-8-add-your-repository-to-helm","text":"Now that your chart is published, you can add your repository to Helm: helm repo add instavote https://xxxxxx.github.io/helm-charts/","title":"Step 8: Add Your Repository to Helm"},{"location":"helm_charts_publish/#step-9-update-helm-repositories","text":"helm repo list helm repo update This will update your local Helm client with the latest charts from your newly published repository.","title":"Step 9: Update Helm Repositories"},{"location":"helm_charts_publish/#step-10-installing-charts","text":"You and others can now install charts from your repository: helm search repo instavote helm install instavote instavote/instavote Replace with a name for your Helm deployment and with the name of the chart in your repository. This process sets up a straightforward, cost-effective, and robust method for hosting and managing your Helm charts using GitHub and GitHub Pages.","title":"Step 10: Installing Charts"},{"location":"ingress/","text":"Lab K201 - Application Routing with Ingress Controllers Pre Requisites Ingress controller such as Nginx, Trafeik needs to be deployed before creating ingress resources. On GCE, ingress controller runs on the master. On all other installations, it needs to be deployed, either as a deployment, or a daemonset. In addition, a service needs to be created for ingress. Daemonset will run ingress on each node. Deployment will just create a highly available setup, which can then be exposed on specific nodes using ExternalIPs configuration in the service. Launch Ingress Controller An ingress controller needs to be created in order to serve the ingress requests. As part of this lab you are going to use Traefik as the ingress controller. Its a fast and lightweight ingress controller and also comes with great documentation and support. +----+----+--+ | ingress | | controller | +----+-------+ You are going to setup ingress controller using helm. Assuming helm is already installed, begin adding the repository to install traefik as a ingress controller from: helm repo add traefik https://helm.traefik.io/traefik helm repo update Pull Traffic Chart helm pull --untar traefik/traefik cd traefik Create a new file my.values.yaml with the following contents ports: traefik: port: 9000 expose: true exposedPort: 9000 nodePort: 30300 protocol: TCP web: port: 8000 expose: true exposedPort: 80 nodePort: 30400 protocol: TCP service: enabled: true single: true type: NodePort Source : traefik deployment customization spec \u00b7 GitHub Apply with the custom values file created as above, kubectl create ns traefik helm install traefik --namespace=traefik --values=my.values.yaml . Validate helm list -A kubectl get all -n traefik Use the following URIs to access Traefik and Traefik Dashboard , Traefik Web : http://xx.xx.xx.xx:30400 (you may see 404 page not found error, which is okay and expected here). Traefik Dashboard : http://xx.xx.xx.xx:30300/dashboard/ (training slash \u201c/\u201c is a must) (you are expected to a nice looking see Traefik dashboard here). replace xx.xx.xx.xx with the external IP used earlier Set up Named Based Routing for Vote App We will direct all our request to the ingress controller now, but with differnt hostname e.g. vote.example.com or results.example.com . And it should direct to the correct service based on the host name. In order to achieve this you, as a user would create a ingress object with a set of rules, +----+----+--+ | ingress | | controller | +----+-------+ | +-----+----+ +---watch----> | ingress | <------- user +----------+ file: vote-ing.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip spec: rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 And apply kubectl get ing kubectl apply -f vote-ing.yaml --dry-run kubectl apply -f vote-ing.yaml Since the ingress controller is constantly monitoring for the ingress objects, the moment it detects, it connects with traefik and creates a rule as follows. +----------+ +--create----> | traefik | | | rules | | +----------+ +----+----+--+ ^ | ingress | : | controller | : +----+-------+ : | +-----+----+ +---watch----> | ingress | <------- user +----------+ where, A user creates a ingress object with the rules. This could be a named based or a path based routing. An ingress controller, in this example traefik constantly monitors for ingress objects. The moment it detects one, it creates a rule and adds it to the traefik load balancer. This rule maps to the ingress specs. You could now see the rule added to ingress controller, Where, vote.example.com is added as frontend. This frontends point to service vote . respective backend also appear on the right hand side of the screen, mapping to each of the service. Add Local DNS You have created the ingress rules based on hostnames e.g. vote.example.com and results.example.com . In order for you to be able to access those, there has to be a dns entry pointing to your nodes, which are running traefik. vote.example.com -------+ +----- vote:81 | +-------------+ | | | ingress | | +===> | node:80 | ===+ | +-------------+ | | | results.example.com -------+ +----- results:82 To achieve this you need to either, Create a DNS entry, provided you own the domain and have access to the dns management console. Create a local hosts file entry. On unix systems its in /etc/hosts file. On windows its at C:\\Windows\\System32\\drivers\\etc\\hosts . You need admin access to edit this file. For example, on a linux or osx, you could edit it as, sudo vim /etc/hosts And add an entry such as , xxx.xxx.xxx.xxx vote.example.com results.example.com kube-ops-view.example.org where, xxx.xxx.xxx.xxx is the actual IP address of one of the nodes running traefik. And then access the app urls using http://vote.example.com or http://results.example.com Add another Web Application with Ingress kubectl create deployment nginx --image=nginx kubectl create service clusterip nginx --tcp=80 Add ingress rule file: web-ing.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web namespace: instavote annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip spec: rules: - host: web.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx port: number: 80 Source: Ingress for Web App \u00b7 GitHub And update hosts file with new route as , xxx.xxx.xxx.xxx vote.example.com web.example.com results.example.com kube-ops-view.example.org Visit http://web.example.com:30400 to see your request being routed to nginx web page. Mini Project : Configure Ingress for Result App Now that you have added the ingress rule for the vote app, follow the same method and add one for the result app as well which is running in the same namespace. The final validation would be, when you access or http://result.example.com you should see it pointing to the result app. Reading Resources Trafeik's Guide to Kubernetes Ingress Controller Annotations DaemonSets References Online htpasswd generator Keywords trafeik on kubernetes kubernetes ingress kubernetes annotations daemonsets","title":"Lab K112 - Ingress"},{"location":"ingress/#lab-k201-application-routing-with-ingress-controllers","text":"","title":"Lab K201 - Application Routing with Ingress Controllers"},{"location":"ingress/#pre-requisites","text":"Ingress controller such as Nginx, Trafeik needs to be deployed before creating ingress resources. On GCE, ingress controller runs on the master. On all other installations, it needs to be deployed, either as a deployment, or a daemonset. In addition, a service needs to be created for ingress. Daemonset will run ingress on each node. Deployment will just create a highly available setup, which can then be exposed on specific nodes using ExternalIPs configuration in the service.","title":"Pre Requisites"},{"location":"ingress/#launch-ingress-controller","text":"An ingress controller needs to be created in order to serve the ingress requests. As part of this lab you are going to use Traefik as the ingress controller. Its a fast and lightweight ingress controller and also comes with great documentation and support. +----+----+--+ | ingress | | controller | +----+-------+ You are going to setup ingress controller using helm. Assuming helm is already installed, begin adding the repository to install traefik as a ingress controller from: helm repo add traefik https://helm.traefik.io/traefik helm repo update Pull Traffic Chart helm pull --untar traefik/traefik cd traefik Create a new file my.values.yaml with the following contents ports: traefik: port: 9000 expose: true exposedPort: 9000 nodePort: 30300 protocol: TCP web: port: 8000 expose: true exposedPort: 80 nodePort: 30400 protocol: TCP service: enabled: true single: true type: NodePort Source : traefik deployment customization spec \u00b7 GitHub Apply with the custom values file created as above, kubectl create ns traefik helm install traefik --namespace=traefik --values=my.values.yaml . Validate helm list -A kubectl get all -n traefik Use the following URIs to access Traefik and Traefik Dashboard , Traefik Web : http://xx.xx.xx.xx:30400 (you may see 404 page not found error, which is okay and expected here). Traefik Dashboard : http://xx.xx.xx.xx:30300/dashboard/ (training slash \u201c/\u201c is a must) (you are expected to a nice looking see Traefik dashboard here). replace xx.xx.xx.xx with the external IP used earlier","title":"Launch Ingress Controller"},{"location":"ingress/#set-up-named-based-routing-for-vote-app","text":"We will direct all our request to the ingress controller now, but with differnt hostname e.g. vote.example.com or results.example.com . And it should direct to the correct service based on the host name. In order to achieve this you, as a user would create a ingress object with a set of rules, +----+----+--+ | ingress | | controller | +----+-------+ | +-----+----+ +---watch----> | ingress | <------- user +----------+ file: vote-ing.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: vote namespace: instavote annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip spec: rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 And apply kubectl get ing kubectl apply -f vote-ing.yaml --dry-run kubectl apply -f vote-ing.yaml Since the ingress controller is constantly monitoring for the ingress objects, the moment it detects, it connects with traefik and creates a rule as follows. +----------+ +--create----> | traefik | | | rules | | +----------+ +----+----+--+ ^ | ingress | : | controller | : +----+-------+ : | +-----+----+ +---watch----> | ingress | <------- user +----------+ where, A user creates a ingress object with the rules. This could be a named based or a path based routing. An ingress controller, in this example traefik constantly monitors for ingress objects. The moment it detects one, it creates a rule and adds it to the traefik load balancer. This rule maps to the ingress specs. You could now see the rule added to ingress controller, Where, vote.example.com is added as frontend. This frontends point to service vote . respective backend also appear on the right hand side of the screen, mapping to each of the service.","title":"Set up Named Based Routing for Vote App"},{"location":"ingress/#add-local-dns","text":"You have created the ingress rules based on hostnames e.g. vote.example.com and results.example.com . In order for you to be able to access those, there has to be a dns entry pointing to your nodes, which are running traefik. vote.example.com -------+ +----- vote:81 | +-------------+ | | | ingress | | +===> | node:80 | ===+ | +-------------+ | | | results.example.com -------+ +----- results:82 To achieve this you need to either, Create a DNS entry, provided you own the domain and have access to the dns management console. Create a local hosts file entry. On unix systems its in /etc/hosts file. On windows its at C:\\Windows\\System32\\drivers\\etc\\hosts . You need admin access to edit this file. For example, on a linux or osx, you could edit it as, sudo vim /etc/hosts And add an entry such as , xxx.xxx.xxx.xxx vote.example.com results.example.com kube-ops-view.example.org where, xxx.xxx.xxx.xxx is the actual IP address of one of the nodes running traefik. And then access the app urls using http://vote.example.com or http://results.example.com","title":"Add Local DNS"},{"location":"ingress/#add-another-web-application-with-ingress","text":"kubectl create deployment nginx --image=nginx kubectl create service clusterip nginx --tcp=80 Add ingress rule file: web-ing.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web namespace: instavote annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip spec: rules: - host: web.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx port: number: 80 Source: Ingress for Web App \u00b7 GitHub And update hosts file with new route as , xxx.xxx.xxx.xxx vote.example.com web.example.com results.example.com kube-ops-view.example.org Visit http://web.example.com:30400 to see your request being routed to nginx web page.","title":"Add another Web Application with Ingress"},{"location":"ingress/#mini-project-configure-ingress-for-result-app","text":"Now that you have added the ingress rule for the vote app, follow the same method and add one for the result app as well which is running in the same namespace. The final validation would be, when you access or http://result.example.com you should see it pointing to the result app.","title":"Mini Project : Configure Ingress for Result App"},{"location":"ingress/#reading-resources","text":"Trafeik's Guide to Kubernetes Ingress Controller Annotations DaemonSets References Online htpasswd generator Keywords trafeik on kubernetes kubernetes ingress kubernetes annotations daemonsets","title":"Reading Resources"},{"location":"install_kubernetes_with_kubeadm/","text":"Setting up Kubernetes Cluster with Kubeadm This documents describes how to setup kubernetes from scratch on your own nodes, without using a managed service. This setup uses kubeadm to install and configure kubernetes cluster. Compatibility Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. The below steps are applicable for the below mentioned OS OS Version Ubuntu ** 22.10 ** Base Setup Refer to base setup document to set up the nodes if you are configuring the nodes from scratch. Initializing Master This tutorial assumes kube-01 as the master and used kubeadm as a tool to install and setup the cluster. This section also assumes that you are using vagrant based setup provided along with this tutorial. If not, please update the IP address of the master accordingly. To initialize master, run this on kube-01 (1st node) replace 192.168.56.101 with the actual IP of your node kubeadm init --apiserver-advertise-address 192.168.56.101 --pod-network-cidr=192.168.0.0/16 Initialization of the Nodes (Previously Minions) After master being initialized, it should display the command which could be used on all worker/nodes to join the k8s cluster. e.g. kubeadm join --token c04797.8db60f6b2c0dd078 192.168.12.10:6443 --discovery-token-ca-cert-hash sha256:88ebb5d5f7fdfcbbc3cde98690b1dea9d0f96de4a7e6bf69198172debca74cd0 dont copy above command as is, this is just a sample, use actual Copy and paste it on all node. Confgure kubectl Client on Master Node mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Validate kubectl get nodes You could also put the above command on a watch to observe the nodes getting ready. watch kubectl get nodes Configure Kubernetes Networking (CNI) Installing overlay network is necessary for the pods to communicate with each other across the hosts. It is necessary to do this before you try to deploy any applications to your cluster. There are various overlay networking drivers available for kubernetes. We are going to use Weave Net . kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml e2e Testing You could validate the status of this cluster, health of pods and whether all the components are up or not by using a few or all of the following commands. To check if nodes are ready kubectl get nodes [ Expected output ] root@kube-01:~# kubectl get nodes NAME STATUS ROLES AGE VERSION kube-01 Ready master 9m v1.8.2 kube-02 Ready 4m v1.8.2 kube-03 Ready 4m v1.8.2 Additional Status Commands kubectl version -o yaml kubectl cluster-info kubectl get pods -n kube-system kubectl get events It will take a few minutes to have the cluster up and running with all the services. Set up Visualiser Fork the repository and deploy the visualizer on kubernetes git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ Visualiser will run on 32000 port. You could access it using a URL such as below and add /#scale=2.0 or similar option where 2.0 = 200% the scale. replace with actual IP of one of your nodes http://:32000/#scale=2.0 Kubernetes visualiser is a third party application which provides a operational view of your kubernetes cluster. Its very useful tool for learning kubernetes as it demonstrates the state of the cluster as well as state of the pods as you make changes. You could read further about it at this link . Download the supporting code Before we proceed further, please checkout the code from the following git repo. This would offer the supporting code for the exercises that follow. run this on the host where you have configured kubectl git clone https://github.com/initcron/k8s-code.git","title":"Lab K400 - Install Kubernetes with Kubeadm"},{"location":"install_kubernetes_with_kubeadm/#setting-up-kubernetes-cluster-with-kubeadm","text":"This documents describes how to setup kubernetes from scratch on your own nodes, without using a managed service. This setup uses kubeadm to install and configure kubernetes cluster.","title":"Setting up Kubernetes Cluster with Kubeadm"},{"location":"install_kubernetes_with_kubeadm/#compatibility","text":"Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. The below steps are applicable for the below mentioned OS OS Version Ubuntu ** 22.10 **","title":"Compatibility"},{"location":"install_kubernetes_with_kubeadm/#base-setup","text":"Refer to base setup document to set up the nodes if you are configuring the nodes from scratch.","title":"Base Setup"},{"location":"install_kubernetes_with_kubeadm/#initializing-master","text":"This tutorial assumes kube-01 as the master and used kubeadm as a tool to install and setup the cluster. This section also assumes that you are using vagrant based setup provided along with this tutorial. If not, please update the IP address of the master accordingly. To initialize master, run this on kube-01 (1st node) replace 192.168.56.101 with the actual IP of your node kubeadm init --apiserver-advertise-address 192.168.56.101 --pod-network-cidr=192.168.0.0/16","title":"Initializing Master"},{"location":"install_kubernetes_with_kubeadm/#initialization-of-the-nodes-previously-minions","text":"After master being initialized, it should display the command which could be used on all worker/nodes to join the k8s cluster. e.g. kubeadm join --token c04797.8db60f6b2c0dd078 192.168.12.10:6443 --discovery-token-ca-cert-hash sha256:88ebb5d5f7fdfcbbc3cde98690b1dea9d0f96de4a7e6bf69198172debca74cd0 dont copy above command as is, this is just a sample, use actual Copy and paste it on all node.","title":"Initialization of the Nodes (Previously Minions)"},{"location":"install_kubernetes_with_kubeadm/#confgure-kubectl-client","text":"on Master Node mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Validate kubectl get nodes You could also put the above command on a watch to observe the nodes getting ready. watch kubectl get nodes","title":"Confgure kubectl Client"},{"location":"install_kubernetes_with_kubeadm/#configure-kubernetes-networking-cni","text":"Installing overlay network is necessary for the pods to communicate with each other across the hosts. It is necessary to do this before you try to deploy any applications to your cluster. There are various overlay networking drivers available for kubernetes. We are going to use Weave Net . kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml","title":"Configure Kubernetes Networking (CNI)"},{"location":"install_kubernetes_with_kubeadm/#e2e-testing","text":"You could validate the status of this cluster, health of pods and whether all the components are up or not by using a few or all of the following commands. To check if nodes are ready kubectl get nodes [ Expected output ] root@kube-01:~# kubectl get nodes NAME STATUS ROLES AGE VERSION kube-01 Ready master 9m v1.8.2 kube-02 Ready 4m v1.8.2 kube-03 Ready 4m v1.8.2 Additional Status Commands kubectl version -o yaml kubectl cluster-info kubectl get pods -n kube-system kubectl get events It will take a few minutes to have the cluster up and running with all the services.","title":"e2e Testing"},{"location":"install_kubernetes_with_kubeadm/#set-up-visualiser","text":"Fork the repository and deploy the visualizer on kubernetes git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ Visualiser will run on 32000 port. You could access it using a URL such as below and add /#scale=2.0 or similar option where 2.0 = 200% the scale. replace with actual IP of one of your nodes http://:32000/#scale=2.0 Kubernetes visualiser is a third party application which provides a operational view of your kubernetes cluster. Its very useful tool for learning kubernetes as it demonstrates the state of the cluster as well as state of the pods as you make changes. You could read further about it at this link .","title":"Set up Visualiser"},{"location":"install_kubernetes_with_kubeadm/#download-the-supporting-code","text":"Before we proceed further, please checkout the code from the following git repo. This would offer the supporting code for the exercises that follow. run this on the host where you have configured kubectl git clone https://github.com/initcron/k8s-code.git","title":"Download the supporting code"},{"location":"kind_create_cluster/","text":"Create Kubernetes Cluster with KIND This lab describes the process of how you could quickly create a multi node Kubernetes Envonment using KIND , which is a simple and quick way to set up a learning environment. Advantage it offers over minikube or docker desktop based kubernetes setup is its a multi node environment closer to the real world setup. Clean up Containers Before proceeding, clean up any containers running on the host. be warned that the following command would DELETE ALL CONTAINRES on the host docker rm -f $(docker ps -aq) Install Kubectl and KIND To install kubectl client, refer to the official documentation here Install Tools | Kubernetes Validate by running kubectl version --client=true kubectl version --client=true -o yaml Install KinD (Kubernetes inside Docker) using operating specific instructions at kind \u2013 Quick Start . Validate by running kind Setup Kubernetes Cluster with KIND Download Cluster Configurations and Create a 3 Node Kubernetes Cluster as git clone https://github.com/initcron/k8s-code.git cd k8s-code/helper/kind/ kind create cluster --config kind-three-node-cluster.yaml Validate kind get clusters kubectl cluster-info --context kind-kind kubectl get nodes [sample output] root@demo:~# kubectl get nodes NAME STATUS ROLES AGE VERSION kind-control-plane Ready master 78s v1.19.1 kind-worker Ready 47s v1.19.1 kind-worker2 Ready 47s v1.19.1 Wait till you see all nodes in Ready state and you have a cluster operational. Wait for a couple of minutes and then validate if the nodes are up and running. Setup Visualiser cd ~ git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ To check whether visualiser has come up, use the following commands, kubectl get pods,services [Expected output ] [root@bbb-01 ~]# kubectl get pods,services NAME READY STATUS RESTARTS AGE pod/kube-ops-view-65466fb5c9-7gwnm 1/1 Running 0 61s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-ops-view NodePort 10.96.54.166 80:32000/TCP 61s service/kubernetes ClusterIP 10.96.0.1 443/TCP 4m28s To access the visualiser, visit http://IPADDRESS:32000 ( where replace IPADDRESS with the actual hostname or IP of the docker host). You shall see a visualiser similar to the following loaded on the browser. If you see this page , Congratulations !! You have the cluster setup. Restarting and Resetting the Cluster (Skip) Note: This is a Optional Topic. Skil this during your initial setup lab. To stop and start the cluster, you could stop and containers created with docker and then start them back docker ps docker stop kind-control-plane kind-worker kind-worker2 to bring it back again, docker start kind-control-plane kind-worker kind-worker2 Even if you restart your system and bring it up using the above command, it should work. To reset the cluster (note you will be deleting the existing environment and create fresh one) asusming your cluster name is k8slab reset it as : kind get clusters kind delete cluster --name k8slab rm -rf ~/.kube kind create cluster --name k8slab --config kind-three-node-cluster.yaml","title":"Lab K601 - Setup Kubernetes Cluster"},{"location":"kind_create_cluster/#create-kubernetes-cluster-with-kind","text":"This lab describes the process of how you could quickly create a multi node Kubernetes Envonment using KIND , which is a simple and quick way to set up a learning environment. Advantage it offers over minikube or docker desktop based kubernetes setup is its a multi node environment closer to the real world setup.","title":"Create Kubernetes Cluster with KIND"},{"location":"kind_create_cluster/#clean-up-containers","text":"Before proceeding, clean up any containers running on the host. be warned that the following command would DELETE ALL CONTAINRES on the host docker rm -f $(docker ps -aq)","title":"Clean up Containers"},{"location":"kind_create_cluster/#install-kubectl-and-kind","text":"To install kubectl client, refer to the official documentation here Install Tools | Kubernetes Validate by running kubectl version --client=true kubectl version --client=true -o yaml Install KinD (Kubernetes inside Docker) using operating specific instructions at kind \u2013 Quick Start . Validate by running kind","title":"Install Kubectl and KIND"},{"location":"kind_create_cluster/#setup-kubernetes-cluster-with-kind","text":"Download Cluster Configurations and Create a 3 Node Kubernetes Cluster as git clone https://github.com/initcron/k8s-code.git cd k8s-code/helper/kind/ kind create cluster --config kind-three-node-cluster.yaml Validate kind get clusters kubectl cluster-info --context kind-kind kubectl get nodes [sample output] root@demo:~# kubectl get nodes NAME STATUS ROLES AGE VERSION kind-control-plane Ready master 78s v1.19.1 kind-worker Ready 47s v1.19.1 kind-worker2 Ready 47s v1.19.1 Wait till you see all nodes in Ready state and you have a cluster operational. Wait for a couple of minutes and then validate if the nodes are up and running. Setup Visualiser cd ~ git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ To check whether visualiser has come up, use the following commands, kubectl get pods,services [Expected output ] [root@bbb-01 ~]# kubectl get pods,services NAME READY STATUS RESTARTS AGE pod/kube-ops-view-65466fb5c9-7gwnm 1/1 Running 0 61s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-ops-view NodePort 10.96.54.166 80:32000/TCP 61s service/kubernetes ClusterIP 10.96.0.1 443/TCP 4m28s To access the visualiser, visit http://IPADDRESS:32000 ( where replace IPADDRESS with the actual hostname or IP of the docker host). You shall see a visualiser similar to the following loaded on the browser. If you see this page , Congratulations !! You have the cluster setup.","title":"Setup Kubernetes Cluster with KIND"},{"location":"kind_create_cluster/#restarting-and-resetting-the-cluster-skip","text":"Note: This is a Optional Topic. Skil this during your initial setup lab. To stop and start the cluster, you could stop and containers created with docker and then start them back docker ps docker stop kind-control-plane kind-worker kind-worker2 to bring it back again, docker start kind-control-plane kind-worker kind-worker2 Even if you restart your system and bring it up using the above command, it should work. To reset the cluster (note you will be deleting the existing environment and create fresh one) asusming your cluster name is k8slab reset it as : kind get clusters kind delete cluster --name k8slab rm -rf ~/.kube kind create cluster --name k8slab --config kind-three-node-cluster.yaml","title":"Restarting and Resetting the Cluster (Skip)"},{"location":"kubernetes-configuration-management-configmaps-secrets/","text":"Configurations Management with ConfigMaps and Secrets Configmap is one of the ways to provide configurations to your application. Injecting env variables with configmaps Create our configmap for vote app file: projects/instavote/dev/vote-cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: vote namespace: instavote data: OPTION_A: Visa OPTION_B: Mastercard In the above given configmap, we define two environment variables, OPTION_A=EMACS OPTION_B=VI Lets create the configmap object kubectl get cm kubectl apply -f vote-cm.yaml kubectl get cm kubectl describe cm vote In order to use this configmap in the deployment, we need to reference it from the deployment file. Check the deployment file for vote add for the following block. file: vote-deploy.yaml ... spec: containers: - image: schoolofdevops/vote imagePullPolicy: Always name: vote envFrom: - configMapRef: name: vote ports: - containerPort: 80 protocol: TCP restartPolicy: Always So when you create your deployment, these configurations will be made available to your application. In this example, the values defined in the configmap (Visa and Mastercard) will override the default values(CATS and DOGS) present in your source code. kubectl apply -f vote-deploy.yaml Watch the monitoring screen for deployment in progress. kubectl get deploy --show-labels kubectl get rs --show-labels kubectl rollout status deploy/vote Note: Automatic Updation of deployments on ConfigMap Updates Currently, updating configMap does not ensure a new rollout of a deployment. What this means is even after updading configMaps, pods will not immediately reflect the changes. There is a feature request for this https://github.com/kubernetes/kubernetes/issues/22368 Currently, this can be done by using immutable configMaps. Create a configMaps and apply it with deployment. To update, create a new configMaps and do not update the previous one. Treat it as immutable. Update deployment spec to use the new version of the configMaps. This will ensure immediate update. Mounting Files with ConfigMap Volume Type If you want to make files e.g. configuration files available inside the container, you could add it to the ConfigMap, reference it as a volume in a pod and then finally mount it inside a container. Lets add a couple of config files for the vote app this way. Begin modifying the ConfigMap for vote app with the content of the config files as: apiVersion: v1 kind: ConfigMap metadata: name: vote namespace: instavote data: OPTION_A: Visa OPTION_B: Mastercard development_config.py: | class DevelopmentConfig: DEBUG = True TESTING = False SECRET_KEY = 'development-key' SQLALCHEMY_DATABASE_URI = 'sqlite:///development.db' SQLALCHEMY_TRACK_MODIFICATIONS = True SQLALCHEMY_ECHO = True production_config.py: | class ProductionConfig: DEBUG = False TESTING = False SECRET_KEY = 'production-secret-key' SQLALCHEMY_DATABASE_URI = 'postgresql://user:password@localhost/production' SQLALCHEMY_TRACK_MODIFICATIONS = False SQLALCHEMY_ECHO = False where, development_config.py and production_config.py keys contain the entire configuration file content within this config map. Apply and validate with, kubectl apply -f vote-cm.yaml kubectl describe cm vote Now, to mount it as a volume, modify vote-deploy.yaml and add the spec.volumes and spec.container.volumeMount using the code reference below: spec: containers: - image: schoolofdevops/vote:v1 name: vote ... ... volumeMounts: - name: config mountPath: \"/app/config\" readOnly: true volumes: - name: config configMap: name: vote items: - key: \"development_config.py\" path: \"development_config.py\" - key: \"production_config.py\" path: \"production_config.py\" optional: true now apply the changes as kubectl apply -f vote-deploy.yaml To validate exec into one of the pods and list the files kubectl exec -it vote-xxxx-yyyy sh cd /app/config ls cat development_config.py [sample output] /app/config # ls development_config.py production_config.py /app/config # cat development_config.py class DevelopmentConfig: DEBUG = True TESTING = False SECRET_KEY = 'development-key' SQLALCHEMY_DATABASE_URI = 'sqlite:///development.db' SQLALCHEMY_TRACK_MODIFICATIONS = True SQLALCHEMY_ECHO = True This validates that the configuration files were created using ConfigMap and made available as a mount path inside the container. Enabling HTTP Authentication for Nginx Ingress with Secrets In this part of the lab, you will Create a secret with an existing file Modify a ingress rule to use that secret via annotations Validate if traefik, the ingress controller, automatically picks up the configurations and enables http auth s Lets create htpasswd spec as Secret apt install -yq apache2-utils htpasswd -c auth devops Or use Online htpasswd generator to generate a htpasswd spec. if you use the online generator, copy the contents to a file by name auth in the current directory. Then generate the secret as, kubectl create secret generic basic-auth --from-file auth kubectl get secret kubectl describe secret basic-auth And then add annotations to the ingress object so that it is read by the ingress controller to update configurations. Reference Nginx Ingress Page to learn more. file: vote-ing.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web namespace: instavote annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - Vote App' spec: ingressClassName: nginx rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 - host: result.example.com http: paths: - path: / pathType: Prefix backend: service: name: result port: number: 80 where, nginx.ingress.kubernetes.io/auth-type: \"basic\" defines authentication type that needs to be added. nginx.ingress.kubernetes.io/auth-secre: \"basic-auth\" refers to the secret created earlier. apply kubectl apply -f vote-ing.yaml kubectl get ing/vote -o yaml Observe the annotations field. No sooner than you apply this spec, ingress controller reads the event and a basic http authentication is set with the secret you added. +----------+ +--update----> | nginx | | | configs | | +----------+ +----+----+--+ ^ | ingress | : | controller | : +----+-------+ : | +-----+-------+ +---watch----> | ingress | <------- user | annotations | +-------------+ imagePullSecrets with Private Repository Create a private repository from DockerHub as docker image pull nginx:alpine docker image tag nginx:alpine xxxx/nginxpvt:alpine docker login docker image push xxxx/nginxpvt:alpine where replace xxxx with actual dockerhub username. Generate a deployment manifest as kubectl create deployment nginxpvt --image=xxxx/nginxpvt:alpine --replicas=1 --dry-run=client -o yaml | tee nginxpvt-deploy.yaml apply this manifest and list the pods kubectl apply -f nginxpvt-deploy.yaml kubectl get pods You will see the pod in error state. e.g. NAME READY STATUS RESTARTS AGE nginxpvt-79f8998f6c-9tvsg 0/1 ErrImagePull 0 4s and describing the pod will show you that there is issue pulling the image from a private repository Warning Failed 21s (x2 over 36s) kubelet Failed to pull image \"initcron/nginxpvt:alpine\": failed to pull and unpack image \"docker.io/initcron/nginxpvt:alpine\": failed to resolve reference \"docker.io/initcron/nginxpvt:alpine\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed 21s (x2 over 36s) kubelet Error: ErrImagePull You could generate a secret with your registry credentials as, kubectl create secret docker-registry dhcreds \\ --docker-server=https://index.docker.io/v1/ \\ --docker-username=xxxx \\ --docker-password=yyyy \\ --docker-email=abc@pqr where, * xxxx : replace with actual registry username * yyyy : replace with actual password * abc@pqr : your registered email on the registry * https://index.docker.io/v1/ : can be changed to your private registry uri kubectl describe secret dhcreds Update nginxpvt-deploy.yaml with the imagePullSecrets as, spec: containers: - image: initcron/nginxpvt:alpine name: nginxpvt resources: {} imagePullSecrets: - name: dhcred and apply as kubectl apply -f nginxpvt-deploy.yaml kubectl get pods This time you shall see the pod running. once done, you could clean up with kubectl delete deploy nginxpvt kubectl delete secret dhcred","title":"Lab K110 - ConfigMaps & Secrets"},{"location":"kubernetes-configuration-management-configmaps-secrets/#configurations-management-with-configmaps-and-secrets","text":"Configmap is one of the ways to provide configurations to your application.","title":"Configurations Management with ConfigMaps and Secrets"},{"location":"kubernetes-configuration-management-configmaps-secrets/#injecting-env-variables-with-configmaps","text":"Create our configmap for vote app file: projects/instavote/dev/vote-cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: vote namespace: instavote data: OPTION_A: Visa OPTION_B: Mastercard In the above given configmap, we define two environment variables, OPTION_A=EMACS OPTION_B=VI Lets create the configmap object kubectl get cm kubectl apply -f vote-cm.yaml kubectl get cm kubectl describe cm vote In order to use this configmap in the deployment, we need to reference it from the deployment file. Check the deployment file for vote add for the following block. file: vote-deploy.yaml ... spec: containers: - image: schoolofdevops/vote imagePullPolicy: Always name: vote envFrom: - configMapRef: name: vote ports: - containerPort: 80 protocol: TCP restartPolicy: Always So when you create your deployment, these configurations will be made available to your application. In this example, the values defined in the configmap (Visa and Mastercard) will override the default values(CATS and DOGS) present in your source code. kubectl apply -f vote-deploy.yaml Watch the monitoring screen for deployment in progress. kubectl get deploy --show-labels kubectl get rs --show-labels kubectl rollout status deploy/vote","title":"Injecting env variables with configmaps"},{"location":"kubernetes-configuration-management-configmaps-secrets/#note-automatic-updation-of-deployments-on-configmap-updates","text":"Currently, updating configMap does not ensure a new rollout of a deployment. What this means is even after updading configMaps, pods will not immediately reflect the changes. There is a feature request for this https://github.com/kubernetes/kubernetes/issues/22368 Currently, this can be done by using immutable configMaps. Create a configMaps and apply it with deployment. To update, create a new configMaps and do not update the previous one. Treat it as immutable. Update deployment spec to use the new version of the configMaps. This will ensure immediate update.","title":"Note: Automatic Updation of deployments on ConfigMap Updates"},{"location":"kubernetes-configuration-management-configmaps-secrets/#mounting-files-with-configmap-volume-type","text":"If you want to make files e.g. configuration files available inside the container, you could add it to the ConfigMap, reference it as a volume in a pod and then finally mount it inside a container. Lets add a couple of config files for the vote app this way. Begin modifying the ConfigMap for vote app with the content of the config files as: apiVersion: v1 kind: ConfigMap metadata: name: vote namespace: instavote data: OPTION_A: Visa OPTION_B: Mastercard development_config.py: | class DevelopmentConfig: DEBUG = True TESTING = False SECRET_KEY = 'development-key' SQLALCHEMY_DATABASE_URI = 'sqlite:///development.db' SQLALCHEMY_TRACK_MODIFICATIONS = True SQLALCHEMY_ECHO = True production_config.py: | class ProductionConfig: DEBUG = False TESTING = False SECRET_KEY = 'production-secret-key' SQLALCHEMY_DATABASE_URI = 'postgresql://user:password@localhost/production' SQLALCHEMY_TRACK_MODIFICATIONS = False SQLALCHEMY_ECHO = False where, development_config.py and production_config.py keys contain the entire configuration file content within this config map. Apply and validate with, kubectl apply -f vote-cm.yaml kubectl describe cm vote Now, to mount it as a volume, modify vote-deploy.yaml and add the spec.volumes and spec.container.volumeMount using the code reference below: spec: containers: - image: schoolofdevops/vote:v1 name: vote ... ... volumeMounts: - name: config mountPath: \"/app/config\" readOnly: true volumes: - name: config configMap: name: vote items: - key: \"development_config.py\" path: \"development_config.py\" - key: \"production_config.py\" path: \"production_config.py\" optional: true now apply the changes as kubectl apply -f vote-deploy.yaml To validate exec into one of the pods and list the files kubectl exec -it vote-xxxx-yyyy sh cd /app/config ls cat development_config.py [sample output] /app/config # ls development_config.py production_config.py /app/config # cat development_config.py class DevelopmentConfig: DEBUG = True TESTING = False SECRET_KEY = 'development-key' SQLALCHEMY_DATABASE_URI = 'sqlite:///development.db' SQLALCHEMY_TRACK_MODIFICATIONS = True SQLALCHEMY_ECHO = True This validates that the configuration files were created using ConfigMap and made available as a mount path inside the container.","title":"Mounting Files with ConfigMap Volume Type"},{"location":"kubernetes-configuration-management-configmaps-secrets/#enabling-http-authentication-for-nginx-ingress-with-secrets","text":"In this part of the lab, you will Create a secret with an existing file Modify a ingress rule to use that secret via annotations Validate if traefik, the ingress controller, automatically picks up the configurations and enables http auth s Lets create htpasswd spec as Secret apt install -yq apache2-utils htpasswd -c auth devops Or use Online htpasswd generator to generate a htpasswd spec. if you use the online generator, copy the contents to a file by name auth in the current directory. Then generate the secret as, kubectl create secret generic basic-auth --from-file auth kubectl get secret kubectl describe secret basic-auth And then add annotations to the ingress object so that it is read by the ingress controller to update configurations. Reference Nginx Ingress Page to learn more. file: vote-ing.yaml --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web namespace: instavote annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: basic-auth # message to display with an appropriate context why the authentication is required nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - Vote App' spec: ingressClassName: nginx rules: - host: vote.example.com http: paths: - path: / pathType: Prefix backend: service: name: vote port: number: 80 - host: result.example.com http: paths: - path: / pathType: Prefix backend: service: name: result port: number: 80 where, nginx.ingress.kubernetes.io/auth-type: \"basic\" defines authentication type that needs to be added. nginx.ingress.kubernetes.io/auth-secre: \"basic-auth\" refers to the secret created earlier. apply kubectl apply -f vote-ing.yaml kubectl get ing/vote -o yaml Observe the annotations field. No sooner than you apply this spec, ingress controller reads the event and a basic http authentication is set with the secret you added. +----------+ +--update----> | nginx | | | configs | | +----------+ +----+----+--+ ^ | ingress | : | controller | : +----+-------+ : | +-----+-------+ +---watch----> | ingress | <------- user | annotations | +-------------+","title":"Enabling HTTP Authentication for Nginx Ingress with Secrets"},{"location":"kubernetes-configuration-management-configmaps-secrets/#imagepullsecrets-with-private-repository","text":"Create a private repository from DockerHub as docker image pull nginx:alpine docker image tag nginx:alpine xxxx/nginxpvt:alpine docker login docker image push xxxx/nginxpvt:alpine where replace xxxx with actual dockerhub username. Generate a deployment manifest as kubectl create deployment nginxpvt --image=xxxx/nginxpvt:alpine --replicas=1 --dry-run=client -o yaml | tee nginxpvt-deploy.yaml apply this manifest and list the pods kubectl apply -f nginxpvt-deploy.yaml kubectl get pods You will see the pod in error state. e.g. NAME READY STATUS RESTARTS AGE nginxpvt-79f8998f6c-9tvsg 0/1 ErrImagePull 0 4s and describing the pod will show you that there is issue pulling the image from a private repository Warning Failed 21s (x2 over 36s) kubelet Failed to pull image \"initcron/nginxpvt:alpine\": failed to pull and unpack image \"docker.io/initcron/nginxpvt:alpine\": failed to resolve reference \"docker.io/initcron/nginxpvt:alpine\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed 21s (x2 over 36s) kubelet Error: ErrImagePull You could generate a secret with your registry credentials as, kubectl create secret docker-registry dhcreds \\ --docker-server=https://index.docker.io/v1/ \\ --docker-username=xxxx \\ --docker-password=yyyy \\ --docker-email=abc@pqr where, * xxxx : replace with actual registry username * yyyy : replace with actual password * abc@pqr : your registered email on the registry * https://index.docker.io/v1/ : can be changed to your private registry uri kubectl describe secret dhcreds Update nginxpvt-deploy.yaml with the imagePullSecrets as, spec: containers: - image: initcron/nginxpvt:alpine name: nginxpvt resources: {} imagePullSecrets: - name: dhcred and apply as kubectl apply -f nginxpvt-deploy.yaml kubectl get pods This time you shall see the pod running. once done, you could clean up with kubectl delete deploy nginxpvt kubectl delete secret dhcred","title":"imagePullSecrets with Private Repository"},{"location":"kubernetes-dynamic-storage-provisioning/","text":"Dynamic Storage Provisioning This tutorial explains how kubernetes storage works and the complete workflow for the dynamic provisioning. The topics include Storage Classes PersistentVolumeClaim persistentVolume Provisioner Pre Reading : Kubernetes Storage Concepts Storage Classes Concepts Deploy Database with a Persistent Volume Claim Lets begin by redeploying the db deployment, this time by configuring it to refer to the persistentVolumeClaim file: db-deploy-pvc.yaml ... spec: containers: - image: postgres:9.4 imagePullPolicy: Always name: db ports: - containerPort: 5432 protocol: TCP #mount db-vol to postgres data path volumeMounts: - name: db-vol mountPath: /var/lib/postgresql/data #create a volume with pvc volumes: - name: db-vol persistentVolumeClaim: claimName: db-pvc Apply db-deploy-pcv.yaml as kubectl apply -f db-deploy-pvc.yaml To monitor resources for this lab, open a new terminal and start watching for relevant objecting using the following command. watch kubectl get pods,pvc,pv,storageclasses We will call the terminal where you are running the above command as your Monitoring Screen . Observe and note if the pod for db is launched. What state is it in ? why? Has the persistentVolumeClaim been bound to a persistentVolume ? Why? Creating a Persistent Volume Claim switch to project directory cd k8s-code/projects/instavote/dev/ Create the following file with the specs below file: db-pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: db-pvc spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 200Mi storageClassName: local-path if you using a KIND bases environment, set StorageClassName to \"standard\" instead of \"local-path\" in the above file. In case of KIND environment, no further configuration of Storageclass/Provisioner is needed. create the Persistent Volume Claim and validate kubectl get pvc kubectl apply -f db-pvc.yaml kubectl get pvc,pv Is persistentVolumeClaim created ? Why ? Is persistentVolume created ? Why ? Is the persistentVolumeClaim bound with a persistentVolume ? Set up Storage Provisioner in kubernetes Skip this step is you are using KIND based environment. Launch a local path provisioner using the following command, kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml This will create all the objects required to setup a local-path provisioner. At this time, you should also see storageclass created for local-path on your monitoring screen. kubectl get storageclass kubectl get pods kubectl logs -f local-path-provisioner-xxxx -n local-path-storage Validate Now, observe the output of the following commands, kubectl get pvc,pv kubectl get pods Do you see pvc bound to pv ? Do you see the pod for db running ? Observe the dynamic provisioning in action. Nano Project Similar to postgres which mounts the data at /var/lib/postgresql/data and consumes it to store the database files, Redis creates and stores the file at /data path. Your task is to have a volume of size 20Mi created and mounted at /data for the redis container. You could follow these steps to complete this task create a pvc by name redis create a volume in the pod spec with type persistentVolumeClaim. Call is redis-data add volumeMounts to the container spec (part of the same deployment file) running redis and have it mount redis-data volume created in the pod spec above. Summary In this lab, you learnt to setup dynamic provisioning provisioner with Storage Classes, Provisioners and PersistentVolumeClaims.","title":"Lab K108 - Storage"},{"location":"kubernetes-dynamic-storage-provisioning/#dynamic-storage-provisioning","text":"This tutorial explains how kubernetes storage works and the complete workflow for the dynamic provisioning. The topics include Storage Classes PersistentVolumeClaim persistentVolume Provisioner Pre Reading : Kubernetes Storage Concepts Storage Classes","title":"Dynamic Storage Provisioning"},{"location":"kubernetes-dynamic-storage-provisioning/#concepts","text":"","title":"Concepts"},{"location":"kubernetes-dynamic-storage-provisioning/#deploy-database-with-a-persistent-volume-claim","text":"Lets begin by redeploying the db deployment, this time by configuring it to refer to the persistentVolumeClaim file: db-deploy-pvc.yaml ... spec: containers: - image: postgres:9.4 imagePullPolicy: Always name: db ports: - containerPort: 5432 protocol: TCP #mount db-vol to postgres data path volumeMounts: - name: db-vol mountPath: /var/lib/postgresql/data #create a volume with pvc volumes: - name: db-vol persistentVolumeClaim: claimName: db-pvc Apply db-deploy-pcv.yaml as kubectl apply -f db-deploy-pvc.yaml To monitor resources for this lab, open a new terminal and start watching for relevant objecting using the following command. watch kubectl get pods,pvc,pv,storageclasses We will call the terminal where you are running the above command as your Monitoring Screen . Observe and note if the pod for db is launched. What state is it in ? why? Has the persistentVolumeClaim been bound to a persistentVolume ? Why?","title":"Deploy Database with a Persistent Volume Claim"},{"location":"kubernetes-dynamic-storage-provisioning/#creating-a-persistent-volume-claim","text":"switch to project directory cd k8s-code/projects/instavote/dev/ Create the following file with the specs below file: db-pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: db-pvc spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 200Mi storageClassName: local-path if you using a KIND bases environment, set StorageClassName to \"standard\" instead of \"local-path\" in the above file. In case of KIND environment, no further configuration of Storageclass/Provisioner is needed. create the Persistent Volume Claim and validate kubectl get pvc kubectl apply -f db-pvc.yaml kubectl get pvc,pv Is persistentVolumeClaim created ? Why ? Is persistentVolume created ? Why ? Is the persistentVolumeClaim bound with a persistentVolume ?","title":"Creating a Persistent Volume Claim"},{"location":"kubernetes-dynamic-storage-provisioning/#set-up-storage-provisioner-in-kubernetes","text":"Skip this step is you are using KIND based environment. Launch a local path provisioner using the following command, kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml This will create all the objects required to setup a local-path provisioner. At this time, you should also see storageclass created for local-path on your monitoring screen. kubectl get storageclass kubectl get pods kubectl logs -f local-path-provisioner-xxxx -n local-path-storage","title":"Set up Storage Provisioner in kubernetes"},{"location":"kubernetes-dynamic-storage-provisioning/#validate","text":"Now, observe the output of the following commands, kubectl get pvc,pv kubectl get pods Do you see pvc bound to pv ? Do you see the pod for db running ? Observe the dynamic provisioning in action.","title":"Validate"},{"location":"kubernetes-dynamic-storage-provisioning/#nano-project","text":"Similar to postgres which mounts the data at /var/lib/postgresql/data and consumes it to store the database files, Redis creates and stores the file at /data path. Your task is to have a volume of size 20Mi created and mounted at /data for the redis container. You could follow these steps to complete this task create a pvc by name redis create a volume in the pod spec with type persistentVolumeClaim. Call is redis-data add volumeMounts to the container spec (part of the same deployment file) running redis and have it mount redis-data volume created in the pod spec above.","title":"Nano Project"},{"location":"kubernetes-dynamic-storage-provisioning/#summary","text":"In this lab, you learnt to setup dynamic provisioning provisioner with Storage Classes, Provisioners and PersistentVolumeClaims.","title":"Summary"},{"location":"kubernetes-replicasets-howto/","text":"Adding HA and Scalability with ReplicaSets If you are not running a monitoring screen, start it in a new terminal with the following command. watch kubectl get all --show-labels PART I - Namespaces Check current config kubectl config view You could also examine the current configs in file cat ~/.kube/config Creating a Namespace and Switching to it Namespaces offers separation of resources running on the same physical infrastructure into virtual clusters. It is typically useful in mid to large scale environments with multiple projects, teams and need separate scopes. It could also be useful to map to your workflow stages e.g. dev, stage, prod. Before you create a namespace, delete all the pods in the default namespace that you may have created earlier and which you would not need. e.g. kubectl delete pod vote db web Lets create a namespace called instavote kubectl get ns kubectl create namespace instavote kubectl get ns And switch to it kubectl config --help kubectl config get-contexts kubectl config current-context kubectl config set-context --help kubectl config set-context --current --namespace=instavote kubectl config get-contexts kubectl config view Exercise : Go back to the monitoring screen and observe what happens after switching the namespace. e.g. kubectl config set-context --current --namespace=default kubectl config set-context --current --namespace=kube-system kubectl config set-context --current --namespace=instavote PART II - ReplicaSets To understand how ReplicaSets works with the selectors lets launch a pod in the new namespace with existing spec. cd k8s-code/pods kubectl apply -f vote-pod.yaml kubectl get pods Adding ReplicaSet Configurations Lets now write the spec for the ReplicaSet. This is going to mainly contain, replicas : define the scalability configs here selector : define configs which provide as a base for checking availibility template (pod spec ) minReadySeconds : duration to wait after pod is ready till its declared as available (used by deployment) From here on, we would switch to the project and environment specific path and work from there. cd projects/instavote/dev edit file: vote-rs.yaml apiVersion: xxx kind: xxx metadata: xxx spec: xxx template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" Above file already containts the spec that you had written for the pod. You would observe its already been added as part of spec.template for replicaset. Lets now add the details specific to replicaset. file: vote-rs.yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: vote spec: replicas: 4 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - key: version operator: Exists template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" The complete file will look similar to above. Lets now go ahead and apply it. kubectl apply -f vote-rs.yaml --dry-run=client kubectl apply -f vote-rs.yaml kubectl get rs kubectl describe rs vote kubectl get pods kubectl get pods --show-labels High Availability Try deleting pods created by the replicaset, replace pod-xxxx and pod-yyyy with actuals kubectl get pods kubectl delete pods vote-xxxx vote-yyyy Observe as the pods are automatically created again. Lets now delete the pod created independent of replica set. kubectl get pods kubectl delete pods vote Observe what happens. * Does replica set take any action after deleting the pod created outside of its spec ? Why? Scalability Scaling out and scaling in your application is as easy as running, kubectl scale rs vote --replicas=8 kubectl get pods --show-labels kubectl scale rs vote --replicas=25 kubectl get pods --show-labels kubectl scale rs vote --replicas=7 kubectl get pods --show-labels Observe what happens Did the number of replicas increase to 8, then to 25 and reduced to 7 ? Exercise: Deploying new version of the application kubectl edit rs vote Update the version of the image from schoolofdevops/vote:v1 to schoolofdevops/vote:v2 From template.metadata.labels update version=v1 to version=v2 Save the file. If you are using kubectl edit it would apply the changes immediately as you save, without needing to use kubectl apply command. Observe what happens ? Did application get updated. Did updating replicaset launched new pods to deploy new version ? Summary With ReplicaSets your application is now high available as well as scalable. However ReplicaSet by itself does not have the intelligence to trigger a rollout if you update the version. For that, you are going to need a deployment which is something you would learn in an upcoming lesson.","title":"Lab K104 - Namespaces and ReplicaSets"},{"location":"kubernetes-replicasets-howto/#adding-ha-and-scalability-with-replicasets","text":"If you are not running a monitoring screen, start it in a new terminal with the following command. watch kubectl get all --show-labels","title":"Adding HA and Scalability with ReplicaSets"},{"location":"kubernetes-replicasets-howto/#part-i-namespaces","text":"Check current config kubectl config view You could also examine the current configs in file cat ~/.kube/config","title":"PART I - Namespaces"},{"location":"kubernetes-replicasets-howto/#creating-a-namespace-and-switching-to-it","text":"Namespaces offers separation of resources running on the same physical infrastructure into virtual clusters. It is typically useful in mid to large scale environments with multiple projects, teams and need separate scopes. It could also be useful to map to your workflow stages e.g. dev, stage, prod. Before you create a namespace, delete all the pods in the default namespace that you may have created earlier and which you would not need. e.g. kubectl delete pod vote db web Lets create a namespace called instavote kubectl get ns kubectl create namespace instavote kubectl get ns And switch to it kubectl config --help kubectl config get-contexts kubectl config current-context kubectl config set-context --help kubectl config set-context --current --namespace=instavote kubectl config get-contexts kubectl config view Exercise : Go back to the monitoring screen and observe what happens after switching the namespace. e.g. kubectl config set-context --current --namespace=default kubectl config set-context --current --namespace=kube-system kubectl config set-context --current --namespace=instavote","title":"Creating a Namespace and Switching to it"},{"location":"kubernetes-replicasets-howto/#part-ii-replicasets","text":"To understand how ReplicaSets works with the selectors lets launch a pod in the new namespace with existing spec. cd k8s-code/pods kubectl apply -f vote-pod.yaml kubectl get pods","title":"PART II - ReplicaSets"},{"location":"kubernetes-replicasets-howto/#adding-replicaset-configurations","text":"Lets now write the spec for the ReplicaSet. This is going to mainly contain, replicas : define the scalability configs here selector : define configs which provide as a base for checking availibility template (pod spec ) minReadySeconds : duration to wait after pod is ready till its declared as available (used by deployment) From here on, we would switch to the project and environment specific path and work from there. cd projects/instavote/dev edit file: vote-rs.yaml apiVersion: xxx kind: xxx metadata: xxx spec: xxx template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" Above file already containts the spec that you had written for the pod. You would observe its already been added as part of spec.template for replicaset. Lets now add the details specific to replicaset. file: vote-rs.yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: vote spec: replicas: 4 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - key: version operator: Exists template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" The complete file will look similar to above. Lets now go ahead and apply it. kubectl apply -f vote-rs.yaml --dry-run=client kubectl apply -f vote-rs.yaml kubectl get rs kubectl describe rs vote kubectl get pods kubectl get pods --show-labels","title":"Adding ReplicaSet Configurations"},{"location":"kubernetes-replicasets-howto/#high-availability","text":"Try deleting pods created by the replicaset, replace pod-xxxx and pod-yyyy with actuals kubectl get pods kubectl delete pods vote-xxxx vote-yyyy Observe as the pods are automatically created again. Lets now delete the pod created independent of replica set. kubectl get pods kubectl delete pods vote Observe what happens. * Does replica set take any action after deleting the pod created outside of its spec ? Why?","title":"High Availability"},{"location":"kubernetes-replicasets-howto/#scalability","text":"Scaling out and scaling in your application is as easy as running, kubectl scale rs vote --replicas=8 kubectl get pods --show-labels kubectl scale rs vote --replicas=25 kubectl get pods --show-labels kubectl scale rs vote --replicas=7 kubectl get pods --show-labels Observe what happens Did the number of replicas increase to 8, then to 25 and reduced to 7 ?","title":"Scalability"},{"location":"kubernetes-replicasets-howto/#exercise-deploying-new-version-of-the-application","text":"kubectl edit rs vote Update the version of the image from schoolofdevops/vote:v1 to schoolofdevops/vote:v2 From template.metadata.labels update version=v1 to version=v2 Save the file. If you are using kubectl edit it would apply the changes immediately as you save, without needing to use kubectl apply command. Observe what happens ? Did application get updated. Did updating replicaset launched new pods to deploy new version ?","title":"Exercise: Deploying new version of the application"},{"location":"kubernetes-replicasets-howto/#summary","text":"With ReplicaSets your application is now high available as well as scalable. However ReplicaSet by itself does not have the intelligence to trigger a rollout if you update the version. For that, you are going to need a deployment which is something you would learn in an upcoming lesson.","title":"Summary"},{"location":"kubernetes-rollouts-rollbacks-deployments/","text":"Defining Release Strategy with Deployment A Deployment is a higher level abstraction which sits on top of replica sets and allows you to manage the way applications are deployed, rolled back at a controlled rate. Deployment provides three features, Availability : Maintain the number of replicas for a type of service/app. Schedule/delete pods to meet the desired count. Scalability : Updating the replica count, allows you to scale in and out, and its the responsibility of the deployment to provide with that scalability. You could scale manually or use horizontalPodAutoscaler to do it automatically. Update Strategy : Define a release strategy and update the pods accordingly. Begin by deleting the previously created replicaset as the presence of it may lead to duplicate set of pods. kubectl delete rs vote Since deployment is just a superset of replicaset specs, lets begin creating a deployment by copying over the existing replicaset code for vote app. /k8s-code/projects/instavote/dev/ cp vote-rs.yaml vote-deploy.yaml Deployment spec (deployment.spec) contains everything that replica set has + strategy. Lets add it as follows, file: vote-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote labels: role: vote spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 replicas: 12 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: In, values: [v1, v2, v3, v4, v5]} template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" In a additional terminal where you have kubectl client setup, launch the monitoring console by running the following command. Ensure you have --show-labels options set. watch -n 1 kubectl get all --show-labels Lets create the Deployment. Do monitor the labels of the pod while applying this. kubectl apply -f vote-deploy.yaml Observe the chances to pod labels, specifically the pod-template-hash . Now that the deployment is created. To validate, kubectl get deployment kubectl get rs --show-labels kubectl get deploy,pods,rs kubectl rollout status deployment/vote kubectl get pods --show-labels Rolling out a new version Now, update the deployment spec to use a new version of the image. file: vote-deploy.yaml ... template: metadata: labels: version: v2 spec: containers: - name: app image: schoolofdevops/vote:v2 and trigger a rollout by applying it kubectl apply -f vote-deploy.yaml Open a couple of additional terminals on the host where kubectl is configured and launch the following monitoring commands. Terminal 1 kubectl rollout status deployment/vote Terminal 2 kubectl get all -l \"role=vote\" What to watch for ? rollout status using the command above following fields for vote deployment on monitoring screen READY count (will reflect surged replicas e.g. 14/12) AVAILABLE count ( will never fall below REPLICAS - maxUnavilable) UP-TO-DATE field will reflect replicas with latest version defined in deployment spec observe that a new replicaset is created for every rollout. Continuously refresh the vote application in the browser to see new version if being rolled out without a downtime. Try updating the version of the image from v2 to v3,v4,v5. Repeat a few times to observe how it rolls out a new version. Breaking a Rollout Introduce an error by using an image which does not exist. file: vote-deploy.yaml spec: containers: - name: app image: schoolofdevops/vote:rgjerdf apply kubectl apply -f vote-deploy.yaml kubectl rollout status Observe how deployment handles the failure in the rolls out. Undo and Rollback To observe the rollout history use the following commands. kubectl rollout history deploy/vote kubectl rollout history deploy/vote --revision=xx where replace xx with revisions Find out the previous revision with sane configs. To undo to a sane version (for example revision 2) kubectl rollout undo deploy/vote --to-revision=2","title":"Lab K107 - Deployments"},{"location":"kubernetes-rollouts-rollbacks-deployments/#defining-release-strategy-with-deployment","text":"A Deployment is a higher level abstraction which sits on top of replica sets and allows you to manage the way applications are deployed, rolled back at a controlled rate. Deployment provides three features, Availability : Maintain the number of replicas for a type of service/app. Schedule/delete pods to meet the desired count. Scalability : Updating the replica count, allows you to scale in and out, and its the responsibility of the deployment to provide with that scalability. You could scale manually or use horizontalPodAutoscaler to do it automatically. Update Strategy : Define a release strategy and update the pods accordingly. Begin by deleting the previously created replicaset as the presence of it may lead to duplicate set of pods. kubectl delete rs vote Since deployment is just a superset of replicaset specs, lets begin creating a deployment by copying over the existing replicaset code for vote app. /k8s-code/projects/instavote/dev/ cp vote-rs.yaml vote-deploy.yaml Deployment spec (deployment.spec) contains everything that replica set has + strategy. Lets add it as follows, file: vote-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote labels: role: vote spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 replicas: 12 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: In, values: [v1, v2, v3, v4, v5]} template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" In a additional terminal where you have kubectl client setup, launch the monitoring console by running the following command. Ensure you have --show-labels options set. watch -n 1 kubectl get all --show-labels Lets create the Deployment. Do monitor the labels of the pod while applying this. kubectl apply -f vote-deploy.yaml Observe the chances to pod labels, specifically the pod-template-hash . Now that the deployment is created. To validate, kubectl get deployment kubectl get rs --show-labels kubectl get deploy,pods,rs kubectl rollout status deployment/vote kubectl get pods --show-labels","title":"Defining Release Strategy with Deployment"},{"location":"kubernetes-rollouts-rollbacks-deployments/#rolling-out-a-new-version","text":"Now, update the deployment spec to use a new version of the image. file: vote-deploy.yaml ... template: metadata: labels: version: v2 spec: containers: - name: app image: schoolofdevops/vote:v2 and trigger a rollout by applying it kubectl apply -f vote-deploy.yaml Open a couple of additional terminals on the host where kubectl is configured and launch the following monitoring commands. Terminal 1 kubectl rollout status deployment/vote Terminal 2 kubectl get all -l \"role=vote\" What to watch for ? rollout status using the command above following fields for vote deployment on monitoring screen READY count (will reflect surged replicas e.g. 14/12) AVAILABLE count ( will never fall below REPLICAS - maxUnavilable) UP-TO-DATE field will reflect replicas with latest version defined in deployment spec observe that a new replicaset is created for every rollout. Continuously refresh the vote application in the browser to see new version if being rolled out without a downtime. Try updating the version of the image from v2 to v3,v4,v5. Repeat a few times to observe how it rolls out a new version.","title":"Rolling out a new version"},{"location":"kubernetes-rollouts-rollbacks-deployments/#breaking-a-rollout","text":"Introduce an error by using an image which does not exist. file: vote-deploy.yaml spec: containers: - name: app image: schoolofdevops/vote:rgjerdf apply kubectl apply -f vote-deploy.yaml kubectl rollout status Observe how deployment handles the failure in the rolls out.","title":"Breaking a Rollout"},{"location":"kubernetes-rollouts-rollbacks-deployments/#undo-and-rollback","text":"To observe the rollout history use the following commands. kubectl rollout history deploy/vote kubectl rollout history deploy/vote --revision=xx where replace xx with revisions Find out the previous revision with sane configs. To undo to a sane version (for example revision 2) kubectl rollout undo deploy/vote --to-revision=2","title":"Undo and Rollback"},{"location":"kubernetes-sample-project/","text":"Mini Project: Deploying Multi Tier Application Stack In this project , you would write definitions for deploying the vote application stack with all components/tiers which include, vote redis worker db result Project Description Apply the deployment and service code for the applications marked as ready Complete the code for deployments and services marked as TODO Apply the definitions that you have completed Validate the application workflow is operational by loading vote and result applications from the browser Following table depicts the state of readiness of the above services. App Deployment Service vote ready ready redis ready ready worker TODO n/a db ready ready result TODO TODO Apply existing code cd k8s-code/projects/instavote/dev/ kubectl config set-context --current --namespace=instavote kubectl apply -f vote-rs.yaml -f vote-svc.yaml kubectl apply -f redis-deploy.yaml -f redis-svc.yaml kubectl apply -f db-deploy.yaml -f db-svc.yaml validate kubectl get all Where you should see, replicaset and service for vote app created deployments and services for redis and db created If you see the above objects, proceed with the next task. Completing Code for worker and result apps You would find the files available in the same directory as above i.e. k8s-code/projects/instavote/dev/ with either partial or blank code. Your job is to complete the deployment and service yaml specs and apply those. While writing the specs, you could refer to the following specification. worker image: schoolofdevops/worker:latest results image: schoolofdevops/vote-result application port: 80 service type: NodePort nodePort : 30100 To Validate: kubectl get all The above command should show, * five deployments and four services created * services for vote and result app should have been exposed with NodePort Find out the NodePort for vote and service apps and load those from your browser. This is how the vote application should look like. You could also load the Result page on the 30100 port of your Node IP/Address as, Above is how the result app should look like. Final validation is, if you submit the vote, you should see the results changing accordingly. You would see this behavior only if all deployments and services are created and configured properly.","title":"Lab K109 - Mini Project"},{"location":"kubernetes-sample-project/#mini-project-deploying-multi-tier-application-stack","text":"In this project , you would write definitions for deploying the vote application stack with all components/tiers which include, vote redis worker db result","title":"Mini Project: Deploying Multi Tier Application Stack"},{"location":"kubernetes-sample-project/#project-description","text":"Apply the deployment and service code for the applications marked as ready Complete the code for deployments and services marked as TODO Apply the definitions that you have completed Validate the application workflow is operational by loading vote and result applications from the browser Following table depicts the state of readiness of the above services. App Deployment Service vote ready ready redis ready ready worker TODO n/a db ready ready result TODO TODO","title":"Project Description"},{"location":"kubernetes-sample-project/#apply-existing-code","text":"cd k8s-code/projects/instavote/dev/ kubectl config set-context --current --namespace=instavote kubectl apply -f vote-rs.yaml -f vote-svc.yaml kubectl apply -f redis-deploy.yaml -f redis-svc.yaml kubectl apply -f db-deploy.yaml -f db-svc.yaml validate kubectl get all Where you should see, replicaset and service for vote app created deployments and services for redis and db created If you see the above objects, proceed with the next task.","title":"Apply existing code"},{"location":"kubernetes-sample-project/#completing-code-for-worker-and-result-apps","text":"You would find the files available in the same directory as above i.e. k8s-code/projects/instavote/dev/ with either partial or blank code. Your job is to complete the deployment and service yaml specs and apply those. While writing the specs, you could refer to the following specification. worker image: schoolofdevops/worker:latest results image: schoolofdevops/vote-result application port: 80 service type: NodePort nodePort : 30100","title":"Completing Code for worker and result apps"},{"location":"kubernetes-sample-project/#to-validate","text":"kubectl get all The above command should show, * five deployments and four services created * services for vote and result app should have been exposed with NodePort Find out the NodePort for vote and service apps and load those from your browser. This is how the vote application should look like. You could also load the Result page on the 30100 port of your Node IP/Address as, Above is how the result app should look like. Final validation is, if you submit the vote, you should see the results changing accordingly. You would see this behavior only if all deployments and services are created and configured properly.","title":"To Validate:"},{"location":"kubernetes-service-networking/","text":"Service Networking, Load Balancing and Service Discovery In this lab, you would not only publish the application deployed with replicaset earlier, but also learn about the load balancing and service discovery features offered by kubernetes. Concepts related to Kubernetes Services are depicted in the following diagram, Publishing external facing app with NodePort Kubernetes comes with four types of services viz. ClusterIP NodePort LoadBalancer ExternalName Lets create a service of type NodePort to understand how it works. To check the status of kubernetes objects, kubectl get all You could also start watching the above output for changes. To do so, open a separate terminal window and run, watch kubectl get all Refer to Service Specs to understand the properties that you could write. filename: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort Apply this file to to create a service kubectl apply -f vote-svc.yaml --dry-run kubectl apply -f vote-svc.yaml kubectl get svc kubectl describe service vote [Sample Output of describe command] Name: vote Namespace: instavote Labels: role=svc tier=front Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"svc\",\"tier\":\"front\"},\"name\":\"vote\",\"namespace\":\"instavote\"},\"spec\":{... Selector: app=vote Type: NodePort IP: 10.108.108.157 Port: 80/TCP TargetPort: 80/TCP NodePort: 31429/TCP Endpoints: 10.38.0.4:80,10.38.0.5:80,10.38.0.6:80 + 2 more... Session Affinity: None External Traffic Policy: Cluster Events: Observe the following Selector TargetPort NodePort Endpoints Go to browser and check http://HOSTIP:NODEPORT Here the node port is 30000 (as defined by nodePort in service spec). Sample output will be: If you refresh the page, you should also notice its sending traffic to diffent pod each time, in round robin fashion. Exercises Change the selector criteria to use a non existant label. Use kubectl edit svc./vote to update and apply the configuration. Observe the output of describe command and check the endpoints. Do you see any ? How are selectors and pod labels related ? Observe the number of endpoints. Then change the scale of replicas created by the replicasets. Does it have any effect on the number of endpoints ? Services Under the Hood Lets traverse the route of the network packet that comes in on port 30000 on any node in your cluster. Connect to a node (If creatd using KIND) with, docker exec -it --privileged kind-worker2 sh and then check the IPTables config as, iptables -nvL -t nat iptables -nvL -t nat | grep 30000 Anything that comes on dpt:3000, gets forwarded to the chain created for that service. iptables -nvL -t nat | grep KUBE-SVC-VIQHAVHDK4QE7NA4 -A 10 Chain KUBE-SVC-VIQHAVHDK4QE7NA4 (2 references) pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-RFJGHFMXUDJXIEW6 all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.20000000019 0 0 KUBE-SEP-GBR5YQCVRYY3BA6U all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.25000000000 0 0 KUBE-SEP-BAI3HQ7SV7RZ2CI6 all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.33332999982 0 0 KUBE-SEP-2EQSLPEP3WDOTI5J all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.50000000000 0 0 KUBE-SEP-2CJQISP4W7F2HCRW all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ Where, count of KUBE-SEP-xxx matches number of pods. KUBE-SEP-BAI3HQ7SV7RZ2CI6 is an example of a chain created for one of the hosts. Examine that next. iptables -nvL -t nat | grep KUBE-SEP-BAI3HQ7SV7RZ2CI6 -A 3 [output] pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 10.32.0.6 0.0.0.0/0 /* instavote/vote: */ 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ tcp to:10.32.0.6:80 -- where the packet is being forwarded to 10.32.0.6, which should corraborate with the ip of the pod e.g. kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE vote-58bpv 1/1 Running 0 1h 10.32.0.6 k-02 vote-986cl 1/1 Running 0 1h 10.38.0.5 k-03 vote-9rrfz 1/1 Running 0 1h 10.38.0.4 k-03 vote-dx8f4 1/1 Running 0 1h 10.32.0.4 k-02 vote-qxmfl 1/1 Running 0 1h 10.32.0.5 k-02 10.32.0.6 matches ip of vote-58bpv to check how the packet is routed next use, ip route show [output] default via 128.199.128.1 dev eth0 onlink 10.15.0.0/16 dev eth0 proto kernel scope link src 10.15.0.10 10.32.0.0/12 dev weave proto kernel scope link src 10.32.0.1 128.199.128.0/18 dev eth0 proto kernel scope link src 128.199.185.90 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown where, 10.32.0.0/12 is going over weave interface. Exposing a Service with ExternalIPs Observe the output of service list, specifically note the EXTERNAL-IP colum in the output. kubectl get svc Now, update the service spec and add external IP configs. Pick IP addresses of any two nodes (You could add one or more) and it to the spec as, kubectl edit svc vote [sample file edit] --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort externalIPs: - xx.xx.xx.xx - yy.yy.yy.yy Where replace xx.xx.xx.xx and yy.yy.yy.yy with IP addresses of the nodes on two of the kubernetes hosts. apply kubectl get svc kubectl apply -f vote-svc.yaml kubectl get svc kubectl describe svc vote [sample output] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vote NodePort 10.107.71.204 206.189.150.190,159.65.8.227 80:30000/TCP 11m where, EXTERNAL-IP column shows which IPs the application is been exposed on. You could go to http:// : to access this application. e.g. http://206.189.150.190:80 where you should replace 206.189.150.190 with the actual IP address of the node that you exposed this on. Internal Service Discovery Kubernetes not only allows you to publish external facing apps with the services, but also allows you to discover other components of your application stack with the clusterIP and DNS attached to it. Before you begin adding service discovery, Visit the vote app from browser Attempt to vote by clicking on one of the options observe what happens. Does it go through? Debugging, kubectl get pod kubectl exec vote-xxxx nslookup redis [replace xxxx with the actual pod id of one of the vote pods ] keep the above command on a watch. You should create a new terminal to run the watch command. e.g. kubectl exec -it vote-xxxx sh watch kubectl exec vote-xxxx ping redis where, vote-xxxx is one of the vote pods that I am running. Replace this with the actual pod id. Now create redis service kubectl apply -f redis-svc.yaml kubectl get svc kubectl describe svc redis Watch the nslookup screen and observe if its able to resolve redis by hostname and its pointing to an IP address. e.g. Name: redis Address 1: 10.104.111.173 redis.instavote.svc.cluster.local where 10.104.111.173 is the ClusterIP assigned to redis service redis.instavote.svc.cluster.local is the dns attached to the ClusterIP above What happened here? Service redis was created with a ClusterIP e.g. 10.102.77.6 A DNS entry was created for this service. The fqdn of the service is redis.instavote.svc.cluster.local and it takes the form of my-svc.my-namespace.svc.cluster.local Each pod points to internal DNS server running in the cluster. You could see the details of this by running the following commands kubectl exec vote-xxxx cat /etc/resolv.conf [replace vote-xxxx with actual pod id] [sample output] nameserver 10.96.0.10 search instavote.svc.cluster.local svc.cluster.local cluster.local options ndots:5 where 10.96.0.10 is the ClusterIP assigned to the DNS service. You could co relate that with, kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 1h kubernetes-dashboard NodePort 10.104.42.73 80:31000/TCP 23m where, 10.96.0.10 is the ClusterIP assigned to kube-dns and matches the configuration in /etc/resolv.conf above. Creating Endpoints for Redis Service is been created, but you still need to launch the actual pods running redis application. Create the endpoints now, kubectl apply -f redis-deploy.yaml kubectl describe svc redis [sample output] Name: redis Namespace: instavote Labels: role=redis tier=back Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"redis\",\"tier\":\"back\"},\"name\":\"redis\",\"namespace\":\"instavote\"},\"spec\"... Selector: app=redis Type: ClusterIP IP: 10.102.77.6 Port: 6379/TCP TargetPort: 6379/TCP Endpoints: 10.32.0.6:6379,10.46.0.6:6379 Session Affinity: None Events: Again, visit the vote app from browser, attempt to register your vote. observe what happens. This time the vote should be registered successfully. Summary In this lab, you have published a front facing application, learnt how services are implemented under the hood as well as added service discovery to provide connection strings automatically. Reading List Debugging Services Kubernetes Services Documentation Service API Specs for Kubernetes Version 1.10","title":"Lab K105 - Service Networking"},{"location":"kubernetes-service-networking/#service-networking-load-balancing-and-service-discovery","text":"In this lab, you would not only publish the application deployed with replicaset earlier, but also learn about the load balancing and service discovery features offered by kubernetes. Concepts related to Kubernetes Services are depicted in the following diagram,","title":"Service Networking, Load Balancing and Service Discovery"},{"location":"kubernetes-service-networking/#publishing-external-facing-app-with-nodeport","text":"Kubernetes comes with four types of services viz. ClusterIP NodePort LoadBalancer ExternalName Lets create a service of type NodePort to understand how it works. To check the status of kubernetes objects, kubectl get all You could also start watching the above output for changes. To do so, open a separate terminal window and run, watch kubectl get all Refer to Service Specs to understand the properties that you could write. filename: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort Apply this file to to create a service kubectl apply -f vote-svc.yaml --dry-run kubectl apply -f vote-svc.yaml kubectl get svc kubectl describe service vote [Sample Output of describe command] Name: vote Namespace: instavote Labels: role=svc tier=front Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"svc\",\"tier\":\"front\"},\"name\":\"vote\",\"namespace\":\"instavote\"},\"spec\":{... Selector: app=vote Type: NodePort IP: 10.108.108.157 Port: 80/TCP TargetPort: 80/TCP NodePort: 31429/TCP Endpoints: 10.38.0.4:80,10.38.0.5:80,10.38.0.6:80 + 2 more... Session Affinity: None External Traffic Policy: Cluster Events: Observe the following Selector TargetPort NodePort Endpoints Go to browser and check http://HOSTIP:NODEPORT Here the node port is 30000 (as defined by nodePort in service spec). Sample output will be: If you refresh the page, you should also notice its sending traffic to diffent pod each time, in round robin fashion.","title":"Publishing external facing app with NodePort"},{"location":"kubernetes-service-networking/#exercises","text":"Change the selector criteria to use a non existant label. Use kubectl edit svc./vote to update and apply the configuration. Observe the output of describe command and check the endpoints. Do you see any ? How are selectors and pod labels related ? Observe the number of endpoints. Then change the scale of replicas created by the replicasets. Does it have any effect on the number of endpoints ?","title":"Exercises"},{"location":"kubernetes-service-networking/#services-under-the-hood","text":"Lets traverse the route of the network packet that comes in on port 30000 on any node in your cluster. Connect to a node (If creatd using KIND) with, docker exec -it --privileged kind-worker2 sh and then check the IPTables config as, iptables -nvL -t nat iptables -nvL -t nat | grep 30000 Anything that comes on dpt:3000, gets forwarded to the chain created for that service. iptables -nvL -t nat | grep KUBE-SVC-VIQHAVHDK4QE7NA4 -A 10 Chain KUBE-SVC-VIQHAVHDK4QE7NA4 (2 references) pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-RFJGHFMXUDJXIEW6 all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.20000000019 0 0 KUBE-SEP-GBR5YQCVRYY3BA6U all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.25000000000 0 0 KUBE-SEP-BAI3HQ7SV7RZ2CI6 all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.33332999982 0 0 KUBE-SEP-2EQSLPEP3WDOTI5J all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ statistic mode random probability 0.50000000000 0 0 KUBE-SEP-2CJQISP4W7F2HCRW all -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ Where, count of KUBE-SEP-xxx matches number of pods. KUBE-SEP-BAI3HQ7SV7RZ2CI6 is an example of a chain created for one of the hosts. Examine that next. iptables -nvL -t nat | grep KUBE-SEP-BAI3HQ7SV7RZ2CI6 -A 3 [output] pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 10.32.0.6 0.0.0.0/0 /* instavote/vote: */ 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* instavote/vote: */ tcp to:10.32.0.6:80 -- where the packet is being forwarded to 10.32.0.6, which should corraborate with the ip of the pod e.g. kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE vote-58bpv 1/1 Running 0 1h 10.32.0.6 k-02 vote-986cl 1/1 Running 0 1h 10.38.0.5 k-03 vote-9rrfz 1/1 Running 0 1h 10.38.0.4 k-03 vote-dx8f4 1/1 Running 0 1h 10.32.0.4 k-02 vote-qxmfl 1/1 Running 0 1h 10.32.0.5 k-02 10.32.0.6 matches ip of vote-58bpv to check how the packet is routed next use, ip route show [output] default via 128.199.128.1 dev eth0 onlink 10.15.0.0/16 dev eth0 proto kernel scope link src 10.15.0.10 10.32.0.0/12 dev weave proto kernel scope link src 10.32.0.1 128.199.128.0/18 dev eth0 proto kernel scope link src 128.199.185.90 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown where, 10.32.0.0/12 is going over weave interface.","title":"Services Under the Hood"},{"location":"kubernetes-service-networking/#exposing-a-service-with-externalips","text":"Observe the output of service list, specifically note the EXTERNAL-IP colum in the output. kubectl get svc Now, update the service spec and add external IP configs. Pick IP addresses of any two nodes (You could add one or more) and it to the spec as, kubectl edit svc vote [sample file edit] --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort externalIPs: - xx.xx.xx.xx - yy.yy.yy.yy Where replace xx.xx.xx.xx and yy.yy.yy.yy with IP addresses of the nodes on two of the kubernetes hosts. apply kubectl get svc kubectl apply -f vote-svc.yaml kubectl get svc kubectl describe svc vote [sample output] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vote NodePort 10.107.71.204 206.189.150.190,159.65.8.227 80:30000/TCP 11m where, EXTERNAL-IP column shows which IPs the application is been exposed on. You could go to http:// : to access this application. e.g. http://206.189.150.190:80 where you should replace 206.189.150.190 with the actual IP address of the node that you exposed this on.","title":"Exposing a Service with ExternalIPs"},{"location":"kubernetes-service-networking/#internal-service-discovery","text":"Kubernetes not only allows you to publish external facing apps with the services, but also allows you to discover other components of your application stack with the clusterIP and DNS attached to it. Before you begin adding service discovery, Visit the vote app from browser Attempt to vote by clicking on one of the options observe what happens. Does it go through? Debugging, kubectl get pod kubectl exec vote-xxxx nslookup redis [replace xxxx with the actual pod id of one of the vote pods ] keep the above command on a watch. You should create a new terminal to run the watch command. e.g. kubectl exec -it vote-xxxx sh watch kubectl exec vote-xxxx ping redis where, vote-xxxx is one of the vote pods that I am running. Replace this with the actual pod id. Now create redis service kubectl apply -f redis-svc.yaml kubectl get svc kubectl describe svc redis Watch the nslookup screen and observe if its able to resolve redis by hostname and its pointing to an IP address. e.g. Name: redis Address 1: 10.104.111.173 redis.instavote.svc.cluster.local where 10.104.111.173 is the ClusterIP assigned to redis service redis.instavote.svc.cluster.local is the dns attached to the ClusterIP above What happened here? Service redis was created with a ClusterIP e.g. 10.102.77.6 A DNS entry was created for this service. The fqdn of the service is redis.instavote.svc.cluster.local and it takes the form of my-svc.my-namespace.svc.cluster.local Each pod points to internal DNS server running in the cluster. You could see the details of this by running the following commands kubectl exec vote-xxxx cat /etc/resolv.conf [replace vote-xxxx with actual pod id] [sample output] nameserver 10.96.0.10 search instavote.svc.cluster.local svc.cluster.local cluster.local options ndots:5 where 10.96.0.10 is the ClusterIP assigned to the DNS service. You could co relate that with, kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 1h kubernetes-dashboard NodePort 10.104.42.73 80:31000/TCP 23m where, 10.96.0.10 is the ClusterIP assigned to kube-dns and matches the configuration in /etc/resolv.conf above.","title":"Internal Service Discovery"},{"location":"kubernetes-service-networking/#creating-endpoints-for-redis","text":"Service is been created, but you still need to launch the actual pods running redis application. Create the endpoints now, kubectl apply -f redis-deploy.yaml kubectl describe svc redis [sample output] Name: redis Namespace: instavote Labels: role=redis tier=back Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"redis\",\"tier\":\"back\"},\"name\":\"redis\",\"namespace\":\"instavote\"},\"spec\"... Selector: app=redis Type: ClusterIP IP: 10.102.77.6 Port: 6379/TCP TargetPort: 6379/TCP Endpoints: 10.32.0.6:6379,10.46.0.6:6379 Session Affinity: None Events: Again, visit the vote app from browser, attempt to register your vote. observe what happens. This time the vote should be registered successfully.","title":"Creating Endpoints for Redis"},{"location":"kubernetes-service-networking/#summary","text":"In this lab, you have published a front facing application, learnt how services are implemented under the hood as well as added service discovery to provide connection strings automatically.","title":"Summary"},{"location":"kubernetes-service-networking/#reading-list","text":"Debugging Services Kubernetes Services Documentation Service API Specs for Kubernetes Version 1.10","title":"Reading List"},{"location":"kubernetes_quickdive/","text":"Kubernetes Quick Dive In this lab you are going to deploy the instavote application stack as described here in a kubernetes environment using kubectl commands. Later, you would learn how to do the same by writing declarive yaml syntax. Purpose of this lab is to quickly get your app up and running and demonstrate kubernetes key features such as scheduling, high availability, scalability, load balancing, service discovery etc. Deploying app with kubernetes Before launching the app, create a new namespace and switch to it kubectl get ns kubectl create namespace instavote kubectl get ns kubectl config get-contexts kubectl config set-context --current --namespace=instavote kubectl config get-contexts Launch vote application with kubernetes in the newly created namespace. kubectl create deployment vote --image=schoolofdevops/vote:v4 You could now validate that the instance of vote app is running by using the following commands, kubectl get pods kubectl get deploy kubectl get all Scalability Scale the vote app to run 4 instances. kubectl scale deployment vote --replicas=4 kubectl get deployments,pods High Availability kubectl get pods The above command will list pods. Try to delete a few pods and observe how it affects the availability of your application. kubectl delete pods vote-xxxx vote-yyyy kubectl get deploy,rs,pods Load Balancing with Services Publish the application (similar to using -P for port mapping) kubectl create service nodeport vote --tcp=80:80 --node-port=30300 kubectl get service Connect to the app, refresh the page to see it load balancing. Also try to vote and observe what happens. Roll Out a New Version kubectl scale deployment vote --replicas=12 kubectl set image deployment vote vote=schoolofdevops/vote:v5 watch the rolling update in action kubectl rollout status deploy/vote Exercise - Deploy Complete Instavote App Deploy the services with the following spec to complete this application stack. Service Name Image Service Type Service Port Node Port redis redis:alpine ClusterIP 6379 N/A worker schoolofdevops/worker:latest No Service Needed N/A N/A db postgres:9.4 ClusterIP 5432 N/A result schoolofdevops/vote-result NodePort 80 30400 If you see db deployment failing, fix it by adding the environment var as, kubectl set env deployment db POSTGRES_HOST_AUTH_METHOD=trust After deploying all services to validate, Browse to vote and result services exposed outside to see the UI When you submit a vote, it should be reflected on result To submit multiple votes, use either a different browser, or use incognito window. Cleaning up Once you are done observing, you could delete it with the following commands, kubectl delete deploy vote redis worker db result kubectl delete service vote redis db result Summary When you deploy an application in kubernetes, you submit it to the api server/ cluster manager. Kubernetes automatically schedules it on a cluster, networks the pods, provides service discovery. In addition as you observed, your application is scalable, high available and is already running behind a load balancer.","title":"Lab K102 - Kubernetes Quickdive"},{"location":"kubernetes_quickdive/#kubernetes-quick-dive","text":"In this lab you are going to deploy the instavote application stack as described here in a kubernetes environment using kubectl commands. Later, you would learn how to do the same by writing declarive yaml syntax. Purpose of this lab is to quickly get your app up and running and demonstrate kubernetes key features such as scheduling, high availability, scalability, load balancing, service discovery etc.","title":"Kubernetes Quick Dive"},{"location":"kubernetes_quickdive/#deploying-app-with-kubernetes","text":"Before launching the app, create a new namespace and switch to it kubectl get ns kubectl create namespace instavote kubectl get ns kubectl config get-contexts kubectl config set-context --current --namespace=instavote kubectl config get-contexts Launch vote application with kubernetes in the newly created namespace. kubectl create deployment vote --image=schoolofdevops/vote:v4 You could now validate that the instance of vote app is running by using the following commands, kubectl get pods kubectl get deploy kubectl get all","title":"Deploying app with kubernetes"},{"location":"kubernetes_quickdive/#scalability","text":"Scale the vote app to run 4 instances. kubectl scale deployment vote --replicas=4 kubectl get deployments,pods","title":"Scalability"},{"location":"kubernetes_quickdive/#high-availability","text":"kubectl get pods The above command will list pods. Try to delete a few pods and observe how it affects the availability of your application. kubectl delete pods vote-xxxx vote-yyyy kubectl get deploy,rs,pods","title":"High Availability"},{"location":"kubernetes_quickdive/#load-balancing-with-services","text":"Publish the application (similar to using -P for port mapping) kubectl create service nodeport vote --tcp=80:80 --node-port=30300 kubectl get service Connect to the app, refresh the page to see it load balancing. Also try to vote and observe what happens.","title":"Load Balancing with Services"},{"location":"kubernetes_quickdive/#roll-out-a-new-version","text":"kubectl scale deployment vote --replicas=12 kubectl set image deployment vote vote=schoolofdevops/vote:v5 watch the rolling update in action kubectl rollout status deploy/vote","title":"Roll Out a New Version"},{"location":"kubernetes_quickdive/#exercise-deploy-complete-instavote-app","text":"Deploy the services with the following spec to complete this application stack. Service Name Image Service Type Service Port Node Port redis redis:alpine ClusterIP 6379 N/A worker schoolofdevops/worker:latest No Service Needed N/A N/A db postgres:9.4 ClusterIP 5432 N/A result schoolofdevops/vote-result NodePort 80 30400 If you see db deployment failing, fix it by adding the environment var as, kubectl set env deployment db POSTGRES_HOST_AUTH_METHOD=trust After deploying all services to validate, Browse to vote and result services exposed outside to see the UI When you submit a vote, it should be reflected on result To submit multiple votes, use either a different browser, or use incognito window.","title":"Exercise - Deploy Complete Instavote App"},{"location":"kubernetes_quickdive/#cleaning-up","text":"Once you are done observing, you could delete it with the following commands, kubectl delete deploy vote redis worker db result kubectl delete service vote redis db result","title":"Cleaning up"},{"location":"kubernetes_quickdive/#summary","text":"When you deploy an application in kubernetes, you submit it to the api server/ cluster manager. Kubernetes automatically schedules it on a cluster, networks the pods, provides service discovery. In addition as you observed, your application is scalable, high available and is already running behind a load balancer.","title":"Summary"},{"location":"kubespray-prereqs/","text":"Provisioning HA Lab Cluster with Vagrant Vagrant Setup: This tutorial assumes you have Vagrant+VirtualBox setup. While Vagrant is used for basic infrastructure requirements, the lessons learned in this tutorial can be applied to other platforms. Start from Set up Kubernetes Using Kubespray (or) Refer to this Document , if you have VMs running elsewhere Software Requirements on Host Machine: Virtual Box (latest) Vagrant (latest) Git Bash (Only for Windows) Conemu (Only for Windows) Set up Learning Environment: Setup the repo git clone https://github.com/schoolofdevops/kubespray-1 Bring up the VMs cd kubespray-1 vagrant up vagrant status Login to nodes Open four different terminals to login to 4 nodes created with above command Terminal 1 vagrant ssh k8s-01 sudo su Terminal 2 vagrant ssh k8s-02 sudo su Terminal 3 vagrant ssh k8s-03 sudo su Terminal 4 vagrant ssh k8s-04 sudo su Once the environment is setup, follow Production Grade Setup with Kubespray","title":"Provisioning HA Lab Cluster with Vagrant"},{"location":"kubespray-prereqs/#provisioning-ha-lab-cluster-with-vagrant","text":"","title":"Provisioning HA Lab Cluster with Vagrant"},{"location":"kubespray-prereqs/#vagrant-setup","text":"This tutorial assumes you have Vagrant+VirtualBox setup. While Vagrant is used for basic infrastructure requirements, the lessons learned in this tutorial can be applied to other platforms. Start from Set up Kubernetes Using Kubespray (or) Refer to this Document , if you have VMs running elsewhere","title":"Vagrant Setup:"},{"location":"kubespray-prereqs/#software-requirements-on-host-machine","text":"Virtual Box (latest) Vagrant (latest) Git Bash (Only for Windows) Conemu (Only for Windows)","title":"Software Requirements on Host Machine:"},{"location":"kubespray-prereqs/#set-up-learning-environment","text":"Setup the repo git clone https://github.com/schoolofdevops/kubespray-1 Bring up the VMs cd kubespray-1 vagrant up vagrant status Login to nodes Open four different terminals to login to 4 nodes created with above command Terminal 1 vagrant ssh k8s-01 sudo su Terminal 2 vagrant ssh k8s-02 sudo su Terminal 3 vagrant ssh k8s-03 sudo su Terminal 4 vagrant ssh k8s-04 sudo su Once the environment is setup, follow Production Grade Setup with Kubespray","title":"Set up Learning Environment:"},{"location":"logging/","text":"Logging References Logging Arcchitecture","title":"Logging"},{"location":"logging/#logging","text":"","title":"Logging"},{"location":"logging/#references","text":"Logging Arcchitecture","title":"References"},{"location":"minikube/","text":"Single node k8s cluster with Minikube Minikube offers one of the easiest zero to dev experience to setup a single node kubernetes cluster. Its also the ideal way to create a local dev environment to test kubernetes code on. This document explains how to setup and work with single node kubernetes cluster with minikube. Install Minikube Instructions to install minikube may vary based on the operating system and choice of the hypervisor. This is the official document which explains how to install minikube. Start all in one single node cluster with minikube minikube status [output] minikube: cluster: kubectl: minikube start [output] Starting local Kubernetes v1.8.0 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components... Kubectl is now configured to use the cluster. Loading cached images from config file. minikube status [output] minikube: Running cluster: Running kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100 Launch a kubernetes dashboard minikube dashboard Setting up docker environment minikube docker-env export DOCKER_TLS_VERIFY=\"1\" export DOCKER_HOST=\"tcp://192.168.99.100:2376\" export DOCKER_CERT_PATH=\"/Users/gouravshah/.minikube/certs\" export DOCKER_API_VERSION=\"1.23\" # Run this command to configure your shell: # eval $(minikube docker-env) Run the command given above, e.g. eval $(minikube docker-env) Now your docker client should be able to connect with the minikube cluster docker ps Additional Commands minikube ip minikube get-k8s-versions minikube logs","title":"Single node k8s cluster with Minikube"},{"location":"minikube/#single-node-k8s-cluster-with-minikube","text":"Minikube offers one of the easiest zero to dev experience to setup a single node kubernetes cluster. Its also the ideal way to create a local dev environment to test kubernetes code on. This document explains how to setup and work with single node kubernetes cluster with minikube.","title":"Single node k8s cluster with Minikube"},{"location":"minikube/#install-minikube","text":"Instructions to install minikube may vary based on the operating system and choice of the hypervisor. This is the official document which explains how to install minikube.","title":"Install Minikube"},{"location":"minikube/#start-all-in-one-single-node-cluster-with-minikube","text":"minikube status [output] minikube: cluster: kubectl: minikube start [output] Starting local Kubernetes v1.8.0 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Starting cluster components... Kubectl is now configured to use the cluster. Loading cached images from config file. minikube status [output] minikube: Running cluster: Running kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100","title":"Start all in one single node cluster with minikube"},{"location":"minikube/#launch-a-kubernetes-dashboard","text":"minikube dashboard","title":"Launch a kubernetes dashboard"},{"location":"minikube/#setting-up-docker-environment","text":"minikube docker-env export DOCKER_TLS_VERIFY=\"1\" export DOCKER_HOST=\"tcp://192.168.99.100:2376\" export DOCKER_CERT_PATH=\"/Users/gouravshah/.minikube/certs\" export DOCKER_API_VERSION=\"1.23\" # Run this command to configure your shell: # eval $(minikube docker-env) Run the command given above, e.g. eval $(minikube docker-env) Now your docker client should be able to connect with the minikube cluster docker ps","title":"Setting up docker environment"},{"location":"minikube/#additional-commands","text":"minikube ip minikube get-k8s-versions minikube logs","title":"Additional Commands"},{"location":"network_policies/","text":"Enforcing Network Policies If you are using KIND based environment, you would have to recreate the cluster to support network policies as the kind-net does not support this feature. Use the following instructions to recreate the cluster with calico and then proceed to create the network policies. Recreate KIND Cluster with Calico For Network Policies to work, you need to have a CNI Plugin which supports it. kind-net which is installed as a defulay CNI with KIND, does not support it. You could recreate the cluster with calico as a CNI plugin by following the instructions here. Delete existing cluster created with KIND as, kind get clusters kind delete cluster --name k8slab assuming k8slab is the name of the cluster. Change it with the actual name. File : k8s-code/helper/kind/kind-three-node-cluster.yaml disable the default network as # three node (two workers) cluster config kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true podSubnet: 192.168.31.0/16 launch the cluster again as cd k8s-code/helper/kind kind create cluster --config kind-three-node-cluster.yaml validate kubectl get nodes the nodes would be in NotReady stat at this time because of no CNI (Network) Plugin. Set up calico as kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/tigera-operator.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/custom-resources.yaml wait for calico pods to be ready watch kubectl get pods -l k8s-app=calico-node -A once the pods for calico are setup, exit from this watch command (use ^c) and validate the node status again as: kubectl get nodes kubectl get pods -A At this time, nodes should be up and running. That You may proceed to create any deployments, services needed at this time. Recreating the Application Deployment kubectl create namespace instavote kubectl config set-context --current --namespace=instavote validate you are switched to instavote namespace as kubectl config get-contexts assuming you have access to all the code to create instavote stack, apply it using command similar to follows kubectl apply -f vote-svc.yaml -f vote-deploy.yaml \\ -f redis-svc.yaml -f redis-deploy.yaml \\ -f db-svc.yaml -f db-deploy.yaml \\ -f worker-deploy.yaml -f results-svc.yaml \\ -f results-deploy.yaml validate you have 5 deployments and 4 services as kubectl get deploy,svc [sample output] NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/db 1/1 1 1 45m deployment.apps/redis 1/1 1 1 45m deployment.apps/result 1/1 1 1 45m deployment.apps/vote 1/1 1 1 45m deployment.apps/worker 1/1 1 1 45m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/db ClusterIP 10.96.180.62 5432/TCP 45m service/redis ClusterIP 10.96.197.185 6379/TCP 45m service/result NodePort 10.96.61.34 80:30100/TCP 45m service/vote NodePort 10.96.242.67 80:30000/TCP 45m Locking down access with a NetworkPolicy Now, define a restrictive network policy which would, Block all incoming connections Block all outgoing connections +-----------------------------------------------------------+ | | | +----------+ +-----------+ | x | | results | | db | | | | | | | | | +----------+ +-----------+ | | | | | | +----+----+--+ | | | worker | | | | | | | +----+-------+ | | | | | | +----------+ +-----------+ | | | vote | | redis | | x | | | | | | | +----------+ +-----------+ | | | +-----------------------------------------------------------+ file: instavote-netpol.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default namespace: instavote spec: podSelector: {} policyTypes: - Ingress - Egress apply kubectl get netpol kubectl apply -f instavote-netpol.yaml kubectl get netpol kubectl describe netpol/default Try accessing the vote and results ui. Can you access it ? Troubleshooting Tip : If you do not see the above policy being in effect (i.e. if you can still access the applications), go back and check if you have applied the label to the namespace as mentioned in the beginning of this section. Enabling external traffic to outward facing applications +-----------------------------------------------------------+ | | | +----------+ +-----------+ | =====> | results | | db | | | | | | | | | +----------+ +-----------+ | | | | | | +----+----+--+ | | | worker | | | | | | | +----+-------+ | | | | | | +----------+ +-----------+ | | | vote | | redis | | =====> | | | | | | +----------+ +-----------+ | | | +-----------------------------------------------------------+ To the same file, add a new network policy object. file: instavote-netpol.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default namespace: instavote spec: podSelector: {} policyTypes: - Ingress - Egress --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: public-ingress namespace: instavote spec: podSelector: matchExpressions: - {key: role, operator: In, values: [vote, result]} policyTypes: - Ingress ingress: - {} where, instavote-ingress is a new network policy which, defines policy for pods with vote and results role and allows them incoming access from anywhere apply kubectl apply -f instavote-netpol.yaml Exercise Try accessing the ui now and check if you are able to. Try to vote, see if that works? Why ? Enabling communication between pods in the same namespace When you tried to vote, you might have observed that it does not work. Thats because the default network policy we created earlier blocks all outgoing traffic. Which is good for securing the environment, however you still need to provide inter connection between services from the same project. Specifically vote , worker and results apps need outgoing connection to redis and db . Lets allow that with a egress policy. +-----------------------------------------------------------+ | | | +------------+ +-----------+ | =====> | results | ------>| db | | | | | | | <-------+ | | +------------+ +-----------+ | | | | | | | | | +----+----+---+ | | | worker | | | | | | | +----+--------+ | | | | | | | | +----------+ +-----------+ | | | | vote | | redis | <-------+ | =====> | | ------> | | | | +----------+ +-----------+ | | | +-----------------------------------------------------------+ Edit the same policy file and add the following snippet, file: instavote-netpol.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default namespace: instavote spec: podSelector: {} policyTypes: - Ingress - Egress ingress: - from: - podSelector: {} # Allows all pods within the same namespace egress: - to: - podSelector: {} # Allows all pods within the same namespace --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: public-ingress namespace: instavote spec: podSelector: matchExpressions: - {key: role, operator: In, values: [vote, result]} policyTypes: - Ingress ingress: - {} where, instavote-egress is a new network policy which, defines policy for pods with vote , worker and results role and allows them outgoing access to any pods in the same namespace, and that includes redis and db Troubleshooting Exercise Applying the above policy has no effect on the communication between vote and redis applications. You could validate this by loading the vote app and submit a vote. It should not work. There is a problem in the network policy file above. Analyse the policies, compare them against the kubernetes api reference document, understand how its being applied and see if you could fix this problem. Your task is to ensure that vote and redis apps are communicating with one another. Solution The reason why communication between vote and redis is broken is because of the name resoulution (DNS Based Service Discovery) is broken. This is because the network policies that you have set up do not allow the services in instavote namespace to communicate to even the DNS server in the cluster running in kube-system namespace. You could allow this by adding one more policiy. You could add it to the same file instavote-netpol.yaml e.g. --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: dns-access namespace: instavote spec: podSelector: {} # Applies to all pods in the namespace policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 now apply and validate kubectl apply -f instavote-netpol.yaml Now you should see vote app connecting with redis . Nano Project The above network policies are a good start. However you could even further restrict access by creating a granular network policy for each application. Create network policies with following specs, vote allow incoming connections from anywhere, only on port 80 allow outgoing connections to redis block everything else, incoming and outgoing redis allow incoming connections from vote and worker , only on port 6379 block everything else, incoming and outgoing worker allow outgoing connections to redis and db block everything else, incoming and outgoing db allow incoming connections from worker and results , only on port 5342 block everything else, incoming and outgoing result allow incoming connections from anywhere, only on port 80 allow outgoing connections to db block everything else, incoming and outgoing","title":"Lab K207 - Network Policies"},{"location":"network_policies/#enforcing-network-policies","text":"If you are using KIND based environment, you would have to recreate the cluster to support network policies as the kind-net does not support this feature. Use the following instructions to recreate the cluster with calico and then proceed to create the network policies.","title":"Enforcing Network Policies"},{"location":"network_policies/#recreate-kind-cluster-with-calico","text":"For Network Policies to work, you need to have a CNI Plugin which supports it. kind-net which is installed as a defulay CNI with KIND, does not support it. You could recreate the cluster with calico as a CNI plugin by following the instructions here. Delete existing cluster created with KIND as, kind get clusters kind delete cluster --name k8slab assuming k8slab is the name of the cluster. Change it with the actual name. File : k8s-code/helper/kind/kind-three-node-cluster.yaml disable the default network as # three node (two workers) cluster config kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true podSubnet: 192.168.31.0/16 launch the cluster again as cd k8s-code/helper/kind kind create cluster --config kind-three-node-cluster.yaml validate kubectl get nodes the nodes would be in NotReady stat at this time because of no CNI (Network) Plugin. Set up calico as kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/tigera-operator.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.3/manifests/custom-resources.yaml wait for calico pods to be ready watch kubectl get pods -l k8s-app=calico-node -A once the pods for calico are setup, exit from this watch command (use ^c) and validate the node status again as: kubectl get nodes kubectl get pods -A At this time, nodes should be up and running. That You may proceed to create any deployments, services needed at this time.","title":"Recreate KIND Cluster with Calico"},{"location":"network_policies/#recreating-the-application-deployment","text":"kubectl create namespace instavote kubectl config set-context --current --namespace=instavote validate you are switched to instavote namespace as kubectl config get-contexts assuming you have access to all the code to create instavote stack, apply it using command similar to follows kubectl apply -f vote-svc.yaml -f vote-deploy.yaml \\ -f redis-svc.yaml -f redis-deploy.yaml \\ -f db-svc.yaml -f db-deploy.yaml \\ -f worker-deploy.yaml -f results-svc.yaml \\ -f results-deploy.yaml validate you have 5 deployments and 4 services as kubectl get deploy,svc [sample output] NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/db 1/1 1 1 45m deployment.apps/redis 1/1 1 1 45m deployment.apps/result 1/1 1 1 45m deployment.apps/vote 1/1 1 1 45m deployment.apps/worker 1/1 1 1 45m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/db ClusterIP 10.96.180.62 5432/TCP 45m service/redis ClusterIP 10.96.197.185 6379/TCP 45m service/result NodePort 10.96.61.34 80:30100/TCP 45m service/vote NodePort 10.96.242.67 80:30000/TCP 45m","title":"Recreating the Application Deployment"},{"location":"network_policies/#locking-down-access-with-a-networkpolicy","text":"Now, define a restrictive network policy which would, Block all incoming connections Block all outgoing connections +-----------------------------------------------------------+ | | | +----------+ +-----------+ | x | | results | | db | | | | | | | | | +----------+ +-----------+ | | | | | | +----+----+--+ | | | worker | | | | | | | +----+-------+ | | | | | | +----------+ +-----------+ | | | vote | | redis | | x | | | | | | | +----------+ +-----------+ | | | +-----------------------------------------------------------+ file: instavote-netpol.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default namespace: instavote spec: podSelector: {} policyTypes: - Ingress - Egress apply kubectl get netpol kubectl apply -f instavote-netpol.yaml kubectl get netpol kubectl describe netpol/default Try accessing the vote and results ui. Can you access it ? Troubleshooting Tip : If you do not see the above policy being in effect (i.e. if you can still access the applications), go back and check if you have applied the label to the namespace as mentioned in the beginning of this section.","title":"Locking down access with a NetworkPolicy"},{"location":"network_policies/#enabling-external-traffic-to-outward-facing-applications","text":"+-----------------------------------------------------------+ | | | +----------+ +-----------+ | =====> | results | | db | | | | | | | | | +----------+ +-----------+ | | | | | | +----+----+--+ | | | worker | | | | | | | +----+-------+ | | | | | | +----------+ +-----------+ | | | vote | | redis | | =====> | | | | | | +----------+ +-----------+ | | | +-----------------------------------------------------------+ To the same file, add a new network policy object. file: instavote-netpol.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default namespace: instavote spec: podSelector: {} policyTypes: - Ingress - Egress --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: public-ingress namespace: instavote spec: podSelector: matchExpressions: - {key: role, operator: In, values: [vote, result]} policyTypes: - Ingress ingress: - {} where, instavote-ingress is a new network policy which, defines policy for pods with vote and results role and allows them incoming access from anywhere apply kubectl apply -f instavote-netpol.yaml Exercise Try accessing the ui now and check if you are able to. Try to vote, see if that works? Why ?","title":"Enabling external traffic to outward facing applications"},{"location":"network_policies/#enabling-communication-between-pods-in-the-same-namespace","text":"When you tried to vote, you might have observed that it does not work. Thats because the default network policy we created earlier blocks all outgoing traffic. Which is good for securing the environment, however you still need to provide inter connection between services from the same project. Specifically vote , worker and results apps need outgoing connection to redis and db . Lets allow that with a egress policy. +-----------------------------------------------------------+ | | | +------------+ +-----------+ | =====> | results | ------>| db | | | | | | | <-------+ | | +------------+ +-----------+ | | | | | | | | | +----+----+---+ | | | worker | | | | | | | +----+--------+ | | | | | | | | +----------+ +-----------+ | | | | vote | | redis | <-------+ | =====> | | ------> | | | | +----------+ +-----------+ | | | +-----------------------------------------------------------+ Edit the same policy file and add the following snippet, file: instavote-netpol.yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default namespace: instavote spec: podSelector: {} policyTypes: - Ingress - Egress ingress: - from: - podSelector: {} # Allows all pods within the same namespace egress: - to: - podSelector: {} # Allows all pods within the same namespace --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: public-ingress namespace: instavote spec: podSelector: matchExpressions: - {key: role, operator: In, values: [vote, result]} policyTypes: - Ingress ingress: - {} where, instavote-egress is a new network policy which, defines policy for pods with vote , worker and results role and allows them outgoing access to any pods in the same namespace, and that includes redis and db","title":"Enabling communication between pods in the same namespace"},{"location":"network_policies/#troubleshooting-exercise","text":"Applying the above policy has no effect on the communication between vote and redis applications. You could validate this by loading the vote app and submit a vote. It should not work. There is a problem in the network policy file above. Analyse the policies, compare them against the kubernetes api reference document, understand how its being applied and see if you could fix this problem. Your task is to ensure that vote and redis apps are communicating with one another.","title":"Troubleshooting Exercise"},{"location":"network_policies/#solution","text":"The reason why communication between vote and redis is broken is because of the name resoulution (DNS Based Service Discovery) is broken. This is because the network policies that you have set up do not allow the services in instavote namespace to communicate to even the DNS server in the cluster running in kube-system namespace. You could allow this by adding one more policiy. You could add it to the same file instavote-netpol.yaml e.g. --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: dns-access namespace: instavote spec: podSelector: {} # Applies to all pods in the namespace policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 now apply and validate kubectl apply -f instavote-netpol.yaml Now you should see vote app connecting with redis .","title":"Solution"},{"location":"network_policies/#nano-project","text":"The above network policies are a good start. However you could even further restrict access by creating a granular network policy for each application. Create network policies with following specs, vote allow incoming connections from anywhere, only on port 80 allow outgoing connections to redis block everything else, incoming and outgoing redis allow incoming connections from vote and worker , only on port 6379 block everything else, incoming and outgoing worker allow outgoing connections to redis and db block everything else, incoming and outgoing db allow incoming connections from worker and results , only on port 5342 block everything else, incoming and outgoing result allow incoming connections from anywhere, only on port 80 allow outgoing connections to db block everything else, incoming and outgoing","title":"Nano Project"},{"location":"operating-docker-containers/","text":"Getting Started with Docker Operations In this chapter, we are going to learn about docker shell, the command line utility and how to use it to launch containers. We will also learn what it means to run a container, its lifecycle and perform basic operations such as creating, starting, stopping, removing, pausing containers and checking the status etc. Using docker cli We can use docker cli to interact with docker daemon. Various functions of docker command is given below. Try this yourself by runnig $sudo docker command docker [Output] Usage: docker [OPTIONS] COMMAND [arg...] docker [ --help | -v | --version ] A self-sufficient runtime for containers. Options: --config=~/.docker Location of client config files -D, --debug Enable debug mode -H, --host=[] Daemon socket(s) to connect to -h, --help Print usage -l, --log-level=info Set the logging level --tls Use TLS; implied by --tlsverify --tlscacert=~/.docker/ca.pem Trust certs signed only by this CA --tlscert=~/.docker/cert.pem Path to TLS certificate file --tlskey=~/.docker/key.pem Path to TLS key file --tlsverify Use TLS and verify the remote -v, --version Print version information and quit Commands: attach Attach to a running container build Build an image from a Dockerfile commit Create a new image from a container's changes cp Copy files/folders between a container and the local filesystem create Create a new container diff Inspect changes on a container's filesystem events Get real time events from the server exec Run a command in a running container export Export a container's filesystem as a tar archive history Show the history of an image images List images import Import the contents from a tarball to create a filesystem image info Display system-wide information inspect Return low-level information on a container, image or task kill Kill one or more running containers load Load an image from a tar archive or STDIN login Log in to a Docker registry. logout Log out from a Docker registry. logs Fetch the logs of a container network Manage Docker networks node Manage Docker Swarm nodes pause Pause all processes within one or more containers port List port mappings or a specific mapping for the container ps List containers pull Pull an image or a repository from a registry push Push an image or a repository to a registry rename Rename a container restart Restart a container rm Remove one or more containers rmi Remove one or more images run Run a command in a new container save Save one or more images to a tar archive (streamed to STDOUT by default) search Search the Docker Hub for images service Manage Docker services start Start one or more stopped containers stats Display a live stream of container(s) resource usage statistics stop Stop one or more running containers swarm Manage Docker Swarm tag Tag an image into a repository top Display the running processes of a container unpause Unpause all processes within one or more containers update Update configuration of one or more containers version Show the Docker version information volume Manage Docker volumes wait Block until a container stops, then print its exit code Getting Information about Docker Setup We can get the information about our Docker setup in several ways. Namely, docker version docker -v docker system info The docker system info command gives a lot of useful information like total number of containers and images along with information about host resource utilization etc. Stream events from the docker daemon Docker events serves us with the stream of events or interactions that are happening with the docker daemon. This does not stream the log data of application inside the container. That is done by docker logs command. Let us see how this command works Open an another terminal. Let us call the old terminal as Terminal 1 and the newer one as Terminal 2 . From Terminal 1, execute docker events . Now you are getting the data stream from docker daemon docker system events Launching our first container Now we have a basic understanding of docker command and sub commands, let us dive straight into launching our very first container docker run alpine:3.4 uptime Where, we are using docker client to run a application/command uptime using an image by name alpine:3.4 [Output] Unable to find image 'alpine:3.4' locally 3.4: Pulling from library/alpine 81033e7c1d6a: Pull complete Digest: sha256:2532609239f3a96fbc30670716d87c8861b8a1974564211325633ca093b11c0b Status: Downloaded newer image for alpine:3.4 15:24:34 up 7:36, load average: 0.00, 0.03, 0.04 What happened? This command will Pull the alpine image file from docker hub , a cloud registry Create a runtime environment/ container with the above image Launch a program (called uptime ) inside that container Stream that output to the terminal Stop the container once the program is exited Where did my container go? docker container ps docker container ps -l The point here to remember is that, when that executable stops running inside the container, the container itself will stop. Let's see what happens when we run that command again, [Output] docker run alpine uptime 07:48:06 up 3:15, load average: 0.00, 0.00, 0.00 Now docker no longer pulls the image again from registry, because it has stored the image locally from the previous run. So once an image is pulled, we can make use of that image to create and run as many container as we want without the need of downloading the image again. However it has created a new instance of the iamge/container. Checking Status of the containers We have understood how docker run commands works. But what if you want to see list of running containers and history of containers that had run and exited? This can be done by executing the following commands docker ps [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES This command doesn't give us any information. Because, docker ps command will only show list of container(s) which are running docker ps -l [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 988f4d90d604 alpine \"uptime\" About a minute ago Exited (0) About a minute ago fervent_hypatia the -l flag shows the last run container along with other details like image it used, command it executed, return code of that command, etc., docker ps -n 2 [Output] NAMES 988f4d90d604 alpine \"uptime\" About a minute ago Exited (0) About a minute ago fervent_hypatia acea3023dca4 alpine \"uptime\" 3 minutes ago Exited (0) 3 minutes ago mad_darwin Docker gives us the flexibility to show the desirable number of last run containers. This can be achieved by using -n #no_of_results flag docker ps -a [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 988f4d90d604 alpine \"uptime\" About a minute ago Exited (0) About a minute ago fervent_hypatia acea3023dca4 alpine \"uptime\" 4 minutes ago Exited (0) 4 minutes ago mad_darwin 60ffa94e69ec ubuntu:14.04.3 \"bash\" 27 hours ago Exited (0) 26 hours ago infallible_meninsky dd75c04e7d2b schoolofdevops/ghost:0.3.1 \"/entrypoint.sh npm s\" 4 days ago Exited (0) 3 days ago kickass_bardeen c082972f66d6 schoolofdevops/ghost:0.3.1 \"/entrypoint.sh npm s\" 4 days ago Exited (0) 3 days ago 0.0.0.0:80->2368/tcp sodcblog This command will show all the container we have run so far. Running Containers in Interactive Mode We can interact with docker containers by giving -it flags at the run time. These flags stand for * i - Interactive * t - tty docker run -it alpine:3.4 sh [Output] Unable to find image 'alpine:3.4' locally latest: Pulling from library/alpine ff3a5c916c92: Already exists Digest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0 Status: Downloaded newer image for alpine:latest / # As you see, we have landed straight into sh shell of that container. This is the result of using -it flags and mentioning that container to run the sh shell. Don't try to exit that container yet. We have to execute some other commands in it to understand the next topic Namespaced: Like a full fledged OS, Docker container has its own namespaces This enables Docker container to isolate itself from the host as well as other containers Run the following commands and see that alpine container has its own namespaces and not inheriting much from host OS NAMESPACED [Run the following commands inside the container] cat /etc/issue ps aux ifconfig hostname Shared(NON NAMESPACED): We have understood that containers have their own namespaces. But will they share something to some extent? the answer is YES . Let's run the following commands on both the container and the host machine [Run the following commands inside the container] uptime uname -a cat /proc/cpuinfo date free As you can see, the container uses the same Linux Kernel from the host machine. Just like uname command, the following commands share the same information as well. In order to avoid repetition, we will see the output of container alone. Now exit out of that container by running exit or by pressing ctrl+d Making Containers Persist Running Containers in Detached Mode So far, we have run the containers interactively. But this is not always the case. Sometimes you may want to start a container without interacting with it. This can be achieved by using \"detached mode\" ( -d ) flag. Hence the container will launch the deafault application inside and run in the background. This saves a lot of time, we don't have to wait till the applications launches successfully. It will happen behind the screen. Let us run the following command to see this in action [Command] docker run -idt schoolofdevops/loop program -d , --detach : detached mode [Output] 2533adf280ac4838b446e4ee1151f52793e6ac499d2e631b2c752459bb18ad5f This will run the container in detached mode. We are only given with full container id as the output Let us check whether this container is running or not [Command] docker ps [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2533adf280ac schoolofdevops/loop \"program\" 37 seconds ago Up 36 seconds prickly_bose As we can see in the output, the container is running in the background Checking Logs To check the logs, find out the container id/name and run the following commands, replacing 08f0242aa61c with your container id [Commands] docker container ps docker container logs docker container logs -f Connecting to running container to execute commands We can connect to the containers which are running in detached mode by using these following commands [Command] docker exec -it sh [Output] / # You could try running any commands on the shell e.g. apk update apk add vim ps aux Now exit the container. Launching a container with a pre built app image To launch vote container run the following command. Don't bother about the new flag -P now. We will explain about that flag later in this chapter docker container run -idt -P schoolofdevops/vote [Output] Unable to find image 'schoolofdevops/vote:latest' locally latest: Pulling from schoolofdevops/vote Digest: sha256:9195942ea654fa8d8aeb37900be5619215c08e7e1bef0b7dfe4c04a9cc20a8c2 Status: Downloaded newer image for schoolofdevops/vote:latest 7d58ecc05754b5fd192c4ecceae334ac22565684c6923ea332bff5c88e3fca2b Lets check the status of the container docker ps -l [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7d58ecc05754 schoolofdevops/vote \"gunicorn app:app -b\u2026\" 27 seconds ago Up 26 seconds 0.0.0.0:32768->80/tcp peaceful_varahamihira Renaming the container We can rename the container by using following command docker rename 7d58ecc05754 vote [replace 7d58ecc05754 with the actual container id on your system ] We have changed container's automatically generated name to vote . This new name can be of your choice. The point to understand is this command takes two arguments. The Old_name followed by New_name Run docker ps command to check the effect of changes docker ps [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7d58ecc05754 schoolofdevops/vote \"gunicorn app:app -b\u2026\" 3 minutes ago Up 3 minutes 0.0.0.0:32768->80/tcp vote As you can see here, the container is renamed to vote . This makes referencing container in cli very much easier. Ready to vote ? Let's see what this vote application does by connecting to that application. For that we need, Host machine's IP Container's port which is mapped to a host's port Let's find out the port mapping of container to host. Docker provides subcommand called port which does this job docker port vote [Output] 80/tcp -> 0.0.0.0:32768 So whatever traffic the host gets in port 2368 will be mapped to container's port 32768 Let's connect to http://IP_ADDRESS:PORT to see the actual application Finding Everything about the running container This topic discusses about finding metadata of containers. These metadata include various parameters like, * State of the container * Mounts * Configuration * Network, etc., Inspecting Lets try this inspect subcommand in action docker inspect vote Data output by above command contains detailed descriptino of the container an its properties. is represented in JSON format which makes filtering these results easier. Copying files between container and client host We can copy files/directories form host to container and vice-versa Let us create a file on the host touch testfile To copy the testfile from host machine to ghsot contanier , try docker cp testfile vote:/opt This command will copy testfile to vote container's /opt directory and will not give any output. To verify the file has been copies or not, let us log into container by running, docker exec -it vote bash Change directory into /opt and list the files cd /opt ls [Output] testfile There you can see that file has been successfully copied. Now exit the container Now you may try to cp some files from the container to the host machine docker cp vote:/app . ls Checking the Stats Docker stats command returns a data stream of resource utilization used by containers. The flag --no-stream disables data stream and displays only first result docker stats --no-stream=true vote docker stats Controlling Resources Docker provides us the granularity to control each container's resource utilization . We have several commands in the inventory to achieve this Putting limits on Running Containers First, let us monitor the utilization docker stats You can see that Memory attribute has max as its value, which means unlimited usage of host's RAM. We can put a cap on that by using update command docker update --memory 400M --memory-swap -1 vote [Output] vote Let us check whether the change has taken effect or not with docker stats terminal docker stat As you can see, the memory utilization of the container is changed from xxxx (unlimited) to 400 mb Limiting Resources while launching new containers The following resources can be limited using the update command * CPU * Memory * Disk IO * Capabilities Open two terminals, lets call them T1, and T2 In T1, start monitoring the stats docker stats [Output] CONTAINER CPU % MEM USAGE / LIMIT MM % NET I/O BLOCK I/O PIDS b28efeef41f8 0.16% 190.1 MiB / 400 MiB 47.51% 1.296 kB / 648 B 86.02 kB / 45.06 kB 0 CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS b28efeef41f8 0.01% 190.1 MiB / 400 MiB 47.51% 1.296 kB / 648 B 86.02 kB / 45.06 kB 0 From T2, launch containers with different CPU shares as well as cpus configurations. Default CPU shares are set to 1024. This is a relative weight. Observe docker stats command after every launch to see the effect of your configurations. [CPU Shares] docker run -d --cpu-shares 1024 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 1024 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 512 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 512 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 4096 schoolofdevops/stresstest stress --cpu 2 [CPUs] docker run -d --cpus 0.2 schoolofdevops/stresstest stress --cpu 2 Close the T2 terminal once you are done with this lab. Checking disk utilisation by docker system df [output] docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 7 5 1.031GB 914.5MB (88%) Containers 8 4 27.97MB 27.73MB (99%) Local Volumes 3 2 0B 0B Build Cache 0B 0B To prune, you could possibly use docker container prune docker system prune e.g. docker system prune WARNING! This will remove: - all stopped containers - all networks not used by at least one container - all dangling images - all build cache Are you sure you want to continue? [y/N] N Make sure you understand what all will be removed before using this command. Stopping and Removing Containers We have learnt about interacting with a container, running a container, pausing and unpausing a container, creating and starting a container. But what if you want to stop the container or remove the container itself Stop a container A container can be stopped using stop command. This command will stop the application inside that container hence the container itself will be stopped. This command basically sends a SIGTERM signal to the container (graceful shutdown) [Command] docker stop Kill a container This command will send SIGKILL signal and kills the container ungracefully [Command] docker kill If you want to remove a container, then execute the following command. Before running this command, run docker ps -a to see the list of pre run containers. Choose a container of your wish and then execute docker rm command. Then run docker ps -a again to check the removed container list or not [Command] docker rm docker rm -f Exercises Launching Containers with Elevated Privileges When the operator executes docker run --privileged, Docker will enable to access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host.","title":"Lab D101 - Operating Containers"},{"location":"operating-docker-containers/#getting-started-with-docker-operations","text":"In this chapter, we are going to learn about docker shell, the command line utility and how to use it to launch containers. We will also learn what it means to run a container, its lifecycle and perform basic operations such as creating, starting, stopping, removing, pausing containers and checking the status etc.","title":"Getting Started with Docker Operations"},{"location":"operating-docker-containers/#using-docker-cli","text":"We can use docker cli to interact with docker daemon. Various functions of docker command is given below. Try this yourself by runnig $sudo docker command docker [Output] Usage: docker [OPTIONS] COMMAND [arg...] docker [ --help | -v | --version ] A self-sufficient runtime for containers. Options: --config=~/.docker Location of client config files -D, --debug Enable debug mode -H, --host=[] Daemon socket(s) to connect to -h, --help Print usage -l, --log-level=info Set the logging level --tls Use TLS; implied by --tlsverify --tlscacert=~/.docker/ca.pem Trust certs signed only by this CA --tlscert=~/.docker/cert.pem Path to TLS certificate file --tlskey=~/.docker/key.pem Path to TLS key file --tlsverify Use TLS and verify the remote -v, --version Print version information and quit Commands: attach Attach to a running container build Build an image from a Dockerfile commit Create a new image from a container's changes cp Copy files/folders between a container and the local filesystem create Create a new container diff Inspect changes on a container's filesystem events Get real time events from the server exec Run a command in a running container export Export a container's filesystem as a tar archive history Show the history of an image images List images import Import the contents from a tarball to create a filesystem image info Display system-wide information inspect Return low-level information on a container, image or task kill Kill one or more running containers load Load an image from a tar archive or STDIN login Log in to a Docker registry. logout Log out from a Docker registry. logs Fetch the logs of a container network Manage Docker networks node Manage Docker Swarm nodes pause Pause all processes within one or more containers port List port mappings or a specific mapping for the container ps List containers pull Pull an image or a repository from a registry push Push an image or a repository to a registry rename Rename a container restart Restart a container rm Remove one or more containers rmi Remove one or more images run Run a command in a new container save Save one or more images to a tar archive (streamed to STDOUT by default) search Search the Docker Hub for images service Manage Docker services start Start one or more stopped containers stats Display a live stream of container(s) resource usage statistics stop Stop one or more running containers swarm Manage Docker Swarm tag Tag an image into a repository top Display the running processes of a container unpause Unpause all processes within one or more containers update Update configuration of one or more containers version Show the Docker version information volume Manage Docker volumes wait Block until a container stops, then print its exit code","title":"Using docker cli"},{"location":"operating-docker-containers/#getting-information-about-docker-setup","text":"We can get the information about our Docker setup in several ways. Namely, docker version docker -v docker system info The docker system info command gives a lot of useful information like total number of containers and images along with information about host resource utilization etc.","title":"Getting Information about Docker Setup"},{"location":"operating-docker-containers/#stream-events-from-the-docker-daemon","text":"Docker events serves us with the stream of events or interactions that are happening with the docker daemon. This does not stream the log data of application inside the container. That is done by docker logs command. Let us see how this command works Open an another terminal. Let us call the old terminal as Terminal 1 and the newer one as Terminal 2 . From Terminal 1, execute docker events . Now you are getting the data stream from docker daemon docker system events","title":"Stream events from the docker daemon"},{"location":"operating-docker-containers/#launching-our-first-container","text":"Now we have a basic understanding of docker command and sub commands, let us dive straight into launching our very first container docker run alpine:3.4 uptime Where, we are using docker client to run a application/command uptime using an image by name alpine:3.4 [Output] Unable to find image 'alpine:3.4' locally 3.4: Pulling from library/alpine 81033e7c1d6a: Pull complete Digest: sha256:2532609239f3a96fbc30670716d87c8861b8a1974564211325633ca093b11c0b Status: Downloaded newer image for alpine:3.4 15:24:34 up 7:36, load average: 0.00, 0.03, 0.04 What happened? This command will Pull the alpine image file from docker hub , a cloud registry Create a runtime environment/ container with the above image Launch a program (called uptime ) inside that container Stream that output to the terminal Stop the container once the program is exited Where did my container go? docker container ps docker container ps -l The point here to remember is that, when that executable stops running inside the container, the container itself will stop. Let's see what happens when we run that command again, [Output] docker run alpine uptime 07:48:06 up 3:15, load average: 0.00, 0.00, 0.00 Now docker no longer pulls the image again from registry, because it has stored the image locally from the previous run. So once an image is pulled, we can make use of that image to create and run as many container as we want without the need of downloading the image again. However it has created a new instance of the iamge/container.","title":"Launching our first container"},{"location":"operating-docker-containers/#checking-status-of-the-containers","text":"We have understood how docker run commands works. But what if you want to see list of running containers and history of containers that had run and exited? This can be done by executing the following commands docker ps [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES This command doesn't give us any information. Because, docker ps command will only show list of container(s) which are running docker ps -l [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 988f4d90d604 alpine \"uptime\" About a minute ago Exited (0) About a minute ago fervent_hypatia the -l flag shows the last run container along with other details like image it used, command it executed, return code of that command, etc., docker ps -n 2 [Output] NAMES 988f4d90d604 alpine \"uptime\" About a minute ago Exited (0) About a minute ago fervent_hypatia acea3023dca4 alpine \"uptime\" 3 minutes ago Exited (0) 3 minutes ago mad_darwin Docker gives us the flexibility to show the desirable number of last run containers. This can be achieved by using -n #no_of_results flag docker ps -a [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 988f4d90d604 alpine \"uptime\" About a minute ago Exited (0) About a minute ago fervent_hypatia acea3023dca4 alpine \"uptime\" 4 minutes ago Exited (0) 4 minutes ago mad_darwin 60ffa94e69ec ubuntu:14.04.3 \"bash\" 27 hours ago Exited (0) 26 hours ago infallible_meninsky dd75c04e7d2b schoolofdevops/ghost:0.3.1 \"/entrypoint.sh npm s\" 4 days ago Exited (0) 3 days ago kickass_bardeen c082972f66d6 schoolofdevops/ghost:0.3.1 \"/entrypoint.sh npm s\" 4 days ago Exited (0) 3 days ago 0.0.0.0:80->2368/tcp sodcblog This command will show all the container we have run so far.","title":"Checking Status of the containers"},{"location":"operating-docker-containers/#running-containers-in-interactive-mode","text":"We can interact with docker containers by giving -it flags at the run time. These flags stand for * i - Interactive * t - tty docker run -it alpine:3.4 sh [Output] Unable to find image 'alpine:3.4' locally latest: Pulling from library/alpine ff3a5c916c92: Already exists Digest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0 Status: Downloaded newer image for alpine:latest / # As you see, we have landed straight into sh shell of that container. This is the result of using -it flags and mentioning that container to run the sh shell. Don't try to exit that container yet. We have to execute some other commands in it to understand the next topic Namespaced: Like a full fledged OS, Docker container has its own namespaces This enables Docker container to isolate itself from the host as well as other containers Run the following commands and see that alpine container has its own namespaces and not inheriting much from host OS NAMESPACED [Run the following commands inside the container] cat /etc/issue ps aux ifconfig hostname Shared(NON NAMESPACED): We have understood that containers have their own namespaces. But will they share something to some extent? the answer is YES . Let's run the following commands on both the container and the host machine [Run the following commands inside the container] uptime uname -a cat /proc/cpuinfo date free As you can see, the container uses the same Linux Kernel from the host machine. Just like uname command, the following commands share the same information as well. In order to avoid repetition, we will see the output of container alone. Now exit out of that container by running exit or by pressing ctrl+d","title":"Running Containers in Interactive Mode"},{"location":"operating-docker-containers/#making-containers-persist","text":"","title":"Making Containers Persist"},{"location":"operating-docker-containers/#running-containers-in-detached-mode","text":"So far, we have run the containers interactively. But this is not always the case. Sometimes you may want to start a container without interacting with it. This can be achieved by using \"detached mode\" ( -d ) flag. Hence the container will launch the deafault application inside and run in the background. This saves a lot of time, we don't have to wait till the applications launches successfully. It will happen behind the screen. Let us run the following command to see this in action [Command] docker run -idt schoolofdevops/loop program -d , --detach : detached mode [Output] 2533adf280ac4838b446e4ee1151f52793e6ac499d2e631b2c752459bb18ad5f This will run the container in detached mode. We are only given with full container id as the output Let us check whether this container is running or not [Command] docker ps [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2533adf280ac schoolofdevops/loop \"program\" 37 seconds ago Up 36 seconds prickly_bose As we can see in the output, the container is running in the background","title":"Running Containers in Detached Mode"},{"location":"operating-docker-containers/#checking-logs","text":"To check the logs, find out the container id/name and run the following commands, replacing 08f0242aa61c with your container id [Commands] docker container ps docker container logs docker container logs -f ","title":"Checking Logs"},{"location":"operating-docker-containers/#connecting-to-running-container-to-execute-commands","text":"We can connect to the containers which are running in detached mode by using these following commands [Command] docker exec -it sh [Output] / # You could try running any commands on the shell e.g. apk update apk add vim ps aux Now exit the container.","title":"Connecting to running container to execute commands"},{"location":"operating-docker-containers/#launching-a-container-with-a-pre-built-app-image","text":"To launch vote container run the following command. Don't bother about the new flag -P now. We will explain about that flag later in this chapter docker container run -idt -P schoolofdevops/vote [Output] Unable to find image 'schoolofdevops/vote:latest' locally latest: Pulling from schoolofdevops/vote Digest: sha256:9195942ea654fa8d8aeb37900be5619215c08e7e1bef0b7dfe4c04a9cc20a8c2 Status: Downloaded newer image for schoolofdevops/vote:latest 7d58ecc05754b5fd192c4ecceae334ac22565684c6923ea332bff5c88e3fca2b Lets check the status of the container docker ps -l [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7d58ecc05754 schoolofdevops/vote \"gunicorn app:app -b\u2026\" 27 seconds ago Up 26 seconds 0.0.0.0:32768->80/tcp peaceful_varahamihira","title":"Launching a container with a pre built app image"},{"location":"operating-docker-containers/#renaming-the-container","text":"We can rename the container by using following command docker rename 7d58ecc05754 vote [replace 7d58ecc05754 with the actual container id on your system ] We have changed container's automatically generated name to vote . This new name can be of your choice. The point to understand is this command takes two arguments. The Old_name followed by New_name Run docker ps command to check the effect of changes docker ps [Output] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7d58ecc05754 schoolofdevops/vote \"gunicorn app:app -b\u2026\" 3 minutes ago Up 3 minutes 0.0.0.0:32768->80/tcp vote As you can see here, the container is renamed to vote . This makes referencing container in cli very much easier.","title":"Renaming the container"},{"location":"operating-docker-containers/#ready-to-vote","text":"Let's see what this vote application does by connecting to that application. For that we need, Host machine's IP Container's port which is mapped to a host's port Let's find out the port mapping of container to host. Docker provides subcommand called port which does this job docker port vote [Output] 80/tcp -> 0.0.0.0:32768 So whatever traffic the host gets in port 2368 will be mapped to container's port 32768 Let's connect to http://IP_ADDRESS:PORT to see the actual application","title":"Ready to vote ?"},{"location":"operating-docker-containers/#finding-everything-about-the-running-container","text":"This topic discusses about finding metadata of containers. These metadata include various parameters like, * State of the container * Mounts * Configuration * Network, etc.,","title":"Finding Everything about the running container"},{"location":"operating-docker-containers/#inspecting","text":"Lets try this inspect subcommand in action docker inspect vote Data output by above command contains detailed descriptino of the container an its properties. is represented in JSON format which makes filtering these results easier.","title":"Inspecting"},{"location":"operating-docker-containers/#copying-files-between-container-and-client-host","text":"We can copy files/directories form host to container and vice-versa Let us create a file on the host touch testfile To copy the testfile from host machine to ghsot contanier , try docker cp testfile vote:/opt This command will copy testfile to vote container's /opt directory and will not give any output. To verify the file has been copies or not, let us log into container by running, docker exec -it vote bash Change directory into /opt and list the files cd /opt ls [Output] testfile There you can see that file has been successfully copied. Now exit the container Now you may try to cp some files from the container to the host machine docker cp vote:/app . ls","title":"Copying files between container and client host"},{"location":"operating-docker-containers/#checking-the-stats","text":"Docker stats command returns a data stream of resource utilization used by containers. The flag --no-stream disables data stream and displays only first result docker stats --no-stream=true vote docker stats","title":"Checking the Stats"},{"location":"operating-docker-containers/#controlling-resources","text":"Docker provides us the granularity to control each container's resource utilization . We have several commands in the inventory to achieve this","title":"Controlling Resources"},{"location":"operating-docker-containers/#putting-limits-on-running-containers","text":"First, let us monitor the utilization docker stats You can see that Memory attribute has max as its value, which means unlimited usage of host's RAM. We can put a cap on that by using update command docker update --memory 400M --memory-swap -1 vote [Output] vote Let us check whether the change has taken effect or not with docker stats terminal docker stat As you can see, the memory utilization of the container is changed from xxxx (unlimited) to 400 mb","title":"Putting limits on Running Containers"},{"location":"operating-docker-containers/#limiting-resources-while-launching-new-containers","text":"The following resources can be limited using the update command * CPU * Memory * Disk IO * Capabilities Open two terminals, lets call them T1, and T2 In T1, start monitoring the stats docker stats [Output] CONTAINER CPU % MEM USAGE / LIMIT MM % NET I/O BLOCK I/O PIDS b28efeef41f8 0.16% 190.1 MiB / 400 MiB 47.51% 1.296 kB / 648 B 86.02 kB / 45.06 kB 0 CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS b28efeef41f8 0.01% 190.1 MiB / 400 MiB 47.51% 1.296 kB / 648 B 86.02 kB / 45.06 kB 0 From T2, launch containers with different CPU shares as well as cpus configurations. Default CPU shares are set to 1024. This is a relative weight. Observe docker stats command after every launch to see the effect of your configurations. [CPU Shares] docker run -d --cpu-shares 1024 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 1024 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 512 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 512 schoolofdevops/stresstest stress --cpu 2 docker run -d --cpu-shares 4096 schoolofdevops/stresstest stress --cpu 2 [CPUs] docker run -d --cpus 0.2 schoolofdevops/stresstest stress --cpu 2 Close the T2 terminal once you are done with this lab.","title":"Limiting Resources while launching new containers"},{"location":"operating-docker-containers/#checking-disk-utilisation-by","text":"docker system df [output] docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 7 5 1.031GB 914.5MB (88%) Containers 8 4 27.97MB 27.73MB (99%) Local Volumes 3 2 0B 0B Build Cache 0B 0B To prune, you could possibly use docker container prune docker system prune e.g. docker system prune WARNING! This will remove: - all stopped containers - all networks not used by at least one container - all dangling images - all build cache Are you sure you want to continue? [y/N] N Make sure you understand what all will be removed before using this command.","title":"Checking disk utilisation by"},{"location":"operating-docker-containers/#stopping-and-removing-containers","text":"We have learnt about interacting with a container, running a container, pausing and unpausing a container, creating and starting a container. But what if you want to stop the container or remove the container itself","title":"Stopping and Removing Containers"},{"location":"operating-docker-containers/#stop-a-container","text":"A container can be stopped using stop command. This command will stop the application inside that container hence the container itself will be stopped. This command basically sends a SIGTERM signal to the container (graceful shutdown) [Command] docker stop ","title":"Stop a container"},{"location":"operating-docker-containers/#kill-a-container","text":"This command will send SIGKILL signal and kills the container ungracefully [Command] docker kill If you want to remove a container, then execute the following command. Before running this command, run docker ps -a to see the list of pre run containers. Choose a container of your wish and then execute docker rm command. Then run docker ps -a again to check the removed container list or not [Command] docker rm docker rm -f ","title":"Kill a container"},{"location":"operating-docker-containers/#exercises","text":"","title":"Exercises"},{"location":"operating-docker-containers/#launching-containers-with-elevated-privileges","text":"When the operator executes docker run --privileged, Docker will enable to access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host.","title":"Launching Containers with Elevated Privileges"},{"location":"operator/","text":"Redis Operator Examine the code at schoolofdevops/redis-operator: Kubernetes Operator to Setup Redis Cluster Clone the repo git clone https://github.com/schoolofdevops/redis-operator.git cd redis-operator Apply the opertor code kubectl apply -f redis-crd.yaml -f redis_operator-rbac.yaml -f redis_operator-deploy.yaml kubectl get all now create an instance of redis cluster File : my-redis.yaml apiVersion: database.example.com/v1 kind: Redis metadata: name: my-redis namespace: default spec: slaveSize: 2 # Specifies the number of slave nodes image: \"redis:6.2.6\" # Redis Docker image version storageClassName: \"standard\" # Specify the Kubernetes storage class for PVCs volumeSize: \"200Mi\" # Each Redis node will use a PVC of this size backupEnabled: true # Enables the backup functionality backupSchedule: \"0 */6 * * *\" # Backup schedule, every 6 hours apply kubectl apply -f my-redis.yaml examine the objects created kubectl get all kubectl delete -f redis-crd.yaml -f redis_operator-rbac.yaml -f redis_operator-deploy.yaml Cleaning Up Once done, to clean up, kubectl delete -f my-redis.yaml","title":"Lab K404 - Kubernetes Operators"},{"location":"operator/#redis-operator","text":"Examine the code at schoolofdevops/redis-operator: Kubernetes Operator to Setup Redis Cluster Clone the repo git clone https://github.com/schoolofdevops/redis-operator.git cd redis-operator Apply the opertor code kubectl apply -f redis-crd.yaml -f redis_operator-rbac.yaml -f redis_operator-deploy.yaml kubectl get all now create an instance of redis cluster File : my-redis.yaml apiVersion: database.example.com/v1 kind: Redis metadata: name: my-redis namespace: default spec: slaveSize: 2 # Specifies the number of slave nodes image: \"redis:6.2.6\" # Redis Docker image version storageClassName: \"standard\" # Specify the Kubernetes storage class for PVCs volumeSize: \"200Mi\" # Each Redis node will use a PVC of this size backupEnabled: true # Enables the backup functionality backupSchedule: \"0 */6 * * *\" # Backup schedule, every 6 hours apply kubectl apply -f my-redis.yaml examine the objects created kubectl get all kubectl delete -f redis-crd.yaml -f redis_operator-rbac.yaml -f redis_operator-deploy.yaml","title":"Redis Operator"},{"location":"operator/#cleaning-up","text":"Once done, to clean up, kubectl delete -f my-redis.yaml","title":"Cleaning Up"},{"location":"pod-adv-specs/","text":"Additional Pod Specs - Resources, Security Specs Resource requests and limits We can control the amount of resource requested and used by all the pods. This can be done by adding following data the deployment template. Resource Request File: code/frontend-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: replicas: 1 template: metadata: labels: app: front-end env: dev spec: containers: - name: front-end image: schoolofdevops/frontend imagePullPolicy: Always ports: - containerPort: 8079 livenessProbe: tcpSocket: port: 8079 initialDelaySeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: / port: 8079 initialDelaySeconds: 5 periodSeconds: 3 resources: requests: memory: \"128Mi\" cpu: \"250m\" This ensures that pod always get the minimum cpu and memory specified. But this does not restrict the pod from accessing additional resources if needed. Thats why we have to use resource limit to limit the resource usage by a pod. Expected output: kubectl describe pod front-end-5c64b7c5cc-cwgr5 [...] Containers: front-end: Container ID: Image: schoolofdevops/frontend Image ID: Port: 8079/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Requests: cpu: 250m memory: 128Mi Resource limit File: code/frontend-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: replicas: 1 template: metadata: labels: app: front-end env: dev spec: containers: - name: front-end image: schoolofdevops/frontend imagePullPolicy: Always ports: - containerPort: 8079 livenessProbe: tcpSocket: port: 8079 initialDelaySeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: / port: 8079 initialDelaySeconds: 5 periodSeconds: 3 resources: requests: memory: \"128Mi\" cpu: \"250m\" limits: memory: \"256Mi\" cpu: \"500m\" Expected output: kubectl describe pod front-end-5b877b4dff-5twdd [...] Containers: front-end: Container ID: docker://d49a08c18fd9651af2f3dd28772da977b238a4010f14372e72e0ca24dcec8554 Image: schoolofdevops/frontend Image ID: docker-pullable://schoolofdevops/frontend@sha256:94b7a0843f99223a8a1d284fdeeb3fd5a731c03aea57a52751c6ebde40be1f50 Port: 8079/TCP State: Running Started: Thu, 08 Feb 2018 17:14:54 +0530 Ready: True Restart Count: 0 Limits: cpu: 500m memory: 256Mi Requests: cpu: 250m memory: 128Mi","title":"Additional Pod Specs - Resources, Security Specs"},{"location":"pod-adv-specs/#additional-pod-specs-resources-security-specs","text":"","title":"Additional Pod Specs - Resources, Security Specs"},{"location":"pod-adv-specs/#resource-requests-and-limits","text":"We can control the amount of resource requested and used by all the pods. This can be done by adding following data the deployment template.","title":"Resource requests and limits"},{"location":"pod-adv-specs/#resource-request","text":"File: code/frontend-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: replicas: 1 template: metadata: labels: app: front-end env: dev spec: containers: - name: front-end image: schoolofdevops/frontend imagePullPolicy: Always ports: - containerPort: 8079 livenessProbe: tcpSocket: port: 8079 initialDelaySeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: / port: 8079 initialDelaySeconds: 5 periodSeconds: 3 resources: requests: memory: \"128Mi\" cpu: \"250m\" This ensures that pod always get the minimum cpu and memory specified. But this does not restrict the pod from accessing additional resources if needed. Thats why we have to use resource limit to limit the resource usage by a pod. Expected output: kubectl describe pod front-end-5c64b7c5cc-cwgr5 [...] Containers: front-end: Container ID: Image: schoolofdevops/frontend Image ID: Port: 8079/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Requests: cpu: 250m memory: 128Mi","title":"Resource Request"},{"location":"pod-adv-specs/#resource-limit","text":"File: code/frontend-deploy.yml apiVersion: apps/v1beta1 kind: Deployment metadata: name: front-end namespace: instavote labels: app: front-end env: dev spec: replicas: 1 template: metadata: labels: app: front-end env: dev spec: containers: - name: front-end image: schoolofdevops/frontend imagePullPolicy: Always ports: - containerPort: 8079 livenessProbe: tcpSocket: port: 8079 initialDelaySeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: / port: 8079 initialDelaySeconds: 5 periodSeconds: 3 resources: requests: memory: \"128Mi\" cpu: \"250m\" limits: memory: \"256Mi\" cpu: \"500m\" Expected output: kubectl describe pod front-end-5b877b4dff-5twdd [...] Containers: front-end: Container ID: docker://d49a08c18fd9651af2f3dd28772da977b238a4010f14372e72e0ca24dcec8554 Image: schoolofdevops/frontend Image ID: docker-pullable://schoolofdevops/frontend@sha256:94b7a0843f99223a8a1d284fdeeb3fd5a731c03aea57a52751c6ebde40be1f50 Port: 8079/TCP State: Running Started: Thu, 08 Feb 2018 17:14:54 +0530 Ready: True Restart Count: 0 Limits: cpu: 500m memory: 256Mi Requests: cpu: 250m memory: 128Mi","title":"Resource limit"},{"location":"pod_security/","text":"Content to this chapter will be added in future. For a tutorial on this chapter, refer to the following page https://github.com/kubernetes/examples/tree/master/staging/podsecuritypolicy/rbac","title":"Pod security"},{"location":"pods-health-probes/","text":"Lab K204 - Adding health checks with Probes Adding health checks Health checks in Kubernetes work the same way as traditional health checks of applications. They make sure that our application is ready to receive and process user requests. In Kubernetes we have two types of health checks, * Liveness Probe * Readiness Probe Probes are simply a diagnostic action performed by the kubelet. There are three types actions a kubelet perfomes on a pod, which are namely, ExecAction : Executes a command inside the pod. Assumed successful when the command returns 0 as exit code. TCPSocketAction : Checks for a state of a particular port on the pod. Considered successful when the state of the port is open . HTTPGetAction : Performs a GET request on pod's IP. Assumed successful when the status code is greater than 200 and less than 400 In cases of any failure during the diagnostic action, kubelet will report back to the API server. Let us study about how these health checks work in practice. Adding Liveness/Readineess Probes Liveness probe checks the status of the pod(whether it is running or not). If livenessProbe fails, then the pod is subjected to its restart policy. The default state of livenessProbe is Success . Readiness probe checks whether your application is ready to serve the requests. When the readiness probe fails, the pod's IP is removed from the end point list of the service. The default state of readinessProbe is Success . Let us add liveness/readiness probes to our vote deployment. file: vote-deploy-probes.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 replicas: 12 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: In, values: [v1, v2, v3, v4, v5]} template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" livenessProbe: tcpSocket: port: 80 initialDelaySeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 where, livenessProbe used a simple tcp check to test whether application is listening on port 80 readinessProbe does httpGet to actually fetch a page using get method and tests for the http response code. Apply this code using, kubectl apply -f vote-deploy-probes.yaml kubectl get pods kubectl describe svc vote Testing livenessProbe kubectl edit deploy vote livenessProbe: failureThreshold: 3 initialDelaySeconds: 5 periodSeconds: 5 successThreshold: 1 tcpSocket: port: 8888 timeoutSeconds: 1 Since you are using edit command, as soon as you save the file, deployment is modified. kubectl get pods kubectl describe pod vote-xxxx where, vote-xxxx is one of the new pods created. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 38s default-scheduler Successfully assigned instavote/vote-668579766d-p65xb to k-02 Normal Pulled 18s (x2 over 36s) kubelet, k-02 Container image \"schoolofdevops/vote:v1\" already present on machine Normal Created 18s (x2 over 36s) kubelet, k-02 Created container Normal Started 18s (x2 over 36s) kubelet, k-02 Started container Normal Killing 18s kubelet, k-02 Killing container with id docker://app:Container failed liveness probe.. Container will be killed and recreated. Warning Unhealthy 4s (x5 over 29s) kubelet, k-02 Liveness probe failed: dial tcp 10.32.0.12:8888: connect: connection refused What just happened ? Since livenessProbe is failing it will keep killing and recreating containers. Thats what you see in the description above. When you list pods, you should see it in crashloopbackoff state with number of restarts incrementing with time. e.g. vote-668579766d-p65xb 0/1 CrashLoopBackOff 7 7m38s 10.32.0.12 k-02 vote-668579766d-sclbr 0/1 CrashLoopBackOff 7 7m38s 10.32.0.10 k-02 vote-668579766d-vrcmj 0/1 CrashLoopBackOff 7 7m38s 10.38.0.8 kube03-01 To fix it, revert the livenessProbe configs by editing the deplyment again. Readiness Probe Readiness probe is configured just like liveness probe. But this time we will use httpGet request . kubectl edit deploy vote readinessProbe: failureThreshold: 3 httpGet: path: /test.html port: 80 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 3 successThreshold: 1 where, readinessProbe.httpGet.path is been changed from / to /test.html which is a non existant path. check kubectl get deploy,rs,pods [output snippet] NAME READY UP-TO-DATE AVAILABLE AGE deployment.extensions/vote 11/12 3 11 2m12s vote-8cbb7ff89-6xvbc 0/1 Running 0 73s 10.38.0.10 kube03-01 vote-8cbb7ff89-6z5zv 0/1 Running 0 73s 10.38.0.5 kube03-01 vote-8cbb7ff89-hdmxb 0/1 Running 0 73s 10.32.0.12 k-02 kubectl describe pod vote-8cbb7ff89-hdmxb where, vote-8cbb7ff89-hdmxb is one of the pods launched after changing readiness probe. [output snippet] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 109s default-scheduler Successfully assigned instavote/vote-8cbb7ff89-hdmxb to k-02 Normal Pulled 108s kubelet, k-02 Container image \"schoolofdevops/vote:v1\" already present on machine Normal Created 108s kubelet, k-02 Created container Normal Started 108s kubelet, k-02 Started container Warning Unhealthy 39s (x22 over 102s) kubelet, k-02 Readiness probe failed: HTTP probe failed with statuscode: 404 kubectl describe svc vote what happened ? Since readinessProbe failed, the new launched batch does not show containers running (0/1) Description of the pod shows it being Unhealthy due to failed HTTP probe Deployment shows surged pods, with number of ready pods being less than number of desired replicas (e.g. 11/12). Service does not send traffic to the pod which are marked as unhealthy/not ready. Reverting the changes to readiness probe should bring it back to working state.","title":"Lab K204 - Health Checks"},{"location":"pods-health-probes/#lab-k204-adding-health-checks-with-probes","text":"","title":"Lab K204 - Adding health checks with Probes"},{"location":"pods-health-probes/#adding-health-checks","text":"Health checks in Kubernetes work the same way as traditional health checks of applications. They make sure that our application is ready to receive and process user requests. In Kubernetes we have two types of health checks, * Liveness Probe * Readiness Probe Probes are simply a diagnostic action performed by the kubelet. There are three types actions a kubelet perfomes on a pod, which are namely, ExecAction : Executes a command inside the pod. Assumed successful when the command returns 0 as exit code. TCPSocketAction : Checks for a state of a particular port on the pod. Considered successful when the state of the port is open . HTTPGetAction : Performs a GET request on pod's IP. Assumed successful when the status code is greater than 200 and less than 400 In cases of any failure during the diagnostic action, kubelet will report back to the API server. Let us study about how these health checks work in practice.","title":"Adding health checks"},{"location":"pods-health-probes/#adding-livenessreadineess-probes","text":"Liveness probe checks the status of the pod(whether it is running or not). If livenessProbe fails, then the pod is subjected to its restart policy. The default state of livenessProbe is Success . Readiness probe checks whether your application is ready to serve the requests. When the readiness probe fails, the pod's IP is removed from the end point list of the service. The default state of readinessProbe is Success . Let us add liveness/readiness probes to our vote deployment. file: vote-deploy-probes.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 replicas: 12 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: In, values: [v1, v2, v3, v4, v5]} template: metadata: name: vote labels: app: python role: vote version: v1 spec: containers: - name: app image: schoolofdevops/vote:v1 resources: requests: memory: \"64Mi\" cpu: \"50m\" limits: memory: \"128Mi\" cpu: \"250m\" livenessProbe: tcpSocket: port: 80 initialDelaySeconds: 5 periodSeconds: 5 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 3 where, livenessProbe used a simple tcp check to test whether application is listening on port 80 readinessProbe does httpGet to actually fetch a page using get method and tests for the http response code. Apply this code using, kubectl apply -f vote-deploy-probes.yaml kubectl get pods kubectl describe svc vote","title":"Adding Liveness/Readineess Probes"},{"location":"pods-health-probes/#testing-livenessprobe","text":"kubectl edit deploy vote livenessProbe: failureThreshold: 3 initialDelaySeconds: 5 periodSeconds: 5 successThreshold: 1 tcpSocket: port: 8888 timeoutSeconds: 1 Since you are using edit command, as soon as you save the file, deployment is modified. kubectl get pods kubectl describe pod vote-xxxx where, vote-xxxx is one of the new pods created. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 38s default-scheduler Successfully assigned instavote/vote-668579766d-p65xb to k-02 Normal Pulled 18s (x2 over 36s) kubelet, k-02 Container image \"schoolofdevops/vote:v1\" already present on machine Normal Created 18s (x2 over 36s) kubelet, k-02 Created container Normal Started 18s (x2 over 36s) kubelet, k-02 Started container Normal Killing 18s kubelet, k-02 Killing container with id docker://app:Container failed liveness probe.. Container will be killed and recreated. Warning Unhealthy 4s (x5 over 29s) kubelet, k-02 Liveness probe failed: dial tcp 10.32.0.12:8888: connect: connection refused What just happened ? Since livenessProbe is failing it will keep killing and recreating containers. Thats what you see in the description above. When you list pods, you should see it in crashloopbackoff state with number of restarts incrementing with time. e.g. vote-668579766d-p65xb 0/1 CrashLoopBackOff 7 7m38s 10.32.0.12 k-02 vote-668579766d-sclbr 0/1 CrashLoopBackOff 7 7m38s 10.32.0.10 k-02 vote-668579766d-vrcmj 0/1 CrashLoopBackOff 7 7m38s 10.38.0.8 kube03-01 To fix it, revert the livenessProbe configs by editing the deplyment again.","title":"Testing livenessProbe"},{"location":"pods-health-probes/#readiness-probe","text":"Readiness probe is configured just like liveness probe. But this time we will use httpGet request . kubectl edit deploy vote readinessProbe: failureThreshold: 3 httpGet: path: /test.html port: 80 scheme: HTTP initialDelaySeconds: 5 periodSeconds: 3 successThreshold: 1 where, readinessProbe.httpGet.path is been changed from / to /test.html which is a non existant path. check kubectl get deploy,rs,pods [output snippet] NAME READY UP-TO-DATE AVAILABLE AGE deployment.extensions/vote 11/12 3 11 2m12s vote-8cbb7ff89-6xvbc 0/1 Running 0 73s 10.38.0.10 kube03-01 vote-8cbb7ff89-6z5zv 0/1 Running 0 73s 10.38.0.5 kube03-01 vote-8cbb7ff89-hdmxb 0/1 Running 0 73s 10.32.0.12 k-02 kubectl describe pod vote-8cbb7ff89-hdmxb where, vote-8cbb7ff89-hdmxb is one of the pods launched after changing readiness probe. [output snippet] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 109s default-scheduler Successfully assigned instavote/vote-8cbb7ff89-hdmxb to k-02 Normal Pulled 108s kubelet, k-02 Container image \"schoolofdevops/vote:v1\" already present on machine Normal Created 108s kubelet, k-02 Created container Normal Started 108s kubelet, k-02 Started container Warning Unhealthy 39s (x22 over 102s) kubelet, k-02 Readiness probe failed: HTTP probe failed with statuscode: 404 kubectl describe svc vote what happened ? Since readinessProbe failed, the new launched batch does not show containers running (0/1) Description of the pod shows it being Unhealthy due to failed HTTP probe Deployment shows surged pods, with number of ready pods being less than number of desired replicas (e.g. 11/12). Service does not send traffic to the pod which are marked as unhealthy/not ready. Reverting the changes to readiness probe should bring it back to working state.","title":"Readiness Probe"},{"location":"rbac-resource-group-mapping/","text":"RBAC Reference kubernetes Instances Configuration GCP NUMBER OF NODE-SIZE INSTANCE TYPE CPU MEMORY 1-5 n1-standard-1 1 6-10 n1-standard-2 2 11-100 n1-standard-4 4 101-250 n1-standard-8 8 251-500 n1-standard-16 16 more than 500 n1-standard-32 32 AWS NUMBER OF NODE_SIZE INSTANCE TYPE CPU MEMORY 1-5 m3.medium 1 3.75 6-10 m3.large 2 7.50 11-100 m3.xlarge 4 15 101-250 m3.2xlarge 8 30 251-500 c4.4xlarge 8 30 more than 500 c4.8xlarge 16 60 api groups and resources apiGroup Resources apps daemonsets, deployments, deployments/rollback, deployments/scale, replicasets, replicasets/scale, statefulsets, statefulsets/scale core configmaps, endpoints, persistentvolumeclaims, replicationcontrollers, replicationcontrollers/scale, secrets, serviceaccounts, services,services/proxy autoscaling horizontalpodautoscalers batch cronjobs, jobs policy poddisruptionbudgets networking.k8s.io networkpolicies authorization.k8s.io localsubjectaccessreviews rbac.authorization.k8s.io rolebindings,roles extensions deprecated (read notes) Notes In addition to the above apiGroups, you may see extensions being used in some example code snippets. Please note that extensions was initially created as a experiement and is been deprecated, by moving most of the matured apis to one of the groups mentioned above. You could read this comment and the thread to get clarity on this.","title":"RBAC apiGroups to Resource Mapping"},{"location":"rbac-resource-group-mapping/#rbac-reference","text":"","title":"RBAC Reference"},{"location":"rbac-resource-group-mapping/#kubernetes-instances-configuration","text":"","title":"kubernetes Instances Configuration"},{"location":"rbac-resource-group-mapping/#gcp","text":"NUMBER OF NODE-SIZE INSTANCE TYPE CPU MEMORY 1-5 n1-standard-1 1 6-10 n1-standard-2 2 11-100 n1-standard-4 4 101-250 n1-standard-8 8 251-500 n1-standard-16 16 more than 500 n1-standard-32 32","title":"GCP"},{"location":"rbac-resource-group-mapping/#aws","text":"NUMBER OF NODE_SIZE INSTANCE TYPE CPU MEMORY 1-5 m3.medium 1 3.75 6-10 m3.large 2 7.50 11-100 m3.xlarge 4 15 101-250 m3.2xlarge 8 30 251-500 c4.4xlarge 8 30 more than 500 c4.8xlarge 16 60","title":"AWS"},{"location":"rbac-resource-group-mapping/#api-groups-and-resources","text":"apiGroup Resources apps daemonsets, deployments, deployments/rollback, deployments/scale, replicasets, replicasets/scale, statefulsets, statefulsets/scale core configmaps, endpoints, persistentvolumeclaims, replicationcontrollers, replicationcontrollers/scale, secrets, serviceaccounts, services,services/proxy autoscaling horizontalpodautoscalers batch cronjobs, jobs policy poddisruptionbudgets networking.k8s.io networkpolicies authorization.k8s.io localsubjectaccessreviews rbac.authorization.k8s.io rolebindings,roles extensions deprecated (read notes)","title":"api groups and resources"},{"location":"rbac-resource-group-mapping/#notes","text":"In addition to the above apiGroups, you may see extensions being used in some example code snippets. Please note that extensions was initially created as a experiement and is been deprecated, by moving most of the matured apis to one of the groups mentioned above. You could read this comment and the thread to get clarity on this.","title":"Notes"},{"location":"rbac_101/","text":"Defining RBAC Policies To understand how to define RBAC Policies, in this lab, you are going to work with the API Tester app, and the add relevant policies to it. To begin with, clone API Tester app and launch it git clone https://github.com/schoolofdevops/k8s-api-tester.git cd k8s-api-tester kubectl apply -f api-tester-deploy.yaml Now list the pod and check the logs kubectl get pods kubectl logs -f api-tester-xxxx You shall see error messages with access to all api resources e.g. pods, deployments, services, pvs, events denied. Lets look at how to configure the RBAC policies to get this app to work. Adding Roles and ClusterRoles Add the following permissions for api-tester app It should be able to list pods, deplyoments and services in namespace default File api-tester-sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: api-tester namespace: default File : api-tester-role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: api-tester namespace: default rules: - apiGroups: [\"\"] resources: [\"pods\", \"Services\"] verbs: [\"get\", \"list\", \"watch\"] - apiGroups: [\"apps\", \"extensions\"] resources: [\"deployments\"] verbs: [\"get\", \"list\", \"watch\"] File : api-tester-rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: api-tester namespace: default subjects: - kind: ServiceAccount name: api-tester namespace: default roleRef: kind: Role name: api-tester apiGroup: rbac.authorization.k8s.io apply kubectl apply -f api-tester-sa.yaml -f api-tester-role.yaml -f api-tester-rolebinding.yaml Now update the deployment spec to refer to Service Account as: File : api-tester-deploy.yaml ... .. spec: replicas: 1 selector: matchLabels: app: api-tester template: metadata: labels: app: api-tester spec: serviceAccountName: api-tester containers: - name: api-tester image: docker.io/schoolofdevops/api-tester:latest .. ... apply kubectl apply -f api-tester-deploy.yaml validate kubectl get pods kubectl logs -f api-tester-xxxx at this time it should show you that it has the permissions to list pods, deployments and services in a namespace. Adding CLusterRoles and ClusterRoleBindings Add the following permissions for api-tester app It should be able to list persistentvolumeclaims in all namespaces It should have ability to create and delete persistentvolumes in all namespaces File: api-tester-clusterrole.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: api-tester-cluster-role rules: - apiGroups: [\"\"] resources: [\"persistentvolumeclaims\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"] - apiGroups: [\"\"] resources: [\"persistentvolumes\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"] File: api-tester-clusterrolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: api-tester-cluster-role-binding subjects: - kind: ServiceAccount name: api-tester namespace: default roleRef: kind: ClusterRole name: api-tester-cluster-role apiGroup: rbac.authorization.k8s.io apply kubectl apply -f api-tester-clusterrole.yaml -f api-tester-clusterrolebinding.yaml validate kubectl get pods kubectl logs -f api-tester-xxxx At this time the logs for the api tester app should show you that it has the authorization to work with PVs and PVCs as well.","title":"Lab K111 - RBAC Policies"},{"location":"rbac_101/#defining-rbac-policies","text":"To understand how to define RBAC Policies, in this lab, you are going to work with the API Tester app, and the add relevant policies to it. To begin with, clone API Tester app and launch it git clone https://github.com/schoolofdevops/k8s-api-tester.git cd k8s-api-tester kubectl apply -f api-tester-deploy.yaml Now list the pod and check the logs kubectl get pods kubectl logs -f api-tester-xxxx You shall see error messages with access to all api resources e.g. pods, deployments, services, pvs, events denied. Lets look at how to configure the RBAC policies to get this app to work.","title":"Defining RBAC Policies"},{"location":"rbac_101/#adding-roles-and-clusterroles","text":"Add the following permissions for api-tester app It should be able to list pods, deplyoments and services in namespace default File api-tester-sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: api-tester namespace: default File : api-tester-role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: api-tester namespace: default rules: - apiGroups: [\"\"] resources: [\"pods\", \"Services\"] verbs: [\"get\", \"list\", \"watch\"] - apiGroups: [\"apps\", \"extensions\"] resources: [\"deployments\"] verbs: [\"get\", \"list\", \"watch\"] File : api-tester-rolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: api-tester namespace: default subjects: - kind: ServiceAccount name: api-tester namespace: default roleRef: kind: Role name: api-tester apiGroup: rbac.authorization.k8s.io apply kubectl apply -f api-tester-sa.yaml -f api-tester-role.yaml -f api-tester-rolebinding.yaml Now update the deployment spec to refer to Service Account as: File : api-tester-deploy.yaml ... .. spec: replicas: 1 selector: matchLabels: app: api-tester template: metadata: labels: app: api-tester spec: serviceAccountName: api-tester containers: - name: api-tester image: docker.io/schoolofdevops/api-tester:latest .. ... apply kubectl apply -f api-tester-deploy.yaml validate kubectl get pods kubectl logs -f api-tester-xxxx at this time it should show you that it has the permissions to list pods, deployments and services in a namespace.","title":"Adding Roles and ClusterRoles"},{"location":"rbac_101/#adding-clusterroles-and-clusterrolebindings","text":"Add the following permissions for api-tester app It should be able to list persistentvolumeclaims in all namespaces It should have ability to create and delete persistentvolumes in all namespaces File: api-tester-clusterrole.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: api-tester-cluster-role rules: - apiGroups: [\"\"] resources: [\"persistentvolumeclaims\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"] - apiGroups: [\"\"] resources: [\"persistentvolumes\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"] File: api-tester-clusterrolebinding.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: api-tester-cluster-role-binding subjects: - kind: ServiceAccount name: api-tester namespace: default roleRef: kind: ClusterRole name: api-tester-cluster-role apiGroup: rbac.authorization.k8s.io apply kubectl apply -f api-tester-clusterrole.yaml -f api-tester-clusterrolebinding.yaml validate kubectl get pods kubectl logs -f api-tester-xxxx At this time the logs for the api tester app should show you that it has the authorization to work with PVs and PVCs as well.","title":"Adding CLusterRoles and ClusterRoleBindings"},{"location":"run_and_publish_image/","text":"Quck Deploy a Docker Image with Kubernetes kubectl run vote --image=schoolofdevops/vote --port 80 kubectl scale --replicas=10 deploy/vote kubectl expose deploy vote --port 80 --target-port 80 --type NodePort","title":"Quck Deploy a Docker Image with Kubernetes"},{"location":"run_and_publish_image/#quck-deploy-a-docker-image-with-kubernetes","text":"kubectl run vote --image=schoolofdevops/vote --port 80 kubectl scale --replicas=10 deploy/vote kubectl expose deploy vote --port 80 --target-port 80 --type NodePort","title":"Quck Deploy a Docker Image with Kubernetes"},{"location":"statefulset/","text":"Running Replicated Redis Setup with statefulsets What will you learn * Statefulsets * initContainers Reading List https://redis.io/topics/replication https://discuss.pivotal.io/hc/en-us/articles/205309278-How-to-setup-Redis-master-and-slave-replication https://www.digitalocean.com/community/tutorials/how-to-configure-redis-replication-on-ubuntu-16-04 Run Replicated Statefulsets Applications Keywords init containers kubernetes statefulsets redis replication","title":"Running Replicated Redis Setup with statefulsets"},{"location":"statefulset/#running-replicated-redis-setup-with-statefulsets","text":"What will you learn * Statefulsets * initContainers","title":"Running Replicated Redis Setup with statefulsets"},{"location":"statefulset/#reading-list","text":"https://redis.io/topics/replication https://discuss.pivotal.io/hc/en-us/articles/205309278-How-to-setup-Redis-master-and-slave-replication https://www.digitalocean.com/community/tutorials/how-to-configure-redis-replication-on-ubuntu-16-04 Run Replicated Statefulsets Applications Keywords init containers kubernetes statefulsets redis replication","title":"Reading List"},{"location":"vote-deployement_strategies/","text":"Release Strategies Releases with downtime using Recreate Strategy When the Recreate deployment strategy is used, * The old pods will be deleted * Then the new pods will be created. This will create some downtime in our stack. Let us change the deployment strategy to recreate and image tag to v4 . cp vote-deploy.yaml vote-deploy-recreate.yaml And edit the specs with following changes Update strategy to Recreate Remove rolling update specs file: vote-deploy-recreate.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote labels: role: vote spec: strategy: type: Recreate revisionHistoryLimit: 4 paused: false ..... and apply kubectl get pods,rs,deploy,svc kubectl apply -f vote-deploy-recreate.yaml kubectl rollout status deplloyment/vote While the deployment happens, use the monitoring/visualiser and observe the manner in which the deployment gets updated. You would observe that All pods wit the current version are deleted first Only after all the existing pods are deleted, pods with new version are launched A/B Testing (Previously Canary) Please note canary section is been renamed to A/B testing as this is more appropriate name for the kind of release strategy being described here In this section of the lab, you will deploy a A/B testing release which will distribute percentage of total traffic to a newer version. This will allow you to deploy and test your code in production, safely, with a subset of your users. Whats the differnce between Canary and A/B Testing ? Well, with canary, you could typically choose which subset of clients you would want to send to the newer versions, based on some criteria, such as client name, user name, type of device, certain specific header etc. A/B testing is simpler version of Canary, where the subset of clients redirected to the new version randomly e.g. 20% of the total traffic going to v2, remaining to v1. This could be easily achieved with Kubernetes using the method described here. Canary needs more advanced configurations, typically lacking with kubernetes, and thats where you may want to consider using a service mesh such as Istio. cd k8s-code/projets/instavote/dev mkdir ab cp vote-deploy.yaml ab/vote-ab-deploy.yaml change the following fields in vote-ab-deploy.yaml metadata.name: vote-ab spec.replicas: 3 spec.selector.matchExpressions: - {key: version, operator: Exists template.metadata.labels.version: v4 template.spec.containers.image: schoolofdevops/vote:v4 File: ab/frontend-ab-deploy.yml apiVersion: apps/v1 kind: Deployment metadata: name: vote-ab spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 paused: false replicas: 3 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: Exists} minReadySeconds: 40 template: metadata: name: vote labels: app: python role: vote version: v4 spec: containers: - name: app image: schoolofdevops/vote:v4 ports: - containerPort: 80 protocol: TCP Before creating this deployment, find out how many endpoints the service has, kubectl describe service/vote [sample output ] Endpoints: 10.32.0.10:80,10.32.0.11:80,10.32.0.4:80 + 12 more... In this example current endpoints are 15 Now create the deployment for ab release kubectl apply -f ab/frontend-ab-deploy.yml And validate, kubectl get rs,deploy,svc kubectl describe service/vote When you describe vote service, observe the number of endpoints [sample output] Endpoints: 10.32.0.10:80,10.32.0.11:80,10.32.0.16:80 + 15 more... Now its 18 , which is 3 more than the previous number. Those are the pods created by the ab deployment. And the above output proves that its actually sending traffic to both versions. Delete A/B Deployment Once validated, you could update the main deployment to rollout the new version 100%(procedure not given here) and clean up ab release using kubectl delete -f ab/vote-ab-deploy.yaml Blue/Green Releases Before proceeding, lets clean up the existing deployment. kubectl delete deploy/vote kubectl delete svc/vote kubectl get pods,deploy,rs,svc And create the work directory for blue-green release definitions. cd k8s-code/projets/instavote/dev mkdir blue-green file: blue-green/vote-blue-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote-blue spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 paused: false replicas: 15 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: Exists} template: metadata: name: vote labels: app: python role: vote version: v3 release: bluegreen code: blue spec: containers: - name: app image: schoolofdevops/vote:v3 ports: - containerPort: 80 protocol: TCP file: blue-green/vote-green-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote-green spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 paused: false replicas: 15 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: Exists} template: metadata: name: vote labels: app: python role: vote version: v3 release: bluegreen code: green spec: containers: - name: app image: schoolofdevops/vote:v4 ports: - containerPort: 80 protocol: TCP file: blue-green/vote-bg-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote-bg labels: role: vote release: bluegreen spec: selector: role: vote release: bluegreen code: green ports: - port: 80 targetPort: 80 nodePort: 30001 type: NodePort file: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote release: bluegreen code: blue ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort Creating blue deployment Now create vote service and observe the endpoints kubectl apply -f vote-svc.yaml kubectl get svc kubectl describe svc/vote [sample output] Name: vote Namespace: instavote Labels: role=vote Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"vote\"},\"name\":\"vote\",\"namespace\":\"instavote\"},\"spec\":{\"externalIPs\":... Selector: code=blue,release=bluegreen,role=vote Type: NodePort IP: 10.111.93.227 External IPs: 206.189.150.190,159.65.8.227 Port: 80/TCP TargetPort: 80/TCP NodePort: 30000/TCP Endpoints: Session Affinity: None External Traffic Policy: Cluster Events: where, * endpoints are None * its selecting pods with code=blue Now lets create the deployment for blue release kubectl get pods,rs,deploy kubectl apply -f blue-green/vote-blue-deploy.yaml kubectl get pods,rs,deploy kubectl rollout status deploy/vote-blue [sample output] Waiting for rollout to finish: 2 of 15 updated replicas are available... deployment \"vote-blue\" successfully rolled out Now if you check the service, it should have the pods launched with blue set as endpoints kubectl describe svc/vote Name: vote Namespace: instavote Labels: role=vote Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"vote\"},\"name\":\"vote\",\"namespace\":\"instavote\"},\"spec\":{\"externalIPs\":... Selector: code=blue,release=bluegreen,role=vote Type: NodePort IP: 10.111.93.227 External IPs: 206.189.150.190,159.65.8.227 Port: 80/TCP TargetPort: 80/TCP NodePort: 30000/TCP Endpoints: 10.32.0.10:80,10.32.0.11:80,10.32.0.4:80 + 12 more... Session Affinity: None External Traffic Policy: Cluster Events: You could observe the Endpoints created and added to the service. Browse to http://IPADDRESS:NODEPORT to see the application deployed. Deploying new version with green release While deploying a new version with blue-green strategy, we would Create a new deployment in parallel Test it by creating another service Cut over to new release by updating selector in the main service Lets create the deployment with new version and a service to test it. Lets call it the green deployment kubectl apply -f blue-green/vote-bg-svc.yaml kubectl apply -f blue-green/vote-bg-svc.yaml kubectl apply -f blue-green/vote-green-deploy.yaml kubectl rollout status deploy/vote-green [sample output] Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 7 of 15 updated replicas are available... deployment \"vote-green\" successfully rolled out Validate kubectl get pods,rs,deploy,svc You could also test it by going to the http://host:nodeport for service vote-bg Switching to new version Now that you have the new version running in parallel, you could quickly switch to it by updating selector for main vote service which is live. Please note, while switching there may be a momentory downtime. Steps visit http://HOST:NODEPORT for vote service update vote service to select green release apply service definition visit http://HOST:NODEPORT for vote service again to validate file: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote release: bluegreen code: green ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort Apply it with kubectl apply -f vote-svc.yaml kubectl describe svc/vote If you visit http://HOST:NODEPORT for vote service, you should see the application version updated Clean up the previous version kubectl delete deploy/vote-blue Clean up blue-green configs Now that you are done testing blue green release, lets revert to our previous configurations. kubectl delete deploy/vote-green kubectl apply -f vote-deploy.yaml Also update the service definition and remove following selectors added for blue green release release: bluegreen code: blue file: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort And apply kubectl apply -f vote-svc.yaml Pause/Unpause live deployments When you are in the middle of a new update for your application and you found out that the application is behaving as intended. In those situations, 1. we can pause the update, 2. fix the issue, 3. resume the update. Let us change the image tag to V4 in pod spec. File: vote-deploy.yaml spec: containers: - name: app image: schoolofdevops/vote:V4 ports: - containerPort: 80 protocol: TCP Apply the changes. kubectl apply -f vote-deploy.yaml kubectl get pods [Output] NAME READY STATUS RESTARTS AGE vote-6c4f7b49d8-g5dgc 1/1 Running 0 16m vote-765554cc7-xsbhs 0/1 ErrImagePull 0 9s Our deployment is failing. From some debugging, we can conclude that we are using a wrong image tag. Now pause the update kubectl rollout pause deploy/vote Set the deployment to use v4 version of the image. Now resume the update kubectl rollout resume deployment vote kubectl rollout status deployment vote [Ouput] deployment \"vote\" successfully rolled out and validate kubectl get pods,rs,deploy [Output] NAME READY STATUS RESTARTS AGE vote-6875c8df8f-k4hls 1/1 Running 0 1m When you do this, you skip the need of creating a new rolling update altogether. Case for Service Mesh When kubernetes offers ways to create Blue/Green release strategy, do zero down time deployments, implement A/B Testing, why do you need Service Mesh ? Well here are a few reasons * Kubernetes does not support Canary out of the box. you could get close by achieving A/B Testing, but still a canary release would be useful in many cases. * Even though you could create A/B testing and Blue/Green deployments, but the percentage of traffic sent to A/B testing is more or less based on the size of the deployments. With Istio, you could pretty much achieve all of this with purely service configurations, and setup many advanced rules. * Its not just about release strategies, Istio offers much more by extending existing awesome kubernetes features. It essentially offloads all the routing, traffic management configurations from developers by managing it all at the platform level.","title":"Lab K403 - Building Deployment Strategies"},{"location":"vote-deployement_strategies/#release-strategies","text":"","title":"Release Strategies"},{"location":"vote-deployement_strategies/#releases-with-downtime-using-recreate-strategy","text":"When the Recreate deployment strategy is used, * The old pods will be deleted * Then the new pods will be created. This will create some downtime in our stack. Let us change the deployment strategy to recreate and image tag to v4 . cp vote-deploy.yaml vote-deploy-recreate.yaml And edit the specs with following changes Update strategy to Recreate Remove rolling update specs file: vote-deploy-recreate.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote labels: role: vote spec: strategy: type: Recreate revisionHistoryLimit: 4 paused: false ..... and apply kubectl get pods,rs,deploy,svc kubectl apply -f vote-deploy-recreate.yaml kubectl rollout status deplloyment/vote While the deployment happens, use the monitoring/visualiser and observe the manner in which the deployment gets updated. You would observe that All pods wit the current version are deleted first Only after all the existing pods are deleted, pods with new version are launched","title":"Releases with downtime using Recreate Strategy"},{"location":"vote-deployement_strategies/#ab-testing-previously-canary","text":"Please note canary section is been renamed to A/B testing as this is more appropriate name for the kind of release strategy being described here In this section of the lab, you will deploy a A/B testing release which will distribute percentage of total traffic to a newer version. This will allow you to deploy and test your code in production, safely, with a subset of your users.","title":"A/B Testing (Previously Canary)"},{"location":"vote-deployement_strategies/#whats-the-differnce-between-canary-and-ab-testing","text":"Well, with canary, you could typically choose which subset of clients you would want to send to the newer versions, based on some criteria, such as client name, user name, type of device, certain specific header etc. A/B testing is simpler version of Canary, where the subset of clients redirected to the new version randomly e.g. 20% of the total traffic going to v2, remaining to v1. This could be easily achieved with Kubernetes using the method described here. Canary needs more advanced configurations, typically lacking with kubernetes, and thats where you may want to consider using a service mesh such as Istio. cd k8s-code/projets/instavote/dev mkdir ab cp vote-deploy.yaml ab/vote-ab-deploy.yaml change the following fields in vote-ab-deploy.yaml metadata.name: vote-ab spec.replicas: 3 spec.selector.matchExpressions: - {key: version, operator: Exists template.metadata.labels.version: v4 template.spec.containers.image: schoolofdevops/vote:v4 File: ab/frontend-ab-deploy.yml apiVersion: apps/v1 kind: Deployment metadata: name: vote-ab spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 paused: false replicas: 3 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: Exists} minReadySeconds: 40 template: metadata: name: vote labels: app: python role: vote version: v4 spec: containers: - name: app image: schoolofdevops/vote:v4 ports: - containerPort: 80 protocol: TCP Before creating this deployment, find out how many endpoints the service has, kubectl describe service/vote [sample output ] Endpoints: 10.32.0.10:80,10.32.0.11:80,10.32.0.4:80 + 12 more... In this example current endpoints are 15 Now create the deployment for ab release kubectl apply -f ab/frontend-ab-deploy.yml And validate, kubectl get rs,deploy,svc kubectl describe service/vote When you describe vote service, observe the number of endpoints [sample output] Endpoints: 10.32.0.10:80,10.32.0.11:80,10.32.0.16:80 + 15 more... Now its 18 , which is 3 more than the previous number. Those are the pods created by the ab deployment. And the above output proves that its actually sending traffic to both versions.","title":"Whats the differnce between Canary and A/B Testing ?"},{"location":"vote-deployement_strategies/#delete-ab-deployment","text":"Once validated, you could update the main deployment to rollout the new version 100%(procedure not given here) and clean up ab release using kubectl delete -f ab/vote-ab-deploy.yaml","title":"Delete A/B Deployment"},{"location":"vote-deployement_strategies/#bluegreen-releases","text":"Before proceeding, lets clean up the existing deployment. kubectl delete deploy/vote kubectl delete svc/vote kubectl get pods,deploy,rs,svc And create the work directory for blue-green release definitions. cd k8s-code/projets/instavote/dev mkdir blue-green file: blue-green/vote-blue-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote-blue spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 paused: false replicas: 15 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: Exists} template: metadata: name: vote labels: app: python role: vote version: v3 release: bluegreen code: blue spec: containers: - name: app image: schoolofdevops/vote:v3 ports: - containerPort: 80 protocol: TCP file: blue-green/vote-green-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: vote-green spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 revisionHistoryLimit: 4 paused: false replicas: 15 minReadySeconds: 20 selector: matchLabels: role: vote matchExpressions: - {key: version, operator: Exists} template: metadata: name: vote labels: app: python role: vote version: v3 release: bluegreen code: green spec: containers: - name: app image: schoolofdevops/vote:v4 ports: - containerPort: 80 protocol: TCP file: blue-green/vote-bg-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote-bg labels: role: vote release: bluegreen spec: selector: role: vote release: bluegreen code: green ports: - port: 80 targetPort: 80 nodePort: 30001 type: NodePort file: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote release: bluegreen code: blue ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort","title":"Blue/Green Releases"},{"location":"vote-deployement_strategies/#creating-blue-deployment","text":"Now create vote service and observe the endpoints kubectl apply -f vote-svc.yaml kubectl get svc kubectl describe svc/vote [sample output] Name: vote Namespace: instavote Labels: role=vote Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"vote\"},\"name\":\"vote\",\"namespace\":\"instavote\"},\"spec\":{\"externalIPs\":... Selector: code=blue,release=bluegreen,role=vote Type: NodePort IP: 10.111.93.227 External IPs: 206.189.150.190,159.65.8.227 Port: 80/TCP TargetPort: 80/TCP NodePort: 30000/TCP Endpoints: Session Affinity: None External Traffic Policy: Cluster Events: where, * endpoints are None * its selecting pods with code=blue Now lets create the deployment for blue release kubectl get pods,rs,deploy kubectl apply -f blue-green/vote-blue-deploy.yaml kubectl get pods,rs,deploy kubectl rollout status deploy/vote-blue [sample output] Waiting for rollout to finish: 2 of 15 updated replicas are available... deployment \"vote-blue\" successfully rolled out Now if you check the service, it should have the pods launched with blue set as endpoints kubectl describe svc/vote Name: vote Namespace: instavote Labels: role=vote Annotations: kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"role\":\"vote\"},\"name\":\"vote\",\"namespace\":\"instavote\"},\"spec\":{\"externalIPs\":... Selector: code=blue,release=bluegreen,role=vote Type: NodePort IP: 10.111.93.227 External IPs: 206.189.150.190,159.65.8.227 Port: 80/TCP TargetPort: 80/TCP NodePort: 30000/TCP Endpoints: 10.32.0.10:80,10.32.0.11:80,10.32.0.4:80 + 12 more... Session Affinity: None External Traffic Policy: Cluster Events: You could observe the Endpoints created and added to the service. Browse to http://IPADDRESS:NODEPORT to see the application deployed.","title":"Creating blue deployment"},{"location":"vote-deployement_strategies/#deploying-new-version-with-green-release","text":"While deploying a new version with blue-green strategy, we would Create a new deployment in parallel Test it by creating another service Cut over to new release by updating selector in the main service Lets create the deployment with new version and a service to test it. Lets call it the green deployment kubectl apply -f blue-green/vote-bg-svc.yaml kubectl apply -f blue-green/vote-bg-svc.yaml kubectl apply -f blue-green/vote-green-deploy.yaml kubectl rollout status deploy/vote-green [sample output] Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 0 of 15 updated replicas are available... Waiting for rollout to finish: 7 of 15 updated replicas are available... deployment \"vote-green\" successfully rolled out Validate kubectl get pods,rs,deploy,svc You could also test it by going to the http://host:nodeport for service vote-bg","title":"Deploying new version with green release"},{"location":"vote-deployement_strategies/#switching-to-new-version","text":"Now that you have the new version running in parallel, you could quickly switch to it by updating selector for main vote service which is live. Please note, while switching there may be a momentory downtime. Steps visit http://HOST:NODEPORT for vote service update vote service to select green release apply service definition visit http://HOST:NODEPORT for vote service again to validate file: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote release: bluegreen code: green ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort Apply it with kubectl apply -f vote-svc.yaml kubectl describe svc/vote If you visit http://HOST:NODEPORT for vote service, you should see the application version updated","title":"Switching to new version"},{"location":"vote-deployement_strategies/#clean-up-the-previous-version","text":"kubectl delete deploy/vote-blue","title":"Clean up the previous version"},{"location":"vote-deployement_strategies/#clean-up-blue-green-configs","text":"Now that you are done testing blue green release, lets revert to our previous configurations. kubectl delete deploy/vote-green kubectl apply -f vote-deploy.yaml Also update the service definition and remove following selectors added for blue green release release: bluegreen code: blue file: vote-svc.yaml --- apiVersion: v1 kind: Service metadata: name: vote labels: role: vote spec: selector: role: vote ports: - port: 80 targetPort: 80 nodePort: 30000 type: NodePort And apply kubectl apply -f vote-svc.yaml","title":"Clean up blue-green configs"},{"location":"vote-deployement_strategies/#pauseunpause-live-deployments","text":"When you are in the middle of a new update for your application and you found out that the application is behaving as intended. In those situations, 1. we can pause the update, 2. fix the issue, 3. resume the update. Let us change the image tag to V4 in pod spec. File: vote-deploy.yaml spec: containers: - name: app image: schoolofdevops/vote:V4 ports: - containerPort: 80 protocol: TCP Apply the changes. kubectl apply -f vote-deploy.yaml kubectl get pods [Output] NAME READY STATUS RESTARTS AGE vote-6c4f7b49d8-g5dgc 1/1 Running 0 16m vote-765554cc7-xsbhs 0/1 ErrImagePull 0 9s Our deployment is failing. From some debugging, we can conclude that we are using a wrong image tag. Now pause the update kubectl rollout pause deploy/vote Set the deployment to use v4 version of the image. Now resume the update kubectl rollout resume deployment vote kubectl rollout status deployment vote [Ouput] deployment \"vote\" successfully rolled out and validate kubectl get pods,rs,deploy [Output] NAME READY STATUS RESTARTS AGE vote-6875c8df8f-k4hls 1/1 Running 0 1m When you do this, you skip the need of creating a new rolling update altogether.","title":"Pause/Unpause live deployments"},{"location":"vote-deployement_strategies/#case-for-service-mesh","text":"When kubernetes offers ways to create Blue/Green release strategy, do zero down time deployments, implement A/B Testing, why do you need Service Mesh ? Well here are a few reasons * Kubernetes does not support Canary out of the box. you could get close by achieving A/B Testing, but still a canary release would be useful in many cases. * Even though you could create A/B testing and Blue/Green deployments, but the percentage of traffic sent to A/B testing is more or less based on the size of the deployments. With Istio, you could pretty much achieve all of this with purely service configurations, and setup many advanced rules. * Its not just about release strategies, Istio offers much more by extending existing awesome kubernetes features. It essentially offloads all the routing, traffic management configurations from developers by managing it all at the platform level.","title":"Case for Service Mesh"},{"location":"wq000-template/","text":"KWQ001 - Kubernetes Security Web Quest Security is a critical aspect while designing, building and maintaining any application infrastructure including kubernetes environemnts. With this exercise you are on a quest to find the measures you could implement to harden your kubernetes environment holistically. Task Your task is to work in your group, research on security measures at every level, and try to find answers to the following questions. You will present on the topic briefly. Questions : * * Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources Article: eBook: video:","title":"KWQ001 - Kubernetes Security Web Quest"},{"location":"wq000-template/#kwq001-kubernetes-security-web-quest","text":"Security is a critical aspect while designing, building and maintaining any application infrastructure including kubernetes environemnts. With this exercise you are on a quest to find the measures you could implement to harden your kubernetes environment holistically.","title":"KWQ001 - Kubernetes Security Web Quest"},{"location":"wq000-template/#task","text":"Your task is to work in your group, research on security measures at every level, and try to find answers to the following questions. You will present on the topic briefly. Questions : * *","title":"Task"},{"location":"wq000-template/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq000-template/#resources","text":"Article: eBook: video:","title":"Resources"},{"location":"wq001-cni/","text":"KWQ001 - CNI Web Quest Kubernetes has adapted Container Network Interface(CNI) as a standard to provide networking between pods running across hosts and to assign those with IP addresses. The purpose of this Web Quest is to compare the CNI plugins available, and make recommendations based on the merit that you find in those. Task Your task is to work in your group, research on CNI plugins, try to find answers to the following questions, and present briefly on it. Questions : Why did kubernetes choose CNI (Container Network Interface) as a networking standard instead of CNM (Container Network Management ) that List down some of the most popular CNI plugins. How do they compare, what are the key differences ? What are some of the advantages of using Calico as a CNI plugin ? Do you see any issues with network implementation in your organisation ? If yes, briefly describe. Which CNI plugin would you choose for your project ? Why ? Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources Article: Choosing a CNI Network Provider for Kubernetes Article: Kubernetes Network Plugins Doc: Official CNI Doc Video: How VXLANs Work Video: Life of a Packet Video: How Calico Works Doc: Official Kubernetes Doc on Cluster Networking","title":"KWQ001 - CNI Web Quest"},{"location":"wq001-cni/#kwq001-cni-web-quest","text":"Kubernetes has adapted Container Network Interface(CNI) as a standard to provide networking between pods running across hosts and to assign those with IP addresses. The purpose of this Web Quest is to compare the CNI plugins available, and make recommendations based on the merit that you find in those.","title":"KWQ001 - CNI Web Quest"},{"location":"wq001-cni/#task","text":"Your task is to work in your group, research on CNI plugins, try to find answers to the following questions, and present briefly on it. Questions : Why did kubernetes choose CNI (Container Network Interface) as a networking standard instead of CNM (Container Network Management ) that List down some of the most popular CNI plugins. How do they compare, what are the key differences ? What are some of the advantages of using Calico as a CNI plugin ? Do you see any issues with network implementation in your organisation ? If yes, briefly describe. Which CNI plugin would you choose for your project ? Why ?","title":"Task"},{"location":"wq001-cni/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq001-cni/#resources","text":"Article: Choosing a CNI Network Provider for Kubernetes Article: Kubernetes Network Plugins Doc: Official CNI Doc Video: How VXLANs Work Video: Life of a Packet Video: How Calico Works Doc: Official Kubernetes Doc on Cluster Networking","title":"Resources"},{"location":"wq002-security/","text":"KWQ002 - Kubernetes Security Web Quest Security is a critical aspect while designing, building and maintaining any application infrastructure including kubernetes environemnts. With this exercise you are on a quest to find the measures you could implement to harden your kubernetes environment holistically. Task Your task is to work in your group, research on security measures at every level, and try to find answers to the following questions. You will present on the topic briefly. Questions : Provide an example of a major kubernetes vulnerability and how it had an adverse impact on an organisation. What are the security measures you would take to harden your kubernetes environment ? In your opinion, which is the most overlooked aspect of Kubernetes Security ? Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources Article: 9 Kubernetes Security Best Practices Everyone Must Follow eBook: Kubernetes Deployment and Security Patterns video: Hacking and Hardening Kubernetes Clusters by Example video: Kubernetes Security Best Practices","title":"KWQ001 - Kubernetes Security Web Quest"},{"location":"wq002-security/#kwq002-kubernetes-security-web-quest","text":"Security is a critical aspect while designing, building and maintaining any application infrastructure including kubernetes environemnts. With this exercise you are on a quest to find the measures you could implement to harden your kubernetes environment holistically.","title":"KWQ002 - Kubernetes Security Web Quest"},{"location":"wq002-security/#task","text":"Your task is to work in your group, research on security measures at every level, and try to find answers to the following questions. You will present on the topic briefly. Questions : Provide an example of a major kubernetes vulnerability and how it had an adverse impact on an organisation. What are the security measures you would take to harden your kubernetes environment ? In your opinion, which is the most overlooked aspect of Kubernetes Security ?","title":"Task"},{"location":"wq002-security/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq002-security/#resources","text":"Article: 9 Kubernetes Security Best Practices Everyone Must Follow eBook: Kubernetes Deployment and Security Patterns video: Hacking and Hardening Kubernetes Clusters by Example video: Kubernetes Security Best Practices","title":"Resources"},{"location":"wq003-ingress/","text":"KWQ003 - Ingress Web Quest Ingress controllers adds layer7 load balancing, reverse proxy and intelligent http routing capabilities to kubernetes cluster. The purpose of this Web Quest is to compare the ingress controllers available, and make recommendations based on the merit that you find in those. Task Your task is to work in your group, research on ingress controllers, and try to find answers to the following questions. You will present on the topic briefly. Questions : If you have to choose between LoadBalancer as a service type and a Ingress Controller which one would you choose? Why? List down some of the most popular ingress controllers ? How do they compare, what are the key differences ? How do you make ingress controllers pick up configurations when a new service/deployment is created. Which ingress controller would you use ? Why? Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources Article: Ingress comparison by kubedex.com eBook: Managing Ingress Controllers on Kubernetes - a Guide by Giant Swarm Doc: Traefik Ingress Controller","title":"KWQ003 - Ingress Web Quest"},{"location":"wq003-ingress/#kwq003-ingress-web-quest","text":"Ingress controllers adds layer7 load balancing, reverse proxy and intelligent http routing capabilities to kubernetes cluster. The purpose of this Web Quest is to compare the ingress controllers available, and make recommendations based on the merit that you find in those.","title":"KWQ003 - Ingress Web Quest"},{"location":"wq003-ingress/#task","text":"Your task is to work in your group, research on ingress controllers, and try to find answers to the following questions. You will present on the topic briefly. Questions : If you have to choose between LoadBalancer as a service type and a Ingress Controller which one would you choose? Why? List down some of the most popular ingress controllers ? How do they compare, what are the key differences ? How do you make ingress controllers pick up configurations when a new service/deployment is created. Which ingress controller would you use ? Why?","title":"Task"},{"location":"wq003-ingress/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq003-ingress/#resources","text":"Article: Ingress comparison by kubedex.com eBook: Managing Ingress Controllers on Kubernetes - a Guide by Giant Swarm Doc: Traefik Ingress Controller","title":"Resources"},{"location":"wq004-monitoring/","text":"KWQ004 - Kubernetes Monitoring Web Quest Monitoring a kubernetes environment including nodes, services, deployments, as well as application availability as well as logs is important for a kubernetes administrator. With this web quest, your task is to dive deeper into the kubernetes monitoring infrastructure and make recommendations. Task Your task is to work in your group, research on monitoring architecture as well as tools available, and try to find answers to the following questions. You will present on the topic briefly. Questions : Explain the Kubernetes Monitoring Architecture briefly Whar are the monitoring solutions available for kubernetes ? Which one would you recommend and why ? What are the aspects of kubernetes environment and yoyr infastructure that you would monitor ? Does introduction to kubernetes have an effect on any of the aspects being monitored ? Which grafana dashboard did you find most useful ? Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources Article: 5 Tools for monitoring Kubernetes at Scale in Production eBook: The State of Kubernetes Ecosystem Video: You are Monitoring Kubernetes Wrong Video: Monitoring Kubernetes Clusters with Prometheus Doc : Kubernetes Monitoring Architecture","title":"KWQ004 - Monitoring Web Quest"},{"location":"wq004-monitoring/#kwq004-kubernetes-monitoring-web-quest","text":"Monitoring a kubernetes environment including nodes, services, deployments, as well as application availability as well as logs is important for a kubernetes administrator. With this web quest, your task is to dive deeper into the kubernetes monitoring infrastructure and make recommendations.","title":"KWQ004 - Kubernetes Monitoring Web Quest"},{"location":"wq004-monitoring/#task","text":"Your task is to work in your group, research on monitoring architecture as well as tools available, and try to find answers to the following questions. You will present on the topic briefly. Questions : Explain the Kubernetes Monitoring Architecture briefly Whar are the monitoring solutions available for kubernetes ? Which one would you recommend and why ? What are the aspects of kubernetes environment and yoyr infastructure that you would monitor ? Does introduction to kubernetes have an effect on any of the aspects being monitored ? Which grafana dashboard did you find most useful ?","title":"Task"},{"location":"wq004-monitoring/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq004-monitoring/#resources","text":"Article: 5 Tools for monitoring Kubernetes at Scale in Production eBook: The State of Kubernetes Ecosystem Video: You are Monitoring Kubernetes Wrong Video: Monitoring Kubernetes Clusters with Prometheus Doc : Kubernetes Monitoring Architecture","title":"Resources"},{"location":"wq005-controllers/","text":"KWQ005 - Controllers Web Quest If you look beyond deployment, which is great for running application with high availability, kubernetes supports variety of workloads. The purpose of this web quest is understand at a depth, the gamut of controllers that kubernetes supports and how each of that behaves. Task Your task is to work in your group, research on the controllers and workloads that the kubernetes supports, the purpose of each and how they behave. You would try to find answers to the following questions and present on the topic briefly. Following are the controllers you would research on deployment statefulset daemonset jobs cron Questions : Describe and compare type of controllers available with kubernetes. Explain the purpose, key feature and the type of work load each is suitable for, Which controllers would you pick to run the following applications ? Why ? a stateless web frontend cassandra db cluster monitoring agents scheduled jobs ad-hoc load tests Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources doc: Statefulsets doc: DaemonSet doc: Jobs doc: Cron doc: Kubernetes Examples","title":"KWQ005 - Controllers Web Quest"},{"location":"wq005-controllers/#kwq005-controllers-web-quest","text":"If you look beyond deployment, which is great for running application with high availability, kubernetes supports variety of workloads. The purpose of this web quest is understand at a depth, the gamut of controllers that kubernetes supports and how each of that behaves.","title":"KWQ005 - Controllers Web Quest"},{"location":"wq005-controllers/#task","text":"Your task is to work in your group, research on the controllers and workloads that the kubernetes supports, the purpose of each and how they behave. You would try to find answers to the following questions and present on the topic briefly. Following are the controllers you would research on deployment statefulset daemonset jobs cron Questions : Describe and compare type of controllers available with kubernetes. Explain the purpose, key feature and the type of work load each is suitable for, Which controllers would you pick to run the following applications ? Why ? a stateless web frontend cassandra db cluster monitoring agents scheduled jobs ad-hoc load tests","title":"Task"},{"location":"wq005-controllers/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq005-controllers/#resources","text":"doc: Statefulsets doc: DaemonSet doc: Jobs doc: Cron doc: Kubernetes Examples","title":"Resources"},{"location":"wq006-persistentvolume/","text":"KWQ001 - Persistent Volume and Storage Web Quest Storage and data persistence is essential in most application environments. With kubernetes, a network storage solution becomes important due to the immutable deployments of pods which can spawn on new nodes every time with a typical deployment. The purpose of this web quest is to understand the storage options, data peristence and dynamic provisioning of data. Task Your task is to work in your group, research on the storage systems and persistent volumes that kubernetes is compatible with and make recommendations. You will try to find answers to the following questions and present on the topic briefly. Questions : Explain the concept of dynamic provisioning Which are the storage classes that have internal provisioners ? Which storage solution would you use in your organisation ? Why? How would your storage and provisioning system ? Explain briefly Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources doc: Storage Classes eBook: State of kubernetes ecosystem Article: The Kubernetes Storage Ecosystem Video: Persistent Storage with Kubernetes in Production - Which Solution and Why? Video: Kubernetes Storage Lingo 101","title":"KWQ006 - Persistent Data Web Quest"},{"location":"wq006-persistentvolume/#kwq001-persistent-volume-and-storage-web-quest","text":"Storage and data persistence is essential in most application environments. With kubernetes, a network storage solution becomes important due to the immutable deployments of pods which can spawn on new nodes every time with a typical deployment. The purpose of this web quest is to understand the storage options, data peristence and dynamic provisioning of data.","title":"KWQ001 - Persistent Volume and Storage Web Quest"},{"location":"wq006-persistentvolume/#task","text":"Your task is to work in your group, research on the storage systems and persistent volumes that kubernetes is compatible with and make recommendations. You will try to find answers to the following questions and present on the topic briefly. Questions : Explain the concept of dynamic provisioning Which are the storage classes that have internal provisioners ? Which storage solution would you use in your organisation ? Why? How would your storage and provisioning system ? Explain briefly","title":"Task"},{"location":"wq006-persistentvolume/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq006-persistentvolume/#resources","text":"doc: Storage Classes eBook: State of kubernetes ecosystem Article: The Kubernetes Storage Ecosystem Video: Persistent Storage with Kubernetes in Production - Which Solution and Why? Video: Kubernetes Storage Lingo 101","title":"Resources"},{"location":"wq007-maintenance/","text":"KWQ001 - Cluster Maintenance Security Web Quest Kubernetes administrators need to be well aware of the cluster maintenance and administration tasks. With this web quest, you would research on the Task Your task is to work in your group, research on cluster maintenance activities in depth, and try to find answers to the following questions. You will present on the topic briefly. Questions : What are the cluster maintenance and administration tasks that you anticipate doing as part of your job ? Briefly explain how you could do the following Prepare nodes for maintenence Define resource quotas Create users and groups Resize a cluster Upgrade the version of kubernetes on the nodes Process Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above. Resources doc: Cluster Management","title":"KWQ007 - Cluster Maintenance Web Quest"},{"location":"wq007-maintenance/#kwq001-cluster-maintenance-security-web-quest","text":"Kubernetes administrators need to be well aware of the cluster maintenance and administration tasks. With this web quest, you would research on the","title":"KWQ001 - Cluster Maintenance Security Web Quest"},{"location":"wq007-maintenance/#task","text":"Your task is to work in your group, research on cluster maintenance activities in depth, and try to find answers to the following questions. You will present on the topic briefly. Questions : What are the cluster maintenance and administration tasks that you anticipate doing as part of your job ? Briefly explain how you could do the following Prepare nodes for maintenence Define resource quotas Create users and groups Resize a cluster Upgrade the version of kubernetes on the nodes","title":"Task"},{"location":"wq007-maintenance/#process","text":"Following is the process you could loosely follow, Start researching on web and reading individually about the topic. Reference the links and articles in the resources section while you do so. Reflect on how it could apply to your work environment. Speak with with the relevant teams in your organisation if needed gain insights into current implementation, what are the issues being faced. Think about how you could you apply the knowledge you have researched on. Have a group discussion with within your group, exchange what you have learnt as well as your experience with the topic in your prior/existing projects. You may come up with the notes and common points you would like to present. Prepare a short 5/10 minutes presentation to explain the topic and answer the questions given in the task section above.","title":"Process"},{"location":"wq007-maintenance/#resources","text":"doc: Cluster Management","title":"Resources"},{"location":"xer-001-pods/","text":"XER001 - Pods 1 Create a Pod manifest which uses \"ghost\" image and open port 2368. apiVersion: v1 kind: Pod metadata: name: ghost spec: containers: - image: xxx name: ghost ports: - containerPort: xxx hostPort: xxx Get the name of the Node in which the Pod is scheduled by running, kubectl describe pod [output] Name: Namespace: default Node: / Start Time: Wed, xx May 201x 15:59:29 +0530 Try to access the application on the host's port 2368. curl :2368 Reference : Ghost Docker image 2 Create a Pod with ubuntu:trusty image and a command to echo \u201cYOUR_NAME\u201d which overrides the default CMD/ENTRYPOINT of the image. Reference : Define command argument in a Pod 3 Apply the following Pod manifest and read the error. Fix it by editing it. apiVersion: v1apps/beta1 kind: Pod metadata: name: mogambo-frontend label: role: frontend spec: containers: - name: frontend image: schoolofdevops/frontend:orange ports: - containerName: web Port: 8079 protocol: TCP Reference : Debugging a unscheduled Pod 4 A Pod with the following pod always crashes with CrashLoopBackOff error. How would you fix it? image: schoolofdevops/nginx:break ports: 80 Reference : Debugging a crashed Pod 5 You are running a Pod with hostPort option. Your Pod status stays \u201cpending\u201d. What could be the issue for the Pod not being scheduled? Reference : Debugging a unscheduled Pod 6 The given manifest for multi-container pod is not working as intended. It does not sync the content between containers like expected. What could be the issue? Find the issue just by reading the manifest. apiVersion: v1 kind: Pod metadata: name: web labels: tier: front app: nginx role: ui spec: containers: - name: nginx image: nginx:stable-alpine ports: - containerPort: 80 protocol: TCP volumeMounts: - name: data mountPath: /var/www/html-sample-app - name: sync image: schoolofdevops/sync:v2 volumeMounts: - name: datanew mountPath: /var/www/app volumes: - name: data emptyDir: {} 7 For the above given manifest, the following command is not working. What could be the issue? kubeclt exec -it web -sh -c synch 8 Try to apply the following manifest. If fails, try to debug. apiVersion: v1 kind: Pod metadata: name: web labels: app: role: role spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP 9 Fix the following manifest. Don't apply it. Just fix it by reading. apiVersion: v1 kind: pod metadata: name: web labels: role: role spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerport: 8080 protocol: TCP 10 Mount /var/www/html from Pod using the follwing manifest. Fill the missing fields. apiVersion: v1 kind: Pod metadata: name: web labels: role: role spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP volumes: - name: roboshop-storage emptyDir: {} 11 Write a Pod manifest with the image nginx which has a volume that mounts /etc/nginx/ directory. Use \"hostPath\" volume type. apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - image: nginx name: nginx ports: - containerPort: 80 volumeMounts: xxx Reference : Volumes","title":"XER001 - Pods"},{"location":"xer-001-pods/#xer001-pods","text":"","title":"XER001 - Pods"},{"location":"xer-001-pods/#1","text":"Create a Pod manifest which uses \"ghost\" image and open port 2368. apiVersion: v1 kind: Pod metadata: name: ghost spec: containers: - image: xxx name: ghost ports: - containerPort: xxx hostPort: xxx Get the name of the Node in which the Pod is scheduled by running, kubectl describe pod [output] Name: Namespace: default Node: / Start Time: Wed, xx May 201x 15:59:29 +0530 Try to access the application on the host's port 2368. curl :2368 Reference : Ghost Docker image","title":"1"},{"location":"xer-001-pods/#2","text":"Create a Pod with ubuntu:trusty image and a command to echo \u201cYOUR_NAME\u201d which overrides the default CMD/ENTRYPOINT of the image. Reference : Define command argument in a Pod","title":"2"},{"location":"xer-001-pods/#3","text":"Apply the following Pod manifest and read the error. Fix it by editing it. apiVersion: v1apps/beta1 kind: Pod metadata: name: mogambo-frontend label: role: frontend spec: containers: - name: frontend image: schoolofdevops/frontend:orange ports: - containerName: web Port: 8079 protocol: TCP Reference : Debugging a unscheduled Pod","title":"3"},{"location":"xer-001-pods/#4","text":"A Pod with the following pod always crashes with CrashLoopBackOff error. How would you fix it? image: schoolofdevops/nginx:break ports: 80 Reference : Debugging a crashed Pod","title":"4"},{"location":"xer-001-pods/#5","text":"You are running a Pod with hostPort option. Your Pod status stays \u201cpending\u201d. What could be the issue for the Pod not being scheduled? Reference : Debugging a unscheduled Pod","title":"5"},{"location":"xer-001-pods/#6","text":"The given manifest for multi-container pod is not working as intended. It does not sync the content between containers like expected. What could be the issue? Find the issue just by reading the manifest. apiVersion: v1 kind: Pod metadata: name: web labels: tier: front app: nginx role: ui spec: containers: - name: nginx image: nginx:stable-alpine ports: - containerPort: 80 protocol: TCP volumeMounts: - name: data mountPath: /var/www/html-sample-app - name: sync image: schoolofdevops/sync:v2 volumeMounts: - name: datanew mountPath: /var/www/app volumes: - name: data emptyDir: {}","title":"6"},{"location":"xer-001-pods/#7","text":"For the above given manifest, the following command is not working. What could be the issue? kubeclt exec -it web -sh -c synch","title":"7"},{"location":"xer-001-pods/#8","text":"Try to apply the following manifest. If fails, try to debug. apiVersion: v1 kind: Pod metadata: name: web labels: app: role: role spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP","title":"8"},{"location":"xer-001-pods/#9","text":"Fix the following manifest. Don't apply it. Just fix it by reading. apiVersion: v1 kind: pod metadata: name: web labels: role: role spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerport: 8080 protocol: TCP","title":"9"},{"location":"xer-001-pods/#10","text":"Mount /var/www/html from Pod using the follwing manifest. Fill the missing fields. apiVersion: v1 kind: Pod metadata: name: web labels: role: role spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP volumes: - name: roboshop-storage emptyDir: {}","title":"10"},{"location":"xer-001-pods/#11","text":"Write a Pod manifest with the image nginx which has a volume that mounts /etc/nginx/ directory. Use \"hostPath\" volume type. apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - image: nginx name: nginx ports: - containerPort: 80 volumeMounts: xxx Reference : Volumes","title":"11"},{"location":"xer-002-rs/","text":"XER002 - Replication Controller and ReplicaSets 1 The following replication controller manifest has some bugs in it. Fix them. apiVersion: v1/beta1 kind: ReplicationController metadata: name: loop spec: replicas: 3 selecter: app: loop templates: metadata: name: loop labels: app: loop specs: container: - name: loop image: schoolofdevops/loop Reference : Replication Controller 2 Create Pods using following manifests. Each Pod has different values for the Label \"platform\". You should create a ReplicaSet with 4 replicas, which also selects both Pods based on \"platform\" Label. Tip: You may have to use matchExpressions attribute in your ReplicaSet. file: sync-aws-po.yml apiVersion: v1 kind: Pod metadata: name: sync-aws labels: version: 2 platform: aws spec: containers: - name: sync image: schoolofdevops/sync:v2 file: sync-gcp-po.yml apiVersion: v1 kind: Pod metadata: name: sync-gcp labels: version: 2 platforms: gcp spec: containers: - name: sync image: schoolofdevops/sync:v2 3 The following manifest is working as intended. Try to debug. apiVersion: v1 kind: ReplicationController metadata: name: web labels: role: role spec: replicas: 3 selector: app: robotshop template: metadata: name: robotshop labels: app: robotshop containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP 4 How do you get the logs from all pods of a Replication Controller? Reference : logs from all pods in a RC 5 The Pods from a Replication Controller has stuck in Terminating status and doesn't actually get deleted. How do you delete these Pods.? Reference : Delete Pods forcefully 6 How do you force pulling an image again with the same tag? apiVersion: v1 kind: Pod metadata: name: kuard spec: containers: - image: gcr.io/kuar-demo/kuard-amd64:1 name: kuard ports: - containerPort: 8080 name: http protocol: TCP Reference : Force image pull 7 When I try to apply the following Pod manifest, I am getting image :latest not found . How would you fix it? Tip : This image is in a private registry. apiVersion: v1 kind: ReplicationController metadata: name: web labels: role: role spec: replicas: 3 selector: app: robotshop template: metadata: name: robotshop labels: app: robotshop containers: - name: web image: my-private-registry/robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP Reference : Pulling images from private registry 8 Launch the following Replication Controller in Prod namespace and use kubectl to scale the replicas from 3 to 6. apiVersion: v1 kind: ReplicationController metadata: name: web labels: role: role spec: replicas: 3 selector: app: robotshop template: metadata: name: robotshop labels: app: robotshop spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP 9 Pod that I've scheduled is in waiting state. How would I debug this issue? Reference : My pod stays waiting","title":"XER002 - ReplicaSets"},{"location":"xer-002-rs/#xer002-replication-controller-and-replicasets","text":"","title":"XER002 - Replication Controller and ReplicaSets"},{"location":"xer-002-rs/#1","text":"The following replication controller manifest has some bugs in it. Fix them. apiVersion: v1/beta1 kind: ReplicationController metadata: name: loop spec: replicas: 3 selecter: app: loop templates: metadata: name: loop labels: app: loop specs: container: - name: loop image: schoolofdevops/loop Reference : Replication Controller","title":"1"},{"location":"xer-002-rs/#2","text":"Create Pods using following manifests. Each Pod has different values for the Label \"platform\". You should create a ReplicaSet with 4 replicas, which also selects both Pods based on \"platform\" Label. Tip: You may have to use matchExpressions attribute in your ReplicaSet. file: sync-aws-po.yml apiVersion: v1 kind: Pod metadata: name: sync-aws labels: version: 2 platform: aws spec: containers: - name: sync image: schoolofdevops/sync:v2 file: sync-gcp-po.yml apiVersion: v1 kind: Pod metadata: name: sync-gcp labels: version: 2 platforms: gcp spec: containers: - name: sync image: schoolofdevops/sync:v2","title":"2"},{"location":"xer-002-rs/#3","text":"The following manifest is working as intended. Try to debug. apiVersion: v1 kind: ReplicationController metadata: name: web labels: role: role spec: replicas: 3 selector: app: robotshop template: metadata: name: robotshop labels: app: robotshop containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP","title":"3"},{"location":"xer-002-rs/#4","text":"How do you get the logs from all pods of a Replication Controller? Reference : logs from all pods in a RC","title":"4"},{"location":"xer-002-rs/#5","text":"The Pods from a Replication Controller has stuck in Terminating status and doesn't actually get deleted. How do you delete these Pods.? Reference : Delete Pods forcefully","title":"5"},{"location":"xer-002-rs/#6","text":"How do you force pulling an image again with the same tag? apiVersion: v1 kind: Pod metadata: name: kuard spec: containers: - image: gcr.io/kuar-demo/kuard-amd64:1 name: kuard ports: - containerPort: 8080 name: http protocol: TCP Reference : Force image pull","title":"6"},{"location":"xer-002-rs/#7","text":"When I try to apply the following Pod manifest, I am getting image :latest not found . How would you fix it? Tip : This image is in a private registry. apiVersion: v1 kind: ReplicationController metadata: name: web labels: role: role spec: replicas: 3 selector: app: robotshop template: metadata: name: robotshop labels: app: robotshop containers: - name: web image: my-private-registry/robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP Reference : Pulling images from private registry","title":"7"},{"location":"xer-002-rs/#8","text":"Launch the following Replication Controller in Prod namespace and use kubectl to scale the replicas from 3 to 6. apiVersion: v1 kind: ReplicationController metadata: name: web labels: role: role spec: replicas: 3 selector: app: robotshop template: metadata: name: robotshop labels: app: robotshop spec: containers: - name: web image: robotshop/rs-web:latest ports: - containerPort: 8080 protocol: TCP","title":"8"},{"location":"xer-002-rs/#9","text":"Pod that I've scheduled is in waiting state. How would I debug this issue? Reference : My pod stays waiting","title":"9"},{"location":"deprecated/2_kube_cluster_vagrant/","text":"Install VirtualBox and Vagrant TOOL VERSION LINK VirtualBox 5.1.26 https://www.virtualbox.org/wiki/Downloads Vagrant 1.9.7 https://www.vagrantup.com/downloads.html Importing a VM Template If you have already copied/downloaded the box file ubuntu-xenial64.box , go to the directory which contains that file. If you do not have a box file, skip to next section. vagrant box list vagrant box add ubuntu/xenial64 ubuntu-xenial64.box vagrant box list Provisioning Vagrant Nodes Clone repo if not already git clone https://github.com/schoolofdevops/lab-setup.git Launch environments with Vagrant cd lab-setup/kubernetes/vagrant-kube-cluster vagrant up Login to nodes Open three different terminals to login to 3 nodes created with above command Terminal 1 vagrant ssh kube-01 sudo su Terminal 2 vagrant ssh kube-02 sudo su Terminal 3 vagrant ssh kube-03 sudo su Once the environment is setup, follow Initialization of Master onwards from the following tutorial https://github.com/schoolofdevops/kubernetes-fundamentals/blob/master/tutorials/1.%20install_kubernetes.md","title":"Install VirtualBox and Vagrant"},{"location":"deprecated/2_kube_cluster_vagrant/#install-virtualbox-and-vagrant","text":"TOOL VERSION LINK VirtualBox 5.1.26 https://www.virtualbox.org/wiki/Downloads Vagrant 1.9.7 https://www.vagrantup.com/downloads.html","title":"Install VirtualBox and Vagrant"},{"location":"deprecated/2_kube_cluster_vagrant/#importing-a-vm-template","text":"If you have already copied/downloaded the box file ubuntu-xenial64.box , go to the directory which contains that file. If you do not have a box file, skip to next section. vagrant box list vagrant box add ubuntu/xenial64 ubuntu-xenial64.box vagrant box list","title":"Importing a VM Template"},{"location":"deprecated/2_kube_cluster_vagrant/#provisioning-vagrant-nodes","text":"Clone repo if not already git clone https://github.com/schoolofdevops/lab-setup.git Launch environments with Vagrant cd lab-setup/kubernetes/vagrant-kube-cluster vagrant up Login to nodes Open three different terminals to login to 3 nodes created with above command Terminal 1 vagrant ssh kube-01 sudo su Terminal 2 vagrant ssh kube-02 sudo su Terminal 3 vagrant ssh kube-03 sudo su Once the environment is setup, follow Initialization of Master onwards from the following tutorial https://github.com/schoolofdevops/kubernetes-fundamentals/blob/master/tutorials/1.%20install_kubernetes.md","title":"Provisioning Vagrant Nodes"},{"location":"deprecated/kube_visualizer/","text":"Kubernetes Visualizer In this chapter we will see how to set up kubernetes visualizer that will show us the changes in our cluster in real time. Set up Fork the repository and deploy the visualizer on kubernetes git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ [Sample Output] serviceaccount \"kube-ops-view\" created clusterrole \"kube-ops-view\" created clusterrolebinding \"kube-ops-view\" created deployment \"kube-ops-view\" created ingress \"kube-ops-view\" created deployment \"kube-ops-view-redis\" created service \"kube-ops-view-redis\" created service \"kube-ops-view\" created Get the nodeport for the service. kubectl get svc [output] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-ops-view NodePort 10.107.204.74 80:**30073**/TCP 1m kube-ops-view-redis ClusterIP 10.104.50.176 6379/TCP 1m kubernetes ClusterIP 10.96.0.1 443/TCP 8m In my case, port 30073 is the nodeport. Visit the port from the browser. You could add /#scale=2.0 or similar option where 2.0 = 200% the scale. http:///#scale=2.0","title":"Kube visualizer"},{"location":"deprecated/kube_visualizer/#kubernetes-visualizer","text":"In this chapter we will see how to set up kubernetes visualizer that will show us the changes in our cluster in real time.","title":"Kubernetes Visualizer"},{"location":"deprecated/kube_visualizer/#set-up","text":"Fork the repository and deploy the visualizer on kubernetes git clone https://github.com/schoolofdevops/kube-ops-view kubectl apply -f kube-ops-view/deploy/ [Sample Output] serviceaccount \"kube-ops-view\" created clusterrole \"kube-ops-view\" created clusterrolebinding \"kube-ops-view\" created deployment \"kube-ops-view\" created ingress \"kube-ops-view\" created deployment \"kube-ops-view-redis\" created service \"kube-ops-view-redis\" created service \"kube-ops-view\" created Get the nodeport for the service. kubectl get svc [output] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-ops-view NodePort 10.107.204.74 80:**30073**/TCP 1m kube-ops-view-redis ClusterIP 10.104.50.176 6379/TCP 1m kubernetes ClusterIP 10.96.0.1 443/TCP 8m In my case, port 30073 is the nodeport. Visit the port from the browser. You could add /#scale=2.0 or similar option where 2.0 = 200% the scale. http:///#scale=2.0","title":"Set up"},{"location":"istio/labs/","text":"Power of Istio - Labs Concepts: Traffic Management : Explains whats Virtual Services, Destination Rules, Gateways, Service Entries, Sidecars Lab: Request Routing Request Routing Lab Steps : * Create virtual services for all apps * Route all traffic to v1 * Route traffic to v2 only for jason user Lab: Fault Injection Fault Injection Lab Steps * inject a 7s delay between reviews/v2 || ratings service, only for user jason. Analyse what happens and possible solutions. * inject http abort fault between reviews/v2 || rating service, for user jason. Lab: Traffic Shifting (A/B Testing) A/B Testing Lab Steps: * Apply default rule to send 100% traffic to v1 * Add weights so that 50% goes to v3 (needs 15 refreshs or more for proper distribution of traffic) * Shift 100% traffic to v3 * Refer here for Canary Releases Lab: Observability with Prometheus and Grafana Prometheus Lab Grafana Lab Access Prometheus and Grafana using NodePort Query and observe metrics generated by istio Observe dashboards created by Grafana Lab : Log Collection Log Collection Lab Components: * instance * handler * rule","title":"Power of Istio - Labs"},{"location":"istio/labs/#power-of-istio-labs","text":"","title":"Power of Istio - Labs"},{"location":"istio/labs/#concepts","text":"Traffic Management : Explains whats Virtual Services, Destination Rules, Gateways, Service Entries, Sidecars","title":"Concepts:"},{"location":"istio/labs/#lab-request-routing","text":"Request Routing Lab Steps : * Create virtual services for all apps * Route all traffic to v1 * Route traffic to v2 only for jason user","title":"Lab: Request Routing"},{"location":"istio/labs/#lab-fault-injection","text":"Fault Injection Lab Steps * inject a 7s delay between reviews/v2 || ratings service, only for user jason. Analyse what happens and possible solutions. * inject http abort fault between reviews/v2 || rating service, for user jason.","title":"Lab: Fault Injection"},{"location":"istio/labs/#lab-traffic-shifting-ab-testing","text":"A/B Testing Lab Steps: * Apply default rule to send 100% traffic to v1 * Add weights so that 50% goes to v3 (needs 15 refreshs or more for proper distribution of traffic) * Shift 100% traffic to v3 * Refer here for Canary Releases","title":"Lab: Traffic Shifting (A/B Testing)"},{"location":"istio/labs/#lab-observability-with-prometheus-and-grafana","text":"Prometheus Lab Grafana Lab Access Prometheus and Grafana using NodePort Query and observe metrics generated by istio Observe dashboards created by Grafana","title":"Lab: Observability with Prometheus and Grafana"},{"location":"istio/labs/#lab-log-collection","text":"Log Collection Lab Components: * instance * handler * rule","title":"Lab : Log Collection"},{"location":"istio/setup/","text":"Setting up Istio Service Mesh on Kubernetes If you already have prometheus and grafana configured, clean it up as istio will come with its own monitoring setup. Assuming you have installed helm version 3, helm list [sample output] NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION grafana instavote 1 2020-02-15 06:55:38.895788799 +0000 UTC deployed grafana-4.6.3 6.6.0 prometheus instavote 1 2020-02-15 06:54:00.177788896 +0000 UTC deployed prometheus-10.4.0 2.15.2 Uninstall prometheus and grafana as, helm uninstall prometheus helm uninstall grafana Install Istio Control Plane Download and install version 1.4.4. of istio using istioctl utility as, cd ~ curl -L https://istio.io/downloadIstio | sh - cd istio-1.4.4/ export PATH=$PWD/bin:$PATH echo \"export PATH=$PWD/bin:$PATH\" >> ~/.bashrc Install istio with demo configuration profile. This is a quick start profile. You could customize the profile and apply it later. istioctl manifest apply --set profile=demo validate kubectl get pods -n istio-system [Sample Output] NAME READY STATUS RESTARTS AGE grafana-6b65874977-8mlrq 1/1 Running 0 6h26m istio-citadel-58c44d6964-9vsl2 1/1 Running 0 6h26m istio-egressgateway-74b6995559-w6hx8 1/1 Running 0 6h26m istio-galley-5bf54d9674-plq6j 1/1 Running 0 6h26m istio-ingressgateway-7f4b56656f-cs6gw 1/1 Running 0 6h26m istio-pilot-568855fb7-lbt2v 1/1 Running 0 6h26m istio-policy-fdb9d6845-prmb5 1/1 Running 4 6h26m istio-sidecar-injector-56b7656fd8-ws7hr 1/1 Running 0 6h26m istio-telemetry-676db4d695-8kmgh 1/1 Running 3 6h26m istio-tracing-c66d67cd9-db2bp 1/1 Running 0 6h26m kiali-8559969566-sjrqm 1/1 Running 0 6h26m prometheus-66c5887c86-vpbnk 1/1 Running 0 6h26m As you notice, Istio is been installed with all components including prometheus and grafana in its own istio-system namespace. Nano Project: Expose istio services with NodePort kubectl get svc -n istio-system [sample output] grafana ClusterIP 10.108.218.221 3000/TCP 6h27m istio-citadel ClusterIP 10.99.97.53 8060/TCP,15014/TCP 6h27m istio-egressgateway ClusterIP 10.101.104.101 80/TCP,443/TCP,15443/TCP 6h27m istio-galley ClusterIP 10.100.6.131