how-to-use-k8s-with-cri-containerd-and-kata.md

How to use Kata Containers and CRI (containerd plugin) with Kubernetes

This document describes how to set up a single-machine Kubernetes (k8s) cluster.

The Kubernetes cluster will use the CRI containerd and Kata Containers to launch untrusted workloads.

Requirements

• Kubernetes, Kubelet, kubeadm
• containerd with cri plug-in
• Kata Containers

Note: For information about the supported versions of these components, see the Kata Containers versions.yaml file.

Install and configure containerd

First, follow the How to use Kata Containers and Containerd to install and configure containerd. Then, make sure the containerd works with the examples in it.

Install and configure Kubernetes

Install Kubernetes

• Follow the instructions for kubeadm installation.

• Check kubeadm is now available

  $ command -v kubeadm

Configure Kubelet to use containerd

In order to allow Kubelet to use containerd (using the CRI interface), configure the service to point to the containerd socket.

• Configure Kubernetes to use containerd

  $ sudo mkdir -p  /etc/systemd/system/kubelet.service.d/
  $ cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
  [Service]                                                 
  Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
  EOF

• Inform systemd about the new configuration

  $ sudo systemctl daemon-reload

Configure HTTP proxy - OPTIONAL

If you are behind a proxy, use the following script to configure your proxy for docker, Kubelet, and containerd:

$ services="
kubelet
containerd
docker
"

$ for service in ${services}; do

    service_dir="/etc/systemd/system/${service}.service.d/"
    sudo mkdir -p ${service_dir}

    cat << EOF | sudo tee "${service_dir}/proxy.conf"
[Service]
Environment="HTTP_PROXY=${http_proxy}"
Environment="HTTPS_PROXY=${https_proxy}"
Environment="NO_PROXY=${no_proxy}"
EOF
done

$ sudo systemctl daemon-reload

Start Kubernetes

• Make sure containerd is up and running

  $ sudo systemctl restart containerd
  $ sudo systemctl status containerd

• Prevent conflicts between docker iptables (packet filtering) rules and k8s pod communication

  If Docker is installed on the node, it is necessary to modify the rule below. See kubernetes/kubernetes#40182 for further details.

  $ sudo iptables -P FORWARD ACCEPT

• Start cluster using kubeadm

  $ sudo kubeadm init --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
  $ export KUBECONFIG=/etc/kubernetes/admin.conf
  $ sudo -E kubectl get nodes
  $ sudo -E kubectl get pods

Configure Pod Network

A pod network plugin is needed to allow pods to communicate with each other. You can find more about CNI plugins from the Creating a cluster with kubeadm guide.

By default the CNI plugin binaries is installed under /opt/cni/bin (in package kubernetes-cni), you only need to create a configuration file for CNI plugin.

$ sudo -E mkdir -p /etc/cni/net.d

$ sudo -E cat > /etc/cni/net.d/10-mynet.conf <<EOF
{
  "cniVersion": "0.2.0",
  "name": "mynet",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "172.19.0.0/24",
    "routes": [
      { "dst": "0.0.0.0/0" }
    ]
  }
}
EOF

Allow pods to run in the master node

By default, the cluster will not schedule pods in the master node. To enable master node scheduling:

$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-

Create runtime class for Kata Containers

By default, all pods are created with the default runtime configured in CRI containerd plugin. From Kubernetes v1.12, users can use RuntimeClass to specify a different runtime for Pods.

$ cat > runtime.yaml <<EOF
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata
EOF

$ sudo -E kubectl apply -f runtime.yaml

Run pod in Kata Containers

If a pod has the runtimeClassName set to kata, the CRI plugin runs the pod with the Kata Containers runtime.

• Create an pod configuration that using Kata Containers runtime

  $ cat << EOF | tee nginx-kata.yaml
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-kata
  spec:
    runtimeClassName: kata
    containers:
    - name: nginx
      image: nginx
      
  EOF

• Create the pod

  $ sudo -E kubectl apply -f nginx-kata.yaml

• Check pod is running

  $ sudo -E kubectl get pods

• Check hypervisor is running

  $ ps aux | grep qemu

Delete created pod

$ sudo -E kubectl delete -f nginx-kata.yaml