Skip to content
This repository has been archived by the owner on May 14, 2024. It is now read-only.

SettingUpAKubernetesCluster

Jump to bottom
oliver-tarrant-tessella edited this page Apr 25, 2019 · 6 revisions

Setup a Kubernetes cluster

This guide describes how to provision a small Kubernetes cluster on a single machine for development purposes.

Requirements

Provisioning

In a console on your local machine, navigate to the root of your cloned repo and run:

vagrant up

This creates 3 Virtual Machines (VMs) running CentOS 7:

  • one named master-k8s, configured as the Kubernetes master node
  • two named node1-k8s and node2-k8s, configured as the Kubernetes worker nodes

These machines are provisioned via Ansible and then connected via Weave to produce a running Kubernetes cluster.

Helm is also installed onto the Kubernetes master. This is a package manager for Kubernetes clusters.

To confirm that the cluster is fully connected, SSH into the master VM and ask Kubelet for the cluster status:

vagrant ssh master-k8s
kubectl get nodes

The response should look like this:

NAME         STATUS   ROLES    AGE   VERSION
master-k8s   Ready    master   1h    v1.13.2
node1-k8s    Ready    <none>   1h    v1.13.2
node2-k8s    Ready    <none>   1h    v1.13.2

Nodes may take a few minutes to become ready (their status will show as NotReady before then) so you may need to wait a little while after first booting up the system.

Configure for spark applications

To configure our Kubernetes cluster to run spark applications we will use helm to install a spark-operator.

From within your master-k8s VM navigate to ~/Kubernetes and run:

./install_spark_operator.sh

To check the installation was completed successfully run kubectl get pods --all-namespaces and check you have a pod containing the name spark-operator that has the status running. Now your machine is provisioned to run spark applications as outlined in Running_examples.md.

The cluster can be configured further to allow extraction of metrics about the cluster and from spark applications by following the instructions in monitoring.

Note certain jobs will require setting up a local docker registry which can be achieved by following the instructions in Setting up a docker registory.

Before deploying the web app you will also be required to install a ingress controller on the Kubernetes master. The process for doing this can be found here.

When done with the cluster run logout to exit the VM. To stop the machines run vagrant halt, to destroy the machines vagrant destroy.

Clone this wiki locally