Bootstrap HA GlusterFS Cluster in GCE

• This project bootstraps off-cluster HA GlusterFS Cluster, if you are interested to run HA GlusterFS Cluster in Kubernetes, check out gluster-kubernetes project.

As you know shared file systems are a tricky problem. One solution to that problem is a distributed file system. Something that your apps can read from and write to. When it comes to distributed file systems, GlusterFS is one of the leading products.

With a few simple scripts on your Mac OS X or Linux machine, you can deploy a multi-zone HA GlusterFS cluster to Google Compute Engine (GCE) that provides scalable, persistent shared storage for your GCE or Google Container Engine (GKE) Kubernetes clusters.

By default it is set to three GlusterFS servers, one server per Google Cloud zone in the same chosen region.

Prerequisites

Before continuing, please make sure you have:

• A Google Cloud account
• The Google Cloud SDK installed
• Git installed

Clone this project and set settings:

$ git clone https://github.com/fdonze/glusterfs-gce
$ cd glusterfs-gce/cluster

• Edit the cluster/settings file and set PROJECT, REGION and ZONES, the rest of settings in this file are probably fine, but can be adjusted if need be.

Bootstrap the cluster

$ ./create_cluster.sh

This command will create three servers.

Each server will have:

• A static IP
• The GlusterFS server package installed
• A Google Cloud persistent disk to be used as a GlusterFS brick, that is: storage space made available to the cluster

Install a GlusterFS management interface in GKE

In order to use the GlusterFS cluster just created from a GKE cluster, a GlusterFS management interface and a storage class need to be created. The management interface can be created by installing the Heketi RESTful volume management interface for GlusterFS.

See https://github.com/gluster/gluster-kubernetes for information about the Heketi RESTful volume management interface for GlusterFS.

Heketi Extras

Here are a few helpers for the installation of Heketi.

• cluster/cluster_topology.sh - output topology json content to be used during heketi installation
• install_heketi.sh - heketi installer helper
• glusterfs-storage-class.yaml - glusterfs storage class template
• heketi-endpoint.yaml - expose the heketi REST end point via a load balancer. This endpoint can then be used in the storage class definition.

Heketi installation in a GKE cluster

Generate the GlusterFS topology json file needed by the Heketi management interface.

$ ./cluster_topology.sh > topology.json

Clone the https://github.com/gluster/gluster-kubernetes repository. The heketi installation script gk-deploy is under the gluster-kubernetes/deploy directory. Use the install_heketi.sh helper script to install heketi REST endpoint in the current namespace of your GKE cluster.

$ ./install_heketi.sh PATH_TO_TOPOLOGY_JSON DIRECTORY_HEKETI_GK_DEPLOY_SCRIPT -s YOUR_GOOGLE_GLUSTERFS_INSTANCE_SSH_KEY -v --ssh-user YOUR_GOOGLE_USERNAME
 

Expose the heketi REST interface

$ kubectl create -f heketi-endpoint.yaml

NOTE that the service created a load balancer and made the port 8080 opened for access to all sources. The firewall rule created for port 8080 should be restricted to the GKE cluster master node(s) only.

Get the heketi service EXTERNAL-IP

$ kubectl get svc heketi-endpoint

Edit the gluster-storage-class.yaml and replace the <HEKETI_SERVICE_EXTERNAL_IP> with the EXTERNAL_IP. Then create the storage class.

$ kubectl create -f gluster-storage-class.yamll

Delete the cluster

$ ./delete_cluster.sh

This command will delete the whole GlusterFS cluster (nodes and disks).