Skip to content

Latest commit

 

History

History
220 lines (197 loc) · 13 KB

README.md

File metadata and controls

220 lines (197 loc) · 13 KB
Raw

GKE nodepool module

This module allows simplified creation and management of individual GKE nodepools, setting sensible defaults (eg a service account is created for nodes if none is set) and allowing for less verbose usage in most use cases.

Example usage

Module defaults

If no specific node configuration is set via variables, the module uses the provider's defaults only setting OAuth scopes to a minimal working set and the node machine type to n1-standard-1. The service account set by the provider in this case is the GCE default service account.

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
}
# tftest modules=1 resources=1 inventory=basic.yaml

Internally managed service account

There are three different approaches to defining the nodes service account, all depending on the service_account variable where the create attribute controls creation of a new service account by this module, and the email attribute controls the actual service account to use.

If you create a new service account, its resource and email (in both plain and IAM formats) are then available in outputs to reference it in other modules or resources.

GCE default service account

To use the GCE default service account, you can ignore the variable which is equivalent to { create = null, email = null }. This is what the first example of this document does.

Externally defined service account

To use an existing service account, pass in just the email attribute. If you do this, will most likely want to use the cloud-platform scope.

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
  service_account = {
    email        = "[email protected]"
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
}
# tftest modules=1 resources=1 inventory=external-sa.yaml

Auto-created service account

To have the module create a service account, set the create attribute to true and optionally pass the desired account id in email.

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
  service_account = {
    create       = true
    email        = "spam-eggs" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
}
# tftest modules=1 resources=2 inventory=create-sa.yaml

Node & node pool configuration

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
  k8s_labels   = { environment = "dev" }
  service_account = {
    create       = true
    email        = "nodepool-1" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
  node_config = {
    machine_type        = "n2-standard-2"
    disk_size_gb        = 50
    disk_type           = "pd-ssd"
    ephemeral_ssd_count = 1
    gvnic               = true
    spot                = true
  }
  nodepool_config = {
    autoscaling = {
      max_node_count = 10
      min_node_count = 1
    }
    management = {
      auto_repair  = true
      auto_upgrade = false
    }
  }
}
# tftest modules=1 resources=2 inventory=config.yaml

GPU Node & node pool configuration

module "cluster-1-nodepool-gpu-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west4-a"
  name         = "nodepool-gpu-1"
  k8s_labels   = { environment = "dev" }
  service_account = {
    create       = true
    email        = "nodepool-gpu-1" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
  node_config = {
    machine_type        = "a2-highgpu-1g"
    disk_size_gb        = 50
    disk_type           = "pd-ssd"
    ephemeral_ssd_count = 1
    gvnic               = true
    spot                = true
    guest_accelerator = {
      type  = "nvidia-tesla-a100"
      count = 1
      gpu_driver = {
        version = "LATEST"
      }
    }
  }
}
# tftest modules=1 resources=2 inventory=guest-accelerator.yaml

Dynamic Workload Scheduler (DWS) & node pool configuration

This example uses Dynamic Workload Scheduler (DWS) to configure a GPU nodepool.

module "cluster-1-nodepool-dws" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west4-a"
  name         = "nodepool-dws"
  k8s_labels   = { environment = "dev" }
  service_account = {
    create       = true
    email        = "nodepool-gpu-1" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
  node_config = {
    machine_type        = "g2-standard-4"
    disk_size_gb        = 50
    disk_type           = "pd-ssd"
    ephemeral_ssd_count = 1
    gvnic               = true
    spot                = true
    guest_accelerator = {
      type  = "nvidia-l4"
      count = 1
      gpu_driver = {
        version = "LATEST"
      }
    }
  }
  nodepool_config = {
    autoscaling = {
      max_node_count = 10
      min_node_count = 0
    }
    queued_provisioning = true
  }
  node_count = {
    initial = 0
  }
  reservation_affinity = {
    consume_reservation_type = "NO_RESERVATION"
  }
}
# tftest modules=1 resources=2 inventory=dws.yaml

Variables

name description type required default
cluster_name Cluster name. string ✓
location Cluster location. string ✓
project_id Cluster project id. string ✓
cluster_id Cluster id. Optional, but providing cluster_id is recommended to prevent cluster misconfiguration in some of the edge cases. string null
gke_version Kubernetes nodes version. Ignored if auto_upgrade is set in management_config. string null
k8s_labels Kubernetes labels applied to each node. map(string) {}
labels The resource labels to be applied each node (vm). map(string) {}
max_pods_per_node Maximum number of pods per node. number null
name Optional nodepool name. string null
node_config Node-level configuration. object({…}) {…}
node_count Number of nodes per instance group. Initial value can only be changed by recreation, current is ignored when autoscaling is used. object({…}) {…}
node_locations Node locations. list(string) null
nodepool_config Nodepool-level configuration. object({…}) null
pod_range Pod secondary range configuration. object({…}) null
reservation_affinity Configuration of the desired reservation which instances could take capacity from. object({…}) null
service_account Nodepool service account. If this variable is set to null, the default GCE service account will be used. If set and email is null, a service account will be created. If scopes are null a default will be used. object({…}) {}
sole_tenant_nodegroup Sole tenant node group. string null
tags Network tags applied to nodes. list(string) null
taints Kubernetes taints applied to all nodes. map(object({…})) {}

Outputs

name description sensitive
id Fully qualified nodepool id.
name Nodepool name.
service_account_email Service account email.
service_account_iam_email Service account email.