From b83594ee8a04e35e899a9356737649742e5d63d5 Mon Sep 17 00:00:00 2001 From: Adam Dyess Date: Tue, 11 Apr 2023 14:09:21 -0500 Subject: [PATCH 1/6] Update AWS Integration page to include cloud-provider and k8s-storage --- pages/k8s/aws-integration.md | 110 +++++++++++++++++++++++++++++++---- 1 file changed, 98 insertions(+), 12 deletions(-) diff --git a/pages/k8s/aws-integration.md b/pages/k8s/aws-integration.md index 60b5c8c9..e463b40b 100644 --- a/pages/k8s/aws-integration.md +++ b/pages/k8s/aws-integration.md @@ -14,8 +14,8 @@ toc: False --- **Charmed Kubernetes** will run seamlessly on AWS. With the addition of the -`aws-integrator`, your cluster will also be able to directly use AWS native -features. +`aws-integrator` and its companion charms, your cluster will also be able +to directly use AWS native features. ## AWS integrator @@ -25,31 +25,73 @@ AWS. Using the credentials provided to **Juju**, it acts as a proxy between Charmed Kubernetes and the underlying cloud, granting permissions to dynamically create, for example, EBS volumes. +## AWS K8S Storage + +The `aws-k8s-storage` moves the AWS specific functions of the EBS csi-driver +out-of-tree. Using this charm, the drivers are installed as workloads in the kubernetes +cluster instead of as natural code paths of the kubernetes binaries. + +## AWS Cloud Provider + +The `aws-cloud-provider` moves the AWS specific functions of the cloud-provider +out-of-tree. The AWS cloud provider provides the interface between a Kubernetes cluster +and AWS service APIs. This project allows a Kubernetes cluster to provision, +monitor and remove AWS resources necessary for operation of the cluster. + ### Installing If you install **Charmed Kubernetes** [using the Juju bundle][install], you can add the -aws-integrator at the same time by using the following overlay file ([download +aws-integrator at the same time by using the following cloud-provider overlay file ([download it here][asset-aws-overlay]): ```yaml description: Charmed Kubernetes overlay to add native AWS support. applications: aws-integrator: - annotations: - gui-x: "600" - gui-y: "300" charm: aws-integrator num_units: 1 trust: true + aws-cloud-provider: + charm: aws-cloud-provider relations: - ['aws-integrator', 'kubernetes-control-plane'] - ['aws-integrator', 'kubernetes-worker'] - ``` + - ["aws-cloud-provider:certificates", "easyrsa"] + - ["aws-cloud-provider:kube-control", "kubernetes-control-plane"] + - ["aws-cloud-provider:external-cloud-provider", "kubernetes-control-plane"] + - ["aws-cloud-provider:aws-integration", "aws-integrator"] +``` + +As well as the storage overlay file ([download it here][asset-aws-storage-overlay]): + +```yaml +description: Charmed Kubernetes overlay to add native AWS support. +applications: + kubernetes-control-plane: + options: + allow-privileged: "true" + aws-integrator: + charm: aws-integrator + num_units: 1 + trust: true + aws-k8s-storage: + charm: aws-k8s-storage + trust: true + options: + image-registry: public.ecr.aws +relations: +- ['aws-k8s-storage:certificates', 'easyrsa:client'] +- ['aws-k8s-storage:kube-control', 'kubernetes-control-plane:kube-control'] +- ['aws-k8s-storage:aws-integration', 'aws-integrator:aws'] +# Include the following relations if not using the aws-cloud-provider charm +# - ['aws-integrator', 'kubernetes-control-plane'] +# - ['aws-integrator', 'kubernetes-worker'] +``` To use this overlay with the **Charmed Kubernetes** bundle, it is specified during deploy like this: ```bash -juju deploy charmed-kubernetes --overlay ~/path/aws-overlay.yaml --trust +juju deploy charmed-kubernetes --overlay ~/path/aws-overlay.yaml --overlay ~/path/aws-storage-overlay.yaml --trust ``` ... and remember to fetch the configuration file! @@ -61,17 +103,54 @@ juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config For more configuration options and details of the permissions which the integrator uses, please see the [charm readme][aws-integrator-readme]. +#### Beginning in Kubernetes 1.27 + +The in-tree cloud-provider is no longer available, and must be deployed +as container workloads in the cluster. Charmed Kubernetes recommends +using the `aws-cloud-provider` charm to have access to AWS Service APIs. + + +#### Prior to Kubernetes 1.27 + +The in-tree cloud-provider is natively available in Kubernetes until the 1.27 +release, and it is not necessary to deploy the `aws-cloud-provider` charm as in the above +overlay. + ### Using EBS volumes -Many pods you may wish to deploy will require storage. Although you can use +Many pods you may wish to deploy will require storage. Although you can use any type of storage supported by Kubernetes (see the [storage documentation][storage]), you also have the option to use the native AWS storage, Elastic Block Store (EBS). +#### Beginning in Kubernetes 1.25 + +The `aws-k8s-storage` charm will need to be installed to make use of EBS Volumes. +Amazon removed CSIMigration away from the in-tree binaries but made them available +as container workload in the cluster. This charm installs and relates to the +existing integrator charm. + +A StorageClass will be created by this charm named `csi-aws-ebs-default` + +You can confirm this has been added by running: + +```bash +kubectl get sc +``` + +which should return: +```bash +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +csi-aws-ebs-default ebs.csi.aws.com Delete WaitForFirstConsumer false 9s +``` + +#### Prior to Kubernetes 1.25 + First we need to create a storage class which can be used by Kubernetes. To start with, we will create one for the 'General Purpose SSD' type of EBS storage: + ```bash kubectl create -f - < -### Upgrading the integrator-charm +### Upgrading the charms -The aws-integrator is not specifically tied to the version of Charmed Kubernetes installed and may -generally be upgraded at any time with the following command: +The charm `aws-integrator`, `aws-cloud-provider` and `aws-k8s-storage` +can be refreshed within the current charm channel without concern and +can be upgraded at any time with the following command, ```bash juju refresh aws-integrator +juju refresh aws-cloud-provider +juju refresh aws-k8s-storage ``` +It isn't recommended to switch charm channels unless a full charm upgrade is planned. + ### Troubleshooting If you have any specific problems with the aws-integrator, you can report bugs on From c6fbf332bd0c9c0c2f3f6cf9ec69a53d47eee822 Mon Sep 17 00:00:00 2001 From: Adam Dyess Date: Wed, 12 Apr 2023 13:02:56 -0500 Subject: [PATCH 2/6] Update docs for gcp storage --- pages/k8s/aws-integration.md | 6 ++- pages/k8s/gcp-integration.md | 77 ++++++++++++++++++++++++++++++++---- 2 files changed, 73 insertions(+), 10 deletions(-) diff --git a/pages/k8s/aws-integration.md b/pages/k8s/aws-integration.md index e463b40b..c5b11968 100644 --- a/pages/k8s/aws-integration.md +++ b/pages/k8s/aws-integration.md @@ -27,7 +27,7 @@ dynamically create, for example, EBS volumes. ## AWS K8S Storage -The `aws-k8s-storage` moves the AWS specific functions of the EBS csi-driver +The `aws-k8s-storage` charm moves the AWS specific functions of the EBS csi-driver out-of-tree. Using this charm, the drivers are installed as workloads in the kubernetes cluster instead of as natural code paths of the kubernetes binaries. @@ -88,7 +88,8 @@ relations: # - ['aws-integrator', 'kubernetes-worker'] ``` -To use this overlay with the **Charmed Kubernetes** bundle, it is specified during deploy like this: +To use these overlays with the **Charmed Kubernetes** bundle, it is specified +during deploy like this: ```bash juju deploy charmed-kubernetes --overlay ~/path/aws-overlay.yaml --overlay ~/path/aws-storage-overlay.yaml --trust @@ -362,6 +363,7 @@ If you are an AWS user, you may also be interested in how to [asset-aws-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/aws-overlay.yaml +[asset-aws-storage-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/aws-stroage-overlay.yaml [quickstart]: /kubernetes/docs/quickstart [storage]: /kubernetes/docs/storage [ebs-info]: https://aws.amazon.com/ebs/features/ diff --git a/pages/k8s/gcp-integration.md b/pages/k8s/gcp-integration.md index 791e22f1..c9afe4b8 100644 --- a/pages/k8s/gcp-integration.md +++ b/pages/k8s/gcp-integration.md @@ -45,6 +45,13 @@ GCP. Using the credentials provided to Juju, it acts as a proxy between **Charmed Kubernetes** and the underlying cloud, granting permissions to dynamically create, for example, storage volumes. +## GCP K8S Storage + +The `gcp-k8s-storage` charm moves the GCP specific functions of the PD csi-driver +out-of-tree. Using this charm, the drivers are installed as workloads in the kubernetes +cluster instead of as natural code paths of the kubernetes binaries. + + ### Installing If you install **Charmed Kubernetes** [using the Juju bundle][install], you can add the @@ -55,9 +62,6 @@ it here][asset-gcp-overlay]): description: Charmed Kubernetes overlay to add native GCP support. applications: gcp-integrator: - annotations: - gui-x: "600" - gui-y: "300" charm: gcp-integrator num_units: 1 trust: true @@ -66,11 +70,34 @@ relations: - ['gcp-integrator', 'kubernetes-worker'] ``` -To use this overlay with the **Charmed Kubernetes** bundle, it is specified +As well as the storage overlay file ([download it here][asset-gcp-storage-overlay]): + +```yaml +description: Charmed Kubernetes overlay to add native GCP storage support. +applications: + kubernetes-control-plane: + options: + allow-privileged: "true" + gcp-integrator: + charm: gcp-integrator + num_units: 1 + trust: true + gcp-k8s-storage: + charm: gcp-k8s-storage + trust: true + options: + image-registry: k8s.gcr.io +relations: +- ['gcp-k8s-storage:certificates', 'easyrsa:client'] +- ['gcp-k8s-storage:kube-control', 'kubernetes-control-plane:kube-control'] +- ['gcp-k8s-storage:gcp-integration', 'gcp-integrator:gcp'] +``` + +To use these overlays with the **Charmed Kubernetes** bundle, it is specified during deploy like this: ```bash -juju deploy charmed-kubernetes --overlay ~/path/gcp-overlay.yaml --trust +juju deploy charmed-kubernetes --overlay ~/path/gcp-overlay.yaml --overlay ~/path/gcp-storage-overlay.yaml --trust ``` ... and remember to fetch the configuration file! @@ -93,6 +120,33 @@ GCP storage currently comes in two types - SSD (pd-ssd) or 'standard'(pd-standard). To use these, we need to create a storage classes in Kubernetes. +#### Beginning in Kubernetes 1.25 + +The `gcp-k8s-storage` charm will need to be installed to make use of PD Volumes. +Google removed CSIMigration away from the in-tree binaries but made them available +as container workload in the cluster. This charm installs and relates to the +existing integrator charm. + +A StorageClass will be created by this charm named `csi-gce-pd-default` + +You can confirm this has been added by running: + +```bash +kubectl get sc +``` + +which should return: +```bash +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +csi-gce-pd-default pd.csi.storage.gke.io Delete WaitForFirstConsumer false 4h19m +``` + +#### Prior to Kubernetes 1.25 + +First we need to create a storage class which can be used by Kubernetes. +To start with, we will create one for the 'General Purpose SSD' type of EBS +storage: + For the standard disks: ```bash @@ -134,6 +188,8 @@ gcp-ssd kubernetes.io/gce-pd 9s gcp-standard kubernetes.io/gce-pd 45s ``` +#### Creating a PVC + To actually create storage using this new class, you can make a Persistent Volume Claim: ```bash @@ -281,15 +337,19 @@ curl 34.76.144.215:8080 Hello Kubernetes! ``` -### Upgrading the integrator-charm +### Upgrading the charms -The gcp-integrator is not specifically tied to the version of **Charmed Kubernetes** installed and may -generally be upgraded at any time with the following command: +The charm `gcp-integrator` and `gcp-k8s-storage` +can be refreshed within the current charm channel without concern and +can be upgraded at any time with the following command, ```bash juju refresh gcp-integrator +juju refresh gcp-k8s-storage ``` +It isn't recommended to switch charm channels unless a full charm upgrade is planned. + ### Troubleshooting If you have any specific problems with the gcp-integrator, you can report bugs on @@ -313,6 +373,7 @@ juju debug-log --replay --include gcp-integrator/0 [owner]: https://console.cloud.google.com/iam-admin/iam [iam-roles]: https://cloud.google.com/compute/docs/access/iam [asset-gcp-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/gcp-overlay.yaml +[asset-gcp-storage-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/gcp-storage-overlay.yaml [operations]: https://console.cloud.google.com/compute/operations [storage]: /kubernetes/docs/storage [bugs]: https://bugs.launchpad.net/charmed-kubernetes From 85f59354956ecfa4e30f55910301450f8c01bff1 Mon Sep 17 00:00:00 2001 From: Adam Dyess Date: Wed, 12 Apr 2023 13:11:58 -0500 Subject: [PATCH 3/6] updates azure integration page --- pages/k8s/azure-integration.md | 23 ++++++++++++++--------- pages/k8s/gcp-integration.md | 2 +- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/pages/k8s/azure-integration.md b/pages/k8s/azure-integration.md index e1624359..b7e073aa 100644 --- a/pages/k8s/azure-integration.md +++ b/pages/k8s/azure-integration.md @@ -28,7 +28,7 @@ dynamically create, for example, storage. ### Installing using the Out-of-Tree Providers If you install **Charmed Kubernetes** [using the Juju bundle][install], -you can add the azure-cloud-provider at the same time by using the following +you can add the `azure-cloud-provider` charm at the same time by using the following overlay file ([download it here][asset-azure-cloud-overlay]): ```yaml @@ -46,14 +46,10 @@ applications: charm: azure-cloud-provider relations: -- - azure-cloud-provider:certificates - - easyrsa:client # or whichever application supplies cluster certs -- - azure-cloud-provider:kube-control - - kubernetes-control-plane:kube-control -- - azure-cloud-provider:external-cloud-provider - - kubernetes-control-plane:external-cloud-provider -- - azure-cloud-provider:azure-integration - - azure-integrator:clients +- [ 'azure-cloud-provider:certificates', 'easyrsa:client' ] # or whichever application supplies cluster certs +- [ 'azure-cloud-provider:kube-control', 'kubernetes-control-plane:kube-control' ] +- [ 'azure-cloud-provider:azure-integration', 'azure-integrator:clients' ] +- [ 'azure-cloud-provider:external-cloud-provider', 'kubernetes-control-plane:external-cloud-provider' ] ``` To use this overlay with the **Charmed Kubernetes** bundle, it is specified @@ -63,6 +59,15 @@ during deploy like this: juju deploy charmed-kubernetes --overlay azure-cloud-overlay.yaml --trust ``` +... and remember to fetch the configuration file! + +```bash +juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config +``` + +For more configuration options and details of the permissions which the +integrator uses, please see the [charm readme][azure-integrator]. + ### Installing using In-Tree Providers The Kubernetes binaries have in-tree providers for common cloud platforms, diff --git a/pages/k8s/gcp-integration.md b/pages/k8s/gcp-integration.md index c9afe4b8..59ac5e35 100644 --- a/pages/k8s/gcp-integration.md +++ b/pages/k8s/gcp-integration.md @@ -377,7 +377,7 @@ juju debug-log --replay --include gcp-integrator/0 [operations]: https://console.cloud.google.com/compute/operations [storage]: /kubernetes/docs/storage [bugs]: https://bugs.launchpad.net/charmed-kubernetes -[gcp-integrator-readme]: https://charmhub.io/containers-gcp-integrator/ +[gcp-integrator-readme]: https://charmhub.io/gcp-integrator/ [target-pool]: https://cloud.google.com/load-balancing/docs/target-pools [install]: /kubernetes/docs/install-manual From 19217dd70da28f56e4ea1866d4832fb75d3594a0 Mon Sep 17 00:00:00 2001 From: Adam Dyess Date: Wed, 12 Apr 2023 13:28:35 -0500 Subject: [PATCH 4/6] updates vsphere integration page --- pages/k8s/vsphere-integration.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/pages/k8s/vsphere-integration.md b/pages/k8s/vsphere-integration.md index 52ca7cf3..8e2101aa 100644 --- a/pages/k8s/vsphere-integration.md +++ b/pages/k8s/vsphere-integration.md @@ -196,14 +196,28 @@ back to the credential data it received via `juju trust`. The vSphere charms can make use of vSphere-backed storage for Kubernetes. The steps below create a busybox pod with a persistent volume claim backed by -vSphere's PersistentDisk as an example. If juju has `vsphere-cloud-provdider` -installed, skip step 1 since it creates the StorageClass `csi-vsphere-default`. +vSphere's PersistentDisk as an example. ### 1. Create a storage class using the `csi.vsphere.vmware.com` provisioner: +* If the `vsphere-cloud-provider` charm is installed, skip this step since + it creates the StorageClass `csi-vsphere-default`. +```bash +SC_NAME=csi-vsphere-default +kubectl get sc $SC_NAME +``` +``` +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +csi-vsphere-default (default) csi.vsphere.vmware.com Delete Immediate false 0s +``` + +* Without the `vsphere-cloud-provider` charm, one will need to create a +storage class which can be used by Kubernetes against the `csi.vsphere.vmware.com` +provisioner. ```bash SC_NAME=mystorage + kubectl create -f - < Date: Fri, 14 Apr 2023 15:27:56 +0100 Subject: [PATCH 5/6] Update aws-integration.md style tweaks --- pages/k8s/aws-integration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pages/k8s/aws-integration.md b/pages/k8s/aws-integration.md index c5b11968..12b5a710 100644 --- a/pages/k8s/aws-integration.md +++ b/pages/k8s/aws-integration.md @@ -28,8 +28,8 @@ dynamically create, for example, EBS volumes. ## AWS K8S Storage The `aws-k8s-storage` charm moves the AWS specific functions of the EBS csi-driver -out-of-tree. Using this charm, the drivers are installed as workloads in the kubernetes -cluster instead of as natural code paths of the kubernetes binaries. +out-of-tree. Using this charm, the drivers are installed as workloads in the Kubernetes +cluster instead of as natural code paths of the Kubernetes binaries. ## AWS Cloud Provider @@ -381,4 +381,4 @@ If you are an AWS user, you may also be interested in how to or file a bug here.

- \ No newline at end of file + From 1cc9a882215700b0a17a996a9201a49708e60581 Mon Sep 17 00:00:00 2001 From: Nick Veitch Date: Tue, 18 Apr 2023 15:17:05 +0100 Subject: [PATCH 6/6] Update aws-integration.md re-org version info --- pages/k8s/aws-integration.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/pages/k8s/aws-integration.md b/pages/k8s/aws-integration.md index 12b5a710..93788abf 100644 --- a/pages/k8s/aws-integration.md +++ b/pages/k8s/aws-integration.md @@ -38,6 +38,21 @@ out-of-tree. The AWS cloud provider provides the interface between a Kubernetes and AWS service APIs. This project allows a Kubernetes cluster to provision, monitor and remove AWS resources necessary for operation of the cluster. +### Version support + +#### From Kubernetes 1.27 + +The in-tree cloud-provider is no longer available, and must be deployed +as container workloads in the cluster. Charmed Kubernetes recommends +using the `aws-cloud-provider` charm to access AWS Service APIs. + +#### Prior to Kubernetes 1.27 + +The in-tree cloud-provider is natively available in Kubernetes until the 1.27 +release, and it is not necessary to deploy the `aws-cloud-provider` charm as in the +above overlay. + + ### Installing If you install **Charmed Kubernetes** [using the Juju bundle][install], you can add the @@ -104,19 +119,6 @@ juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config For more configuration options and details of the permissions which the integrator uses, please see the [charm readme][aws-integrator-readme]. -#### Beginning in Kubernetes 1.27 - -The in-tree cloud-provider is no longer available, and must be deployed -as container workloads in the cluster. Charmed Kubernetes recommends -using the `aws-cloud-provider` charm to have access to AWS Service APIs. - - -#### Prior to Kubernetes 1.27 - -The in-tree cloud-provider is natively available in Kubernetes until the 1.27 -release, and it is not necessary to deploy the `aws-cloud-provider` charm as in the above -overlay. - ### Using EBS volumes Many pods you may wish to deploy will require storage. Although you can use