diff --git a/pages/k8s/aws-integration.md b/pages/k8s/aws-integration.md
index 60b5c8c9..93788abf 100644
--- a/pages/k8s/aws-integration.md
+++ b/pages/k8s/aws-integration.md
@@ -14,8 +14,8 @@ toc: False
---
**Charmed Kubernetes** will run seamlessly on AWS. With the addition of the
-`aws-integrator`, your cluster will also be able to directly use AWS native
-features.
+`aws-integrator` and its companion charms, your cluster will also be able
+to directly use AWS native features.
## AWS integrator
@@ -25,31 +25,89 @@ AWS. Using the credentials provided to **Juju**, it acts as a proxy between
Charmed Kubernetes and the underlying cloud, granting permissions to
dynamically create, for example, EBS volumes.
+## AWS K8S Storage
+
+The `aws-k8s-storage` charm moves the AWS specific functions of the EBS csi-driver
+out-of-tree. Using this charm, the drivers are installed as workloads in the Kubernetes
+cluster instead of as natural code paths of the Kubernetes binaries.
+
+## AWS Cloud Provider
+
+The `aws-cloud-provider` moves the AWS specific functions of the cloud-provider
+out-of-tree. The AWS cloud provider provides the interface between a Kubernetes cluster
+and AWS service APIs. This project allows a Kubernetes cluster to provision,
+monitor and remove AWS resources necessary for operation of the cluster.
+
+### Version support
+
+#### From Kubernetes 1.27
+
+The in-tree cloud-provider is no longer available, and must be deployed
+as container workloads in the cluster. Charmed Kubernetes recommends
+using the `aws-cloud-provider` charm to access AWS Service APIs.
+
+#### Prior to Kubernetes 1.27
+
+The in-tree cloud-provider is natively available in Kubernetes until the 1.27
+release, and it is not necessary to deploy the `aws-cloud-provider` charm as in the
+above overlay.
+
+
### Installing
If you install **Charmed Kubernetes** [using the Juju bundle][install], you can add the
-aws-integrator at the same time by using the following overlay file ([download
+aws-integrator at the same time by using the following cloud-provider overlay file ([download
it here][asset-aws-overlay]):
```yaml
description: Charmed Kubernetes overlay to add native AWS support.
applications:
aws-integrator:
- annotations:
- gui-x: "600"
- gui-y: "300"
charm: aws-integrator
num_units: 1
trust: true
+ aws-cloud-provider:
+ charm: aws-cloud-provider
relations:
- ['aws-integrator', 'kubernetes-control-plane']
- ['aws-integrator', 'kubernetes-worker']
- ```
+ - ["aws-cloud-provider:certificates", "easyrsa"]
+ - ["aws-cloud-provider:kube-control", "kubernetes-control-plane"]
+ - ["aws-cloud-provider:external-cloud-provider", "kubernetes-control-plane"]
+ - ["aws-cloud-provider:aws-integration", "aws-integrator"]
+```
-To use this overlay with the **Charmed Kubernetes** bundle, it is specified during deploy like this:
+As well as the storage overlay file ([download it here][asset-aws-storage-overlay]):
+
+```yaml
+description: Charmed Kubernetes overlay to add native AWS support.
+applications:
+ kubernetes-control-plane:
+ options:
+ allow-privileged: "true"
+ aws-integrator:
+ charm: aws-integrator
+ num_units: 1
+ trust: true
+ aws-k8s-storage:
+ charm: aws-k8s-storage
+ trust: true
+ options:
+ image-registry: public.ecr.aws
+relations:
+- ['aws-k8s-storage:certificates', 'easyrsa:client']
+- ['aws-k8s-storage:kube-control', 'kubernetes-control-plane:kube-control']
+- ['aws-k8s-storage:aws-integration', 'aws-integrator:aws']
+# Include the following relations if not using the aws-cloud-provider charm
+# - ['aws-integrator', 'kubernetes-control-plane']
+# - ['aws-integrator', 'kubernetes-worker']
+```
+
+To use these overlays with the **Charmed Kubernetes** bundle, it is specified
+during deploy like this:
```bash
-juju deploy charmed-kubernetes --overlay ~/path/aws-overlay.yaml --trust
+juju deploy charmed-kubernetes --overlay ~/path/aws-overlay.yaml --overlay ~/path/aws-storage-overlay.yaml --trust
```
... and remember to fetch the configuration file!
@@ -63,15 +121,39 @@ please see the [charm readme][aws-integrator-readme].
### Using EBS volumes
-Many pods you may wish to deploy will require storage. Although you can use
+Many pods you may wish to deploy will require storage. Although you can use
any type of storage supported by Kubernetes (see the
[storage documentation][storage]), you also have the option to use the native
AWS storage, Elastic Block Store (EBS).
+#### Beginning in Kubernetes 1.25
+
+The `aws-k8s-storage` charm will need to be installed to make use of EBS Volumes.
+Amazon removed CSIMigration away from the in-tree binaries but made them available
+as container workload in the cluster. This charm installs and relates to the
+existing integrator charm.
+
+A StorageClass will be created by this charm named `csi-aws-ebs-default`
+
+You can confirm this has been added by running:
+
+```bash
+kubectl get sc
+```
+
+which should return:
+```bash
+NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
+csi-aws-ebs-default ebs.csi.aws.com Delete WaitForFirstConsumer false 9s
+```
+
+#### Prior to Kubernetes 1.25
+
First we need to create a storage class which can be used by Kubernetes.
To start with, we will create one for the 'General Purpose SSD' type of EBS
storage:
+
```bash
kubectl create -f - <
-### Upgrading the integrator-charm
+### Upgrading the charms
-The aws-integrator is not specifically tied to the version of Charmed Kubernetes installed and may
-generally be upgraded at any time with the following command:
+The charm `aws-integrator`, `aws-cloud-provider` and `aws-k8s-storage`
+can be refreshed within the current charm channel without concern and
+can be upgraded at any time with the following command,
```bash
juju refresh aws-integrator
+juju refresh aws-cloud-provider
+juju refresh aws-k8s-storage
```
+It isn't recommended to switch charm channels unless a full charm upgrade is planned.
+
### Troubleshooting
If you have any specific problems with the aws-integrator, you can report bugs on
@@ -276,6 +365,7 @@ If you are an AWS user, you may also be interested in how to
[asset-aws-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/aws-overlay.yaml
+[asset-aws-storage-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/aws-stroage-overlay.yaml
[quickstart]: /kubernetes/docs/quickstart
[storage]: /kubernetes/docs/storage
[ebs-info]: https://aws.amazon.com/ebs/features/
@@ -293,4 +383,4 @@ If you are an AWS user, you may also be interested in how to
or
file a bug here.
-
\ No newline at end of file
+
diff --git a/pages/k8s/azure-integration.md b/pages/k8s/azure-integration.md
index e1624359..b7e073aa 100644
--- a/pages/k8s/azure-integration.md
+++ b/pages/k8s/azure-integration.md
@@ -28,7 +28,7 @@ dynamically create, for example, storage.
### Installing using the Out-of-Tree Providers
If you install **Charmed Kubernetes** [using the Juju bundle][install],
-you can add the azure-cloud-provider at the same time by using the following
+you can add the `azure-cloud-provider` charm at the same time by using the following
overlay file ([download it here][asset-azure-cloud-overlay]):
```yaml
@@ -46,14 +46,10 @@ applications:
charm: azure-cloud-provider
relations:
-- - azure-cloud-provider:certificates
- - easyrsa:client # or whichever application supplies cluster certs
-- - azure-cloud-provider:kube-control
- - kubernetes-control-plane:kube-control
-- - azure-cloud-provider:external-cloud-provider
- - kubernetes-control-plane:external-cloud-provider
-- - azure-cloud-provider:azure-integration
- - azure-integrator:clients
+- [ 'azure-cloud-provider:certificates', 'easyrsa:client' ] # or whichever application supplies cluster certs
+- [ 'azure-cloud-provider:kube-control', 'kubernetes-control-plane:kube-control' ]
+- [ 'azure-cloud-provider:azure-integration', 'azure-integrator:clients' ]
+- [ 'azure-cloud-provider:external-cloud-provider', 'kubernetes-control-plane:external-cloud-provider' ]
```
To use this overlay with the **Charmed Kubernetes** bundle, it is specified
@@ -63,6 +59,15 @@ during deploy like this:
juju deploy charmed-kubernetes --overlay azure-cloud-overlay.yaml --trust
```
+... and remember to fetch the configuration file!
+
+```bash
+juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config
+```
+
+For more configuration options and details of the permissions which the
+integrator uses, please see the [charm readme][azure-integrator].
+
### Installing using In-Tree Providers
The Kubernetes binaries have in-tree providers for common cloud platforms,
diff --git a/pages/k8s/gcp-integration.md b/pages/k8s/gcp-integration.md
index 791e22f1..59ac5e35 100644
--- a/pages/k8s/gcp-integration.md
+++ b/pages/k8s/gcp-integration.md
@@ -45,6 +45,13 @@ GCP. Using the credentials provided to Juju, it acts as a proxy between
**Charmed Kubernetes** and the underlying cloud, granting permissions to
dynamically create, for example, storage volumes.
+## GCP K8S Storage
+
+The `gcp-k8s-storage` charm moves the GCP specific functions of the PD csi-driver
+out-of-tree. Using this charm, the drivers are installed as workloads in the kubernetes
+cluster instead of as natural code paths of the kubernetes binaries.
+
+
### Installing
If you install **Charmed Kubernetes** [using the Juju bundle][install], you can add the
@@ -55,9 +62,6 @@ it here][asset-gcp-overlay]):
description: Charmed Kubernetes overlay to add native GCP support.
applications:
gcp-integrator:
- annotations:
- gui-x: "600"
- gui-y: "300"
charm: gcp-integrator
num_units: 1
trust: true
@@ -66,11 +70,34 @@ relations:
- ['gcp-integrator', 'kubernetes-worker']
```
-To use this overlay with the **Charmed Kubernetes** bundle, it is specified
+As well as the storage overlay file ([download it here][asset-gcp-storage-overlay]):
+
+```yaml
+description: Charmed Kubernetes overlay to add native GCP storage support.
+applications:
+ kubernetes-control-plane:
+ options:
+ allow-privileged: "true"
+ gcp-integrator:
+ charm: gcp-integrator
+ num_units: 1
+ trust: true
+ gcp-k8s-storage:
+ charm: gcp-k8s-storage
+ trust: true
+ options:
+ image-registry: k8s.gcr.io
+relations:
+- ['gcp-k8s-storage:certificates', 'easyrsa:client']
+- ['gcp-k8s-storage:kube-control', 'kubernetes-control-plane:kube-control']
+- ['gcp-k8s-storage:gcp-integration', 'gcp-integrator:gcp']
+```
+
+To use these overlays with the **Charmed Kubernetes** bundle, it is specified
during deploy like this:
```bash
-juju deploy charmed-kubernetes --overlay ~/path/gcp-overlay.yaml --trust
+juju deploy charmed-kubernetes --overlay ~/path/gcp-overlay.yaml --overlay ~/path/gcp-storage-overlay.yaml --trust
```
... and remember to fetch the configuration file!
@@ -93,6 +120,33 @@ GCP storage currently comes in two types - SSD (pd-ssd) or
'standard'(pd-standard). To use these, we need to create a storage classes in
Kubernetes.
+#### Beginning in Kubernetes 1.25
+
+The `gcp-k8s-storage` charm will need to be installed to make use of PD Volumes.
+Google removed CSIMigration away from the in-tree binaries but made them available
+as container workload in the cluster. This charm installs and relates to the
+existing integrator charm.
+
+A StorageClass will be created by this charm named `csi-gce-pd-default`
+
+You can confirm this has been added by running:
+
+```bash
+kubectl get sc
+```
+
+which should return:
+```bash
+NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
+csi-gce-pd-default pd.csi.storage.gke.io Delete WaitForFirstConsumer false 4h19m
+```
+
+#### Prior to Kubernetes 1.25
+
+First we need to create a storage class which can be used by Kubernetes.
+To start with, we will create one for the 'General Purpose SSD' type of EBS
+storage:
+
For the standard disks:
```bash
@@ -134,6 +188,8 @@ gcp-ssd kubernetes.io/gce-pd 9s
gcp-standard kubernetes.io/gce-pd 45s
```
+#### Creating a PVC
+
To actually create storage using this new class, you can make a Persistent Volume Claim:
```bash
@@ -281,15 +337,19 @@ curl 34.76.144.215:8080
Hello Kubernetes!
```
-### Upgrading the integrator-charm
+### Upgrading the charms
-The gcp-integrator is not specifically tied to the version of **Charmed Kubernetes** installed and may
-generally be upgraded at any time with the following command:
+The charm `gcp-integrator` and `gcp-k8s-storage`
+can be refreshed within the current charm channel without concern and
+can be upgraded at any time with the following command,
```bash
juju refresh gcp-integrator
+juju refresh gcp-k8s-storage
```
+It isn't recommended to switch charm channels unless a full charm upgrade is planned.
+
### Troubleshooting
If you have any specific problems with the gcp-integrator, you can report bugs on
@@ -313,10 +373,11 @@ juju debug-log --replay --include gcp-integrator/0
[owner]: https://console.cloud.google.com/iam-admin/iam
[iam-roles]: https://cloud.google.com/compute/docs/access/iam
[asset-gcp-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/gcp-overlay.yaml
+[asset-gcp-storage-overlay]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/gcp-storage-overlay.yaml
[operations]: https://console.cloud.google.com/compute/operations
[storage]: /kubernetes/docs/storage
[bugs]: https://bugs.launchpad.net/charmed-kubernetes
-[gcp-integrator-readme]: https://charmhub.io/containers-gcp-integrator/
+[gcp-integrator-readme]: https://charmhub.io/gcp-integrator/
[target-pool]: https://cloud.google.com/load-balancing/docs/target-pools
[install]: /kubernetes/docs/install-manual
diff --git a/pages/k8s/vsphere-integration.md b/pages/k8s/vsphere-integration.md
index 52ca7cf3..8e2101aa 100644
--- a/pages/k8s/vsphere-integration.md
+++ b/pages/k8s/vsphere-integration.md
@@ -196,14 +196,28 @@ back to the credential data it received via `juju trust`.
The vSphere charms can make use of vSphere-backed storage for Kubernetes.
The steps below create a busybox pod with a persistent volume claim backed by
-vSphere's PersistentDisk as an example. If juju has `vsphere-cloud-provdider`
-installed, skip step 1 since it creates the StorageClass `csi-vsphere-default`.
+vSphere's PersistentDisk as an example.
### 1. Create a storage class using the `csi.vsphere.vmware.com` provisioner:
+* If the `vsphere-cloud-provider` charm is installed, skip this step since
+ it creates the StorageClass `csi-vsphere-default`.
+```bash
+SC_NAME=csi-vsphere-default
+kubectl get sc $SC_NAME
+```
+```
+NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
+csi-vsphere-default (default) csi.vsphere.vmware.com Delete Immediate false 0s
+```
+
+* Without the `vsphere-cloud-provider` charm, one will need to create a
+storage class which can be used by Kubernetes against the `csi.vsphere.vmware.com`
+provisioner.
```bash
SC_NAME=mystorage
+
kubectl create -f - <