-
Notifications
You must be signed in to change notification settings - Fork 4.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor terraform GKE and add bitnami kafka module (#27949)
* Refactor GKE and kafka-cluster * Add cloudresourcemanager to required services * Add additional required services * Move cloudresourcemanager to documented prerequisite
- Loading branch information
1 parent
fe974de
commit b34abc4
Showing
30 changed files
with
312 additions
and
1,187 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
| --> | ||
|
|
||
| # Overview | ||
|
|
||
| This module provisions a Bitnami Kafka Cluster based on its | ||
| [helm chart](https://github.com/bitnami/charts/tree/main/bitnami/kafka). | ||
| It uses the [terraform helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest/docs) | ||
| Therefore, you DO NOT need helm to apply this module. | ||
|
|
||
| # Requirements | ||
|
|
||
| - [terraform](https://terraform.io) | ||
| - Connection to a kubernetes cluster (See: [.test-infra/terraform/google-cloud-platform/google-kubernetes-engine/google-kubernetes-engine](../../terraform/google-cloud-platform/google-kubernetes-engine) in this repository) | ||
| - [kubectl](https://kubernetes.io/docs/reference/kubectl/) cli | ||
|
|
||
| # Usage | ||
|
|
||
| Simply follow standard terraform workflow to apply this module. | ||
|
|
||
| ``` | ||
| terraform init | ||
| terraform apply | ||
| ``` | ||
|
|
||
| # Special note about GKE Autopilot | ||
|
|
||
| When applying this module to | ||
| [Google Kubernetes Engine (GKE) Autopilot](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview) | ||
| you will see an "Unschedulable" status. This is because, GKE Autopilot | ||
| needs time to scale up the node. After some time, the kubernetes cluster | ||
| will provision the kafka cluster when these compute resources are available. | ||
|
|
||
| # Debugging and Troubleshooting | ||
|
|
||
| This module deploys a kafka client on the cluster to help with debugging and | ||
| troubleshooting. | ||
|
|
||
| ## Query the kafka client pod name | ||
|
|
||
| Run the following command to query the pod name. | ||
|
|
||
| ``` | ||
| kubectl get po -l app=kafka-client | ||
| ``` | ||
|
|
||
| You should see something similar to the following: | ||
| ``` | ||
| NAME READY STATUS RESTARTS AGE | ||
| kafka-client-cdc7c8885-nmcjc 1/1 Running 0 4m12s | ||
| ``` | ||
|
|
||
| ## Get a shell to the running container | ||
|
|
||
| Run the following command to shell into the running container. | ||
|
|
||
| ``` | ||
| kubectl exec --stdin --tty kafka-client-cdc7c8885-nmcjc -- /bin/bash | ||
| ``` | ||
|
|
||
| ## Execute kafka commands | ||
|
|
||
| The container executes using the latest [bitnami/kafka](https://hub.docker.com/r/bitnami/kafka/) | ||
| image. It has installed all the necessary `kafka-*.sh` scripts in its path. | ||
|
|
||
| In all of the commands, you can use the flag: `--bootstrap-server kafka:9092` | ||
| because the pod is in the same kubernetes cluster and takes advantage | ||
| of its domain name service (DNS). The bitnami helm operator creates a Kubernetes | ||
| service called `kafka` that exposes port `9092`. | ||
|
|
||
| ### Get the cluster-id | ||
|
|
||
| The following command gives you the cluster ID and validates you can connect to | ||
| the cluster. | ||
|
|
||
| ``` | ||
| kafka-cluster.sh cluster-id --bootstrap-server kafka:9092 | ||
| ``` | ||
|
|
||
| ### Create a topic | ||
|
|
||
| The following command creates a Kafka topic. | ||
|
|
||
| ``` | ||
| kafka-topics.sh --create --topic some-topic --partitions 3 --replication-factor 3 --bootstrap-server kafka:9092 | ||
| ``` | ||
|
|
||
| ### Get information about a topic | ||
|
|
||
| The following command queries information about a topic | ||
| (assuming the name `some-topic`). | ||
|
|
||
| ``` | ||
| kafka-topics.sh --describe --topic some-topic --bootstrap-server kafka:9092 | ||
| ``` | ||
|
|
||
| See https://kubernetes.io/docs/tasks/debug/debug-application/get-shell-running-container/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| // Provision the kafka cluster using the Bitnami helm chart. | ||
| resource "helm_release" "kafka" { | ||
| wait = false | ||
| repository = "https://charts.bitnami.com/bitnami" | ||
| chart = "kafka" | ||
| name = "kafka" | ||
| set { | ||
| name = "listeners.client.protocol" | ||
| value = "PLAINTEXT" | ||
| } | ||
| set { | ||
| name = "listeners.interbroker.protocol" | ||
| value = "PLAINTEXT" | ||
| } | ||
| set { | ||
| name = "listeners.external.protocol" | ||
| value = "PLAINTEXT" | ||
| } | ||
| set { | ||
| name = "externalAccess.enabled" | ||
| value = "true" | ||
| } | ||
| set { | ||
| name = "externalAccess.autoDiscovery.enabled" | ||
| value = "true" | ||
| } | ||
| set { | ||
| name = "rbac.create" | ||
| value = "true" | ||
| } | ||
| set { | ||
| name = "service.annotations" | ||
| value = yamlencode({"networking.gke.io/load-balancer-type": "Internal" }) | ||
| } | ||
| set { | ||
| name = "externalAccess.service.broker.ports.external" | ||
| value = "9094" | ||
| } | ||
| set { | ||
| name = "externalAccess.service.controller.containerPorts.external" | ||
| value = "9094" | ||
| } | ||
| set_list { | ||
| name = "externalAccess.controller.service.loadBalancerAnnotations" | ||
| value = [ | ||
| yamlencode({"networking.gke.io/load-balancer-type": "Internal" }), | ||
| yamlencode({"networking.gke.io/load-balancer-type": "Internal" }), | ||
| yamlencode({"networking.gke.io/load-balancer-type": "Internal" }), | ||
| ] | ||
| } | ||
| set_list { | ||
| name = "externalAccess.broker.service.loadBalancerAnnotations" | ||
| value = [ | ||
| yamlencode({"networking.gke.io/load-balancer-type": "Internal" }), | ||
| yamlencode({"networking.gke.io/load-balancer-type": "Internal" }), | ||
| yamlencode({"networking.gke.io/load-balancer-type": "Internal" }), | ||
| ] | ||
| } | ||
| } | ||
|
|
||
| // Provision a kafka client to validate and debug the cluster. | ||
| resource "kubernetes_deployment" "kafka_client" { | ||
| wait_for_rollout = false | ||
| metadata { | ||
| name = "kafka-client" | ||
| labels = { | ||
| app = "kafka-client" | ||
| } | ||
| } | ||
| spec { | ||
| selector { | ||
| match_labels = { | ||
| app = "kafka-client" | ||
| } | ||
| } | ||
| template { | ||
| metadata { | ||
| labels = { | ||
| app = "kafka-client" | ||
| } | ||
| } | ||
| spec { | ||
| container { | ||
| name = "kafka-client" | ||
| image = "bitnami/kafka:latest" | ||
| image_pull_policy = "IfNotPresent" | ||
| command = ["/bin/bash"] | ||
| args = [ | ||
| "-c", | ||
| "while true; do sleep 2; done", | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.