diff --git a/src/current/_data/redirects.yml b/src/current/_data/redirects.yml index 8c8610f1880..dcc198caa1e 100644 --- a/src/current/_data/redirects.yml +++ b/src/current/_data/redirects.yml @@ -666,6 +666,9 @@ - destination: cockroachcloud/cmek.md#faq sources: ['cockroachcloud/cmek-faq.md'] +- destination: cockroachcloud/managing-cmek.md + sources: ['cockroachcloud/cmek-ops-*.md'] + - destination: cockroachcloud/connect-to-your-cluster.md sources: ['cockroachcloud/stable/cockroachcloud-connect-to-your-cluster.md'] diff --git a/src/current/_includes/v24.2/cdc/cluster-iam-role-step.md b/src/current/_includes/v24.2/cdc/cluster-iam-role-step.md new file mode 100644 index 00000000000..ce6ab2dec86 --- /dev/null +++ b/src/current/_includes/v24.2/cdc/cluster-iam-role-step.md @@ -0,0 +1,43 @@ +1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. +1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. +1. On the **Add permissions** page, click **Next**. +1. Name the role (for example, `ec2-role`) and click **Create role**. +1. Once the role has finished creating, copy the ARN in the **Summary** section. Click on the **Trust relationships** tab. You'll find a **Trusted entities** policy: + + {% include_cached copy-clipboard.html %} + ~~~json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "ec2.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } + ] + } + ~~~ + +1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/) and search for the role (`msk-role`) you created in Step 2 that contains the MSK policy. Select the role, which will take you to its summary page. +1. Click on the **Trust relationships** tab, and click **Edit trust policy**. Add the ARN of the EC2 IAM role (`ec2-role`) to the JSON policy: + + {% include_cached copy-clipboard.html %} + ~~~json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "ec2.amazonaws.com", + "AWS": "arn:aws:iam::{account ID}:role/{ec2-role}" + }, + "Action": "sts:AssumeRole" + } + ] + } + ~~~ + + Once you've updated the policy, click **Update policy**. \ No newline at end of file diff --git a/src/current/_includes/v24.2/cdc/msk-dedicated-support.md b/src/current/_includes/v24.2/cdc/msk-dedicated-support.md new file mode 100644 index 00000000000..9c3c119b838 --- /dev/null +++ b/src/current/_includes/v24.2/cdc/msk-dedicated-support.md @@ -0,0 +1 @@ +If you would like to connect a changefeed running on a CockroachDB Dedicated cluster to an Amazon MSK Serverless cluster, contact your Cockroach Labs account team. \ No newline at end of file diff --git a/src/current/_includes/v24.2/cdc/msk-iam-policy-role-step.md b/src/current/_includes/v24.2/cdc/msk-iam-policy-role-step.md new file mode 100644 index 00000000000..0758d426419 --- /dev/null +++ b/src/current/_includes/v24.2/cdc/msk-iam-policy-role-step.md @@ -0,0 +1,51 @@ +1. In the AWS Management Console, go to the [IAM console](https://console.aws.amazon.com/iam/), select **Policies** from the navigation, and then **Create Policy**. +1. Using the **JSON** tab option, update the policy with the following JSON. These permissions will allow you to connect to the cluster, manage topics, and consume messages. You may want to adjust the permissions to suit your permission model. For more details on the available permissions, refer to the AWS documentation on [IAM Access Control](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#kafka-actions) for MSK. + + Replace the instances of `arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}` with the MSK ARN from your cluster's summary page and add `/*` to the end, like the following: + + {% include_cached copy-clipboard.html %} + ~~~json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "kafka-cluster:Connect", + "kafka-cluster:AlterCluster", + "kafka-cluster:DescribeCluster" + ], + "Resource": [ + "arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}/*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "kafka-cluster:*Topic", + "kafka-cluster:WriteData", + "kafka-cluster:ReadData" + ], + "Resource": [ + "arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}/*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "kafka-cluster:AlterGroup", + "kafka-cluster:DescribeGroup" + ], + "Resource": [ + "arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}/*" + ] + } + ] + } + ~~~ + +1. Once you have added your policy, add a policy name (for example, `msk-policy`), click **Next**, and **Create policy**. +1. Return to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. +1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. +1. On the **Add permissions** page, search for the IAM policy (`msk-policy`) you just created. Click **Next**. +1. Name the role (for example, `msk-role`) and click **Create role**. \ No newline at end of file diff --git a/src/current/_includes/v24.2/cdc/msk-tutorial-crdb-setup.md b/src/current/_includes/v24.2/cdc/msk-tutorial-crdb-setup.md new file mode 100644 index 00000000000..40de46f2af7 --- /dev/null +++ b/src/current/_includes/v24.2/cdc/msk-tutorial-crdb-setup.md @@ -0,0 +1,33 @@ +1. (Optional) On the EC2 instance running CockroachDB, run the [Movr]({% link {{ page.version.version }}/movr.md %}) application workload to set up some data for your changefeed. + + Create the schema for the workload: + + {% include_cached copy-clipboard.html %} + ~~~shell + cockroach workload init movr + ~~~ + + Then run the workload: + + {% include_cached copy-clipboard.html %} + ~~~shell + cockroach workload run movr --duration=1m + ~~~ + +1. Start a SQL session. For details on the available flags, refer to the [`cockroach sql`]({% link {{ page.version.version }}/cockroach-sql.md %}) page. + + {% include_cached copy-clipboard.html %} + ~~~ shell + cockroach sql --insecure + ~~~ + + {{site.data.alerts.callout_info}} + To set your {{ site.data.products.enterprise }} license, refer to the [Licensing FAQs]({% link {{ page.version.version }}/licensing-faqs.md %}#set-a-license) page. + {{site.data.alerts.end}} + +1. Enable the `kv.rangefeed.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}): + + {% include_cached copy-clipboard.html %} + ~~~ sql + SET CLUSTER SETTING kv.rangefeed.enabled = true; + ~~~ \ No newline at end of file diff --git a/src/current/_includes/v24.2/sidebar-data/stream-data.json b/src/current/_includes/v24.2/sidebar-data/stream-data.json index 5a0dc3d276b..a218a190237 100644 --- a/src/current/_includes/v24.2/sidebar-data/stream-data.json +++ b/src/current/_includes/v24.2/sidebar-data/stream-data.json @@ -87,15 +87,26 @@ "title": "Changefeed Tutorials", "items": [ { - "title": "Connect to a Changefeed Kafka Sink with OAuth Using Okta", - "urls": [ - "/${VERSION}/connect-to-a-changefeed-kafka-sink-with-oauth-using-okta.html" + "title": "Stream a Changefeed to an Amazon MSK Cluster", + "items": [ + { + "title": "Amazon MSK", + "urls": [ + "/${VERSION}/stream-a-changefeed-to-amazon-msk.html" + ] + }, + { + "title": "Amazon MSK Serverless", + "urls": [ + "/${VERSION}/stream-a-changefeed-to-amazon-msk-serverless.html" + ] + } ] }, { - "title": "Stream a Changefeed to Amazon MSK Serverless", + "title": "Connect to a Changefeed Kafka Sink with OAuth Using Okta", "urls": [ - "/${VERSION}/stream-a-changefeed-to-amazon-msk-serverless.html" + "/${VERSION}/connect-to-a-changefeed-kafka-sink-with-oauth-using-okta.html" ] }, { diff --git a/src/current/_includes/v24.3/cdc/cluster-iam-role-step.md b/src/current/_includes/v24.3/cdc/cluster-iam-role-step.md new file mode 100644 index 00000000000..64980c725bc --- /dev/null +++ b/src/current/_includes/v24.3/cdc/cluster-iam-role-step.md @@ -0,0 +1,41 @@ +1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. +1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. +1. On the **Add permissions** page, click **Next**. +1. Name the role (for example, `ec2-role`) and click **Create role**. +1. Once the role has finished creating, copy the ARN in the **Summary** section. Click on the **Trust relationships** tab. You'll find a **Trusted entities** policy: + + ~~~json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "ec2.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } + ] + } + ~~~ + +1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/) and search for the role (`msk-role`) you created in Step 2 that contains the MSK policy. Select the role, which will take you to its summary page. +1. Click on the **Trust relationships** tab, and click **Edit trust policy**. Add the ARN of the EC2 IAM role (`ec2-role`) to the JSON policy: + + ~~~json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "ec2.amazonaws.com", + "AWS": "arn:aws:iam::{account ID}:role/{ec2-role}" + }, + "Action": "sts:AssumeRole" + } + ] + } + ~~~ + + Once you've updated the policy, click **Update policy**. \ No newline at end of file diff --git a/src/current/_includes/v24.3/cdc/msk-dedicated-support.md b/src/current/_includes/v24.3/cdc/msk-dedicated-support.md new file mode 100644 index 00000000000..9c3c119b838 --- /dev/null +++ b/src/current/_includes/v24.3/cdc/msk-dedicated-support.md @@ -0,0 +1 @@ +If you would like to connect a changefeed running on a CockroachDB Dedicated cluster to an Amazon MSK Serverless cluster, contact your Cockroach Labs account team. \ No newline at end of file diff --git a/src/current/_includes/v24.3/cdc/msk-iam-policy-role-step.md b/src/current/_includes/v24.3/cdc/msk-iam-policy-role-step.md new file mode 100644 index 00000000000..0758d426419 --- /dev/null +++ b/src/current/_includes/v24.3/cdc/msk-iam-policy-role-step.md @@ -0,0 +1,51 @@ +1. In the AWS Management Console, go to the [IAM console](https://console.aws.amazon.com/iam/), select **Policies** from the navigation, and then **Create Policy**. +1. Using the **JSON** tab option, update the policy with the following JSON. These permissions will allow you to connect to the cluster, manage topics, and consume messages. You may want to adjust the permissions to suit your permission model. For more details on the available permissions, refer to the AWS documentation on [IAM Access Control](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#kafka-actions) for MSK. + + Replace the instances of `arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}` with the MSK ARN from your cluster's summary page and add `/*` to the end, like the following: + + {% include_cached copy-clipboard.html %} + ~~~json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "kafka-cluster:Connect", + "kafka-cluster:AlterCluster", + "kafka-cluster:DescribeCluster" + ], + "Resource": [ + "arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}/*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "kafka-cluster:*Topic", + "kafka-cluster:WriteData", + "kafka-cluster:ReadData" + ], + "Resource": [ + "arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}/*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "kafka-cluster:AlterGroup", + "kafka-cluster:DescribeGroup" + ], + "Resource": [ + "arn:aws:kafka:{region}:{account ID}:cluster/{msk-cluster-name}/*" + ] + } + ] + } + ~~~ + +1. Once you have added your policy, add a policy name (for example, `msk-policy`), click **Next**, and **Create policy**. +1. Return to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. +1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. +1. On the **Add permissions** page, search for the IAM policy (`msk-policy`) you just created. Click **Next**. +1. Name the role (for example, `msk-role`) and click **Create role**. \ No newline at end of file diff --git a/src/current/_includes/v24.3/cdc/msk-tutorial-crdb-setup.md b/src/current/_includes/v24.3/cdc/msk-tutorial-crdb-setup.md new file mode 100644 index 00000000000..40de46f2af7 --- /dev/null +++ b/src/current/_includes/v24.3/cdc/msk-tutorial-crdb-setup.md @@ -0,0 +1,33 @@ +1. (Optional) On the EC2 instance running CockroachDB, run the [Movr]({% link {{ page.version.version }}/movr.md %}) application workload to set up some data for your changefeed. + + Create the schema for the workload: + + {% include_cached copy-clipboard.html %} + ~~~shell + cockroach workload init movr + ~~~ + + Then run the workload: + + {% include_cached copy-clipboard.html %} + ~~~shell + cockroach workload run movr --duration=1m + ~~~ + +1. Start a SQL session. For details on the available flags, refer to the [`cockroach sql`]({% link {{ page.version.version }}/cockroach-sql.md %}) page. + + {% include_cached copy-clipboard.html %} + ~~~ shell + cockroach sql --insecure + ~~~ + + {{site.data.alerts.callout_info}} + To set your {{ site.data.products.enterprise }} license, refer to the [Licensing FAQs]({% link {{ page.version.version }}/licensing-faqs.md %}#set-a-license) page. + {{site.data.alerts.end}} + +1. Enable the `kv.rangefeed.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}): + + {% include_cached copy-clipboard.html %} + ~~~ sql + SET CLUSTER SETTING kv.rangefeed.enabled = true; + ~~~ \ No newline at end of file diff --git a/src/current/_includes/v24.3/sidebar-data/stream-data.json b/src/current/_includes/v24.3/sidebar-data/stream-data.json index 5a0dc3d276b..a218a190237 100644 --- a/src/current/_includes/v24.3/sidebar-data/stream-data.json +++ b/src/current/_includes/v24.3/sidebar-data/stream-data.json @@ -87,15 +87,26 @@ "title": "Changefeed Tutorials", "items": [ { - "title": "Connect to a Changefeed Kafka Sink with OAuth Using Okta", - "urls": [ - "/${VERSION}/connect-to-a-changefeed-kafka-sink-with-oauth-using-okta.html" + "title": "Stream a Changefeed to an Amazon MSK Cluster", + "items": [ + { + "title": "Amazon MSK", + "urls": [ + "/${VERSION}/stream-a-changefeed-to-amazon-msk.html" + ] + }, + { + "title": "Amazon MSK Serverless", + "urls": [ + "/${VERSION}/stream-a-changefeed-to-amazon-msk-serverless.html" + ] + } ] }, { - "title": "Stream a Changefeed to Amazon MSK Serverless", + "title": "Connect to a Changefeed Kafka Sink with OAuth Using Okta", "urls": [ - "/${VERSION}/stream-a-changefeed-to-amazon-msk-serverless.html" + "/${VERSION}/connect-to-a-changefeed-kafka-sink-with-oauth-using-okta.html" ] }, { diff --git a/src/current/cockroachcloud/managing-cmek.md b/src/current/cockroachcloud/managing-cmek.md index 3929754a7bc..dffd36a08e3 100644 --- a/src/current/cockroachcloud/managing-cmek.md +++ b/src/current/cockroachcloud/managing-cmek.md @@ -60,6 +60,8 @@ Follow these steps to create a cross-account IAM role and give it permission to
+### Step 1. Provision the cross-tenant service account + 1. In CockroachDB Cloud, visit the CockroachDB {{ site.data.products.cloud }} [organization settings page](https://cockroachlabs.cloud/settings). Copy your organization ID, which you will need to create the cross-tenant service account. 1. Visit the [Clusters page](https://cockroachlabs.cloud/clusters). Click on the name of your cluster to open its cluster overview page. In the URL, copy the cluster ID: `https://cockroachlabs.cloud/cluster/{YOUR_CLUSTER_ID}/overview`. @@ -96,12 +98,12 @@ If you intend to use an existing key as the CMEK, skip this step. You can create the CMEK directly in the AWS Console or using [HashiCorp Vault]({% link {{site.current_cloud_version}}/hashicorp-integration.md %}). -
- - -
+
+ + +
-
+
1. In the AWS console, visit [AWS KMS](https://console.aws.amazon.com/kms/). 1. To create the key, select **Customer managed keys** and click **Create Key**. @@ -142,9 +144,9 @@ You can create the CMEK directly in the AWS Console or using [HashiCorp Vault]({ 1. Finish creating the key. -
+
-
+
{% capture vault_client_steps %}1. [Install Vault Enterprise Edition installed locally](https://learn.hashicorp.com/tutorials/nomad/hashicorp-enterprise-license?in=vault/enterprise). @@ -198,7 +200,7 @@ You can create the CMEK directly in the AWS Console or using [HashiCorp Vault]({ 1. Set the permissions policy for your key with the `crdb-cmek-kms` IAM policy provided in the [Appendix](#appendix-iam-policy-for-the-cmek-key). 1. Save to finish creating the key. -
+
@@ -206,12 +208,12 @@ You can create the CMEK directly in the AWS Console or using [HashiCorp Vault]({ You can create the CMEK directly in the GCP Console or using [HashiCorp Vault]({% link {{site.current_cloud_version}}/hashicorp-integration.md %}). -
- - -
+
+ + +
-
+
1. In the GCP console, visit the [KMS page](https://console.cloud.google.com/security/kms). 1. Click **+ CREATE KEY RING** and fill in the details to complete the key ring. @@ -224,9 +226,9 @@ You can create the CMEK directly in the GCP Console or using [HashiCorp Vault]({ Make a note of the key ring name. -
+
-
+
1. In the GCP Console, visit the [KMS page](https://console.cloud.google.com/security/kms) and click **+ CREATE KEY RING** to create a key ring for your encryption key. Make a note of the name of the key ring, which you will provide to Vault to create your encryption key. @@ -272,7 +274,7 @@ Make a note of the key ring name. Click **SAVE**. Make a note of the key ring name. -
+
### Step 3. Build your CMEK configuration manifest diff --git a/src/current/releases/cloud.md b/src/current/releases/cloud.md index 04875971718..5dcf432b777 100644 --- a/src/current/releases/cloud.md +++ b/src/current/releases/cloud.md @@ -14,6 +14,10 @@ Get future release notes emailed to you: {% include marketo.html formId=1083 %} +## October 22, 2024 + +- The deprecated `is_regex` field has been removed from the [CockroachDB {{ site.data.products.cloud }} API for Java Web Tokens (JWTs)](https://cockroachlabs.com/docs/api/cloud/v1.html#post-/api/v1/jwt-issuers). The API now handles regular expressions seamlessly. + ## October 1, 2024 Metering for usage-based billing of data transfer, managed backup storage, and changefeeds is now in [Preview]({% link {{site.current_cloud_version }}/cockroachdb-feature-availability.md %}) for all CockroachDB Cloud organizations. @@ -21,7 +25,7 @@ Metering for usage-based billing of data transfer, managed backup storage, and c - [Usage metrics]({% link cockroachcloud/costs.md %}) for data transfer, managed backup storage, and changefeeds are now visible for CockroachDB Standard and Advanced clusters in the CockroachDB Cloud Console. You can view your usage across these metrics on the [Billing page](https://cockroachlabs.cloud/billing/overview) and on invoices. - There will be no usage-based charges associated with these metrics during the preview period, which is in effect through November 30, 2024. During this time, line items with a charge of $0 will be shown for each metric on your monthly invoice. - We will share pricing for these usage-based costs by November 1, 2024. -- On December 1, 2024, once the preview has ended, pricing for these metrics goes into effect immediately for new customers and customers billed monthly, and upon contract renewal for customers billed by invoice. +- On December 1, 2024, once the preview has ended, pricing for these metrics goes into effect immediately for new customers and customers billed monthly, and upon contract renewal for customers billed by invoice. {{site.data.alerts.callout_info}} Bytes transferred for [managed backups]({% link cockroachcloud/managed-backups.md %}) on CockroachDB Standard are not yet metered under [Data Transfer]({% link cockroachcloud/costs.md %}#data-transfer) metrics. This will be implemented during the Preview period and be announced in a future release note. diff --git a/src/current/v24.2/stream-a-changefeed-to-amazon-msk-serverless.md b/src/current/v24.2/stream-a-changefeed-to-amazon-msk-serverless.md index af2616c1619..50ead73bb38 100644 --- a/src/current/v24.2/stream-a-changefeed-to-amazon-msk-serverless.md +++ b/src/current/v24.2/stream-a-changefeed-to-amazon-msk-serverless.md @@ -4,7 +4,7 @@ summary: Learn how to connect a changefeed to stream data to an Amazon MSK Serve toc: true --- -{% include_cached new-in.html version="v24.2" %} [Changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}) can stream change data to [Amazon MSK Serverless clusters](https://docs.aws.amazon.com/msk/latest/developerguide/serverless.html) (Amazon Managed Streaming for Apache Kafka), which is an Amazon MSK cluster type that automatically scales your capacity. +[Changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}) can stream change data to [Amazon MSK Serverless clusters](https://docs.aws.amazon.com/msk/latest/developerguide/serverless.html) (Amazon Managed Streaming for Apache Kafka), which is an Amazon MSK cluster type that automatically scales your capacity. MSK Serverless requires [IAM authentication](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html?icmpid=docs_iam_console) for the changefeed to connect to the cluster. @@ -12,14 +12,15 @@ In this tutorial, you'll set up an MSK Serverless cluster and connect a changefe ## Before you begin +You'll need: + - An [AWS account](https://signin.aws.amazon.com/signup?request_type=register). - A CockroachDB {{ site.data.products.core }} cluster hosted on AWS. You can set up a cluster using [Deploy CockroachDB on AWS EC2]({% link {{ page.version.version }}/deploy-cockroachdb-on-aws.md %}). You must create instances in the same VPC that the MSK Serverless cluster will use in order for the changefeed to authenticate successfully. - A Kafka client to consume the changefeed messages. You **must** ensure that your client machine is in the same VPC as the MSK Serverless cluster. This tutorial uses a client set up following the AWS [MSK Serverless guide](https://docs.aws.amazon.com/msk/latest/developerguide/create-serverless-cluster-client.html). -- A CockroachDB [{{ site.data.products.enterprise }} license]({% link {{ page.version.version }}/enterprise-licensing.md %}). - {% include {{ page.version.version }}/cdc/tutorial-privilege-check.md %} {{site.data.alerts.callout_info}} -If you would like to connect a changefeed running on a CockroachDB Dedicated cluster to an Amazon MSK Serverless cluster, contact your Cockroach Labs account team. +{% include {{ page.version.version }}/cdc/msk-dedicated-support.md %} {{site.data.alerts.end}} ## Step 1. Create an MSK Serverless cluster @@ -33,110 +34,20 @@ If you would like to connect a changefeed running on a CockroachDB Dedicated clu MSK Serverless clusters only support IAM authentication. In this step, you'll create an IAM policy that contains the permissions to interact with the MSK Serverless cluster. Then, you'll create an IAM role, which you'll associate with the IAM policy. In a later step, both the CockroachDB cluster and Kafka client machine will use this role to work with the MSK Serverless cluster. -1. In the AWS Management Console, go to the [IAM console](https://console.aws.amazon.com/iam/), select **Policies** from the navigation, and then **Create Policy**. -1. Using the **JSON** tab option, update the policy with the following JSON. These permissions will allow you to connect to the cluster, manage topics, and consume messages. You may want to adjust the permissions to suit your permission model. For more details on the available permissions, refer to the AWS documentation on [IAM Access Control](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#kafka-actions) for MSK. - - Replace the instances of `arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}` with the MSK Serverless ARN from your cluster's summary page. - - {% include_cached copy-clipboard.html %} - ~~~json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "kafka-cluster:Connect", - "kafka-cluster:AlterCluster", - "kafka-cluster:DescribeCluster" - ], - "Resource": [ - "arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}/*" - ] - }, - { - "Effect": "Allow", - "Action": [ - "kafka-cluster:*Topic", - "kafka-cluster:WriteData", - "kafka-cluster:ReadData" - ], - "Resource": [ - "arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}/*" - ] - }, - { - "Effect": "Allow", - "Action": [ - "kafka-cluster:AlterGroup", - "kafka-cluster:DescribeGroup" - ], - "Resource": [ - "arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}/*" - ] - } - ] - } - ~~~ - -1. Once you have added your policy, add a policy name (for example, `msk-serverless-policy`), click **Next**, and **Create policy**. -1. Return to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. -1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. -1. On the **Add permissions** page, search for the IAM policy (`msk-serverless-policy`) you just created. Click **Next**. -1. Name the role (for example, `msk-serverless-role`) and click **Create role**. +{% include {{ page.version.version }}/cdc/msk-iam-policy-role-step.md %} ## Step 3. Set up the CockroachDB cluster role -In this step, you'll create a role, which contains the `sts:AssumeRole` permission, for the EC2 instance that is running your CockroachDB cluster. The `sts:AssumeRole` permission will allow the EC2 instance to obtain temporary security credentials to access the MSK Serverless cluster according to the `msk-serverless-policy` permissions. To achieve this, you'll add the EC2 role to the trust relationship of the `msk-serverless-role` you created in the [previous step](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). - -1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. -1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. -1. On the **Add permissions** page, click **Next**. -1. Name the role (for example, `ec2-role`) and click **Create role**. -1. Once the role has finished creating, copy the ARN in the **Summary** section. Click on the **Trust relationships** tab. You'll find a **Trusted entities** policy: - - ~~~json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] - } - ~~~ - -1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/) and search for the role (`msk-serverless-role`) you created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster) that contains the MSK Serverless policy. Select the role, which will take you to its summary page. -1. Click on the **Trust relationships** tab, and click **Edit trust policy**. Add the ARN of the EC2 IAM role (`ec2-role`) to the JSON policy: - - ~~~json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com", - "AWS": "arn:aws:iam::{account ID}:role/{ec2-role}" - }, - "Action": "sts:AssumeRole" - } - ] - } - ~~~ +In this step, you'll create a role, which contains the `sts:AssumeRole` permission, for the EC2 instance that is running your CockroachDB cluster. The `sts:AssumeRole` permission will allow the EC2 instance to obtain temporary security credentials to access the MSK Serverless cluster according to the `msk-policy` permissions. To achieve this, you'll add the EC2 role to the trust relationship of the `msk-role` you created in the [previous step](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). - Once you've updated the policy, click **Update policy**. +{% include {{ page.version.version }}/cdc/cluster-iam-role-step.md %} ## Step 4. Connect the client to the MSK Serverless cluster In this step, you'll prepare the client to connect to the MSK Serverless cluster and create a Kafka topic. 1. Ensure that your client can connect to the MSK Serverless cluster. This tutorial uses an EC2 instance running Kafka as the client. Navigate to the summary page for the client EC2 instance. Click on the **Actions** dropdown. Click **Security**, and then select **Modify IAM role**. -1. On the **Modify IAM role** page, select the role you created for the MSK Serverless cluster (`msk-serverless-role`) that contains the policy created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). Click **Update IAM role**. +1. On the **Modify IAM role** page, select the role you created for the MSK Serverless cluster (`msk-role`) that contains the policy created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). Click **Update IAM role**. 1. Open a terminal and connect to your Kafka client. Check that your `client.properties` file contains the correct SASL and security configuration, like the following: ~~~ @@ -176,47 +87,7 @@ In this step, you'll prepare the client to connect to the MSK Serverless cluster In this step, you'll prepare your CockroachDB cluster to start the changefeed. -1. (Optional) On the EC2 instance running CockroachDB, run the [Movr]({% link {{ page.version.version }}/movr.md %}) application workload to set up some data for your changefeed. - - Create the schema for the workload: - - {% include_cached copy-clipboard.html %} - ~~~shell - cockroach workload init movr - ~~~ - - Then run the workload: - - {% include_cached copy-clipboard.html %} - ~~~shell - cockroach workload run movr --duration=1m - ~~~ - -1. Start a SQL session. For details on the available flags, refer to the [`cockroach sql`]({% link {{ page.version.version }}/cockroach-sql.md %}) page. - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach sql --insecure - ~~~ - -1. Set your organization name and [{{ site.data.products.enterprise }} license]({% link {{ page.version.version }}/enterprise-licensing.md %}) key: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING cluster.organization = ''; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING enterprise.license = ''; - ~~~ - -1. Enable the `kv.rangefeed.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}): - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING kv.rangefeed.enabled = true; - ~~~ +{% include {{ page.version.version }}/cdc/msk-tutorial-crdb-setup.md %} 1. To connect the changefeed to the MSK Serverless cluster, the URI must contain the following parameters: - The MSK Serverless cluster endpoint prefixed with the `kafka://` scheme, for example: `kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098`. @@ -224,11 +95,11 @@ In this step, you'll prepare your CockroachDB cluster to start the changefeed. - `sasl_enabled` set to `true`. - `sasl_mechanism` set to `AWS_MSK_IAM`. - `sasl_aws_region` set to the region of the MSK Serverless cluster. - - `sasl_aws_iam_role_arn` set to the ARN for the IAM role (`msk-serverless-role`) that has the permissions outlined in [Step 2.2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). + - `sasl_aws_iam_role_arn` set to the ARN for the IAM role (`msk-role`) that has the permissions outlined in [Step 2.2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). - `sasl_aws_iam_session_name` set to a string that you specify to identify the session in AWS. ~~~ - 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-serverless-role}&sasl_aws_iam_session_name={user-specified session name}' + 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}' ~~~ You can either specify the Kafka URI in the `CREATE CHANGEFEED` statement directly. Or, create an [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) for the MSK Serverless URI. @@ -237,7 +108,7 @@ In this step, you'll prepare your CockroachDB cluster to start the changefeed. {% include_cached copy-clipboard.html %} ~~~ sql - CREATE EXTERNAL CONNECTION msk_serverless AS 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-serverless-role}&sasl_aws_iam_session_name={user-specified session name}'; + CREATE EXTERNAL CONNECTION msk_serverless AS 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}'; ~~~ 1. Use the [`CREATE CHANGEFEED`]({% link {{ page.version.version }}/create-changefeed.md %}) statement to start the changefeed using either the external connection (`external://`) or full `kafka://` URI: diff --git a/src/current/v24.2/stream-a-changefeed-to-amazon-msk.md b/src/current/v24.2/stream-a-changefeed-to-amazon-msk.md new file mode 100644 index 00000000000..5dded49a574 --- /dev/null +++ b/src/current/v24.2/stream-a-changefeed-to-amazon-msk.md @@ -0,0 +1,348 @@ +--- +title: Stream a Changefeed to Amazon MSK +summary: Learn how to connect a changefeed to stream data to an Amazon MSK cluster. +toc: true +--- + +[Changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}) can stream change data to [Amazon MSK clusters](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) (Amazon Managed Streaming for Apache Kafka) using [AWS IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html?icmpid=docs_iam_console) or [SASL/SCRAM]({% link {{ page.version.version }}/security-reference/scram-authentication.md %}) authentication to connect to the MSK cluster. + +In this tutorial, you'll set up an MSK cluster and connect a changefeed with either IAM or SCRAM authentication: + +- For [IAM authentication]({% link {{ page.version.version }}/stream-a-changefeed-to-amazon-msk.md %}?filters=iam-setup-steps#step-1-create-an-msk-cluster-with-iam-authentication), you'll create the MSK cluster with an IAM policy and role. CockroachDB and a Kafka client will assume the IAM role in order to connect to the MSK cluster. Then, you'll set up the Kafka client to consume the changefeed messages and start the changefeed on the CockroachDB cluster. +- For [SCRAM authentication]({% link {{ page.version.version }}/stream-a-changefeed-to-amazon-msk.md %}?filters=scram-setup-steps#step-1-create-an-msk-cluster-with-scram-authentication), you'll create the MSK cluster and then store your SCRAM credentials in [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/). You'll set up the Kafka client configuration and consume the changefeed messages from the CockroachDB cluster. + +{{site.data.alerts.callout_info}} +CockroachDB changefeeds also support IAM authentication to MSK Serverless clusters. For a setup guide, refer to [Stream a Changefeed to Amazon MSK Serverless]({% link {{ page.version.version }}/stream-a-changefeed-to-amazon-msk-serverless.md %}). +{{site.data.alerts.end}} + +## Before you begin + +You'll need: + +- An [AWS account](https://signin.aws.amazon.com/signup?request_type=register). +- A CockroachDB {{ site.data.products.core }} cluster hosted on AWS. You can set up a cluster using [Deploy CockroachDB on AWS EC2]({% link {{ page.version.version }}/deploy-cockroachdb-on-aws.md %}). You must create instances in the same VPC that the MSK cluster will use in order for the changefeed to authenticate successfully. +- A Kafka client to consume the changefeed messages. You **must** ensure that your client machine is in the same VPC as the MSK cluster. This tutorial uses a client set up following the AWS [MSK guide](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html). +- {% include {{ page.version.version }}/cdc/tutorial-privilege-check.md %} + +{{site.data.alerts.callout_info}} +{% include {{ page.version.version }}/cdc/msk-dedicated-support.md %} +{{site.data.alerts.end}} + +**Select the authentication method that you'll use to connect the changefeed to your MSK cluster:** + +
+ + +
+ +
+ +## Step 1. Create an MSK cluster with IAM authentication + +1. In the AWS Management Console, go to the [Amazon MSK console](https://console.aws.amazon.com/msk/home) and click **Create cluster**. +1. Select **Custom create**, name the cluster, and select **Provisioned** as the cluster type. Click **Next**. +1. Select the VPC for the MSK cluster with the subnets and security group. The VPC selection is important because the MSK cluster must be in the same VPC as the CockroachDB instance and Kafka client machine. Click **Next**. +1. Under **Access control methods** select **IAM role-based authentication**. Click **Next**. +1. Continue to select the required configuration options for your cluster. Click **Next**. +1. Review the cluster details, and then click **Create cluster**. +1. Once the cluster is running, click **View client information** in the **Cluster summary** box. Copy the endpoint addresses, which will be similar to `b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098,b-2.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098,b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098`. Click **Done** to return to the cluster's overview page. + +## Step 2. Create an IAM policy and role to access the MSK cluster + +In this step, you'll create an IAM policy that contains the permissions to interact with the MSK cluster. Then, you'll create an IAM role, which you'll associate with the IAM policy. In a later step, both the CockroachDB cluster and Kafka client machine will use this role to work with the MSK cluster. + +{% include {{ page.version.version }}/cdc/msk-iam-policy-role-step.md %} + +## Step 3. Set up the CockroachDB cluster role + +In this step, you'll create a role, which contains the `sts:AssumeRole` permission, for the EC2 instance that is running your CockroachDB cluster. The `sts:AssumeRole` permission will allow the EC2 instance to obtain temporary security credentials to access the MSK cluster according to the `msk-policy` permissions. To achieve this, you'll add the EC2 role to the trust relationship of the `msk-role` you created in the [previous step](#step-2-create-an-iam-policy-and-role-to-access-the-msk-cluster). + +{% include {{ page.version.version }}/cdc/cluster-iam-role-step.md %} + +## Step 4. Connect the client to the MSK cluster + +In this step, you'll prepare the client to connect to the MSK cluster, create a Kafka topic, and consume messages that the changefeed sends. + +1. Ensure that your client can connect to the MSK cluster. This tutorial uses an EC2 instance running Kafka as the client. Navigate to the summary page for the client EC2 instance. Click on the **Actions** dropdown. Click **Security**, and then select **Modify IAM role**. +1. On the **Modify IAM role** page, select the role you created for the MSK cluster (`msk-role`) that contains the policy created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-cluster). Click **Update IAM role**. +1. Open a terminal and connect to your Kafka client. Check that the `client.properties` file in your Kafka installation contains the correct SASL and security configuration, like the following: + + {% include_cached copy-clipboard.html %} + ~~~ + security.protocol=SASL_SSL + sasl.mechanism=AWS_MSK_IAM + sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; + sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler + ~~~ + + If you need further detail on setting up the Kafka client, refer to the [AWS setup guide](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html). +1. Move to the directory of your Kafka installation: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. To create a topic, run the following: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ~/kafka_2.12-2.8.1/bin/kafka-topics.sh --bootstrap-server {msk_endpoint} --command-config client.properties --create --topic {users} --partitions {1} --replication-factor {3} + ~~~ + + Replace: + - `{msk_endpoint}` with your endpoint copied in [Step 1](#step-1-create-an-msk-cluster-with-iam-authentication). + - `{users}` with your topic name. This tutorial will use the CockroachDB `movr` workload and will run a changefeed on the `movr.users` table. + - `{1}` with the number of partitions you require. + - `{3}` with the replication you require. + + You will receive confirmation output: + + ~~~ + Created topic users. + ~~~ + +## Step 5. Start the changefeed + +In this step, you'll prepare your CockroachDB cluster to start the changefeed. + +{% include {{ page.version.version }}/cdc/msk-tutorial-crdb-setup.md %} + +1. To connect the changefeed to the MSK cluster, the URI must contain the following parameters: + - An MSK cluster endpoint prefixed with the `kafka://` scheme, for example: `kafka://b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098`. + - `tls_enabled` set to `true`. + - `sasl_enabled` set to `true`. + - `sasl_mechanism` set to `AWS_MSK_IAM`. + - `sasl_aws_region` set to the region of the MSK cluster. + - `sasl_aws_iam_role_arn` set to the ARN for the IAM role (`msk-role`) that has the permissions outlined in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-cluster). + - `sasl_aws_iam_session_name` set to a string that you specify to identify the session in AWS. + + ~~~ + 'kafka://b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}' + ~~~ + + You can either specify the Kafka URI in the `CREATE CHANGEFEED` statement directly. Or, create an [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) for the MSK URI. + + External connections define a name for an external connection while passing the provider URI and query parameters: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE EXTERNAL CONNECTION msk AS 'kafka://b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}'; + ~~~ + +1. Use the [`CREATE CHANGEFEED`]({% link {{ page.version.version }}/create-changefeed.md %}) statement to start the changefeed using either the external connection (`external://`) or full `kafka://` URI: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE CHANGEFEED FOR TABLE movr.users INTO `external://msk` WITH resolved; + ~~~ + ~~~ + job_id + ----------------------- + 1002677216020987905 + ~~~ + + To view a changefeed job, use [`SHOW CHANGEFEED JOBS`]({% link {{ page.version.version }}/show-jobs.md %}#show-changefeed-jobs). + +## Step 6. Consume the changefeed messages on the client + +1. Return to the terminal that is running the Kafka client. Move to the Kafka installation directory: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. Run the following command to start a consumer. Set `--topic` to the topic you created: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ~/kafka_2.12-2.8.1/bin/kafka-console-consumer.sh --bootstrap-server {msk_endpoint} --consumer.config client.properties --topic users --from-beginning + ~~~ + ~~~ + {"after": {"address": "49665 Green Highway", "city": "amsterdam", "credit_card": "0762957951", "id": "10acc68c-5cea-4d32-95db-3254b8a1170e", "name": "Jimmy Gutierrez"}} + {"after": {"address": "1843 Brandon Common Apt. 68", "city": "amsterdam", "credit_card": "3414699744", "id": "53d95b9a-abf3-4af2-adc8-92d4ee026327", "name": "James Hunt"}} + {"after": {"address": "87351 David Ferry Suite 24", "city": "amsterdam", "credit_card": "7689751883", "id": "58f66df9-e2ef-48bf-bdbe-436e8caa0fae", "name": "Grant Murray"}} + {"after": {"address": "35991 Tran Flats", "city": "amsterdam", "credit_card": "6759782818", "id": "6e8d430d-9a3b-4519-b7ab-987d21043f6a", "name": "Mr. Alan Powers"}} + {"after": {"address": "65320 Emily Ports", "city": "amsterdam", "credit_card": "7424361516", "id": "74e8c91b-9534-4e40-9d19-f23e14d24114", "name": "Michele Grant"}} + {"after": {"address": "85363 Gary Square Apt. 39", "city": "amsterdam", "credit_card": "0267354734", "id": "99d2c816-2216-40f3-b60c-f19bc3e9f455", "name": "Mrs. Wendy Miller"}} + {"after": {"address": "68605 Shane Shores Suite 22", "city": "amsterdam", "credit_card": "5913104602", "id": "ae147ae1-47ae-4800-8000-000000000022", "name": "Crystal Sullivan"}} + {"after": {"address": "41110 Derrick Walk Apt. 42", "city": "amsterdam", "credit_card": "2866469885", "id": "b3333333-3333-4000-8000-000000000023", "name": "Michael Lewis"}} + {"after": {"address": "47781 Robinson Villages Apt. 41", "city": "amsterdam", "credit_card": "6596967781", "id": "b40e7c51-7e68-43f1-a4df-92fa8a05c961", "name": "Abigail Sellers"}} + ... + {"resolved":"1725982589714395510.0000000000"} + ... + ~~~ + +
+ +
+ +## Step 1. Create an MSK cluster with SCRAM authentication + +1. In the AWS Management Console, go to the [Amazon MSK console](https://console.aws.amazon.com/msk/home) and click **Create cluster**. +1. Select **Custom create**, name the cluster, and select **Provisioned** as the cluster type. Click **Next**. +1. Select the VPC for the MSK cluster with the subnets and security group. The VPC selection is important because the MSK cluster must be in the same VPC as the CockroachDB instance and Kafka client machine. Click **Next**. +1. Under **Access control methods** select **SASL/SCRAM authentication**. Click **Next**. +1. Continue to select the required configuration options for your cluster. Click **Next**. +1. Review the cluster details, and then click **Create cluster**. +1. Once the cluster is running, click **View client information** in the **Cluster summary** box. Copy the endpoint addresses, which will be similar to `b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-2.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096`. Click **Done** to return to the cluster's overview page. + +## Step 2. Store the SCRAM credentials + +In this step, you'll store the SCRAM credentials in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/) and then associate the secret with the MSK cluster. + +1. In the AWS Management Console, go to the [Amazon Secrets Manager console](https://console.aws.amazon.com/secretsmanager/home) and click **Store a new secret**. +1. For **Secret type**, select **Other type of secret**. +1. In the **Key/value pairs** box, enter the user and password in **Plaintext** in the same format as the following: + + {% include_cached copy-clipboard.html %} + ~~~ + { + "username": "your_username", + "password": "your_password" + } + ~~~ + +1. Select or add a new encryption key. (You cannot use the default AWS KMS key with an Amazon MSK cluster.) Click **Next**. +1. Add the **Secret name**, you must prefix the name with `AmazonMSK_`. +1. After selecting any other relevant configuration for your secret, click **Store** to complete. +1. Copy the **Secret ARN** for your secret on the **Secret details** page. +1. Return to your MSK cluster in the [Amazon MSK console](https://console.aws.amazon.com/msk/home). +1. Click on the **Properties** tab and find the **Security settings**. Under **SASL/SCRAM authentication**, click **Associate secrets** and paste the ARN of your secret. Click **Associate secrets**. + +## Step 3. Set up the SCRAM authentication on the client + +In this step, you'll configure the Kafka client for SASL/SCRAM authentication and create a Kafka topic. + +1. Open a terminal window and connect to your Kafka client. Check that your `client.properties` file contains the correct SASL/SCRAM, security configuration, and your SASL username and password, like the following: + + {% include_cached copy-clipboard.html %} + ~~~ + security.protocol=SASL_SSL + sasl.mechanism=SCRAM-SHA-512 + sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ + username="your_username" \ + password="your_password"; + ~~~ + +1. Create an environment variable for your broker endpoints: + + {% include_cached copy-clipboard.html %} + ~~~ shell + export brokers=b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-2.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096 + ~~~ + +1. Move to the directory of your Kafka installation: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. To allow the user to interact with Kafka, grant them permission with the Kafka ACL. Replace `your_username` in the following: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ./kafka-acls.sh --bootstrap-server $brokers --add --allow-principal User:{your_username} --operation All --cluster --command-config client.properties + ~~~ + + For more details on permissions and ACLs, refer to [Use ACLs](https://docs.confluent.io/platform/current/security/authorization/acls/manage-acls.html#use-acls). + +1. Create a topic: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ./kafka-topics.sh --create --bootstrap-server $brokers --replication-factor {3} --partitions {1} --topic {users} --command-config client.properties + ~~~ + + Replace: + - `{users}` with your topic name. This tutorial will use the CockroachDB `movr` workload and will run a changefeed on the `movr.users` table. + - `{3}` with the replication factor you require. + - `{1}` with the number of partitions you require. + + You will receive confirmation output: + + ~~~ + Created topic users. + ~~~ + +## Step 4. Start the changefeed + +In this step, you'll prepare your CockroachDB cluster to start the changefeed. + +{% include {{ page.version.version }}/cdc/msk-tutorial-crdb-setup.md %} + +1. To connect the changefeed to the MSK cluster, the URI in the changefeed statement must contain the following parameters: + - One of the MSK cluster endpoints prefixed with the `kafka://` scheme, for example: `kafka://b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096`. + - `tls_enabled` set to `true`. + - `sasl_enabled` set to `true`. + - `sasl_mechanism` set to `SCRAM-SHA-512`. + - `sasl_user` set to `your_username`. + - `sasl_password` set to `your_password`. + + ~~~ + 'kafka://b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096?tls_enabled=true&sasl_enabled=true&sasl_user={your_username}&sasl_password={your_password}-secret&sasl_mechanism=SCRAM-SHA-512' + ~~~ + + You can either specify the Kafka URI in the `CREATE CHANGEFEED` statement directly. Or, create an [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) for the MSK URI. + + External connections define a name for an external connection while passing the provider URI and query parameters: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE EXTERNAL CONNECTION msk AS 'kafka://b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096?tls_enabled=true&sasl_enabled=true&sasl_user={your_username}&sasl_password={your_password}-secret&sasl_mechanism=SCRAM-SHA-512'; + ~~~ + +1. Use the [`CREATE CHANGEFEED`]({% link {{ page.version.version }}/create-changefeed.md %}) statement to start the changefeed using either the external connection (`external://`) or full `kafka://` URI: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE CHANGEFEED FOR TABLE movr.users INTO `external://msk` WITH resolved; + ~~~ + ~~~ + job_id + ----------------------- + 1002677216020987905 + ~~~ + + To view a changefeed job, use [`SHOW CHANGEFEED JOBS`]({% link {{ page.version.version }}/show-jobs.md %}#show-changefeed-jobs). + +## Step 5. Consume the changefeed messages + +1. Return to the terminal that is running the Kafka client. Move to the Kafka installation directory: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. Run the following command to start a consumer. Set `--topic` to the topic you created in [Step 3](#step-3-set-up-the-scram-authentication-on-the-client): + + {% include_cached copy-clipboard.html %} + ~~~ shell + ./kafka-console-consumer.sh --bootstrap-server $brokers --consumer.config client.properties --topic users --from-beginning + ~~~ + ~~~ + {"after": {"address": "49665 Green Highway", "city": "amsterdam", "credit_card": "0762957951", "id": "10acc68c-5cea-4d32-95db-3254b8a1170e", "name": "Jimmy Gutierrez"}} + {"after": {"address": "1843 Brandon Common Apt. 68", "city": "amsterdam", "credit_card": "3414699744", "id": "53d95b9a-abf3-4af2-adc8-92d4ee026327", "name": "James Hunt"}} + {"after": {"address": "87351 David Ferry Suite 24", "city": "amsterdam", "credit_card": "7689751883", "id": "58f66df9-e2ef-48bf-bdbe-436e8caa0fae", "name": "Grant Murray"}} + {"after": {"address": "35991 Tran Flats", "city": "amsterdam", "credit_card": "6759782818", "id": "6e8d430d-9a3b-4519-b7ab-987d21043f6a", "name": "Mr. Alan Powers"}} + {"after": {"address": "65320 Emily Ports", "city": "amsterdam", "credit_card": "7424361516", "id": "74e8c91b-9534-4e40-9d19-f23e14d24114", "name": "Michele Grant"}} + {"after": {"address": "85363 Gary Square Apt. 39", "city": "amsterdam", "credit_card": "0267354734", "id": "99d2c816-2216-40f3-b60c-f19bc3e9f455", "name": "Mrs. Wendy Miller"}} + {"after": {"address": "68605 Shane Shores Suite 22", "city": "amsterdam", "credit_card": "5913104602", "id": "ae147ae1-47ae-4800-8000-000000000022", "name": "Crystal Sullivan"}} + {"after": {"address": "41110 Derrick Walk Apt. 42", "city": "amsterdam", "credit_card": "2866469885", "id": "b3333333-3333-4000-8000-000000000023", "name": "Michael Lewis"}} + {"after": {"address": "47781 Robinson Villages Apt. 41", "city": "amsterdam", "credit_card": "6596967781", "id": "b40e7c51-7e68-43f1-a4df-92fa8a05c961", "name": "Abigail Sellers"}} + ... + {"resolved":"1725982589714395510.0000000000"} + ... + ~~~ + +
+ +## See also + +For more resources, refer to the following: + +- [Changefeed Sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}) page for details on parameters that sinks support. +- [Monitor and Debug Changefeeds]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}) for details on monitoring the changefeed job. \ No newline at end of file diff --git a/src/current/v24.3/stream-a-changefeed-to-amazon-msk-serverless.md b/src/current/v24.3/stream-a-changefeed-to-amazon-msk-serverless.md index 9597fa5d71b..05ae612b5ee 100644 --- a/src/current/v24.3/stream-a-changefeed-to-amazon-msk-serverless.md +++ b/src/current/v24.3/stream-a-changefeed-to-amazon-msk-serverless.md @@ -12,13 +12,15 @@ In this tutorial, you'll set up an MSK Serverless cluster and connect a changefe ## Before you begin +You'll need: + - An [AWS account](https://signin.aws.amazon.com/signup?request_type=register). - A CockroachDB {{ site.data.products.core }} cluster hosted on AWS. You can set up a cluster using [Deploy CockroachDB on AWS EC2]({% link {{ page.version.version }}/deploy-cockroachdb-on-aws.md %}). You must create instances in the same VPC that the MSK Serverless cluster will use in order for the changefeed to authenticate successfully. - A Kafka client to consume the changefeed messages. You **must** ensure that your client machine is in the same VPC as the MSK Serverless cluster. This tutorial uses a client set up following the AWS [MSK Serverless guide](https://docs.aws.amazon.com/msk/latest/developerguide/create-serverless-cluster-client.html). - {% include {{ page.version.version }}/cdc/tutorial-privilege-check.md %} {{site.data.alerts.callout_info}} -If you would like to connect a changefeed running on a CockroachDB Dedicated cluster to an Amazon MSK Serverless cluster, contact your Cockroach Labs account team. +{% include {{ page.version.version }}/cdc/msk-dedicated-support.md %} {{site.data.alerts.end}} ## Step 1. Create an MSK Serverless cluster @@ -32,110 +34,20 @@ If you would like to connect a changefeed running on a CockroachDB Dedicated clu MSK Serverless clusters only support IAM authentication. In this step, you'll create an IAM policy that contains the permissions to interact with the MSK Serverless cluster. Then, you'll create an IAM role, which you'll associate with the IAM policy. In a later step, both the CockroachDB cluster and Kafka client machine will use this role to work with the MSK Serverless cluster. -1. In the AWS Management Console, go to the [IAM console](https://console.aws.amazon.com/iam/), select **Policies** from the navigation, and then **Create Policy**. -1. Using the **JSON** tab option, update the policy with the following JSON. These permissions will allow you to connect to the cluster, manage topics, and consume messages. You may want to adjust the permissions to suit your permission model. For more details on the available permissions, refer to the AWS documentation on [IAM Access Control](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#kafka-actions) for MSK. - - Replace the instances of `arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}` with the MSK Serverless ARN from your cluster's summary page. - - {% include_cached copy-clipboard.html %} - ~~~json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "kafka-cluster:Connect", - "kafka-cluster:AlterCluster", - "kafka-cluster:DescribeCluster" - ], - "Resource": [ - "arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}/*" - ] - }, - { - "Effect": "Allow", - "Action": [ - "kafka-cluster:*Topic", - "kafka-cluster:WriteData", - "kafka-cluster:ReadData" - ], - "Resource": [ - "arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}/*" - ] - }, - { - "Effect": "Allow", - "Action": [ - "kafka-cluster:AlterGroup", - "kafka-cluster:DescribeGroup" - ], - "Resource": [ - "arn:aws:kafka:{region}:{account ID}:cluster/{msk-serverless-cluster-name}/*" - ] - } - ] - } - ~~~ - -1. Once you have added your policy, add a policy name (for example, `msk-serverless-policy`), click **Next**, and **Create policy**. -1. Return to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. -1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. -1. On the **Add permissions** page, search for the IAM policy (`msk-serverless-policy`) you just created. Click **Next**. -1. Name the role (for example, `msk-serverless-role`) and click **Create role**. +{% include {{ page.version.version }}/cdc/msk-iam-policy-role-step.md %} ## Step 3. Set up the CockroachDB cluster role -In this step, you'll create a role, which contains the `sts:AssumeRole` permission, for the EC2 instance that is running your CockroachDB cluster. The `sts:AssumeRole` permission will allow the EC2 instance to obtain temporary security credentials to access the MSK Serverless cluster according to the `msk-serverless-policy` permissions. To achieve this, you'll add the EC2 role to the trust relationship of the `msk-serverless-role` you created in the [previous step](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). - -1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/), select **Roles** from the navigation, and then **Create role**. -1. Select **AWS service** for the **Trusted entity type**. For **Use case**, select **EC2** from the dropdown. Click **Next**. -1. On the **Add permissions** page, click **Next**. -1. Name the role (for example, `ec2-role`) and click **Create role**. -1. Once the role has finished creating, copy the ARN in the **Summary** section. Click on the **Trust relationships** tab. You'll find a **Trusted entities** policy: - - ~~~json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] - } - ~~~ - -1. Navigate to the [IAM console](https://console.aws.amazon.com/iam/) and search for the role (`msk-serverless-role`) you created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster) that contains the MSK Serverless policy. Select the role, which will take you to its summary page. -1. Click on the **Trust relationships** tab, and click **Edit trust policy**. Add the ARN of the EC2 IAM role (`ec2-role`) to the JSON policy: - - ~~~json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com", - "AWS": "arn:aws:iam::{account ID}:role/{ec2-role}" - }, - "Action": "sts:AssumeRole" - } - ] - } - ~~~ +In this step, you'll create a role, which contains the `sts:AssumeRole` permission, for the EC2 instance that is running your CockroachDB cluster. The `sts:AssumeRole` permission will allow the EC2 instance to obtain temporary security credentials to access the MSK Serverless cluster according to the `msk-policy` permissions. To achieve this, you'll add the EC2 role to the trust relationship of the `msk-role` you created in the [previous step](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). - Once you've updated the policy, click **Update policy**. +{% include {{ page.version.version }}/cdc/cluster-iam-role-step.md %} ## Step 4. Connect the client to the MSK Serverless cluster In this step, you'll prepare the client to connect to the MSK Serverless cluster and create a Kafka topic. 1. Ensure that your client can connect to the MSK Serverless cluster. This tutorial uses an EC2 instance running Kafka as the client. Navigate to the summary page for the client EC2 instance. Click on the **Actions** dropdown. Click **Security**, and then select **Modify IAM role**. -1. On the **Modify IAM role** page, select the role you created for the MSK Serverless cluster (`msk-serverless-role`) that contains the policy created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). Click **Update IAM role**. +1. On the **Modify IAM role** page, select the role you created for the MSK Serverless cluster (`msk-role`) that contains the policy created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). Click **Update IAM role**. 1. Open a terminal and connect to your Kafka client. Check that your `client.properties` file contains the correct SASL and security configuration, like the following: ~~~ @@ -175,47 +87,7 @@ In this step, you'll prepare the client to connect to the MSK Serverless cluster In this step, you'll prepare your CockroachDB cluster to start the changefeed. -1. (Optional) On the EC2 instance running CockroachDB, run the [Movr]({% link {{ page.version.version }}/movr.md %}) application workload to set up some data for your changefeed. - - Create the schema for the workload: - - {% include_cached copy-clipboard.html %} - ~~~shell - cockroach workload init movr - ~~~ - - Then run the workload: - - {% include_cached copy-clipboard.html %} - ~~~shell - cockroach workload run movr --duration=1m - ~~~ - -1. Start a SQL session. For details on the available flags, refer to the [`cockroach sql`]({% link {{ page.version.version }}/cockroach-sql.md %}) page. - - {% include_cached copy-clipboard.html %} - ~~~ shell - cockroach sql --insecure - ~~~ - -1. Set your organization name and [{{ site.data.products.enterprise }} license]({% link {{ page.version.version }}/licensing-faqs.md %}#types-of-licenses) key: - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING cluster.organization = ''; - ~~~ - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING enterprise.license = ''; - ~~~ - -1. Enable the `kv.rangefeed.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}): - - {% include_cached copy-clipboard.html %} - ~~~ sql - SET CLUSTER SETTING kv.rangefeed.enabled = true; - ~~~ +{% include {{ page.version.version }}/cdc/msk-tutorial-crdb-setup.md %} 1. To connect the changefeed to the MSK Serverless cluster, the URI must contain the following parameters: - The MSK Serverless cluster endpoint prefixed with the `kafka://` scheme, for example: `kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098`. @@ -223,11 +95,11 @@ In this step, you'll prepare your CockroachDB cluster to start the changefeed. - `sasl_enabled` set to `true`. - `sasl_mechanism` set to `AWS_MSK_IAM`. - `sasl_aws_region` set to the region of the MSK Serverless cluster. - - `sasl_aws_iam_role_arn` set to the ARN for the IAM role (`msk-serverless-role`) that has the permissions outlined in [Step 2.2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). + - `sasl_aws_iam_role_arn` set to the ARN for the IAM role (`msk-role`) that has the permissions outlined in [Step 2.2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-serverless-cluster). - `sasl_aws_iam_session_name` set to a string that you specify to identify the session in AWS. ~~~ - 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-serverless-role}&sasl_aws_iam_session_name={user-specified session name}' + 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}' ~~~ You can either specify the Kafka URI in the `CREATE CHANGEFEED` statement directly. Or, create an [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) for the MSK Serverless URI. @@ -236,7 +108,7 @@ In this step, you'll prepare your CockroachDB cluster to start the changefeed. {% include_cached copy-clipboard.html %} ~~~ sql - CREATE EXTERNAL CONNECTION msk_serverless AS 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-serverless-role}&sasl_aws_iam_session_name={user-specified session name}'; + CREATE EXTERNAL CONNECTION msk_serverless AS 'kafka://boot-vab1abab.c1.kafka-serverless.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}'; ~~~ 1. Use the [`CREATE CHANGEFEED`]({% link {{ page.version.version }}/create-changefeed.md %}) statement to start the changefeed using either the external connection (`external://`) or full `kafka://` URI: diff --git a/src/current/v24.3/stream-a-changefeed-to-amazon-msk.md b/src/current/v24.3/stream-a-changefeed-to-amazon-msk.md new file mode 100644 index 00000000000..5dded49a574 --- /dev/null +++ b/src/current/v24.3/stream-a-changefeed-to-amazon-msk.md @@ -0,0 +1,348 @@ +--- +title: Stream a Changefeed to Amazon MSK +summary: Learn how to connect a changefeed to stream data to an Amazon MSK cluster. +toc: true +--- + +[Changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}) can stream change data to [Amazon MSK clusters](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) (Amazon Managed Streaming for Apache Kafka) using [AWS IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html?icmpid=docs_iam_console) or [SASL/SCRAM]({% link {{ page.version.version }}/security-reference/scram-authentication.md %}) authentication to connect to the MSK cluster. + +In this tutorial, you'll set up an MSK cluster and connect a changefeed with either IAM or SCRAM authentication: + +- For [IAM authentication]({% link {{ page.version.version }}/stream-a-changefeed-to-amazon-msk.md %}?filters=iam-setup-steps#step-1-create-an-msk-cluster-with-iam-authentication), you'll create the MSK cluster with an IAM policy and role. CockroachDB and a Kafka client will assume the IAM role in order to connect to the MSK cluster. Then, you'll set up the Kafka client to consume the changefeed messages and start the changefeed on the CockroachDB cluster. +- For [SCRAM authentication]({% link {{ page.version.version }}/stream-a-changefeed-to-amazon-msk.md %}?filters=scram-setup-steps#step-1-create-an-msk-cluster-with-scram-authentication), you'll create the MSK cluster and then store your SCRAM credentials in [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/). You'll set up the Kafka client configuration and consume the changefeed messages from the CockroachDB cluster. + +{{site.data.alerts.callout_info}} +CockroachDB changefeeds also support IAM authentication to MSK Serverless clusters. For a setup guide, refer to [Stream a Changefeed to Amazon MSK Serverless]({% link {{ page.version.version }}/stream-a-changefeed-to-amazon-msk-serverless.md %}). +{{site.data.alerts.end}} + +## Before you begin + +You'll need: + +- An [AWS account](https://signin.aws.amazon.com/signup?request_type=register). +- A CockroachDB {{ site.data.products.core }} cluster hosted on AWS. You can set up a cluster using [Deploy CockroachDB on AWS EC2]({% link {{ page.version.version }}/deploy-cockroachdb-on-aws.md %}). You must create instances in the same VPC that the MSK cluster will use in order for the changefeed to authenticate successfully. +- A Kafka client to consume the changefeed messages. You **must** ensure that your client machine is in the same VPC as the MSK cluster. This tutorial uses a client set up following the AWS [MSK guide](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html). +- {% include {{ page.version.version }}/cdc/tutorial-privilege-check.md %} + +{{site.data.alerts.callout_info}} +{% include {{ page.version.version }}/cdc/msk-dedicated-support.md %} +{{site.data.alerts.end}} + +**Select the authentication method that you'll use to connect the changefeed to your MSK cluster:** + +
+ + +
+ +
+ +## Step 1. Create an MSK cluster with IAM authentication + +1. In the AWS Management Console, go to the [Amazon MSK console](https://console.aws.amazon.com/msk/home) and click **Create cluster**. +1. Select **Custom create**, name the cluster, and select **Provisioned** as the cluster type. Click **Next**. +1. Select the VPC for the MSK cluster with the subnets and security group. The VPC selection is important because the MSK cluster must be in the same VPC as the CockroachDB instance and Kafka client machine. Click **Next**. +1. Under **Access control methods** select **IAM role-based authentication**. Click **Next**. +1. Continue to select the required configuration options for your cluster. Click **Next**. +1. Review the cluster details, and then click **Create cluster**. +1. Once the cluster is running, click **View client information** in the **Cluster summary** box. Copy the endpoint addresses, which will be similar to `b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098,b-2.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098,b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098`. Click **Done** to return to the cluster's overview page. + +## Step 2. Create an IAM policy and role to access the MSK cluster + +In this step, you'll create an IAM policy that contains the permissions to interact with the MSK cluster. Then, you'll create an IAM role, which you'll associate with the IAM policy. In a later step, both the CockroachDB cluster and Kafka client machine will use this role to work with the MSK cluster. + +{% include {{ page.version.version }}/cdc/msk-iam-policy-role-step.md %} + +## Step 3. Set up the CockroachDB cluster role + +In this step, you'll create a role, which contains the `sts:AssumeRole` permission, for the EC2 instance that is running your CockroachDB cluster. The `sts:AssumeRole` permission will allow the EC2 instance to obtain temporary security credentials to access the MSK cluster according to the `msk-policy` permissions. To achieve this, you'll add the EC2 role to the trust relationship of the `msk-role` you created in the [previous step](#step-2-create-an-iam-policy-and-role-to-access-the-msk-cluster). + +{% include {{ page.version.version }}/cdc/cluster-iam-role-step.md %} + +## Step 4. Connect the client to the MSK cluster + +In this step, you'll prepare the client to connect to the MSK cluster, create a Kafka topic, and consume messages that the changefeed sends. + +1. Ensure that your client can connect to the MSK cluster. This tutorial uses an EC2 instance running Kafka as the client. Navigate to the summary page for the client EC2 instance. Click on the **Actions** dropdown. Click **Security**, and then select **Modify IAM role**. +1. On the **Modify IAM role** page, select the role you created for the MSK cluster (`msk-role`) that contains the policy created in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-cluster). Click **Update IAM role**. +1. Open a terminal and connect to your Kafka client. Check that the `client.properties` file in your Kafka installation contains the correct SASL and security configuration, like the following: + + {% include_cached copy-clipboard.html %} + ~~~ + security.protocol=SASL_SSL + sasl.mechanism=AWS_MSK_IAM + sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; + sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler + ~~~ + + If you need further detail on setting up the Kafka client, refer to the [AWS setup guide](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html). +1. Move to the directory of your Kafka installation: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. To create a topic, run the following: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ~/kafka_2.12-2.8.1/bin/kafka-topics.sh --bootstrap-server {msk_endpoint} --command-config client.properties --create --topic {users} --partitions {1} --replication-factor {3} + ~~~ + + Replace: + - `{msk_endpoint}` with your endpoint copied in [Step 1](#step-1-create-an-msk-cluster-with-iam-authentication). + - `{users}` with your topic name. This tutorial will use the CockroachDB `movr` workload and will run a changefeed on the `movr.users` table. + - `{1}` with the number of partitions you require. + - `{3}` with the replication you require. + + You will receive confirmation output: + + ~~~ + Created topic users. + ~~~ + +## Step 5. Start the changefeed + +In this step, you'll prepare your CockroachDB cluster to start the changefeed. + +{% include {{ page.version.version }}/cdc/msk-tutorial-crdb-setup.md %} + +1. To connect the changefeed to the MSK cluster, the URI must contain the following parameters: + - An MSK cluster endpoint prefixed with the `kafka://` scheme, for example: `kafka://b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098`. + - `tls_enabled` set to `true`. + - `sasl_enabled` set to `true`. + - `sasl_mechanism` set to `AWS_MSK_IAM`. + - `sasl_aws_region` set to the region of the MSK cluster. + - `sasl_aws_iam_role_arn` set to the ARN for the IAM role (`msk-role`) that has the permissions outlined in [Step 2](#step-2-create-an-iam-policy-and-role-to-access-the-msk-cluster). + - `sasl_aws_iam_session_name` set to a string that you specify to identify the session in AWS. + + ~~~ + 'kafka://b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}' + ~~~ + + You can either specify the Kafka URI in the `CREATE CHANGEFEED` statement directly. Or, create an [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) for the MSK URI. + + External connections define a name for an external connection while passing the provider URI and query parameters: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE EXTERNAL CONNECTION msk AS 'kafka://b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9098/?tls_enabled=true&sasl_enabled=true&sasl_mechanism=AWS_MSK_IAM&sasl_aws_region=us-east-1&sasl_aws_iam_role_arn=arn:aws:iam::{account ID}:role/{msk-role}&sasl_aws_iam_session_name={user-specified session name}'; + ~~~ + +1. Use the [`CREATE CHANGEFEED`]({% link {{ page.version.version }}/create-changefeed.md %}) statement to start the changefeed using either the external connection (`external://`) or full `kafka://` URI: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE CHANGEFEED FOR TABLE movr.users INTO `external://msk` WITH resolved; + ~~~ + ~~~ + job_id + ----------------------- + 1002677216020987905 + ~~~ + + To view a changefeed job, use [`SHOW CHANGEFEED JOBS`]({% link {{ page.version.version }}/show-jobs.md %}#show-changefeed-jobs). + +## Step 6. Consume the changefeed messages on the client + +1. Return to the terminal that is running the Kafka client. Move to the Kafka installation directory: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. Run the following command to start a consumer. Set `--topic` to the topic you created: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ~/kafka_2.12-2.8.1/bin/kafka-console-consumer.sh --bootstrap-server {msk_endpoint} --consumer.config client.properties --topic users --from-beginning + ~~~ + ~~~ + {"after": {"address": "49665 Green Highway", "city": "amsterdam", "credit_card": "0762957951", "id": "10acc68c-5cea-4d32-95db-3254b8a1170e", "name": "Jimmy Gutierrez"}} + {"after": {"address": "1843 Brandon Common Apt. 68", "city": "amsterdam", "credit_card": "3414699744", "id": "53d95b9a-abf3-4af2-adc8-92d4ee026327", "name": "James Hunt"}} + {"after": {"address": "87351 David Ferry Suite 24", "city": "amsterdam", "credit_card": "7689751883", "id": "58f66df9-e2ef-48bf-bdbe-436e8caa0fae", "name": "Grant Murray"}} + {"after": {"address": "35991 Tran Flats", "city": "amsterdam", "credit_card": "6759782818", "id": "6e8d430d-9a3b-4519-b7ab-987d21043f6a", "name": "Mr. Alan Powers"}} + {"after": {"address": "65320 Emily Ports", "city": "amsterdam", "credit_card": "7424361516", "id": "74e8c91b-9534-4e40-9d19-f23e14d24114", "name": "Michele Grant"}} + {"after": {"address": "85363 Gary Square Apt. 39", "city": "amsterdam", "credit_card": "0267354734", "id": "99d2c816-2216-40f3-b60c-f19bc3e9f455", "name": "Mrs. Wendy Miller"}} + {"after": {"address": "68605 Shane Shores Suite 22", "city": "amsterdam", "credit_card": "5913104602", "id": "ae147ae1-47ae-4800-8000-000000000022", "name": "Crystal Sullivan"}} + {"after": {"address": "41110 Derrick Walk Apt. 42", "city": "amsterdam", "credit_card": "2866469885", "id": "b3333333-3333-4000-8000-000000000023", "name": "Michael Lewis"}} + {"after": {"address": "47781 Robinson Villages Apt. 41", "city": "amsterdam", "credit_card": "6596967781", "id": "b40e7c51-7e68-43f1-a4df-92fa8a05c961", "name": "Abigail Sellers"}} + ... + {"resolved":"1725982589714395510.0000000000"} + ... + ~~~ + +
+ +
+ +## Step 1. Create an MSK cluster with SCRAM authentication + +1. In the AWS Management Console, go to the [Amazon MSK console](https://console.aws.amazon.com/msk/home) and click **Create cluster**. +1. Select **Custom create**, name the cluster, and select **Provisioned** as the cluster type. Click **Next**. +1. Select the VPC for the MSK cluster with the subnets and security group. The VPC selection is important because the MSK cluster must be in the same VPC as the CockroachDB instance and Kafka client machine. Click **Next**. +1. Under **Access control methods** select **SASL/SCRAM authentication**. Click **Next**. +1. Continue to select the required configuration options for your cluster. Click **Next**. +1. Review the cluster details, and then click **Create cluster**. +1. Once the cluster is running, click **View client information** in the **Cluster summary** box. Copy the endpoint addresses, which will be similar to `b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-2.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096`. Click **Done** to return to the cluster's overview page. + +## Step 2. Store the SCRAM credentials + +In this step, you'll store the SCRAM credentials in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/) and then associate the secret with the MSK cluster. + +1. In the AWS Management Console, go to the [Amazon Secrets Manager console](https://console.aws.amazon.com/secretsmanager/home) and click **Store a new secret**. +1. For **Secret type**, select **Other type of secret**. +1. In the **Key/value pairs** box, enter the user and password in **Plaintext** in the same format as the following: + + {% include_cached copy-clipboard.html %} + ~~~ + { + "username": "your_username", + "password": "your_password" + } + ~~~ + +1. Select or add a new encryption key. (You cannot use the default AWS KMS key with an Amazon MSK cluster.) Click **Next**. +1. Add the **Secret name**, you must prefix the name with `AmazonMSK_`. +1. After selecting any other relevant configuration for your secret, click **Store** to complete. +1. Copy the **Secret ARN** for your secret on the **Secret details** page. +1. Return to your MSK cluster in the [Amazon MSK console](https://console.aws.amazon.com/msk/home). +1. Click on the **Properties** tab and find the **Security settings**. Under **SASL/SCRAM authentication**, click **Associate secrets** and paste the ARN of your secret. Click **Associate secrets**. + +## Step 3. Set up the SCRAM authentication on the client + +In this step, you'll configure the Kafka client for SASL/SCRAM authentication and create a Kafka topic. + +1. Open a terminal window and connect to your Kafka client. Check that your `client.properties` file contains the correct SASL/SCRAM, security configuration, and your SASL username and password, like the following: + + {% include_cached copy-clipboard.html %} + ~~~ + security.protocol=SASL_SSL + sasl.mechanism=SCRAM-SHA-512 + sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ + username="your_username" \ + password="your_password"; + ~~~ + +1. Create an environment variable for your broker endpoints: + + {% include_cached copy-clipboard.html %} + ~~~ shell + export brokers=b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-1.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096,b-2.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096 + ~~~ + +1. Move to the directory of your Kafka installation: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. To allow the user to interact with Kafka, grant them permission with the Kafka ACL. Replace `your_username` in the following: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ./kafka-acls.sh --bootstrap-server $brokers --add --allow-principal User:{your_username} --operation All --cluster --command-config client.properties + ~~~ + + For more details on permissions and ACLs, refer to [Use ACLs](https://docs.confluent.io/platform/current/security/authorization/acls/manage-acls.html#use-acls). + +1. Create a topic: + + {% include_cached copy-clipboard.html %} + ~~~ shell + ./kafka-topics.sh --create --bootstrap-server $brokers --replication-factor {3} --partitions {1} --topic {users} --command-config client.properties + ~~~ + + Replace: + - `{users}` with your topic name. This tutorial will use the CockroachDB `movr` workload and will run a changefeed on the `movr.users` table. + - `{3}` with the replication factor you require. + - `{1}` with the number of partitions you require. + + You will receive confirmation output: + + ~~~ + Created topic users. + ~~~ + +## Step 4. Start the changefeed + +In this step, you'll prepare your CockroachDB cluster to start the changefeed. + +{% include {{ page.version.version }}/cdc/msk-tutorial-crdb-setup.md %} + +1. To connect the changefeed to the MSK cluster, the URI in the changefeed statement must contain the following parameters: + - One of the MSK cluster endpoints prefixed with the `kafka://` scheme, for example: `kafka://b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096`. + - `tls_enabled` set to `true`. + - `sasl_enabled` set to `true`. + - `sasl_mechanism` set to `SCRAM-SHA-512`. + - `sasl_user` set to `your_username`. + - `sasl_password` set to `your_password`. + + ~~~ + 'kafka://b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096?tls_enabled=true&sasl_enabled=true&sasl_user={your_username}&sasl_password={your_password}-secret&sasl_mechanism=SCRAM-SHA-512' + ~~~ + + You can either specify the Kafka URI in the `CREATE CHANGEFEED` statement directly. Or, create an [external connection]({% link {{ page.version.version }}/create-external-connection.md %}) for the MSK URI. + + External connections define a name for an external connection while passing the provider URI and query parameters: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE EXTERNAL CONNECTION msk AS 'kafka://b-3.msk-cluster_name.1a2b3c.c4.kafka.us-east-1.amazonaws.com:9096?tls_enabled=true&sasl_enabled=true&sasl_user={your_username}&sasl_password={your_password}-secret&sasl_mechanism=SCRAM-SHA-512'; + ~~~ + +1. Use the [`CREATE CHANGEFEED`]({% link {{ page.version.version }}/create-changefeed.md %}) statement to start the changefeed using either the external connection (`external://`) or full `kafka://` URI: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE CHANGEFEED FOR TABLE movr.users INTO `external://msk` WITH resolved; + ~~~ + ~~~ + job_id + ----------------------- + 1002677216020987905 + ~~~ + + To view a changefeed job, use [`SHOW CHANGEFEED JOBS`]({% link {{ page.version.version }}/show-jobs.md %}#show-changefeed-jobs). + +## Step 5. Consume the changefeed messages + +1. Return to the terminal that is running the Kafka client. Move to the Kafka installation directory: + + {% include_cached copy-clipboard.html %} + ~~~ shell + cd kafka_2.12-2.8.1/bin + ~~~ + +1. Run the following command to start a consumer. Set `--topic` to the topic you created in [Step 3](#step-3-set-up-the-scram-authentication-on-the-client): + + {% include_cached copy-clipboard.html %} + ~~~ shell + ./kafka-console-consumer.sh --bootstrap-server $brokers --consumer.config client.properties --topic users --from-beginning + ~~~ + ~~~ + {"after": {"address": "49665 Green Highway", "city": "amsterdam", "credit_card": "0762957951", "id": "10acc68c-5cea-4d32-95db-3254b8a1170e", "name": "Jimmy Gutierrez"}} + {"after": {"address": "1843 Brandon Common Apt. 68", "city": "amsterdam", "credit_card": "3414699744", "id": "53d95b9a-abf3-4af2-adc8-92d4ee026327", "name": "James Hunt"}} + {"after": {"address": "87351 David Ferry Suite 24", "city": "amsterdam", "credit_card": "7689751883", "id": "58f66df9-e2ef-48bf-bdbe-436e8caa0fae", "name": "Grant Murray"}} + {"after": {"address": "35991 Tran Flats", "city": "amsterdam", "credit_card": "6759782818", "id": "6e8d430d-9a3b-4519-b7ab-987d21043f6a", "name": "Mr. Alan Powers"}} + {"after": {"address": "65320 Emily Ports", "city": "amsterdam", "credit_card": "7424361516", "id": "74e8c91b-9534-4e40-9d19-f23e14d24114", "name": "Michele Grant"}} + {"after": {"address": "85363 Gary Square Apt. 39", "city": "amsterdam", "credit_card": "0267354734", "id": "99d2c816-2216-40f3-b60c-f19bc3e9f455", "name": "Mrs. Wendy Miller"}} + {"after": {"address": "68605 Shane Shores Suite 22", "city": "amsterdam", "credit_card": "5913104602", "id": "ae147ae1-47ae-4800-8000-000000000022", "name": "Crystal Sullivan"}} + {"after": {"address": "41110 Derrick Walk Apt. 42", "city": "amsterdam", "credit_card": "2866469885", "id": "b3333333-3333-4000-8000-000000000023", "name": "Michael Lewis"}} + {"after": {"address": "47781 Robinson Villages Apt. 41", "city": "amsterdam", "credit_card": "6596967781", "id": "b40e7c51-7e68-43f1-a4df-92fa8a05c961", "name": "Abigail Sellers"}} + ... + {"resolved":"1725982589714395510.0000000000"} + ... + ~~~ + +
+ +## See also + +For more resources, refer to the following: + +- [Changefeed Sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}) page for details on parameters that sinks support. +- [Monitor and Debug Changefeeds]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}) for details on monitoring the changefeed job. \ No newline at end of file