Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support populating Spanner source database id, instance id and a custom metadata field in Spanner CDC to Pub Sub template #1769

Merged
merged 4 commits into from
Sep 11, 2024

Conversation

ShuranZhang
Copy link
Contributor

@ShuranZhang ShuranZhang commented Aug 2, 2024

Introduce two new options for Spanner Change Streams to Pub/Sub template.

  1. includeSpannerSource: boolean option default to false.
    If this config is set to true, two new fields spannerDatabaseId and spannerInstanceId will be populated to the message data field with the associate values.

  2. outputMessageData: string option default to empty.
    If this config has non-empty string value, the string value will be populated to the outputMessageMetadata field in output pub sub message body.

I verified correctness for both AVRO and JSON data formats through integration tests in SpannerChangeStreamsToPubSubIT.java.

@ShuranZhang ShuranZhang marked this pull request as ready for review August 2, 2024 20:43
@pull-request-size pull-request-size bot added size/XL and removed size/L labels Sep 5, 2024
@ShuranZhang ShuranZhang changed the title Support populating Spanner source database and instance id in Spanner CDC to Pub Sub template Support populating Spanner source database id, instance id and a custom metadata field in Spanner CDC to Pub Sub template Sep 6, 2024
Copy link

codecov bot commented Sep 10, 2024

Codecov Report

Attention: Patch coverage is 62.50000% with 48 lines in your changes missing coverage. Please review.

Project coverage is 43.33%. Comparing base (1e49734) to head (8205dc6).
Report is 15 commits behind head on main.

Files with missing lines Patch % Lines
...rt/v2/transforms/WriteDataChangeRecordsToAvro.java 72.60% 17 Missing and 3 partials ⚠️
...FileFormatFactorySpannerChangeStreamsToPubSub.java 0.00% 14 Missing ⚠️
...ort/v2/templates/SpannerChangeStreamsToPubSub.java 0.00% 12 Missing ⚠️
...rt/v2/transforms/WriteDataChangeRecordsToJson.java 93.10% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1769      +/-   ##
============================================
+ Coverage     43.28%   43.33%   +0.04%     
- Complexity     3514     3515       +1     
============================================
  Files           832      832              
  Lines         48553    48647      +94     
  Branches       5222     5235      +13     
============================================
+ Hits          21016    21080      +64     
- Misses        25848    25874      +26     
- Partials       1689     1693       +4     
Components Coverage Δ
spanner-templates 63.66% <ø> (+0.02%) ⬆️
spanner-import-export 63.83% <ø> (+0.02%) ⬆️
spanner-live-forward-migration 75.11% <ø> (ø)
spanner-live-reverse-replication 68.93% <ø> (ø)
spanner-bulk-migration 84.19% <ø> (+0.05%) ⬆️
Files with missing lines Coverage Δ
...rt/v2/transforms/WriteDataChangeRecordsToJson.java 87.09% <93.10%> (+27.09%) ⬆️
...ort/v2/templates/SpannerChangeStreamsToPubSub.java 0.00% <0.00%> (ø)
...FileFormatFactorySpannerChangeStreamsToPubSub.java 0.00% <0.00%> (ø)
...rt/v2/transforms/WriteDataChangeRecordsToAvro.java 60.60% <72.60%> (+9.35%) ⬆️

... and 6 files with indirect coverage changes

Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approval based on passed tests

@Abacn Abacn merged commit ae3161e into GoogleCloudPlatform:main Sep 11, 2024
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants