Dataflow Templates 2024-08-13-00_RC00

Release week of 2024-08-12

Dataflow Templates 2024-08-06-00_RC00

Release Week of 2024-08-06

Improvements

• Add App Profile ID to CBT Dataflow Templates: Bigtable to Avro/Parquet/JSON (#1762)
• Add string mapper case for IT (#1757)
• Add support for ALO in SpannerChangeStreamsToBigQuery template (#1750)
• Adding autoReconnect parameters to URL (#1760)
• DatastreamToSQL - log schema map details (#1747)
• Handling for String Primary Keys in Bulk Reader. (#1743)
• Spanner to Source basic impl (#1707)
• Terraform template updates for custom transformation (#1746)
• [BULK] Convert MySQL Time to unified TIME_INTERVAL types (#1741)
• docs: adding file for load testing data generation (#1733)
• fix CdcToBigQueryChangeApplierPipelineIT debezium wait logic (#1729)
• pipeline parameter updates (#1740)

Bug Fixes

• Fixing Index Discovery for 5.7 and removing innodb_parallel_read_threads from init sequences for MySql5.7 compliance. (#1758)
• Fixing exception in String .isSplittable (#1755)

Contributors

Aditya
Astha Mohta
Deep1998
Derek Yau
Dip Patel
Dylan Hercher
Jeff Kinard
Vardhan Vinay Thigle
aksharauke
shreyakhajanchi

Dataflow Templates 2024-07-30-00_RC00

This release was abandoned due to infrastructure issues and wasn't deployed to production. We'll update the current production version of templates directly to 2024-08-06-00_RC00.

Dataflow Templates 2024-07-23-00_RC00

Release Week of 2024-07-23

Note: This release is being rolled out. It may not be in your region yet.

New Templates

Spanner to Source basic impl (#1707)

Improvements

Add support for ALO in SpannerChangeStreamsToBigQuery template (#1750)
Add table counters and separate metrics on a table level (#1728)
Added sample implmentation and cross functional UT for custom transformation (#1732)
Terraform template updates for custom transformation (#1746)
[BULK] Convert MySQL Time to unified TIME_INTERVAL types (#1741)
Use Beam YAML main.py (#1709)
Upgrade jackson version (#1739)
Handling for String Primary Keys in Bulk Reader. (#1743)

Bug Fixes

Fix annotation processor (#1744)
Fix for mutated failsafe event (#1734)
Fixing exception in String .isSplittable (#1755)
Remove PIP_NO_DEPS from YamlTemplate Dockerfile (#1748)

Contributors

Deep1998
Dip Patel
Jeff Kinard
Shreya Khajanchi
Vardhan Vinay Thigle
aksharauke
shreyakhajanchi

Dataflow Templates 2024-07-16-00_RC00

Release Week of 2024-07-16

Note: This release is being rolled out. It may not be in your region yet.

New Templates

• [Terraform] [Cloud Datastream to Spanner] Added terraform template for MySQL sharded source to Spanner

Improvements

• [Spanner Change Streams to BigQuery] Add support for FLOAT32 columns
• [Cloud Datastream to Spanner] Add support for FLOAT32 columns
• [Terraform] [Cloud Datastream to Spanner] Add support for shared VPC

Bug Fixes

• [Cloud Spanner to BigQuery] Fix a NPE when one of the source fields contains a NULL value.

Contributors

@arawind
@damondouglas
@Polber
@manitgupta
@shreyakhajanchi
@an2x

Dataflow Templates 2024-07-09-00_RC00

Release Week of 2024-07-09

Note: This release is being rolled out. It may not be in your region yet.

New Templates

N/A

Improvements

• added fk and referenced tables
• Mongo to BQ - advanced filters
• Remove CustomMySQLResourceManager
• Terraform: Improvements to existing Terraform templates for live migrations
• Support FKs in bulk

Bug Fixes

• Minor fixes for custom transformation
• Fixing timeout handling for uniform partitioning and (temporarily) disabling splitting overhad

Contributors

• aksharauke
• darshan-sj
• manitgupta
• Polber
• shreyakhajanchi
• stankiewicz
• VardhanThigle

Full Changelog: 2024-07-01-00_RC00...2024-07-09-00_RC00

Dataflow Templates 2024-07-01-00_RC00

Release Week of 2024-07-01

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• Updated all templates to Beam 2.57.0
• Added DLQDirectory parameter to Sourcedb To Spanner template
• PubSub to Datadog template now includes full PubSub message by default
• Add support for using pre-existing private connectivity configuration for Datastream to Spanner terraform module
• Various IT's added

Contributors

@AdibH
@aksharauke
@AnandInguva
@dtru-ddog
@Dippatel98
@Polber
@justinuang
@manitgupta
@shreyakhajanchi
@Abacn
@darshan-sj

Dataflow Templates 2024-06-18-00_RC01

Release week of 2024-06-17

Dataflow Templates 2024-06-18-00_RC00

Release week of 2024-06-17

Dataflow Templates 2024-06-11-00_RC00

Release Week of 2024-06-11

New Templates

N/A

Improvements

• Add terraform module to Any SourceDB to Cloud Spanner template
• Update bigtable-beam-import to 2.14.2 to support Data Boost
• Update Template groupName details
• Update the skipOptions in Template annotation for kafka template
• Allow flags to be used as jinja template variables

Bug Fixes

• Add TLS authentication support for Kafka Templates

Contributors

• bharadwaj-aditya
• justinuang
• shreyakhajanchi
• akashorabek
• AnandInguva
• Dippatel98
• robertwb