[temp] Changes from 2.7.2.Final #159
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add manifests for source/sink connectors, converters and transformations.
In the description of the `database.ssl.keystore.password` property, the xref to the `database.ssl.keystore` property is incorrectly prefixed with the `{context}` attribute
(cherry picked from commit 13703c7)
add new line add new line DBZ-7259 Fix shouldParseDefiner unittest fix formatting fix formatting Revert "DBZ-7259 Fix shouldParseDefiner unittest" This reverts commit e7de21d. Revert "DBZ-7259 Solve Parser Issue in MySQL Create Procedure" This reverts commit 91976c8. DBZ-7259 Solve Parser Issue in MySQL Create Procedure add new line add new line DBZ-7259 Fix shouldParseDefiner unittest fix formatting fix formatting Revert "DBZ-7259 Fix shouldParseDefiner unittest" This reverts commit cfb2cd6. Revert "# This is a combination of 4 commits." This reverts commit 6d53494. DBZ-7259 Solve Parser Issue in MySQL Create Procedure add new line add new line DBZ-7259 Fix shouldParseDefiner unittest fix formatting fix formatting Revert "DBZ-7259 Fix shouldParseDefiner unittest" This reverts commit e7de21d. Revert "DBZ-7259 Solve Parser Issue in MySQL Create Procedure" This reverts commit 91976c8. DBZ-7259 Solve Parser Issue in MySQL Create Procedure add new line add new line DBZ-7259 Fix shouldParseDefiner unittest fix formatting fix formatting Revert "DBZ-7259 Fix shouldParseDefiner unittest" This reverts commit cfb2cd6. Revert "# This is a combination of 4 commits." This reverts commit 6d53494. resolve
If the logical replication slot has absolutely zero events in it, then the searchWalPosition could be looping for quite some time. During this time, it wasn't sending heartbeats. This commit fixes that function to send heartbeats.
(cherry picked from commit a4e660f)
(cherry picked from commit e2c0a59)
This fix uses the relational table primary key by default; however, as a user can define `message.key.columns` to even override the primary key configuration or to handle keyless tables, the user can override the `reselect.use.event.key` option with `true` so use the event key fields instead for the re-select so that keyless tables can also participate with the column reselection process.
DBZ-7360 Resolve DDL Parser in Oracle Alter Table
DBZ-7360 Resolve DDL Parser in Oracle Alter Table DBZ-7420 Modify alterSpecification Clause DBZ-7420 Modify alterSpecification Clause
This PR enabled some of the replica identity related tests that were disabled earlier because of the underlying operation not being supported by `yugabyte-db`.
…ies (#130) This PR adds the changes to the connector required to support all kinds of operations supported by YugabyteDB when coupled with various replica identities. One change to note here would be that in case of `DEFUALT` replica identity, Postgres sends out the before image of the primary key column in case of `DELETE` and `UPDATE` operations. Meanwhile, YugabyteDB only sends out the before image of primary key in case of `DELETE` operations, and if a primary key itself is updated, YugabyteDB sends two events: 1. Delete for the existing row 2. Insert with the new value of the primary key Additionally to test out everything, we have a new test class `YugabyteReplicaIdentityIT`.
Currently the PG debezium connector makes use of smart driver for creating connections to YugabyteDB. The smart driver has in built support for load balancing. However to enable this, the property `load-balance=true` needs to be set in the connection URL. This PR adds the `load-balance=true` property to the multi host URL used for creating load-balanced connections.
Fixed and trimmed log messages. PG connector skips empty txns and these txns were being logged as a warning with the message "possible data loss", which is not true. Therefore, we are now removing the "possible data loss" log message.
Based on our testing so far, whenever the PG connector has skipped txns, those txns have always been empty txns.
…put (#140) This PR changes the default plugin in the connector to `yboutput`. This closes yugabyte/yugabyte-db#23153
…llow YB standards (#136) This PR also changes the versioning scheme of the connector where the version would now look like: * `dz.<DebeziumVersion>.yb.<YBVersion>` * For example: `dz.2.5.2.yb.2024.1` This partially addresses yugabyte/yugabyte-db#23081
…g convention (#142) The PR renamed the main class PostgresConnector to YBPostgresConnector to avoid name clashes while loading the classes. Additionally, this closes yugabyte/yugabyte-db#23081
…bezium-core changes (#137) This PR reverts the changes made in `debezium-core` and converts those changes to be in the local module only. Changes include: 1. Adding a custom field definition for `HOSTNAME` in `PostgresConnectorConfig` which uses a new validator `PostgresConnectorConfig#validateYBHostname` a. Currently the validator definition returns a dummy positive result but we can modify this in future to set a validation rule if needed. 2. Removing the method `SnapshotResult#isSkipped` and keeping it localised in `PostgresSnapshotChangeEventSource` 3. Modifying `Dockerfile` to not copy custom `debezium-core-*.jar` - this will enforce that the images use the prepackaged jar only. This closes yugabyte/yugabyte-db#23082
This PR fixes the flakiness and false negatives by changing the
following:
1. Disabling the tests which are unsupported
i. Tests with postgis are not supported
2. Filtering heartbeat records while consuming
3. Fixing other test issues with incorrect record value matching or
value assertions
…s for datatypes (#144) * This PR adds a test class `YBRecordsStreamProducerIT` which is essentially a copy of `RecordsStreamProducerIT`. * The tests are modified in order to support the structure generated by the plugin `yboutput`. * Some tests are disabled owing to the fact that the underlying feature is not yet supported. * Altering not allowed for column under replication --> yugabyte/yugabyte-db#16625 * Altering the replica identity is not allowed
) Some of the tests were not working while working with YugabyteDB's plugin `yboutput` (default) and the default replica identity `CHANGE` because the record structure is different from the vanilla `PostgresConnector`, these tests required change on what is being asserted and the way values are being read from the source records being produced by the plugin `yboutput`. This PR enables the following tests by adding the changes described: a. `TransactionMetadataIT` b. `SourceInfoTest` c. `ReplicaIdentityTestMapperTest` d. `PostgresSkipMessagesWithoutChangeConfigIT` e. `PostgresOffsetContextTest` f. `PostgresMoneyIT` g. `PostgresMetricsIT` h. `PostgresErrorHandlerTest` i. `PostgresConnectorConfigDefTest` j. `FieldTest` k. `DebeziumEngineIT`
This PR changes the name of the jar file which gets generated once the connector is compiled. We have changed the name from `debezium-connector-postgres` to `debezium-connector-yugabytedb`. Additionally, this PR modifies the `name` tag in the `pom.xml` file with the name of the source connector.
…tity CHANGE (#147) ## Problem Suppose we create a table which has a non-key column which also has a constraint `NOT NULL` - when the connector will start and CDC service will send a `RELATION` message, the column will be marked as "required" since a `NOT NULL` column means that the value should be present. But for a table with replica identity `CHANGE` the column will not be present in `UPDATE` operations if the column is not updated. This will violate the assumption of the column being marked as "required" since the value will not be present and the connector will end up throwing error of a similar format as: ``` org.apache.kafka.connect.errors.DataException: Invalid value: null used for required field: "name", schema type: STRUCT ``` ## Solution This PR introduces a fix to the above problem by adding a check while decoding the relation message - the check essentially verifies that if the replica identity of the table is `CHANGE` then we explicitly mark the column as optional **if it is a non key column**.
## Problem The Postgres JDVC driver jar being used currently has a `CRITICAL` vulnerability `CVE-2024-1597` as identified by `trivy` in the version `42.6.0`. ## Solution For the fix, this PR upgrades the jar to a driver version `42.6.1` which does not have the vulnerability.
This PR adds a GitHub action to the repository which can be run manually to publish a Docker image to Quay and create a GitHub release draft with a fat jar.
This PR fixes the path for the connector jar file to pick up the correct artefact to be packaged in the Docker image.
## Problem
Suppose we have a column with a non-basic value like `timestamp` - the
code for converting the values here is following:
```java
if (connectorConfig.plugin().isYBOutput()) {
if (value != null && !UnchangedToastedReplicationMessageColumn.isUnchangedToastedValue(value)) {
value = converter.convert(((Object[]) value)[0]);
Struct cell = new Struct(fields[i].schema());
cell.put("value", value);
cell.put("set", true);
result.put(fields[i], cell);
} else {
result.put(fields[i], null);
}
} else {
result.put(fields[i], value);
}
```
The `else` block above has a missing line which should utilise a
converter for the value.
## Solution
This PR modifies the `PostgresSchema#getTableSchemaBuilder` method to
return the default `TableSchemaBuilder` when the plugin is NOT
`yboutput`.
Additionally, this PR fixes the `else` block in `PGTableSchemaBuilder`
with the required conversion and adds tests verifying the same for
`pgoutput` by adding the following logic:
```java
...
else {
value = converter.convert(value);
result.put(fields[i], value);
}
...
```
This PR adds the logic to throw an exception if a connector is deployed with `decimal.handling.mode=PRECISE`.
This PR adds a test which works with both `pgoutput` and `yboutput` and tests that if we update the column of any type, we should get the relevant change event.
This PR introduces the following change: * Version of the pom in the jar file will now be the same as the one being specified in the input form
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR upgrades the base debezium version from
2.5.2.Finalto2.7.2.Final