Explicit checkpoint wait on Tablet Split is now based on LastRecordCheckpoint

[adithya-kb]

We will populate the last seen record's checkpoint in the "lastRecordCheckpoint" field. This is meant to have the explicit checkpoint details of the last seen record from any response (i,e the from_op_id and the record’s commit_time). We will try to update this value in every GetChanges call.

After “lastRecordCheckpoint” is updated as needed, we will then see the response checkpoint from the current GetChanges response. If the response checkpoint is higher :

1. We will store the response’s checkpoint, if the response’s checkpoint is higher than “lastReadRecordCheckpoint”
2. We will add this tablet to a wait-list, where we will first wait until the ExplicitCheckpoint of the tablet is equal to: “lastRecordCheckpoint”.
3. After the ExplicitCheckpoint was equal to “lastRecordCheckpoint”, then we can update the ExplicitCheckpoint value of the tablet to the response’s checkpoint i.e directly update ExplicitCheckpoint in the “tabletToExplicitCheckpoint“ map.

This will take care of updating the ExplicitCheckpoint in cases of NoOp messages in the RAFT WAL , and also take care of tablets splitting right after a previous split in which case there will be no records to checkpoint.

[vaibhav-yb]

The changes seem fine, I am a little skeptical about the comment I provided, please confirm once if that will not have unintended effects.

Another change I remember you added on the service side was a change in proto files, I believe you will need to update the the yb-client version here in the pom.xml as well. The change is purely a connector side change.

[vaibhav-yb]

This will call the GetChanges API by setting the from_op_id as lastRecordCheckpoint, are we sure that won't have any side effects?

[adithya-kb]

Yes, need to revert this to cp again

[vaibhav-yb]

Two things, in the YugabyteDBStreamingChangeEventSource class:

1. In the handleTabletSplit method, add a line to remove the instance of split tablet from the explicit map:

// Remove the corresponding entry for explicit checkpoint as well.
tabletToExplicitCheckpoint.remove(splitTabletId);

2. Initialise the explicit checkpoint for children tablets in the addTabletIfNotPresent:

tabletToExplicitCheckpoint.put(p.getId(), OpId.from(pair.getCdcSdkCheckpoint()).toCdcSdkCheckpoint());