-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explicit checkpoint wait on Tablet Split is now based on LastRecordCheckpoint #246
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes seem fine, I am a little skeptical about the comment I provided, please confirm once if that will not have unintended effects.
Another change I remember you added on the service side was a change in proto files, I believe you will need to update the the The change is purely a connector side change.yb-client
version here in the pom.xml
as well.
// At this position, we know we have received a callback for split tablet | ||
// handle tablet split and delete the tablet from the waiting list. | ||
|
||
// Call getChanges to make sure checkpoint is set on the cdc_state table. | ||
LOGGER.info("Setting explicit checkpoint is set to {}.{}", explicitCheckpoint.getTerm(), explicitCheckpoint.getIndex()); | ||
setCheckpointWithGetChanges(tableIdToTable.get(part.getTableId()), part, | ||
cp, explicitCheckpoint, schemaNeeded.get(part.getId()), | ||
lastRecordCheckpoint, explicitCheckpoint, schemaNeeded.get(part.getId()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will call the GetChanges
API by setting the from_op_id
as lastRecordCheckpoint
, are we sure that won't have any side effects?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, need to revert this to cp again
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two things, in the YugabyteDBStreamingChangeEventSource
class:
- In the
handleTabletSplit
method, add a line to remove the instance of split tablet from the explicit map:
// Remove the corresponding entry for explicit checkpoint as well.
tabletToExplicitCheckpoint.remove(splitTabletId);
- Initialise the explicit checkpoint for children tablets in the
addTabletIfNotPresent
:
tabletToExplicitCheckpoint.put(p.getId(), OpId.from(pair.getCdcSdkCheckpoint()).toCdcSdkCheckpoint());
We will populate the last seen record's checkpoint in the "lastRecordCheckpoint" field. This is meant to have the explicit checkpoint details of the last seen record from any response (i,e the from_op_id and the record’s commit_time). We will try to update this value in every GetChanges call.
After “lastRecordCheckpoint” is updated as needed, we will then see the response checkpoint from the current GetChanges response. If the response checkpoint is higher :
This will take care of updating the ExplicitCheckpoint in cases of NoOp messages in the RAFT WAL , and also take care of tablets splitting right after a previous split in which case there will be no records to checkpoint.