Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not reset state when there's only addition of new partitions to the consumer #28

Open
mosyp opened this issue Jun 11, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@mosyp
Copy link
Contributor

mosyp commented Jun 11, 2021

Whenever rebalance event is happening, each workers is revoked and then assigned the list of partitions each. As a result, consumer might get new partitions or remove existing. In sake of simplicity and safety in both cases the consumer state is reset and it start consuming messages from previous commit/offset.

However, in theory if the partitions is only added to the consumers and all of existing are kept with a consumer it makes sense to not reset the state for performance reasons.

See https://github.com/delta-io/kafka-delta-ingest/pull/27/files/8e598e35406f5d2978306134ddde9d9d54b4dccc#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R420 for inspiration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant