Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send DB backend messages by partition instead of partition_key #23

Open
dorner opened this issue Sep 10, 2019 · 0 comments
Open

Send DB backend messages by partition instead of partition_key #23

dorner opened this issue Sep 10, 2019 · 0 comments
Labels
big change This is a significant amount of work. enhancement New feature or request

Comments

@dorner
Copy link
Member

dorner commented Sep 10, 2019

Currently, DB backend producers can scale out the number of topics but not a single large topic with lots of partitions. To ensure order, only a single thread can send messages from a topic right now.

We should be changing this so that we save the partition of a message instead of a partition_key (using RubyKafka's Partitioner class to generate the partition). We should then add a configuration to allow locking particular topics (or all topics) by partition instead of just by topic. (For small topics this would be overkill.)

This will require a PR to Phobos to allow sending messages with a partition, which it currently does not support, as well as changing the save logic, the read logic, and the migration template.

@dorner dorner added enhancement New feature or request big change This is a significant amount of work. labels Sep 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
big change This is a significant amount of work. enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant