Support avro schema evolution #1305
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In order to support avro schema evolution, add naive default values to each field (except the timestamp, which ldms metric sets will always contain). Technically for the default form of schema evolution in kafka, we only need to have defaults for any _new_ fields that we add when evolving a schema. However, we are not tracking the various iterations of the schema over time, so just setting defaults for all fields allows us greater flexibility.
The name alone of the avro schema in the kafka registry is insufficient to match an ldms schema. A schema in the kafka registry is uniquely identified by a combination of a name and an integer ID. The schema with the same name can evolve over time, and each iteration if given new unique ID. We want to _always_ generate an avro schema, and attempt to push it to the registry. When the schema hasn't changed, the kafka registry handles the attempt to add a duplicate gracefully. The existing ID is used. But when the schema is different, the registry applies schema compatibility checks, and if they pass it updates the schema with a new ID. This patch first looks up the schema name in the registry, then generates its own schema and attemps to push it. This allows us to generate informational messages about whether the schema is new, existing, or updated/evolved.
|
Fixes #1262 |
|
@tom95858 ping. I've been able to evolve schemas multiple times with these patches, and I'm carrying these in the toss release of ldms. We couldn't be able to function without them. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pair of patches allows avro schema evolution in a kafka registry. It required two changes: