Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add Avro Schema input processor #1753
feat: add Avro Schema input processor #1753
Changes from 15 commits
be40416
4c4af48
a36cafe
cd8edf9
34d707c
c508005
73f47d7
3b924d4
277ff53
fc2ab8e
3db2eb2
017d635
ba51c45
bdb1dd5
3f35f10
e41e4d6
0f78593
e5249ce
7d927ab
ef792c6
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Incorrect, we support enum as well, is there any other types that you can define with Avro that we dont support after merging this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jonaslagoni As of now, we do not support
map
,fixed
,byte
datatypes. I don't get your point aboutenum
, it is being handled as anEnum Model
separately.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My question is, why are you making this
Note
, and which cases of Avro are we NOT supporting since you highlightrecord
here? 🤨There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In Avro Schema, a JSON Object is defined using one of the two datatypes:
map
andrecord
We do not handle the
map
datatype as I am not able to think of any use case. What would we been seeking in input object with single datatype. This is the reason I mentioned this explicitly in note.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maps can easily be converted to our DictionaryMetaModel, with key being StringMetaModel and the value what ever is defined 🙂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't know this could be converted to
DictionaryMetaModel
.Below is the example I am considering. Please suggest any changes if required 😄
Also, Just wanna check if there is any use case of
fixed
andbyte
datatype defined in Avro Schema or not.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is the Avro specification so bad at explaining what is possible... Can you have the
value
be an array or record? 🤨 Or is only primitive types allowed?fixed
andbyte
I would convert to string models. We don't have official byte representation in the core models.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need to check for this non primitive value type. However in my opinion I don't think non primitive is supposed to be there else it would have been explicitly mentioned there as done for the fields attribute in record type
https://avro.apache.org/docs/1.11.1/specification/#schema-record
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.