feat(python, rust): add drop column operation [BLOCKED]
#2710
Conversation
ion-elgreco
requested review from
wjones127,
fvaleye,
roeap,
rtyler and
hntd187
as code owners
ion-elgreco
changed the title
drop column operationdrop column operation
ion-elgreco
force-pushed
the
feat/drop_columns
branch
from
2add092
to
8c65c58
Compare
github-actions
bot
added
binding/python
| /// Specify if you want to raise if the specified column does not exist | ||
| pub fn with_raise_if_not_exists(mut self, raise: bool) -> Self { | ||
| self.raise_if_not_exists = raise; | ||
| self | ||
| } |
There was a problem hiding this comment.
Question out of curiosity: What would be the use case or reason to allow drop column commits for columns that are missing from the schema?
There was a problem hiding this comment.
Let's say I drop 10 columns, and one is a typo, it could still be useful for the commit to pass.
I probably need to add another check though to check if the schema actually changed, otherwise the commits are pointless
|
Don't drop columns require column mapping otherwise you have to re-write all the datafiles? |
Why would it? Column mapping would help when we rename and don't want to rewrite but dropping is just a logical change Maybe I'm missing something that column mapping would bring though. |
|
https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mapping says so right here in the protocol |
It doesn't describe though why it's needed though? Is it because you could potentially re-add the same column name but have completely different type or data? |
Co-authored-by: Guilhem de Viry <[email protected]>
ion-elgreco
force-pushed
the
feat/drop_columns
branch
from
b463fd6
to
dcca23c
Compare
|
@hntd187 added a check to see if column mapping is enabled. I'll put in DRAFT until we have actual column mapping support |
Pull request was converted to draft
ion-elgreco
changed the title
drop column operationdrop column operation [BLOCKED]
Description
Adds a drop column operation, can be used to drop root fields and nested fields in a struct.