Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option for distcp with STORAGE_MIGRATION #51

Open
dstreev opened this issue Apr 28, 2022 · 0 comments
Open

Add option for distcp with STORAGE_MIGRATION #51

dstreev opened this issue Apr 28, 2022 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@dstreev
Copy link
Owner

dstreev commented Apr 28, 2022

STORAGE_MIGRATION current handles the movement of data through SQL using Hive and the compute layer to run the jobs. There are benefits to this in the sense it provides an opportunity to consolidate and rewrite the data. But sometimes it might just be more efficient to use distcp to move the data and simply change the table's LOCATION information.

This won't work for ACID tables, but could be used for EXTERNAL tables. Changing locations will also mean that we'll need to modify all the partition location information as well.

@dstreev dstreev added the enhancement New feature or request label Apr 28, 2022
@dstreev dstreev self-assigned this Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant