Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migration of storage layer to delta-rs #307

Merged
merged 21 commits into from
Mar 14, 2023
Merged

Migration of storage layer to delta-rs #307

merged 21 commits into from
Mar 14, 2023

Commits on Feb 23, 2023

  1. Initial migration of create and insert table paths to delta-rs

    @gruuya
    gruuya committed Feb 23, 2023
    Configuration menu
    Copy the full SHA
    7362d0e View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2023

  1. Revise the schema provider implementation to fit with non-cloneable D… 

    …eltaTable
    @gruuya
    gruuya committed Feb 24, 2023
    Configuration menu
    Copy the full SHA
    88b3e53 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2023

  1. Use root internal object_store instead of storage options for managin… 

    …g delta tables
    @gruuya
    gruuya committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    9707fdf View commit details
    Browse the repository at this point in the history

  2. Enable time-travel for Delta tables

    @gruuya
    gruuya committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    97de330 View commit details
    Browse the repository at this point in the history

  3. Implement deletion of table objects on DROP TABLE

    @gruuya
    gruuya committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    6dce354 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2023

  1. Enable ALTER TABLE on object stores by table object renaming

    @gruuya
    gruuya committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    6ae318f View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2023

  1. Fix tests or ignore them if the implementation is still missing

    @gruuya
    gruuya committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    dc2018f View commit details
    Browse the repository at this point in the history

  2. Remove testing crutches used during development

    @gruuya
    gruuya committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    77179e6 View commit details
    Browse the repository at this point in the history

  3. Format Cargo.toml as per tomlfmt

    @gruuya
    gruuya committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    13ce11c View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2023

  1. Pivot to using a uuid as delta table root directory

    @gruuya
    gruuya committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    6838409 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2023

  1. Test legacy tables by bootstraping from an existing catalog

    @gruuya
    gruuya committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    aead9eb View commit details
    Browse the repository at this point in the history

  2. Extend legacy table test to check migration works ok

    @gruuya
    gruuya committed Mar 7, 2023
    Configuration menu
    Copy the full SHA
    c3faf29 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Re-implement DELETE in accordance with delta-rs 

    This uses some private delta-rs code, as well as some WIP code for native delta-rs DELETE support.
    @gruuya
    gruuya committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    513ee02 View commit details
    Browse the repository at this point in the history

  2. Enable UPDATE statements for delta tables

    @gruuya
    gruuya committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    fa65b16 View commit details
    Browse the repository at this point in the history

  3. Skip eager deletion of schema files used initially

    @gruuya
    gruuya committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    dd09c7b View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2023

  1. Implement repository changes needed for lazy cleanup of dropped tables

    @gruuya
    gruuya committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    6b54c1d View commit details
    Browse the repository at this point in the history

  2. Add dropped_tables system table and extend tests

    @gruuya
    gruuya committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    b3e71dd View commit details
    Browse the repository at this point in the history

  3. Persist new Delta table versions in Seafowl catalog

    @gruuya
    gruuya committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    0cbfab9 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Bump to Datafusion 19

    @gruuya
    gruuya committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    0249337 View commit details
    Browse the repository at this point in the history

  2. Move copied delta-rs methods to a separate module

    @gruuya
    gruuya committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    0e2543e View commit details
    Browse the repository at this point in the history

  3. Make table building logic more explicit

    @gruuya
    gruuya committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    9f56b12 View commit details
    Browse the repository at this point in the history