Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write virtual references to Icechunk #1

Draft
wants to merge 43 commits into
base: main
Choose a base branch
from
Draft

Write virtual references to Icechunk #1

wants to merge 43 commits into from
+437 −53

Commits on Sep 27, 2024

  1. move vds_with_manifest_arrays fixture up

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    7b00e41 View commit details
    Browse the repository at this point in the history

  2. sketch implementation

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c82221c View commit details
    Browse the repository at this point in the history

  3. test that we can create an icechunk store

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    d29362b View commit details
    Browse the repository at this point in the history

  4. fixture to create icechunk filestore in temporary directory

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    2aa3cb5 View commit details
    Browse the repository at this point in the history

  5. get the async fixture working properly

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    f2c095c View commit details
    Browse the repository at this point in the history

  6. split into more functions

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    6abe32d View commit details
    Browse the repository at this point in the history

  7. change mode

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    93080b3 View commit details
    Browse the repository at this point in the history

  8. try creating zarr group and arrays explicitly

    @TomNicholas
    TomNicholas committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    bebf370 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. create root group from store

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    833e5f0 View commit details
    Browse the repository at this point in the history

  2. todos

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    9853140 View commit details
    Browse the repository at this point in the history

  3. do away with the async pytest fixtures/functions

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    030a96e View commit details
    Browse the repository at this point in the history

  4. successfully writes root group attrs

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    90ca5cf View commit details
    Browse the repository at this point in the history

  5. check array metadata is correct

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    b138dde View commit details
    Browse the repository at this point in the history

  6. try to write array attributes

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    6631102 View commit details
    Browse the repository at this point in the history

  7. sketch test for checking virtual references have been set correctly

    @TomNicholas
    TomNicholas committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    e92b56c View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. test setting single virtual ref

    @TomNicholas
    TomNicholas committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    2c8c0ee View commit details
    Browse the repository at this point in the history

  2. use async properly

    @TomNicholas
    TomNicholas committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a2ce1ed View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. better separation of handling of loadable variables

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9393995 View commit details
    Browse the repository at this point in the history

  2. fix chunk key format

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    956e324 View commit details
    Browse the repository at this point in the history

  3. use require_array

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    2d7d5f6 View commit details
    Browse the repository at this point in the history

  4. check that store supports writes

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    8726e23 View commit details
    Browse the repository at this point in the history

  5. removed outdated note about awaiting

    @TomNicholas
    TomNicholas committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    387f345 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. fix incorrect chunk key in test

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    b2a0700 View commit details
    Browse the repository at this point in the history

  2. absolute path

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    4ffb55e View commit details
    Browse the repository at this point in the history

  3. convert to file URI before handing to icechunk

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    f929fcb View commit details
    Browse the repository at this point in the history

  4. test that without encoding we can definitely read one chunk

    @TomNicholas
    TomNicholas committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    e9c1287 View commit details
    Browse the repository at this point in the history

  5. Work on encoding test

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    2fe3012 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/icechunk' into matt/icechunk-enc… 

    …oding
    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    33d8ce8 View commit details
    Browse the repository at this point in the history

  7. Update test to match

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    8aa6034 View commit details
    Browse the repository at this point in the history

  8. Quick comment

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    aa2d415 View commit details
    Browse the repository at this point in the history

  9. more comprehensive

    @mpiannucci
    mpiannucci committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    7e4e2ce View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. add attrtirbute encoding

    @mpiannucci
    mpiannucci committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    9a03245 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Merge pull request #2 from earth-mover/matt/icechunk-encoding 

    Matt/icechunk encoding
    @TomNicholas
    TomNicholas authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9676485 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Fix array dimensions

    @mpiannucci
    mpiannucci committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    bbaf3ba View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Merge pull request #3 from earth-mover/matt/array-dims

    @mpiannucci
    mpiannucci authored Oct 11, 2024
    Configuration menu
    Copy the full SHA
    31945cd View commit details
    Browse the repository at this point in the history

  2. Fix v3 codec pipeline

    @mpiannucci
    mpiannucci committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    49daa7e View commit details
    Browse the repository at this point in the history

  3. Put xarray dep back

    @mpiannucci
    mpiannucci committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    756ff92 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Handle codecs, but get bad results

    @mpiannucci
    mpiannucci committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    8c7242e View commit details
    Browse the repository at this point in the history

  2. Gzip an d zlib are not directly working

    @mpiannucci
    mpiannucci committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    666b676 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2024

  1. Get up working with numcodecs zarr 3 codecs

    @mpiannucci
    mpiannucci committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    9076ad7 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Update codec pipeline

    @mpiannucci
    mpiannucci committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    7a160fd View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Merge pull request #4 from earth-mover/matt/v3-codecs 

    Fix v3 codec pipeline
    @mpiannucci
    mpiannucci authored Oct 15, 2024
    Configuration menu
    Copy the full SHA
    286a9b5 View commit details
    Browse the repository at this point in the history

  2. oUdpate to latest icechunk using sync api

    @mpiannucci
    mpiannucci committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    8f1f96e View commit details
    Browse the repository at this point in the history