Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make kerchunk dependency entirely optional #258

Open
TomNicholas opened this issue Oct 17, 2024 · 0 comments · May be fixed by #259
Open

Make kerchunk dependency entirely optional #258

TomNicholas opened this issue Oct 17, 2024 · 0 comments · May be fixed by #259
Labels
dependencies Updates a dependency Kerchunk Relating to the kerchunk library / specification itself

Comments

@TomNicholas
Copy link
Member

Virtualizarr doesn't need to have an explicit dependency on Kerchunk. Kerchunk is one (well it reads many formats) amongst many readers for creating virtual references, and it is now one of two options for writing virtual references (the other being Icechunk). Issue #78 points this out for the readers, but we could make the entire package only optionally depend on kerchunk.

A more urgent reason to change this is Kerchunk does not currently support zarr-python v3 (fsspec/kerchunk#516), which is preventing us ensuring that the rest of this package works with zarr v3 (#182), which is needed for testing Icechunk compatibility (#256).

Changing the virtualizarr business logic to make kerchunk optional is easy - the hard part is that lots of our tests are more reliant on kerchunk than they need to be. Ideally we would either rewrite those integration tests to start from in-memory references rather than files on disk, or use a non-kerchunk approach to generate references from example HDF5 files (#87). I think the former would be neater, but alternatively once the #87 is merged the latter would be a simple change.

@mpiannucci @norlandrhagen @ghidalgo3 @sharkinsspatial

@TomNicholas TomNicholas added Kerchunk Relating to the kerchunk library / specification itself dependencies Updates a dependency labels Oct 17, 2024
@TomNicholas TomNicholas linked a pull request Oct 17, 2024 that will close this issue
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Updates a dependency Kerchunk Relating to the kerchunk library / specification itself
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant