Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use xarray and Dask to load and save results #59

Open
wtbarnes opened this issue Jul 22, 2019 · 2 comments
Open

Use xarray and Dask to load and save results #59

wtbarnes opened this issue Jul 22, 2019 · 2 comments

Comments

@wtbarnes
Copy link
Member

xarray has good support for gridded data. Dask is good for lazily loading data. Might be worth investigating how this would make loading and quick looking data easier without having to load all of the data from a single run into memory.

@wtbarnes
Copy link
Member Author

If we're going to use xarray anyway, it may be worth investigating the xgcm package for representing the numerical grid in each strand.

@wtbarnes
Copy link
Member Author

Thinking about this more, I think this may be the most important issue on this repo. It would constitute a significant refactor of a lot of the code, but would also mean we could do away with almost all of the visualization code which would be great!

There are a few ways we could go about this.

  • Every Profile becomes a thin wrapper around an xarray.DataArray object with a single coordinate corresponding to the strand length.
  • Every Strand becomes a thin wrapper around an xarray.Dataset object with an additional coordinate that spans the Profile objects that constitutes time. I'm not completely sure how to do this w.r.t. the time dimension spanning multiple data arrays, but I think it is possible.

Alternatively, we could just build one big coordinate grid for all spatial grids at all timesteps. Computationally, this may be challenging given the possible number of timesteps and the size of the grids, unless we can somehow load them lazily.

Another challenge to consider is unit support. Currently, pydrad uses astropy.units everywhere for unitful computations. Unfortunately, xarray has no support for unitful computations and especially has no support for astropy units. As such, we would need to think about ways to still support returning astropy.units.Quantity objects when applicable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant