Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory consuption in axis_world_coords (again) #798

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Cadair
Copy link
Member

@Cadair Cadair commented Dec 17, 2024

This PR works, as an alternative to #780 by not generating mesh coords for unneeded pixel axes.

i.e. if the pixel axes are not correlated with a requested world axis then don't generate the coords in the mesh.

If the pixel axes are not correlated with a requested world axis then
don't generate the coords in the mesh
@Cadair Cadair marked this pull request as draft December 17, 2024 17:39
@Cadair
Copy link
Member Author

Cadair commented Dec 17, 2024

This PR needs tests which test at least with some DKIST or IRIS WCSes that you can get low-memory axis world coords for uncorrelated axes, and as a counterpoint that correlated axes still work properly.

For example, I have tested that axis_world_coords("time") works fine for this dataset but not that axis_world_coords("lat") works for instance (which should still generate a large grid).

ndcube/ndcube.py Outdated Show resolved Hide resolved
@nabobalis nabobalis added this to the 2.3.0 milestone Dec 17, 2024
@nabobalis
Copy link
Contributor

I do question why I bother to do any programming if its always wrong.

This solution works:

Command line: /home/nabil/.mamba/envs/iris-dev/bin/memray run -f -o output.bin examples/umbral_flashes.py
Start time: Tue Dec 17 2024 11:08:16 GMT-0800 (Pacific Standard Time)
End time: Tue Dec 17 2024 11:08:42 GMT-0800 (Pacific Standard Time)
Duration: 0:00:26.707000
Total number of allocations: 6674037
Total number of frames seen: 17921
Peak memory usage: 2.3 GiB
Python allocator: pymalloc

The biggest allocations are from loading the data:

Screenshot_20241217_111209

@DanRyanIrish
Copy link
Member

This is great news!

@Cadair
Copy link
Member Author

Cadair commented Dec 19, 2024

@nabobalis I can't add a visp WCS without a dkist dependency, I assume you can't either?

@nabobalis
Copy link
Contributor

@nabobalis I can't add a visp WCS without a dkist dependency, I assume you can't either?

I am the same

@nabobalis
Copy link
Contributor

Could we try Wills WCS from his issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants