Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dask Chunking Approach: Example in Documentation #65

Open
dougbrn opened this issue Oct 17, 2024 · 0 comments
Open

Dask Chunking Approach: Example in Documentation #65

dougbrn opened this issue Oct 17, 2024 · 0 comments
Assignees
Labels
documentation Improvements or additions to documentation

Comments

@dougbrn
Copy link
Collaborator

dougbrn commented Oct 17, 2024

We've discussed there being motivation for a chunking approach as an alternative to sending massive task graphs to Dask. The main appeal being that chunking can potentially provide a more memory-stable compute at the cost of adding some looping overhead to the overall performance, which would help users that run into dask issues avoid dask troubleshooting as their only path forward.

@wilsonbb and I talked about this in more depth, and we came to the conclusion that the likely best output of this would be to have an example within our documentation that shows how one would do this on something like workflow in #42 . This is preferable to building a bespoke chunk function, as a built-in function would have many limitations regarding the graphs it can chunk (for example anything where a global value is computed) and therefore may set bad expectations for users. And building something that's more general would risk building an entire dask streaming interface that directly competes with Dask's workflow.

The first step to this is to actually verify that a chunking approach performs well, which @wilsonbb has agreed to explore as part of his exploration in #42

@dougbrn dougbrn added the documentation Improvements or additions to documentation label Oct 17, 2024
@wilsonbb wilsonbb self-assigned this Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

2 participants