[Feature]: Define partial chunk shape for GenericDataChunkIterator

[bendichter]

What would you like to see added to HDMF?

Right now, for the GenericDataChunkIterator, it's possible to define chunk_mb or chunk_shape. I would like to enable a hybrid approach, where a user could input chunk_mb=10.0, chunk_shape=(None, 64), and the GenericDataChunkIterator would identify the remaining dimension that gets you close to the target chunk size.

Is your feature request related to a problem?

It is pretty common for users to have some insight into the likely read patterns of a dataset.

What solution would you like?

I would like GenericDataChunkIterator to find the maximum size (prod of dims) that is <= the target size. I also would like the chunk to be as cube-like as possible, so I would like to minimize the sum of the dimensions of the array. Previously, we tried building chunks that were scaled down versions of the data shape, similar to h5py, but experience with Jeremy has shown that this approach is poorly suited for common data reading routines, and I think a better naive assumption would be that (hyper-) cube chunks are a good default.

Do you have any interest in helping implement the feature?

Yes.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this change was not already requested?

[oruebel]

@CodyCBakerPhD is this an issue you could help with, since you are most familiar with GenericDataChunkIterator?

[bendichter]

Above is a proposed solution. Obv this needs tests, but I wanted to run it by the group before moving forward

[oruebel]

To be honest, the functionality you describe to me sounds more like a utility function that would be more broadly useful for DataChunk iterators. I.e., this could be a method (e.g., max_chunk_shape that a user would call to get suggested chunk sized that they would then hand to the iterator, e.g.:

GenericDataChunkIterator(chunk_shape=max_chunk_shape(chunk_mb=10.0, chunk_shape=(None, 64)), ...)

This function could either live on it's own in the same module as GenericDataChunkIterator or maybe be a static method on AbstractDataChunkIterator or DataChunk (but I think separately may make sense). The main reason I think this would be useful to do as a utility function is that it:

• Makes the logic reusable
• Makes the logic explicit, since the user calls the function ,rather than being additional "magic" inside the constructor of GenericDataChunkIterator

[oruebel]

Sorry, I didn't see that you made a draft PR, I was referring to the solution you suggested in the issue. Let me take a look at the PR.