Does materialize require the dataset to be loaded into memory?

[davidfstein]

I have a large dataset, ~30 million x ~500. I would like to stream this data from the disk in batches, but I'm wondering if materialize is compatible with this? Does materialize load the entire data into memory to compute column statistics?

[akihironitta]

Currently, yes, unfortunately. To unblock your use case, you'd need to patch Dataset.materialize and data loading in a way that you don't have to load the entire dataset in memory when sampling mini-batches.

[davidfstein]

Got it thank you. Do you anticipate supporting this use case? I can probably help writing a pull but maybe this is already on the roadmap?

[akihironitta]

No, I don't think there's any plan at the moment to support the use case.