Big Dataset Examples

[satyaog]

No description provided.

[satyaog]

quick urls :

• https://mila-docs--163.org.readthedocs.build/en/163/Minimal_examples.html#data-handling-during-training
• https://github.com/satyaog/mila-docs/tree/big_datasets/docs/examples/data/hf
• https://github.com/satyaog/mila-docs/tree/big_datasets/docs/examples/data/torchvision

[satyaog]

I actually failed to understand how HF can allow the use of a custom download of the dataset the pile yet but I plan to add another example with that dataset

[lebrice]

Wouldn't we want to extract the archives into SLURM_TMPDIR in the sbatch script?

[satyaog]

Yes I was also thinking about that and the currently strategy in job.sh should work with most of the torchvision datasets (particularly for ImageNet which needs reorganization steps done by torchvision) but for datasets that are not in the torchvision's list then yes it's much simpler to do the extracting in job.sh so I'll find another dataset for it

[btravouillon]

Waiting for merge of #161.

[lebrice]

This might fit nicely in good_practices, what do you think?

[lebrice]

Suggested change

	.. literalinclude:: examples/data/torchvision/job.sh.diff
	.. literalinclude:: job.sh.diff

[lebrice]

Suggested change

	.. literalinclude:: examples/data/torchvision/main.py.diff
	.. literalinclude:: main.py.diff

[lebrice]

Suggested change

	.. literalinclude:: examples/data/torchvision/data.py
	.. literalinclude:: data.py

[satyaog]

@lebrice did you had time to check the recent updates to this PR?

[lebrice]

@lebrice did you had time to check the recent updates to this PR?

Not fully, but a glance, my comment here doesnt seem to have been addressed: #163 (comment)

Edit: Okay I've looked at it now, my previous comments about the content are still relevant (for the most part).

[lebrice]

Sorry, same comment (third time I make it): #163 (comment)
Let me know what you think.

[satyaog]

So I think the only issues remaining were the main.py diff and the good_practices general section. As I failed to make sphinx land directly on the data example if it is alone in it's category, I've move it to good_practices.

[satyaog]

https://mila-docs--163.org.readthedocs.build/en/163/examples/good_practices/data/index.html#

[lebrice]

Let me clarify the comment #163 (comment) :

What I'm saying is that I don't really see the value in having the main.py file included in this example, or showing a diff with respect to the single-gpu job's main.py (you did address this part by removing the diff, thanks!). In my opinion, the main "body" of the example is data.py, and showing how to use srun to launch the data pre-processing commands (with all the resources of each node, instead of the usual srun with one task per gpu) before training.

What do you think?

[lebrice]

To be clear, if you feel like you want to merge this, then sure, it's fine as-is. I was just hoping that perhaps we could re-focus the example a bit so it doesn't dilute or mix up the important part of the content with what's already in the GPU job example.

One other thing: Why do we allow customizing the number of workers for data preparation? Is there a context in which we don't want to use the number of data preparation workers = number of cpus per node?

[satyaog]

Nah not on the cluster, people will use all CPUs available, this is mostly a left over from the scripts I'm personally using to preprocess datasets (at least the bash version). We're also showing a very good practice which is to not override environnement variables if they exists but I'm ok with removing it.

I agree for the main.py but then I think we could have a main that benches the dataloading time for a couple of epochs so people can use it to check the difference between multiple filesystems. At the same time, dataloading time is really not much if it is compared to the time it takes to train on GPU so might as well keep the training part and remove the accuracy logs. Later on, this example could also be used as a base for an example which shows (or not) the loss in performance for overzealous logs when a lot of cpu-gpu syncs are involved