Thanks for opening this issue @msaroufim !

On top of model training time and accuracy, I think we'll also want to monitor the time for the DataLoader to yield an entire epoch (or 5), without a training loop. Ultimately we do care about training time, but it depends a lot on the GPU (and the number of GPUs).

Regarding the vision models to benchmarks, I would suggest the following instead of Resnet50 and Resnet128:

• large batch + small model: IO bound. mobilenet_v3_large with batch-size 128
• small batch + large model: compute heavy. resnext50_32x4d batch-size 32

(this is taken from past investigations from @datumbox (unrelated to datapipes)).

I spent a lot of time porting the torchvision training references to use datapipes. I don't think they're suitable for the kind of benchmark we want to do here (because they support tons of other training features, so they're too complex to be public as-is), but they could be a good start. Happy to get you started if you need.

I spent a lot of time porting the torchvision training references to use datapipes. I don't think they're suitable for the kind of benchmark we want to do here (because they support tons of other training features, so they're too complex to be public as-is), but they could be a good start. Happy to get you started if you need.

FYI I just published this PR pytorch/vision#6196 which adds datapipe support to torchvision's classification training reference (without all the complex async-io stuff).

DataLoaderV2 doesn't support the DistributedReadingService right now so I'm sticking to DL1, but I'll start running more intensive benchmarks on my side as well.

Some basic results, which are consistent with what I had a few months ago:

Benchmarking mobilenet_v3_large (io bound) from the torchvision training references (pytorch/vision#6196) on the AWS cluster, distributed over 8 A100 GPUs with 12 workers each. This is a very typical setup that we use constantly.

• On fsx (pretty slow file system):
  • training with datapipes is ~30% faster than with map-style datasets. But strangely, datapipe epochs seem to take increasinly longer (see details below).
• On ontap (fast file system):
  • training with datapipes is ~10% slower than with map-style datasets.

The ontap reports are more relevant, because in general there is no reason to use the slow fsx file system.

I will start running more in-depth experiments, e.g. completely removing the model-training part, to see if we can identify what could cause such stark differences.

Details

python -u ~/slurm/run_with_submitit.py --ngpus 8 --nodes 1 --model mobilenet_v3_large --epochs 5 --batch-size 128 --workers 12 --ds-type $ds_type --fs $fs

For ref: running just the model training with a pre-loaded dataset (no IO, no transforms) takes ~13 mins both both datapipes and mapstyle datasets. This is the "best" possible training time, assuming data-loading time is zero.

Note: we should ignore the first epoch because these file-system are sensitive to warm-up / caching.

file-system = fsx
ds-type=dp
Epoch: [0] Total time: 0:15:07
Epoch: [1] Total time: 0:15:36
Epoch: [2] Total time: 0:16:42
Epoch: [3] Total time: 0:19:26
Epoch: [4] Total time: 0:20:41
Training time 1:32:03

file-system = fsx
ds-type=mapstyle
Epoch: [0] Total time: 0:22:09
Epoch: [1] Total time: 0:24:51
Epoch: [2] Total time: 0:26:21
Epoch: [3] Total time: 0:25:50
Epoch: [4] Total time: 0:25:40
Training time 2:10:07

file-system = ontap
ds-type=dp
Epoch: [0] Total time: 0:10:02
Epoch: [1] Total time: 0:04:12
Epoch: [2] Total time: 0:04:10
Epoch: [3] Total time: 0:04:10
Epoch: [4] Total time: 0:04:10
Training time 0:28:01

file-system = ontap
ds-type=mapstyle
Epoch: [0] Total time: 0:07:32
Epoch: [1] Total time: 0:03:46
Epoch: [2] Total time: 0:03:47
Epoch: [3] Total time: 0:03:46
Epoch: [4] Total time: 0:03:45
Training time 0:23:40