[Example] Multi node training on XPU device

[zhouyu5]

This is a follow-up PR for issue #9464 .

I refer to distributed_sampling_multinode.py for a XPU version of multi node training.

I use two nodes for testing, each nodes with 1 XPU card.

See the following screenshot, it has passed the test in internal cluster.

[zhouyu5]

@DamianSzwichtenberg I plan to add doc for it, where should I put the doc, in the current folder?

[rusty1s]

Thanks. Is there any reason we cannot merge this with multi_gpu/distributed_sampling.py?

[zhouyu5]

Thanks. Is there any reason we cannot merge this with multi_gpu/distributed_sampling.py?

@rusty1s Good question. Actually, I had the same confusion as you. I think it is possible to merge it with multi_gpu/distributed_sampling.py.

However, I want to clarify the reason why I didn't merge them. As a new user, I believe it’s essential to follow the existing conventions of the repo. I noticed that for NVDA GPUs, there are two files provided: distributed_sampling.py and distributed_sampling_multinode.py. Following this format, I created separate files for XPU, which is distributed_sampling_xpu.py and distributed_sampling_multinode_xpu.py (note the xpu suffix).

Additionally, if you closely examine the code under multi_gpu, you'll find several redundant pieces of code. Some variations are due to different datasets or training configurations. I counted a total of 12 files for NVDA GPUs. Ideally, by introducing some conditional branches in the code, we could consolidate many of these files, potentially reducing them to 3-4 files. What are your thoughts on this approach?

[zhouyu5]

@rusty1s Update with readme and guide. Please have a check. Thanks.

[akihironitta]

This is great! Do you have an idea which part of the example script is XPU-specific? We could consider merging this file with distributed_sampling_multinode.py if the only change is in the launch command and the torch.device.