[Stateful DataLoader] CI takes too long for Mac and WIndows

[andrewkho]

🐛 Describe the bug

CI for Ubuntu / Linux in stateful dataloader takes about 20 seconds, but for mac and windows takes about 30+ minutes. Not only burning a huge amount of CI compute budget but slowing down iteration.

Linux probably runs so quickly because it's able to fork a process instead of spawning. The reason not to drop them is if the CI would have caught bugs that would not appear on Linux tests, and since we're not building native code in Stateful DataLoader, I'd assume it's unlikely.

Versions

Nightly