Is there a way to use multi GPU for more VRAM?

[sazabidesu]

Would there be a way to be able to use more than one GPU to have more total VRAM available?

[ZeroCool940711]

Multi-GPU would be something really nice to have, we could for example load some of the models on one GPU and the rest on another GPU, something that comes to mind is loading GFPGAN and RealESRGAN both in one or each on a different GPUs to speed up those parts of the generation while the main model is on the main GPU. I'm giving this example because in my case I have three GPUs on my computer, two of them are GTX 750 Ti with 4 GB of VRAM each while the main one is an RTX 3059 with 8 GB of VRAM. I normally use the RTX for everything as it has more VRAM but the other GPUs could still be used for other stuff, in my case if we could split the models into each GPU it would speed up things a lot because after the main GPU generates the image the other GPUs would then upscale and fix the faces in parallel and the main GPU would continue to generate images without stopping to do anything else, that would be one way to implement that. Of course not everyone has three GPUs on their computer, with two GPUs doing the same process would speed things up, it could also be used to do everything using all the GPUs at the same time and that would also speed things up a lot.

[hlky]

Try this branch please
#264
https://github.com/hlky/stable-diffusion-webui/tree/264-extra-models-gpu-gfpgan-gpu-id-esrgan-gpu-id

[sazabidesu]

Yeah, that would be awesome, in my case, my second card is a 1660 TI that gives full green images unless the -- full precision parameter is used which also increments vram consumption. But if I could use the vram it has and add it to the first card, or like you mention, load specific models into it. Would make creating bigger and more detailed images, easily without the need to go for a single bigger card.

[hlky]

Loading the extra models to the other gpu would be easy enough to implement afaik, good opportunity to take a look at the cpu options too as some people have reported issues with those, although RealESRGAN on cpu worked for me when I implemented it (that was with --optimized too)

I could add --extra-models-gpu <id> to load both to a certain gpu, and individual options --esrgan-gpu <id> and --gfpgan-gpu <id>

[ZeroCool940711]

I could add --extra-models-gpu to load both to a certain gpu, and individual options --esrgan-gpu and --gfpgan-gpu

Having those options would be nice and they do what I mentioned before, the important part is whether the main GPU continue to generate the other images while on the other GPUs is being processed the face fix and upscaling, if those work in parallel that would provide a huge boost to the speed generation. Also, I think that the reason some people can't use old GPUs with low VRAM is because if you load all three models to the same GPU it uses a lot of VRAM but I think that if we split them even older GPUs with 2-4GB could probably load ESRGAN or GFPGAN.

[hlky]

It would need to used in combination with --gpu to set the main model on that gpu, could probably rename --gpu to --sd-gpu or something to make it clearer

The simplest workaround for now is to remove the esrgan and gfpgan models, then they simply won't load and those features don't activate/appear in the ui

[hlky]

Try this branch please
#264
https://github.com/hlky/stable-diffusion-webui/tree/264-extra-models-gpu-gfpgan-gpu-id-esrgan-gpu-id

[hlky]

Combining vram to be able to generate a larger image? not right now, it might be possible, it might not

If you want to just use a different gpu, that's already implemented with the command line option --gpu

If you'd like to generate on all your gpus, there's a workaround discussed here.
Essentially you just run multiple copies each with their own --gpu option set.
Splitting generation of batches or anything like that across gpus is probably possible but would increase complexity and could cause issues, but if it's going to be a popular feature it can be done.
From what i've seen there aren't many people who would want to do specifically that, it seems most people will use the --gpu option because they have 2 graphics cards and only 1 is good enough.

[sazabidesu]

Thanks, I will try that branch you mentioned. If loading the upscalers on a dedicated gpu could free extra vram on the power horse, then that's good enough.

[hlky]

if you have wanted to use esrgan and gfpgan on different gpus please try this branch and report back

#264
https://github.com/hlky/stable-diffusion-webui/tree/264-extra-models-gpu-gfpgan-gpu-id-esrgan-gpu-id

[sazabidesu]

I tried that branch and got errors. I'm not really sure if I installed it in the wrong way and broke something.

What I did was:

1. Get main repo.
2. Extract main repo.
3. Get --extra-models-gpu --gfpgan-gpu <id> --esrgan-gpu <id> #264 and extract it on top of main repo, replacing anything that comes up.
4. Launch webuildm.cmd

When the ui was working i tried a prompt and got this:

Traceback (most recent call last): File "C:\Users\delfi\.conda\envs\ldm\lib\site-packages\gradio\routes.py", line 247, in run_predict output = await app.blocks.process_api( File "C:\Users\delfi\.conda\envs\ldm\lib\site-packages\gradio\blocks.py", line 641, in process_api predictions, duration = await self.call_function(fn_index, processed_input) File "C:\Users\delfi\.conda\envs\ldm\lib\site-packages\gradio\blocks.py", line 556, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\delfi\.conda\envs\ldm\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\Users\delfi\.conda\envs\ldm\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "C:\Users\delfi\.conda\envs\ldm\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run result = context.run(func, *args) File "scripts/webui.py", line 1007, in txt2img output_images, seed, info, stats = process_images( File "scripts/webui.py", line 761, in process_images if variant_amount > 0.0: TypeError: '>' not supported between instances of 'NoneType' and 'float'

And that's not even trying out the new parameters.

[mchaker]

I have 4 to 8 GPUs with 16GB VRAM each. I would like to run one instance of stablediffusion that spans across all the cards for combined VRAM, as OP mentioned. Will this be possible? I have tried HuggingFace Accelerate, but that did not span multiple GPUs.

[mchaker]

@hlky there has been successful work to use multiple GPUs with a single instance of the web UI:
NickLucche/stable-diffusion-nvidia-docker#5

This seems to be the branch: https://github.com/NickLucche/stable-diffusion-nvidia-docker/tree/dp

Would you be able to integrate such working multi-GPU capabilities into this stable-diffusion-webui? :)

[mchaker]

update: multi-gpu work has been merged into the main branch of NickLucche's repo

[Omegadarling]

@hlky Can you incorporate what NickLucche has done with multi-GPU support (spawning new instances for idle GPUs when GPUs are busy)?
NickLucche/stable-diffusion-nvidia-docker#7