AMD GPUs (even NAVI22 and lower) working in Linux

[cstueckrath]

So, after some hard work, I got this running.
https://www.youtube.com/watch?v=d_CgaHyA_n4 helped a lot to get the base understanding, what went wrong in my previous attempts

I will try to get a version running on Ubuntu 20.04 without the docker stuff, but if you follow the video, you'll get a more or less working version.

Changes:

1. if on ubuntu, pay attention to the radeon repositories you'll add on the host. I'll comment on this later
2. Triple check the group membership (video, render, docker)
3. run ./webui.sh once until it breaks because of missing nVidia GPU
4. you have to use pip3 install --force-reinstall torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.2 inside (ldm), once the script is finished (in the video, this is done outside of the venv. Conda needs to be initialized for bash before, but it will tell you what to do, if you run conda activate)
5. you have to export HSA_OVERRIDE_GFX_VERSION=10.3.0 and (optionally) export PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:128 (add them to /etc/environment) if you want to use an "unsupported" GPU. If you have NAVI21 or better, this is not necessary

[mchaker]

WOO AMD!
Oh, I just used a docker container. It worked almost from default when using the amdih ROCm container... let me see if I can find my command history for the missing dependencies

[cstueckrath]

ah, btw.
no need to use the navi21 image from pytorch. Just use :latest

[mchaker]

starting from:

docker run -it --privileged --workdir="/data" --volume /etc/group:/etc/group:ro --volume /etc/passwd:/etc/passwd:ro --volume /etc/shadow:/etc/shadow:ro --volume /etc/sudoers:/etc/sudoers:ro --volume /data:/data:rw --group-add plugdev --group-add video --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined -p 7860:7860 --name pytorch-rocm amdih/pytorch:rocm5.0_ubuntu18.04_py3.7_pytorch_1.10.0

cd /data/github/hlky/stable-diffusion-webui/
pip install gradio pydantic pynvml einops omegaconf gfpgan basicsr realesrgan pytorch-lightning clip transformers kronia 
cd /data/github/harubaru/waifu-diffusion/
pip install -r requirements.txt
cd /data/github/hlky/stable-diffusion-webui/
python3 relauncher.py --port 9999

[cstueckrath]

ok, no need for docker at all.

Steps to reproduce a working environment with Ubuntu 20.04:

1. add apt sources (key: https://repo.radeon.com/rocm/rocm.gpg.key)

• deb https://repo.radeon.com/amdgpu/latest/ubuntu focal main (can be added with current Linux driver package from AMD)
• deb [arch=amd64] https://repo.radeon.com/rocm/apt/latest ubuntu main

2. install rock-dkms
3. install a rocm runtime like rocm-opencl-runtime (hip might work too, didn't try) and rocminfo
4. sudo update-initramfs -u
5. reboot!
6. add yourself to groups "video" and "render" and check with "groups" that it worked (might have to log off)
7. lsmod | grep amdgpu

amdgpu              10477568  34
amddrm_ttm_helper      16384  1 amdgpu
amdttm                 86016  2 amdgpu,amddrm_ttm_helper
iommu_v2               24576  1 amdgpu
amd_sched              53248  1 amdgpu
amdkcl                 28672  3 amd_sched,amdttm,amdgpu
drm_kms_helper        307200  3 amdgpu
drm                   618496  17 drm_kms_helper,amd_sched,amdttm,amdgpu,amddrm_ttm_helper
i2c_algo_bit           16384  1 amdgpu

Working so far? Good!

8. install git and anaconda and initialize anaconda, so you have (base) at your prompt in a new terminal window (don't use sudo to run the installer. Conda should not install to /root)
9. export HSA_OVERRIDE_GFX_VERSION=10.3.0 if running an unsupported GPU like 6700XT
10. make a directory for sd somewhere in your home, git clone the repo and run ./webui.sh . Configure and wait for it to fail.
11. conda activate ldm
12. pip3 install --force-reinstall torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.2
13. run script again -> SD and webui is working!

[rokwenelenya]

I am going to try this solution as I have a 6800xt, however I don't understand one thing, you install git and anaconda first and then run the script for SD, but the script will also install anaconda and git right? so why you need to install them first and then run the script? sorry if I am asking a dumb question

[cstueckrath]

the version of the script I was using didn't handle those missing prerequisites at that time.

[rokwenelenya]

I imagined that, the whole SD scene is moving really fast, BTW, this solution works great thank you for posting it, I was using one with docker but using this one is a breeze, I found out that every time you run it the first time you get this message while everything is loading:

MIOpen(HIP): Warning [SQLiteBase] Missing system database file: gfx1030_36.kdb Performance may degrade. Please follow instructions to install: https://github.com/ROCmSoftwarePlatform/MIOpen#installing-miopen-kernels-package

I went to check out the link but I don't know if it's worth the effort, for what I understand installing this will help that first run to go faster, but it doesn't seem to do much else, I mean everything already works without it, did you tried that?

[cstueckrath]

yeah, kernel has to be compiled each time you start. Docker had those preinstalled. I cant't figure out how to do that, yet.
But it would only safe a few seconds and even that only on the first run, so i won't touch my system until I know how to do this properly.

[rokwenelenya]

I fully agree with you brother, if i'ts working, why mess with it? :D

[Hydroidev]

I got this running on Manjaro 21.2 with my RX 6700 XT. I didn't find any newer instructions elsewhere and since there are different packages involved, I thought I'd make a post for getting it running on Manjaro/Arch so everything is in one place. Mainly just the packages and dependencies are different:

1. Install packages rocm-hip-sdk rocm-opencl-sdk from the AUR, either manually or with an AUR helper like yay, paru, or similar.
   • There are a lot of packages and a couple of them had issues building for me. The following issue pages helped resolve those:
     • pkgconf is required for building these packages, so be sure to install that first if you need it
     • Compilation error in format.h; requires manual edit to a library header file
     • The compiler packaged with hip (hipcc) is never added to path so at some point you may need to run PATH="$PATH:/opt/rocm/bin" to add it temporarily while building
     • rocm-llvm-mlir bad validity check; you'll have to edit the PKGBUILD and run makepkg -i to fix this one until the maintainers fix this
   • Building all of these took an obscene amount of time on my system w/ R9 3900X, so you may want to add arch4edu to download the binaries directly, although there have been some problems with this repo recently from what I can tell.
   • See more instructions for this step in the rocm-arch repo
2. Run mkinitcpio -P to refresh initramfs
3. Reboot
4. Check that the gpu is recognized with lsmod | grep amdgpu

amdgpu               9379840  47
gpu_sched              49152  1 amdgpu
drm_ttm_helper         16384  1 amdgpu
ttm                    94208  2 amdgpu,drm_ttm_helper
drm_display_helper    180224  1 amdgpu

5. Install the webui as per the guide
6. conda activate ldm
7. pip3 install --upgrade torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.1.1
8. export HSA_OVERRIDE_GFX_VERSION=10.3.0
9. Then you should be able to run the WebUI script

I had a rough time following the install guide, but I did manage to get Steamlit to work. RealESRGAN locks up my 6700 XT for reasons I don't understand yet, but everything else works as expected.