Script doesn't use GPU

[bjoernapfel]

I run my pyGeNN script on ubuntu, despite following the steps V and VI in the GeNN installation process (and ensuring that my GPU, CUDA and everything is compatible and set up), my script somehow only seems to run over my CPU and doesn´t connect to the GPU - it's very slow and when checking with the top program, 100% of my CPU is being used for the process which shouldn´t be the case when actually using the GPU. Nothing changes upon un- and reinstalling. Is this maybe some sort of known error you have a simple solution for?
Maybe it helps clarifying my problem, when running the script I also get multiple messages like

[neworderofjamie]

Did you set the CUDA_PATH environment variable before building PyGeNN?

[neworderofjamie]

You can also verify whether CUDA was correctly detected when building PyGeNN by running the following:

import pygenn
print(pygenn.genn_model.backend_modules)

If the first entry in the dictionary doesn't have a key of "CUDA" then the CUDA backend was not build correctly

[bjoernapfel]

my output when running these lines:

[bjoernapfel]

so I don´t think it is connected to to CUDA not being build correctly

[bjoernapfel]

do you maybe have any other suggestions?

[neworderofjamie]

Apologies, I've been out of the office for a few days. Your output suggests that the CUDA backend was built and loaded correctly so it should be being used. Could you post the output from the nvidia-smi command line tool?

[neworderofjamie]

Interesting! I will investigate this further and fix ASAP. In the mean time, I imagine if you increase the constantCacheOverhead somewhat you'll find a point where it works

[neworderofjamie]

Also, if it's possible, you could try downgrading to CUDA 11.X - knowing whether this is a CUDA 12 issue or not would be very helpful

[bjoernapfel]

interestingly, after downgrading to CUDA 11.8, none of the above problems showed up and the script works just fine, thanks! so it actually appears to have been a CUDA 12 issue

[neworderofjamie]

For now downgrading to CUDA 11.X is probably the best workaround but clearly something is going awry with CUDA 12 which needs investigating - will raise a bug

[neworderofjamie]

Turns out this had nothing to do with CUDA versions, whether it occurred or not was dependent on the non-deterministic construction of this model. Nonetheless, it is fixed in #589