current prod CUDA version too new for our GPU container

[homework36]

Our codes run on TensorFlow 2.5.1, which does not support the latest CUDA version, 12.4 (I think Arbutus upgraded it from 11.4 to 12.4). As a result, current production cannot use GPU.
According to the official guide, there is now a mismatch in package versions.

We need to spend a decent amount of time updating relevant codes to support the latest GPU, and Python and other packages also need to be updated, which will be a huge pain.
There are ongoing online discussions regarding failures to use TensorFlow (many versions) with CUDA 12.4. It looks like we cannot do much about this. We have to blame Arbutus (a lot, as always).

[homework36]

Also, it seems that every time Arbutus updates the GPU driver, we lose the GPU driver and have to reinstall and reconfigure everything. I don't want to say much, but I don't think any of us like this.