Performing LES on devices

[Shiyu-Sandy-Du]

In this PR,

• ax_helm_full_device is implemented for hip and cuda
• sigma and vreman SGS models are implemented for hip and cuda as well
• bug is fixed in the pnpn pressure res computation for full stress formulation on devices

Some other things to be discussed:

• In pnpn_res_stress_device.F90, device math is called for multiple times. If we merge those kernels together as much as possible, how much can we benefit from it considering the solving time for the pressure equation is not short?
• Implementation for 1D in ax_helm_full_device has not been implemented and thus autotune is not yet implemented as well. But do we really need it? I cannot see it in the vector version of ax_helm_device implementation. Is the 1D or kstep discussion for vector version of ax_helm happened before?

[njansson]

Nice!

[njansson]

To answer some of the questions,

• In pnpn_res_stress_device.F90, device math is called for multiple times. If we merge those kernels together as much as possible, how much can we benefit from it considering the solving time for the pressure equation is not short?

There are gains to be made here, by fusing the e.g. the cluster of col3 and sub2. Remember that each call to a device math functions comes with a launch latency, which is not negligible.

• Implementation for 1D in ax_helm_full_device has not been implemented and thus autotune is not yet implemented as well. But do we really need it? I cannot see it in the vector version of ax_helm_device implementation. Is the 1D or kstep discussion for vector version of ax_helm happened before?

Exactly, there's no 1d version of these, since 1d will run out of shared memory for most polynomial orders.

[MartinKarp]

Great stuff Shiyu! I just added two comments, feel free to take them to heart or not.

One thing I noted is the use of quite a lot of math functions such as pow, divisions and so on. I think we should just be aware this might be an issue if people run in single precision and one can perhaps use xreal in some places then. This is just a note for the future though.

[Shiyu-Sandy-Du]

Great stuff Shiyu! I just added two comments, feel free to take them to heart or not.

One thing I noted is the use of quite a lot of math functions such as pow, divisions and so on. I think we should just be aware this might be an issue if people run in single precision and one can perhaps use xreal in some places then. This is just a note for the future though.

I changed the unnecessary pow into multiplications here. And I think we should note the sin, cos and sqrt here for future maybe.