Help understanding syncSolutions() when sub-cycling

[aprilnovak]

Hi all,

I'm wrapping nekRS as a MOOSE app, and wanted to get some clarification on the syncSolutions() function when using subcycling. Suppose I run nekRs with a smaller dt than a master app, and I have a transfer of heat flux to nekRS on timestep_end. If I'm subcycling nekRS, is that heat flux field fixed for each of the smaller steps taken by nekRS?

• If it is, isn't it extra work to keep calling syncSolutions() for each nekRS dt? In this function, I'm writing into nekRS arrays and doing a problem-wide normalization to conserve power, which requires some global communication in nekRS.

• If it isn't, what is going on under the hood here?

Thanks!

-April

[aeslaughter]

If I'm subcycling nekRS, is that heat flux field fixed for each of the smaller steps taken by nekRS?

I believe so, the sub-application is just solved with smaller timesteps until it reaches the desired time from the main application. There is no communication during this and the main is just waiting for sub-cycling to finish.

* If it is, isn't it extra work to keep calling `syncSolutions()` for each nekRS dt?

I am not familiar with this function, I have not used ExternalProblem. But I imagine that you are correct. Perhaps @permcody will weigh in.

[aprilnovak]

Thanks! I'm basically wondering if I can add some type of check like the following:

syncSolutions(Direction)
{
  case ExternalProblem::Direction::TO_EXTERNAL_APP:
  {
    if (!first_step_in_subcycle) // already sent all info to nekRS
      return;

    send_heat_flux_to_nekrs();
    // other stuff in syncSolutions()
  }
}

We'll be running nekRS with a time step about 10x smaller than it's master App, so this could help with speedup.

[aeslaughter]

I would give it a shot, don't see why not.

[aprilnovak]

Thanks!

For data transfer in the reverse direction (nekRS -> master), I had also suggested to @friedmud that we could save some overhead by only copying the nekRS solution from the GPU back to the host CPU at those master-sub synchronization steps. But if I recall correctly, I think @friedmud said that MOOSE does some type of interpolation operation between the actual Transfer points? (thereby requiring copying from device -> host each nekRS step). That discussion was basically the origin of my question here. If @friedmud wants to comment here that'd be helpful as well!