Weird transfer behavior for multiple source/target boundaries

[aprilnovak]

Hi all,

I'm hoping someone can help me figure out some strange behavior going on with the MultiAppNearestNodeTransfer when there are multiple source/target boundaries for the direction sub -> master. Let me describe the setup -

I have a 7-pin hexagonal bundle. For sake of demonstration, I've trimmed out all the physics to just focus on the transfer issue. The domains for the master app and sub app are identical, and in my application are coupled through boundary conditions on the clad surface (boundary ID 5) and duct inner surface (boundary ID 10). The transfers are as follows:

1: master -> sub: sends a temperature (nek_temp) from boundaries 5 and 10 in the master app to boundaries 5 and 10 in the sub app. This works just fine - the first picture below is nek_temp in the master app, while the second is nek_temp in the sub app after the transfer.

variable in master before transfer TO sub

variable in sub after transfer TO sub

2. sub -> master: sends a flux (avg_flux) from boundaries 5 and 10 in the sub app to boundaries 5 and 10 in the master app. I cannot seem to get this to work correctly if I use a source_boundary, which I want to do because my actual problems are really big, but the boundaries I'm passing data for are a relatively small part of the problem. Before I describe what I did, here's a visual of the avg_flux in the sub that I want to get sent to the master:

variable in sub before transfer FROM sub

Attempt 1: Try using source_boundary FROM sub
First, I obviously tried just using source_boundary for the data transfer for both the pins and duct. But, it seems that the second transfer in the input file (for the duct) overwrites whatever transfer I had just done for the pins. This picture shows that it looks like everywhere just takes the closest value based on the duct boundary. And, it doesn't even seem like I sent data just to one boundary in the master app, because every element has flux data.

Attempt 2: Try using source_boundary FROM sub, but switch the order
To confirm my hunch that using multiple source boundaries overwrites the first transfer, I switched the order of the pin and duct transfers, and it again seems like the duct flux in the master app is overwritten by the pin transfer based on an input file ordering. I definitely don't think there should be an order dependence-here, so this feels like a bug.

Attempt 3: Get rid of source_boundary FROM sub:
This seems to work correctly. But, I really want to use this feature because my problems are really big, and it should reduce the transfer time.

Questions

1. Is there a bug here? Why should the order of transfers matter (difference between attempts 1 and 2)?
2. Why does putting a source_boundary at all for sub->master seem to transfer data to the entire master domain?
3. How can I leverage the source_boundary feature from sub->master?

[aprilnovak]

Here are all my input files.
example.zip

[aprilnovak]

I figured this out in case anyone is interested!

If I change the variable I'm transferring in this example to a Lagrange variable, there's no strange transfer behavior. The issue is that target_boundary isn't used at all for elemental transfers, so because I was restricted to setting a single source/target boundary (because a vector is not yet implemented as an input option), I was basically overwriting the first transfer

source_boundary = '5'
target_boundary = '5' # not used, so this transfer was sending from boundary 5 to everywhere

with the second one

source_boundary = '10'
target_boundary = '10'  # not used, so this transfer was sending from boundary 10 to everywhere

I'm going to add an error if target_boundary is specified but you're transferring an elemental variable, because the transfer just doesn't make sense in that case, and you can run into unexpected behavior like I was seeing.