MPI ERROR

[Leni-Yeo]

Good afternoon,
I am trying to do a simulation for polycrystal grain growth. I saw the example that applies the polycrystalvariables and polycrystalkernel actions. I modified the polycrystalkernel action to include the following kernels: ADAllenCahn, ADTimeDerivative, and ADACInterfaceKobayashi2 for a basic simulation. The problem is that I get a "MPI had an error" when running the simulation. I cannot find the exact problem. I assumed it is in the action kernel but still could not solve it. Please help.

I attached my input file and the kernel action I am using to set up the kernels.

##################### I COPIED THE ERROR BELOW ##########################

Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 69
MPI had an error

Invalid communicator, error stack:
MPI_Comm_test_inter(112): MPI_Comm_test_inter(comm=0xf2e8f3e0, flag=0x7ffcce88cfc0) failed
MPI_Comm_test_inter(70).: Invalid communicator

application called MPI_Abort(MPI_COMM_WORLD, 612613) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=612613
:
system msg for write_line failure : Bad file descriptor

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 69
MPI had an error

Invalid communicator, error stack:
MPI_Comm_test_inter(112): MPI_Comm_test_inter(comm=0x4ce493e0, flag=0x7ffd8f4d8be0) failed
MPI_Comm_test_inter(70).: Invalid communicator

application called MPI_Abort(MPI_COMM_WORLD, 612613) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=612613
:
system msg for write_line failure : Bad file descriptor
Assertion failed in file src/mpi/comm/comm_test_inter.c at line 77: ((comm)&(0x03ffffff)) < MPIR_COMM_N_BUILTIN
/home/user/miniconda3/envs/moose/lib/libmpi.so.12(+0x1c7d93) [0x7fa949f80d93]
/home/user/miniconda3/envs/moose/lib/libmpi.so.12(MPL_backtrace_show+0x18) [0x7fa949f80e73]
/home/user/miniconda3/envs/moose/lib/libmpi.so.12(+0x15bb94) [0x7fa949f14b94]
/home/user/miniconda3/envs/moose/lib/libmpi.so.12(MPI_Comm_test_inter+0x2f0) [0x7fa949e27840]
/lib/x86_64-linux-gnu/libmpi_cxx.so.40(_ZN3MPI9IntracommC1EP19ompi_communicator_t+0x8b) [0x7fa9443623db]
/lib/x86_64-linux-gnu/libmpi_cxx.so.40(+0x1125a) [0x7fa94435e25a]
/lib64/ld-linux-x86-64.so.2(+0x11b8a) [0x7fa94e9abb8a]
/lib64/ld-linux-x86-64.so.2(+0x11c91) [0x7fa94e9abc91]
/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0xe5) [0x7fa94abed915]
/lib64/ld-linux-x86-64.so.2(+0x160bf) [0x7fa94e9b00bf]
/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88) [0x7fa94abed8b8]
/lib64/ld-linux-x86-64.so.2(+0x155fa) [0x7fa94e9af5fa]
/lib/x86_64-linux-gnu/libdl.so.2(+0x134c) [0x7fa94717734c]
/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88) [0x7fa94abed8b8]
/lib/
internal ABORT - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor
Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 69
MPI had an error

Invalid communicator, error stack:
MPI_Comm_test_inter(112): MPI_Comm_test_inter(comm=0xa292c3e0, flag=0x7ffe75b7b580) failed
MPI_Comm_test_inter(70).: Invalid communicator

application called MPI_Abort(MPI_COMM_WORLD, 537483525) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=537483525
:
system msg for write_line failure : Bad file descriptor
Generic Warning: In ../Parallel/MPI/vtkMPICommunicator.cxx, line 69
MPI had an error

Invalid communicator, error stack:
MPI_Comm_test_inter(112): MPI_Comm_test_inter(comm=0x62a903e0, flag=0x7ffcd7088450) failed
MPI_Comm_test_inter(70).: Invalid communicator

files.zip

[aeslaughter]

It seems like you might be getting different VTK included in your build. Are you using the conda "moose-libmesh" package? @milljm Have you seen something like this before?

[Leni-Yeo]

Hi. Yes I am. Someone advised me to use the "Adaptivity" block (It was off on in the original input file and in mine too); and it solved it. But I am not sure where the problem was exactly considering the original version worked well without it.

[permcody]

You may be running out of memory: Adaptivity can drastically increase the mesh, and everything else that goes with it.

[Leni-Yeo]

Oh, okay, I see. I have a large memory so its not a problem for now I guess.