Solve failure with no information #16615
-
|
My solve fails after multiple time steps without producing any error messages. The solution fails on the second nonlinear iteration. Here is the output using '-snes_converged_reason -ksp_converged_reason -ksp_monitor_true_residual -ksp_view': Performing automatic scaling calculation 0 Nonlinear |R| = [32m6.158493e+01[39m After this line the program exits with the following response on the command line: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 I'm not sure how to go about debugging this issue, so any help would be greatly appreciated. Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 27 replies
-
|
Hi Dusty Can you run this through a debugger, see moose-debugging ? Best, |
Beta Was this translation helpful? Give feedback.
-
|
Add a breakpoint on |
Beta Was this translation helpful? Give feedback.
-
|
I installed MOOSE using the conda environment method so I'm assuming it is using moose-mpich |
Beta Was this translation helpful? Give feedback.
-
|
Is it called PMPI_Abort in that one? |
Beta Was this translation helpful? Give feedback.
-
|
With both MPI_Abort and PMPI_Abort I'm getting this output: Performing automatic scaling calculation 0 Nonlinear |R| = 6.158493e+01 |
Beta Was this translation helpful? Give feedback.
-
|
Could you show the complete command-line that you used to debug the code? |
Beta Was this translation helpful? Give feedback.
-
|
When use lldb, I have something like this: The same thing should happen for gdb as well |
Beta Was this translation helpful? Give feedback.
-
|
gdb --args ../my-app/my-app-dbg -i debug_input.i Reading symbols from ../my-app/my-app-dbg... (gdb) b MPI_Abort Lots of initialization here... |
Beta Was this translation helpful? Give feedback.
-
|
When I use lldb I don't get the last line you have there. Instead I get: (lldb) b MPI_Abort |
Beta Was this translation helpful? Give feedback.
-
|
@milljm Do you have any thought on this? Did we use "-g" when compiling mpich? |
Beta Was this translation helpful? Give feedback.
-
|
@dcolema4 appears to be running Linux (/home instead of /Users as it would be on the Mac). And is using LLVM compilers (lldb)... which we do not create. So no... I have no idea how mpich was created on this machine. Conda will not give you LLVM compilers when installing moose-libmesh, moose-petsc, etc on Linux machines. This would appear to be a custom build at first glance (unless your use of /home was a typo). |
Beta Was this translation helpful? Give feedback.
-
|
I'm using Ubuntu on WSL and originally went through the conda environment installation. Looking again I see there I may not have executed the hostname change on my Windows side. Perhaps that's the issue? |
Beta Was this translation helpful? Give feedback.
-
|
Looking again, I see you are also using gdb (GCC). That and you mentioning conda, probably means you are using conda :) I don't know what -g actually does... |
Beta Was this translation helpful? Give feedback.
-
|
My guess here is is that you have crossed up your MPI installations somehow (i.e. you are compiling/linking against one MPI and running with a different one). Can you describe what your installation process was like? |
Beta Was this translation helpful? Give feedback.
-
|
I am. When I run the debugger on my input file it gets to the end of the first nonlinear iteration to show: 0 Nonlinear |R| = 6.158493e+01 Then no matter what I type it exits out showing: [Inferior 1 (process 23255) exited with code 0220] I'm kind of at a loss for what to do. |
Beta Was this translation helpful? Give feedback.
-
|
That is absolutely bizarre. To be able to make it that far and then go wrong is just odd. I literally can't think of anything that would cause that. @milljm I'm punting - any other ideas? |
Beta Was this translation helpful? Give feedback.
-
|
I am going to try and debug that same problem on my windows WSL... just gotta build in debug mode on my little 8 core machine :) I'll post when I am able/done. The only thing that comes to mind, is if the Windows host file is not configured to use MPI. But @dcolema4 mentioned they saw that step later on, so... I figure that was completed. |
Beta Was this translation helpful? Give feedback.
-
|
Here are my results: I had to install conda install gdbI ended up grabbing the latest version (9.2). I suppose the larger issue, is that my solve converges and works. Can you perform an ldd on moose_test-dbg? I want to see what libraries it is linked to: |
Beta Was this translation helpful? Give feedback.
-
|
@milljm I get the same results as you when I run the 2d diffusion case. It is on my problem where the issue is showing up. Despite not being able to get the debugger to work, I was able to determine that the issue was coming from a vector material property. I still don't know why it caused the program to fail, but when I changed it to a scalar and updated my kernels to accept the scalar value everything worked fine. |
Beta Was this translation helpful? Give feedback.
My guess here is is that you have crossed up your MPI installations somehow (i.e. you are compiling/linking against one MPI and running with a different one). Can you describe what your installation process was like?