Simulation freezes without any notification, HYPRE suspected #17058
-
Dear MOOSE users, I am currently investigation a very strange issue, which is hard to reproduce (hence no minimum working example), and only occurs in MPI parallelized simulations. The key facts:
A simple example involving 130,000 DOFs occasionally freezes at the same time step, while most of the time the simulation works perfectly without any issue.
Attaching
This is valid for ALL MPI processes. Currently, I have no clue what to do. Please, can anybody give me a hint what to do? Thank you and best regards, Matthias |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 5 replies
-
@fdkong Can you help out here? |
Beta Was this translation helpful? Give feedback.
-
I have not much to contribute here, other than, the hypre source code is a joy to read: https://github.com/hypre-space/hypre/blob/master/src/parcsr_ls/par_relax_more.c ... not! |
Beta Was this translation helpful? Give feedback.
-
I just want to point out that this is not necessarily a problem with hypre. It could be that hypre+PETSc_snes+MPI happen to have forced your simulation into a "bad place" in parameter space, such that the Jacobian is very poor, or even has NaNs/bad_things in it. This happens occasionally for me when trying to solve very stiff or otherwise difficult systems: the parallel linear solve is not accurate enough, which throws the system into a "bad place" from which it essentially never recovers. Then i have to choose a stronger preconditioner or more accurate linear-solve. I know it's unlikely to be your problem, but wanted to mention it, just in case. |
Beta Was this translation helpful? Give feedback.
-
Thank you for all the comments, despite the difficult nature of the problem! My current understanding is that the issue is related to the Eigenvalue estimation required by the In principle, HYPRE would allow to use the algorithm My motivation for Thank you all! |
Beta Was this translation helpful? Give feedback.
-
That shouldn't be hard to fix though, @fdkong , right? |
Beta Was this translation helpful? Give feedback.
I have not much to contribute here, other than, the hypre source code is a joy to read: https://github.com/hypre-space/hypre/blob/master/src/parcsr_ls/par_relax_more.c ... not!