Simulation freezes without any notification, HYPRE suspected

[matthiasneuner]

Dear MOOSE users,

I am currently investigation a very strange issue, which is hard to reproduce (hence no minimum working example), and only occurs in MPI parallelized simulations. The key facts:

• A TensorMechanics like simulation involving multiple fields (displacement, rotations, etc.) using my own MOOSE app
• Monolithic SMP preconditioning with hypre boomeramg using following setup:

petsc_options_iname = '     -pc_type
                                -pc_hypre_type
                                -ksp_type
                                -ksp_gmres_restart
                                -pc_hypre_boomeramg_relax_type_all
                                -pc_hypre_boomeramg_strong_threshold
                                -pc_hypre_boomeramg_agg_nl
                                -pc_hypre_boomeramg_agg_num_paths
                                -pc_hypre_boomeramg_max_levels
                                -pc_hypre_boomeramg_coarsen_type
                                -pc_hypre_boomeramg_interp_type
                                -pc_hypre_boomeramg_P_max
                                -pc_hypre_boomeramg_truncfactor' 

    petsc_options_value = '     hypre
                                boomeramg
                                gmres
                                301
                                Chebyshev
                                0.65
                                5 
                                2
                                25
                                HMIS
                                ext+i
                                4
                                0.4 '

A simple example involving 130,000 DOFs occasionally freezes at the same time step, while most of the time the simulation works perfectly without any issue.

• The freeze occurs without any warning or error message
• No FPE is issued by libMESH
• The freeze does not occur for threaded parallelization
• The freeze does not occur when using a direct solver (e.g., strumpack)

Attaching gdb to such a freezing simulation reveals that all MPI processes are stuck in HYPRE:

0x00007f91dd7f3f28 in hypre_LINPACKcgpthy () from /home/matthias/anaconda3/envs/moose/lib/libHYPRE-2.19.0.so                                                                             
(gdb) bt full                                                                                                                                                                            
#0  0x00007f91dd7f3f28 in hypre_LINPACKcgpthy () from /home/matthias/anaconda3/envs/moose/lib/libHYPRE-2.19.0.so                                                                         
No symbol table info available.
#1  0x00007f91dd7f4175 in hypre_LINPACKcgtql1 () from /home/matthias/anaconda3/envs/moose/lib/libHYPRE-2.19.0.so
No symbol table info available.
#2  0x00007f91dd7f4b6d in hypre_ParCSRMaxEigEstimateCG () from /home/matthias/anaconda3/envs/moose/lib/libHYPRE-2.19.0.so
No symbol table info available.
#3  0x00007f91dd7632d5 in hypre_BoomerAMGSetup () from /home/matthias/anaconda3/envs/moose/lib/libHYPRE-2.19.0.so
No symbol table info available.
#4  0x00007f91de6c00b2 in PCSetUp_HYPRE () from /home/matthias/anaconda3/envs/moose/lib/libpetsc.so.3.14
No symbol table info available.
#5  0x00007f91de6fb951 in PCSetUp () from /home/matthias/anaconda3/envs/moose/lib/libpetsc.so.3.14
No symbol table info available.
#6  0x00007f91de7301b4 in KSPSetUp () from /home/matthias/anaconda3/envs/moose/lib/libpetsc.so.3.14
No symbol table info available.
#7  0x00007f91de730e32 in KSPSolve_Private () from /home/matthias/anaconda3/envs/moose/lib/libpetsc.so.3.14
No symbol table info available.
#8  0x00007f91de733443 in KSPSolve () from /home/matthias/anaconda3/envs/moose/lib/libpetsc.so.3.14
No symbol table info available.

This is valid for ALL MPI processes.

Currently, I have no clue what to do. Please, can anybody give me a hint what to do?

Thank you and best regards, Matthias

[aeslaughter]

@fdkong Can you help out here?

[dschwen]

I have not much to contribute here, other than, the hypre source code is a joy to read: https://github.com/hypre-space/hypre/blob/master/src/parcsr_ls/par_relax_more.c ... not!

[aeslaughter]

Could you just be running out of memory?

[matthiasneuner]

Could you just be running out of memory?

Actually, I can exclude that. Other examples with 1 million++ DOFs, 'more nonlinearity' and a similar solver setup work very well. For the present example, the freeze occurs quite early, which means that most parts of the domain still behave linear elastic.

[matthiasneuner]

I have not much to contribute here, other than, the hypre source code is a joy to read: https://github.com/hypre-space/hypre/blob/master/src/parcsr_ls/par_relax_more.c ... not!

Maybe a stupid question ... but does hypre_ParCSRMaxEigEstimateCG expect a symmetric matrix? Because my system is definitely nonsymmetric.

[fdkong]

Yes, it could be an issue.

Could you delete option -pc_hypre_boomeramg_relax_type_all Chebyshev? GS will be used by default, IIRC. I do not have any issue with the default relax solvers for even larger problems( around 10 billions DoFs).

You might have a motivaiton to add -pc_hypre_boomeramg_relax_type_all Chebyshev ?

[WilkAndy]

I just want to point out that this is not necessarily a problem with hypre. It could be that hypre+PETSc_snes+MPI happen to have forced your simulation into a "bad place" in parameter space, such that the Jacobian is very poor, or even has NaNs/bad_things in it.

This happens occasionally for me when trying to solve very stiff or otherwise difficult systems: the parallel linear solve is not accurate enough, which throws the system into a "bad place" from which it essentially never recovers. Then i have to choose a stronger preconditioner or more accurate linear-solve. I know it's unlikely to be your problem, but wanted to mention it, just in case.

[matthiasneuner]

Thank you for all the comments, despite the difficult nature of the problem!

My current understanding is that the issue is related to the Eigenvalue estimation required by the Chebyshev relaxation:
The default algorithm hypre_ParCSRMaxEigEstimateCG for estimating the max. and mix. Eigenvalues uses CG and expects a symmetric matrix:
https://github.com/hypre-space/hypre/blob/be18e595aed8dcd1af818eb563c8a962d96a162c/src/parcsr_ls/par_amg_setup.c#L3088

In principle, HYPRE would allow to use the algorithm hypre_ParCSRMaxEigEstimate (estimating only the max. value) via hypre_ParAMGDataChebyEigEst, but the respective option is not exposed to us by PETSc, so we cannot use it.

My motivation for Chebyshev relaxation is that I observed it to be far superior to SOR/Jacobi (very good experiences in particular for Trilinos' ML). I have successfully applied it to other examples. Switching back to SOR/Jacobi considerably decreases performance due to increasing gmres iterations, but I guess that I have no choice here :-) . Just for confirmation: No freezes occur for SOR/Jacobi.

Thank you all!

[dschwen]

but the respective option is not exposed to us by PETSc, so we cannot use it.

That shouldn't be hard to fix though, @fdkong , right?

[fdkong]

It is very easy to fix, right. @matthiasneuner You are free to create an issue in PETSc, and tag me.