Reorganize tasks to use shared steps

[xylar]

This is the follow-up to #116 that actually reorganizes the test cases to use shared steps and that changes to the proposed new work-directory structure.

This branch builds on #116 to implements the design described in #109.

Tasks in the work directory are organized into planar and spherical (currently only cosine_bell).

Under planar, tests are largely organized as they were before except that:

• baroclinic_channel uses shared init steps for each resolution
• inertial_gravity_wave and manufactured_soluion are now the names of tasks (there is no longer a single convergence task within each)
• the resolution, nx, and ny are now config options in single_column (rather than lx and ly), and there is no subdirectory for resolution

Under spherical:

• tasks are now organized into icos and qu (with other meshes and mesh types to come later)
• an ocean framework function has been added to create shared base_mesh/<res>km steps within ocean/spherical/icos and ocean/spherical/qu.
• the cosine_bell tasks use the shared base_mesh/* steps, and make local symlinks to each res under their own base_mesh subdirectory (so that the organization looks much like before)
• the cosine_bell/with_viz tasks share these base_mesh/* steps, and they also share the init/*, forward/* and analysis steps with cosine_bell tasks. They make local symlinks so it should be clear. See the updated docs and the design doc in New shared steps capability design doc #109 for more details.

I went through the entire documentation (it took a full day) and it seems like it's in pretty good shape now.

Checklist

• User's Guide has been updated
• Developer's Guide has been updated
• API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
• Documentation has been built locally and changes look as expected
• Testing comment in the PR documents testing used to verify the changes
• Test suites have been updated
• Cached files have been updated

[xylar]

Testing

I successfully ran all test cases except RPE 1 km and 4 km on Chrysalis. I will rerun once more before merging to make sure changes during updating the docs and review don't inadvertently affect the results.

[cbegeman]

I'm happy with these changes. I ran the nightly, pr and cosine_bell tests on chrys with intel, openmpi. I also tried polaris serial at the task and step levels, with and without cleaning. I made a few suggestions in #116. The only suggestion that may affect #117 is #116 (comment). Thanks for all your work on this @xylar!

[sbrus89]

@xylar, these changes look good to me. The pr and nightly suites run successfully and the manufactured_solution test fails with the expected convergence error on Perlmutter. However, I am seeing that the cosine_bell forward step fails with debug flags with the error:

Error termination. Backtrace:
At line 689 of file mpas_ocn_tracer_advection_mono.F
Fortran runtime error: Index '2' of dimension 1 of array 'tracercur' above upper bound of 1

I'm sure the case runs fine with standard compiler flags given @cbegeman's testing. I will verify to be sure.

[sbrus89]

Thanks for this improvement.

[sbrus89]

In a later PR, we should probably move these to being config options

[xylar]

Yes, I agree. I decided not to do that here but it should be changed.

[xylar]

@sbrus89, do you know if that issue (Index '2' of dimension 1 of array 'tracercur' above upper bound of 1) is new in this PR or if that's always been present? It seems like it's a problem of not having initialized 3 debug tracers, and that didn't change here as far as I know.

[xylar]

Hmm, I really don't get that error because we do initialize all three debug tracers to the same value. Is it possible that error was introduced in MPAS-Ocean and is unrelated to Polaris? But then why don't we see it in Compass? Mysterious!

[xylar]

Error termination. Backtrace:
At line 689 of file mpas_ocn_tracer_advection_mono.F
Fortran runtime error: Index '2' of dimension 1 of array 'tracercur' above upper bound of 1

This appears to be an issue with indexing for single-layer runs. Maybe no one has been running in debug mode with single layer? In any case, it seems like that particular line in MPAS-O hasn't been changed in 2 years so I don't think it's a new issue and I don't think it's introduced here. Can you verify that you see it with main? If so, I would suggest we make an E3SM issue about it since it doesn't seem to be a Polaris problem as far as I can tell.

[sbrus89]

@xylar, the cosine_bell suite passes DEBUG=false so the issue is on the MPAS-O side, as we had suspected.

[xylar]

Thanks @sbrus89! Do you have time to make an E3SM issue for this? I'd hate for us to lose track of it. Presumably, we should add a single-layer test to the Compass pr suite so this gets flagged in testing that is regularly done on E3SM, since Polaris is not used that way yet.