Reorganize tasks to use shared steps

docs/conf.py

@@ -72,14 +72,14 @@
 
 intersphinx_mapping = {
     'geometric_features':
-        ('http://mpas-dev.github.io/geometric_features/stable', None),
-    'matplotlib': ('http://matplotlib.org/stable', None),
-    'mpas_tools': ('http://mpas-dev.github.io/MPAS-Tools/stable', None),
+        ('https://mpas-dev.github.io/geometric_features/stable', None),
+    'matplotlib': ('https://matplotlib.org/stable', None),
+    'mpas_tools': ('https://mpas-dev.github.io/MPAS-Tools/stable', None),
     'numpy': ('https://numpy.org/doc/stable', None),
     'python': ('https://docs.python.org', None),
-    'scipy': ('http://docs.scipy.org/doc/scipy/reference', None),
+    'scipy': ('https://docs.scipy.org/doc/scipy/reference', None),
     "sphinx": ("https://www.sphinx-doc.org/en/master", None),
-    'xarray': ('http://xarray.pydata.org/en/stable', None)
+    'xarray': ('https://xarray.pydata.org/en/stable', None)
 }
 
 # -- MyST settings ---------------------------------------------------

docs/developers_guide/api.md

@@ -77,7 +77,7 @@ seaice/api
    unpickle_suite
    setup_config
    load_dependencies
-   pickle_step_after_run
+   complete_step_run
    serial.run_tasks
    serial.run_single_step
 

docs/developers_guide/building_docs.md

@@ -10,8 +10,7 @@ Then, run the following script to build the docs:
 
 ```bash
 cd docs
-rm -rf developers_guide/generated/ developers_guide/*/generated/ _build/
-make html
+make clean && make html
 ```
 
 You can view the documentation by opening `_build/html/index.html`.

docs/developers_guide/command_line.md

@@ -23,16 +23,17 @@ The `polaris list` command is used to list tasks, suites, and
 supported machines.  The command-line options are:
 
 ```none
-polaris list [-h] [-t TEST] [-n NUMBER] [--machines] [--suites] [-v]
+$ polaris list --help
+usage: polaris list [-h] [-t TASK] [-n NUMBER] [--machines] [--suites] [-v]
 ```
 
 By default, all tasks are listed:
 
 ```none
 $ polaris list
-Testcases:
-   0: examples/example_compact/1km/test1
-   1: examples/example_compact/1km/test2
+Tasks:
+   0: ocean/planar/baroclinic_channel/10km/default
+   1: ocean/planar/baroclinic_channel/10km/decomp
 ...
 ```
 
@@ -63,15 +64,14 @@ the configuration and the names of its steps:
 
 ```none
 $ polaris list -n 0 -v
-path:          examples/example_compact/1km/test1
-description:   Tempate 1km test1
-name:          test1
-core:          examples
-configuration: example_compact
-subdir:        1km/test1
+path:          ocean/planar/baroclinic_channel/10km/default
+name:          default
+component:     ocean
+subdir:        planar/baroclinic_channel/10km/default
 steps:
- - step1
- - step2
+ - init:    ocean/planar/baroclinic_channel/10km/init
+ - forward: ocean/planar/baroclinic_channel/10km/default/forward
+ - viz:     ocean/planar/baroclinic_channel/10km/default/viz
 ```
 
 See {ref}`dev-list` for more about the underlying framework.
@@ -90,21 +90,24 @@ want to run before setting up a polaris task.
 The command-line options are:
 
 ```none
-polaris setup [-h] -w PATH [-t PATH] [-n NUM [NUM ...]] [-f FILE] [-m MACH]
-               [-b PATH] [-p PATH] [--suite_name SUITE]
+$ polaris setup --help
+usage: polaris setup [-h] [-t PATH] [-n NUM [NUM ...]] [-f FILE] [-m MACH] -w
+                     PATH [-b PATH] [-p PATH] [--suite_name SUITE]
+                     [--cached STEP [STEP ...]] [--copy_executable] [--clean]
+
 ```
 
 The `-h` or `--help` options will display the help message describing the
 command-line options.
 
 The tasks to set up can be specified either by relative path or by number.
-The `-t` or `--test` flag is used to pass the relative path of the test
-case within the resulting work directory.  The is the path given by
+The `-t` or `--task` flag is used to pass the relative path of the task
+within the resulting work directory.  The is the path given by
 {ref}`dev-polaris-list`.  Only one task at a time can be supplied to
 `polaris setup` this way.
 
-Alternatively, you can supply the test numbers of any number of tasks to
-the `-n` or `--case_number` flag.  Multiple test numbers are separated by
+Alternatively, you can supply the task numbers of any number of tasks to
+the `-n` or `--task_number` flag.  Multiple test numbers are separated by
 spaces.  These are the test numbers  given by {ref}`dev-polaris-list`.
 
 `polaris setup` requires a few basic pieces of information to be able to set
@@ -152,6 +155,19 @@ Tasks within the custom suite are run in the order they are supplied to
 cases that depend on the output of other tasks must run after their
 dependencies.
 
+You can uses `--cached` to specify steps of a test case to download from
+pre-generated files if they are available (see {ref}`dev-polaris-cache`.)
+
+If you specify `--copy_executable`, the model executable will be copied to the 
+work directory rather than just symlinked.  This is useful if wish to run
+the tasks again later but anticipate that you may have removed (or replaced)
+the model code.
+
+Finally, if you specify `--clean`. The base work directory pointed to with the
+`-w` flag will be deleted before setting up the tasks.  This is useful if you
+want to do a fresh run, since polaris will not rerun steps that have already
+been run.
+
 See {ref}`dev-setup` for more about the underlying framework.
 
 (dev-polaris-suite)=
@@ -162,8 +178,9 @@ The `polaris suite` command is used to set up a suite. The command-line
 options are:
 
 ```none
-polaris suite [-h] -c COMPONENT -t SUITE -w PATH [-f FILE] [-v]
-              [-m MACH] [-b PATH] [-p PATH]
+$ polaris suite --help
+usage: polaris suite [-h] -c COMPONENT -t SUITE [-f FILE] [-m MACH] [-b PATH]
+                     -w PATH [-p PATH] [--copy_executable] [--clean]
 ```
 
 The `-h` or `--help` options will display the help message describing the
@@ -184,6 +201,9 @@ comparison with `-b` or `--baseline_dir`.  If supplied, each task in the
 suite that includes {ref}`dev-validation` will be validated against the 
 previous run in the baseline.
 
+The flags `--copy_executable`and `--clean` are the same as in 
+{ref}`dev-polaris-setup`.
+
 See {ref}`dev-suite` for more about the underlying framework.
 
 (dev-polaris-run)=
@@ -195,9 +215,11 @@ parallel) a suite, task or step  that has been set up in the current
 directory:
 
 ```none
-polaris serial [-h] [--steps STEPS [STEPS ...]]
-                 [--skip_steps SKIP_STEPS [SKIP_STEPS ...]]
-                 [suite]
+$ polaris serial --help
+usage: polaris serial [-h] [--steps STEPS [STEPS ...]]
+                      [--skip_steps SKIP_STEPS [SKIP_STEPS ...]] [-q]
+                      [--step_is_subprocess]
+                      [suite]
 ```
 
 Whereas other `polaris` commands are typically run in the local clone of the
@@ -244,29 +266,37 @@ over the config option.
 To see which steps are are available in a given task, you need to run
 {ref}`dev-polaris-list` with the `-v` or `--verbose` flag.
 
+The `--step_is_subprocess` flag is for internal use by the framework so you
+shouldn't need to use that flag.
+
 See {ref}`dev-run` for more about the underlying framework.
 
 (dev-polaris-cache)=
 
 ## polaris cache
 
-`polaris` supports caching outputs from any step in a special database
+Polaris supports caching outputs from any step in a special database
 called `polaris_cache` (see {ref}`dev-step-input-download`). Files in this
 database have a directory structure similar to the work directory (but without
 the component subdirectory, which is redundant). The files include a date stamp
 so that new revisions can be added without removing older ones (supported by
 older polaris versions).  See {ref}`dev-step-cached-output` for more details.
 
-A new command, `polaris cache` has been added to aid in updating the file
-`cached_files.json` within a component.  This command is only available on
-Anvil and Chrysalis, since developers can only copy files from a polaris work
-directory onto the LCRC server from these two machines.  Developers run
-`polaris cache` from the base work directory, giving the relative paths of
-the step whose outputs should be cached:
+The command `polaris cache` is used to update the file `cached_files.json` 
+within a component.  This command is only available on Anvil and Chrysalis, 
+since developers can only copy files from a Polaris work directory onto the 
+LCRC server from these two machines.
+```none
+$ polaris cache --help
+usage: polaris cache [-h] [-i STEP [STEP ...]] [-d DATE] [-r]
+```
+
+Developers run `polaris cache` from the base work directory, giving the 
+relative paths of the step whose outputs should be cached:
 
 ```bash
-polaris cache -i ocean/global_ocean/QU240/mesh/mesh \
-    ocean/global_ocean/QU240/PHC/init/init
+polaris cache -i ocean/spherical/*/base_mesh/* \
+    ocean/spherical/*/cosine_bell/init/*
 ```
 
 This will:
@@ -281,11 +311,18 @@ The resulting `ocean_cached_files.json` will look something like:
 
 ```json
 {
-    "ocean/global_ocean/QU240/mesh/mesh/culled_mesh.nc": "global_ocean/QU240/mesh/mesh/culled_mesh.210803.nc",
-    "ocean/global_ocean/QU240/mesh/mesh/culled_graph.info": "global_ocean/QU240/mesh/mesh/culled_graph.210803.info",
-    "ocean/global_ocean/QU240/mesh/mesh/critical_passages_mask_final.nc": "global_ocean/QU240/mesh/mesh/critical_passages_mask_final.210803.nc",
-    "ocean/global_ocean/QU240/PHC/init/init/initial_state.nc": "global_ocean/QU240/PHC/init/init/initial_state.210803.nc",
-    "ocean/global_ocean/QU240/PHC/init/init/init_mode_forcing_data.nc": "global_ocean/QU240/PHC/init/init/init_mode_forcing_data.210803.nc"
+    "ocean/spherical/icos/base_mesh/120km/mesh.msh": "spherical/icos/base_mesh/120km/mesh.230914.msh",
+    "ocean/spherical/icos/base_mesh/120km/base_mesh.nc": "spherical/icos/base_mesh/120km/base_mesh.230914.nc",
+    "ocean/spherical/icos/base_mesh/120km/cellWidthVsLatLon.nc": "spherical/icos/base_mesh/120km/cellWidthVsLatLon.230914.nc",
+    "ocean/spherical/icos/base_mesh/120km/graph.info": "spherical/icos/base_mesh/120km/graph.230914.info",
+    "ocean/spherical/icos/base_mesh/240km/mesh.msh": "spherical/icos/base_mesh/240km/mesh.230914.msh",
+    "ocean/spherical/icos/base_mesh/240km/base_mesh.nc": "spherical/icos/base_mesh/240km/base_mesh.230914.nc",
+    "ocean/spherical/icos/base_mesh/240km/cellWidthVsLatLon.nc": "spherical/icos/base_mesh/240km/cellWidthVsLatLon.230914.nc",
+    "ocean/spherical/icos/base_mesh/240km/graph.info": "spherical/icos/base_mesh/240km/graph.230914.info",
+    "ocean/spherical/qu/cosine_bell/init/210km/initial_state.nc": "spherical/qu/cosine_bell/init/210km/initial_state.230914.nc",
+    "ocean/spherical/qu/cosine_bell/init/240km/initial_state.nc": "spherical/qu/cosine_bell/init/240km/initial_state.230914.nc",
+    "ocean/spherical/qu/cosine_bell/init/60km/initial_state.nc": "spherical/qu/cosine_bell/init/60km/initial_state.230914.nc",
+    "ocean/spherical/qu/cosine_bell/init/90km/initial_state.nc": "spherical/qu/cosine_bell/init/90km/initial_state.230914.nc"
 }
 ```
 

docs/developers_guide/docs.md

@@ -133,11 +133,8 @@ be added to `docs/<component>/api.md`:
 .. autosummary::
    :toctree: generated/
 
-   BaroclinicChannel
+   add_baroclinic_channel_tasks
 
-   BaroclinicChannelTestCase
-   BaroclinicChannelTestCase.configure
-
    forward.Forward
    forward.Forward.compute_cell_count
    forward.Forward.dynamic_model_config

docs/developers_guide/framework/commands.md

@@ -31,23 +31,14 @@ pickle files are not intended for users (or developers) to read or modify.
 Properties of the task and step objects are not intended to change between
 setting up and running a suite, task or step.
 
-(dev-clean)=
-
-## clean module
-
-The {py:func}`polaris.clean.clean_cases()` function is used by
-`polaris clean` and `polaris suite` to delete the constants of a task
-subdirectory in the work directory.
-
 (dev-suite)=
 
 ## suite module
 
-The {py:func}`polaris.suite.setup_suite()` and {py:func}`polaris.suite.clean_suite()`
-functions are used by `polaris suite` to set up or clean up a suite in a
-work directory.  Setting up a suite includes setting up the tasks
-(see {ref}`dev-setup`), writing out a {ref}`dev-provenance` file, and saving
-a pickle file containing a python dictionary that defines the suite for
+The {py:func}`polaris.suite.setup_suite()` function is used by `polaris suite`
+to set up a suite in a work directory.  Setting up a suite includes setting up 
+the tasks (see {ref}`dev-setup`), writing out a {ref}`dev-provenance` file, and
+saving a pickle file containing a python dictionary that defines the suite for
 later use by `polaris run`.  The "target" and "minimum" number of cores
 required for running the suite are displayed.  The "target" is determined
 based on the maximum product of the `ntasks` and `cpus_per_task`
@@ -90,8 +81,8 @@ and finally indicates if the overall task execution was `SUCCESS` or `ERROR`.
 Execution times are provided for individual steps, tasks and the suite as a
 whole.
 
-{py:func}`polaris.run.run_single_step()` runs only the selected step from a
-given task, skipping any others, displaying the output in the terminal
+{py:func}`polaris.run.serial.run_single_step()` runs only the selected step 
+from a given task, skipping any others, displaying the output in the terminal
 window rather than a log file.
 
 (dev-cache)=

docs/developers_guide/framework/config.md

@@ -15,21 +15,20 @@ The {py:meth}`mpas_tools.config.MpasConfigParser.add_from_package()` method can
 be used to add the contents of a config file within a package to the config
 options. Examples of this can be found in many tasks as well as
 {py:func}`polaris.setup.setup_task()`. Here is a typical example from
-{py:func}`polaris.ocean.tasks.global_ocean.make_diagnostics_files.MakeDiagnosticsFiles.configure()`:
+{py:func}`polaris.ocean.tasks.baroclinic_channel.decomp.Decomp.configure()`:
 
 ```python
 def configure(self):
     """
-    Modify the configuration options for this task
+    Add the config file common to baroclinic channel tests
     """
-    self.config.add_from_package(
-       'polaris.ocean.tasks.global_ocean.make_diagnostics_files',
-       'make_diagnostics_files.cfg', exception=True)
+    self.config.add_from_package('polaris.ocean.tasks.baroclinic_channel',
+                                 'baroclinic_channel.cfg')
 ```
 
 The first and second arguments are the name of a package containing the config
 file and the name of the config file itself, respectively.  You can see that
-the file is in the path `polaris/ocean/tasks/global_ocean/make_diagnostics_files`
+the file is in the path `polaris/ocean/tasks/baroclinic_channel`
 (replacing the `.` in the module name with `/`).  In this case, we know
 that the config file should always exist, so we would like the code to raise
 an exception (`exception=True`) if the file is not found.  This is the

docs/developers_guide/framework/index.md

@@ -2,8 +2,8 @@
 
 # Framework
 
-All of the {ref}`dev-packages` that are not in the two components (`landice`
-and `ocean`) belong to the polaris framework.  Some of these
+All of the {ref}`dev-packages` that are not in the two components (`ocean`
+and `seaice`) belong to the polaris framework.  Some of these
 modules and packages are used by the {ref}`dev-command-line`, while others are
 meant to be called within tasks and steps to simplify tasks like adding
 input and output files, downloading data sets, building up config files,