Skip to content

Commit

Permalink
fix slurm docs with a note about --export defaults (#895)
Browse files Browse the repository at this point in the history
  • Loading branch information
johrstrom authored Dec 1, 2023
1 parent e95af1b commit 1f067ab
Show file tree
Hide file tree
Showing 2 changed files with 39 additions and 19 deletions.
48 changes: 33 additions & 15 deletions source/installation/resource-manager/slurm.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,26 +6,38 @@ Slurm
A YAML cluster configuration file for a Slurm resource manager on an HPC
cluster looks like:

.. warning::
Open OnDemand's Slurm support defaults to issuing CLI commands with
the ``--export`` flag set to ``NONE``, when Slurms default is ``ALL``.
This can cause issues with jobs that require ``srun``.

Work arounds are currently to ``export SLURM_EXPORT_ENV=ALL``
in a :ref:`script_wrapper <script_wrapper>` before any job scripts run.

Alternatively, you can use ``copy_enviornment`` below with the caveat
that the PUNs environment is very different from regular shell sessions.

.. code-block:: yaml
:emphasize-lines: 8-
# /etc/ood/config/clusters.d/my_cluster.yml
---
v2:
metadata:
title: "My Cluster"
login:
host: "my_cluster.my_center.edu"
job:
adapter: "slurm"
cluster: "my_cluster"
bin: "/path/to/slurm/bin"
conf: "/path/to/slurm.conf"
# bin_overrides:
# sbatch: "/usr/local/bin/sbatch"
# squeue: ""
# scontrol: ""
# scancel: ""
metadata:
title: "My Cluster"
login:
host: "my_cluster.my_center.edu"
job:
adapter: "slurm"
cluster: "my_cluster"
bin: "/path/to/slurm/bin"
conf: "/path/to/slurm.conf"
# bin_overrides:
# sbatch: "/usr/local/bin/sbatch"
# squeue: ""
# scontrol: ""
# scancel: ""
copy_enviornment: false
with the following configuration options:

Expand All @@ -36,7 +48,7 @@ cluster

.. warning::
Using the ``cluster`` option is discouraged. This is because maintenance
outages on the Slurm DB will propogate to Open OnDemand. Instead sites
outages on the Slurm database will propogate to Open OnDemand. Instead sites
should use different ``conf`` files for each cluster to limit maintenance outages.
bin
The path to the Slurm client installation binaries.
Expand All @@ -54,6 +66,12 @@ bin_overrides
- `scontrol`
- `scancel`

copy_enviornment
Copies the enviornment of the PUN when issuing CLI commands. Default behaviour
for Open OnDemand is to use ``--export=NONE`` flag. Setting this to true will
cause Open OnDemand to issue CLI commands with ``--export=ALL``. Though this may
cause issues as the PUN's environment is very different than a regular shell session.

.. note::

If you do not have a multi-cluster Slurm setup you can remove the ``cluster:
Expand Down
10 changes: 6 additions & 4 deletions source/reference/files/submit-yml/basic-bc-options.rst
Original file line number Diff line number Diff line change
Expand Up @@ -119,20 +119,22 @@ Basic Batch Connect Options
header: 'echo "all done at $(date)"'
.. _script_wrapper:

.. describe:: script_wrapper (String, "%s")

wrap the script ('%s' being the script content) with commands before
and after
Wrap the script (``%s`` being the script content) with commands before
and after.

Default
the script has no wrapper
The script has no wrapper.

.. code-block:: yaml
script_wrapper: "%s"
Example
load a module before the script and echo a statement after it
Load a module before the script and echo a statement after it.

.. code-block:: yaml
Expand Down

0 comments on commit 1f067ab

Please sign in to comment.