Deploying Flux "System Instance"

[wkharold]

The docs (the Admin guide specifically) mention Flux system instances. Most of the tutorial examples use Slurm for resource management & process launching. With "system instances" I assume that Slurm (or an equivalent resource manager) wouldn't be necessary. Is there documentation, or examples, describing how to set up a "pure" Flux environment?

[garlick]

That was the intent of the admin guide. If it's falling short please let us know what we missed (docs are a work in progress). Thanks!

[wkharold]

OK, my bad. Now that I've read the admin guide in more depth I understand better.

Next question: are there Spack packages for flux-security & flux-accounting? My Spack install did not find them so I suspect not, although I would love to be wrong since I try to install things via Spack when possible.

[garlick]

Sorry, I don't think we've got those. Most of the sites we deal with would require native OS packages (like RPM) for system software.

If this is for experimentation we do have https://hub.docker.com/r/fluxrm/fluxorama which allows a system instance to be started in a docker container. Hmm, I know we have some instructions for that stashed somewhere but I'm not finding them. Let us know if that's of interest and we'll get that for you.

[wkharold]

No worries. I'm using Rocky Linux 8. Would the TOSS packages work there? Or, I can build it from source.

[garlick]

Very likely as the current TOSS release is based on RHEL 8. They are currently hosted on our internal build farm so let me check on how we can get them to you.

[wkharold]

Progress report. I have flux installed on four cloud-based nodes, gflux-01-4. flux-core and flux-security are built and installed on all the ranks. flux-sched is build and installed on rank 0 (gflux-01). Everything starts clean. but flux resource list shows all four nodes as 'down'. I'm not sure where to go from here since I don't see any errors in the logs.

[wkharold]

And ... following up with flux resource undrain gflux-0[1-4] yields

[wkh@gflux-01 job-submit-cli]$ flux resource list
     STATE NNODES   NCORES    NGPUS NODELIST
      free      4        8        0 gflux-[01-04]
 allocated      0        0        0 
      down      0        0        0 

but then I get:

[wkh@gflux-01 job-submit-cli]$ flux mini submit --nodes=2 --ntasks=4 --cores-per-task=2 ./compute.lua 120
[Errno 1] only instance owner can use sign-type=none
[wkh@gflux-01 job-submit-cli]$ sudo -u flux /usr/local/bin/flux mini submit --nodes=2 --ntasks=4 --cores-per-task=2 ./compute.lua 120
[Errno 22] Direct job submission is disabled for this instance. Please use the batch or alloc subcommands of flux-mini(1), Unsatisfiable request

[grondo]

These are cloud (Google Cloud Platform) based nodes. They have four vCPUs. If I do --cores=0-3 I see flux complain about waiting for cores 2-3. So I've been doing --cores=0-1.

Interesting! Flux is also using hwloc for discovery of cores. What does flux R encode --local show on these nodes (that should use the same code Flux is using to gather the local resources)

[grondo]

[Errno 22] Direct job submission is disabled for this instance. Please use the batch or alloc subcommands of flux-

This is because the require-instance validator is configured, which may not be what you want for your system instance.

Find the [ingest.validator] TOML configuration and remove "require-instance" and you should be able to submit normal jobs to your instance.

[grondo]

If I do --cores=0-3 I see flux complain about waiting for cores 2-3. So I've been doing --cores=0-1.

Oh, actually I see the problem. Flux only supports real cores at this time. 2 of your vCPUs appear as hyperthreads to hwloc.

If you'd like to run with your current configuration, you can try setting noverify = true in the [resource] TOML config. You may run into warnings from the job shell when it tries to set CPU affinity to cores 2-3 though.

[wkharold]

Right. We actually see that with OpenMPI as well. It's kind of OK since for many workloads turning off hyper-threading yields slightly better performance. As expected flux R encode --local recognizes two cores.

This is great progress:

[wkh@gflux-01 tmp]$ flux R encode --local
{"version": 1, "execution": {"R_lite": [{"rank": "0", "children": {"core": "0-1"}}], "starttime": 0.0, "expiration": 0.0, "nodelist": ["gflux-01"]}}
[wkh@gflux-01 tmp]$ sudo -u flux /usr/local/bin/flux mini run -N4 -l hostname
0: gflux-01
1: gflux-02
3: gflux-04
2: gflux-03
[wkh@gflux-01 tmp]$ flux mini run -N4 -l hostname
flux-mini: ERROR: [Errno 1] only instance owner can use sign-type=none

All I'm left wondering about, today, is that final error. What do I need to configure to allow an arbitrary user to do useful things?

[garlick]

[wkh@gflux-01 job-submit-cli]$ flux mini submit --nodes=2 --ntasks=4 --cores-per-task=2 ./compute.lua 120
[Errno 1] only instance owner can use sign-type=none

Was flux core not configured --with-flux-security. That is required for a system installation of flux.

[wkharold]

Progress report: rebuilt from scratch on a new collection of nodes using --with-flux-security. Everything seems to work as advertised!

Thanks for your patience and support.

Right now I just have a set of "compute" nodes, rank 0-n. Where should I look for information on a) adding "management" and "login" nodes to the cluster (do they end up in the resource encoding R?) and b) supporting heterogeneous nodes (N with x cores, M with y cores, T with attached GPUs, etc.)?

Thanks again.

[garlick]

Yes we usually include management and login nodes in R, and then add them to the exclude list in the [resource] table so they are not allocated to jobs. Usually the rank 0 node is the management node.

To generate a more complex R you can run flux R encode multiple times to generate fragments, then combine them with flux R append, e.g.

$ flux R encode --ranks=0-1 --hosts=foo[0-1] --cores=0-3 >R1
$ flux R encode --ranks=2-3 --hosts=foo[2-3] --cores=0-1 --gpus=0-1 >R2
$ cat R1 R2 | flux R append >R

You can also hand edit the result since it's not too complicated. @grondo may have additional suggestions for you.

[grondo]

You can also run multiple flux R encode in a subshell when that is more convenient:

$ (flux R encode --ranks=0-1 --hosts=foo[0-1] --cores=0-3 && flux R encode --ranks=2-3 --hosts=foo[2-3] --cores=0-1 --gpus=0-1) | flux R append | jq
{
  "version": 1,
  "execution": {
    "R_lite": [
      {
        "rank": "0-1",
        "children": {
          "core": "0-3"
        }
      },
      {
        "rank": "2-3",
        "children": {
          "core": "0-1",
          "gpu": "0-1"
        }
      }
    ],
    "starttime": 0,
    "expiration": 0,
    "nodelist": [
      "foo[0-3]"
    ]
  }
}

[wkharold]

Thanks again for all your support. Here's what I've got:

wkh@gfluxx-01 flux]$ flux resource list
     STATE PROPERTIES NNODES   NCORES    NGPUS NODELIST
      free                 9       54        0 gfluxx-[02-04,11-16]
      free compute         4      120        0 gfluxx-[21-24]
 allocated                 0        0        0 
      down                 1        2        0 gfluxx-01

gfluxx-01 is the mgmt node; the 'compute' nodes are 60 vCPU (30 sans hyperthreading) compute optimized instances. A minor UX nit: it would be nice if excluded nodes showed up as something other than down.

I'm going through the Workflow Examples, so far everything works as advertised.

Quick question: I set up hostbased authentication to make admin things easier across nodes. Beyond admin use is password-less ssh between nodes necessary, e.g., to do MPI?

[grondo]

A minor UX nit: it would be nice if excluded nodes showed up as something other than down.

flux resource list queries the scheduler state of resources. The scheduler should not see excluded nodes unless the nodes were excluded after the scheduler module was started, then they do appear as down (the scheduler only knows about up/allocated/down etc. states. To view administrative status of resources try flux resource status.

[garlick]

Nice!

I'm not sure why gfluxx-01 is showing up as down in flux resource list output. It shouldn't if it is excluded. Has flux picked up the exclusion config? You can try flux config reload.

Passwordless ssh is useful if you want to connect to a flux instance that you've started as a job. For instance, you can start a batch job and then watch what it's running with flux top JOBID. The JOBID will get resolved to a ssh:// URI that won't work if ssh prompts for a password (at least I'm pretty sure not). It shouldn't be necessary for MPI as long as flux is launching mpi "natively" as opposed to using another launcher that requires ssh like hydra (if I recall?)

[grondo]

For instance, you can start a batch job and then watch what it's running with flux top JOBID

Perhaps even more useful, you can "proxy" to a batch job with flux proxy JOBID, which lets you interact with a batch job that is also an instance of Flux, e.g. list jobs, cancel them, submit more work, etc.

For flux proxy at least, the underlying ssh will prompt for a password and if correctly typed should work fine.

ƒ(s=15,d=0,builddir) grondo@asp:~ $ flux jobs
       JOBID USER     NAME       ST NTASKS NNODES  RUNTIME NODELIST
   ƒSP5mhocT grondo   flux        R      1      1   1.915s asp
ƒ(s=15,d=0,builddir) grondo@asp:~ $ flux proxy ƒSP5mhocT
Password: 
ƒ(s=1,d=1,builddir) grondo@asp:~ $ flux jobs
$ flux jobs
       JOBID USER     NAME       ST NTASKS NNODES  RUNTIME NODELIST
     ƒSqrUud grondo   sleep       R      1      1   2.069m asp

For other tools (like perhaps flux top) typing your password will be onerous though. E.g. there are recursive jobs listing support where you will definitely want passwordless ssh, e.g.:

$ flux pstree -x
       JOBID USER     ST NTASKS NNODES  RUNTIME
   ƒUB4tk8nB grondo    R      1      1   16.76s flux
     ƒRo4zMV grondo    R      1      1   14.98s ├── sleep
     ƒRmb15A grondo    R      1      1   14.98s ├── sleep
     ƒRmb159 grondo    R      1      1   14.98s └── sleep
   ƒSP5mhocT grondo    R      1      1   4.212m flux
     ƒSqrUud grondo    R      1      1   4.181m └── sleep

[wkharold]

Re: UX nit: I think I just did a systemctl reload flux rather than a restart; when I bounced everything the mgmt node no longer show up as down. Great stuff.

I've started playing with properties and the --requires switch to get a little more control over the machine types a job gets assigned to. So far it's working well. Is there a way to get the PROPERTIES field in the resource listing wider? I know what the properties are but others might not and I will have forgotten by Monday.

Thanks!

[grondo]

The output of flux resource list can be set with the -o, --format=FORMAT option, but unfortunately there is no way at this time to change the default output.

The default format string is

{state:>10} {properties:<10.10} {nnodes:>6} {ncores:>8} {ngpus:>8} {nodelist}

When there are properties present.
For now you can try changing the width available to the properties field, e.g.:

$ flux resource list -o "{state:>10} {properties:<25.25} {nnodes:>6} {ncores:>8} {ngpus:>8} {nodelist}"
     STATE PROPERTIES                NNODES   NCORES    NGPUS NODELIST
      free long-property-name             3       12        0 asp,asp,asp
      free                               12       48        0 asp,asp,asp,asp,asp,asp,asp,asp,asp,asp,asp,asp
 allocated                                0        0        0 
      down                                0        0        0 

Alternately, users could "query" properties with something like

$ flux resource list -no "{nnodes:>6} {ncores:>8} {properties}"
     3       12 long-property-name
    12       48 

Sorry there isn't a better way to handle this!