proper flux-sched configuration

[wkharold]

I have a management node with flux-security, flux-core (--with-flux-security), and flux-sched installed.

I have a compute node with flux-security, and flux-core (--with flux-security) installed.

The management node exports /usr/local/etc/flux and the compute node mounts it.

flux resource list returns:

      free manager         1       30        0 gf-manager-1
      free compute         1       30        0 gf-compute-1
 allocated                 0        0        0 
      down                 0        0        0 

Most commands work as expected. When, however, I do flux mini alloc -N1 -n12 I get

2022-07-06T19:05:21.949309Z broker.err[0]: rc1.0: flux-module: sched-fluxion-resource: not found in module search path
2022-07-06T19:05:21.949372Z broker.err[0]: rc1.0: /usr/local/etc/flux/rc1 Exited (rc=1) 0.2s
[detached: session exiting]

but, if I do flux mini alloc -N2 -n12 the command executes successfully and flux resource list says

     STATE PROPERTIES NNODES   NCORES    NGPUS NODELIST
      free manager         1        6        0 gf-manager-1
      free compute         1        6        0 gf-compute-1
 allocated                 0        0        0 
      down                 0        0        0 

As might be expected flux mini batch ... behaves similarly. If I exclude the management node then any flux mini alloc ... attempt fails with the message above.

While the FLUX_MODULE_PATH directory on the management node contains the shared lib for sched-fluxion-resource the corresponding directory on the compute node does not. Which stands to reason since flux-sched is not installed there. So the error seems understandable.

In what seems like a hack to get things working I have exported the /usr/local/lib/flux directory from the management node and mounted it on the compute node. With the necessary share libs in place things work as expected. But that begs the question of what is the point of not building and installing flux-sched on the compute nodes?

I'm no doubt missing something simple but I would like to get a correct, per LLNL operations, config nailed down before I make my work on reproducibly deploying flux available to my colleagues.

[garlick]

alloc and batch start new Flux instances so they will need a scheduler on whatever node is their rank 0 (not necessarily the management node). Since you're sharing the etc directory, and that is where fluxion installs its rc scripts, when the new instance starts, it tries to run fluxion's rc script to load the fluxion modules, but doesn't find them.

My first thought would be to either install fluxion everywhere or don't share the entire etc directory (maybe do a selective rsync or some sort of config management?)

[wkharold]

Thanks! I went with installing fluxion (flux-sched) everywhere for now. Perhaps I'll try to narrow what gets shared out of the etc dir in the future.

Follow-up question: when alloc | batch start a new flux instance how do I determine which allocated node is rank 0? For example, in my latest cluster I did flux mini alloc -N3 -n3 and flux job info <jobid> R yields

{"version": 1, "execution": {"R_lite": [{"rank": "1-3", "children": {"core": "29"}}], "nodelist": ["gflux-[002-004]"], "properties": {"compute": "1-3"}, "starttime": 1657147893, "expiration": 1657752693}}

is gflux-002 rank 0 for this flux instance?

[garlick]

Yup. Maybe easier: flux jobs also shows this under the NODELIST column (hosts are in rank order), e.g.

 garlick@picl0:~$ flux mini submit sleep 60
ƒ2TYHhzmkP9
 garlick@picl0:~$ flux mini submit -N2 sleep 60
ƒ2TYSY93eg7
 garlick@picl0:~$ flux jobs
       JOBID USER     NAME       ST NTASKS NNODES  RUNTIME NODELIST
 ƒ2TYSY93eg7 garlick  sleep       R      2      2   2.263s picl[4,6]
 ƒ2TYHhzmkP9 garlick  sleep       R      1      1   48.35s picl2

[wkharold]

OK, great. And if I were to narrow down what is shared from etc, so that the rc scripts aren't visible, this new instance would just use the simple scheduler, yes?

[garlick]

Yes.

[grondo]

You should setuid /usr/local/libexec/flux/flux-imp, not flux-shell (which will make all your job tasks run as root)

…

On Thu, Jul 7, 2022 at 3:40 PM wkharold ***@***.***> wrote: Actually, it doesn't go away with flux-shell setuid everywhere instead I get a bunch of 0.046s: flux-imp[2]: stderr: flux-shell: ERROR: output: shell_output_write: Operation not permitted 0.046s: flux-imp[3]: stderr: flux-shell: ERROR: output: shell_output_write: Operation not permitted 0.046s: flux-imp[2]: stderr: flux-shell: ERROR: output: shell_output_write: Operation not permitted — Reply to this email directly, view it on GitHub <#4394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFVEUS6AEYTONX6LHFFSYDVS5ME3ANCNFSM52236M5Q> . You are receiving this because you are subscribed to this thread.Message ID: <flux-framework/flux-core/repo-discussions/4394/comments/3103657@ github.com>

[wkharold]

Got it, thanks! That whole question thread turned out to be the result of mismatched uid/gid assignments across the cloud instances.