Running Flux on Argonne machines

[jameshcorbett]

I am trying to get a Flux instance up and running on Theta, an ALCF machine that uses Cobalt as its resource manager. After spack-installing flux-sched@master, I tried to start a multi-node Flux instance:

corbett8@thetamom2:/gpfs/mira-home/corbett8> aprun -n 2 -N 1 -d 64 -j 4 -cc depth flux start flux mini run -n1 hostname
sh: ldconfig: command not found
sh: ldconfig: command not found
nid00000
nid00001

(The -N option determines the number of tasks per node, not the number of allocated nodes. And the -d 64 -j 4 -cc depth says to give Flux access to all of the hardware threads on the node. The KNL nodes have 256 hardware threads per node.)

It seems that Flux isn't picking up the fact that it's been launched under MPI---it looks like I get two independent Flux instances. It also seems that Flux isn't registering all of the resources available to it:

corbett8@thetamom2:/gpfs/mira-home/corbett8> aprun -n 2 -N 1 -d 64 -j 4 -cc depth flux start flux mini run -n2 -c16 hostname
sh: ldconfig: command not found
sh: ldconfig: command not found
0.048s: job.exception type=alloc severity=0 unsatisfiable request
0.047s: job.exception type=alloc severity=0 unsatisfiable request
2021-02-24T00:46:44.975130Z broker.err[0]: rc2.0: flux mini run -n2 -c16 hostname Exited (rc=1) 3.2s
2021-02-24T00:46:44.977393Z broker.err[0]: rc2.0: flux mini run -n2 -c16 hostname Exited (rc=1) 3.2s
Application 22467445 exit codes: 1
Application 22467445 resources: utime ~8s, stime ~18s, Rss ~18532, inblocks ~1232, outblocks ~0

Any idea what might be going on here? How could I help you find the source of the issue?

In the meanwhile I should be able to make good progress even with Flux instances that can only see part of a single node's resources.

[dongahn]

@jameshcorbett: try

flux resource list

And see what this version reports.

It could be that aprun binds flux brokers to a subset of cores, which keeps flux from discovering all resources.

If this is the case, we need to turn off binding.

[dongahn]

Cool. But the bootstrapping (see below) is still an issue?

[dongahn]

sh: ldconfig: command not found

Probably red herring. But where does it come from?

[jameshcorbett]

Yeah, it is still an issue. And how could I tell where the ldconfig message is coming from?

[dongahn]

which ldconfig. Is your standard environment be able to locate it at least?

And how could I tell where the ldconfig message is coming from?

I think we will have to do some debugging. But I think you probably want to resolve the PMI issues (below) first...

[jameshcorbett]

Oh I thought you meant what application (aprun, flux, or something else) was generating that message. There's no ldconfig in my PATH.

[dongahn]

It seems that Flux isn't picking up the fact that it's been launched under MPI---it looks like I get two independent Flux instances.

We also need to debug this. Can you set FLUX_PMI_DEBUG=1 to see if this cause flux to print out PMI debug traces?

[garlick]

Interesting. We are up and communicating with PMI because get_params worked and gave us reasonable values. The kvs_get is expected to fail when flux is launched by a foreign resource manager. But the kvs_put ought to have succeeded. It's using the provided kvsname. The value is not very long. Hmmm.

[SteVwonder]

kvs_put (kvsname=22468287 key=0 value=S1@Qqw*:U%sG!9k3vv!my>jU]/EMfGlJ:*@Ph8>l,tcp://10.236.16.120:49152)

Looks like a bunch of gibberish followed by a useful tcp endpoint. Is the beginning bit garbage? Or is it encrypted? Weird.

[dongahn]

@jameshcorbett: what specific Cray/HPE system is this exactly? If this is an old Cray, I wonder if there is a specific vendor extension to PMI such a way that the user software (e.g., Cray MPI) needs to comply with additionally. If that is the case, we should probably wrap that up into what akin to pmi-shim on CORAL: https://flux-framework.readthedocs.io/en/latest/coral.html as our compat layer w/ Flux.

@SteVwonder: do you still have your note on your exchanges with Cray on this?

[garlick]

Looks like a bunch of gibberish followed by a useful tcp endpoint. Is the beginning bit garbage? Or is it encrypted? Weird

It's a Z85 encoded CURVE key followed by the endpoint. Together they are the broker's "business card" (borrowed that concept from MPICH). I'll accept weird though.

[SteVwonder]

@SteVwonder: do you still have your note on your exchanges with Cray on this?

Sadly the notes I have boil down to a random link to a gitlab repo with a Cray PMI header in it (not sure if it was ever intended to be released like that, so I'm hesitant to post it here) and a verbal exchange with Larry where he said the person that dealt with XC is no longer at Cray. I could easily be wrong about this, but my understanding was that the Cray-specific extensions were new interfaces to PMI as opposed to changing the semantics of existing PMI interfaces. So IIUC, Cray MPI will only bootstrap under a Cray-enabled launcher, but a Cray-enabled launcher can launch any MPI (not just Cray MPI). Even if all that is true, it could still be the case that the PMI implementation provided on the system is broken in some way.

Looks like a bunch of gibberish followed by a useful tcp endpoint. Is the beginning bit garbage?

Comments in the broker PMI code state: "Each broker writes a "business card" consisting of (currently): pubkey[,URI]. The URI and separator are omitted if broker is a leaf in the TBON and won't be creating its own endpoint." And that pubkey is the output of zcert_public_txt which outputs the public key side of the public/private pair. So that value actually makes sense. Please ignore my comment about that 🤦

[jameshcorbett]

After talking with @dongahn and @SteVwonder yesterday, it turns out installing Flux on Argonne's machines is really not that important---at least for me. The collaborations I was hoping to do on Argonne machines I can do on LC machines instead. So I am just going to forget about these issues for now.

[dongahn]

For the record, there are two things that should be resolved properly before being able to support that platform. This way, we can focus on these issues when we circle back to this.

1. ANL's Conda packaging and Flux's Spack packaging can interplay badly if we are not careful enough. We found ways to mix two. At least we need to document this in platform specific sections like https://flux-framework.readthedocs.io/en/latest/coral.html

2. Cray/HPE PMI is a big big issue. It has a couple of vendor-proprietary extensions and I suspect two problems we see arise from this:

• Failure to bootstrap multi-node flux instance with aprun. Will need more investigation.
• Failure to flux mini run an application compiled/linked with Cray MPI (in our case we try with mpi4py. My hope in getting the info from the vendor is kind of low because of our conversation with the vendor folks the other day. My proposal would be similar to what we did on CORAL: 1) enable low performing MPI solution first; 2) then enable high performing one including MVAPICH and finally Cray MPI (if possible).