nm751 and MPI issues

[afacius]

I tried to use a (modified) version of NONMEM.Dockerfile. It seems to work well for single-threaded nm, but I can’t get the mpi support up and running. When I apt-get install libmpich-dev mpich and replace the /opt/NONMEM/nm7.5.1/mpi/mpi_ling/libmpich.a with /usr/lib/x86_64-linux-gnu/libmpich.a as proposed, I get the following error when testing the installation:

*** An error occurred in MPI_Comm_rank
*** reported by process [3785162753,6]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_COMM: invalid communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)

Any ideas how to fix that?

Axel

The base image I'm using is FROM rocker/verse:4.2.3. I installed nm751 on top of it and it appears to work fine (CONTROL5 gives the expected OFV). Happy to provide my modified scripts if that helps.

[billdenney]

Can you share the exact variant Dockerfile that you're using? There are some subtleties about MPI that I have worked through over time. Important details that I see from the Dockerfile (but I don't recall all of the issues that I had getting there):

I believe that libmpich.a must be replaced in the NONMEM directory after setting up NONMEM (order is important). I think that trying to do it before setup did not work.

You're using rocker for your base installation which starts from Debian while I'm starting from Ubuntu. I don't know how differently they may handle MPI.

What OS are you using? (There was a recent issue, #10, with the NONMEM side of OS detection on Mac due to the interaction of M1 chips and using Linux in the Docker container.)

[billdenney]

Your Dockerfile didn't come through; I think that you have to upload it directly to GitHub rather than using an email attachment.

[afacius]

If the issue is because of debian vs ubuntu, I could probably also change this. Just tried to inherit as much as possible with my starting point. But I guess I could install R and RStudio also myself on top of ubuntu.

I’m using MacOS Ventura 13.3.1 (a) with Intel chips, so I would assume the M1chip issue should not apply.

Anyway, here is my Dockerfile along with the install scripts and an empty resources folder. After unzipping, the folder needs to be populated with a valid nonmem.lic, a nm751 installation archive (NONMEM751.zip downloaded from https://nonmem.iconplc.com/) and a simple text file NONMEM751.pwd containing the zip password y*** as a single line of text (no newline at the end). Eventually, this is what you should have to run docker build -t nonmem-workbench:7.5.1 . from within the parent folder.

NONMEM-Workbench-7.5.1/
|- Dockerfile
+- resources/
| |- NONMEM751.pwd
| |- NONMEM751.zip
| '- nonmem.lic
+- scripts/
|- install_MPI.sh
|- install_NONMEM.sh
'- worker_user.sh

NONMEM-Workbench-7.5.1.zip

[billdenney]

You're changing several things simultaneously, in your installation settings, and I'm not sure which is causing the issue.

One change that sticks out is that you seem to be installing NONMEM into one directory and then running it from another (the mpi directory that you create). I think that the installation may still be looking to the original directory.

I'd suggest having a single NONMEM installation and using MPI and single-thread from the same location. That works with the Dockerfile that I've created, and it's the way that I've typically seen it run.

[afacius]

This is only a single installation but I was creating subdirectories for testing NONMEM in single-thread and mpi mode. I will try and use your Dockerfile unchanged and see if this works in my environment (should have done this to start with...). If this is the case, I will try to apply my changes to see when things break. Let you know my progress on this.

[billdenney]

Did you have success in your testing?

[afacius]

Sorry, I had little time testing this during the last weeks. But yes, I have some success. I could build a docker image based on your docker file and there things seem to work well. Somehow this seems to be related with my base layer (I tried to build on rocker/verse:4.2.3). When I change this to ubuntu:22.04 my Dockerfile produces a container with both linear and MPI executions running.
Now I need to see how I can get the rocker elements back (RStudio etc.)

[billdenney]

Thanks for letting me know. That is odd because rocker/verse:4.2.3 is based on 22.04 (https://github.com/rocker-org/rocker-versioned2/blob/master/dockerfiles/r-ver_4.2.3.Dockerfile). Please share your findings as I'm also interested in making a combined rocker/verse and NONMEM container.

FYI, there is another project that makes an nlmixr2 container (https://github.com/RichardHooijmaijers/nlmixr.docker/). It may be useful to use that as your base.

[ldbro0]

I've also experienced this same error with a rocker based image, @afacius are you able to share your final dockerfile?

[afacius]

Hi, I only use docker in batch mode to execute NONMEM runs in a controlled Linux environment while the rest of my stuff (R, RStudio, etc) is on my local Mac. Eventually I did not found the Rstudio on Docker a good solution as I need to use a browser to go there and can only run a single instance, etc (the latter could be avoided with the non-free RStudio of course). So I do have a docker image which installs NONMEM, PsN and MPI support on top of ubuntu but not on top of rocker/verse.