How to run MPI compatible applications from Jupyter notebooks?

Question

So I have a trobule with gmsh.

Direct execution works fine:

!gmsh -3 -algo meshadapt tmp_0.geo -o SFM.msh

While execution from code fails:

try:
    out = subprocess.check_output(
            ["gmsh", "gmsh -3 -algo meshadapt tmp_0.geo -o SFM.msh"],
            stderr=subprocess.STDOUT
            ).strip().decode('utf8')
except subprocess.CalledProcessError as e:
    out = e.output
print(out)

with:

b"--------------------------------------------------------------------------\n[[23419,1],0]: A high-performance Open MPI point-to-point messaging module\nwas unable to find any relevant network interfaces:\n\nModule: OpenFabrics (openib)\n Host: 931136e3f6fe\n\nAnother transport will be used instead, although this may result in\nlower performance.\n--------------------------------------------------------------------------\n\x1b[1m\x1b[31mFatal : Can't open display: (FLTK internal error)\x1b[0m\n--------------------------------------------------------------------------\nMPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD \nwith errorcode 1.\n\nNOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.\nYou may or may not see output from other processes, depending on\nexactly when Open MPI kills them.\n--------------------------------------------------------------------------\n"

So how to emulate ! execution in jupyter from Python 3 code?

---

@Hristo:

_=/opt/conda/bin/jupyter SHLVL=1 PATH=/opt/conda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=931136e3f6fe HOME=/root LC_ALL=C.UTF-8 PWD=/ JPY_PARENT_PID=1 LANG=C.UTF-8 TERM=xterm-color CLICOLOR=1 PAGER=cat GIT_PAGER=cat MPLBACKEND=module://ipykernel.pylab.backend_inline env DISPLAY=:0 gmsh -3 -algo meshadapt tmp_0.geo -o SFM.msh

@Gilles: Same result.

Acupuncture

Can you export OMPI_MCA_btl=^openib and try again ? It is hard to figure out whether the root cause is infiniband (MPI) or the display issue (likely related to the app) – Nettles

It's complaining that it cannot open a connection to the display server, which means the DISPLAY environment variable is not set properly. Try running the command as ["env", "env DISPLAY=:0 gmsh -3 -algo ..."]. Do an echo $DISPLAY in a graphical terminal to obtain the proper value. If the Jupyter server is running on under a different account, it will likely not work unless xhost + is issued in a terminal (dangerous - disables display server authentication). It will likely not work at all if Jupyter is running on a different host. – Housecoat

Answer 1

It seems the root cause is the $DISPLAY environment variable is not set.

first make sure $DISPLAY is set when your Jupyter notebook starts. you might also have to direct mpirun to export it to all the MPI tasks.

starting from Open MPI 3.0.0, you can achieve this with export OMPI_MCA_mca_base_env_list=DISPLAY before starting your Jupyter notebook

By the way, should your application need to open the X display ? If it does not do any graphics, then it could be adjusted to work correctly when no display is available.

[ADDENDUM]

An other possibility is that gmsh thinks a display is available since DISPLAY is set, so it tries to open it and fails. You can try to unset this environment variable, and see how things go, both from the command line (e.g. interactive mode) and via the notebook (e.g. batch mode)

Nettles