mpirun retaining redirected stdout
Asked Answered
G

1

6

I am running in a cluster. I tried to run my executable with 4 different forms:

  1. In serial, with

    myexec
    

    This starts giving output in stdout right away, as expected.

  2. In serial, redirecting stdout and stderr, with

    myexec > out-err.log 2>&1
    

    This starts giving output in out-err.log right away, as expected (verified with cat out-err.log in another terminal).

  3. In parallel, with

    mpirun -n 2 myexec
    

    This starts giving output in stdout right away, as expected.

  4. In parallel, redirecting stdout and stderr, with

    mpirun -n 2 myexec > out-err.log 2>&1
    

    This retains output until job is finished (due to completion or time allowance).

Is there any way of having stdout/stderr "flushed" at runtime in case 4, so I can check out-err.log?

Geum answered 9/6, 2017 at 17:0 Comment(0)
G
4

This is a known feature/issue with redirection in mpi. I found how to solve this:

  1. Add export OMPI_MCA_opal_event_include=poll in ~/.bashrc, or

  2. Add opal_event_include=poll in ~/.openmpi/mca-params.conf (create the dir and/or file if they do not exist).

The sources used to get info are:

https://www.cfd-online.com/Forums/openfoam-installation/162664-openfoam-2-4-0-openmpi-epoll-warning-parallel-job.html

https://github.com/open-mpi/ompi/issues/341

https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php#sect20

Geum answered 9/6, 2017 at 21:50 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.