How to reproduce deadlock hinted to by Boost process documentation?

Question

According to the Boost documentation (section 'Why does the pipe not close?'), the following code will result in a deadlock:

#include <boost/process.hpp>

#include <iostream>

namespace bp = ::boost::process;

int main(void)
{
  bp::ipstream is;
  bp::child c("ls", bp::std_out > is);

  std::string line;
  while (std::getline(is, line))
  {
    std::cout << line << "\n";
  }

  return 0;
}

The documentation says:

This will also deadlock, because the pipe does not close when the subprocess exits. So the ipstream will still look for data even though the process has ended.

However, I am not able to reproduce the deadlock (under Linux). Furthermore, I do not understand why the deadlock would occur in the first place. Once the child process exits it closes the write-end of the pipe. The read-end of the pipe will still be available for the parent process to read from, and std::getline() will fail once no more data is available in the pipe buffer, and the write-end was closed, correct? In case the pipe buffer fills up during execution of the child process, the child process will block waiting for the parent process to read enough data from the pipe so that it can continue.

So in case the above code can deadlock, is there an easy way to reproduce the deadlock scenario?

Update:

Indeed, the following piece of code deadlocks using Boost process:

#include <boost/process.hpp>
#include <iostream>

namespace bp = ::boost::process;

int main() 
{
    bp::ipstream is;
    bp::child c("/bin/bash", bp::args({"-c", "ls >&40"}), bp::posix::fd.bind(40, is.rdbuf()->pipe().native_sink()));

    std::string line;
    while (std::getline(is, line))
    {
        std::cout << line << "\n";
    }

    c.wait();

    return 0;
}

I wonder whether this really is some unavoidable property of process spawning under Linux though. Reproducing the above example using Subprocess from Facebook's Folly library at least does not deadlock:

#include <folly/Subprocess.h>
#include <iostream>

int main()
{
   std::vector<std::string> arguments = {"/bin/bash", "-c", "ls >&40"};

   folly::Subprocess::Options options;
   options.fd(40, STDOUT_FILENO);

   folly::Subprocess p(arguments, options);
   std::cout << p.communicate().first;
   p.wait();

   return 0;
}

Scribbler

Answer 1

Once the child process exits it closes the write-end of the pipe.

This seems to be the assumption. What program closes what pipe?

If /bin/ls does, what happens for

bp::child c("/bin/bash", bp::args({"-c", "ls; ls"}));

If ls really does close it, then it should be closed twice.

Perhaps bash duplicates the handles under the hood, so the subprocesses close different copies of the same pipe. I'm not sure about the reliability of these semantics¹

So, apparently stdout is well-catered for. However, I can reproduce the deadlock when using a non-standard file-descriptor for output on linux:

#include <boost/process.hpp>
#include <iostream>

namespace bp = ::boost::process;

int main() {
    bp::ipstream is;
    bp::child c("/bin/bash", bp::args({"-c", "exec >&40; ls"}), bp::posix::fd.bind(40, is.rdbuf()->pipe().native_sink()));

    std::string line;
    while (std::getline(is, line)) {
        std::cout << line << "\n";
    }
}

I'm not sure why the "closing stdout" behaviour of sub processes in bash should behave differently when it was redirected to an fd, but there you go.

Another nice way to demonstrate a related deadlock is:

{
    bp::child c("/bin/bash", bp::args({"-c", "ls -R /"}), bp::std_out > is);
    c.wait();
    return c.exit_code();
}

This answer is not conclusive but does observe some points and demonstrate them on linux:

• not all filedescriptors seem to be covered the same as stdout
• deadlocks can occur in many scenarios where subprocesses handles their IO asynchronously but the calling process treats them synchronously.

I think the latter was the point in the documentation.

---

¹ indeed the documentation explicitly suggests difference in these semantics are the problem in Win32:

It is not possible to use automatically pipe-closing in this library, because a pipe might be a file-handle (as for async pipes on windows)

Godman

Thanks for the pointers. It looks like the example in Boost does not demonstrate a deadlock scenario (at least not under Linux). I can not compile the code in your first example. It seems this is an issue in pipe.hpp at least in the version that I have? Your last example is related to the deadlock I describe in my question; the child process deadlocks since the parent process does not read from is, therefore the internal buffer for the pipe forwarding stdout from the child process to the parent process fills up, and the child process waits for the parent process to process output. – Scribbler

Regarding the issue in pipe.hpp, see: github.com/boostorg/process/blob/develop/include/boost/process/… rdbuf() should return a pointer, but returns a value object _buf... – Scribbler

@TonvandenHeuvel Oh, aha: indeed. I totally forgot I paved over that. Let me PR...: github.com/klemens-morgenstern/boost-process/pull/117 – Godman

Thanks! It indeed results in a deadlock. – Scribbler