Can popen() make bidirectional pipes like pipe() + fork()?

Question

I'm implementing piping on a simulated file system in C++ (with mostly C). It needs to run commands in the host shell but perform the piping itself on the simulated file system.

I could achieve this with the pipe(), fork(), and system() system calls, but I'd prefer to use popen() (which handles creating a pipe, forking a process, and passing a command to the shell). This may not be possible because (I think) I need to be able to write from the parent process of the pipe, read on the child process end, write the output back from the child, and finally read that output from the parent. The man page for popen() on my system says a bidirectional pipe is possible, but my code needs to run on a system with an older version supporting only unidirectional pipes.

With the separate calls above, I can open/close pipes to achieve this. Is that possible with popen()?

For a trivial example, to run ls -l | grep .txt | grep cmds I need to:

• Open a pipe and process to run ls -l on the host; read its output back
• Pipe the output of ls -l back to my simulator
• Open a pipe and process to run grep .txt on the host on the piped output of ls -l
• Pipe the output of this back to the simulator (stuck here)
• Open a pipe and process to run grep cmds on the host on the piped output of grep .txt
• Pipe the output of this back to the simulator and print it

man popen

From Mac OS X:

The popen() function 'opens' a process by creating a bidirectional pipe, forking, and invoking the shell. Any streams opened by previous popen() calls in the parent process are closed in the new child process. Historically, popen() was implemented with a unidirectional pipe; hence, many implementations of popen() only allow the mode argument to specify reading or writing, not both. Because popen() is now implemented using a bidirectional pipe, the mode argument may request a bidirectional data flow. The mode argument is a pointer to a null-terminated string which must be 'r' for reading, 'w' for writing, or 'r+' for reading and writing.

Levorotatory

Ignore this bad manual, which is making it seem like unidirectionality of pipes is a legacy behavior. Your implementation's behavior is nonstandard and not likely to ever be supported in other environments. Simply setup the pipes and fork and exec yourself, and all will be happy. – Hoot

This entire thread is made of win, all the answers and the question deserve +1. – Dreyfus

I just did a man popen on Ubuntu 13.04 and it states:Since a pipe is by definition unidirectional, the type argument may specify only reading or writing, not both; the resulting stream is correspondingly read-only or write-only. I'm surprised this is the "older" popen... – Adumbral

See also Trying to use pipe() to read from/write to another program. – Yes

Answer 1

You seem to have answered your own question. If your code needs to work on an older system that doesn't support popen opening bidirectional pipes, then you won't be able to use popen (at least not the one that's supplied).

The real question would be about the exact capabilities of the older systems in question. In particular, does their pipe support creating bidirectional pipes? If they have a pipe that can create a bidirectional pipe, but popen that doesn't, then I'd write the main stream of the code to use popen with a bidirectional pipe, and supply an implementation of popen that can use a bidirectional pipe that gets compiled in an used where needed.

If you need to support systems old enough that pipe only supports unidirectional pipes, then you're pretty much stuck with using pipe, fork, dup2, etc., on your own. I'd probably still wrap this up in a function that works almost like a modern version of popen, but instead of returning one file handle, fills in a small structure with two file handles, one for the child's stdin, the other for the child's stdout.

Merit

I think you've helped me realize that my question was really, "Is the bidirectional capability 'common'?", and it seems it is not. Thanks for the great response. – Levorotatory

+1 for the popen for one direction + pipe() for the other. Saves me from rewriting most of the cout statements, and only have to change cin instead. – Maltzman

Answer 2

I'd suggest writing your own function to do the piping/forking/system-ing for you. You could have the function spawn a process and return read/write file descriptors, as in...

typedef void pfunc_t (int rfd, int wfd);

pid_t pcreate(int fds[2], pfunc_t pfunc) {
    /* Spawn a process from pfunc, returning it's pid. The fds array passed will
     * be filled with two descriptors: fds[0] will read from the child process,
     * and fds[1] will write to it.
     * Similarly, the child process will receive a reading/writing fd set (in
     * that same order) as arguments.
    */
    pid_t pid;
    int pipes[4];

    /* Warning: I'm not handling possible errors in pipe/fork */

    pipe(&pipes[0]); /* Parent read/child write pipe */
    pipe(&pipes[2]); /* Child read/parent write pipe */

    if ((pid = fork()) > 0) {
        /* Parent process */
        fds[0] = pipes[0];
        fds[1] = pipes[3];

        close(pipes[1]);
        close(pipes[2]);

        return pid;

    } else {
        close(pipes[0]);
        close(pipes[3]);

        pfunc(pipes[2], pipes[1]);

        exit(0);
    }

    return -1; /* ? */
}

You can add whatever functionality you need in there.

Doralynn

Answer 3

POSIX stipulates that the popen() call is not designed to provide bi-directional communication:

The mode argument to popen() is a string that specifies I/O mode:

1. If mode is r, when the child process is started, its file descriptor STDOUT_FILENO shall be the writable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the readable end of the pipe.
2. If mode is w, when the child process is started its file descriptor STDIN_FILENO shall be the readable end of the pipe, and the file descriptor fileno(stream) in the calling process, where stream is the stream pointer returned by popen(), shall be the writable end of the pipe.
3. If mode is any other value, the result is unspecified.

Any portable code will make no assumptions beyond that. The BSD popen() is similar to what your question describes.

Additionally, pipes are different from sockets and each pipe file descriptor is uni-directional. You would have to create two pipes, one configured for each direction.

Yes

Answer 4

In one of netresolve backends I'm talking to a script and therefore I need to write to its stdin and read from its stdout. The following function executes a command with stdin and stdout redirected to a pipe. You can use it and adapt it to your liking.

static bool
start_subprocess(char *const command[], int *pid, int *infd, int *outfd)
{
    int p1[2], p2[2];

    if (!pid || !infd || !outfd)
        return false;

    if (pipe(p1) == -1)
        goto err_pipe1;
    if (pipe(p2) == -1)
        goto err_pipe2;
    if ((*pid = fork()) == -1)
        goto err_fork;

    if (*pid) {
        /* Parent process. */
        *infd = p1[1];
        *outfd = p2[0];
        close(p1[0]);
        close(p2[1]);
        return true;
    } else {
        /* Child process. */
        dup2(p1[0], 0);
        dup2(p2[1], 1);
        close(p1[0]);
        close(p1[1]);
        close(p2[0]);
        close(p2[1]);
        execvp(*command, command);
        /* Error occured. */
        fprintf(stderr, "error running %s: %s", *command, strerror(errno));
        abort();
    }

err_fork:
    close(p2[1]);
    close(p2[0]);
err_pipe2:
    close(p1[1]);
    close(p1[0]);
err_pipe1:
    return false;
}

https://github.com/crossdistro/netresolve/blob/master/backends/exec.c#L46

(I used the same code in popen simultaneous read and write)

Jasik

Answer 5

Here's the code (C++, but can be easily converted to C):

#include <unistd.h>

#include <cstdlib>
#include <cstdio>
#include <cstring>
#include <utility>

// Like popen(), but returns two FILE*: child's stdin and stdout, respectively.
std::pair<FILE *, FILE *> popen2(const char *__command)
{
    // pipes[0]: parent writes, child reads (child's stdin)
    // pipes[1]: child writes, parent reads (child's stdout)
    int pipes[2][2];

    pipe(pipes[0]);
    pipe(pipes[1]);

    if (fork() > 0)
    {
        // parent
        close(pipes[0][0]);
        close(pipes[1][1]);

        return {fdopen(pipes[0][1], "w"), fdopen(pipes[1][0], "r")};
    }
    else
    {
        // child
        close(pipes[0][1]);
        close(pipes[1][0]);

        dup2(pipes[0][0], STDIN_FILENO);
        dup2(pipes[1][1], STDOUT_FILENO);

        execl("/bin/sh", "/bin/sh", "-c", __command, NULL);

        exit(1);
    }
}

Usage:

int main()
{
    auto [p_stdin, p_stdout] = popen2("cat -n");

    if (p_stdin == NULL || p_stdout == NULL)
    {
        printf("popen2() failed\n");
        return 1;
    }

    const char msg[] = "Hello there!";
    char buf[32];

    printf("I say \"%s\"\n", msg);

    fwrite(msg, 1, sizeof(msg), p_stdin);
    fclose(p_stdin);

    fread(buf, 1, sizeof(buf), p_stdout);
    fclose(p_stdout);

    printf("child says \"%s\"\n", buf);

    return 0;
}

Possible Output:

I say "Hello there!"
child says "     1      Hello there!"

Seidler

Answer 6

No need to create two pipes and waste a filedescriptor in each process. Just use a socket instead. https://mcmap.net/q/394270/-open-a-cmd-program-with-full-functionality-i-o

Buckler

Can you please add more details why a filedescriptor for each process would be a "waste"? I mean what does it cost? What is cheaper with the domain socket? – Blacklist

I'm also not convinced that socket is required in this case. But if it is indeed better than pipes, I'd like to know how and why. – Kelso

"Socket descriptors are implemented as file descriptors in the UNIX System." - then I guess it is the same in the end – Homophonic

What more details do you need? You need two pipes to support bidirectional communication, which uses 2 filedescriptors each. You only need one socket to support bidirectional communication, so it only uses 2 filedescriptors total. If you're writing a server that will handle a lot of client connections, the cost of descriptors will add up. – Buckler