Force a program created using `exec` to perform unbuffered I/O

Question

I am trying to interact with an external program in C using pipe, fork and exec. I want to force the external program to perform unbuffered I/O. Here's a relevant snippet from my code so far:

...
pid = fork();
if (pid == (pid_t) 0)
{
    /* child process */

    /* make pipe connections to standard streams */             
    dup2(wpipe[0], STDIN_FILENO);
    dup2(rpipe[1], STDOUT_FILENO);

    /* close pipe endings */
    close(wpipe[0]); close(rpipe[0]);
    close(wpipe[1]); close(rpipe[1]);

    /* unbuffered I/O */
    setvbuf(stdin, NULL, _IONBF, BUFSIZ);
    setvbuf(stdout, NULL, _IONBF, BUFSIZ); 

    if (execl("path/to/some_binary", "path/to/some_binary", (char *) NULL) == -1)
    {
        fprintf(stderr, "exec failed\n");
        return EXIT_FAILURE;
    }   
    return EXIT_SUCCESS;
}
...

This does not work because streams do not survive across exec calls. So using setvbuf to force unbuffered I/O does not work, because the program image (some_binary) creates stdin and stdout streams of its own, and does not use the streams that I called setvbuf on.

The program does work when I re-build some_binary after adding the setvbuf calls in its code. But how can this be done if you do not have any control over the binary you are passing to exec? How can this be made to work for, say, a unix command like cat or ls?

Laceration

Check the unbuffer command that comes with expect. – Oreste

Why do you ask?? What is the use-case???? – Zenda

When doing two-way communication with a sub-process using two pipes. With I/O buffering, read operations in the parent process get blocked simply because the output has been buffered by the child sub-process. – Laceration

What exactly is the program running (after execve) in the child process? – Zenda

related: There are existing utilities that can do it for you. Ignore Python-specific code or reimplement it in C e.g., there is a select loop over pseudo-tty. – Brython

Answer 1

You cannot do what you want in the general case (unbuffering after execve(2) of arbitrary executables...)

Buffering is done by code (e.g. by some libc code related to <stdio.h>). And the code is defined by the program being execve-ed.

You might perhaps play with LD_PRELOAD tricks which might call setvbuf(stdin, NULL, _IONBF, BUFSIZ); after the execve (but before the main....); but this would work only with dynamically linked executables.

Perhaps using some constructor function attribute in some initialization function of your LD_PRELOAD-ed shared object might sometimes do the trick. Or redefine printf, fopen, .... in that shared object...

addenda

You commented that you do :

two-way communication with a sub-process using two pipes.

Then your approach is wrong. The parent process should monitor the two pipes, probably with a multiplexing call like poll(2), then (according to the result of that multiplexing) decide to read or to write to the child process. In reality, you want some event loop: either implement a simple event loop yourself (with e.g. poll [iteratively called many times] inside a repeated loop) or use some existing one (see libevent or libev, or the one provided by some toolkit like GTK or Qt etc...)

^{You might also multiplex with select(2) but I recommend poll because of the C10K problem}

You won't lose your time by reading Advanced Linux Programming.

See also this answer to your next related question.

Zenda

Thank you for the addendum! Is the poll function available in GNU C? Why does it not appear in the documentation of GNU C? – Laceration

What do you call GNU C? Is it GCC the compiler (which is unrelated to your question) or Glibc (a.k.a. GNU libc) the standard C library? poll is documented in POSIX poll and in the man pages!! And GNU libc does not document all of POSIX (but assumes it is known). BTW, GNU libc documents select -which is an old-fashion way to multiplex. – Zenda

I meant glibc. I couldn't find the poll function documented on this page: gnu.org/software/libc/manual/html_mono/libc.html – Laceration

I tried to implement polling, but again ran into problems because of buffering - #20581966 – Laceration

You need an event loop the poll should be repeated inside a loop... (with different actions when you get readable and writable file descriptors given by poll) – Zenda

Answer 2

You can actually do that!

One way to do that is to just replace your pipe/socketpair with openpty():

int master, slave, pid;

if (openpty(&master, &slave, NULL, NULL, NULL) == -1) {
    perror("openpty");
    return -1;
}

pid = fork();
if (pid == -1) {
    perror("fork");
    return -1;
}

if (pid == 0) {
    setsid();
    if (ioctl(slave, TIOCSCTTY, NULL) == -1) {
        perror("ioctl slave TIOCSCTTY");
        exit(EXIT_FAILURE);
    }

    dup2(slave, STDIN_FILENO);
    dup2(slave, STDOUT_FILENO);
    dup2(slave, STDERR_FILENO);

    execvp(args[0], args);
    perror("execvp");
    exit(EXIT_FAILURE);
}

// Sample read/write using select, add a flow control by yourself

struct timeval tv;

fd_set fdr, fdw;
FD_ZERO(&fdr);
FD_SET(master, &fdr);
tv.tv_sec = 1;
tv.tv_usec = 0;
if (select(master+1, &fdr, NULL, NULL, &tv) < 0) {
    printf("Nothing to read\n");
}
if (FD_ISSET(master, &fdr)) {
    // Read stuff...
}

FD_ZERO(&fdw);
FD_SET(master, &fdw);
tv.tv_sec = 1;
tv.tv_usec = 0;
if (select(master+1, NULL, &fdw, NULL, &tv) < 0) {
    printf("Not ready to write\n");
}
if (FD_ISSET(master, &fdw)) {
    // Write stuff...
}

Or alternatively you may use stdbuf in your exec() path to start your program, see bash: force exec'd process to have unbuffered stdout

Both methods worked for me.

Path

Answer 3

You might try to pause the task, then modify its image. You can do it via gdb or the /proc filesystem. In particular, you have the access to all fds the task created via the /proc filesystem.

UPDATE:

If the buffering is done by glibc, just override the relevant function in glibc, by defining your own function, compiling it into a shared library and loading with the program.

Harpy