C Control buffering of Child process
Asked Answered
V

3

0

My goal is to control buffering of child process when executed with execvp.

More precisely, I want to redirect stdout and stderr to the same file descriptor (this is wanted behaviour, I cannot change that). But, how the default buffering mechanism work provokes unexpected behaviour. For example, when this python script is executed in your terminal:

print("test sdout recup")
print("test stderr recup", file=sys.stderr)

stdout is line buffered, so the first print is instantly flushed (print adds by default a newline in python) then stderr is unbuffered, and thus directly prompted which lead to:

 test sdout recup
 test stderr recup

When I execute the same script with my C code (see the code in the end), I got all the time:

test stderr recup
test sdout recup

Since stdout is not a terminal, (it's a pipe), stdout become fully buffered, while stderr is still unbuffered, leading to this order.

I need a way to control those modes (from C, not via another process) to preserve the same terminal output, and also later to unbuffer stdout as well (for another goal) but I really don't know how to do. I have seen some code that work with files pointer instead of file descriptors (FD) but I cannot find same functions for the field.

Also, I am not even sure that this is controllable from the parent process. So here I am.

Here is the main code: output.h:

#include <stddef.h>//size_t
typedef struct Output
{
  char* out;
  int status;
  double times;
} Output;


Output* Output_new();
/*Return an correctly initialized Ouput in regard to the buffer size*/
size_t read_append_into_Output( int fd, Output* out, size_t* current_size );
/*Append the result in the output buffer and manage size properly(actualize constructor with new default size, prevent overflow...*/

executor.c:

#include "executor.h"
#include "argv.h"//buildarg

#include <unistd.h>//fork
#include <stdio.h>//pipe
#include <stdlib.h>//EXIT_SUCCESS
#include <sys/wait.h>
#include <string.h> //strlen
#include <errno.h>//perror


#define READ 0
#define WRITE 1

void child_life(char** argv){
    /*Do child stuff*/
    // char* expected[] = {"test.py", "test", NULL};
    execvp(*argv, argv);
    perror("Process creation failed");


}

//TODO better control over when to send in pipe
void parent_life(int read_fd, int write_fd, char** prompt, size_t prompt_number, Output* output){
    //inject prompt
    for (int i=0; i<prompt_number; i++){
        write(write_fd, prompt[i], strlen(prompt[i]));//TODO dont call strlen and control ourself the size?
    }
    size_t readed=0;
    size_t max_read=0;
    while (max_read==readed){//we stop when we read less what we should or error
        max_read= read_append_into_Output(read_fd, output,&readed);
    }
    output->out[readed]=0;
}

Output* executor_get_output(char* command, char** prompt, size_t prompt_number, double timout)
{

    Output* output=Output_new();
    int pipe_father[2];
    int pipe_son[2];
    
    pipe(pipe_father);
    pipe(pipe_son);

    pid_t cpid;

    int argc;
    char** argv= buildargv(command,&argc); // We do it here because code betwen fork and exec is dangerous (must not contain malloc for exemple)

    cpid = fork();



    if (cpid == 0) {    /* Child reads from pipe */
        /*Listening on father pipe*/
        close(pipe_father[WRITE]);          /* Close unused write end */
        dup2(pipe_father[READ], STDIN_FILENO); /*Replace STDIN by our pipe*/


        /*Redirecting stdout and stder to the write pipe*/
        close(pipe_son[READ]);          
        dup2(pipe_son[WRITE], STDOUT_FILENO); /*Replace STDOUT by our pipe*/
        dup2(pipe_son[WRITE], STDERR_FILENO);

        child_life( argv);

        //EXIT (executed only if exevp failed)
        close(pipe_father[READ]);
        close(pipe_son[WRITE]);
        _exit(EXIT_FAILURE);

    } 
    //Parent code
    close(pipe_father[READ]);          /* Close unused read end */
    close(pipe_son[WRITE]);
    parent_life( pipe_son[READ], pipe_father[WRITE], prompt, prompt_number, output);

    //EXIT
    close(pipe_father[WRITE]);          /* Reader will see EOF */
    waitpid(cpid, NULL,0);                /* Wait for child terminaison*/
    
    close (pipe_son[READ]);
    return output;


}

You can find on github a convenient build to compile not shown dependencies of the code you saw plus a test to mess around if you want:

git clone -b dev https://github.com/crazyhouse33/authbreak.git
cd authbreak/build
cmake ..
make executor

This commands create the binary in bin/tests directory.

The associated test source code is in tests/execution/executor.c

The test runs an execution of such a showed python script and compare the obtained output with my already presented expectations. For some reason the test segfault when ran from make test (ctest) but now when you run it manually.

Vasileior answered 14/4, 2020 at 17:59 Comment(4)
Linux or Windows? (it looks like Linux)Brenna
Can you please finish this sentence: ... So, what I need is to: <enter your concise problem statement here> in 30 words or less.Brenna
It's for linux. Can you tell me what it's not clear in "my goal is to control buffering of child process when executed with execvp"? I gave exemple of expected/obtained behaviour and explained why the unexpected behaviour is happening, thinking it would help to understand what's bothering me. I am gonna try to reformul: I want to control how my child process output to my pipe is buffered by the system. If you search around there are 3 modes: unbuffered, line buffered or fully buffered. You can modify those modes when you use file pointer I have seen it. I can't find an equivalent for fd.Vasileior
in more words than necessary I am just suggesting you pare down your question to a minimal reproducible example.Brenna
P
2

my goal is to control buffering of child process when executed with execvp

So basically you want to replicate the job of stdbuf.

Fast screening at coreutils stdbuf sources it shows that stdbuf just sets the environment variables _STDBUF_I=$MODE _STDBUF_O=$MODE and _STDBUF_E=$MODE and then ld preloads the libstdbuf.so library. The libstdbuf is then executed inside the childs process right before it starts - it reads the environment variables _STDBUF_I _STDBUF_O and _STDBUF_E and just applies the proper buffering mode obtained from environment variables inside the childs process.

As you seem to want to reinvent stdbuf, just do the same as it does. (Or, as your question really seems to me like an XY problem, just use stdbuf. It's typical in the shell to use stdbuf -oL in pipes...).

Perplexed answered 14/4, 2020 at 23:14 Comment(1)
Hey, I am gonna try a last technique before validating your solution. I did not want to use external software in the begining because I must time the execution of the command with the best precision possible ( yes this is really important ). Seem it's not that big of a deal since how stdbuf work. Let me check if I dont find something better.Vasileior
L
0

Since the child is a separate program, its will buffer how it wishes, so you need to tell it not to buffer, or change the program.

Since your child seems to be a python script, you can probably make it work by setting the PYTHONUNBUFFERED environment variable, but that is specific to python:

putenv("PYTHONUNBUFFERED=1");
Lagerkvist answered 14/4, 2020 at 23:1 Comment(1)
Hi, I took python just for testing purpose, my code must monitor any program. So this solution is not for me :(Vasileior
V
0

If somebody comes across against the same problem. It seems impossible to do from the parent code whatever you do. I tried to open the pipe as a stream (fdopen) and call setvbuf on every pipe ends, in child and parents. It's silently failing.

So I guess that whatever you do before execve, the loader setup stdin and stdout buffering again and erase your setup. So I did a C module dedicated to play around with envp and argv in order to use environment variables tricks such as the stdbuf solution proposed by KamilCuk from C. Here is how you can use a subset of it in order to answer this question:

#include "argv.h"
#include <unistd.h>


void execvp_buffer_control( char* command, char** argv, char** buffering_mode){
//Like excevp but offer control over child buffering
char **stdbuf_trick_envp = build_stdbuf_exec_envp(buffering_mode);
execve(command, argv,get_envp_appended(stdbuf_trick_envp));// This return an envp vector equal to current envp concatenated with the good stuff to use stdbuff trick 
}
//Where buffering_mode is something as below:
char *mode[] = {DEFAULT_BUFFERING, LINE_BUFFERED, "65536"};# 65536 is the size of the fully buffered stderr stream buffer 

Now if you compile this code and define the variable LIBSTDBUF_PATH at build time to the path of the stdbuflib, it will work as expected. If you don't set it, it will emit a warning at compile time and your execvp_buffer_control will act as a basic execvp.

I created this module with flexibility in mind. It contains a function to merge envp tricks such as the stdbuf with other ones that also use LD_PRELOAD, in order to mix features even if some envp variables collides. I wish to see people pull request me their own function, creating an appropriate envp vector such as build_stdbuf_exec_envp so this module would be really useful to tune the behaviour of child process in combinable ways.

Everything is tested. I will probably do a repository for the module alone, but for now it's in the first posted repository, the sources are in src/common-stuff. Argv.h use unistd.h to retrieve the current envp. But if you replace it with a crossplaform function to do the work, it would become compilable on all platforms.

However, in regard to this question, it obviously suffers from the same drawbacks of stdbuf, and for example won't work with process managing their buffering themself, such as python.

Vasileior answered 25/4, 2020 at 20:17 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.