How do I prevent a C shared library to print on stdout in python?

Question

I work with a python lib that imports a C shared library that prints on stdout. I want a clean output in order to use it with pipes or to redirect in files. The prints are done outside of python, in the shared library.

At the beginning, my approach was:

# file: test.py
import os
from ctypes import *
from tempfile import mktemp

libc = CDLL("libc.so.6")

print # That's here on purpose, otherwise hello word is always printed

tempfile = open(mktemp(),'w')
savestdout = os.dup(1)
os.close(1)
if os.dup(tempfile.fileno()) != 1:
    assert False, "couldn't redirect stdout - dup() error"

# let's pretend this is a call to my library
libc.printf("hello world\n")

os.close(1)
os.dup(savestdout)
os.close(savestdout)

This first approach is half working:
- For some reason, it needs a "print" statement just before moving stdout, otherwise hello word is always printed. As a result it will print an empty line instead of all the fuzz the library usually outputs.
- More annoying, it fails when redirecting to a file:

$python test.py > foo && cat foo

hello world

My second python attempt was inspired from another similar thread given in the comments:

import os
import sys
from ctypes import *
libc = CDLL("libc.so.6")

devnull = open('/dev/null', 'w')
oldstdout = os.dup(sys.stdout.fileno())
os.dup2(devnull.fileno(), 1)

# We still pretend this is a call to my library
libc.printf("hello\n")

os.dup2(oldstdout, 1)

This one also fails to prevent "hello" from printing.

Since I felt this was a bit low level, I then decided to go completely with ctypes. I took inspiration from this C program, which does not print anything:

#include <stdio.h>

int main(int argc, const char *argv[]) {
    char buf[20];
    int saved_stdout = dup(1);
    freopen("/dev/null", "w", stdout);

    printf("hello\n"); // not printed

    sprintf(buf, "/dev/fd/%d", saved_stdout);
    freopen(buf, "w", stdout);

    return 0;
}

I built the following example:

from ctypes import *
libc = CDLL("libc.so.6")

saved_stdout = libc.dup(1)
stdout = libc.fdopen(1, "w")
libc.freopen("/dev/null", "w", stdout);

libc.printf("hello\n")

libc.freopen("/dev/fd/" + str(saved_stdout), "w", stdout)

This prints "hello", even if I libc.fflush(stdout) just after the printf. I am starting to think it may be not possible to do what I want in python. Or maybe the way I get a file pointer to stdout is not right.

What do you think?

Lagan

Where do you get the segfault (stack trace)? And, sincerely... shame on the developper of the shared lib. Writing directly to stdout from within a shared library without providing a means to change this behaviour is baaad. – Hon

Unfortunately, I can't find any way of redirecting stdout from within Python. I think you're on the right track here with wrapping your shared library in C, making a dll out of the wrapper and using ctypes to call that. I believe your segfault is due to the sprintf, but I can't really tell what the problem is. – Ephrayim

possible duplicate of Suppressing output of module calling outside library – Catarrhine

I don't get a segfault anymore (and I no more use sprintf), sorry for making your comments outdated but I felt the post is long enough without a stack trace. – Lagan

related: Redirect stdout to a file in Python? – Shearwater

Answer 1

Based on @Yinon Ehrlich's answer. This variant tries to avoid leaking file descriptors:

import os
import sys
from contextlib import contextmanager

@contextmanager
def stdout_redirected(to=os.devnull):
    '''
    import os

    with stdout_redirected(to=filename):
        print("from Python")
        os.system("echo non-Python applications are also supported")
    '''
    fd = sys.stdout.fileno()

    ##### assert that Python and C stdio write using the same file descriptor
    ####assert libc.fileno(ctypes.c_void_p.in_dll(libc, "stdout")) == fd == 1

    def _redirect_stdout(to):
        sys.stdout.close() # + implicit flush()
        os.dup2(to.fileno(), fd) # fd writes to 'to' file
        sys.stdout = os.fdopen(fd, 'w') # Python writes to fd

    with os.fdopen(os.dup(fd), 'w') as old_stdout:
        with open(to, 'w') as file:
            _redirect_stdout(to=file)
        try:
            yield # allow code to be run with the redirected stdout
        finally:
            _redirect_stdout(to=old_stdout) # restore stdout.
                                            # buffering and flags such as
                                            # CLOEXEC may be different

Shearwater

Marked as best answer because this one does not leak. However, I believe that using a decorator, like @Yinon Ehrlich would be more elegant. – Lagan

@user48678: I don't see any decorators in @Yinon Ehrlich's answer. HideOutput is a context manager defined using a class; stdout_redirected is also a context manager but it is defined using contextlib.contextmanager decorator (I could have used a class here with the same result) – Shearwater

Sebastian: Well HideOutput is a decorator. The usage is 'with HideOutput:'. But I did not know contextlib.contextmanager. From the documentation, it is actually a decorator too. I had missed that. – Lagan

@user48678: You've mixed the concepts. with-statement uses context managers, examples: files, locks, decimal.localcontext() (all are objects with __enter__, __exit__ methods). They are different from decorators that are callables (e.g., functions) that accept a callable and return a callable (as a rule), examples: staticmethod, classmethod, makebold,makeitalic. Python supports special @ syntax for decorators. – Shearwater

Absolutely, you are right. I just checked my facts, and a decorator is a completely different thing from what I meant. Apologies for the mistake. – Lagan

@J.F. Sebastian: thanks for clarifying the files descriptors leak and for introducing me with the contextlib.contextmanager decorator – Dimitry

Can anyone explain me what is the problem with the code above, that gave me the following: Traceback (most recent call last): File "...", line 474, in open with stdout_redirected(): File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "...", line 191, in stdout_redirected with os.fdopen(os.dup(fd), 'w') as old_stdout: OSError: [Errno 9] Bad file descriptor ??? – Dimitry

@YinonEhrlich: what is fd? Is it open for writing? – Shearwater

@YinonEhrlich: I meant, are you sure that sys.stdout.fileno() returns an appropriate value? For example, idle, bpython overwrite sys.stdout. Notice the comment: "assert that Python and C stdio write using the same file descriptor". Also the function is not reentrant and it is not safe to use it from multiple threads. – Shearwater

Can this be modified to work with a filename or a file-like object like StringIO or ByteIO ? – Necrophobia

@Shearwater - easily? I've been poking around and it seems you need to use a tempfile.SpooledTemporaryFile to have all the proper os-level file interfaces like a descriptor. – Necrophobia

@BrandonDube click the link – Shearwater

@Shearwater as far as I understand (I'm not a linux user) /dev/null is somewhere you send data to die. I want to recover the data from print statements in C/Fortran linked code. I could use this with a filename and write to a .txt file, then read from it and delete it, or I could do it in memory with e.g. StringIO. It is non-obvious how to do this with this answer -- the .fileno() attribute of to is not present on a BytesIO or StringIO object. – Necrophobia

@BrandonDube pass a different name instead of /dev/null – Shearwater

@Shearwater - that will write to a file. I would prefer to not thrash the filesystem by needlessly making a real file, only to immediately read and delete it. – Necrophobia

@BrandonDube read your first comment. Do you see "filename" word in it? Then read your comment that says io.BytesIO() doesn't provide .fileno(). It seems you've answered your question. – Shearwater

@Shearwater -- allow me to rephrase -- when I wrote or, I meant or as in interoperably, the function works with either a filename input, or a BytesIO or a StringIO object seamlessly. – Necrophobia

@Shearwater Great work, however it look like the descriptor does still leak in a multithreading app. Do you think it is fixable? – Stepper

@Solon: the function manipulates global data. If you need to use multiple threads, then the access to the data should be serialized e.g., use a mutex around the code – Shearwater

I obtained the following error in Python 3.7: AttributeError: 'DupStdoutFileWriter' object has no attribute 'fileno'. Would you have a suggestion for a fix? Thanks. – Levine

Beware that this can cause problems in other places where sys.stdout is used, for example, in logging (StreamHandler). If you have a logger with a StreamHandler using sys.stdout, the old sys.stdout (the one from before this context manager was executed) will still be attached to the StreamHandler on the logger. Given that this function closes the old sys.stdout, the logger will be using a closed stream and will raise errors. – Ulu

@RodrigoBonadia: see whether the answer without explicit sys.stdout.close() will work: Redirect stdout to a file in Python? – Shearwater

@RodrigoBonadia it caused some problems for me, the only solution i found for now is using StreamHandler(stream=sys.stderr) – Senter

Answer 2

Combining both answers - https://mcmap.net/q/25706/-how-do-i-prevent-a-c-shared-library-to-print-on-stdout-in-python & https://mcmap.net/q/25705/-suppressing-output-of-module-calling-outside-library to context manager that blocks print to stdout only for its scope (the code in the first answer blocked any external output, the latter answer missed the sys.stdout.flush() at end):

class HideOutput(object):
    '''
    A context manager that block stdout for its scope, usage:

    with HideOutput():
        os.system('ls -l')
    '''

    def __init__(self, *args, **kw):
        sys.stdout.flush()
        self._origstdout = sys.stdout
        self._oldstdout_fno = os.dup(sys.stdout.fileno())
        self._devnull = os.open(os.devnull, os.O_WRONLY)

    def __enter__(self):
        self._newstdout = os.dup(1)
        os.dup2(self._devnull, 1)
        os.close(self._devnull)
        sys.stdout = os.fdopen(self._newstdout, 'w')

    def __exit__(self, exc_type, exc_val, exc_tb):
        sys.stdout = self._origstdout
        sys.stdout.flush()
        os.dup2(self._oldstdout_fno, 1)

Dimitry

This very neat. The entry from Dietrich Epp was the one that solved my issue at the time I asked this question and he still gets my vote, but yours is more elegant. I am marking it as best. – Lagan

This will leak file descriptors every time you use it. Any descriptor created with os.dup() or os.open() must be somehow closed, and this doesn't happen to _old_stdout_fileno, and if you don't use it in a context then _devnull will also leak. Leaking file descriptors is pretty serious because you only get something like 256 or 1024 of them. – Wheelbase

@DietrichEpp: I've posted solution that tries to avoid leaking file descriptors. – Shearwater

Answer 3

Yeah, you really want to use os.dup2 instead of os.dup, like your second idea. Your code looks somewhat roundabout. Don't muck about with /dev entries except for /dev/null, it's unnecessary. It's also unnecessary to write anything in C here.

The trick is to save the stdout fdes using dup, then pass it to fdopen to make the new sys.stdout Python object. Meanwhile, open an fdes to /dev/null and use dup2 to overwrite the existing stdout fdes. Then close the old fdes to /dev/null. The call to dup2 is necessary because we can't tell open which fdes we want it to return, dup2 is really the only way to do that.

Edit: And if you're redirecting to a file, then stdout is not line-buffered, so you have to flush it. You can do that from Python and it will interoperate with C correctly. Of course, if you call this function before you ever write anything to stdout, then it doesn't matter.

Here is an example that I just tested that works on my system.

import zook
import os
import sys

def redirect_stdout():
    print "Redirecting stdout"
    sys.stdout.flush() # <--- important when redirecting to files
    newstdout = os.dup(1)
    devnull = os.open(os.devnull, os.O_WRONLY)
    os.dup2(devnull, 1)
    os.close(devnull)
    sys.stdout = os.fdopen(newstdout, 'w')

zook.myfunc()
redirect_stdout()
zook.myfunc()
print "But python can still print to stdout..."

The "zook" module is a very simple library in C.

#include <Python.h>
#include <stdio.h>

static PyObject *
myfunc(PyObject *self, PyObject *args)
{
    puts("myfunc called");
    Py_INCREF(Py_None);
    return Py_None;
}

static PyMethodDef zookMethods[] = {
    {"myfunc",  myfunc, METH_VARARGS, "Print a string."},
    {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC
initzook(void)
{
    (void)Py_InitModule("zook", zookMethods);
}

And the output?

$ python2.5 test.py
myfunc called
Redirecting stdout
But python can still print to stdout...

And redirecting to files?

$ python2.5 test.py > test.txt
$ cat test.txt
myfunc called
Redirecting stdout
But python can still print to stdout...

Wheelbase

Thanks @YinonEhrlich for the suggestion to use os.devnull instead of '/dev/null'. The edit was rejected by reviewers and I disagree with the rejection. – Wheelbase

can this approach be used to redirect to a file created from the logger module? – Enthronement

In theory, yes, but how would you know which file descriptor to replace? How would you know which object to flush? Think about it like using dynamite to open a door because you can't be bothered to find the key. – Wheelbase

That's the crux of the question .. if and how can it be done with a Logger object? – Enthronement

I don't even know what a "Logger object" is, and there is probably a larger discussion to be had here (like, what are you trying to do? rotate logs? redirect logs?) that I would lean towards asking this as its own question. – Wheelbase

Answer 4

Here is how I finally did. I hope this can be useful for other people (this works on my linux station).

I proudly present the libshutup, designed for making external libraries shut up.

1) Copy the following file

// file: shutup.c
#include <stdio.h>
#include <unistd.h>

static char buf[20];
static int saved_stdout;

void stdout_off() {
    saved_stdout = dup(1);
    freopen("/dev/null", "w", stdout);
}

void stdout_on() {
    sprintf(buf, "/dev/fd/%d", saved_stdout);
    freopen(buf, "w", stdout);
}

2) Compile it as a shared library

gcc -Wall -shared shutup.c -fPIC -o libshutup.so

3) Use it in you code like this

from ctypes import *
shutup = CDLL("libshutup.so")

shutup.stdout_off()

# Let's pretend this printf comes from the external lib
libc = CDLL("libc.so.6")
libc.printf("hello\n")

shutup.stdout_on()

Lagan

All that to replace 5 lines of Python. – Catarrhine

If you have a better answer, I'd be delighted to have it. You can run every examples above and see they all print something. For now, this is the only solution that really works for me. – Lagan

This problem is solved in the question that this one is a duplicate of. – Catarrhine

@Ignacio No it's not. Just try to run the piece of code given in my second attempt, which is exactly your advice in the post, and you'll see that. – Lagan

Nice solution, It works for my specific case. None of the solutions that I found before didn't completely work. One thing I want to do is that to return stdout as a char array so to control in Python instead of writing into "/dev/fd/%d". Is there a way to do that? I'm very new to both C & ctypes. Sorry for my ignorance if this is too obvious question. – Underfur

Have you tried the answer I accepted? Because yes, creating my own C library was working for me, but in the end the accepted answer enabled me to achieve the same in pure python, so I consider it a better approach. By the way this question is 8 years old, it's funny how it is still relevant today :-) – Lagan

@Lagan Thanks for the reply. I just tried the accepted answer and it works well. But I would like to capture the stdout into a variable instead of just redirecting, so I can use later on. Is it possible to get the contents of the redirected std_out? – Underfur

Answer 5

The top answer here is very good. However, it requires sys.stdout.close() which conflicts with Juypter, if one is using Python notebooks. There is an awesome project called Wurlitzer that solves the underlying problem with a context manager as well as being not only usable in Jupter, but also provides a native Jupyer extension.

https://github.com/minrk/wurlitzer

https://pypi.org/project/wurlitzer/

pip install wurlitzer

from wurlitzer import pipes

with pipes() as (out, err):
    call_some_c_function()

stdout = out.read()

from io import StringIO
from wurlitzer import pipes, STDOUT

out = StringIO()
with pipes(stdout=out, stderr=STDOUT):
    call_some_c_function()

stdout = out.getvalue()

from wurlitzer import sys_pipes

with sys_pipes():
    call_some_c_function()

And the most magical part: it supports Jupyter:

%load_ext wurlitzer

Swaziland

Answer 6

All previous answers did not work for me, either resulting in a Segmentation Fault or not closing the outputs properly. Copying this answer and adding an additional close statement looked like it does not seem leak any file descriptors and worked in my case:

class HideOutput(object):
    '''
    A context manager that block stdout for its scope, usage:

    with HideOutput():
        os.system('ls -l')
    '''

    def __init__(self, *args, **kw):
        sys.stdout.flush()
        self._origstdout = sys.stdout
        self._oldstdout_fno = os.dup(sys.stdout.fileno())
        self._devnull = os.open(os.devnull, os.O_WRONLY)

    def __enter__(self):
        self._newstdout = os.dup(1)
        os.dup2(self._devnull, 1)
        os.close(self._devnull)
        sys.stdout = os.fdopen(self._newstdout, 'w')

    def __exit__(self, exc_type, exc_val, exc_tb):
        sys.stdout = self._origstdout
        sys.stdout.flush()
        os.dup2(self._oldstdout_fno, 1)
        os.close(self._oldstdout_fno) # Additional close to not leak fd

99% copied from this and closing the file as mentioned in this answer.

Underdog

Answer 7

jfs's answer gives me an error so I came up with another solution based on this answer.

ValueError: I/O operation on closed file.

import contextlib

@contextlib.contextmanager
def silence_stderr():
    stderr_fd = sys.stderr.fileno()
    orig_fd = os.dup(stderr_fd)
    null_fd = os.open(os.devnull, os.O_WRONLY)
    os.dup2(null_fd, stderr_fd)
    try:
        yield
    finally:
        os.dup2(orig_fd, stderr_fd)
        os.close(orig_fd)
        os.close(null_fd)

Usage is quite simple, as expected.

with silence_stderr():
    # call python module: stderr will be silenced
    # call c/c++ library: stderr will be silenced

You can easily modify the code to silence stdout instead of stderr by a simple find-replace.

Bohemianism

Answer 8

Wouldn't you be able to do this the same as you would in Python? You'd import sys and point sys.stdout and sys.stderr to something that isn't the default sys.stdout and sys.stderr? I do this all the time in a few apps where I have to slurp up output from a library.

Fabron

No, because sys.std* are used only by Python code, whereas C libraries use FDs 0 through 2 directly. – Catarrhine

This should be a comment, not an answer. – Terribly

Actually, it was an answer but without working code, I could see how it didn't get the message across. Back in 2011, I didn't realize the amount of hand holding you had to do on StackOverflow. Regardless, not sure what you get out of picking nits and giving it a downvote...2.5 years later. – Fabron