Sending data to a program via stdin and ostream. (C++)

Question

I would like to send data from within my C++ program to an external pipeline, like so:

FILE* file = popen("my_prog -opt | other_prog", "w");
std::ostream fileStream = some_function(file);
fileStream << "some data";

I understand there is no simple, cross-platform way to do the second line, but is there any way to accomplish the same thing using something other than popen? I don't need to use popen, but I do need to use ostream. It would need to compile with clang and gcc at minimum, but preferably it would work with any compiler. I could also change how I handle the piping, but I don't have the source for my_prog or other_prog.

Clyte

Try to look at boost::iostreams::file_descriptor. It has very bad docs, but it can help you and it is cross-platform. – Goodwill

Is there any specific reason you need to use ostream – Dorene

Can you put all your input into fileStream before creating the process, then feed that as input to my_prog, or is some of the input not known until after the process is created? – Indistinct

@James_Parsons There is a lot of other code in my codebase that works with ostreams. – Clyte

@Indistinct I could easily tweak my program so that all of the input to the process is known at process creation time. Can you provide some code that shows how to connect the pipeline once I have done that? – Clyte

Have you considered creating your own class that implements std::ostream? I am not sure about complexity but simple not-so-optimized version, as I guess, could be completed in one day. What I am sure about is that you shouldn't just write popen in your code: use RAII class with popen in constructor and pclose in destructor. – Shrub

Would it be OK if it used an inherited class of std::basic_ostream ? std::ostream is not meant to be inherited from; all std::basic_* classes are. – Gauvin

The easiest soluton I could come up with would be to implement a custom std::streambuf that can be passed to the constructor of std::ostream. A short web search brought up this site which seems to already have code for FILE* that could easily be adjusted to work cross platform. – Maudemaudie

Answer 1

It is straight forward to create a stream buffer using a FILE* as underlying destination and create a corresponding std::ostream using such a stream buffer. It would roughly look something like this:

#include <stdio.h>
#include <streambuf>
#include <ostream>

class stdiobuf
    : public std::streambuf {
    enum { bufsize = 2048 };
    char buffer[bufsize];
    FILE* fp;
    int   (*close)(FILE*);
    int overflow(int c) {
        if (c != std::char_traits<char>::eof()) {
            *this->pptr() = std::char_traits<char>::to_char_type(c);
            this->pbump(1);
        }
        return this->sync()
            ? std::char_traits<char>::eof()
            : std::char_traits<char>::not_eof(c);
    }
    int sync() {
        std::streamsize size(this->pptr() - this->pbase());
        std::streamsize done(this->fp? fwrite(this->pbase(), 1, size, this->fp): 0);
        this->setp(this->pbase(), this->epptr());
        return size == done? 0: -1;
    }
public:
    stdiobuf(FILE* fp, int(*close)(FILE*) = fclose)
        : fp(fp)
        , close(close) {
        this->setp(this->buffer, this->buffer + (this->fp? bufsize - 1: 0));
    }
    ~stdiobuf() {
        this->sync();
        this->fp && this->close(this->fp);
    }
};
class opipestream
    : private virtual stdiobuf
    , public std::ostream {
public:
    opipestream(std::string const& pipe)
        : stdiobuf(popen(pipe.c_str(), "w"), pclose)
        , std::ios(static_cast<std::streambuf*>(this))
        , std::ostream(static_cast<std::streambuf*>(this)) {
    }
};

int main()
{
    opipestream out("/usr/bin/sed -e 's/^/xxxx /'");
    out << "Hello\n";
    out << "world\n";
}

The basic idea is that you can create a new stream by implementing a stream buffer. The implementation above should be fairly complete. The error handling when an incomplete buffer could be improved although the most likely case of an error is that the pipe was closed and there isn't really much what could be done.

Topazolite

Thank you this was exactly what I needed. – Clyte

Answer 2

My old answer did only work under Windows due to the missing std::filebuf ctor. As user4815162342 pointed out that an alternative might be using __gnu_cxx::stdio_filebuf<char>.

I patched something together that should now work with windows and linux, yet there is a chance that your platforms might not all work.

#ifdef __GNUC__
    #include <ext/stdio_sync_filebuf.h>    
    typedef __gnu_cxx::stdio_sync_filebuf<char> popen_filebuf;
#elif _MSC_VER
    #include<fstream>
    typedef std::filebuf popen_filebuf;
    FILE*(*popen)(const char*, const char*) = _popen;
#else
    static_assert(false, "popen_filebuf is not available for this platform");
#endif

int main()
{
    FILE* file = popen("my_prog -opt | other_prog", "w");

    popen_filebuf buffer(file);
    std::ostream fileStream(&buffer);

    fileStream << "some data";

    return 0;
}

Maudemaudie

The standard doesn't mandate the std::filebuf is implemented in terms of FILE* although multiple implementation do (or did) it that way. – Calamitous

As shown in an edit of my answer, g++ on Linux also supports __gnu_cxx::stdio_sync_filebuf, so you can simply typedef popen_filebuf to __gnu_cxx::stdio_sync_filebuf<char>. – Forgiving

@Forgiving That results in error: no matching function for call to '__gnu_cxx::stdio_filebuf<char>::stdio_filebuf(FILE*&)' – Maudemaudie

stdio_filebuf and stdio_sync_filebuf are different classes, and you need to use the latter (as shown in the comment) to directly construct from FILE *. – Forgiving

@Forgiving Adjusted the answer. Thanks for the hint. It's much easier now. – Maudemaudie

Answer 3

If you know in advance which platforms you support, you can use platform-specific extensions to create an iostream out of a file descriptor. For example, as shown here, GNU libstdc++ provides __gnu_cxx::stdio_sync_filebuf which can be used to create a filebuf from a FILE *. A class that inherits from std::iostream and initializes it with the appropriate filebuf, tested with g++ 5.2 and clang++ 3.7 on Linux, can look like this:

#include <iostream>
#include <ext/stdio_sync_filebuf.h>

class StdioStream:
  private __gnu_cxx::stdio_sync_filebuf<char>, public std::iostream {
public:
  explicit StdioStream(FILE *fp):
    __gnu_cxx::stdio_sync_filebuf<char>(fp), std::iostream(this) { }
};

int main()
{
    FILE* file = popen("my_prog -opt | other_prog", "w");
    StdioStream fileStream(file);
    fileStream << "some data";
}

Note that one cannot just define a some_function that returns an std::ostream because the latter has a private copy constructor, and because the intermediate filebuf needs to be destructed along with the stream. The above example uses multiple inheritance to be able to initialize stdio_sync_filebuf (which would otherwise be a member) before the iostream base class - see base-from-member for details.

The referenced answer also contains equivalent code for MSVC++ iostreams.

Forgiving

why don't you derived from std::iostream? – Idiomorphic

@Idiomorphic Good point, I've now updated the answer to inherit from iostream. The member-to-base problem requires multiple inheritance which reduces readability, but that buys simplicity of use for StdioStream, more than justifying the tradeoff. – Forgiving

@Forgiving Why not combine our answers: rextester.com/PFJNUL66673 and goo.gl/NzKDeh – Maudemaudie

@SimonKraemer You kind of already did that, by incrementally editing your original answer to take the approach from mine. Now that Dietmar has provided a tested version of his portable answer, and his answer is accepted, the point is moot anyway. – Forgiving

@Forgiving Ah - I didn't notice Dietmar's new answer. ;-) – Maudemaudie

Answer 4

An expansion on my comment to the original question.

You can save all the input into a file on disk, then feed that file as input to my_prog.

ofstream fileStream("input.txt");
fileStream << "some data";
// insert other input
fileStream.close();
FILE *file = popen("my_prog -opt <input.txt | other_prog", "w");

You could use something other than popen here, possibly a straight system call. Once you were all done, you'd want to delete the input.txt file.

Indistinct

While not portable, I've also seen ways of creating a process and using an alternative handle in place of stdin, stdout, etc., using native APIs . – Indistinct