How can I compose output streams, so output goes multiple places at once?

Question

I'd like to compose two (or more) streams into one. My goal is that any output directed to cout, cerr, and clog also be outputted into a file, along with the original stream. (For when things are logged to the console, for example. After closing, I'd like to still be able to go back and view the output.)

I was thinking of doing something like this:

class stream_compose : public streambuf, private boost::noncopyable
{
public:
    // take two streams, save them in stream_holder,
    // this set their buffers to `this`.
    stream_compose;

    // implement the streambuf interface, routing to both
    // ...

private:
    // saves the streambuf of an ios class,
    // upon destruction restores it, provides
    // accessor to saved stream
    class stream_holder;

    stream_holder mStreamA;
    stream_holder mStreamB;
};

Which seems straight-forward enough. The call in main then would be something like:

// anything that goes to cout goes to both cout and the file
stream_compose coutToFile(std::cout, theFile);
// and so on

I also looked at boost::iostreams, but didn't see anything related.

Are there any other better/simpler ways to accomplish this?

Lexicographer

I like the answer here: https://mcmap.net/q/582927/-using-freopen-to-print-to-file-and-screen – Cannibalize

Answer 1

You mention having not found anything in Boost.IOStreams. Did you consider tee_device?

Reactor

Unfortunately it is not compatible with std::ostream - you can't pass it to a function that expects an ostream. – Renferd

@Renferd It is not an ostream, but can be adapted into one using "stream" – Bistre

Answer 2

You do have the right design—if you want to do this purely within the stdlib.

One thing: instead of teeing to each streambuf on every output, implement it to use the same put area as one of the streambufs it's given, and copy to the others on overflow and sync. This will minimize virtual calls, which is one of the goals of how streambufs work.

Alternatively, and if you want to only handle stdout & stderr (which is common), run your program through the standard Unix tee program (or the equivalent on your platform), either by doing it yourself when invoking the program, or within the program by forking, setting up the streams as appropriate, etc.

Edit: You got me thinking, and I should know how to get this right. Here's my first approximation. (When this breaks, you get to keep both pieces.)

#ifndef INCLUDE_GUARD_A629F54A136C49C9938CB33EF8EDE676
#define INCLUDE_GUARD_A629F54A136C49C9938CB33EF8EDE676

#include <cassert>
#include <cstring>
#include <streambuf>
#include <map>
#include <vector>

template<class CharT, class Traits=std::char_traits<CharT> >
struct basic_streamtee : std::basic_streambuf<CharT, Traits> {
    typedef std::basic_ios<CharT, Traits> Stream;
    typedef std::basic_streambuf<CharT, Traits> StreamBuf;

    typedef typename StreamBuf::char_type char_type;
    typedef typename StreamBuf::traits_type traits_type;
    typedef typename StreamBuf::int_type int_type;
    typedef typename StreamBuf::pos_type pos_type;
    typedef typename StreamBuf::off_type off_type;

    basic_streamtee() : _key_buf(0) {}
    basic_streamtee(Stream& a, Stream& b) : _key_buf(0) {
        this->pubimbue(a.rdbuf()->getloc());
        _set_key_buf(a.rdbuf());
        insert(a);
        insert(b);
    }
    ~basic_streamtee() {
        sync();
        for (typename std::map<Stream*, StreamBuf*>::iterator i = _bufs.begin();
             i != _bufs.end();
             ++i)
        {
            StreamBuf* old = i->first->rdbuf(i->second);
            if (old != this) {
                old->pubsync();
            }
        }
    }

    // add this functionality?
    // streambufs would be unconnected with a stream
    // easy to do by changing _bufs to a multimap
    // and using null pointers for the keys
    //void insert(StreamBuf* buf);
    //void remove(StreamBuf* buf);

    void insert(Stream& s) {
        sync();
        if (!_bufs.count(&s)) {
            if (!_key_buf) {
                _set_key_buf(s.rdbuf());
            }
            _bufs[&s] = s.rdbuf(this);
        }
    }
    void remove(Stream& s) {
        sync();
        typename std::map<Stream*, StreamBuf*>::iterator i = _bufs.find(&s);
        if (i != _bufs.end()) {
            StreamBuf* old = i->second;
            i->first->rdbuf(i->second);
            _bufs.erase(i);

            if (old == _key_buf) {
                _set_key_buf(_bufs.empty() ? 0 : _bufs.begin()->second);
            }
        }
    }

private:
    basic_streamtee(basic_streamtee const&); // not defined
    basic_streamtee& operator=(basic_streamtee const&); // not defined

    StreamBuf* _key_buf;
    std::map<Stream*, StreamBuf*> _bufs;

    void _set_key_buf(StreamBuf* p) {
        //NOTE: does not sync, requires synced already
        _key_buf = p;
        _update_put_area();
    }
    void _update_put_area() {
        //NOTE: does not sync, requires synced already
        if (!_key_buf) {
            this->setp(0, 0);
        }
        else {
            this->setp((_key_buf->*&basic_streamtee::pbase)(),
                       (_key_buf->*&basic_streamtee::epptr)());
        }
    }


#define FOREACH_BUF(var) \
for (typename std::map<Stream*, StreamBuf*>::iterator var = _bufs.begin(); \
var != _bufs.end(); ++var)


    // 27.5.2.4.1 Locales
    virtual void imbue(std::locale const& loc) {
        FOREACH_BUF(iter) {
            iter->second->pubimbue(loc);
        }
    }


    // 27.5.2.4.2 Buffer management and positioning
    //virtual StreamBuf* setbuf(char_type* s, std::streamsize n); // not required
    //virtual pos_type seekoff(off_type off, std::ios_base::seekdir way,
    //                         std::ios_base::openmode which); // not required
    //virtual pos_type seekpos(pos_type sp, std::ios_base::openmode which); // not required
    virtual int sync() {
        if (!_key_buf) {
            return -1;
        }
        char_type* data = this->pbase();
        std::streamsize n = this->pptr() - data;
        (_key_buf->*&basic_streamtee::pbump)(n);
        FOREACH_BUF(iter) {
            StreamBuf* buf = iter->second;
            if (buf != _key_buf) {
                buf->sputn(data, n); //BUG: ignores put errors
                buf->pubsync(); //BUG: ignroes errors
            }
        }
        _key_buf->pubsync(); //BUG: ignores errors
        _update_put_area();
        return 0;
    }


    // 27.5.2.4.3 Get area
    // ignore input completely, teeing doesn't make sense
    //virtual std::streamsize showmanyc();
    //virtual std::streamsize xsgetn(char_type* s, std::streamsize n);
    //virtual int_type underflow();
    //virtual int_type uflow();


    // 27.5.2.4.4 Putback
    // ignore input completely, teeing doesn't make sense
    //virtual int_type pbackfail(int_type c);


    // 27.5.2.4.5 Put area
    virtual std::streamsize xsputn(char_type const* s, std::streamsize n) {
        assert(n >= 0);
        if (!_key_buf) {
            return 0;
        }

        // available room in put area? delay sync if so
        if (this->epptr() - this->pptr() < n) {
            sync();
        }
        // enough room now?
        if (this->epptr() - this->pptr() >= n) {
            std::memcpy(this->pptr(), s, n);
            this->pbump(n);
        }
        else {
            FOREACH_BUF(iter) {
                iter->second->sputn(s, n);
                //BUG: ignores put errors
            }
            _update_put_area();
        }
        return n;
    }
    virtual int_type overflow(int_type c) {
        bool const c_is_eof = traits_type::eq_int_type(c, traits_type::eof());
        int_type const success = c_is_eof ? traits_type::not_eof(c) : c;
        sync();
        if (!c_is_eof) {
            char_type cc = traits_type::to_char_type(c);
            xsputn(&cc, 1);
            //BUG: ignores put errors
        }
        return success;
    }

#undef FOREACH_BUF
};

typedef basic_streamtee<char> streamtee;
typedef basic_streamtee<wchar_t> wstreamtee;

#endif

Now, this test is far from complete, but it seems to work:

#include "streamtee.hpp"

#include <cassert>
#include <iostream>
#include <sstream>

int main() {
    using namespace std;
    {
        ostringstream a, b;
        streamtee tee(a, b);
        a << 42;
        assert(a.str() == "42");
        assert(b.str() == "42");
    }
    {
        ostringstream a, b;
        streamtee tee(cout, a);
        tee.insert(b);
        a << 42 << '\n';
        assert(a.str() == "42\n");
        assert(b.str() == "42\n");
    }
    return 0;
}

Put it together with a file:

#include "streamtee.hpp"

#include <iostream>
#include <fstream>

struct FileTee {
  FileTee(std::ostream& stream, char const* filename)
  : file(filename), buf(file, stream)
  {}

  std::ofstream file;
  streamtee buf;
};

int main() {
  using namespace std;

  FileTee out(cout, "stdout.txt");
  FileTee err(clog, "stderr.txt");
  streambuf* old_cerr = cerr.rdbuf(&err.buf);

  cout << "stdout\n";
  clog << "stderr\n";

  cerr.rdbuf(old_cerr);
  // watch exception safety

  return 0;
}

Eldridgeeldritch

So I got mine implemented as well, our code was fairly close, though I split mine into two classes (one tee's streambufs, the other tee'd streams, which used the former tee). However, Eric has pointed out that boost implements this already, so if I can get his working how I'm hoping, I'll accept his answer. But you should know this post was very helpful, I learn a lot from it. :) – Lexicographer

Nice solution! Non-broken link: bitbucket.org/kniht/scraps/src/01ecc1346bc5/cpp/kniht/… – Drambuie

Okay, so I am super late here, but what is that include guard??? What's wrong with just #ifndef FILENAME_H? – Eason

Answer 3

I would write a custom stream buffer that just forwards data to the buffers of all your linked streams.

#include <iostream>
#include <fstream>
#include <vector>
#include <algorithm>
#include <functional>

class ComposeStream: public std::ostream
{
    struct ComposeBuffer: public std::streambuf
    {
        void addBuffer(std::streambuf* buf)
        {
            bufs.push_back(buf);
        }
        virtual int overflow(int c)
        {
            // std::for_each(bufs.begin(),bufs.end(),std::bind2nd(std::mem_fun(&std::streambuf::sputc),c));

            // In C++20 we can simplify this:
            // Thanks: @nabelekt
            for (auto& buf: bufs) {
                buf->sputc(c);
            }
  
            return c;
        }

        private:
            std::vector<std::streambuf*>    bufs;
            
    };  
    ComposeBuffer myBuffer;
    public: 
        ComposeStream()
            :std::ostream(NULL)
        {
            std::ostream::rdbuf(&myBuffer);
        }   
        void linkStream(std::ostream& out)
        {
            out.flush();
            myBuffer.addBuffer(out.rdbuf());
        }
};
int main()
{
    ComposeStream   out;
    out.linkStream(std::cout);
    out << "To std::cout\n";

    out.linkStream(std::clog);
    out << "To: std::cout and std::clog\n";

    std::ofstream   file("Plop");
    out.linkStream(file);
    out << "To all three locations\n";
}

Personage

You have a few errors in there, by the way. This is similar to what I had so far, actually. The problem is I'll need to make three, because I can't have cout and cerr linked to each other. (They output to the same place, by default) – Lexicographer

compiles and runs for me. If clog and cout are going to the same place is not the problem of the ComposeStream. – Personage

Nice and simple. However, it won't allow code already using cout/clog/etc. to get their output into the composed stream. (This may certainly be a benefit, but it's not what was asked for, the way I read it.) – Eldridgeeldritch

cout and cerr are not linked to each other, unless your implementation does that. One writes to stdout and the other to stderr. clog and cerr both write to stderr though, is that what you meant? – Eldridgeeldritch

The error's are that bind2nd requires functional, and ostream needs to be initialized in the constructor with &myBuffer – Lexicographer

@GMan: No you can't construct ostream with &myBuffer. As mybuffer will not have been constructed at the point where the ostream constructor is called. Hence you must call rdbuf() to set the buffer after the ostream has been created. (Of course there are ways around this like using dynamic creation of ComposeBuffer object but that is more work than is necessary for this code). Have updated to include <functional> – Personage

Oops, didn't notice the first one. You'll have to pass it 0, since the standard doesn't give it a default value. – Lexicographer

Though I didn't end up using your or Pate's designs, this answer was still helpful, so thank you. :) – Lexicographer

This doesn't seem to flush anything to the output file until its closed? – Preen

@NPike: Most streams are buffered. Thus the three methods to flush are fill the buffer, close the stream or manually flush it. Alternatively don't use a buffered stream. See std::flush and std::endl for more information. – Personage

@Martin so should out << "Test" << std::endl; cause the buffer to be flushed to the file? Is there a quick way to make this use unbuffered streams? – Preen

Came here to say that in 2023, the std::for_each(bufs.begin(),bufs.end(),std::bind2nd(std::mem_fun(&std::streambuf::sputc),c)); line throws some -Wdeprecated-declarations warnings. That line can be replaced with for (auto buffer : bufs) buffer->sputc(c);. – Denver