Why does `<< std::endl` not call the operator I want it to call?

Question

I was looking for a solution to write to a file and the console at the same time. I found a nice solution here.

As I am working pre C++11 I had to make a small change to the code from Lightness Races in Orbit:

#include <iostream>
#include <fstream>
#include <string>

struct OutputAndConsole : std::ofstream
{
    OutputAndConsole(const std::string& fileName)
       : std::ofstream(fileName.c_str())   // constructor taking a string is C++11
       , fileName(fileName)
    {};

    const std::string fileName;
};

template <typename T>
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var)
{
    std::cout << var;    
    static_cast<std::ofstream&>(strm) << var;
    return strm;
};

It works nicely apart from a small thing takes puzzles me. If I use it like this:

int main(){
    OutputAndConsole oac("testLog.dat");
    double x = 5.0;
    oac << std::endl;
    static_cast<OutputAndConsole&>(oac << "foo \n" << x << "foo").operator<<(std::endl);  
    oac << "foo" << std::endl;
}

then all the std::endl are ignored for the output on the console while they appear correctly in the file. My guess is that when I use std::endl the ostream::operator<< is called which will print to the file but not to the console. The line with the static_cast<OutputAndConsole&> is my dilettantish attempt to call the correct operator, but still only the line break from \n appears on the console.

Why for std::endl the wrong operator is called?

How can I call the correct one?

PS: I know that I can use \n without problems, but still I would like to know what is going on here and how to fix it.

Opinionated

If you use '\n' to end a line you won't have this problem. Do you really need the extra stuff that std::endl does? – Wally

@PeteBecker you are absolutely right, and I dont need it, but still I would like to understand what is going wrong and how to fix it. – Opinionated

Stop. ostream already have operators which handle those. Let it do its job and handle formatted output. You just need to duplicate raw output over two other streams. It is a job for streambuf class. Just implement it properly and redirect all operatons to other two buffers: Dr.Dobb's article, another link – Pastoral

For your PPS, you have problem with overload priority. – Ensheathe

@Pastoral in that case you should write an answer to this question. Actually I had my doubts if it is a good idea to post Lightness code here, but it was the best solution I could find and there was only this small thing that I didnt understand – Opinionated

@Pastoral actually I was looking for a solution to write to console and file at the same time and the best I could find was the question i mentioned in the last comment. Just now I realized that this question specifically asks for inheriting from ofstream. Maybe I should ask a new question about writing to console and file at the same time, if there is no dupe that I missed. – Opinionated

@tobi303: And in the linked question, there is an other answer which is similar to given links – Ensheathe

@Ensheathe hm ok thanks I will take a look at it. To be honest I just picked to one that looked more simple to me. – Opinionated

PPS should be a comment on that answer, and PPPS should not be here at all. – Specie

@Specie probably you are right. edited – Opinionated

Answer 1

struct OutputAndConsole : std::ofstream
{
  // ...
};

template <typename T>
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var);

As others have mentioned, std::endl is a template function. A template function is not a value, it is just a name.

A template function can be converted to a value if you try to pass it to a function expecting function of a compatible signature. It is not converted to a value if passed to a template function taking T or const T&, because the name of a template function represents a whole host of possible values.

Because std::endl is not a valid argument for your custom-written operator<<, it looks elsewhere. It finds the std::ofstream's operator<< that takes an io manipulator function by explicit function pointer.

That one works, it can convert endl to that function pointer type! So, gleefully, it calls it.

To fix this problem, add an operator<< overload to OutputAndConsole that takes io manipulator function pointers.

The easiest way to do that is to write a helper function:

template <class T>
void output_to(OutputAndConsole& strm, const T& var)
{
  std::cout << var;    
  static_cast<std::ofstream&>(strm) << var;
};

then two << overloads:

template<class T>
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var) {
  output_to(strm, var);
  return strm;
}
OutputAndConsole& operator<<(OutputAndConsole& strm, std::ostream& (*var)(std::ostream&)) {
  output_to(strm, var);
  return strm;
}

which causes the std::endl template to find a matching <<.

Germanophile

first answer that answers both, why and how, thus I will accept it (honestly I dont care for who was first or who gets the rep :P). Actually it was my mistake to have two questions in one...sorry folks for that – Opinionated

Answer 2

Let us try something simpler:

#include <iostream>

struct Foo { };

template <typename T>
Foo& operator<<(Foo& foo, const T& var)
{
    std::cout << var;
    return foo;
};

int main(){
    Foo foo;
    foo << std::endl;
}

This does not compile:

a1.cpp: In function ‘int main()’:
a1.cpp:14:9: error: no match for ‘operator<<’ (operand types are ‘Foo’ and ‘<unresolved overloaded function type>’)
     foo << std::endl;
         ^
a1.cpp:14:9: note: candidates are:
a1.cpp:6:6: note: template<class T> Foo& operator<<(Foo&, const T&)
 Foo& operator<<(Foo& foo, const T& var)
      ^
a1.cpp:6:6: note:   template argument deduction/substitution failed:
a1.cpp:14:17: note:   couldn't deduce template parameter ‘T’

Why? What does the compiler try to tell us?

The definition of std::endl can be found here: http://en.cppreference.com/w/cpp/io/manip/endl

template< class CharT, class Traits >
std::basic_ostream<CharT, Traits>& endl( std::basic_ostream<CharT, Traits>& os );

So std::endl is not a variable. It is a template. More precisely, a template function. In my little code, the compiler is unable to instantiate the template.

When we directly call std::cout << std::endl; , the compiler instantiates std::endl from CharT and Traits of decltype(std::cout) .

In your code, the compiler instead instantiates the template using CharT and Traits from std::ofstream, because your OutputAndConsole is a descendant of std::ofstream. When std::cout tries to output the wrong instantiation of std::endl, it fails.

PS: The last paragraph is only partially correct. When you write

oac << something;

where something has type T,

It can, theoretically, call either of two

std::ofstream& std::ofstream::operator<<(T)  // or operator<<(const T&)
// -- or --
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var);

The first definition is possible because OutputAndConsole inherited the member function operator<< from std::ofstream. The second form is provided by you.

When something is a variable, it uses the second definition.

When something is a template, it cannot use the second definition, as there is no way to determine the parameters of the template. So it uses the first definition. Therefore,

oac << std::endl;  // std::endl is a template

is equivalent to

static_cast<ofstream&>(oac) << std::endl;

We can see it by the following code:

#include <iostream>

struct Foo : std::ofstream {};

template <typename T>
Foo& operator<<(Foo& strm, const T& var)
{
    std::cout << "X" << std::endl;
    return strm;
};

int main() {
    Foo oac;
    oac << std::endl;
}

This code does NOT print "X".

Parvati

I will have to think about it, but it sounds like a good explanation – Opinionated

Answer 3

std::endl is a function, not a string. Your overloaded method takes a string for the overloading, so it isnt this one who get called when you do << std::endl

You need to create an operator who takes a function who has the same signature as std:endl to do your overload.

 std::ostream& operator<<( std::ostream& (*f)(std::ostream&) )

Incurious

I dont really understand, my overloaded operator takes a const T& not a string – Opinionated

@tobi303 Yeah, const T, not string. Still, you need to give him the exact same signature when it comes to std:endl. i've edited my answer with the signature of std::endl used in the operator. – Incurious

OK thanks that made it work, but I still dont understand why this was not covered by the template. I also tried T instead of const T& and then I would expect that std::ostream& (*)(std::ostream&) is a valid template parameter, no? – Opinionated

@tobi303 const T& is a variable in your case. what the operator was expecting was a function pointer, a delegate, a method signature. When you declare a function to take another function as a parameter rather than a variable, you can't use the same syntax, since you must tell how the signature of said function is formed. – Incurious

sorry still dont understand. Maybe I am just a bit too confused atm... If I have a template that takes a parameter T i can pass whatever I want, as long as the template is well formed. Why cant I pass a function (or why template parameter deduction fails in that case)? – Opinionated

Also why does it work for the file but not for cout? The first paragraph of your answer is wrong and this doesn't really explain why this is happening. Yes it fixes it but it still seems like something is missing. – Stephine

I added a PPS on the question because it was too long for a comment – Opinionated

@Stephine it works for the file because for std::endl the "normal" operator<< is called. Note that the struct inherits from ofstream to write to the file. At least this is my theory ;) – Opinionated

Answer 4

To make it work, I would create my own set of manipulators:

struct ManipEndl {};
const ManipEndl manip_endl;
OutputAndConsole& operator<<(OutputAndConsole& strm, const ManipEndl& foo)
{
    std::cout << std::endl;    
    static_cast<std::ofstream&>(strm) << '\n'; // no need for std::endl here
    return strm;
};

int main(){
    OutputAndConsole oac("testLog.dat");
    double x = 5.0;
    oac << manip_endl;
    oac << x << manip_endl << "foo" << manip_endl;  
    oac << "foo" << manip_endl;
}

Parvati

I answered why it is get called in my initial answer. This answer is how to fix it. Note that std::endl and line break are two different things. x << std::endl is equivalent to x << '\n' << std::flush – Parvati

Answer 5

I suggest not to implement standard I/O stream functionality via the stream-interface, but the streambuffer-interface. A customized stream-interface leads usually to trouble as soon as the stream is recognized as a std::istream/std::ostream, only (due to some operator, manipulator or function taking a reference to a stream).

You may utilize:

#include <array>
#include <iostream>
#include <sstream>
#include <vector>

// BasicMultiStreamBuffer
// ============================================================================

/// A (string) stream buffer for synchronizing writes into multiple attached buffers.
template<class Char, class Traits = std::char_traits<Char>, class Allocator = std::allocator<Char> >
class BasicMultiStreamBuffer : public std::basic_stringbuf<Char, Traits, Allocator>
{
    // Types
    // =====

    private:
    typedef typename std::basic_stringbuf<Char, Traits> Base;

    public:
    typedef typename std::basic_streambuf<Char, Traits> buffer_type;
    typedef typename buffer_type::char_type char_type;
    typedef typename buffer_type::traits_type traits_type;
    typedef typename buffer_type::int_type int_type;
    typedef typename buffer_type::pos_type pos_type;
    typedef typename buffer_type::off_type off_type;

    private:
    typedef typename std::vector<buffer_type*> container_type;

    public:
    typedef typename container_type::size_type size_type;
    typedef typename container_type::value_type value_type;
    typedef typename container_type::reference reference;
    typedef typename container_type::const_reference const_reference;
    typedef typename container_type::iterator iterator;
    typedef typename container_type::const_iterator const_iterator;


    // Construction/Destructiion
    // =========================

    public:
    BasicMultiStreamBuffer()
    {}

    template <typename...Buffers>
    BasicMultiStreamBuffer(Buffers* ...buffers) {
        std::array<buffer_type*, sizeof...(Buffers)> buffer_array{buffers...};
        m_buffers.reserve(buffer_array.size());
        for(auto b : buffer_array) {
            if(b)
                m_buffers.push_back(b);
        }
    }

    template <typename Iterator>
    BasicMultiStreamBuffer(Iterator first, Iterator last)
    :   m_buffers(first, last)
    {}

    ~BasicMultiStreamBuffer() {
        sync();
    }


    private:
    BasicMultiStreamBuffer(BasicMultiStreamBuffer const&); // No Copy.
    BasicMultiStreamBuffer& operator=(BasicMultiStreamBuffer const&); // No Copy.


    // Capacity
    // ========

    public:
    bool empty() const { return m_buffers.empty(); }
    size_type size() const { return m_buffers.size(); }


    // Iterator
    // ========

    public:
    iterator begin() { return m_buffers.begin(); }
    const_iterator begin() const { return m_buffers.end(); }
    iterator end() { return m_buffers.end(); }
    const_iterator end() const { return m_buffers.end(); }


    // Modifiers
    // =========

    public:
    /// Attach a buffer.
    void insert(buffer_type* buffer) {
        if(buffer) m_buffers.push_back(buffer);
    }

    /// Synchronize and detach a buffer.
    void erase(buffer_type* buffer) {
        iterator pos = this->begin();
        for( ; pos != this->end(); ++pos) {
            if(*pos == buffer) {
                char_type* p = this->pbase();
                std::streamsize n = this->pptr() - p;
                if(n)
                    sync_buffer(*pos, p, n);
                m_buffers.erase(pos);
                break;
            }
        }
    }


    // Synchronization
    // ===============

    private:
    int sync_buffer(buffer_type* buffer, char_type* p, std::streamsize n) {
        int result = 0;
        std::streamoff offset = 0;
        while(offset < n) {
            int k = buffer->sputn(p + offset, n - offset);
            if(0 <= k) offset += k;
            else {
                result = -1;
                break;
            }
            if(buffer->pubsync() == -1)
                result = -1;
        }
        return result;
    }

    protected:
    /// Synchronize with the attached buffers.
    /// \ATTENTION If an attached buffer fails to synchronize, it gets detached.
    virtual int sync() override {
        int result = 0;
        if( ! m_buffers.empty()) {
            char_type* p = this->pbase();
            std::streamsize n = this->pptr() - p;
            if(n) {
                iterator pos = m_buffers.begin();
                while(pos != m_buffers.end()) {
                    if(0 <= sync_buffer(*pos, p, n)) ++pos;
                    else {
                        pos = m_buffers.erase(pos);
                        result = -1;
                    }
                }
            }
        }
        this->setp(this->pbase(), this->epptr());
        if(Base::sync() == -1)
            result = -1;
        return result;
    }

    private:
    container_type m_buffers;
};

typedef BasicMultiStreamBuffer<char> OStreamBuffers;


// BasicMultiStream
// ============================================================================

template<class Char, class Traits = std::char_traits<Char>, class Allocator = std::allocator<Char> >
class BasicMultiStream : public std::basic_ostream<Char, Traits>
{
    // Types
    // =====

    private:
    typedef std::basic_ostream<Char, Traits> Base;

    public:
    typedef BasicMultiStreamBuffer<Char, Traits, Allocator> multi_buffer;
    typedef std::basic_ostream<Char, Traits> stream_type;

    typedef typename multi_buffer::buffer_type buffer_type;
    typedef typename multi_buffer::char_type char_type;
    typedef typename multi_buffer::traits_type traits_type;
    typedef typename multi_buffer::int_type int_type;
    typedef typename multi_buffer::pos_type pos_type;
    typedef typename multi_buffer::off_type off_type;

    typedef typename multi_buffer::size_type size_type;
    typedef typename multi_buffer::value_type value_type;
    typedef typename multi_buffer::reference reference;
    typedef typename multi_buffer::const_reference const_reference;
    typedef typename multi_buffer::iterator iterator;
    typedef typename multi_buffer::const_iterator const_iterator;


    // Construction
    // ============

    public:
    BasicMultiStream()
    :   Base(&m_buffer)
    {}

    template <typename ...Streams>
    BasicMultiStream(Streams& ...streams)
    :   Base(&m_buffer), m_buffer(streams.rdbuf()...)
    {}

    private:
    BasicMultiStream(const BasicMultiStream&); // No copy.
    const BasicMultiStream& operator = (const BasicMultiStream&); // No copy.

    // Capacity
    // ========

    public:
    bool empty() const { return m_buffer.empty(); }
    size_type size() const { return m_buffer.size(); }


    // Iterator
    // ========

    public:
    iterator begin() { return m_buffer.begin(); }
    const_iterator begin() const { return m_buffer.end(); }
    iterator end() { return m_buffer.end(); }
    const_iterator end() const { return m_buffer.end(); }


    // Modifiers
    // =========

    public:
    template <typename StreamIterator>
    void insert(StreamIterator& first, StreamIterator& last)
    {
        while(first != last)
            insert(*first++);
    }
    void insert(stream_type& stream) { m_buffer.insert(stream.rdbuf()); }
    void erase(stream_type& stream) { m_buffer.erase(stream.rdbuf()); }

    private:
    multi_buffer m_buffer;
};

typedef BasicMultiStream<char> MultiStream;


int main() {
    MultiStream s(std::cout, std::cerr, std::clog);
    s << "Hello World" << std::endl;
    printf("[Three lines of output]\n");
}

Note, the only function applying changes to the std:: basic_streambuf interface is virtual int sync() override.

The standard basic stream classes do not provide any interface besides deriving and initializing a custom stream class. The actual (virtual) interface is the standard stream buffer basic_streambuf.

Tallboy

Thanks for sharing. However, I think this would be better as an asnwer to a question asking how to write to console and file at the same time, while this one was specifically about the overload in the code and why it does not work for std::endl. Once I have time I will make a new question and you may move this asnwer. – Opinionated

Answer 6

I had a similar problem, and fixed it by making my operator<< function a friend like this:

struct OutputAndConsole : std::ofstream
{
    OutputAndConsole(const std::string& fileName)
       : std::ofstream(fileName.c_str())   // constructor taking a string is C++11
       , fileName(fileName)
    {};

    const std::string fileName;

    template <typename T>
    friend OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var)
    {
        std::cout << var;    
        std::ofstream::operator << (var); //*See note at end
        return strm;
    }
};

* This is the preferred syntax for calling a base class method/operator from a derived class method/operator. While your syntax will work in most cases, it would fail if std::ofstream operator << were declared pure-virtual in ofstream.

Moldboard

this has the same issue with std::endl as my code, no? – Opinionated