Using std::endl on an ostream makes my file binary

Question

I am working on a project that uses libzip. I'm working in c++14 and I wrote a tiny wrapper around libzip to make my life easier.

I have an std::ostream object built around custom class that inherits std::streambuf. This streambuf uses the libzip functions to write in a file in the archive.

Everything works great until I use std::endl. When I do this, the output file is read as binary by all my text readers (only writing strings).

My text reader detect that its binary because in the place I used std::endl there a NUL byte, any file with a NUL byte inside of it is seen as binary.

So my question is : Is this normal ? Is there a way for me to use std::endl ?

My code (extracted so it may not be exactly the same).

source.hpp

// my attributes
std::unique_ptr<zip_source_t, std::function<void(zip_source_t*)>> _source;
std::unique_ptr<std::ostream> _stream;
std::unique_ptr<_ZipBuffer> _buffer;

class _ZipBuffer : public std::streambuf {
    private:
        zip_source_t* _source;

        std::streamsize xsputn(char const* s, std::streamsize n) override;
        int overflow(int c) override;

    public:
        _ZipBuffer(zip_source_t* file);
};

source.cpp

// create the streambuf and send it to the ostream
_buffer.reset(new _ZipBuffer(_source.get()));
_stream.reset(new std::ostream(_buffer.get()));

// the implementation of _ZipBuffer
Zip::Source::_ZipBuffer::_ZipBuffer(zip_source_t* source) {
    _source = source;
}

std::streamsize Zip::Source::_ZipBuffer::xsputn(char const* s, std::streamsize n) {
    return zip_source_write(_source, s, n * sizeof(char));
}

int Zip::Source::_ZipBuffer::overflow(int c) {
    return zip_source_write(_source, &c, sizeof(int));
}

main.cpp

Zip::Source src;

src << "Some text and a number : " << 2.5 << std::endl;
src << "another line !";

// zip is an object of class Zip that takes my source and write it in the archive
zip.addFile("test.txt", src);

If I remove the std::endl in my main, the text file is recognized as a text file. If I add it, it it recognized as binary file.

The binary file is a valid utf-8 output (except for the NUL byte) :

496c 2065 7374 2070 6f73 7369 626c 6520
6427 c3a9 6372 6972 6520 6465 7320 6e6f
6d62 7265 7320 c3a0 2076 6972 6775 6c65
203a 2032 2e35 0a00 0000 736f 6d65 7468
696e 6720 656c 7365 

Thanks!

Outflow

Why is this a problem? Emacs seems to read such a file just fine. – Brussels

Even though overflow takes an int, it still represents one character. You are supposed to write one character's worth of data, not sizeof(int) worth of data. – Levitation

It's a problem because the NUL byte should not exist, I didn't knew where it was coming from. But I found the problem and added an answer. – Outflow

@Outflow just as a side note, this question was really good, and besides one formatting mistake, it seems to be that you tried really hard to ask a good question. Keep up the good work! (Just for you other people, it doesn't seem that good newbies get recognition as to the fact they asked a good question, only lack of correction. So I thought I would add this.) – Haberman

This isn't the problem, but names that begin with an underscore followed by a capital letter (_ZipBuffer) and names that contain two consecutive underscores are reserved for use by the implementation. Don't use them in your code. – Spectatress

Answer 1

You implemented overflow() as follows:

int Zip::Source::_ZipBuffer::overflow(int c) {
   return zip_source_write(_source, &c, sizeof(int));
}

Your C++ library apparently implements std::endl, by calling overflow() with '\n' getting passed as a parameter.

This is perfectly compliant with C++ spec. Your implementation of overflow() has a bug.

The parameter to overflow() is a single character, passed as an int. Your implementation writes an entire binary int to the output file, which is exactly what you're seeing. Your sizeof(int) is, apparently, 4, so you see 0x0a and three more null bytes written to the output.

Purpurin

Answer 2

Ok, just found the problem...

In the std::streambuf::overflow overload I wrote, I'm writing an integer when I'm receiving a char. So the leading 0 in my int was written in the file.

I had to cast my int to char and the problem disappeared :)

Outflow

A cast sounds like the wrong solution. In Zip::Source::_ZipBuffer::overflow, just write 1 byte instead of sizeof(int). – Spectatress