Taking ownership of streambuf/stringbuf data

Question

I'd like an interface for writing to an automatically resizing array. One way to do this is with a generic std::ostream *.

Then consider if ostringstream is the target:

void WritePNG(ostream *out, const uint8_t *pixels);

void *WritePNGToMemory(uint8_t *pixels)
{
  ostringstream out;
  WritePng(&out, pixels);

  uint8_t *copy = new uint8_t[out.tellp()];
  memcpy(copy, out.str().c_str(), out.tellp()];
  return copy;
}

But I want to avoid the memcpy(). Is there a way to take ownership of the array in the underlying stringbuf class and return that?

I get the feeling this can't be done using standard library, since the stream buffer might not even be a contiguous array.

Anywheres

No. You're not really supposed to hang on to the value returned by c_str() at all. – Thadthaddaus

There are two subquestions here: (1) How do I extract the std::string from a std::ostringstream? Starting in C++20, you can use std::move(out).str() for that. (2) How do I extract an owning (delete[]able) char* from a std::string? You can't do that, and probably will never be able to. So for your purposes, you must either replace your void* with std::string, or replace your stringstream/stringbuf with a custom child of streambuf as indicated in the existing answers. – Wacker

Answer 1

IIRC the whole reason stringstream exists (vs strstream) was to sort out the fuzzy questions of memory ownership that would come up by giving direct buffer access. e.g. I think that change was to specifically prevent what you are asking to do.

One way or another I think you'd have to do it yourself, by overriding the stream buffer. To answer a similar question I suggested something for input streams that wound up getting quite a few upvotes. But honestly I didn't know what I was talking about then, nor now when I suggest the following:

Hacking up this link from the web for doing an "uppercasing stream buffer" to one that just echoes and gives you a reference to its buffer might give:

#include <iostream>
#include <streambuf>

class outbuf : public std::streambuf {
    std::string data;

protected:
    virtual int_type overflow (int_type c) {
        if (c != EOF)
            data.push_back(c);
        return c;
    }

public:
    std::string& get_contents() { return data; }
};

int main() {
    outbuf ob;
    std::ostream out(&ob);
    out << "some stuff";
    std::string& data = ob.get_contents();
    std::cout << data;
    return 0;
}

I'm sure it's broken in all kinds of ways. But the uppercase-buffer-authors seemed to think that overriding the overflow() method alone would let them uppercase all output to the stream, so I guess one could argue that it's enough to see all output if writing to one's own buffer.

But even so, going one character at a time seems suboptimal...and who knows what overhead you get from inheriting from streambuf in the first place. Consult your nearest C++ iostream expert for what the actual right way is. But hopefully it's proof that something of the sort is possible.

Aphra

Answer 2

If you're willing to use the old, deprecated <strstream> interface, this is fairly easy - just create a std::strstreambuf pointing at your storage, and it will work by magic. std::ostrstream even has a constructor to do this for you:

#include <iostream>
#include <strstream>

int main()
{
    char copy[32] = "";

    std::ostrstream(copy, sizeof copy) << "Hello, world!"
        << std::ends << std::flush;

    std::cout << copy << '\n';
}

With the more modern <sstream> interface, you need to access the string stream's buffer, and call pubsetbuf() to make it to point at your storage:

#include <iostream>
#include <sstream>

int main()
{
    char copy[32] = "";

    {
        std::ostringstream out{};
        out.rdbuf()->pubsetbuf(copy, sizeof copy);

        out << "Hello, world!" << std::ends << std::flush;
    }

    std::cout << copy << '\n';
}

Obviously, in both cases, you'll need a way to know in advance how much memory to allocate for copy, because you can't wait until tellp() is ready for you...

Watershed

neat trick, but my output size is unknown (hopefully less than uncompressed size). But I think you're on to something. Can you use pubsetbuf() or set some internal variable to NULL to fool streambuf into thinking there's no memory to deallocate? – Anywheres

I think the only reasonable option would be to implement a buffer yourself - inherit from std::basic_stringbuf with a std::vector as storage - read about setp for an example using std::array. Because a vector can move its contents when it's resized, you'll need to override overflow() to call setp after using push_back or any other vector method that invalidates iterators. – Watershed

Answer 3

Here's the solution I ended up using. The idea is the same as the one proposed by HostileFork - only needing to implement overflow(). But as already hinted, it has much better throughput by buffering. It also optionally supports random access (seekp(), tellp()).

class MemoryOutputStreamBuffer : public streambuf
{
public:
    MemoryOutputStreamBuffer(vector<uint8_t> &b) : buffer(b)
    {
    }
    int_type overflow(int_type c)
    {
        size_t size = this->size();   // can be > oldCapacity due to seeking past end
        size_t oldCapacity = buffer.size();

        size_t newCapacity = max(oldCapacity + 100, size * 2);
        buffer.resize(newCapacity);

        char *b = (char *)&buffer[0];
        setp(b, &b[newCapacity]);
        pbump(size);
        if (c != EOF)
        {
            buffer[size] = c;
            pbump(1);
        }
        return c;
    }
  #ifdef ALLOW_MEM_OUT_STREAM_RANDOM_ACCESS
    streampos MemoryOutputStreamBuffer::seekpos(streampos pos,
                                                ios_base::openmode which)
    {
        setp(pbase(), epptr());
        pbump(pos);
        // GCC's streambuf doesn't allow put pointer to go out of bounds or else xsputn() will have integer overflow
        // Microsoft's does allow out of bounds, so manually calling overflow() isn't needed
        if (pptr() > epptr())
            overflow(EOF);
        return pos;
    }
    // redundant, but necessary for tellp() to work
    // https://mcmap.net/q/779432/-why-does-the-standard-have-both-seekpos-and-seekoff
    streampos MemoryOutputStreamBuffer::seekoff(streamoff offset,
                                                ios_base::seekdir way,
                                                ios_base::openmode which)
    {
        streampos pos;
        switch (way)
        {
        case ios_base::beg:
            pos = offset;
            break;
        case ios_base::cur:
            pos = (pptr() - pbase()) + offset;
            break;
        case ios_base::end:
            pos = (epptr() - pbase()) + offset;
            break;
        }
        return seekpos(pos, which);
    }
#endif    
    size_t size()
    {
        return pptr() - pbase();
    }
private:
    std::vector<uint8_t> &buffer;
};

They say a good programmer is a lazy one, so here's an alternate implementation I came up with that needs even less custom code. However, there's a risk for memory leaks because it hijacks the buffer inside MyStringBuffer, but doesn't free MyStringBuffer. In practice, it doesn't leak for GCC's streambuf, which I confirmed using AddressSanitizer.

class MyStringBuffer : public stringbuf
{
public:
  uint8_t &operator[](size_t index)
  {
    uint8_t *b = (uint8_t *)pbase();
    return b[index];
  }
  size_t size()
  {
    return pptr() - pbase();
  }
};

// caller is responsible for freeing out
void Test(uint8_t *&_out, size_t &size)
{
  uint8_t dummy[sizeof(MyStringBuffer)];
  new (dummy) MyStringBuffer;  // construct MyStringBuffer using existing memory

  MyStringBuffer &buf = *(MyStringBuffer *)dummy;
  ostream out(&buf);

  out << "hello world";
  _out = &buf[0];
  size = buf.size();
}

Anywheres