Copy a streambuf's contents to a string

Question

Apparently boost::asio::async_read doesn't like strings, as the only overload of boost::asio::buffer allows me to create const_buffers, so I'm stuck with reading everything into a streambuf.
Now I want to copy the contents of the streambuf into a string, but it apparently only supports writing to char* (sgetn()), creating an istream with the streambuf and using getline().

Is there any other way to create a string with the streambufs contents without excessive copying?

Countess

Answer 1

I mostly don't like answers that say "You don't want X, you want Y instead and here's how to do Y" but in this instance I'm pretty sure I know what tstenner wanted.

In Boost 1.66, the dynamic string buffer type was added so async_read can directly resize and write to a string buffer.

Countess

Can you provide an example? The official doc does not even show how to construct one, since it's a sort of template... – Southward

Answer 2

I don't know whether it counts as "excessive copying", but you can use a stringstream:

std::ostringstream ss;
ss << someStreamBuf;
std::string s = ss.str();

Like, to read everything from stdin into a string, do

std::ostringstream ss;
ss << std::cin.rdbuf();
std::string s = ss.str();

Alternatively, you may also use a istreambuf_iterator. You will have to measure whether this or the above way is faster - i don't know.

std::string s((istreambuf_iterator<char>(someStreamBuf)), 
               istreambuf_iterator<char>());

Note that someStreamBuf above is meant to represent a streambuf*, so take its address as appropriate. Also note the additional parentheses around the first argument in the last example, so that it doesn't interpret it as a function declaration returning a string and taking an iterator and another function pointer ("most vexing parse").

Figurine

Thanks, istreambuf_iterator was what I've been looking for. – Countess

Here's something strange. I can't assume why, but istreambuf_iterator cuts last symbols (if it's more than 20 symbols in last line). Is there any ideas of why could it be? – Endearment

ss << someStreamBuf copies the 11 bytes '0xFFFFFFFFF' which represent the address of the streambuf in memory. thanks. very helpful :D – Rooks

@AlexKremer obviously someStreamBufPointer was meant – Loach

Since C++11, the "most vexing parse" problem can also be avoided by istreambuf_iterator<char>{someStreamBuf}. – Aniconic

Answer 3

It's really buried in the docs...

Given boost::asio::streambuf b, with size_t buf_size ...

boost::asio::streambuf::const_buffers_type bufs = b.data();
std::string str(boost::asio::buffers_begin(bufs),
                boost::asio::buffers_begin(bufs) + buf_size);

Pitchstone

This works well! You can use b.size() instead of buf_size if you want the whole thing put into the string. – Barthold

It is supposed that the answer of tstenner above where he says that it is considered dirty and won't work with 'normal streambufs' apply to this answer or this way is considered safe? – Marcello

Answer 4

Another possibility with boost::asio::streambuf is to use boost::asio::buffer_cast<const char*>() in conjunction with boost::asio::streambuf::data() and boost::asio::streambuf::consume() like this:

const char* header=boost::asio::buffer_cast<const char*>(readbuffer.data());
//Do stuff with header, maybe construct a std::string with std::string(header,header+length)
readbuffer.consume(length);

This won't work with normal streambufs and might be considered dirty, but it seems to be the fastest way of doing it.

Countess

where does length come from? – Henni

It's the number of bytes you took from the stream. See boost.org/doc/libs/1_53_0/doc/html/boost_asio/reference/… – Countess

What you mean with 'normal streambufs' ? The answer of Sean DeNigris bellow based on the documentation wouldn't be valid? – Marcello

Just as a note to people seeing this now, this will only work if your stream has a single buffer object allocated and populated. If the stream has more than one buffer, you're screwed this way. What you'll need to do and what you should do for this to be guaranteed to work is to get an iterator to the buffer(s) and then cast the iterators. You can get these iterators by accessing buffer->data().begin(). – Pogey

I could not find any infromation on if and how boost::asio::steambuf allocates multiple buffer objects. Could you tell me what the requierements are for streambuf to return more than one buffer. – Rame

@Rame You gotta tag people! I didn't see your comment, and I actually came back here and found my own comment because I forgot about this gotcha lol. Anyway the streambuf internally will allocate multiple internal buffers which the single streambuf spans. To be safe, you absolutely must use the iterators. Using a straight up cast like this will give you a pointer that will lead to undefined behavior, whether it's that the ordering of data is messed up, or the length you use to read from the pointer is out of bounds. – Pogey

Answer 5

For boost::asio::streambuf you may find a solution like this:

    boost::asio::streambuf buf;
    /*put data into buf*/

    std::istream is(&buf);
    std::string line;
    std::getline(is, line);

Print out the string :

    std::cout << line << std::endl;

You may find here: http://www.boost.org/doc/libs/1_49_0/doc/html/boost_asio/reference/async_read_until/overload3.html

Maracanda

This answer is definitely better if you're using async_read_until because that call can return more than one line of data, and this answer correctly leaves the extra data in the streambuf. – Peduncle

Answer 6

One can also obtain the characters from asio::streambuf using std::basic_streambuf::sgetn:

asio::streambuf in;
// ...
char cbuf[in.size()+1]; int rc = in.sgetn (cbuf, sizeof cbuf); cbuf[rc] = 0;
std::string str (cbuf, rc);

Dianadiandra

Answer 7

The reason you can only create const_buffer from std::string is because std::string explicitly doesn't support direct pointer-based writing in its contract. You could do something evil like resize your string to a certain size, then const_cast the constness from c_str() and treat it like a raw char* buffer, but that's very naughty and will get you in trouble someday.

I use std::vector for my buffers because as long as the vector doesn't resize (or you are careful to deal with resizing), you can do direct pointer writing just fine. If I need some of the data as a std::string, I have to copy it out, but the way I deal with my read buffers, anything that needs to last beyond the read callback needs to be copied out regardless.

Justino

Answer 8

I didn't see an existing answer for reading exactly n chars into a std::stringstream, so here is how that can be done:

std::stringstream ss;
boost::asio::streambuf sb;
const auto len = 10;

std::copy_n(boost::asio::buffers_begin(sb.data()), len,
            std::ostream_iterator<decltype(ss)::char_type>(ss));

Compiler explorer

Kneehigh

Answer 9

A simpler answer would be to convert it in std::string and manipulate it some what like this

 std::string buffer_to_string(const boost::asio::streambuf &buffer)
 {
  using boost::asio::buffers_begin;
  auto bufs = buffer.data();
  std::string result(buffers_begin(bufs), buffers_begin(bufs) + buffer.size());
 return result;
}

Giving a very concise code for the task.

Doelling

This is just the same answer as Sean DeNigris... which he posted 4 years earlier! – Salomo

Answer 10

I tested the first answer and got a compiler error when compiling using "g++ -std=c++11" What worked for me was:

        #include <string>
        #include <boost/asio.hpp>
        #include <sstream>           
        //other code ...
        boost::asio::streambuf response;
        //more code
        std::ostringstream sline;
        sline << &response; //need '&' or you a compiler error
        std::string line = sline.str(); 

This compiled and ran.

Threshold

Answer 11

I think it's more like:

streambuf.commit( number_of_bytes_read );

istream istr( &streambuf );
string s;
istr >> s;

I haven't looked into the basic_streambuf code, but I believe that should be just one copy into the string.

Maley

operator>>(istream&,string&) copies only up to the first whitespace – Countess

@Countess There is flag to disable that behavior, as well as other flags add different behaviors. – Patricia

This solution only reads until the first whitespace character is reached, hence the downmark – Permanency