Get an istream from a char*

Question

I have a char* and the data length that I'm receiving from a library, and I need to pass the data to a function that takes an istream.

I know I can create a stringstream but that will copy all the data. And also, the data will surely have 0s since it's a zip file, and creating a stringstream will take the data until the first 0 I think.

Is there any way to create an istream from a char* and it's size without copying all the data?

Vidavidal

"will take the data until the first 0 I think." Why do you think that? – Oration

@FailedDev: Because constructing an std::string from a char* will stop at the first 0. However, you can get around this with the iterator range constructor, as in: std::string(buffer, buffer + buffer_size), this doesn't get around the copying problem though. – Concoff

@FailedDev: That's the character '0', usually equivalent to it's ASCII value 48. It's not the integer value 0, which would be represented in a string by '\0'. Your example, revised: ideone.com/UpPSf – Concoff

@BenjaminLindley I didn't realize the op was talking about null terminating chars.. – Oration

Answer 1

Here's a non-deprecated method found on the web, has you derive your own std::streambuf class, but easy and seems to work:

#include <iostream>
#include <istream>
#include <streambuf>
#include <string>

struct membuf : std::streambuf
{
    membuf(char* begin, char* end) {
        this->setg(begin, begin, end);
    }
};

int main()
{
    char buffer[] = "I'm a buffer with embedded nulls\0and line\n feeds";

    membuf sbuf(buffer, buffer + sizeof(buffer));
    std::istream in(&sbuf);
    std::string line;
    while (std::getline(in, line)) {
        std::cout << "line: " << line << "\n";
    }
    return 0;
}

Which outputs:

line: I'm a buffer with embedded nullsand line
line:  feeds

Glop

Nice, simple solution (I am not the downvoter). But be aware that tellg and seekg on the iostream will not work unless you implement seekoff and seekpos in the streambuf. You might or might not want to provide setbuf as well. – Reexamine

There's an excellent blog post here (see Example 2) that explains this in detail. – Whiting

Is there a similar approach to get a char* from an ostream? – Mistrial

See #13059591 for a slightly improved version that works with const – League

what about output buffer? – Miguelinamiguelita

foo(char* buf, int len) { membuf sbuf(buf, buf + len); … } if you're given a char* and length – Glide

Answer 2

A non deprecated solution using Boost:

#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/device/array.hpp>
using namespace boost::iostreams;

basic_array_source<char> input_source(my_ptr_to_char, byte_count);
stream<basic_array_source<char> > input_stream(input_source);

or even simpler:

#include <boost/interprocess/streams/bufferstream.hpp>
using namespace boost::interprocess;

bufferstream input_stream(my_ptr_to_char, byte_count);

Faux

Answer 3

The only (simple) portable way includes making the copy:

std::istringstream ss(std::string(buf,len));

In fact, this is likely to copy the data twice, once to create the string and once to create the istringstream. (Maybe C++11 can avoid one of the copies via a move constructor; I am not sure.)

However, if you are lucky, your C++ implementation will let you do this:

std::istringstream ss;
ss.rdbuf()->pubsetbuf(buf,len);

Under GNU C++ (and, I believe, some other implementations), this will create the stringstream without copying the data. But this is "implementation-defined" behavior according to the spec. (See also this question.)

By including the len parameter, you ensure that both of these will have no problem with null characters.

The only portable way to do what you want is to implement your own subclass of stringbuf and use it to initialize the stringstream. Not for the faint of heart.

Reexamine

So istringstream copies the input string instead of its reference? P.S. if you implement your own streambuf then you don't need istringstream; istream will work. – Benil

Answer 4

I needed a solution that supports tellg and seekg and didn't require boost.

char_array_buffer from A beginner's guide to writing a custom stream buffer (std::streambuf) gave a got starting point.

byte_array_buffer.h:

#include <cstdio>
#include <string>
#include <list>
#include <fstream>
#include <iostream>

//
// http://www.mr-edd.co.uk/blog/beginners_guide_streambuf
//

class byte_array_buffer : public std::streambuf
{
public:
    byte_array_buffer(const uint8_t *begin, const size_t size);

private:
    int_type underflow();
    int_type uflow();
    int_type pbackfail(int_type ch);
    std::streamsize showmanyc();
    std::streampos seekoff ( std::streamoff off, std::ios_base::seekdir way,
                            std::ios_base::openmode which = std::ios_base::in | std::ios_base::out );
    std::streampos seekpos ( std::streampos sp,
                            std::ios_base::openmode which = std::ios_base::in | std::ios_base::out);

    // copy ctor and assignment not implemented;
    // copying not allowed
    byte_array_buffer(const byte_array_buffer &);
    byte_array_buffer &operator= (const byte_array_buffer &);

private:
    const uint8_t * const begin_;
    const uint8_t * const end_;
    const uint8_t * current_;
};

byte_array_buffer.cpp:

#include "byte_array_buffer.h"

#include <cassert>


byte_array_buffer::byte_array_buffer(const uint8_t *begin, const size_t size) :
begin_(begin),
end_(begin + size),
current_(begin_)
{
    assert(std::less_equal<const uint8_t *>()(begin_, end_));
}

byte_array_buffer::int_type byte_array_buffer::underflow()
{
    if (current_ == end_)
        return traits_type::eof();

    return traits_type::to_int_type(*current_);
}

byte_array_buffer::int_type byte_array_buffer::uflow()
{
    if (current_ == end_)
        return traits_type::eof();

    return traits_type::to_int_type(*current_++);
}

byte_array_buffer::int_type byte_array_buffer::pbackfail(int_type ch)
{
    if (current_ == begin_ || (ch != traits_type::eof() && ch != current_[-1]))
        return traits_type::eof();

    return traits_type::to_int_type(*--current_);
}

std::streamsize byte_array_buffer::showmanyc()
{
    assert(std::less_equal<const uint8_t *>()(current_, end_));
    return end_ - current_;
}


std::streampos byte_array_buffer::seekoff ( std::streamoff off, std::ios_base::seekdir way,
                                           std::ios_base::openmode which )
{
    if (way == std::ios_base::beg)
    {
        current_ = begin_ + off;
    }
    else if (way == std::ios_base::cur)
    {
        current_ += off;
    }
    else if (way == std::ios_base::end)
    {
        current_ = end_ + off;
    }

    if (current_ < begin_ || current_ > end_)
        return -1;


    return current_ - begin_;
}

std::streampos byte_array_buffer::seekpos ( std::streampos sp,
                                           std::ios_base::openmode which )
{
    current_ = begin_ + sp;

    if (current_ < begin_ || current_ > end_)
        return -1;

    return current_ - begin_;
}

Arwood

That seems to be wrong (in seekof()): else if (way == std::ios_base::end) { current_ = end_; } I think it needs to be: else if (way == std::ios_base::end) { current_ = end_ + off; } ` – Merchandising

Thx Marco. Could explain this in more detail? (Sorry it's really early in the morning and I haven't touched that code in a year). – Arwood

Did that code work for you? I'm trying to do exactly the same thing, the documentation of streambuf to me looks as it might be more complicated than that. In regards to why: parameter specification for seekoff says: off := Offset value, relative to the way parameter. In other words, seekoff( offset, end, ... ) shall position the _current pointer RELATIVE to end, not to end. seekoff( -5, end, ... ) shall position 5 elements before the end, for example ... – Merchandising

Yes it is working for me. But it is possible that it is not covering all cases ... a test case would help but I don't have the time to write one. – Arwood

@catlan: I think Marco Freudenberger is correct - end means seek relative to the end. I.e. end + offset. – Circumambient

@Arwood I confirm what Marco and Timmmm are saying. Confirmed from std::istringstream behavior with test case: std::istringstream stream("Hello"); stream.seekg(-1, ios_base::end); stream.read(&ch, 1); char ch; assert(ch == 'o'); assert(stream.good());. Your code would fail in same situation. – Cottle

Also I'm wondering if assigning current_ before checking for error is actually correct or result should be to kept in temporary variable first to keep the internal state clean. – Cottle

Answer 5

An extension to the accepted answer that supports tellg and seekg:

struct membuf : std::streambuf
{
    membuf(char* begin, char* end)
    {
        this->setg(begin, begin, end);
    }

    pos_type seekoff(off_type off, std::ios_base::seekdir dir, std::ios_base::openmode which = std::ios_base::in) override
    {
        if (dir == std::ios_base::cur)
            gbump(off);
        else if (dir == std::ios_base::end)
            setg(eback(), egptr() + off, egptr());
        else if (dir == std::ios_base::beg)
            setg(eback(), eback() + off, egptr());
        return gptr() - eback();
    }

    pos_type seekpos(pos_type sp, std::ios_base::openmode which) override
    {
        return seekoff(sp - pos_type(off_type(0)), std::ios_base::beg, which);
    }
};

Usage of this class stays the same.

Jeri

Answer 6

Have you tried std::istrstream? http://stdcxx.apache.org/doc/stdlibref/istrstream.html

Technically, I think that it is deprecated, but still part of the standard.

Soilure

It's been deprecated since 1998, and it's highly unlikely to be removed any time soon (since it addresses the OP's common problem), but it causes annoying "deprecation" warnings on pretty much all compilers these days, which is a good enough reason to want something else. – Jensen

With C++23 you may use std::spanstream, which is practically the same but not deprecated – Astro

Answer 7

Try the Boost.Iostreams array source and sink classes.

http://www.boost.org/doc/libs/1_47_0/libs/iostreams/doc/index.html

Concoff