A custom ostream

Question

I need some guidance or pointers understanding how to implement a custom ostream. My requirements are:

1. A class with a '<<' operator for several data types.
2. The intention is to send output to database. Each "line" should go to a separate record.
3. Each record most important field would be the text (or blob), but some other fields such as time, etc. can be mostly deduced automatically
4. buffering is important, as I don't want to go to database for every record.

First, does it worth deriving from ostream? What do I get by deriving from ostream? What if my class simply implements few operator<< methods (including some custom data types). Which functionality do I get from ostream?

Assuming what I want is a class derived from ostream, I need some guidance understanding the relationship between the ostream and the streambuf classes. Which one do I need to implement? Looking at some samples, it appears that I don't need to derive from ostream at all, and just give the ostream constructor a custom streambuf. Is that true? is that the canonical approach?

Which virtual functions at the custom streambuf do i need to implement? I've seen some samples (including this site: here and here, and few more), some override the sync method, and other override the overflow method. Which one should I override? Also, looking at the stringbuf and filebuf sources (Visual Studio or GCC) both those buffer classes implement many methods of the streambuf.

If a custom class derived from streambuf is required, would there be any benefit deriving from stringbuf (or any other class) instead of directly from streambuf?

As for "lines". I would like at least when my users of the class using the 'endl' manipulator to be a new line (i.e. record in database). Maybe - depends on effort - every '\n' character should be considered as a new record as well. Who do my custom ostream and/or streambuf get notified for each?

Merely

You should probably create your own streambuf class, which handles all the heavy work, and then create a very simple ostream class that inherits std::basic_ostream and initializes itself with your streambuf object. – Sleet

You should check out Boost.Iostreams - it make creating custom streams and buffers a lot simpler. – Burgeon

Thank you for your edit, @MarkusParker – Merely

Answer 1

A custom destination for ostream means implementing your own ostreambuf. If you want your streambuf to actually buffer (i.e. don't connect to the database after each character), the easiest way to do that is by creating a class inheriting from std::stringbuf. The only function that you'll need to override is the sync() method, which is being called whenever the stream is flushed.

class MyBuf : public std::stringbuf
{
public:
    virtual int sync() {
        // add this->str() to database here
        // (optionally clear buffer afterwards)
    }
};

You can then create a std::ostream using your buffer:

MyBuf buff;
std::ostream stream(&buf)

Most people advised against redirecting the stream to a database, but they ignored my description that the database basically has a single blob field where all text is going to. In rare cases, I might send data to a different field. This can be facilitated with custom attributes understood by my stream. For example:

MyStream << "Some text " << process_id(1234) << "more text" << std::flush

The code above will create a record in the database with:

blob: 'Some text more text'
process_id: 1234

process_id() is a method returning a structure ProcessID. Then, in the implementation of my ostream, I have an operator<<(ProcessID const& pid), which stores the process ID until it gets written. Works great!

Merely

Does your MyStream class inherit from std::ostream? Does it override any methods? I'm asking this because I'm getting an error stating that the constructor is protected – Ehrsam

the best way I found to "clear the buffer" is this->str("") – Bethannbethanne

Does sync get called after a while even if you don't flush the stream? I know this happens with fstream. – Kolinsky

@Kolinsky - If by "after a while" you mean after certain amount of bytes, than yes. – Merely

@Uri what I don't understand then is, why do we need an overflow function? No space available -> flush() -> put character in stringbuf. Why is overflow() needed too? – Kolinsky

Answer 2

The simplest way is to inherit std::streambuf and override just two methods:

• std::streamsize xsputn(const char_type* s, std::streamsize n) – to append a given buffer with size provided to your internal buffer, std::string for example;
• int_type overflow(int_type c) – to append a single char to your internal buffer.

Your streambuf can be constructed from whatever you want (DB connection for example). After append something into the internal buffer you may try to split it into lines and push something into DB (or just buffer an SQL statements to execute later).

To use it: just attach your streambuf to any std::ostream using constructor.

Simple! I've done something like this to output strings to syslog – everything works fine with any custom operator<< for user defined classes.

Lucianolucias

Wouldn't implementing xsputn defeat the all purpose of streambuf? Both filebuf and stringbuf don't override this method, but only overflow (which is called by stringbuf). – Merely

In fact only overflow method is to be overriden. By default sputn executes sputc on every character. – Senile

Answer 3

my2c - I think you are tackling this the wrong way. A stream may sound like a nice idea, but you'll need a way to indicate the end of the row too (and then what if someone forgets?) I would suggest something along the lines of how the java PreparedStatements and batches work, as in provide a set of methods which accept the types and a column index, then a "batch" method which explicitly makes it clear that you are indeed batching that row and then an execute to push the batch in.

Any stream based operation will rely on type (typically) to indicate which column to fill - but what if you have two ints? IMO, as a user, it doesn't feel like a natural way of inserting records into a database...

Fugue

Answer 4

To add a new source or destination of character input/output to the iostreams mechanism, you should create a new streambuf class. The task of the stream buffer classes is to communicate with the 'external device' that will store the characters and to provide buffering facilities.

The problem with using iostreams to communicate with your database is that a database table does not match with the concept of a character sequence. A bit like pushing a round peg in a square hole. A streambuf only operates on characters. That is the only thing ever presented to it. This means the streambuf has to parse the character stream presented to it to find the field and record separators. If you decide to go this route, I predict you will end up writing a CSV-to-SQL converter in your streambuf, just to get it working.

You will probably be better of with just adding a few operator<< overloads to your class(es). You could look at the Qt framework for ideas here. They also have the possibility to use operator<< to add items to a collections and such.

Succentor