boost beast message with body_limit

Question

My starting point is to create a simple downloader code from the boost beast http_client_async example at boost http_client_async. In this scenario i want to write the received body into a file.

So I exchanged the string body into a file_body, to write the received data:

 http::response_parser<http::file_body> res_;

And simply rewrote the on_write method to

void on_write( boost::system::error_code ec,
               std::size_t bytes_transferred )
{
    boost::ignore_unused(bytes_transferred);

    if(ec)
        return fail(ec, "write");

    boost::system::error_code ec_file;
    res_.body().open("myTest.txt", boost::beast::file_mode::write, ec_file);

    // test for ec_file missing 

    // Receive the HTTP response
    http::async_read(socket_, buffer_, res_,
        std::bind(
            &session::on_read,
            shared_from_this(),
            std::placeholders::_1,
            std::placeholders::_2));
}

So but now, some of the received data bodies are to big:

read: body limit exceeded

and I try to increase the body limit.

In case of using a parser instead of a message, the size limit of the requested body can be changed with the body_limit() method.

Is there an easy way to increase the body size limit from a message as well?

Scribbler

to use the response_parser<http::file_body> instead of response<http::string_body> is the best way to go, I think. It got confused here. A response parser is not a message. When trying to get the body for calling open, res_ needs a call to res_.get().body().open("myTest.txt", boost::beast::file_mode::write, ec_file);. The response_parser res_ is able to handle the member variable body_limit(). – Scribbler

Answer 1

Beast's HTTP interfaces are grouped into layers. The first layer has the message-oriented stream algorithms, which operate on an HTTP message container. These are designed for simplicity, but allow for very little customization. The next layer is the serializer/parser oriented interface. This requires maintaining the lifetime of a serializer (for writing) or a parser (for reading) for the duration of the stream operation. It is a little more complex but correspondingly allows for more customization.

Adjusting the maximum size of the message body requires using the parser-oriented interface, as you have noted in your comment:

namespace http = boost::beast::http;
http::response_parser<http::file_body> parser;

// Allow for an unlimited body size
parser.body_limit((std::numeric_limits<std::uint64_t>::max)());
...
http::read(socket, buffer, parser);

Tying

Thanks for the clarification! I worked with the code for a while now. Reading the code + ducumentation API helps more than the documentation examples at the moment as the possibilities to use beast are really huge! – Scribbler

I think std::numeric_limits<std::uint64_t>::max should be std::numeric_limits<std::uint64_t>::max(). – Sacramentarian

Yes, you are right! Also, the function name should be surrounded with parenthesis in case someone defines max as a macro (tsk tsk Windows.h) – Tying

http::response_parser<http::string_body> is not move_insertable. This prevents std::vector::reserve, ::resize and more. How can I deal with this? All I really need is a usable vector of HTTP responses with no body limit. – Goodish

@VinnieFalco I've basically gone through the same steps as the OP. It works, but I was wondering about using async_read_some as I'll be working a relatively resource constrained target. So far, I've failed and I get a "bad line ending" error on the 2nd read. I'm obviously doing something wrong; do I need to use async_read_some at all? If so, can you give me any pointers on where to look (do I need a serializer, or a custom parser?) Is there a better place to ask for help!? Many thanks. – Polaroid

You need to inspect the bytes that you are receiving. Most likely, the other end is sending you invalid HTTP. – Tying