Iterable object and Django StreamingHttpResponse

Question

I want to connect to internal http services with django and I need to buffer the output the http response of those services because some contents are very large.

I am using python 3.6, django 2.0, http.client and the following code:

class HTTPStreamIterAndClose():
    def __init__(self, conn, res, buffsize):
        self.conn = conn
        self.res = res
        self.buffsize = buffsize
        self.length = 1

        bytes_length = int(res.getheader('Content-Length'))

        if buffsize < bytes_length:
            self.length = math.ceil(bytes_length/buffsize)

    def __iter__(self):
        return self

    def __next__(self):
        buff = self.res.read(self.buffsize)

        if buff is b'':
            self.res.close()
            self.conn.close()

            raise StopIteration
        else:

            return buff

    def __len__(self):
        return self.length


def passthru_http_service(request, server, timeout, path):
    serv = HTTPService(server, timeout)
    res = serv.request(path)

    response = StreamingHttpResponse(
        HTTPStreamIterAndClose(serv.connection, res, 200),
        content_type='application/json'
    )
    response['Content-Length'] = res.getheader('Content-Length')

    return response

And the reponse is empty, I test the iterator with:

b''.join(HTTPStreamIterAndClose(serv.connection, res, 200)

And everything works fine, I don't know why is not working.

Skvorak

Are there any errors in browser console? – Belak

No, there's no error – Skvorak

What do you mean "not working"? Do you get http 500? 200 with a blank output? Provide curl -v http://localhost:8000/x/y/ (or Postman or ...) output in your question please. – Tlingit

As I say, empty response, nothing =( – Skvorak

Use an external connection as client, you will be see really "what happened" ! According to your question, it is impossible to connect to a loopback address this way (define another loopback for the stream job). – Aerophone

That may seem a bit of a long-shot @FelipeBuccioni but can you try and change the content_type to text/event-stream (or even remove it all together)? – Sine

What WSGI server are you using? Is there a nginx-like proxy in front of it? – Bridie

Answer 1

https://andrewbrookins.com/django/how-does-djangos-streaminghttpresponse-work-exactly/

First, some conditions must be true:

• The client must be speaking HTTP/1.1 or newer
• The request method wasn’t a HEAD
• The response does not include a Content-Length header
• The response status wasn’t 204 or 304

If these conditions are true, then Gunicorn will add a Transfer-Encoding: chunked header to the response, signaling to the client that the response will stream in chunks.

In fact, Gunicorn will respond with Transfer-Encoding: chunked even if you used an HttpResponse, if those conditions are true!

To really stream a response, that is, to send it to the client in pieces, the conditions must be true, and your response needs to be an iterable with multiple items.

Basically, you need to decide: streaming or Content-Length.

If you want resumable downloads use Range.

Larina

Does this apply to other WSGI servers as well? – Inhale

IDK, I didn't experiment. – Larina

@Inhale the answer states gUnicorn, so this must be applicable for WSGI servers right? – Gadabout

Answer 2

At the end, the problem was in the fact that the http request disconnects after a some ms, thats why when I iterate inmediatly after the connection and before the creation of the objects works find and not when I use after the response, but when I start the connection when start to iterate, all works perfectly =/.

Cheers.

Skvorak