How to stream an HttpResponse with Django

Asked 27/5, 2010 at 16:21 Answered 17/11, 2012 at 10:33

I'm trying to get the 'hello world' of streaming responses working for Django (1.2). I figured out how to use a generator and the yield function. But the response still not streaming. I suspect there's a middleware that's mucking with it -- maybe ETAG calculator? But I'm not sure how to disable it. Can somebody please help?

Here's the "hello world" of streaming that I have so far:

def stream_response(request):
    resp = HttpResponse( stream_response_generator())
    return resp

def stream_response_generator():
    for x in range(1,11):
        yield "%s\n" % x  # Returns a chunk of the response to the browser
        time.sleep(1)

Incite answered 27/5, 2010 at 16:21 Comment(1)

@Tomasz: the WSGI protocol specification python.org/dev/peps/pep-0333 – Salinas 27/5, 2010 at 17:35

You can disable the ETAG middleware using the condition decorator. That will get your response to stream back over HTTP. You can confirm this with a command-line tool like curl. But it probably won't be enough to get your browser to show the response as it streams. To encourage the browser to show the response as it streams, you can push a bunch of whitespace down the pipe to force its buffers to fill. Example follows:

from django.views.decorators.http import condition

@condition(etag_func=None)
def stream_response(request):
    resp = HttpResponse( stream_response_generator(), content_type='text/html')
    return resp

def stream_response_generator():
    yield "<html><body>\n"
    for x in range(1,11):
        yield "<div>%s</div>\n" % x
        yield " " * 1024  # Encourage browser to render incrementally
        time.sleep(1)
    yield "</body></html>\n"

Tutu answered 27/5, 2010 at 16:48 Comment(5)

In my tests the Django GZipMiddleware can prevent this from working. – Vegetative 21/12, 2010 at 5:55

Yeah, I expect a lot of middlewares are likely to mess with it, so if it's not working, try disabling all your middlewares and re-enable them incrementally. GZip requires having the entire response before it compresses it, so it won't let you stream. – Tutu 3/1, 2011 at 1:25

@Xealot: I'm having a similar problem with GzipMiddleware. Filed a bug because that middleware does not support generators (it actually clears the generator accidentally): Django ticket #15066 – Unrivalled 12/1, 2011 at 23:54

I disabled the GZipMiddleware on this call by inserting a fake header into the StreamingHttpResponse: response['Content-Encoding'] = 'identity'. This is probably invalid HTTP in eighteen different ways, but it was just an internal endpoint in my case, so this was good enough for me. – Kilimanjaro 19/10, 2015 at 21:57

Maybe a better way would be to subclass it and customize it to ignore all responses with a header X-Streaming or something like than on. – Ardithardme 11/1, 2016 at 15:31

A lot of the django middleware will prevent you from streaming content. Much of this middleware needs to be enabled if you want to use the django admin app, so this can be an annoyance. Luckily this has been resolved in the django 1.5 release. You can use the StreamingHttpResponse to indicate that you want to stream results back and all the middleware that ships with django is aware of this and acts accordingly to not buffer your content output but send it straight down the line. Your code would then look like the following to use the new StreamingHttpResponse object.

def stream_response(request):
    return StreamingHttpResponse(stream_response_generator())

def stream_response_generator():
    for x in range(1,11):
        yield "%s\n" % x  # Returns a chunk of the response to the browser
        time.sleep(1)

Note on Apache

I tested the above on Apache 2.2 with Ubuntu 13.04. The apache module mod_deflate which was enabled by default in the setup I tested will buffer the content you are trying to stream until it reaches a certain block size then it will gzip the content and send it to the browser. This will prevent the above example from working as desired. One way to avoid this is to disable mod_deflate by putting the following line in your apache configuration:

SetEnvIf Request_URI ^/mysite no-gzip=1

This is discussed more in the How to disable mod_deflate in apache2? question.

Amora answered 17/11, 2012 at 10:33 Comment(5)

It works! but how am I supposed to render it to the template real-time ? Currently I am using Javascript with an ajax call.. but it posts the output.. only when it's finished processing.. So it's not Streaming when it comes to rendering it to the browser. Any help would be appreciated.. Thank you :) – Tribunal 5/11, 2013 at 20:56

@FirstBlood With ajax you can get callbacks as it is streaming and not just when the download is complete. Check out this answer https://mcmap.net/q/169678/-what-is-the-best-way-of-showing-progress-on-an-ajax-call for showing progress on an Ajax call. – Amora 6/11, 2013 at 12:2

Thanks.. I want to achieve Ping output status.. to be printed real-time on the browser.. So for ex: ping -c 3 www.google.com .. how the command prompt gives the output.. is it possible in Django, to spill out the output to the browser as I read via subprocess? I tried the StreamingHttpResponse, but it doesn't seem to stream real-time.. I even tried setting headers: Keep-Alive.. but no effect :( – Tribunal 6/11, 2013 at 20:30

your webserver (eg Apache) will likely also buffer the output, if the process is not very long running or doesn't send much output this may have the effect that the webserver buffers it all and then delivers it in one lump at the end – Dermatome 29/3, 2014 at 9:38

@Dermatome That is true. I have updated the answer with a note about Apache – Amora 29/3, 2014 at 13:19

Recommended topics

Hot tags