Flask broken pipe with requests

Question

I'd like to send a local REST request in a flask app, like this:

from flask import Flask, url_for, request
import requests

app = Flask(__name__)

@app.route("/<name>/hi", methods=["POST"])
def hi_person(name):
    form = {"name": name}
    return requests.post(url_for("hi", _external=True), data=form)

@app.route("/hi", methods=["POST"])
def hi():
    return 'Hi, %s!' % request.form["name"]

Sending curl -X POST http://localhost:5000/john/hi causes the entire flask app to freeze. When I send a kill signal, I get a broken pipe error. Is there a way to prevent flask from freezing here?

Spirochete

Answer 1

Run your flask app under a proper WSGI server capable of handling concurrent requests (perhaps gunicorn or uWSGI) and it'll work. While developing, enable threads in the Flask-supplied server with:

app.run(threaded=True)

but note that the Flask server is not recommended for production use. As of Flask 1.0, threaded is enabled by default, and you'd want to use the flask command on the command line, really, to run your app.

What happens is that using requests you are making a second request to your flask app, but since it is still busy processing the first, it won't respond to this second request until it is done with that first request.

Incidentally, under Python 3 the socketserver implementation handles the disconnect more gracefully and continues to serve rather than crash.

Quarles

I run the wsgi app threaded but I get the Broken Pipe anyways: app.run(debug=True, threaded=True, host='0.0.0.0', port=8080) – Symbolic

@loretoparisi: Without details that is impossible to diagnose. Running the code in the question with your app.run() line and adjusted curl command works fine. Perhaps you should post a new question? – Quarles

the problem wont be solved by threaded=True right ? @Martijn Pieters – Costumier

@Uddhav: if you are running Flask as a WSGI app under a WSGI server, then app.run() isn't used, and so threaded=True won't make a difference, no. – Quarles

Thank you for your solution. This issue was really bugging me. – Sipper

Answer 2

There are several things at play here, and I'll try to address them one-at-a-time.

First, you're probably using the toy development server. This server has many limitations; chiefly among these limitations is that it can only handle one request at a time. When you create a second request during your first request, you are locking up your application: The requests.post() function is waiting for Flask to respond, but Flask itself is waiting for post() to return! The solution to this particular problem is to run your WSGI application in a multithreaded or multiprocess environment. I prefer http://twistedmatrix.com/trac/wiki/TwistedWeb for this, but there are several other options.

With that out of the way... This is an antipattern. You almost certainly don't want to invoke all of the overhead of an HTTP request just to share some functionality between two views. The correct thing to do is to refactor to have a separate function that does that shared work. I can't really refactor your particular example, because what you have is very simple and doesn't really even merit two views. What did you want to build, exactly?

Edit: A comment asks whether multithreaded mode in the toy stdlib server would be sufficient to keep the deadlock from occurring. I'm going to say "maybe." Yes, if there aren't any dependencies keeping both threads from making progress, and both threads make sufficient progress to finish their networking tasks, then the requests will complete correctly. However, determining whether two threads will deadlock each other is undecidable (proof omitted as obtuse) and I'm not willing to say for sure that the stdlib server can do it right.

Marable

+1 making a request within a request handler is bad in most cases, an http request to yourself ... ick – Sporule

I ran into the problem because I was merging two web services that had previously been running as separate processes. The only interaction between the two web services was that POST. I combined them into the same process using blueprints and boom, it froze. So, a little bit more refactoring is in order. – Spirochete

@Marable so does the threaded=True option not enough as TwistedWeb regarding multiprocessing e multi threaded execution at every request? – Symbolic

Answer 3

The bug that caused the crash was fixed in Version 0.12, Released on December 21st 2016. Yeah! This is an important fix that many have been waiting for.

From the Flask changelog:

• Revert a behavior change that made the dev server crash instead of returning a Internal Server Error (pull request #2006).

Potomac

Yes! Thanks for the tip. Upgrading to flask 0.12 fixed the problem. Now these errors are merely logged without crashing my server. – Exhalation

Answer 4

I have had the same issue with post method, in general my post method was doing nothing, thats why this issues came

return _socket.socket.send(self._sock, data, flags) urllib3.exceptions.ProtocolError:
('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

if request.method == 'POST':
    print(len(request.data))
return 'dummy'

this print did the trick

Colonel