Python HTTP server keep connection alive

Question

I am trying to test a HTTP client that is written in C, that sends an HTTP POST request to a local sever on my computer. I have added the headers keep-alive in my POST request that looks like this on the python3 HTTP server running on my computer:

<ip-address-1> - - [29/Apr/2018 18:27:49] "POST /html HTTP/1.1" 200 -
Host: <ip-address-2>
Content-Type: application/json
Content-Length: 168
Connection: Keep-Alive
Keep-Alive: timeout=5, max=100


INFO:root:POST request,
Body:
{
"field": "abc",
"time": "2018-04-29T01:27:50.322000Z" 
}

The HTTP server POST handler looks like this:

class S(BaseHTTPRequestHandler):
    def _set_response(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.send_header("Connection", "keep-alive")
        self.send_header("keep-alive", "timeout=5, max=30")
        self.end_headers()

    def do_POST(self):
        content_length = int(self.headers['Content-Length']) # <--- Gets the size of data
        post_data = self.rfile.read(content_length) # <--- Gets the data itself
        print(self.headers)
        logging.info("POST request,\nBody:\n%s\n", post_data.decode('utf-8'))

        self._set_response()
        self.wfile.write("POST request for {}".format(self.path).encode('utf-8'))

def run(server_class=HTTPServer, handler_class=S, port=8080):
    logging.basicConfig(level=logging.INFO)
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    logging.info('Starting httpd...\n')
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    logging.info('Stopping httpd...\n')

The header response I see on the client side is:

HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.5.2
Date: Tue, 29 April 2018 16:07:42 GMT
Content-type: text/html
Connection: keep-alive
keep-alive: timeout=5, max=30

I still do end up getting a disconnection callback, so my question is how can I set keep-alive connection parameters from the server side?

Homopolar

Answer 1

By default the BaseHTTPRequestHandler emits HTTP/1.0 responses, as you can see in HTTP/1.0 200 OK. HTTP/1.1 is required for keep alive responses, as seen in the doc (or for v3):

protocol_version

This specifies the HTTP protocol version used in responses. If set to 'HTTP/1.1', the server will permit HTTP persistent connections; however, your server must then include an accurate Content-Length header (using send_header()) in all of its responses to clients. For backwards compatibility, the setting defaults to 'HTTP/1.0'.

Then as you can see in the quote, you'll have to set the right Content-Length for your response also.

Note that currently you send responses without body, you should use a 204 (no content) code for that and add the Content-length: 0 header, or add a small body (with the right bytes count in Content-Length, warning, that's not a character counter, that's a byte counter, almost the same in ascii7 but not with other encodings).

Macassar

Keep alive seems to work with the default HTTP/1.0, at least for answering GET requests. I also came across setting it on TCP sockets themselves. – Johansen

Answer 2

I will mostly show how to implement what accepted answer proposed. The step2 which is fundamental to have it working is missing in accepted answer.

1/ Make HTTPServer serving HTTP/1.1 instead of HTTP/1.0

class HTTP11RequestHandler(BaseHTTPRequestHandler):
    def __init__(self, request, client_address, server):
         self.protocol_version = 'HTTP/1.1'
         super().__init__(request, client_address, server)
         
    #...as you would do in a regular RequestHandler, except that you must specify the Content-Length header
    def do_GET(self):
        message = 'the backend answer'
        self.send_response(200)
        self.send_header('Content-Type','text/plain')
        self.send_header('Content-Length', len(message))
        self.end_headers()

2/ HTTPServer is unable to process multiple connection simultaneously. The connection being kept-alive, this will not work.

=> use https://docs.python.org/3.8/library/http.server.html#http.server.ThreadingHTTPServer ThreadingHTTPServer instead of HTTPServer

def start_backend_server(ip,port,requestHandler):
    backend_server = ThreadingHTTPServer((ip,port), requestHandler)
    f = lambda : backend_server.serve_forever()
    backend_thread = threading.Thread(target=f)
    backend_thread.daemon=True
    backend_thread.start()
    return backend_thread

call it like that

backend_thread = start_backend_thread('127.0.0.1', '80', HTTP11RequestHandler)

Ornamental