How to stop BaseHTTPServer.serve_forever() in a BaseHTTPRequestHandler subclass?

Question

I am running my HTTPServer in a separate thread (using the threading module which has no way to stop threads...) and want to stop serving requests when the main thread also shuts down.

The Python documentation states that BaseHTTPServer.HTTPServer is a subclass of SocketServer.TCPServer, which supports a shutdown method, but it is missing in HTTPServer.

The whole BaseHTTPServer module has very little documentation :(

Nuremberg

Answer 1

I should start by saying that "I probably wouldn't do this myself, but I have in the past". The serve_forever (from SocketServer.py) method looks like this:

def serve_forever(self):
    """Handle one request at a time until doomsday."""
    while 1:
        self.handle_request()

You could replace (in subclass) while 1 with while self.should_be_running, and modify that value from a different thread. Something like:

def stop_serving_forever(self):
    """Stop handling requests"""
    self.should_be_running = 0
    # Make a fake request to the server, to really force it to stop.
    # Otherwise it will just stop on the next request.
    # (Exercise for the reader.)
    self.make_a_fake_request_to_myself()

Edit: I dug up the actual code I used at the time:

class StoppableRPCServer(SimpleXMLRPCServer.SimpleXMLRPCServer):

    stopped = False
    allow_reuse_address = True

    def __init__(self, *args, **kw):
        SimpleXMLRPCServer.SimpleXMLRPCServer.__init__(self, *args, **kw)
        self.register_function(lambda: 'OK', 'ping')

    def serve_forever(self):
        while not self.stopped:
            self.handle_request()

    def force_stop(self):
        self.server_close()
        self.stopped = True
        self.create_dummy_request()

    def create_dummy_request(self):
        server = xmlrpclib.Server('http://%s:%s' % self.server_address)
        server.ping()

Fredfreda

I think you can call "self.serve_forever()" after "self.stopped = True" and avoid implementing "create_dummy_request(self)". It's worked for me so far, but there may be a subtlety that I'm missing. – Synchrocyclotron

As far as I'm looking at BaseServer.serve_forever() sources in Python 2.7, I see it already has a flag implemented, __shutdown_request, plus __is_shut_down threading.Event on top. So the actual answer I see here is the allow_reuse_address = True line. – Celadon

Answer 2

Another way to do it, based on http://docs.python.org/2/library/basehttpserver.html#more-examples, is: instead of serve_forever(), keep serving as long as a condition is met, with the server checking the condition before and after each request. For example:

import CGIHTTPServer
import BaseHTTPServer

KEEP_RUNNING = True

def keep_running():
    return KEEP_RUNNING

class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
    cgi_directories = ["/cgi-bin"]

httpd = BaseHTTPServer.HTTPServer(("", 8000), Handler)

while keep_running():
    httpd.handle_request()

Glassco

After seeing all other hacks here and there, this is a rare answer who relies on the officially documented behavior only, therefore it is considered cannonical. Up-voted! – Brodeur

I used this code and added some site updating code to keep_running() and found out that it only executes when getting a request. +1 Works perfect. This is the true answer. – Stroke

Answer 3

The event-loops ends on SIGTERM, Ctrl+C or when shutdown() is called.

server_close() must be called after server_forever() to close the listening socket.

import http.server

class StoppableHTTPServer(http.server.HTTPServer):
    def run(self):
        try:
            self.serve_forever()
        except KeyboardInterrupt:
            pass
        finally:
            # Clean-up server (close socket, etc.)
            self.server_close()

Simple server stoppable with user action (SIGTERM, Ctrl+C, ...):

server = StoppableHTTPServer(("127.0.0.1", 8080),
                             http.server.BaseHTTPRequestHandler)
server.run()

Server running in a thread:

import threading

server = StoppableHTTPServer(("127.0.0.1", 8080),
                             http.server.BaseHTTPRequestHandler)

# Start processing requests
thread = threading.Thread(None, server.run)
thread.start()

# ... do things ...

# Shutdown server
server.shutdown()
thread.join()

Snafu

Answer 4

In my python 2.6 installation, I can call it on the underlying TCPServer - it still there inside your HTTPServer:

TCPServer.shutdown


>>> import BaseHTTPServer
>>> h=BaseHTTPServer.HTTPServer(('',5555), BaseHTTPServer.BaseHTTPRequestHandler)
>>> h.shutdown
<bound method HTTPServer.shutdown of <BaseHTTPServer.HTTPServer instance at 0x0100D800>>
>>> 

Push

ah. yes, and I am reading the 2.6 docs - alas, I'm running 2.5. Bummer. Me idiot. – Nuremberg

Note that even after shutdown, the socket will still listen on the interface/port (tested on Python v2.7). You should also call: h.socket.close(). – Oxytetracycline

The important part to remember with the shutdown method is that is must be called from a thread different than the serving thread. Thus, if you want to shutdown in reaction to a request handler, you need to kick off a (daemon) thread which has its target= set to your h.shutdown function. – Phosphorate

Answer 5

I think you can use [serverName].socket.close()

Frolic

This was the only way that worked for me, tried interrupting with while not some_flag, with server.shutdown(). Only socket.close() achieved the desired. – Maynord

I had the same issue. I believe that my issue was that I was trying to call server.shutdown() from the inside handler and that sequence results in a deadlock since I suspect that the server is waiting for the handler to complete before returning from the shutdown() call. The downside to this is that I get a scary-looking exception printed on stdout that I'd like to suppress somehow. – Pampas

I've tried this but it throws exception. recent version have shutdown() function exported. – Corrosion

Answer 6

This is a simplified version of Helgi's answer for python 3.7:

import threading
import time
from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler


class MyServer(threading.Thread):
    def run(self):
        self.server = ThreadingHTTPServer(('localhost', 8000), SimpleHTTPRequestHandler)
        self.server.serve_forever()
    def stop(self):
        self.server.shutdown()


if __name__ == '__main__':
    s = MyServer()
    s.start()
    print('thread alive:', s.is_alive())  # True
    time.sleep(2)
    s.stop()
    print('thread alive:', s.is_alive())  # False

Altagraciaaltaic

Nice answer. Tested on 3.7 and works like a charm. – Alenaalene

Very nice answer. The best one. This changed my whole day. Thanks a lot. – Proxy

Super answer, works on Python 3.8.2. Thank you very much! – Christabella

The example works great. One thing to add: Often a web server is shut down from the outside using a signal. If the signal handler calls server.shutdown() you may end in a deadlock as you have no control about what Thread handles the signal (and you must not call shutdown() from the server thread). Possible solution (works for me): Thread(target=myServer.shutdown).start() in the signal handler. – Pride

Answer 7

In python 2.7, calling shutdown() works but only if you are serving via serve_forever, because it uses async select and a polling loop. Running your own loop with handle_request() ironically excludes this functionality because it implies a dumb blocking call.

From SocketServer.py's BaseServer:

def serve_forever(self, poll_interval=0.5):
    """Handle one request at a time until shutdown.

    Polls for shutdown every poll_interval seconds. Ignores
    self.timeout. If you need to do periodic tasks, do them in
    another thread.
    """
    self.__is_shut_down.clear()
    try:
        while not self.__shutdown_request:
            # XXX: Consider using another file descriptor or
            # connecting to the socket to wake this up instead of
            # polling. Polling reduces our responsiveness to a
            # shutdown request and wastes cpu at all other times.
            r, w, e = select.select([self], [], [], poll_interval)
            if self in r:
                self._handle_request_noblock()
    finally:
        self.__shutdown_request = False
        self.__is_shut_down.set()

Heres part of my code for doing a blocking shutdown from another thread, using an event to wait for completion:

class MockWebServerFixture(object):
    def start_webserver(self):
        """
        start the web server on a new thread
        """
        self._webserver_died = threading.Event()
        self._webserver_thread = threading.Thread(
                target=self._run_webserver_thread)
        self._webserver_thread.start()

    def _run_webserver_thread(self):
        self.webserver.serve_forever()
        self._webserver_died.set()

    def _kill_webserver(self):
        if not self._webserver_thread:
            return

        self.webserver.shutdown()

        # wait for thread to die for a bit, then give up raising an exception.
        if not self._webserver_died.wait(5):
            raise ValueError("couldn't kill webserver")

Yesteryear

Answer 8

This method I use successfully (Python 3) to stop the server from the web application itself (a web page):

import http.server
import os
import re

class PatientHTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
    stop_server = False
    base_directory = "/static/"
    # A file to use as an "server stopped user information" page.
    stop_command = "/control/stop.html"
    def send_head(self):
        self.path = os.path.normpath(self.path)
        if self.path == PatientHTTPRequestHandler.stop_command and self.address_string() == "127.0.0.1":
            # I wanted that only the local machine could stop the server.
            PatientHTTPRequestHandler.stop_server = True
            # Allow the stop page to be displayed.
            return http.server.SimpleHTTPRequestHandler.send_head(self)
        if self.path.startswith(PatientHTTPRequestHandler.base_directory):
            return http.server.SimpleHTTPRequestHandler.send_head(self)
        else:
            return self.send_error(404, "Not allowed", "The path you requested is forbidden.")

if __name__ == "__main__":
    httpd = http.server.HTTPServer(("127.0.0.1", 8080), PatientHTTPRequestHandler)
    # A timeout is needed for server to check periodically for KeyboardInterrupt
    httpd.timeout = 1
    while not PatientHTTPRequestHandler.stop_server:
        httpd.handle_request()

This way, pages served via base address http://localhost:8080/static/ (example http://localhost:8080/static/styles/common.css) will be served by the default handler, an access to http://localhost:8080/control/stop.html from the server's computer will display stop.html then stop the server, any other option will be forbidden.

Shirashirah

That while loop is using ≈100% CPU. See the latest "server_forever" method source in "socketserver.py". It's doing "select(poll_interval)" which is essentially doing a time.Sleep(0.5) interrupted by a server socket read available. So the internal loop is taking only a few percent CPU normally (as long as there is no socket activity). – Horst

Answer 9

import http.server
import socketserver
import socket as sck
import os
import threading


class myserver:
    def __init__(self, PORT, LOCATION):
        self.thrd = threading.Thread(None, self.run)
        self.Directory = LOCATION
        self.Port = PORT
        hostname = sck.gethostname()
        ip_address = sck.gethostbyname(hostname)
        self.url = 'http://' + ip_address + ':' + str(self.Port)
        Handler = http.server.SimpleHTTPRequestHandler
        self.httpd = socketserver.TCPServer(("", PORT), Handler)
        print('Object created, use the start() method to launch the server')
    def run(self):
        print('listening on: ' + self.url )
        os.chdir(self.Directory)
        print('myserver object started')        
        print('Use the objects stop() method to stop the server')
        self.httpd.serve_forever()
        print('Quit handling')

        print('Sever stopped')
        print('Port ' + str(self.Port) + ' should be available again.')


    def stop(self):
        print('Stopping server')
        self.httpd.shutdown()
        self.httpd.server_close()
        print('Need just one more request before shutting down'


    def start(self):
        self.thrd.start()

def help():
    helpmsg = '''Create a new server-object by initialising
NewServer = webserver3.myserver(Port_number, Directory_String)
Then start it using NewServer.start() function
Stop it using NewServer.stop()'''
    print(helpmsg)

Not a experience python programmer, just wanting to share my comprehensive solution. Mostly based on snippets here and there. I usually import this script in my console and it allows me to set up multiple servers for different locations using their specific ports, sharing my content with other devices on the network.

Fading

Answer 10

I tried all above possible solution and ended up with having a "sometime" issue - somehow it did not really do it - so I ended up making a dirty solution that worked all the time for me:

If all above fails, then brute force kill your thread using something like this:

import subprocess
cmdkill = "kill $(ps aux|grep '<name of your thread> true'|grep -v 'grep'|awk '{print $2}') 2> /dev/null"
subprocess.Popen(cmdkill, stdout=subprocess.PIPE, shell=True)

Intangible

Answer 11

Here's a context-flavored version for Python 3.7+ which I prefer because it cleans up automatically and you can specify the directory to serve:

from contextlib import contextmanager
from functools import partial
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread


@contextmanager
def http_server(host: str, port: int, directory: str):
    server = ThreadingHTTPServer(
        (host, port), partial(SimpleHTTPRequestHandler, directory=directory)
    )
    server_thread = Thread(target=server.serve_forever, name="http_server")
    server_thread.start()

    try:
        yield
    finally:
        server.shutdown()
        server_thread.join()


def usage_example():
    import time

    with http_server("127.0.0.1", 8087, "."):
        # now you can use the web server
        time.sleep(100)

Alenaalene

Answer 12

In latest version of Python 3, BaseServer now implements the shutdown method.

http_server = HTTPServer(('localhost', 8080), SimpleHTTPRequestHandler)
http_thread = Thread(target=http_server.serve_forever)
http_thread.start()

time.sleep(10)
http_server.shutdown()
http_thread.join()

Affrica