How can I cleanly exit a Pyro Daemon by client request?

Question

I'm trying to use Pyro to control a slave machine. I rsync the necessary python files, start a Pyro server, perform some actions by remote control, and then I want to tell the Pyro server to shut down.

I'm having trouble getting the Pryo Daemon to shut down cleanly. It either hangs in the Daemon.close() call, or if I comment out that line it exits without shutting down its socket correctly, resulting in socket.error: [Errno 98] Address already in use if I restart the server too soon.

It don't think that SO_REUSEADDR is the right fix, as unclean socket shutdown still results in a socket hanging around in the TIME_WAIT state, potentially causing some clients to experience problems. I think the better solution is to convince the Pyro Daemon to close its socket properly.

Is it improper to call Daemon.shutdown() from within the daemon itself?

If I start a server and then press CTRL-C without any clients connected I don't have any problems (no Address already in use errors). That makes a clean shutdown seem possible, most of the time (assuming an otherwise sane client and server).

Example: server.py

import Pyro4

class TestAPI:
    def __init__(self, daemon):
        self.daemon = daemon
    def hello(self, msg):
        print 'client said {}'.format(msg)
        return 'hola'
    def shutdown(self):
        print 'shutting down...'
        self.daemon.shutdown()

if __name__ == '__main__':
    daemon = Pyro4.Daemon(port=9999)
    tapi = TestAPI(daemon)
    uri = daemon.register(tapi, objectId='TestAPI')
    daemon.requestLoop()
    print 'exited requestLoop'
    daemon.close() # this hangs
    print 'daemon closed'

Example: client.py

import Pyro4

if __name__ == '__main__':
        uri = 'PYRO:TestAPI@localhost:9999'
        remote = Pyro4.Proxy(uri)
        response = remote.hello('hello')
        print 'server said {}'.format(response)
        try:
            remote.shutdown()
        except Pyro4.errors.ConnectionClosedError:
            pass
        print 'client exiting'

Brandybrandyn

Hey Eric. I never had the Address already in use for the Pyro server, but I get it all the time for the Name Server. Hitting CTRL+C on the NameServer has a 50% chance of causing that error if I run the name server again within 30 seconds. Have you had this before? – Jeffersonjeffery

Answer 1

I think this can be done without using timeout or loopCondition, by having your shutdown() call the daemon's shutdown. According to http://pythonhosted.org/Pyro4/servercode.html#cleaning-up:

Another possibility is calling Pyro4.core.Daemon.shutdown() on the running bdaemon object. This will also break out of the request loop and allows your code to neatly clean up after itself, and will also work on the threaded server type without any other requirements.

The following works on Python3.4.2 on Windows. The @Pyro4.oneway decorator for shutdownis not needed here, but it is in some situations.

server.py

import Pyro4
# using Python3.4.2

@Pyro4.expose
class TestAPI:
    def __init__(self, daemon):
        self.daemon = daemon
    def hello(self, msg):
        print('client said {}'.format(msg))
        return 'hola'
    @Pyro4.oneway   # in case call returns much later than daemon.shutdown
    def shutdown(self):
        print('shutting down...')
        self.daemon.shutdown()

if __name__ == '__main__':
    daemon = Pyro4.Daemon(port=9999)
    tapi = TestAPI(daemon)
    uri = daemon.register(tapi, objectId='TestAPI')
    daemon.requestLoop()
    print('exited requestLoop')
    daemon.close()
    print('daemon closed')

client.py

import Pyro4
# using Python3.4.2

if __name__ == '__main__':
    uri = 'PYRO:TestAPI@localhost:9999'
    remote = Pyro4.Proxy(uri)
    response = remote.hello('hello')
    print('server said {}'.format(response))
    remote.shutdown()
    remote._pyroRelease()
    print('client exiting')

Elysium

Answer 2

I think I am close to a solution: a combination of using the loopCondition parameter to requestloop() and the config value COMMTIMEOUT.

server.py

import Pyro4
Pyro4.config.COMMTIMEOUT = 1.0 # without this daemon.close() hangs

class TestAPI:
    def __init__(self, daemon):
        self.daemon = daemon
        self.running = True
    def hello(self, msg):
        print 'client said {}'.format(msg)
        return 'hola'
    def shutdown(self):
        print 'shutting down...'
        self.running = False

if __name__ == '__main__':
    daemon = Pyro4.Daemon(port=9999)
    tapi = TestAPI(daemon)
    uri = daemon.register(tapi, objectId='TestAPI')
    def checkshutdown():
        return tapi.running
    daemon.requestLoop(loopCondition=checkshutdown) # permits self-shutdown
    print 'exited requestLoop'
    daemon.close()
    print 'daemon closed'

Unfortunately, there is one condition where it still leaves a socket behind in the TIME_WAIT state. If the client closes his socket after the server, then the next attempt to start the server returns the same Address already in use error.

The only way I can find to work around this is to make the server COMMTIMEOUT longer (or sleep for several seconds before calling daemon.close()), and make sure the client always calls _pyroRelease() right after the shutdown call:

client.py

import Pyro4

if __name__ == '__main__':
        uri = 'PYRO:TestAPI@localhost:9999'
        remote = Pyro4.Proxy(uri)
        response = remote.hello('hello')
        print 'server said {}'.format(response)
        remote.shutdown()
        remote._pyroRelease()
        print 'client exiting'

I suppose that's good enough, but given the unfairness of scheduling and network delays it's still disappointing to have that race condition lurking.

Brandybrandyn

I have found in testing that using COMMTIMEOUT too aggressively leads to spurious failures, so I had to back this off to 5 seconds. Yet another reason why this solution doesn't feel exactly right. – Brandybrandyn