Python - Running Autobahn|Python asyncio websocket server in a separate subprocess or thread

Question

I have a tkinter based GUI program running in Python 3.4.1. I have several threads running in the program to get JSON data from various urls. I am wanting to add some WebSocket functionality to be able to allow program to act as a server and allow several clients to connect to it over a WebSocket and exchange other JSON data.

I am attempting to use the Autobahn|Python WebSocket server for asyncio.

I first tried to run the asyncio event loop in a separate thread under the GUI program. However, every attempt gives 'AssertionError: There is no current event loop in thread 'Thread-1'.

I then tried spawning a process with the standard library multiprocessing package that ran the asyncio event loop in another Process. When I try this I don't get any exception but the WebSocket server doesn't start either.

Is it even possible to run an asyncio event loop in a subprocess from another Python program?

Is there even a way to integrate an asyncio event loop into a currently multithreaded/tkinter program?

UPDATE Below is the actual code I am trying to run for an initial test.

from autobahn.asyncio.websocket import WebSocketServerProtocol
from autobahn.asyncio.websocket import WebSocketServerFactory
import asyncio
from multiprocessing import Process

class MyServerProtocol(WebSocketServerProtocol):

   def onConnect(self, request):
      print("Client connecting: {0}".format(request.peer))

   def onOpen(self):
      print("WebSocket connection open.")

   def onMessage(self, payload, isBinary):
      if isBinary:
         print("Binary message received: {0} bytes".format(len(payload)))

      else:
         print("Text message received: {0}".format(payload.decode('utf8')))

      ## echo back message verbatim
      self.sendMessage(payload, isBinary)

   def onClose(self, wasClean, code, reason):
      print("WebSocket connection closed: {0}".format(reason))

def start_server():
   factory = WebSocketServerFactory("ws://10.241.142.27:6900", debug = False)
   factory.protocol = MyServerProtocol
   loop = asyncio.get_event_loop()
   coro = loop.create_server(factory, '10.241.142.27', 6900)
   server = loop.run_until_complete(coro)
   loop.run_forever()
   server.close()
   loop.close()


websocket_server_process = Process(target = start_server)
websocket_server_process.start()

Most of it is straight from the Autobahn|Python example code for asyncio. If I try to run it as a Process it doesn't do anything, no client can connect to it, if I run netstat -a there is no port 6900 being used. If just use start_server() in the main program it creates the WebSocket Server.

Studner

Answer 1

First, you're getting AssertionError: There is no current event loop in thread 'Thread-1'. because asyncio requires each thread in your program to have its own event loop, but it will only automatically create an event loop for you in the main thread. So if you call asyncio.get_event_loop once in the main thread it will automatically create a loop object and set it as the default for you, but if you call it again in a child thread, you'll get that error. Instead, you need to explicitly create/set the event loop when the thread starts:

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

Once you've done that, you should be able to use get_event_loop() in that specific thread.

It is possible to start an asyncio event loop in a subprocess started via multiprocessing:

import asyncio
from multiprocessing import Process 

@asyncio.coroutine
def coro():
    print("hi")

def worker():
    loop = asyncio.get_event_loop()
    loop.run_until_complete(coro())

if __name__ == "__main__":
    p = Process(target=worker)
    p.start()
    p.join()

Output:

hi

The only caveat is that if you start an event loop in the parent process as well as the child, you need to explicitly create/set a new event loop in the child if you're on a Unix platform (due to a bug in Python). It should work fine on Windows, or if you use the 'spawn' multiprocessing context.

I think it should be possible to start an asyncio event loop in a background thread (or process) of your Tkinter application and have both the tkinter and asyncio event loop run side-by-side. You'll only run into issues if you try to update the GUI from the background thread/process.

Maggot

I updated the original post with the exact code I am trying to run and use in a separate process. The code I am using for some reason doesn't work. Also I do update the GUI from some of the background threads. What kind of issues might I have if I try run tkinter and asyncio? – Studner

@Studner As far as I know, Tkinter doesn't support updating the GUI from anything but the main thread. Trying to do so will likely not work or cause exceptions to be raised. – Maggot

@Studner Also, If I run your exact code above, I do end up with a server listening on port 6900. (This is using Python 2.7/trollius instead of Python 3.4/asyncio, though. No access to Python 3.4 right now). The only difference is I'm using localhost instead of the IP you listed,. – Maggot

Thank you! I was having trouble with this error message. I forgot to set the event loop. – Markley

Answer 2

The answer by @dano might be correct, but creates an new process which is unnessesary in most situations.

I found this question on Google because i had the same issue myself. I have written an application where i wanted an websocket api to not run on the main thread and this caused your issue.

I found my alternate sollution by simply reading about event loops on the python documentation and found the asyncio.new_event_loop and asyncio.set_event_loop functions which solved this issue.

I didn't use AutoBahn but the pypi websockets library, and here's my solution

import websockets
import asyncio
import threading

class WebSocket(threading.Thread):    
    @asyncio.coroutine
    def handler(self, websocket, path):
        name = yield from websocket.recv()
        print("< {}".format(name))
        greeting = "Hello {}!".format(name)
        yield from websocket.send(greeting)
        print("> {}".format(greeting))

    def run(self):
        start_server = websockets.serve(self.handler, '127.0.0.1', 9091)
        eventloop = asyncio.new_event_loop()
        asyncio.set_event_loop(eventloop)
        eventloop.run_until_complete(start_server)
        eventloop.run_forever()

if __name__ == "__main__":
    ws = WebSocket()
    ws.start()

Uralite

How does one send messages over websocket "on demand", i.e. not through a callback? I have a game server that wants to communicate to clients via websockets, and I want to call socket_server.sendMessage("my message"), but I can only do that if i put it in a callback... – Paly

@andy you could add each new connection in the onConnect() to a list stored in a classvariable and then use a classmethod to iterate over the list to send a message to all available connections (this looks somehow dirty - but it works for me). don't forget to remove the connection form the list within onClose(). – Jaf

Answer 3

"Is there even a way to integrate an asyncio event loop into a currently multithreaded/tkinter program?"

Yes, run your tkinter program with an asyncio event loop. Proof of concept.

'''Proof of concept integrating asyncio and tk loops.

Terry Jan Reedy
Run with 'python -i' or from IDLE editor to keep tk window alive.
'''

import asyncio
import datetime as dt
import tkinter as tk

loop = asyncio.get_event_loop()
root = tk.Tk()

# Combine 2 event loop examples from BaseEventLoop doc.
# Add button to prove that gui remain responsive between time updates.
# Prints statements are only for testing.

def flipbg(widget, color):
    bg = widget['bg']
    print('click', bg, loop.time())
    widget['bg'] = color if bg == 'white' else 'white'

hello = tk.Label(root)
flipper = tk.Button(root, text='Change hello background', bg='yellow',
                    command=lambda: flipbg(hello, 'red'))
time = tk.Label(root)
hello.pack()
flipper.pack()
time.pack()

def hello_world(loop):
    hello['text'] = 'Hello World'
loop.call_soon(hello_world, loop)

def display_date(end_time, loop):
    print(dt.datetime.now())
    time['text'] = dt.datetime.now()
    if (loop.time() + 1.0) < end_time:
        loop.call_later(1, display_date, end_time, loop)
    else:
        loop.stop()

end_time = loop.time() + 10.1
loop.call_soon(display_date, end_time, loop)

# Replace root.mainloop with these 4 lines.
def tk_update():
    root.update()
    loop.call_soon(tk_update)  # or loop.call_later(delay, tk_update)
# Initialize loop before each run_forever or run_until_complete call    
tk_update() 
loop.run_forever()

I have experimentally run IDLE with those 4 extra lines, with a slowdown only noticeable when syntax highlighting 1000s of lines.

Gettings