Use asyncio and Tkinter (or another GUI lib) together without freezing the GUI

Question

I want to use asyncio in combination with a tkinter GUI. I am new to asyncio and my understanding of it is not very detailed. The example here starts 10 task when clicking on the first button. The task are just simulating work with a sleep() for some seconds.

The example code is running fine with Python 3.6.4rc1. But the problem is that the GUI is freezed. When I press the first button and start the 10 asyncio-tasks I am not able to press the second button in the GUI until all tasks are done. The GUI should never freeze - that is my goal.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from tkinter import *
from tkinter import messagebox
import asyncio
import random

def do_freezed():
    """ Button-Event-Handler to see if a button on GUI works. """
    messagebox.showinfo(message='Tkinter is reacting.')

def do_tasks():
    """ Button-Event-Handler starting the asyncio part. """
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(do_urls())
    finally:
        loop.close()

async def one_url(url):
    """ One task. """
    sec = random.randint(1, 15)
    await asyncio.sleep(sec)
    return 'url: {}\tsec: {}'.format(url, sec)

async def do_urls():
    """ Creating and starting 10 tasks. """
    tasks = [
        one_url(url)
        for url in range(10)
    ]
    completed, pending = await asyncio.wait(tasks)
    results = [task.result() for task in completed]
    print('\n'.join(results))


if __name__ == '__main__':
    root = Tk()

    buttonT = Button(master=root, text='Asyncio Tasks', command=do_tasks)
    buttonT.pack()
    buttonX = Button(master=root, text='Freezed???', command=do_freezed)
    buttonX.pack()

    root.mainloop()

A _side problem

...is that I am not able to run the task a second time because of this error.

Exception in Tkinter callback
Traceback (most recent call last):
  File "/usr/lib/python3.6/tkinter/__init__.py", line 1699, in __call__
    return self.func(*args)
  File "./tk_simple.py", line 17, in do_tasks
    loop.run_until_complete(do_urls())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 443, in run_until_complete
    self._check_closed()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Multithreading

Whould multithreading be a possible solution? Only two threads - each loop has it's own thread?

EDIT: After reviewing this question and the answers it is related to nearly all GUI libs (e.g. PygObject/Gtk, wxWidgets, Qt, ...).

Gwyn

Your code works for me using Python 3.6.3 (on Windows). It takes a while, but eventually prints out 10 lines in this format url: 3 sec: 6, url: 4 sec: 4, etc. Perhaps you're encountering a bug that has been fixed (if you're using an earlier version of Python). – Greaseball

@Greaseball With 3.7.0a2 on win10, I get the same error when pressing the Asyncio Tasks button a second time. – Glide

@Greaseball But the GUI freezes while the tasks are running, right? That is the problem. About the bug: Please provide a link to the bug report. – Gwyn

buhtz: Yes, it freezes for a short time. If I click the other button while it's frozen, then when it unfreezes, the pop-up comes up. I didn't have a specific bug in mind, which is why I said perhaps it was one. – Greaseball

Your question is similar to this one, and my answer works with your setup. All you need is re-struct your program (like @Terry suggested) a little and bind your coroutines properly (via command/bind). First problem is obvious - you stuck in the inner loop (asyncio), while the outer loop in unreachable (tkinter), hence GUI in unresponsive state. Second one - you already closed asyncio loop. You either should close it once at the end (as @Terry proposed) or should create a new one each time. – Kersten

Answer 1

I had similar task solved with multiprocessing.

Major parts:

1. Main process is Tk's process with mainloop.
2. daemon=True process with aiohttp service that executes commands.
3. Intercom using duplex Pipe so each process can use it's end.

Additionaly, I'm making Tk's virtual events to simplify massage tracking on app's side. You will need to apply patch manually. You can check python's bug tracker for details.

I'm checking Pipe each 0.25 seconds on both sides.

$ python --version
Python 3.7.3

main.py

import asyncio
import multiprocessing as mp

from ws import main
from app import App


class WebSocketProcess(mp.Process):

    def __init__(self, pipe, *args, **kw):
        super().__init__(*args, **kw)
        self.pipe = pipe

    def run(self):
        loop = asyncio.get_event_loop()
        loop.create_task(main(self.pipe))
        loop.run_forever()


if __name__ == '__main__':
    pipe = mp.Pipe()
    WebSocketProcess(pipe, daemon=True).start()
    App(pipe).mainloop()

app.py

import tkinter as tk


class App(tk.Tk):

    def __init__(self, pipe, *args, **kw):
        super().__init__(*args, **kw)
        self.app_pipe, _ = pipe
        self.ws_check_interval = 250;
        self.after(self.ws_check_interval, self.ws_check)

    def join_channel(self, channel_str):
        self.app_pipe.send({
            'command': 'join',
            'data': {
                'channel': channel_str
            }
        })

    def ws_check(self):
        while self.app_pipe.poll():
            msg = self.app_pipe.recv()
            self.event_generate('<<ws-event>>', data=json.dumps(msg), when='tail')
        self.after(self.ws_check_interval, self.ws_check)

ws.py

import asyncio

import aiohttp


async def read_pipe(session, ws, ws_pipe):
    while True:
        while ws_pipe.poll():
            msg = ws_pipe.recv()

            # web socket send
            if msg['command'] == 'join':
                await ws.send_json(msg['data'])

            # html request
            elif msg['command'] == 'ticker':
                async with session.get('https://example.com/api/ticker/') as response:
                    ws_pipe.send({'event': 'ticker', 'data': await response.json()})

        await asyncio.sleep(.25)


async def main(pipe, loop):
    _, ws_pipe = pipe
    async with aiohttp.ClientSession() as session:
        async with session.ws_connect('wss://example.com/') as ws:
            task = loop.create_task(read_pipe(session, ws, ws_pipe))
            async for msg in ws:
                if msg.type == aiohttp.WSMsgType.TEXT:
                    if msg.data == 'close cmd':
                        await ws.close()
                        break
                    ws_pipe.send(msg.json())
                elif msg.type == aiohttp.WSMsgType.ERROR:
                    break

Shellfish

This is a high quality answer! I am not sure how Pipe works, but if there really is a new process created then the overhead should keept in mind. A new process come with a lot of overhead instead of just a new Thread. further details timber.io/blog/… – Gwyn

Answer 2

Trying to run both event loops at the same time is a dubious proposition. However, since root.mainloop simply calls root.update repeatedly, one can simulate mainloop by calling update repeatedly as an asyncio task. Here is a test program that does so. I presume adding asyncio tasks to the tkinter tasks would work. I checked that it still runs with 3.7.0a2.

"""Proof of concept: integrate tkinter, asyncio and async iterator.

Terry Jan Reedy, 2016 July 25
"""

import asyncio
from random import randrange as rr
import tkinter as tk


class App(tk.Tk):
    
    def __init__(self, loop, interval=1/120):
        super().__init__()
        self.loop = loop
        self.protocol("WM_DELETE_WINDOW", self.close)
        self.tasks = []
        self.tasks.append(loop.create_task(self.rotator(1/60, 2)))
        self.tasks.append(loop.create_task(self.updater(interval)))

    async def rotator(self, interval, d_per_tick):
        canvas = tk.Canvas(self, height=600, width=600)
        canvas.pack()
        deg = 0
        color = 'black'
        arc = canvas.create_arc(100, 100, 500, 500, style=tk.CHORD,
                                start=0, extent=deg, fill=color)
        while await asyncio.sleep(interval, True):
            deg, color = deg_color(deg, d_per_tick, color)
            canvas.itemconfigure(arc, extent=deg, fill=color)

    async def updater(self, interval):
        while True:
            self.update()
            await asyncio.sleep(interval)

    def close(self):
        for task in self.tasks:
            task.cancel()
        self.loop.stop()
        self.destroy()


def deg_color(deg, d_per_tick, color):
    deg += d_per_tick
    if 360 <= deg:
        deg %= 360
        color = '#%02x%02x%02x' % (rr(0, 256), rr(0, 256), rr(0, 256))
    return deg, color

loop = asyncio.get_event_loop()
app = App(loop)
loop.run_forever()
loop.close()

Both the tk update overhead and time resolution increase as the interval is decreased. For gui updates, as opposed to animations, 20 per second may be enough.

I recently succeeded in running async def coroutines containing tkinter calls and awaits with mainloop. The prototype uses asyncio Tasks and Futures, but I don't know if adding normal asyncio tasks would work. If one wants to run asyncio and tkinter tasks together, I think running tk update with an asyncio loop is a better idea.

EDIT: A least as used above, exception without async def coroutines kill the coroutine but are somewhere caught and discarded. Silent error are pretty obnoxious.

EDIT2: Additional code and comments at https://bugs.python.org/issue27546

Glide

Could you please adapt my MWE to your solution. – Gwyn

At least for the present, I leave it to you to experiment. – Glide

Then your answer does not fit to my question and dies not help other readers what would be the main goal of stackexchange. – Gwyn

The answer does help other people. – Overabound

I recently found a different way to drive coroutines with tk.mainloop that does not used anything from asyncio. It should work better for applications that do not use asyncio. For applications that use both, update (my answer) or update_idletasks(martineau's answer, and likely better if sufficient) in the main thread or asynio loop in a separate thread (oat's answer) are possible solutions. I don't plan to experiment more with using mainloop when using both (asyncio's loop was faster in one experiment I did). – Glide

A minor suggestion: As the first action in updater, it may make sense to call the undocumented, but apparently public, self.tk.willupdate(). That sets a flag otherwise set only by mainloop, to indicate that the loop is running; without it, attempts to invoke Tk methods from other threads will die with an error, instead of marshaling the command from the other thread to the main thread. – Gi

Similarly, tk._tkinter.getbusywaitinterval() gets the time, in milliseconds, that the mainloop waits between attempts to update; you could use that (divided by 1000 to convert from millis to fractional seconds) as a decent default for the asyncio.sleep call in updater, to match tkinter more precisely. – Gi

Blech, in my first comment, I said willupdate when I meant willdispatch. Sorry for confusion. – Gi

I've recently tried this technique of using an asyncio loop and then updating tk with root.update() but it appears to cause a "memory leak". Even running Terry's example code shows rising memory consumption with each update. – Fourflusher

@chmedly: How much of a leak are we talking? I've got a monitoring GUI I use daily (started at ~8 AM, running until ~4 PM) that uses a trivial variation on Terry's example code (explicitly calling self.tk.willdispatch() before the while True: loop, and prebinding names to trivially reduce overhead, so I do tkupdate = self.update and sleep = functools.partial(asyncio.sleep, 0.02, loop=self.loop) where self.loop is a cached reference to the event loop, making the body of the loop just tkupdate() followed by await sleep(). Memory usage (virtual and resident in top) remains stable. – Gi

@ShadowRanger. I ran Terry's proof code as it is on this page with Python 3.7.3 on Mac. I see about 32MiB of usage at the start and 143MiB at 500 secs when tracking with mprof. It's basically a steadily climbing graph. I also tried it with the willdispatch(). Perhaps I'm not using it correctly but I see no difference in behavior or memory use. I've also run other pieces of example code on the web where I see this same rise in memory use when using the async loop to drive tk.update(). I'm not sure if this problem is actually caused by calling tk.update() but appears to me to be so. – Fourflusher

Conversely, I found this server/client example code for asyncio (with tkinter) that uses threading instead of the tk.update() approach described by Terry. github.com/cjrh/chat I'm not sure that this is part of the asyncio documentation but I think it's in the proposal stage at the moment. So, I'm getting the impression that threading will be the Python recommended way of running tkinter with asyncio. – Fourflusher

The github.com/cjrh/chat client runs the tkinter gui in the main thread and the asyncio loop for socket communication in a separate thread. This should work fine as long as the traffic from the server does not hog the cpu, which should not happen. tcl/tk has a unix-only function to read/write disk functions using a poll loop in a separate thread. On unix, asyncio uses the same unix poll functions. Its main innovation is adding a Windows proactor loop with mostly the same API (not trivial at all). – Glide

Update: This memory issue appears to be limited to MacOS. I've run the code in Linux Mint with Python 3.7.3 and the memory flat lines after about 30 seconds. And it appears to be the same on Windows 10. I suppose this means that which method you choose might depend on the platform you expect to deploy to.. – Fourflusher

And another Update: I think I've narrowed this memory issue down to the Tk version 8.5 that comes with MacOs. Upgrading to 8.6 fixes the issue. Unfortunately, if you install Python3 with homebrew, getting an upgraded Tk to connect to Python is very tricky. If you install Python3 from Python.org you will get the right stuff. – Fourflusher

Note that one drawback of updating the GUI as an asyncio task is that tk.update() blocks until all events are processed, thus suspending the execution of other coroutines. This can be reproduced by resizing the window for a few seconds, and the canvas arc will stop updating. – Katiakatie

Answer 3

In a slight modification to your code, I created the asyncio event_loop in the main thread and passed it as an argument to the asyncio thread. Now Tkinter won't freeze while the urls are fetched.

from tkinter import *
from tkinter import messagebox
import asyncio
import threading
import random

def _asyncio_thread(async_loop):
    async_loop.run_until_complete(do_urls())


def do_tasks(async_loop):
    """ Button-Event-Handler starting the asyncio part. """
    threading.Thread(target=_asyncio_thread, args=(async_loop,)).start()

    
async def one_url(url):
    """ One task. """
    sec = random.randint(1, 8)
    await asyncio.sleep(sec)
    return 'url: {}\tsec: {}'.format(url, sec)

async def do_urls():
    """ Creating and starting 10 tasks. """
    tasks = [one_url(url) for url in range(10)]
    completed, pending = await asyncio.wait(tasks)
    results = [task.result() for task in completed]
    print('\n'.join(results))


def do_freezed():
    messagebox.showinfo(message='Tkinter is reacting.')

def main(async_loop):
    root = Tk()
    Button(master=root, text='Asyncio Tasks', command= lambda:do_tasks(async_loop)).pack()
    Button(master=root, text='Freezed???', command=do_freezed).pack()
    root.mainloop()

if __name__ == '__main__':
    async_loop = asyncio.get_event_loop()
    main(async_loop)

Diverticulitis

What is the reason for calling asyncio.get_event_loop() in the main thread instead of in the worker thread? – Annabal

Why did you use buttonX = Button(...).pack()? Please look at this to see the problem. – Gender

@Gender - Yes you are right about it. However since we do not use buttonX variable after that line, it really doesn't mater in this example. – Diverticulitis

@Diverticulitis It can confuse people that see this in the future. It's better to just use Button(...).pack(). So many new people make this mistake. – Gender

Answer 4

I'm a bit late to the party but if you are not targeting Windows you can use aiotkinter to achieve what you want. I modified your code to show you how to use this package:

from tkinter import *
from tkinter import messagebox
import asyncio
import random

import aiotkinter

def do_freezed():
    """ Button-Event-Handler to see if a button on GUI works. """
    messagebox.showinfo(message='Tkinter is reacting.')

def do_tasks():
    task = asyncio.ensure_future(do_urls())
    task.add_done_callback(tasks_done)

def tasks_done(task):
    messagebox.showinfo(message='Tasks done.')

async def one_url(url):
    """ One task. """
    sec = random.randint(1, 15)
    await asyncio.sleep(sec)
    return 'url: {}\tsec: {}'.format(url, sec)

async def do_urls():
    """ Creating and starting 10 tasks. """
    tasks = [
        one_url(url)
        for url in range(10)
    ]
    completed, pending = await asyncio.wait(tasks)
    results = [task.result() for task in completed]
    print('\n'.join(results))

if __name__ == '__main__':
    asyncio.set_event_loop_policy(aiotkinter.TkinterEventLoopPolicy())
    loop = asyncio.get_event_loop()
    root = Tk()
    buttonT = Button(master=root, text='Asyncio Tasks', command=do_tasks)
    buttonT.pack()
    buttonX = Button(master=root, text='Freezed???', command=do_freezed)
    buttonX.pack()
    loop.run_forever()

Rice

Answer 5

You can keep the GUI alive after pressing the Button by adding a call to root.update_idletasks() in the right spot:

from tkinter import *
from tkinter import messagebox
import asyncio
import random

def do_freezed():
    """ Button-Event-Handler to see if a button on GUI works. """
    messagebox.showinfo(message='Tkinter is reacting.')

def do_tasks():
    """ Button-Event-Handler starting the asyncio part. """
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(do_urls())
    finally:
        loop.close()

async def one_url(url):
    """ One task. """
    sec = random.randint(1, 15)
    root.update_idletasks()  # ADDED: Allow tkinter to update gui.
    await asyncio.sleep(sec)
    return 'url: {}\tsec: {}'.format(url, sec)

async def do_urls():
    """ Creating and starting 10 tasks. """
    tasks = [one_url(url) for url in range(10)]
    completed, pending = await asyncio.wait(tasks)
    results = [task.result() for task in completed]
    print('\n'.join(results))


if __name__ == '__main__':
    root = Tk()

    buttonT = Button(master=root, text='Asyncio Tasks', command=do_tasks)
    buttonT.pack()
    buttonX = Button(master=root, text='Freezed???', command=do_freezed)
    buttonX.pack()

    root.mainloop()

Greaseball

It is not working. The behaviour doesn't changed. Have you tested your solution with my MWE? – Gwyn

Yes, I ran it with the code in your question—whatever it was when I posted my answer anyway (in case you've changed it since then). – Greaseball

In that case would you please post your complete working code. – Gwyn

swdmnd: Of course it does, a fact @Gwyn never acknowledged after I posted the complete working code. – Greaseball

Answer 6

A solution using async_tkinter_loop module (which is written by me).

Internally, the approach is similar to the code from the answer of Terry Jan Reedy, but the usage is much simpler: you just need to wrap your asynchronous handlers into async_handler function calls, and use them as a command or an event handlers, and use async_mainloop(root) in place of root.mainloop().

from tkinter import *
from tkinter import messagebox
import asyncio
import random
from async_tkinter_loop import async_handler, async_mainloop


def do_freezed():
    """ Button-Event-Handler to see if a button on GUI works. """
    messagebox.showinfo(message='Tkinter is reacting.')


async def one_url(url):
    """ One task. """
    sec = random.randint(1, 15)
    await asyncio.sleep(sec)
    return 'url: {}\tsec: {}'.format(url, sec)


async def do_urls():
    """ Creating and starting 10 tasks. """
    tasks = [
        asyncio.create_task(one_url(url))  # added create_task to remove warning "The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11."
        for url in range(10)
    ]
    print("Started")
    completed, pending = await asyncio.wait(tasks)
    results = [task.result() for task in completed]
    print('\n'.join(results))
    print("Finished")


if __name__ == '__main__':
    root = Tk()

    # Wrap async function into async_handler to use it as a button handler or an event handler
    buttonT = Button(master=root, text='Asyncio Tasks', command=async_handler(do_urls))
    buttonT.pack()
    buttonX = Button(master=root, text='Freezed???', command=do_freezed)
    buttonX.pack()

    # Use async_mainloop(root) instead of root.mainloop()
    async_mainloop(root)

Concuss

Answer 7

Using Python3.9, it could be done by making several async functions with one of them responsible to the Tk update(). While in the main loop, ensure_future() can be used to invoke all these async functions before starting the asyncio loop.

#!/usr/bin/env python3.9

import aioredis
import asyncio
import tkinter as tk 
import tkinter.scrolledtext as st 
import json

async def redis_main(logs):
    redisS = await aioredis.create_connection(('localhost', 6379))  
    subCh = aioredis.Channel('pylog', is_pattern=False)
    await redisS.execute_pubsub('subscribe', subCh)
    while await subCh.wait_message():
            msg = await subCh.get()
            jmsg = json.loads(msg.decode('utf-8'))
            logs.insert(tk.INSERT, jmsg['msg'] + '\n')

async def tk_main(root):
    while True:
        root.update()
        await asyncio.sleep(0.05)

def on_closing():
    asyncio.get_running_loop().stop()

if __name__ == '__main__':
    root = tk.Tk()
    root.protocol("WM_DELETE_WINDOW", on_closing)
    logs = st.ScrolledText(root, width=30, height=8)
    logs.grid()
    
    tkmain = asyncio.ensure_future(tk_main(root))
    rdmain = asyncio.ensure_future(redis_main(logs))
    
    loop = asyncio.get_event_loop()
    try:
        loop.run_forever()
    except KeyboardInterrupt:
        pass

    tkmain.cancel()
    rdmain.cancel()

Sudarium

It is a question not an answer. – Gwyn

@buhtz, the style is in form of a question but that is actually a solution. Did you spend sometime to look at it? – Sudarium

As written, it’s not clear that it’s an answer. Can you edit your wording so future readers aren’t confused? – Dekker

Answer 8

I tried this module (https://github.com/davidhozic/Tkinter-Async-Execute) and it seemed to work well. I'm not much of a programmer and the other answers on this page are probably good but are a little beyond my capability to implement into my code.

Disconsider

Answer 9

I've had great luck running an I/O loop on another thread, started at the beginning of the app creation, and tossing tasks onto it using asyncio.run_coroutine_threadsafe(..).

I'm kind of surprised that I can make changes to the tkinter widgets on the other asyncio loop/thread, and maybe it's a fluke that it works for me -- but it does work.

Notice that while the asyncio tasks are happening, the other button is still alive and responding. I always like to the disable/enable thing on the other button so you don't fire off multiple tasks accidentally, but that's just a UI thing.

import threading
from functools import partial
from tkinter import *
from tkinter import messagebox
import asyncio
import random


# Please wrap all this code in a nice App class, of course

def _run_aio_loop(loop):
    asyncio.set_event_loop(loop)
    loop.run_forever()
aioloop = asyncio.new_event_loop()
t = threading.Thread(target=partial(_run_aio_loop, aioloop))
t.daemon = True  # Optional depending on how you plan to shutdown the app
t.start()

buttonT = None

def do_freezed():
    """ Button-Event-Handler to see if a button on GUI works. """
    messagebox.showinfo(message='Tkinter is reacting.')

def do_tasks():
    """ Button-Event-Handler starting the asyncio part. """
    buttonT.configure(state=DISABLED)
    asyncio.run_coroutine_threadsafe(do_urls(), aioloop)

async def one_url(url):
    """ One task. """
    sec = random.randint(1, 3)
    # root.update_idletasks()  # We can delete this now
    await asyncio.sleep(sec)
    return 'url: {}\tsec: {}'.format(url, sec)

async def do_urls():
    """ Creating and starting 10 tasks. """
    tasks = [one_url(url) for url in range(3)]
    completed, pending = await asyncio.wait(tasks)
    results = [task.result() for task in completed]
    print('\n'.join(results))
    buttonT.configure(state=NORMAL)  # Tk doesn't seem to care that this is called on another thread


if __name__ == '__main__':
    root = Tk()

    buttonT = Button(master=root, text='Asyncio Tasks', command=do_tasks)
    buttonT.pack()
    buttonX = Button(master=root, text='Freezed???', command=do_freezed)
    buttonX.pack()

    root.mainloop()

Pfaff