How to implement a python REPL that nicely handles asynchronous output?

Question

I have a Python-based app that can accept a few commands in a simple read-eval-print-loop. I'm using raw_input('> ') to get the input. On Unix-based systems, I also import readline to make things behave a little better. All this is working fine.

The problem is that there are asynchronous events coming in, and I'd like to print output as soon as they happen. Unfortunately, this makes things look ugly. The "> " string doesn't show up again after the output, and if the user is halfway through typing something, it chops their text in half. It should probably redraw the user's text-in-progress after printing something.

This seems like it must be a solved problem. What's the proper way to do this?

Also note that some of my users are Windows-based.

TIA

Edit: The accepted answer works under Unixy platforms (when the readline module is available), but if anyone knows how to make this work under Windows, it would be much appreciated!

Rubefaction

Answer 1

Maybe something like this will do the trick:

#!/usr/bin/env python2.6

from __future__ import print_function

import readline
import threading

PROMPT = '> '

def interrupt():
    print() # Don't want to end up on the same line the user is typing on.
    print('Interrupting cow -- moo!')
    print(PROMPT, readline.get_line_buffer(), sep='', end='')

def cli():
    while True:
        cli = str(raw_input(PROMPT))

if __name__ == '__main__':
    threading.Thread(target=cli).start()
    threading.Timer(2, interrupt).start()

I don't think that stdin is thread-safe, so you can end up losing characters to the interrupting thread (that the user will have to retype at the end of the interrupt). I exaggerated the amount of interrupt time with the time.sleep call. The readline.get_line_buffer call won't display the characters that get lost, so it all turns out alright.

Note that stdout itself isn't thread safe, so if you've got multiple interrupting threads of execution, this can still end up looking gross.

Coincidentally

Thanks, readline.get_line_buffer() is a good start. If anyone has any tips on making this work in Windows (which has no readline module), let me know. – Rubefaction

Is this still the way, or a close approximation thereof, to do it in 2021? Anything I should know that has changed? I'm making a discord bot using discord.py and am running into the same issue as the asker. I get non-blocking input using aioconsole, but I can't seem to get it without different input/output sources interleaving one another in visually unpleasing ways – Boss

Answer 2

Why are you writing your own REPL using raw_input()? Have you looked at the cmd.Cmd class? Edit: I just found the sclapp library, which may also be useful.

Note: the cmd.Cmd class (and sclapp) may or may not directly support your original goal; you may have to subclass it and modify it as needed to provide that feature.

Perkins

cmd.Cmd looks pretty awesome and I should probably start using it. However, it doesn't seem to accomplish my original goal, any tips? – Rubefaction

sclapp seems to be defunct. The link provided didn't work, and I've not found its new home. – Promenade

Answer 3

run this:

python -m twisted.conch.stdio

You'll get a nice, colored, async REPL, without using threads. While you type in the prompt, the event loop is running.

Warship

Answer 4

Another method is to use the prompt_toolkit library (I believe this is what IPython uses internally).

Example from the linked answer:

from prompt_toolkit import prompt
from prompt_toolkit.patch_stdout import patch_stdout
import threading

def fn():
    print('async event')

with patch_stdout():
    threading.Timer(3, fn).start()
    a = prompt('Cli> ')

Murguia

Answer 5

look into the code module, it lets you create objects for interpreting python code also (shameless plug) https://github.com/iridium172/PyTerm lets you create interactive command line programs that handle raw keyboard input (like ^C will raise a KeyboardInterrupt).

Ignoble

Answer 6

It's kind of a non-answer, but I would look at IPython's code to see how they're doing it.

Compartment

Answer 7

I think you have 2 basic options:

1. Synchronize your output (i.e. block until it comes back)
2. Separate your input and your (asyncronous) output, perhaps in two separate columns.

Groundhog