redirect sys.stdout to specific Jupyter Notebook cell

Question

Jupyter==4.1.0, Python==2.7.10, IPython==4.2.0

I'm writing a SQL UI for my Jupyter Notebooks and would like to incorporate multithreading so that I can run a query in one cell and continue to work in other cells while the query is running.

The problem I'm having is that if I execute a query in one cell, the output will be displayed in the last-executed cell's output prompt instead of in the output prompt of the cell that executed the query.

I scoured the interwebs and discovered this clever trick, but I think it's outdated and/or no longer works in my version of Jupyter. When I run it, I only get output for whatever cell was last executed. So if I run both, I only get the last-executed output, instead of the output printing to separate cells simultaneously.

So I have my context manager which sets the parent_header:

import sys
import threading
from contextlib import contextmanager

# we need a lock so that other threads don't snatch control
# while we have set a temporary parent
stdout_lock = threading.Lock()

@contextmanager
def set_stdout_parent(parent):
    """a context manager for setting a particular parent for sys.stdout 
    the parent determines the destination cell of the output
    """
    save_parent = sys.stdout.parent_header
    with stdout_lock:
        sys.stdout.parent_header = parent
        try:
            yield
        finally:
            # the flush is important, because that's when the parent_header actually has its effect
            sys.stdout.flush()
            sys.stdout.parent_header = save_parent

I essentially want to be able to get the parent_header of a cell In[1] and redirect the output of cell In[2] to the output of In[1].

Example:

Get parent_header of In[1]:

In[1]: t = sys.stdout.parent_header

Then the following code will run, but the output should print to Out[1] (currently, I get no output when I run this code):

In [2]: with set_stdout_parent(t):
            print 'FOO'

Which should produce:

In[1]: t = sys.stdout.parent_header
Out[1]:'FOO'

Seyler

My subpar fix for this is to run the query on its on thread and print out an HTML container with a unique id, then when the query finishes, use IPython's display and Javascript functions to append it to the appropriate DOM object. – Seyler

How about saving the output to a file then in another cell opening and printing that file? – Lynching

Have you tried something like this – Reverent

how about this: ipykernel.iostream.OutStream ? – Polygamous

Answer 1

The documentation for ipywidgets.Output has a section about interacting with output widgets from background threads. Using the Output.append_stdout method there is no need for locking. The final cell in this answer can then be replaced with:

def t1_main():
    for i in range(10):
        output1.append_stdout(f'thread1 {i}\n')
        time.sleep(0.5)


def t2_main():
    for i in range(10):
        output2.append_stdout(f'thread2 {i}\n')
        time.sleep(0.5)

output1.clear_output()
output2.clear_output()
        
t1 = Thread(target=t1_main)
t2 = Thread(target=t2_main)
t1.start()
t2.start()
t1.join()
t2.join()

Condillac

Answer 2

You can use a combination of ipywidgets.Output (docs) and locking:

Code in jupyter cells:

# In[1]:


from threading import Thread, Lock
import time
from ipywidgets import Output


# In[2]:


output1 = Output()
output1


# In[3]:


output2 = Output()
output2


# In[4]:


print_lock = Lock()
def t1_main():    
    for i in range(10):
        with print_lock, output1:
            print('thread1', i)
        time.sleep(0.5)

def t2_main():
    for i in range(10):
        with print_lock, output2:
            print('thread2', i)
        time.sleep(0.5)

output1.clear_output()
output2.clear_output()
        
t1 = Thread(target=t1_main)
t2 = Thread(target=t2_main)
t1.start()
t2.start()
t1.join()
t2.join()

Condillac