Jupyter notebook, how to run multiple cells simultaneously?

Question

I defined a python function which run a bash script. Let's say the function is: calc(x,y,z). If I run this function in python with some variables,

>>> calc(1,2,3)

It generates a C code which simulate something using the variables (x=1, y=2, z=3), compiles the C code and executes the compiled output file.

I want to run multiple calc(x,y,z)s with different (x,y,z)s in jupyter notebook simultaneously. As you may noticed, the problem is that cells in jupyter notebook are executed sequentially. If I run three calc functions, it takes three times longer than the one function running time.

I tried two ways but they didn't work well.

1. Use multiprocessing module: By using the module, it is possible to execute multiple calcs simultaneously in "one cell". But for later analysis, I would like to execute multiple cells simultaneously which include only one calc each using different processors (or cpu cores).

2. Use ipyparallel cell magic (inspired by this answer): I tried as following after import ipyparallel

   # Cell 1
   %%px --targets 0 # use processor 0
   calc(1,1,1)
   

   .

   # Cell 2
   %%px --targets 1 # use processor 1
   calc(2,2,2)        
   

   .

   # Cell 3
   %%px --targets 2 # use processor 2
   calc(3,3,3) 
   

But the cells are executed sequentially: Cell 2 is executed after Cell 1 simulation was done, similar for Cell 3.

How to run multiple jupyter cells using different cores?

Pianissimo

Using different cores is not a parallel processing. It is a just concurrent, uncoordinated, independently run coincidence of processing flows. – Pirouette

Answer 1

In your solution, the cells are executed by different engines as you expected. The issue is caused by default blocking behavior. You can simply add --noblock argument to execute a cell in non-blocking mode. Then the cell returns AsyncResult object that makes it possible to read the output once the execution is finished by calling method display_outputs(). Go to the documentation for details targets-and-blocking

# Cell 1
%%px --targets 0 --noblock
calc(1,1,1)

.

# Cell 2
%%px --targets 1 --noblock
calc(2,2,2)   

.

# Cell 3
%%px --targets 2 --noblock
calc(3,3,3) 

If you need to access the output you can call display_outputs() as I explained above.

# output of the first the cell 1
___.display_outputs()

# output of the first the cell 2
__.display_outputs()

# output of the first the cell 1
_.display_outputs()

I used underscore notation to access AsyncResult objects returned by the cells 1-3. There is a number of other ways to access those objects, for example by using Out[x] where x is the cell's execution number visible in the notebook after executing a cell.

Preface

Answer 2

Maybe my problem is your answer! read this thread. because I run jupyter cells via looping javascript module, they run almost simultaneously. if you know number of cells you want to run, just customize cell_n such as cell_n = [0,1,2] (the first three cells) and update cell_n value in cell B. Alternatively, you can also put a similar keyword for each necessary cell to be run together following my routine work: running cell A once >>> modifying text widget >>> running cell B.

Cell A

### trying to make Jupyter ipynb to run cells containing especial keywords
# cell A >> preparing functions and the input widget
# kw is short form of keyword(s)

import numpy as np 
import codecs
import ipynbname  #pip install ipynbname
from IPython.display import Javascript, display
import ipywidgets as widgets
from ipywidgets import Layout, Textarea



ID = '5oG9gNy7myn/k86/+/8cemphdkEsK237SjQltj2GZGU=' # unique ID for finding current code cell number inside ipynb file


def indice_matcher(j01,j02): # if positive difference of j02[n] and j01[n] is less than j01[n] and j01[+1] then they have matching indices
    '''j01 has the bigger range '''
    jj = []
    for j1 in range(len(j01)-1):
        for j2 in range(len(j02)):
            if j02[j2] - j01[j1] < j01[j1+1] - j01[j1] and j02[j2] - j01[j1] > 0: 
                jj.append(j1)
            
    return jj  # returns list indices of matched indices

def kwsearch(x,char):  # x is the big string and char is the small string
    f = 0
    i = []
    f = x.find(char)
    i.append(f) 
    while f != -1:
        f = x[i[-1]+1:].find(char)
        i.append(f+1+i[-1]) 
    
    return i[:-1] # returns all indices of ONE searched character inside x string

def flat(lis):
    ''' # Converts a nested list into a flat list; 
    source : https://www.geeksforgeeks.org/python-convert-a-nested-list-into-a-flat-list/'''
    flatList = []
    # Iterate with outer list
    for element in lis:
        if type(element) is list:
            # Check if type is list than iterate through the sublist
            for item in element:
                flatList.append(item)
        else:
            flatList.append(element)
    return flatList

def kwrun(kw_string):
    kw_string.append(ID)
    N = len(kw_string)
    # nb_fname = ipynbname.name()  # current ipynb file's name
    nb_path = ipynbname.path()  # current ipynb file's path
    f=codecs.open(nb_path, 'r')
    current_ipynb = f.read()    

    kw_indices = [] # indices of all inputted keywords
    for n in range(N): # the loop repeats itself till N keywords are searched
        kw_indices.append(kwsearch(current_ipynb,kw_string[n]))

    cell_indices = kwsearch(current_ipynb,'"cell_type"')   # searching "cell_type" inside ipynb file helps us to find indices for all cells 
    # calculating selected cell number
    cell_n = [] 
    for n in range(N): # the loop repeats till N kewords are searched
        cell_n.append(indice_matcher(cell_indices,kw_indices[n]))  # returns list of cell numbers that should be run ; # cell numbers in this way is equal to cell_indices' number of each index
    cell_n = flat(cell_n)
    print(f'Running Cells: {cell_n[:-1]} \n current code cell number: {cell_n[-1]}')    
    return cell_n


w1 = widgets.Text(description='Keywords') # an elegant way to input keywords
display(w1)

Cell B

# cell B >> search and run
cell_n = kwrun((w1.value).split()) # w1.value is value of inputted words inside cell 1 widget; the keywords should be separated by space
for n in range(len(cell_n)-1): # the -1 in range removes loops to run current code cell number (cell A)
    display(Javascript(f'IPython.notebook.execute_cells([{str(cell_n[n])}])')) #if a name is not found, no cell will be run

Positively