Simple way to choose which cells to run in ipython notebook during run all

Question

I have an ipython notebook that runs several steps in a data processing routine and saves information in files along the way. This way, while developing my code (mostly in a separate .py module), I can skip to and run various steps. I'd like to set it up so that I can Cell->run all but only have it execute certain chosen steps that would be easily chosen. e.g., I'd envision defining the steps I want to run in a dict like so:

process = {
    'load files':False,
    'generate interactions list':False,
    'random walk':True,
    'dereference walk':True,
    'reduce walk':True,
    'generate output':True
}

then the steps would run based on this dict. BTW, each step comprises multiple cells.

I think %macro is not quite what I want since anytime I changed anything or restarted the kernel I'd have to redefine the macro, with changing cell numbers.

Is there like a %skip or %skipto magic or something along those lines? Or perhaps a clean way to put at the beginning of cells, if process[<current step>]: %dont_run_rest_of_cell?

Canard

I have the same need - I use notebooks as a template for automatically generated reports. I want to be able to define which sections of my notebook are executed based on some condition, such as whether a certain input file exists (i.e. if this file is provided, run the next 6 cells). The idea reminds me of #define, #ifdef compiler macros from C-family languages. – Immemorial

Answer 1

You can create your own skip magic with the help of a custom kernel extension.

skip_kernel_extension.py

def skip(line, cell=None):
    '''Skips execution of the current line/cell if line evaluates to True.'''
    if eval(line):
        return

    get_ipython().ex(cell)

def load_ipython_extension(shell):
    '''Registers the skip magic when the extension loads.'''
    shell.register_magic_function(skip, 'line_cell')

def unload_ipython_extension(shell):
    '''Unregisters the skip magic when the extension unloads.'''
    del shell.magics_manager.magics['cell']['skip']

Load the extension in your notebook:

%load_ext skip_kernel_extension

Run the skip magic command in the cells you want to skip:

%%skip True  #skips cell
%%skip False #won't skip

You can use a variable to decide if a cell should be skipped by using $:

should_skip = True
%%skip $should_skip

Vasyuta

Cool, but where does one put a "custom kernel extension"? – Paginate

@Arthur: You can put your extension modules anywhere you want, as long as they can be imported by Python’s standard import mechanism. However, to make it easy to write extensions, you can also put your extensions in extensions/ within the IPython directory. This directory is added to sys.path automatically. – Vasyuta

instead of get_ipython().ex(cell) better get_ipython().run_cell(cell) – Jinx

Answer 2

If you are using nbconvert to execute your notebook, you can write a custom preprocessor that looks at cell metadata to know which cells to execute.

class MyExecutePreprocessor(nbconvert.preprocessors.ExecutePreprocessor):

    def preprocess_cell(self, cell, resources, cell_index):
        """
        Executes a single code cell. See base.py for details.
        To execute all cells see :meth:`preprocess`.

        Checks cell.metadata for 'execute' key. If set, and maps to False, 
          the cell is not executed.
        """

        if not cell.metadata.get('execute', True):
            # Don't execute this cell in output
            return cell, resources

        return super().preprocess_cell(cell, resources, cell_index)

By editing cell metadata, you can specify whether that cell should be executed.

You can get fancier by adding a master dictionary to your notebook metadata. This would look like the dictionary in your example, mapping sections to a boolean specifying whether that section would be called.

Then, in your cell metadata, you can use a "section" keyword mapping to the section ID in your notebook metadata.

When executing nbconvert, you can tell it to use your preprocessor.

See the docs on Notebook preprocessors for more information.

Immemorial

I added a few more details when responding to this related question: #33518400. – Immemorial

Thanks @Gordon Bean. What are the most convenient ways to edit cell metadata? Is there some way to do it with a command in a Jupyter notebook? – Perplex

@Perplex - you can edit the metadata via the notebook editor. I'm not sure if there is a way to edit the metadata using a command (though it sounds like a great question for SO ;). – Immemorial

Answer 3

I am new to Jupyter Notebook and am loving it. I had heard of IPython before but didn't look into it seriously until a recent consulting job.

One trick my associate showed me to disable blocks from execution is to change them from "Code" type to "Raw NBConvert" type. This way I sprinkle diagnostic blocks through my notebook, but only turn them on (make them "Code") if I want them to run.

This method isn't exactly dynamically selectable in a script, but may suit some needs.

Disbranch

Answer 4

Adding to what Robbe said above (I can't comment because I'm new), you could just do the following in your first cell if you don't want to create a custom extension that you might just forget about:

def skip(line, cell=None):
    '''Skips execution of the current line/cell if line evaluates to True.'''
    if eval(line):
        return

    get_ipython().ex(cell)

def load_ipython_extension(shell):
    '''Registers the skip magic when the extension loads.'''
    shell.register_magic_function(skip, 'line_cell')

def unload_ipython_extension(shell):
    '''Unregisters the skip magic when the extension unloads.'''
    del shell.magics_manager.magics['cell']['skip']
    
    
load_ipython_extension(get_ipython())

Maiocco

instead of get_ipython().ex(cell) better get_ipython().run_cell(cell) – Jinx

Answer 5

You can use nbconvert and tags option in metadata: In my case I edited the cell metadata:

{
    "deletable": true,
    "colab_type": "code",
    "id": "W9i6oektpgld",
    "tags": [
        "skip"
    ],
    "colab": {},
    "editable": true
}

Create a preprocess.py file.

from nbconvert.preprocessors import Preprocessor

class RemoveCellsWithNoTags(Preprocessor):
    def preprocess(self, notebook, resources):
        executable_cells = []
        for cell in notebook.cells:
            if cell.metadata.get('tags'):
                if "skip" in cell.metadata.get('tags'):
                    continue
            executable_cells.append(cell)
        notebook.cells = executable_cells
        return notebook, resources

Then export notebook:

jupyter nbconvert --Exporter.preprocessors=[\"preprocess.RemoveCellsWithNoTags\"] --ClearOutputPreprocessor.enabled=True --to notebook --output=getting-started-keras-test getting-started-keras.ipynb

Won

Answer 6

Explicit is always better that implicit. Simple is better than complicated. So why don't use plain python?

With one cell per step you can do:

if process['load files']:
    load_files()
    do_something()

and

if process['generate interactions list']:
    do_something_else()

If you want to stop the execution when a particular step is skipped you could use:

if not process['reduce walk']:
    stop
else:
    reduce_walk()
    ...

stop is not a command so it will generate an exception and stop the execution when using Cell -> Run all.

You can also make conditional steps like:

if process['reduce walk'] and process['save output']:
    save_results()
    ...

But, as a rule of thumb, I wouldn't make conditions that are much more complex than that.

Selenium

Answer 7

Or from another point of view, you can skip the cells that you do not want to run (i.e. by adding the following code at the first line of your cell that needs to be skipped).

%%script echo skipping

Creolized

Answer 8

Adding on to @Robbe's answer (after reading the Defining custom magics tutorial). You can also make it a class with a register method:

# path/to/my_jupy_pkg.magics.py

from IPython.core.magic import magics_class, line_cell_magic, Magics
from ipywidgets import get_ipython

@magics_class
class SkipMagic(Magics):
    '''
    A class that provides a magic to skip the execution of a cell based on a boolean value.
    '''
    def __init__(self, shell):
        super().__init__(shell=shell)
    
    @line_cell_magic
    def skip(self, line, cell=None):
        '''
        Skips execution of the current line/cell if line evaluates to True.
        
        Parameters
        ----------
        line : str
            The line to evaluate for skipping.
        cell : str, optional
            The cell contents to execute if line evaluates to False.
            
        Returns
        -------
        None
        '''
        if eval(line, self.shell.user_ns):
            return

        if cell:  # Only execute if there's cell content to execute
            get_ipython().ex(cell)


    def load_ipython_extension(self, shell):
        '''
        Registers the skip magic when the extension loads.
        
        Parameters
        ----------
        shell : IPython InteractiveShell
            The IPython shell instance to which the magic should be registered.
            
        Returns
        -------
        None
        '''
        shell.register_magics(self)

    def unload_ipython_extension(self, shell):
        '''
        Unregisters the skip magic when the extension unloads.
        
        Parameters
        ----------
        shell : IPython InteractiveShell
            The IPython shell instance from which the magic should be unregistered.
            
        Returns
        -------
        None
        '''
        del shell.magics_manager.magics['cell']['skip']
        del shell.magics_manager.magics['line']['skip']


    @classmethod
    def register(cls) -> 'SkipMagic':
        ipy = get_ipython()
        ins = cls(shell=ipy)
        ipy.register_magics(ins)
        return ins

As before one can import and run:

from my_jupy_pkg.magics import SkipMagic
mgc = SkipMagic.register()

Electroanalysis