How to Copy from IPython session without terminal prompts

Question

Frequently, my workflow involves data cleaning/munging in an IPython shell. This has become particularly wonderful since IPython version 5.0 with all the great upgrades to the terminal interface. So, let's say I make an attempt at sprucing up some piece of unstructured data:

In [11]: for i, (num, header, txt) in enumerate(data):
    ...:     header = [e.strip() for e in header.strip().split('\n')]
    ...:     header[4] = header[4].strip(',').split(',')
    ...:     data[i] = (num, header, txt)
    ...:

Fantastic, it works! But now, I would really like to add this to a script in my editor. If I copy and paste from my terminal, I capture all the junk on the left. I can clean this up more-or-less easily in an editor, but it would be great if I could copy the code directly to my clipboard from the terminal without touching the mouse and without grabbing the extra stuff either. Is there such a functionality in IPython?

Malvasia

Looks like people have implemented the feature, but it doesn't seem to come with IPython. – Outbuilding

If you don't want to customize your IPython setup, print(In[11]) should give you something to copy without the junk on the left. – Outbuilding

@Outbuilding Hm, although that isn't ideal, that could work for me. If you add that as an answer, and nothing better shows up, I'll accept. – Malvasia

Not really an answer to this question, but I tend to do the reverse – type the code in vim, then use jupyter-vim plugin to send the code to jupyter to execute it. That way the code is already in vim, no need to copy. – Lynching

@Lynching this isn't about jupyter, this is about the IPython REPL. The whole point is using the REPL to experiment. I dont' generally use Jupyter notebooks. – Malvasia

I meant send the code from vim to IPython console (or jupyter console). Despite the plugin name, it works with both. – Lynching

Answer 1

You can use the %history magic to extract the interesting parts from your session. They will be shown in terminal without any of the junk.

Example

In [1]: import numpy as np    
In [2]: a = np.random(10)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-83ce219ad17b> in <module>()
----> 1 a = np.random(10)

TypeError: 'module' object is not callable

In [3]: a = np.random.random(10)
In [4]: for i in a:
   ...:     print(i)
   ...:     
0.688626523886
[...]
0.341394850998

If I want to save a part of the session above I can use:

In [5]: %history 1 3-4

import numpy as np
a = np.random.random(10)
for i in a:
    print(i)

In the example above I used %history 1 3-4 to assemble all the commands I want to keep and omit the ones I do not need (Line 2, the one with the error). Now you have version of your session that can be nicely copied.

Writing a file

You can also directly write this to file using the -f FILENAME as parameter.

In [8]: %history 1 3-4 -f /tmp/foo.py

Be careful though, this will overwrite existing files. More Details can be found in the documentation of the %history magic.

Ence

It's a useful function, although it also the gives the commands which we use in output, example if I use %history it will give that as well next time. I think that's what history does. – Hillhouse

Answer 2

So, I have finally found a great solution that is essentially exactly what I wanted: Use Vi mode in IPython. On version 5, this requires:

$ ipython --TerminalInteractiveShell.editing_mode=vi

Now I can use handy vi-like visual mode and yank whatever I need!

Which leads to the following new alias in my .bash_profile/.bash_rc:

alias vpython='ipython --TerminalInteractiveShell.editing_mode=vi'

Malvasia

Instead of using an alias, you can add c.TerminalInteractiveShell.editing_mode = 'vi' to ~/.ipython/profile_default/ipython_config.py. Credit to and more details here. – Actinouranium

great answer but still how to copy code with terminal prompts?? – Valvular

The yank to regular clipboard functionality doesn't seem to work in IPython vim mode as it does in Vim. "+y doesn't work. – Cleveland

The yanking also doesn't register to my clipboard in vim mode – Adrastus

Answer 3

The save magic command [documentation] saves the input lines you want to a file; the -a option is for "append" mode so that the lines are added at the end of the file instead of overwriting the file. I use it all the time.

With your example:

%save -a myfile.py 11
# the '%' is not necessary
save -a myfile.py 11

Then you can keep coding in IPython.

When there is another command you want to write to the same file, you can just type save then use the up arrow to bring back the last use of "save" (so that the -a option and the filename are already there) and just edit the line number.

Note that you can give several lines to save and also line ranges:

save -a myfile.py 15 18 19-25

Degrading

What I like about this solution is that it also displays the code without terminal prompts in the ipython shell. – Comprehension

This is really nice in combination with auto-revert-mode in emacs. – Gemoets

Answer 4

In the shell you can first convert the IPython file to a regular Python file (.py) and then do the clean up:

http://ipython.org/ipython-doc/3/notebook/nbconvert.html (see --to script format)

You can also download the file in the notebook editor as Python file and perform the cleanup after this step.

Carraway

... I don't think you understand, there is no IPython file (by that I assume you mean Jupyter notebook). The whole point is that I am in a terminal, using the shell, and ideally, I don't want to use my mouse at all. The %history magic came closest, which is why I accepted, but it still requires a mouse, unfortunately. – Malvasia

Yes you are correct Jupyter notebook file (ipynb) format. I thought you might be able to run the follow type of command in the terminal: $ ipython nbconvert --to script <filename>.ipynb to create a workable file to cleanup in the terminal (my apologies, just starting with Python) TI learned from your post. – Carraway

Yeah, unfortunately, I'm not working with a jupyter notebook, i'm working in a REPL. Sometimes I do, but then, the problem is moot because of the tooling you've pointed out, and the fact that code is already in a cleaned-up text format. – Malvasia

Interesting that you refer to REPL previously as shell or terminal. In the case of code in the REPL window, most of my editors allow me to copy in rectangles by holding the shift - alt key and using arrows. Place the cursor right for the for keyword in your example. This works in Visual Studio, Atom, VIM etc. I copy from REPL into the editor all the time without problems. If this is important to you and not working in your IDE/text editor then I recommend to look at alternative IDE/text editor. – Carraway

But the issue isn't my IDE, I don't use an IDE, I use a terminal emulator and a text-editor, and the editor doesn't matter (I use vim and sometimes Atom), but the issue was copying from the REPL session in the terminal – Malvasia

Answer 5

I don't think terminal applications really get access to the copy/paste buffer. You're going to have to use the mouse. How do do it depends on what terminal you're using. Most modern terminals have some sort of "rectangular select" or "block select" mode.

With Windows, rectangular select is the default for cmd.exe and Powershell. If you're using Cygwin's mintty, hold Alt and then select the region with the mouse. The same goes for PuTTY.

On Linux (which I don't have in front of me - take these with a grain of salt), xterm doesn't support it, Gnome Terminal uses Ctrl as the modifier, and KDE's Konsole uses Ctrl+Alt.

For OS X Terminal, the Internet tells me that you use ⌘ while clicking.

Other terminals (and GNU Screen) likely have the feature, it's just a matter of figuring out how to activate it.

Illtimed

Terminal applications get the same access to the clipboard that any other application gets. – Outbuilding

Answer 6

This is a bit of a hack, but you can use these keystrokes:

ggIprint("""<Esc>GA""")

This will make the contents of the entire cell into a multiline string, then print it. As long as the number of lines isn't massive, this is usually convenient enough.

It may also be possible to convert it into a keybinding/macro, though I haven't investigated this.

Stoss