Convert generator object to list for debugging [duplicate]

Question

When I'm debugging in Python using IPython, I sometimes hit a break-point and I want to examine a variable that is currently a generator. The simplest way I can think of doing this is converting it to a list, but I'm not clear on what's an easy way of doing this in one line in ipdb, since I'm so new to Python.

Janie

This is a valid question because the "duplicate question" linked above makes no mention of "generators". So a person searching for "generators" would never find the other question. – Kalagher

Answer 1

Simply call list on the generator.

lst = list(gen)
lst

Be aware that this affects the generator which will not return any further items.

You also cannot directly call list in IPython, as it conflicts with a command for listing lines of code.

Tested on this file:

def gen():
    yield 1
    yield 2
    yield 3
    yield 4
    yield 5
import ipdb
ipdb.set_trace()

g1 = gen()

text = "aha" + "bebe"

mylst = range(10, 20)

which when run:

$ python code.py 
> /home/javl/sandbox/so/debug/code.py(10)<module>()
      9 
---> 10 g1 = gen()
     11 

ipdb> n
> /home/javl/sandbox/so/debug/code.py(12)<module>()
     11 
---> 12 text = "aha" + "bebe"
     13 

ipdb> lst = list(g1)
ipdb> lst
[1, 2, 3, 4, 5]
ipdb> q
Exiting Debugger.

General method for escaping function/variable/debugger name conflicts

There are debugger commands p and pp that will print and prettyprint any expression following them.

So you could use it as follows:

$ python code.py 
> /home/javl/sandbox/so/debug/code.py(10)<module>()
      9 
---> 10 g1 = gen()
     11 

ipdb> n
> /home/javl/sandbox/so/debug/code.py(12)<module>()
     11 
---> 12 text = "aha" + "bebe"
     13 

ipdb> p list(g1)
[1, 2, 3, 4, 5]
ipdb> c

There is also an exec command, called by prefixing your expression with !, which forces debugger to take your expression as Python one.

ipdb> !list(g1)
[]

For more details see help p, help pp and help exec when in debugger.

ipdb> help exec
(!) statement
Execute the (one-line) statement in the context of
the current stack frame.
The exclamation point can be omitted unless the first word
of the statement resembles a debugger command.
To assign to a global variable you must always prefix the
command with a 'global' command, e.g.:
(Pdb) global list_options; list_options = ['-l']

Sheliasheline

hi @Jan Vlcinsky, first thanks for your answer, this methode work great with small simples I am workin with data like 100000000000000000 in generator is there an other way to convert it to fast, because this method can take days to give me wath I need, and thanks again. – Alisaalisan

@WalidBousseta If you have a generator with so many potential items, any attempt to convert it completely into list will consume all the RAM. – Sheliasheline

Agreed with Jan. The utility of a generator as I understand it is to provide a very convenient way to access data in a way that it's a) intrinsically sequential and b) of undetermined length. I'd see whether the vendor can provide bulk transfer options / data dumps if that's possible. But if calculation of one depends on previous items and it's raw horsepower, consider a compiled language. You can get 1000+x speed up that way (e.g. inner loops in Swift are 9000x faster than Python for matrix multiplications) – Mernamero

keep in mind that list(gen) will deplete the generator and it will be unusable afterwards – Banana

Oh man why is it when I try list(gen) it just puts the generator object in a list? [<generator object main.<locals>.<genexpr> at 0x0000023D3F8D2130>] – Flatboat

Ahhh - this worked: list(*gen) – Flatboat