How to get Cartesian product in Python using a generator?

Question

I'm trying to get the Cartesian product of multiple arrays but the arrays are pretty large and I am trying to optimize memory usage. I have tried implementing a generator by using the below code but it just returns that there is a generator at a certain location.

import itertools

x = [[1,2],[3,4]]

def iter_tools(*array):
    yield list(itertools.product(*array))

print(iter_tools(*x))

When I try the same code but with return instead of yield it works fine. How could I get the cartesian product by implementing a generator?

Mossback

Your generator materializes the whole product immediately defeating the entire purpose. In any case, itertools.product(*array) is already an efficient iterator. There is no need to wrap it in an generator. So just use itertools.product(*array). Your generator does work, generator functions return generators. It seems like you have a fundamental misunderstanding. – Pitchstone

@juanpa Thanks! I posted an answer partly based on your comment. – Coquelicot

Answer 1

If you want to yield individual item from the cartesian product, you need to iterate over the product:

import itertools

x = [[1,2],[3,4]]

def iter_tools(*array):
    for a in itertools.product(*array):
        yield a

for a in iter_tools(*x):
    print(a)

Yu

itertools.product is already an iterator, making this iter_tools pointless. – Coquelicot

Answer 2

Bottom line, itertools.product is already an iterator. You don't need to write your own. (A generator is a kind of iterator.) For example:

>>> x = [[1, 2], [3, 4]]
>>> p = itertools.product(*x)
>>> next(p)
(1, 3)
>>> next(p)
(1, 4)

---

Now, to explain, it seems like you're misunderstanding something fundamental. A generator function returns a generator iterator. That's what you're seeing from the print:

>>> iter_tools(*x)
<generator object iter_tools at 0x7f05d9bc3660>

Use list() to cast an iterator to a list.

>>> list(iter_tools(*x))
[[(1, 3), (1, 4), (2, 3), (2, 4)]]

Note how it's a nested list. That's because your iter_tools yields one list then nothing else. On that note, that part makes no sense because casting itertools.product to a list defeats the whole purpose of an iterator - lazy evaluation. If you actually wanted to yield the values from an iterator, you would use yield from:

def iter_tools(*array):
    yield from itertools.product(*array)

In this case iter_tools is pointless, but if your actual iter_tools is more complex, this might be what you actually want.

See also:

• what's the difference between yield from and yield in python 3.3.2+
• How to Use Generators and yield in Python - Real Python

---

This answer is partly based on juanpa.arrivillaga's comment

Coquelicot

Answer 3

The idea of a generator is that you don't do all the calculation at the same time, as you do with your call list(itertools.product(*array)). So what you want to do is generate the results one by one. For example like this:

def iter_tools(*array):
    for i in array[0]:
        for j in array[1]:
            yield (i, j)

You can then do something with each resulting tuple like this:

for tup in iter_tools(*x):
    print(tup)

Of course you can easily adapt the generator so that it yields each row or columns per call.

Or if you are happy with what itertools provides:

for i in itertools.product(*x):
    print(i)

What you need depends on your use-case. Hope I could help you :)

Icj