Is it possible to pickle itertools.product in python?

Question

I would like to save the state of itertools.product() after my program quits. Is it possible to do this with pickling? What I am planning to do is to generate permutations and if the process is interrupted (KeyboardInterrupt), I can resume the process the next time I run the program.

def trywith(itr):
     try:
         for word in itr:
             time.sleep(1)
             print("".join(word))
     except KeyboardInterrupt:
         f=open("/root/pickle.dat","wb")
         pickle.dump((itr),f)
         f.close()

if os.path.exists("/root/pickle.dat"):
    f=open("/root/pickle.dat","rb")
    itr=pickle.load(f)
    trywith(itr)
else:
    try:
        itr=itertools.product('abcd',repeat=3)
        for word in itr:
            time.sleep(1)
            print("".join(word))
    except KeyboardInterrupt:
        f=open("/root/pickle.dat","wb")
        pickle.dump((itr),f)
        f.close()

Solemnize

Have a look at https://mcmap.net/q/1746608/-using-itertools-product-and-want-to-seed-a-value/3001761 – Guillerminaguillermo

Nice but I was looking for a simpler explanation on how to do this since I am still exploring python. – Solemnize

The short answer is no, you can't just pickle it; there isn't a much simpler solution, I don't think. – Guillerminaguillermo

ok, appreciated jonrsharpe..I will see if I can try to understand and examine the inputs at #9865309. – Solemnize

actually, product created out of range objects could just be pickled and unpickled with no issues here (Python 3.11) – Dreyfus

@Guillerminaguillermo - as it turns out, your short answer is incorrect. It is pickleable if the iterators are pickleable. I don't know why the OP had not simply trying to run the snippet here. – Dreyfus

Answer 1

Perhaps the easiest thing to do is to save the next group that would be generated. Then a future run can rebuild a new product instance that starts with that group:

def restart_product_at(start_group, *pools):
    n = 0                           # Position of the start_group
    for element, pool in zip(start_group, pools):
        n *= len(pool)
        n += pool.index(element)
    p = product(*pools)             # New fresh iterator
    next(islice(p, n, n), None)     # Advance n steps ahead
    return p

For example:

>>> p = restart_product_at(('c', 'e', 'm'), 'abcd', 'efg', 'hijklm')
>>> next(p)
('c', 'e', 'm')
>>> next(p)
('c', 'f', 'h')
>>> next(p)
('c', 'f', 'i')

Overtire

Can I ask why you deleted your previous answer? I think it would be better to leave it visible with a disclaimer / link to the deprecation issue rather than delete it altogether – Lindseylindsley

Could be faster to skip the already used part of the first pool, i.e., something like del pools[0][:pools[0].index(start_group[0])]. But I don't see a "nice" way to do it... – Warmup

I actually ended up going all the way, similarly skipping in all pools, producing only what's needed. – Warmup

Answer 2

From the "works for me" department:

In [31]: a = product("abcd", repeat=3)

In [32]: next(a), next(a)
Out[32]: (('a', 'a', 'a'), ('a', 'a', 'b'))

In [33]: b = pickle.loads(pickle.dumps(a))

In [34]: next(b)
Out[34]: ('a', 'a', 'c')

In [35]: next(a)
Out[35]: ('a', 'a', 'c')

Dreyfus

Note that this is not supported in Python 2 and in some versions of Python 3, and relies on undocumented and (as of Python 3.12) deprecated behavior. See also the deleted answer from a Python core developer and gh-101588 – Lindseylindsley

Answer 3

Like Raymond's this one creates a new iterator given the pools and start tuple, but instead of generating the full product of the pools and throwing some of it away with islice, I only produce what's needed. This is done by splitting the remaining product into parts and chaining them. For example, with

pools = 'ABCDEFG', 'abcde', '123456'
start = 'D', 'c', '4'

I basically create and chain these three parts:

product('D', 'c', '456')
product('D', 'de', '123456')
product('EFG', 'abcde', '123456')

My solution, followed by Raymond's and testing:

def restart_product_at(start_group, *pools):
    pools = tuple(map(tuple, pools))
    ps = list(zip(start_group))
    def parts():
        inc = 0
        for i in reversed(range(len(ps))):
            ps[i] = pools[i][pools[i].index(*ps[i])+inc:]
            yield product(*ps)
            ps[i] = pools[i]
            inc = 1
    return chain.from_iterable(parts())


# Raymond's
def restart_product_at_Raymond(start_group, *pools):
    n = 0                           # Position of the start_group
    for element, pool in zip(start_group, pools):
        n *= len(pool)
        n += pool.index(element)
    p = product(*pools)             # New fresh iterator
    next(islice(p, n, n), None)     # Advance n steps ahead
    return p


# Testing
from itertools import *

pools = 'ABCDEFG', 'abcde', '123456'
start = 'D', 'c', '4'

expect = list(restart_product_at_Raymond(start, *pools))
result = list(restart_product_at(start, *pools))

print(result == expect)
print(*map(''.join, expect))
print(*map(''.join, result))

Attempt This Online!

Warmup