multiprocessing returns "too many open files" but using `with...as` fixes it. Why?

Question

I was using this answer in order to run parallel commands with multiprocessing in Python on a Linux box.

My code did something like:

import multiprocessing
import logging

def cycle(offset):
    # Do stuff

def run():
    for nprocess in process_per_cycle:
        logger.info("Start cycle with %d processes", nprocess)
        offsets = list(range(nprocess))
        pool = multiprocessing.Pool(nprocess)
        pool.map(cycle, offsets)

But I was getting this error: OSError: [Errno 24] Too many open files
So, the code was opening too many file descriptor, i.e.: it was starting too many processes and not terminating them.

I fixed it replacing the last two lines with these lines:

    with multiprocessing.Pool(nprocess) as pool:
        pool.map(cycle, offsets)

But I do not know exactly why those lines fixed it.

What is happening underneath of that with?

Coupling

Here is the source code each process is calling self.terminate() – Locomobile

The with version is implicitly calling pool.close() after pool.map() returns. According to the docs, that "Prevents any more tasks from being submitted to the pool. Once all the tasks have been completed the worker processes will exit". This likely causes the open files each task has open to be closed. – Orbit

Am I wrong, or @COLDSPEED did answer the question and now it is erased? I was not able to read it deeply, but I would like to... But now it is gone... Anyone knows why the question is down-voted? – Coupling

Hi @cᴏʟᴅsᴘᴇᴇᴅ ... Any hints on this? – Coupling

@Coupling I'd deleted it because I didn't think it was helpful to anyone. I've undeleted it now. – Erdman

Had the same issue +1 for question! – Anorthosite

Answer 1

You're creating new processes inside a loop, and then forgetting to close them once you're done with them. As a result, there comes a point where you have too many open processes. This is a bad idea.

You could fix this by using a context manager which automatically calls pool.terminate, or manually call pool.terminate yourself. Alternatively, why don't you create a pool outside the loop just once, and then send tasks to the processes inside?

pool = multiprocessing.Pool(nprocess) # initialise your pool
for nprocess in process_per_cycle:
    ...       
    pool.map(cycle, offsets) # delegate work inside your loop

pool.close() # shut down the pool

For more information, you could peruse the multiprocessing.Pool documentation.

Erdman

Calling manually to pool.terminate probably will be the thing to do here. I could not create the pool outside, because I wanted to change it on each iteration. So in each one, the number of spawned processes was increasing. For example, if process_per_cycle was [2, 4, 8], each iteration would spawn 2, 4 and 8 processes. – Coupling

@Coupling Yes, that would be the other option, if you can't use the context manager. – Erdman

@Coupling I explained why... because with closes the process pool automatically once you're done with it. – Erdman

Why does the OS treat processes like files in this case? – Giddens

@Giddens When a process is created it must be explicitly destroyed unless it's a daemon. – Erdman

Answer 2

I was already terminating and closing pool but there was a limit on number of file descriptors and I changed my ulimit to 4096 from 1024 and it worked. Following is the procedure:

Check:

ulimit -n

I updated it to 4096 and it worked.

ulimit -n 4096

Spinster

Answer 3

It is context manger. Using with ensures that you are opening and closing files properly. To understand this in detail, I'd recommend this article https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/

Illconditioned

It's obvious it is a context manager. But why does its use here cause this to work? – Erdman

You can have a look at this explanation. – Attalie

Answer 4

This can happen when you use numpy.load too, make sure close those files too, or avoid using it and use pickle or torch.save torch.load etc.

Inversion