"WindowsError: Access is denied" on calling Process.terminate

Question

I enforce a timeout for a block of code using the multiprocessing module. It appears that with certain sized inputs, the following error is raised:

WindowsError: [Error 5] Access is denied

I can replicate this error with the following code. Note that the code completes with '467,912,040' but not with '517,912,040'.

import multiprocessing, Queue

def wrapper(queue, lst):
    lst.append(1)
    queue.put(lst)
    queue.close()

def timeout(timeout, lst):
    q = multiprocessing.Queue(1)
    proc = multiprocessing.Process(target=wrapper, args=(q, lst))
    proc.start()
    try:
        result = q.get(True, timeout)
    except Queue.Empty:
        return None
    finally:
        proc.terminate()
    return result

if __name__ == "__main__":
    # lst = [0]*417912040 # this works fine
    # lst = [0]*467912040 # this works fine
    lst = [0] * 517912040 # this does not
    print "List length:",len(lst)
    timeout(60*30, lst)

The output (including error):

List length: 517912040

Traceback (most recent call last):
  File ".\multiprocessing_error.py", line 29, in <module>
    print "List length:",len(lst)
  File ".\multiprocessing_error.py", line 21, in timeout
    proc.terminate()
  File "C:\Python27\lib\multiprocessing\process.py", line 137, in terminate
    self._popen.terminate()
  File "C:\Python27\lib\multiprocessing\forking.py", line 306, in terminate
    _subprocess.TerminateProcess(int(self._handle), TERMINATE)
WindowsError: [Error 5] Access is denied

Am I not permitted to terminate a Process of a certain size?

I am using Python 2.7 on Windows 7 (64bit).

Smitty

I don't think you angered python... but its a large list you want to transfer and you likely timed out while the child process was in a strange state (maybe not even started yet). Just pickling your list took 70 seconds on my system. You could print the handle when the terminate fails and see if the process exists. Maybe you could sleep a bit and try again. – Dorella

Looks like the problem has something to do with the len(lst) call which could be because the queue has been corrupted. There's a warning about this being liable to happen highlighted in the terminate() documentation. – Lely

Answer 1

While I am still uncertain regarding the precise cause of the problem, I have some additional observations as well as a workaround.

Workaround.

Adding a try-except block in the finally clause.

finally:
    try:
        proc.terminate()
    except WindowsError:
        pass

This also seems to be the solution arrived at in a related (?) issue posted here on GitHub (you may have to scroll down a bit).

Observations.

1. This error is dependent on the size of the object passed to the Process/Queue, but it is not related to the execution of the Process itself. In the OP, the Process completes before the timeout expires.
2. proc.is_alive returns True before and after the execution of proc.terminate() (which then throws the WindowsError). A second or two later, proc.is_alive() returns False and a second call to proc.terminate() succeeds.
3. Forcing the main thread to sleep time.sleep(1) in the finally block also prevents the throwing of the WindowsError. Thanks, @tdelaney's comment in the OP.
4. My best guess is that proc is in the process of freeing memory (?, or something comparable) while being killed by the OS (having completed execution) when the call to proc.terminate() attempts to kill it again.

Smitty