IronPython throw InsufficientMemoryException when using numpy in threads

Question

I have some IronPython code that being called from within a C# application.
This code worked fine until I decided to change one function to run in a thread.
when numpy functions called in a python thread a InsufficientMemoryException exception is thrown.
I Searched for solutions but didn't find. can someone explain why it is happening and how can I fix it?

I think this is happening only when I have two threads that use numpy

I run code like this:

C#:

_python.functionA(); # _python was created with "Python.CreateEngine()"
_python.functionA(); # twice on purpose

Python:
my_python_script.py

import threading
import time
import numpy

def blah():    
    print numpy.array([100,100,0])

def functionA():
    t = threading.Timer(0,blah)    
    t.start()
    time.sleep(2)

And I got this Exception:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\Program Files\IronPython 2.7.1\Lib\threading.py", line 552, in _Thread__bootstrap_inner
    self.run()
  File "c:\Program Files\IronPython 2.7.1\Lib\threading.py", line 756, in run
    self.function(*self.args, **self.kwargs)
  File "C:\workspace\my_python_script.py", line 113, in blah
    print numpy.array([100,100,0])
MemoryError: Exception of type 'System.InsufficientMemoryException' was thrown.

Thanks

UPDATE 13/07/14

I get this exception even when I run only one thread and via IronPython interpreter, without the C#:

C:\>"c:\Program Files\IronPython 2.7.1\ipy.exe"
IronPython 2.7.1 (2.7.0.40) on .NET 4.0.30319.18063
Type "help", "copyright", "credits" or "license" for more information.
>>> execfile(r"c:\workspace\my_python_script.py")
>>> functionA()
>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\Program Files\IronPython 2.7.1\Lib\threading.py", line 552, in _Thread__bootstrap_inner
    self.run()
  File "c:\Program Files\IronPython 2.7.1\Lib\threading.py", line 756, in run
    self.function(*self.args, **self.kwargs)
  File "c:\workspace\my_python_script.py", line 6, in blah
    print numpy.array([100,100,0])
MemoryError: Exception of type 'System.InsufficientMemoryException' was thrown.

Babbette

How many processors do you have? – Nefertiti

I'm not sure right now, it on my work computer. But I believe it is at least dual, maybe quad. Is it important how exactly? – Babbette

You might read this and this and see if that fixes your issue. – Nefertiti

quad core, and that didn't fixed the problem :( – Babbette

Are you sure that numpy is thread-safe? And the other libraries used? – Hankow

in cpython it works fine. I don't know if it is thread-safe but i doubt that its not. (Im adding an update to the question) – Babbette

Are you using the same numpy package for ironpython and cpython, or are you using the ported version of numpy? blog.enthought.com/python/scipy-for-net/#.U8TA_vldVSQ (a bit old, so it may have changed in the past 3 years) – Edmonson

@Edmonson I use different versions. pytools.codeplex.com/… – Babbette

Answer 1

I believe that numpy is not thread-safe. I had a similar problem where using np.asarray() with threading would crash my program. It seems that the way that numpy's array function builds the array is not thread-safe. The way around this I found was to use np.fromiter() instead. Apparently, it is thread safe. It is slightly slower, which makes since if it is not using threading, but it works. Try putting your data into a list (or some other iterable data structure) and using np.fromiter() to convert it to a numpy array.

Also, just so you know, it actually runs fine on my computer so it could just be that you don't have enough memory to handle threading (or at least not when using numpy).

Commode

I expected for a solution to this problem and not a workaround so I don't accept this answer yet (maybe solutions will come later). However, It might be that there is no better solution and your answer is the best so far, so you got the bounty – Babbette