Using Numpy in different platforms

Question

I have a piece of code which computes the Helmholtz-Hodge Decomposition. I've been running on my Mac OS Yosemite and it was working just fine. A month ago, however, my Mac got pretty slow (it was really old), and I opted to buy a new notebook (Windows 8.1, Dell).

After installing all Python libs and so on, I continued my work running this same code (versioned in Git). And then the result was pretty weird, completely different from the one obtained in the old notebook.

For instance, what I do is to construct to matrices a and b(really long calculus) and then I call the solver:

s = numpy.linalg.solve(a, b)

This was returning a (wrong, and different of the result obtained in my Mac, which was right).

Then, I tried to use:

s = scipy.linalg.solve(a, b)

And the program exits with code 0 but at the middle of it. Then, I just made a simple test of:

print 'here1'
s = scipy.linalg.solve(a, b)
print 'here2'

And here2 is never printed.

I tried:

print 'here1'
x, info = numpy.linalg.cg(a, b)
print 'here2'

And the same happens.

I also tried to check the solution after using numpy.linalg.solve:

print numpy.allclose(numpy.dot(a, s), b)

And I got a False (?!).

I don't know what is happening, how to find a solution, I just know that the same code runs in my Mac, but it would be very good if I could run it in other platforms. Now I'm stucked in this problem (don't have a Mac anymore) and with no clue about the cause.

The weirdest thing is that I don't receive any error on runtime warning, no feedback at all.

Thank you for any help.

EDIT:

Numpy Suit Test Results:

Scipy Suit Test Results:

Dafodil

Did you double check for version consistency? Perhaps something changed between the versions on your old Mac and the (presumably) newer versions on the new computer. – Donnelly

Yes, @Ajean. I also plugged a pen-drive in my Mac to get the old code and re-run it on Mac and Windows systems. It worked (Mac), but the system is too slow, so I really would like to have it running on a Windows too. This seems not to make sense at all. – Dafodil

if det(b)=0, even due to precision loss, it means your problem is not well defined. Fighting against badly conditioned matrix problem is just not the right way. – Sheeting

It rather feels like this is more due to a change in the version of numpy, scipy rather than to Mac/Windows differences. Also try running the numpy/scipy test suite both on your Mac and Windows and compare the results. The fact that the the program exists on scipy.linalg.solve looks like a segfault, make sure BLAS/LAPACK libraries are correctly installed. – Principate

Hi, @rth. With numpy tests, I'm getting memory error. As my matrix a is 10000 x 10000, I think this can be a problem (although I don't receive any error or warning in my console). If that's the problem, do you know how to solve it? Thank you. – Dafodil

To run the matrix multiplication, you need 10000x10000 multiplied by float 4 bytes, which is atleast 400mb of RAM. If not it won't run. Try sparse matrices :) – Greaser

if you didn't already, it's probably worth reporting this as a numpy bug – Pallas

Hi, @pbu. That is was I though the conjugate gradient method would be useful (it's an iterative method to solve systems), but it doesn't run either. – Dafodil

how much available RAM do you have on this new machine? I asked because I just did a numpy.linalg.solve(A,b) on a in R^10000, and it consumed about 1.9GB. – Derina

Hi, @tipanverella. Both, Mac and Windows, with 4GB of RAM. It doesn't run, no matter what I try (just for small systems). – Dafodil

I would check the installations of numpy and scipy – Derina

What do you get when you run numpy.__version__ and scipy.__version__ ? – Derina

Numpy version = 1.9.2 and Scipy version = 0.15.1 in both computers, @tipanverella. – Dafodil

How really old was the Mac? Was it super, super old? So old that it wasn't an x86 Mac? Apple switched over to using Intel processors in 2005 so if your Mac was made before 2005 it's possible that it's using a different CPU from your new Dell PC (PowerPC versus x86) and that MIGHT explain the results. en.wikipedia.org/wiki/… – Mortgagor

Rs, no, it's a mid 2009 Mac, @MikeSandford. And the Dell is brand new, 2015 model. – Dafodil

Something else to check would be the versions of the libraries that underlie numpy. BLAS, LAPACK, and many others are installed and the builds might well be different in some way between the Windows and the Mac versions. What installation method did you use on OSX and Windows? – Mortgagor

I agree with @MikeSandford about BLAS and LAPACK. The underlying libraries of numpy have huge impact on how numpy behaves. Which also leads to the question, how did you install numpy? Are you using anaconda or something similar? – Eye

@MikeSandford , I installed numpy through the official site (sourceforge.net/projects/numpy/files/NumPy/1.9.2). When I try to run np.__config__.show() I got a bunch of "Information not available". – Dafodil

I am looking at the files there and I suspect that is where your problem lies. The files are only a few megabytes, so I'm going to guess that they didn't install the dependencies underlying Numpy. If you look here, the numpy package is 27 megabytes. I think if you install that package and the scipy one on the same page, your program may start working. lfd.uci.edu/~gohlke/pythonlibs/#numpy – Mortgagor

It is not clear to me if you get this problem for any size of arrays a and b or only for large sizes. I've edited my answer to include a potentially reproducible example. – Affusion

Answer 1

Download Anaconda package manager

http://continuum.io/downloads

When you download this it will already have all the dependencies for numpy worked out for you. It installs locally and will work on most platforms.

Bargeboard

Answer 2

This is not really an answer, but this blog discusses in length the problems of having a numpy ecosystem that evolves fast, at the expense of reproducibility.

By the way, which version of numpy are you using? The documentation for the latest 1.9 does not report any method called cg as the one you use...

I suggest the use of this example so that you (and others) can check the results.

>>> import numpy as np
>>> import scipy.linalg
>>> np.random.seed(123)
>>> a = np.random.random(size=(10000, 10000))
>>> b = np.random.random(size=(10000,))
>>> s_np = np.linalg.solve(a, b)
>>> s_sc = scipy.linalg.solve(a, b)
>>> np.allclose(s_np,s_sc)
>>> s_np
array([-15.59186559,   7.08345804,   4.48174646, ..., -16.43310046,
    -8.81301553, -10.77509242])

Affusion

Hi @RamonCrehuet. I posted the versions in the comments above. They are the last ones for both Scipy and Numpy. cg comes from Scipy. docs.scipy.org/doc/scipy/reference/sparse.linalg.html – Dafodil

I know cg is in scipy.sparse, but you seem to be calling a numpy function. Please, clarify this. – Affusion

I've edited my post to suggest a reproducible example that we can all test. – Affusion

I've could'n reproduce your code here. I get memory error. However, I have 4GB of RAM when calling the solver. – Dafodil

Answer 3

I hope you can find the answer - one option in the future is to create a virtual machine for each of your projects, using Docker. This allows easy portability.

See a great article here discussing Docker for research.

Pyatt

Docker does not create virtual machines, it runs processes. Since the host kernel and the process kernel are shared, this almost certainly won't change any fundamental platform behavior. – Unctuous