Make sure numpy is using MKL library on mac pro

Question

I am using Enthought's Canopy/EPD version of python which ships with numpy linked against MKL. I am currently running a set of simulations in parallel (using PP) on both my work computer (Windows 7, Quad Core i5 @ 3.33 Ghz, 4 GB ram) and my home workstation (Mac Pro 3.1, Ubuntu 12.04, 2x Quad Core Xeon @ 2.8 Ghz, 6 GB ram).

But when I benchmark my simulations, they run much quicker on the work computer (35 seconds per iteration vs. 60 on the mac pro). The problem is being perfectly balanced between cores (embarrassingly parallel problem), so I suspect there is an issue with the MKL library on the linux workstation at home. Is there a way to verify that the MKL library is actually being used in python. I've read threads that say you can check to see if python is linked to it, but it doesn't ensure that it was built correctly and is actually being used.

Chick

Is scipy.show_config() not what you're looking for? Not sure if this counts as "ensuring" that it's actually being used. – Humph

when I type that, all of the 'library_dirs' and 'include_dirs' say: "/home/vagrant/src/master-env/...". Is that correct (my user account is not 'vagrant' and when i check for that folder, it is not there). Is this field important, or not used? The libraries correctly report "mkl_lapack95_lp64" etc. – Chick

Answer 1

>>> numpy.show_config()

You will see output something like this, showing that MKL is indeed linked.

lapack_opt_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
blas_opt_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
openblas_info:
NOT AVAILABLE
lapack_mkl_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
blas_mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']

Kapok

Regarding your underlying concern, have you taken into account physical cores vs logical cores? You could try tweaking environment variable MKL_NUM_THREADS to avoid possible thrashing. – Kapok

I'll look at that again, but the operations I'm doing aren't multi-threaded (a lot of add(), multiply(), etc. but no dot()). Also, I'm not sure why this would be a problem on the XEON but not on the i5 – Chick

these paths are hardcoded lists you can find in config.py and not necessarily the what numpy uses. – Arbor