Anaconda numpy: how to enable multiprocessing?

Question

I am trying to enable multithreading/multiprocessing in an Anaconda installation of Numpy. My test program is the following:

import os
import numpy as np
from timeit import timeit

size = 1024
A = np.random.random((size, size)),
B = np.random.random((size, size))
print 'Time with %s threads: %f s' \
      %(os.environ.get('OMP_NUM_THREADS'),
        timeit(lambda: np.dot(A, B), number=4))

I change the environmental variable OMP_NUM_THREADS, but regardless its value, it always takes the same amount of time to run the code and always a single core is being used.

It appears that my Numpy is linked against OpenBlas:

numpy.__config__.show()
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/home/myuser/anaconda3/envs/py2env/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/home/myuser/anaconda3/envs/py2env/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/home/myuser/anaconda3/envs/py2env/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blis_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/home/myuser/anaconda3/envs/py2env/lib']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE

and this is my relevant part of conda list:

conda list | grep blas
blas                      1.1                    openblas    conda-forge
libblas                   3.9.0           1_h6e990d7_netlib    conda-forge
libcblas                  3.9.0           3_h893e4fe_netlib    conda-forge
numpy                     1.14.6          py27_blas_openblashd3ea46f_200  [blas_openblas]  conda-forge
openblas                  0.2.20                        8    conda-forge
scikit-learn              0.19.2          py27_blas_openblasha84fab4_201  [blas_openblas]  conda-forge

I also tried setting OPENBLAS_NUM_THREADS but did make any difference. I use a Python2.7 environment in conda 4.12.0.

Dorso

Numpy random number generation probably doesn’t use multiple cores, but other operations do. Try with matrix multiplication. – Martinelli

@jkr, this is matrix multiplication of two random matrices. np.dot() is called, that's what I expect to be parallel. – Dorso

Answer 1

A bit late, but maybe it helps others coming here.

Your numpy distribution needs to be compiled with threading support. I.e. NPY_ALLOW_THREADS=1 during compilation. See https://numpy.org/doc/stable/reference/c-api/array.html#threading-support for details.

When I numpy from conda-forge it did not have threading support.

So removing numpy and installing via pip using

NPY_ALLOW_THREADS=1 pip install numpy

solved the problem for me.

Some Notes:

• In the numpy docs (https://numpy.org/doc/stable/reference/global_state.html#number-of-threads-used-for-linear-algebra), the envrionment variables (like OMP_NUM_THREADS) are mentioned to control the number of threads. But this can also be done from within python using threadpoolctl.

• I had the additional problem, that when I used np.random.default_rng().integers() to build the arrays there were no BLAS threads. (Even though I saw them when using something like htop they where just idling around).

Clingfish

Answer 2

If you want to build parallel processing, you will have to break down the problem and use python's multi-threading or multi-processing tools to implement it. Here is a scipy doc on how to get started.

If you need to do more sophisticated calculations you can also consider using mpi4py. If most of your calculations are just numpy calculations, I would also consider using dask. It helps chunk and parallelizes your code and lots of the numpy functions are already supported.

Gavotte

I don’t think this is true. Some numpy operations do use multiple cores. – Martinelli

there's np.dot() that should be parallel by numpy – Dorso

Maybe I am wrong about it not being built in for some fuctions- edited my answer to reflect this. For building anything more complicated, I think the rest of the answer still stands. – Gavotte