How to build NumPy from source linked to Apple Accelerate framework?

Question

It is my understanding that NumPy dropped support for using the Accelerate BLAS and LAPACK at version 1.20.0. According to the release notes for NumPy 1.21.1, these bugs have been resolved and building NumPy from source using the Accelerate framework on MacOS >= 11.3 is now possible again: https://numpy.org/doc/stable/release/1.21.0-notes.html, but I cannot find any documentation on how to do so. This seems like it would be an interesting thing to try and do because the Accelerate framework is supposed to be highly-optimized for M-series processors. I imagine the process is something like this:

1. Download numpy source code folder and navigate to this folder.
2. Make a site.cfg file that looks something like:

[DEFAULT]
library_dirs = /some/directory/
include_dirs = /some/other/directory/

[accelerate]
libraries = Accelerate, vecLib

3. Run python setup.py build

The problem is I do not know 1. what the variables library_dirs and include_dirs should be so that NumPy knows to use Accelerate BLAS and LAPACK and 2. if there are any other additional steps that need to be taken. If anyone knows how to do this or can provide any insight, it would be greatly appreciated.

Clachan

Answer 1

I actually attempted this earlier today and these are the steps I used:

• In the site.cfg file, put

[accelerate]
libraries = Accelerate, vecLib

• Build with NPY_LAPACK_ORDER=accelerate python3 setup.py build

• Install with python3 setup.py install

Afterwards, np.show_config() returned the following

blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
accelerate_info:
    extra_compile_args = ['-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
blas_opt_info:
    extra_compile_args = ['-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
  NOT AVAILABLE
openblas_clapack_info:
  NOT AVAILABLE
flame_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_compile_args = ['-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
Supported SIMD extensions in this NumPy install:
    baseline = NEON,NEON_FP16,NEON_VFPV4,ASIMD
    found = ASIMDHP,ASIMDDP
    not found = 

and my quick test suggest significant performance boost relative to OpenBlas.

Eclipse

Thanks! That worked! And it is indeed much MUCH faster. Out of curiosity, I tried doing something analogous for Scipy, but without luck. – Clachan

Just tried to do this myself, but somehow the build keeps erroring out. I think it has to do with -march=native not being supported on M1s, but I certainly don't understand how the build system works. Would you mind sharing a bit more about your setups? I basically tried git clone numpy && cd numpy && <create site.cfg> && NPY_LAPACK_ORDER=accelerate python3 setup.py build && pip3 install . on a new machine – Admittedly

In my opinion, the most straightforward way to get an M1-native environment setup is to get the Macosx-arm64 version of miniconda3 at: github.com/conda-forge/miniforge#download. Anything installed in this environment will default to building for arm64 or installing published arm64 binaries. If this is done and the necessary compilers are installed, most of what you have said sounds correct. Except I'm not sure you can create the site.cfg file in the command line like that. you have to manually make it and type in the lines above. Perhaps if you showed what error you get we can help. – Clachan

You might also have to delete the site.cfg.example file, but that's just me guessing. It's difficult to say what is going wrong without more information about the error. Maybe you don't have one or more of the build dependencies installed. – Clachan

Answer 2

No it doesn't have to be that complicated. I used these two commands and was able to install numpy with Apple Accelerate on Mac M1.

pip install cython pybind11
pip install --no-binary :all: --no-use-pep517 numpy

Reference: How to install SciPy on Apple Silicon (ARM / M1)

Thrift

I've spent my whole morning trying to setup performant numpy on an M1 Mac, and this is by far the cleanest solution I've found! I can confirm that this works under OS X 12.3, in a clean Python 3.10 virtualenv, for numpy 1.22. Using gist.github.com/markus-beuckelmann/… as a benchmark, I was seeing total run times in the range of 80-110 seconds with various pre-built numpy wheels. A convoluted setup with conda clocked 18 seconds, which I'm treating as "best possible". @Dat's answer clocked 26 seconds, which while slower is also much simpler! – Elbring

Also, --no-use-pep517 can be omitted, I installed successfully with just the no-binary flag. – Elbring

I like this so much. Thank you! – Sepulcher

Answer 3

Starting from the 2.0 release, Numpy will have wheels built against the Accelerate library on macOS platforms with OS version greater than Sonoma 14 (source).

Weldonwelfare