How to build LLVM (clang,clang++) for Apple M1?

Question

I am trying to build LLVM compilers so that I can enable OpenMP on the Apple M1. I am using the LLVM development tree, (since I saw some OpenMP runtime go into that for this recently).

I have ended up with this script to invoke cmake:

# Xcode, Ninja
BUILD_SYSTEM=Ninja
BUILD_TAG=Ninja

cmake ../llvm \
      -G$BUILD_SYSTEM -B ${BUILD_TAG}_build \
      -DCMAKE_OSX_ARCHITECTURES='arm64' \
      -DCMAKE_C_COMPILER=`which clang` \
      -DCMAKE_CXX_COMPILER=`which clang++` \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_BUILD_WITH_INSTALL_RPATH=1 \
      -DCMAKE_INSTALL_PREFIX=$HOME/software/clang-12.0.0/arm64 \
      -DLLVM_ENABLE_WERROR=FALSE \
      -DLLVM_TARGETS_TO_BUILD='AArch64' \
      -DLLVM_ENABLE_PROJECTS='clang;openmp,polly' \
      -DLLVM_DEFAULT_TARGET_TRIPLE='aarch64-apple-darwin20.1.0'

The compilers used here are

$ /usr/bin/clang --version
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: arm64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

ninja can then successfully build clang, clang++ and the OpenMp runtime and install them. (As simple, Arm64 images targeting Arms64)

$ file ~/software/clang-12.0.0/arm64/bin/clang
/Users/jcownie/software/clang-12.0.0/arm64/bin/clang: Mach-O 64-bit executable arm64
$ ~/software/clang-12.0.0/arm64/bin/clang --version
clang version 12.0.0 (https://github.com/llvm/llvm-project.git 879c15e890b4d25d28ea904e92497f091f796019)
Target: aarch64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Users/jcownie/software/clang-12.0.0/arm64/bin

Which all looks sane, except that when I try to compile anything with them they are missing the include path to get system headers.

$ ~/software/clang-12.0.0/arm64/bin/clang hello.c
hello.c:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.

So, after all that,

1. Does anyone know how to fix that include path problem?
2. Does anyone know how to configure and build a fat binary for the compilers (and libraries) so that the x86_64 embedded compiler targets x86_64 and the aarch64 binary aarch64? (This is what the Xcode clang and clang++ do...) My attempt at this ended up with a compiler fat binary where both architectures targeted x86_64 :-(

Thanks

Bronchitis

Answer 1

You can set -DDEFAULT_SYSROOT=/path/to/MacOSX11.1.sdk at build time or do export SDKROOT=/path/to/MacOSX11.1.sdk at runtime.

You need to compile with clang -arch arm64 -arch x86_64 to get a fat binary out of clang. You need to do this for Apple clang as well.

Trix

Thanks for answering. I have tried with -DDEFAULT_SYSROOT="$(xcrun --show-sdk-path)" and just trying to build an Arm target, but that still doesn't work fully. I can compile code with the new compiler, but that compiled code won't link against the OpenMP runtime because the new compiler does not include its own library path, but only the system one. If I force that by explicitly using -L, the code links, but won't run because the RPATH isn't set either. – Bronchitis

You can change the default at github.com/llvm/llvm-project/blob/… and see what happens. – Trix

I really don't want to change the LLVM code unless that is essential, because this doesn't seem as if it should require that (at least to build a single architecture compiler). Also this isn't really OpenMP related; that's just the first library that caused a problem... So hacking a special case for that seems the wrong answer. This should surely be soluble with the right magic CMAKE flags :-) – Bronchitis

clang -arch arm64 -arch x86_64 gives a clang that always defaults to target = host. While the system installed fat binary behaves differently and correctly. – Perforce

For system installed: $ arch -arm64 clang -c a.c ; produces arm64 a.o $ arch -x86_64 clang -c a.c ; produces x86_64 a.o How do we build a fat binary like that? The fat binary I built always produces x86_64 a.o, irrespective of the arch being executed. – Perforce

@Perforce You may need to use some other technique to build the fat binary, possibly "lipo". cpufun.substack.com/p/setting-up-the-apple-m1-for-native has some (now possibly outdated) info on the general issue. (Note that I have not built fat binaries myself...) – Bronchitis

Answer 2

UPDATED 8 Feb 2021 Homebrew now supports the M1 based Arm machines, so using that is a better answer than the one below. The info below is potentially still useful if you want to do this on your own, but using brew is likely to be much simpler.

Pre-brew answer

I haven't found a clean solution, but in case it helps anyone else, I do have a horrible hack.

The full recipe, then is configure with this script, then build and install.

# Xcode, Ninja
BUILD_SYSTEM=Ninja
BUILD_TAG=ninja
INSTALLDIR=$HOME/software/clang-12.0.0/arm64

cmake ../llvm \
      -G$BUILD_SYSTEM -B ${BUILD_TAG}_build \
      -DCMAKE_OSX_ARCHITECTURES='arm64' \
      -DCMAKE_C_COMPILER=`which clang` \
      -DCMAKE_CXX_COMPILER=`which clang++` \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=$INSTALLDIR \
      -DLLVM_LOCAL_RPATH=$INSTALLDIR/lib \
      -DLLVM_ENABLE_WERROR=FALSE \
      -DLLVM_TARGETS_TO_BUILD='AArch64' \
      -DLLVM_DEFAULT_TARGET_TRIPLE='aarch64-apple-darwin20.1.0' \
      -DDEFAULT_SYSROOT="$(xcrun --show-sdk-path)" \
      -DLLVM_ENABLE_PROJECTS='clang;openmp;polly;clang-tools-extra;libcxx;libcxxabi' \
#      -DLLVM_ENABLE_PROJECTS='clang;openmp;polly' 

That gives a compiler that finds the right headers, but won't link successfully if OpenMP is used because it doesn't pass on any useful -L path or add a necessary rpath.

To overcome that I created a small shell script that sits in my ~/bin, at the front of my $PATH, which adds those extra linker flags.

#
# A truly awful hack, but it seems necessary.
# Install this with execute permissions as clang and clang++ in
# a directory early in your path, so that it is executed when clang or
# clang++ is needed.
#
# For brew...
INSTALLDIR=/usr/local/opt/llvm
# For a local build.
INSTALLDIR=${HOME}/software/clang-12.0.0/arm64/

# Find out the name of this file, and then invoke the same file in the
# compiler installation, adding the necessary linker directives
CMD=`echo $0 | sed "s/\/.*\///"`
${INSTALLDIR}/bin/${CMD} -L${INSTALLDIR}/lib -Wl,-rpath,${INSTALLDIR}/lib $*

I am not recommending this particularly; there should clearly be a better way to make it work, but it'll do for now, and lets me get back to using the compiler rather than building it!

Bronchitis

This is a dumb question probably, but is cross-compiling on Linux and for M1 impossible now? – Edmon

Answer 3

I was able to build with -DDEFAULT_SYSROOT="$(xcrun --show-sdk-path)" -DCMAKE_INSTALL_PREFIX=/Users/foo/lokal/ and install into the lokal/bin lokal/lib path. Once that is done you can use LD_LIBRARY_PATH=/Users/foo/lokal/lib and all the libraries should be found without mucking with anything else rpath related.

Demote

That set of flags is what I have used, but (as I said before) it doesn't automatically search the non-system library path (that inside the CMAKE_INSTALL_PREFIX path) so libraries such as libomp.dylib (which are not present in the Apple compiler) are not found at link time. Also, requiring LD_LIBRARY_PATH to be set is a hack which shouldn't be required. – Bronchitis