OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized

Asked 6/12, 2018 at 9:58 Answered 10/3, 2021 at 14:24

I'm trying to run a test program to check if my Anaconda environment is configured correctly. However, when I run my test program I get this error message when the program is setting up graph (on_train_end() callback to be precise):

OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

I'm running the test program on my MacBook Pro 15" 2015 where it is installed macOS Mojave 10.14.1. The Anaconda distribution that I have currently installed is https://repo.anaconda.com/archive/Anaconda2-5.3.0-MacOSX-x86_64.sh.

Here is the test program:

#!/usr/bin/env python

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

from tensorflow import keras

Xs = np.array([
    [0, 0],
    [0, 1],
    [1, 1],
    [1, 0]
])

Ys = np.array([
    [0],
    [1],
    [0],
    [1]
])

class MyCallback(keras.callbacks.Callback):
    def __init__(self):
        super(MyCallback, self).__init__()
        self.stats = []

    def on_epoch_end(self, epoch, logs=None):
        self.stats.append({
            'loss': logs['loss'],
            'acc': logs['acc'],
            'epoch': epoch
        })

    def on_train_end(self, logs=None):
        loss_x = []
        loss_y = []
        acc_x = []
        acc_y = []
        for e in self.stats:
            loss_x.append(e['epoch'])
            loss_y.append(e['loss'])
            acc_x.append(e['epoch'])
            acc_y.append(e['acc'])
        plt.plot(loss_x, loss_y, 'r', label='Loss')
        plt.plot(acc_x, acc_y, 'b', label='Accuracy')
        plt.xlabel('Epochs')
        plt.ylabel('Loss / Accuracy')
        plt.legend(loc='upper left')
        plt.show()

with tf.Session() as session:
    model = keras.models.Sequential()

    model.add(keras.layers.Dense(10, activation=keras.activations.elu, input_dim=2))
    model.add(keras.layers.Dense(1, activation=keras.activations.sigmoid))

    model.compile(optimizer=keras.optimizers.Adam(lr=0.05),
                  loss=keras.losses.mean_squared_error,
                  metrics=['accuracy'])

    model.fit(x=Xs, y=Ys, batch_size=4, epochs=50, callbacks=[MyCallback()])

    print("Training complete")

    loss, acc = model.evaluate(Xs, Ys)

    print(f"loss: {loss} - acc: {acc}")

    predictions = model.predict(Xs)

    print("predictions")
    print(predictions)

I already tried to fix the issue referencing to the answer of this related question. Thus, adding the following lines of code after the import section:

import os
os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'

What I get is another error message, this is the full stack trace:

2018-12-06 10:18:34.262 python[19319:371282] -[NSApplication _setup:]: unrecognized selector sent to instance 0x7ff2b07a3d00
2018-12-06 10:18:34.266 python[19319:371282] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NSApplication _setup:]: unrecognized selector sent to instance 0x7ff2b07a3d00'
*** First throw call stack:
(
        0   CoreFoundation                      0x00007fff2ccf0e65 __exceptionPreprocess + 256
        1   libobjc.A.dylib                     0x00007fff58d47720 objc_exception_throw + 48
        2   CoreFoundation                      0x00007fff2cd6e22d -[NSObject(NSObject) __retain_OA] + 0
        3   CoreFoundation                      0x00007fff2cc92820 ___forwarding___ + 1486
        4   CoreFoundation                      0x00007fff2cc921c8 _CF_forwarding_prep_0 + 120
        5   libtk8.6.dylib                      0x0000000b36aeb31d TkpInit + 413
        6   libtk8.6.dylib                      0x0000000b36a4317e Initialize + 2622
        7   _tkinter.cpython-36m-darwin.so      0x0000000b3686ba16 _tkinter_create + 1174
        8   python                              0x000000010571c088 _PyCFunction_FastCallDict + 200
        9   python                              0x00000001057f2f4f call_function + 143
        10  python                              0x00000001057f0abf _PyEval_EvalFrameDefault + 46847
        11  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        12  python                              0x00000001057f3b1c _PyFunction_FastCallDict + 364
        13  python                              0x000000010569a8b0 _PyObject_FastCallDict + 320
        14  python                              0x00000001056c1fe8 method_call + 136
        15  python                              0x00000001056a1efe PyObject_Call + 62
        16  python                              0x0000000105743385 slot_tp_init + 117
        17  python                              0x00000001057478c1 type_call + 241
        18  python                              0x000000010569a821 _PyObject_FastCallDict + 177
        19  python                              0x00000001056a2a67 _PyObject_FastCallKeywords + 327
        20  python                              0x00000001057f3048 call_function + 392
        21  python                              0x00000001057f0b6f _PyEval_EvalFrameDefault + 47023
        22  python                              0x00000001057f330c fast_function + 188
        23  python                              0x00000001057f2fac call_function + 236
        24  python                              0x00000001057f0abf _PyEval_EvalFrameDefault + 46847
        25  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        26  python                              0x00000001057f3b1c _PyFunction_FastCallDict + 364
        27  python                              0x000000010569a8b0 _PyObject_FastCallDict + 320
        28  python                              0x00000001056c1fe8 method_call + 136
        29  python                              0x00000001056a1efe PyObject_Call + 62
        30  python                              0x00000001057f0cc0 _PyEval_EvalFrameDefault + 47360
        31  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        32  python                              0x00000001057f33ba fast_function + 362
        33  python                              0x00000001057f2fac call_function + 236
        34  python                              0x00000001057f0abf _PyEval_EvalFrameDefault + 46847
        35  python                              0x00000001057f330c fast_function + 188
        36  python                              0x00000001057f2fac call_function + 236
        37  python                              0x00000001057f0abf _PyEval_EvalFrameDefault + 46847
        38  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        39  python                              0x00000001057f33ba fast_function + 362
        40  python                              0x00000001057f2fac call_function + 236
        41  python                              0x00000001057f0abf _PyEval_EvalFrameDefault + 46847
        42  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        43  python                              0x00000001057f33ba fast_function + 362
        44  python                              0x00000001057f2fac call_function + 236
        45  python                              0x00000001057f0b6f _PyEval_EvalFrameDefault + 47023
        46  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        47  python                              0x00000001057f33ba fast_function + 362
        48  python                              0x00000001057f2fac call_function + 236
        49  python                              0x00000001057f0abf _PyEval_EvalFrameDefault + 46847
        50  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        51  python                              0x00000001057f33ba fast_function + 362
        52  python                              0x00000001057f2fac call_function + 236
        53  python                              0x00000001057f0abf _PyEval_EvalFrameDefault + 46847
        54  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        55  python                              0x00000001057f33ba fast_function + 362
        56  python                              0x00000001057f2fac call_function + 236
        57  python                              0x00000001057f0b6f _PyEval_EvalFrameDefault + 47023
        58  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        59  python                              0x00000001057f33ba fast_function + 362
        60  python                              0x00000001057f2fac call_function + 236
        61  python                              0x00000001057f0b6f _PyEval_EvalFrameDefault + 47023
        62  python                              0x00000001057e4209 _PyEval_EvalCodeWithName + 425
        63  python                              0x000000010583cd4c PyRun_FileExFlags + 252
        64  python                              0x000000010583c224 PyRun_SimpleFileExFlags + 372
        65  python                              0x0000000105862d66 Py_Main + 3734
        66  python                              0x0000000105692929 main + 313
        67  libdyld.dylib                       0x00007fff59e1608d start + 1
        68  ???                                 0x0000000000000002 0x0 + 2
)
libc++abi.dylib: terminating with uncaught exception of type NSException

Here is a list of the related dependencies installed in the environment (not related dependencies are omitted for brevity):

Name                |     Version                      Build
--------------------|----------------|----------------------
_tflow_select       |     2.3.0      |                   mkl
blas                |     1.0        |                   mkl
intel-openmp        |     2019.1     |                   144
matplotlib          |     3.0.1      |        py36h54f8f79_0
mkl                 |     2018.0.3   |                     1
mkl_fft             |     1.0.6      |        py36hb8a8100_0
mkl_random          |     1.0.1      |        py36h5d10147_1
numpy               |     1.15.4     |        py36h6a91979_0
numpy-base          |     1.15.4     |        py36h8a80b8c_0
tensorboard         |     1.12.0     |        py36hdc36e2c_0
tensorflow          |     1.12.0     |    mkl_py36h2b2bbaf_0
tensorflow-base     |     1.12.0     |    mkl_py36h70e0e9a_0

Gambier answered 6/12, 2018 at 9:58 Comment(1)

Related: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized – Silverware 9/9, 2021 at 11:42

In most cases, this solves the problem:

conda install nomkl

Garonne answered 9/12, 2018 at 13:19 Comment(5)

Thanks, this worked for me. Do you know why it fixes the issue and what caused it? – Acrolein 27/1, 2019 at 10:4

@JulienPerrenoud Sorry don't know why or what - just found this link after searching for days in the depths of a hidden GitHub thread... – Garonne 27/1, 2019 at 10:16

I thought this means that MKL won't be used at all, and performance would suffer. But apparently this is not the case – MKL is used indirectly. See https://mcmap.net/q/193844/-error-15-initializing-libiomp5-dylib-but-found-libiomp5-dylib-already-initialized. Performance with and without nomkl was the same for a simple multi-layer perceptron on my MacBook with Core i9 CPU. – Savonarola 26/11, 2019 at 2:16

This didn't work for me at first, it seems like I had a broken install of Anaconda (maybe because I started with miniconda and then tried to update with full Anaconda packages). I had to remove Anaconda completely and reinstall it, and this fix worked. – Arrant 27/12, 2020 at 13:2

I did a lot of reasearch and then finally landed on this solution which did worked for me, thanks for the solution @AgileBean – Trudytrue 3/8, 2022 at 8:41

I have tried the following solutions that I have come across. Unfortunately, many of them did not work out and the reasons behind them are also not very clear:

I was using conda installed Tensorflow 2.0 MKL with python3.6 in mac OS Mojave

To downgrade matplotlib. What does it have to do something with OpenMP? Reason not clear but it did not work out.
```
conda install matplotlib==2.2.3 
```
Allow duplication of OpenMP library because of multiple copies of it exist. This works out but in the warning log, it says this is a workaround and silently produce incorrect results. So, definitely, this is not the way to go, therefore still a proper solution/fix is required.
```
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
```
To install nomkl. I guess this is to not use MKL based binaries for all the libraries (scipy, numpy, tensorflow etc) but then I do not get the point why to use Tensorflow-MKL? because the whole point is to use MKL binaries to take advantage of the Intel architecture to do fast processing (AVX2 instructions etc). Most of the people said this worked out for them, however, this did not work out for me:
```
conda install nomkl
```
Update MKL. It did not work out.
```
conda install -c intel mkl
```
Uninstall OpenMP and install it again. It did not work out.
```
conda uninstall openmp
conda install openmp
```
Finally, what I did is to uninstall conda installed tensorflow (tf-mkl) and install it again via pip. This has worked out!!! I think this is a proper solution. So, it might mean Intel TF-MKL binaries are broken for macOS. I have an observation this is common for Intel and macOS since other libraries like OpenVINO, pyrealsense2 etc are also not working well in macOS.
```
conda uninstall tensorflow
pip install tensorflow==2.0.0 
```

Some useful links:

Orsay answered 12/8, 2020 at 10:24 Comment(1)

Uninstalling conda's tensorflow and re-installing from pip is what worked for me. Thanks for sharing! – Nonpareil 1/9, 2020 at 2:44

I had a similar experience and solutions posted elsewhere were not fixing things for me. Eventually I got unblocked by downgrading my version of matplotlib, i.e. conda install matplotlib=2.2.3

Perak answered 7/12, 2018 at 0:9 Comment(2)

Thank you so much!! I was going nuts. – Gambier 7/12, 2018 at 10:52

I would suggest to use conda install -n myenv -c intel matplotlib if you're using an Intel architecture. This installs the LTS version of matplotlib (namely version 2.2.3) and it prevents to upgrade to 3.x.y when you upgrade all your packages using conda update --all -y. – Gambier 7/12, 2018 at 11:0

i keep running into this error and it seems to have to do with dependency based installation and conda missing symlinks after that.

Example: I installed a package with pip in my conda environment which had a torch dependency, and it did install successfully - but when importing i got the error above. The lib/ looked the following way:

~/opt/anaconda3/lib  ll|grep libomp
lrwxr-xr-x    1 user  staff    12B Dec 31 12:17 libgomp.1.dylib -> libomp.dylib
lrwxr-xr-x    1 user  staff    12B Dec 31 12:17 libgomp.dylib -> libomp.dylib
lrwxr-xr-x    1 user  staff    12B Dec 31 12:17 libiomp5.dylib -> libomp.dylib
-rwxrwxr-x    1 iser  staff   642K Dec 31 12:17 libomp.dylib

I then used conda install pytorch which did install additional packages. After that my lib/looked like this:

 ~/opt/anaconda3/lib  ll|grep libomp
lrwxr-xr-x    1 user  staff    12B Dec 31 12:17 libgomp.1.dylib -> libomp.dylib
lrwxr-xr-x    1 user  staff    12B Dec 31 12:17 libgomp.dylib -> libomp.dylib
lrwxr-xr-x    1 user  staff    12B Mar 10 14:59 libiomp5.dylib -> libomp.dylib
-rwxrwxr-x    2 user  staff   646K Jan 15 22:21 libomp.dylib

Hence the libomp.dylib and libiomp5.dylib symlink got updated. Import worked then.

I also fixed this issue before by manually creating a symlink between those libraries... so check if this looks valid for you!

Beige answered 10/3, 2021 at 14:24 Comment(0)

Recommended topics

Hot tags