sklearn OMP: Error #15 ("Initializing libiomp5md.dll, but found mk2iomp5md.dll already initialized.") when fitting models

Question

I have recently uninstalled a nicely working copy of Enthought Canopy 32-bit and installed Canopy version 1.1.0 (64 bit). When I try to use sklearn to fit a model my kernel crashes, and I get the following error:

The kernel (user Python environment) has terminated with error code 3. This may be due to a bug in your code or in the kernel itself.

Output captured from the kernel process is shown below.

OMP: Error #15: Initializing libiomp5md.dll, but found mk2iomp5md.dll already initialized.

OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

The same code was running just fine under Canopy's 32 bit. The code is really just a simple fit of a

linear_model.SGDClassifier(loss='log')

(same error for Logistic Regression, haven't tried other models)

How do I fix this?

Oratory

Answer 1

I had the same problem, coming from conflicting installations in numpy and from canopy. Resolved it by writing:

import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"

Not an elegant solution, but it did the job for me.

Gabriellagabrielle

-1, The error message explicitly states this is unsafe: The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. – Petterson

Is there a better solution? – Zany

Did the job for me as well, unlike other solutions that might have come up. should also be linked to this question: #56759612 – Angelicangelica

@Zany yes! use conda install nomkl (see my answer) – Sizar

For beginners, os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE" should be added before you invoke the library , else it will not work. – Robot

Answer 2

I tried manually deleting the old libiomp5md.dll file. This file is in your anaconda3/lib directory. You should remove the old libiomp5.dll file. Then it should work.

Chiachiack

Yeah, thank you iAvesh and JW Geertsma! it works after manually deleted the libiomp5.dll file. I also found other group tried this out zhuanlan.zhihu.com/p/371649016 – Palliative

file is in your anaconda3/envs/XXX/Library/bin folder where XXX is your env name or just anaconda3/Library/bin if not in an env – Greyson

any idea why conda duplicates the file ? – Pedaias

Confirming this works. I renamed mambaforge/envs/XXX/Library/bin/libiomp5md.dll as a defensive precaution. – Fazeli

This works for me as well, using Anaconda on Windows 11 Pro. I prefer this solution over some of the other solutions. The 'os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"' solution seems risky as it can lead to incorrect results. And the "nomkl" solution was not an option for me as it's my understanding that performance will suffer without mkl in case of a Windows environment with Intel processors. – Educated

This solution works. After deleting the file the file, the app started working twice as faster than before. However I ruined my conda env at first and had to reinstall some packages. After reinstall was done, I deleted the file again and it worked ok. – Bouldin

thanks for the solution, it worked perfectly. for those using Windows and can't find the libiomp5md.dll file, you can open a command prompt, go to Anaconda repository using cd, then type in this command to look for all instances of the ddl : dir /s libiomp5md.dll – Lebron

Answer 3

You will almost certainly be able to get past this error by setting the environment parameter

import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"

However, setting this parameter is not the recommended solution (as it says in the error message). Instead, you may try setting up your conda environment without using the Intel Math Kernel Library, by running:

conda install nomkl --channel conda-forge

(Note, the --channel conda-forge option tells conda to install the package from the conda-forge channel, which is a good idea generally, and may be necessary in this case)

You may need to install some packages again after doing this, if the versions you had were based on MKL (though conda should just do this for you). If not, then you will need to do the following:

install packages that would normally include MKL or depend on packages that include MKL, such as scipy, numpy, and pandas. Conda will install the non-MKL versions of these packages together with their dependencies (see here)

For macOS, nomkl is a good option, since the optimization MKL provides is in fact already provided by Apple's Accelerate Framework, which uses OpenMP already. This is in fact the reason that the error ('...multiple copies of the OpenMP runtime...') is triggered, it seems (as stated in this answer).

Sizar

Installing nomkl works for me for now. Had to use the forge-install though. – Geronto

thanks for that @HansVK, I have updated the answer to be explicit about this (I personally actually have conda-forge as my default, achieved with conda config --add channels conda-forge; conda config --set channel_priority strict) – Sizar

Thank you so much! Been struggling for ages with a different duplicate lib issue where matplotlib would crash the kernel almost every time it was accessed (super inconsistent and therefore hard to track down though). This fixed it immediately. – Bowerman

Answer 4

Had the same issue with easyocr, which is also mkl dependent.

Reinstalling other mkl dependent modules seemed to work. For example, I did pip uninstall numpy, and then pip install numpy, and that made import easyocr work.

Discolor

Thanks a lot, you saved a life. – Terhune

Answer 5

OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.

I was facing the similar problem for python, I fixed it by deleting the libiomp5md.dll duplicate file from Anaconda environment folder C:\Users\aqadir\Anaconda3\envs\your_env_name\Library\bin\libiomp5md.dll

In my case the file libiomp5md.dll was already in the base Anaconda bin folder C:\Users\aqadir\Anaconda3\Library\bin

Bridge

Answer 6

I have read the docs on Intel support research (http://www.intel.com/software/products/support/) and the reason in this case, incl. for me, was the numpy library. I had installed it separately and also as part of the PyTorch install. So it was giving an error. Basically you should create a new environment, and install all your dependencies there.

Overboard

Answer 7

I solved this in a different manner to all the ones here. My script had a matplotlib.pyplot show at the beginning of the script. Moving it to AFTER training removed the error. I'm not an expert on these libraries, but I'm guessing the plot counts as a copy of OpenMP runtime, hence the " multiple copies" error.

Brayton

This is IT! Using conda, we end up with two duplicate OpenMP running simultaneously. In my case, one is used by Tensorflow, the other by matplotlib. I wonder which of the two used the OpenMP installed locally, but either way, they now both use the same, and it's OK! – Nominal

Answer 8

Perhaps this solution will help for sklearn as well. Confronted with the same error #15 for tensorflow, none of the solutions to-date (5 Feb 2021) fully worked despite being helpful. However, I did manage to solve it while avoiding: dithering with dylib libraries, installing from source, or setting the environment variable KMP_DUPLICATE_LIB_OK=TRUE and its downsides of being an “unsafe, unsupported, undocumented workaround” and its potential “crashes or silently produce incorrect results”.

The trouble was that conda wasn’t picking up the non-mkl builds of tensorflow (v2.0.0) despite loading the nomkl package. What finally made this solution work was to:

• ensure I was loading packages from the defaults channel (ie. from a channel with a non-mkl version of tensorflow. As of 5 Feb 2021, conda-forge does not have a tensorflow version of 2.0 or greater).
• specify the precise build of the tensorflow version I wanted: tensorflow>=2.*=eigen_py37h153756e_0. Without this, conda kept loading the mkl_... version of the package despite the nomkl package also being loaded.

I created a conda environment using the following environment.yml file (as per the conda documentation for managing environments) :

name: tf_nomkl
channels:
  - conda-forge
  - defaults
dependencies:
  - nomkl
  - python>=3.7
  - numpy
  - scipy
  - pandas
  - jupyter
  - jupyterlab
  - nb_conda
  - nb_conda_kernels
  - ipykernel
  - pathlib
  - matplotlib
  - seaborn
  - tensorflow>=2.*=eigen_py37h153756e_0

You could try to do the same without an environment.yml file, but it’s better to load all the packages you want in an environment in one go if you can. This solution works on MacOS Big Sur v11.1.

Dur

Answer 9

Only this command solved my problem

pip install numpy --upgrade

Gehrke

Answer 10

the error:

OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.

OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

solving this error can be done by running these two lines

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

instead of

import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"

For an example of this error see the image below

Tiedeman

Answer 11

To solve this and allow the code to continue running, I added the following to the Windows environment variables:

Key: KMP_DUPLICATE_LIB_OK Value: TRUE

Then, started a new command line and ran the code again, it worked without any issues.

Hynda

This answer and its flaw are already provided among existing answers. – Library