Why " NumExpr defaulting to 8 threads. " warning message shown in python?

Question

I am trying to use the lux library in python to get visualization recommendations. It shows warnings like NumExpr defaulting to 8 threads..

import pandas as pd
import numpy as np
import opendatasets as od
pip install lux-api
import lux
import matplotlib

And then:

link = "https://www.kaggle.com/noordeen/insurance-premium-prediction"
od.download(link) 
df = pd.read_csv("./insurance-premium-prediction/insurance.csv")

But, everything is working fine. Is there any problem or should I ignore it? Warning shows like this:

Pyriform

Maybe you don't have 8 threads availables on the machine this in run onto ? I guess there is no reason to worry as long as NumExpr doesn't need the 8 threads, but if the threadpool configuration needs more threads than available you could have a problem, maybe a crash (even if I'd be surprised). So if you don't have 8 threads or if you don't want to give all your CPU cores to NumExpr, you should explicitely reconfigure the NumExpr threadpool You should look at this anyway numexpr.readthedocs.io/projects/NumExpr3/en/latest/… – Harker

Answer 1

This is not really something to worry about in most cases. The warning comes from this function, here the most important part:

...
    env_configured = False
    n_cores = detect_number_of_cores()
    if 'NUMEXPR_MAX_THREADS' in os.environ:
        # The user has configured NumExpr in the expected way, so suppress logs.
        env_configured = True
        n_cores = MAX_THREADS
...
    if 'NUMEXPR_NUM_THREADS' in os.environ:
        requested_threads = int(os.environ['NUMEXPR_NUM_THREADS'])
    elif 'OMP_NUM_THREADS' in os.environ:
        requested_threads = int(os.environ['OMP_NUM_THREADS'])
    else:
        requested_threads = n_cores
        if not env_configured:
            log.info('NumExpr defaulting to %d threads.'%n_cores)

So if neither NUMEXPR_MAX_THREADS nor NUMEXPR_NUM_THREADS nor OMP_NUM_THREADS are set, NumExpr uses so many threads as there are cores (even if the documentation says "at most 8", yet this is not what I see in the code).

You might want to use another number of threads, e.g. while really huge matrices are calculated and one could profit from it or to use less threads, because there is no improvement. Set the environment variables either in the shell or prior to importing numexpr, e.g.

import os
os.environ['NUMEXPR_MAX_THREADS'] = '4'
os.environ['NUMEXPR_NUM_THREADS'] = '2'
import numexpr as ne 

Choosey

Not quite. From version 2.8.1 (same code as link): python INFO - _init_num_threads: Note: detected 96 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable. INFO - _init_num_threads: Note: NumExpr detected 96 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. INFO - _init_num_threads: NumExpr defaulting to 8 threads. – Pasahow