Unable to solve, Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized

Question

Unable to solve this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError:  logits and labels must have the same first dimension, got logits shape [4,1] and labels shape [12]
     [[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at /Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py:86) ]] [Op:__inference_train_function_978]

Function call stack:
train_function

2020-07-17 01:50:11.552216: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
     [[{{node PyFunc}}]]

Process finished with exit code 1

I am trying to classify Images based on three categories,

Here is my code if it helps:

# Classification of cases from Chest-xray images
import os
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.image import imread
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, MaxPool2D, Dropout, Flatten
from tensorflow.keras.callbacks import EarlyStopping

data_dir = 'Covid19-dataset'

# Creating paths
train_path = 'Covid19-dataset/train'
test_path = 'Covid19-dataset/test'

# Displaying one image
normal_xray_path = train_path + '/Normal/01.jpeg'
normal_xray_path_arr = imread(normal_xray_path)

# print(len(os.listdir(os.path.join(train_path, 'Viral Pneumonia'))))
# number of images for Normal, Covid, Viral Pneumonia = 70,111,70

# Dimensionality study
dim1 = []
dim2 = []
test_covid_path = test_path + '/Covid'
for image_name in os.listdir(os.path.join(test_path, 'Covid')):
    img = imread(os.path.join(test_covid_path, image_name))
    # print(img.shape)
    '''debug for tuple length 2'''
    # if len(img.shape)==2:
    #     print(image_name)
    d1, d2, colors = img.shape
    dim1.append(d1)
    dim2.append(d2)
# print(np.mean(dim1), np.mean(dim2)) = 728.2 782.6

# Keeping dimensions of images same
image_shape = (540, 583, 3)

img_gen = ImageDataGenerator(rotation_range=5, width_shift_range=0.1, height_shift_range=0.1,
                             rescale=1 / 255, shear_range=0.1, zoom_range=0.1)

img_gen.flow_from_directory(test_path)
# plt.imshow(normal_xray_path_arr)
# plt.show()
# plt.imshow(img_gen.random_transform(normal_xray_path_arr))
# plt.show()

model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=128, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(1, activation='softmax'))

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
early_stop = EarlyStopping(monitor='val_loss', patience=2)

# Settings
batch_size = 4

train_image_gen = img_gen.flow_from_directory(train_path, target_size=image_shape[:2],
                                              color_mode='rgb',
                                              batch_size=batch_size,
                                              class_mode='categorical')
test_image_gen = img_gen.flow_from_directory(test_path, target_size=image_shape[:2],
                                             color_mode='rgb',
                                             batch_size=batch_size,
                                             class_mode='categorical',
                                             shuffle=False)
model.summary()

# Result index
# print(train_image_gen.class_indices) ={'Covid': 0, 'Normal': 1, 'Viral Pneumonia': 2}

results = model.fit_generator(train_image_gen, epochs=30, validation_data=test_image_gen,
                              callbacks=[early_stop])

The output:

C:\Users\astro\AppData\Local\Programs\Python\Python38\python.exe C:/Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py
2020-07-17 01:50:05.579136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Found 57 images belonging to 3 classes.
2020-07-17 01:50:07.487490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-07-17 01:50:07.513725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.515GHz coreCount: 14 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 178.84GiB/s
2020-07-17 01:50:07.514010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-07-17 01:50:07.517418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-17 01:50:07.520744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-07-17 01:50:07.521789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-07-17 01:50:07.525789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-07-17 01:50:07.527902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-07-17 01:50:07.535988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-17 01:50:07.536194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-17 01:50:07.536719: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-07-17 01:50:07.543508: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1195281a180 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-17 01:50:07.543717: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-17 01:50:07.543983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.515GHz coreCount: 14 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 178.84GiB/s
2020-07-17 01:50:07.544272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-07-17 01:50:07.544416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-17 01:50:07.544560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-07-17 01:50:07.544703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-07-17 01:50:07.544846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-07-17 01:50:07.544997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-07-17 01:50:07.545141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-17 01:50:07.545307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-17 01:50:08.078548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-17 01:50:08.078710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-07-17 01:50:08.078803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-07-17 01:50:08.079063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2917 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-07-17 01:50:08.081913: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1197b02d2c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-17 01:50:08.082107: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1650, Compute Capability 7.5
Found 209 images belonging to 3 classes.
WARNING:tensorflow:From C:/Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py:86: Model.fit_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.fit, which supports generators.
Found 57 images belonging to 3 classes.
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 538, 581, 32)      896       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 269, 290, 32)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 267, 288, 64)      18496     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 133, 144, 64)      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 131, 142, 64)      36928     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 65, 71, 64)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 63, 69, 128)       73856     
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 31, 34, 128)       0         
_________________________________________________________________
flatten (Flatten)            (None, 134912)            0         
_________________________________________________________________
dense (Dense)                (None, 128)               17268864  
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 129       
=================================================================
Total params: 17,399,169
Trainable params: 17,399,169
Non-trainable params: 0
_________________________________________________________________
Epoch 1/30
2020-07-17 01:50:09.168943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-17 01:50:09.706730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-17 01:50:10.841108: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
Traceback (most recent call last):
  File "C:/Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py", line 86, in <module>
    results = model.fit_generator(train_image_gen, epochs=30, validation_data=test_image_gen,
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1465, in fit_generator
    return self.fit(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 848, in fit
    tmp_logs = train_function(iterator)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\def_function.py", line 644, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 1661, in _filtered_call
    return self._call_flat(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 1745, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 593, in call
    outputs = execute.execute(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  logits and labels must have the same first dimension, got logits shape [4,1] and labels shape [12]
     [[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at /Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py:86) ]] [Op:__inference_train_function_978]

Function call stack:
train_function

2020-07-17 01:50:11.552216: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
     [[{{node PyFunc}}]]

Process finished with exit code 1

The error also includes:

tensorflow.python.framework.errors_impl.InvalidArgumentError:  logits and labels must have the same first dimension, got logits shape [4,1] and labels shape [12]

I don't know how to solve this issue

Aspergillosis

I can't solve this issue without knowing more about the problem. But this error is basically saying the output from your model is a tensor of shape [4, 1]. The four is coming from your batch size, and the 1 from the last dense layer of your model which forces the output to a single value. On the other hand, your label is of shape [12]. Tensorflow is telling you it can't calculate the loss of the model because it doesn't know how to compare a a tensor of shape [4, 1] to a tensor of shape [12]. – Mould

Hello @Mould I have just solved the issue by changing the loss to categorical_crossentropy and the number of neurons in the last layer to be 3. – Aspergillosis

I solved the same issue changing the number of class in the last layer. There was a mistake. Thank you @Aspergillosis – Shirlyshiroma

Hi @Aspergillosis Can share what are the solutions provide for the mentioned issues, I am facing similar problems. – Disjoined

Answer 1

The following solved the issue for me:

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession


def fix_gpu():
    config = ConfigProto()
    config.gpu_options.allow_growth = True
    session = InteractiveSession(config=config)


fix_gpu()

Call this function at the start of your script

Rabi

where do you use the "session" later? I use TensorFLow, and the "fit" function I use is model.fit( – Indore

Answer 2

You need to specify input_shape only for the first Conv2D layer. Actually, the input_shape to next Conv2D layers would be different than image_shape. These layers look like:

model.add(Conv2D(filters=32, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=128, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))

Canonize

Answer 3

I have the same problem using tf.keras.losses.CategoricalCrossentropy(). So this works for me:

model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate=0.001), loss = 'categorical_crossentropy', metrics = ["accuracy"])

Internode

Answer 4

This error message can be triggered by a lot of different things.

In my case, I had the right layers and the right dimensions for the inputs and outputs, but the wrong padding. The original code used tf.contrib.layers, and used arg_scope to change the apparent default for the 'padding' parameter.

Pincer

Answer 5

The problem for me was the Jupyter notebook running was attached to the kernel and I was trying to use it from a script which wasn't allowing it. Closing the jupyter-labs session fixed it.

Aitch

Answer 6

I got the same issue. It is solved.

 physical_devices = tf.config.experimental.list_physical_devices('GPU')
    # physical_devices = tf.config.experimental.list_physical_devices('CPU')
    print("physical_devices-------------", len(physical_devices))
    tf.config.experimental.set_memory_growth(physical_devices[0], True)

Also, here

It like this:

if __name__ == '__main__':
    physical_devices = tf.config.experimental.list_physical_devices('GPU')
    # physical_devices = tf.config.experimental.list_physical_devices('CPU')
    print("physical_devices-------------", len(physical_devices))
    tf.config.experimental.set_memory_growth(physical_devices[0], True)
    main()

Cosmorama