Type ERROR when upgrading to tensorflow 2.9

Question

After upgrading to tensorflow 2.9 I got the following Erro message when calling model.fit() with tf 2.8 there were no error. The fit runs anways but its worrying.

2022-06-21 12:42:58.930086: W tensorflow/core/common_runtime/forward_type_inference.cc:231] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_BOOL
    }
  }
}
 is neither a subtype nor a supertype of the combined inputs preceding it:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_LEGACY_VARIANT
    }
  }
}

    while inferring type of node 'calculate/cond/output/_10'

Any idea what can cause this or how to fix it?

Oleomargarine

Can you post your code with the issue? Just the error message itself is a bit vague. – Elwaine

Please share some minimal reproducible code to understand this error. – Sodomy

Unfortunatly I cant really provide more information, since I got the message for the following code: self.train_model.fit( train_generator.get(), epochs=self.epochs, validation_data=val_generator.get(), callbacks=self.callbacks ) I got the same message for different model architectures, so its hard to reduce the search space – Tonsillotomy

I think I've narrowed it down to tf.train, tf.io, or tf.data... I probably have to redo the code for saving TFRecords and loading them again. Hopefully it will be better because it takes a million years to save anything with the current code (developed on 2.2, upgraded to 2.4, now trying to upgrade to 2.9). – Fifteenth

Did anyone find out the reason for this error? – Adamok

Answer 1

This should be a comment instead of an answer, but I don't have enough reputation for that. I have seen the same type of error message appear in the output type of the tensorflow guide here https://www.tensorflow.org/guide/migrate/evaluator and here https://www.tensorflow.org/guide/migrate/migrating_feature_columns . You can easily find the lines with the error by searching for "type inference failed" after following the links.

Roxana

It also shows up here: tensorflow.org/responsible_ai/fairness_indicators/tutorials/… – Fifteenth

Answer 2

At the time of writing, this seems to be still an open issue. For those who may read this in the future, keep an eye on the pertinent issues posted on TensorFlow and Keras repos. Although there has been some pushback against the statements like ...is just a warning, you can safely ignore it. Given code executed without any error message., for now, one MAY want to ignore this warning as suggested by Keras Team:

I think this should be reproducible without involving any Keras logic, at which point the TF folks will definitely look at it. But anyway, as said before, this is just a warning, not something critical. You can ignore it.

Novitiate

Answer 3

I have solved a similar error in my code and here's how I did it.

I think the problem lies when using @tf.function or using any function with a condition while running tf graph. In my case during model.fit method.

Problem indicates that invalid graph escaped type checking. When using if-else statement in @tf.function code keras API converts if-else conditions into tf.cond (AutoGraph converts if-statement to tf.cond().) however, during model.fit() tensorflow gives a warning when using elif but if you want to avoid that error remove elif statements with normal if-else statements and I think that might solve this problem.

Implementation of function before error and it was used in loss function which was used in mode.compile and later model.fit method

import tensorflow as tf
class RescaleImage():
    def __init__(self) -> None:
        super().__init__()
    
    @tf.function
    def normalize(self, x:tf.Tensor, min_val: float=0.0, max_val: float=1.0)->tf.Tensor:
        min_val = tf.cast(min_val,tf.float32)
        max_val = tf.cast(max_val, tf.float32)
        if tf.reduce_max(x)>1.0 and tf.reduce_min(x)>=0.0:
            if min_val==0.0 and max_val==1.0:
                x = x/255.0
            elif min_val==-1.0 and max_val==1.0:
                x = (x - 127.5)/127.5
        elif tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=-1.0 and tf.reduce_min(x)<0.0:
            if min_val==0.0 and max_val==1.0:
                x = (x+1.0)/2.0
            elif min_val==0.0 and max_val==255.0:
                x = (x+1.0)*255.0/2.0

        elif tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=0.0:
            if min_val==-1.0 and max_val==1.0:
                x = (x-0.5)/0.5
            elif min_val==0.0 and max_val==255.0:
                x = x*255.0
        return x
    
    @tf.function
    def normalize_individual(self, x:tf.Tensor, min_val: float=0.0, max_val: float=1.0)->tf.Tensor:
        min_val = tf.cast(min_val,tf.float32)
        max_val = tf.cast(max_val, tf.float32)
        if tf.reduce_max(x)>1.0 and tf.reduce_min(x)>=0.0:
            factor = (max_val-min_val)/(tf.math.reduce_max(x)-tf.math.reduce_min(x))
            x = factor*(x - tf.math.reduce_min(x))+min_val
            
        elif tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=-1.0 and tf.reduce_min(x)<0.0:
            if min_val==0.0 and max_val==1.0:
                x = (x+1.0)/2.0
            elif min_val==0.0 and max_val==255.0:
                x = (x+1.0)*255.0/2.0

        elif tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=0.0:
            if min_val==-1.0 and max_val==1.0:
                x = (x-0.5)/0.5
            elif min_val==0.0 and max_val==255.0:
                x = x*255.0
        return x

Code after solving the error (using normal if statements):

import tensorflow as tf

class RescaleImage():
    def __init__(self) -> None:
        super().__init__()
    
    @tf.function
    def normalize(self, x:tf.Tensor, min_val: float=0.0, max_val: float=1.0)->tf.Tensor:
        min_val = tf.cast(min_val,tf.float32)
        max_val = tf.cast(max_val, tf.float32)
        if tf.reduce_max(x)>1.0 and tf.reduce_min(x)>=0.0:
            if min_val==0.0 and max_val==1.0:
                x = x/255.0
            if min_val==-1.0 and max_val==1.0:
                x = (x - 127.5)/127.5

        if tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=-1.0 and tf.reduce_min(x)<0.0:
            if min_val==0.0 and max_val==1.0:
                x = (x+1.0)/2.0
            if min_val==0.0 and max_val==255.0:
                x = (x+1.0)*255.0/2.0

        if tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=0.0:
            if min_val==-1.0 and max_val==1.0:
                x = (x-0.5)/0.5
            if min_val==0.0 and max_val==255.0:
                x = x*255.0

        return x
    
    @tf.function
    def normalize_individual(self, x:tf.Tensor, min_val: float=0.0, max_val: float=1.0)->tf.Tensor:
        min_val = tf.cast(min_val,tf.float32)
        max_val = tf.cast(max_val, tf.float32)
        if tf.reduce_max(x)>1.0 and tf.reduce_min(x)>=0.0:
            factor = (max_val-min_val)/(tf.math.reduce_max(x)-tf.math.reduce_min(x))
            x = factor*(x - tf.math.reduce_min(x))+min_val
            
        if tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=-1.0 and tf.reduce_min(x)<0.0:
            if min_val==0.0 and max_val==1.0:
                x = (x+1.0)/2.0
            if min_val==0.0 and max_val==255.0:
                x = (x+1.0)*255.0/2.0

        if tf.reduce_max(x)<=1.0 and tf.reduce_min(x)>=0.0:
            if min_val==-1.0 and max_val==1.0:
                x = (x-0.5)/0.5
            if min_val==0.0 and max_val==255.0:
                x = x*255.0

        return x

Urethroscope