What is the way to use Tensor flow 2.0 object in open cv2 python and why is it so circuitous?

Question

I load an image using tensor flow api (2.0) like so :

def load(image_file):
  image = tf.io.read_file(image_file)
  image = tf.image.decode_jpeg(image)

Now that I have this object, I want to show this image, I can simply use matplotlib.pyplot, and this works.

plt.figure()
plt.imshow(re/255.0)
plt.show()

However attempting this with OpenCV2 is problematic from the start, most of the examples are from 1.0 with .eval() session based suggestion for numpy conversion. One way would be to first convert tensor flow object to numpy, here is the function to do that from API documentation :

TensorFlow
API r2.0
TensorFlow Core 2.0a
Python
tf.make_ndarray
Create a numpy ndarray from a tensor.

I dont understand why this does not works and I get a number of errors while all I want is to do something simple and then use some open cv2 functions like remap, resize etc.:

File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\def_function.py", line 426, in call self._initialize(args, kwds, add_initializers_to=initializer_map) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\def_function.py", line 370, in _initialize *args, **kwds)) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\function.py", line 1313, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\function.py", line 1580, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\function.py", line 1512, in _create_graph_function capture_by_value=self._capture_by_value), File "C:\Python\Python37\lib\site-packages\tensorflow\python\framework\func_graph.py", line 694, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\def_function.py", line 317, in wrapped_fn return weak_wrapped_fn().wrapped(*args, **kwds) File "C:\Python\Python37\lib\site-packages\tensorflow\python\framework\func_graph.py", line 686, in wrapper ), args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 392, in converted_call result = converted_f(*effective_args, **kwargs) File "C:\Users\syeda\AppData\Local\Temp\tmpnahp3og4.py", line 32, in tf__random_deform im2 = ag__.converted_call('make_ndarray', tf, ag__.ConversionOptions(recursive=True, verbose=0, strip_decorators=(tf.function, defun_9, ag__.convert, ag__.do_not_convert, ag__.converted_call), force_conversion=False, optional_features=(), internal_convert_user_code=True), (real_image,), {}) File "C:\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 267, in converted_call return _call_unconverted(f, args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 188, in _call_unconverted return f(*args, **kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 596, in MakeNdarray shape = [d.size for d in tensor.tensor_shape.dim] AttributeError: 'Tensor' object has no attribute 'tensor_shape'

Update 5/5/2018 : After searching more I found out that this has something to do with Tensorflow graph execution. I have a function

def load_image_train(image_file):
  input_image, real_image = load(image_file)
 print(type(real_image))
  print(real_image.shape)
  some_image = Open CV operations like filtering, jitter etc performed on real_image
return some_image

This works nicely when called eagerly with .numpy() attribute, however when called like following code and when you try to inspect what real_image is and its type returns

class 'tensorflow.python.framework.ops.Tensor' (None, None, None)

Please advice.

# Input pipeline
train_dataset = tf.data.Dataset.list_files(PATH+'train/*.jpg')
train_dataset = train_dataset.shuffle(BUFFER_SIZE)
train_dataset = train_dataset.map(load_image_train,
                               num_parallel_calls=tf.data.experimental.AUTOTUNE)
train_dataset = train_dataset.batch(1)

Update 5/5/2018 : I decided to do a preprocessing of the data so I don't have to worry about the using any opencv functionality during the load time of the data. However during training time I still want to do some openCV operations. Now as per the suggestion of @giser_yugang I tried using py_function, I wrap opencv operations in py_function and call that function in a wrapper tf.function. This wrapper tf.function I call in train step. However the output I get from this wrapper function is like so :

class 'tensorflow.python.framework.ops.Tensor'
unknown

Then if I try to consume this tensor in the next train step operation I get a

incompatible with the layer: its rank is undefined, but the layer requires a defined rank.

If I don't use this py_function wrapper in my train step and directly try the numpy operations using opencv I get another error

AttributeError: 'Tensor' object has no attribute 'numpy'

I guess both ways you cant win !

Cyndy

I tried np.array(tensor) conversion but it was also useless as the tensor type remains. – Cyndy

Have you tried tensor.numpy(). That's supposed to be the way to do it in TF 2.0a. You might have a lot of problems since you are using TF 2 already and not a lot of people are, many resources will be out of day for you. Use TF 1.13 for better support. – Sweeny

I had tried image.numpy() I got the error earlier, documenting it here : AttributeError: 'Tensor' object has no attribute 'numpy'. Also opencv is compatible directly with numpy array (don't need to convert it to cv2.UMat) img = cv2.imread('Image0.png') X = img[:,:,1] print(X.shape) print(type(X)) gets you this output (1080, 1920) <class 'numpy.ndarray'> – Cyndy

I did some searching and I found out that I can use .numpy() when its eager tensor and not when its just tensor, <class 'tensorflow.python.framework.ops.EagerTensor'> The .numpy() method explicitly converts a Tensor to a numpy array <class 'numpy.ndarray'> <class 'numpy.ndarray'> float32 <class 'numpy.ndarray'> float32 <class 'numpy.ndarray'> float32 <class 'numpy.ndarray'> float32 <class 'tensorflow.python.framework.ops.Tensor'> <dtype: 'float32'> – Cyndy

I think you need Applying arbitrary Python logic with tf.py_func(). – Alterable

Answer 1

Using OpenCV + Tensorflow 2.0 is straightforward.

Let's say we have the image "wink.jpg" (see wink image attached) in the current directory, then it is possible to use Tensorflow 2.0 to read the JPEG image and get a tf.Tensor with dtype=uint8, get a numpy array from it and use OpenCV to visualize it (in BGR format, as OpenCV needs).

import tensorflow as tf
import cv2


def load(image_file):
    image = tf.io.read_file(image_file)
    image = tf.image.decode_jpeg(image)
    return image


wink = load("wink.jpg")
print(wink.shape, wink.dtype)

# Get a Numpy BGR image from a RGB tf.Tensor
image = cv2.cvtColor(wink.numpy(), cv2.COLOR_RGB2BGR)

cv2.imshow("image", image)
cv2.waitKey()

If you have some problem related to the Graph architecture, it is probably because you're:

• Or using tf.function to convert the code the graph (in this case just remove the annotation)
• Or using OpenCV inside a tf.data.Dataset method (in this case just do not use OpenCV or use tf.py_func where OpenCV is needed)
• Or using a wrong version of Tensorflow, that is not 2 with eager mode enabled by default (check if everything is ok by running pip list |grep tf and pip list | grep tensor and if you see something weird like more than 1 version of TF installed, I suggest to delete the environment a start with a new installation).

Tantalous

I think here the question has been misunderstood. I have already used opencv with tensor object since both are compatible with numpy arrays, see update section "This works nicely when called eagerly with .numpy() attribute, ..." However during a training step or other involved operation I could not use the same function since now the numpy attribute will come missing and debugging it will yields no shape, [None, none, none] as a W x H x C array. – Cyndy

That's because you've annotated the train function with tf.function, I guess. If you run the training loop in pure eager it should continue working. Btw you should post all the code that reproduces your issue, otherwise, it is difficult to help only by guessing what happens – Tantalous

Use the code from here, tensorflow.org/alpha/tutorials/generative/pix2pix , and if you can show me how to do an opencv operation like lets say cv::remap, cv::gaussian etc. to one of the images in the train_step I will accept and close this. Thanks for your response. – Cyndy

You can't use opencv inside the input data pipeline, every operation should be a tensorflow operation. You can try by using a tf.py_func inside the map method, but I don't guarantee it will work – Tantalous

I have already tried tf.py_func , see my Update 5/5/2018 it does not work. – Cyndy