How do I pass an OpenCV Mat into a C++ Tensorflow graph?

Question

In Tensorflow C++ I can load an image file into the graph using

tensorflow::Node* file_reader =  tensorflow::ops::ReadFile(tensorflow::ops::Const(IMAGE_FILE_NAME, b.opts()),b.opts().WithName(input_name));
tensorflow::Node* image_reader = tensorflow::ops::DecodePng(file_reader, b.opts().WithAttr("channels", 3).WithName("png_reader"));
tensorflow::Node* float_caster = tensorflow::ops::Cast(image_reader, tensorflow::DT_FLOAT, b.opts().WithName("float_caster"));
tensorflow::Node* dims_expander = tensorflow::ops::ExpandDims(float_caster, tensorflow::ops::Const(0, b.opts()), b.opts());
tensorflow::Node* resized = tensorflow::ops::ResizeBilinear(dims_expander, tensorflow::ops::Const({input_height, input_width},b.opts().WithName("size")),b.opts());

For an embedded application I would like to instead pass an OpenCV Mat into this graph.

How would I convert the Mat to a tensor that could be used as input to tensorflow::ops::Cast or tensorflow::ops::ExpandDims?

Illusive

Answer 1

It's not directly from a CvMat, but you can see an example of how to initialize a Tensor from an in-memory array in the TensorFlow Android example: https://github.com/tensorflow/tensorflow/blob/0.6.0/tensorflow/examples/android/jni/tensorflow_jni.cc#L173

You would start off by creating a new tensorflow::Tensor object, with something like this (all code untested):

tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({1, height, width, depth}));

This creates a Tensor object with float values, with a batch size of 1, and a size of widthxheight, and with depth channels. For example a 128 wide by 64 high image with 3 channels would pass in a shape of {1, 64, 128, 3}. The batch size is just used when you need to pass in multiple images in a single call, and for simple uses you can leave it as 1.

Then you would get the underlying array behind the tensor using a line like this:

auto input_tensor_mapped = input_tensor.tensor<float, 4>();

The input_tensor_mapped object is an interface to the data in your newly-created tensor, and you can then copy your own data into it. Here I'm assuming you've set source_data as a pointer to your source data, for example:

const float* source_data = some_structure.imageData;

You can then loop through your data and copy it over:

for (int y = 0; y < height; ++y) {
    const float* source_row = source_data + (y * width * depth);
    for (int x = 0; x < width; ++x) {
        const float* source_pixel = source_row + (x * depth);
        for (int c = 0; c < depth; ++c) {
           const float* source_value = source_pixel + c;
           input_tensor_mapped(0, y, x, c) = *source_value;
        }
    }
}

There are obvious opportunities to optimize this naive approach, and I don't have sample code on hand to show how to deal with the OpenCV side of getting the source data, but hopefully this is helpful to get you started.

Cowey

what is the number of depth in tensorflow::TensorShape for a greyscale image? – Diarmid

Answer 2

Here is complete example to read and feed:

Mat image;
image = imread("flowers.jpg", CV_LOAD_IMAGE_COLOR);
cv::resize(image, image, cv::Size(input_height, input_width), 0, 0, CV_INTER_CUBIC);

int depth = 3;
tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT,
                                tensorflow::TensorShape({1, image.rows, image.cols, depth}));

for (int y = 0; y < image.rows; y++) {
    for (int x = 0; x < image.cols; x++) {
        Vec3b pixel = image.at<Vec3b>(y, x);

        input_tensor_mapped(0, y, x, 0) = pixel.val[2]; //R
        input_tensor_mapped(0, y, x, 1) = pixel.val[1]; //G
        input_tensor_mapped(0, y, x, 2) = pixel.val[0]; //B
    }
}

auto result = Sub(root.WithOpName("subtract_mean"), input_tensor, {input_mean});
ClientSession session(root);
TF_CHECK_OK(session.Run({result}, out_tensors));

Corin

Answer 3

I had tried to run inception model on the opencv Mat file and following code worked for me https://gist.github.com/kyrs/9adf86366e9e4f04addb. Although there are some issue with integration of opencv and tensorflow. Code worked without any issue for .png files but failed to load .jpg and .jpeg. You can follow this for more info https://github.com/tensorflow/tensorflow/issues/1924

Nore

Answer 4

Tensor convertMatToTensor(Mat &input)
{
    int height = input.rows;
    int width = input.cols;
    int depth = input.channels();

    Tensor imgTensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({height, width, depth}));

    float* p = imgTensor.flat<float>().data();
    Mat outputImg(height, width, CV_32FC3, p);
    input.convertTo(outputImg, CV_32FC3);

    return imgTensor;
}

Cathrin