TensorFlow: How to handle void labeled data in image segmentation?

Question

I was wondering how to handle not labeled parts of an image in image segmentation using TensorFlow. For example, my input is an image of height * width * channels. The labels are too of the size height * width, with one label for every pixel.

Some parts of the image are annotated, other parts are not. I would wish that those parts have no influence on the gradient computation whatsoever. Furthermore, I am not interested in the network predicting this “void” label.

Is there a label or a function for this? At the moment I am using tf.nn.sparse_softmax_cross_entropy_with_logits.

Verniavernice

Rhetorical question: how do you know which parts are incorrectly labeled? Or how would your network know which parts are incorrectly labeled? – Dreamadreamer

Look at this data set for example: host.robots.ox.ac.uk/pascal/VOC/voc2012/segexamples/index.html The images have pixels labeled as "void" or even as background data. That is how I would know. And how does the network know is pretty much my question. – Verniavernice

OK, so it isn't that parts of the image are not correctly annotated, just that they are annotated with some indication that they should be ignored? – Dreamadreamer

What you call "possibly void data" could be a case where you need semi-supervised learning, and there are multiple ways to make it work, as well as to achieve an implementation with Tensorflow. Your question is too broad and, by the looks of it, not even about software development. Consider doing more research on semi-supervised learning tasks, because currently there cannot be a canonical answer that fits in a Stack Overflow answer. – Dilettantism

possible duplicate of: https://mcmap.net/q/661069/-tensorflow-how-to-ignore-specific-labels-during-semantic-segmentation/1714410 – Philharmonic

ideally, you would use pixel-wise "info gain" loss (which is a generalization of "cross entropy" loss) – Brande

you can use a custom loss function to mask the loss of unlabeled data dlology.com/blog/… – Roger

Answer 1

I'm not 100% familiar with TF. However, have you considered using the weights parameter of the loss?
Looking at tf.loses.sparse_softmax_cross_entropy it has a parameter weights

weights: Coefficients for the loss. This must be scalar or of same rank as labels

You can set weightof "void" pixels to zero, thus making the loss ignore them.

You can also remove the reduction from tf.nn.sparse_softmax_cross_entropy_with_logits and use tf.losses.compute_weighted_loss to perform the weighting.

Philharmonic

Just FYI, and reference, tf.losses.softmax_cross_entropy also exists, for your non-sparse needs – Fawnfawna

tf.keras.losses.SparseCategoricalCrossentropy: It has the parameter "ignore_class" (tensorflow.org/api_docs/python/tf/keras/losses/…) – Axseed

Answer 2

If I understand correctly you have a portion of each image with label void in which you are not interested at all. Since there is not a easy way to obtain the real value behind this void spots, why don't you map these points to background label and try to get results for your model? I would try in a preprocessing state to clear the data labels from this void label and substitute them with background label.

Another possible strategy ,if you don's simply want to map void labels to background, is to run a mask (with a continuous motion from top to bottom from right to left) to check the neigthbooring pixels from a void pixel (let's say an area of 5x5 pixels) and assign to the void pixels the most common label besides void.

Also you can always keep a better subset of the data, filtering data where the percentage of void labels is over a threshold. You can keep only images with no void labels, or more likeley you can keep images that have only under a threshold (e.g. 5%) of non-labeled points. In this images you can implement the beforementioned strategies for replacing the void labels.

Dugas

I have tried your first recommendation. I've labeled all the void data to label 0, unfortunately the network then makes label 0 predictions for non void areas. This is due to the nature of my data, I guess. Thats why I hoped for a void label like one can find in caffe. – Verniavernice

The network will always misclassify pixels, it can not have perfect accuracy even in training data. How many pixels did you substitute and what percentage of your images have void labels? Maybe you should filter all images with void labels and run your model to see if you have another problem with your code and not with the images that have void labels – Dugas