Does tensorflow's object detection api support multi-class multi-label detection?

Question

After hours of research, I could not find any example on multi-label predictions with object detection API. Basically I would like to predict more than one label per instance in an image. As the image shown below:

I would like to predict clothing categories, but also the attributes such as color and pattern.

From my understanding, I need to attach more classification head per each attribute to the 2nd stage ROI feature map, and sums each attribute's loss? However, I have trouble implement this in the object detection code. Can somebody give me some tips on which functions should I start to modify? Thank you.

Arleta

It would be helpful to include the results your research yielded. For sure you found some suitable approaches or other people's implementations regarding this problem? By showing us what you have tried so far it gets far easier for us to point you in the right direction. – Forcible

Currently I have two approaches in mind: 1. Modify the github.com/matterport/Mask_RCNN implementation of Mask_RCNN, by changing the mask prediction head to multiple attribute classification heads. 2. Directly modify the Object detection API code, maybe start by adding more classification heads in the model.py and meta_architectures? However this approach seems to be more complex, and as I have pointed out, I had trouble finding examples on multi-label problem with object detection API. – Arleta

Answer 1

One approach, depending on performance requirements and platform, might be to simplify the problem by using multiple classifiers in a pipeline. For example, you might use a multi-class object detector ('tee' and 'pant' in your example) to capture the areas of interest; then crop those areas based on the bounding box and feed those cropped areas into another model, which in this case doesn't have a locality requirement ('color' and 'pattern').

For the multi-class object detection guidance I might recommend the Eager Few Shot Object Detection Colab from the Tensorflow Git repo. In the 'Preparing data for training' cell note:

num_classes = 1

category_index = {duck_class_id: {'id': duck_class_id, 'name': 'rubber_ducky'}}

Updating this would enable multi-class detection without altering the model. For example, the following would create two classes:

num_classes = 2

category_index = {
    1: {'id': 1, 
        'name': 'tee'},
    2: {'id': 2, 
        'name': 'pant'}
}

Then based on this output, feed these cropped area into another classifier.

Platon

How do you then assign a bounding box to a particular class? – Mascara

For A bounding box to a particular class , here – Euraeurasia

Answer 2

In order for the mono-class detection detection tutorial: Rubber Ducky object detector or Zombie detector . Change it to work with multi-class. You need to totally change the way you get the gt_classes_one_hot_tensors changes like these need to be made:

How to perform object detection model training on more than 1 class?

Of course make the configuration change of the pipeconfig file (as shown in the tutorials) model_config.ssd.num_classes = num_classes

---

Tutorials to consider (optional):

How implement, detection learning boxes transferred multi-label (cats, dogs and zombies) .

The following tutorial is multi labels with boxes, it is perfectly scalabel:

• TF2 Rubber Ducky object detector With 2 Classes: (without object detection API) Rubber Ducky object detector 2 Classes look for MultiClasses in eager_few_shot_od_training_tf2_colab.zip multiclass label_map= { 0: "Allomyrina dichotomus" 1: "Lucanidae"}

• Read .tfRecord format (or in case you have not it explains how to create it), how to edit the pipeconfig.config of the downloaded models, how to save it and how to load and evaluate it. Object solar panel detector

• Another longer tutorial, that I recommend to use as support for the first one is: tutorial recommend to use as support (part 5 and 6)

• Good tutorial with multi-label this time for TF_v2. How to Train Your Own Object Detector Using TensorFlow Object Detection API Part_1 Part_2

• Fantastico trains the object detection model, starting from pre-trained models and finally deploys it on Android with TFlite. Training link the rest check the whole github project tanks vs. fighter jets Extra one: Udemy, Mastering-Computer-Vision-with-TensorFlow-2.0

• To TF2 until Android with TFlite: TF2 until Android

• TF2 Object Detection API to train an SSD-MobileNet model or EfficientDet model with a custom dataset and convert it to TensorFlow Lite format.TF2 until tflite

• Other good tutorial: updated best tutorial

These tutorials are based on TF_v1 , if you want to use TF_v2, special care is required in the ~/models/research/object_detection/model_main.py style commands to replace them by model_main_tf2.py

Things to consider when starting (optional):

• Mono-label tutorials are perfectly valid for loading, pre-processing, predicting and saving images (not for multi-label training). in particular Zombie detector
• Check the size of most of the images and choose a pre-trained model with a similar long_x_width , for example for my images 640x640 resnet101_v1_640x640 can be chose and downloaded from here Zoo_Models if you are looking for a more specific model https://tfhub.dev/
• Use an annotation tool that allows easy transfer to tfRecord. COCO format is easily transferable to tfRecord, check the transfer of your format.
• Use some image augmentation(optional) TF_augmentation_oficial or simpler multi-label augmentation (dot Image augmentation)
• TF models can be saved in 2 ways .pb can be collected, evaluated and retrained or raw .h5 or .tflite (Andorid and rasperri Pi) only to evaluate images.

Euraeurasia