How can I use MS COCO test-dev dataset for instance segmentation?

Question

Now I'm reproducing the Mask R-CNN(Instance segmentation task.)

I can't figure out how to use the MS COCO test dataset.

There exists 'instances_train2014', 'instances_val2014' which have specific annotations.

But in testing data, there are only image_id, image_url, image height and width. I couldn't figure out how to use it at all.

Could anyone figure this out? :( Sorry for terrible English

Odetteodeum

Answer 1

The test data is just that, test data. They did not release annotations for it. Instead, you train on the train/val datasets and submit results for the test data to the evaluation server. This ensures a fair comparison between different methods.

Bloom

As @Bloom said, test data is meant for evaluating candidates to the leaderboard on the COCO website, here is a link with a more detailed description about this challenge : COCO test guidelines – Kutz

Answer 2

So could you tell me how Researcher evaluate mAP of a model, like ssd_mobilenet_v1_coco have COCO mAP[^1] = 21(according to This)? Did they calculate on COCO validation set instead of COCO Test set and record the mAP number?

Harsho

See here: github.com/AlexeyAB/darknet/issues/2145 - it can't be done locally, but check out CodaLab. – Coagulum

This didn't answer my question. Did Researcher calculate mAP[^1] on COCO validation set instead of COCO Test set, or they calculate mAP[^1] on COCO Test set using CodaLab? – Harsho

They calculated mAP on COCO validation set. COCO2014 minival but different split. Can be referred to here: [^1]: See MSCOCO evaluation protocol. The COCO mAP numbers here are evaluated on COCO 14 minival set (note that our split is different from COCO 17 Val). A full list of image ids used in our split could be fould here. – Bozovich