A guide to convert_imageset.cpp

Question

I am relatively new to machine learning/python/ubuntu.

I have a set of images in .jpg format where half contain a feature I want caffe to learn and half don't. I'm having trouble in finding a way to convert them to the required lmdb format.

I have the necessary text input files.

My question is can anyone provide a step by step guide on how to use convert_imageset.cpp in the ubuntu terminal?

Thanks

Mcmaster

Answer 1

A quick guide to Caffe's convert_imageset

Build

First thing you must do is build caffe and caffe's tools (convert_imageset is one of these tools).
After installing caffe and makeing it make sure you ran make tools as well.
Verify that a binary file convert_imageset is created in $CAFFE_ROOT/build/tools.

Prepare your data

Images: put all images in a folder (I'll call it here /path/to/jpegs/).
Labels: create a text file (e.g., /path/to/labels/train.txt) with a line per input image . For example:

img_0000.jpeg 1
img_0001.jpeg 0
img_0002.jpeg 0

In this example the first image is labeled 1 while the other two are labeled 0.

Convert the dataset

Run the binary in shell

~$ GLOG_logtostderr=1 $CAFFE_ROOT/build/tools/convert_imageset \
    --resize_height=200 --resize_width=200 --shuffle  \
    /path/to/jpegs/ \
    /path/to/labels/train.txt \
    /path/to/lmdb/train_lmdb

Command line explained:

• GLOG_logtostderr flag is set to 1 before calling convert_imageset indicates the logging mechanism to redirect log messages to stderr.
• --resize_height and --resize_width resize all input images to same size 200x200.
• --shuffle randomly change the order of images and does not preserve the order in the /path/to/labels/train.txt file.
• Following are the path to the images folder, the labels text file and the output name. Note that the output name should not exist prior to calling convert_imageset otherwise you'll get a scary error message.

Other flags that might be useful:

• --backend - allows you to choose between an lmdb dataset or levelDB.
• --gray - convert all images to gray scale.
• --encoded and --encoded_type - keep image data in encoded (jpg/png) compressed form in the database.
• --help - shows some help, see all relevant flags under Flags from tools/convert_imageset.cpp

You can check out $CAFFE_ROOT/examples/imagenet/convert_imagenet.sh for an example how to use convert_imageset.

Beige

Hi, thanks for the great guide. I took a slightly different route and edited the create_imagenet script. However upon running it I got the follow error; i.e. " a total of 0 images" Creating train lmdb... I0715 16:54:06.121748 4120 convert_imageset.cpp:79] Shuffling data I0715 16:54:06.122463 4120 convert_imageset.cpp:82] A total of 0 images. I0715 16:54:06.123065 4120 db.cpp:34] Opened lmdb /home/pwhc/caffe/GPRLearn/lmdb/GPR_train_lmdb Any thoughts? – Mcmaster

@Mcmaster it seems like no image files were found. check the path to images and the image names in the labels gile – Beige

when reference in the train and test text files i included the file trype; as in $DATA/train.txt when it should have simply been $DATA/train. Appreciate the guidance – Mcmaster

how can you have all your images in a single folder and yet still have a train/test text file? where does the train/test split take place then? – Mcmaster

@Mcmaster you need to construct different lmdb/leveldb for train/test. Therefore, you need two different files /path/to/labels/train.txt and /path/to/labels/test.txt the image names in those files should be different, but they can point to images in the same or in different folders - it's up t o you to organize them. – Beige

so the lmdb train/test databases are defined by the text files? – Mcmaster

@Beige best Tutorial for caffe! Thanks – Noodlehead

Can you please explain why first image is 1 and the other two are 0? you mean there is only 1 positive image and the other two are negative images? what if I have positives in one folder and negatives in another folder? – Goidelic

@SaeidYazdani it's all the way you list the images in the list – Beige

@Shai: Regarding what you mentioned that "Therefore, you need two different files /path/to/labels/train.txt and /path/to/labels/test.txt the image names in those files should be different, but they can point to images in the same or in different folders - it's up t o you to organize them.", is it OK if I use the same filename but it's put in different directories i.e. in train and val directories, just like I did in (#42560263)? – Fumatorium

@Fumatorium if the paths (directories) are different then the files are not the same. It's up to you to organize the images. Just make sure you do not test on images that were in the training set -this is cheating – Beige

@Shai: Thanks for the answer. Yes, they both are totally different images although having the same name, so as to avoid over-fitting (remembering instead of having generalization capability). – Fumatorium

@Beige I can't create the binary. When I run make tools I get make: Nothing to be done for 'tools'.. What is happening? – Coincidentally

@yellow01 this usually means your tools are already compiled and up to date – Beige