Tensorflow object detection API not displaying global steps

Question

I am new here. I recently started working with object detection and decided to use the Tensorflow object detection API. But, when I start training the model, it does not display the global step like it should, although it's still training in the background.

Details: I am training on a server and accessing it using OpenSSH on Windows. I trained a custom dataset, by collecting pictures and labeling them. I trained it using model_main.py. Also, until a couple of months back, the API was a little different, and only recently they changed to the latest version. For instance, earlier it used to use train.py for training, instead of model_main.py. All the online tutorial I can find use train.py, so it might be a problem with the latest commit. But I don't find anyone else fining this problem.

Thanks in advance!

Cestoid

Welcome to the site. It is indeed a bit difficult to help with limited information. You could try to read How do I ask a good question? from the Help center. – Coolish

If you use putty kind of a client and invoke the train.py from the putty you will be able to see the steps and progress – Nb

@SrinivasBringu Hi, thanks for the reply. I am using openSSH on windows. And it's still not showing the steps. Also, it no more uses train.py, they changed their API and it used model_main.py now. – Cestoid

@Coolish I added more details! – Cestoid

Thanks for this amazing question!!! i've been wasting quite a lot of time thinking training is not happening – Pulmonate

@PiyalGeorge I am glad it helped! – Cestoid

Answer 1

Add tf.logging.set_verbosity(tf.logging.INFO) after the import section of the model_main.py script. It will display a summary after every 100th step.

Claudicant

It worked, thanks! I had a few more doubts: 1. Why did it not work initially? 2. Can I make it display every step? – Cestoid

Cool, would you mind to accept my answer as the right one, so that it helps others too? To your questions: 1. I actually don't know why they disabled this feature, maybe someone just forgot to commit it in the official repository. 2. Should be possible but I've not figured out how. But having an output after every 100th seem sufficient for debugging (to at least check if the process is still running). – Claudicant

I figured it out! Thanks for the help, it saved me a lot of time. Great community! – Cestoid

Answer 2

As Thommy257 suggested, adding tf.logging.set_verbosity(tf.logging.INFO) after the import section of model_main.py prints the summary after every 100 steps by default.

Further, to specify the frequency of the summary, change

config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir)

to

config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir, log_step_count_steps=k)

where it will print after every k steps.

Cestoid

Is there a way to disable the global_step/sec INFO? Pretty redundant if you ask me. – Fanjet

Answer 3

Regarding the recent change to model_main , the previous version is available at the folder "legacy". I use train.py and eval.py from this legacy folder with the same functionality as before.

Kohl