Tensorflow Object Detection: use Adam instead of RMSProp

Question

I'm training a CNN with this [.config file][1]:

rms_prop_optimizer: {
    learning_rate: {
      exponential_decay_learning_rate {
        initial_learning_rate: 0.004
        decay_steps: 800720
        decay_factor: 0.95
      }
    }
   momentum_optimizer_value: 0.9
   decay: 0.9
   epsilon: 1.0
}   

}
As you can see there is a rms_prop as optimizer. What if I would like to use Adam? How am I supposed to edit this file?

Homily

Please show use the code or the lines that you focused on and show us more details about this question. – Edington

@InfiniteLoops there is no code. Using Tensorflow Object Detection you have to configure a pipeline with all details of your network: input size, mini-batch, gradient discent, learning rate etc. Then you run a script (legacy/train.py) which takes as input this .config file. So the main part is the configuration. I hope I was clear enough. – Homily

What I mean is that, instead of just pointing the link to the file, you can directly copy and paste the lines that you are referring to make it easier for everyone to see. Because your question look too short. – Edington

ok I will edit the post – Homily

Answer 1

if I'm right, you're trying to use the object_detection model with a pre-trained network offered by Tensorflow, am I right? Then, if you know a little of programming, you can take a look at models/research/object_detection/builders/optimizer_builder.py and see which are the optimizer that can be used and with which parameters. Instead if you just want a out-of-the-box solution, this is how I did:

optimizer {
    # momentum_optimizer {
    adam_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: .0002
          schedule {
            step: 4500
            learning_rate: .0001
          }
          schedule {
            step: 7000
            learning_rate: .00008
          }
          schedule {
            step: 10000
            learning_rate: .00004
          }
        }
      }
      # momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }

In my (little) experience I noticed that using the same learning_experience as momentum_optimizer makes the learning too fast and/or brings to NaN Losses, so I usually decrease it of 10 times or more. I'm trying just now. :)

Dynasty

Thanks a lot Giovanni for your answer. I sucessfully run a new training using adam optimizer and, as a programmer, I'm glad I understood finally how optimizer_builder.py works :-) – Homily

Hello, I tried to use Adam optimizer but I got this error: Key Conv/biases/Adam not found in checkpoint [[node save/RestoreV2 (defined at /truong/object-detection-tf-api/models/research/object_detection/model_lib.py:490) ]] – Shope

I think that you are trying to restore a model that you have already tried to train with another optimizer. If this is the case, unfortunately you cannot change optimizer during train - even if you are resuming it. So you should start the training again with another optimizer and see if it works. – Dynasty

But I was planning to use pretrained faster rcnn on my own dataset, and because I got some error regarding NaN loss, I decided to change the optimizer to Adam (previously it was momentum optimizer). And I can't even change the optimizer? :( – Shope

Maybe I understood badly. If you train your network with momentum and then, from a checkpoint of you, try to continue with another optimizer, it is not possible. Although if you start the fine-tuning "from scratch" (so no already-tuned-by-you checkpoints) with another optimizer, then it should work. Is your case the first or the second one? – Dynasty

@Giovannie Cavaillin I have used SSD Mobile pretrained as transferred learning. From reading the config file, the pretrained model was using momentum optimizer. For my training, I was able to switch to Adam optimizer. – Allayne

Answer 2

Sorry if I am late but I am researching this field nowadays.

I realized we can check inside this directory for some examples of ADAM optimizers:

Tensorflow/models/research/object_detection/builders/optimizer_builder_tf2_test.py

Here is the optimizer config into my pipeline.config without scheduling:

optimizer {
    adam_optimizer: {
      learning_rate: {
        constant_learning_rate {
          learning_rate: 0.0001
        }
      }
    }
    use_moving_average: false
  }

Amaty

Hi @Dennis. Not sure that you are using the same api version. – Homily

Hello. I am using Tensorflow Object Detection v2 with SSD MobileNet v2 320x320 from github.com/tensorflow/models/blob/master/research/… and Adam works better than Momentum (default optimizer) in some examples I made. – Amaty