Gradient clipping in keras

Question

I have a fully implemented LSTM RNN using Keras, and I want to use gradient clipping with the gradient norm limited to 5 (I'm trying to reproduce a research paper). I'm quite a beginner with regards to implementing Neural Networks, how would I implement this ?

Is it just (I'm using rmsprop optimizer):

 sgd = optimizers.rmsprop(lr=0.01, clipnorm=5)

 model.compile(optimizer=sgd,
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])

Correspondence

The accurate answer to this question would be "Yes, this is how you do it". Did you try it? Is it not working? Are you experiencing any problems with this code? – Pro

Answer 1

According to the official documentation, any optimizer can have optional arguments clipnorm and clipvalue. If clipnorm provided, gradient will be clipped whenever gradient norm exceeds the threshold.

Elevenses