Is there cudnnLSTM or cudNNGRU alternative in tensorflow 2.0

Question

The CuDNNGRU in TensorFlow 1.0 is really fast. But when I shifted to TensorFlow 2.0 i am unable to find CuDNNGRU. Simple GRU is really slow in TensorFlow 2.0.

Is there any way to use CuDNNGRU in TensorFlow 2.0?

Adept

Answer 1

The importable implementations have been deprecated - instead, LSTM and GRU will default to CuDNNLSTM and CuDNNGRU if all conditions are met:

1. activation = 'tanh'
2. recurrent_activation = 'sigmoid'
3. recurrent_dropout = 0
4. unroll = False
5. use_bias = True
6. Inputs, if masked, are strictly right-padded
7. reset_after = True (GRU only)

---

Also ensure TensorFlow uses GPU:

import tensorflow as tf
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))

---

Update: there appears to be a problem w/ TF 2.0.0 when running on Colab in getting CuDNN to work; try !pip install tensorflow==2.1.0 instead.

Marquand

i tried this ` x, x_h, x_c = Bidirectional(GRU(50, activation= 'tanh',recurrent_activation ='sigmoid',recurrent_dropout=0, unroll =False,use_bias =True,reset_after=True,return_sequences = True, return_state = True))(x)` but it take the same time as without these parameters. this is 10 time more than CuDNNGRU – Adept

@TalhaAnwar Does your TensorFlow use GPU? See updated answer; also, are your inputs padded or masked? – Marquand

yes, i am using colab gpu and my inputs are post padded to make all sentence of equal length, which i think is right-padded – Adept

@TalhaAnwar Nevermind my earlier suggestion, the docstring is falsely phrased; the inputs, if padded, should be strictly right-padded (i.e. never left). Is this the case, and do you use masking? – Marquand

i even tried pre paddings but it took around the same time, with post paddings. No, i am not doing masking. – Adept

i am simply trying to train GRU with glove embeddings. Previously i am with TF 1.0 and CuDNN saves a lot of time. I just switched to TF 2.0 and its too slow – Adept

@TalhaAnwar Strange, I was able to reproduce your problem without any padding - but it was resolved after upgrading to 2.1; try !pip install tensorflow==2.1.0. – Marquand

Thanks this truly worked for me by setting the told perimeters. – Ghostwrite