How can I feed last output y(t-1) as input for generating y(t) in tensorflow RNN?

Question

I want to design a single layer RNN in Tensorflow such that last output (y(t-1)) is participated in updating the hidden state.

h(t) = tanh(W_{ih} * x(t) + W_{hh} * h(t) + **W_{oh}y(t - 1)**)
y(t) = W_{ho}*h(t)

How can I feed last input y(t - 1) as input for updating the hidden state?

Donkey

Currently, I am looking into this tutorial which seems promising: github.com/ematvey/tensorflow-seq2seq-tutorials/blob/master/… – Cassell

Related post: #39681526 – Finally

Answer 1

Is y(t-1) the last input or output? In both cases it is not a straight fit with the TensorFlow RNN cell abstraction. If your RNN is simple you can just write the loop on your own, then you have full control. Another way that I would use is to pre-process your RNN input, e.g., do something like:

processed_input[t] = tf.concat(input[t], input[t-1])

Then call the RNN cell with processed_input and split there.

Selfsupporting

It uses the last generated output by itself. How can I write a loop for a simple RNN? In the way that the behind-the-scene optimizations be ok. – Donkey

Answer 2

One possibility is to use tf.nn.raw_rnn which I found in this article. Check my answer to this related post.

Finally

Answer 3

I would call what you described an "autoregressive RNN". Here's an (incomplete) code snippet that shows how you can create one using tf.nn.raw_rnn:

import tensorflow as tf

LSTM_SIZE = 128
BATCH_SIZE = 64
HORIZON = 10

lstm_cell = tf.nn.rnn_cell.LSTMCell(LSTM_SIZE, use_peepholes=True)


class RnnLoop:
    def __init__(self, initial_state, cell):
        self.initial_state = initial_state
        self.cell = cell

    def __call__(self, time, cell_output, cell_state, loop_state):
        emit_output = cell_output  # == None for time == 0
        if cell_output is None:  # time == 0
            initial_input = tf.fill([BATCH_SIZE, LSTM_SIZE], 0.0)
            next_input = initial_input
            next_cell_state = self.initial_state
        else:
            next_input = cell_output
            next_cell_state = cell_state

        elements_finished = (time >= HORIZON)
        next_loop_state = None
        return elements_finished, next_input, next_cell_state, emit_output, next_loop_state


rnn_loop = RnnLoop(initial_state=initial_state_tensor, cell=lstm_cell)
rnn_outputs_tensor_array, _, _ = tf.nn.raw_rnn(lstm_cell, rnn_loop)
rnn_outputs_tensor = rnn_outputs_tensor_array.stack()

Here we initialize internal state of LSTM with some vector initial_state_tensor, and feed zero array as input at t=0. After that, the output of the current timestep is the input for the next timestep.

Jerryjerrybuild

How would you implement this in Keras @Alexey Petrenko? – Raycher

Not sure about Keras, but this is absolutely trivial in PyTorch! :D – Jerryjerrybuild