Input Data Format for RNN

Question

I am confused how exactly to encode a sequence of data as an input to an LSTM RNN.

In a vanilla DNN, there is an input for every label. What is the "input" in an RNN? Doesnt it have to be a set (or sequence) of data, in order to train sequential events associated with a label?

Im confused how to encode sequential information, because it seems that there should be more than a single input associated with a given label.

Stroup

Answer 1

Let's draw up an example in code.

Say we have some sentences where each word in the sentence is encoded as a vector (vectors from word2vec maybe).

Suppose we want to classify each sentence into one of two class (0, 1). We might build a simple classifier like so:

import numpy as np
from keras.models import Sequential
from keras.layers import LSTM, Dense

# each example (of which we have a 100) is a sequence of 10 words and
# each words is encoded as 16 element vectors

X = np.random.rand(100, 10, 16) 
y = np.random.choice(1, 100)

model = Sequential()
model.add(LSTM(128, input_shape=(10, 16)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='sgd')

# fit model
model.fit(X, y, epochs=3, batch=16)

Sporozoite

model.add(LSTM(128, input_shape=(10, 16)) <- missing a closing parenthesis – Hogle

Answer 2

For example, if you have a timeseries (X1,..,Xt) and you want to train a predictor to predict at horizon of +1 and use sequence of length 3, your input and output will be :

[[X1,X2,X3]]    [X4]
[[X2,X3,X4]]    [X5]
...
[[Xt-3,Xt-2,Xt-1]] [Xt]

So, there are t-3 sequences, each of this sequence has length 3 and has 1 features. The dimension should be (t-3,3,1).

Yoicks

Answer 3

it seems that there should be more than a single input associated with a given label

Yes you are right. Actually your input need to be a 3D matrix. For example if you have n sequences, each sequence is of length m and each of your sequence data has d features the input of your RNN must be of dimension (n,m,d).

Mada

Answer 4

The book (page 19) by Graves augments the answers with explicit dimensionalities:

Consider a length T input sequence x presented to an RNN with I input units, H hidden units, and K output units. Let $x_i^t$ be the value of input i at time t.
In each of your input sentence of words t= 1,...,T, word t has a length I embedding vector. The input on the scalar level ( $x_i^t$ ) is the i-th component of that vector.

Immediately