Keras LSTM Autoencoder time-series reconstruction

Question

I am trying to reconstruct time series data with LSTM Autoencoder (Keras). Now I want train autoencoder on small amount of samples (5 samples, every sample is 500 time-steps long and have 1 dimension). I want to make sure that model can reconstruct that 5 samples and after that I will use all data (6000 samples).

window_size = 500
features = 1
data = data.reshape(5, window_size, features)

model = Sequential()

model.add(LSTM(256, input_shape=(window_size, features), 
return_sequences=True))
model.add(LSTM(128, input_shape=(window_size, features), 
return_sequences=False))
model.add(RepeatVector(window_size))

model.add(LSTM(128, input_shape=(window_size, features), 
return_sequences=True))
model.add(LSTM(256, input_shape=(window_size, features), 
return_sequences=True))
model.add(TimeDistributed(Dense(1)))

model.compile(optimizer='adam', loss='mse')
model.fit(data, data, epochs=100, verbose=1)

Model

Training:

Epoch 1/100
5/5 [==============================] - 2s 384ms/step - loss: 0.1603
...
Epoch 100/100
5/5 [==============================] - 2s 388ms/step - loss: 0.0018

After training, I tried reconstruct one of 5 samples:

yhat = model.predict(np.expand_dims(data[1,:,:], axis=0), verbose=0)

Reconstitution: Blue
Input: Orange

Why is reconstruction so bad when loss is small? How can I make model better? Thanks.

Sharpen

Would you show all graphs from data[0,:,:] to data[4,:,:]? – Latrena

Answer 1

Update: The answer below is based on the old version and based on the current LSTM doc, the input should be shaped as [batch, timesteps, feature]! See this: https://github.com/keras-team/keras/blob/b80dd12da9c0bc3f569eca3455e77762cf2ee8ef/keras/layers/rnn/lstm.py#L481

---

Old Answer:

It seems to me, a time series should be given to the LSTMs in this format:

 (samples, features , window_size)

So, if you change the format, for example I exchanged the variables, and look at the results:

[![enter image description here][1]][1]

Code for reproducing the result(I didn't change the name of the variables, so please don't be confused :)):

import numpy as np
import keras
from keras import Sequential
from keras.layers import Dense, RepeatVector,        TimeDistributed
from keras.layers import LSTM

N = 10000
data = np.random.uniform(-0.1, 0.1, size=(N, 500))
data = data.cumsum(axis=1)
print(data.shape)
window_size = 1
features = 500
data = data.reshape(N, window_size, features)

model = Sequential()

model.add(LSTM(32, input_shape=
(window_size,features), 
return_sequences=True))
model.add(LSTM(16, input_shape=(window_size,   
features), 
return_sequences=False))
model.add(RepeatVector(window_size))

model.add(LSTM(16, input_shape=(window_size, 
features), 
return_sequences=True))
model.add(LSTM(32, input_shape=(window_size,   
features), 
return_sequences=True))
model.add(TimeDistributed(Dense(500)))

model.compile(optimizer='adam', loss='mse')
model.fit(data, data, epochs=100, verbose=1)


yhat = model.predict(np.expand_dims(data[1,:,:],   axis=0), verbose=0)
plot(np.arange(500), yhat[0,0,:])
plot(np.arange(500), data[1,0,:])

Credit to sobe86: I used the proposed data by him/her. [1]: https://i.sstatic.net/5JUDN.png

Mannerly

this is not correct. Keras explicitly states the format (samples, window_size , n_features) should be given to the lstm. Read more here blog.keras.io/building-autoencoders-in-keras.html and here keras.io/api/layers/recurrent_layers/lstm so the format you suggested, is not implementing an lstm at all, it would behave more like a dense layer since you have 1 feature. – Permeability

Answer 2

I tried running your code on the following data

data = np.random.uniform(-0.1, 0.1, size=(5, 500))
data = data.cumsum(axis=1)

so the data is just the cumalative sum of some random uniform noise. I ran for 1000 epochs, and my results are not as bad as yours, the LSTM seems to make some effort to follow the line, though it seems to just be hovering around the running mean (as one might expect).

Note that this is running the model on the TRAINING data (which you seem to imply you were doing in your question) - if we try to look at performance on data that the model was not trained on, we can get bad results.

This is not surprising in the least, with such a small training set, we should fully expect the model to overfit, and not generalise to new data.

Abbacy

Answer 3

One thing I understood from my experience trying to fit auto encoders, is that they are not easy to fit. But I would check these elements:

1. LSTM doesn't do good with non-stationary data. Instead of learning the variability in the data it would try to learn the trend. So de-trending would be a good step to add to your data before hand. Now, to do that, one easy way is to calculate the difference of data with its previous timestamp. Then at each timestep you would have x[i]-x[i-1] instead of x[i]. You can experiment with different orders of de-trending based on your data and its trend/seasonality. For example, if you expect the data has weekly seasonality, another order to check would be 7 days (if each timestep is a day) and your data would be x[i]-x[i-7].
2. Experiment with the architecture of the auto-encoder. depending on the sequence length 32 hidden units might not be enough to encode the data properly and keep enough information.
3. Use Bidirectional layers. Sometimes I use Conv1D as well.
4. Don't need to be Symmetrical. So be creative.

Permeability