Python/Keras/Theano wrong dimensions for Deep Autoencoder

Question

I'm trying to follow the Deep Autoencoder Keras example. I'm getting a dimension mismatch exception, but for the life of me, I can't figure out why. It works when I use only one encoded dimension, but not when I stack them.

Exception: Input 0 is incompatible with layer dense_18:
expected shape=(None, 128), found shape=(None, 32)*

The error is on the line decoder = Model(input=encoded_input, output=decoder_layer(encoded_input))

from keras.layers import Dense,Input
from keras.models import Model

import numpy as np

# this is the size of the encoded representations
encoding_dim = 32

#NPUT LAYER
input_img = Input(shape=(784,))

#ENCODE LAYER
# "encoded" is the encoded representation of the input
encoded = Dense(encoding_dim*4, activation='relu')(input_img)
encoded = Dense(encoding_dim*2, activation='relu')(encoded)
encoded = Dense(encoding_dim, activation='relu')(encoded)

#DECODED LAYER
# "decoded" is the lossy reconstruction of the input
decoded = Dense(encoding_dim*2, activation='relu')(encoded)
decoded = Dense(encoding_dim*4, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)

#MODEL
autoencoder = Model(input=input_img, output=decoded)


#SEPERATE ENCODER MODEL
encoder = Model(input=input_img, output=encoded)

# create a placeholder for an encoded (32-dimensional) input
encoded_input = Input(shape=(encoding_dim,))

# retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]

# create the decoder model
decoder = Model(input=encoded_input, output=decoder_layer(encoded_input))

#COMPILER
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

Ahrens

Amazing how most people struggle at the very same points. Thanks for sharing – Transparency

#47843431 any suggestions? – Ageless

Answer 1

Thanks for the hint from Marcin. Turns out all the decoder layers need to be unrolled in order to get it to work.

# retrieve the last layer of the autoencoder model
decoder_layer1 = autoencoder.layers[-3]
decoder_layer2 = autoencoder.layers[-2]
decoder_layer3 = autoencoder.layers[-1]

# create the decoder model
decoder = Model(input=encoded_input, output=decoder_layer3(decoder_layer2(decoder_layer1(encoded_input))))

Ahrens

in plain english, what is this 'unrolling' means? I assume the layers using Keras functional API are linked correctly, so using the last one only should work. Anyone willing to give a simple explanation? – Evanesce

The last one only wouldn't work because you need something that will transform a vector of dimension 'encoding_dim' to a vector of dimension 784. decoder_layer3 alone goes from encoding_dim*4 to 784. decoder_layer1 alone goes from encoding_dim to encoding_dim*2. – Krute

#47843431 any suggestions? – Ageless

Answer 2

The problem lies in:

# retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]

In previous model - the last layer was the only decoder layer. So it input was also an input to decoder. But right now you have 3 decoding layer so you have to go back to the first one in order to obtain decoder first layer. So changing this line to:

# retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-3]

Should do the work.

Polyptych

I don't think this will work, don't you need to apply the transformation from each decoder layer, not just the first? – Colosseum

Actually - with keras 1.1.1 version my solution works perfectly. And this was the version which I used at this time. So your remark is invalid. – Crashland

oh weird, with keras 2.0.3 it doesn't work. But also, I don't understand how it could work, even with keras 1.x. If you build a decoder model with decoder = Model(encoded_input, autoencoder.layers[-3](encoded_input)) you're only applying the transformation from that one single layer. To build the full decoder model you need to apply the transformations from every layer (in the decoder bit), either manually, as @chris has done, or automatically, as I've done below. Or maybe I've misunderstood the syntax of the functional API. – Colosseum

#47843431 any suggestions? – Ageless

Answer 3

You need to apply the transformation from each decoder layer to the previous. You can manually unroll and hard code these as in the accepted answer, or the following loop should take care of it:

# create a placeholder for an encoded (32-dimensional) input
encoded_input = Input(shape=(encoding_dim,))

# retrieve the decoder layers and apply to each prev layer
num_decoder_layers = 3
decoder_layer = encoded_input
for i in range(-num_decoder_layers, 0):
    decoder_layer = autoencoder.layers[i](decoder_layer)

# create the decoder model
decoder = Model(encoded_input, decoder_layer)

Colosseum

#47843431 any suggestions – Ageless

@Colosseum I think this needs a better explanation but once you know what you're doing with some of the answers above your solution for looping through the layers works nicely. Especially in a keras antotuner environment. I gave you an upvote for that but I think you should explain this a little better. – Allopathy