Keras: How to feed input directly into other hidden layers of the neural net than the first?

Question

I have a question about using Keras to which I'm rather new. I'm using a convolutional neural net that feeds its results into a standard perceptron layer, which generates my output. This CNN is fed with a series of images. This is so far quite normal.

Now I like to pass a short non-image input vector directly into the last perceptron layer without sending it through all the CNN layers. How can this be done in Keras?

My code looks like this:

# last CNN layer before perceptron layer
model.add(Convolution2D(200, 2, 2, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Dropout(0.25))

# perceptron layer
model.add(Flatten())

# here I like to add to the input from the CNN an additional vector directly

model.add(Dense(1500, W_regularizer=l2(1e-3)))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))

Any answers are greatly appreciated, thanks!

Gaylenegayler

Answer 1

Provided your Keras's backend is Theano, you can do the following:

import theano
import numpy as np

d = Dense(1500, W_regularizer=l2(1e-3), activation='relu') # I've joined activation and dense layers, based on assumption you might be interested in post-activation values
model.add(d)
model.add(Dropout(0.5))
model.add(Dense(1))

c = theano.function([d.get_input(train=False)], d.get_output(train=False))
layer_input_data = np.random.random((1,20000)).astype('float32') # refer to d.input_shape to get proper dimensions of layer's input, in my case it was (None, 20000)
o = c(layer_input_data)

Hostess

Thanks for your help, Serj. I think I now understand the concept. – Gaylenegayler

Answer 2

You didn't show which kind of model you use to me, but I assume that you initialized your model as Sequential. In a Sequential model you can only stack one layer after another - so adding a "short-cut" connection is not possible.

For this reason authors of Keras added option of building "graph" models. In this case you can build a graph (DAG) of your computations. It's a more complicated than designing a stack of layers, but still quite easy.

Check the documentation site to look for more details.

Toboggan

Oh, I see. Yes, I really used a 'sequential' set up. Thanks for your help and the link! – Gaylenegayler

Answer 3

The answer here works. It is more high level and works also for the tensorflow backend:

input_1 = Input(input_shape)
input_2 = Input(input_shape)

merge = merge([input_1, input_2], mode="concat")  # could also to "sum", "dot", etc.
hidden = Dense(hidden_dims)(merge)
classify = Dense(output_dims, activation="softmax")(hidden)

model = Model(input=[input_1, input_2], output=hidden)

Paleolithic