Keras VGG extract features

Question

I have loaded a pre-trained VGG face CNN and have run it successfully. I want to extract the hyper-column average from layers 3 and 8. I was following the section about extracting hyper-columns from here. However, since the get_output function was not working, I had to make a few changes:

Imports:

import matplotlib.pyplot as plt
import theano
from scipy import misc
import scipy as sp
from PIL import Image
import PIL.ImageOps
from keras.models import Sequential
from keras.layers.core import Flatten, Dense, Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.optimizers import SGD
import numpy as np
from keras import backend as K

Main function:

#after necessary processing of input to get im
layers_extract = [3, 8]
hc = extract_hypercolumn(model, layers_extract, im)
ave = np.average(hc.transpose(1, 2, 0), axis=2)
print(ave.shape)
plt.imshow(ave)
plt.show()

Get features function:(I followed this)

def get_features(model, layer, X_batch):
    get_features = K.function([model.layers[0].input, K.learning_phase()], [model.layers[layer].output,])
    features = get_features([X_batch,0])
    return features

Hyper-column extraction:

def extract_hypercolumn(model, layer_indexes, instance):
    layers = [K.function([model.layers[0].input],[model.layers[li].output])([instance])[0] for li in layer_indexes]
    feature_maps = get_features(model,layers,instance)
    hypercolumns = []
    for convmap in feature_maps:
        for fmap in convmap[0]:
            upscaled = sp.misc.imresize(fmap, size=(224, 224),mode="F", interp='bilinear')
            hypercolumns.append(upscaled)
    return np.asarray(hypercolumns)

However, when I run the code, I'm getting the following error:

get_features = K.function([model.layers[0].input, K.learning_phase()], [model.layers[layer].output,])
TypeError: list indices must be integers, not list

How can I fix this?

NOTE:

In the hyper-column extraction function, when I use feature_maps = get_features(model,1,instance) or any integer in place of 1, it works fine. But I want to extract the average from layers 3 to 8.

Farthest

Answer 1

It confused me a lot:

1. After layers = [K.function([model.layers[0].input],[model.layers[li].output])([instance])[0] for li in layer_indexes], layers is list of extracted feature.
2. And then you send that list into feature_maps = get_features(model,layers,instance).
3. In def get_features(model, layer, X_batch):, they second parameter, namely layer, is used to index in model.layers[layer].output.

What you want is:

1. feature_maps = get_features(model,layer_indexes,instance): passing layer indices rather than extracted features.
2. get_features = K.function([model.layers[0].input, K.learning_phase()], [model.layers[l].output for l in layer]): list cannot be used to indexing list.

Still, your feature abstracting function is horribly written. I suggest you to rewrite everything rather than mixing codes.

Shay

Answer 2

I rewrote your function for a single channel input image (W x H x 1). Maybe it will be helpful.

def extract_hypercolumn(model, layer_indexes, instance):
    test_image = instance
    outputs    = [layer.output for layer in model.layers]          # all layer outputs
    comp_graph = [K.function([model.input]+ [K.learning_phase()], [output]) for output in outputs]  # evaluation functions

    feature_maps = []
    for layerIdx in layer_indexes:
        feature_maps.append(layer_outputs_list[layerIdx][0][0])


    hypercolumns = []
    for idx, convmap in enumerate(feature_maps):
        #        vv = np.asarray(convmap)
        #        print(vv.shape)
        vv = np.asarray(convmap)
        print('shape of feature map at layer ', layer_indexes[idx], ' is: ', vv.shape)

        for i in range(vv.shape[-1]):
            fmap = vv[:,:,i]
            upscaled = sp.misc.imresize(fmap, size=(img_width, img_height),
                                    mode="F", interp='bilinear')
            hypercolumns.append(upscaled)  

    # hypc = np.asarray(hypercolumns)
    # print('shape of hypercolumns ', hypc.shape)

    return np.asarray(hypercolumns)

Chelsea