Integrating Keras model into TensorFlow

Question

I am trying to use a pre-trained Keras model within TensorFlow code, as described in this Keras blog post under section II: Using Keras models with TensorFlow.

I want to use the pre-trained VGG16 network available in Keras to extract convolutional feature maps from images, and add my own TensorFlow code over that. So I've done this:

import tensorflow as tf
from tensorflow.python.keras.applications.vgg16 import VGG16, preprocess_input
from tensorflow.python.keras import backend as K

# images = a NumPy array containing 8 images

model = VGG16(include_top=False, weights='imagenet')
inputs = tf.placeholder(shape=images.shape, dtype=tf.float32)
inputs = preprocess_input(inputs)
features = model(inputs)

with tf.Session() as sess:
    K.set_session(sess)
    output = sess.run(features, feed_dict={inputs: images})
    print(output.shape)

However, this gives me an error:

FailedPreconditionError: Attempting to use uninitialized value block1_conv1_2/kernel
     [[Node: block1_conv1_2/kernel/read = Identity[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](block1_conv1_2/kernel)]]
     [[Node: vgg16_1/block5_pool/MaxPool/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_132_vgg16_1/block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Instead, if I run an initializer op before running the network:

with tf.Session() as sess:
    K.set_session(sess)
    tf.global_variables_initializer().run()
    output = sess.run(features, feed_dict={inputs: images})
    print(output.shape)

Then I get the expected output:

(8, 11, 38, 512)

My question is, upon running tf.global_variables_initializer(), have the variables been initialized randomly or with the ImageNet weights? I ask this because the blog post referenced above does not mention that an initializer needs to be run when using pre-trained Keras models, and indeed it makes me feel a bit uneasy.

I suspect that it does use the ImageNet weights, and that one needs to run the initializer only because TensorFlow requires all variables to be explicitly initialized. But this is just a guess.

Teamster

Answer 1

TLDR

When using Keras,

1. Avoid using Session if you can (in the spirit of agnostic Keras)
2. Use Keras-handled Session through tf.keras.backend.get_session otherwise.
3. Use Keras' set_session for advanced uses (e.g. when you need profiling or device placement) and very early in your program — contrary to common practice and good usage in "pure" Tensorflow.

More about that

Variables must be initialized before they can be used. Actually, it's a bit more subtle than that: Variables must be initialized in the session they are used. Let's look at this example:

import tensorflow as tf

x = tf.Variable(0.)

with tf.Session() as sess:
    tf.global_variables_initializer().run()
    # x is initialized -- no issue here
    x.eval()
    
with tf.Session() as sess:
    x.eval()
    # Error -- x was never initialized in this session, even though
    # it has been initialized before in another session

So it shouldn't come as a surprise that variables from your model are not initialized, because you create your model before sess.

However, VGG16 not only creates initializer operations for the model variables (the ones you are calling with tf.global_variables_initializer), but actually does call them. Question is, within which Session?

Well, since none existed at the time you built your model, Keras created a default one for you, that you can recover using tf.keras.backend.get_session(). Using this session now works as expected because variables are initialized in this session:

with tf.keras.backend.get_session() as sess:
    K.set_session(sess)
    output = sess.run(features, feed_dict={inputs: images})
    print(output.shape)

Note that you could also create your own Session and provide it to Keras, through keras.backend.set_session — and this is exactly what you have done. But, as this example shows, Keras and TensorFlow have different mindsets.

A TensorFlow user would typically first construct a graph, then instantiate a Session, perhaps after freezing the graph.

Keras is framework-agnostic and does not have this built-in distinction between construction phases — in particular, we learned here that Keras may very well instantiate a Session during graph construction.

For this reason, when using Keras, I would advise against managing a tf.Session yourself and instead rely on tf.keras.backend.get_session if you need to handle TensorFlow specific code that requires a tf.Session.

World

Good, detailed answer. The usage of tf.keras.backend.get_session should have been mentioned in the official blog post where they talk about using a pretrained VGG16 model. – Teamster

I have a question (perhaps unrelated, if you suggest I'll post it as a new question): if I define the further layers after VGG16 in TensorFlow, including a loss and training op, then upon running the training op, will the whole model (including VGG16) be trained end-to-end? – Teamster

@RohanSaxena Thanks for the edits :-) Yes, VGG16 variables are defined as trainable, so you should be able to retrain the model – World

This particular answer did not work for me when I tried it in my project. If I read it right, we get here the session created within Keras with tf.keras.backend.get_session and then set the same session again back to Keras with K.set_session(sess). (I am maybe reading this wrong..) But I replaced "with tf.keras.backend.get_session() as sess:" with "with keras.backend.get_session() as sess:" and removed the line "K.set_session(sess)" and it worked. What is the difference here? – Allina

Answer 2

As a complement to @P-Gn's answer, if you insist on explicitly creating a new session (like the tutorial you are reading) you should put these lines:

sess = tf.Session()
K.set_session(sess)

before creating the model (i.e. model = VGG16(...)) and then use the created session like:

with sess.as_defualt():
    output = sess.run(features, feed_dict={inputs: images})

Pain

Good note. One question on that: if I set the session in Keras (with K.set_session(sess)) before creating the VGG16 model, then will Keras use sess for initializing the model variables – and not generate a default Session via tf.keras.backend.get_session as @World said? – Teamster

@RohanSaxena Yes, Keras will use sess as its active session and the model variables will be initialized using sess session. If you want to make sure, use print(sess) and print(K.get_session()) after setting Keras session (i.e. K.set_session(sess)) and use print(K.get_session()) before setting Keras session. You will see that the first two (i.e. sess and K.get_session() after setting Keras session) point to the same object. – Pain