Deploying Keras Models via Google Cloud ML

Question

I am looking to use Google Cloud ML to host my Keras models so that I can call the API and make some predictions. I am running into some issues from the Keras side of things.

So far I have been able to build a model using TensorFlow and deploy it on CloudML. In order for this to work I had to make some changes to my basic TF code. The changes are documented here: https://cloud.google.com/ml/docs/how-tos/preparing-models#code_changes

I have also been able to train a similar model using Keras. I can even save the model in the same export and export.meta format as I would get with TF.

from keras import backend as K

saver = tf.train.Saver()
session = K.get_session()
saver.save(session, 'export')

The part I am missing is how do I add the placeholders for input and output into the graph I build on Keras?

Talbot

Answer 1

After training your model on Google Cloud ML Engine (check out this awesome tutorial ), I named the input and output of my graph with

signature = predict_signature_def(inputs={'NAME_YOUR_INPUT': new_Model.input},
                                  outputs={'NAME_YOUR_OUTPUT': new_Model.output})

You can see the full exporting example for an already trained keras model 'model.h5' below.

import keras.backend as K
import tensorflow as tf
from keras.models import load_model, Sequential
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import tag_constants, signature_constants
from tensorflow.python.saved_model.signature_def_utils_impl import predict_signature_def

# reset session
K.clear_session()
sess = tf.Session()
K.set_session(sess)

# disable loading of learning nodes
K.set_learning_phase(0)

# load model
model = load_model('model.h5')
config = model.get_config()
weights = model.get_weights()
new_Model = Sequential.from_config(config)
new_Model.set_weights(weights)

# export saved model
export_path = 'YOUR_EXPORT_PATH' + '/export'
builder = saved_model_builder.SavedModelBuilder(export_path)

signature = predict_signature_def(inputs={'NAME_YOUR_INPUT': new_Model.input},
                                  outputs={'NAME_YOUR_OUTPUT': new_Model.output})

with K.get_session() as sess:
    builder.add_meta_graph_and_variables(sess=sess,
                                         tags=[tag_constants.SERVING],
                                         signature_def_map={
                                             signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature})
    builder.save()

You can also see my full implementation.

edit: And if my answer solved your problem, just leave me an uptick here :)

Christ

The full implementation link is broken. I think this might be where it's supposed to be now, but I can't even find any Python code there. – Slipsheet

Answer 2

I found out that in order to use keras on google cloud one has to install it with a setup.py script and put it on the same place folder where you run the gcloud command:

├── setup.py
└── trainer
    ├── __init__.py
    ├── cloudml-gpu.yaml
    ├── example5-keras.py

And in the setup.py you put content such as:

from setuptools import setup, find_packages

setup(name='example5',
  version='0.1',
  packages=find_packages(),
  description='example to run keras on gcloud ml-engine',
  author='Fuyang Liu',
  author_email='[email protected]',
  license='MIT',
  install_requires=[
      'keras',
      'h5py'
  ],
  zip_safe=False)

Then you can start your job running on gcloud such as:

export BUCKET_NAME=tf-learn-simple-sentiment
export JOB_NAME="example_5_train_$(date +%Y%m%d_%H%M%S)"
export JOB_DIR=gs://$BUCKET_NAME/$JOB_NAME
export REGION=europe-west1

gcloud ml-engine jobs submit training $JOB_NAME \
  --job-dir gs://$BUCKET_NAME/$JOB_NAME \
  --runtime-version 1.0 \
  --module-name trainer.example5-keras \
  --package-path ./trainer \
  --region $REGION \
  --config=trainer/cloudml-gpu.yaml \
  -- \
  --train-file gs://tf-learn-simple-sentiment/sentiment_set.pickle

To use GPU then add a file such as cloudml-gpu.yaml in your module with the following content:

trainingInput:
  scaleTier: CUSTOM
  # standard_gpu provides 1 GPU. Change to complex_model_m_gpu for 4 
GPUs
  masterType: standard_gpu
  runtimeVersion: "1.0"

Nootka

These are great instructions for running training, but the original question is about serving. Do you mind posting a new question about how to train Keras models on CloudML and self-answering with this valuable information? – Fibre

Oh, that totally makes sense. I will do that :) – Nootka

Answer 3

I don't know much about Keras. I consulted with some experts, and the following should work:

from keras import backend as k

# Build the model first
model = ...    

# Declare the inputs and outputs for CloudML
inputs = dict(zip((layer.name for layer in model.input_layers),
                  (t.name for t in model.inputs)))
tf.add_to_collection('inputs', json.dumps(inputs))

outputs = dict(zip((layer.name for layer in model.output_layers),
                   (t.name for t in model.outputs)))
tf.add_to_collection('outputs', json.dumps(outputs))

# Fit/train the model
model.fit(...)

# Export the model
saver = tf.train.Saver()
session = K.get_session()
saver.save(session, 'export')

Some important points:

• You have to call tf.add_to_collection after you create the model but before you ever call K.get_session(), fit etc.,
• You should be sure set the name of input and output layers when you add them to the graph because you'll need to refer to them when you send prediction requests.

Fibre

I also have a problem when making the prediction due to the keras learning phase, I train the model (learning phase = 1) then before I save I do K.set_learning_phase(0) but I still get an error: ERROR: (gcloud.beta.ml.local.predict) ERROR:root:Exception during running the graph: You must feed a value for placeholder tensor 'keras_learning_phase' with dtype bool [[Node: keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]] – Talbot

You could include {"keras_learning_phase": 0, ...} as part of the inputs to each instance. Does that work? – Fibre

Do you mean in the inputs from your code above? If so yes, it didn't work. The prediction complained the input tensor for keras_learning_phase had the wrong shape due to the first dimension ie there could be multiple inputs. Not sure how to just pass in a scalar. – Talbot

I added the input as inputs['keras_learning_phase'] = K.learning_phase().name and included {'keras_learning_phase': 0} in my input data for my cloud ml prediction. I get the error: InvalidArgumentError (see above for traceback): The second input must be a scalar, but it has shape [2], where 2 is the number of data points. – Talbot

If locally I do test_fn = K.function([model.layers[0].input, K.learning_phase()], [model.layers[4].output]) / results = test_fn([test_x, 0]) this works. I am not sure what CloudML is using to get the predictions but I guess it is something similar, not sure if you have any ideas? – Talbot

Backing up a bit, the first error that's complaining about the keras_learning_phase tensor: that's an indication that Keras has added a placeholder to the graph that's being exported. I'm not sure whether or not that can be suppressed. Assuming it cannot, then each request you send must treat that as one of the inputs to the graph. However, I'm not sure that particular placeholder is compatible with CloudML which assumes that each input is of rank at least 1, where the outermost dimension is a batch. It appears this placeholder is always a scalar in Keras. I can investigate a bit further. – Fibre

Yea, you are exactly right. That is where I'm up to in my investigations. I managed to replicate the model in TensorFlow and get it to work by adding a placeholder_with_default set to 1 for each dropout layer that way I can train using a value in the feed dict and cloud ML doesn't complain because it has a default scalar value. I haven't figure out how to get it working with Keras, perhaps the Keras code should have the option to set a default value. – Talbot

When you K.set_learning_phase(0) did you do it before the line "model = ..."? – Fibre

I have tried both, if I do it before "model = ..." I get a similar error saying it was expecting placeholder shape []. I have also tried training the model, saving the weights and config and loading the model as new_model = Model.from_config(config) new_model.set_weights(weights) as suggested here in section IV for TF serving. When I do that I get an error AttributeError: 'function' object has no attribute 'get_shape' which is traced back to the Model.from_config call. – Talbot

The solution I came up with for the same issue in pure TF was to have my dropout placeholders set to a default value of 1.0 i.e don't do drop out. That way GCP predict doesn't complain when I don't pass any value in t he feed dict. I am wondering if the solution would be in the Keras source code to make a change where by instead of K.set_learning_phase setting a constant to initialise the placeholder in ithe dropout layer with a default of 1.0. – Talbot

Sorry for the delayed response; I've been away. Let me follow up with some folks. – Fibre

No problem, thanks for getting back to me. I haven't had time to look into it any further myself yet. – Talbot

Here's the info I was given: Setting K.set_learning_phase(0) before building the model (before is important) does more than providing a default value for the learning phase placeholder: it literally skips all operations involving the learning phase (e.g. dropout), so that the resulting graph is an inference-only graph, with no dependency whatsoever on the learning phase placeholder. If you do that, no further errors can be related to the learning phase." – Fibre

Hi, sorry it has been so long since I got back to you. There were changes in CloudML so I had a lot of work to redo! Anyway, I have finally been able to get things working with dropout. Thank you so much for all your help. – Talbot

@MatthewJackson: you should post a self answer explaining what fixed it for you, so that this question won't keep popping up as unsolved. – Tripterous

@Tripterous I will get around to doing this, on vacation at the moment. – Talbot

Answer 4

Here's another answer that may help. Assuming you already have a keras model you should be able to append this to the end of your script and get an ML Engine compatible version of the model (protocol buffer). Note that you need to upload the saved_model.pb file and the sibling directory with variables to ML Engine for it to work. Note also that the .pb file must be named saved_model.pb or saved_model.pbtxt.

Assuming your model is name model

from tensorflow import saved_model

model_builder = saved_model.builder.SavedModelBuilder("exported_model")                                                     
inputs = {                                                                          
    'input': saved_model.utils.build_tensor_info(model.input)                    
}                                                                                   
outputs = {                                                                         
    'earnings': saved_model.utils.build_tensor_info(model.output)                
}                                                                                                                                                
signature_def = saved_model.signature_def_utils.build_signature_def(             
    inputs=inputs,                                                                  
    outputs=outputs,                                                                
    method_name=saved_model.signature_constants.PREDICT_METHOD_NAME              
)                                                                            
model_builder.add_meta_graph_and_variables(                                         
    K.get_session(),                                                                
    tags=[saved_model.tag_constants.SERVING],                                    
    signature_def_map={saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature_def
    })                                                                                                                                                                       
model_builder.save()   

will export the model to directory /exported_model.

Jem