Unable to make predictions on google cloud ml, whereas same model is working on the local machine

Question

I am trying to train a machine learning model usinf tensorflow library in the google cloud. I am able to train the model in the cloud after creating a bucket. I am facing the issue when I am tring to make predictions using the existing model. The code and the data is available in the following Github directory. https://github.com/terminator172/game-price-predictions

The tensorflow version on the cloud is 1.8 and the tensorflow version on my system is also 1.8

I tried to make predictions by giving the following input "gcloud ml-engine predict --model=earnings --version=v8 --json-instances=sample_input_prescaled.json"

It errored out with the following error "{ "error": "Prediction failed: Error during model execution: AbortionError(code=StatusCode.FAILED_PRECONDITION, details=\"Attempting to use uninitialized value output/biases4\n\t [[Node: output/biases4/read = IdentityT=DT_FLOAT, _output_shapes=[[1]], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]]\")" }"

Headmistress

Answer 1

The error message indicates that not all variables have been initialized. There is some sample code in the CloudML samples that demonstrate how to take care of initialization (link) Also, I recommend using tf.saved_model.simple_save on newer versions of TF. Try the following changes to your code:

def main_op():
  init_local = variables.local_variables_initializer()
  init_tables = lookup_ops.tables_initializer()
  return control_flow_ops.group(init_local, init_tables)

[...snip...]    

# This replaces everything from your SavedModelBuilder on
tf.saved_model.simple_save(
    session,
    export_dir='exported_model',
    inputs={'input': X},
    outputs={'earnings': prediction},
    legacy_init_op=main_op)  # This line is important

Ster

Answer 2

Your model directory in gcloud (the one that you provide with the --model flag) should contain 2 things:

1. The saved_model.pb file, containing the actual TensorFlow program, or model, and a set of named signatures, each identifying a function that accepts tensor inputs and produces tensor outputs.

2. The variables directory, containing a standard training checkpoint.

In case your variables directory is missing and you have only the saved_model.pb file, you can get this Attempting to use uninitialized value error. In order to fix it you just need to add the variables directory to your model directory in gcloud.

Reference: Tensorflow SavedModel format

Dairen