Hyperparameter tuning locally -- Tensorflow Google Cloud ML Engine

Question

Is it possible to tune hyperparameters using ML Engine to train the model locally? The documentation only mentions training with hyperparameter tuning in the cloud (submitting a job), and has no mention to doing so locally.

Otherwise, is there another commonly used hyperparameter tuning that passes in command arguments to task.py as in the census estimator tutorial?

https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/census

Daryl

Answer 1

As Puneith said, hyperparamater tuning cannot run locally in ML-Engine.

SciKit Optimize provides an easy to use wrapper that works with any model including estimators. Just put the code that runs training for N epochs into its own function, which returns the evaluation 1-accuracy, 1-auroc or loss metric for minimizing.

import numpy as np
from skopt import gp_minimize

def train(hyperparam_config):
    # set from passed in hyperparameters
    learning_rate = hyperparam_config[0]
    num_layers = hyperparam_config[2]
    # run training
    res = estimator.train_and_evaluate()...
    return res['loss']  # return metric to minimize

hyperparam_config = [Real(0.0001, 0.01, name="learning_rate"),
                      Integer(3, 10, name="num_layers")]
res = gp_minimize(train, hyperparam_config)
with open('results.txt', 'w') as wf:
    wf.write(str(res))
print(res)

Source: https://github.com/scikit-optimize/scikit-optimize/blob/master/examples/hyperparameter-optimization.ipynb

Daryl

Answer 2

You cannot perform HPTuning (Bayesian Optimization based HPTuning which Cloud ML Engine supports) locally, since it's a managed service which Cloud ML Engine offers. There are other ways to perform Hyperparameter tuning e.g., Scikit-learn GridSearch but they are far less effective in this task.

Justinajustine

Answer 3

Check Sherpa, excellent Hyperparameter optimization library.

It says:

Hyperparameter optimization that enables researchers to experiment, visualize, and scale quickly

There are many HyperParameter optimization libraries out there, but using Sherpa one can visualize the results.

Bellbottoms