Reload Keras-Tuner Trials from the directory

Question

I'm trying to reload or access the Keras-Tuner Trials after the Tuner's search has completed for inspecting the results. I'm not able to find any documentation or answers related to this issue.

For example, I set up BayesianOptimization to search for the best hyper-parameters as follows:

## Build Hyper Parameter Search
tuner = kt.BayesianOptimization(build_model,
                     objective='val_categorical_accuracy',
                     max_trials=10,
                     directory='kt_dir',
                     project_name='lstm_dense_bo')

tuner.search((X_train_seq, X_train_num), y_train_cat,
             epochs=30,
             batch_size=64,
             validation_data=((X_val_seq, X_val_num), y_val_cat),
             callbacks=[callbacks.EarlyStopping(monitor='val_loss', patience=3, 
                                                restore_best_weights=True)])

I see this creates trial files in the directory kt_dir with project name lstm_dense_bo such as below:

Now, if I restart my Jupyter kernel, how can I reload these trials into a Tuner object and subsequently inspect the best model or the best hyperparameters or the best trial?

I'd very much appreciate your help. Thank you

Walker

Answer 1

I was trying to do the same thing. I was looking into the keras docs for an easier way than this but could not find one - so if any other SO-ers have a better idea, please let us know!

1. Load the previous tuner. Make sure overwrite=False or else you'll delete your trials.

workdir = "mlp_202202151345"
obj = "val_recall"
tuner = kt.Hyperband(
    hypermodel=build_model,
    metrics=metrics,
    objective=kt.Objective(obj, direction="max"),
    executions_per_trial=1,
    overwrite=False,
    directory=workdir,
    project_name="keras_tuner",
)

2. Look for a trial you want to load. Note that TensorBoard works really well for this. In this example, I'm loading 1a38ebaba07b77501999cb1c4ab9413e.

3. Here's the part that I could not find in Keras docs. This might be dependent on the tuner you use (I am using Hyperband):

tuner.oracle.get_trial('1a38ebaba07b77501999cb1c4ab9413e')

Returns a Trial object (also could not find in the docs). The Trial object has a hyperparameters attribute that will return that trial's hyperparameters. Now:

tuner.hypermodel.build(trial.hyperparameters)

Gives you the trial's model for training, evaluation, predictions, etc.

NOTE This seems convuluted and hacky, would love to see a better way.

Tillis

Why are you minimizing recall? – Larocca

Good catch! Should be max. I edited it. – Tillis

Answer 2

j7skov has correctly mentioned that you need to reload previous tuner and set the parameter overwrite=False(so that tuner will not overwrite already generated trials).

Further if you want to load first K best models then we need to use tuner's get_best_models method as below

# This will load 10 best hyper tuned models with the weights 
# corresponding to their best checkpoint (at the end of the best epoch of best trial).
best_model_count = 10
bo_tuner_best_models = tuner.get_best_models(num_models=best_model_count)

Then you can access a specific best model as below

best_model_id = 7
model = bo_tuner_best_models[best_model_id]

This method is for querying the models trained during the search. For best performance, it is recommended to retrain your Model on the full dataset using the best hyperparameters found during search, which can be obtained using tuner.get_best_hyperparameters().

tuner_best_hyperparameters = tuner.get_best_hyperparameters(num_trials=best_model_count)
best_hp = tuner_best_hyperparameters[best_model_id]
model = tuner.hypermodel.build(best_hp)

If you want to just display hyperparameters for the K best models then use tuner's results_summary method as below

tuner.results_summary(num_trials=best_model_count)

For further reference visit this page.

Sheikh

The output of get_best_models is a list sorted by objective metric. Calling the index trial_id is misleading. – Larocca

Answer 3

Inspired by j7skov, I found that the models can be reloaded by manipulating tuner.oracle.trials and tuner.load_model.

By assigning tuner.oracle.trials to a variable, we can find that it is a dict object containing all relavant trials in the tuning process. The keys of the dictionary are the trial_id, and the values of the dictionary are the instance of the Trial object. Alternatively, we can return the best few trials by using tuner.oracle.get_best_trials.

To inspect the hyperparameters of the trial, we can use the summary method of the instance.

To load the model, we can pass the trial instance to tuner.load_model.

Beware that different versions can lead to incompatibilities. For example the directory structure is a little different between keras-tuner==1.0 and keras-tuner==1.1 as far as I know.

Using your example, the working flow may be summarized as follows.

# Recreate the tuner object
tuner = kt.BayesianOptimization(build_model,
            objective='val_categorical_accuracy',
            max_trials=10,
            directory='kt_dir',
            project_name='lstm_dense_bo',
            overwrite=False)

# Return all trials from the oracle
trials = tuner.oracle.trials

# Print out the ID and the score of all trials
for trial_id, trial in trials.items():
    print(trial_id, trial.score)

# Return best 5 trials
best_trials = tuner.oracle.get_best_trials(num_trials=5)
for trial in best_trials:
    trial.summary()
    model = tuner.load_model(trial)
    # Do some stuff to the model

Zosema

Answer 4

using

tuner = kt.BayesianOptimization(build_model,
                         objective='val_categorical_accuracy',
                         max_trials=10,
                         directory='kt_dir',
                         project_name='lstm_dense_bo')

will load the tuner again.

Grand

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Doxy

Answer 5

The answer from hgh worked perfectly for me using the Hyperband search but I had to add tuner.reload() as follows:

tuner = kt.Hyperband(
       hypermodel=build_model,
       objective= kt.Objective("val_categorical_accuracy", direction="max"),
       max_epochs=50,
       factor=3 ,
       seed = 17,
       hyperband_iterations=1,
       distribution_strategy=tf.distribute.MirroredStrategy(),
       directory='models',
       project_name=EXPERIMENT,
       tuner_id = EXPERIMENT,
       overwrite=False,
       logger=NeptuneLogger()
       )
# Return all trials from the oracle
tuner.reload()
trials = tuner.oracle.trials

# Print out the ID and the score of all trials
for trial_id, trial in trials.items():
       print(trial_id, trial.score)

hp = tuner.oracle.get_trial("0028").hyperparameters

model = build_model(hp)

Discharge

Answer 6

I have found this way to load the keras_tuner trials into a tuner.

tuner = RandomSearch.load_from_dir('path/to/dir/tuner1')

and then you would continue with the training or select the best performing model as per your need.

N.B This works for RandomSearch but was not tested on your specific 'search' approach.

Conventionalism