use trial.suggest_int to pick values from given list in optuna, just like trial.suggest_categorical does

Question

I'm working with optuna for hyperparameter tuning of ML models in Python. While defining the objective function for tuning a Deep Learning model I tried to define a list of choices from which the trail.suggest_int can pick up values from. For example -

'batch_size': trial.suggest_int('batch_size', [16, 32, 64, 128, 256])

optuna documentation suggest that trial.suggest_int should be in the following format

'some_param': trial.suggest_int('some_param', low, high, step)

my code looks something like below

def objective(trial):
        DL_param = {
            'learning_rate': trial.suggest_float('learning_rate', 1e-3, 1e-1),
            'optimizer': trial.suggest_categorical('optimizer', ["Adam", "RMSprop", "SGD"]),
            'h_units': trial.suggest_int('h_units', 50, 250, step = 50),
            'alpha': trial.suggest_float('alpha', [0.001,0.01, 0.1, 0.2, 0.3]),
            'batch_size': trial.suggest_int('batch_size', [16, 32, 64, 128, 256]),
        }
        DL_model = build_model(DL_param)
        DL_model.compile(optimizer=DL_param['optimizer'], loss='mean_squared_error')
        DL_model.fit(x_train, y_train, validation_split = 0.3, shuffle = True,
                                  batch_size = DL_param['batch_size'], epochs = 30)
        y_pred_2 = DL_model.predict(x_test)
        return mse(y_test_2, y_pred_2, squared=True)

I'm facing problem in defining a list for the parameters 'alpha' and 'batch_size'. Is there a way? something like trial.suggest_categorical can pick strings from the given list like in the above code

'optimizer': trial.suggest_categorical('optimizer', ["Adam", "RMSprop", "SGD"])

Any suggestions are welcome. Thanks in advance.

Exhalant

Answer 1

It turns out that you can just use trial.suggest_categorical to achieve your goal:

import optuna
def objective(trial):
        # define two variables:
        A = trial.suggest_categorical('A', [1,2,3])
        B = trial.suggest_categorical('B', [5,6])
        # minimize this toy objective:
        obj = A/B
        return obj
study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=20)

Unique

Answer 2

There is a module GridSampler in optuna that solves this question:

import optuna

def objective(trial):
        # define two variables:
        A = trial.suggest_float('A', 0.001, 0.01)
        B = trial.suggest_int('B', 10, 70)
        # minimize this toy objective:
        obj = A/B
        return obj

def optimization():
        # define custom values to search on:
        search_space = {'A':[0.0015, 0.003, 0.0075], 'B':[11, 23]}
        sampler = optuna.samplers.GridSampler(search_space)
        study = optuna.create_study(study_name="Optimization over given values", sampler=sampler)
        study.optimize(objective, n_trials = 6)

if __name__=='__main__':
    optimization()  

The out put is:

[I 2023-02-03 10:21:01,912] A new study created in memory with name: Optimization over given values
[I 2023-02-03 10:21:01,914] Trial 0 finished with value: 0.0006818181818181818 and parameters: {'A': 0.0075, 'B': 11}. Best is trial 0 with value: 0.0006818181818181818. 
[I 2023-02-03 10:21:01,916] Trial 1 finished with value: 0.00013043478260869567 and parameters: {'A': 0.003, 'B': 23}. Best is trial 1 with value: 0.00013043478260869567.
[I 2023-02-03 10:21:01,917] Trial 2 finished with value: 0.0003260869565217391 and parameters: {'A': 0.0075, 'B': 23}. Best is trial 1 with value: 0.00013043478260869567.
[I 2023-02-03 10:21:01,921] Trial 3 finished with value: 6.521739130434783e-05 and parameters: {'A': 0.0015, 'B': 23}. Best is trial 3 with value: 6.521739130434783e-05. 
[I 2023-02-03 10:21:01,927] Trial 4 finished with value: 0.00013636363636363637 and parameters: {'A': 0.0015, 'B': 11}. Best is trial 3 with value: 6.521739130434783e-05.
[I 2023-02-03 10:21:01,951] Trial 5 finished with value: 0.00027272727272727274 and parameters: {'A': 0.003, 'B': 11}. Best is trial 3 with value: 6.521739130434783e-05.

Propagate

Answer 3

Just add one comment to @mohammad-joshaghani’s solution, when using search_space and GridSampler to define custom values for hyper-parameters, one has to provide custom values for all the hyper-parameters that need to be tuned and define in objective() function. Otherwise, any attempts to run the optimisation for ML models would be terminated with complained error info from optuna/samplers/_grid.py, line 186, in sample_independent raise ValueError(message) as follows:

“ValueError: The parameter name, learning_rate, is not found in the given grid.”

On the other hand, if someone prefers to provide user-specified values for only a subset of hyper-parameters, @youjun-hu’s solution is recommended.

Brownson