purpose of 'num_samples' in Tune of Ray package for hyperprameter optimization

Question

I am trying to perform a hyper parameter optimization task for a LSTM (pure Tensorflow) with Tune. I followed their example on the hyperopt algorithm. In the example they have used the below line inside the 'config' section.

"num_samples": 10 if args.smoke_test else 1000,

The documenatation does not explain what this is. I am uable to determine if this is a useful piece of code or how am I supposed to alter this for my scenario. So it will be great if I can know the meaning of this line of code.

The example hyperopt code can be found through this link

Valonia

this is called a ternary expression - it sets the "num_samples" variable to 10 or 1000 depending on some other given variable. From the names I would guess it samples some given data at 10 or 1000 points to create whatever it does ... you would need to look through the documentation to see for yourself. "num_samples" is provided as kwargs param. Maybesomewhere around here: ray.readthedocs.io/en/latest/… – Higginbotham

Answer 1

You can find the parameter in the documentation of run_experiments.

By default, each random variable and grid search point is sampled once. To take multiple random samples, add num_samples: N to the experiment config. If grid_search is provided as an argument, the grid will be repeated num_samples of times.

Essentially the parameter is part of the configuration and can be used to sample your data multiple times instead of only once.

Your demo code however uses run_experiment:

config = {
    "my_exp": {
        "run": "exp",
        "num_samples": 10 if args.smoke_test else 1000,
        "config": {
            "iterations": 100,
        },
        "stop": {
            "timesteps_total": 100
        },
    }
}
algo = HyperOptSearch(space, max_concurrent=4, reward_attr="neg_mean_loss")
scheduler = AsyncHyperBandScheduler(reward_attr="neg_mean_loss")
run_experiments(config, search_alg=algo, scheduler=scheduler)  # here the config is passed

Higginbotham

Thanks, any idea what 'args.smoke_test' is supposed to do? – Valonia

args.smoke_test is just a boolean we use to make running tests easy. You can simply consider the line to basically be "num_samples": 10. – Liturgist

@Sul 10 if args.smoke_test else 1000, is a ternary expression that says "if args.smoke_test is true use 10 else 1000 - think of it like a "small" and a "big" testingscenario based on some input. Hint: if you put a @ before a username it auto_completes it for you if this person was inside this answer/qeustion and more importantly the person gets a message that someone sain something - somehow your comment eluded me :) – Higginbotham

Answer 2

As per the documentation:

num_samples (int) – Number of times to sample from the hyperparameter space. Defaults to 1. If grid_search is provided as an argument, the grid will be repeated num_samples of times.

Substitue of repeat:

repeat (int) – Deprecated and will be removed in future versions of Ray. Use num_samples instead

Usage:

"num_samples": 10

num_samples=10

class ray.tune.Experiment(name,run,stop=None,config=None,trial_resources=None,
repeat=1,num_samples=1,local_dir=None,upload_dir=None,checkpoint_freq=0,
checkpoint_at_end=False,max_failures=3,restore=None)

Micky