What are hp.Discrete and hp.Realinterval? Can I include more values in hp.realinterval instead of just 2?

Question

I am using Hyperparameter using HParams Dashboard in Tensorflow 2.0-beta0 as suggested here https://www.tensorflow.org/tensorboard/r2/hyperparameter_tuning_with_hparams

I am confused in step 1, I could not find any better explanation. My questions are related to following lines:

HP_NUM_UNITS = hp.HParam('num_units', hp.Discrete([16, 32]))
HP_DROPOUT = hp.HParam('dropout', hp.RealInterval(0.1, 0.2))
HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd']))

My question: I want to try more dropout values instead of just two (0.1 and 0.2). If I write more values in it then it throws an error- 'maximum 2 arguments can be given'. I tried to look for documentation but could not find anything like from where these hp.Discrete and hp.RealInterval functions came. Any help would be appreciated. Thank you!

Sportive

I would also like some clarification about the same docs. In their example linked below, the discrete and RealInterval act the same way. tensorflow.org/tensorboard/… – Wasserman

Answer 1

Good question. They notebook tutorial lacks in many aspects. At any rate, here is how you do it at a certain resolution res

for dropout_rate in tf.linspace(
      HP_DROPOUT.domain.min_value,
      HP_DROPOUT.domain.max_value,
      res,): 

Suddenly

Answer 2

By looking at the implementation to me it really doesn't seem to be GridSearch but MonteCarlo/Random search (note: this is not 100% correct, please see my edit below)

So on every iteration a random float of that real interval is chosen

If you want GridSearch behavior just use "Discrete". That way you can even mix and match GridSearch with Random search, pretty cool!

Edit: 27th of July '22: (based on the comment of @dpoiesz)

Just to make it a little more clear, as it is sampled from the intervals, concrete values are returned. Therefore, those are added to the grid dimension and grid search is performed using those

Reversioner

In the guide the question linked to, both discrete and RealInterval act the same way. Their example performs a grid search – Wasserman

Answer 3

RealInterval is a min, max tuple in which the hparam will pick a number up.
Here a link to the implementation for better understanding. The thing is that as it is currently implemented it does not seems to have any difference in between the two except if you call the sample_uniform method.

Oui

Answer 4

Note that tf.linspace breaks the mentioned sample code when saving current value.

See https://github.com/tensorflow/tensorboard/issues/2348

In particular OscarVanL's comment about his quick&dirty workaround.

Aleppo