Hyperopt: Define parameter which is dependent on other parameter

Question

I am using python package hyperopt and I have a parameter a which requires to be larger than parameter b.

For example, I hope my parameter space is like

from hyperopt import hp

space = {"b": hp.uniform(0, 0.5), "a": hp.uniform(b, 0.5)}

Which, requires a to be at least larger than b, how can I do that?

Thanks in advance

Foofaraw

Well one option is to run many evaluations and then choose the minimum from the cases which satisfy your constraint. – Headman

Answer 1

A simple option is to use the ability of hyperopt to nest parameters. You can thus define a hyper-parameter space like you want:

space = hp.uniform("a", hp.uniform("b", 0, 0.5), 0.5)

Only "a"'s value is passed to the function that you optimize (because this is the hyper-parameter space), but hyperopt.fmin() will return both parameters.

A similar option, but where the function to be optimized receives both parameters is:

b_var = hp.uniform("b", 0, 0.5)
space = {"b": b_var, "a": hp.uniform("a", b_var, 0.5)}

Finally, it might be simpler to change a bit the inputs to the optimized function: parameter a can be replaced by a_fraction running between 0 and 1 and interpolating between b and 0.5 (i.e. a_fraction = 0 yields a = b and a_fraction = 1 gives a = 0.5 inside the modified function to be optimized). The parameter space thus has the usual form:

space = {"b": hp.uniform("b", 0, 0.5), "a_fraction": hp.uniform("a_fraction", 0, 1)}

There is an interesting discussion at https://github.com/hyperopt/hyperopt/issues/175#issuecomment-29401501.

Thagard

That's very comprehensive. Thanks a lot. – Foofaraw

I have hyperopt 0.2.3 and when I try your first two solutions that involve putting one hp variable as an argument to another I get TypeError: len of pyll.Apply either undefined or unknown. Is this expected? Do these solutions no longer work? – Elise

Unfortunately, many things seem to have changed: hyperopt 0.2.3 doesn't even accept the very simple hp.uniform(0, 0.5) from the original question! – Thagard

@EricOLebigot any idea how to do this on the current version of hyperopt? – Noncombatant

Unfortunately no. However, you might want to look at the more recent and potentially more convenient optuna package. – Thagard

Answer 2

Maybe my find could help someone. I was using HyperOpt to optimise parameters of a piecewise affine function. Therefore I need nested parameters to properly set x parameters parts:

As nested parameters don't work (anymore) for HyperOpt, nor for Optuna... I recommend the pwlf library for all those who have the same problem.

This lib proposes 2 ways to extract the parameters of a piecewise linear function :

• By providing the number of parts
• By providing the breakpoint locations

If like me you are interested in finding the breakpoint locations you can provide the number of parts:

import pwlf
import pandas as pd

data = pd.DataFrame.from_dict({
    "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
    "y": [0, 4, 6, 7, 8, 9, 10, 7, 3, 1],
})

model = pwlf.PiecewiseLinFit(data.x, data.y)
nb_parts = 3
model.fit(nb_parts)

print(f"x part intersections: {model.fit_breaks}")
# x part intersections: [0. 1.99 6.00 9.]
print(f"linear coefficient of each part: {model.slopes}")
# linear coefficient of each part: [ 4.00,  1.00, -3.00]
print(f"linear bias of each part: {model.intercepts}")
# linear bias of each part: [-7.11e-06,  5.99e+00,  3.00e+01]

Hope it helps!

Tucana

I consider this an answer (not judging quality). So I removed your own statement that you consider it a comment. Please understand that if it really IS a comment you should not post it. meta.stackexchange.com/questions/214173/… – Mercie

Please add a demonstration of how the lib you recommend can be used to solve the problem discussed at the top of this page. Only mentioning a tool/lib is not really considered an answer. You spend much more effort and I am generous in judging the "only" here. But please add some more info. – Mercie