You can find a good general guide to hyperparameter optimization in the sklearn docs.
One simple but effective technique which you can apply to optimizing a LightFM model is random search. Roughly, it consists of the following steps:
- Split your data into a training set, a validation set, and a test set.
- Define a distribution for each hyperparameter you would like to optimize. For example, if you are optimizing your learning rate, you could use an exponential distribution with a mean of 0.05; if you are optimizing the loss function, you could sample uniformly from
['warp', 'bpr', 'warp-kos'].
- In each iteration of the optimization, sample all your hyperparameters and use them to fit the model on the training data. Evaluate the model's performance on the validation set.
- After performing a number of optimization steps, select the one with the best validation performance.
To gauge the performance of the final model you should use the test set: simply evaluate the best validation model on the test set.
The following script illustrates this:
import itertools
import numpy as np
from lightfm import LightFM
from lightfm.evaluation import auc_score
def sample_hyperparameters():
"""
Yield possible hyperparameter choices.
"""
while True:
yield {
"no_components": np.random.randint(16, 64),
"learning_schedule": np.random.choice(["adagrad", "adadelta"]),
"loss": np.random.choice(["bpr", "warp", "warp-kos"]),
"learning_rate": np.random.exponential(0.05),
"item_alpha": np.random.exponential(1e-8),
"user_alpha": np.random.exponential(1e-8),
"max_sampled": np.random.randint(5, 15),
"num_epochs": np.random.randint(5, 50),
}
def random_search(train, test, num_samples=10, num_threads=1):
"""
Sample random hyperparameters, fit a LightFM model, and evaluate it
on the test set.
Parameters
----------
train: np.float32 coo_matrix of shape [n_users, n_items]
Training data.
test: np.float32 coo_matrix of shape [n_users, n_items]
Test data.
num_samples: int, optional
Number of hyperparameter choices to evaluate.
Returns
-------
generator of (auc_score, hyperparameter dict, fitted model)
"""
for hyperparams in itertools.islice(sample_hyperparameters(), num_samples):
num_epochs = hyperparams.pop("num_epochs")
model = LightFM(**hyperparams)
model.fit(train, epochs=num_epochs, num_threads=num_threads)
score = auc_score(model, test, train_interactions=train, num_threads=num_threads).mean()
hyperparams["num_epochs"] = num_epochs
yield (score, hyperparams, model)
if __name__ == "__main__":
from lightfm.datasets import fetch_movielens
data = fetch_movielens()
train = data["train"]
test = data["test"]
(score, hyperparams, model) = max(random_search(train, test, num_threads=2), key=lambda x: x[0])
print("Best score {} at {}".format(score, hyperparams))
