Spark CrossValidatorModel access other models than the bestModel?

Question

I am using Spark 1.6.1:

Currently I am using a CrossValidator to train my ML Pipeline with various parameters. After the training process I can use the bestModel property of the CrossValidatorModel to get the Model that performed best during the Cross Validation. Are the other models of the cross validation automatically discarded or can I select a model that performed worse than the bestModel?

I am asking because I am using the F1 Score metric for the cross validation but I am also interested in the weighedRecall of all of the models and not just of the model that has performed best during the crossvalidation

val folds = 6
val cv = new CrossValidator()
  .setEstimator(pipeline)
  .setEvaluator(new MulticlassClassificationEvaluator)
  .setEstimatorParamMaps(paramGrid)
  .setNumFolds(folds)

val avgF1Scores = cvModel.avgMetrics

val predictedDf = cvModel.bestModel.transform(testDf)

// Here I would like to predict as well with the other models of the cross validation

Manlove

Answer 1

Spark >= 2.4.0 ( >= 2.3.0 in Scala)

SPARK-21088 CrossValidator, TrainValidationSplit should collect all models when fitting - adds support for collecting submodels.

cv = CrossValidator(..., collectSubModels=True)

model = cv.fit(...)
model.subModels

Spark < 2.4

If you want to access all intermediate models you'll have to create custom cross validator from scratch. o.a.s.ml.tuning.CrossValidator discards other models, and only the best one and metrics are copied to the CrossValidatorModel.

See also Pyspark - Get all parameters of models created with ParamGridBuilder

Omnivore

Scala 2.3(maybe earlier) has .setCollectSubModels() – Powder

I used model.subModels but it returns an array. How can I get the F1 score or accuracy for each of those sub-models? – Samaniego

@savinaynarendra I've been working through a similar problem and still don't quite have the answer, but these model objects can be accessed by, e.g., model.subModels[1][0] . Which would give you the second model object in the array. – Fibster

Answer 2

If you're just looking to do this for experimentation as opposed to a production implementation of something, I recommend monkey-patching. Here is what I did to print out the intermediate training results. Just use CrossValidatorVerbose as a drop-in replacement for CrossValidator.

import numpy as np

from pyspark.ml.tuning import CrossValidator, CrossValidatorModel
from pyspark.sql.functions import rand


class CrossValidatorVerbose(CrossValidator):

    def _fit(self, dataset):
        est = self.getOrDefault(self.estimator)
        epm = self.getOrDefault(self.estimatorParamMaps)
        numModels = len(epm)

        eva = self.getOrDefault(self.evaluator)
        metricName = eva.getMetricName()

        nFolds = self.getOrDefault(self.numFolds)
        seed = self.getOrDefault(self.seed)
        h = 1.0 / nFolds

        randCol = self.uid + "_rand"
        df = dataset.select("*", rand(seed).alias(randCol))
        metrics = [0.0] * numModels

        for i in range(nFolds):
            foldNum = i + 1
            print("Comparing models on fold %d" % foldNum)

            validateLB = i * h
            validateUB = (i + 1) * h
            condition = (df[randCol] >= validateLB) & (df[randCol] < validateUB)
            validation = df.filter(condition)
            train = df.filter(~condition)

            for j in range(numModels):
                paramMap = epm[j]
                model = est.fit(train, paramMap)
                # TODO: duplicate evaluator to take extra params from input
                metric = eva.evaluate(model.transform(validation, paramMap))
                metrics[j] += metric

                avgSoFar = metrics[j] / foldNum
                print("params: %s\t%s: %f\tavg: %f" % (
                    {param.name: val for (param, val) in paramMap.items()},
                    metricName, metric, avgSoFar))

        if eva.isLargerBetter():
            bestIndex = np.argmax(metrics)
        else:
            bestIndex = np.argmin(metrics)

        bestParams = epm[bestIndex]
        bestModel = est.fit(dataset, bestParams)
        avgMetrics = [m / nFolds for m in metrics]
        bestAvg = avgMetrics[bestIndex]
        print("Best model:\nparams: %s\t%s: %f" % (
            {param.name: val for (param, val) in bestParams.items()},
            metricName, bestAvg))

        return self._copyValues(CrossValidatorModel(bestModel, avgMetrics))

NOTE: this solution also corrects what I see as a bug in v2.0.0 where the CrossValidationModel.avgMetrics are set to the sum of the metrics instead of the average.

Here is an example of the output for a simple 5-fold validation of ALS:

Comparing models on fold 1
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 1.122425  avg: 1.122425
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 1.123537  avg: 1.123537
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.123651  avg: 1.123651
Comparing models on fold 2
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 0.992541  avg: 1.057483
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 0.992541  avg: 1.058039
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 0.992541  avg: 1.058096
Comparing models on fold 3
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 1.141786  avg: 1.085584
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 1.141786  avg: 1.085955
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.141786  avg: 1.085993
Comparing models on fold 4
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 0.954110  avg: 1.052715
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 0.952955  avg: 1.052705
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 0.952873  avg: 1.052713
Comparing models on fold 5
params: {'regParam': 0.1, 'rank': 5, 'maxIter': 10}     rmse: 1.140098  avg: 1.070192
params: {'regParam': 0.01, 'rank': 5, 'maxIter': 10}    rmse: 1.139589  avg: 1.070082
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.139535  avg: 1.070077
Best model:
params: {'regParam': 0.001, 'rank': 5, 'maxIter': 10}   rmse: 1.070077

Dimitris

I had this problem but I wanted to extract more metrics than the one used for evaluation. I made some changes to the method you implemented link. Thanks for your answer, it was very usefull. – Vandal