Why does spark-ml ALS model returns NaN and negative numbers predictions?

Question

Actually I'm trying to use ALS from spark-ml with implicit ratings.

I noticed that some predictions given by my trained model are negative or NaN, why is it?

Jerz

experienced similar issue. possible reason seen in the answer of this question, #37380251 – Hep

Can you show us how you created your model ? – Validate

@Validate databricks-prod-cloudfront.cloud.databricks.com/public/… – Jerz

I'll take a look at it. You are using Spark 2+ ? – Validate

@Validate Yes, 2.1 – Jerz

Ok set "nonnegative=True" for ALS ! That should remove negative values. – Validate

I see but is it natural in the algorithm? And by the way I used the whole dataset don't know why it's returning NaN – Jerz

Well you are just setting the nonnegative constraint to compute least squares – Validate

Let us continue this discussion in chat. – Validate

Answer 1

Apache Spark provides an option to force non negative constraints on ALS.

Thus, to remove these negative values, you'll just need to set :

Python:

nonnegative=True

Scala:

setNonnegative(true)

when creating your ALS model, i.e :

>>> als = ALS(rank=10, maxIter=5, seed=0, nonnegative=True)

Non-negative matrix factorization (NMF or NNMF), also called non-negative matrix approximation is a group of algorithms in multivariate analysis and linear algebra where a matrix V is factorized into (usually) two matrices W and H, with the property that all three matrices have nonnegative elements [Ref. Wikipedia].

If you want to read more about NMF , I'd recommend reading the following paper :

• Collaborative Filtering via Ensembles of Matrix Factorizations

As for NaN values, usually it's due to splitting your dataset which can lead to unseen items or users if one of them isn't present in the training set and for the matter just present in the testing set. This might also happen if you cross validated your training. For the matter, there is a couple of JIRAs that are marked resolved for 2.2 :

• https://issues.apache.org/jira/browse/SPARK-14489.
• https://issues.apache.org/jira/browse/SPARK-19345.

The latest will allow you set the cold start strategy to use when creating your model.

Validate