Why netlib-java native blas/lapack libraries doesn't give performance improvement?

Question

I am using this piece of code to calculate spark recommendations:

    SparkSession spark = SparkSession
            .builder()
            .appName("SomeAppName")
            .config("spark.master", "local[" + args[2] + "]")
            .config("spark.local.dir",args[4])
            .getOrCreate();
    JavaRDD<Rating> ratingsRDD = spark
            .read().textFile(args[0]).javaRDD()
            .map(Rating::parseRating);
    Dataset<Row> ratings = spark.createDataFrame(ratingsRDD, Rating.class);
    ALS als = new ALS()
            .setMaxIter(Integer.parseInt(args[3]))
            .setRegParam(0.01)
            .setUserCol("userId")
            .setItemCol("movieId")
            .setRatingCol("rating").setImplicitPrefs(true);

    ALSModel model = als.fit(ratings);
    model.setColdStartStrategy("drop");
    Dataset<Row> rowDataset = model.recommendForAllUsers(50);

These are maven dependencies to make this piece of code work:

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>

Calculating recommendations with this code takes ~70sec for my data file. This code produces following warning:

WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK

Now I try to enable netlib-java by adding this dependency in maven:

    <dependency>
        <groupId>com.github.fommil.netlib</groupId>
        <artifactId>all</artifactId>
        <version>1.1.2</version>
        <type>pom</type>
    </dependency>

to avoid crashing of this new environment I had to do this extra trick:

LD_PRELOAD=/usr/lib64/libopenblas.so

Now it also works, it gives no warnings, but it works slower and it takes ~170sec on average to perform the same calculation. I am running this on CentOS.

Shouldn't it be faster with native libraries? Is it possible to make it faster?

Holloman

I was able to reproduce the warnings. However, I am able to get the result and all the results in the Spark example (Spark docs) within 8 seconds and even with show() I got it in 16 seconds. What parameters are using for setMaxIter() and master"local[" + args[2] + "]"? I am using 10 and 2 respectively. – Keilakeily

Can you share the dataset may be I am using the smaller one? – Keilakeily

drive.google.com/file/d/16a-U43TDUp8_U3oRG30bq08t51HhZbgd/view – Holloman

you can set maxiter to ~100 for example, to get long time running – Holloman

Answer 1

1. first you can check you centos version ,for centos 6 may not using the native libraries, check this

2. As far as I konw , the ALS algorithm has been improved since 2.0 version , you can check Highlights in 2.2

   And the source code from 2.2 as blow :

   so the native libraries has not help!

Tortfeasor