I'm trying to use spark mllib lda to summarize my document corpus. My problem setting is as bellow. about 100,000 documents about 400,000 unique words 100 cluster I have 16 servers (each has ...

How can we do ARIMA modeling in spark scala? Can we directly import any ARIMA package like regression or classification? In Spark's ml library, we do not have anything like ARIMA model.

I am using this piece of code to calculate spark recommendations: SparkSession spark = SparkSession .builder() .appName("SomeAppName") .config("spark.master", "local[" + args[2] + "]") .confi...

I would like to do some DBSCAN on Spark. I have currently found 2 implementations: https://github.com/irvingc/dbscan-on-spark https://github.com/alitouka/spark_dbscan I have tested the first on...

I have read somewhere that MLlib local vectors/matrices are currently wrapping Breeze implementation, but the methods converting MLlib to Breeze vectors/matrices are private to org.apache.spark.mll...

When we do a k-fold Cross Validation we are testing how well a model behaves when it comes to predict data it has never seen. If split my dataset in 90% training and 10% test and analyse the model ...

I'm trying to extract the feature importance's of a random forest classifier model I have trained using Pyspark. I referred to the following article to get the feature importance scores for the ran...

I am trying to build decision tree and random forest classifier on the UCI bank marketing data -> https://archive.ics.uci.edu/ml/datasets/bank+marketing. There are many categorical features (having...

I've created a PipelineModel for doing LDA in Spark 2.0 (via PySpark API): def create_lda_pipeline(minTokenLength=1, minDF=1, minTF=1, numTopics=10, seed=42, pattern='[\W]+'): """ Create a pipel...

I made a random forest model using python's sklearn package where I set the seed to for example to 1234. To productionise models, we use pyspark. If I was to pass the same hyperparmeters and same s...

Seeing a weird problem when trying to generate one-hot encoded vectors for categorical features using pyspark's OneHotEncoder (https://spark.apache.org/docs/2.1.0/ml-features.html#onehotencoder) wh...

I noticed there are two LinearRegressionModel classes in SparkML, one in ML package (spark.ml) and another one in MLLib (spark.mllib) package. These two are implemented quite differently - e.g. the...

I have a spark Dataframe with two coulmn "label" and "sparse Vector" obtained after applying Countvectorizer to the corpus of tweet. When trying to train Random Forest Regressor model i found that...

I am using the standard (string indexer + one hot encoder + randomForest) pipeline in spark, as shown below labelIndexer = StringIndexer(inputCol = class_label_name, outputCol="indexedLabel").fi...

How can one apply some function in parallel on chunks of a sparse CSR array saved on disk using Python? Sequentially this could be done e.g. by saving the CSR array with joblib.dump opening it with...

I am working to create a LDA model. Here is what I have done so far- created a unigram and converted the dataframe to RDD based on this post. Here is the code: countVectors = CountVectorizer(...

I am attempting to fill in missing values in my Spark dataframe with the previous non-null value (if it exists). I've done this type of thing in Python/Pandas but my data is too big for Pandas (on ...

I want to overwrite a spark column with a new column which is a binary flag. I tried directly overwriting the column id2 but why is it not working like a inplace operation in Pandas? How to do it...

I'm doing some testing with Apache Spark, for my final project in college. I have a data set that I use to generate a decision tree, and make some predictions on new data. In the future, I think t...

I am trying to build a Movie Recommender System Using Apache Spark MLlib. I have written a code for recommender in java and its working fine when run using spark-submit command. My run command loo...

I unable to save random forest model generated using ml package of python/spark. >>> rf = RandomForestClassifier(labelCol="label", featuresCol="features") >>> pipeline = Pipeline...

According to mllib.feature.Word2Vec - spark 1.3.1 documentation [1]: def setNumIterations(numIterations: Int): Word2Vec.this.type Sets number of iterations (default: 1), which should be smalle...

I'm trying to figure out how an ALS model can predict values for new users in between it being updated by a batch process. In my search, I came across this stackoverflow answer. I've copied the ans...

I want to change List to Vector in pySpark, and then use this column to Machine Learning model for training. But my spark version is 1.6.0, which does not have VectorUDT(). So what type shoul...

I have a saved PipelineModel: pipe_model = pipe.fit(df_train) pipe_model.write().overwrite().save("/user/pipe_text_2") And now I want to add to this Pipe a new already fited PipelineModel: pipe...