I want to switch my notebook easily between different kernels. One use case is to quickly test a piece of code in tensorflow 2, 2.2, 2.3, and there are many similar use cases. However I prefer to d...

I know Apache Livy is the rest interface for interacting with spark from anywhere. So what is the benefits of using Apache Livy instead of spark-jobserver. What are the drawbacks of spark-jobserver...

I created an EMR cluster on AWS with Spark and Livy. I submitted a custom JAR with some additional libraries (e.g. datasources for custom formats) as a custom JAR step. However, the stuff from the ...

When launching my code with scala -cp assembly.jar class.A --config-path confFile I get java.lang.IllegalStateException: No LivyClientFactory implementation was found But when launching throu...

I've been trying to create a new spark session with Livy 0.7 server that runs on Ubuntu 18.04. On that same machine I have a running spark cluster with 2 workers and I'm able to create a normal spa...

I am running a Spark job using Scala, but it gets stuck not executing and tasks by my worker nodes. Currently I am submitting this to Livy, which submits to our Spark Cluster with 8 cores and 12GB...

We are trying to run a spark program using NiFi. This is the basic sample we tried to follow. We have configured Apache-Livy server in 127.0.0.1:8998. ExecutiveSparkInteractive processor is used ...

I am currently trying to mock a livy server from scala to run unit tests. Basically, I would like to test submitting jar to a livy client. I'm trying to adapt the code I found here (livy)HttpClient...

I'm using Zeppelin v0.7.3 notebook to run Pyspark scripts. In one paragraph, I am running script to write data from dataframe to a parquet file in a Blob folder. File is partitioned per country. Nu...

I am creating an EMR cluster, and using jupyter notebook to run some spark tasks. My tasks die after approximately 1 hour of execution, and the error is: An error was encountered: Invalid status c...

I am running Pyspark scripts to write a dataframe to a csv in jupyter Notebook as below: df.coalesce(1).write.csv('Data1.csv',header = 'true') After an hour of runtime I am getting the below err...

As per title, I would like to request a calculation to a Spark cluster (local/HDInsight in Azure) and get the results back from a C# application. I acknowledged the existence of Livy which I under...

I am trying to run local jar file with spark-submit which is working perfectly fine. Here is the command- spark-submit --class "SimpleApp" --master local myProject/target/scala-2.11/simple-project...

I'm trying to use Livy to remotely submit several Spark jobs. Lets say I want to perform following spark-submit task remotely (with all the options as-such) spark-submit \ --class com.company.driv...

I want to use Airflow for orchestration of jobs that includes running some pig scripts, shell scripts and spark jobs. Mainly on Spark jobs, I want to use Apache Livy but not sure whether it is goo...

I recently started working with Spark Scala, HDFS, sbt and Livy. Currently I tried to create livy batch. Warning: Skip remote jar hdfs://localhost:9001/jar/project.jar. java.lang.ClassNotFoundExc...

Is it possible to submit and run Spark jobs concurrently in the same AWS EMR cluster ? If yes then could you please elaborate ?

I’ve written a Java client for parts of Livy’s REST API at https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md. The client uses Spring’s RestTemplate.getForObject() and postForObj...

I am executing a statement in Livy Server using HTTP POST call to localhost:8998/sessions/0/statements, with the following body { "code": "spark.sql(\"select * from test_table limit 10\")" } I ...

I am trying to submit Python file to REST API but it is always giving the error. I am using the local mode and the command which I am running is given below: $curl -X POST --data '{"file":"/Users/...

I'm building a RESTful API on top of Apache Spark. Serving the following Python script with spark-submit seems to work fine: import cherrypy from pyspark.sql import SparkSession spark = SparkSess...