I've got a local Kafka running using the following docker-compose.yml file: version: '2' services: zookeeper: image: "confluentinc/cp-zookeeper:5.0.1" environment: ZOOKEEPER_CLIENT_PO...

How to get current_date - 1 day in sparksql, same as cur_date()-1 in mysql.

A legacy app has the below working Gatling test private val getUserByTuid = scenario("GetActiveUserInfoWrapper") .feed(users.toArray.circular) .exec( http("GET /v1/foo/{id}") .get("/v1/foo/${i...

Taken from "Scala with cats" (page 18): Implicit Conversions When you create a type class instance constructor using an implicit def, be sure to mark the parameters to the method as impl...

I'm trying to compile a mixed java-scala code using intelliJ 14.1.4 but it keeps giving me this error: Error Compiling SBT component 'compiler-interface-2.9.2-52.0' I have tried to downgrade my sc...

I have a Spark Streaming application that reads data from multiple Kafka topics. Each topic has a different type of data, and thus requires a different processing pipeline. My initial solution was...

I keep getting Forward reference extends over definition of value a error while trying to compile my application (inside SBT). a is just val a = "", the error is triggered by accessing a partic...

Checkpoint version: val savePath = "/some/path" spark.sparkContext.setCheckpointDir(savePath) df.checkpoint() Write to disk version: df.write.parquet(savePath) val df = spark.read.parque...

I am trying to build a Scala project on IntelliJ using Gradle. I am able to make but compile keeps failing with a stack overflow error. I looked through other posts with similar issues which sugg...

I'm trying to use Apache Iceberg for writing data to a specified location(S3/local). Following is the configuration used below. SBT: libraryDependencies += "org.apache.spark" %% "spa...

I'm using sbt-native-packager 1.0.0-M5 to create my docker image. I need to add a file that's not a source file or in the resource folder. My docker commands are as follows: dockerCommands := Seq(...

I have my timestamp in UTC and ISO8601, but using Structured Streaming, it gets automatically converted into the local time. Is there a way to stop this conversion? I would like to have it in UTC. ...

Given Table 1 with one column "x" of type String. I want to create Table 2 with a column "y" that is an integer representation of the date strings given in "x". Essential is to keep null values in...

Im using scala Map#get function, and for every accurate query it returns as Some[String] IS there an easy way to remove the Some? Example: def searchDefs{ print("What Word would you like define...

I have a method that should convert a list to an Option of an object, or None if the list is empty. def listToOption(myList: List[Foo]): Option[Bar] = { if(myList.nonEmpty) Some(Bar(myList)) els...

Let's say that I have a trait, Parent, with one child, Child. scala> sealed trait Parent defined trait Parent scala> case object Boy extends Parent defined module Boy I write a function t...

I use EMR Notebook connected to EMR cluster. Kernel is Spark and language is Scala. I need some jars that are located in S3 bucket. How can I add jars? In case of 'spark-shell' it's easy: spar...

I read data from a csv file ,but don't have index. I want to add a column from 1 to row's number. What should I do,Thanks (scala)

I am using Spark 2 and Scala 2.11 in a Zeppelin 0.7 notebook. I have a dataframe that I can print like this: dfLemma.select("text", "lemma").show(20,false) and the output looks like: +---------...

Problem I am trying to run a remote Spark Job through IntelliJ with a Spark HDInsight cluster (HDI 4.0). In my Spark application I am trying to read an input stream from a folder of parquet files f...

What are the possible ways to make a HashSet thread safe? Saw some samples as given below. var test = new mutable.HashSet[Long] with mutable.SynchronizedSet[Long] SynchronizedSet is deprecated ...

I was looking at the DataFrame API, i can see two different methods doing the same functionality for removing duplicates from a data set. I can understand dropDuplicates(colNames) will remove dupl...

I have three strings, for example "A", "B", "C". I have to produce the string which results from concatenating them, only the second string must be padded with whitespace to a given length. This w...

Using Sbt in order to run scalatest tests, you can put all test of a suite in parallel using the trait ParallelTestExecution and this works out of the box. How to obtain the same result running ...

Look at this code snippet: userService.insert(user) match { case Success(f) => Logger.debug("User created successfully") case Failure(e) => { // how do I determine the type of `e`? } } ...