scala Questions
2
Solved
PS. There's a similar question here, but that is in mvn and my project is in sbt.
First up, a few required informations:
Spark Installed Version: 2.4.0
Scala Installed Version: 2.11.12
I'm try...
Pelagic asked 18/4, 2019 at 20:14
1
I have to do simple remove "." from the strings before doing word count. It is working fine in two statements but giving me below error while writing in single statement. Am I doing something like ...
Guillerminaguillermo asked 2/5, 2018 at 1:28
2
For our use case we need to load in json files from an S3 bucket. As processing tool we are using AWS Glue. But because we will soon be migrating to Amazon EMR, we are already developing our Glue j...
Weinrich asked 24/1, 2023 at 15:38
3
Solved
I am building a Spark Structured Streaming application where I am doing a batch-stream join. And the source for the batch data gets updated periodically.
So, I am planning to do a persist/unpersist...
Rizzo asked 11/2, 2021 at 12:32
2
Solved
I have created one small program and in order to test it I have write small Scala Test class. But when I tried to execute scala test I was getting below error, please advise,
java.lang.NoSuchMetho...
Aphonic asked 12/12, 2014 at 23:9
4
Solved
I got the error "IOException: Cannot run program "javac": CreateProcess error=2, The system cannot find the file specified" while running my scala project in play 2. why does this error occur . Ple...
Lounging asked 23/9, 2013 at 10:55
4
Running into issues when trying to use kerberos auth with connecting to Kafka. Using scala and my jaas.config looks something like this.
KafkaClient {
com.sun.security.auth.module.Krb5LoginModul...
Paramorphism asked 3/3, 2020 at 4:31
2
Solved
I'm looking for a stripped down (so NOT like squeryl, or anything that creates typesafe queries where I have to respecify the table just to get some data) wrapper for jdbc.
i'm using scala 2.9 an...
Interlaken asked 3/2, 2012 at 18:34
5
I am relatively new to Scala Test, and so I consulted the documentation on how to test on Either values.
I tried to replicate the instructions like that:
import org.scalatest.EitherValues
import or...
7
In jenkins I'm seeing the following error message in the jenkins console while building any of our projects that are integrated with Gitlab.
Failed to update Gitlab commit status for project '120'...
Exsert asked 22/3, 2018 at 4:32
2
Solved
I have a JSON string and I created a function which parses this JSON as an object using Scala case class. I wrote the below code to parse it in a generic way. However, It gave me an error:
def get...
5
Solved
Besides using match, is there an Option-like way to getOrElse the actual content of the Right or Left value?
scala> val x: Either[String,Int] = Right(5)
scala> val a: String = x match {
ca...
4
Solved
I'm unable to use sbt or run tests from within IntelliJ on an existing Scala project. My setup seems basic.
% brew install coursier/formulas/coursier && cs setup
% cs setup
Checking if a JV...
7
Solved
I've got a fairly simple job coverting log files to parquet. It's processing 1.1TB of data (chunked into 64MB - 128MB files - our block size is 128MB), which is approx 12 thousand files.
Job works...
Asphyxia asked 9/6, 2017 at 13:40
3
Solved
I want to write one large sized dataframe with repartition, so I want to calculate number of repartition for my source dataframe.
numberofpartition = {size of dataframe/default_blocksize}
How to c...
Crept asked 21/4, 2020 at 7:45
1
I have a server-side application that runs through a large number of image URLs and uploads the images from these URLs to S3.
The files are served over HTTP. I download them using InputStream I get...
Outpost asked 13/2, 2019 at 18:3
1
I know functions in Python are 1st class citizens, meaning that they are objects of the function class similar to how 5 is an object of the int class. This means that at some point in their life-ti...
10
Solved
How would you find minValue below?
I have my own solution but want to see how others would do it.
val i1: Option[Int] = ...
val i2: Option[Int] = ...
val defaultValue: Int = ...
val minValue = ?
...
Lubra asked 28/9, 2012 at 12:22
7
Solved
I would like to include null values in an Apache Spark join. Spark doesn't include rows with null by default.
Here is the default Spark behavior.
val numbersDf = Seq(
("123"),
("456"),
(null),...
Locally asked 18/1, 2017 at 20:21
4
Spark can use Hadoop S3A file system org.apache.hadoop.fs.s3a.S3AFileSystem. By adding the following into the conf/spark-defaults.conf, I can get spark-shell to log to the S3 bucket:
spark.jars.pa...
Cauchy asked 6/10, 2016 at 22:23
5
Solved
I'm trying to port a Scala system to Mockito2. There are a few test cases that use doReturn and now in Mockito 2.18.0 I get this error:
Error:(34, 5) ambiguous reference to overloaded definition,
...
1
Solved
I have small csv file with a counties dictionary which I want to use in a large spark job. I load this file via --files option of spark-submit:
spark-submit --master yarn <some config stuff> ...
Consistency asked 17/6 at 7:48
2
Solved
In Scala fs2 library for functional streams:
I am trying to understand the difference between flatMap, flatTap, evalMap and evalTap. They all seem to perform the same thing, which is transformatio...
Helenehelenka asked 12/11, 2019 at 14:58
2
Solved
Hi I'm runnign example on Spark site at http://spark.apache.org/docs/1.2.0/quick-start.html#self-contained-applications
spark version: spark-1.4.0
sbt version: 0.13.8
And I ran command "sbt run"...
Lexie asked 27/8, 2015 at 5:41
4
Solved
I am running a Spark application (version 1.6.0) on a Hadoop cluster with Yarn (version 2.6.0) in client mode. I have a piece of code that runs a long computation, and I want to kill it if it takes...
Incorporeal asked 25/9, 2016 at 10:17
1 Next >
© 2022 - 2024 — McMap. All rights reserved.