scala Questions
3
I am trying to implement topological sort using Spark's GraphX library.
This is the code I've written so far:
MyObject.scala
import java.util.ArrayList
import scala.collection.mutable.Queue
im...
Eventide asked 18/10, 2016 at 13:11
3
The Spark API Doc's show how to get a pretty-print snippit from a dataset or dataframe sent to stdout.
Can this output be directed to a log4j logger? Alternately: can someone share code which will...
Kori asked 11/1, 2017 at 20:42
2
Solved
I am trying to use the should matchers on a case class
case class ListOfByteArrayCaseConfig(
@BeanProperty
permissions: java.util.List[Array[Byte]]
)
With the following test case
val orig ...
Hanger asked 22/10, 2012 at 21:4
9
Solved
I would like to know what the best Scala imitation of Groovy's safe-dereference operator (?.), or at least some close alternatives are?
I've discussed it breifly on Daniel Spiewak's blog, but woul...
3
Having seen the answers coming out of questions like this one involving horror shows like trying to catch the NPE and dredge the mangled name out of the stack trace, I am asking this question so I ...
Congressman asked 1/9, 2009 at 20:1
2
We have a project with huge configuration files built using hocon configs.
There is an intention to use variables to create template_section and set up some values in the template based on some opt...
Presently asked 15/2, 2016 at 14:54
4
Solved
Given following code:
import java.sql.Date
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
object SortQuestion extends App{
val spark = SparkSession.builder().ap...
Dace asked 5/4, 2018 at 11:34
7
How to find size (in MB) of dataframe in pyspark,
df = spark.read.json("/Filestore/tables/test.json")
I want to find how the size of df or test.json
Noteworthy asked 16/6, 2020 at 15:15
7
Solved
Is there an easy way to convert a case class into a tuple?
I can, of course, easily write boilerplate code to do this, but I mean without the boilerplate.
What I'm really after is a way to easily...
Coraliecoraline asked 10/11, 2011 at 23:42
5
Solved
I have two DataFrames in Spark SQL (D1 and D2).
I am trying to inner join both of them D1.join(D2, "some column")
and get back data of only D1, not the complete data set.
Both D1 and D2 are ha...
Striking asked 2/8, 2016 at 13:2
3
Solved
I have a scenario to compare two different tables source and destination from two separate remote hive servers, can we able to use two SparkSessions something like I tried below:-
val spark = Spa...
Altimetry asked 6/7, 2017 at 12:43
1
Solved
Specifically using the new enum keyword provided by Scala 3...
enum Translation(val bit: Byte):
case FlipX extends Translation(1)
case FlipY extends Translation(2)
case RotateClockwise extends T...
Leesa asked 19/5 at 19:14
2
Intellij Idea add Scala support to existing Java project.
Just cannot figure it out How to add Scala worksheet to existing Java project?
It would be just nice to have Scala REPL or worksheet.
But a...
Hardheaded asked 22/3, 2015 at 23:15
5
I am trying to follow some tutorials to learn more about Scala and I am trying to use it in IntelliJ in my computer (MacBook Pro M1). I installed sdk using sdkman. However, I get certain errors reg...
11
Solved
When I create a DataFrame from a JSON file in Spark SQL, how can I tell if a given column exists before calling .select
Example JSON schema:
{
"a": {
"b": 1,
"c": 2
}
}
This is what I want ...
Easiness asked 9/3, 2016 at 22:40
2
How to make a list lazy, i.e. create a LazyList given a usual one? I tried to find suitable method in Scala documentation, but there is no such function.
Jipijapa asked 30/7, 2020 at 20:30
9
Solved
I have a text file on HDFS and I want to convert it to a Data Frame in Spark.
I am using the Spark Context to load the file and then try to generate individual columns from that file.
val myFile...
Praemunire asked 21/4, 2016 at 10:6
4
Solved
I've got a set of rows in a database, and I'd like to provide an interface to spin through them like this:
def findAll: Iterable[MyObject]
Where we don't require having all the instances in memo...
4
Solved
Is there a way to destructure input parameters of a function in Scala (akin to Clojure)?
So, instead of
scala> def f(p: (Int, Int)) = p._1
f: (p: (Int, Int))Int
I'd like to have this (it doe...
4
Solved
How to split a string by a delimiter from the right?
e.g.
scala> "hello there how are you?".rightSplit(" ", 1)
res0: Array[java.lang.String] = Array(hello there how are, you?)
Python has a ....
Ladylove asked 2/4, 2013 at 2:30
3
Solved
I have a text file of size 100-200 GB. So I wish to store in a compressed format (such as zip). However, I need to process it one line at a time due to its size. Though it is straightforward to rea...
Hateful asked 24/5, 2021 at 21:9
6
Solved
I'm not able to locate error logs or message's from println calls in Scala while running jobs on Spark in EMR.
Where can I access these?
I'm submitting the Spark job, written in Scala to EMR usin...
Metastasis asked 27/5, 2015 at 23:38
6
Solved
I want to create a hive table using my Spark dataframe's schema. How can I do that?
For fixed columns, I can use:
val CreateTable_query = "Create Table my table(a string, b string, c double)"
spa...
Indore asked 15/2, 2017 at 22:58
6
Solved
I'm wondering if there is any idiomatic way to chain multiple InputStreams into one continual InputStream in Java (or Scala).
What I need it for is to parse flat files that I load over the network ...
Scalf asked 12/1, 2013 at 16:2
3
Solved
I am trying to deploy a simple web application written using Play Framework in Scala to Amazon web service.
The web application is running OK in development mode and production mode in my local m...
Turboelectric asked 13/7, 2017 at 1:47
© 2022 - 2024 — McMap. All rights reserved.