spark-graphx Questions
3
I am trying to implement topological sort using Spark's GraphX library.
This is the code I've written so far:
MyObject.scala
import java.util.ArrayList
import scala.collection.mutable.Queue
im...
Eventide asked 18/10, 2016 at 13:11
1
My aim is to find whether the max path length between two vertices is <= 4.
I have a graph dataframe and a test file of the below format.
I am trying to get the output column(OP) from bfs fun...
Livelong asked 2/12, 2019 at 9:45
5
Just started my excursion to graph processing methods and tools. What we basically do - count some standard metrics like pagerank, clustering coefficient, triangle count, diameter, connectivity etc...
Prebo asked 19/2, 2015 at 14:40
6
Solved
I am running a Bash Script in MAC. This script calls a spark method written in Scala language for a large number of times. I am currently trying to call this spark method for 100,000 times using a ...
Outing asked 22/11, 2016 at 11:32
1
I have created a GraphFrame in Spark and the graph currently looks as following:
Basically, there will be lot of such subgraphs where each of these subgraphs will be disconnected to each other. ...
Scad asked 26/5, 2016 at 14:41
5
I have a cluster of two worker nodes.
Worker_Node_1 - 64GB RAM
Worker_Node_2 - 32GB RAM
Background Summery :
I am trying to execute spark-submit on yarn-cluster to run Pregel on a Graph to calcul...
Poetry asked 17/11, 2015 at 14:34
2
Solved
GraphX comes with an algorithm for finding connected components of a graph.
I did not find a statement about the complexity of their implementation.
Generally, finding connected components can be...
Aristaeus asked 28/4, 2016 at 20:59
1
Solved
When I was trying to implement an algorithm in Graphx with Scala, I didn't find it possible to activate all the vertices in the next ietration.. How can I send a message to all my graph vertices?
...
Joaquinajoash asked 29/11, 2018 at 10:48
4
I am looking for a way to visualize the graph constructed in Spark's Graphx. As far as I know Graphx doesn't have any visualization methods so I need to export the data from Graphx to another graph...
Nerta asked 3/8, 2016 at 5:51
0
We have hit a bug with GraphX when calling the connectedComponents function, where it errors with the following error java.lang.ArrayIndexOutOfBoundsException: -1
I've found this bug report:
https:...
Savoirfaire asked 13/10, 2017 at 15:17
1
Solved
I'm trying to find the connected components for friends in a city. My data is a list of edges with an attribute of city.
City | SRC | DEST
Houston Kyle -> Benny
Houston Benny -> Charles
Houston...
Potato asked 25/9, 2017 at 1:59
2
I am trying to use connected components but having issue with scaling. My Here is what I have -
// get vertices
val vertices = stage_2.flatMap(x => GraphUtil.getVertices(x)).cache
// get edges...
Relic asked 26/10, 2016 at 15:47
1
Solved
I'm trying to compute the sum of node values in a spark graphx graph. In short the graph is a tree and the top node (root) should sum all children and their children. My graph is actually a tree th...
Group asked 3/1, 2017 at 20:38
2
Solved
I have met an Out Of Memeory error with unknown reasons, I have released the useless RDDs immediately, but after several round of loop, OOM error still come out. My code is as following:
// single...
Anora asked 12/12, 2016 at 5:2
1
Solved
I have following directed graph as given by the nodes and edges below.
Nodes
1,2,3,4,5
Edges
(1,2),(1,3),(1,4),(2,5),(3,4),(3,5),(4,5)
How do I convert this directed graph to undirected gr...
Movie asked 21/11, 2016 at 20:22
1
I've created a directed graph, using graphx.
#src->dest
a -> b 34
a -> c 23
b -> e 10
c -> d 12
d -> c 12
c -> d 11
I want to get all two hop neighbors like this:
a -> e...
Lest asked 8/10, 2016 at 4:31
1
Solved
Spark version 1.6.1
Creating Edge and Vertex RDDs
val vertices_raw = sqlContext.read.json("vertices.json.gz")
val vertices = vertices_raw.rdd.map(row=> ((row.getAs[String]("toid").stripPrefix...
Kalynkam asked 17/8, 2016 at 10:30
1
I use graphx api in a iterative alogrithm. Although I have carefully cache/ unpersist rdd, and take care of the vertices partition num. The time cost still seems increases per round in a lineary tr...
Graphitize asked 22/6, 2016 at 8:23
1
Solved
I am using spark-shell for running my code. In my code, I have defined a function and I call that function with its parameters.
The problem is that I get the below error when I call the function.
...
Auction asked 27/5, 2016 at 6:58
1
Solved
I try to run connected components on logNormalGraph.
val graph: Graph[Long, Int] = GraphGenerators.
logNormalGraph(context.spark, numEParts = 10, numVertices = 1000000,
mu = 0.01, sigma = 0.01)
...
Incertitude asked 12/4, 2016 at 13:34
1
Solved
This is my code:
class FNNode(val name: String)
case class Ingredient(override val name: String, category: String) extends FNNode(name)
val ingredients: RDD[(VertexId, FNNode)] =
sc.textFile(P...
Bouchier asked 8/4, 2016 at 17:14
1
Solved
I have created a graph in Spark GraphX using the following codes. (See my question and solution)
import scala.math.random
import org.apache.spark._
import org.apache.spark.graphx._
import org.apac...
Bevan asked 22/12, 2015 at 17:25
1
Solved
I am moving from MPI based systems to Apache Spark. I need to do the following in Spark.
Suppose, I have n vertices. I want to create an edge list from these n vertices. An edge is just a tuple of...
Mendymene asked 15/12, 2015 at 18:19
2
I have constructed a graph in Spark's GraphX. This graph is going to have potentially 1 billion nodes and upwards of 10 billion edges, so I don't want to have to build this graph over and over agai...
Incorrigible asked 4/8, 2015 at 6:54
1
Solved
Suppose we have got the input in Apache GraphX as :
Vertex RDD:
val vertexArray = Array(
(1L, "Alice"),
(2L, "Bob"),
(3L, "Charlie"),
(4L, "David"),
(5L, "Ed"),
(6L, "Fran")
)
Edge RDD:
...
Tempo asked 16/9, 2015 at 2:36
1 Next >
© 2022 - 2024 — McMap. All rights reserved.