Unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16 when "sbt update" in scalding

S

3

9

After getting code from git using clone https://github.com/twitter/scalding.git and doing ./sbt update I get:

::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found

And later:

sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:213)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)

I have removed ~/.sbt directory, installed latest Scala and cloned repository anew. The problem remains.

My configuration:

[dk@localhost scalding]$ scala -version
Scala code runner version 2.10.3 -- Copyright 2002-2013, LAMP/EPFL

[dk@localhost scalding]$ uname -a
Linux localhost.localdomain 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

CentOS 6.5 (Final)

What can be possibly wrong with ./sbt update? Should I install the dependency hadoop-lzo separately and set CLASSPATH to install Scalding? Any other solutions?

Here is the complete installation log:

[dk@localhost wks]$ cd scalding
[dk@localhost scalding]$ ./sbt update
Downloading sbt launcher for 0.13.0:
  From  http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.0/sbt-launch.jar
    To  /home/dk/.sbt/launchers/0.13.0/sbt-launch.jar
Getting org.scala-sbt sbt 0.13.0 ...
:: retrieving :: org.scala-sbt#boot-app
    confs: [default]
    43 artifacts copied, 0 already retrieved (12440kB/229ms)
Getting Scala 2.10.2 (for sbt)...
:: retrieving :: org.scala-sbt#boot-scala
    confs: [default]
    5 artifacts copied, 0 already retrieved (24390kB/150ms)
[info] Loading project definition from /home/dk/wks/scalding/project
[info] Updating {file:/home/dk/wks/scalding/project/}scalding-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 3 Scala sources to /home/dk/wks/scalding/project/target/scala-2.10/sbt-0.13/classes...
[warn] there were 10 deprecation warning(s); re-run with -deprecation for details
[warn] there were 2 feature warning(s); re-run with -feature for details
[warn] two warnings found
[info] Set current project to scalding (in build file:/home/dk/wks/scalding/)
[info] Updating {file:/home/dk/wks/scalding/}scalding-args...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-date...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}maple...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-core...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-json...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-parquet...
[info] Resolving com.hadoop.gplcompression#hadoop-lzo;0.4.16 ...
[warn]  module not found: com.hadoop.gplcompression#hadoop-lzo;0.4.16
[warn] ==== local: tried
[warn]   /home/dk/.ivy2/local/com.hadoop.gplcompression/hadoop-lzo/0.4.16/ivys/ivy.xml
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== snapshots: tried
[warn]   http://oss.sonatype.org/content/repositories/snapshots/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== releases: tried
[warn]   http://oss.sonatype.org/content/repositories/releases/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Concurrent Maven Repo: tried
[warn]   http://conjars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Clojars Repository: tried
[warn]   http://clojars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Twitter Maven: tried
[warn]   http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[info] Updating {file:/home/dk/wks/scalding/}scalding-jdbc...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-avro...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/home/dk/wks/scalding/}scalding-commons...
[info] Resolving com.hadoop.gplcompression#hadoop-lzo;0.4.16 ...
[warn]  module not found: com.hadoop.gplcompression#hadoop-lzo;0.4.16
[warn] ==== local: tried
[warn]   /home/dk/.ivy2/local/com.hadoop.gplcompression/hadoop-lzo/0.4.16/ivys/ivy.xml
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== snapshots: tried
[warn]   http://oss.sonatype.org/content/repositories/snapshots/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== releases: tried
[warn]   http://oss.sonatype.org/content/repositories/releases/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Concurrent Maven Repo: tried
[warn]   http://conjars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Clojars Repository: tried
[warn]   http://clojars.org/repo/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[warn] ==== Twitter Maven: tried
[warn]   http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/hadoop-lzo-0.4.16.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[info] Updating {file:/home/dk/wks/scalding/}scalding-repl...
[info] Resolving org.codehaus.jackson#jackson-core-asl;1.8.8 ...
[info] Done updating.
sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:213)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
    at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
    at 
....
.... many lines of exception stack-trace are skipped ...
....

[error] (scalding-commons/*:update) sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[error] (scalding-parquet/*:update) sbt.ResolveException: unresolved dependency: com.hadoop.gplcompression#hadoop-lzo;0.4.16: not found
[error] Total time: 53 s, completed Feb 14, 2014 4:51:51 PM
[dk@localhost scalding]$ ./sbt update^C
[dk@localhost scalding]$ uname -a
Linux localhost.localdomain 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[dk@localhost scalding]$ scala -version
Scala code runner version 2.10.3 -- Copyright

Scornful answered 14/2, 2014 at 13:45 Comment(7)

<s>The problem is that you're using CentOS instead of the god blessed Debian</s> Joking apart, that dependency comes from maven.twttr.com so make sure it is listed in repositories and is not blocked somehow by your host (e.g. can you open it in a browser?). Having said this, on my system ./sbt update on a fresh repo runs flawlessly. – Catechumen 14/2, 2014 at 14:11

@Catechumen The repo is already defined in project/Build.scala in the project github.com/twitter/scalding/blob/develop/project/… so there must be something different. – Buggy 14/2, 2014 at 14:56

@anton-ashanin you don't need to install Scala to run SBT projects. I'd go as far as to say you should remove it and sbt console is your buddy now. – Buggy 14/2, 2014 at 14:58

@anton-ashanin Can you paste the entire sbt update output? I'm pretty there's a bit more details on the root cause of the issue you face. – Buggy 14/2, 2014 at 14:59

@Catechumen You were right. This happens at my company environment and their it-security blocked access to maven.twttr.com. Debian is my favorite too, but I have to use CentOS as a project requirement. Notwithstanding with Debian at home I have some other installation problems with Scalding that I hope to write about later today. – Scornful 14/2, 2014 at 15:18

@Jacek Laskowski Added installation log to my original question – Scornful 14/2, 2014 at 15:29

Could be that you're blocked to access maven.twttr.com repository and hence maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.16/…? Can you check using your browser and when it's worked curl from the command line? – Buggy 15/2, 2014 at 21:22

P

11

SBT is unable to fetch com.hadoop.gplcompression#hadoop-lzo;0.4.16 from resolvers defined in your build.sbt

Adding maven.twttr.com: "Twitter Maven Repo" at "http://maven.twttr.com" should work.

Panoptic answered 11/9, 2014 at 10:27 Comment(1)

This worked for me (it should be marked as the correct answer). – Spadix 4/3, 2016 at 23:9

F

9

I don't think that would work anymore, use: "https://maven.twttr.com" instead. (Notice the https) They changed it in Jan 2018

Hope it helps

Finsen answered 1/2, 2018 at 10:3 Comment(0)

S

0

Today I got a similar error but in lein. I come through a lot of articles and Jira issue. All recommend put https in URL and tag twitter repository directly. But this does not help. Does someone know another way?

list of checked sources.

https://mcmap.net/q/1137912/-unresolved-dependency-com-hadoop-gplcompression-hadoop-lzo-0-4-16-when-quot-sbt-update-quot-in-scalding
https://issues.apache.org/jira/browse/PARQUET-1556

Stanislaw answered 15/11, 2019 at 14:3 Comment(0)

Recommended topics

Hot tags