Why does sbt compile fail with StackOverflowError?

Question

I am working on a Scala project that has been in existence for a few years but is new to me. My task is to upgrade it from Scala 2.9.3 to 2.11.7, along with its dependencies. I have gotten past the errors and warnings, but I cannot get the project to compile successfully in SBT. I always get a StackOverflowError in pretty much the same place. The stacktrace looks like this, but details vary with the Xss setting (currently 4M, but have tried as high as 24M):

java.lang.StackOverflowError
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5369)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5373)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5471)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5479)
at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:644)
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)

SBT_OPTS looks like this:

-Xmx2G -Xss4M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

I can 'make' the project successfully in Intellij, and others can pull my changes from GitHub and compile the project in sbt, so the issue seems to be local to my machine (a recent quad-core Macbook Pro with 16GB RAM). Other Scala/sbt projects compile successfully for me on this machine.

Here are other relevant details:

Scala version: 2.11.7
Java version: java version "1.8.0_66" (build 1.8.0_66-b17)
sbt version: 0.13.7 (have also tried 0.13.9)

I have completely rebuilt the ivy2 cache and cleared the lib_managed directory. The version of the scala-compiler.jar is the same as is used on at least one machine that can 'sbt compile' the code successfully. I did a clean reinstall of sbt (via brew remove sbt, manual removal of ~/.sbt directory, then brew install sbt).

I have not tried to isolate the line of source code being compiled when the error occurs. I have assumed it would be more productive to look for a configuration issue or dependency conflict somewhere.

Any suggestions for further troubleshooting will be appreciated.

[Added...] It may be helpful to add that, as an experiment, I downloaded the Scala language source code from https://github.com/scala/scala and got the following very similar error trying to sbt compile it:

java.lang.StackOverflowError
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.outerValue(ExplicitOuter.scala:229)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:441)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1345)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:44)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.scala$reflect$internal$Trees$UnderConstructionTransformer$$super$transform(ExplicitOuter.scala:219)
at scala.reflect.internal.Trees$UnderConstructionTransformer$class.transform(Trees.scala:1693)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.transform(ExplicitOuter.scala:291)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:459)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1347)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)

Here is something interesting. From this post I found out about launching sbt with a -d flag for debugging info. Got the following output:

Kevins-MacBook-Pro:scala kdoherty$ sbt -d
[process_args] java_version = '1.8.0_66'
# Executing command line:
java
-Xmx2G
-Xss4M
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Xmx384m
-Xss512k
-XX:+UseCompressedOops
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
-jar
/usr/local/Cellar/sbt/0.13.9/libexec/sbt-launch.jar

So somewhere my SBT_OPTS settings are being overridden (by defaults, I guess). Now I need to find where those defaults are coming from.

Sexpot

When you say you have the same version of scala-compiler.jar as another computer that can compile this, do you mean that scalac -version returns the same thing on both computers, or that both computers have 2.11.7 defined in the .sbt build file? – Resourceful

@soong: both have 2.11.7 specified in the build.sbt file. – Sexpot

@KevinDoherty Do you happen to know the version of java installed on this other computer, and what all of the JVM settings for it are? – Resourceful

@soong My other Mac (13-inch MBP), which will compile the project successfully, is using Java 1.8.0-b132. No special JVM settings are used, just the defaults. – Sexpot

they don't look like they are getting overwritten, they are getting duplicated! (I see both memory properties getting listed twice but with different values) – Outsoar

Answer 1

I figured it out. Once I knew that the -d flag would tell me what settings SBT was actually using, I saw that the values in my SBT_OPTS environment variable were being clobbered by other, lower settings. Where were those coming from? From my JAVA_OPTS env variable! I should have noticed them sooner, but now I know I can keep those Java options as they are and override them by adding the SBT-specific settings to my /usr/local/etc/sbtopts file, using the somewhat awkward format of

-J-Xmx2G
-J-Xss2M 

Using the values shown I was able to run sbt compile successfully on my project.

I hope someone finds this useful.

Sexpot

If I run this: sbt -Pyarn -pHadoop 2.7 assembly -J-Xss2M I get: Java HotSpot(TM) Client VM warning: ignoring option MaxPermSize=256m; support wa s removed in 8.0 - Any ideas? – Fernandafernande

Upgrade sbt and the warning goes away. – Daveen

MaxPermSize was replaced in java 8 with -XX:MaxMetaspaceSize – Outsoar

it would be nice if there was a solution that works inside the sbt project and therefore could be run anywhere on anything that is trying to do the build – Outsoar

the sbt -d param doesn't work for me I get a "The - command is deprecated in favor of onFailure and will be removed in 0.14.0". That said, "sbt show scalacOptions" is showing the correct memory settings for me. My problem is that the compiler is still somehow ignoring / not honoring the values. – Outsoar

Answer 2

Add to the bottom of /usr/local/etc/sbtopts

-J-Xmx4G 
-J-Xss4M

All set.

Audacious

Answer 3

I just added -Xss in my Intellij sbt properties and the issue is resolved.

Intellij SBT properties

Jessalyn

Answer 4

Relevant parts from the output of sbt -h:

  # jvm options and output control
  JAVA_OPTS          environment variable, if unset uses ""
  .jvmopts           if this file exists in the current directory, its contents
                     are appended to JAVA_OPTS
  SBT_OPTS           environment variable, if unset uses ""
  .sbtopts           if this file exists in the current directory, its contents
                     are prepended to the runner args

Thus, in my case I solved the problem by creating a file .sbtopts with content

-J-Xmx4G 
-J-Xss4M

in the project directory.

Note: Running sbt -d shows what settings have been used, for instance:

$ sbt -d
[addSbt] arg = '-debug'
[process_args] java_version = '8'
# Executing command line:
java
-Xms1024m
-XX:ReservedCodeCacheSize=128m
-XX:MaxMetaspaceSize=256m
-Xmx2G
-Xss2M
-jar
/path/to/sbt-launch.jar
-debug

Amplify

Answer 5

I was unable to get this to work via the provided answers. build.sbt settings and the sbtopts file both failed to solve this error for me. What I had to do was run sbt with the -mem flag, for example:

sbt -mem 2048 compile

and now my projects built. If using IntelliJ you can also go to

Preferences > Build, Execution, Deployment > Build Tools > sbt 

and set Maximum heap size, MB to whatever target you need.

Arpent

Answer 6

I discovered the SBT_OPT setting in the bin/sbt file of my sbt install was affecting the memory values set in my projects build.sbt

updating the existing -Xss value in this file from 1M to 8M raised the memory size of the Scalac stack to a point I stopped getting StackOverflow exceptions in the sbt-invoked compiler. This seemed odd because the sbt documented approach to setting stack size in the compiler is to do this is with the -J-Xss setting.

Sbt doesn't seem to actually enable you to set the compiler's stack memory. While a build.sbt accepts the following configuration as a valid setting, it doesn't seem to apply the value in the compiler:

scalacOptions in ThisBuild ++= Seq(-J-Xss8M)

I suspect that is a bug or non-implemented functionality

Outsoar

Answer 7

There are multiple Correct Answers already. But what worked for me is below,

// Created and Added a File: .jvmopts in the Project Root Folder with below Parameters.
-Xms3022m
-Xmx4048m
-Xss124m
-XX:MaxPermSize=4048m
-XX:MaxMetaspaceSize=512m
-XX:+CMSClassUnloadingEnabled
-XX:ReservedCodeCacheSize=128m

Mogul

Why do you have MaxMetaspaceSize twice? – Quartana

for me just -Xss8m in the file was enought – Armrest

Answer 8

I too run into this problem recently and I discovered a working solution for it with sbt 1.3.13.

1. create a .sbtopts file under project root
2. add -J-Xss100M (or any thread stack size you think suitable) to the .sbtopts file

Layman