Scala Script: Explain this class cast error to ScalaClassLoader on Windows

Question

Consider following scala script:

import scala.reflect.internal.util.ScalaClassLoader

object Test {
  def main(args: Array[String]) {
    val classloaderForScalaLibrary = classOf[ScalaClassLoader.URLClassLoader].getClassLoader
    println(classloaderForScalaLibrary)
    val classloaderForTestClass = this.getClass.getClassLoader
    println(classloaderForTestClass)
    this.getClass.getClassLoader.asInstanceOf[ScalaClassLoader.URLClassLoader]
  }
}

The output is:

scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@71c8becc
scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@71c8becc
java.lang.ClassCastException: scala.reflect.internal.util.ScalaClassLoader$URLClassLoader cannot be cast to scala.reflect.internal.util.ScalaClassLoader$URLClassLoader
        at Main$.main(Test.scala:8)
        at Main.main(Test.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:98)
        at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:32)
...

Why can't I cast ScalaClassLoader$URLClassLoader to ScalaClassLoader$URLClassLoader?

Edit:

On running:

scala -J-verbose:class Test.scala | grep ScalaClassLoader

The output is:

[Loaded scala.reflect.internal.util.ScalaClassLoader$URLClassLoader from file:/C:/Development/Software/scala-2.12.2/lib/scala-reflect.jar]
...
...
[Loaded scala.reflect.internal.util.ScalaClassLoader$URLClassLoader from file:/C:/DEVELO~1/Software/SCALA-~1.2/lib/scala-reflect.jar]

So there is definitely some shady class loading going on. Now trying to investigate why this is so

Schwartz

Weird, my classloader(this.getClass.getClassLoader) is sun.misc.Launcher$AppClassLoader@68de145 and I can assert as instance of URLClassLoader (this.getClass.getClassLoader.asInstanceOf[URLClassLoader]). What about new ScalaClassLoader.URLClassLoader(Seq(), null).asInstanceOf[ScalaClassLoader.URLClassLoader]?? – Johnsen

How are you starting your program? When I use scala Test.scala it works for me, when I run it from sbt or IntelliJ I get a ClassCastException with a different class loader. – Sefton

@prayagupd You're probably running from an IDE which uses a different classloader. Try running from commandline – Schwartz

You are right. I was using IDE and class loader was Sun one. scalac gives me proper result and casting is working fine. scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@e6ea0c6. I'm using scala 2.12.2 – Johnsen

ClassCastExceptions for the same class can happen when the class was loaded twice by different classloaders. In that case the classes are not considered equal by the JVM even though the have the same name. – Sefton

@Harald As you can see in the screenshot. I don't have any customised classloaders – Schwartz

@Harald should scala.reflect.internal.util.ScalaClassLoader$URLClassLoader not be loaded only once. Curios who can load it twice? – Johnsen

@KshitizSharma I am using scala 2.12.2 on MacOS. – Johnsen

@prayagupd This is really weird. I though maybe something wrong with my scala installation, so just downloaded and tried 2.11.11. Same results – Schwartz

I think Herald's theory is right. https://mcmap.net/q/498292/-getting-class-cast-exception-where-both-classes-are-exactly-the-same – Johnsen

@prayagupd Yes, normally it should be loaded only once, especially in such a simple case. But I've seen CCE in containers with dynamic classloaders where the same class could be loaded by unrelated classloaders. – Sefton

@prayagupd That sounds reasonable. But the classloader seems to be the same. See my edit to question – Schwartz

@KshitizSharma: AFAICT the two classloaders have identical names, but the classloaders in question (that are also Java classes) were in their turn loaded by different classloaders, likely from different jars. So, despite being "the same class", they cannot be cast to each other, because they potentially come from different source code, just named identically. – Whimsicality

@Whimsicality The two classloaders seem to have same hashcode (see output). Which makes me think its the same instance. Assuming what you say is true, do you think there is a way to debug this further? – Schwartz

@KshitizSharma should this.getClass.getClassLoader.getParent tell who is the root ClassLoader? To me on scalac gives null which means itself is a Parent Loader. But IDE gives me sun.misc.Launcher$ExtClassLoader@5034c75a. – Johnsen

Also, do scala Test.scala -J-verbose:class | grep ScalaClassLoader which gives all the ScalaClassLoader being loaded. for me there is one loading only [Loaded scala.reflect.internal.util.ScalaClassLoader$URLClassLoader from /usr/local/scala-2.12.2/lib/scala-reflect.jar – Johnsen

@prayagupd Thanks for pointing out the -J-verbose:class flag. It did reveal the double class loading. Now I need to figure out why – Schwartz

Sorry to see you're on Windows. Managing paths & classpaths on Windows is always challenging. Why does anyone use Windows? I need a corporate vpn client available only on Windows. – Robbyrobbyn

@Robbyrobbyn Windows 10 has much better hardware and app support. Its command line is crap (even powershell), but that is solved by Cygwin. – Schwartz

@KshitizSharma :) I just updated cygwin and my environment broke. Of course it's not easy. :) Someday I'll try the Windows support for bash, on a day when I am less lazy. – Robbyrobbyn

Answer 1

If you extend your code a bit more as following:

import scala.reflect.internal.util.ScalaClassLoader

object test {

  def main(args: Array[String]) {
    val cl1 = this.getClass.getClassLoader
    println(cl1)
    val c1 = cl1.getClass 
    println(cl1.getClass)
    println(cl1.getClass.getClassLoader)

    println("-------")

    var c2 = classOf[ScalaClassLoader.URLClassLoader]
    println(c2)
    println(c2.getClassLoader)
    println("-------")
    println(c1 == c2)

  }
}

you'll get following output:

scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@5cee5251
class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader
sun.misc.Launcher$AppClassLoader@4554617c
-------
class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader
scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@5cee5251
-------
false

Notice matching hashes @5cee5251. It means that first Scala interpreter loads ScalaClassLoader$URLClassLoader using root Java class loader and then uses that class loader to load all the classes in your script and when you ask for ScalaClassLoader$URLClassLoader in your code it is loaded with another (already loaded) instance of ScalaClassLoader$URLClassLoader. In this way your script is isolated from the "runtime environment" that executes it.

You may find some details at ScalaClassLoader.asContext method, that you can see in your stack trace, that uses Thread.setContextClassLoader to set itself as the main classloader for the thread that executes your script.

Update (Why it works on Mac but doesn't work on Windows)

The major difference between scala shell script for *nix and scala.bat for Windows is that by default on *nix platforms standard Scala libraries are added to Boot Classpath (see usebootcp in the script) while on Windows they are added to the "Usual Classpath". This is important because it defines which class loader will load scala.reflect.internal.util.ScalaClassLoader that is used by scala.tools.nsc.MainGenericRunner: will it be the root class loader (which is represented as null if you call getClassLoader) or Application Class Loader (i.e. an instance of sun.misc.Launcher$AppClassLoader). This is important because CommonRunner.run creates an instance of ScalaClassLoader using just urls without parent

def run(urls: Seq[URL], objectName: String, arguments: Seq[String]) {
  (ScalaClassLoader fromURLs urls).run(objectName, arguments)
} 

This means that the parent class loader for the "main" ScalaClassLoader will be boot class loader rather than sun.misc.Launcher$AppClassLoader and thus when you ask this "main" ScalaClassLoader for class scala.reflect.internal.util.ScalaClassLoader it can't find it among classes loaded by its class loaders chain and thus has to load it again. This is the reason why you have two different instances of ScalaClassLoader class in your script.

There are two obvious workarounds (and both are not so good):

• Change CommonRunner.run in the Scala source to actually pass current context class loader as the parent to the new ScalaClassLoader (might be not that easy ☺)
• Change scala.bat to use -Xbootclasspath/a: instead of -cp for %_TOOL_CLASSPATH%. However looking into the usebootcp in the *nix script I can see following comment:

# default to the boot classpath for speed, except on cygwin/mingw/msys because
# JLine on Windows requires a custom DLL to be loaded.
unset usebootcp
if [[ -z "$cygwin$mingw$msys" ]]; then
  usebootcp="true"
fi

So I suspect that if you want to use scala.bat for REPL, moving all Scala libs to the Boot Classpath might be a bad idea. If this is the case, you probably need to create a copy of scala.bat (such as scala_run_script.bat) change it and use it to run your Scala scripts leaving standard scala.bat for REPL.

Justus

Interestingly on Mac OS with Scala 2.12.2 I get the follwing output scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@5fdef03a class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader null ------- class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader null ------- true i.e. no sun.misc.Launcher$AppClassLoader involved and the final comparison returns true – Sefton

Just like Herald, c1 == c2 is true for me as well on scala 2.12.2 on Mac OS (using scalac compiler) but IDE has false. – Johnsen

@Harald, @prayagupd do you compile it with scalac and then run or run this script directly using scala as OP seems to do? – Justus

I did both with the above result – Sefton

@Harald, I only have Scala 2.11 on the Mac I have access to but surprisingly I got the same ("good") results there. Moreover, original example with asInstanceOf seems to work OK on that Mac as well. Could you check it in your environment? – Justus

@Harald Could this be a bug with handling of windows paths? Has anyone managed to run this on Windows? – Schwartz

@KshitizSharma, please take a look at my update with more details on reasons – Justus

@Justus Awesome! Thanks for all the effort. +1 – Schwartz