Kafka start error on MAC .. something related to java and scala ... NoSuchMethodError: scala.Predef$.refArrayOps

Question

i have downloaded kafka from here https://www.apache.org/dyn/closer.cgi?path=/kafka/3.4.0/kafka_2.13-3.4.0.tgz

Zookeeper starts with the command

bin/zookeeper-server-start.sh config/zookeeper.properties

but when i ran this command

bin/kafka-server-start.sh config/server.properties

i got this

[2023-03-12 14:25:29,772] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Ljava/lang/Object;
    at kafka.Kafka$.getPropsFromArgs(Kafka.scala:43) ~[kafka_2.13-3.4.0.jar:?]
    at kafka.Kafka$.main(Kafka.scala:92) [kafka_2.13-3.4.0.jar:?]
    at kafka.Kafka.main(Kafka.scala) [kafka_2.13-3.4.0.jar:?]
Exception in thread "main" java.lang.NoSuchMethodError: scala.Option.orNull(Lscala/$less$colon$less;)Ljava/lang/Object;
    at kafka.utils.Exit$.exit(Exit.scala:28)
    at kafka.Kafka$.main(Kafka.scala:128)
    at kafka.Kafka.main(Kafka.scala)

I tried to check if it has scala lib in libs and orNull method exists in in that scala version

Judaic

Look's strange. Out of curiosity, have you tried to download another version on the official website? – Spickandspan

#40329448 #35686 – Muniz

Please show java.class.path. There should be line INFO Server environment:java.class.path=... in the output of zookeeper-server-start.sh and INFO Client environment:java.class.path=... in the output of kafka-server-start.sh. It's better to show the whole output of the both commands. Maybe something confuses scala 2.13 and 2.12. – Muniz

The signature of refArrayOps is different in 2.12 def refArrayOps(scala.Array[scala.Any]): scala.Array[scala.Any] (scalap) public <T> T[] refArrayOps(T[]) (javap) api and 2.13 def refArrayOps(scala.Array[scala.Any]): scala.Any (scalap) public <T> T[] refArrayOps(T[]) (javap) api – Muniz

If you've installed Scala separately, your kafka version needs to match. Looks like you still have Scala 2.12 on your system – Stickup

@Stickup Well, I have Scala 2.12 installed in my system (Ubuntu) but zookeeper-server-start.sh and kafka-server-start.sh from kafka_2.13-3.4.0 run. Doesn't Kafka use scala-library from kafka_2.13-3.4.0/libs and not installed in the system? At least at the site it's written that Java 8+ should be installed, not Scala. But maybe this can be different at MacOS. – Muniz

@Dmytro Zookeeper doesn't use Scala. Kafka still relies on OS PATH variables, of which scala binaries would conflict with JVM classpath. Yes, Java needs installed, but Kafka is compiled with Scala bytecode, which is version specific... In other words, if Scala is not installed on the system, Kafka works fine because it'll only use its local libraries – Stickup

@Stickup Well, maybe I misunderstand but you said that if Scala is installed then Kafka version needs to match. I'm telling you that at my system Scala 2.12 is installed and kafka_2.13-3.4.0 doesn't match but runs (because uses Kafka local Scala library and not the one that is installed). In my output of kafka-server-start.sh, java.class.path has libs/scala-library-2.13.10.jar and not Scala 2.12 from my system $PATH. But maybe something is different in OP's setting. – Muniz

@Gireeshd Glad you figured it out, but Kafka doesn't need Python, so it's not clear why you saw that in any logs – Stickup

I apologize for the confusion earlier. Regarding the issue with the conflicting class in Hive and Kafka, I have commented out the line that sets the CLASSPATH environment variable to include hive libraries . This was necessary to resolve the conflict and avoid issues. By doing this, the class that exists in both libraries will only be loaded from one location, preventing conflicts... sorry for pasting clion cmake build log earlier.. – Judaic

#75947949 – Muniz