Hadoop NoSuchMethodError apache.commons.cli

Question

I'm using hadoop-2.7.2 and I did a MapReduceJob with IntelliJ. In my job, I'm using apache.commons.cli-1.3.1 and I put the lib in the jar.

When I use the MapReduceJob on my Hadoop cluster I have a NoSuchMethodError:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;

I don't understand because the method exist in the class Option and the class Option is extracted from the commons-cli.jar to my application jar. Moreover, I don't have this issue with my others libraries.

Thank you for your time.

Soubrette

I just found that hadoop use commons-CLi 1.2. I think this is the source of my issue but I don't know how to fix it. – Soubrette

You can try to exclude commons-cli-1.2 from Hadoop dependency in your pom.xml, then Hadoop willl use your commons-cli-1.3.1. If it causes any errors, you'd better use 1.2 in your code. – Hyder

I did something like that but my boss doesn't want this way because it doesn't fix the issue. If we have the same issue later with another lib, he doesn't want to fix the issue again. But thank you for your advice. – Soubrette

I fixed it by changing the sequence in which the jars are declared in the classpath. Please check my answer. – Ikeikebana

Answer 1

We were able to fix this error using maven class relocations. If you are using the shade plugin to build your jar add following to pom.xml under appropriate section:

<!-- necessary to fix NoSuchMethodError: org.apache.commons.cli.Option.builder -->
                            <relocations>
                                <relocation>
                                    <pattern>org.apache.commons.cli</pattern>
                                    <shadedPattern>org.shaded.commons.cli</shadedPattern>
                                </relocation>
                            </relocations>

Also an explicit reference to v1.3+ of commons-cli needs to be added at the TOP of the dependencies section before any dependency that may have a transitive reference to an older version of commons-cli.

Corsair

this solution worked for me, thanks – Lobby

Answer 2

The problem seems to be related to how the classloader is loading the classes. Because the static Builder class was in common-cli 1.4, while some of the hadoop dependencies were still referring to older version - the issue occurred.

In my case, the issue resolved by changing the sequence of jar file addition into the classpath in the shell script responsible for setting up environment before program execution. Earlier, I was adding the jar into classpath like

CLASSPATH=<Hadoop Jars>:<Common CLI jar>:$CLASSPATH

is changed to

CLASSPATH=<Common CLI jar>:<Hadoop Jars>:$CLASSPATH

and it fixed the problem.

Ikeikebana

Answer 3

We solved this issue with the next gradle configuration:

 compile('org.apache.parquet:parquet-tools:1.9.0'){
  exclude module:"commons-cli"
 }

Nonparous