Failed to construct instance from factory method DataflowRunner#fromOptions in beamSql, apache beam

Question

I'm specifying dataflow runner in my beamSql program below :

DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
    options.setStagingLocation("gs://gcpbucket/staging");
    options.setTempLocation("gs://gcpbucket/tmp");
    options.setProject("beta-19xxxx");
    options.setRunner(DataflowRunner.class);
    Pipeline p = Pipeline.create(options);

But I'm getting below exception :

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at my.proj.StarterPipeline.main(StarterPipeline.java:34)Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@782830e, only URLClassLoaders are supported.
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:43)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:262)

can anyone help me out understand what exactly is the issue ?

Kapoor

Are you using Java 9? Java 9 is currently not supported by Apache Beam. Take a look at this Stack Overflow post: #48292991. This is the same issue. Recommend downgrading to Java 8 for now. – Sterilant

Well yes !! Java 9 was the problem, I downgraded it to Java 8 and it worked. Thanks Andrew – Kapoor

I got the similar issue but I'm using java 8. Can someone tell me what may be the reason of this issue. – Indeliberate

Answer 1

Downgrading java 9 to java 8 did the trick.

Kapoor

Answer 2

This problem can happen when using Beam SDK with JVM > 8.

Support for JVM 11 (or 9) is only experimental, starting from Beam SDK 2.12 and at least up to Beam SDK 2.16. This answer provides a bit more context: https://mcmap.net/q/2032834/-apache-beam-2-12-0-with-java-11-support

Regarding this specific stack trace, it seems related to this issue: https://issues.apache.org/jira/browse/BEAM-3718

Pinchhit

Answer 3

For me the fix was different. In my pom.xml, below dependency was set to 2.20 while apache beam was set to 2.19

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
  <version>2.20</version>
</dependency>

After changing it to same as beam version, build was working fine.

Net Net, just make sure all the beam related sdks are set to same version. Use variable for the same, like ${beamVersion} so you don't end up facing strange errors.

Cavender