Maven conflict in Java app with google-cloud-core-grpc dependency

Question

(I've also raised a GitHub issue for this - https://github.com/googleapis/google-cloud-java/issues/4095)

I have the latest versions of the following 2 dependencies for Apache Beam:

Dependency 1 - google-cloud-dataflow-java-sdk-all

(A distribution of Apache Beam designed to simplify usage of Apache Beam on Google Cloud Dataflow service - https://mvnrepository.com/artifact/com.google.cloud.dataflow/google-cloud-dataflow-java-sdk-all)

<dependency>
  <groupId>com.google.cloud.dataflow</groupId>
  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
  <version>2.5.0</version>
</dependency>

Dependency 2 - beam-runners-google-cloud-dataflow-java

(I'm guessing that this allows the actual running of a Beam pipeline inside a Google Cloud Dataflow)

https://mvnrepository.com/artifact/org.apache.beam/beam-runners-google-cloud-dataflow-java

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
  <version>2.8.0</version>
</dependency>

The Maven install works fine with these 2 dependencies. I need to add the following (3rd) dependency for independently subscribing to a Google Cloud pub/sub outside of an Apache Beam pipeline:

google-cloud-pubsub

https://mvnrepository.com/artifact/com.google.cloud/google-cloud-pubsub

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-pubsub</artifactId>
  <version>1.53.0</version>
</dependency>

(This is the latest version). After this is added I get the following conflict after I do mvn clean install:

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I don't get any Maven install conflict if I remove the Apache Beam dependencies and keep the google-cloud-pubsub dependency.

There's a separate Google 'BOM' (Bill of Materials) dependency manager that's supposed to manage sub-dependencies of Google Cloud dependencies, but this doesn't solve the conflict for me:

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bom</artifactId>
        <version>0.71.0-alpha</version>
        <type>pom</type>
        <scope>import</scope>
       </dependency>
     </dependencies>
  </dependencyManagement>

I followed the answer to this question -

How do I resolve a dependency conflict in Maven?

and I identified that

---

• com.google.cloud:google-cloud-pubsub:jar:1.53.0

has a sub-dependency on:

io.grpc:grpc-core:jar:1.16.1

---

• com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0

has sub-dependencies on various versions of:

io.grpc:grpc-core:jar

---

(GRPC is A high performance, open-source universal RPC framework)

If I add io.grpc:grpc-core:jar:1.6.1 as an explicit dependency to my POM file, it doesn't 'pin' the dependency as it still gets pulled in as a sub-dependency by com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0.

On the following GitHub issue, garrettjonesgoogle confirms on 21 Feb that google-cloud-bom doesn't pin the version for io.grpc:grpc-core:

https://github.com/googleapis/google-cloud-java/issues/2890

It seems that the Apache Beam dependency manages the varying versions of io.grpc:grpc-core:jar internally, however this conflicts with the (consistent) version used by the Google pub/sub dependency.

Is there anything I can do to work around this? Can I maybe isolate sub-dependencies somehow using Maven?

Franci

Any luck? I'm having the exact same issue at this moment. – Prognosticate

@Iso I've added more info but still haven't solved my issue. You can look at the answer to this question which may help you identify what the problem dependency is - #37595422. If you follow the directions, you should get a text file showing dependencies, comma separated by package. Look for different version numbers of the same sub-dependency across 2 or more different packages. Any package you can see duplicated with different version numbers may be your problem package. – Franci

@Iso you could also try this to identify your conflicting dependency, although it didn't work for me it just gave me the same error output that I'm already getting - maven.apache.org/plugins/maven-dependency-plugin/examples/… – Franci

@Iso I've added an 'answer'. It's not an actual solution but it identifies the cause of my problem. I'm working around this for now. Can you do the same by pollling instead of using pub/sub? – Franci

Answer 1

I was able to workaround this issue buy just excluding grpc-core from google-cloud-pubsub artifact. It gets incorporated in the dependency tree afterwards with the same version 1.13.1 as a dependency of beam-runners-google-cloud-dataflow-java

This is the exceprt of the pom.xml which worked for me.

<dependencies>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-pubsub</artifactId>
        <version>1.53.0</version>
        <exclusions>
            <exclusion>
                <groupId>io.grpc</groupId>
                <artifactId>grpc-core</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.beam</groupId>
        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
        <version>2.8.0</version>
    </dependency>
</dependencies>

Prevalent

Answer 2

This isn't really a solution however I did identify that the issue is mainly with this dependency:

<dependency>
  <groupId>com.google.cloud.dataflow</groupId>
  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
  <version>2.5.0</version>
</dependency>

This artifact has a number of inconsistent dependencies on io.grpc:grpc-core:jar and isolation between the Apache Beam and Google pub sub artifacts aren't really working properly. Whatever version of google-cloud-pubsub is being used, its own dependency on io.grpc:grpc-core:jar will conflict with one of the versions of of io.grpc:grpc-core:jar.

I've raised the following Git issue against the Google artifact:

https://github.com/googleapis/google-cloud-java/issues/4095

And I've raised the following against the Apache Beam artifact (it points back to the Google issue I raised):

https://issues.apache.org/jira/browse/BEAM-6118

I'm addressing the issue by working around it. I can poll the data I need instead of consuming it from a subscription to the pub/sub however this isn't optimal and will lead to significant increased costs on Google Cloud.

Franci

Hey Chris. Excuse me for ignoring your earlier reply. I've got this over-simplified setup for a masterthesis proof-of-concept. I worked around the conflict by separating the projects (as in my case, I was dealing with a bundled Pub/Sub publisher and Dataflow project). This is indeed not optimal, but it will do for the moment being. Thanks for the heads up, i'll keep an eye on the issue. – Prognosticate