Confluent Maven repository not working?

Question

I need to use the Confluent kafka-avro-serializer Maven artifact. From the official guide I should add this repository to my Maven pom

<repository>
  <id>confluent</id>
  <url>http://packages.confluent.io/maven/</url>
</repository>

The problem is that the URL http://packages.confluent.io/maven/ seems to not work at the moment as I get the response below

<Error>
  <Code>NoSuchKey</Code>
  <Message>The specified key does not exist.</Message>
  <Key>maven/</Key>
  <RequestId>15E287D11E5D4DFA</RequestId>
  <HostId>
    QVr9lCF0y3SrQoa1Z0jDWtmxD3eJz1gAEdivauojVJ+Bexb2gB6JsMpnXc+JjF95i082hgSLJSM=
  </HostId>
</Error>

In fact Maven does not find the artifact

<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-avro-serializer</artifactId>
  <version>3.1.1</version>
</dependency>

Do you know what the problem could be? Thank you

Tunicate

The URL needs to be in https instead of the http – Mulct

Answer 1

The file is available, since you can download it if you go to it directly: http://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/3.1.1/kafka-avro-serializer-3.1.1.jar

You could try adding the -U flag to your maven command to force download of cached files.

The root of the repo isn't browsable which is why you are getting the message when browsing to http://packages.confluent.io/maven/

Rudder

Thanks @Kevin. I'm using a central Artifactory mirror so I should add the Confluent repository to the mirrored ones in Artifactory. But when I try to add it, I get an error Input may not be null. How can I let Artifactory resolve the Confluent artifacts if the repository is not browsable? – Tunicate

With not browsable i mean through a browser by us, which makes us guess about the contents instead of being able to verify it visually. When adding to Artifactory: You should enter "maven" as repository key and "http:// packages.confluent.io/" as url (ignore the space, otherwise it didnt show the http) – Rudder

I tried adding only http://packages.confluent.io/ and the error does not appear, but no artifact is fetched from it. – Tunicate

I think the artifacts are only fetched when you call for them, so you should now be able to add your artifactory as a repository (instead of the coonfluent one) and your dependency should be loaded through your artifactory (and will then be visible there) – Rudder

Note that this repo is availaible in https as well just as https://repo.maven.apache.org/maven2/ – Parathion

This command works fine: spark-shell --packages org.apache.spark:spark-avro_2.11:2.4.4 --repositories http://packages.confluent.io/maven/ --packages io.confluent:kafka-avro-serializer:5.2.2 – Beslobber

Answer 2

Needs to add confluent repositories in pom.xml
Please add below lines in the pom.xml

<repositories>
    <repository>
        <id>confluent</id>
        <url>https://packages.confluent.io/maven/</url>
    </repository>
</repositories>

Dominant

this helped my url was <url>packages.confluent.io/</url> – Parabasis

It worked for me! Thanks! – Paronychia

Worked for me with a slight change of http to https. – Aver

Answer 3

Just like you I use a company repository (Sonatype Nexus) and was not able to proxy the confluent's repository.

Then I changed my maven settings.xml to exclude confluent form the mirrored repository:

    <mirrors>
        <mirror>
            <id>nexus</id>
            <mirrorOf>*,!confluent</mirrorOf> <!-- mirror anything but confluent as Nexus cannot proxy it -->
            <url>repository.company.local/nexus/content/groups/public</url>
        </mirror>
    </mirrors>
    ...
        <repositories>
            ...
            <repository>
                <id>confluent</id>
                <url>http://packages.confluent.io/maven/</url>
            </repository>
        </repositories>

This way, artifacts resolution works for confluents' artifacts as well.

Not as neat as proxying the repo but at least less cumbersome than downloading and registering each dependency manually.

Feverish

Just a additional help that the mirror should be adjusted in conf/settings.xml of m2. And the confluent dependency should be adjusted in the module where it is listed. My additional requirement was to point it to local .m2 so confluent url I put as: file:///c/Users/<login>/.m2/repository/. Hope it will help someone searching to point to a local folder. – Spoil

Answer 4

seams jar file removed from http url or http url not working. https url worked for me.

<repositories>
<repository>
    <id>confluent</id>
    <url>https://packages.confluent.io/maven/</url>
</repository>

Grimsley

Answer 5

You can add a mirror in you maven settings file to fetch the jars from confluent repo along with repository config . Changes needed are Add a mirror in settings.xml

   <mirror>
      <id>confluent</id>
      <mirrorOf>confluent</mirrorOf>
      <name>Nexus public mirror</name>
      <url>http://packages.confluent.io/maven/</url>
</mirror>

In repository section of maven settings add this

<repository>
          <id>confluent</id>
          <url>http://packages.confluent.io/maven/</url>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
</repository>

Phillisphilly

just the mirror worked fine for me, so have an up-vote. Thank you – Caprifig

Answer 6

When trying to connect Artifactory to the Confluent Maven repository you have to set the repo URL in Artifactory to be either http://packages.confluent.io/maven or https://packages.confluent.io/maven (both schemes seem to work fine). The confusing part is that when you ask Artifactory to test that URL it will fail with the message "Input may not be null". You are also unable to browse the repository in Artifactory. However, regardless of these problems, artifacts will be downloaded and cached when clients request them.

Mortality

Hi @rndgstn, my problem is that the Kafka repository is not "proxable" by the company's Artifactory repository, so in our company we cannot use the Artifacotry to access also the Kafka repo. At the moment we manually download the artifacts and upload them to the company repo. This is not the ideal solution, so I hope that in the future it will be proxable – Tunicate

It is working fine for us. What error are you seeing when a client makes a request? – Mortality

FWIW, the http scheme worked for me as a proxied repo with Sonatype Nexus. https scheme did not work. – Clobber

Answer 7

Had issue with:

<repositories>
        <repository>
            <id>confluent</id>
            <url>http://packages.confluent.io/maven/</url>
        </repository>
</repositories>

working:

<repositories>
        <repository>
            <id>confluent</id>
            <url>https://packages.confluent.io/maven/</url>
        </repository>
</repositories>

Changed "http" with "https" worked for me as maven has blocked the HTTP due to security reasons.

Conk

Answer 8

Adding confluent repository worked in POM file for Maven project.

Use below in build.sbt for SBT projects

resolvers += "confluent" at "https://packages.confluent.io/maven/"

Rufous

Answer 9

In my case I had everything written in the pom.xml correctly but it was still highlighting my dependency as an unknown and so all I had to do was go to the Maven tab on the right sidebar and hit the "Reload all Maven Projects" button.

Subtorrid

Answer 10

For those who are using gradle,

repositories {
    mavenCentral()
    maven {
        url 'https://packages.confluent.io/maven/'
    }
}

Waylan