Use Confluent Hub without Confluent Platform installation

Question

I am using these installation instructions for getting confluent hub client https://docs.confluent.io/current/connect/managing/confluent-hub/client.html

But, when I get to the line to install the kafka-connect-elasticsearch connector

confluent-hub install confluentinc/kafka-connect-elasticsearch:latest

I continue to get this error message:

Unable to detect Confluent Platform installation. Specify --component-dir and --worker-configs explicitly.

Error: Invalid options or arguments

I have ElasticSearch and Kafka all installed via Homebrew on my Mac

Cagliari

Answer 1

Specify --component-dir and --worker-configs explicitly

You'll have to pass both options to include the Homebrew location of Kafka Connect's plugin.path setting (where Confluent Hub would install to) and your connect-distributed.properties (so that the plugin path can be setup if it isn't already)

e.g. (on a Mac, after brew install kafka)

confluent-hub install <name> \
   --component-dir /usr/local/Cellar/kafka/plugins \
   --worker-configs /usr/local/etc/kafka/connect-distributed.properties

This command should download and extract the connector zip into /usr/local/Cellar/kafka/plugins and modify the connect-distributed.properties file to include that path in its plugin.path property

Here is an example using a Kafka Connect docker image that I've built

Stanislaw

are there suggested locations for both these parameters? – Hairpin

@Hairpin I would say a location where the service account running Kafka Connect has access to read them. By default in Confluent Platform, components are installed under /usr/share/java and configs are placed within /etc – Stanislaw

How to find these paths? If installed from docker? – Watt

@Megan The confluentinc connect container is already pre-configured for using Confluent Hub, so you don't need to specify them. You can find examples of usage by blog posts from rmoff – Stanislaw

Answer 2

MacOS

In my case I have installed Kafka and Confluent Hub with brew and the installation path for Kafka is /usr/local/Cellar/kafka. And created a directory to install the connector.

Move to /usr/local/Caskroom/confluent-hub-client/6.2.1/bin, it is the directory confluent-hub is located.

Then I've executed

confluent-hub install confluentinc/kafka-connect-azure-blob-storage:1.6.2 \
  --component-dir /Users/Rustam_Zokirov/BigData/connectors \
  --worker-configs /usr/local/Cellar/kafka/3.0.0/libexec/config/connect-distributed.properties

Viola

Thanks! For M1 Mac, homebrew installed kafka here instead: /opt/homebrew/Cellar/kafka/3.3.1_1/libexec/config/connect-distributed.properties . Then I created the other directory for --component-dir – Ioves