How to change the "kafka connect" component port?

Question

On port 8083 I am running Influxdb for which I am even getting the GUI on http://localhost:8083

Now come to kafka, Here I am referring the setup as per https://kafka.apache.org/quickstart

starting the zookeeeper which is in folder /opt/zookeeper-3.4.10 by the command: bin/zkServer.sh start

So zookeeper is started now starting kafka under /opt/kafka_2.11-1.1.0 folder as :
bin/kafka-server-start.sh config/server.properties

create a topic named "test" with a single partition and only one replica:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Topic is created and can be checked in with command :
bin/kafka-topics.sh --list --zookeeper localhost:2181

Uptill here everything is fine and tuned.

Now I need to use "Kafka connect" component to import/export data. So I am creating a seed data as: echo -e "foo\nbar" > test.txt

Now using connector configuration for "kafka connect" to work : bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties

after running above command I am getting : Address already in use

Kafka connect has stopped

I even changed the rest.port=8084 in the /opt/kafka_2.11-1.1.0/config/connect-distributed.properties so as it don't get conflict with influxdb which already on 8083. Still I am getting the same Address already in use, Kafka connect has stopped as shown in screenshots.

Lipinski

Please be a bit more specific when asking a question: What have you tried so far with a code example? (I downvoted because there is no code) / What do you expect? / What error do you get? For Help take a look at "How to ask" – Glue

Can you provide the command that you're using to launch Kafka Connect? From this we can understand if you're using Standalone or Distributed mode, and thus which config file to edit. – Reimport

I have phrased the whole problem step by step in main section...Hope this will be easy to know the issues @RobinMoffatt – Lipinski

Answer 1

Since you're using Kafka Connect in Standalone mode, you need to change the REST port in config/connect-standalone.properties:

rest.port=18083

To understand more about Standalone vs Distributed you can read the doc here.

Reimport

there is nothing like rest.port= option in config/connect-standalone.properties Should I put it manually ?? – Lipinski

My goodness @Robin Moffatt I wrote rest.port=8084 in config/connect-standalone.properties and it worked as per kafka.apache.org/quickstart tutorial. But why in kafka.apache.org/quickstart it is not given to mention to write a rest api port for connect-standalone.properties Thanks a lot !!! I have been working on it from past 3 days. – Lipinski

Nowadays, with Kafka 3.x, we have to use listeners=http://0.0.0.0:8084 as described here – Bilious

Answer 2

Kafka 3.0.0

Since Kafka Connect is intended to be run as a service, it also provides a REST API for managing connectors. The REST API server can be configured using the listeners configuration option. This field should contain a list of listeners in the following format: protocol://host:port,protocol2://host2:port2. Currently supported protocols are http and https. For example: listeners= http://localhost:8080,https://localhost:8443 By default, if no listeners are specified, the REST server runs on port 8083 using the HTTP protocol.

More details: https://kafka.apache.org/documentation/#connect_rest

Servetnick

Answer 3

UPDATE

If you still have no luck after putting 'listensers' in connect-standalone.properties, this is what worked for me. Instead of mentioning both protocols(http & https) in the listeners value, try putting only one protocol like 'http' there and instead of writing 'localhost' as hostname, write '127.0.0.1'. So the value of listeners would be:-

listeners=http://127.0.0.1:8084

Macintyre

Answer 4

Kafka Connect Standalone mode, uses Port 8084 as the Rest API post, by default. Due to this reason, if someone else is using that port already, the process with throw a BindException.

To change the port used above, navigate to the config/connect-standalone.properties file in the Kafka Root directory.

Add the following key value property to change the Port being used for Rest API opening. (Kafka should have included this in the properties file by default, else many developers go nuts trying to find the port mapping used in the standalone mode). Put a different port as you wish.

rest.port=11133

Tusk

Answer 5

Change the port definition in config/server.properties:

# The port the socket server listens on
port=9092

Hachman

That is the Kafka Broker port, not Kafka Connect REST API – Florez

OP didn't ask for the "REST" port. – Hachman

They did ask for Connect, though, which runs a REST server, if you see my answer instead of downvoting. Also, the broker running on 9092 would not conflict in InfluxDB on post 8083, which is what prompted the question – Florez

No he said he wanted to "connect" (lowercase) to the server. The edited question wasn't asked by OP. – Hachman

"change the port for Kafka connect" was the original question... Again, I'll point out that port 9092 would never conflict with port 8083 – Florez

Which simply means the port he "connects" to. Again, he didn't say what you claimed. Instead of making assumptions, let OP clarify. – Hachman

Please see revision 1... I'm done here. – Florez

Guys I meant to "change the port for Kafka connect" and @cricket_007 is right. Connecting to Kafka server and Kafka connect are two different aspects. Please share the knowledge that you guys have experienced without any conflicts of interest. We all are here to learn and share our knowledge on this great platform of stackoverflow to make each others work easier. : ) – Lipinski

@suryarahul: The meaning of your original statement is simply that you want to "connect" to Kafka. This correctly answers the question as originally stated. If you wanted something else, you should revise your question to reflect that. See stackoverflow.com/help/how-to-ask – Hachman

Nope ... I meant "Kafka connect" not "connect" to Kafka ...apologies from my side if my sentence was not clear and went misunderstood. – Lipinski

Now, coming to real problem... I am still getting the error that Address is already in use even though I changed rest.port=8084 in the /opt/kafka_2.11-1.1.0/config/connect-distributed.properties Kindly bail me out guys @cricket_007, @Hachman . I am not able to figure out a solution for this from past 2 days :( – Lipinski

What you meant and what you wrote were different in this case. Also, why did you mark it correct if it wasn't? You need to update your question with more information. – Hachman

What I meant, I wrote that, I didn't do any edit afterwards putting the question, so one can understand "Kafka connect" as ''connect to kafka", how in this my writing would be fully responsible for this if someone misunderstand "Kafka connect" as ''connect to kafka". @Hachman The point is misunderstanding can happen, but I really don't feel that issue need to be exaggerated this much, working on open source technologies need patience and knowledge sharing which is the prime moto of stackoverflow and that is what should be done with a smile :) and a cool mind. – Lipinski

The question was edited to say "port that Kafka Connect uses" which is completely different from your original question which simply asked how to connect to Kafka. In any case, you should update your question with more information. – Hachman

Really...? The original question said nothing like "port that Kafka uses". Or how to "connect to Kafka". The OP has been updated. – Florez

Wrong @cricket_007. The meaning was clear and your interpretation wasn't correct. The edited question is completely different from the original one. If you believe otherwise, then you should revert the edits. – Hachman

The question was edited to clarify. The original intent never changed. – Florez

Are you trying to be funny? Despite you upvoting your own answer, it was wrong. The meaning was changed in subsequent edits, otherwise the edits would've been unnecessary. – Hachman