kafka connect exception, Replication factor: 3 larger than available brokers: 1

Question

kafka connect 5.4 only one connector, one worker and use connect-distributed.

Below is the error message:

[2020-06-22 19:09:58,700] ERROR [Worker clientId=connect-1, groupId=test-cluster] 
Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:290)
org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'test-connect-offsets'
    at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:262)
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99)
    at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:128)
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109)
    at org.apache.kafka.connect.runtime.Worker.start(Worker.java:186)
    at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:121)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:277)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
    at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:229)
    ... 11 more
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

Donohoe

Answer 1

For Kafka Connect to run in distributed mode it uses three topics that are stored on the Kafka cluster and hold information about configuration etc. You need to set in the Kafka Connect worker properties:

config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

If you're using the Docker image then you need to set the environment variables to override these, which in Docker Compose looks like:

CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"

Ref: Configuring Kafka Connect distributed workers

Haviland

The default of 3 is dumb in a Docker container. Most people will start by getting one server working first and then scale up later. – Landlubber

Flip it on its head and someone picking that container up for Production use would say it was "dumb" to set it to 1 by default 🤷‍♂️ :-D – Haviland

You start by crawling, then walking, then running. You don't start with the most complicated configuration by default. If you're intention is to get started (everyone was a new person at some point), you want it easy with sensible defaults. If you are in the smaller group of people who decide to scale up, then you're likely not surprise you'll need to set a config that says use a higher replication factor. The confluent container forces users to understand way more than they should up-front to get started. Unlike the debezium container for example – Landlubber

Answer 2

In addition to what Robin Moffatt answered, you also need to set in Docker:

CONNECT_CONFLUENT_TOPIC_REPLICATION_FACTOR: 1

And you may also want to change the Connector's setting like below:

"topic.creation.default.replication.factor": "1"

(This is Debezium's setting)

Counterproductive

Answer 3

it works after adding below to config.properties

offsets.topic.replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

Donohoe

You've found the answer but the additional configuration options you've listed are not applicable and would be misleading for people finding this. – Haviland