How to determine API version of Kafka?

Question

I am using kafka-python for accessing Kafka. I try to create a Kafka Producer:

kafka_producer = KafkaProducer(bootstrap_servers=['kafka:9092'])

but this fails with exception kafka.errors.NoBrokersAvailable: NoBrokersAvailable.

I've found out I need to add api_version parameter to the KafkaProducer:

kafka_producer = KafkaProducer(bootstrap_servers=['kafka:9092'],
                               api_version=(0, 10, 1))

This command works.

The question is: how to determine value of api_version?

kafka-broker-api-versions.sh --bootstrap-server localhost:9092 gives me something, but I am not sure if there's a number I can use. I tried random values like api_version=(20, 2, 1) and it also worked.

Sechrist

Ideally, it's the version of Kafka you've installed, or the lowest version you're targeting. 0.10.2 is a safe default – Strobile

I am more interested in why is it a safe default. Source? – Cowpea

That's that point at which any version of newer clients should work - cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix – Strobile

@cricket_007 Thanks, that's useful. I assume I can set any version to api_version parameter in KafkaProducer constructor, if it does not exist the class selects something on its own. – Cowpea

Answer 1

This was solved in the discussion in the comments already:

Ideally, it's the version of Kafka you've installed, or the lowest version you're targeting. 0.10.2 is a safe default

That's that point at which any version of newer clients should work - cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix

Melaniamelanic

Answer 2

Let kafka-python guess your Kafka API version:

from kafka import KafkaProducer  # pip3 install kafka-python

k = KafkaProducer(bootstrap_servers="my-kafka.host.internal")
print(k.config['api_version'])

Queenstown