Kafka Connect in debug mode

Question

I am using native apache kafka and not confluent's edition of kafka.

I am using using kafka jdbc connector pluggin. I intend to put a break point and inspect through my IDE (intelliJ).

How to make connect listen on a debug port and suspend. I referred What is a simple, effective way to debug custom Kafka connectors? but that's for confluent kafka

I am looking for below confluent configuration equivalents for apache kafka

CONNECT_DEBUG=y; export DEBUG_SUSPEND_FLAG=y;

Becnel

Answer 1

confluent's edition of kafka

There is no such thing. Both are the same Apache Kafka, so the scripts are the same (which is kafka-run-class)

What you reference, the confluent CLI command, is intended for local deployments of the Confluent stack, just redirects CONNECT_DEBUG into KAFKA_DEBUG

Therefore, it is always KAFKA_DEBUG for all Kafka Components (including Zookeeper, surprisingly) not CONNECT_DEBUG specifically for Connect.

Underfoot

Answer 2

So this won't work with kafka in docker compose. You need to bind *:5005 and KAFKA_DEBUG=y won't do this. I found out the hard way and found this https://www.mrjamiebowman.com/software-development/kafka/debugging-kafka-connect-connectors-smts-in-java/ to help me.

  connect:
    image: quay.io/debezium/connect:2.6
    ports:
      - 8083:8083
      - 5005:5005
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_source_connect_statuses
      - CONNECT_PLUGIN_PATH=/kafka/connect,/opt/connectors
      - JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,address=*:5005,server=y,suspend=n"
#      - KAFKA_DEBUG=y
#      - DEBUG_SUSPEND_FLAG=y
    depends_on:
      kafka:
        condition: service_healthy
    volumes:
      - ./kafka-connect-connectors-jar:/opt/connectors

Smear