Kafka : Running Confluent in a Windows environment

Question

I set up Kafka for local run. I have written sample producer and consumer in Java and running from local, by starting server and zookeeper.
I want to use oracle as producer, that will require to write the configuration file(already written), confluent shell script to run it on Unix.

Is there any way to run confluent on Windows, I could not find the batch file confluent in setup?

Also, is there any way to run Oracle as producer, without using confluent script?

Cavan

command to run same in unix : $ ./bin/confluent load jdbc_source_mysql_foobar_01 -d /tmp/kafka-connect-jdbc-source.json – Cavan

Answer 1

Confluent Platform is not supported on Windows. Best option if you have to use Windows is the Docker images, which are fully supported.

Dichroic

Documentation can be found here docs.confluent.io/current/installation/docker/index.html – Gloxinia

Answer 2

The Confluent software has plenty of shell scripts and you can leverage them on Windows if you can use Cygwin ( https://www.cygwin.com/ ).

Pre-requisite: The Java SDK to use should be installed on a file path without space.

Pre-requisite: Cygwin with curl installed

Download the Confluent distribution and install (unpack) on a file path without space.

For each of the following start a Cygwin session and set JAVA_HOME to the SDK and goto /bin

Start zookeeper with

./zookeeper-server-start ../etc/kafka/zookeeper.properties

Start Kafka broker with

./kafka-server-start ../etc/kafka/server.properties

Start Confluent Schema Registry with

./schema-registry-start ../etc/schema-registry/schema-registry.properties

Linell

Answer 3

yes you can run it in windows. Below is window patch for confluent 5.0.1.

https://github.com/mduhan/confluent-windows-5.0.1

Lala

Answer 4

confluent command is written in Bash, so you would need something like the WASL or Cygwin to run it successfully natively (outside of Docker / a VM)

By "oracle" sounds like you are trying to run Kafka Connect JDBC.

You can find connect-standalone.bat in the bin/windows directory.

It takes a properties configuration file, which would map accordingly to the kafka-connect-jdbc-source.json file, not the JSON file itself, though.

If you want to POST JSON to the Connect API, you need to use connect-distributed script.

Apropos

Answer 5

The latest confluent package has windows support. https://docs.confluent.io/4.0.0/installation/installing_cp.html#installation-archive

They have zookeeper and Kafka server bat files. But Schema registry bat files are not there. However, you can download the below files.

https://github.com/renukaradhya/confluentplatform/blob/master/bin/windows/schema-registry-run-class.bat https://github.com/renukaradhya/confluentplatform/blob/master/bin/windows/schema-registry-start.bat

Save above bat files in CONFLUENT_HOME\bin\windows directory and run as below:

schema-registry-start.bat ....\etc\schema-registry\schema-registry.properties

Behoof

Obtained this message Error: Could not find or load main class .\config\schema-registry.properties Caused by: java.lang.ClassNotFoundException: /\config\schema-registry/properties – Virgenvirgie

Answer 6

I know this is old but since it sill comes up in google search so it is important to have a practical answer on it.

In this answer I assumed that you are going to also deploy your app on Windows Server at some point

As Robin Moffatt said, Kafka is not supported "cleanly"(I added this ^^) for windows and docker images for linux containers on windows are fully supported.

But do not develop your app based on linux containers on docker windows, because the linux containers are only working with windows 10 and you don't have them on windows server (Yeah I know there is an experimental vesion ready but you can not trust that with production and it doesn't really work on old build versions of windows server 2016 if your customers like ours are old school type people).

I am also currently struggling with the same problem and I am contacting our project manager back and forth to choose from one of the solutions below:

1. Find a good alternative to Kafka that works on windows and also runs on windows containers
2. Run your app on a Linux machine and have docker installed there. if you are having microservices architecture with .net core it runs perfectly on Linux.
3. This way is like reinventing some parts of a wheel. You need to create windows services for the confluent Kafka that they provide to you in a zip file. You need to try having schema-registry, kafka connect and also control center up and running yourself. Controlling Kafka without the help of control center is a pain specially in production environment when you are not gonna be the one gonna install everything (If you were able to go with option 3 you probably can create your own windows image for Kafka (The wheel invented))

I know this is not really helpful but it is the result of my research about this Issue and I hope either Microsoft finishes the development of Linux containers or confluent releases a windows container image. For me the solution number 2 if possible is the best solution.

Weldonwelfare

Answer 7

Best way is to run Kafka, Zookeeper through Docker, example : https://hub.docker.com/r/wurstmeister/kafka/

Lodicule

tihs is a dockerized solution, also confluent has it's own docker images of zookeeper and kafka ... – Brat