How to publish messages directly from existing Mainframe application to Kafka topics?

Question

We are planning to replace IBM MQ with Apache Kafka to implement pub-sub use-cases in future. Currently Mainframe(running Z/OS) applications are written using COBOL programming language & pushing messages to MQ. I went through the documentation to find out about Clients supported by Kafka and seems, it doesn't support COBOL.

What's the best approach to build producer on Mainframe and publish messages directly to Kafka topics with minimal change in existing applications ?

Wimbush

Answer 1

Lots of options. You can publish to Kafka using any REST/HTTP client in any language. You can leave WebSphere MQ on zOS and run an external Kafka Source Connector to pull from MQ and put into Kafka. You can write to DB2 or another database and use a number of CDC tools (including IBM InfoSphere) to send the database updates to Kafka. You can use the Confluent JMS client to publish using Java. You can help finish the work to get librdkafka C client to run native on zOS. You could use Datapower as an ESB gateway between your mainframe app and Kafka via the Confluent REST Proxy. It all depends on your use case which option is best.

Some Links of interests:

Confluent Kafka JMS Client http://docs.confluent.io/current/clients/kafka-jms-client/docs/index.html

DataMountaineer Kafka Connectors for JMS Source: http://docs.datamountaineer.com/en/latest/jms-source.html

Sink: http://docs.datamountaineer.com/en/latest/jms.html

Ginsburg

Thanks Hans for sharing the links. Is there any Kafka source connector for IBM MQ as well? – Wimbush

Btw isn't the use of JMS client only for existing JMS brokers? Can we leverage the same for IBM MQ as well running on zOS? Ask here is to replace MQ and start publishing messages directly from Mainframe application to Kafka topics. Is that feasible without any additional hop ? – Wimbush

Has anybody even started a z/OS port of librdkafka? From a quick browse of the github repo it doesn't seem so. It looks like it wouldn't be too difficult if there was a market for it. – Racialism

@HansJespersen, So can we say there is no Kafka source connector for IBM MQ ? we'll need to leverage some client libraries such as librdkafka C client or Java which can run native on ZOS.. Otherwise, use IIB to pull messages from MQ and publish to Kafka topics ? – Wimbush

There are source connectors for MQ but the connector instances would run off the mainframe and connect as an MQ client to the MQ Manager running on z/OS. – Ginsburg

Contact Confluent directly if you want to learn more about all the integration options as StackOverflow is not the best place to go over all the details or pricing. – Ginsburg

Answer 2

I'm assuming you want to publish z/OS DB2 changes into Kafka. The solution that takes the lowest MIPS for us is IBM InfoSphere CDC for z/OS, together with the other IBM part# D1RC9LL. If your budget allows high CAPEX and low OPEX, then this is the way to go for you.

Bergschrund

Answer 3

now CICS support sending messages into Kafka, take a look at this link

https://developer.ibm.com/cics/2020/01/06/cics-and-kafka-integration/

Dire