Python: Mocking out Kafka for integration tests

Question

I'm somewhat new to integration tests. I have two services that pass messages to one another using Kafka. However, for my integration tests, I don't necessarily want to have Kafka running in order to run my tests. Is there a standard way to mock out Kafka? Or is this something I need to create myself, some MockKafka queue and patch where appropriate in the app? Additionally is this in violation of what an integration test should do? My take on this is that I am not testing any functionality of Kafka, and there for the sake of integration tests should be mocked out.

Johiah

no answer since a year, any updates on this please? I have a similar scenario. – Maritamaritain

If part of what you are testing depends on how Kafka behaves then it is essential to use a real Kafka server. If you are testing how your service might receive messages from Kafka and what it does with them, you're quite easily able to stub this out. – Johiah

Answer 1

Mocking the Kafka itself is not viable, since its protocol is not generic RFC-backed standard (unlike HTTP) and has only one implementation.

On the unit test level, you can abstract and mock the functions that require Kafka, but that's not your case. You want the integration test, so, most likely you would have to involve actual Kafka to have actual interactions tested.

To involve real Kafka with smallest effort, I'd use special self-contained Docker image (like this one: https://github.com/up9inc/async-ms-demo/tree/main/kafka). Build it and run it with 9092 port exposed.

For integration testing with Kafka, my principle is to not mock Kafka itself, and do mock "undesired/unrelated" services with what I call "Mock Actor".

"Mock Actors" approach is described here in detail: https://up9.com/mock-actor-approach-for-kafka (article written by me). There is a tool implementing those features, too (http://mockintosh.io).

Scow