NServiceBus and Rabbit MQ or Kafka

Question

I am trying to learn messaging system. I have found that RabbitMq and NServiceBus are using together in few places. My questions are

1. If I am using the RabbitMQ then why do i need NServiceBus? and vice versa
2. What NServiceBus can do but RabbitMQ or Kafka cannot?
3. Can I use NServiceBus and kafka together? Or Apache-Kafka does not require NServiceBus

Recusancy

Related to your second question – Jackboot

@Jackboot Thanks for this. Is it same for Kafka? – Recusancy

To your question about Kafka: particular.net/blog/lets-talk-about-kafka – Correia

Answer 1

Years ago, I asked myself the same question. I was looking at NServiceBus to work with a different message queue, but the question was the same.

I decided not to use NServiceBus.

6 Month later, I realized I had re-built half of what NServiceBus did... only much more poorly.

The equivalent question of why would you need NServiceBus with RabbitMQ, is to ask why you would need the .NET Framework with ASP.NET MVC, or WinForms, or XAML, or any of the built-in libraries that .NET ships with, when you have the Common Language Runtime.

Shouldn't the CLR be enough, after all?

Of course not. Having the runtime on which code can execute - the MSIL interpreter and execution engine - is not nearly enough to be productive.

Sure, you can write command-line applications that take input and produce output. But try to build a real application without the common libraries - without the built-in SQL Server drivers; without any 3rd party controls or libraries. Build a Windows Desktop app without the System.Windows namespace.

You need those libraries to give you collections, and database access, and window objects and UI controls.

Similarly, RabbitMQ gives you everything you need to get started and working, but not enough to maintain productivity.

Sure, you can grab the .NET driver for RabbitMQ and start producing and consuming messages.

For a while, this will work just fine.

Pretty soon, you'll find yourself creating a wrapper around the driver, so you can reduce the amount of code you need to write.

Then you'll find yourself needing to deal with ack vs nack, and you'll create a simple API for that.

Then the need for dead-letter queues will pop up with nack calls, and you'll wrap that up in your API - simplified compared to the rabbitmq driver, of course.

Eventually, you'll want to deal with poison messages - messages that are malformed and causing exceptions. Once again, you don't want to write one-off code for this, so you'll write a library to handle it.

The list goes on and on.

6 months from now, you'll find yourself working with a half-written, barely specified, untestable library that only mimics the value and capabilities of NServiceBus (or MassTransit or whatever other service bus library you choose).

I won't say you have to use NServiceBus. And I would say you should learn how RabbitMQ works, without it. But once you get beyond the basics of sending and receiving messages, the value of NServiceBus and other service bus implementations, becomes very apparent, very quickly.

Orgasm

I finally have a concise answer! "The equivalent question of why would you need NServiceBus with RabbitMQ, is to ask why you would need the .NET Framework with ASP.NET MVC, or WinForms, or XAML, or any of the built-in libraries that .NET ships with, when you have the Command Language Runtime." annnd I'm going to tweet that. – Danette

A very nice clear explanation @Derick. Many thanks for taking your time to explan this. It is now clear enough – Recusancy

Thanks, Well written. – Sublease

You can always take a look at www.masstransit-project.com/ – Riegel

Perfect! I have been scratching my head around this question since the product I support involves NServiceBus but i never got why it is required in first place!! – Atrabilious

I've been struggling explaining our reasoning of choosing NServiceBus over just going with RabbitMQ and your summary nails it. We decided early that we would like to have all these features for free and not reinvent the wheel. The small drawback of this was that we didn't learn from our own mistakes and those lessons last longer. – Birdt

This recently came up for my org which has been working with rabbitmq for a while now and our org has done exactly what you've outlined in your answer - poor rmq wrapper. – Shockproof

@ChristianPaulin "for free" is maybe exaggerated when talking about NServiceBus :) – Coprology

Answer 2

It seems there is community support for Kafka transport in NServiceBus now: https://docs.particular.net/nservicebus/kafka/ (haven't tried it myself yet).

Derivation

That's a dead link now I'm afraid. – Llamas

github.com/pablocastilla/NServiceBus.Kafka It doesn't seem to be maintained though. – Aubin

Check out particular.net/blog/lets-talk-about-kafka for Particular's viewpoint on Kafka. – Maximamaximal