What is the best practice for naming kafka topics?

Question

We are new to kafka and we have a few teams working on a few applications that publish/subscribe events to/from each other. Since the kafka topic names are going to be shared across teams, is there any best practice for naming?

Basically we don't want to see team A naming a topic companyname-appname-events while team B naming another topic productname_functionB in totally different styles.

Any suggestions are appreciated !

Note this probably sounds more like the following asked question: What should be naming convention of topic and partition of Kafka? However, the author there was asking something more specific.

Hahnke

Answer 1

https://cnr.sh/essays/how-paint-bike-shed-kafka-topic-naming-conventions helped us answering that same question.

As a summary this article suggest to follow similar best practices to naming databases and tables, and it provides these additional points of advice:

1. Avoid topic names based on things that change
2. Avoid topic names based on information that would be stored in other places
3. Avoid topic names based on their planned consumers/producers. This is essentially a special case of the first advice :D.
4. Decide casing early on, and consider enforcing it or at least check/monitor it. This way you catch offenders early on.

Setup

The message-types should say something about the structure of the message. (log) indicates a relatively small message with a lot of dynamism and some stable meta-data. (queue) indicates a stable meta-data that is one of many in an ongoing collection. – Aeneus

I've seen some people suggest adding verbs to topics. Based on the link posted, I just want to clarify: should we never do this? For example, if we delete a contact, we should publish onto the topic queue.address_book.contact the cancellation in the data instead, like { change: 'deleted', other: 'data' }. Should we avoid having a topic like queue.address_book.contact.deleted? – Incognito

I guess one could say "if it makes sense in your world, maybe?". But typically the topic would be consumed by an entity, and it feels more useful then to have all events describing that entity in the same topic. There is an important reason why you may want to use a single topic: You can only rely on order of events through offsets when they are in the same topic (and partition!). Splitting events that somehow require relative ordering into different topics means you need to work out a way to order them explicitly :) – Setup

@Incognito I agreen with @Setup that in this case you should use a single topic. Another reason is that the event implementation classes are often organized in hierarchies. So, you could have a ContactEvent that is the superclass of ContactAddedEvent, ContactDeletedEvent, etc. – Neurath

I am not very happy with the use of application names in the blog article. From an architectural point of view, you should avoid coupling to applications via application names. This may not be a problem at the beginning of a project. But software will be replaced, extended, etc... at some point... it really helps if not every application has to be modified afterwards because of a new topic name. – Yettie

Answer 2

I would like to present an alternative approach to the above answer, which has worked well in practice so far and which does not cause any coupling to the product or application name.

My recommendation:

1. Use of business domains and sub-domains, e.g.:

   public.sales.ecommerce.shoppingcarts

   private.risk.portfolio.analysis.loans.csvimport

2. In addition, domain internal topics can be marked with "private" or external topics (i.e. after quality assurance for example) with "public". Another example:

   private.risk.portfolio.pricingengine.assetpricing

The number of domains and subdomains and whether you need to mark streams as public or private obviously depends on your organization's size. As always, overengineering should be avoided.

I have written a blog post "Topic naming conventions: How do I name my topics? 5 recommendations with examples", which goes into more details.

Yettie