Is Event sourcing using Database CDC considered good architecture?

Question

When we talk about sourcing events, we have a simple dual write architecture where we can write to database and then write the events to a queue like Kafka. Other downstream systems can read those events and act on/use them accordingly.

But the problem occurs when trying to make both DB and Events in sync as the ordering of these events are required to make sense out of it.

To solve this problem people encourage to use database commit logs as a source of events, and there are tools build around it like Airbnb's Spinal Tap, Redhat's Debezium, Oracle's Golden gate, etc... It solves the problem of consistency, ordering guaranty and all these.

But the problem with using the Database commit log as event source is we are tightly coupling with DB schema. DB schema for a micro-service is exposed, and any breaking changes in DB schema like datatype change or column name change can actually break the downstream systems.

So is using the DB CDC as an event source a good idea?

A talk on this problem and using Debezium for event sourcing

Terza

We can rather decouple both the source (say DB) and target systems (downstream) from the actual implementation. Adapters might just come in handy. In particular to Kafka landscape, we might choose to use Avro to handle version compatibility of different message schema. – Dubbing

Note that in general, questions about architecture are more topical at Software Engineering than here; Stack Overflow is more focused on specific, narrow, tactical issues. – Tryst

Answer 1

If you are using Event sourcing:

Then the coupling should not exist. The Event store is generic, it doesn't care about the internal state of your Aggregates. You are in the worst case coupled with the internal structure of the Event store itself but this is not specific to a particular Microservice.

If you are not using Event sourcing:

In this case there is a coupling between the internal structure of the Aggregates and the CDC component (that captures the data change and publish the event to an Message queue or similar). In order to limit the effects of this coupling to the Microservice itself, the CDC component should be part of it. In this way when the internal structure of the Aggregates in the Microservice changes then the CDC component is also changed and the outside world doesn't notice. Both changes are deployed at the same time.

Andres

When you say "CDC should be part of it" , does that mean that the microservice itself will consume the CDC and convert it into an event and publish. – Terza

Also the CDC has two use cases , 1. Creating events for other services ,2. Sharing data with the data pipeline. – Terza

@Terza I mean the CDC component, the process or whatever that tails the commit log and creates the events from it – Andres

@Terza The CDC component could for example be a sidecar (process or event container) that runs along the Microservice process – Andres

Answer 2

Extending Constantin's answer:

TLDR;

Transaction log tailing/mining should be hidden from others.

It is not strictly an event-stream, as you should not access it directly from other services. It is generally used when transitioning a legacy system gradually to a microservices based. The flow could look like this:

1. Service A commits a transaction to the DB
2. A framework or service polls the commit log and maps new commits to Kafka as events
3. Service B is subscribed to a Kafka stream and consumes events from there, not from the DB

Longer story:

Service B doesn't see that your event is originated from the DB nor it accesses the DB directly. The commit data should be projected into an event. If you change the DB, you should only modify your projection rule to map commits in the new schema to the "old" event format, so consumers must not be changed. (I am not familiar with Debezium, or if it can do this projection).

Your events should be idempotent as publishing an event and committing a transaction atomically is a problem in a distributed scenario, and tools will guarantee at-least-once-delivery with exactly-once-processing semantics at best, and the exactly-once part is rarer. This is due to an event origin (the transaction log) is not the same as the stream that will be accessed by other services, i.e. it is distributed. And this is still the producer part, the same problem exists with Kafka->consumer channel, but for a different reason. Also, Kafka will not behave like an event store, so what you achieved is a message queue.

I recommend using a dedicated event-store instead if possible, like Greg Young's: https://eventstore.org/. This solves the problem by integrating an event-store and message-broker into a single solution. By storing an event (in JSON) to a stream, you also "publish" it, as consumers are subscribed to this stream. If you want to further decouple the services, you can write projections that map events from one stream to another stream. Your event consuming should be idempotent with this too, but you get an event store that is partitioned by aggregates and is pretty fast to read.

If you want to store the data in the SQL DB too, then listen to these events and insert/update the tables based on them, just do not use your SQL DB as your event store cuz it will be hard to implement it right (failure-proof).

For the ordering part: reading events from one stream will be ordered. Projections that aggregates multiple event streams can only guarantee ordering between events originating from the same stream. It is usually more than enough. (btw you could reorder the messages based on some field on the consumer side if necessary.)

Venipuncture

The flow that you mentioned : "A framework or service polls the commit log and maps new commits to Kafka as events" => How do you keep code/deployment of service A and that framework/service which polls the commit log in sync ? Like a breaking change in the DB schema by Service A can break that framework/service . – Terza

Debezium/other TxLog parsing tools will provide idempotent events. Events will be only published once DB commit happened. For example , if salary increased for an employee , it will give Employee table : Old { Salary : 100} , New { Salary: 150}. – Terza

If the CDC is a framework and it runs inside service A then you can simply rewrite the projection rule and commit it with the DB schema change. E.g. you renamed salary to payment, then modify the projection to map payment to salary, as consumers should not know you renamed a field. If cannot write a projection that hides the change, then you have to modify the clients too. You could also use versioned events, and if schema changes are incompatible, then a new event type or version is published. – Venipuncture

If it is a service, then it depends on how you deploy. E.g. you could stop the CDC, deploy the new service A with new schema, then modify the projection for the CDC and redeploy it too. It will catch up eventually if new transactions are already committed in service A before the CDC is started. The most important part is to convert the events published from the CDC to an unified event format, maybe you can even do this in Kafka instead of the CDC. – Venipuncture

Answer 3

So is using the DB CDC as an event source a good idea?

"Is it a good idea?" is a question that is going to depend on your context, the costs and benefits of the different trade offs that you need to make.

That said, it's not an idea that is consistent with the heritage of event sourcing as I learned it.

Event sourcing - the idea that our book of record is a ledger of state changes - has been around a long long time. After all, when we talk about "ledger", we are in fact alluding to those documents written centuries ago that kept track of commerce.

But a lot of the discussion of event sourcing in software is heavily influenced by domain driven design; DDD advocates (among other things) aligning your code concepts with the concepts in the domain you are modeling.

So here's the problem: unless you are in some extreme edge case, your database is probably some general purpose application that you are customizing/configuring to meet your needs. Change data capture is going to be limited by the fact that it is implemented using general purpose mechanisms. So the events that are produced are going to look like general purpose patch documents (here's the diff between before and after).

But if we trying to align our events with our domain concepts (ie, what does this change to our persisted state mean), then patch documents are a step in the wrong direction.

For example, our domain might have multiple "events" that make changes to the same, or very similar, sets of fields in our model. Trying to rediscover the motivation for a change by reverse engineering the diff is kind of a dumb problem to have; especially when we have already fought with the same sort of problem learning user interface design.

In some domains, a general purpose change is good enough. In some contexts, a general purpose change is good enough for now. Horses for courses.

But it's not really the sort of implementation that the "event sourcing" community is talking about.

Backslide

Answer 4

Besides Constantin Galbenu mentioned CDC component side, you can also do it in event storage side like Kafka stream API.

What is Kafka stream API? Input is read from one or more topics in order to generate output to one or more topics, effectively transforming the input streams to output streams.

After transfer detailed data to abstract data, your DB schema is only bind with the transformation now and can release the tightly relation between DB and subscribers.

If your data schema need to change a lot, maybe you should add a new topic for it.

Concent