Need defense against wacky challenge to Event Sourcing architecture w/CosmosDB

Question

In the current plan, incoming commands are handled via Function Apps, resulting in Events being sent to an Event Hub, and then materializing the views

Someone is arguing that instead of storing events in something like table storage, and materializing views based on events and snapshots, that we should:

Just stream events to a log in Azure Monitor to have auditing

We can make changes to a domain object immediately in response to a command and use the change feed as our source of events for materialized views.

He doesn’t see the advantage of even having a materialized view. Why not just use a query? Argument is we don’t expect a lot of traffic.

He wants to fulfill the whole audit log by saving events to the azure monitor log - Just an application log. Instead, that commands should just directly modify the representation of an entity in cosmos, and we'd use the change feed from CosmosDB as our domain object events, or we would create new events off of that via subscribers to that stream.

Is this actually an advantageous approach? Can ya'll think of any reasons why we wouldn't want to do that? Seems like we'd be losing something here.

He's saying we'd no longer need to be concerned with eventual consistency, as we'd have immediate consistency.

Every reference implementation I've evaluated does NOT do it the way he's suggesting. I'm not deeply versed in the advantages/disadvantages of the event sourcing / CQRS paradigm so I'm at a loss at the moment.. Currently researching furiously

This is a conceptual issue so there's not so much a code example. However, here's some references that seem to back up the approach I'm taking..

https://medium.com/@thomasweiss_io/planet-scale-event-sourcing-with-azure-cosmos-db-48a557757c8d

https://sajeetharan.com/2019/02/03/event-sourcing-with-azure-eventhub-and-cosmosdb/

https://learn.microsoft.com/en-us/azure/architecture/patterns/event-sourcing

Bryanbryana

plus one just for the awesome title – Steiermark

Answer 1

If your goal is only to have the audit log, state-based persistence could be a good choice. Event sourcing adds some complexity to the implementation side and unless you can identify more advantages of using it, you might not convince your team to bring this complexity to the system. There are numerous questions and answers on SO, as well as in some blog posts, about pros and cons of event sourcing, so I won't get into that discussion here.

I can warn you, though, that the second article in your list is very weak and would most probably lead you to many difficulties. The role of Event Hub there is completely unclear and it doesn't explain anything about projections and read-models (what you call "materialised views"). Only a very limited number of use-cases can live with only getting one entity by id and without being able to execute a query across multiple entities. That also probably answers your concern of having read-models at all. You will need them very soon when for the first time you will start figuring out how to get a list of entities based on some condition (query).

Using CosmosDb as the event store is completely feasible, as described in the first article if you can manage the costs involved. Just remember to set the change feed TTL to -1, otherwise, you won't be able to replay your projections when you need to.

To summarise:

• Keeping the audit log can be done without event-sourcing, but you need to ensure that events are published reliably, preferably in the same transaction as the entity state update. It is often hard or impossible but you might accept the risk of your audit requirement is not strict. You can also base your audit log on the CosmosDb change feed, just collecting document changes and logging them somewhere.
• Event sourcing is a powerful technique but it has both pros and cons. The most common prejudice against using event sourcing is its implementation complexity. It might not be a big issue if you have a team that is somewhat experienced in building event-sourced systems. If you don't have such a team, you might want to build a small-scale spike to get some experience.
• If you don't get full buy-in from the team to use event sourcing, you will later get all the blame if anything goes wrong. And it will go wrong at some point, especially with little experience in this area.
• Spend some time reading books and trying out things yourself, before going wild in production.
• Don't use Event Hub for anything that it is not designed for. Event Hub is the powerful event ingestion transport with limited TTL and it should be used for that purpose.
• Don't use Table Storage as the event store, unless you only read entities by id. I used it in production for such a scenario and it worked (to some extent) but you can't project read-models from there.

Wrac

would love some guidance on cosmosdb/table storage solutions for projections – Steiermark

Well, as I said, table storage won't work for projections because there's no way to create a real-time update feed. For CosmosDb, there are articles available, but to summarise - you need to create a collection where you put your events, which will serve as an event store and then subscribe to the change feed that would serve your projections. CosmosDb change feed can be configured to have the unlimited size and it enables to have features like full replay and such. – Wrac

thank you. i was just going to ask you for recommendations on a book on event sourcing, and saw that you wrote a book !!! – Steiermark

what is the benefit of being able to replay events? can you give an example? – Steiermark

By replaying you can reconstruct the past. Lets say you created a order with properties and then updated a few properties over time. With event sourcing you will have a event for each update(versions of the order). You can reconstruct the state of order to a specific time in the past. – Cavender

Answer 2

A simple rule of thumb is to not use products for tasks they weren't designed for.

Azure Monitor was not designed to store application domain data. Azure Monitor is designed to store telemetry data from your applications and services and provides features such as alerts and other types of integration into DevOps tools for managing the operation and health of your apps.

There is a simple reason why you were able to find articles on event sourcing using Cosmos DB and why our own docs talk about it. Because it was designed to be used this way. It is simple to set up Cosmos DB to be an append only event store for your applications and use Change Feed to fire off messages in other apps or services or, in your case, to maintain a materialized view state of domain objects within your app.

Gracioso