Occasionally connected CQRS system

Question

Problem:

Two employees (A & B) go off-line at the same time while editing customer #123, say version #20, and while off-line continue making changes...

Scenarios:

1 - The two employees edit customer #123 and make changes to one or more identical attributes.

2 - The two employees edit customer #123 but DO NOT make the same changes (they cross each other without touching).

... they then both come back on-line, first employee A appends, thereby changing the customer to version #21, then employee B, still on version #20

Questions:

Who's changes do we keep in scenario 1?

Can we do a merge in scenario 2, how?

Context:

1 - CQRS + Event Sourcing style system

2 - Use Event Sourcing Db as a Queue

3 - Eventual Consistency on Read Model

4 - RESTful APIs

EDIT-1: Clarifications based on the answers so far:

In order to perform fined grained merging, I'll need to have one command for each of field in a form for example?

Above, finely grained commands for ChangeName, ChangeSupplier, ChangeDescription, etc., each with their own timestamp would allow for auto-merging in the event A & B both updated ChangedName?

Edit-2: Follow up based on the the use of a particular event store:

It seems as though I'll make use of @GetEventStore for the persistence of my event streams.

They make use of Optimistic Concurrency as follows:

• Each event in a stream increments stream version by 1

• Writes can specify an expected version, making use of the ES-ExpectedVersion header on writers

  • -1 specifies stream should not already exist

  • 0 and above specifies a stream version

  • Writes will fail if the stream is not at the version, you either retry with a new expected version number or you reprocessed the behavior and decided it's OK if you so choose.

• If no ES-Expected Version specified, optimistic concurrency control is disabled

• In this context, the Optimistic Concurrency is not only based on the Message ID, but also on the Event #

Passerby

Just a heads up, I asked another question related to this one here: #25257513 – Passerby

For EDIT-2: I'm not familiar with GetEventStore, but the ES-ExpectedVersion should be specified by the command handler, based on the aggregate version it loaded from the database. You should usually keep it. Your commands itself usually wouldn't contain a specific version, since commands represent use cases and should be oblivous to the underlying domain model (and hence, version of particular aggregates) – Jacquelyn

I'd like to use exactly the same model for our system, what reading is available that talks about turning commands into events, and possibly the offline aspect? – Edgebone

Answer 1

If I understand your design picture correctly, then the occasionally connected users enqueue commands, i.e., change requests, and when the user reconnects the queued commands are sent together; there is only one database authority (that the command handlers query to load the most recent versions of their aggretates); only the view model is synced to the clients.

In this setup, Scenario 2 is trivially auto-merged by your design, if you choose your commands wisely, read: make them fine-grained: For every possible change, choose one command. Then, on re-connection of the client, the commands are processed in any order, but since they only affect disjunct fields, there is no problem:

1. Customer is at v20.
2. A is offline, edits changes against stale model of v20.
3. B is offline, edits changes against stale model of v20.
4. A comes online, batch sends an queued ChangeName command, the Customer of v20 is loaded and persisted as v21.
5. B comes online, batch sends an queued ChangeAddress command, the Customer of v21 is loaded and persisted as v22.
6. The database contains the user with their correct name and address, as expected.

In Scenario 1, with this setup, both employees will overwrite the other employees' changes:

1. Customer is at v20.
2. A is offline, edits changes against stale model of v20.
3. B is offline, edits changes against stale model of v20.
4. A comes online, batch sends an queued ChangeName command to "John Doe", the Customer of v20 is loaded and persisted as v21 with name "John Doe"
5. B comes online, batch sends an queued ChangeName command to "Joan d'Arc", the Customer of v21 (named "John Doe") is loaded and persisted as v22 (with name "Joan d'Arc').
6. Database contains a user with name "Joan d'Arc".

If B comes online before A, then it's vice versa:

1. Customer is at v20.
2. A is offline, edits changes against stale model of v20.
3. B is offline, edits changes against stale model of v20.
4. B comes online, batch sends an queued ChangeName command to "Joan d'Arc", the Customer of v20 is loaded and persisted as v21 (with name "Joan d'Arc').
5. A comes online, batch sends an queued ChangeName command to "John Doe", the Customer of v21 is loaded and persisted as v22 with name "John Doe".
6. Database contains a user with name "John Doe".

There are two ways to enable conflict detection:

1. Check whether the command's creation date (i.e., the time of the employees modification) is after the last modification date of the Customer. This will disable the auto-merge feature of Scenario 2, but will give you full conflict detection against concurrent edits.
2. Check whether the command's creation date (i.e., the time of the employees modification) is after the last modification date of the individual field of the Customer it is going to change. This will leave the auto-merge of Scenario 2 intact, but will give you auto-conflict-detection in Scenario 1.

Both are easy to implement with event sourcing (since the timestamps of the individual events in the event stream are probably known).

As for your question "Who's changes do we keep in scenario 1?" -- this depends on your business domain and its requirements.

EDIT-1: To answer on the clarification question:

Yes, you'll need one command for each field (or group of fields, respectively) that can be changed individually.

Regarding your mockup: What you are showing is a typical "CRUD" UI, i.e., multiple form fields and, e.g., one "Save" button. CQRS is usually and naturally combined with a "task based" UI, where there would be, say, the Status field be displayed (read-only), and if a user wants to change the status, one clicks, say, a "Change Status" button, which opens a dialog/new window or other UI element, where one can change the status (in web based systems, in-place-editing is also common). If you are doing a "task based" UI, where each task only affects a small subset of all fields, then finely grained commands for ChangeName, ChangeSupplier etc are a natural fit.

Jacquelyn

Thanks for your answer. I'll accept it, but I edited my question to ask for clarification. – Passerby

Answer 2

Here's a generic overview of some solutions:

Scenario 1

Someone has to decide, preferably a human. You should ask the user or show that there is a conflict.

Dropbox solves this by picking the later file and keeping a file.conflict file in the same directory for the user to delete or use.

Scenario 2

Keep the original data around and see which fields actually changed. Then you can apply employee 1's changes and then employee 2's changes without stepping on any toes.

Scenario 3 (Only when the changes come online at different times)

Let the second user know that there were changes while they were offline. Attempt Scenario 2 and show the second user the new result (because this might change his inputs). Then ask him if he wants to save his changes, modify them first, or throw them out.

Ochre

Answer 3

Aaron, where the events do actually conflict, i.e. in scenario 1 then I would expect a concurrency exception of some sort to be thrown.

The second scenario is much more interesting. Assuming your commands and events are reasonably well defined, i.e. not a wrapper for CRUD then you would be able to test if the events committed since your command was issued actually conflict. I use a concurrency conflict registry for this purpose. Essentially when I detect a potential conflict I grab the events that have been committed since the version I currently have and ask the registry to check if any of them actually conflict.

If you want to see a code example and and a bit more detail on this I put together a post outlining my approach. Take a look at it here: handling concurrency issues in cqrs es systems

Hope this helps!

Clayborne

Answer 4

In this case, maybe you can use the "aggregate root" concept, for the Item which powered by CEP Engine (Complex Event Process Engine) to perform these complex operations.

Liba