In practice, how eventual is the "eventual consistency" in HRD?

Question

I am in the process of migrating an application from Master/Slave to HRD. I would like to hear some comments from who already went through the migration.

1. I tried a simple example to just post a new entity without ancestor and redirecting to a page to list all entities from that model. I tried it several times and it was always consistent. Them I put 500 indexed properties and again, always consistent...

2. I was also worried about some claims of a limit of one 1 put() per entity group per second. I put() 30 entities with same ancestor (same HTTP request but put() one by one) and it was basically no difference from puting 30 entities without ancestor. (I am using NDB, could it be doing some kind of optimization?)

I tested this with an empty app without any traffic and I am wondering how much a real traffic would affect the "eventual consistency".

I am aware I can test "eventual consistency" on local development. My question is:

Do I really need to restructure my app to handle eventual consistency?

Or it would be acceptable to leave it the way it is because the eventual consistency is actually consistent in practice for 99%?

Socialist

How did you list all entities under 1. ? – Interrupt

What does your application do? Would there be visible, negative effects if your writes were eventually consistent? – Enrapture

ndb can do auto-batching of your put, see code.google.com/p/appengine-ndb-experiment/source/browse/ndb/… – Zwick

Answer 1

If you have a small app then your data probably live on the same part of the same disk and you have one instance. You probably won't notice eventual consistency. As your app grows, you notice it more. Usually it takes milliseconds to reach consistency, but I've seen cases where it takes an hour or more.

Generally, queries is where you notice it most. One way to reduce the impact is to query by keys only and then use ndb.get_multi() to load the entities. Fetching entities by keys ensures that you get the latest version of that entity. It doesn't guarantee that the keys list is strongly consistent, though. So you might get entities that don't match the query conditions, so loop through the entities and skip the ones that don't match.

From what I've noticed, the pain of eventual consistency grows gradually as your app grows. At some point you do need to take it seriously and update the critical areas of your code to handle it.

Tibia

Answer 2

What's the worst case if you get inconsistent results?

• Does a user see some unimportant info that's out of date? That's probably ok.

• Will you miscalculate something important, like the price of something? Or the number of items in stock in a store? In that case, you would want to avoid that chance occurence.

From observation only, it seems like eventually consistent results show up more as your dataset gets larger, I suspect as your data is split across more tablets.

Also, if you're reading your entities back with get() requests by key/id, it'll always be consistent. Make sure you're doing a query to get eventually consistent results.

Hurtle

Answer 3

The replication speed is going to be primarily server-workload-dependent. Typically on an unloaded system the replication delay is going to be milliseconds.

But the idea of "eventually consistent" is that you need to write your app so that you don't rely on that; any replication delay needs to be allowable within the constraints of your application.

Operation