Making POST requests idempotent

Question

I have been looking for a way to design my API so it will be idempotent, meaning that some of that is to make my POST request routes idempotent, and I stumbled upon this article.

(If I have understood something not the way it is, please correct me!)

In it, there is a good explanation of the general idea. but what is lacking are some examples of the way that he implemented it by himself.

Someone asked the writer of the article, how would he guarantee atomicity? so the writer added a code example.

Essentially, in his code example there are two cases,

the flow if everything goes well:

• Open a transaction on the db that holds the data that needs to change by the POST request
• Inside this transaction, execute the needed change
• Set the Idempotency-key key and the value, which is the response to the client, inside the Redis store
• Set expire time to that key
• Commit the transaction

the flow if something inside the code goes wrong:

• and exception inside the flow of the function occurs.
• a rollback to the transaction is performed

Notice that the transaction that is opened is for a certain DB, lets call him A. However, it is not relevant for the redis store that he also uses, meaning that the rollback of the transaction will only affect DB A.

So it covers the case when something happends inside the code that make it impossible to complete the transaction.

But what will happend if the machine, which the code runs on, will crash, while it is in a state when it has already executed the Set expire time to that key and it is now about to run the committing of the transaction?

In that case, the key will be available in the redis store, but the transaction has not been committed. This will result in a situation where the service is sure that the needed changes have already happen, but they didn't, the machine failed before it could finish it.

I need to design the API in such a way that if the change to the data or setting of the key and value in redis fail, that they will both roll back.

What is the solution to this problem?

How can I guarantee the atomicity of a changing the needed data in one database, and in the same time setting the key and the needed response in redis, and if any of them fails, rollback them both? (Including in a case that a machine crashes in the middle of the actions)

Please add a code example when answering! I'm using the same technologies as in the article (nodejs, redis, mongo - for the data itself)

Thanks :)

Overman

What technologies are you using? Node.JS? .NET? We can’t provide code if we don’t know what language/tech you’re in :) – Quentin

I'm using the same technologies as in the article, I will point it out in the question. thanks! – Overman

Answer 1

Per the code example you shared in your question, the behavior you want is to make sure there was no crash on the server between the moment where the idempotency key was set into the Redis saying this transaction already happened and the moment when the transaction is, in fact, persisted in your database.

However, when using Redis and another database together you have two independent points of failure, and two actions being executed sequentially in different moments (and even if they are executed asynchronously at the same time there is no guarantee the server won’t crash before any of them completed).

What you can do instead is include in your transaction an insert statement to a table holding relevant information on this request, including the idempotent key. As the ACID properties ensure atomicity, it guarantees either all the statements on the transaction to be executed successfully or none of them, which means your idempotency key will be available in your database if the transaction succeeded.

You can still use Redis as it’s gonna provide faster results than your database.

A code example is provided below, but it might be good to think about how relevant is the failure between insert to Redis and database to your business (could it be treated with another strategy?) to avoid over-engineering.

async function execute(idempotentKey) {
  try {
    // append to the query statement an insert into executions table.
    // this will be persisted with the transaction
    query = ```
        UPDATE firsttable SET ...;
        UPDATE secondtable SET ...;
        INSERT INTO executions (idempotent_key, success) VALUES (:idempotent_key, true);
    ```;

    const db = await dbConnection();
    await db.beginTransaction();
    await db.execute(query);

    // we're setting a key on redis with a value: "false".
    await redisClient.setAsync(idempotentKey, false, 'EX', process.env.KEY_EXPIRE_TIME);

    /*
      if server crashes exactly here, idempotent key will be on redis with false as value.
      in this case, there are two possibilities: commit to database suceeded or not.
      if on next request redis provides a false value, query database to verify if transaction was executed.
    */

    await db.commit();

    // you can now set key value to true, meaning commit suceeded and you won't need to query database to verify that.
    await redis.setAsync(idempotentKey, true);
  } catch (err) {
    await db.rollback();
    throw err;
  }
}

Loafer

Okay, so that is using the same DB to ensure the atomicity, and it might be the only way it is possible to do such a thing, but if so, then why would the writer of the article (that I referenced at the beginning) use two completely different databases? Are we missing something here? Or is he just misleading? – Overman

Anyway, I'm not sure if that is the best way to do it, it looks like saving the data about the idempotent key and the result of the request in the same DB as the data itself it a bit wierd... Also, what if the DB for the data itself is not the right fit for the key-value pairs of the idempotency, makes it pretty complicated logic to check if a request has been made before if we use two DB's that have duplicated data. – Overman

The database is a fallback for an edge case. For almost all the time, if not every time, you must only access Redis. As I mentioned in my answer, considering this edge case can be overengineering, I'd abstract that if data is saved in Redis, commit to the database also succeeded. That's what the author of the post is also considering. Redis is designed for many use cases including cache, with all the related implementations as expiry, and your database won't provide that. Only if it's crucial, you can check the idempotent table in your database if idempotentKey is false on Redis (edge case). – Loafer

And wont that data duplication (even for a while, given that the redis is a cache and eventually the data in it will be whiped) be considered a problem? Is this the solution for large scale applications? to write the same data to both data bases? – Overman

The duplication itself is not a problem. You're doing like this as you want to still benefit from Redis as it's designed for what you need. The database table with the information is just there so you can fallback when the Redis value for the idempotent key is false, as it's set to true only after commit transaction to database completes. But it should be true in all cases though as if there will be any problem, it will likely be on client-side (user connection failure causing a duplication action by the user). And for this case, you'll be safe. I hope it makes sense for you now! – Loafer

But again: when I refer to overengineering it's because you'll add a lot of complexity to your code for something that might never happen. Not to mention this table can grow up quickly as it will save all your requests. If your business logic can handle this edge case (although as I mentioned before, if there's a problem it will likely be on the user side) and you can keep data only on Redis, you can go for this direction. So the author's approach is completely understandable. – Loafer

Yeah thats what I'm thinking, it might be overengineering... But the problem is that I have to design the API in such a way that there cant be anything that will cause a non reliable answer from the service, as if I'm working with money transactions... From your experience, have you ever seen such implementation on production environments? maybe you've encountered a better solution? – Overman

Let us continue this discussion in chat. – Overman

I made some comments on the chat on previous experiences I've had with two-phase commit, queues where duplicates can't be tolerated and some architectures you can consider for achieving the behavior you need. – Loafer

@Loafer the code example that the author of the article you referenced provided is incorrect and it was rightfully pointed to the author in the comments. The reason could be that the author does not understand fully the concepts of ACID and that you can't achieve it when using two independent stores (redis and DB). I recommend to read this guy instead: brandur.org/idempotency-keys he is definitely knowledgable about this matter. – Tanah

@max how is storing false and then updating with true in redis better than simply storing the final key after committing a transaction? Say we store the key in redis after the transaction, then upon second request execution if the key is not in redis, we query the DB and if it is not there, we repeat the request execution, otherwise return previous result. Am I missing something? We get the same result: most of the times, redis will be the only destination and in rare cases of computer death right after transaction, request will have to consult the DB. – Tanah