How to prevent "Given transaction number 1 does not match any in-progress transactions" with Mongoose Transactions?

Question

I am using Mongoose to access to my database. I need to use transactions to make an atomic insert-update. 95% of the time my transaction works fine, but 5% of the time an error is showing :

"Given transaction number 1 does not match any in-progress transactions"

It's very difficult to reproduce this error, so I really want to understand where it is coming from to get rid of it. I could not find a very clear explanation about this type of behaviour.

I have tried to use async/await key words on various functions. I don't know if an operation is not done in time or too soon.

Here the code I am using:

export const createMany = async function (req, res, next) {
  if (!isIterable(req.body)) {
    res.status(400).send('Wrong format of body')
    return
  }
  if (req.body.length === 0) {
    res.status(400).send('The body is well formed (an array) but empty')
    return
  }

  const session = await mongoose.startSession()
  session.startTransaction()
  try {
    const packageBundle = await Package.create(req.body, { session })
    const options = []
    for (const key in packageBundle) {
      if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
        options.push({
          updateOne: {
            filter: { _id: packageBundle[key].id },
            update: {
              $set: {
                custom_id_string: 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
                  minimumIntegerDigits: 14,
                  useGrouping: false
                })
              },
              upsert: true
            }
          }
        })
      }
    }
    await Package.bulkWrite(
      options,
      { session }
    )
    for (const key in packageBundle) {
      if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
        packageBundle[key].custom_id_string = 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
          minimumIntegerDigits: 14,
          useGrouping: false
        })
      }
    }
    res.status(201).json(packageBundle)
    await session.commitTransaction()
  } catch (error) {
    res.status(500).end()
    await session.abortTransaction()
    throw error
  } finally {
    session.endSession()
  }
}

I expect my code to add in the database and to update the entry packages in atomic way, that there is no instable database status. This is working perfectly for the main part, but I need to be sure that this bug is not showing anymore.

Abruzzi

Answer 1

You should use the session.withTransaction() helper function to perform the transaction, as pointed in mongoose documentation. This will take care of starting, committing and retrying the transaction in case it fails.

const session = await mongoose.startSession();
await session.withTransaction(async () => {
    // Your transaction methods
});

---

Explanation:

The multi-document transactions in MongoDB are relatively new and might be a bit unstable in some cases, such as described here. And certainly, it has also been reported in Mongoose here. Your error most probably is a TransientTransactionError due to a write-conflict happening when the transaction is committed.

However, this is a known and expected issue from MongoDB and these comments explain their reasoning behind why they decided it to be like this. Moreover, they claim that the user should be handling the cases of write conflicts and retrying the transaction if that happens.

Therefore, looking at your code, the Package.create(...) method seems to be the reason why the error gets triggered, since this method is executing a save() for every document in the array (from mongoose docs).

A quick solution might be using Package.insertMany(...) instead of create(), since the Model.insertMany() "only sends one operation to the server, rather than one for each document" (from mongoose docs).

However, MongoDB provides a helper function session.withTransaction() that will take care of starting and committing the transaction and retry it in case of any error, since release v3.2.1. Hence, this should be your preferred way to work with transactions in a safer way; which is, of course, available in Mongoose through the Node.js API.

Elisha

¿Is it posible to explicitly use abortTransaction using session.withTransaction()? – Miscreant

@VincentGuyard yes, you can pass handle as first argument and use that to abort or commit at any time during the flow await session.withTransaction(async (handle) => { // Your transaction methods }); – Propagate

How write conflict is happen while we push await before session.commitTransaction(). It's should be committed success before create a new session, it mean has only one session one time? NOT parallel multiple sessions run the same time? It's right – Armlet

withTransaction is deprecated mongodb.com/docs/drivers/node/v4.14/fundamentals/transactions/… – Exsert

Answer 2

I was using a Promise.all([]) to execute several queries concurrently, but was passing the same session to each query. I tried doing as suggested above, and still encountered the same error. I removed the promise all (which was redundant anyway due to the nature of transactions) and my issue was solved!

Variant

I had exactly the same case! Thank you! But I'm still wondering why it works this way. – Stoic

I asked the GitHub Copilot and this is what it said: "When you use Promise.all, operations are executed in parallel and may attempt to use the session simultaneously. This can lead to conflicts because MongoDB does not guarantee that the session will work correctly when used concurrently from multiple operations." – Stoic

I don't understand. I am also facing this issue, and am doing a Promise.all within a transaction, and passing the same session, but why is it redundant due to the nature of transactions? Otherwise I would have to await multiple getters synchronously, and I'd prefer to not do this? – Incapacitate

At a database level transaction items are processed serially, one after another. Even if you add items to a transaction using a promise.all, the async aspect of the code isn't occurring at this stage. Its when you commit the transaction that the async code is executed, it will then execute each request in the order it was added to the transaction. So by using a promise.all, you just add items to your transaction at the "same time", but they are then actually executed serially. Rendering the promise.all redundant. Even if the method looks async in node, its syncronous when using a transaction – Variant

Answer 3

The accepted answer is great. In my case, I was running multiple transactions serially within a session. I was still facing this issue every now and then. I wrote a small helper to resolve this.

File 1:

// do some work here
await session.withTransaction(() => {});

// ensure the earlier transaction is completed
await ensureTransactionCompletion(session);

// do some more work here
await session.withTransaction(() => {});

Utils File:

async ensureTransactionCompletion(session: ClientSession, maxRetryCount: number = 50) {
        // When we are trying to split our operations into multiple transactions
        // Sometimes we are getting an error that the earlier transaction is still in progress
        // To avoid that, we ensure the earlier transaction has finished
        let count = 0;
        while (session.inTransaction()) {
            if (count >= maxRetryCount) {
                break;
            }
            // Adding a delay so that the transaction get be committed
            await new Promise(r => setTimeout(r, 100));
            count++;
        }
    }

Tupi

Answer 4

I have same issue, after a while trial & error, I found out adding a checking truthy ( if (result) ) works for me

const result = await Package.bulkWrite(
  options,
  { session }
)
for (const key in packageBundle) {
  if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
    packageBundle[key].custom_id_string = 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
      minimumIntegerDigits: 14,
      useGrouping: false
    })
  }
}
if (result) await session.commitTransaction()
else await session.abortTransaction()

Can't explain why, but it's worth to try

Misesteem