PyMongo transaction error:Transaction numbers are only allowed on a replica set member or mongos

Question

When I am using pymongo 3.7 transaction function to connect to mongo server 4.0, this error "Transaction numbers are only allowed on a replica set member or mongos" occurred and I cannot find any answer to solve this problem. My code is:

from pymongo import MongoClient
conn = MongoClient(host, port)
tb = conn.collector_gateway.try_table
with conn.start_session() as session:
    with session.start_transaction():
        tb.insert_one({"sku": "abc123", "qty": 100}, session=session)

The error is:

Traceback (most recent call last):
  File "/Users/yuzgu/PycharmProjects/seller_loss_alert/try_mongo.py", line 22, in <module>
    tb.insert_one({"sku": "abc123", "qty": 100}, session=session)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 693, in insert_one
    session=session),
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 607, in _insert
    bypass_doc_val, session)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 595, in _insert_one
    acknowledged, _insert_command, session)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1243, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1196, in _retry_with_session
    return func(session, sock_info, retryable)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/collection.py", line 590, in _insert_command
    retryable_write=retryable_write)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/pool.py", line 579, in command
    unacknowledged=unacknowledged)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/network.py", line 150, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/Users/yuzgu/anaconda3/lib/python3.6/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Transaction numbers are only allowed on a replica set member or mongos

Since mongo transaction function was published this year, I couldn't find the solution, can anyone help me with this problem?

Supply

Answer 1

Transactions are only available in a replica set setup (https://docs.mongodb.com/master/core/transactions/#transactions-and-replica-sets):

Multi-document transactions are available for replica sets only.

This error message is shown when you are attempting to do a transactional operation on a standalone mongod instance.

Please see Transactions page for more details & requirements.

Vitamin

Thanks for your answer, but I want to use transaction on a standalone mongod instance to ensure all my operations committed at the same time. Do you know any way to realize this? – Supply

You can start a replica set with one node for development purposes. Start a mongod —replSet <replsetname> then do rs.initiate() in the mongo shell. Be advised that this is not a recommended setup for a production environment. – Vitamin

Thanks. I see your profile on stackoverflow and know that you worked in MongoDb. Just out of curiosity, why didn't MongoDb open this functionality to standalone mongod instance? It seems to be a common scenario for users. – Supply

It’s because MongoDB is inherently a distributed database by design, and was not designed to be used as a standalone deployment. It is particularly challenging to provide transactional guarantees in a distributed system, while providing those guarantees in a single node is quite trivial by comparison. If you’re interested to know more about the design, please see mongodb.com/blog/post/multi-document-transactions – Vitamin

Since it's trivial to do multi-document transactions on a single node why doesn't MongoDB allow that? One of MongoDB's strengths has always been that it's very easy to get a basic system working - but now to use transactions (a core feature) requires extra steps to set up a replica set - and of course thus extra problems (e.g. rs.initiate() is putting localhost as its hostname in my docker container, and clients then can't connect to it after reading the list of members...) – Sukkah

@Sukkah because transactions are dependent on the features of the oplog. If you want to deploy a testing environment to test transaction, you can run a single-node replica set, and essentially it will behave like a standalone deployment with transaction. – Vitamin

@kevinadi, You said that creating a replica set with one node is not good for production. Do you have a solution that is good for a production environment as well? I've also posted a question for this issue here and after reading your answer I got confused and didn't accept the suggested solution there. – Pontianak

@LiranFriedman it depends on how much you value your data. If you don't really care about it (i.e. for caching) then you can do a single-node replica set in production. If you really value the data and must have it available and consistent (i.e. as a main database), then you should use at least a 3-node replica set. For the transaction design, MongoDB assumed that your data is valuable and thus need it to be consistent. However, your use case may differ and the data may not be valuable, so in specific cases a single node replica set may be ok. – Vitamin

@Vitamin Thank you for responding. The data is HIGHLY important, I have a collection of products and a collection with several different kinds of their inventory, the inventory update query touches 3 different collections so a transaction here sounds very recommended in case one of the queries fails. Do you have any suggestions on how to best approach this kind of issue? – Pontianak

@LiranFriedman I agree that transaction is likely the best method here. I would suggest you to use at least a 3-node replica set, all setup following the production notes. Note that for prod environment with highly valuable data, it's best if you have an experienced ops person, since it's not trivial to manage (even if it's hosted in EC2). Alternatively, you can see if Atlas is the right fit for you, since the ops are done for you. – Vitamin

@Vitamin Thank you for this awesome advice, I've talked to the MongoDB guys in my country and I'm starting to use Atlas. – Pontianak

I'm re-opening this thread because I wonder why setting your only one MongoDB instance in Replica is an issue in prod environment. @kevinadi, I would appreciate your feedback about it :) Thanks! – Paulson

@CyrilN. short answer is, a replica set's primary function is to provide high availability for your database (see Replication). Having only one node in a replica set does not provide this. Since high availability is the recommended practice in prod env, changestreams was designed with this prod env in mind. It wouldn't make sense to design changestreams based on a dev env (i.e. a standalone node), and have a separate design for prod. – Vitamin

Thank you @Vitamin for your response. In terms of performance, is there any differences between a replica set alone, and a standalone node? I'm asking because I'm using Mongo for a queueing system and need the transaction. – Paulson

@CyrilN. I think it's best if you open a new question for this case. The discussion thread is way too long already, and it won't be a short answer :) – Vitamin

@Vitamin you are right, I've posted a new question: https://mcmap.net/q/668013/-why-starting-a-solo-mongodb-instance-as-replica-set-is-not-recommended-in-production/330867 - Thanks! – Paulson

Answer 2

added this in mongod.conf file

replication:
    replSetName: set1

restarted mongod service with this command

sudo service mongod restart

went to mongosh

rs.initiate()

Done

Amplify