MongoDB upgrade from 3.6 to 4.0: How to fix the "collection does not have uuid in kvcatalog" exception?

Question

I run a MongoDB replica set in a containerization environment orchestrated by Rancher. After rolling out the latest update, MongoDB as been upgraded from 3.6.x to 4.0.x (latest).

No upgrade path has been followed and, as a result, when trying to bind the volume holding the database storage, the service fails to start, exiting with the following exception:

STORAGE  [initandlisten] exception in initAndListen: MustDowngrade: Collection does not have UUID in KVCatalog. Collection: admin.system.version, terminating

I get the same outcome even attempting to launch mongod with the --repair option. I have also tried to rollback the container to MongoDB 3.6.16, but the journal version is now incompatible.

We don't have snapshots of the volume, hence restoring the data is not an option (this is not a production environment). The only solution I've found online is not applicable, since it suggests to leverage mongodump and mongorestore, requiring a running database with the data storage attached.

I'm running out of ideas, any advice about how to fix this?

Montoya

Try to downgrade to 3.6.16 and restore as at this answer https://mcmap.net/q/243213/-mongodb-won-39-t-start-after-server-crash . – Obstetrics

@Obstetrics this was already attempted: > I get the same outcome even attempting to launch mongod with the --repair option. I have also tried to rollback the container to MongoDB 3.6.16, but the journal version is now incompatible. – Montoya

I attempted to use a Mongo 3.6 DB with a Mongo 4.0 server and got into the same situation, but in my case the journal was not modified by the newer server, so I managed to downgrade. How do you know your journal is now incompatible? – Considering

@MondKin is the exit error of the container, when trying to rollback to MongoDB 3.6.* – Montoya

Answer 1

A bug report on this issue recommended following the full upgrade path and that worked for me. I was able to go back to my old version when I ran into this error, and then upgrade properly.

Specifically: upgrade just one major version at a time and check the prerequisites and upgrade steps carefully. The most important part of this is checking your compatibility with db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) and make sure it matches the current version before upgrading to the next version. Update it with command like db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )

Read the full docs at pages like https://docs.mongodb.com/manual/release-notes/3.6/#upgrade-procedures https://docs.mongodb.com/manual/release-notes/4.0/#upgrade-procedures etc

Villosity

Answer 2

I was updating MongoDB from 3.6 to 4.2 with Homebrew on Mac OS, and ran into this similar situation.

Since formula mongodb was removed from homebrew-core, the team of mongodb maintained a custom Homebrew tap.

Following steps is my solution for upgrading MongoDB from 3.6 to 4.2 with Homebrew:

1. Uninstall old version mongodb from Homebrew

# If you run mongodb in background service, stop it first.
brew services stop mongodb
# If you have multiple version of mongodb, you might need to add `--force` option for deleting all of them at once.
brew uninstall mongodb

2. Install new mongodb from custom Homebrew tap

brew tap mongodb/brew
brew install [email protected]

3. Add [email protected] to your PATH

echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.bash_profile

4. Restart your terminal, and run mongod

5. Open another tab or terminal windows to run mongodump

6. Terminate mongod

7. Uninstall [email protected] and reinstall mongodb-community

brew uninstall [email protected]
brew install mongodb-community

8. Delete old data with old mongo data structure

rm -rf /data/db/*

9. Run mongod

10. Open another tab or terminal windows to run mongorestore dump/

Once mongorestore completed, mongod will be terminated. And you can run mongodb as a background service or just run mongod again. Open MongoDB with mongo or Robo 3T(formerly Robomongo), you will see your data is good as usual.

Hope it helps.

Sum

I think in step 10 the / is not needed as long you run mongorestore from the same directory mongodump was run. – Chervil

My bad. Put the / in the wrong place. It should be dump/. Thanks for the reminder. – Sum

Answer 3

Alright after about 3hours of trying to fix exactly what you are describing, here are the steps I took. Here I will show how to retain the data while downgrading to 3.6 and also how to update it 4.0 with the retained data.

The reason this happens for mac users is because there was an update for people using brew can no longer download Mongodb directly from brew. Then people tend to download the newer community edition instead from brew which then gets in conflict the old version of 3.6 db folder still present in the directory. Instead you would need to download it from the actual mongodb website:

https://www.mongodb.com/download-center/community

Here what you would want to do, is download everything manually. Follow these instructions here, but do this with the 3.6 version:

https://medium.com/@saurabhkumar_4718/install-mongodb-without-homebrew-on-mac-os-2a98b68ab09c

NOTE: when you copy .bash_profile please make sure export commands are not

Once you have successfully installed 3.6 manually you can mongodump to be safe, and or export all your mongo data into json with mongoexport for any data that you must really really save.

After that you can fresh install 4.0 manually again by following exactly the same steps as previously done for 3.6

This worked for me without having to do any 'mongorestore' as the 'db' folder was left alone and all my data was there, but maybe extra steps might be needed to restore the data. Worst comes to worst you have .jsons that needs to be rewritten. 'mongo --version' also showed the right version :) Hope it helps

Primulaceous

Answer 4

• Backing-up all the databases (using mongodump)
• Removing every file in /data/db, due to their incompatibility with the newer version: rm -rf /data/db/*
• Restarting the MongoDB service by mongod
• Restoring all the databases (using mongorestore)

https://dba.stackexchange.com/a/252393

Chou

Answer 5

For Docker:

I had this issue when I was upgrading a project that used Docker from 3.6 to 4.2 (and 4.4).

The way the project is set up means I don't have a physical /data/db directory to purge.

However, there is a volume defined that acts as the /data/db directory.

In this case all you need to do is backup any data you need and remove the associated volume.

If using vs code with the docker extension, you can simply open your docker tab in the sidebar, expand the volumes section, locate the volume in question (for me it was called mongo-4-2-upgrade-test_mongo-data which depends on how your various containers are named for your project), right-click and remove.

Ensure that the containers that use this volume have been properly shut down or you'll get a 409 conflict error. I did the same as above but found my mongo container (in the containers section of the docker tab) instead. then when I restarted my docker-compose file the new version of mongo was installed and ran as expected.

Rhona