MongoDB dump from 3.2, restore with 3.4, error index safe = null

Question

I get the following error (dump MongoDB 3.2) (restore MongoDB 3.4):

Failed: ngivr-dev.ledgerhelpers: error creating indexes for ngivr-dev.ledgerhelpers: **createIndex error:** **The field 'safe' is not valid for an index specification.** Specification: **{ unique: true, name: "ledgerId_1", safe: null, ns: "ngivr-dev.ledgerhelpers", background: true, key: { ledgerId: 1 } }**

Looks like the safe index is null. But how can i use it with MongoDB 3.4? 3.2 is ok.

Sublett

Answer 1

safe=true is not an index specification.

In previous versions of MongoDB, lower than 3.4, extra indexes specifications can be added. Those were used by specific drivers.

In 3.4, mongodb added a validation on indexes specification:

Ensuring that the specified index options are valid. Previous versions ignored invalid options.

That's why you have this error. I am afraid you need to ensure that the index in your 3.2 version does not have invalid index specificaitons, and after that do the mongodump.

As kz_sergey says in his answer, you can mongorestore using --noIndexRestore, that should work fine.

Baritone

The --noIndexRestore option should be used in the mongorestore command, not the mongodump command. – Tadich

You're totally right : I'm using mLab to host my mongodb, updated versions when needed (from 2.x, to 3.2), I ignored that some of my old indexes (created in 2.x) was containing attribute safe: null. For me, it was this old specification that created the import issue. Recreated these indexes without it fixed the problem. Tanks ! – Nickelsen

I also got an error with index safe = null when going from 3.4 mmapv1 to wiredTiger – Thundery

Answer 2

Why do you restore indexes? --noIndexRestore and create them again.

Clinton

manually from scratch? it's going to demand solid portion of time – Estival

But restoring of collection without indexes is much faster. Maybe my way is faster at all, you can check this. – Clinton

of course it's faster, but app without indexes is not going to work assuming db has more or less decent size. this is what this fuss is all about – Estival

It's only faster initially-- then you have to add back the human and computer time to add back the indexes in additional step. – Backpedal

is there a use case where we need to optimise database restore performance? – Barytone

Answer 3

In the spirit of Aymeric's comment, you can use this awk one-liner to replace the "safe" property in your .metadata.json files.

awk -i inplace '{gsub(",\"safe\":null", ""); print}' *.metadata.json

Run it in the directory of your MongoDB export. This approach allows you to keep the indexes, but drop the "safe" option.

Annora

Answer 4

find . -type f -name "*.metadata.json" -exec sed -i 's/,"safe":null//g' {} \;

This works and you will keep your indexes! It find all files in the present location (.) then using the same process (exec) replace in file (sed -i) according to the following regex which is basically saying all occurrences of "safe":null with nothing.

Replace the "." argument with the path to the directory where your mongodb exports are stored.

Triquetrous

some explanation would be nice. – Kurbash

While this code may answer the question, providing additional context regarding how and why it solves the problem would improve the answer's long-term value. – Prostyle

This works and you will keep your indexes! What it does/means: find all files in the present location (.) then using the same process (exec) replace in file (sed -i) according to the following regex which is basically saying all occurrences of "safe":null with nothing. I know its not the perfect explanation of the regex! – Ibbetson

Answer 5

Since I wanted to keep all indexes, and none of the methods above worked in my case, I've just edited all *.metadata.json files and manually removed all occurrences of "safe":true.

Context: I'm using an old database which I cannot upgrade to 4.x or above - it was originally running on 3.2, but there's no version of mongo running on new Macs with Apple Silicon, so I had to use 3.4 (the first version available) and do a dump from the old Mac and a restore from the new Mac.

Cursive