Deleted database accidentally by a bash script, rescue please

Question

My developer committed a huge mistake and we cannot find our mongo database anyone in the server. Rescue please!!!

He logged into the server, and saved the following shell under ~/crontab/mongod_back.sh:

And then he run ./mongod_back.sh, then there were lots of permission denied, then he did Ctrl+C. Then the server shut down automatically.

He tried to restart the server, then he got an grub error:

He then contacted AliCloud, the engineer connected the disk to another working server, so that he could check the disk. Then, he realized that some folders have gone, including /data/ where the mongodb is!!!

1) We just don't understand how the bash could destroy the disk including /data/;

2) And of course, is it possible to get the /data/ back?

PS: he did not take a snapshot of the disk before.

Twannatwattle

Sorry, it is about filesystem, not just database... – Twannatwattle

It is related to the shell. We don't understand how come the shell could delete those folders. Maybe there are other reasons? – Twannatwattle

Understanding how the shell deleted those folders will help us recover them. – Twannatwattle

"I don't believe the script you posted ( nor it's image, which you are not supposed to be posting here ) has anything to do with this either. " ==> What do you mean? You don't think it is the script that deleted the folders? If so, we could just leave the script here for a moment and see what others think about that. We try other sites and solutions at the same time. – Twannatwattle

@NeilLunn: You can delete a directory without mentioning it. – Circumscription

The answers show that it is indeed related to the bash. Remind me of this post. – Twannatwattle

@SoftTimur: If you want the data back, you're in the wrong place. If you want to know what is wrong with the code and how to do it better, then you are right here. – Namara

@Cyrus, there are related too, understanding how our database was destroyed helps us recover the data. – Twannatwattle

I think this also has to be said: someone who thinks // introduces a comment in bash has no business touching a production server. – Gratia

Answer 1

Question 1

1. We just don't understand how the bash could destroy the disk including /data/;

Reason: $OUT_DIR was unset

In bash and sh comments are written as # comment, not // comment.
The following line will have the following effects

someVariable=someValue // not a comment

• Assign someValue to the environment variable someVariable, but only for this command. After that command the variable will go back to its old value, which is null in this case.
• Execute the "command" // not a comment, that is the program // with the parameters not, a, and comment. Since // is just a directory (the same as /) this will cause an error message and nothing more.

Right now this behavior might seem strange, but you may have already used it in well known idioms like IFS= read -r line or LC_ALL=C sort.

Looking at your script the following lines probably caused the problem:

OUT_DIR=/data/backup/mongodb/tmp // ...
...
rm -rf $OUT_DIR/*

I'm sorry to bring this to you, but you basically executed rm -rf /* since $OUT_DIR expanded to the empty string.

Potential Risk On Other Systems

Even if $OUT_DIR wasn't empty the effect could have been the same since there is a // "comment" after rm. Consider the command

rm -rf some // thing

This is supposed to delete the three files/directories some, //, and thing. As already pointed out // is the same directory as /.

However, most implementations of rm on Linux have a guard for this case and won't delete / so easily. On Ubuntu you will get the following warning (don't try this at home. Would suck if your rm differs.)

$ rm -rf //
rm: it is dangerous to operate recursively on '//' (same as '/')
rm: use --no-preserve-root to override this failsafe

Question 2

2. And of cause, is it possible to get the /data/ back?

This is off-topic for StackOverflow. However, you can find many answers to this question on other stackexchange sites.

There are recovery tools you can try, but there is no guarantee that you can restore your data if you don't have a backup.

Aimeeaimil

i would also include set -u -e in the script -- this will bail as soon as an error occurred or when there is unassigned variable... and maybe use bash -n my_script.sh – Diphthong

if you use pipes and traps in bash, set -Eeuo pipefail is going to keep it safer – Ithunn

I tried to execute someVariable=someValue echo "someVariable=$someVariable" but it output someVariable= only. The answer said the variable assignment is only for that one line. If that is correct, why I am getting an empty value? – Jemima

@Jemima Because the variable is expanded before it is set. Bash processes commands in multiple steps, expanding variables ($var) is one of the first steps, assigning values var=... comes later -- If it was the other way around you couldn't write things like var="...$var...".¶ With var=test bash -c 'echo "$var"' you will get your expected output (note the single quotes). – Aimeeaimil

nitpick: there was another // in the arguments of rm, so it would have been a disaster even if $OUT_DIR had been set properly – Illyes

@Illyes Very good point! Thank you for sharing. I added your observation to this answer. – Aimeeaimil

Answer 2

Switching between languages can be tricky! // is not a comment starter in shell, it's #. All the commands with these "comments" were parsed wrongly and skipped:

$ VAR=whatever // comment
bash: //: Is a directory
[$?=126]
$ echo "($VAR)"
()

Therefore, OUT_DIR=... was ignored and $OUT_DIR was empty. It's easy to guess what

rm -rf $OUT_DIR/*

then did. It was basically equivalent to

rm -rf /*

Use your backups to restore the database.

Circumscription

And let's hope the backup scripts were written by somebody else. – Zurheide

Answer 3

I can read the Chinese wordings in comment field, from line 10, the user wants to create a temp folder but used cd, so if /data/backup/mongodb/tmp does not exist in the first place, then $OUT_DIR is empty or null, after that line 11 became rm -rf /*.

Tempietempla