Undelete folders from AWS S3

Question

I have a S3 bucket with versioning enabled. It is possible to undelete files, but how can I undelete folders?

I know, S3 does not have folders... but how can I undelete common prefixes? Is there a possibility to undelete files recursively?

Floc

Answer 1

I found a satisfying solution here, which is described in more details here.

To sum up, there is no out-of-the-box tool for this, but a simple bash script wraps the AWS tool "s3api" to achieve the recursive undelete.

The solution worked for me. The only drawback I found is, that Amazon seems to throttle the restore operations after about 30.000 files.

Floc

Answer 2

I created this simple bash script to restore all the files in an S3 folder I deleted:

#!/bin/bash
    
recoverfiles=$(aws s3api list-object-versions --bucket MyBucketName  --prefix TheDeletedFolder/ --query "DeleteMarkers[?IsLatest && starts_with(LastModified,'yyyy-mm-dd')].{Key:Key,VersionId:VersionId}")
for row in  $(echo "${recoverfiles}" | jq -c '.[]'); do
    key=$(echo "${row}" | jq -r '.Key'  )
    versionId=$(echo "${row}" | jq -r '.VersionId'  )
    echo aws s3api delete-object --bucket MyBucketName --key $key --version-id $versionId
done

yyyy-mm-dd = the date the folder was deleted

Spectra

This script echos all the commands that you need to run. You can redirect the output of this script (>) to a text file and then run cat myTextFile.txt | bash to actually execute all the undelete commands – Keelia

You are saviour :) – Goodnatured

Answer 3

You cannot undelete a common prefix. You would need to undelete one object at a time. When an object appears, any associated folder will also reappear.

Undeleting can be accomplished in two ways:

• Delete the Delete Marker that will reverse the deletion, or
• Copy a previous version of the object to itself, which will make the newest version newer than the Delete Marker, so it will reappear. (I hope you understood that!)

Thuggee

Answer 4

If a folder and its contents are deleted you can recover them using the below script inspired by a previous answer

The script is applicable to an S3 bucket where versioning is enabeled before hand. It uses the delete marker tag to restore files in an S3 prefix.

#!/bin/bash
# Inspired by https://www.dmuth.org/how-to-undelete-files-in-amazon-s3/

# This script can be used to undelete objects from an S3 bucket.
# When run, it will print out a list of AWS commands to undelete files, which you
# can then pipe into Bash.
#

#
# You will need the AWS CLI tool from https://aws.amazon.com/cli/ in order to run this script.
#
# Note that you must have the following permissions via IAM:
#
# Bucket permissions:
#
#   s3:ListBucket
#   s3:ListBucketVersions
#
# File permissions:
#
#   s3:PutObject
#   s3:GetObject
#   s3:DeleteObject
#   s3:DeleteObjectVersion
#
# If you want to do this in a "quick and dirty manner", you could just grant s3:* to
# the account, but I don't really recommend that.
#
                
# profile = company
# bucket = company-s3-bucket
# prefix = directory1/directory2/directory3/lastdirectory/
# pattern = (.*)

# USAGE
# bash undelete.sh  > recover_files.txt  | bash

read -p "Enter your aws  profile: " PROFILE
read -p "Enter your S3 bucket name: " BUCKET
read -p "Enter your S3 directory/prefix to be recovered from, leave empty for to recover all of the S3 bucket: " PREFIX
read -p "Enter the file pattern looking to recover, leave empty for all: " PATTERN

# Make sure Profile and Bucket are entered
[[ -z "$PROFILE" ]] && { echo "Profile is empty" ; exit 1; }
[[ -z "$BUCKET" ]] && { echo "Bucket is empty" ; exit 1; }

# Fill PATTERN to match all if empty
PATTERN=${PATTERN:-(.*)}

# Errors are fatal
set -e


if [ "$PREFIX" = "" ]; 

# To recover all of the S3 bucket
then

    aws --profile ${PROFILE} --output text s3api list-object-versions --bucket ${BUCKET} \
            | grep -i $PATTERN \
            | grep -E "^DELETEMARKERS" \
            | awk -v PROFILE=$PROFILE -v BUCKET=$BUCKET -v PREFIX=$PREFIX  \
            -F "[\t]+" '{ print "aws --profile " PROFILE " s3api delete-object --bucket " BUCKET "--key \""$3"\" --version-id "$5";"}' 


# To recover a directory
else

    aws --profile ${PROFILE} --output text s3api list-object-versions --bucket ${BUCKET} --prefix ${PREFIX} \
            | grep -E $PATTERN \
            | grep -E "^DELETEMARKERS" \
            | awk -v PROFILE=$PROFILE -v BUCKET=$BUCKET -v PREFIX=$PREFIX  \
            -F "[\t]+" '{ print "aws --profile " PROFILE " s3api delete-object --bucket " BUCKET "--key \""$3"\" --version-id "$5";"}' 
fi

Clout

Answer 5

If you enabled the version for the bucket, Use the below command to retrieve the files or folder.

For folder with respective: -

echo '#!/bin/bash' > undeleteScript.sh && aws --output text s3api list-object-versions --bucket bucketname --prefix path/to/retrieve| grep -E "^DELETEMARKERS" | awk '{FS = "[\t]+"; print "aws s3api delete-object --bucket buckername --key \42"$3"\42 --version-id "$5";"}' >> undeleteScript.sh && . undeleteScript.sh; rm -f undeleteScript.sh;

For Files in the Bucket: -

echo '#!/bin/bash' > undeleteScript.sh && aws --output text s3api list-object-versions --bucket bucketname --prefix | grep -E "^DELETEMARKERS" | awk '{FS = "[\t]+"; print "aws s3api delete-object --bucket buckername --key \42"$3"\42 --version-id "$5";"}' >> undeleteScript.sh && . undeleteScript.sh; rm -f undeleteScript.sh;

Intendment

Answer 6

I wrote an optimize version to recover in batch of 1000 files:

#!/bin/bash

if [[ "$#" -lt 3 ]]; then
    echo "Run: $0 <BUCKET> <PREFIX> <PROFILE>"
    exit 1
fi

BUCKET=$1
PREFIX=$2
PROFILE=$3

echo "Restore bucket PREFIX - ${BUCKET}/${PREFIX}"

while true; do
  result="$(aws --profile ${PROFILE} s3api list-object-versions --max-items 1000 --bucket ${BUCKET} --prefix ${PREFIX} --query '{Objects: DeleteMarkers[0:999].{Key:Key,VersionId:VersionId}}')"
  
  if [ "$(echo $result|jq '.Objects')" == "null" ]; then
    echo "No more files to undelete."
    break
  fi
  
  echo "Restoring:"
  aws --profile ${PROFILE} --no-cli-pager s3api delete-objects --bucket ${BUCKET} --delete "$result" |grep Key
done

Titoism

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Taradiddle