what is the best way to download the contents of an AWS EBS volume?

Question

I have a number of large (100GB-400GB) files stored on various EBS volumes in AWS. I need to have local copies of these files for offline use. I am wary to attempt to scp such files down from AWS considering their size. I've considered cutting the files up into smaller pieces and reassembling them once they all successfully arrive. But I wonder if there is a better way. Any thoughts?

Furgeson

Answer 1

There are multiple ways, here are some:

1. Copy your files to S3 and download them from there. S3 has a lot more support in the backend for downloading files (It's handled by Amazon)

2. Use rsync instead of scp. rsync is a bit more reliable than scp and you can resume your downloads.

   rsync -azv remote-ec2-machine:/dir/iwant/to/copy /dir/where/iwant/to/put/the/files

3. Create a private torrent for your files. If your using Linux mktorrent is a good utility you can use: http://mktorrent.sourceforge.net/

Ultrasonics

Don't do 1. Because downloading data from S3 is chargeable. It will cost a lot given the huge data size. – Feltner

Answer 2

Here is one more option you can consider if you are wanting to transfer large amounts of data:

AWS Import/Export is a service that accelerates transferring data into and out of AWS using physical storage appliances, bypassing the Internet. AWS Import/Export Disk was originally the only service offered by AWS for data transfer by mail. Disk supports transfers data directly onto and off of storage devices you own using the Amazon high-speed internal network.

Basically from what I understand, you send amazon your HDD and they will copy the data onto it for you and send it back.

As far as I know this is only available in USA but it might have been expanded to other regions.

Lailalain

Why the down vote? Is this not a valid way? The OP did not mention the purpose they are doing this. For all I know they may just be wanting a 1 time backup and this is a valid solution to that. Sure its not a programming solution but it is still one way of solving the problem. – Lailalain