Improving Data Transfer Rate on Amazon EC2

Question

I've got a 1-Gig EBS Volume mounted to an EC2 instance, I am copying 600MB of binary data from a Local Hard Drive (via RDS Connection) and the copying process windows is showing 10 Hours remaining. Though I have a High Speed connection (100+Mbps)

Whatever the data volume, data transfer rate is 1 Min / MB (i.e 16Kbs / Sec)
I am hesitating between reading Moby Dick in front of my workstation or just taking a day-off.

Are there any reasonable options to speed up this transfer rate ?
(Ideally 512 Kbps/Sec at the minimum)

I am very open to ANY solution to shorten up the uploading/downloading time to/from and EC2 instance.

Thanks in advance.

EDIT :
I just stumbled upon [Amazon Export/Import Service][1] "AWS Import/Export accelerates transferring large amounts of data between the AWS cloud and portable storage devices that you mail to us"
By "mail to us", they literally mean you "materially" shipping your storage device to Amazon.
Don't say this is Stoneage, this is BRAND new TECHNOLOGY, Dude ! :-)

EDIT2 :
This sounded great: [Aspera for AWs][2] But unfortunately way too expensive;
Tailored for Fortune 500 with big needs and big cash.

Synchronic

Would S3 be an alternative ? I know S3 is meant for Web Access which is not my use case. But if it increases transfer rate and is accessible from a local hard drive why not. – Synchronic

I found this : forums.aws.amazon.com/message.jspa?messageID=183555. I am in Europe, but I had no other choices than "US East" when creating by EBS Volume. – Synchronic

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." —Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6. – Unrivaled

At least they're not asking you to transfer via snail... blog.makezine.com/2006/02/16/data-transfer-via-snail – Britisher

It's true, there is still nothing that beats tape for high volume. Starting to suspect that these transfer rates are intentional to pad the instance hours for "batch" users. Much slower than other providers. I would hate to think there is any slicing or multiplexing going on. (2.5 years later, and still slow). – Vizier

Answer 1

Sometimes, if you want something done it's better to do it yourself :-)

I did not find anything satisfying over the net so I spent the evening doing a rather Complete Bench Test of my own. I have tested a few alternatives /scenarios and here are the results :

FTP server installed on EC2 was Filezilla Server (correct configuration is touchy)
FTP Client used for this Bench test was WinSPC (Filezilla Client didn't work. see this other post
Legend : [HC] stands for Home Connection (100MBps)

Upload Bandwidth
RDS Upload : 15 Kb/S => Worse Ever
FTP Upload [FTP Server Installed on EC2]: 100 Kb/S
Upload to S3 from AWS Management Console from HC: 60 Kb/S
Upload to S3 using AWS Console interface from EC2 : 145 Kb/S
Upload to S3 using S3 Browser from HC: 120 Kb/S
Upload to S3 using S3 Browser from EC2 : 2000 Kb/S

Download Bandwidth
RDS Download and Upload: 15 Kb/S => Worse Ever
FTP Download [FTP Server Installed on EC2]: 360 Kb/S
Download from S3 AWS Console interface from EC2 : 350 Kb/S
Download from S3 using S3 Browser: from HC: 380 to 620 Kb/S
Download from S3 using S3 Browser: from EC2 : 3000 Kb/S

Conclusions :

So, as of now, Amazon S3 combined with S3 Browser give the best results. (S3 Browser is just a layer over S3 I don't get it why the upload rate is better)

However, one should keep in mind that an FTP Server on an EC2 instance has the great advantage of Mapping directly a local EC2 Directory into EC2. Unlike S3, there is only one transfer involved. Indeed, S3 requires 2 Transfers : Form Local Resource to S3/and from S3 to EC2 and the other way round, while FTP Access grants immediacy by shortening transfer cycles. Besides, it spares the cost of S3 Buckets.

It is also interesting to mention that EC2 Instance's bandwidth is really strong. It is hence far more interesting -needless to say, to use protocols that really take advantage of it such as S3 or FTP, rather than RDS.

I hope this will be useful to other people facing the same issue and spare them precious time.

Synchronic

Are you sure the rates are just 2000 Kb/s -> 2Mb/s 256 KB/s is just too low. There must be some sort of problem. The rates I'm getting over HTTPS are around 80 Mb/s (10 MB/s). – Holey

Answer 2

Use Aspera or Tsunami UDP and move 600 MB to your Amazon EC2 infra (jump box). Then internally copy from jumpbox to windows ec2. My benchmark shows tsunami UDP is quite faster compared to traditional modes.

Orthocephalic