Basic Problem:
We've been experiencing very strange behaviour in our current infrastructure setup:
- file download speed from Amazon S3 drops to <2 kb/s (after ~10 downloads that have perfectly normal download speed) if the file is downloaded from the same IP/machine it has been uploaded from
- on our other machines we can download the file a couple of thousand times and don't see this behaviour
Additional details:
- the machines are setup identical using puppet
- they are all virtual machines running ubuntu 12.04.4 on KVM with libvirtd on ubuntu 12.04.4 and 13.04 hosts
- each VM has it's own public IP from which the traffic is originating
- after a couple of minutes-hours it's possible to download the file again with >5 mb/s for a couple of times (seems to be 10 times)
- files are uploaded from rails applications using the fog gem
Tests with wget:
Using wget, you see this output on the affected machines for a file we uploaded:
--2014-07-31 16:33:38-- http://s3-eu-west-1.amazonaws.com/not_the_real_file_url
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 178.236.6.160
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|178.236.6.160|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2801149 (2.7M) [text/plain]
Saving to: `/dev/null'
0% [ ] 10,111 1.05K/s eta 68m 26s
and it stays like this for 68m! (it does finish the download after that time though)
And this output for a random file hosted on amazon s3 by somebody else:
--2014-07-31 16:39:21-- https://s3.amazonaws.com/Minecraft.Download/versions/14w31a/minecraft_server.14w31a.jar
Resolving s3.amazonaws.com (s3.amazonaws.com)... 72.21.211.199
Connecting to s3.amazonaws.com (s3.amazonaws.com)|72.21.211.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10342238 (9.9M) [application/octet-stream]
Saving to: `/dev/null'
32% [====================================> ] 3,370,945 747K/s eta 12s
Our current workaround
Our current solution, is to use our HAProxy as a transparent HTTP proxy.
Meaning we have a frontend "cloud.example.com" defined, and a backend that first replaces the requests HOST with "s3-eu-west-1.amazonaws.com" and then uses s3-eu-west-1.amazonaws.com:80 as a server. For amazon it then looks like the request is coming from our proxy and we can download the files we stored on S3 thousands of times again. :)
[2014-07-31 16:56:57 +0200] RUN[28] AVG: '0.9612743812142854' s, LAST_RUN: '0.711118431' s
--2014-07-31 16:56:57-- https://cloud.example.com/not_the_real_file_url
Resolving cloud.example.com (cloud.example.com)... 1.2.3.4
Connecting to cloud.example.com (cloud.example.com)|1.2.3.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2801149 (2.7M) [text/plain]
Saving to: `/dev/null'
100%[====================>] 2,801,149 2.47M/s in 1.1s