AWS S3 Disabling SSLv3 Support

Question

We received an email from AWS that basically says 'S3 is disabling SSLv3 Support, access will be cut-off in 15 days'. They then listed some buckets we have (one in production) that are 'currently accepting request from clients that specify SSLv3'. The full email is here, and other AWS users seem to have received one too:

https://gist.github.com/anonymous/4240c8af5208782c144c

My question is how do we test for this scenario, and what do we need to do to prepare for this cut-off date?

We use Rails 4.1 and the Fog (~> 1.28.0) and right_aws (~> 3.1.0) gems for AWS access and we're on Heroku. Our app provides signed HTTPS links to S3 resources to our browser users in our UI.

Is this just a client (browser) issue or something we need to understand better and test/fix?

Schall

My understanding is it's entirely a browser issue, and this essentially cuts off support for browsers less than IE7. I would love to be able to run some tests before they cut this off to production data though, hoping we can find more info out soon. – Midland

Thanks for the response. Yes, we suspect this is just about depreciated browsers, which makes sense. Our concern, like yours, is that we miss something and we really want to test out this scenario before any drop-dead date. So far AWS support aren't saying anything, so we'll have to try premium support if nothing comes out soon: forums.aws.amazon.com/thread.jspa?threadID=176062 – Schall

Answer 1

fog uses excon for its http(s) transport. excon is a low-level pure-ruby http client, which relies on the ruby openssl bindings to work. Though it is possible to explicitly set an ssl version to use, excon doesn't, which to the best of my knowledge should mean that it negotiates with the server to choose what to use (so if the server asks for not SSLv3, it should cooperate).

I believe that should mean no action would be required here, but the specifics of all that vary a bit across Ruby and OpenSSL versions (not to mention that it is just a bit hard to introspect/understand the specifics of those bindings), so it is hard to say for certain. excon does support an ssl_version argument, which can be used to force a specific version if it does end up being a problem (this is just not a good general choice because it disallows negotiation and the specifics vary between ruby versions).

Hope that helps.

Leahleahey

Thanks, that does help and gives us more to go on. We're on ruby 2.2.1p85 so I'm almost certain excon will negotiate ok up from SSLv3 both for fog and right_aws. I'll leave this open for a bit longer to add more info, but I think we'll pursue the path of (a) using wireshark etc to check our server to S3 calls are TLS and (b) right_aws (which we just use for S3 headers) is older and less maintained than fog, so just to reduce the number of moving parts we'll look at removing that and using something else. Thanks again. – Schall

We've been in contact with AWS support and they are setting up a S3 bucket for testing with SSLv3 access disabled. That way we'll soon be able to verify for sure that everything in excon/fog is working fine for sure. I'll come back and update this when we have that answer, as it might help others. – Schall

Great, thanks for the update. Definitely will be good to have that clarification. – Leahleahey

We've got a test IP from AWS with S3 and disabled SSLv3. We used a xyz.s3.amazonaws.com in our /etc/hosts to map to that test endpoint. It looks like there is an issue with excon/fog, in that we get this error trying to read a directory: "excon::Errors::SocketError: hostname "dev-test-xyz1.s3.amazonaws.com" does not match the server certificate (OpenSSL::SSL::SSLError)". Is it best to contact you via the excon or fog gem issues? Any help would be appreciated as this could be serious. – Schall

Ok, more info and some good news. Through the support of AWS (which was excellent btw) we figured out that we could get fog to connect once we used the :path_style => true setting. The bucket name had '-' (not dots), so maybe that was causing the 'does not match server cert' OpenSSL issue. Anyway, with that fog, via excon worked correctly. I'll raise an info issue on fog/fog as well just so other fog users don't get a shock on April 30th when AWS hit the switch on SSLv3. – Schall

So the only thing we don't know know is how to use our test IP with fog, as in to replace the s3.amazonaws.com part of the hostname for the AWS provider. We want to test a staging on heroku, where /etc/hosts is off limits (and all we have to the AWS test endpoint is an IP). – Schall

Cool, yeah, dashes in hostnames prevent ssl verification on S3, it is a known issue that fog (at least in some cases) will warn about. – Leahleahey

I'm not sure what the easiest way to test the other endpoint would be. You might be able to just try setting :host when making the connection, but I think that might not work with region stuff. It might take just hacking that in to a local client and trying it (since it is kind of an unusual use case that we didn't really plan for). Hope that helps and thanks again for helping out with this investigation and sharing your findings. – Leahleahey

This helps, but see my updated answers for the steps I have taken to resolve this. Basically, Excon should be updated to latest version. – Thigmotropism

Answer 2

Deadline has been moved:

Based on the feedback received we are extending the deadline for discontinuing support of SSLv3 for securing connections to S3 buckets to 12:00 AM PDT May 20, 2015.

Seer

Answer 3

Update May 7 2015, 11:26 AM IST

In carrierwave initializer, put things as following,

CarrierWave.configure do |config|
  config.fog_credentials = {
      :provider               => 'AWS',       # required
      :aws_access_key_id      => Settings.carrier_wave.amazon_s3.access_key,       # required
      :aws_secret_access_key  => Settings.carrier_wave.amazon_s3.secret_key,       # required
      :region                 => 'external-1'  # optional, defaults to 'us-east-1'
  }
  config.fog_directory  = Settings.carrier_wave.amazon_s3.bucket                    # required
  #config.fog_host       = 'http://aws.amazon.com/s3/'            # optional, defaults to nil
  config.fog_public     = false                                   # optional, defaults to true
  config.fog_authenticated_url_expiration = 600
  config.fog_attributes = {ssl_version: :TLSv1_2} #{'Cache-Control'=>'max-age=315576000'}  # optional, defaults to {}
end

This worked for me, and have a look at the wireshark trace log.

1577    22.611358000    192.168.0.113   8.8.8.8 DNS 87  Standard query 0xffd8  A s3-external-1.amazonaws.com
1578    22.611398000    192.168.0.113   8.8.8.8 DNS 87  Standard query 0xbf2f  AAAA s3-external-1.amazonaws.com
1580    22.731084000    8.8.8.8 192.168.0.113   DNS 103 Standard query response 0xffd8  A 54.231.1.234
1586    22.849595000    54.231.10.34    192.168.0.113   TLSv1.2 107 Encrypted Alert

1594    23.012866000    192.168.0.113   54.231.1.234    TLSv1.2 347 Client Hello
1607    23.310950000    192.168.0.113   54.231.1.234    TLSv1.2 204 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
1608    23.578966000    54.231.1.234    192.168.0.113   TLSv1.2 129 Change Cipher Spec, Encrypted Handshake Message
1609    23.579480000    192.168.0.113   54.231.1.234    TLSv1.2 427 Application Data
1610    23.868725000    54.231.1.234    192.168.0.113   TLSv1.2 299 Application Data

---

Update May 6 2015, 6-53 PM IST

Ok, After updating the Excon gem, we are able to see the TLSv1.2 protocol between our server and S3 servers.

bundle update excon

Wireshark trace log statements,

29  1.989230000 192.168.0.115   54.231.32.0 SSL 336 Client Hello
34  2.215461000 54.231.32.0 192.168.0.115   TLSv1.2 1494    Server Hello
40  2.219301000 54.231.32.0 192.168.0.115   TLSv1.2 471 Certificate
42  2.222127000 192.168.0.115   54.231.32.0 TLSv1.2 204 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message

---

UPDATE May 6, 2015, 4-29 PM IST

After updating the hosts file, following is the wireshark trace log.

14  2.012094000 192.168.0.115   54.231.32.0 SSLv3   192 Client Hello 
17  2.242423000 54.231.32.0 192.168.0.115   SSLv3   61  Alert (Level:  Fatal, Description: Handshake Failure)

---

Please see the above wireshark request capture, when I upload a file from my local development rails on S3. As it shows, on initial handshake Amazon server uses SSLv3 and so my rails server sends all future requests with SSLv3.

Now, the question is, How can I change the bucket settings so that it would accept/initiate the process using TLS only? I have checked in amazon settings, there is nothing like that.

I have already changed my nginx to use TLS, but I think that is not needed because Rails will talk to S3 in the background using Excon as mentioned in above comment.

So, Please suggest what could be the best possible way to test this before 20th May, to make sure that it will not break on that day.

Any help would be great.

Just for information - My bucket name is like xyz.abc.com, so no - in the name.

Thigmotropism

thank you for documenting your investigation. I would like to verify my updates using wireshark/tshark but I only have command line access. tshark -f 'tcp port 80 and http' isn't showing the TSL handshake. Do you have any insight into this? – Damaris

i installed wireshare on Ubuntu, and run it using sudo wireshark because it does not allow to trace requests until ran from root mode. And I didn't want to spend more time researching it, so use sudo mode. Please use wireshark in root mode, ask your Admin to do it for you for some days at least. – Thigmotropism

Answer 4

AWS's official FAQ https://forums.aws.amazon.com/thread.jspa?threadID=179904&tstart=0

54.231.32.0 s3.amazonaws.com
54.231.32.1 <bucket name>.s3.amazonaws.com
54.231.32.3 <bucket name>.s3-external-1.amazonaws.com

Configure the above in your /etc/hosts, replacing <bucket name> with your bucket name.

NOTE: when using with a non us-east-1 bucket you may get redirect and failure responses. This has more to do with their adhoc infrastructure for testing this than anything else. So ignore that.

Create a "standard US bucket" and test with that instead. Remember to configure your app to use s3 region external-1

FWIW, my app using paperclip (4.2.0) on ruby 2.1.4 works fine.

Lanilaniard

Yes, I found these Hosts configs, but when i do that with existing bucket it throws, Excon::Errors::SocketError (SSL_connect returned=1 errno=0 state=SSLv3 read server hello A: sslv3 alert handshake failure (OpenSSL::SSL::SSLError)) I use Carrierwave 0.6.2, and fog 1.6.2 – Thigmotropism

Ok, When I added the external-1 as a region in carrierwave config, it works. But when I track the requests through wireshark, it first tries to lookup DNS for s3-external-1.amazonaws.com and gets new IP address starting with 54.231.xx.xx. From this IP, the server hello handshake shows SSLv3. The file uploading and removal is working fine, but I doubt it is still using SSLv3. What can we do to stop it? – Thigmotropism

So, As per your comment. - If i use default region us-east-1, it is not working. - if i use exernal-1, its working but it is resolving diff. IP than the specified one. Please suggest. – Thigmotropism

Please see the above updated answer, it seems that Rails server uses SSLv3 for handshake. – Thigmotropism

as long as you're talking to the ip address they gave, you're good. fwiw, my packets show comms with 54.231.32.3 is in TLSv1.2 – Lanilaniard

Answer 5

It is a client-side issue entirely , if the protocol that the client (e.g the browser) uses to issue requests over https is SSLv3 , than the ssl handshake will not succeed and these requests will fail. So it's the client that needs to disable SSLv3.

AWS's action is a follow-up on the POODLE vulnerability uncovered last year , and since then also all the AWS CloudFront distributions that use the *.cloudfront.net domain name have been updated with discontinued SSLv3 support .Now AWS is moving on to S3 to do the same.

Unequal

Thanks for the response, that helps. As well as the browser clients, our server does communicate with S3 (via the fog and right_aws gems) then I guess it is these 'clients' that we wanted to understand and test. This then leads to needing more info on how to set that up before the cut-off date. – Schall

I know this is kind of OT because the question is about the Fog gem, but does it mean that if, let's say, we use FactoryGirl, it doesn't really matter as long as the users' browsers use TLS? – Ruby

Today the aws support sent me a list with the ip adress that are involved (only 7 based on a 3 days period during april) and indeed it was totaly a browser-side issue ! I think that all of us guys could have save a lot of time if Aws have been less dramatic and more precise in the first email notice we received ! – Doublefaced

And what about mobile applications that access S3 data? How can i test if they will run properly in Android and iOS apps? – Toyatoyama

Answer 6

I was able to force TLS using the following setting in my fog config:

connection_options: { ssl_version: :TLSv1_2 }

To test, update your host file (instructions from AWS):

54.231.32.0 s3.amazonaws.com
54.231.32.1 bucket.s3.amazonaws.com   #replace bucket with your bucket name
54.231.32.3 bucket.s3-external-1.amazonaws.com   #replace bucket with your bucket name

I was able to connect successfully. Also, If you change the setting to :SSLv3 you'll get an error. Good Luck!

Fagaly

How would you do that if you used Carrierwave? Part of the fog_credentials hash? – Schall

Yes, Same question as above. How to define in carrierwave, and is it needed? – Thigmotropism

Ok, I think I got it. Please see my updated answer for carrier_wave initializer configuration details. – Thigmotropism

You can also set it this way in the carrierwave initializer config.fog_credentials = { provider: "AWS", aws_access_key_id: 'blah' aws_secret_access_key: 'blah', path_style: true, connection_options: { ssl_version: :TLSv1_2 } } – Fagaly

@Fagaly @Paarth @Schall Can Some one please help me on this: I am using Amazon s3 with fog papaerclip. I have sepcified the connection_options in my initializer but still I am getting excon:: Error while creating data in Amazon s3. I am using Heroku server to run my rails application. Please help me its very need for me. – Cesium

Please see the details mentioned, you need to make sure that you do not use SSLV3. And check it by tracing the request as shown. – Thigmotropism