Get site content over SSL with httr in R

Question

I'm trying to fetch a JSON array from my server using the HTTP POST method in R.

I've tried using both the POSTfunction from httrand the getURLfunction from RCurl but both return errors.

cafile <- system.file("CurlSSL", "cacert.pem", package = "RCurl")
url    <- "https://example.com/query/getData.php"

POST(url,body=NULL)
POST(url,body=NULL,config(cainfo=cafile))

getURL(url)
getURL(url,cainfo=cafile)

The error given by the POST function is (for both calls):

Error in curl::curl_fetch_memory(url, handle = handle) : 
  SSL peer certificate or SSH remote key was not OK

The error given by the getURL function is (without config(cainfo=cafile)):

* Hostname was NOT found in DNS cache
*   Trying 162.xxx.xxx.xxx...
* connect to 162.xxx.xxx.xxx port 443 failed: Connection refused
*   Trying 130.yyy.yyy.yyy...
* Connected to example.com (130.yyy.yyy.yyy) port 443 (#0)
* found 175 certificates in /etc/ssl/certs/ca-certificates.crt
* gnutls_handshake() warning: The server name sent was not recognized
* failed to get server cert
* Closing connection 0
Error in function (type, msg, asError = TRUE)  : 
  gnutls_handshake() warning: The server name sent was not recognized

I'm suspecting this has something to do with R since running:

curl 'https://example.com/query/getData.php'

from the command line return the expected result.

The server is a apache2 server with COMODO SSL certificate. In /etc/apache2/sites-enabled/000-default.conf the server name is set to

ServerName www.example.com  

Any help would be most apreciated

Mickelson

My guess is that R is not trusting the cert coming from example.com. To remedy this, you will need to add this cert to your trust store. – Hoyt

Shouldn't R trust the certificate if the certificate is trusted when browsing the site in a web-browser, I've also verified the certificate online? Is the trust store R specific? – Mickelson

Yes, I would expect this. Can you try manually adding the base 64 encoded cert to your cacert.pem file? Yes, this is a hack but it will let us see what is going on. – Hoyt

I've tried this but I'm getting the same error, (i don't have a .pem file but I have .ca-bundle, .crt, .p7b .csr and .key files and I've tried with all of them). I tried the same command with the verbose option set to true so I'll edit my question to include this. – Mickelson

Can it be that the certificate and the server returns different server names? – Mickelson

Your code implies that you have a file called cacert.pem. Is this not the case? This is the cause of your woes. R cannot verify the cert which example.com is sending back. – Hoyt

Let us continue this discussion in chat. – Mickelson

download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem") ... you need to get a truststore file. – Hoyt

Same error gnutls_handshake() warning: The server name sent was not recognized – Mickelson

removing the www part from ServerName resolved this, I'm not sure if it's in combination with updating cacert.pem. Thanks for all your help! – Mickelson

Please look into this and then answer your own question. – Hoyt

Answer 1

The httr package includes it's own CA bundle so this probably not the issue. More likely a server side SNI config problem or a problem with your certificate

Unfortunately you haven't posted a reproducible example with an actual URL. But with the latest version of the new openssl package you can easily debug your server cert:

library(openssl)
cert <- download_ssl_cert("www.r-project.org")
print(cert)
print(as.list(cert[[1]]))

Also try validating it

cert_verify(cert, ca_bundle())

This might give a hint on what's wrong with your certificate.

Specs

Thank you for your help, I didn't want to disclose the domain name of my server though. It seems like this was a problem with the server setup. – Mickelson

Answer 2

It seems like changing

ServerName www.example.com

To

ServerName example.com

fixed this issue. I tried this solution from another computer and I was able to use the httr POST function with this fix with the default httr CA bundle.

Mickelson