python foursquare - SSL3 certificate verify failed

Question

I'm trying to make a userless request to the Foursquare API using Mike Lewis' Python wrapper - https://github.com/mLewisLogic/foursquare:

client = foursquare.Foursquare(client_id=Client_ID, client_secret=Client_Secret)
categs = client.venues.categories()

Intermittently, I get a "Error connecting to Foursquare API" msg. Running a logger to catch a more detailed message produces:

"SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"

This didn't use to happen and occurs both on my local Windows development machine and on a server running ubuntu. Am I missing something basic here about security certificates? The problem is intermittent and sometimes just leaving it a minute and retrying fixes the problem temporarily.

I've downloaded the latest 20120716 version of the wrapper although in the code for init.py it still says API_VERSION = '20120608'. I'm using Python 2.73 and have also signed up for the Foursquare API key, using the id and secret in the above code. I listed the urls, using my local IP:8000 and tried using separate keys for the local and dev machines but this seems to make no difference.

Help much appreciated. Thanks.

Corsica

Answer 1

I encountered the exact same problem yesterday. I searched a lot and tried a lot, and seem the following strategy works for my case (I'm not 100% sure but it's not reporting the error any longer).

1) Download http://curl.haxx.se/ca/cacert.pem

wget http://curl.haxx.se/ca/cacert.pem

2) Go to your Python httplib2 dir. Mine is at /usr/local/lib/python2.7/dist-packages/httplib2

cd /usr/local/lib/python2.7/dist-packages/httplib2

3) Back up the current certificate

cp cacerts.txt backup_cacerts.txt

4) And then copy the downloaded file there and rename it as cacerts.txt

mv cacert.pem cacerts.txt

That's it.

Hope it helps. If not recover from the backup file and try other methods.

My foursquare scripts work all right after I change this : )

Prem

This solution also works for the requests package by replacing the site-packages/requests/cacert.pem file. – Munniks

And also works for boto in site-packages/boto/cacerts/cacerts.txt – Ammadis

You are a lifesaver. Works inside virtual environment too. Make sure the downloaded cacerts.txt is located at the base of the httplib2 package folder. – Thissa

Is it just me or does downloading something like this over http seem rather strange to anybody else? – Barbarous

@mlissner: Not to mention that you are downloading it from a site with the word 'hax' in the URL! – Gilchrist

there seems to be no package named httplib2 under site-packages directory, why? and where i should put this pem file? – Eadwine

Answer 2

Install or upgrade the certifi package:

pip install --upgrade certifi

This is the solution recommended by urllib3 (which is used by many projects including requests, which in turn is used by the foursquare library linked above and many other libraries): http://urllib3.readthedocs.io/en/latest/user-guide.html#certificate-verification

If you need to use this for a packages which doesn't use urllib3, but httplib2 for example, you copy the certifi/cacerts.pem file to the httplib2/cacerts.txt. The directories to copy from/to can be found by doing:

python -c 'import httplib2; import os.path; print(os.path.dirname(httplib2.__file__))'

python -c 'import certifi; import os.path; print(os.path.dirname(certifi.__file__))' 

If you are just using urllib2, then the httplib2/ directory is not used, but a system location. You can check the files being loooked with the something like following:

strace python -c "import urllib2; urllib2.urlopen(urllib2.Request('https://google.com/'))" 2>&1 | grep 'open' | grep 'cert'

Turbofan