How to install libcurl with nss backend in aws ec2? (Python 3.6 64bit Amazon Linux)

Question

I have an ec2 instance in AWS running Python3.6 (Amazon Linux/2.8.3) where I need to install pycurl with NSS ssl backend.

First I tried it by adding pycurl==7.43.0 --global-option="--with-nss" to my requirement.txt file but I was getting errors installation errors. So I ended up doing it by adding a .config file in .ebextensions (that runs during deployment):

container_commands:
  09_pycurl_reinstall:
    # the upgrade option is because it will run after PIP installs the requirements.txt file.
    # and it needs to be done with the virtual-env activated
    command: 'source /opt/python/run/venv/bin/activate && PYCURL_SSL_LIBRARY=nss pip3 install pycurl --global-option="--with-nss" --upgrade --no-cache-dir --compile --ignore-installed'

Pycurl seems to be correctly installed, but the celery worker is not running. The celery worker logs show:

__main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory

If I ssh connect to the instance and run python 3.6 -c 'import pycurl' I get a more detailed error:

ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (nss)

So I guess that my problem is that I had previously installed libcurl with openSSL instead of NSS, and hence the mismatch between libcurl and pycurl.

According to another stackoverflow question, for libcurl to be installed with NSS backend I should have installed it with:

sudo apt libcurl4-nss-dev

But since the server is running Amazon Linux I can't use the apt command. So I did instead:

yum install libcurl-devel

And I guess this is the problem: this installs libcurl with OpenSSL support when I need it with NSS support.

How can I install libcurl with NSS in Amazon Linux?? (I need NSS because I'm running a Django app with celery using SQS as the broker, and SQS requires NSS).

Thank you very much!

Kwok

I have pretty much the same stack as you (saw it from other questions): Django / SQS / Celery / Beanstalk. I got it running and had also the issues with pycurl, but I used openssl. I don't think that SQS needs nss, at least for me openssl works fine with Python 3.6 on Amazon Linux 2.8.6. – Pains

Yes, I ended up using openssl and it worked fine. I don't know why I thought SQS needed nss. Thanks anyway! – Kwok

Answer 1

I just ran into the same issue and did manage to fix it :)

1. Retrieve Amazon Linux configure options for libcurl:

curl-config --configure

All options referring to paths can be ignored, but other must be kept if you want the same features than system libcurl. Of course, --with-ssl will be replaced with --without-ssl --with-nss.

1.1 Install prerequisites:

sudo yum install libssh2-devel nss-devel

(of course you should rather add then to the packages > yum section of your ebextensions)

2. Compile libcurl from source (i chose 7.61.1 to match the one used by Amazon Linux 2018.03):

wget https://curl.haxx.se/download/curl-7.61.1.tar.gz
tar xf curl-7.61.1.tar.gz
cd curl-7.61.1
./configure '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-amazon-linux-gnu' '--program-prefix=' '--cache-file=../config.cache' '--disable-static' '--enable-symbol-hiding' '--enable-ipv6' '--enable-threaded-resolver' '--with-gssapi' '--with-nghttp2' '--without-ssl' '--with-nss' '--with-ca-bundle=/etc/pki/tls/certs/ca-bundle.crt' '--enable-ldap' '--enable-ldaps' '--enable-manual' '--with-libidn2' '--with-libpsl' '--with-libssh2' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-amazon-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
make
sudo make install

(of course this should be deployed properly as a Shell script through the files section of your ebextensions).

Now you should see libcurl.so.4 created in /usr/local/lib.

3. Define the following envvars for pycurl to be compiled using your custom libcurl:

LD_LIBRARY_PATH=/usr/local/lib
PYCURL_CURL_CONFIG=/usr/local/bin/curl-config
PYCURL_SSL_LIBRARY=nss

4. Run your pip install

You can check pycurl linked to the right libcurl:

ldd /opt/python/run/venv/local/lib64/python3.6/site-packages/pycurl.cpython-36m-x86_64-linux-gnu.so

should show you libcurl.so.4 => /usr/local/lib64/libcurl.so.4

And of course python 3.6 -c 'import pycurl' should work.

That's it! You should be able to run Celery with SQS.

Shown

Answer 2

I had loads of issues getting pycurl installed on my Python ec2 instance. First up, pip wouldn't install pycurl at all. I had to ssh and run

yum install libcurl-devel

After this, I added the following line to my requirements.txt file so that it included the relevant ssl option (--with-openssl) as an install option:

pycurl==7.43.0.5 --install-option="--with-openssl"

Once I deployed a new version with this change in requirements.txt, pycurl worked fine, which I tested with:

python 'import pycurl'

I knew I needed to use openssl from the error message

libcurl link-time ssl backend (openssl)

Parricide