haproxy - unable to load SSL private key from PEM file

Question

haproxy does not start anymore, it shows the error

bind <ip>:443' : unable to load SSL private key from PEM file ...

We did not change anything on the certificates or configuration. Since the last start we only made normal updates to the system.

To find the error, I generated a completely new certificate (self signed) but the error still exists.

This is the structure of the PEM file:

-----BEGIN CERTIFICATE-----
MIIDXjCCAkY...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKC....
-----END RSA PRIVATE KEY-----

I also tried to convert the private key with

openssl pkcs8 -topk8 -inform pem -in server.key -outform pem -nocrypt -out server_new.key

but haproxy still shows the same error.

I'm trying for hours now but I can not find the reason. Please help! Thank you!

Update:

The problem has something to do with file access. The PEM file was stored at /data/ssl/domainname/domainname.pem. File rights are ok. When I move the PEM file to /etc/haproxy then everything is ok.

Regazzi

If you use a RSA certificate the key needs to have at least 1024 bits, on ubuntu even more; found this in a note here access.redhat.com/solutions/6813181. i.e. openssl genrsa -out key 2048 does work – Corticosterone

Answer 1

The problem I was running into on CentOS was SELinux was getting in the way. To test if SELinux is the problem execute the following as root: setenforce 0, then try restarting the haproxy. If it works, there is an SELinux problem. (You can re-enable SELinux now and try to fix the underlying problem with the command setenforce 1).

Since I have the certificates in the folder /etc/haproxy/certificates, the following command worked to get the right permissions on the files restorecon -v -R /etc/haproxy (depending on your OS and SELinux config this may or may not work).

Crepe

Answer 2

The order in which the cert and key files appear in the pem is important. Use the following to create the pem file.

cat example.com.crt example.com.key > example.com.pem

Sporocarp

this is the order in my pem file as you can see in my question...but thanks – Regazzi

This answer solved my problem. Thank you with the same error! I forgot to concatenate files. – Ursas

This may have changed because I got it working with the private key coming before the public cert in the PEM file. I had been getting the same error, but in my case it was because I was running HAProxy in Docker but forget to add a volume to the container so HAProxy could see the PEM. – Fredericton

Thanks. I have the same issue while I am giving the server.pem file to haproxy – Pourparler

Answer 3

For me the problem was caused by this line in combined PEM file:

-----END CERTIFICATE----------BEGIN RSA PRIVATE KEY-----

After I split it I could start HaProxy and load it OK:

-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----

Dissimilation

Similar for me, had to break this: -----END CERTIFICATE----------BEGIN CERTIFICATE----- – Susann

Answer 4

For the latest version of letsencrypt certbot,fullchain.pem and privkey.pem files will be generated for you in /etc/letsencrypt/live/example.com folder. They need to be combined in order to HAProxy to read it properly.

cat fullchain.pem privkey.pem > example.com.pem

In HAProxy configuraion /etc/haproxy/haproxy.cfg

bind *:443 ssl crt /etc/letsencrypt/live/example.com/example.com.pem

Millpond

Answer 5

I also encountered this error. You might want to try to remove the passphrase from the private key before you begin ripping your hair out. It solved the problem for me. I think HAProxy is supposed to ask you for the password on restart, but it didn't in my case using 'sudo /etc/init.d/haproxy restart

To remove the password, try 'openssl rsa -in [PRIVATE_KEY_FILE] -out nopassphrase.key'

Is passphrase necesssary? There's a discussion in the link below. https://security.stackexchange.com/questions/70495/ssl-certificate-is-passphrase-necessary-and-how-does-apache-know-it

Valdemar

Answer 6

Did you append your certificate's private key to the end of the file?

HAProxy requires a "full chain" - certificate, intermediate authority (if you have one), and then private key. E.g.:

cat cert.pem cert.key > /haproxy/certs/fullchain.pem

Seato

Answer 7

The problem for me was a strange character at the beginning of the key.

This character did not show up when I cated the file because the character was <feff> otherwise known as the UTF-8 BOM (Byte Order Mark). It only showed up when I opened the file in vim.

I wouldn't expect this to be very common, but hopefully it saves someone some headache.

Algonkian

Answer 8

Just for information, in my case I had space character in front of "-----BEGIN RSA PRIVATE KEY-----" sequence and that broke the pem file.

Boykin

Answer 9

I'd like to add, for people which join here and have the same issue, that you have to keep your intermediate certificates in the chain as well... So if you have a chain with some layers, don't only take the rootca but also the intermediate certificates into your pem file

Refreshment

Answer 10

SElinux was the problem for me as well. HAProxy reported it could not read the file due to permissions even though the permissions matched other pem files in the folder. Our process is automated which is likely why SELinux is involved. The solution that seems to work for me so far (leaving SELinux running) is:

#!/bin/sh

if [ "$2" == "add" ]; then
  sudo touch /etc/haproxy/ssl/$1
  sudo cat $1 > /etc/haproxy/ssl/$1
  sudo chmod 644 /etc/haproxy/ssl/$1
fi
if [ "$2" == "delete" ]; then
  sudo rm /etc/haproxy/ssl/$1
fi

echo "performed $2 on $1";

Farmland