curl: (60) SSL certificate problem: when uploading behind proxy

Question

I need to do curl uploading behind company proxy. and I've getting the following two type of problems depending on the site that I try,

• curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
• curl: (60) SSL certificate problem: unable to get local issuer certificate

Here are the details:

Case 1:

. . . 
< HTTP/1.1 200 Connection established
< Proxy-agent: CCProxy
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Case 2:

$ curl -vX POST -d "userId=5&title=Hello World&body=Post body." https://jsonplaceholder.typicode.com/posts
Note: Unnecessary use of -X or --request, POST is already inferred.
* Uses proxy env variable https_proxy == 'http://10.xx.xx.xx:808/'
*   Trying 10.xx.xx.xx:808...
* TCP_NODELAY set
* Connected to 10.xx.xx.xx port 808 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to jsonplaceholder.typicode.com:443
> CONNECT jsonplaceholder.typicode.com:443 HTTP/1.1
> Host: jsonplaceholder.typicode.com:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< Proxy-agent: CCProxy
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

The problem is not the above CCProxy, but our company is using the Zscaler transparent proxy which is intercepting SSL requests with its own certificate.

Is there any way to fix it pls?

$ curl --version
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1g zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.8.0 nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08

$ lsb_release -a 
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux bullseye/sid
Release:        testing
Codename:       bullseye

Enculturation

Answer 1

Step 1 in both options will extract the Zscaler certificates.

OPTION 1 Direct curl

1. Download the certificates (all certificates are included in a single file)
2. Execute the curl command passing the certificateS you want to use.

# 1
openssl s_client -showcerts \
-connect jsonplaceholder.typicode.com:443 </dev/null 2>/dev/null \
| sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p'  > typicode.crt
# 2
curl --cacert typicode.crt -v \
-d "userId=5&title=Hello World&body=Post body." \
 https://jsonplaceholder.typicode.com/posts

OPTION 2 (installer script)

In case the curl command is executed by an installer you don't have control, then, update your certificates:

1. Extract the certificates from server (use the FQDN or IP and PORT, i.e: jsonplaceholder.typicode.com:443)
2. Move the XXX.crt certificate to your certificates directory
3. Update certificates
4. Execute installation script

# 1
openssl s_client -showcerts \
-connect jsonplaceholder.typicode.com:443 </dev/null 2>/dev/null \
| sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p'  > typicode.crt
# 2
sudo mv typicode.crt /usr/local/share/ca-certificates/
# 3
sudo update-ca-certificates
# 4 execute your installer script

Bonus

In case you need/want to get the Zscaler certificates only, get the IP from: https://ip.zscaler.com

openssl s_client -showcerts -servername server -connect 165.225.216.33:443  </dev/null 2>/dev/null | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | grep -m1 -B-1 -- '-----END CERTIFICATE-----'  > zscaler.crt

UPDATED (11/19/21):

• Adding option 1, when is a direct curl and no need of install the certificates.
• Optimized the command for extracting the certificates (creating the file)
• Bonus: Getting the Zscaler IP

Tested on Ubuntu 20 and 18 behind Zscaler proxy.

Without certificate

With certificate

References:

• How to install certificates for command line
• unable to connect to server: x509: certificate signed by unknown authority

Yawn

Thanks for the answer, however the problem is that "our company is using the Zscaler transparent proxy which is intercepting SSL requests with its own certificate", which doesn't seems to have been addressed, as it's quite different from getting a normal site certificate. – Enculturation

I had same issue, my company is using Zscaler, I executed those steps and then, I was able to re-try an installation (snap) and went through without issues. Try: openssl s_client -showcerts -servername server -connect 165.225.216.33:443 > zscaler.pem Get the IP from ip.zscaler.com – Yawn

If you are installing from script, review what is the address used by the curl command, execute with verbose to get the IP and Port that is failing, i.e.: curl -vv https://get.helm.sh * Rebuilt URL to: https://get.helm.sh/ * Trying 152.195.19.97... * TCP_NODELAY set * Connected to get.helm.sh (152.195.19.97) port 443 (#0) – Yawn

Last option... TEMPORAL if you trust in the scripts you are using and REMOVE the file after executing the installation https://mcmap.net/q/18951/-curl-error-60-ssl-certificate-problem-certificate-has-expired – Yawn

Thanks Adrian. would you put your new comments (except the last one) into your answer to make it a whole and integrated answer please? I.e., in order to post to jsonplaceholder.typicode.com behind Zscaler, what are the necessary steps and their orders pls. thx. – Enculturation

Done :) I improved the commands and split it in two options – Yawn

Your grep command at the end returns "grep: -1: invalid context length argument" and I am currently working this to install the needed certs in Docker. Any ideas on what could be wrong with the grep portion? – Ardeth

@CameronKilgore, probably your certificate is in a single line, hence you may not need the grep part. Remove the grep part and review the resulting file to troubleshoot. – Yawn

Try: openssl s_client -showcerts -servername server -connect 165.225.216.33:443 > </dev/null 2>/dev/null | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | grep -m1 -B-1 -- '-----END CERTIFICATE-----' > zscaler.crt The answer was updated (community), and I think introduced an error. Test it, and let me know – Yawn

Answer 2

The answer is to "add that proxy's certificate to the CA bundle", thanks to Daniel Stenberg's answer. Then I guess I am suppose to fill in the rest. So here it is my attempt solving the remaining of the problems/questions --

• Q: What is the easiest way to get that Zscaler certificate?
  A: From here:

Go to Policy > SSL Inspection. In the Intermediate Root Certificate Authority for SSL Interception section, click Download Zscaler Root Certificate. Navigate to the ZscalerRootCerts. zip file and unzip it.

• Q: How to add that certificate to the CA bundle?
  A: See How to install company proxy certificate:

• You can use curl --cacert <CA certificate> to supply your company CA cert.
• Or you can add your company CA cert to /etc/pki/tls/certs/ and run make there to make it available system-wide.

Enculturation

Is there a way to make it globally on CentOS 7 ? running make doesn't make it system-wide – Utley

Answer 3

This error (SSL certificate problem) means that the CA store that curl uses to verify the server's peer did not contain the cert and therefore the server couldn't be verified.

If want curl to work with a transparent proxy that terminates TLS you must add that proxy's certificate to the CA bundle or completely ignore the certificate check (which I recommend against).

A transparent proxy for TLS will of course make the connection completely unreliable and have broken security properties.

Virulent

Answer 4

I exported RootCA cert as PEM and copied the file contents into the file I downloaded from https://curl.se/docs/caextract.html. And now CURL works for all requests—even those that end up being intercepted by ZScaler. I also set the SSL_CERT_FILE environment variable that file that contains both certs.

Cuneate