Force cURL to use GET on proxy for HTTPs requests

Question

I'm trying to use a forward proxy server (Apache Traffic Server or Squid) on my local machine, as a local HTTP cache for my cURL calls.

I've set up the proxy using:

curl_setopt($ch, CURLOPT_PROXY, 'http://localhost:8080');

When I query an HTTP website, cURL performs a standard HTTP GET proxy request, which can be cached properly:

GET http://example.com/ HTTP/1.1

However, when querying an HTTPs website, cURL performs a CONNECT instead, effectively using the proxy as a TCP tunnel, and preventing it from caching the response:

CONNECT example.com:80 HTTP/1.1

Is there a way to force cURL to perform a GET request even for HTTPs websites?

I can understand the rationale behind using a TCP tunnel for HTTPs requests over an HTTP proxy for security, but because my proxy server is on localhost, I don't care using an insecure HTTP connection to the proxy, and would like cURL to perform a GET request:

GET https://example.com/ HTTP/1.1

I tried using:

curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, false);

But this didn't change anything.

Aliaalias

no good way around it AFAIK; you need a fake SSL certificate, and an intelligent intercepting (SOCKS?) proxy - that replace the target website's ssl with it's own - an example is the Fiddler proxy at telerik.com/fiddler – Loving

Could you try using curl via command line? If you are using SOCKS proxy: curl --socks5 127.0.0.1:8889 https://www.example.com/ -v If you are using HTTP proxy: curl -x http://PROXY_HOST:PROXY_PORT https://www.example.com/ -v – Lydon

and did you check curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); option? ;) – Lydon

@MooYeolPrescottLee setting CURLOPT_SSL_VERIFYPEER to false just prevents cURL from verifying the validity of the SSL certificate, which is unrelated. – Aliaalias

Answer 1

I don't believe this can be done using some common configuration setting on the client side as it would deny the whole purpose of HTTPS (that no one can eavesdrop your HTTPS traffic).

Your only option is thus to configure your proxy to basically create a man-in-the-middle attack to decrypt the HTTPS traffic going through it. Squid proxy should support this using their "SSL bump" feature. There is a nice intro for it on this wiki and more setup docs here.

What squid does in this regime is that it gets the address of the remote server from the client's CONNECT request and instead of creating just a blind tunnel to the server, it starts a new direct HTTPS request to the server by itself and saves the reply. Thus Squid has access to all the traffic and can cache it or do anything else Squid can do with it.

When sending replies back to the client, it itself needs to present a HTTPS certificate (the client expects HTTPS traffic), so in Squid there is a feature to automatically generate certificates for all proxied domains. To configure it you will essentially have to create a local Certificate Authority. Note that these auto-generated certificates will be simple self-signed certificates so, on the client side, this will look as untrusted certificates and you'll need to switch off peer verification (CURLOPT_SSL_VERIFYPEER = false).

I could not find any similar feature in Apache traffic server. They seem to support only SSL termination in the reverse proxy mode.

Last note: please bear in mind that this is still rather a hack and decrypting HTTPS may bring legal or ethical problems. Never do this without the clients consent!

Geof

Thanks for this answer. There's no problem in decrypting HTTPS, as it all happens as local traffic on the same machine, apart from the complexity and performance issues. I'm surprised that cURL does not allow to override this behaviour (for proxies on localhost), as on a technical side there's absolutely nothing preventing what I want to achieve. – Aliaalias

#15156 – Catalan

Thanks, and sorry :). To my knowledge this really is a special mode of operation that has to be supported on the proxy side and cannot be just "switched on" on the client side. – Daiseydaisi

Answer 2

I don't think the other answers here understand what you want to do. But its possible.

You want to make an https request and have it complete like this:

client <--http--> cache <--https--> remote server

So you send the https request insecurely over http on your local network, to a local cache, then have the cache fetch it securely as https across the open internet.

To do this you just have to hack the first hop. Your client program makes a plain http request to the cache, but adds a header that says to convert to https on the next hop like:

x-use-protocol: https

Invent whatever header you like. For this to work both the client and the cache have to understand this header to make the conversion happen. This is not good for general web browsing - or any time that you can't control the client. But is a good answer if you're writing both the client and the cache.

Cruzeiro

You undestood my problem correctly. I'm not writing the cache, though. I'm using a stock software, currently Apache Traffic Server. The real problem is cURL not allowing to force a GET request instead of a CONNECT request on https URLs. – Aliaalias