How do I respond to a "CONNECT" method request in a proxy server using socket in python?

Question

I am currently programming a proxy server using httplib, and when I try to connect to HTTPS websites (such as facebook and google) my client sends me "CONNECT" requests that look like this:

CONNECT www.google.co.il:443 HTTP/1.1\r\n
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0\r\n
Proxy-Connection: keep-alive\r\n
Connection: keep-alive\r\n
Host: www.google.co.il:443\r\n
\r\n

I took a working proxy from the internet and put it on, then sniffed the network on wireshark, and the response to this request should look this way:

HTTP/1.1 200 Connection established\n
Proxy-agent: Python Proxy/0.1.0 Draft 1\n
\n

I noticed that the client sends the request to the proxy itself, so I decided to use socket, and send the response to the client in this way:

if getmethod(clientreq) is "CONNECT":
    text="HTTP/1.1 200 Connection established\nProxy-Agent: THE BB Proxy\n\n"
    client.send(text)

I really hoped that handling those "CONNECT" requests would be the solution and that my server will finally take care of HTTPS requests but it doesn't, and the response packets that I send to the client don't even appear on wireshark.

So my questions are: 1. What does the "CONNECT" method really do? 2. What else do I need except handling "CONNECT" method requests in order to communicate with a HTTPS servers?

Exemplar

For the correct use of CONNECT see RFC2817 – Whomsoever

For the tl;dr version, CONNECT essentially establishes a tunnel between the client and origin server. You will need to use I/O multiplexing (for example using the select module, or something like Twisted) to implement this correctly. – Rajab

i did the same and found that the SSL handshake was indeed being attempted; but in my case the handshake stopped midway after the client hello, server hello, and server certificate (exchanging server key)... the clinet sent Client Key Exchange and Cipher Spec Change request but surprisingly that did not reach the proxy or the proxy ate up this message. and i am stuck there. if you had proceeded further and implemented HTTPS proxy, then would you mind sharing your experiences on how you overcame the problem you mentioned in the post? thanks, Samba – Lacefield

Answer 1

I am replying after this long time because I recently worked with this concept. It may help others.

To work with CONNECT http method proxy need to create socket connection with the server's https port (ex. 443). Once connection is established you can send "HTTP/1.1 200 Connection established" as response.

After this client and server with communicate with each other through proxy. Proxy has to just transfer data from client socket to server socket and vice versa. Client and server will exchange certificate information for handshaking, once handshaking is done they will start sharing data in encrypted format so proxy will not be able to understand anything.

May the following code helps you.

def _read_write(self):
    socs = [self.client, self.target]
    count = 0
    while 1:
        count += 1
        (recv, _, error) = select.select(socs, [], socs, 3)
        if error:
            break
        if recv:
            for in_ in recv:
                data = in_.recv(BUFLEN)
                if in_ is self.client:
                    out = self.target
                else:
                    out = self.client
                if data:
                    out.send(data)
                    print(data)
                    count = 0
        if count == time_out_max:
            break

Hope this answer helps anyone in need. As I had to go through a lot of things to find this answer.

Vibes

Your problem seems similar to mine: would you help me with my question? #36355136 – Monogamy

Answer 2

I met basically the same problem and the way I finally solved this is to look up for sample code on GitHub. It turns out that the proxy2 project is quite helpful. Some relevant code that is pretty similar to rushikesh's answer:

    def connect_relay(self):
        address = self.path.split(':', 1)
        address[1] = int(address[1]) or 443
        try:
            s = socket.create_connection(address, timeout=self.timeout)
        except Exception as e:
            self.send_error(502)
            return
        self.send_response(200, 'Connection Established')
        self.end_headers()

        conns = [self.connection, s]
        self.close_connection = 0
        while not self.close_connection:
            rlist, wlist, xlist = select.select(conns, [], conns, self.timeout)
            if xlist or not rlist:
                break
            for r in rlist:
                other = conns[1] if r is conns[0] else conns[0]
                data = r.recv(8192)
                if not data:
                    self.close_connection = 1
                    break
                other.sendall(data)

You can find more information in the repo.

Clyburn