HTTPS proxy tunneling with the ssl module

Asked 8/12, 2010 at 22:37 Answered 10/4, 2019 at 3:27

I'd like to manually (using the socket and ssl modules) make an HTTPS request through a proxy which itself uses HTTPS.

I can perform the initial CONNECT exchange just fine:

import ssl, socket

PROXY_ADDR = ("proxy-addr", 443)
CONNECT = "CONNECT example.com:443 HTTP/1.1\r\n\r\n"

sock = socket.create_connection(PROXY_ADDR)
sock = ssl.wrap_socket(sock)
sock.sendall(CONNECT)
s = ""
while s[-4:] != "\r\n\r\n":
    s += sock.recv(1)
print repr(s)

The above code prints HTTP/1.1 200 Connection established plus some headers, which is what I expect. So now I should be ready to make the request, e.g.

sock.sendall("GET / HTTP/1.1\r\n\r\n")

but the above code returns

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</body></html>

This makes sense too, since I still need to do an SSL handshake with the example.com server to which I'm tunneling. However, if instead of immediately sending the GET request I say

sock = ssl.wrap_socket(sock)

to do the handshake with the remote server, then I get an exception:

Traceback (most recent call last):
  File "so_test.py", line 18, in <module>
    ssl.wrap_socket(sock)
  File "/usr/lib/python2.6/ssl.py", line 350, in wrap_socket
    suppress_ragged_eofs=suppress_ragged_eofs)
  File "/usr/lib/python2.6/ssl.py", line 118, in __init__
    self.do_handshake()
  File "/usr/lib/python2.6/ssl.py", line 293, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [Errno 1] _ssl.c:480: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

So how can I do the SSL handshake with the remote example.com server?

EDIT: I'm pretty sure that no additional data is available before my second call to wrap_socket because calling sock.recv(1) blocks indefinitely.

Abohm answered 8/12, 2010 at 22:37 Comment(2)

my rough guess is that ssl.wrap_socket cares for socket connection state. usually you'd create socket, then wrap it, then connect. Here you create socket, connect, then wrap. perhaps ssl is just confused by already-connected underlying socket state. github.com/kennethreitz/requests/blob/… – Coal 25/11, 2013 at 15:55

hey, did you have any luck? I am stuck with the same problem, but also did not find anything... – Granado 25/11, 2013 at 21:45

This should work if the CONNECT string is rewritten as follows:

CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)

Not sure why this works, but maybe it has something to do with the proxy I'm using. Here's an example code:

from OpenSSL import SSL
import socket

def verify_cb(conn, cert, errun, depth, ok):
        return True

server = 'mail.google.com'
port = 443
PROXY_ADDR = ("proxy.example.com", 3128)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY_ADDR)
s.send(CONNECT)
print s.recv(4096)      

ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
ss = SSL.Connection(ctx, s)

ss.set_connect_state()
ss.do_handshake()
cert = ss.get_peer_certificate()
print cert.get_subject()
ss.shutdown()
ss.close()

Note how the socket is first opened and then open socket placed in SSL context. Then I manually initialize SSL handshake. And output:

HTTP/1.1 200 Connection established

<X509Name object '/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com'>

It's based on pyOpenSSL because I needed to fetch invalid certificates too and Python built-in ssl module will always try to verify the certificate if it's received.

Lamberto answered 16/3, 2012 at 13:46 Comment(5)

Does this work for you even if you're connecting to an HTTPS proxy? In your example, you're connecting to a regular proxy, which also works for me. It's when I need to double-wrap the socket that it fails. – Abohm 16/3, 2012 at 19:10

Good answer, but why is it not possible to use ssl.wrap_socket? – Coal 25/11, 2013 at 15:53

This does not work as well for the case of HTTPS-over-HTTPS and results in the same error – Granado 25/11, 2013 at 22:3

I get Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')] and no binary rubbish in ssl depth-1 output. I suspect that instead of wrapping data in SSL twice, OpenSSL reuses the underlying socket/fd and only wraps data once. – Coal 2/12, 2013 at 14:2

You are not using a HTTPS proxy I think, it's just a HTTPS in HTTP proxy example – Krystenkrystin 22/3, 2016 at 9:35

Judging from the API of the OpenSSL and GnuTLS library, stacking a SSLSocket onto a SSLSocket is actually not straightforwardly possible as they provide special read/write functions to implement the encryption, which they are not able to use themselves when wrapping a pre-existing SSLSocket.

The error is therefore caused by the inner SSLSocket directly reading from the system socket and not from the outer SSLSocket. This ends in sending data not belonging to the outer SSL session, which ends badly and for sure never returns a valid ServerHello.

Concluding from that, I would say there is no simple way to implement what you (and actually myself) would like to accomplish.

Granado answered 26/11, 2013 at 20:31 Comment(5)

Sounds like a sound explanation NPI. Do you perhaps know an alternative? – Coal 27/11, 2013 at 13:8

Sadly not, if you have any idea, I am all ears – Granado 27/11, 2013 at 17:18

I got somewhere looping data back through socket.socketpair ;-) – Coal 2/12, 2013 at 15:1

@qarma so you read it from the SSLSocket, write it into the socketpair and then read again from the second SSLSocket on the other end of the socketpair?! – Granado 2/12, 2013 at 19:5

Yep that's basically it. Meanwhile I discovered that twisted package appears to support SSL-in-SSL via custom BIO in its SSL/TLS module, but that's a lot of dependencies. – Coal 3/12, 2013 at 9:52

Finally I got somewhere expanding on @kravietz and @02strich answers.

Here's the code

import threading
import select
import socket
import ssl

server = 'mail.google.com'
port = 443
PROXY = ("localhost", 4433)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)


class ForwardedSocket(threading.Thread):
    def __init__(self, s, **kwargs):
        threading.Thread.__init__(self)
        self.dest = s
        self.oursraw, self.theirsraw = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
        self.theirs = socket.socket(_sock=self.theirsraw)
        self.start()
        self.ours = ssl.wrap_socket(socket.socket(_sock=self.oursraw), **kwargs)

    def run(self):
        rl, wl, xl = select.select([self.dest, self.theirs], [], [], 1)
        print rl, wl, xl
        # FIXME write may block
        if self.theirs in rl:
            self.dest.send(self.theirs.recv(4096))
        if self.dest in rl:
            self.theirs.send(self.dest.recv(4096))

    def recv(self, *args):
        return self.ours.recv(*args)

    def send(self, *args):
        return self.outs.recv(*args)


def test():
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect(PROXY)
    s = ssl.wrap_socket(s, ciphers="ALL:aNULL:eNULL")
    s.send(CONNECT)
    resp = s.read(4096)
    print (resp, )

    fs = ForwardedSocket(s, ciphers="ALL:aNULL:eNULL")
    fs.send("foobar")

Don't mind custom cihpers=, that only because I didn't want to deal with certificates.

And there's depth-1 ssl output, showing CONNECT, my response to it ssagd and depth-2 ssl negotiation and binary rubbish:

[dima@bmg ~]$ openssl s_server  -nocert -cipher "ALL:aNULL:eNULL"
Using default temp DH parameters
Using default temp ECDH parameters
ACCEPT
-----BEGIN SSL SESSION PARAMETERS-----
MHUCAQECAgMDBALAGQQgmn6XfJt8ru+edj6BXljltJf43Sz6AmacYM/dSmrhgl4E
MOztEauhPoixCwS84DL29MD/OxuxuvG5tnkN59ikoqtfrnCKsk8Y9JtUU9zuaDFV
ZaEGAgRSnJ81ogQCAgEspAYEBAEAAAA=
-----END SSL SESSION PARAMETERS-----
Shared ciphers: [snipped]
CIPHER is AECDH-AES256-SHA
Secure Renegotiation IS supported
CONNECT mail.google.com:443 HTTP/1.0
Connection: close

sagq
�u\�0�,�(�$��
�"�!��kj98���� �m:��2�.�*�&���=5�����
��/�+�'�#��     ����g@32��ED���l4�F�1�-�)�%���</�A������
                                                        ��      ������
                                                                      �;��A��q�J&O��y�l

Coal answered 2/12, 2013 at 15:1 Comment(0)

It doesn't sound like there's anything wrong with what you're doing; it's certainly possible to call wrap_socket() on an existing SSLSocket.

The 'unknown protocol' error can occur (amongst other reasons) if there's extra data waiting to be read on the socket at the point you call wrap_socket(), for instance an extra \r\n or an HTTP error (due to a missing cert on the server end, for instance). Are you certain you've read everything available at that point?

If you can force the first SSL channel to use a "plain" RSA cipher (i.e. non-Diffie-Hellman) then you may be able to use Wireshark to decrypt the stream to see what's going on.

Drill answered 8/12, 2010 at 23:15 Comment(3)

I'm pretty sure that there's nothing available on the socket, because if I call sock.recv(1) then it blocks indefinitely. However, thanks for confirming that I can double-wrap a socket. I can't change the server's SSL settings, though I appreciate the Wireshark suggestion - please let me know if you have any other ideas. – Abohm 8/12, 2010 at 23:26

Do what SimonJ says. 1) SSL sockets work differently then regular sockets. Even if there is raw received SSL data none of it will be returned until a complete and valid SSL record is received. 2) You do not need to change anything at the server to force RSA, simply modify the client ciphersuites to exclude any that use diffie-hellman. Of course you'll also need to get the server's private key to decrypt so if you can't get that then all you can see is cipher. Wireshark gives you ground truth: give it a try. – Laceylach 9/12, 2010 at 12:38

Can the client connect directly to the server using SSL? It may be that your network topology doesn't allow this, but it'd be good to confirm there's not some protocol-level mismatch (SSL version or cipher suite incompatibility) that's preventing the endpoints from communicating. – Drill 11/12, 2010 at 14:6

Building on @kravietz answer. Here is a version that works in Python3 through a Squid proxy:

from OpenSSL import SSL
import socket

def verify_cb(conn, cert, errun, depth, ok):
        return True

server = 'mail.google.com'
port = 443
PROXY_ADDR = ("<proxy_server>", 3128)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY_ADDR)
s.send(str.encode(CONNECT))
s.recv(4096)

ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
ss = SSL.Connection(ctx, s)

ss.set_connect_state()
ss.do_handshake()
cert = ss.get_peer_certificate()
print(cert.get_subject())
ss.shutdown()
ss.close()

This works in Python 2 also.

Imposing answered 10/4, 2019 at 3:27 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags