Throttling bandwidth of an SSL connection

Question

I asked a question about how to throttle a python upload, which sent me to this answer, where I was informed of a little helper library called socket-throttle. That's all fine and dandy for regular HTTP and probably also for most plain uses of the socket. However, I'm trying to throttle an SSL connection, and trying to combine socket-throttle with the stock SSL library (used implicitly by requests) causes an exception deep in the guts of the library:

  File "***.py", line 590, in request
    r = self.session.get(url, headers=extra_headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 394, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 324, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 478, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 95, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util.py", line 643, in ssl_wrap_socket
    ssl_version=ssl_version)
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 211, in __init__
    socket.__init__(self, _sock=sock._sock)
  File "***/socket_throttle.py", line 54, in __getattr__
    return getattr(self._wrappedsock, attr)
AttributeError: '_socket.socket' object has no attribute '_sock'

Well, that's a downer. As you can tell, the ssl package is trying to use one of the socket's private fields, _sock rather than the socket itself. (Isn't the point of private fields that you're not supposed to access them from the outside? Grr.) If I try to inject myself into that field on my ThrottledSocket object, I run into this problem:

    File "/home/alex/dev/jottalib/src/jottalib/JFS.py", line 590, in request
    r = self.session.get(url, headers=extra_headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 394, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 324, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 478, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 95, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util.py", line 643, in ssl_wrap_socket
    ssl_version=ssl_version)
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 241, in __init__
    ciphers)
TypeError: must be _socket.socket, not ThrottledSocket

Now what? Is there somewhere else in this where I could rate-limit the python communication? Or is there a cleaner way to do it than having to override the socket implementation? Which turns out to be moot anyway, since the ssl package just tries to bypass it altogether.

Krasnoyarsk

The reason ssl is rooting around in private socket fields is, the underlying C library for TLS, openssl, wants very much to talk directly to the OS-level socket descriptor. What might work instead is changing socket-throttle so that it monkeypatches ssl.wrap_socket instead of socket.socket -- you need the throttle wrapper outside the TLS wrapper. I'm not going to post that as an answer because I don't know whether it'll work, and even if it does it'll probably be a lot of tinkering. Good luck? – Cowskin

The ssl library is not mucking around in the private guts of a socket.socket object. ssl.wrap_socket returns a new object which stores the original instance of socket.socket on it as _sock. It has every right to be using its own private attribute. Next time, please read the source before making statements like that about other libraries. Also, as a point of order, the exception is coming from ssl/socket-throttle but bubbles up through requests. Requests has no responsibility for this. – Rosy

Answer 1

Depending on your requirements, you can and maybe should solve this particular problem on the OS level instead of on the application level.

Approaching this on the OS level has two advantages. First, it does not make a difference how the sockets involved are used (HTTP or HTTPS or IRC or some ping of death packets -- it does not matter). Secondly, the more you decouple the different components of your system, the easier it is to make changes afterwards and to debug issues.

There are tools (at least for POSIX-compliant systems) for throttling bandwidth of network interfaces and/or processes. You might want to have a look at these, for example:

• trickle (for shaping traffic of processes)
• wondershaper (for shaping traffic of entire network interfaces, I have actually used this from within a modern Ubuntu, and it works perfectly fine)

These discussions might be relevant for you:

• https://superuser.com/questions/66574/how-to-throttle-bandwidth-on-a-linux-network-interface
• http://jwalanta.blogspot.de/2009/04/easy-bandwidth-shaping-in-linux.html
• https://unix.stackexchange.com/questions/28198/how-to-limit-network-bandwidth

Enlist

I was hoping to provide the user with a configurable upload (and possibly download) setting in the program itself. The trickle solution looks like it would solve my problem, but it also removes control of the situation from my program to an external entity... still, I'm really starting to lose hope about this situation. Either Python simply can't do what I'm asking, or it's just so complicated or esoteric that nobody on Stack Overflow knows how. Honestly I thought this would be a relatively common scenario, but I'm starting to think I was wrong in that assumption. – Krasnoyarsk

Taking control over network traffic surely is possible, but it is much less trivial than most people think. The TCP/IP stack is incredibly complex and well-engineered, incorporating many special algorithms. Your operating system does a very good job of hiding this complexity from you. You can achieve what you want, just not in a short project. Maybe this make you some home: If trickle works for you, you can ship it together with your application, and put everything into a "wrapper", whereas the wrapper configures trickle as well as your app. – Enlist

By the way, you may want to upvote those answers that provided you with some insight, even if it wasn't the perfect solution that you hoped to obtain. ;) – Enlist

True. I sometimes forget. – Krasnoyarsk

Oh, and you might want to check out github.com/praus/shapy which I just came across. – Enlist

Answer 2

It looks like you're trying to throttle HTTP requests. If that's the case, you can try RequestsThrottler instead. Python requests is way nicer than httplib too.

Stumper

I'm actually using the requests library (which in turn uses httplib). This RequestsThrottler seems promising, but I'll have to test it to make sure before I accept your answer. :) I wasn't having issues with throttling HTTP connections, but HTTPS connections is where I've been having trouble. – Krasnoyarsk

Either I don't understand how RequestsThrottler works, or it only affects downloads, not uploads. I need to throttle uploads, as the first line in my question says. Any chance you could provide me with a working sample, if you've gotten it to work before? – Krasnoyarsk