Python unpickling stack underflow

Question

I've been working on a python application, where the client sends a clock signal to the server and the server responds with an audio signal.
I have two buttons, one to start the clock and one to pause the track.

Main Class

# function I call when I hit the play button
def play(self):
    start_song = [250]
    global IS_FIRST_PLAY
    if IS_FIRST_PLAY:
        IS_FIRST_PLAY = False
        self.startClock()
    if IS_CONNECTED:
        client.sendMessage(start_song)

# here I start the clock and send a constant clock signal to the client
def startClock(self):
    clock = midi.startClock()
    for i in clock:
        if IS_CONNECTED:
            client.sendMessage(i)
    midi.playing = True

# here I pause the track
def pause(self):
    stop_song = [252]
    if IS_CONNECTED:
        client.sendMessage(stop_song)
    midi.sendMidiMessage(stop_song)
    midi.playing = False
    midi.mtClock = [0, 0, 0, 0]

Client class

# this is the client.sendMessage() function
def sendMessage(self, message):
    self.s.sendall(pickle.dumps(message))

Server class

# this is the class that handles the incoming clock signal for the server
class MyTCPHandler(socketserver.BaseRequestHandler):

    def handle(self):
        global IS_FIRST_PLAY, IS_PLAYING      
        thread1 = threading.Thread(target=self.sendAudio)
        thread1.start()
        while True:
            # here throws python an error
            self.data = pickle.loads(self.request.recv(12).strip())

This all works fine, except for a random moment, when I change pause to play, I keep getting this error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socketserver.py", line 306, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socketserver.py", line 332, in process_request
    self.finish_request(request, client_address)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socketserver.py", line 345, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socketserver.py", line 666, in __init__
    self.handle()
  File "/Users/cedricgeerinckx/Dropbox/Redux/OSX/Server.py", line 85, in handle
    self.data = pickle.loads(self.request.recv(12).strip())
_pickle.UnpicklingError: unpickling stack underflow

What can this issue possibly be?

Anticosti

Looks like you've got a bad pickle. Why are you using recv(12).strip()? – Excrescent

I also used pickle without strip(), but no difference. I was sending my data with send(1024) and recv(1024), but the wrong data was send, because of reading 1024 bytes instead of just one line. Maybe I should change this to readlineor something? – Anticosti

Answer 1

unpickling stack underflow can happen when a pickle ends unexpectedly.

Here self.request.recv(12), you're receiving only 12 bytes max, your pickled object must be longer than 12 bytes and so it gets cut off.

I would not recommend handling TCP sockets directly unless you're very, very familiar with networking and you need very high performance. I'd suggest to use HTTP to wrap your messages and use an HTTP library.

If you do have to deal with TCP directly though, you'll have two choices:

1. You can agree on a terminator string between your client and server, say the '\0' (null) character; and your messages would be delimited with this terminator string. The terminator string must never occur inside the message body (or you'll have to figure out a way to escape the terminator string in the body); you would also need to buffer your packets so that you can receive the entire message if your read size are smaller or larger than your objects and split the messages on the terminator string. Note that you'll also need to handle the situation where if multiple small messages are sent in quick succession, the receiver may receive multiple messages in a single .recv().

2. Perhaps easier is to start all messages by its length as the first four bytes it sends. The receiver will always start by reading four bytes from the stream, decode that into an integer, and read that many bytes from the stream, that's one full message.

Alternatively, you may be able to rearchitect your program to use a multiprocessing Queue if both sender and receiver is in Python.

I'd say using an HTTP library as your transport protocol is probably going to be easiest as it will handle all these details for chunking messages for you as well as being usable across multiple machines and technologies.

Barcroft

Is the integer the max value? Or does it need to be exact? Because at first I was sending my data with send(1024) and recv(1024), but the wrong data was send, because of reading 1024 bytes instead of just one line. – Anticosti

@Barto: the integer you pass to read() is the maximum amount of bytes that read() will return, read() may return less bytes than requested but never more. – Barcroft

Thanks for the explanation. As soon as I get the chance, I'll test it out! – Anticosti

You should also be aware that with TCP sockets, the data from a single send() can appear across multiple recv() calls. A send() of 100 bytes might appear in a recv() of the first 25 followed by a recv() of the remaining 75 (or, conceivably, 100 recv()s of 1 byte each!). Length prefixing is definitely the way to go if you must use TCP. – Palaeolithic