Python struct.pack() for individual elements in a list?

Question

I would like to pack all the data in a list into a single buffer to send over a UDP socket. The list is relatively long, so indexing each element in the list is tedious. This is what I have so far:

NumElements = len(data)
buf = struct.pack('d'*NumElements,data[0],data[1],data[2],data[3],data[4])

But I would like to do something more pythonic that doesn't require I change the call if I added more elements to the list... something like:

NumElements = len(data)
buf = struct.pack('d'*NumElements,data)  # Returns error

Is there a good way of doing this??

Diffuser

Answer 1

Yes, you can use the *args calling syntax.

Instead of this:

buf = struct.pack('d'*NumElements,data)  # Returns error

… do this:

buf = struct.pack('d'*NumElements, *data) # Works

See Unpacking Argument Lists in the tutorial. (But really, read all of section 4.7, not just 4.7.4, or you won't know what "The reverse situation…" is referring to…) Briefly:

… when the arguments are already in a list or tuple but need to be unpacked for a function call requiring separate positional arguments… write the function call with the *-operator to unpack the arguments out of a list or tuple…

Rambow

Ah that's just what I needed! Thank you! – Diffuser

'd' * NumElements isn't very good practice, it could end up making a very large string, only to have to waste time parsing, only to free afterwards. Instead include the number directly in the string: buf = struct.pack("{0:d}d".format(NumElements), *data) – Pointdevice

@ideasman42: Good point. But with a huge number of values, unpacking them into 500000 arguments might be as much of a problem as passing 'd'*500000, so you might want to test that against b''.join(struct.pack('d', elem) for elem in data), or consider using something like array or ctypes instead of struct. – Rambow

@abarnert, of course if performance is a priority... its worth looking into alternatives as you suggest, even so, when an API is provided that exposes a convenient & fast way to handle data. I'm not sure why you'd use the slower option. The example you give using b''.join is the slowest method by far, See benchmark gist.github.com/ideasman42/662dd741f31eaf33b006 Also, since this is an answer to a question for someone learning the API, its worth using good-practices. – Pointdevice

@ideasman42: My point isn't that b''.join(…) is going to be the fastest, it's that you shouldn't try to guess the fastest in advance; you should look at all of the reasonable options (with the same algorithmic complexity) and test them to see which really is fastest. Also, you really want to test with timeit, not a loop over time.time; read the docs on the module for why. – Rambow

@abarnert, Yep, agree re: optimization, however, when there is an option to allocate a string, potentially unknown amount of memory, parse every character... then throw it away. If theres a convenient alternative which avoids doing that... just use it, theres really no good reason not to. Python devs dont do this in their code (not counting tests), and probably if you submitted code to Python.org standard library it probably wouldn't pass review. – Pointdevice

@ideasman42: Well, sure, but we don't need to improve the OP's code until it meets the quality of the stdlib, we just need to fix the problem he's asking about. Of course adding comments explaining other things that might be worth improving is definitely helpful (that's why I upvoted your comment), but within the answer there's a tough balance between failing to correct irrelevant but bad things and swamping the asker in things he didn't ask about. (I think I usually err on the latter side, but if you think I did the opposite here, I can edit the answer.) – Rambow

Answer 2

The format string for struct.pack(...) and struct.unpack(...) allows to pass number(representing count) in front of type, with meaning how many times is the specific type expected to be present in the serialised data:

Simple case

data = [1.2, 3.4, 5.6]
struct.pack('3d', data[0], data[1], data[2])
struct.pack('3d', *[1.2, 3.4, 5.6])

or more generally:

data = [1.0, 1.234, 1.9, 3.14, 6.002, 7.4, 9.2]
struct.pack('{}d'.format(len(data)), *data)

Isabelleisac