struct pack return is too long

Question

I'm trying to use the struct.pack function

import struct
values = (0, 44)
s = struct.Struct('HI')
b = s.pack(*values)
print(b)
print(str(len(b)))

and it gives me this output:

b'\x00\x00\x00\x00,\x00\x00\x00'
8

while the python docs say:

Format - C Type         - Python type - Standard size - Notes

H      - unsigned short - integer     - 2             - (3)

I      - unsigned int   - integer     - 4             - (3)

so len() should be 2 + 4 = 6, and I need bytes with size = 6

Any ideas?

I'm using Python 3.6 on Windows 10

Ocrea

Answer 1

pack will add pad bytes so that the second integer is 4 byte aligned. From the documentation:

By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; To ... omit implicit pad bytes, use standard size and alignment instead of native size and alignment: see Byte Order, Size, and Alignment for details

Ergo

That got me an answer, should use struct.Struct('=HI'), struct.Struct('<HI') or struct.Struct('>HI'), – Ocrea

Answer 2

That's a consequence of "Data structure padding". It will pad the H (2 bytes + 2 bytes padding) so that it aligns with the I (4 bytes).

However you can experiment with the order if you need to decrease the size. To quote Wikipedia:

It is possible to change the alignment of structures to reduce the memory they require (or to conform to an existing format) by reordering structure members or changing the compiler’s alignment (or “packing”) of structure members.

For example on my computer it works if you just swap the H and I:

import struct
values = (0, 1)
s = struct.Struct('IH')         # swapped H and I
b = s.pack(*values)
print(b)                        # b'\x00\x00\x00\x00\x01\x00'
print(str(len(b)))              # 6
print(struct.calcsize('IH'))    # 6
print(struct.calcsize('HI'))    # 8

Catch

Answer 3

The size computation is not directly additive for the contained native types. You should compute the size using struct.calcsize:

In [8]: struct.calcsize('HI')
Out[8]: 8

Appalling