How to pack a UUID into a struct in Python?

Question

I have a UUID that I was thinking of packing into a struct using UUID.int, which turns it into a 128-bit integer. But none of the struct format characters are large enough to store it, how to go about doing this?

Sample code:

s = struct.Struct('L')
unique_id = uuid.uuid4()    
tuple = (unique_id.int)
packed = s.pack(*tuple)

The problem is, struct format 'L' is only 4 bytes...I need to store 16. Storing it as a 32-char string is a bit much.

Hartman

Post your code sample please. – Jarlathus

Answer 1

It is a 128-bit integer, what would you expect it to be turned into? You can split it into several components — e.g. two 64-bit integers:

max_int64 = 0xFFFFFFFFFFFFFFFF
packed    = struct.pack('>QQ', (u.int >> 64) & max_int64, u.int & max_int64)
# unpack
a, b     = struct.unpack('>QQ', packed)
unpacked = (a << 64) | b

assert u.int == unpacked

Unwind

Works beautifully. I was thinking there might be an easier way like using UUID.bytes but bitshifting is pretty simple already. – Hartman

Answer 2

As you're using uuid module, you can simply use bytes member, which holds UUID as a 16-byte string (containing the six integer fields in big-endian byte order):

u = uuid.uuid4()
packed = u.bytes # packed is a string of size 16
assert u == uuid.UUID(bytes=packed)

Neuberger

But since struct can only store a char, I would have to iterate through the bytes as 16 separate characters. Then upon unpacking, put together 16 chars again. EDIT: Looks like it can store a string too, I will try this out. – Hartman

The version of Python I'm using also seems to do some implicit conversion with strings in pack, I think I'm safer using long. – Hartman

I can't see a problem, you should be fine with stuct.pack("16s", packed) which will be equivalent to struct.pack("16c", *[c for c in packed]) (and will just recopy your string). – Neuberger

Answer 3

TL;DR

struct.pack('<QBBHHL', *uuid_foo.fields[::-1])

Introduction

Even though Cat++'s answer is really great, it breaks the UUID in half to fit it into two unsigned long longs. I wanted to pack each field, which left me with the following:

def maxsize(size: typing.Union[int,str]):
    """ Useful for playing with different struct.pack formats """
    if isinstance(size, str):
        size = struct.calcsize(size)
    return 2 ** (4 * size) - 1

uuid_max = uuid.UUID('ffffffff-ffff-ffff-ffff-ffffffffffff')
tuple(maxsize(len(f)) for f in str(u).split('-'))
# (4294967295, 65535, 65535, 65535, 281474976710655)

uuid_max.fields
# (4294967295, 65535, 65535, 255, 255, 281474976710655)

uuid_foo = UUID('909822be-c5c4-432f-95db-da1be79cf067')
uuid_foo.fields
# (2425889470, 50628, 17199, 149, 219, 239813384794215)

The first five fields are easy since they already line up as unsigned 8, 4, 4, 2, 2 size integers. The last one required a little extra help from another answer.

Notes: Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct.

No padding is added when using non-native size and alignment, e.g. with ‘<’, ‘>’, ‘=’, and ‘!’.

To align the end of a structure to the alignment requirement of a particular type, end the format with the code for that type with a repeat count of zero. See Examples.

struct.pack('>LHHBBQ', *uuid_foo.fields)
# b'\x90\x98"\xbe\xc5\xc4C/\x95\xdb\x00\x00\xda\x1b\xe7\x9c\xf0g'
#                                    ^^  ^^ these empty bytes won't work!

The actual answer

Since the last field is size 12, you'll have to pack it and unpack it backwards, little endian. That'll leave zeros at the end, instead of between the fifth and sixth fields.

struct.unpack('<QBBHHL', struct.pack('<QBBHHL', *uuid_foo.fields[::-1]))
# (281474976710655, 255, 255, 65535, 65535, 4294967295)

uuid_foo.fields
# (4294967295, 65535, 65535, 255, 255, 281474976710655)

Regenerating this requires you reverse it one more time.

uuid_packed = struct.pack('<QBBHHL', *uuid_foo.fields[::-1])
uuid_unpacked = struct.unpack('<QBBHHL', uuid_packed)[::-1]
uuid.UUID(fields=uuid_unpacked)
# UUID('909822be-c5c4-432f-95db-da1be79cf067')

Grievous