How to shift bits in a 2-5 byte long bytes object in python?

Question

I am trying to extract data out of a byte object. For example: From b'\x93\x4c\x00' my integer hides from bit 8 to 21. I tried to do bytes >> 3 but that isn't possible with more than one byte. I also tried to solve this with struct but the byte object must have a specific length.

How can I shift the bits to the right?

Campagna

In Perl, you'd use my $i = ( unpack('N', substr("\x00\x00".$s, -4)) >> 8) & 0x1FFF;. – Chloro

Note that bits are counted from right to left starting at 0; bit 8 is the last bit of the \x4c byte, bit 21 through 16 are the last right-most 5 bits of the first byte. Your shift by 3 implies that you are counting bits from left to right; perhaps you are looking for a different set of bits from the convention? – Paramorphism

Answer 1

Don't use bytes to represent integer values; if you need bits, convert to an int:

value = int.from_bytes(your_bytes_value, byteorder='big')
bits_21_to_8 = (value & 0x1fffff) >> 8

where the 0x1fffff mask could also be calculated with:

mask = 2 ** 21 - 1

Demo:

>>> your_bytes_value = b'\x93\x4c\x00'
>>> value = int.from_bytes(your_bytes_value, byteorder='big')
>>> (value & 0x1fffff) >> 8
4940

You can then move back to bytes with the int.to_bytes() method:

>>> ((value & 0x1fffff) >> 8).to_bytes(2, byteorder='big')
b'\x13L'

Paramorphism

For those who find their way here via Google: note that this solution is only appropriate if you are using Python 3. The int class in Python 2 does not have a from_bytes method. – Bartko

@larsks: yes, the question is specifically tagged with the python-3.x tag. :-) – Paramorphism

to get 8-21 bits, you could drop the first 8 bits and take 13 bits from what left: (value >> 8) & 0b1_1111_1111_1111. To get n bits from x starting with offset: get_bits = lambda x, n, offset=0: (x >> offset) & ~-(1 << n) – Fourposter

Answer 2

As you have a bytes string and you want to strip the right-most eight bits (i.e. one byte), you can simply it from the bytes string:

>>> b'\x93\x4c\x00'[:-1]
b'\x93L'

If you want to convert that then to an integer, you can use Python’s struct to unpack it. As you correctly said, you need a fixed size to use structs, so you can just pad the bytes string to add as many zeros as you need:

>>> data = b'\x93\x4c\x00'
>>> data[:-1]
b'\x93L'
>>> data[:-1].rjust(4, b'\x00')
b'\x00\x00\x93L'
>>> struct.unpack('>L', data[:-1].rjust(4, b'\x00'))[0]
37708

Of course, you can also convert it first, and then shift off the 8 bits from the resulting integer:

>>> struct.unpack('>Q', data.rjust(8, b'\x00'))[0] >> 8
37708

If you want to make sure that you don’t actually interpret more than those 13 bits (bits 8 to 21), you have to apply the bit mask 0x1FFF of course:

>>> 37708 & 0x1FFF
4940

(If you need big-endianness instead, just use <L or <Q respectively.)

---

If you are really counting the bits from left to right (which would be unusual but okay), then you can use that padding technique too:

>>> struct.unpack('>Q', data.ljust(8, b'\x00'))[0] >> 43
1206656

Note that we’re adding the padding to the other side, and are shifting it by 43 bits (your 3 bits plus 5 bytes for the padded data we won’t need to look at)

Frodeen

Note that int.from_bytes() doesn't require padding and can take byte sizes larger than 8 as well. – Paramorphism

Answer 3

Another approach that works for arbitrarily long byte sequences is to use the bitstring library which allows for bitwise operations on bitstrings e.g.

>>> import bitstring
>>> bitstring.BitArray(bytes=b'\x93\x4c\x00') >> 3
BitArray('0x126980')

Ciera

small aside: BitArray is mutable; Bits is the base class for general use. cf. bytearray and bytes. – Balustrade

Answer 4

You could convert your bytes to an integer then multiply or divide by powers of two to accomplish the shifting

Adda