WebSocket protocol binary data endianness

Question

I am writing server and client based on WebSocket protocol.

Server is made using Python and Twisted.

Right now I can send binary data from server to client and back, only problem is that, according to some sources, endianness of binary data sent from browser is based on machine endianness. I want to make sure, is it true?

If it's true, then should I somehow check what endianness client has and read/send data from/to him using his endianness? What is the best way to check client endianness, just send from client

var view_buffer = new UInt8Array(new ArrayBuffer(1));
view_buffer[0] = 1;

this data, and check on server if it returns 1 or 128?

Ruscio

Note that one bit in a byte doesn't tell you endianess. Bit are very rarely swapped by processors, and for sure never swapped when sending over the network. A better example would have been to use two bytes. – Ellieellinger

Answer 1

To check endianess you'd need to send at least a two byte value.

You can't check just by sending a single byte because the bit-level endianess has already been sorted out for you at that point.

Hence the endianess only matters for purposes of byte swapping (if required) and if you're trying to pack individual bits using endian-specific rules, such as when you use bitfields in C.

If your question relates to your own messages carried within WebSocket payloads, the usual process is to choose a byte order yourself, and then do whatever packing or unpacking you have to do to convert from the native byte order to your preferred byte order. Most protocols I know of use big-endian, aka "network order".

So, if you have a 16 bit value, convert it into two 8-bit bytes yourself with shifts and bitwise operators, and then send the top byte first followed by the second byte. At the server side, reverse that process. Most server side languages make it pretty easy to byte-swap values from network order to native byte order.

Immoralist

Thanks for answer. Look at my comment on Mark Ransom answer please. – Gammadion

@RafałŁużyński are you concerned about the messages that form the WebProtocol protocol itself, or the transport of your data within that protocol? – Immoralist

I did what you mentioned. I split data into 8-bits values on client side and send it like that to server. But on server I'm not doing any reversal process. I just unpack data with big-endian and values are right (struct.unpack('>H', data)). Is that ok? – Gammadion

Answer 2

According to RFC 6455:

Multibyte length quantities are expressed in network byte order.

Network byte order is big endian. Both the server and the client should use this byte order, no matter what their native order is.

In Python the struct module can be used to ensure the proper byte order with the '>' specifier. I'm not sure how it would be done in Javascript.

Plagiary

I have packing of message types in 2bytes (first 4 bits is a type, and last 12bits is the subtype). # shift type value 12 bits to left message_type_value = message_type << 12 # alternative of subtype value message_types_value = message_type_value | message_subtype. I checked it and when I send from client value 4096 (which means type 1, subtype 0) on server I unpack (276, 0) using big endian, using little endian it's (1,0) and it looks like I can't force browser to use big-endian – Gammadion

@RafałŁużyński, see #9283593 – Plagiary

Thanks, looks like it works when I splited data into 8-bits values. But I accepted Alnitak answer since he gave me more detailed answer. – Gammadion

Answer 3

In RFC 6455 there is this note about the order of the close message:

If there is a body, the first two bytes of the body MUST be a 2-byte unsigned integer (in network byte order) representing a status code with value /code/ defined in Section 7.4.

So it sounds like the protocol would generally expect you to use big endian if you are to use binary data in your WS messages. I would imagine it's not mandatory, through.

If you'd like to allow either one, what you can do is add a magic number at the start of your message. That would be 2 bytes which are different such as (('M' << 8) | 'G'). If you receive "M" then "G", it's big endian. If you receive "G" then "M", it's little endian. Now each message tells you its endianness. No more guessing work.

As pointed out by Mark Ransom in a comment, it's possible to send data in either endian from JavaScript, it can be annoying to have to swap all the bytes, though. So having a 2 byte value which indicates the current endianness is often a really good solution to the problem. The server that receives the data can check these two bytes and swap the short, int, long int data accordingly.

Ellieellinger

Answer 4

Endianess is only relevant for data types which are composed of more than one byte. Since you are working with an array of uint8 endianess does not matter.

Coachman

I didnt say I'm working only with UInt8. I just wanted to send it for endianess check, but you are right, with 1 byte it doesn't matter. – Gammadion

I was just refering to your comment and the UInt8Aray. A uint8 is exactly the same on a big endian and little endian system. If you want to detect the endianess by transferring test data you need at least an uint16. A big endian 1 will become a little endian 256 and vice verse: 0x0001 <-> 0x0100 – Coachman

Answer 5

Have the client send a text message of 126 bytes or more (but less than 65535) (200 for example) to the server. The Payload_Length field will be 126.

Then check the next two bytes for the Extended Payload Length to determine if the client sent it in big endian (probably) or if the client messed up and is sending in little endian format.

For big endian, the byte right next to the 8-bit payload length (which should be 126 to indicate the next two bytes actually contain the real payload length) should be the Most Significant Byte of the 16-bit length.

Twocolor