Are packed structs portable?

Question

I have some code on a Cortex-M4 microcontroller and'd like to communicate with a PC using a binary protocol. Currently, I'm using packed structs using the GCC-specific packed attribute.

Here is a rough outline:

struct Sensor1Telemetry {
    int16_t temperature;
    uint32_t timestamp;
    uint16_t voltageMv;
    // etc...
} __attribute__((__packed__));

struct TelemetryPacket {
    Sensor1Telemetry tele1;
    Sensor2Telemetry tele2;
    // etc...
} __attribute__((__packed__));

My question is:

• Assuming that I use the exact same definition for the TelemetryPacket struct on the MCU and the client app, will the above code be portable accross multiple platforms? (I'm interested in x86 and x86_64, and need it to run on Windows, Linux and OS X.)
• Do other compilers support packed structs with the same memory layout? With what syntax?

EDIT:

• Yes, I know packed structs are non-standard, but they seem useful enough to consider using them.
• I'm interested in both C and C++, although I don't think GCC would handle them differently.
• These structs are not inherited and don't inherit anything.
• These structs only contain fixed-size integer fields, and other similar packed structs. (I've been burned by floats before...)

Dragelin

possibly find some similar answers at stackoverflow.com/questions/8568432/… ; +1 b/c of my interest too. I think, it is always a good idea to put largest type, here uint32t at the first place, smaller data types at the last place of the struct or union. – Posey

Is this a question about C or C++? They are not the same language. Please specify and remove the tag for the other in your question. – Braid

packed is always your best bet for any communication protocol. – Permafrost

one obvious possible problem is endianness of the integers – Dafna

EABI v5 doesn't support passing packed structs by value either – Dafna

@Braid just curiosity, would the answer differ if it's supposed to be answered to C++ or C? – Selfexecuting

"Yes, I know packed structs are non-standard, but they seem useful enough to consider using them." I think people creating C standard do not care about networking. endian.h is also non standard. – Felike

Answer 1

You should never use structs across compile domains, against memory (hardware registers, picking apart items read from a file or passing data between processors or the same processor different software (between an app and a kernel driver)). You are asking for trouble as the compiler has somewhat free will to choose alignment and then the user on top of that can make it worse by using modifiers.

No there is no reason to assume you can do this safely across platforms, even if you use the same gcc compiler version for example against different targets (different builds of the compiler as well as the target differences).

To reduce your odds of failure start with the largest items first (64 bit then 32 bit the 16 bit then lastly any 8 bit items) Ideally align on 32 minimum perhaps 64 which one would hope arm and x86 do, but that can always change as well as the default can be modified by whomever builds the compiler from sources.

Now if this is a job security thing, sure go ahead, you can do regular maintenance on this code, likely going to need a definition of each structure for each target (so one copy of the source code for the structure definition for ARM and another for x86, or will need this eventually if not immediately). And then every or every few product releases you get to be called in to do work on the code...Nice little maintenance time bombs that go off...

If you want to safely communicate between compile domains or processors the same or different architectures, use an array of some size, a stream of bytes a stream of halfwords or a stream of words. Significantly reduces your risk of failure and maintenance down the road. Do not use structures to pick apart those items that just restores the risk and failure.

The reason why folks seem to think this is okay because of using the same compiler or family against the same target or family (or compilers derived from other compilers choices), as you understand the rules of the language and where the implementation defined areas are you will eventually run across a difference, sometimes it takes decades in your career, sometimes it takes weeks...Its the "works on my machine" problem...

Toggle

"picking apart items read from a file" - I disagree. You can pack structs to make sure there is no padding and then use that struct to represent items read from a file (given fixed offsets of data in file, ofcourse). – Matthaeus

I disagree with the disagreement. I JUST NOW have to deal with a compiler, that chooses to ignore the pack pragma, because it is free to do so. It is a PITA, First Class!! – Spurgeon

Addendum: Used the __attribute__ mentioned below instead of #pragma, and that is working as intended. God bless stackoverflow!!! – Spurgeon

@U.W. : years later, I know, but just to know a safety trick: when using packed structure, ALWAYS add an assertion (better, if possible, a static assertion...) in your code checking sizeof(your_packed_struct)==expected_size_on_wire - where expected_size_on_wire is obviously the sum of all field's sizes. This way, you'll ensure that the expected size is the correct one, without padding, so that your compiler properly understood that the struct must be packed. You would have detected that your #pragma was ignored maybe even at compile time, and at least obviously on first launch. – Excellence

@Wisblade: Years later I am still here! ;-) And you are right, everything that can be checked/enforced at compile time should be done! – Spurgeon

Answer 2

Considering the mentioned platforms, yes, packed structs are completely fine to use. x86 and x86_64 always supported unaligned access, and contrary to the common belief, unaligned access on these platforms has (almost) the same speed as aligned access for a long time (there's no such thing that unaligned access is much slower). The only drawback is that the access may not be atomic, but I don't think it matters in this case. And there is an agreement between compilers, packed structs will use the same layout.

GCC/clang supports packed structs with the syntax you mentioned. MSVC has #pragma pack, which can be used like this:

#pragma pack(push, 1)
struct Sensor1Telemetry {
    int16_t temperature;
    uint32_t timestamp;
    uint16_t voltageMv;
    // etc...
};
#pragma pack(pop)

Two issues can arise:

1. Endianness must be the same across platforms (your MCU must be using little-endian)
2. If you assign a pointer to a packed struct member, and you're on an architecture which doesn't support unaligned access (or use instructions which have alignment requirements, like movaps or ldrd), then you may get a crash using that pointer (gcc doesn't warn you about this, but clang does).

---

Here's the doc from GCC:

The packed attribute specifies that a variable or structure field should have the smallest possible alignment—one byte for a variable

So GCC guarantees that no padding will be used.

MSVC:

To pack a class is to place its members directly after each other in memory

So MSVC guarantees that no padding will be used.

The only "dangerous" area I've found, is the usage of bitfields. Then the layout may differ between GCC and MSVC. But, there's an option in GCC, which makes them compatible: -mms-bitfields

---

Tip: even, if this solution works now, and it is highly unlikely that it will stop working, I recommend you keep dependency of your code on this solution low.

Note: I've considered only GCC, clang and MSVC in this answer. There are compilers maybe, for which these things are not true.

Fiann

Unaligned accesses can even be atomic, if they do not cross a cache line boundary - but there is little control over that – Ellynellynn

@harold: thanks, I've modified my answer a little bit. – Fiann

I believe unaligned accesses still make a difference: the DIMMs have a 64-bit data channel, doing some math on the number of signals in the datasheets it seems that memory is addressed in multiple of 64-bit. So reading a DWORD at address 3 reads 64-bit from address 0 (a single access) but reading a DWORD at address 6 reads 64-bit from address 0 and 64-bit from address 8 (double access). All this without considering the cache subsystem. – Edgy

@MargaretBloom will that affect cached reads then? Maybe give them a slightly higher latency (the first time that is) since critical-word-first doesn't work? – Ellynellynn

Unaligned access is still slower on x86 than aligned access. The difference compared to other architectures is that x86 forgives attempts to access unaligned data, silently fixing up the misalignment. (Unless you're using aligned SIMD instructions, then it faults. Use the unaligned ones. The performance difference is minimal on the latest architectures, although that hasn't been true for "a long time", unless your definition of "long" is quite different from mine.) – Lymphatic

@CodyGray: That's why I written "(almost)". I've written a little benchmark (which is already very far from how a typical program uses memory - it summed 64-bit numbers from 8 streams), and the difference is 20%. If I use 32-bit numbers instead, I get 9% difference. Yes, one can argue about that 20% is not "almost", and I kinda agree. Maybe I'll edit my answer. About "long". I cannot remember when, about 10 years ago, I did a similar benchmark on my current computer, and I saw similar numbers. Unfortunately I don't have a 10-year old machine to do the test. (I did the test on a Haswell CPU) – Fiann

@CodyGray: and maybe my little test is flawed, maybe one can create a test which shows even bigger differences. But my point is, that for a typical program, aligned/unaligned access is not a huge factor of speed. – Fiann

@CodyGray: here's my test: stackoverflow.com/questions/45128763/…. As it turned out, for 128-bit access, when data is in the cache, the difference grows to 40% – Fiann

I think the summary is for that for recent Intel x86, misaligned access is free or nearly so, for accesses that don't span a 64B cache line. There is hardware that still lets you do 2 reads even if both are misaligned. Your throughput is cut in half for line spanning loads. Larger loads (like 128-bit) are more likely to span a line (indeed, nearly half of 256-bit loads will cross a boundary). – Holdall

No it is not really fine on x86 either. There are instructions for SSE that require aligned access, otherwise a fault occurs. If you ever pass a pointer to a double that is misaligned somewhere, you never know what will happen. There is no good reason to use packed, like, ever; there are only some where you need to judge the alternatives. – Stepmother

@AnttiHaapala: if packed struct is used, then the compiler won't use these instructions. The only problem if the compiler doesn't know about misalignment. That's what I described in 2. – Fiann

@Fiann while it might be true, however you can never safely reference a pointer to a member to a packed struct if the type of member has fundamental alignment requirements – Stepmother

@AnttiHaapala: it not might be true. It is true. If not, then it's a bug in the compiler. One typical use case for packed structs is data transfer, where pointer to member is not typical. But I've already described this problem in my answer. – Fiann

Your answer states that ~"x86_64 always supports aligned access" and therefore it would be safe to always use a pointer to a member; that is not true. – Stepmother

@AnttiHaapala: It is almost safe to use on x86_64. The only exception I know is SSE 16 byte move. Which is highly unlikely that is used in this scenario. But even, there's a warning for this in my answer. But I'll edit my answer to contain this information. – Fiann

Answer 3

If

• endianness is not an issue
• both compilers handle packing correctly
• the type definitions on both C implementations are accurate (Standard compliant).

then yes, "packed structures" are portable.

For my taste too many "if"s, do not do this. It's not worth the hassle to arise.

Dareece

Shouldn't we be able to assume implementation correctness? Same (maybe to a lesser degree) for compiler correctness? – Galleass

@DavidBowling: "assume implementation correctness" As far as I remember at least "packing" isn't part of the C Standard, so well, what would be "correct" though. – Dareece

@alk-- I meant implementation correctness with respect to "the type definitions on both C implementations are accurate," and compiler correctness with respect to "compilers handle packing correctly", in the sense that if a compiler supports packing, it should get it right (but may not). – Galleass

I would add a couple more ifs:1) if all members have a fixed size, no int, short, long, and of course no pointers. 2) no bit-fields. – Strontian

@chqrlie: Well yes, true. Still my answer refers to the particular case the OP is showing, which uses fixed sizes already. – Dareece

@alk: which uses fixed sizes already. The OP only shows 3 members, we can only guess what lies behind // etc... – Strontian

Actually, using packed structs work. If the endianness is the same, they work like a charm, there's no hassle at all. – Fiann

@Fiann Actually, using packed structs work. If the endianness is the same, they work like a charm, there's no hassle at all. Is that because they've worked for you so far? Not having run into any issues yet does not mean they "work like a charm" in general. – Spit

@AndrewHenle: Always apply the compiler specific declaration to say pack this struct. If a certain compiler doesn't have one, don't use that one. – Danit

@Danit You're also assuming whatever choice you make in compiler won't change its implementation in the future and that the deployment platform won't change. I'd use a more portable, stable way to pass the data that's not based on "do it this way with this compiler on just this platform". Because in general, it doesn't work. Fixed-width sizes such as int16_t are in fact optional and are not required by the C standard. – Spit

@AndrewHenle: Yes. And I think that I can depend on it, it is very unlikely that it will ever change. It is a very simple concept, why would it change? Even if it is not standardized. Who cares? All major compilers have this option. In the viewpoint of standard, it is not portable. In the real world, it is. Besides, standards can change too. You usually don't fear that your code's behavior changes when a new standard comes out. But, sometimes standards change, and your code will behave differently. It is the same scenario. – Fiann

@geza: You name it. And that's exactly why I recommend to rule out such uncertainties by concept: Simply use another, a well defined approach: Use (hardware/implementation independent) serialisation. :-) – Dareece

People are different :) In this case, I happily go with packed structs. They worked in the last 20 years, and in my opinion they will work 20 years later. Some other people will choose some other (in this case, more complex) way, just because, there is a possibility that packed struct won't work in the future. Or maybe they have to port the program to a compiler, which doesn't support it. I've no problems with that, as I said, people are different, and choose different solutions for a problem :) – Fiann

@AndrewHenle: about int16_t, I've just asked this question: https://mcmap.net/q/14614/-c-11-on-exotic-hardware, hopefully it won't get closed. – Fiann

@AndrewHenle: I'm talking about packed, not integer types. Indeed I would use the intX types declaring a portable struct. – Danit

@Danit Indeed I would use the intX types declaring a portable struct. But fixed-width intX types such as int16_t are optional in C. Per 7.20.1.1 Exact-width integer types of the C Standard: "The typedef ame intN_t designates a signed integer type with width N, no padding bits, and a two’s complement representation. ... These types are optional. ..." – Spit

(cont) And C++ merely follows the C <stdint.h> header with the C++ equivalent <cstdint>. Per cplusplus.com/reference/cstdint: "Notice that some types are optional (and thus, with no portability guarantees)." The fixed-width integer types are again optional. Strictly speaking, int32_t makes for a non-portable structure for strictly-conforming code. – Spit

Answer 4

You could do that, or use a more reliable alternative.

For the hard core amongst the serialisation fanatics out there, there's CapnProto. This gives you a native structure to deal with, and undertakes to ensure that when it's transferred across a network and lightly worked on, it'll still make sense the other end. To call it a serialisation is nearly inaccurate; it aims to do a little as possible to the in-memmory representation of a structure. Might be amenable to porting to an M4

There's Google Protocol Buffers, that's binary. More bloaty, but pretty good. There's the accompanying nanopb (more suited to microcontrollers), but it doesn't do the whole of GPB (I don't think it does oneof). Many people use it successfully though.

Some of the C asn1 runtimes are small enough for use on micro controllers. I know this one fits on M0.

Granger

I kinda think that using these libraries to transfer some little data from a sensor is overkill. – Fiann

@geza, they're not if one wants that little sensor to reliably talk to software running on Windows, Linux and OSX, as the original poster asked, without having to do some boring, error prone structure layout verification work. Using a reputable library designed for the job helps avoid having to do all that work oneself. Look at all the comments (including your own) for alk's excellent post; endianness, layout, integer width; boring, pointless discussion in my opinion; it's a solved problem. – Granger

if you use packed structs with intXX_t, then I'm absolutely sure that you don't have to do anything like "layout verification". It will work out of the box, on any of the platforms mentioned. Put a static_assert after the struct to assert that the sizeof is OK (to catch possible future errors), and that's all. – Fiann

I echo alk's post. Too many if's for it to be worthwhile. Why not just use a lightweight serialisation library and never, ever have to worry about it ever again? – Granger

@Granger because it's 1) another dependency, 2) more development and maintenance overhead, 3) I've yet to get Cap'nProto working correctly cross-platform, 4) I don't wish to support Google by using Protobufs, 5) other serialization libraries are generally too high level and do too much processing at the cost of performance. People have reasons for things, sometimes. – Gulosity

Answer 5

If you want something maximally portable, you can declare a buffer of uint8_t[TELEM1_SIZE] and memcpy() to and from offsets within it, performing endianness conversions such as htons() and htonl() (or little-endian equivalents such as the ones in glib). You could wrap this in a class with getter/setter methods in C++, or a struct with getter-setter functions in C.

Abie

uint8_t is only defined on platforms supporting it. You may want to used unsigned char instead for better portability. – Cristinacristine

@Cristinacristine Or uint_least8_t has a stronger guarantee that it’s the smallest type at least 8 bits wide. You probably can’t pack data using the same logic on your oddball 9-bit or 12-bit computer from 1970 that has a C99 compiler anyway. – Abie

Does the standard make any alignment guarantees for uint_least8_t as it does for unsigned char (alignment of 1)? Or can there be padding between array elements on my oddball computer? – Cristinacristine

@Cristinacristine There can be padding bits between uint_least8_t, but not uint8_t if it exists. – Abie

@Cristinacristine As you know. :) – Abie

Answer 6

It strongly depends on what struct is, bear in mind that in C++ struct is a class with default visibility public.

So you can inherit and even add virtual to this so this could break things for you.

If it is a pure data class (in C++ terms a standard layout class) this should work in combination with packed.

Also bear in mind, that if you start doing this you might get problems with strict aliasing rules of your compiler, because you will have to look at the byte representation of your memory (-fno-strict-aliasing is your friend).

Note

That being said I would strongly advise against using that for serialization. If you use tools for this (i.e.: protobuf, flatbuffers, msgpack, or others) you get a ton of features:

• language independence
• rpc (remote procedure call)
• data specification languages
• schemas/validation
• versioning

Viscounty

Strict aliasing should't be an issue in this case because of exception to allow aliasing through character types (that is; view structure bytes through char pointer). – Sykes

"because you will have to look at the byte representation of your memory". What do you mean by this? – Fiann

@Fiann you want to write this to a socket or a file right? Thus you need to cast your struct to char * or void * if you do this wrong you might see it serializing stale data. – Viscounty

@Alex: can you give an example of this stale data? I don't see why -fno-strict-aliasing is needed in this case. Looking at any data with char * doesn't violate strict aliasing rules. It is allowed. But why would I need to do that? I simply call write(fd, &myStruct, sizeof(myStruct)), and it will be written perfectly. No stale data. – Fiann

if you call write it's fine. – Viscounty

Answer 7

Speaking about alternatives and considering your question Tuple-like container for packed data (for which I don't have enough reputation to comment on), I suggest having a look at Alex Robenko's CommsChampion project:

COMMS is the C++(11) headers only, platform independent library, which makes the implementation of a communication protocol to be an easy and relatively quick process. It provides all the necessary types and classes to make the definition of the custom messages, as well as wrapping transport data fields, to be simple declarative statements of type and class definitions. These statements will specify WHAT needs to be implemented. The COMMS library internals handle the HOW part.

Since you're working on a Cortex-M4 microcontroller, you may also find interesting that:

The COMMS library was specifically developed to be used in embedded systems including bare-metal ones. It doesn't use exceptions and/or RTTI. It also minimises usage of dynamic memory allocation and provides an ability to exclude it altogether if required, which may be needed when developing bare-metal embedded systems.

Alex provides an excellent free ebook titled Guide to Implementing Communication Protocols in C++ (for Embedded Systems) which describes the internals.

Kendakendal

Thanks for the link! It looks like something similar to, although more complicated than what I had in mind with my tuple-like stuff. – Dragelin

Answer 8

Here is pseudo code towards an algorithm that may fit your needs to ensure the use with the proper target OS and platform.

If using the C language you will not be able to use classes, templates and a few other things, but you can use preprocessor directives to create the version of your struct(s) you need based on the OS, the architect CPU-GPU-Hardware Controller Manufacturer {Intel, AMD, IBM, Apple, etc.}, platform x86 - x64 bit, and finally the endian of the byte layout. Otherwise the focus here would be towards C++ and the use of templates.

Take your struct(s) for example:

struct Sensor1Telemetry {
    int16_t temperature;
    uint32_t timestamp;
    uint16_t voltageMv;
    // etc...
} __attribute__((__packed__));

struct TelemetryPacket {
    Sensor1Telemetry tele1;
    Sensor2Telemetry tele2;
    // etc...
} __attribute__((__packed__));

You could template these structs as such:

enum OS_Type {
    // Flag Bits - Windows First 4bits
    WINDOWS    = 0x01  //  1
    WINDOWS_7  = 0x02  //  2 
    WINDOWS_8  = 0x04, //  4
    WINDOWS_10 = 0x08, //  8

    // Flag Bits - Linux Second 4bits
    LINUX      = 0x10, // 16
    LINUX_vA   = 0x20, // 32
    LINUX_vB   = 0x40, // 64
    LINUX_vC   = 0x80, // 128

    // Flag Bits - Linux Third Byte
    OS         = 0x100, // 256
    OS_vA      = 0x200, // 512
    OS_vB      = 0x400, // 1024
    OS_vC      = 0x800  // 2048

    //....
};

enum ArchitectureType {
    ANDROID = 0x01
    AMD     = 0x02,
    ASUS    = 0x04,
    NVIDIA  = 0x08,
    IBM     = 0x10,
    INTEL   = 0x20,
    MOTOROALA = 0x40,
    //...
};

enum PlatformType {
    X86 = 0x01,
    X64 = 0x02,
    // Legacy - Deprecated Models
    X32 = 0x04,
    X16 = 0x08,
    // ... etc.
};

enum EndianType {
    LITTLE = 0x01,
    BIG    = 0x02,
    MIXED  = 0x04,
    // ....
};

// Struct to hold the target machines properties & attributes: add this to your existing struct.

struct TargetMachine {
    unsigned int os_;
    unsigned int architecture_;
    unsigned char platform_;
    unsigned char endian_;

    TargetMachine() : 
      os_(0), architecture_(0),
      platform_(0), endian_(0) {
    }

    TargetMachine( unsigned int os, unsigned int architecture_, 
                   unsigned char platform_, unsigned char endian_ ) :
      os_(os), architecture_(architecture),
      platform_(platform), endian_(endian) {
    }    
};

template<unsigned int OS, unsigned int Architecture, unsigned char Platform, unsigned char Endian>
struct Sensor1Telemetry {       
    int16_t temperature;
    uint32_t timestamp;
    uint16_t voltageMv;
    // etc...
} __attribute__((__packed__));

template<unsigned int OS, unsigned int Architecture, unsigned char Platform, unsigned char Endian>
struct TelemetryPacket {
    TargetMachine targetMachine { OS, Architecture, Platform, Endian };
    Sensor1Telemetry tele1;
    Sensor2Telemetry tele2;
    // etc...
} __attribute__((__packed__));

With these enum identifiers you could then use class template specialization to set the up this class to its needs depending on the above combinations. Here I would take all the common cases that would seem to work fine with default class declaration & definition and set that as the main class's functionality. Then for those special cases, such as different Endian with byte order, or specific OS versions doing something in a different way, or GCC versus MS compilers with the use of __attribute__((__packed__)) versus #pragma pack() can then be the few specializations that need to be accounted for. You shouldn't need to specify a specialization for every possible combination; that would be too daunting and time consuming, should only need to do the few rare case scenarios that can occur to make sure you always have proper code instructions for the target audience. What also makes the enums very handy too is that if you pass these as a function argument, you can set multiple ones at a time as they are designed as bit flags. So if you want to create a function that takes this template struct as its first argument, then supported OS's as its second you could then pass in all available OS support as bit flags.

This may help to ensure that this set of packed structures is being "packed" and or aligned correctly according to the appropriate target and that it will always perform the same functionality to maintain portability across different platforms.

Now you may have to do this specialization twice between the preprocessor directives for different supporting compilers. Such that if the current compiler is GCC as it defines the struct in one way with its specializations, then Clang in another, or MSVC, Code Blocks etc. So there is a little overhead to get this initially set up, but it should, could highly ensure that it is being properly used in the specified scenario or combination of attributes of the target machine.

Succeed

Answer 9

Not always. When you send data to different architect processor, you need to consider about Endianness, primitive data type, etc. Better to use Thrift or Message Pack. If not, create yourself Serialize and DeSerialize methods instead.

Jason