Can memcpy be used for type punning?

Question

This is a quote from the C11 Standard:

6.5 Expressions
...

6 The effective type of an object for an access to its stored value is the declared type of the object, if any. If a value is stored into an object having no declared type through an lvalue having a type that is not a character type, then the type of the lvalue becomes the effective type of the object for that access and for subsequent accesses that do not modify the stored value. If a value is copied into an object having no declared type using memcpy or memmove, or is copied as an array of character type, then the effective type of the modified object for that access and for subsequent accesses that do not modify the value is the effective type of the object from which the value is copied, if it has one. For all other accesses to an object having no declared type, the effective type of the object is simply the type of the lvalue used for the access.

7 An object shall have its stored value accessed only by an lvalue expression that has one of the following types:

— a type compatible with the effective type of the object,
— a qualified version of a type compatible with the effective type of the object,
— a type that is the signed or unsigned type corresponding to the effective type of the object,
— a type that is the signed or unsigned type corresponding to a qualified version of the effective type of the object,
— an aggregate or union type that includes one of the aforementioned types among its members (including, recursively, a member of a subaggregate or contained union), or
— a character type.

Does this imply that memcpy cannot be used for type punning this way:

double d = 1234.5678;
uint64_t bits;
memcpy(&bits, &d, sizeof bits);
printf("the representation of %g is %08"PRIX64"\n", d, bits);

Why would it not give the same output as:

union { double d; uint64_t i; } u;
u.d = 1234.5678;
printf("the representation of %g is %08"PRIX64"\n", d, u.i);

What if I use my version of memcpy using character types:

void *my_memcpy(void *dst, const void *src, size_t n) {
    unsigned char *d = dst;
    const unsigned char *s = src;
    for (size_t i = 0; i < n; i++) { d[i] = s[i]; }
    return dst;
}

---

EDIT: EOF commented that The part about memcpy() in paragraph 6 doesn't apply in this situation, since uint64_t bits has a declared type. I agree, but, unfortunately, this does not help answer the question whether memcpy can be used for type punning, it just makes paragraph 6 irrelevant to assess the validity of the above examples.

Here here is another attempt at type punning with memcpy that I believe would be covered by paragraph 6:

double d = 1234.5678;
void *p = malloc(sizeof(double));
if (p != NULL) {
    uint64_t *pbits = memcpy(p, &d, sizeof(double));
    uint64_t bits = *pbits;
    printf("the representation of %g is %08"PRIX64"\n", d, bits);
}

Assuming sizeof(double) == sizeof(uint64_t), Does the above code have defined behavior under paragraph 6 and 7?

---

EDIT: Some answers point to the potential for undefined behavior coming from reading a trap representation. This is not relevant as the C Standard explicitly excludes this possibility:

7.20.1.1 Exact-width integer types

1 The typedef name intN_t designates a signed integer type with width N, no padding bits, and a two’s complement representation. Thus, int8_t denotes such a signed integer type with a width of exactly 8 bits.

2 The typedef name uintN_t designates an unsigned integer type with width N and no padding bits. Thus, uint24_t denotes such an unsigned integer type with a width of exactly 24 bits.

These types are optional. However, if an implementation provides integer types with widths of 8, 16, 32, or 64 bits, no padding bits, and (for the signed types) that have a two’s complement representation, it shall define the corresponding typedef names.

Type uint64_t has exactly 64 value bits and no padding bits, thus there cannot be any trap representations.

Emission

The way I read section 6 is that whether your example is equivalent. With the memcpy() the value is the same since you assign the double then do a memcpy() to the uint64_t so the value copied is of type double. Similarly with the union you assign a value to the double part of the union so the value in the memory area is of type double and you access it via the uint64_t. Either way there is no type conversion of the actual value. So no conversion from double to uint64_t. memcpy() copies the specified number of bytes. Why do you think the two examples would be different? – Meiny

I'm no expert on the C standard, but there's an important difference between the two. The union version ensures alignment restrictions of both fields are met by the starting address of u. The memcpy version doesn't guarantee that bits is positioned on a double boundary. It could fail if double alignment was more restrictive than uint64_t, e.g. with a bus error when printf attempts to compute the double string representation of bits. – Randolphrandom

@RichardChambers: I am aware that no actual type conversion occurs, merely a reinterpretation of the bitwise representation of double as a uint64_t. I am perplexed by the wording of paragraph 6, especially regarding the copy done via memcpy. I tough it should be safe, but other savvy C experts differ. – Emission

@Gene: I don't see any alignment issues: memcpy is explicitly safe to copy between non aligned blocks and the values passed to printf are read from their effective types. – Emission

I can understand being perplexed by the wording of para 6. It has that lawyerly rhythm and construction that is not truly accessible to mere mortals. When I read it, my impression is that it is the same thing and should be safe. Like a union, using memcpy() copies a value from a memory location whose value is of a particular type and puts it in a different location and the value is still the same type. However when accessed through an lvalue of a different type, the value is retrieved as the type of the lvalue and not the original type. – Meiny

@RichardChambers: However when accessed through an lvalue of a different type, the value is retrieved as the type of the lvalue and not the original type. which is exactly the definition of type punning and the purpose of the above code. So is your answer: yes, memcpy can be used for type punning. – Emission

I would consider memcpy() safe for type punning. And actually I have seen a lot of embedded code that depends on it with memcpy() used to divide up struct objects with offset address calculations and really weird stuff. The struct were defined with pragma for byte alignment so it all works. That was not C11 but older C98 and some of it old K&R C. This might actually be an improvement in the spec to actually specify the behavior that has been canonized as the defacto behavior for years. – Meiny

@RichardChambers: can you write an answer? – Emission

I put together an answer. probably a bit too wordy as that is a problem I have. <laughter>. – Meiny

Note: Difference in union v memcpy() should sizeof double != sizeof uint64_t - but I take their equal size is taken as a given. – Bobbobb

@chux, in the memcpy() the number of bytes specified is the sizeof the destination so the max number of bytes copied depends on the size of the destination. If the source type is different in a union you will still have the same issue of interpretation of the bytes when you access the bytes using a type different from the type used to store those bytes. – Meiny

@Emission Of course the memcpy is safe. But for a compiler pushing a uint64_t on the stack for printf to know it should be aligned on a double boundary, it would have to inspect the format string, push every varargs argument at the lcm of all alignment boundaries, or dynamically check each printf argument for alignment and copy as needed on the fly. All that seems very un-C-like. But as I said, I'm not an expert. – Randolphrandom

The part about memcpy() in paragraph 6 doesn't apply in this situation, since uint64_t bits has a declared type. – Carduaceous

@EOF: good point. I edited the question. – Emission

Relevant: blog.regehr.org/archives/959 John Regehr says that in some cases, memcpy is his preferred way of doing it and enables the majority of compilers to generate optimal object code. – Physique

Suggest uint64_t bits = *pi; --> uint64_t bits = *pbits; – Bobbobb

@ChrisBeck: What would facilitate optimal code would be if the Standard were to require that if a pointer cast from one specific type to another, then all accesses made via the cast pointer which precede any other accesses to the same storage must be recognized as potential accesses of either the old or new type. The simplest way of achieving on traditional compilers would be to say that if e.g. an int* is cast to a short* then any cached int values which might be identified by that pointer must be flushed. Code reordering complicates things, but the proper remedy for that would be... – Preponderate

...to use things like restrict. I really doubt that most of the people voting for the languages in the standard would expect that quality compilers would not recognize that something like ((uint16_t*)someUint32Ptr)[IS_BIG_ENDIAN]; might modify a value of type uint32_t [and might be the most efficient way to clear the lower half of it], or would think that optimization is furthered by requiring programmers to write code that would force a compiler to assume a pointer might alias almost anything, anywhere, of any type. – Preponderate

Answer 1

There are two cases to consider: memcpy()ing into an object that has a declared type, and memcpy()ing into an object that does not.

In the second case,

double d = 1234.5678;
void *p = malloc(sizeof(double));
assert(p);
uint64_t *pbits = memcpy(p, &d, sizeof(double));
uint64_t bits = *pbits;
printf("the representation of %g is %08"PRIX64"\n", d, bits);

The behavior is indeed undefined, since the effective type of the object pointed to by p will become double, and accessing an object of effective type double though an lvalue of type uint64_t is undefined.

On the other hand,

double d = 1234.5678;
uint64_t bits;
memcpy(&bits, &d, sizeof bits);
printf("the representation of %g is %08"PRIX64"\n", d, bits);

is not undefined. C11 draft standard n1570:

7.24.1 String function conventions
3 For all functions in this subclause, each character shall be interpreted as if it had the type unsigned char (and therefore every possible object representation is valid and has a different value).

And

6.5 Expressions
7 An object shall have its stored value accessed only by an lvalue expression that has one of the following types: 88)

— a type compatible with the effective type of the object,
— a qualified version of a type compatible with the effective type of the object,
— a type that is the signed or unsigned type corresponding to the effective type of the object,
— a type that is the signed or unsigned type corresponding to a qualified version of the effective type of the object,
— an aggregate or union type that includes one of the aforementioned types among its members (including, recursively, a member of a subaggregate or contained union), or
— a character type.

Footnote 88) The intent of this list is to specify those circumstances in which an object may or may not be aliased.

So the memcpy() itself is well-defined.

Since uint64_t bits has a declared type, it retains its type even though its object representation was copied from a double.

As chqrlie points out, uint64_t cannot have trap representations, so accessing bits after the memcpy() is not undefined, provided sizeof(uint64_t) == sizeof(double). However, the value of bits will be implementation-dependent (for example due to endianness).

Conclusion: memcpy() can be used for type-punning, provided that the destination of the memcpy() does have a declared type, i.e. is not allocated by [m/c/re]alloc() or equivalent.

Carduaceous

Is memcpy used for type-punning concerned about the source type? Can the source have no declared type (m/c/re/alloced) providing that we know its effective type? – Ehrenburg

@SomeName Considering that the act of storing to an object causes it to have an effective type for further (reading) accesses, even if it had no effective type before, the only way to copy from an object without an effective type would be to access an object that is neither initialized nor written to, which would make its value indeterminate and the read of it undefined behavior. – Carduaceous

@EOF: The Effective Type rules as written only appear simple because they make no effort to avoid creating ambiguous and unworkable corner cases. Further, they use an abstraction model which is bears no relation to the things compilers actually care about. The situations where blind allowance for aliasing would be needlessly expensive are generally those where storage is accessed via one type, accessed via pointer that has no discernible relationship with that type, and then accessed again as the first type using a pointer with no discernible relationship to any other type. – Preponderate

@EOF: If a compiler which is processing an access of the intermediate type doesn't know about both preceding and following accesses using some other type, it wouldn't have any reason to care about how such accesses might have affected the object's "Effective Type". In cases where it does know about such accesses, but a pointer is converted from the earlier type to another type between the last access using the earlier type and the first with the new one, treating the conversion as a barrier to the consolidation of loads using the original type would be unlikely... – Preponderate

...to impede useful optimizations. When one factors in questions about interactions involving things like memcpy, direct operations on struct and union member lvalues, and use of address-of on such lvalues, pretending that storage holds Effective Types ends up being a far less useful model than would be one based upon the sequences of actions that access storage or derive pointers to it. – Preponderate

Answer 2

You propose 3 ways which all have different problems with C standard.

1. standard library memcpy

   double d = 1234.5678;
   uint64_t bits;
   memcpy(&bits, &d, sizeof bits);
   printf("the representation of %g is %08"PRIX64"\n", d, bits);
   

   The memcpy part is legal (provided in your implementation sizeof(double) == sizeof(uint64_t) which is not guaranteed per standard): you access two objects through char pointers.

   But the printf line is not. The representation in bits is now a double. it might be a trap representation for an uint64_t, as defined in 6.2.6.1 General §5

   Certain object representations need not represent a value of the object type. If the stored value of an object has such a representation and is read by an lvalue expression that does not have character type, the behavior is undefined. If such a representation is produced by a side effect that modifies all or any part of the object by an lvalue expression that does not have character type, the behavior is undefined. Such a representation is called a trap representation.

   And 6.2.6.2 Integer types says explicitely

   For unsigned integer types other than unsigned char, the bits of the object representation shall be divided into two groups: value bits and padding bits ... The values of any padding bits are unspecified.⁵³

   With note 53 saying:

   Some combinations of padding bits might generate trap representations,

   If you know that in your implementation there are no padding bits (still never seen one...) every representation is a valid value, and the print line becomes valid again. But it is only implementation dependant and can be undefined behaviour in the general case

2. union

   union { double d; uint64_t i; } u;
   u.d = 1234.5678;
   printf("the representation of %g is %08"PRIX64"\n", d, u.i);
   

   The members of the union do not share a common subsequence, and you are accessing a member which is not the last value written. Ok common implementation will give expected results but per standard it is not explicitely defined what should happen. A footnote in 6.5.2.3 Structure and union members §3 says that if leads to same problems as previous case:

   If the member used to access the contents of a union object is not the same as the member last used to store a value in the object, the appropriate part of the object representation of the value is reinterpreted as an object representation in the new type as described in 6.2.6 (a process sometimes called "type punning"). This might be a trap representation.

3. custom memcpy

   Your implementation only does character accesses which is always allowed. It is exactly the same thing as the first case: implementation defined.

The only way that would be explicitely defined per standard would be to store the representation of the double in an char array of the correct size, and then display the bytes values of the char array:

double d = 1234.5678;
unsigned char bits[sizeof(d)];
memcpy(&bits, &d, sizeof(bits));
printf("the representation of %g is ", d);
for(int i=0; i<sizeof(bits); i++) {
    printf("%02x", (unsigned int) bits[i]);
}
printf("\n");

And the result will only be useable if the implementation uses exactly 8 bits for a char. But it would be visible because it would display more than 8 hexa digits if one of the bytes had a value greater than 255.

---

All of the above is only valid because bits has a declared type. Please see @EOF's answer to understand why it would be different for an allocated object

Sounder

The union case is explicitly (but sadly non-normatively) documented to be implementation-defined in C11 draft standard n1570 6.5.2.3 Structure and union members [Footnote] 95 If the member used to read the contents of a union object is not the same as the member last used to store a value in the object, the appropriate part of the object representation of the value is reinterpreted as an object representation in the new type as described in 6.2.6 (a process sometimes called ‘‘type punning’’). This might be a trap representation. – Carduaceous

The one part that is missing here now is that the behavior is explicitly undefined in the memcpy() variant if the object that is copied to is allocated by [m/c/re]alloc(). Have a look at my answer for why that is. – Carduaceous

@EOF: I agree with you but it is a different problem, that I see as loosely related to xalloc. If I change your example to void *p = malloc(sizeof(double)); uint64_t *pbits = p; *pbits = 0 // effective type is now uint64_t; memcpy(p, &d, sizeof(double));, I am only left with the implementation defined behaviour because of possible trap representation. Once an allocated variable receives a type it keeps it until it is released. – Sounder

No. Remember that C11 draft standard n1570 : 6.5 Expressions 6 [...]If a value is copied into an object having no declared type using memcpy or memmove, or is copied as an array of character type, then the effective type of the modified object for that access and for subsequent accesses that do not modify the value is the effective type of the object from which the value is copied, if it has one. The memcpy() does modify the value, so the effective type of the object changes again. – Carduaceous

Actually, the relevant part for your specific example is a sentence earlier: If a value is stored into an object having no declared type through an lvalue having a type that is not a character type, then the type of the lvalue becomes the effective type of the object for that access and for subsequent accesses that do not modify the stored value. [...] , but the point is the same. – Carduaceous

In the rationale for the C99 standard 3. Terms and definitions The definition of object does not employ the notion of type. Thus an object has no type in and of itself. However, since an object may only be designated by an lvalue (see §6.3.2.1), the phrase “the type of an object” is taken to mean, here and in the Standard, “the type of the lvalue designating this object,” and “the value of an object” means “the contents of the object interpreted as a value of the type of the lvalue designating the object.” Thus an object's lifetime is not bound to its type. – Carduaceous

@EOF: That's an interesting question, because the standard says nothing on how an allocated object can be built. I've recently asked a question on this and 6 days later proposed an answer. Please read it and comment either the question or the answer if you can sched some light on that. But for what remains, I've now understood your point and linked to your answer. – Sounder

The problem is not trap representations. As a matter of fact, uint64_t cannot have trap representations since the representation must be 64 value bits and no padding bits. The problem is whether the compiler is smart/obnoxious enough to assume that bits was not changed since it was last read or written, assumption that could be supported by the absence of any write access to it through an appropriate type. memcpy obviously must be assumed to modify bits since a non const pointer to it is passed to memcpy, converted to void*. – Emission

Answer 3

I read paragraph 6 as saying that using the memcpy() function to copy a series of bytes from one memory location to another memory location can be used for type punning just as using a union with two different types can be used for type punning.

The first mention of using memcpy() indicates that if it copies the specified number of bytes and that those bytes will have the same type as the variable at the source destination when that variable (lvalue) was used to store the bytes there.

In other words if you have a variable double d; and you then assign a value to this variable (lvalue) the type of the data stored in that variable is type double. If you then use the memcpy() function to copy those bytes to another memory location, say a variable uint64_t bits; the type of those copied bytes is still double.

If you then access the copied bytes through the destination variable (lvalue), the uint64_t bits; in the example, then the type of that data is seen as the type of the lvalue used to retrieve the data bytes from that destination variable. So the bytes are interpreted (not converted but interpreted) as the destination variable type rather than the type of the source variable.

Accessing the bytes through a different type means the bytes are now interpreted as the new type even though the bytes have not actually changed in any way.

This is also the way a union works. A union does not do any kind of conversion. You store bytes into a union member which is of one type and then you pull the same bytes back out through a different union member. The bytes are the same however the interpretation of the bytes depends on the type of the union member that is used to access the memory area.

I have seen the memcpy() function used in older C source code to help divide up a struct into pieces by using struct member offset along with the memcpy() function to copy portions of the struct variable into other struct variables.

Because the type of the source location used in the memcpy() is the type of the bytes stored there the same kinds of problems that you can run into with the use of a union for punning also apply to using memcpy() in this way such as the Endianness of the data type.

The thing to remember is that whether using a union or using the memcpy() approach the type of the bytes copied are the type of the source variable and when you then access the data as another type, whether through a different member of a union or through the destination variable of the memcpy() the bytes are interpreted as the type of the destination lvalue. However the actual bytes are not changed.

Meiny

I wish your interpretation is correct, but If paragraph 6 says the effective type of the representation at the destination of memcpy is that of the source, paragraph 7 says accessing it with the type of the destination is not one of the enumerated cases, because the type of the lvalue expression, uint64_t, is not compatible with the effective type of the object whose value is stored, namely double, and is not a character type. – Emission

Answer 4

CHANGED--SEE BELOW

While I have never observed a compiler to interpret a memcpy of non-overlapping source and destination as doing anything that would not be equivalent to reading all of the bytes of the source as a character type and then writing all of the bytes of the destination as a character type (meaning that if the destination had no declared type, it would be left with no effective type), the language of the Standard would allow obtuse compilers to make "optimizations" which--in those rare instances where a compiler would be able to identify and exploit them--would be more likely to break code which would otherwise work (and would be well-defined if the Standard were better written) than to actually improve efficiency.

As to whether that means that it's better to use memcpy or a manual byte-copy loop whose purpose is sufficiently well-disguised as to be unrecognizable as "copying an array of character type", I have no idea. I would posit that the sensible thing would be to shun anyone so obtuse as to suggest that a good compiler should generate bogus code absent such obfuscation, but since behavior that would have been considered obtuse in years past is presently fashionable, I have no idea whether memcpy will be the next victim in the race to break code which compilers had for decades treated as "well-defined".

UPDATE

GCC as of 6.2 will sometimes omit memmove operations in cases where it sees that the destination and source identify the same address, even if they are pointers of different types. If storage which had been written as the source type is later read as the destination type, gcc will assume that the latter read cannot identify the same storage as the earlier write. Such behavior on gcc's part is justifiable only because of the language in the Standard which allows the compiler to copy the Effective Type through the memmove. It's unclear whether that was an intentional interpretation of the rules regarding memcpy, however, given that gcc will also make a similar optimization in some cases where it is clearly not allowed by the Standard, e.g. when a union member of one type (e.g. 64-bit long) is copied to a temporary and from there to a member of a different type with the same representation (e.g. 64-bit long long). If gcc sees that the destination will be bit-for-bit identical to the temporary, it will omit the write, and consequently fail to notice that the effective type of the storage was changed.

Preponderate

not sure what this has to do with the question. there are two memory copying functions available, memcpy() for non-overlapping memory regions and memmov() for possibly overlapping memory regions. I assume this is to allow library optimizations for non-overlapping memory regions which are dangerous for overlapping memory regions. https://mcmap.net/q/24001/-memcpy-vs-memmove – Meiny

@RichardChambers: If the destination of a memcpy or memmove does not have a declared type, a compiler is allowed to regard as unreachable any code which would try to read the destination using any type which differs from the source; a compiler can also do that if it thinks data is being copied as a character array (whatever that means). If data is copied as a sequence of totally-unrelated character operations, however, a compiler is not allowed to make such an inference, but the Standard isn't clear how unrelated the operations must be. – Preponderate

Answer 5

It might give the same result, but the compiler does not need to guarantee it. So you simply cannot rely on it.

Tumefaction

Hence the question: Why would it not? Paragraph 6 is obscure to me. Can someone explain? What if memcpy() is implemented as reading and writing bytes, does the last item in paragraph 7 save the day? – Emission

@chqrlie: The behavior was historically 100% reliable, but major compiler writers are so desperate to eke an extra percentage point or two out of benchmark performance that they're willing to throw decades of precedent out the window (and force programmers who aren't writing benchmarks to write less efficient code) to do it. – Preponderate