Can realloc fail (return NULL) when trimming?

Question

If do the next:

int* array = malloc(10 * sizeof(int));

and them I use realloc:

array = realloc(array, 5 * sizeof(int));

On the second line (and only it), can it return NULL?

Okhotsk

somewhat related: #1736933 – Plenty

Answer 1

Yes, it can. There are no implementation guarantees on realloc(), and it can return a different pointer even when shrinking.

For example, if a particular implementation uses different pools for different object sizes, realloc() may actually allocate a new block in the pool for smaller objects and free the block in the pool for larger objects. Thus, if the pool for smaller objects is full, it will fail and return NULL.

---

Or it may simply decide it's better to move the block

I just used the following program to get size of actually allocated memory with glibc:

#include <stdlib.h>                                                          
#include <stdio.h>                                                           

int main()                                                                   
{                                                                            
    int n;                                                                   

    for (n = 0; n <= 10; ++n)                                                
    {                                                                        
        void* array = malloc(n * sizeof(int));                               
        size_t* a2 = (size_t*) array;                                        

        printf("%d -> %zu\n", n, a2[-1]);                                    
    }                                                                        
}

and for n <= 6, it allocates 32 bytes, and for 7-10 it is 48.

So, if it shrank int[10] to int[5], the allocated size would shrink from 48 to 32, effectively giving 16 free bytes. Since (as it just has been noted) it won't allocate anything less than 32 bytes, those 16 bytes are lost.

If it moved the block elsewhere, the whole 48 bytes will be freed, and something could actually be put in there. Of course, that's just a science-fiction story and not a real implementation ;).

---

The most relevant quote from the C99 standard (7.20.3.4 The realloc function):

Returns

4 The realloc function returns a pointer to the new object (which may have the same value as a pointer to the old object), or a null pointer if the new object could not be allocated.

'May' is the key-word here. It doesn't mention any specific circumstances when that can happen, so you can't rely on any of them, even if they sound obvious at a first glance.

---

By the way, I think you could consider realloc() somewhat deprecated. If you'd take a look at C++, the newer memory allocation interfaces (new / delete and allocators) don't even support such a thing. They always expect you to allocate a new block. But that's just a loose comment.

Gunwale

I must object to calling realloc deprecated just because C++ doesn't have an analogue in the new/delete regime. C++ is a very different language from C, and in particular, support for moving objects in C++ would require some way for the implementation to notify an object that it's being relocated and allow it to update its own internal references. C on the other hand doesn't automate or encapsulate any of this, so it's up to the caller (and thus perfectly fine) to be responsible for whether the object's contents need to be changed after realloc. – Condition

Generally, I find it a bit strange to reply to a C question with C++ code and C++ thoughts about deprecation. – Polivy

Indeed, I hadn't even bothered to read the code... That should really be fixed since this question is about C, not C++. – Condition

I'd prefer malloc_usable_size over a negative index. – Inchmeal

This looks incorrect to me, because there is memory available, which is the current allocation. If realloc() fails just because it wants to move memory somewhere else, then it's incorrect for it to return ENOMEM. It's not lack of memory, it's something else. – Indonesia

Answer 2

The other answers have already nailed the question, but assuming you know the realloc call is a "trimming", you can wrap it with:

void *safe_trim(void *p, size_t n) {
    void *p2 = realloc(p, n);
    return p2 ? p2 : p;
}

and the return value will always point to an object of size n.

In any case, since the implementation of realloc knows the size of the object and can therefore determine that it's "trimming", it would be pathologically bad from a quality-of-implementation standpoint not to perform the above logic internally. But since realloc is not required to do this, you should do it yourself, either with the above wrapper or with analogous inline logic when you call realloc.

Condition

Out of curiosity, is it practically useful? I'd consider that if realloc() just failed, there will be soon a random failing malloc()… – Noyade

Yes, I believe it is. Code that's trimming storage for an existing result may not be able to "back out" its progress on failure and report failure in a meaningful way to the higher-level code. Thus, it's very valuable to be able to write the code in a way that it cannot fail. Even if the next call to malloc is going to fail somewhere else, that will (in a robust program, at least) be at a point where the program can handle the failure case, back out any partial work, and report the error. – Condition

Is the old pointer still valid if realloc fails? I was under the impression that it frees it first, but I could be wrong. – Restharrow

Yes, of course it is. If it weren't, realloc would be utterly useless in robust programs. This is actually an extremely common form of memory leak (i.e. p=realloc(p,newsize); which loses the old memory if realloc fails). – Condition

@R..: Yes, you're right—I see that now after reading the man page for realloc(3). I've just fixed a leak in my code where I didn't free the old memory when realloc fails (although it's not extremely important since I call exit right after). – Restharrow

@R..: Is there reason why an even-remotely-decent-quality implementation that would be unable to usefully reduce the size of an allocation shouldn't simply ignore the request? The Standard makes no attempt to forbid "conforming" implementations that are of such poor quality as to be useless, but I see no reason programmers targeting good implementations should cater to the quirks of abysmal ones. – Brazell

@supercat: Considering it again, if the only way an implementation could reduce the size is by leaving the remainder permanently unusable, or unusable for the lifetime of the object, it would arguably be a higher-quality behavior to report failure to the caller and let it know it can still use the full original size rather than hiding the extra space. – Condition

@R..: The one realistic situation in which a quality allocator should fail to shrink an object would occur when the amount of contiguous space that would be freed up would be insufficient to hold a new allocation. If code repeatedly tries to shrink an allocation by one byte, some operations would succeed and some would fail, but the final size of the object would likely be within 16 or maybe 32 bytes of the last requested size. – Brazell

@supercat: That's not a realistic situation, it's a nonsensical one. If allocation granularity is, for example, 32 bytes, trying to resize an object of size 32 down to size 31 should not fail. There is no benefit whatsoever to having it do so. On the other hand, if an object has size 10000000 and you want to resize it down to size 10, but no memory is available in zones that support partitioning into small sizes, there is value in reporting failure rather than succeeding just by leaving the caller an object that wastes ~10MB of unusable space. – Condition

@R..: Some platforms may have an allocation granularity which is smaller than the size of an allocation header. A quality implementation which is asked to resize a 10000000 byte object to 10 bytes should always succeed. If an implementation that has a granularity of 4 bytes but uses 8-byte headers is asked to shrink a 10000000-byte allocation to 9999996 bytes, it would be reasonable for it to leave the allocation at 100000000 bytes. Even if asked to shrink it to 9999992 bytes, it might not be unreasonable for it to decide that the value of creating space for a zero-sized object... – Brazell

...would be unlikely to be sufficient to justify the effort. If the following object gets freed before the object that got realloc'ed, the zero-sized allocation might get consolidated into the resulting free space, and there may be some scenarios where that could end up being useful, but for the most part it would be better not to add such tiny chunks to the free list. – Brazell

@supercat: You're not making sense, and mixing up implementation details which are not visible to the application with properties that have severe effects on the application. – Condition

@R..: On a quality implementation, following a realloc() that shrinks a block, the block should be left with a size close to what's requested. If the block isn't shrunk very much, the application shouldn't care about whether the block was actually shrunk. – Brazell

Return NULL does not always mean failure. If n==0, it is possible that the old memory is freed and no longer valid. – Malcolmmalcom

@12431234123412341234123: Per the spec that's false, though many erroneous implementations did/do it. – Condition

@R..GitHubSTOPHELPINGICE Can you quote the part of the standard that forbids that? I found this in 7.22.3 Memory management functions - 1: If the size of the space requested is zero, the behavior is implementation-defined: either a null pointer is returned, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object. – Malcolmmalcom

@12431234123412341234123: 7.22.3.5 ¶4: "If memory for the new object cannot be allocated, the old object is not deallocated and its value is unchanged." However this was a topic of much controversy; see DR400 and glibc issue 12547. Note that the result of DR 400 allows the wrong behavior for future and deprecates realloc with size 0. – Condition

Note further that current POSIX (see pubs.opengroup.org/onlinepubs/9699919799/functions/realloc.html) does not allow the wrong behavior; I'm not sure if future is allowing it. – Condition

Answer 3

The language (and library) specification makes no such guarantee, just like it does not guarantee that a "trimming" realloc will preserve the pointer value.

An implementation might decide to implement realloc in the most "primitive" way: by doing an unconditional malloc for a new memory block, copying the data and free-ing the old block. Obviously, such implementation can fail in low-memory situations.

Outstand

Answer 4

Don't count on it. The standard makes no such provision; it merely states "or a null pointer if the new object could not be allocated".

You'd be hard-pressed to find such an implementation, but according to the standard it would still be compliant.

Amey

I believe you shouldn't be calling such an implementation braindead. It may be actually more optimal. – Noyade

@MichałGórny In my language "more optimal" is considered a pleonasm, so I would avoid saying it. But yes, I edited :-) – Amey

@cnicutar: For some reason, some people who write implementations seem to regard "clever" and "stupid" as antonyms. – Brazell

Answer 5

I suspect there may be a theoretical possibility for failure in the scenario you describe.

Depending on the heap implementation, there may be no such a thing as trimming an existing allocation block. Instead a smaller block is allocated first, then the data is copied from the old one, and then it's freed.

For instance this may be the case with bucket-heap strategy (used by some popular heaps, such as tcmalloc).

Dingbat

It would still be valid to just return the original pointer in this case. Whether that's more helpful, I'm not sure. Reporting the error is more informative and allows the caller to make the choice to use the existing oversized allocation, but it also has a high chance of breaking bad code that assumes a "trimming" realloc never fails. – Condition

tcmalloc can fail if it shrinks, check the source tcmalloc.cc function do_realloc() which is used in tc_realloc(), (github.com/gperftools/gperftools/blob/master/src/…) – Malcolmmalcom

@R..: The Standard really should define some standard macros to indicate how an implementation will behave in various corner cases, and allow code to refuse to run on wacky implementations, rather than requiring extra code to handle cases that shouldn't arise on quality implementations. Even better might be to have a more generalized allocation-control function with a parameter to indicate whether an allocation is expected to grow or shrink, and indicate whether relocation is acceptable. Implementations would not be allowed to ignore information about expectations, and have requests... – Brazell

...to expand a block if possible without relocation always fail, but quality implementations that optimize based upon information and requests they receive may be able to outperform those that don't. In any case, programmers targeting good implementations shouldn't have to bend over backward to accommodate bottom-of-the-barrel ones. – Brazell

Answer 6

A bit late, but there is at least one popular implementation which realloc() with a smaler size can fail: TCMalloc. (At least as far as i understand the code)

If you read the file tcmalloc.cc, in the function do_realloc_with_callback(), you will see that if you shrink enough (50% of alloced memory, otherwise it will be ignored), TCMalloc will alloc the new memory first (and possible fail) and then copy it and remove the old memory.

I do not copy the source code, because i am not sure if the copyrights (of TCMalloc and Stackoverflow) will allow that, but here is a link to the source (revision as at May 17, 2019).

Malcolmmalcom

Answer 7

realloc will not fails in shrinking the existing memory, so it will not return NULL. It can return NULL only if fails during expansion.

But shrinking can fail in some architecture, where realloc can be implemented in a different manner like allocating a smaller size memory separately and freeing the old memory to avoid fragmentation. In that case shrinking memory can return NULL. But its very rare implementation.

But its better to be in a safer side, to keep NULL checks after shrinking the memory also.

Raila

Is this implementation guaranteed? Or can an implementation still try and move the allocated memory (e.g. a "free" and "malloc") on a realloc and thus fail? – Plenty

So then the statement "will not fail" is incorrect/misleading :) – Plenty

In some RTOS architecutre, realloc can be implemented by free and malloc(smallersize) for avoiding fragmentation. – Raila

(I'm just pointing out that your first two sentences and the rest of the answer disagree. This is why it does not have any up-votes .. it either can fail or will never fail. Pick one.) – Plenty

If an architecture tries to shrink then it will not fail, but if it do malloc(smallersize) and free(oldblock) then it can fail(but this type of implementation is very rare). – Raila

What if you shrink it to zero? Then it will return NULL. – Crosslet