Is there an equivalent of set_new_handler() for malloc() failures?

Question

In C++, you can arrange for a function to be called whenever new fails. Is there a way to have a function called whenever malloc fails? Assume that malloc is being called from third-party libraries that I don't want to change.

I don't think there's a portable answer, so I'll happily accept platform-specific ones. I'm using Linux+uclibc on some platforms and Linux+glibc on others. I'm planning to use setrlimit to limit the amount of memory malloc can return.

Ptarmigan

We're assuming you've turned overcommit off. Right? – Buskus

Conrad: I hope so, but I don't own the entire system, I'm just providing an app. I have been promised a certain amount of RAM by the people designing the system, and in return I've got to promise to stay within that limit. (This is an embedded device without swap, so people get quite serious about memory budgets and making sure everything fits). – Ptarmigan

Answer 1

malloc returns NULL if it fails. You should be handling that, and other failures from CRT memory functions (realloc especially is easy to get wrong).

In the general case, I think you'd have to wrap all CRT usage of memory in functions of your own devising to redirect on error.

On Windows you can hook into the CRT using Allocation Hook Functions, that might be a way to do what you want. This gives you a hook for handling CRT calls via logic for onalloc, onrealloc, onfree, effectively.

I make no guarantees since I am a Windows guy but it looks like malloc_hook on Linux offers the same as that Windows hook allows. These methods should enable you to capture all CRT memory calls without changing code in the third-party libraries, assuming they all use the same CRT at runtime - always a good idea, but not guaranteed on Windows at least...

Gasteropod

Yes, __malloc_hook() is the correct function on Linux (glibc at least - I don't know about uclibc). – Custodial

From a more recent man page: "The use of these hook functions is not safe in multithreaded programs, and they are now deprecated. Programmers should instead preempt calls to the relevant functions by defining and exporting functions such as "malloc" and "free"." kernel.org/doc/man-pages/online/pages/man3/malloc_hook.3.html – Ptarmigan