Problems with LD_PRELOAD and calloc() interposition for certain executables

Question

Relating to a previous question of mine

I've successfully interposed malloc, but calloc seems to be more problematic.

That is with certain hosts, calloc gets stuck in an infinite loop with a possible internal calloc call inside dlsym. However, a basic test host does not exhibit this behaviour, but my system's "ls" command does.

Here's my code:

// build with: g++ -O2 -Wall -fPIC -ldl -o libnano.so -shared Main.cc
#include <stdio.h>
#include <dlfcn.h>

bool gNanoUp = false;// global

// Function types
typedef void* (*MallocFn)(size_t size);
typedef void* (*CallocFn)(size_t elements, size_t size);

struct MemoryFunctions {
    MallocFn   mMalloc;
    CallocFn   mCalloc;
};

MemoryFunctions orgMemFuncs;

// Save original methods.
void __attribute__((constructor)) __nano_init(void) {
    fprintf(stderr, "NANO: init()\n");

    // Get address of original functions
    orgMemFuncs.mMalloc = (MallocFn)dlsym(RTLD_NEXT, "malloc");
    orgMemFuncs.mCalloc = (CallocFn)dlsym(RTLD_NEXT, "calloc");

    fprintf(stderr, "NANO: malloc() found @%p\n", orgMemFuncs.mMalloc);
    fprintf(stderr, "NANO: calloc() found @%p\n", orgMemFuncs.mCalloc);

    gNanoUp = true;
}

// replacement functions
extern "C" {
    void *malloc(size_t size) {
        if (!gNanoUp) __nano_init();
        return orgMemFuncs.mMalloc(size);
    }

    void* calloc(size_t elements, size_t size) {
        if (!gNanoUp) __nano_init();
        return orgMemFuncs.mCalloc(elements, size);
    }
}

Now, When I do the following, I get an infinite loop followed by a seg fault, eg:

% setenv LD_PRELOAD "./libnano.so"
% ls
...
NANO: init()
NANO: init()
NANO: init()
Segmentation fault (core dumped)

However if I comment out the calloc interposer, it almost seems to work:

% setenv LD_PRELOAD "./libnano.so"
% ls
NANO: init()
NANO: malloc() found @0x3b36274dc0
NANO: calloc() found @0x3b362749e0
NANO: init()
NANO: malloc() found @0x3b36274dc0
NANO: calloc() found @0x3b362749e0
<directory contents>
...

So somethings up with "ls" that means init() gets called twice.

EDIT Note that the following host program works correctly - init() is only called once, and calloc is successfully interposed, as you can see from the output.

// build with: g++ test.cc -o test
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[]) {

    void* p = malloc(123);
    printf("HOST p=%p\n", p);
    free(p);

    char* c = new char;
    printf("HOST c=%p\n", c);
    delete c;

    void* ca = calloc(10,10);
    printf("HOST ca=%p\n", ca);
    free(ca);
}

% setenv LD_PRELOAD "./libnano.so"
% ./test 
NANO: init()
NANO: malloc() found @0x3b36274dc0
NANO: calloc() found @0x3b362749e0
HOST p=0x601010
HOST c=0x601010
HOST ca=0x601030

Sachikosachs

check this solution: blog.bigpixel.ro/2010/09/interposing-calloc-on-linux It works even tho it should not... – Macdermot

Answer 1

With regard to __nano_init() being called twice: You've declared the function as a constructor, so it's called when the library is loaded, and it's called a second time explicitly when your malloc() and calloc() implementations are first called. Pick one.

With regard to the calloc() interposer crashing your application: Some of the functions you're using, including dlsym() and fprintf(), may themselves be attempting to allocate memory, calling your interposer functions. Consider the consequences, and act accordingly.

Konstanze

See my edit with an example host that appears to work correctly. – Sachikosachs

The second part of your answer is correct. The first part about the constructor isn't, because the replacement happens before the constructor - only for certain executables, which end up calling the init twice directly (not via constructor). – Sachikosachs

It's possible that what was happening the first time around was that some executables would call malloc()/calloc() in their own initializers before yours ran. – Konstanze

Answer 2

I know I am a bit late (6 years). But I wanted to override calloc() today and faced a problem because dlsym() internally uses calloc(). I solved it using a simple technique and thought of sharing it here:

static unsigned char buffer[8192];

void *calloc(size_t nmemb, size_t size)
{
    if (calloc_ptr == NULL) // obtained from dlsym
            return buffer;

    init(); // uses dlsym() to find address of the real calloc()

    return calloc_ptr(len);
}

void free(void *in)
{
    if (in == buffer)
        return;

    free_ptr(in);
}

buffer satisfies the need of dlsym() till the real calloc() has been located and my calloc_ptr function pointer initialized.

Mycobacterium

Answer 3

Using dlsym based hooking can result in crashes, as dlsym calls back into the memory allocator. Instead use malloc hooks, as I suggested in your prior question; these can be installed without actually invoking dlsym at all.

Protege

Ok I'm just going to assume in this case dlsym won't work - despite the many examples all over the internet that show replacing memory functions with it. I'll try to formulate a new SO question showing one of these examples. – Sachikosachs

@Justicle, just because it might have worked if you only hooked some of the functions, or maybe worked at some point in the past, doesn't mean it'll work with newer versions of glibc. Use the hook functions; they're the only ones that are guaranteed to work properly. – Protege

Also, there's no mention of problems with dlsym in any examples I can find, and until now I haven't found anyone stating why hooks would be better than preloading. I guess I just can't believe I'm the first person to try this ... – Sachikosachs

Perhaps they're not trying with calloc? In any case, it's clear enough that it's not working, and glibc has this nice feature that does what you want in a supported manner, so just use it already :) – Protege

Ok the penny just dropped :D replacement allocators that work via LD_PRELOAD provide a complete replacement for the memory functions, not just a pass through. So 1) There's no question "my_awesum_calloc" implementation being called by libc functions like dlsym. and 2) The dlsym call is not necessary, because you don't need the old implementation. – Sachikosachs

@Justicle, and it provides a direct function pointer to the original implementation without bothering with dlsym :) [note that dlsym and friends will call back into your replacement calloc - but since you have the pointer to the original without using dlsym, it's not a problem] – Protege

Hooks are deprecated, as they are not thread safe. See NOTES section in this link man7.org/linux/man-pages/man3/malloc_hook.3.html. Furthermore, I do not think hooks cover calloc. – Emileemilee

Answer 4

You can get away with a preliminary poor calloc that simply returns NULL. This actually works on Linux, YMMV.

static void* poor_calloc(size_t nmemb, size_t size)
{
    // So dlsym uses calloc internally, which will lead to infinite recursion, since our calloc calls dlsym.
    // Apparently dlsym can cope with slightly wrong calloc, see for further explanation:
    // http://blog.bigpixel.ro/2010/09/interposing-calloc-on-linux
    return NULL; // This is a poor implementation of calloc!
}

Restaurateur

Answer 5

You can also use sbrk to allocate the memory for "poor calloc".

Pubis