Replace default STL allocator
Asked Answered
S

3

27

I have the source for a large (>250 files) library that makes heavy use of STL containers and strings. I need to run it in an embedded environment with limited heap, so I want to ensure that this library itself is limited in its heap usage.

The obvious solution is to create an allocator, but modifying the entire code base to include the allocator template parameter is a big job of last resort, and undesirable in case I ever want to take a new version of the source. Globally replacing new and delete is not feasible since that affects the entire image, not just this library.

My next thought was a stupid C macro trick, but that doesn't seem like it would be possible, although I admit to not being a clever macro author.

So I thought "is there a compiler or pragma switch to specify the allocator<> class at compile time"? But I'm open for anything.

The next question I'll ask, if anyone can come up with a solution, is how to do the same for new/delete within the set of files comprising this library.

I'm using the gcc 3.4.4 toolchain to run this under Cygwin, with a target of VxWorks, if that sparks any ideas.

Stein answered 23/11, 2011 at 18:53 Comment(8)
gcc 3.4.4 That's old enough to be labeled 'completely broken'. Any reason why you don't switch to a newer version?Overreact
I think in GCC that's just a simple macro somewhere in the interna, which you should be able to switch. The default is the "new allocator", but GCC comes with several alternatives, like the "malloc allocator" and a pool allocator and whatnot.Retread
Suppose you could create a limited heap and then overload operator new for this library only, so all operator new calls in the library went to your limited heap. What behavior would you want when the limited heap is full and the overloaded operator new is called? Would you really want the operator new to fail under those circumstances? I don't believe you would, unless that library can still function when operator new is failing. Maybe if the library is a database cache, or something like that. What does the library do?Miguelmiguela
The way we did it is by creating a template class that wraps each container we use. Take a look at a question I posted: stackoverflow.com/q/8248076/1037208 . It saves you some trouble, and makes it harder to make a mistake. I am not entirely happy with our solution, so if someone answers your question with a better solution I would probably consider revising our solution.Bren
@sehe, the reason for 3.4.4 is that our target system is satellite hardware running VxWorks, and that toolchain is still on 3.4.4. I'm not privy to the 'why', just the requirement.Stein
@James Brock, my library is a fault analysis package. You describe a model of your system to the package, and then provide realtime sensor inputs. It determines whether the system is in a 'consistent' state, and tries to find the faulty component when not. It creates a number of 'fault candidates' to search the fault space. The mem usage is not deterministic. This capability is desirable, but not critical. So if it throws 'out of memory', we don't mind starting it over at time t_n.Stein
@Oren S, this still requires me to touch many of the 250+ files, and kills me when the vendor releases a new version. If I'm forced to do that, then I'll do it. But I'm still looking for 'clever'. I wonder how much damage I could do writing a script to automate the changes? That would probably take longer than the changes!Stein
@Reilly: that sounds like a good enough reason :) Provided my answerOverreact
S
10

I resorted to the preprocessor to get a possible solution, although it currently relies upon the GCC 3.4.4 implementation to work.

The GCC <memory> implementation includes the file <bits/allocator.h>, which in turn includes another file <bits/c++allocator.h>, which defines a macro that defines the class implementing the default allocator base class.

Since is found in a platform-dependent path (/lib/gcc/i686-pc-cygwin/3.4.4/include/c++/i686-pc-cygwin/bits), I don't feel (very) dirty in supplanting it with my own "platform-dependent" implementation.

So I just create a folder bits/ in the root of my source's include path, and then create the file c++allocator.h in that folder. I define the required macro to be the name of my allocator class and it works like a charm, since gcc searches my include paths prior to searching the system includes.

Thanks for all your responses. I think I can go with this "solution", which will only work as long as I'm using 3.4.4 probably.

Stein answered 28/11, 2011 at 16:15 Comment(1)
any thoughts about how I do this in Visual Studio? I tried to locate the default allocator header file, but got lost :) thanks!Bren
O
6

You could benefit from using EASTL (Enterprise Arts STL (partial) implementation)

EASTL -- Electronic Arts Standard Template Library

This was intended for embedded/game development, in environments where global heap is really scarce, non-existent or problematic in general.

The allocator model of EASTL was inspired on (or resembles?) the ideas in the famous Towards a Better Allocator Model publication (PDF).

The EASTL lends itself well for custom allocators. In fact, it doesn't ship with an allocator, so providing (a minimal) one is required to even get you application to link.

Here is the github repo for EASTL: https://github.com/electronicarts/EASTL

Overreact answered 25/11, 2011 at 14:14 Comment(1)
This is a possible solution, although I'd have to edit either EASTL or my library due to std vs eastl namespaces.Stein
C
3

So I thought "is there a compiler or pragma switch to specify the allocator<> class at compile time"? But I'm open for anything.

No there isn't.

Take a look here.

Allocators are a template argument in every stl container. You will need to change them. I have done the same thing in the past, when working on embedded. I could give you some pointers if you like :

Basic template allocator :

namespace PFM_MEM {
    template <class T>
    class CTestInstAllocator {
    public:
        // type definitions
        typedef size_t    size_type;
        typedef ptrdiff_t difference_type;
        typedef T*        pointer;
        typedef const T*  const_pointer;
        typedef T&        reference;
        typedef const T&  const_reference;
        typedef T         value_type;

        // rebind CTestInstAllocator to type U
        template <class U>
        struct rebind {
            typedef CTestInstAllocator<U> other;
        };

        // return address of values
        pointer address (reference value) const {
            return &value;
        }
        const_pointer address (const_reference value) const {
            return &value;
        }

        /* constructors and destructor
        * - nothing to do because the CTestInstAllocator has no state
        */
        CTestInstAllocator() {
        }
        CTestInstAllocator(const CTestInstAllocator&) {
        }
        template <class U>
        CTestInstAllocator (const CTestInstAllocator<U>&) {
        }
        ~CTestInstAllocator() {
        }

        // return maximum number of elements that can be allocated
        size_type max_size () const {
            return std::numeric_limits<size_t>::max() / sizeof(T);
        }

        // pvAllocate but don't initialize num elements of type T by using our own memory manager
        pointer allocate (size_type num) {
            /**
            * pvAllocate memory custom memory allocation scheme
            */
            return(pointer)(CPfmTestInstMemManager::pvAllocate(num*sizeof(T)));
        }
        // initialize elements of allocated storage p with value value
        void construct (pointer p, const T& value) {
            // initialize memory with placement new
            new((void*)p)T(value);
        }

        // destroy elements of initialized storage p
        void destroy (pointer p) {
            // destroy objects by calling their destructor
            p->~T();
        }
        // vDeallocate storage p of deleted elements
        void deallocate (pointer p, size_type num) {
            /**
            *Deallocate memory with custom memory deallocation scheme
            */
            CPfmTestInstMemManager::vDeallocate((void*)p);
        }
    };

    // return that all specializations of this CTestInstAllocator are interchangeable
    template <class T1, class T2>
    bool operator== (const CTestInstAllocator<T1>&,
        const CTestInstAllocator<T2>&) {
            return true;
    }
    template <class T1, class T2>
    bool operator!= (const CTestInstAllocator<T1>&,
        const CTestInstAllocator<T2>&) {
            return false;
    }
}

Take particular note at these lines :

/**
* pvAllocate memory custom memory allocation scheme
*/
return(pointer)(CPfmTestInstMemManager::pvAllocate(num*sizeof(T)));

// vDeallocate storage p of deleted elements
void deallocate (pointer p, size_type num) {
/**
*Deallocate memory with custom memory deallocation scheme
*/
CPfmTestInstMemManager::vDeallocate((void*)p);

Here is the place where you call your new and delete which work on your heap.

I could provide you with an example of how to construct some basic memory manager to help you further.

Clein answered 23/11, 2011 at 19:11 Comment(4)
So you are saying there is no way to tell STL to use a custom allocator globally; you must make sure that when you use an stl type, you define it with the custom allocator. right?Bren
@OrenS. Well for specific compilers, there may be an option but I am talking from my experience, alone. I was in the same spot as the OP and had to do this. Edit corrected answer.Clein
@FailedDev, I saw that page before I posted here and was hopeful. If I could figure out how to make my arbitrary allocator an 'extension' allocator, I would be golden. I just don't know how to do that, either. If you know that, I'll name my next dog "FailedDev"...Stein
Has anything changed with this topic and c++ 17? Is t here now a common way to replacem the global allocator ?Venicevenin

© 2022 - 2024 — McMap. All rights reserved.