How to deal with symbol collisions between statically linked libraries?
Asked Answered
E

3

96

One of the most important rules and best practices when writing a library, is putting all symbols of the library into a library specific namespace. C++ makes this easy, due to the namespace keyword. In C the usual approach is to prefix the identifiers with some library specific prefix.

Rules of the C standard put some constraints on those (for safe compilation): A C compiler may look at only the first 8 characters of an identifier, so foobar2k_eggs and foobar2k_spam may be interpreted as the same identifiers validly – however every modern compiler allows for arbitrary long identifiers, so in our times (the 21st century) we should not have to bother about this.

But what if you're facing some libraries of which you cannot change the symbol names / idenfiers? Maybe you got only a static binary and the headers or don't want to, or are not allowed to adjust and recompile yourself.

Evslin answered 4/8, 2011 at 11:14 Comment(1)
See also: #6539001Stome
E
169

At least in the case of static libraries you can work around it quite conveniently.

Consider those headers of libraries foo and bar. For the sake of this tutorial I'll also give you the source files

examples/ex01/foo.h

int spam(void);
double eggs(void);

examples/ex01/foo.c (this may be opaque/not available)

int the_spams;
double the_eggs;

int spam()
{
    return the_spams++;
}

double eggs()
{
    return the_eggs--;
}

example/ex01/bar.h

int spam(int new_spams);
double eggs(double new_eggs);

examples/ex01/bar.c (this may be opaque/not available)

int the_spams;
double the_eggs;

int spam(int new_spams)
{
    int old_spams = the_spams;
    the_spams = new_spams;
    return old_spams;
}

double eggs(double new_eggs)
{
    double old_eggs = the_eggs;
    the_eggs = new_eggs;
    return old_eggs;
}

We want to use those in a program foobar

example/ex01/foobar.c

#include <stdio.h>

#include "foo.h"
#include "bar.h"

int main()
{
    const int    new_bar_spam = 3;
    const double new_bar_eggs = 5.0f;

    printf("foo: spam = %d, eggs = %f\n", spam(), eggs() );
    printf("bar: old spam = %d, new spam = %d ; old eggs = %f, new eggs = %f\n", 
            spam(new_bar_spam), new_bar_spam, 
            eggs(new_bar_eggs), new_bar_eggs );

    return 0;
}

One problem becomes apparent immediately: C doesn't know overloading. So we have two times two functions with identical name but of different signature. So we need some way to distinguish those. Anyway, lets see what a compiler has to say about this:

example/ex01/ $ make
cc    -c -o foobar.o foobar.c
In file included from foobar.c:4:
bar.h:1: error: conflicting types for ‘spam’
foo.h:1: note: previous declaration of ‘spam’ was here
bar.h:2: error: conflicting types for ‘eggs’
foo.h:2: note: previous declaration of ‘eggs’ was here
foobar.c: In function ‘main’:
foobar.c:11: error: too few arguments to function ‘spam’
foobar.c:11: error: too few arguments to function ‘eggs’
make: *** [foobar.o] Error 1

Okay, this was no surprise, it just told us, what we already knew, or at least suspected.

So can we somehow resolve that identifer collision without modifying the original libraries' source code or headers? In fact we can.

First lets resolve the compile time issues. For this we surround the header includes with a bunch of preprocessor #define directives that prefix all the symbols exported by the library. Later we do this with some nice cozy wrapper-header, but just for the sake of demonstrating what's going on were doing it verbatim in the foobar.c source file:

example/ex02/foobar.c

#include <stdio.h>

#define spam foo_spam
#define eggs foo_eggs
#  include "foo.h"
#undef spam
#undef eggs

#define spam bar_spam
#define eggs bar_eggs
#  include "bar.h"
#undef spam
#undef eggs

int main()
{
    const int    new_bar_spam = 3;
    const double new_bar_eggs = 5.0f;

    printf("foo: spam = %d, eggs = %f\n", foo_spam(), foo_eggs() );
    printf("bar: old spam = %d, new spam = %d ; old eggs = %f, new eggs = %f\n", 
           bar_spam(new_bar_spam), new_bar_spam, 
           bar_eggs(new_bar_eggs), new_bar_eggs );

    return 0;
}

Now if we compile this...

example/ex02/ $ make
cc    -c -o foobar.o foobar.c
cc   foobar.o foo.o bar.o   -o foobar
bar.o: In function `spam':
bar.c:(.text+0x0): multiple definition of `spam'
foo.o:foo.c:(.text+0x0): first defined here
bar.o: In function `eggs':
bar.c:(.text+0x1e): multiple definition of `eggs'
foo.o:foo.c:(.text+0x19): first defined here
foobar.o: In function `main':
foobar.c:(.text+0x1e): undefined reference to `foo_eggs'
foobar.c:(.text+0x28): undefined reference to `foo_spam'
foobar.c:(.text+0x4d): undefined reference to `bar_eggs'
foobar.c:(.text+0x5c): undefined reference to `bar_spam'
collect2: ld returned 1 exit status
make: *** [foobar] Error 1

... it first looks like things got worse. But look closely: Actually the compilation stage went just fine. It's just the linker which is now complaining that there are symbols colliding and it tells us the location (source file and line) where this happens. And as we can see those symbols are unprefixed.

Let's take a look at the symbol tables with the nm utility:

example/ex02/ $ nm foo.o
0000000000000019 T eggs
0000000000000000 T spam
0000000000000008 C the_eggs
0000000000000004 C the_spams

example/ex02/ $ nm bar.o
0000000000000019 T eggs
0000000000000000 T spam
0000000000000008 C the_eggs
0000000000000004 C the_spams

So now we're challenged with the exercise to prefix those symbols in some opaque binary. Yes, I know in the course of this example we have the sources and could change this there. But for now, just assume you have only those .o files, or a .a (which actually is just a bunch of .o).

objcopy to the rescue

There is one tool particularily interesting for us: objcopy

objcopy works on temporary files, so we can use it as if it were operating in-place. There is one option/operation called --prefix-symbols and you have 3 guesses what it does.

So let's throw this fella onto our stubborn libraries:

example/ex03/ $ objcopy --prefix-symbols=foo_ foo.o
example/ex03/ $ objcopy --prefix-symbols=bar_ bar.o

nm shows us that this seemed to work:

example/ex03/ $ nm foo.o
0000000000000019 T foo_eggs
0000000000000000 T foo_spam
0000000000000008 C foo_the_eggs
0000000000000004 C foo_the_spams

example/ex03/ $ nm bar.o
000000000000001e T bar_eggs
0000000000000000 T bar_spam
0000000000000008 C bar_the_eggs
0000000000000004 C bar_the_spams

Lets try linking this whole thing:

example/ex03/ $ make
cc   foobar.o foo.o bar.o   -o foobar

And indeed, it worked:

example/ex03/ $ ./foobar 
foo: spam = 0, eggs = 0.000000
bar: old spam = 0, new spam = 3 ; old eggs = 0.000000, new eggs = 5.000000

Now I leave it as an exercise to the reader to implement a tool/script that automatically extracts the symbols of a library using nm, writes a wrapper header file of the structure

/* wrapper header wrapper_foo.h for foo.h */
#define spam foo_spam
#define eggs foo_eggs
/* ... */
#include <foo.h>
#undef spam
#undef eggs
/* ... */

and applies the symbol prefix to the static library's object files using objcopy.

What about shared libraries?

In principle the same could be done with shared libraries. However shared libraries, the name tells it, are shared among multiple programs, so messing with a shared library in this way is not such a good idea.

You will not get around writing a trampoline wrapper. Even worse you cannot link against the shared library on the object file level, but are forced to do dynamic loading. But this deserves its very own article.

Stay tuned, and happy coding.

Evslin answered 4/8, 2011 at 11:14 Comment(13)
Impressive! Didn't expect it to be so easy with objcopy.Khanna
Did you just... answer your own question within 1 minute of asking it?Bickerstaff
@Alex B: This is an tutorial article and I followed a path suggested to me at meta.stackoverflow.com how one could place tutorials about (interesting?) questions and their solutions. The question about clashing libraries came up and I thought "hm, I know how to deal with this kind of solution", wrote an article and posted it here in form of Q&A. meta.stackexchange.com/questions/97240/…Evslin
@Evslin any idea about solving this problem for iOS libraries. As I found out, objcopy doesn't support iOS libraries :/Discord
@EgeAkpinar: Just read your comment today. Well, iOS is a very different beast, because, being a relative of MacOS X it makes heavy use of Objective-C. And within the runtime model of Objective-C there's more than just statically assigned symbol names. There are messages passed around, and those would require modification as well. I know too little about the internals of the Apple Obj-C runtime to give any valueable advice.Evslin
Here's another guy looking for help with doing this on iOS. Dropping a comment here hoping that someone comes along with an answer for objcopy on iOS.Arabesque
Is there any way to do it on Windows as easily as with objcopy? If not as easily, then how to do it at all?Beore
@astraujums: objcopy is available for Windows as well. A pre built binary is available as part of the MinGW compiler suite.Evslin
Blah blah blah objcopy --prefix-symbols ... +1!Contemporary
+1 for objcopy, Can you please look at this question and see if you have some say about it: #30080948Roving
@Shchvova: a) this is about renaming symbols if you don't have access to the full source code (you have only the headers). And b) I wrote this down, because someone sitting in the office next door did ask me about that and I thought what I wrote down then might be a nice fit here.Evslin
FYI, this works on very limited number of architectures. objcopy can't ARMCalamity
@Shchvova: Sure it can. But if you're working on a machine with cross compilation tools, you must use the ARM variant of it, i.e. prefix objcopy with the target triple. Something like arm-none-eabi-objcopy or the likes.Evslin
O
8

Rules of the C standard put some constraints on those (for safe compilation): A C compiler may look at only the first 8 characters of an identifier, so foobar2k_eggs and foobar2k_spam may be interpreted as the same identifiers validly – however every modern compiler allows for arbitrary long identifiers, so in our times (the 21st century) we should not have to bother about this.

This is not just an extension of modern compilers; the current C standard also requires the compiler to support reasonably long external names. I forget the exact length but it's something like 31 characters now if I remember right.

But what if you're facing some libraries of which you cannot change the symbol names / idenfiers? Maybe you got only a static binary and the headers or don't want to, or are not allowed to adjust and recompile yourself.

Then you're stuck. Complain to the author of the library. I once encountered such a bug where users of my application were unable to build it on Debian due to Debian's libSDL linking libsoundfile, which (at least at the time) polluted the global namespace horribly with variables like dsp (I kid you not!). I complained to Debian, and they fixed their packages and sent the fix upstream, where I assume it was applied, since I never heard of the problem again.

I really think this is the best approach, because it solves the problem for everyone. Any local hack you do will leave the problem in the library for the next unfortunate user to encounter and fight with again.

If you really do need a quick fix, and you have source, you could add a bunch of -Dfoo=crappylib_foo -Dbar=crappylib_bar etc. to the makefile to fix it. If not, use the objcopy solution you found.

Outdated answered 4/8, 2011 at 15:1 Comment(1)
You are right of course, however sometimes you need a dirty hack, like the one I showed above. For example if you're stuck with some legacy library where the vendor went out of business or similar. I specifically wrote this up for static libraries.Evslin
G
4

If you're using GCC, the --allow-multiple-definition linker switch is a handy debugging tool. This hogties the linker into using the first definition (and not whining about it). More about it here.

This has helped me during development when I have the source to a vendor-supplied library available and need to trace into a library function for some reason or other. The switch allows you to compile and link in a local copy of a source file and still link to the unmodified static vendor library. Don't forget to yank the switch back out of the make symbols once the voyage of discovery is complete. Shipping release code with intentional name space collisions is prone to pitfalls including unintentional name space collisions.

Godden answered 17/6, 2017 at 21:18 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.