Houston, we have an undefined reference

Question

MEGAEDIT 3000

I have discovered the cause of undefined references. I checked my .o files for symbols, and they were just missing. Just weren't there. For no apparent reason. I have checked the source of the compromiting .o file, but everything seemed to be OK. I've thought that it may be one of those nasty #ifdef ... #endif blocks, cunningly hidden amongst lines of the code like a bear trap, waiting for its victim. But I couldn't find any. After moving some function definitions to the very end of the file and enclosing them in #ifdef ... #endif block suitable for Linux, the symbols magically appeared and everything was fine. Thank you for your time.

---

Outdated, do not read.

I have been assigned to one big project lately, my job is to make it run on Linux platforms. Unfortunately, the programmer who wrote the whole project before me did a very poor job, creating whole lots of circular dependencies by including file B3D_Base.h, where almost every header is #included, to almost every header and source file in the project (that makes 170+ files with #include "B3D_Base.h" line). This caused no problems during compilation on both Windows and Linux. However, this causes lots of 'undefined reference' errors during linking phase on Linux, even though all object files and libraries are linked.

Being aware of ld's limitation of single iteration through all object files during linking, I'm asking:

Is there a way to force ld to do multiple iterations through objects and libraries, possibly solving all the 700+ undefined reference errors, or is my only chance to actually link the project just replacing all the #include "B3D_Base.h" lines with proper includes to needed headers?

APPENDIX #1:

Here is an example of what mess I'm supposed to make run on Linux:

There is one header file B3D_Base.h, to which is included B3D_Loading3.h, which contains class B3D_LOADING3. File B3D_Loading3.cpp is including only B3D_Base.h and a memory manager class not included in Base. An instance B3D_LOADING3 g_Loading3; is declared in B3D_Base.h and initializes+used in many other parts of the project. Everything compiles just fine, but when it comes to linking, I get many 'undefined reference to g_Loading3' errors. And there are many more errors like this, complaining about undefined references to both class instances and functions alike. So much for clean code.

APPENDIX #2:

I'll be providing an SSCCE example, as kfsone requested:

B3D_Base.h

... (many other includes)

#include "B3D_Loading3.h"
extern B3D_LOADING3 g_Loading3;

... (includes go on)

B3D_Loading3.h/B3D_Loading.cpp

a full definition of class B3D_LOADING3, but no real declaration of g_Loading3

AW_Game.cpp (one of the files where g_Loading3 is used)

#include "B3D_Base.h"
... (some lines)
extern B3D_LOADING3 g_Loading3; 

Thanks to kfsone who made me go through the code thoroughly, I managed to find out where the problem lies: g_Loading3 is defined everywhere as extern, thus not being properly defined. All I needed was to remove the extern keyword from one file. Fixed me ~ 30 linking errors. Thanks, kfsone.

APPENDIX #3:

But there is still problem with undefined references to class functions. After some code scanning, I suppose that the functions are not even defined due to overextended use of #ifdef ... #endif clauses (portability) and thus it may be my fault, overlooking functions defined only in some passive ifdef block. I will try to look at this issue and later report if anything changed.

APPENDIX #3 REPORT:

Even after properly defining functions, the undefined-reference-list did not shrink. The problem is therefore still open for discussion.

APPENDIX #3.1

I'll be providing some example code.

Foo.h

#ifndef FOO_H
#define FOO_H

class Foo
{
    unsigned int    m_size;
    bool            m_lock;
    friend class ASDF;

public:
    unsigned int    m_id;

    void    Release() {delete this;}
    int     Lock(UINT OffsetToLock, UINT SizeToLock, VOID ** ppbData, DWORD Flags);
    int     Unlock();
};

// other class declarations to follow

#endif

Foo.cpp

#include "Foo.h"
// other includes and function definitions to follow

int Foo::Lock(UINT OffsetToLock, UINT SizeToLock, VOID ** ppbData, DWORD Flags)
{
    if(!m_lock)
    {
        // some locking code
    }
    return 0;
}

Baz.cpp

#include "Foo.h"  // although included, defined and declared, causes undef reference

Foo *m_pFoo;
// several lines later
m_pFoo->Lock(0,0,(void)&Asdf,NULL); // <-- compiles fine, causes undefined reference

Dappled

#includes will not change undefined reference errors. What you need to modify is the linker commandline, by adding -lmylib statements, or adding .o files to the linker to resolve undefined references. – Athalee

@Athalee The problem does not lie in unlinked libraries/object files. All needed libraries/object files are linked. – Dappled

AFAIK gnu ld doesn't provide such an option unfortunately. Are you using the same toolchain when building between windows and linux or is MSVC used for the former? – Talich

@rotflesh - however altering includes will not alter the linker issues - you ned to look at the order of linking with libraries and these could also be circular – Godless

@Talich Unfortunately, MSVC was used in the former. I want to ask, is Visual Studio compiler using something as -fpermissive by default? (It may sound stupid, but I don't code on Win at all) I'm asking because g++ threw many 'taking address of temporary' errors while compiling, yet I have been told by the guys working on Windows/XBOX version that it compiles properly. – Dappled

@rotflesh MSVC seems to be more lenient on these things. If I recall correctly, using cl /Za disables microsoft's "extensions" to the C++ language. That might help catch more non-conformant c++ code. – Talich

Would this https://mcmap.net/q/1014836/-how-do-you-link-correctly-in-c-to-stop-symbols-being-stripped help in any way? – Marrissa

@Marrissa No, not actually. As you may (or may not) have learned from appendix #2, the problem does not lie within libraries, but within the project source files. Anyway, thanks for teaching me something new. Hopefully, I'll never be forced to use this. – Dappled

I have to ask, are there any unit test for this project? If there are, you can try building the most basic usage and hopefully there'll be less undefined refs. – Talich

@Talich Guess what? Nope, there are no unit tests in this project. Not even in Debug builds. I'll be adding more detail to question. – Dappled

@rotflesh unit tests is a conscious choice. It's something you have to design with when you create the project. google for TDD. If the original authors wrote it with unit tests, it'll probably be in a directory named unittest, test or something to that effect. – Talich

@Talich I'm aware of pros of unit tests, I do them when working on my own projects. But there aren't any unit tests in this mostrosity. The core of the project was originally written in 2006 and has gone through many ugly morphs and changes, leaving me this abomination to port on Linux systems. – Dappled

Answer 1

One thing worth checking for is determining whether it's a dead code stripping issue. I've had "undefined reference" linker errors in the past because of this.

The MS compiler will remove code if it's not used. GCC will not do this by default, and if that code uses symbols that aren't defined, you'll get errors. Since the MS compiler removes those calls, you get no errors there. In order to see if this is your problem, add these flags to your compilation flags (both for C++ as well as for C, if you also have C sources):

-fdata-sections -ffunction-sections

And this flag to your link flags:

-Wl,--gc-sections

(Of course make sure you're using g++ for linking, not ld.)

Other than that, my only recommendation is to use a proper build system. I highly recommend CMake. It's easy, reliable, and you should be able to learn it quickly. Creating a CMake-based build for a 300k LOC project with over 100 source files took me less than 3 hours (and at that point it was the first time I ever used CMake.)

Trinia

Answer 2

Without any kind of example, it's really unclear where your problem might be, except that it's not the ordering of header files - it's what is in those header files and, subsequently, the order in which you link modules.

GCC/LD should only be complaining about undefined references to things that are referenced not just declared.

// t1.h
#pragma once
#include "t2.h"

class Foo1 {};
class Foo3 {}; // This is going to be our undef ref

extern Foo1 g_foo1_a;
extern Foo1& foo1_a();
extern Foo1 g_foo1_b;
extern Foo1& foo1_b();
extern Foo2 g_foo2;
extern Foo3 g_foo3;

// Uncomment this line to generate an undef ref
//static Foo3* g_foo3p = &g_foo3;


// t2.h
#pragma once
#include "t1.h"

class Foo2 {};

extern Foo2 g_foo2;

// t1.cpp
#include "t1.h"
#include "t2.h"

#if defined(_MSC_VER) && _MSC_VER > 9000
Foo3 g_foo3; // gcc won't produce one of these.
#endif

Foo1& foo1_b() { return g_foo1_b; }

int main(int argc, const char** argv)
{
}

// t2.cpp
#include "t1.h"

Foo1 g_foo1_b;

// Makefile
all:
    g++ -Wall -o t1.o -c t1.cpp
    g++ -Wall -o t2.o -c t2.cpp
    g++ -Wall -o t t1.o t2.o
    g++ -Wall -o t t2.o t1.o

Compiles without any undef references until you uncomment the static, at which point you get both compile warnings (unused variable) and undefined reference warnings, because nobody actually instantiated g_foo3.

So - while I realize you can't give us your exact code, if you want to resolve the issue you're probably going to need to come up with an SSCCE. Like as not, in the process of replicating the problem, you'll probably figure it out, which is why it's generally a requirement of asking questions here.

Amenity

Your post helped me fix the part of the problem, see Appendices #2 and #3. Thanks a lot. – Dappled

Answer 3

On Linux, you will need multiple copies of your -l arguments unless you want to learn esoteric ld arguments. List all the libs three (yes, 3) times:

 -la -lb -lc -la -lb -lc -la -lb -lc

to ensure that all dependencies between the libraries are satisfied. Note that the number 3 is independent of the number of libraries. Some versions of ld have -load_all or -( as less verbose solutions to this problem.

The reason is that the linker processes undefined symbols right-to-left. When it first sees -la it only picks up .o files files from liba.a that are required by the base exec. When it sees -lb it loads those required for the base and liba. If some of liba is required for libb, you're out of luck unless you list liba a second time, etc, etc.

Cultivation

If I understand this correctly, you're saying given n library dependencies you potentially have to repeat them up to n times on the CLI for them to fully resolve properly. Who's bright idea was it to design the linker this way? – Talich

Thanks, I will try it. Am I supposed to do this with .o files as well if the linker throws undefined references to variables/functions from the source code e.g. g++ foo.o bar.o baz.o foo.o bar.o baz.o ... ? – Dappled

@Talich One iteration is not a problem unless some fancypants makes circular dependencies all around the project. At least I suppose so. – Dappled

@rotflesh There is some good news. Unlike -llibraries, you don't have to do that for object files as the linker is smart enough to consider everything its seen already. – Talich

@Talich Then I have no idea where the problem may lie. I'll give you a brief example. Class B3D_Loading3.h is included (along with another 80+ headers) in file B3D_Base.hm which is included in B3D_Loading.h, yet the linker screams about undefined reference to 'g_Loading3' which is an instance of class B3D_LOADING from B3D_Loading.h ... Pretty much of a mess. – Dappled

@rotflesh should probably add that to your question. – Talich

@rotflesh I don't much time atm to help but I can later if you still need ideas. Just add some more details to the problems you're running into. – Talich

Not n times, always 3 times. – Cultivation

@Cultivation Why always 3 times? n times makes more sense to me, could you please explain this on and example where n > 3? – Dappled

Note that relinking libraries like this is only relevant to static libraries (.a files on Linux; .lib on Windows). Once a shared library is loaded, all the symbols defined in it are available; there is no need to rescan a shared library. Make sure you list the libraries after all the object files. If you need to link the (static) libraries multiple times, you have circular dependencies between the libraries. One easy way to resolve that problem is to combine the libraries into one; failing that, it is best to reorganize them so that circular dependencies are removed. – Unsound