Creating a module system (dynamic loading) in C

Question

How would one go about loading compiled C code at run time, and then calling functions within it? Not like simply calling exec().

EDIT: The the program loading the module is in C.

Heptameter

Excellent question. Many people know how to do this, but those who don't would do well to learn this valuable technique. – Beanie

Answer 1

In Linux/UNIX you can use the POSIX dlopen / dlsym / dlerror / dlclose functions to dynamically open shared libraries and access the symbols (including functions) they provide, see the man page for details.

Amyl

Is the principle of the poco library like this? – Howlett

Answer 2

dlopen is the way to go. Here are a few examples:

Loading a plugin with dlopen:

#include <dlfcn.h>
...
int
main (const int argc, const char *argv[])
{

  char *plugin_name;
  char file_name[80];
  void *plugin;
  ...
  plugin = dlopen(file_name, RTLD_NOW);
  if (!plugin)
  {
     fatal("Cannot load %s: %s", plugin_name, dlerror ());
  }

Compiling the above:

% cc  -ldl -o program program.o 

Then, assuming this API for the plugins:

/* The functions we will find in the plugin */
typedef void (*init_f) ();
init_f init;
typedef int (*query_f) ();
query_f query;

Finding the address of init() in the plugin:

init = dlsym(plugin, "init");
result = dlerror();
if (result)
{
   fatal("Cannot find init in %s: %s", plugin_name, result);
}
init();

With the other function, query(), which returns a value:

query = dlsym (plugin, "query");
result = dlerror();
if (result)
{
    fatal("Cannot find query in %s: %s", plugin_name, result);
}
printf("Result of plugin %s is %d\n", plugin_name, query ());

You can retrieve the complete example on line.

Subcelestial

Would you mind putting the complete example to github? Would be easier to read there. – Sounding

if using a c++ compiler, is it standard to use the mangled string function name when you use dlsym? or extern "c" on the function to just use the normal function name on dlsym? – Corncob

Answer 3

for GNU/Linux users

dynamic loading library is a mechanism with which we can run our program, and at runtime, decide what function, we want to use / call. I think in some cases static variable is possible as well.

First start seeing man 3 dlopen or see it online

The header file that is required is: dlfcn and since this is not part of the standard you should link it to your object file with this library: libdl.(so/a) and therefore you need something like:

gcc yours.c -ldl

then you have a file name a.out and you can run it BUT it does not work properly and I will explain it why.

---

A complete example:

first crate 2 files func1.c and func2.c respectively. We want to call these functions at runtime.

func.c

int func1(){
    return 1;
}

func2.c

const char* func2(){
    return "upgrading to version 2";
}

Now we have 2 functions, let's make our modules:

ALP ❱ gcc -c -fPIC func1.c
ALP ❱ gcc -c -fPIC func2.c
ALP ❱ gcc -o libfunc.so -shared -fPIC func1.o func2.o 

for inquiring mind about -fPIC => PIC

Now you have a dynamic library names: libfunc.so

Let's create the main program (= temp.c) that wants to use those functions.

header files

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h> 

and the main program

int main()
{
    // pointer function to func1 and func2
    int         ( *f1ptr )();
    const char* ( *f2ptr )();

    // for pointing to the library
    void* handle = NULL;

    // for saving the error messages
    const char* error_message = NULL;

    // on error dlopen returns NULL
    handle = dlopen( "libfunc.so", RTLD_LAZY );

    // check for error, if it is NULL
    if( !handle )
    {
        fprintf( stderr, "dlopen() %s\n", dlerror() );
        exit( 1 );
    }

    /*
        according to the header file:

        When any of the above functions fails, call this function
        to return a string describing the error.  Each call resets
        the error string so that a following call returns null.

        extern char *dlerror (void) __THROW;
    */

    // So, reset the error string, of course we no need to do it just for sure
    dlerror();

    // point to func1
    f1ptr = (int (*)()) dlsym( handle, "func1" );

    // store the error message to error_message
    // because it is reseted if we use it directly
    error_message = dlerror();
    if( error_message ) //   it means if it is not null
    {
        fprintf( stderr, "dlsym() for func1 %s\n", error_message );
        dlclose( handle );
        exit( 1 );
    }

    // point the func2
    f2ptr = (const char* (*)()) dlsym( handle, "func2" );

    // store the error message to error_message
    // because it is reseted if we use it directly
    error_message = dlerror();
    if( error_message ) //   it means if it is not null
    {
        fprintf( stderr, "dlsym() for func2 %s\n", error_message );
        dlclose( handle );
        exit( 1 );
    }

    printf( "func1: %d\n", ( *f1ptr )() );
    printf( "func2: %s\n", ( *f2ptr )() );

    // unload the library
    dlclose( handle );

    // the main return value
    return 0;
}

Now we just need to compile the this code (= temp.c), thus try:

ALP ❱ gcc temp.c -ldl
ALP ❱ ./a.out
libfunc.so: cannot open shared object file: No such file or directory

It does not work! WHY easy; because our a.out program does not know where to find the related library: libfunc.so and therefore it tells us cannot not open ...

how to tell the program (= a.out) to find its library?

1. using ld linker
2. using environment variable LD_LIBRARY_PATH
3. using the standard path

first way, with help of ld

use -Wl,-rpath, and pwd and put the path as a argument for it

ALP ❱ gcc temp.c -ldl
ALP ❱ ./a.out
libfunc.so: cannot open shared object file: No such file or directory
ALP ❱ pwd
/home/shu/codeblock/ALP
ALP ❱ gcc temp.c -ldl -Wl,-rpath,/home/shu/codeblock/ALP
ALP ❱ ./a.out
func1: 1
func2: upgrading to version 2

second way

ALP ❱ gcc temp.c -ldl
ALP ❱ ./a.out
libfunc.so: cannot open shared object file: No such file or direc
ALP ❱ export LD_LIBRARY_PATH=$PWD
ALP ❱ echo $LD_LIBRARY_PATH
/home/shu/codeblock/ALP
ALP ❱ ./a.out
func1: 1
func2: upgrading to version 2
ALP ❱ export LD_LIBRARY_PATH=
ALP ❱ ./a.out
libfunc.so: cannot open shared object file: No such file or 

and third way

you have libfunc.so in you current path, thus you can copy it into a standard path for libraries.

ALP $ sudo cp libfunc.so /usr/lib
ALP ❱ gcc temp.c -ldl
ALP ❱ ./a.out
func1: 1
func2: upgrading to version 2

you can remove it from /usr/lib and use it. It is up to you.

NOTE

how to find out that our a.out knows about its path?
easy:

ALP ❱ gcc temp.c -ldl -Wl,-rpath,/home/shu/codeblock/ALP
ALP ❱ strings a.out  | grep \/
/lib/ld-linux.so.2
/home/shu/codeblock/ALP

how we can use it in c++?
As long as I know you cannot because g++ mangles the function names whereas gcc does not thus you should use: extern "C" int func1(); for example.

For any more details see man pages and Linux programing books.

Bascule

Nice! There is a 4th way, according dlopen man page "If filename contains a slash ("/"), then it is interpreted as a (relative or absolute) pathname." So 'handle = dlopen( "./libfunc.so", RTLD_LAZY );' allows to compile as described, and just execute "./a.out" successfully without doing anything else. – Kathlyn

Answer 4

See this question has been answered but thought others interested in this topic may appreciate a cross platform example from an old plugin based application. The example works on win32 or linux, and seaches for and calls a function called 'constructor' in the dynamically loaded .so or .dll specified in the file argument. The example is in c++ but the procedures should be the same for c.

//firstly the includes
#if !defined WIN32
   #include <dlfcn.h>
   #include <sys/types.h>
#else
   #include <windows.h>
#endif

//define the plugin's constructor function type named PConst
typedef tcnplugin* (*PConst)(tcnplugin*,tcnplugin*,HANDLE);

//loads a single specified tcnplugin,allmychildren[0] = null plugin
int tcnplugin::loadplugin(char *file) {
    tcnplugin *hpi;
#if defined WIN32               //Load library windows style
    HINSTANCE hplugin=LoadLibrary(file);
    if (hplugin != NULL) {
            PConst pinconstruct = (PConst)GetProcAddress(hplugin,"construct");
#else                                   //Load it nix style
    void * hplugin=dlopen(file,RTLD_NOW);
    if (hplugin != NULL) {
            PConst pinconstruct = (PConst)dlsym(hplugin,"construct");
#endif   
            if (pinconstruct != NULL) { //Try to call constructor function in dynamically loaded file, which returns a pointer to an instance of the plugin's class
                    hpi = pinconstruct(this, this, hstdout);
            } else {
                    piprintf("Cannot find constructor export in plugin!\n");
                    return 0;
            }
    } else {
            piprintf("Cannot open plugin!\n");
#if !defined WIN32
            perror(dlerror());
#endif
            return 0;
    }
    return addchild(hpi); //add pointer to plugin's class to our list of plugins
}

Might also be mentioned that if the module who's functions you wish to call is written in c++ your must declare the function with extern "C" such as:

extern "C" pcparport * construct(tcnplugin *tcnptr,tcnplugin *parent) {
    return new pcparport(tcnptr,parent,"PCPARPORT",0,1);
}

Unhandsome

Which headers are needed to make it run on Linux? And the '::' means it is C++, rather than C, doesn't it? – Septuple

Answer 5

Also you can look at cpluff. It is a plugin management library on pure c.

Saltwater

Answer 6

Dynamic languages like Perl do this all the time. The Perl interpreter is written in C, and many Perl modules are partially written in C. When those modules are required, the compiled C components are dynamically loaded on the fly. As noted in another answer, the mechanism for storing those modules is DLLs on windows, and shared libraries (.so files) on UNIX. I believe the call for loading a shared library on UNIX is dlopen(). You can probably find pointers for how to accomplish this on UNIX by starting with the documentation for that call. For Windows, you would need to research DLLs and learn how to load them dynamically at runtime. [Or possibly go through the Cygwin UNIX emulation layer, which would probably allow you to use the same calls on Windows as you would on UNIX, but I wouldn't recommend that unless you're already using and compiling against Cygwin.]

Note that this is different from just linking against a shared library. If you know ahead of time exactly what code you will call, you can build against a shared library and the build will be "dynamically linked" to that library; without any special handling from you the routines from the library will be loaded into memory only when and if your program actually calls them. But you can't do that if you're planning to write something capable of loading any arbitrary object code, code that you can't identify now, at build time, but are instead waiting to be selected somehow at run time. For that you'll have to use dlopen() and its Windows cousins.

You might look at the way Perl or other dynamic languages do this to see some real examples. The Perl library responsible for this kind of dynamic loading is DynaLoader; it has both a Perl and a C component, I believe. I'm certain that other dynamic languages like Python have something similar which you might rather look at; and Parrot, the virtual machine for the unreleased Perl 6, surely has a mechanism for doing this as well (or will in the future).

For that matter, Java accomplishes this through its JNI (Java Native Interface) interface, so you could probably look at the source code for OpenJDK to see how Java accomplishes this on both UNIX and Windows.

Confinement

Answer 7

If you are willing to consider the framework, Qt provides QPluginLoader: Qt 5 docs (or for old Qt 4.8 docs see here)

If you need/want more fine grained control, Qt also provides a means to load libraries on the fly with QLibrary: Qt 5 docs (or for old Qt 4.8 docs see here)

Even better, these are portable across platforms.

Cutoff

Answer 8

There is a DIY approach. While the method (and possibility) of doing this varies from system to system, the general idea is to open a file, read the contents of the file into memory, make said memory executable, initialise a function pointer to a valid position within this memory, and there you are.

Of course this is assuming that it's just executable code - quite unlikely. The code probably requires data to be loaded into RAM too, and may require space for global/static variables. You could load this all yourself, but you'd need to go into the executable code and adjust all the memory references in it.

Most operating systems allow dynamic linking, which does all this for you.

Kg

Reading an executable into memory, getting all the protection settings correct, and finding correct symbols is hard. Why reinvent the wheel when there are standard operating system functions which can do a better job for you? – Bendite

The parts about "read the contents of the file into memory, make said memory executable" covers a lot, because there's usually a lot of relocation and code adjustment at load-time. I actually tried it once. Not for wimps. – Beanie

Answer 9

Under Windows, this is how I do it:

• Generate code (in C because it's easy to find compilers, and library requirements are minimal)
• spawn a job to compile/link it into a DLL
• load it with LoadLibrary
• get function pointers with GetProcAddress

The generate/compile/link steps generally take less than a second.

Beanie