F

7

23

Here is what I want to do:

Run a program and initialize some data structures.
Then compile additional code that can access/modify the existing data structures.
Repeat step 2 as needed.

I want to be able to do this with both C and C++ using gcc (and eventually Java) on Unix-like systems (especially Linux and Mac OS X). The idea is to basically implement a read-eval-print loop for these languages that compiles expressions and statements as they are entered and uses them to modify existing data structures (something that is done all the time in scripting languages). I am writing this tool in python, which generates the C/C++ files, but this should not be relevant.

I have explored doing this with shared libraries but learned that modifying shared libraries does not affect programs that are already running. I have also tried using shared memory but could not find a way to load a function onto the heap. I have also considered using assembly code but have not yet attempted to do so.

I would prefer not to use any compilers other than gcc unless there is absolutely no way to do it in gcc.

If anyone has any ideas or knows how to do this, any help will be appreciated.

Fletafletch answered 12/5, 2012 at 14:39 Comment(3)

You should look at llvm/clang. But a REPL for C or C++ sounds like quite a task. – Valvule 12/5, 2012 at 14:43

Something to do with this? – Equilibrium 12/5, 2012 at 14:43

I'd rather use real scripting language, except you can tell what's all this for – Mischief 12/5, 2012 at 14:44

S

14

I think you may be able to accomplish this using dynamic libraries and loading them at runtime (using dlopen and friends).

void * lib = dlopen("mynewcode.so", RTLD_LAZY);
if(lib) {
    void (*fn)(void) = dlsym(lib, "libfunc");

    if(fn) fn();
    dlclose(lib);
}

You would obviously have to be compiling the new code as you go along, but if you keep replacing mynewcode.so I think this will work for you.

Spoils answered 12/5, 2012 at 14:46 Comment(9)

Loading should be supported, I'm not sure that un loading is supported in all cases however. – Gazetteer 12/5, 2012 at 15:13

@ChrisStratton: I'll confess I'm far from an expert on runtime loading, but the man page leads me to believe the symbols are unloaded at dlclose (specifically the RTLD_NODELETE flag). Take all that with a grain of salt though :). – Spoils 12/5, 2012 at 22:41

@ChrisStratton I don't know of 'all' cases but in one project of mine I have never seen dlclose() not unload the symbols. Unless of course RTLD_NODELETE is passed in which case it does not unload them. – Jeffiejeffrey 27/10, 2019 at 13:56

N.B. cosine = (double (*)(double)) dlsym(handle, "cos"); .. According to the ISO C standard, casting between function pointers and 'void *', as done above, produces undefined results. POSIX.1-2003 and POSIX.1-2008 accepted this state of affairs and proposed the following workaround: *(void **) (&cosine) = dlsym(handle, "cos"); (1/2) – Jeffiejeffrey 27/10, 2019 at 14:1

This (clumsy) cast conforms with the ISO C standard and will avoid any compiler warnings. The 2013 Technical Corrigendum to POSIX.1-2008 (a.k.a. POSIX.1-2013) improved matters by requiring that conforming implementations support casting 'void *' to a function pointer. Nevertheless, some compilers (e.g., gcc with the '-pedantic' option) may complain about the cast used in this program. (From dlopen(3)) Just as a note about this issue :) (2/2) – Jeffiejeffrey 27/10, 2019 at 14:2

Oh one more thing: In the past I noticed that without matching CFLAGS in the shared libraries and the binary that loads them crashes (obviously not making it a .so but otherwise yes and making sure the binary can load them properly maybe with - and I am vague on this one - -rdynamic). I don't remember the specifics as this was decades ago but looking at it more deeply matching the flags stopped the (random) crashing. And depending on the environment you might need different PIC compiler options. – Jeffiejeffrey 27/10, 2019 at 14:11

Make that one more thing now. As for RTLD_LAZY just to clarify what it means: for functions (not variables) it only resolves the unresolved symbols when the function is called. Depending on the use and importance of the functions this might not be desirable - as in it might be desirable to abort the program if the symbols cannot be resolved at the time. In which case you can use RTLD_NOW. You can use dlerror() to report the error of the most recent related function. You of course need the linker flag for these functions -ldl. I think that's all I have now! – Jeffiejeffrey 27/10, 2019 at 14:14

Where is the libary that has all these functions? – Nagari 20/9, 2020 at 8:14

@Nagari - From the man page: Link with -ldl. If you don't have libdl.so on your system, check your package manager or distribution's documentation. – Spoils 20/9, 2020 at 14:21

G

16

There is one simple solution:

create own library having special functions
load created library
execute functions from that library, pass structures as function variables

To use your structures you have to include same header files like in host application.

structs.h:

struct S {
    int a,b;
};

main.cpp:

#include <iostream>
#include <fstream>
#include <dlfcn.h>
#include <stdlib.h>

#include "structs.h"

using namespace std;

int main ( int argc, char **argv ) {

    // create own program
    ofstream f ( "tmp.cpp" );
    f << "#include<stdlib.h>\n#include \"structs.h\"\n extern \"C\" void F(S &s) { s.a += s.a; s.b *= s.b; }\n";
    f.close();

    // create library
    system ( "/usr/bin/gcc -shared tmp.cpp -o libtmp.so" );

    // load library        
    void * fLib = dlopen ( "./libtmp.so", RTLD_LAZY );
    if ( !fLib ) {
        cerr << "Cannot open library: " << dlerror() << '\n';
    }

    if ( fLib ) {
        int ( *fn ) ( S & ) = dlsym ( fLib, "F" );

        if ( fn ) {
            for(int i=0;i<11;i++) {
                S s;
                s.a = i;
                s.b = i;

                // use function
                fn(s);
                cout << s.a << " " << s.b << endl;
            }
        }
        dlclose ( fLib );
    }

    return 0;
}

output:

You can also create mutable program that will be changing itself (source code), recompiling yourself and then replace it's actual execution with execv and save resources with shared memory.

Gillie answered 12/5, 2012 at 15:42 Comment(4)

very useful info but how would you go about including the main.cpp in the tmp.cpp? – Eggnog 22/9, 2014 at 0:56

Okey, I was going to edit the question to answer you, but there is no need :) You can't include the main.cpp in tmp. If you want to share some data, then you have to use headers (or write it directly to file) and pass structures into dynamically created function :) – Gillie 24/9, 2014 at 18:19

thnx! when using headers however the values of the variables shared are not the same. so i end up having to pass them to the function. i am wondering if there is some way to go around passing variables – Eggnog 24/9, 2014 at 20:32

If you include header into cpp, it's same as if you have written its content into cpp. So you end up with two instances of variables ( main.cpp and dynamic library ), but if u had the header(defining variables) included in two objects(cpp-s) in same library, then it will throw you error in compilation. You have to use "extern" keyword in header, to tell compiler, these variables are not instantiated in current object(cpp) and will be linked by linker. You can make variables 'static', they will be instantiated privately in every object, but you won't share anything anyway. – Gillie 28/9, 2014 at 11:56

S

14

I think you may be able to accomplish this using dynamic libraries and loading them at runtime (using dlopen and friends).

void * lib = dlopen("mynewcode.so", RTLD_LAZY);
if(lib) {
    void (*fn)(void) = dlsym(lib, "libfunc");

    if(fn) fn();
    dlclose(lib);
}