C/C++ Dynamic loading of functions with unknown prototype

Question

I'm in the process of writing a kind of runtime system/interpreter, and one of things that I need to be able to do is call c/c++ functions located in external libraries.

On linux I'm using the dlfcn.h functions to open a library, and call a function located within. The problem is that, when using dlsysm() the function pointer returned need to be cast to an appropriate type before being called so that the function arguments and return type are know, however if I’m calling some arbitrary function in a library then obviously I will not know this prototype at compile time.

So what I’m asking is, is there a way to call a dynamically loaded function and pass it arguments, and retrieve it’s return value without knowing it’s prototype?

So far I’ve come to the conclusion there is not easy way to do this, but some workarounds that I’ve found are:

• Ensure all the functions I want to load have the same prototype, and provide some sort mechanism for these functions to retrieve parameters and return values. This is what I am doing currently.

• Use inline asm to push the parameters onto the stack, and to read the return value. I really want to steer clear of doing this if possible!

If anyone has any ideas then it would be much appreciated.

Edit:

I have now found exactly what I was looking for:

http://sourceware.org/libffi/

"A Portable Foreign Function Interface Library"

(Although I’ll admit I could have been clearer in the original question!)

Biddle

I dont understand your claim that you cannot know the functions prototype at compile time. I mean, you call dlsym() and do what with the result? Where are you getting the 'arguments' from exactly? The presence of arguments implies - to me - that you have some idea of the functions prototype. – Malpighi

How do you call a function if you don't know the parameters or return type? – Moniz

These functions are specified in the source code of a different language, much like they are for Java JNI. So I can easily get the name of a function to call dlysm, but the arguments and types are represented as various AST nodes which I cannot simply pass to the loaded function. – Biddle

Answer 1

I believe the ruby FFI library achieves what you are asking. It can call functions in external dynamically linked libraries without specifically linking them in.

http://wiki.github.com/ffi/ffi/

You probably can't use it directly in your scripting language but perhapps the ideas are portable.

-- Brad Phelan http://xtargets.heroku.com

Suiting

Answer 2

What you are asking for is if C/C++ supports reflection for functions (i.e. getting information about their type at runtime). Sadly the answer is no.

You will have to make the functions conform to a standard contract (as you said you were doing), or start implementing mechanics for trying to call functions at runtime without knowing their arguments.

Since having no knowledge of a function makes it impossible to call it, I assume your interpreter/"runtime system" at least has some user input or similar it can use to deduce that it's trying to call a function that will look like something taking those arguments and returning something not entirely unexpected. That lookup is hard to implement in itself, even with reflection and a decent runtime type system to work with. Mix in calling conventions, linkage styles, and platforms, and things get nasty real soon.

Stick to your plan, enforce a well-defined contract for the functions you load dynamically, and hopefully make due with that.

Vally

Answer 3

Can you add a dispatch function to the external libraries, e.g. one that takes a function name and N (optional) parameters of some sort of variant type and returns a variant? That way the dispatch function prototype is known. The dispatch function then does a lookup (or a switch) on the function name and calls the corresponding function.

Obviously it becomes a maintenance problem if there are a lot of functions.

Pellitory

Answer 4

I'm in the process of writing a kind of runtime system/interpreter, and one of things that I need to be able to do is call c/c++ functions located in external libraries.

You can probably check for examples how Tcl and Python do that. If you are familiar with Perl, you can also check the Perl XS.

General approach is to require extra gateway library sitting between your interpreter and the target C library. From my experience with Perl XS main reasons are the memory management/garbage collection and the C data types which are hard/impossible to map directly on to the interpreter's language.

So what I’m asking is, is there a way to call a dynamically loaded function and pass it arguments, and retrieve it’s return value without knowing it’s prototype?

No known to me.

Ensure all the functions I want to load have the same prototype, and provide some sort mechanism for these functions to retrieve parameters and return values. This is what I am doing currently.

This is what in my project other team is doing too. They have standardized API for external plug-ins on something like that:

typedef std::list< std::string > string_list_t;
string_list_t func1(string_list_t stdin, string_list_t &stderr);

Common tasks for the plug-ins is to perform transformation or mapping or expansion of the input, often using RDBMS.

Previous versions of the interface grew over time unmaintainable causing problems to both customers, products developers and 3rd party plug-in developers. Frivolous use of the std::string is allowed by the fact that the plug-ins are called relatively seldom (and still the overhead is peanuts compared to the SQL used all over the place). The argument stdin is populated with input depending on the plug-in type. Plug-in call considered failed if inside output parameter stderr any string starts with 'E:' ('W:' is for warnings, rest is silently ignored thus can be used for plug-in development/debugging).

The dlsym is used only once on function with predefined name to fetch from the shared library array with the function table (function public name, type, pointer, etc).

Maybe

Answer 5

My solution is that you can define a generic proxy function which will convert the dynamic function to a uniform prototype, something like this:

#include <string>
#include <functional>

using result = std::function<std::string(std::string)>;

template <class F>
result proxy(F func) {
  // some type-traits technologies based on func type
}

In user-defined file, you must add define to do the convert:

double foo(double a) { /*...*/ }
auto local_foo = proxy(foo);

In your runtime system/interpreter, you can use dlsym to define a foo-function. It is the user-defined function foo's responsibility to do calculation.

Stuff