Changing Calling Convention

Question

I have a 3rd party C API that expects a __stdcall callback function.
My code has an externally-provided __cdecl callback function.

I cannot pass my function pointer to the C-API as they are considered different types.
Bypassing the type system and using reinterpret_cast<> naturally results in a runtime error.

Here's an example from here:

// C-API
// the stdcall function pointer type:
typedef CTMuint(__stdcall *CTMwritefn)(const void *aBuf, CTMuint aCount, void *aUserData);

// A function needing the callback: 
CTMEXPORT void __stdcall ctmSaveCustom(CTMcontext aContext, CTMwritefn aWriteFn, void *aUserData, int *newvertexindex);
                                                            ^^^^^^^^^^^^^^^^^^^

//////////////////////////////////////////////////////////////////////////////

// C++
CTMuint __cdecl my_func(const void *aBuf, CTMuint aCount, void *aUserData);

// I want to call here:
ctmSaveCustom(context, my_func, &my_data, nullptr);
//                     ^^^^^^^

Is there a way to safely convert and/or wrap a function with one calling convention into another?

I did find a way to do it by passing a casted captureless-lambda that calls a second capturing lambda. The first is passed as the callback, the second through the void* user_data. This works and is type-safe. But it is quite convoluted for something that seems so simple.

Yokoyokohama

Can you not just create a cdecl wrapper that would forward the call to your real callback? – Munger

@krzaq: How would you do that? It is not as obvious as it would seem... I will add an example. – Yokoyokohama

Can you - apart from the function pointer - also pass "user data" (usually a void *)? Otherwise a wrapper function is going to be a bit of a hassle regarding thread safety / multiple callbacks. – Arissa

@DanielJour: Yes, see my extended question and workaround using that above. – Yokoyokohama

But it is quite convoluted for something that seems so simple -- Why do you think it's "simple"? A calling convention is much more than just a keyword. It defines how to pass parameters and return values at a low level. A function wants a __stdcall function, there is no way around it -- it is what it is. – Mukund

@AdiShavit: would this be something you want? – Munger

@PaulMcKenzie: I said "seems" simple - obviously it isn't. However, do note that capture-less lambdas can be seamlessly / automatically cast into whatever calling convention function pointer you can't. – Yokoyokohama

@krzaq: Yes! Looking at the sample, I think this is what I wanted - will test it more thoroughly. You should put that as an answer :-) – Yokoyokohama

Answer 1

You can make a wrapper for translation between different calling conventions:

template<typename Func, Func* callback>
auto make_callback()
{
    return &detail::callback_maker<Func, callback>::call;
}

with callback_maker defined as

template<typename T, T*>
struct callback_maker;

template<typename R, typename... Params, R(*Func)(Params...)>
struct callback_maker<R(Params...), Func>
{
    static R __stdcall call(Params... ps)
    {
        return Func(std::forward<Params>(ps)...);
    }
};

This is aimed to be a fairly general solution, allowing you to specify the function prototype. You can use it as follows:

//  external_api(&not_stdcall_func); // error
external_api(make_callback<void(int,int), &not_stdcall_func>());

demo

---

If the pointer is to be determined at runtime, you could keep the callback in the user data. You'd have to manage the lifetime of that correctly, but it's likely that you already need to to that. Again, attempting a generic solution. Make a callback and tell it which argument is the user data pointer:

template<typename Callback, size_t N>
auto make_callback()
{
    using callback_maker = detail::callback_maker<Callback, N>;
    return &callback_maker::call;
}

With callback_maker defined as

template<typename T, size_t N>
struct callback_maker;

template<typename R, typename... Params, size_t N>
struct callback_maker<R(*)(Params...), N>
{
    using function_type = R(Params...);

    static R __stdcall call(Params... ps)
    {
        void const* userData = get_nth_element<N>(ps...);
        auto p = static_cast<pair<function_type*, void*> const*>(userData);
        return p->first(ps...);
    }
};

and get_nth_element as

template<size_t N, typename First, typename... Ts>
decltype(auto) get_nth_element_impl(false_type, First&& f, Ts&&...);

template<size_t N, typename First, typename... Ts>
decltype(auto) get_nth_element_impl(true_type, First&&, Ts&&... ts)
{
    return get_nth_element_impl<N-1>(integral_constant<bool, (N > 1)>{}, forward<Ts>(ts)...);
}

template<size_t N, typename First, typename... Ts>
decltype(auto) get_nth_element_impl(false_type, First&& f, Ts&&...)
{
    return forward<First>(f);
}

template<size_t N, typename... Ts>
decltype(auto) get_nth_element(Ts&&... ts)
{
    return get_nth_element_impl<N>(integral_constant<bool, (N > 0)>{}, forward<Ts>(ts)...);
}

Now, on the call site

using callback_t = CTMuint(*)(const void *aBuf, CTMuint aCount, void *aUserData);
auto runtime_ptr = &not_stdcall_func;

pair<callback_t, void*> data;
data.first = runtime_ptr;
data.second = nullptr; // actual user data you wanted

auto callback = make_callback<callback_t, 2>();

ctmSaveCustom({}, callback, &data, nullptr);

demo

---

As per Andrey Turkin's suggestion, you can replace the user data pointer in the parameter list. Along with forward_as_tuple, it obviates the need for get_nth_element. The upgraded call function:

static R __stdcall call(Params... ps)
{
    auto params_tuple = forward_as_tuple(ps...);
    void const* userData = get<N>(params_tuple);
    auto p = static_cast<pair<function_type*, void*> const*>(userData);
    get<N>(params_tuple) = p->second;
    return apply(p->first, move(params_tuple));
}

and here's a simplistic implementation of C++17's apply:

template<typename Func, typename T, size_t... Is>
decltype(auto) apply_impl(Func f, T&& t, index_sequence<Is...>)
{
    return f(get<Is>(t)...);
}

template<typename Func, typename... Ts>
decltype(auto) apply(Func f, tuple<Ts...>&& tup)
{
    return apply_impl(f, move(tup), index_sequence_for<Ts...>{});
}

demo

Munger

I've never seen this way of decomposing (reflecting) the function (pointer) types. Very cool using the template specialization. – Yokoyokohama

As discussed in the comments for @StoryTeller's answer, and thus expected I get the following error: error C2975: 'callback': invalid template argument for 'make_callback', expected compile-time constant expression – Yokoyokohama

@AdiShavit added my take on the runtime callback. Maybe you'll find this useful. Either way, this was a fun question – Munger

@Munger Kudos on selecting user_data argument. As a possible improvement - I bet it is possible to replace N-th argument in ps... with actual userData thus avoiding need for union hacks. – Federate

@AndreyTurkin It's definitely possible. Whether it's worth the implementation effort is debatable - at least I cannot think of an easy way to do this. That being said, feel free to enhance this answer or post your own (and please notify me if you do). I'd gladly learn. – Munger

@krzaq: Very cool generic solution - Very similar to my non-generic one. I'd probably default N to the last argument, as this is the most common convention - to have user_data as the last arg. Too bad I can't vote more than +1. – Yokoyokohama

@AndreyTurkin well, it was easier than I thought. Thanks for making me think – Munger

Awesome. Final though: I think it can be even more generic if it would take any N-argument callable and make it into N+1-argument __cdecl function. It would take user_data from the arguments (which is used to store internal state) and call this callable with remaining arguments. That callable would/could add original user_data back and call original callback. It would be still very easy to wrap callbacks with same signatures (by using generic helper callable to insert user_data into same place) but this way different signatures can be accomodated too (with std::bind or lambda). – Federate

Answer 2

In the case of visual c++ (starting from VC11), state-less lambdas implement a conversion operator to function pointers of all calling conventions.

So this, can work just as well

#include <iostream>
using namespace std;

int __cdecl foo()
{
    return 2;
}

void bar (int (__stdcall *pFunc)() )
{
    cout << pFunc()*2;
}

int main() {

    bar([](){ return foo(); });

    return 0;
}

Coinsure

Actually, though the principle is true, this cannot work as you show since in my case foo() is not a function declaration (which can be used in a captureless lambda), but a function-pointer which needs to be passed as a parameter to the lambda making it capture-full and thus uncastable. – Yokoyokohama

@AdiShavit, if you (& the compiler) don't have access to the full function definition, the entire premise is impossible. Even krzaq's answer won't work if you get the pointer at runetime. – Coinsure

@AdiShavit what he said. Notice that I use function pointer as a template parameter - it must be known at compile (well, link?) time – Munger

@krzaq: Yes, you're right, I realize this now. I do get the fn-pointer at run time, so I guess neither solutions will work. – Yokoyokohama

@AdiShavit can I assume that you always get userData pointer in the callback that you control? – Munger

@krzaq: In my case, I do have userData and I used it solve it like explained here I was just hoping there is a simpler way to do this. – Yokoyokohama

@AdiShavit, than post that as an answer & accept. It's good to have for posterity. – Coinsure

Answer 3

If callback is not known at compile time you have following options:

• Use single wrapper function and pass target callback around in user_data. Pro - reasonably easy to use; con - needs user_data for its own use; requires very similar function signatures
• Use wrapper class, allocate instance of the class and pass this in user_data. Pro - more versatily as it can capture some data in each instance (e.g. it can store user_data for target callback or pass additional data to target callback); con - need to manage wrapper instance lifetime
• Build separate thunks for each distinct target callback. Pro - doesn't require using user_data; con - pretty low-level and pretty nonportable (in both compiler in OS); might be hard to do; hard to do in C++ without resorting to assembly.

First option would look something like that (shamelessly ripping off @krzaq):

template<typename T> struct callback_maker;
template<typename R, typename... Params> struct callback_maker<R(Params...)> {
    static R __stdcall call_with_userdata_as_last_parameter(Params... ps, void* userData) {
        R(__cdecl *Func)(Params...) = reinterpret_cast<R(__cdecl *)(Params...)>(userData);
        return Func(std::forward<Params>(ps)...);
    }
};
template<typename Func> constexpr auto make_callback() {
    return &callback_maker<Func>::call_with_userdata_as_last_parameter;
}

...
extern void external_api(void(__stdcall*)(int,int,void*), void* userdata);
extern void __cdecl not_stdcall_func(int,int);
external_api(make_callback<void(int,int)>(), &not_stdcall_func);

Probably not usable for you since you need userData for both callbacks.

Second option:

template<typename T> struct CallbackWrapper;
template<typename R, typename... Params> struct CallbackWrapper<R(Params...)> {
    using stdcall_callback_t = R(__stdcall*)(Params..., void*);
    using cdecl_callback_t = R(__cdecl*)(Params..., void*);
    using MyType = CallbackWrapper<R(Params...)>;
    CallbackWrapper(cdecl_callback_t target, void* target_userdata) : _target(target), _target_userdata(target_userdata) {}
    stdcall_callback_t callback() const { return &MyType::callback_function; }
private:
    static R __stdcall callback_function(Params... ps, void* userData) {
        auto This = reinterpret_cast<MyType*>(userData);
        return This->_target(std::forward<Params>(ps)..., This->_target_userdata);
    }
    cdecl_callback_t _target;
    void* _target_userdata;
};

...
extern void external_api(void(__stdcall*)(int,int,void*), void* userdata);
extern void __cdecl not_stdcall_func(int,int, void*);

void * userdata_for_not_stdcall_func = nullptr;
CallbackWrapper<void(int, int)> wrapper(&not_stdcall_func, userdata_for_not_stdcall_func);
external_api(wrapper.callback(), &wrapper);
// make sure wrapper is alive for as long as external_api is using the callback!

Federate

Answer 4

Answering myself, hoping somebody has a simpler solution.
The approach is the same as explained here.

We are going to use the following:

1. Captureless lambdas can be automatically cast to function-pointers with any desired calling convention.
2. The C-API function provides a void* user_data way to pass data to the call-back.

We will pass the C-API two labmdas:

1. One is captureless cast to the correct calling convention;
2. The other captures the callback fn-ptr and is passed as the user_data to the captureless lambda to call. It captures both the original callback and the original user_data for internal use.

Here's the code:

// This is a lambda that calls the (cdecl) callback via capture list
// However, you can't convert a non-captureless lambda to a function pointer
auto callback_trampoline = [&callback, &user_data](const void *aBuf, CTMuint aCount) -> CTMuint
{
    return callback(aBuf, aCount, user_data);
};

using trampoline_type = decltype(callback_trampoline);

// so we create a capture-less wrapper which will get the lambda as the user data!
// this CAN be cast to a function pointer!
auto callback_wrapper_dispatcher = [](const void *aBuf, CTMuint aCount, void *aUserData) -> CTMuint
{
    auto& lambda = *reinterpret_cast<trampoline_type*>(aUserData);
    return lambda(aBuf, aCount);
};

ctmSaveCustom(context_, callback_wrapper_dispatcher, &callback_trampoline, nullptr);

This is type safe and works as expected.

It would be cool to make this into a generic tool similar to what is suggested in the answer by @krzaq.

UPDATE:
Here's a simpler formulation with just a single captureless lambda, but the same concept:

auto payload = std::tie(callback, user_data);
using payload_type = decltype(payload);
auto dispatcher = [](const void *aBuf, CTMuint aCount, void *aUserData)->CTMuint
{
    // payload_type is visible to the captureless lamda
    auto& payload = *reinterpret_cast<payload_type*>(aUserData);
    return std::get<0>(payload)(aBuf, aCount, std::get<1>(payload));
};
ctmSaveCustom(context_, dispatcher, &payload, nullptr);

Yokoyokohama

@krzaq: see the updated formulation with a single lambda, basically what you suggested but non-generic. – Yokoyokohama