What's the protocol for calling Raku code from C code?

Question

2023 update The last person to edit this Q deleted the critically important "LATEST LATEST UPDATE" part that @zentrunix had added near the top. I'm reinstating it.

LATEST LATEST UPDATE

Please see my answer below.

Thanks to everyone who took the time to answer and understand this question.

Original question

Say I have my event-driven TCP communications library in C.

From my Raku application, I can call a function in the C library using NativeCall.

my $server = create-server("127.0.0.1", 4000);

Now, from my callback in C (say onAccept) I want to call out to a Raku function in my application (say on-accept(connection) where connection will be a pointer to a C struct).

So, how can I do that: call my Raku function on-accept from my C function onAccept ?

ps. I tried posting using a simple title "How to call Raku code from C code", but for whatever reason stackoverflow.com wouldn't let me do it. Because of that I concocted this fancy title.

I was creating a 32-bit DLL. We have to explicitly tell CMake to configure a 64-bit build.

cmake -G "Visual Studio 14 2015 Win64" ..

Anyway, now that the code runs, it's not really what I asked for, because the callback is still in C.

It seems that what I asked for it's not really possible.

I tried to use the approach suggested by Haakon, though I'm afraid I don't understand how it would work.

I'm in Windows, and unfortunately, Raku can't find my dlls, even if I put them in C:\Windows\System32. It finds "msvcrt" (C runtime), but not my dlls.

The dll code (Visual Studio 2015).

#include <stdio.h>

#define EXPORTED __declspec(dllexport)

typedef int (*proto)(const char*);

proto raku_callback;

extern EXPORTED void set_callback(proto);
extern EXPORTED void foo(void);

void set_callback(proto arg)
{
  printf("In set_callback()..\n");
  raku_callback = arg;
}

void foo(void)
{
  printf("In foo()..\n");
  int res = raku_callback("hello");
  printf("Raku return value: %d\n", res);
}

Cmake code for the

CMAKE_MINIMUM_REQUIRED (VERSION 3.1)
add_library (my_c_dll SHARED my_c_dll.c)

Raku code.

use v6.d;

use NativeCall;

sub set_callback(&callback (Str --> int32))
  is native("./my_c_dll"){ * }

sub foo()
  is native("./my_c_dll"){ * }

sub callback(Str $str --> Int) {
  say "Raku callback.. got string: {$str} from C";
  return 32;
}

## sub _getch() returns int32 is native("msvcrt") {*};
## print "-> ";
## say "got ", _getch();

set_callback(&callback);
# foo();

When I run

$ raku test-dll.raku
Cannot locate native library '(null)': error 0xc1
  in method setup at D:\tools\raku\share\perl6\core\sources
    \947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 298
  in block set_callback at D:\tools\raku\share\perl6\core\sources
     \947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 594
  in block <unit> at test-dll.raku line 21

Raku version.

$ raku -v
This is Rakudo version 2020.05.1 built on MoarVM version 2020.05
implementing Raku 6.d.

Filature

I'm not frustrated, I'm enlightened :) ... as to the error in loading the dll, it was my fault – Filature

for a little more context, in Lua, C code can call Lua code,so maybe it would also be possible to do something like that in Raku ... so, that's why I posted this question – Filature

According to Wikipedia, for Lua, "...the interpreter of compiled bytecode is written in ANSI C, and Lua has a relatively simple C API to embed it into applications." So if the question you're posting is closer to the title @raiph suggests (i.e. interoperability), it's natural to assume that Lua/C interoperability will be easier than Raku/C interoperability. – Automobile

@Automobile I made a simple question, "Is it possible to call Raku from C ?", and that's it ... it seems it's not possible, and that's ok by me, I'm not frustrated, nor angry, it was simply something that I wanted to know – Filature

.@Automobile I see I never replied to your comment that "it's natural to assume that Lua/C interoperability will be easier than Raku/C interoperability." The accepted answer by Håkon Hægland shows how Raku interop is both easier and safer. Using Lua you have to write C code that embeds Lua (luaL_newstate, luaL_openlibs, etc), explicitly manually marshal arguments (luaL_loadstring etc), and do explicit manual stack management (lua_pop etc). It's not just lua_pcall. – Vibrations

I see I haven't left a reply to .@zentrunix's last comment which preceded us clearing up the confusion after they first asked their question. The answer to the simple question "Is it possible to call Raku from C?" is YES. Not only that, but it's easier and safer than calling Lua from C. This is why .@Filature accepted Håkon Hægland's answer, and why the conclusion of .@zentrunix's own answer was "What amazes me is that the Raku signature for my_raku_function maps cleanly to the C signature ... isn't Raku wonderful ?". – Vibrations

Answer 1

Another approach could be to save a callback statically in the C library, for example (libmylib.c):

#include <stdio.h>

static int (*raku_callback)(char *arg);

void set_callback(int (*callback)(char * arg)) {
    printf("In set_callback()..\n");
    raku_callback = callback;
}

void foo() {
    printf("In foo()..\n");
    int res = raku_callback("hello");
    printf("Raku return value: %d\n", res);
}

Then from Raku:

use v6;
use NativeCall;

sub set_callback(&callback (Str --> int32)) is native('./libmylib.so') { * }
sub foo() is native('./libmylib.so') { * }

sub callback(Str $str --> Int) {
    say "Raku callback.. got string: {$str} from C";
    return 32;
}

set_callback(&callback);
foo();

Output:

In set_callback()..
In foo()..
Raku callback.. got string: hello from C
Raku return value: 32

Burkhard

although it's not the answer I was hoping for, it's a nice answer, because it shows how easy it is to use native calls (to C) in Raku – Filature

When .@Filature wrote their comment they had not realized this answer in fact was what they were hoping for, because it not only shows how to call C code from Raku code, but also how to call Raku code from C code. (Not to mention that it's easier and safer than the equivalents back and forth between C and Lua.) – Vibrations

is native("./libmylib.so") is now spelled ``````is native("./mylib")``` [not tested] - raku will them follow the dll/so naming convention of the platform (*nix / windows) – Spatterdash

Answer 2

Raku is a compiled language; depending on the implementation you've got, it will be compiled to MoarVM, JVM or Javascript. Through compilation, Raku code becomes bytecode in the corresponding virtual machine. So it's never, actually, binary code.

However, Raku code seems to be cleverly organized in a way that an object is actually a pointer to a C endpoint, as proved by Haakon Hagland answer.

WRT to your latest problem, please bear in mind that what you are calling is not a path, but a name that is converted to a navive shared library name and also uses local library path conventions to look for them (it's `PATH' on Windows). So if it's not finding it, add local path to it of simply copy the DLL to one of the searched directories.

Kernel

In the case I've used in my question the connections would also be handled by C code, but yes, I think you're right, callbacks (especially asynchronous callbacks generated by multi-threaded applications) would have to be handle by C code ... the interaction between Raku code and C code would have to serialized (as in not muliti-threaded), but I think there's no other way to do that (in fact, I believe that's how node.js works) ... thanks for your input – Filature

@Vibrations I do know the "actors" concept, although I have zero experince with it; now, right after I wrote the comment above I realized that, with some care, maybe Raku code can interact with C code in a non-serialized way: say, 2 "start" blocks in Raku, each calling a different C function which wait for events delivered by other parts of the C code; as for C calling Raku functions directly, I don't know if there's an easy way of doing that like (as I said above) Lua allows it: see here the function lua_pcall – Filature

@Vibrations please note that I'm not an expert neither in Raku nor in Lua :) – Filature

@Vibrations no I don't. I can't delete the answer, though. I'll see. – Kernel

Answer 3

First of all, my apologies to @Håkon and @raiph. Sorry for being so obtuse. :)

Håkon's answer does indeed answer my question, although for whatever reason I have failed to see that until now.

Now the code I played with in order to understand Håkon's solution.

---

// my_c_dll.c
// be sure to create a 64-bit dll

#include <stdio.h>

#define EXPORTED __declspec(dllexport)

typedef int (*proto)(const char*);

proto raku_function;

extern EXPORTED void install_raku_function(proto);
extern EXPORTED void start_c_processing(void);

void install_raku_function(proto arg)
{
  printf("installing raku function\n");
  raku_function = arg;
}

void start_c_processing(void)
{
  printf("* ----> starting C processing..\n");

  for (int i = 0; i < 100; i++)
  {
    printf("* %d calling raku function\n", i);
    int res = raku_function("hello");
    printf("* %d raku function returned: %d\n", i, res);
    Sleep(1000);
  }
}

---

# test-dll.raku

use v6.d;

use NativeCall;

sub install_raku_function(&raku_function (Str --> int32))
  is native("./my_c_dll.dll") { * }

sub start_c_processing()
  is native("./my_c_dll.dll") { * }

sub my_raku_function(Str $str --> Int)
{
  say "@ raku function called from C with parameter [{$str}]";
  return 32;
}

install_raku_function &my_raku_function;

start { start_c_processing; }

for ^1000 -> $i
{
  say "# $i idling in raku";
  sleep 1;
}                               

---

$ raku test-dll.raku                   
installing raku function               
# 0 idling in raku                     
* ----> starting C processing..        
* 0 calling raku function              
@ 0 raku function called from C with parameter [hello]
* 0 raku function returned: 32         
# 1 idling in raku                     
* 1 calling raku function              
@ 1 raku function called from C with parameter [hello]
* 1 raku function returned: 32         
# 2 idling in raku                     
* 2 calling raku function              
@ 2 raku function called from C with parameter [hello]
* 2 raku function returned: 32         
# 3 idling in raku                     
* 3 calling raku function              
@ 3 raku function called from C with parameter [hello]
* 3 raku function returned: 32         
# 4 idling in raku                     
* 4 calling raku function              
@ 4 raku function called from C with parameter [hello]
* 4 raku function returned: 32         
# 5 idling in raku                     
* 5 calling raku function              
@ 5 raku function called from C with parameter [hello]
* 5 raku function returned: 32         
^CTerminate batch job (Y/N)?           
^C                                     

What amazes me is that the Raku signature for my_raku_function maps cleanly to the C signature ... isn't Raku wonderful ? :)

Filature

It really is :-) – Kernel

Cool that you made it work on Windows, I am trying on Windows now, but I get "Cannot locate native library '(null)': error 0xc1" when I run the script. How did you compile the shared library? I compiled with Visual Studio 2019 from the developer command prompt: cl.exe /c my_c_dll.c and then link /DLL /OUT:my_c_dll.dll my_c_dll.obj – Rebarebah

@HåkonHægland you're creating a 32-bit dll ... 0xc1 ---> 193 msdn ... raku wants a 64-bit dll ... I've used cmake to create the solution, so I had to tell cmake to create a 64-bit solution ... in your case, you have to open a 64-bit developer prompt ... I have VS2017, it has 5 "developer prompts" in the start menu ... one of them will be the 64-bit shell for visual studio for sure – Filature

@Filature Interesting! Yes there are many prompts in the start menu, I now tried the one called "x86 Native Tools Command Prompt for VS 2019", but it gives the same error here too.. – Rebarebah

@Filature My bad, I needed to use the "x64 Native Tools Command Prompt for VS 2019" (not the x86 of course) It now works! I posted a summary here for reference – Rebarebah