interop with nim return Struct Array containing a string /char* member

Question

interoping nim dll from c# i could call and execute the code below

if i will add another function (proc) that Calls GetPacks() and try to echo on each element's buffer i could see the output in the C# console correctly but i could not transfer the data as it is, i tried everything but i could not accomplish the task

proc GetPacksPtrNim(parSze: int, PackArrINOUT: var DataPackArr){.stdcall,exportc,dynlib.} =
  PackArrINOUT.newSeq(parSze)
  var dummyStr = "abcdefghij"
  for i, curDataPack in PackArrINOUT.mpairs:
    dummyStr[9] = char(i + int8'0')
    curDataPack = DataPack(buffer:dummyStr, intVal: uint32 i)

type
  DataPackArr = seq[DataPack]
  DataPack = object
    buffer: string
    intVal: uint32

when i do same in c/c++ the type i am using is either an IntPtr or char* that is happy to contain returned buffer member

EXPORT_API void __cdecl c_returnDataPack(unsigned int size, dataPack** DpArr)
{
    unsigned int dumln, Index;dataPack* CurDp = {NULL};
    char dummy[STRMAX];
    *DpArr = (dataPack*)malloc( size * sizeof( dataPack ));
    CurDp = *DpArr;
    strncpy(dummy, "abcdefgHij", STRMAX);

    dumln = sizeof(dummy);

    for ( Index = 0; Index < size; Index++,CurDp++)
    {
        CurDp->IVal = Index;
        dummy[dumln-1] = '0' + Index % (126 - '0');
        CurDp->Sval = (char*) calloc (dumln,sizeof(dummy));
        strcpy(CurDp->Sval, dummy);
    }

}

c# signature for c code above

    [DllImport(@"cdllI.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
    private static extern uint c_returnDataPack(uint x, DataPackg.TestC** tcdparr);

C# Struct

public unsafe static class DataPackg
{

   [StructLayout(LayoutKind.Sequential)]
    public struct TestC
    {
        public uint Id;
        public IntPtr StrVal;
    }

}

finally calling the function like so:

    public static unsafe List<DataPackg.TestC> PopulateLstPackC(int ArrL)
    {
        DataPackg.TestC* PackUArrOut;
        List<DataPackg.TestC> RtLstPackU = new List<DataPackg.TestC>(ArrL);
        c_returnDataPack((uint)ArrL, &PackUArrOut);
        DataPackg.TestC* CurrentPack = PackUArrOut;
        for (int i = 0; i < ArrL; i++, CurrentPack++)
        {

            RtLstPackU.Add(new DataPackg.TestC() { StrVal = CurrentPack->StrVal, Id = CurrentPack->Id });
        }
        //Console.WriteLine("Res={0}", Marshal.PtrToStringAnsi((IntPtr)RtLstPackU[1].StrVal));//new string(RtLstPackU[0].StrVal));
        return RtLstPackU;
    }

how could i produce similar c code as above from Nim ?

_{it doesn't have to be same code, but same effect, that in c# i would be able to read the content of the string. for now, the int is readable but the string is not}

Edit:

this is what i tried to make things simple struct array of int members

Update:

it seem that the problem is to do with my settings of nim in my windows OS. i will be updating as soon as i discover what exactly is wrong.

Afteryears

Answer 1

The string type in Nim is not equivalent to the C's const char* type. Strings in Nim are represented as pointers, pointing into a heap-allocated chunk of memory, which has the following layout:

NI length;   # the length of the stored string
NI capacity; # how much room do we have for growth
NIM_CHAR data[capacity]; # the actual string, zero-terminated

Please beware that these types are architecture specific and they are really an implementation detail of the compiler that can be changed in the future. NI is the architecture-default interger type and NIM_CHAR is usually equivalent to a 8-bit char, since Nim is leaning towards the use of UTF8.

With this in mind, you have several options:

1) You can teach C# about this layout and access the string buffers at their correct location (the above caveats apply). An example implementation of this approach can be found here: https://gist.github.com/zah/fe8f5956684abee6bec9

2) You can use a different type for the buffer field in your Nim code. Possible candidates are ptr char or the fixed size array[char]. The first one will require you to give up the automatic garbage collection and maintain a little bit of code for manual memory management. The second one will give up a little bit of space efficiency and it will put hard-limits on the size of these buffers.

EDIT: Using cstring may also look tempting, but it's ultimately dangerous. When you assign a regular string to a cstring, the result will be a normal char * value, pointing to the data buffer of the Nim string described above. Since the Nim garbage collector handles properly interior pointers to allocated values, this will be safe as long as the cstring value is placed in a traced location like the stack. But when you place it inside an object, the cstring won't be traced and nothing prevents the GC from releasing the memory, which may create a dangling pointer in your C# code.

Petiole

in the same spirit and hours after trying every possible way on the c# side , what about just changing string to cstring ? i got it working like that few minutes before you posted your answer , now that it works with cstring as the buffer on nim's side, please just let me know how to fix the trailing digit value char in the loop and i will close the deal (: – Afteryears

so looping over a struct-like type assigning its cstring member with some random values(here the trailingdigit just simulates random data could be file names or database query fed in to) so until what point it's safe and where does it looses it's safty , as soon as loop is finished , control returns to c# there i put it in to a list, the proc in nim has allready exited... – Rebut

@zach could you show little example as to using option for buffer of type ptr char as i'd like to test it (also as my code is a bit unstable while looping in c# over the returned collection i see some disorder when printing to console) so i'll epriciate some code sample implementing candidate #1 – Afteryears

Let's collaborate on a gist for some final working code that I'll include in the answer: gist.github.com/zah/fe8f5956684abee6bec9 . By the way, you may have everything a bit easier if you create a single managed C++ module that uses the magic of IJW to implement all the marshalling: msdn.microsoft.com/en-us/library/aa712982(v=vs.71).aspx – Petiole

first of all thanks a lot for the effort, i will now have a look at that gist thanks, and second, i will be happy to learn about that option "C++ module that uses magic of IJW" i did visit that second link i need to better understand what is that IJW.. – Afteryears

zah i have updated the gist , left you a comment and made a fork. please chek it if you can. thanks. – Afteryears

i have an update on regarding the reason i have failed to implement, it seem that with every iteration the fields exchange place when marashalling now it has nothing to do with data types (yet) first there's a problem with the layout (i have tried to set different layouts and i have tried to change call to stdcall insted of cdecl) .. but no chang its spinning the array is not sequent something is wrong and it's nothing to do with the string vs cstring at this point did u ever encounter that ? – Afteryears

@AviaAfer, I've updated the gist with a working example, involving sequences and strings. – Petiole

@Petiole i have updated comments in nimSharp Gist, and added my benchmark project folder thanks for the effort again. – Afteryears

@Petiole first problem solved, i don't know how to thank you enough, and for now, i suggest that i will further investigate what is it that i do wrong with setting of the nim compiler, as soon as i rebuild/compile nims dll it's not running against the executable of the call from c#. – Afteryears

Answer 2

Try to change your struct to:

public unsafe static class DataPackg
{
   [StructLayout(LayoutKind.Sequential)]
   public struct TestC
   {
      public uint Id;
      [MarshalAs(UnmanagedType.LPStr)]
      public String StrVal;
   }
}

Circumfluous

kodre you got it wrong, as i did not post the c# code( i will edit and add), the return type is an array of structs, not a string – Afteryears

i will post the usage code too, thanks for trying to help – Afteryears

any way what did you suggest? the problem is not with the c implementation but the nim one, the c code is working, the c# call and signature translates the content as expected, but the nim code returns allmost same but i could properly see only the int value, but not the string value – Afteryears

I thought that is something wrong with translating nim/c char* to string in c#. Because of that i set [MarshalAs(UnmanagedType.LPStr)] for this extern method – Circumfluous

can u add an attribute like [MarshalAs(UnmanagedType.LPStr)] for the string on the return type which is a pointer to an array and it would apply on the content of the array ? – Rebut

in your edit, you added the struct with a string which could not be marashaled via a pointer to the array did you try it? if you actually tested it to work, please edit the signature on the dll import – Rebut

@Raj Felix I don't know if you can. I didn't saw that return type was array, i thougth it is char*. He says that his dll import signature is working ok, so only struct should be modified – Circumfluous

the return is DataPackg.TestC* you could test the whole code by copying the c# code posted with the question and then you should see the implementation with c and C# does work as it is written – Rebut

@Circumfluous the code in the question regarding the c implementation is perfectly fine, i guess you could manipulate it to work with a string to be marshaled in another approach, though the issue here is not about c problem, but as the implementation with nim to try do same with nim as i did with the c implementation is problematic for now, so c code is working good nim not, are you familiar with nim – Afteryears

@AviaAfer I'm not familiar with nim, but i saw that nim is compiled to c and then to native code. Maybe you should look at generated c code to see how DataPack structure was generated. – Circumfluous

thanks i did, it's not that clear to read an autogenerated code, i could follow until i reach a point where the typedef is not clear or i can't follow where it's declared, i'll try again, thanks for looking into it – Rebut