PInvoke lmdif1 signature

Question

I am trying to Pinvoke the lmdif1 method in the cminpack_dll.dll from c# and i am running into some curious errors.

The 2nd and third parameters passed to lmdif1 are ints, and for my test the values in order are 12 and 9. Now when ther are in the C code, the value that was 12 is now 9 and the value that was 9 varies between 308000 and 912000. No idea why.

First of all, I was wondering if the signature I am using is valid

the C# signature:

[DllImport("cminpack_dll.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int lmdif1(IntPtr fcn, int m, int n, double[] x, double[] fVec, double tol, int[] iwa, double[] wa, int lwa);

The C signature:

int __cminpack_func__(lmdif1)(__cminpack_decl_fcn_mn__ void *p, int m, int n, real *x, 
real *fvec, real tol, int *iwa, 
real *wa, int lwa)

And the way I am calling it:

//All of the variables passed in match the types in C# signature above.
var info =lmdif1(
            functionPointer,
            pointsetLength,
            initialGuessLength,
            initialGuess,
            fVec,
            tol,
            iwa,
            wa,
            lwa);

Now this is my first time dealing with PInvoke, and my C isn't great, having never really done it before, so any help would be great. My suspicion is that I may need to Marshal things, but I've tried marshalling the ints as I4 and as U4 and it still does the same.

Cheers in Advance for your help.

EDIT: Here is the comment that describes the C function, if that helps:

/*     ********** */

/*     subroutine lmdif1 */

/*     the purpose of lmdif1 is to minimize the sum of the squares of */
/*     m nonlinear functions in n variables by a modification of the */
/*     levenberg-marquardt algorithm. this is done by using the more */
/*     general least-squares solver lmdif. the user must provide a */
/*     subroutine which calculates the functions. the jacobian is */
/*     then calculated by a forward-difference approximation. */

/*     the subroutine statement is */

/*       subroutine lmdif1(fcn,m,n,x,fvec,tol,info,iwa,wa,lwa) */

/*     where */

/*       fcn is the name of the user-supplied subroutine which */
/*         calculates the functions. fcn must be declared */
/*         in an external statement in the user calling */
/*         program, and should be written as follows. */

/*         subroutine fcn(m,n,x,fvec,iflag) */
/*         integer m,n,iflag */
/*         double precision x(n),fvec(m) */
/*         ---------- */
/*         calculate the functions at x and */
/*         return this vector in fvec. */
/*         ---------- */
/*         return */
/*         end */

/*         the value of iflag should not be changed by fcn unless */
/*         the user wants to terminate execution of lmdif1. */
/*         in this case set iflag to a negative integer. */

/*       m is a positive integer input variable set to the number */
/*         of functions. */

/*       n is a positive integer input variable set to the number */
/*         of variables. n must not exceed m. */

/*       x is an array of length n. on input x must contain */
/*         an initial estimate of the solution vector. on output x */
/*         contains the final estimate of the solution vector. */

/*       fvec is an output array of length m which contains */
/*         the functions evaluated at the output x. */

/*       tol is a nonnegative input variable. termination occurs */
/*         when the algorithm estimates either that the relative */
/*         error in the sum of squares is at most tol or that */
/*         the relative error between x and the solution is at */
/*         most tol. */

/*       info is an integer output variable. if the user has */
/*         terminated execution, info is set to the (negative) */
/*         value of iflag. see description of fcn. otherwise, */
/*         info is set as follows. */

/*         info = 0  improper input parameters. */

/*         info = 1  algorithm estimates that the relative error */
/*                   in the sum of squares is at most tol. */

/*         info = 2  algorithm estimates that the relative error */
/*                   between x and the solution is at most tol. */

/*         info = 3  conditions for info = 1 and info = 2 both hold. */

/*         info = 4  fvec is orthogonal to the columns of the */
/*                   jacobian to machine precision. */

/*         info = 5  number of calls to fcn has reached or */
/*                   exceeded 200*(n+1). */

/*         info = 6  tol is too small. no further reduction in */
/*                   the sum of squares is possible. */

/*         info = 7  tol is too small. no further improvement in */
/*                   the approximate solution x is possible. */

/*       iwa is an integer work array of length n. */

/*       wa is a work array of length lwa. */

/*       lwa is a positive integer input variable not less than */
/*         m*n+5*n+m. */

/*     subprograms called */

/*       user-supplied ...... fcn */

/*       minpack-supplied ... lmdif */

/*     argonne national laboratory. minpack project. march 1980. */
/*     burton s. garbow, kenneth e. hillstrom, jorge j. more */

/*     ********** */

/*     check the input parameters for errors. */

Parisi

It would appear from the header on Windows there is additional first parameter in this function, cminpack_func_nn fcn_nn, which is another IntPtr. – Chanterelle

That would explain the mismatch in the ints, as the one that varies constantly will be the memory address of the function pointer. As I said above, my C isn't the best, i have no idea what I would need to pass as that intptr. I'll do alittle more digging and let you know. If this is the solution, would you add this comment as an answer, and I will accept it. – Parisi

It would appear it is a pointer to int, not a function pointer (I'm not a C person either). I have no idea what it should point to, probaly there is an answer in the docs. – Chanterelle

I have updated the question with the comment describing the method that i am trying to access. If your eyes can see something that mine can't feel free to let me know. – Parisi

Re-Reading your comment @Chanterelle this does not use the fcn_nn, it says fcn_mn, and that has no additional first parameter. – Parisi

It doesn't, it is macro soup. Fortran doesn't get any uglier than this. I also randomly guess at "real" being float. Try getting this going in C++/CLI instead, better odds for the macro soup to work. – Tirewoman

real is double. In the header it is cminpack_real double – Parisi

@Hans C++/CLI isn't needed here. All you need is a pre-processor. Once you understand what the macros do it's easy peasy. – Biogeography

Answer 1

I think the fundamental issue here is that you've no real way to tell what the native code actually is. You need to understand all the macros to do that.

So, here's what I did. I downloaded the library and passed the lmdif1.c file through a C pre-processor, and hence expanded the macros. I used the one from my mingw compiler. It produced the following output:

int __attribute__((__dllexport__)) lmdif1(cminpack_func_mn fcn_mn,
    void *p, int m, int n, double *x, double *fvec, double tol,
    int *iwa, double *wa, int lwa);

Then looking at the definition of cminpack_func_mn we have:

typedef int (*cminpack_func_mn)(void *p, int m, int n, const double *x, 
    double *fvec, int iflag);

So there is an extra void pointer that lmdif1 receives. Since the function pointer type cminpack_func_mn also receives a void pointer with the same non-descript name, I am prepared to bet that the pointer you pass to lmdif1 is passed back to your callback function fcn_mn. This mechanism is typically used to allow the consumer of the library to write a callback function that has access to extra state.

If you don't need that, and you surely won't with a C# delegate, you can pass IntPtr.Zero and ignore the value in the callback.

To fix this you'll need to make three changes:

1. Add the void pointer to the C# declaration of lmdif1.
2. Add the void pointer to the C# declaration of the callback delegate, and your function that implements the callback.
3. Pass IntPtr.Zero to the extra void pointer parameter of lmdif1.

I'm not sure why you declared your callback parameter as IntPtr. You can use the delegate type here and also use the UnmanagedFunctionPointer attribute to enforce Cdecl.

Update: For completeness I dug into the implementation of lmdif1 and looked at how the callback is called. And yes, the void pointer p is passed back to the callback as described above.

Biogeography

Do you mean an extra void pointer that isn't p? I have gotten the pointer to P by using Marshal.GetFunctionPointerForDelegate, but you say there needs to be another void pointer before p? @DavidHeffernan Apologies if these questions seem stupid, Like i claimed above, I am no C buff – Parisi

Read the code in my answer. There's the function pointer first, and then an extra void pointer that you don't need. Count the parameters in the function as written in my answer. The code there is the code after macro expansion. You probably need to learn about macros to see what is happening. For sure my answer is accurate. – Biogeography

As for the function pointer callback you are doing that the hard way. You should make that param be the delegate type and not IntPtr. – Biogeography

Answer 2

Adding to @David Heffernan's excellent answer, it might be worth noting that for the callback, using double[] for the x and fvec arrays will not quite suffice, as the information on array lengths is lost in the process. As such, the final signatures for the C function and the delegate may look something like

[DllImport("cminpack_dll.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int lmdif1(CminpackFuncMn fcn, IntPtr p, int m, int n, double[] x,
    double[] fvec, double tol, int[] iwa, double[] wa, int lwa);

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate int CminpackFuncMn(IntPtr p, int m, int n, IntPtr x, IntPtr fvec, int iflag);

With this, an example of usage is the following, in which we fit a quadratic to some test data:

// Define some test data by 5i + 3i^2. The plan is to let cminpack figure out
// the values 5 and 3.
var data = Enumerable.Range(0, 20)
    .Select(i => 5 * i + 3 * Math.Pow(i, 2))
    .ToList();

CminpackFuncMn residuals = (p, m, n, x, fvec, iflag) =>
{
    unsafe
    {
        // Update fvec with the values of the residuals x[0]*i + x[1]*i^2 - data[i].
        var fvecPtr = (double*)fvec;
        var xPtr = (double*)x;
        for (var i = 0; i < m; i++)
            *(fvecPtr + i) = *xPtr * i + *(xPtr + 1) * Math.Pow(i, 2) - data[i];
    }
    return 0;
};

// Define an initial (bad, but not terrible) guess for the value of the parameters x.
double[] parameters = { 2d, 2d };
var numParameters = parameters.Length;
var numResiduals = data.Count;
var lwa = numResiduals * numParameters + 5 * numParameters + numResiduals;

// Call cminpack
var info = lmdif1(
    fcn: residuals,
    p: IntPtr.Zero,
    m: numResiduals,
    n: numParameters,
    x: parameters,
    fvec: new double[numResiduals],
    tol: 0.00001,
    iwa: new int[numParameters],
    wa: new double[lwa],
    lwa: lwa);

// info is now 2, and parameters are { 5, 3 }.
Console.WriteLine($"Return value: {info}, x: {string.Join(", ", parameters)}");

Riendeau