Mex function not updated after recompile
Asked Answered
S

2

7

I have a simple mex function, which calls another C++ function from a library. I compile the source with

mex -cxx mymexfunction.cpp -I/some/include -L/some/lib -lmylib

The mylib library is dynamic (.so) and is linked itself against some other libraries (boost, OpenCV and some more).

The problem I am having is that once I have called the function mymexfunction once, it will not get updated when I recompile the source code. I have tried

clear
clear all
clear mex
clear functions
clear fun('mymexfunction')
munlock('mymexfunction')
unloadlibrary('mymexfunction')

...but nothing helps! I have to restart Matlab in order to be able to see the updated mexfunction. Even if I delete the compiled mex file and recompile, I still get the old version of the mex function (not on disk, but in memory).

All is fine if I don't link against mylib, but I have no idea what could be the culprit that prevents the update. The library is unfortunately too large and too interwoven to remove single modules one by one.

Are there some known conditions which could cause such problems?

Clarification:

I only update the content of the mex function, not the library.

Update:

It works under Ubuntu 11.04 with Matlab R2011a! I tried to reproduce the same environment on my OpenSUSE machine (R2011a, Boost 1.42, OpenCV 2.2 dynamically linked, ...) but still no luck. So I conclude that nothing is actually wrong with my library (otherwise it wouldn't work under Ubuntu) but it must be some collision of the dependencies and Matlab internal libraries. I officially give up. Praetorian and Amro, thank you for your help!

Starve answered 10/8, 2011 at 14:27 Comment(0)
T
4

The mex command automatically clears the mex function if it is currently loaded in memory. Are you sure your mex function is closing whatever handle it holds to the other library? If such a handle exists it might prevent the OS from unloading the mex file.

I've used the following set of commands to clear mex functions manually, and from my experience, using a full path to the mex file when calling clear works. So give this a try and if it still doesn't get unloaded you might want to start looking at the code for loading and unloading the other library.

[~,f] = inmem( '-completenames' );
result = strfind( f, ['mymexfile' '.' mexext] );
result = f(cellfun( @isempty, result, 'UniformOutput', true ) == 0);
clear( result{:} )

Try running the inmem command again after the above and see if your mex file is still listed.

Something that might help you with making sure the other library gets unloaded is maybe using an std::shared_ptr to hold the handle to this library. Then, at the beginning of the mexFunction() entry point load the library and stick the handle into the shared_ptr. The shared_ptr will also need to use a custom deleter to unload the library (on Windows the custom deleter would call FreeLibrary).

Of course, if this is being caused by a bug in the other library none of this is going to help.

Tampa answered 10/8, 2011 at 15:19 Comment(11)
Thanks! I could confirm with your code that the mex file is indeed unloaded (f is empty afterwards). BUT: Matlab still does use the previous version if I call the function again!! :(Starve
@Starve But inmem is supposed to report whether there is a cached version in memory! You're not trying to run the mex file in one MATLAB instance and compile it in another instance are you?Tampa
What kind of bug could cause something like this? I don't know how to debug this, or even where to start...Starve
No, it's only one MATLAB instance.Starve
@Starve Start looking for handles to the second library that are not being closed by the mex file. As I mentioned in the answer, if that happens the OS may not unload the mex file even if MATLAB tries to. Maybe inmem tries to unload the file and doesn't (or can't) get confirmation from the OS that it has actually been unloaded. Do you know whether the second library opens any other shared libraries?Tampa
@Pratorian ldd on the library gives the following: linux-vdso.so.1 => (0x00007fff19fff000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f7a4a460000) libtbb.so.2 => /cvhci/distlibs/dists/opensuse-11.4-x86_64/tbb/lib/libtbb.so.2 (0x00007f7a4a234000) librt.so.1 => /lib64/librt.so.1 (0x00007f7a4a02b000) libz.so.1 => /lib64/libz.so.1 (0x00007f7a49e12000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f7a49b09000) libm.so.6 => /lib64/libm.so.6 (0x00007f7a498b2000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7a4969b000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7a4947e000)Starve
libc.so.6 => /lib64/libc.so.6 (0x00007f7a49111000) /lib64/ld-linux-x86-64.so.2 (0x00007f7a4b1a9000)Starve
sorry about the missing format... The only "non-standard" library would be libtbb which is Intel's Threading Building BlocksStarve
OK, I could confirm that both the mex file and my library remain loaded AFTER clear mex (although inmem shows them removed). I did a cat /proc/<matlab pid>/maps and it is still there. Pratorian, thanks for your help so far, I will check back tomorrow...Starve
@Starve Good, now you have a direction to start looking in! Best of luck.Tampa
@Martin: I posted an complete example to help solve the problem, please have a look..Aryl
A
4

Trying to reproduce the problem, I've written a minimum working example for your case: a MEX file that links against a dynamic library and use use one of its exposed functions. I've tested the following on WinXP 32-bit using MATLAB R2010b with VS2010 as compiler (for both the DLL and MEX).

The example simply adds floating-point numbers. The MEX file accepts matrices/vectors and loops over the elements calling the add() function from the library on each pair.

Adder.h

#ifndef ADDER_H
#define ADDER_H

#ifdef __cplusplus
extern "C" {
#endif

#ifdef _WIN32
#   ifdef BUILDING_DLL
#       define DLL_IMPORT_EXPORT __declspec(dllexport)
#   else
#       define DLL_IMPORT_EXPORT __declspec(dllimport)
#   endif
#else
#   define DLL_IMPORT_EXPORT
#endif

DLL_IMPORT_EXPORT double add(double x, double y);

#ifdef __cplusplus
}
#endif

#endif

Adder.c

#include "Adder.h"

double add(double x, double y)
{
    return x+y;
}

mymexfunction.c

#include "mex.h"
#include "Adder.h"

#define X_IN    input[0]
#define Y_IN    input[1]
#define Z_OUT   output[0]

void mexFunction(int output_size, mxArray *output[], int input_size, const mxArray *input[])
{
    double *inX, *inY, *outZ;
    mwSize m,n;
    int i;

    /* check for proper number of arguments */
    if (input_size != 2) {
        mexErrMsgTxt("Two input arguments required.");
    }
    if (output_size > 1) {
        mexErrMsgTxt("Too many output arguments.");
    }

    /* check input argument sizes */
    m = mxGetM(X_IN);
    n = mxGetN(X_IN);
    if ( !mxIsDouble(X_IN) || !mxIsDouble(Y_IN) ) {
        mexErrMsgTxt("Input arguments must be matrices/vectors of doubles.");
    }
    if ( mxGetM(Y_IN)!=m || mxGetN(Y_IN)!=n ) {
        mexErrMsgTxt("X and Y must be of same size.");
    }

    /* Create a matrix for the return argument */
    Z_OUT = mxCreateDoubleMatrix(m, n, mxREAL);

    // get pointers to data
    inX =  (double *) mxGetPr(X_IN);
    inY =  (double *) mxGetPr(Y_IN);
    outZ = (double *) mxGetPr(Z_OUT);

    // compute and store result
    for(i=0; i<m*n; ++i) {
        outZ[i] = add(inX[i], inY[i]);
    }

    return;
}

First we build the dynamic library, as I mentioned I am using VC++ for the job. On Unix-based systems with GCC, I think this step goes like (correct me if I am wrong):

gcc -c -DBUILDING_DLL Adder.c -o Adder.o -I.
gcc -shared -o libAdder.so Adder.o -Wl,--out-implib,libAdder.a

then in MATLAB, we compile the MEX file:

>> mex mymexfunction.c -I. -L. -lAdder

(Note: I put everything in the same folder to avoid dealing with path issues.)

Next, we can test the function in MATLAB:

>> mymexfunction([1 2;3 4], [5 6; 7 8])
ans =
     6     8
    10    12

Using the Process Explorer tool from Sysinternals, we can view the loaded DLLs by the MATLAB process, the MEX function and our custom dynamic library:

sysinternals

If we issue the command clear mex then both modules are unloaded as expected (which is verified using Process Explorer). This is also confirmed by INMEM as @Praetorian showed:

clear mex
[~,m] = inmem('-completenames');
any( ismember(m,fullfile(pwd,['mymexfunction.' mexext])) )

Finally if we make some changes to mymexfunction.c:

// add 10 to all results
outZ[i] = add(inX[i], inY[i]) + 10.0;

recompile the MEX, and test it again (all in the same session, no restart). The result will reflect the changes made as you can see:

>> mymexfunction([1 2;3 4], [5 6; 7 8])
ans =
    16    18
    20    22

Please try to repeat the above on your Mac/Linux machine. If you still receive the old sums, then it must be a bug specific to non-Windows platforms, and should be reported to MathWorks... Otherwise I suspect that in your code, there must be some un-released resources causing the module to remain in memory?

Aryl answered 11/8, 2011 at 15:25 Comment(5)
Thanks for the detailed example! I tried it and of course it works. I also suspect something fishy going on in either my library or one of the libraries it depends upon. However, I was still not able to find out what causes the actual problem...Starve
Is there still no simple solution to this problem?Decare
@mangledor: as was showed in my example above, I was not able to reproduce the problem in the first place. The conclusion at the time was that there were unreleased resources in the OP's external library causing it to remain in memory.. If you have a specific counter-example, consider creating a new question.Aryl
Any clues on how to debug such a problem, then?Decare
@mangledor: it is hard to tell without being able to reproduce the problem. Try running version -modules after calling clear mex to see what modules are still loaded in memory. Otherwise all I can suggest is stepping through your code using a debugger paying special attention to allocated resources..Aryl

© 2022 - 2024 — McMap. All rights reserved.