Connect to already running MATLAB with MATLAB Engine

Question

The MATLAB Engine API allows accessing MATLAB functionality from a C program. To set up a MATLAB Engine session, one calls the engOpen function. By default, this launches a new instance of MATLAB, for the exclusive use of Engine applications.

What I want is to use an already running MATLAB session from an Engine application so I can access its variables. I need to do this on Linux and OS X.

The Engine interface has very different implementations on Windows (where it uses a COM server) and Unix. On Unix we have to pass the location of the MATLAB executable to engOpen, while on Windows (where the Engine uses a COM server) we don't. On Windows it is possible to use an existing instance of MATLAB for Engine applications: just run enableservice('AutomationServer', true). Is there any solution on Unix systems?

I am hoping that there is a general solution—perhaps undocumented—because, based on the documentation, the Python interface seems to allow this. I tested this on OS X and it works. I can, for example, start up an interactive MATLAB session, set a variable a=5, then start Python, connect to the same session and be able to retrieve this variable. It is also possible to connect to an already running session from Java.

I need to do this from C however, not from Python, as it will be used in MATLink, the Mathematica-MATLink interface. How does the Python interface achieve this? How can I reproduce the same using C?

If there is an alternative C API than the "MATLAB Engine" that makes this possible (perhaps some other documented or undocumented C API that the Python interface is based on), I can accept that as an answer. I want a C program that can make use of an already running interactive MATLAB session.

---

Update:

Some poking around in the Python interface reveals that it makes use of a library called libmwengine_api. This is not the same as the documented MATLAB Engine C API. We can look at the symbols in this library. On OS X,

nm -g libmwengine_api.dylib | c++filt

Then we can google for these symbols or grep the MATLAB installation directory for files containing them. There is nothing in plain text that turns up.

Based on this I believe that the Python interface uses an undocumented C++ Engine API which is distinct from the old, documented C one.

Token

I think that if there is an alternative API, it must be related to the libmwengine_api library. Listing symbols in this library reveals that it is C++, not C. That is not a problem for me, I use C++ anyway. grepping the whole MATLAB installation directory for e.g. findSharedMATLAB does not reveal any associated header files (only binaries), which strongly suggests that this interface is undocumented. – Token

see this pragmatec.net/CDROM/ARM9/CDROM_ARM9_v1_09/Linux_26/Dev/user/… – Courthouse

It is part of library that was obseleted long back ago. – Courthouse

Did you found Your Answer ? – Courthouse

@SurajJain Unfortunately, no. It looks like this alternative engine API is the only way through C/C++, and it is undocumented. I don't think that you are correct that is it obsolete, as your link mentions other libraries with similar but distinct names. I think it was never documented. One solution could be to use the Java API, which has the required functionality, and is documented. The MathWorks support told me that it is not currently possible to do what I want from C (or C++) on OS X and Linux. It is only possible on Windows. – Token

The Link Said "This affects libmwengine, libmwdrivers, libmwfonts * * and libmwinlib. " , and you are talking about libmwengine_api. – Courthouse

@SurajJain I think that link is to the Microwindows sources and has nothing to do with MATLAB ... – Token

So Should i remove my answer or edit it to just include that you can check the python implementation source here. – Courthouse

Answer 1

"On the Mac and Linux® platforms, you cannot make an engine program connect to an existing MATLAB session."

Mariehamn

Thanks for pointing to that. Perhaps this was the case in the past, but now it is possible to connect to a running session. The proof is that the python interface, which is also branded as "MATLAB Engine" can do this on OS X. I just tested it, and updated the question accordingly. Perhaps the solution uses undocumented functions. That's okay, I can accept an undocumented solution as the answer. – Token

@Token I search about it maybe it's use a named pipe or socket or IPC. It could be an interesting hack. – Mariehamn

I am going to ask MathWorks support and will report back. – Token

@Token I didn't know that MATLAB is not free(2000€, at this price I hope they had a good support). Good luck in your search. – Mariehamn

They were not very helpful. I think there isn't any good way to do it, as you said. – Token

Answer 2

Here is the source of its Python Implementation , see if it helps you.

Some Searching Gave Me The Impression that libmwengine is part of a library that is used in embedded system and which was long back obsoleted.

See This link , Library libmwengine_api was Obsoleted long back.Maybe That is why it is undocumented.

Courthouse

It looks like this is the same as the answer by @Stargateur. Please see my comment there. – Token

@Token SO basically you are saying you want it to work on linux ? – Courthouse

If any part of my question is confusing, please point it out. The Python and Java interfaces can connect to an already running session on OS X or Linux. How do they do it? How can I replicate this from C, using any documented or undocumented C API that MATLAB offers? – Token

@Token See Again. – Courthouse

Suraj, I wrote MATLink myself. I am asking this question so I can improve MATLink. I want to be able to have two interactive MATLAB and a Mathematica session side by side and access the MATLAB workspace from either one. I am looking to find any APIs that makes this possible on OS X and Linux as well. On Windows it is already working. – Token

@Token Yeah , i understand now , i saw this discussion in.mathworks.com/matlabcentral/answers/… – Courthouse

@Token See this if it can help you phidgets.com/docs/Language_-_MATLAB#Use_our_Examples_3 – Courthouse

Thanks! I'll take a look at how they do things. – Token

@Token Also This undocumentedmatlab.com/blog/jmi-wrapper-remote-matlabcontrol – Courthouse

@Token See its python implementation source here pypi.python.org/pypi/pymatlab – Courthouse

Answer 3

I am able to connect to an already-running MATLAB session in my command line on MacOS (should work on any Linux). I haven't figured out how to connect to the already-running MATLAB session from the MATLAB IDE, but my personal motivation was to not have to use the IDE. I keep a MATLAB kernel running in the background indefinitely and connect to it as I like. This allows my to run MATLAB scripts in batch mode from my text editor without having to spin up a new kernel each time.

Here's how I did it. If you want more details, I would be happy to provide the full scripts that accomplish it:

1. Start a MATLAB kernel/session from Python. Rather than the MATLAB Engine API, which I found hard to use, I opted for an existing iPython Jupyter Kernel interface called imatlab. I had to install this (pip install imatlab) as well as Jupyter (pip install jupyter):

from notebook.services.kernels.kernelmanager import MappingKernelManager
m = MappingKernelManager()
m.start_kernel(kernel_name="imatlab")
...

This creates a .json "connection file" in the working directory with the name kernel-kernel_id.json where kernel_id a UID (e.g. 4931ac70-e8fd-4d35-81b2-de53e07956c8). I also write the name of this kernel to a file in my Python script:

id = m.list_kernel_ids()[0]
text_file = open("/Users/Zach/.matlab_kernel/matlab_kernel_name.txt", "w")
text_file.write(id)
text_file.close()

I actually do this in a background TMUX session so I don't have to leave my terminal window open. Happy to comment on that as well.

2. Connect to this MATLAB kernel using Jupyter's console interface (though I could connect a Jupyter notebook connected to this kernel as well). First I have to retrieve the kernel_name from the file I wrote it to:

MATLABKERNELNAME=`cat /Users/Zach/.matlab_kernel/matlab_kernel_name.txt`
cd /Users/Zach/.matlab_kernel
jupyter console --existing $MATLABKERNELNAME

I also made a variant of this bash command that opens the kernel and sends the command to source a MATLAB .m file:

mtlb_existing_run_expect(){
        thefilename=`to_abs_path $1`
        thecurrentdirectory=`pwd`
        thekernelname=`cat /Users/Zach/.matlab_kernel/matlab_kernel_name.txt`
        expect <(cat <<EOF
            cd /Users/zach/.matlab_kernel
            spawn jupyter console --existing $thekernelname --simple-prompt --no-confirm-exit && exit
            expect ": "
            send "cd '$thecurrentdirectory'\n"
            expect ": "
            send "run('$thefilename')\n"
            interact -u "exp0"
EOF
)
}

You'll notice I always cd into the directory in which I started the kernel. That's because Jupyter creates that .json "connection file" there, and it automatically reads it if you're in the directory and give the kernel UID as the argument to the --existing flag.

I hope this is helpful to somebody!

Kaiulani

Thanks for taking the time to answer, but notice that the question is about using the MATLAB Engine API. This answer is not about the MATLAB Engine, and is not helpful for the application I described. – Token

It may seem that way, but I tried very hard to achieve this using only the MATLAB Engine and could not find a way to do it (on Unix). I believe there’s no way, and I even opened an issue with Mathworks since I feel strongly it should be possible. In fact, as far as I can tell there’s no good way to start the engine in MATLAB, and connect to it from any of the language APIs in such a way that you have a live interpreter. On the other hand, my solution allows you to open a kernel and connect to it (via iPython, which implements its own REPL environment), to use as a REPL or to run scripts. – Kaiulani

To be clearer, if you specifically want to achieve a REPL environment from an external MATLAB kernel, the MATLAB Engine provides absolutely no mechanism. You’d have to build it from scratch. That led me to think, “maybe someone’s built a REPL framework before,” which led me to Jupyter/iPython. Since they’re not interpreted languages (jshell notwithstanding), I doubt C/Java have mature frameworks that can be combined with their MATLAB Engine APIs to achieve this. – Kaiulani

It seems there is a way with the new C++ Engine API that I think came out around R2017, but I have not yet had the time to investigate. This new API is much more advanced than the old Engine API for C. – Token

I don't want a REPL. As I wrote in the question, this is for the MATLink system, a Mathematica package that makes it possible to call MATLAB from Mathematica. – Token

Okay! I think if it were me, I’d give up and have Mathematica launch MATLAB code through its API to the UNIX shell. If you’d be okay with that, this would be a solution to run MATLAB code on an already-running engine from Mathematica. This is definitely a detour from your original question, but I hope it’s relevant to someone someday! – Kaiulani