Use the Eigen library with cppyy

Question

I've been successfully using cppyy for automatic python bindings for a C++ project I'm working on. I recently included the Eigen library, but I'm having trouble using this together with cppyy. Does anyone have any experience doing this, or know how I should do this?

I have the following structure for the repo (only relevant parts shown):

.
├── CMakeLists.txt
├── build
├── external
   ── eigen
├── include
   ── all .hpp files
├── src
   ── all .cpp files
├── python
   ── qmc.py

Here the external/eigen is a copy of the Eigen GitHub repo. The qmc.py file is where the cppyy magic happens, and it looks like this (before trying to add Eigen, this works just fine)

import cppyy
import tempfile
import os
import glob

try:
    current_dir = os.path.dirname(__file__)
except NameError:
    current_dir = os.getcwd()
source_dir = os.path.dirname(current_dir)
install_dir = os.path.join(source_dir, 'build')
include_dir = os.path.join(source_dir, 'include')
eigen_dir = os.path.join(source_dir, 'external', 'eigen')
print(current_dir, source_dir, include_dir, install_dir)

def cmake_run(build_type='Release', c_compiler='gcc', cxx_compiler='g++'):
    os.environ['CC'] = c_compiler
    os.environ['CXX'] = cxx_compiler
    os.system('cd {} && cmake {} -DCMAKE_BUILD_TYPE={}'.format(install_dir, source_dir, build_type))

def load_library():
    os.system('cd {} && make engine'.format(install_dir))
    libraries = glob.glob(os.path.join(install_dir, 'libengine.*'))
    print('Found libraries: {}'.format(libraries))
    library = libraries[0]
    cppyy.load_library(library)
    for header in glob.glob(os.path.join(include_dir, '*.hpp')):
        print('Loading {}'.format(header))
        cppyy.include(header)

The build part works, but as soon as I try to load any header that uses Eigen, I get an error. I've tried just about everything I can think of (include needed headers manually one by one, copying the entire library into the build dir etc.) but regardless of what I do the same type of errors pop up. Something like

In file included from 
/path/to/repo/projects/include/myheader.hpp:3:10: fatal error: 'Eigen/Dense' file not found
#include <Eigen/Dense>
         ^~~~~~~~~~~~~

Any help with what to change here would be much appreciated!

Edit: To be clear, the build step works just fine, i.e. the code compiles, links and runs as it should. Loading the library with cppyy also works. The issue is that cppyy also needs to include the header files. Again, this works for my own headers, but it is unable to find the Eigen headers...

Enterovirus

Can you manually look for the Dense file? What is the absolute path? – Ole

@AviGinsburg I can have cppyy include any one particular header, for instance, including Eigen/Dense with the full path. For instance cppyy.include(os.path.join(eigen_dir, 'Eigen', 'Dense')) finds the file. Here however I get other file-not-found problems, strangely it complains it cannot find assert.h. I fear this is just some configuration bug that's quite specific to my system, and as such quite hard to get help with... – Enterovirus

Does external/eigen appears in CMakeLists.txt? Try to add include_directories(external/eigen) if not. – Akmolinsk

@Akmolinsk I have that in there, yes. I've updated the question to be a bit clearer, but the issue is not with cmake, as the code builds just fine. The issue is related to the use of cppyy. – Enterovirus

Answer 1

When calling help(), there is:

>>> import cppyy
>>> help(cppyy)
    """
    add_include_path(path)
        Add a path to the include paths available to Cling.
    """
>>>

So with eigen_dir being the path to Eigen, this should be the ticket:

cppyy.add_include_path(eigen_dir)

There are better ways, though, since you are using cmake already. See this repo: https://github.com/jclay/cppyy-knearestneighbors-example. With that, auto-loading should work. I.e. no need to deal with libraries and headers in your own code.

Maricela

Oo, help from the man himself, thanks! I wasn’t aware of add_include_path, but that does sound very much like the solution. I’ll try it out soon and accept the answer if/when that works. And that example looks neat, I’ll give that a shot as well! – Enterovirus

Sure enough, that line fixes it. I got very curious about the CMake approach, but I've been stuck for hours. Finally found the issue, I can't get pyhon to work with clang, and so FindLibClang.cmake fails... Guess I'll ask for that next! – Enterovirus