Link Waf target to a library generated by external build system (CMake)

Question

My waf project has two dependencies, built with CMake.

What I'm trying to do, is following the dynamic_build3 example found in waf git repo, create a tool which spawns CMake and after a successful build, performs an install into waf's output subdirectory:

@extension('.txt')
def spawn_cmake(self, node):
    if node.name == 'CMakeLists.txt':
        self.cmake_task = self.create_task('CMake', node)
        self.cmake_task.name = self.target


@feature('cmake')
@after_method('process_source')
def update_outputs(self):
    self.cmake_task.add_target()


class CMake(Task.Task):
    color = 'PINK'

    def keyword(self):
        return 'CMake'

    def run(self):
        lists_file = self.generator.source[0]
        bld_dir = self.generator.bld.bldnode.make_node(self.name)
        bld_dir.mkdir()

        # process args and append install prefix
        try:
            cmake_args = self.generator.cmake_args
        except AttributeError:
            cmake_args = []
        cmake_args.append(
            '-DCMAKE_INSTALL_PREFIX={}'.format(bld_dir.abspath()))

        # execute CMake
        cmd = '{cmake} {args} {project_dir}'.format(
            cmake=self.env.get_flat('CMAKE'),
            args=' '.join(cmake_args),
            project_dir=lists_file.parent.abspath())
        try:
            self.generator.bld.cmd_and_log(
                cmd, cwd=bld_dir.abspath(), quiet=Context.BOTH)
        except WafError as err:
            return err.stderr

        # execute make install
        try:
            self.generator.bld.cmd_and_log(
                'make install', cwd=bld_dir.abspath(), quiet=Context.BOTH)
        except WafError as err:
            return err.stderr

        try:
            os.stat(self.outputs[0].abspath())
        except:
            return 'library {} does not exist'.format(self.outputs[0])

        # store the signature of the generated library to avoid re-running the
        # task without need
        self.generator.bld.raw_deps[self.uid()] = [self.signature()] + self.outputs

    def add_target(self):
        # override the outputs with the library file name
        name = self.name
        bld_dir = self.generator.bld.bldnode.make_node(name)
        lib_file = bld_dir.find_or_declare('lib/{}'.format(
            (
                self.env.cshlib_PATTERN
                if self.generator.lib_type == 'shared' else self.env.cstlib_PATTERN
            ) % name))
        self.set_outputs(lib_file)

    def runnable_status(self):
        ret = super(CMake, self).runnable_status()
        try:
            lst = self.generator.bld.raw_deps[self.uid()]
            if lst[0] != self.signature():
                raise Exception
            os.stat(lst[1].abspath())
            return Task.SKIP_ME
        except:
            return Task.RUN_ME
        return ret

I'd like to spawn the tool and then link the waf target to the installed libraries, which I perform using the "fake library" mechanism by calling bld.read_shlib():

def build(bld):
    bld.post_mode = Build.POST_LAZY
    # build 3rd-party CMake dependencies first
    for lists_file in bld.env.CMAKE_LISTS:
        if 'Chipmunk2D' in lists_file:
            bld(
                source=lists_file,
                features='cmake',
                target='chipmunk',
                lib_type='shared',
                cmake_args=[
                    '-DBUILD_DEMOS=OFF',
                    '-DINSTALL_DEMOS=OFF',
                    '-DBUILD_SHARED=ON',
                    '-DBUILD_STATIC=OFF',
                    '-DINSTALL_STATIC=OFF',
                    '-Wno-dev',
                ])
    bld.add_group()

    # after this, specifying `use=['chipmunk']` in the target does the job
    out_dir = bld.bldnode.make_node('chipmunk')
    bld.read_shlib(
        'chipmunk',
        paths=[out_dir.make_node('lib')],
        export_includes=[out_dir.make_node('include')])

I find this VERY UGLY because:

1. The chipmunk library is needed ONLY during final target's link phase, there's no reason to block the whole build (by using Build.POST_LAZY mode and bld.add_group()), though unblocking it makes read_shlib() fail. Imagine if there was also some kind of git clone task before that...
2. Calling read_shlib() in build() command implies that the caller knows about how and where the tool installs the files. I'd like the tool itself to perform the call to read_shlib() (if necessary at all). But I failed doing this in run() and in runnable_status(), as suggested paragraph 11.4.2 of Waf Book section about Custom tasks, seems that I have to encapsulate in some way the call to read_shlib() in ANOTHER task and put it inside the undocumented more_tasks attribute.

And there are the questions:

1. How can I encapsulate the read_shlib() call in a task, to be spawned by the CMake task?
2. Is it possible to let the tasks go in parallel in a non-blocking way for other tasks (suppose a project has 2 or 3 of these CMake dependencies, which are to be fetched by git from remote repos)?

Butterfat

Answer 1

Well in fact you have already done most of the work :)

read_shlib only create a fake task pretending to build an already existing lib. In your case, you really build the lib, so you really don't need read_shlib. You can just use your cmake task generator somewhere, given that you've set the right parameters.

The keyword use recognizes some parameters in the used task generators:

• export_includes
• export_defines

It also manage libs and tasks order if the used task generator has a link_task.

So you just have to set the export_includes and export_defines correctly in your cmake task generator, plus set a link_task attribute which reference your cmake_task attribute. You must also set your cmake_task outputs correctly for this to work, ie the first output of the list must be the lib node (what you do in add_target seems ok). Something like:

@feature('cmake')
@after_method('update_outputs')
def export_for_use(self):
    self.link_task = self.cmake_task
    out_dir = self.bld.bldnode.make_node(self.target)
    self.export_includes = out_dir.make_node('include')

This done, you will simply write in your main wscript:

def build(bld):
    for lists_file in bld.env.CMAKE_LISTS:
        if 'Chipmunk2D' in lists_file:
            bld(
                source=lists_file,
                features='cmake',
                target='chipmunk',
                lib_type='shared',
                cmake_args=[
                    '-DBUILD_DEMOS=OFF',
                    '-DINSTALL_DEMOS=OFF',
                    '-DBUILD_SHARED=ON',
                    '-DBUILD_STATIC=OFF',
                    '-DINSTALL_STATIC=OFF',
                    '-Wno-dev',
                ])

    bld.program(source="main.cpp", use="chipmunk")

You can of course simplify/factorize the code. I think add_target should not be in the task, it manages mainly task generator attributes.

Hogle