This is the sixth post in my SCons series. The topic of this post is building reusable infrastructure that can extremely simplify your module-level SConscript files.

Starting with the first non-trivial SCons project, the module-level SConscript files contained too much repetitive code. The goal of this enhancement is to go back to minimalistic SConscript files. The objective is to let the developer define the module-level targets with minimal code, and no hassle.

I continue using the same C++ project that I introduced in the basic example. In this post I present SCons shortcuts that are available in module-level SConscript files. These shortcuts are Python functions that take care of dirty details behind the scenes.

The final result is available on my GitHub scons-series repository.

As a reminder, the (seemingly silly) C++ project is a simple address book program. Refer to a previous post if you need more details.

Bottom Line Up Front

The SConscript files I start with look something like this:

Import('*')

module_targets = dict()
module_targets['addressbook'] = env.Library('addressbook', ['addressbook.cc'])

Return('module_targets')

Import('*')

module_targets = dict()
module_targets['writer'] = env.Program('writer', ['writer.cc'] + env.get_targets('addressbook'))

Return('module_targets')

You can see the repetitiveness I’m talking about. Why does a developer need to worry about maintaining and returning a module_targets dictionary? Is it really necessary to mention the name of every target twice? Is it possible to avoid referring to env explicitly all over the place?

The SCons shortcuts I introduce allow for much simpler SConscript files:

Import('*')

Lib('addressbook', 'addressbook.cc')

Import('*')

Prog('writer', 'writer.cc', with_libs='AddressBook::addressbook')

Now, that’s how I like my build recipes!

Implementation Details

The starting point for this enhancement is the last episode (see on GitHub).

The “trick” is to take advantage of the exports argument of the SConscript() function, to make the shortcut functions available to the module-level SConscript file. You can see this in the process_module() method in site_scons/site_init.py file. Here’s an abbreviated version:

def process_module(self, module):
    print 'scons: |- Reading module', module, '...'
    # Prepare shortcuts to export to SConscript
    shortcuts = dict(
        Lib       = self._lib_wrapper(self._env.Library, module),
        StaticLib = self._lib_wrapper(self._env.StaticLibrary, module),
        SharedLib = self._lib_wrapper(self._env.SharedLibrary, module),
        Prog      = self._prog_wrapper(module),
    )
    # Execute the SConscript file, with variant_dir set to the
    #  module dir under the project flavored build dir.
    self._env.SConscript(
        sconscript_path,
        variant_dir=os.path.join('$BUILDROOT', module),
        exports=shortcuts)

The exports argument takes a dictionary whose keys are names of symbols to export, and values are the exported values. The shortcuts are the shorthand names I want to make available in SConscript files (Lib, Prog, etc.). The values assigned to those shortcuts are customized builders created by the wrapper methods, _lib_wrapper and _prog_wrapper. Each wrapper method returns a function that wraps an underlying SCons builder (e.g. Library, Program), and also performs the extra steps I want to hide from the SConscript. Specifically, the lib-wrapper needs to populate the known libraries dictionary, and the prog-wrapper needs to take care of adding library nodes specified via the with_libs argument.

The Library wrappers

The custom library builder is returned by the _lib_wrapper method in site_scons/site_init.py. It’s pretty straight forward:

def _lib_wrapper(self, bldr_func, module):
    """Return a wrapped customized flavored library builder for module.

    @param  builder_func        Underlying SCons builder function
    @param  module              Module name
    """
    def build_lib(lib_name, sources, *args, **kwargs):
        """Customized library builder.

        @param  lib_name    Library name
        @param  sources     Source file (or list of source files)
        """
        # Create unique library key from module and library name
        lib_key = self.lib_key(module, lib_name)
        assert lib_key not in self._libs
        # Store resulting library node in shared dictionary
        self._libs[lib_key] = bldr_func(lib_name, sources, *args, **kwargs)
    return build_lib

The _lib_wrapper() method returns the nested function build_lib. If you’re not familiar with returning nested functions to statically capture the scope, you might be interested in my post on Python closures. Practically, it allows someone else (e.g. a SConscript file) to invoke the build_lib function, and reference the bldr_func and module variables that were captured in the closure, without the caller ever knowing it existed.

The build_lib function simply uses the underlying SCons builder bldr_func to compile a library. The resulting library node is saved in the shared libraries dictionary, using a unique key generated from the module and library name (modname::libname).

The Prog wrapper

The custom program builder is returned by the _prog_wrapper method in site_scons/site_init.py.

def _prog_wrapper(self, module, default_install=True):
    """Return a wrapped customized flavored program builder for module.

    @param  module              Module name
    @param  default_install     Whether built program nodes should be
                                installed in bin-dir by default
    """
    def build_prog(prog_name, sources, with_libs=None, *args, **kwargs):
        """Customized program builder.

        @param  prog_name   Program name
        @param  sources     Source file (or list of source files)
        @param  with_libs   Library name (or list of library names) to
                            link with.
        @param  install     Binary flag to override default value from
                            closure (`default_install`).
        """
        # Make sure sources is a list
        sources = listify(sources)
        install_flag = kwargs.pop('install', default_install)
        # Process library dependencies - add libs specified in `with_libs`
        for lib_name in listify(with_libs):
            lib_keys = listify(self._get_matching_lib_keys(lib_name))
            if len(lib_keys) == 1:
                # Matched internal library
                lib_key = lib_keys[0]
                # Extend prog sources with library nodes
                sources.extend(self._libs[lib_key])
            elif len(lib_keys) > 1:
                # Matched multiple internal libraries - probably bad!
                raise StopError('Library identifier "%s" matched %d '
                                'libraries (%s). Please use a fully '
                                'qualified identifier instead!' %
                                (lib_name, len(lib_keys),
                                 ', '.join(lib_keys)))
            else:  # empty lib_keys
                raise StopError('Library identifier "%s" didn\'t match '
                                'any library. Is it a typo?' % (lib_name))
        # Build the program and add to prog nodes dict if installable
        prog_nodes = self._env.Program(prog_name, sources, *args, **kwargs)
        if install_flag:
            # storing each installable node in a dictionary instead of
            #  defining InstallAs target on the spot, because there's
            #  an "active" variant dir directive messing with paths.
            self._progs[module].extend(prog_nodes)
    return build_prog

When the Writer/SConscript calls Prog('writer', 'writer.cc', with_libs='AddressBook::addressbook'), it actually executes build_prog(target_name='writer', sources='writer.cc', with_libs='AddressBook::addressbook', args=[], kwargs={}) with captured variable module=Writer from the enclosing scope.

The custom builder takes care of the stuff that were previously done in the SConscript itself. It uses a simplified version of the _get_targets() method that was introduced in an earlier episode to add dependencies to the sources list. In previous episodes, the SConscript used get_targets() explicitly to extend the sources list. To enable the simpler, more readable, with_libs=[...] approach shown here, I allow the custom builder to take an additional arguments with_libs (that defaults to None). The custom builder uses with_libs to dynamically extend the sources list with the _get_matching_lib_keys() method.

Passing through *args, **kwargs to the underlying SCons builder allows you to use SCons features (like environment override) without the custom builder interfering with it.

Simplified library matching

The _get_targets function introduced in a previous episode did many things. It supported taking variable list of library queries, because it was meant to be used in SConscript files. If you linked with libraries A and B, you could use _get_targets('A', 'B') instead of two separate calls. It also supported wildcard-queries, and contained the logic to deal with multiple results, no results, and printing warnings. This made the function less than simple, allowing to do more without adding overhead to the SConscript file.

This shortcuts refactor changes things completely. The SConscript maintainer uses with_libs instead of calling a function. The wrapped build_prog is responsible for processing with_libs, so it makes sense to perform a more sensible role separation. This is the reason I replaced _get_targets with the much simpler _get_matching_lib_keys method:

def _get_matching_lib_keys(self, lib_query):
    """Return list of library keys for given library name query.

    A "library query" is either a fully-qualified "Module::LibName" string
     or just a "LibName".
    If just "LibName" form, return all matches from all modules.
    """
    if self.is_lib_key(lib_query):
        # It's a fully-qualified "Module::LibName" query
        if lib_query in self._libs:
            # Got it. We're done.
            return [lib_query]
    else:
        # It's a target-name-only query. Search for matching lib keys.
        lib_key_suffix = '%s%s' % (self._key_sep, lib_query)
        return [lib_key for lib_key in self._libs
                if lib_key.endswith(lib_key_suffix)]

This function doesn’t return library nodes, just matching library keys. It doesn’t take multiple queries – just one. It doesn’t deal with raising errors or printing warnings – it just returns what it finds. It doesn’t support wildcard-queries.

Summary

That’s it. Some refactoring of previous episodes, along with clever use of exports dictionary, allow simple and readable SConscript files. The actual functionality did not change from previous episodes.

The final result is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.

See the scons tag for more in my SCons series. You may want to stay tuned for the next episode, that will show how to remove the last bit of overhead in SConscript files – the Import('*') line. Another future episode that may interest you will deal with removing the requirement for forward-only module dependencies.