This is the fifteenth post in my SCons series. This post introduces a small enhancement to my SCons shortcuts system – nicer support for external libraries via the with_libs keyword argument.
In recent episodes, I needed to link with the protobuf library to use the Protocol-Buffers-based AddressBook library. I did this by adding the LIBS=['protobuf'] argument to the Program target, which works just fine.
If this works “just fine”, why mess with it? Well, I already mentioned my OCD, haven’t I? I already have a nicer way to refer to libraries I use, so why can’t I write with_libs=['AddressBook::addressbook', 'protobuf']? It looks a bit cleaner.
The reason this would not work as is, is because I lookup the with_libs entries in a shared dictionary of project-specific libraries (more no that in the post that introduced the shortcuts system), and “protobuf” is not a project library.
This post extends the shortcuts system to support also external libraries. In addition to improved aesthetics, I add a couple of useful features:
- Support configuration-based list of “supported external libraries”. This allows for centralized control of external libraries used across the project, which can be very useful in projects that want to enforce policies about library-usage (e.g. licensing requirements etc.).
- Simpler support for libraries that are not installed system-wide, taking care of icky details, like CPPPATH and LIBPATH crap.
- Protection against potentially difficult troubleshooting due to library name typo’s.
- External library aliases and groups.
This episode picks up where the previous episode left off. Read on for the full details, or check out the final result on my GitHub scons-series repository.
BLUF
The end-game here is to make these SConscript files work:
"""AddressBook Writer SConscript script"""
Import('*')
Prog('writer', 'writer.cc',
with_libs=['AddressBook::addressbook', 'protobuf'])
"""AddressBook Reader SConscript script"""
Import('*')
Prog('reader', 'reader.cc',
with_libs=['AddressBook::addressbook', 'protobuf'])
If I implement just this change, the build breaks:
itamar@legolas sconseries (episodes/14-extlibs) $ scons scons: Reading SConscript files ... scons: + Processing flavor debug ... scons: |- First pass: Reading module AddressBook ... scons: |- First pass: Reading module Reader ... scons: |- First pass: Reading module Writer ... scons: |- Second pass: Reading module AddressBook ... scons: |- Second pass: Reading module Reader ... scons: *** Library identifier "protobuf" didn't match any library. Is it a typo? Stop.
The ExtLib data structure
In its simplest form, an external library is just a string in the LIBS keyword argument of the Program SCons target. This is sufficient when the library is installed system-wide, and the linker is configured correctly to lookup libraries in the relevant system directories (e.g. /usr/lib, /usr/local/lib, etc.).
I’d like to nicely support the more general case, even if the library isn’t installed system-wide. To do this, I start with a simple data structure to represent relevant attributes of an external library:
"""External libraries data structures."""
from site_utils import listify
class ExtLib(object):
"""External Library class."""
def __init__(self, lib_name, libs=None, include_paths=None, lib_paths=None):
"""Initialize external library instance.
@param lib_name Symbolic name of library (or library-group)
@param libs Identifiers of libraries to link with
(if not specified, `lib_name` is used)
@param include_paths Additional include search paths
@param lib_paths Additional library search paths
"""
super(ExtLib, self).__init__()
self.name = lib_name
self.libs = listify(libs) if libs is not None else [lib_name]
self.cpp_paths = listify(include_paths)
self.lib_paths = listify(lib_paths)
def __repr__(self):
return u'%s' % (self.name)
class HeaderOnlyExtLib(ExtLib):
"""Header-only external library class.
Same as ExtLib, supporting only extra include paths.
This is useful to enforce header-only external libraries
(like many boost sub-libraries).
"""
def __init__(self, *args, **kwargs):
"""Initialize header-only external library instance."""
# Extract keyword-arguments not allowed with header-only libraries
kwargs.pop('libs')
kwargs.pop('lib_paths')
if len(args) >= 3:
assert 'include_paths' not in kwargs
kwargs['include_paths'] = args[2]
super(HeaderOnlyExtLib, self).__init__(args[0], **kwargs)
With this data structure in place, I can specify external libraries I’d like to support in site_scons/site_config.py:
ENV_EXTENSIONS = {
'_common': dict(
...,
# List of supported external libraries
EXTERNAL_LIBRARIES = [
ExtLib('protobuf'),
],
),
....
Injecting external libraries in the SCons shortcut handlers
All that is left now, is to make sure that I lookup and add the external libraries when processing with_libs in the Prog shortcut handler.
To make this easy, I implemented a helper method in site_scons/site_init.py FlavorBuild class:
def _get_external_library(self, lib_name):
"""Return external library object with name `lib_name` (or None)."""
for lib in self._env['EXTERNAL_LIBRARIES']:
if lib.name == lib_name:
return lib
This method looks up an external library by its “nice name” in the environment configuration, and returns the ExtLib instance it finds (or None).
I use this method in site_scons/site_init.py wrapped build_prog function (the changes are highlighted):
def build_prog(prog_name, sources=None, with_libs=None, **kwargs):
"""Customized program builder.
@param prog_name Program name
@param sources Source file (or list of source files)
@param with_libs Library name (or list of library names) to
link with.
kwargs params:
@param install Binary flag to override default value from
closure (`default_install`).
@param protos Names of proto (or protos) to add to target.
"""
# Extend sources list with protos from generated code manager
sources = self._extend_proto_sources(sources, kwargs)
install_flag = kwargs.pop('install', default_install)
# Extract optional keywords arguments that we might extend
cpp_paths = listify(kwargs.pop('CPPPATH', None))
ext_libs = listify(kwargs.pop('LIBS', None))
lib_paths = listify(kwargs.pop('LIBPATH', None))
# Process library dependencies - add libs specified in `with_libs`
for lib_name in listify(with_libs):
lib_keys = listify(self._get_matching_lib_keys(lib_name))
if len(lib_keys) == 1:
# Matched internal library
lib_key = lib_keys[0]
# Extend prog sources with library nodes
sources.extend(self._libs[lib_key])
elif len(lib_keys) > 1:
# Matched multiple internal libraries - probably bad!
raise StopError('Library identifier "%s" matched %d '
'libraries (%s). Please use a fully '
'qualified identifier instead!' %
(lib_name, len(lib_keys),
', '.join(lib_keys)))
else: # empty lib_keys
# Maybe it's an external library
ext_lib = self._get_external_library(lib_name)
if ext_lib:
# Matched external library - extend target parameters
cpp_paths.extend(ext_lib.cpp_paths)
ext_libs.extend(ext_lib.libs)
lib_paths.extend(ext_lib.lib_paths)
else:
raise StopError(
'Library identifier "%s" didn\'t match any '
'library. Is it a typo?' % (lib_name))
# Return extended construction environment parameters to kwargs
if cpp_paths:
kwargs['CPPPATH'] = cpp_paths
if ext_libs:
kwargs['LIBS'] = ext_libs
if lib_paths:
kwargs['LIBPATH'] = lib_paths
# Build the program and add to prog nodes dict if installable
prog_nodes = self._env.Program(prog_name, sources, **kwargs)
The essence of the changes: when going over with_libs elements, if no project-specific library matched, try matching a supported external library. If an external library matched, use the data structure fields to extend the relevant construction environment parameters (LIBS, CPPPATH, LIBPATH).
I still want to allow SConscript writers to pass things directly. This is the reason that I extract these keyword arguments in the beginning, extend them, and put them back in kwargs before passing it into the underlying SCons Program builder.
With this implemented, the build now works as expected:
itamar@legolas sconseries (episodes/14-extlibs) $ scons scons: Reading SConscript files ... scons: + Processing flavor debug ... scons: |- First pass: Reading module AddressBook ... scons: |- First pass: Reading module Reader ... scons: |- First pass: Reading module Writer ... scons: |- Second pass: Reading module AddressBook ... scons: |- Second pass: Reading module Reader ... scons: |- Second pass: Reading module Writer ... scons: + Processing flavor release ... scons: |- First pass: Reading module AddressBook ... scons: |- First pass: Reading module Reader ... scons: |- First pass: Reading module Writer ... scons: |- Second pass: Reading module AddressBook ... scons: |- Second pass: Reading module Reader ... scons: |- Second pass: Reading module Writer ... scons: done reading SConscript files. scons: Building targets ... scons: `.' is up to date. scons: done building targets.
Since nothing essential changed, and I didn’t clean the previous build, nothing was rebuilt.
Summary
That was a quick and simple enhancement to my SCons shortcuts framework. While being simple, and providing marginal value-add, it is a building block in the way to useful valuable features planned for future episodes:
- Adding “required libraries” semantics to Lib targets, allowing to propagate library requirements automatically.
- Using SCons to enforce library policies in a project.
The final result is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.
Leave a Reply