Supporting External Libraries In My SCons Shortcuts

By Thursday, February 26, 2015 0 , , Permalink 0

This is the fifteenth post in my SCons series. This post introduces a small enhancement to my SCons shortcuts system – nicer support for external libraries via the with_libs keyword argument.

In recent episodes, I needed to link with the protobuf library to use the Protocol-Buffers-based AddressBook library. I did this by adding the LIBS=['protobuf'] argument to the Program target, which works just fine.

If this works “just fine”, why mess with it? Well, I already mentioned my OCD, haven’t I? I already have a nicer way to refer to libraries I use, so why can’t I write with_libs=['AddressBook::addressbook', 'protobuf']? It looks a bit cleaner.

The reason this would not work as is, is because I lookup the with_libs entries in a shared dictionary of project-specific libraries (more no that in the post that introduced the shortcuts system), and “protobuf” is not a project library.

This post extends the shortcuts system to support also external libraries. In addition to improved aesthetics, I add a couple of useful features:

  • Support configuration-based list of “supported external libraries”. This allows for centralized control of external libraries used across the project, which can be very useful in projects that want to enforce policies about library-usage (e.g. licensing requirements etc.).
  • Simpler support for libraries that are not installed system-wide, taking care of icky details, like CPPPATH and LIBPATH crap.
  • Protection against potentially difficult troubleshooting due to library name typo’s.
  • External library aliases and groups.

This episode picks up where the previous episode left off. Read on for the full details, or check out the final result on my GitHub scons-series repository.

Continue Reading…

Adding SCons Proto Builder Shortcut

This is the fourteenth post in my SCons series. The topic of this post is adding a shortcut for the custom SCons Protoc builder from the previous episodes.

The shortcut is in line with the SConscript simplification approach described in an earlier episode. In this installment, I add a new Proto shortcut to the collection, so the address book SConscript can look like this:

"""AddressBook proto-based library SConscript script"""

Import('*')

AbProtos = ['person.proto', 'addressbook.proto']

Proto(AbProtos)
Lib('addressbook', protos=AbProtos)

The final result is available on my GitHub scons-series repository.

Continue Reading…

Fixing the Protoc SCons Builder

This is the thirteenth post in my SCons series.

In the previous episode I integrated Protoc, an existing custom builder, in my SCons project.

This post will demonstrate how this builder fails with non-trivial projects, and suggest some fixes and improvements.

I wanted to share my proposed fix via the SCons wiki page, but I couldn’t create a user… I’d appreciate if someone with access to that wiki could assist 🙂 .

The final result is available on my GitHub scons-series repository.

Continue Reading…

The Protoc Builder: Compiling Protocol Buffers With SCons

This is the twelfth post in my SCons series. This posts continues exploring ways to work with protocol buffers in a SCons project.

In the previous episode I covered the manual approach to using protocol buffers in a SCons project. As mentioned there, SCons does not know how to compile .proto files into C++ and Python code out of the box.

This post will take the integration a step further, by actually using SCons to compile .proto files.

I am definitely not the first one to suggest this SCons extension. I started using the SCons ProtocBuilder by Scott Stafford. It worked fine for my needs, until it didn’t. This post focuses on integrating Scott’s builder as is. Future posts will deal with fixes and improvements.

The final result is available on my GitHub scons-series repository.

Continue Reading…

How To Use Protocol Buffers In a SCons Project, Take 1

This is the eleventh post in my SCons series. This post starts exploring ways to work with protocol buffer files in a SCons project.

Protocol buffers are a structured-data-serialization mechanism from Google. This is not a tutorial on protocol buffers. I will use the address book project example that appears in the official protocol buffers tutorial.

When using protocol buffers, you write .proto files to describe your structured data. You use the protoc compiler to generate C++ and Python code that allows you to serialize, load, and manipulate your protocol buffers data.

Out of the box, SCons does not know how to compile .proto files into C++ and Python code. The purpose of this post is to start exploring ways to integrate protocol buffers in the build process. The first iteration is based on manual “integration”.

The final result is available on my GitHub scons-series repository.

Continue Reading…

Supporting the SCons Help Command and Quiet Flag

This is the tenth post in my SCons series. The topic of this post is adding support for the silent / quiet flag, and the help command.

In previous episodes I’ve added various enhancements. Some of them print progress information (like processing modules, and two-pass processing). This makes my enhancements quite chatty, which isn’t a problem by itself. The problem is that these messages are printed also in SCons “silent mode” (scons -s), and this episode fixes that.

On the way, I also add a custom help message. Just because it’s fun.

The final result is available on my GitHub scons-series repository.

Continue Reading…

Automating Module Discovery in a SCons Project

By Thursday, December 25, 2014 0 , , Permalink 0

This is the ninth post in my SCons series. The topic of this post is automating module discovery.

Up until now, the module directories were hardcoded in site_scons/site_config.py. This was a natural choice, because you needed to specify the modules in the correct order. In the previous episode, I implemented a solution that supports arbitrary modules order. Now it’s much more natural to automate the process of module discovery, over manual specification.

As a developer in the project, you know that when you create a new module you need to create a SConscript to describe how to build it. It makes sense that the build system will be able to locate all modules to build, by looking for the SConscript files recursively from the project base directory.

In this episode, I describe how I implemented module auto-discovery by walking the project directory tree.

My implementation also correctly handles common caveats:

  • Avoid walking the build directory.
  • Skip “hidden” directories (like .git, and .whatever from your favorite IDE metadata).

In addition, my implementation provides several useful features:

  • Ability to limit recursion depth.
  • Support for “stop marker files”. If a stop marker file (e.g. .noscons) exists in a project directory, the directory (and all sub-directories) will be skipped.

The episode builds on top of the previous episode. The final result is available on my GitHub scons-series repository.

Continue Reading…

Supporting Arbitrary Modules Order In SCons

By Thursday, December 18, 2014 2 , , Permalink 0

This is the eighth post in my SCons series. In this post, I describe how to support arbitrary modules order.

In an earlier episode, I presented the multi-module C++ SCons project. In that episode, I explained that modules need to be specified at the order of dependence.

This restriction can be annoying, and painfully limiting, once your project “gets serious”.

I promised a better solution, and now I provide one 🙂 . In a nutshell, the solution is based on a two-pass approach. In the first pass, all library-targets are processed and collected across all modules. In the second pass, all program-targets are processed, using the libraries already collected.

The rest of the post goes into further detail about the solution. The result, as usual, is available on my GitHub scons-series repository. It builds on top of the SCons shortcuts enhancement, in case you need to refresh your memory 🙂 .

Continue Reading…

Getting Rid of Redundant Import In SConscripts

This is the seventh post in my SCons series. The topic of this post is getting rid of the last bit of overhead in SConscript files – the Import('*') line.

According the the SCons user guide, a SConscript files needs to Import('...') shared symbols. It’s possible to import all exported symbols with Import('*'). This is the method I used in previous episodes to make shortcuts available in module-level SConscript files

Perhaps you’re fine with this little remaining overhead in every SConscript file. After all, it’s not such a big deal. With the wildcard syntax, you will never need to go over old SConscript files and update their import list when you add a new shortcut.

If you prefer your SConscript files as minimal as possible, here’s a dirty little hack I use. Instead of passing the shortcuts dictionary to a delegated SConscript file using the exports argument, I modify the _SConscript.GlobalDictdirectly before invoking the SConscript() function.

Using my example project, you may recall from the previous episode how the delegation looks like:

def process_module(self, module):
    print 'scons: |- Reading module', module, '...'
    # Execute the SConscript file, with variant_dir set to the
    #  module dir under the project flavored build dir.
    self._env.SConscript(
        sconscript_path,
        variant_dir=os.path.join(self._env['BUILDROOT'], module),
        exports=shortcuts)

The hack is a minor modification of this snippet:

def process_module(self, module):
    print 'scons: |- Reading module', module, '...'
    # Execute the SConscript file, with variant_dir set to the
    #  module dir under the project flavored build dir.
    SCons.Script._SConscript.GlobalDict.update(shortcuts)
    self._env.SConscript(
        sconscript_path,
        variant_dir=os.path.join(self._env['BUILDROOT'], module))

The result of the hack is that the shortcuts are injected directly into the global dictionary. This means that these symbols are available in SConscript files, without any call to Import(..)!

As always, the entire project is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.

See the scons tag for more in my SCons series.

Continue Reading…

How to Simplify Your SConscripts

This is the sixth post in my SCons series. The topic of this post is building reusable infrastructure that can extremely simplify your module-level SConscript files.

Starting with the first non-trivial SCons project, the module-level SConscript files contained too much repetitive code. The goal of this enhancement is to go back to minimalistic SConscript files. The objective is to let the developer define the module-level targets with minimal code, and no hassle.

I continue using the same C++ project that I introduced in the basic example. In this post I present SCons shortcuts that are available in module-level SConscript files. These shortcuts are Python functions that take care of dirty details behind the scenes.

The final result is available on my GitHub scons-series repository.

Continue Reading…