This is the fifth post in my SCons series. The topic of this post is improving the previous multi-flavor project via terminal integration.

It can be a hassle to handle multi-flavor projects. In the multi-flavor project post, I suggested a solution to simplify the multi-flavor build process. Using that solution, you just run scons flavor_name to build a specific flavor. But there’s still room for improvement! If you want to run a program you just built, you still need to specify the path to the flavored executable.

For example, say you built a project with a program named say_hi in the module hello. You built it by running scons debug. To run it you execute ./build/debug/hello/say_hi. It can be a hassle to write ./build/debug over and over. Even worse, it’s the first time you need to know about the layout of the build directory. Up until now, such details were nicely hidden in the config file.

In addition, you may often want to work with just one flavor. You may be developing a new feature, and you want to only build and run the debug flavor. If you run scons without the debug argument, all flavors will be built. This can be annoying and time consuming.

In this post, I suggest a helper script to make things simpler. The purpose of the script is to allow you to activate a flavor in a terminal session. While a flavor is active, magical things happen:

1. Running scons with no arguments builds only the active flavor.
2. The executable programs of the active flavor can be executed more conveniently.
3. The active flavor is indicated in the terminal prompt.

The final result is available on my GitHub scons-series repository. In the rest of this post I go into the details of the helper script and related changes.

The Final Result

Lets start by seeing how the final result behaves.

Using the same silly Address Book project, there’s a new mode script in the project base directory.

The script is meant to be sourced:

itamar@legolas sconseries (episodes/04-flavterm) $ source mode
Usage: source mode [debug|release|clear]

When sourced with no arguments, it prints out the usage – source mode [flavor] to activate a flavor (for all known flavors), or source mode clear to deactivate any active flavor.

We activate the “debug” flavor by running source mode debug, which adds a flavor marker to the session prompt:

itamar@legolas sconseries (episodes/04-flavterm) $ source mode debug
(debug) itamar@legolas sconseries (episodes/04-flavterm) $

With an active flavor, running scons builds only that flavor:

(debug) itamar@legolas sconseries (episodes/04-flavterm) $ scons
scons: Reading SConscript files ...
scons: Using active flavor "debug" from your environment
scons: + Processing flavor debug ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
clang++ -o build/debug/AddressBook/addressbook.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug build/debug/AddressBook/addressbook.cc
ar rc build/debug/AddressBook/libaddressbook.a build/debug/AddressBook/addressbook.o
ranlib build/debug/AddressBook/libaddressbook.a
clang++ -o build/debug/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug build/debug/Writer/writer.cc
clang++ -o build/debug/Writer/writer build/debug/Writer/writer.o build/debug/AddressBook/libaddressbook.a
Install file: "build/debug/Writer/writer" as "build/debug/bin/Writer.writer"
scons: done building targets.

Other flavors (namely, release) are not built, nor even processed.

In addition, the Writer/writer program that was built can also be executed using the “shortcut” Writer.writer – from any directory:

(debug) itamar@legolas sconseries (episodes/04-flavterm) $ which Writer.writer
/..../sconseries/build/debug/bin/Writer.writer

You can see that the program resolves to an executable under the build/debug directory. This is the same executable as the one in build/debug/Writer/writer.

Changing the active flavor to release and repeating the exercise produces the expected behavior:

(debug) itamar@legolas sconseries (episodes/04-flavterm) $ source mode release
(release) itamar@legolas sconseries (episodes/04-flavterm) $ scons
scons: Reading SConscript files ...
scons: Using active flavor "release" from your environment
scons: + Processing flavor release ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
clang++ -o build/release/AddressBook/addressbook.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/AddressBook/addressbook.cc
ar rc build/release/AddressBook/libaddressbook.a build/release/AddressBook/addressbook.o
ranlib build/release/AddressBook/libaddressbook.a
clang++ -o build/release/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release build/release/Writer/writer.cc
clang++ -o build/release/Writer/writer build/release/Writer/writer.o build/release/AddressBook/libaddressbook.a
Install file: "build/release/Writer/writer" as "build/release/bin/Writer.writer"
scons: done building targets.
(release) itamar@legolas sconseries (episodes/04-flavterm) $ which Writer.writer
/..../sconseries/build/release/bin/Writer.writer

As before, only the active flavor is processed and built, and the Writer.writer executable is resolved to the correct flavor.

Clearing the active flavor and repeating the exercise once again:

(release) itamar@legolas sconseries (episodes/04-flavterm) $ source mode clear
itamar@legolas sconseries (episodes/04-flavterm) $ scons
scons: Reading SConscript files ...
scons: + Processing flavor debug ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: + Processing flavor release ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
itamar@legolas sconseries (episodes/04-flavterm) $ which Writer.writer
itamar@legolas sconseries (episodes/04-flavterm) $

This time all flavors were processed (nothing built though), and Writer.writer didn’t resolve to anything. Also, the prompt string flavor indicator was removed.

The sections that follow explain how this behavior is implemented.

Installing Executable Program In Flavor bin Directory

As apparent in the last section, the executable programs that are built are copied to a bin directory under the flavor build directory, and their names are prefixed with their module name.

This is accomplished using the SCons InstallAs function. I added an install loop to the process_module function, in the highlighted lines:

def process_module(env, module):
    """Delegate build to a module-level SConscript using the specified env.

    @param  env     Construction environment to use
    @param  module  Directory of module

    @raises AssertionError if `module` does not contain SConscript file
    """
    # Verify the SConscript file exists
    sconscript_path = os.path.join(module, 'SConscript')
    assert os.path.isfile(sconscript_path)
    print 'scons: |- Reading module', module, '...'
    # Execute the SConscript file, with variant_dir set to the
    #  module dir under the project flavored build dir.
    targets = env.SConscript(
        sconscript_path,
        variant_dir=os.path.join('$BUILDROOT', module),
        exports={'env': env})
    # Add the targets built by this module to the shared cross-module targets
    #  dictionary, to allow the next modules to refer to these targets easily.
    for target_name in targets:
        # Target key built from module name and target name
        # It is expected to be unique (per flavor)
        target_key = '%s::%s' % (module, target_name)
        assert target_key not in env['targets']
        env['targets'][target_key] = targets[target_name]
        # Add Install-to-binary-directory for Program targets
        for target in targets[target_name]:
            # Program target determined by name of builder
            # Probably a little hacky... (TODO: Improve)
            if target.get_builder().get_name(env) in ('Program',):
                bin_name = '%s.%s' % (module, os.path.basename(str(target)))
                env.InstallAs(os.path.join('$BINDIR', bin_name), target)

The loop in the highlighted lines goes over the nodes of a target, and uses the InstallAs function to copy and rename the built program to the flavor-bin-dir.

The loop tries to copy only executable programs by examining the target builder name. So far I used only the Program builder to build programs, so it worked as expected. I suspect that there might be other builders that produce executable programs, or other types of outputs that need to be installed as well. For now, I gracefully ignore them, by leaving a TODO comment.

Resolving Executables to the Correct Flavor bin Directory

I’m sure it will not surprise you if I tell you that the correct Writer.writer program was resolved thanks to some simple PATH manipulations:

itamar@legolas sconseries (episodes/04-flavterm) $ echo $PATH
/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
itamar@legolas sconseries (episodes/04-flavterm) $ source mode debug
(debug) itamar@legolas sconseries (episodes/04-flavterm) $ echo $PATH
/..../sconseries/build/debug/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
(debug) itamar@legolas sconseries (episodes/04-flavterm) $ source mode release
(release) itamar@legolas sconseries (episodes/04-flavterm) $ echo $PATH
/..../sconseries/build/release/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
(release) itamar@legolas sconseries (episodes/04-flavterm) $ source mode clear
itamar@legolas sconseries (episodes/04-flavterm) $ echo $PATH
/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
itamar@legolas sconseries (episodes/04-flavterm) $

Building Only the Active Flavor

As you may recall from the multi-flavor post, the list of flavors to build is set in the get_base_env function.

If the active flavor is stored in the session environment variable BUILD_FLAVOR, then the function can use it instead of the list of all flavors. Here it is, in the highlighted lines:

def get_base_env(*args, **kwargs):
    """Initialize and return a base construction environment."""
    # Initialize new construction environment
    env = Environment(*args, **kwargs)
    # If a flavor is activated in the external environment - use it
    if 'BUILD_FLAVOR' in os.environ:
        active_flavor = os.environ['BUILD_FLAVOR']
        if not active_flavor in flavors():

            raise StopError('%s (from env) is not a known flavor.' % (active_flavor))
        print ('scons: Using active flavor "%s" from your environment' % (active_flavor))
        env.flavors = [active_flavor]
    else:
        # If specific flavor target specified, skip processing other flavors
        # Otherwise, include all known flavors
        env.flavors = (set(flavors()).intersection(COMMAND_LINE_TARGETS) or flavors())
    # Perform base construction environment customizations from site_config
    if '_common' in ENV_OVERRIDES:
        env.Replace(**ENV_OVERRIDES['_common'])
    if '_common' in ENV_EXTENSIONS:
        env.Append(**ENV_EXTENSIONS['_common'])
    return env

The mode Flavor Activation Script

The mode script is where the PATH manipulation happen. It also sets the BUILD_FLAVOR environment variable.

It’s a bash script (so portability is limited). The main logic is contained in the second half of the script, as pasted below:

# Iterate over known flavors, removing them from PATH, and adding the selected flavor
FLAVORS_STR="["
FOUND_FLAV="0"
for FLAVOR in $FLAVORS; do

    if [ "clear" == "$FLAVOR" ]; then
        echo "WARNING: Flavor 'clear' collides with clearing active flavor!" >&2
    fi
    FLAV_BASE="$BASE_DIR/$BUILD_SUBDIR/$FLAVOR"
    FLAV_BIN="$FLAV_BASE/$BIN_SUBDIR"
    FLAVORS_STR="${FLAVORS_STR}${FLAVOR}|"
    if [ "$REQ_FLAVOR" == "$FLAVOR" ]; then
        # Found requested flavor - mark found and update path and env
        export BUILD_FLAVOR="$FLAVOR"
        FOUND_FLAV="1"
        path_prepend "$FLAV_BIN"
        # Update prompt with colored flavor decoration
        export PS1="\[\e[0;36m\]($FLAVOR)\[\e[m\] $CLEAN_PS"
    else
        # Not requested flavor - remove from PATH
        path_remove "$FLAV_BIN"
    fi
done

if [ "clear" == "$REQ_FLAVOR" ]; then
    unset BUILD_FLAVOR
    export PS1="$CLEAN_PS"
else
    if [ "0" == "$FOUND_FLAV" ]; then
        # not "clear" and no matching flavor - print usage
        FLAVORS_STR="${FLAVORS_STR}clear]"
        echo "Usage: source mode $FLAVORS_STR"
        return 1
    fi
fi

It loops over known flavors (in $FLAVORS). For each flavor:

• If it matches the requested flavor:
  1. The flavor name is exported as BUILD_FLAVOR.
  2. The flavor-bin-directory is prepended to the PATH.
  3. The prompt string is updated.
• If it doesn’t match the requested flavor:
  • The flavor-bin-directory is removed from the PATH. If that path wasn’t in PATH before – nothing is changed.

Finally, if clear matches the requested flavor, the BUILD_FLAVOR variable is cleared, and the prompt string is restored to normal.

In case no flavor matched, and “clear” didn’t match too, a usage string is printed. Note that the usage string known-flavors list is constructed dynamically.

The first half of the script performs all the initializations you would expect, as used throughout the second half.

Getting Config Values From the SCons Config Script

Since this script is a bash script, it cannot share the configuration used by SCons, which is written in Python.

A simple approach would be to repeat the required parts of the config in the bash script. Setting the build dir to build, list of known flavors to [debug, release], etc.

It can work. But it introduces duplicity, that will probably break when someone changes the Python config without updating this one too.

For that reason, I preferred to treat the Python config as the canonical source for configuration data. To allow the bash script to access configuration variables it needs, I added a main section to the site_config.py script. The Python main treats its argument as a variable query, writing the variable value(s) to STDOUT:

def main():

    """Main procedure - print out a requested variable (value per line)"""
    import sys
    if 2 == len(sys.argv):
        var = sys.argv[1].lower()
        items = list()
        if var in ('flavors',):
            items = flavors()
        elif var in ('modules',):
            items = modules()
        elif var in ('build', 'build_dir', 'build_base'):
            items = [_BUILD_BASE]
        elif var in ('bin', 'bin_subdir'):
            items = [_BIN_SUBDIR]
        # print out the item values
        for val in items:
            print val

if '__main__' == __name__:
    main()

Thanks to this addition, the bash script can get configuration variables by parsing the output of python site_scons/site_config.py variable_name. This can be seen in several places in the script initialization:

REQ_FLAVOR="$1"
# Get base directory of this script
BASE_DIR="$( cd "$(dirname "${BASH_SOURCE[0]}" )" && pwd )"
SITE_CONFIG_SCRIPT="$BASE_DIR/site_scons/site_config.py"
# Check that site config script exists
if [ ! -f "$SITE_CONFIG_SCRIPT" ]; then
    echo "Missing site_config.py script in site_scons dir." >&2
    return 5
fi
# Remember the clean prompt
if [ -z "$CLEAN_PS" ]; then
    export CLEAN_PS="$PS1"
fi
# Get build & bin dirs from the config script
BUILD_SUBDIR="$( $PYTHON "$SITE_CONFIG_SCRIPT" build )"
BIN_SUBDIR="$( $PYTHON "$SITE_CONFIG_SCRIPT" bin )"
# Get known flavors from the config script
FLAVORS="$( $PYTHON "$SITE_CONFIG_SCRIPT" flavors )"

Other interesting stuff that happen in the quoted snippet:

1. The requested flavor is stored.
2. The BASE_DIR of the project is determined by running pwd after cd to dirname ${BASH_SOURCE[0]} in a sub-shell. If you want to better understand what’s going on here, refer to my post on getting the directory of a sourced bash script.
3. The clean prompt string is saved (unless it was already saved during a previous sourcing).

If the Python config file cannot be located, the script prints an error and terminates (using return, as appropriate for a sourced script).

Other Initialization Snippets

Helper functions for PATH manipulations:

path_append ()  { path_remove $1; export PATH="$PATH:$1"; }
path_prepend () { path_remove $1; export PATH="$1:$PATH"; }
path_remove ()  { export PATH=`echo -n $PATH | awk -v RS=: -v ORS=: '$0 != "'$1'"' | sed 's/:$//'`; }

The script terminates (using exit, as appropriate for a process) if it detects that it is not sourced:

# Exit if not sourced
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    echo "This script must be sourced (e.g. \"source mode [flavor|clear]\")" >&2
    exit 37
fi

The script terminates (using return) if the Python binary could not be located:

PYTHON="$( type -P python )"
# Check that Python is available
if [ "x$PYTHON" == "x" ]; then
    echo "Could not find Python" >&2
    return 17
fi

This is done by running type -P command, that prints the path to the Python executable if the exists in the system. If it doesn’t exist, then an empty string is printed – so this is what I test for.

Interesting Gotcha’s

Contradiction between active flavor and explicit target

(debug) itamar@legolas sconseries (episodes/04-flavterm) $ scons release
scons: Reading SConscript files ...
scons: Using active flavor "debug" from your environment
scons: + Processing flavor debug ...
scons: |- Reading module AddressBook ...
scons: |- Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
scons: *** Do not know how to make File target `release' (<...>/sconseries/release).  Stop.
scons: building terminated because of errors.

When a flavor is active, other flavors are not processed. So if an inactive flavor is given as an explicit target name, SCons will cry about unknown target.

Using “clear” as a flavor name

If I add clear as a name for a new flavor:

(debug) itamar@legolas sconseries (episodes/04-flavterm) $ source mode
WARNING: Flavor 'clear' collides with clearing active flavor!
Usage: source mode [debug|release|clear|clear]

A warning is printed.

Active flavor is not a known flavor

itamar@legolas sconseries (episodes/04-flavterm) $ export BUILD_FLAVOR=foo
itamar@legolas sconseries (episodes/04-flavterm) $ scons
scons: Reading SConscript files ...
scons: *** foo (from env) is not a known flavor.  Stop.

It doesn’t really makes sense to manually override the BUILD_FLAVOR variable like this. But this can happen in practice. For example, you can have a legitimate flavor foo enabled, and then switch to another branch where foo is not a valid flavor.

If it happens, simply source mode valid_flavor or clear to restore.

Multiple projects don’t play nice

If you have multiple projects that implement this approach, they may interfere. The flavor settings are stored in the shell session environment. These settings are there also when you navigate away from the project directory. If you navigate to another project, you may experience the “flavor not known” warning mentioned above (if flavors differ across projects).

I don’t think that’s so bad though. It is reasonable to assume that all projects will have debug and release flavors. So in most cases this would work fine.

It may be annoying to see the flavor marker in the prompt string, even when you’re not working on the project. You could run source mode clear before leaving the project directory. I agree that it could have been convenient if this could be automated.

Summary

This was my suggestion for complementing the previously described multi-flavor SCons project with terminal integration. The integration allows a developer to easily activate a flavor in a terminal session. An active flavor saves typing by building only that flavor by default. It also makes the flavored project binaries available in the PATH, for easy execution.

The final result is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.

See the scons tag for more in my SCons series. An interesting next post might be my extreme SConscript simplification using custom SCons shortcuts.