This is the fourteenth post in my
SCons series. The topic of this post is adding a shortcut for the custom SCons Protoc builder from the previous episodes.
The shortcut is in line with the SConscript simplification approach described in an earlier episode. In this installment, I add a new Proto shortcut to the collection, so the address book SConscript can look like this:
"""AddressBook proto-based library SConscript script""" Import('*') AbProtos = ['person.proto', 'addressbook.proto'] Proto(AbProtos) Lib('addressbook', protos=AbProtos)
The final result is available on my GitHub scons-series repository.
Adding a new Proto shortcut
shortcuts = dict( Lib = self._lib_wrapper(self._env.Library, module), StaticLib = self._lib_wrapper(self._env.StaticLibrary, module), SharedLib = self._lib_wrapper(self._env.SharedLibrary, module), Proto = self._proto_wrapper(), Prog = nop, )
The _proto_wrapper() method returns a wrapped custom Protoc builder:
def _proto_wrapper(self): """Return a wrapped Protoc builder.""" def compile_proto(proto_sources, **kwargs): """Customized Protoc builder. Uses Protoc builder to compile proto files specified in `proto_sources`. Optionally pass `cpp=False` to disable C++ code generation. Optionally, pass `python=True` to enable Python code generation. Optionally pass `PROTOPATH=[...]` to override default list of proto search paths (default is [$BUILDROOT]). Optionally pass `PROTOCPPOUT=path` and `PROTOPYOUT=path` to override default output path for C++ / Python outputs (respectively), when output for this language is enabled. Default output paths are $BUILDROOT (so expect output files in the module directory under the flavor build dir). Tip: Don't mess with these... """ if not hasattr(self._env, 'Protoc'): raise StopError('Protoc tool not installed.') # use user-specified value, or set default kwargs.setdefault('PROTOPATH', ['$BUILDROOT']) any_output = False for gen_flag_name, default_gen_flag, path_name, default_path in [ ('cpp', True, 'PROTOCPPOUT', '$BUILDROOT'), ('python', False, 'PROTOPYOUT', '$BUILDROOT'), ]: gen_output_flag = kwargs.pop(gen_flag_name, default_gen_flag) if gen_output_flag: any_output = True # use user-specified value, or set default kwargs.setdefault(path_name, default_path) else: kwargs[path_name] = '' # check that at least one output language is enabled if any_output: targets = self._env.Protoc(, proto_sources, *args, **kwargs) for gen_node in targets: gen_filename = os.path.basename(gen_node.path) if gen_filename.endswith('.pb.cc'): # Save generated .pb.cc sources in proto_cc dictionary # (without the ".pb.cc" suffix) self._proto_cc[gen_filename[:-6]] = gen_node else: sprint('warning: Proto target with no output directives') return compile_proto
When the Proto shortcut is used in a SConscript file, it invokes the nested
compile_proto() function. This custom builder expects to receive a list of
.proto sources, with optional keyword arguments to customize the build. The custom builder then uses the Protoc builder (as-is from the previous episode) to do the work, passing the source list to it.
After compiling the proto files, the builder saves generated C++ nodes in a dictionary, keyed by source filename without the “.pb.cc” suffix. This is required so other targets will be able to refer to these nodes later.
By default, the custom builder sets up the
PROTOCPPOUT arguments to
$BUILDROOT. This makes the Protoc builder work exactly as it did in the previous episode. It executes relative to the build directory, and generates only C++ code in the same build directory.
All of the Protoc builder arguments can be directly manipulated by the caller, by passing them as additional keyword arguments. This is made possible by passing
**kwargs to the underlying Protoc builder.
In addition, the shortcut builder supports special keyword arguments to control its own behavior. The
python keyword arguments are boolean flags that control whether the custom builder generates C++ and Python code. The default values (
python=False) can be overridden by passing them as keyword arguments to the Proto shortcut. Since the underlying Protoc builder doesn’t know about these flags, they are extracted from
kwargs before passing it down (using
Note that passing
PROTOPYOUT to set a directory for Python generated code is not sufficient. You will also need to explicitly pass
python=True, or change the default flag value to True.
The shortcut prints a warning if it recognizes that all valid output languages are disabled. It also raises a
StopError if the Protoc builder is missing.
Adding support for the protos keyword argument
The changes described so far are sufficient for this SConscript to work:
"""AddressBook proto-based library SConscript script""" Import('*') AbProtos = ['person.proto', 'addressbook.proto'] Proto(AbProtos) Lib('addressbook', ['person.pb.cc', 'addressbook.pb.cc'])
This may be just fine, but not enough for my OCD. I don’t like it that I need to repeat the names of the proto’s, and even worse – that I need to know the naming convention of generated C++ files!
To be able to use
Lib('addressbook', protos=AbProtos) instead of
Lib('addressbook', ['person.pb.cc', 'addressbook.pb.cc']), I need to modify the existing wrappers that handle the other shortcuts. Since all wrappers already accept
**kwargs, I can check if
protos key exists, and use it to extend the sources list. I added this line to both existing wrappers (lib and prog):
sources = self._extend_proto_sources(sources, kwargs)
The _extend_proto_sources() method pops the
protos member if it exists, and uses the
_proto_cc dictionary to add generated .pb.cc nodes to the sources list:
def _extend_proto_sources(self, sources, kwargs_dict): """Return the sources list, extended with proto-cc-sources. @param sources The original list of sources @param kwargs_dict The keyword argument dictionary The protos list, if specified, is at kwargs_dict['protos']. If it is specified, the sources list is extended accordingly, and the 'protos' key is removed (so it's safe to pass-through). """ return listify(sources) + [ self._proto_cc[re.sub(r'\.proto$', '', proto)] for proto in listify(kwargs_dict.pop('protos', None))]
Notice that he method allows specifying the names of the protos with or without the “.proto” suffix, by stripping it out if it’s there (using
Now my OCD can relax a bit, with the SConscript clean and neat, while nothing is changed in the actual compile & link commands and outputs.
The final result is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.
Another thing that still bothers me, is the way I needed to specify linking with the external protobuf library. Although the AddressBook library is the one using protocol buffers, I needed to specify
LIBS=['protobuf'] on the Writer and Reader program targets! I understand this from a compile/link point of view, but I still don’t like it. Ideally, I’d like to “label” the AddressBook library as “requiring” the external protobuf library, and have SCons propagate this information for me. If you feel this way too, stay tuned for a future episode that will deal with propagating required libraries.