This is the twelfth post in my SCons series. This posts continues exploring ways to work with protocol buffers in a SCons project.
In the previous episode I covered the manual approach to using protocol buffers in a SCons project. As mentioned there, SCons does not know how to compile .proto files into C++ and Python code out of the box.
This post will take the integration a step further, by actually using SCons to compile .proto files.
I am definitely not the first one to suggest this SCons extension. I started using the SCons ProtocBuilder by Scott Stafford. It worked fine for my needs, until it didn’t. This post focuses on integrating Scott’s builder as is. Future posts will deal with fixes and improvements.
The final result is available on my GitHub scons-series repository.
Integrating the original Protoc tool by Scott Stafford
I copied the SCons ProtocBuilder by Scott Stafford into site_scons/site_tools/protoc.py, following the SCons user guide instructions for custom builders.
To instruct SCons to load this builder, I changed one line in the main SConsturct file (highlighted in the snippet):
"""Flavor-based project main SConstruct script with SCons shortcuts,
and support for compiling Protocol Buffers.
"""
OSTRICH_SCONS_HELP = """..."""
if GetOption('help'):
# Skip it all if user just wants help
Help(OSTRICH_SCONS_HELP)
else:
# Get the base construction environment
_BASE_ENV = get_base_env(tools=['default', 'protoc'])
# Build every selected flavor
for flavor in _BASE_ENV.flavors:
sprint('+ Processing flavor %s ...', flavor)
flav_bldr = FlavorBuilder(_BASE_ENV, flavor)
flav_bldr.build()
I then added a new target in the AddressBook/SConscript:
"""AddressBook proto-based library SConscript script"""
Import('*')
Protoc([], 'addressbook.proto', PROTOCPROTOPATH='.',
PROTOCOUTDIR='.', PROTOCPYTHONOUTDIR=None)
Lib('addressbook', 'addressbook.pb.cc')
The integration is supposedly complete, but if I try to build it now, it breaks:
itamar@legolas sconseries (episodes/11-protoc) $ rm AddressBook/*.pb.*
itamar@legolas sconseries (episodes/11-protoc) $ rm -r build/
itamar@legolas sconseries (episodes/11-protoc) $ scons
scons: Reading SConscript files ...
scons: + Processing flavor debug ...
scons: |- First pass: Reading module AddressBook ...
scons: |- First pass: Reading module Reader ...
scons: |- First pass: Reading module Writer ...
scons: |- Second pass: Reading module AddressBook ...
scons: |- Second pass: Reading module Reader ...
scons: |- Second pass: Reading module Writer ...
scons: + Processing flavor release ...
scons: |- First pass: Reading module AddressBook ...
scons: |- First pass: Reading module Reader ...
scons: |- First pass: Reading module Writer ...
scons: |- Second pass: Reading module AddressBook ...
scons: |- Second pass: Reading module Reader ...
scons: |- Second pass: Reading module Writer ...
scons: done reading SConscript files.
scons: Building targets ...
protoc -I. --cpp_out=. build/debug/AddressBook/addressbook.proto
clang++ -o build/debug/AddressBook/addressbook.pb.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug build/debug/AddressBook/addressbook.pb.cc
build/debug/AddressBook/addressbook.pb.cc:5:10: fatal error: 'build/debug/AddressBook/addressbook.pb.h' file not found
#include "build/debug/AddressBook/addressbook.pb.h"
^
1 error generated.
scons: *** [build/debug/AddressBook/addressbook.pb.o] Error 1
scons: building terminated because of errors.
This occurred because the generated addressbook.pb.cc included addressbook.pb.h using path relative to the project root with explicit build directory. This does not fit my include-path policy, that assumes header files are included with paths relative to the build directory itself. This is somewhat annoying, but easy to accommodate by adding # to the C++ compiler search path. This is done in site_scons/site_config.py:
...
ENV_EXTENSIONS = {
'_common': dict(
# Common flags for all C++ builds
CCFLAGS = ['-std=c++11', '-Wall', '-fvectorize', '-fslp-vectorize'],
# Modules should be able to include relative to build root dir
CPPPATH = ['#$BUILDROOT', '#'],
),
...
Now protocol buffers code generation is a part of the SCons build chain!
itamar@legolas sconseries (episodes/11-protoc) $ rm -r build/ itamar@legolas sconseries (episodes/11-protoc) $ scons scons: Reading SConscript files ... scons: + Processing flavor debug ... scons: |- First pass: Reading module AddressBook ... scons: |- First pass: Reading module Reader ... scons: |- First pass: Reading module Writer ... scons: |- Second pass: Reading module AddressBook ... scons: |- Second pass: Reading module Reader ... scons: |- Second pass: Reading module Writer ... scons: + Processing flavor release ... scons: |- First pass: Reading module AddressBook ... scons: |- First pass: Reading module Reader ... scons: |- First pass: Reading module Writer ... scons: |- Second pass: Reading module AddressBook ... scons: |- Second pass: Reading module Reader ... scons: |- Second pass: Reading module Writer ... scons: done reading SConscript files. scons: Building targets ... protoc -I. --cpp_out=. build/debug/AddressBook/addressbook.proto clang++ -o build/debug/AddressBook/addressbook.pb.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug -I. build/debug/AddressBook/addressbook.pb.cc ar rc build/debug/AddressBook/libaddressbook.a build/debug/AddressBook/addressbook.pb.o ranlib build/debug/AddressBook/libaddressbook.a clang++ -o build/debug/Reader/reader.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug -I. build/debug/Reader/reader.cc clang++ -o build/debug/Reader/reader build/debug/Reader/reader.o build/debug/AddressBook/libaddressbook.a -lprotobuf clang++ -o build/debug/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -g -DDEBUG -Ibuild/debug -I. build/debug/Writer/writer.cc clang++ -o build/debug/Writer/writer build/debug/Writer/writer.o build/debug/AddressBook/libaddressbook.a -lprotobuf Install file: "build/debug/Reader/reader" as "build/debug/bin/Reader.reader" Install file: "build/debug/Writer/writer" as "build/debug/bin/Writer.writer" protoc -I. --cpp_out=. build/release/AddressBook/addressbook.proto clang++ -o build/release/AddressBook/addressbook.pb.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release -I. build/release/AddressBook/addressbook.pb.cc ar rc build/release/AddressBook/libaddressbook.a build/release/AddressBook/addressbook.pb.o ranlib build/release/AddressBook/libaddressbook.a clang++ -o build/release/Reader/reader.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release -I. build/release/Reader/reader.cc clang++ -o build/release/Reader/reader build/release/Reader/reader.o build/release/AddressBook/libaddressbook.a -lprotobuf clang++ -o build/release/Writer/writer.o -c -std=c++11 -Wall -fvectorize -fslp-vectorize -O2 -DNDEBUG -Ibuild/release -I. build/release/Writer/writer.cc clang++ -o build/release/Writer/writer build/release/Writer/writer.o build/release/AddressBook/libaddressbook.a -lprotobuf Install file: "build/release/Reader/reader" as "build/release/bin/Reader.reader" Install file: "build/release/Writer/writer" as "build/release/bin/Writer.writer" scons: done building targets. itamar@legolas sconseries (episodes/11-protoc) $ . mode release (release) itamar@legolas sconseries (episodes/11-protoc) $ Reader.reader foo.adbook Person ID: 123 Name: Oogi E-mail address: oogi@oogi.oogi Work phone #: 12321
Summary
That’s it for integrating Scott’s existing Protoc builder in my SCons environment. It required some adjustments, and the SConscript syntax can be neater, but it works!
From my experience, this Protoc builder is sufficient for simple projects. It becomes insufficient when the project becomes more complex, as I will demonstrate in the next episode (spoiler: try import a proto from another proto).
There’s much more to cover around SCons-ProtoBuf integration. Some of the future episodes:
- Fixing the Protoc builder.
- Integrating protocol buffers with my SCons shortcuts.
- Expanding the integration to handle generated Python code.
The code for this episode is available on my GitHub scons-series repository. Feel free to use / fork / modify. If you do, I’d appreciate it if you share back improvements.
Leave a Reply