How To Use Protocol Buffers In a SCons Project, Take 1

This is the eleventh post in my SCons series. This post starts exploring ways to work with protocol buffer files in a SCons project.

Protocol buffers are a structured-data-serialization mechanism from Google. This is not a tutorial on protocol buffers. I will use the address book project example that appears in the official protocol buffers tutorial.

When using protocol buffers, you write .proto files to describe your structured data. You use the protoc compiler to generate C++ and Python code that allows you to serialize, load, and manipulate your protocol buffers data.

Out of the box, SCons does not know how to compile .proto files into C++ and Python code. The purpose of this post is to start exploring ways to integrate protocol buffers in the build process. The first iteration is based on manual “integration”.

The final result is available on my GitHub scons-series repository.

Shell Foo: Parallelizing Multiple wget Downloads

Got a bunch of files to download? Got an open terminal session? Want to use wget to parallelize the download?

How about this:

echo http://dl.whatever.com/dl/file{1..1000} | xargs -n 1 -P 16 wget -q

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Running a Time-Limited Subprocess In Python (concurrency caveats inside!)

By Tuesday, January 13, 2015 0 Permalink 0

I tried to write a “simple” Python function. All it had to do is to run a command line in a subprocess, and enforce a timeout on that subprocess.

Turns out that this “simple” goal is not so simple. I considered multiple approaches to solve this, and ran into interesting concurrency issues.

In this post I describe a solution based on Python subprocess and various threading constructs. I demonstrate a concurrency inconsistency I discovered when testing my solution on Linux and OS X.

The conclusion, if that’s all what you’re looking for, is that timer.is_alive() is not a safe way to test whether a timer had expired!

Note: My experience is based on Python 2.7 with the default subprocess module from the standard library. If you’re on Python 3.3+, a timeout argument was added to subprocess. You can also install python-subprocess32, which brings this joy to Python 2.4-2.7.

Weekly Review, January 10

By Saturday, January 10, 2015 0 Permalink 0

I thought I’d skip this weekly review too, but if I can get away with a minimalistic one, it will save me from a single-post week!

The reason for the relative silence is a rare combination of approaching DayJob milestone, and being sick for the entire week… Be warned that these conditions are expected to exist during the next week as well!

So I will leave you with the Google Analytics stats of this site from September 1st through December 31st 2014, while I go tend to my Sinusitis. Maybe next week I’ll try to analyze it a little.

The Weekly Review is a recurring (sort-of-)weekly summary, reviewing highlights from the last week.

App Highlights: Yatse, the XBMC / Kodi Remote

Yatse is my Kodi (formerly XBMC) remote Android app of choice.

Kodi is a free and open source media center application. It’s what I currently use on my dedicated HTPC computer in the living room (via OpenELEC).

The Yatse Remote app allows me to control the media center from my phone. It’s so good doing that, that ever since I installed it, I never needed to pull out the HTPC keyboard / mouse!

There are plenty of alternative Kodi remote apps, including an official one from the XBMC team. I briefly used the official app a while ago (maybe early 2013?), and got quickly frustrated.

The Yatse app is free, with a $6.99 unlocker for extra features (and developer support). I never had any need for the extra features, so I purchased the upgrade only to support the developer.

Yatse also has various plugins for even more features!

App Highlights is a recurring series. From time to time, I highlight one Android app that I found useful. Feel free to suggest apps for me to highlight, but be advised that I focus on apps that I actually use.

Weekly Review, January 3

By Saturday, January 3, 2015 0 Permalink 1

For the first time since starting this weekly review, I skipped a week! Did you notice? ;-)

Anyway, I just finished “audio-reading” The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution. It was a long one! At 17:30 hours net play time, it took me about 4 weeks to finish it at 1.25x rate, but it was worth it! I followed a recommendation by Jamie Todd Rubin, and enjoyed the excellent stories laying out the evolution of the digital age. I always thought that “historical subjects” can become interesting easily if told through stories of actual people involved, as opposed to dry facts and numbers. This book proved that point for me. As a common geek, I admit that it isn’t hard for me to find interest in stories about people ranging from Ada Lovelace through Sergey Brin, but I really think also non-geeks will find this book interesting.

The Weekly Review is a recurring (sort-of-)weekly summary, reviewing highlights from the last week(s).

Supporting the SCons Help Command and Quiet Flag

This is the tenth post in my SCons series. The topic of this post is adding support for the silent / quiet flag, and the help command.

In previous episodes I’ve added various enhancements. Some of them print progress information (like processing modules, and two-pass processing). This makes my enhancements quite chatty, which isn’t a problem by itself. The problem is that these messages are printed also in SCons “silent mode” (scons -s), and this episode fixes that.

On the way, I also add a custom help message. Just because it’s fun.

The final result is available on my GitHub scons-series repository.

My Favorite Blog Feeds, 2014 Edition

By Tuesday, December 30, 2014 0 , Permalink 2

I like being up-to-date on topics that interest me. The main tool I use to do that is RSS feeds. I use Feedly to subscribe to RSS feeds of information sources that I want to follow (mostly blogs).

While Feedly is available via web and mobile apps, I use the Android app almost exclusively. I might use the web version to organize my subscriptions from time to time, but I do the actual reading on my Nexus 5.

I made it a habit to catch up on my Feedly Must Read list as a filler, whenever I have “down time” (commute, waiting for something, coffee break, etc.). Much better than burning time on Facebook and such, in my opinion :-) . I religiously clear out my Must Read queue at least once a week (I estimate it’s currently around 1,200 items per week).

This post lists my favorite RSS feeds, as of December 2014.

Quick and Dirty Personal Social Analytics With Google App Engine

I want to track basic metrics of my main social thingies – The Ostrich Facebook page, The Ostrich Twitter account, and The Ostrich Google+ page.

My short-term goal is to have simple graphs of some basic metrics over time:

  • Facebook page likes and shares, posts, post likes and shares.
  • Twitter followers (and following), tweets, favorites, retweets.
  • Google+ followers, views, +1’s, shares.

I’m not sure why I want this data, but if I do find a good reason in the future, I’d better start tracking it now!

In this post, I describe how I went from the idea to start collecting the data, to a deployed data collecting app using Google App Engine, in under 5 hours.

Automating Module Discovery in a SCons Project

By Thursday, December 25, 2014 0 , , Permalink 0

This is the ninth post in my SCons series. The topic of this post is automating module discovery.

Up until now, the module directories were hardcoded in site_scons/site_config.py. This was a natural choice, because you needed to specify the modules in the correct order. In the previous episode, I implemented a solution that supports arbitrary modules order. Now it’s much more natural to automate the process of module discovery, over manual specification.

As a developer in the project, you know that when you create a new module you need to create a SConscript to describe how to build it. It makes sense that the build system will be able to locate all modules to build, by looking for the SConscript files recursively from the project base directory.

In this episode, I describe how I implemented module auto-discovery by walking the project directory tree.

My implementation also correctly handles common caveats:

  • Avoid walking the build directory.
  • Skip “hidden” directories (like .git, and .whatever from your favorite IDE metadata).

In addition, my implementation provides several useful features:

  • Ability to limit recursion depth.
  • Support for “stop marker files”. If a stop marker file (e.g. .noscons) exists in a project directory, the directory (and all sub-directories) will be skipped.

The episode builds on top of the previous episode. The final result is available on my GitHub scons-series repository.