From Markdown To Pastable Formatted Text In OS X Terminal

By Tuesday, August 4, 2015 0 , , Permalink 3

I use Markdown a lot. I use it right now, to compose this post 🙂 . It’s my go-to syntax for quick plain text files for notes and such. It’s what I use when I code something that outputs “formatted” text to terminal.

Recently, I wanted to take Markdown output from a script I wrote, and send a nicely formatted version of it by email.

Read on for the solution I eventually came up with, after several iterations, and essential tips from Brett Terpstra, the Markdown Master.

tl;dr:

cat foo.md | \
  pandoc --from markdown --to html | \
  textutil -convert rtf -stdin -stdout -format html | \
  pbcopy -Prefer rtf

Continue Reading…

Shell Foo: Comparing the Output of Two Commands – Without Temp Files!

You probably know to use the diff file1 file2 command to compare file1 and file2 line by line.

Sometimes you don’t want to compare files, but output of commands. For example, compare sort file1 and sort file2. Or maybe compare grep foo file1 and grep foo file2.

Of course, there’s the obvious way – run the commands, redirect their output to temp files, and compare the temp files:

$ sort file1 >tmpf1
$ sort file2 >tmpf2
$ diff tmpf1 tmpf2
...
$ rm tmpf*

But, really..? 3 extra commands just for the one command you really wanted?

There must be a better way! This post is about the better way 🙂 .

Even better, the solution is more general than diff. It allows using the STDOUT of any command as a “file” input for other commands!

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Shell Foo: Subtracting Lists With comm

The comm utility is powerful, yet not well known.

Named after the word “common”, you can use the comm utility to select or reject lines common to two files.

Practically, once I learned about this tool, I use it often to:

  1. Get all lines that exist in both files (“select common” lines).
  2. Find lines that are missing from one file compared to another (“reject common” lines).

Just don’t forget that comm requires that the input files are sorted lexically! That shouldn’t be a problem, since you can always use sort before calling comm. But you do need to remember to do this, otherwise you might get unexpected results (which may or may not happen to me on a regular basis).

Shell-Foo credit for this one: Eyal Fink.

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

How To Use SSH With Multiple GitHub Accounts

All I wanted is to have two GitHub accounts, and use them both from the same computer, using SSH. Is this too much to ask?

I’ve been using my personal GitHub account for some time, and associated my default SSH keys with that account. When I created a work account on GitHub, I expected I’d be able to associate the same SSH keys with the work account, so I can git pull/push as easily.

This proved to be a problem. GitHub does not allow reusing SSH keys across accounts. Here is the message I got when trying to do that:

I managed to overcome this problem by creating new SSH keys for the work account. Read on for the details.

Continue Reading…

Shell Foo: Simple Distributed grep on a GCE Cluster

You just finished running a distributed multi-phase pipeline on a cluster of 200 GCE VM’s. Good for you! But something doesn’t look right, and you’d like to investigate by grepping the local log files across all nodes and do something with the results. How would you do that?

Let’s say that if you had 1-2 machines, you would ssh into each, and run this command:

$ grep ERROR /tmp/*.log.* >errors

This post describes a simple one-liner that scales the local grep to cluster-scale.

Shell-Foo credit for this one: Eyal Fink.

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Supporting External Libraries In My SCons Shortcuts

By Thursday, February 26, 2015 0 , , Permalink 0

This is the fifteenth post in my SCons series. This post introduces a small enhancement to my SCons shortcuts system – nicer support for external libraries via the with_libs keyword argument.

In recent episodes, I needed to link with the protobuf library to use the Protocol-Buffers-based AddressBook library. I did this by adding the LIBS=['protobuf'] argument to the Program target, which works just fine.

If this works “just fine”, why mess with it? Well, I already mentioned my OCD, haven’t I? I already have a nicer way to refer to libraries I use, so why can’t I write with_libs=['AddressBook::addressbook', 'protobuf']? It looks a bit cleaner.

The reason this would not work as is, is because I lookup the with_libs entries in a shared dictionary of project-specific libraries (more no that in the post that introduced the shortcuts system), and “protobuf” is not a project library.

This post extends the shortcuts system to support also external libraries. In addition to improved aesthetics, I add a couple of useful features:

  • Support configuration-based list of “supported external libraries”. This allows for centralized control of external libraries used across the project, which can be very useful in projects that want to enforce policies about library-usage (e.g. licensing requirements etc.).
  • Simpler support for libraries that are not installed system-wide, taking care of icky details, like CPPPATH and LIBPATH crap.
  • Protection against potentially difficult troubleshooting due to library name typo’s.
  • External library aliases and groups.

This episode picks up where the previous episode left off. Read on for the full details, or check out the final result on my GitHub scons-series repository.

Continue Reading…

Adding SCons Proto Builder Shortcut

This is the fourteenth post in my SCons series. The topic of this post is adding a shortcut for the custom SCons Protoc builder from the previous episodes.

The shortcut is in line with the SConscript simplification approach described in an earlier episode. In this installment, I add a new Proto shortcut to the collection, so the address book SConscript can look like this:

"""AddressBook proto-based library SConscript script"""

Import('*')

AbProtos = ['person.proto', 'addressbook.proto']

Proto(AbProtos)
Lib('addressbook', protos=AbProtos)

The final result is available on my GitHub scons-series repository.

Continue Reading…

Shell Foo: Merging Common Files In a Directory

Say you have two directories with a bunch of text files. How can you create a third directory that will contain all the files that are common to both input directories, such that every output file is a concatenation of both input files?

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Shell Foo: Parallelizing Multiple wget Downloads

Got a bunch of files to download? Got an open terminal session? Want to use wget to parallelize the download?

How about this:

echo http://dl.whatever.com/dl/file{1..1000} | xargs -n 1 -P 16 wget -q

Shell-Foo is a series of fun ways to take advantage of the powers of the shell. In the series, I highlight shell one-liners that I found useful or interesting. Most of the entries should work on bash on Linux, OS X and other UNIX-variants. Some probably work with other shells as well. Your mileage may vary.

Feel free to suggest your own Shell-Foo one-liners!

Continue Reading…

Supporting the SCons Help Command and Quiet Flag

This is the tenth post in my SCons series. The topic of this post is adding support for the silent / quiet flag, and the help command.

In previous episodes I’ve added various enhancements. Some of them print progress information (like processing modules, and two-pass processing). This makes my enhancements quite chatty, which isn’t a problem by itself. The problem is that these messages are printed also in SCons “silent mode” (scons -s), and this episode fixes that.

On the way, I also add a custom help message. Just because it’s fun.

The final result is available on my GitHub scons-series repository.

Continue Reading…