Closures, finally explained!

Programming language weirdos (by which I mostly mean people who program in Lisp) are forever telling you that you need closures, and that your language is unbearably primitive if it doesn’t support them.  Turns out that they’re right, but their rightness is completely obscured by the horrible, uninformative term “closure”, and even more so by the fact that the definition given for the term is usually some opaque hunk of verbiage like “a first-class function with free variables that are bound in the lexical environment” (from the Wikipedia article on the subject.)

It also doesn’t help that the examples people usually give are so lame: it’s all “Here’s a function that returns a function that yields the value of a hidden variables that is incremented every time it’s called”.  Whenever I see examples like that, I just think “Why would I ever want such a thing”?

Ruby to the rescue!  It’s my new Programming Language Of Choice (for reasons I’ll talk about in a future article), and one of the things it’s done for me is make me go “Ohhhhh … I get it!” about closures.  It did that by sneaking up on me.  In the Ruby world, no-one talks about closures, they talk about blocks, which sound much nicer and more solid.  When I started writing Ruby, I happily ploughed in with all the blocks that one uses all the time in that language without even thinking about it; and only after a few days of this did I realise I’d been had … WAITTA MINNIT — THESE “BLOCK” THINGS ARE CLOSURES!

Here is my definition of a closure, which I think should replace the boring version currently in the Wikipedia article.  Get ready:

Definition: a closure is a callback that Just Works.

That’s all.

I will now demonstrate, using a function in C and in Ruby.

Back in the 1990s, when I worked for System Simulation (great guys, by the way), we had a big in-house text-database system, written in C (hey, it was started in the early eighties, what else was it going to be written in?).  It was very nice, modular code, and one of the modules represented record lists stored on disk.  A function in that module allowed you to traverse such a list, sending each datum in turn to a callback function that you specified.  So that you could do what you wanted to with the datum, an additional piece of information, a typeless pointer, had to be passed into the traversal function, and was then passed into the callback.  We did this all the time — the same pattern in many modules. We conventionally called the additional-information pointer “udata”.  So we’d invoke the traverser with code like this (slightly simplified for clarity):

void print_list(LIST *l, FILE *fp) {
    list_traverse(l, printitem, (void*) fp);
}

static void printitem(LISTITEM item, void *udata) {
    fprintf((FILE*) udata, "%lu %s\n", item.address, item.val);
}

I’m sure you’ve seen similar code many times, in C, C++ and Java.  The udata pointer and the type-casts are necessary because the caller print_list() and the receiver printitem() both know that a file-pointer is needed, but list_traverse() quite properly knows nothing about file-pointers.

Now this is nicely designed C code; but it’s still C code, and it suffers accordingly from residual ugliness which comes from several sources:

• Because C is not object-oriented, we need to fake OO with module-specific functions like list_traverse() that take the list as argument rather than a receiver.  OO languages — even primitive ones like C++ — let us fix this by calling l.traverse((void*) fp) rather than list_traverse(l, (void*) fp)
• In C and C++, the callback code — in this case printitem() — has to be a separate function (so that it has an address which we can pass to list_traverse()).  More modern languages — even primitive ones like Java — have facilities for writing the callback code inline, which helps.
• Although print_list() and printitem() both know that the extra information pointer being passed around really points to a FILE structure, the traverser doesn’t; so the poor pointer has to be cast to (void*) for the traverser’s benefit, and then cast back to (FILE*) when we want to actually use it.  This is not only ugly, it also drives a truck right through the static typing that these languages use.  It may be that Java has ways to avoid this, in these days of generics and suchlike — I can certainly imagine an ethically horrifying C++ “solution” using templates that would be much worse than the problem it solves.  Other languages, such as Ruby, just ignore the problem by not declaring the types of objects at all — an approach which, to be honest, I am still ambivalent about.  (Topic for a future post, I think.)
• Finally, what the heck is that udata doing in there at all?  It’s ugly, it’s dumb, we surely want to just not have to mention it — have our callback Do The Right Thing all on its own.  To get rid of the udata, you need — get ready for the actual point of this article — a closure.

Putting it all together, here’s how the print_list code looks in Ruby:

def print_list(l, fp)
    list.traverse { |item| fp.printf "%lu %s\N, item.address, item.val }
end

That’s it — one line of actual code, wrapped into a function.  This is much more compact and readable than the C version, and rather better than the C++ or Java versions would be — note the specific improvements (based on the bullets above):

• The trivial one: OO syntax lets us call list.traverse()
• The callback code is inline — it’s a closure^H^H^H^H^H^H^Hblock.
• We don’t have to cast the type of fp (in fact we don’t even state its type)
• In fact, we don’t even pass it into the callback!

That last point is the magic that make it a closure.  Remember, Wikipedia defined a closure as “a first-class function with free variables that are bound in the lexical environment” and noted its intoxicating effect on certain carbon-based life-forms.  That “bound in the lexical environment” bit just means that from within the closure, you can still see the local variables that you had when you defined it — in this case, the file-pointer fp.

Folks, that’s all a closure is.  It’s a callback that can see the variables you had when you defined it.  And it turns out that when you have simple syntax for them, you use them all the time without even noticing.  Like in this trivial Ruby program for printing the first five integers to a specific file-pointer:

(1..5).each { |i| fp.puts i }

Who would even stop to look at that code and think about what it’s doing?  It’s so natural that it just doesn’t register.  But the fact that the code inside the curlies knows what fp is, is thanks to the magic of closures.

Here is another neat but trivial example.  Have you ever written logging code like this?

maybePrintDebugInfo(5, generateDebugInfo(obj))

The 5 in there is the debugging level of the output that might get emitted — maybePrintDebugInfo() has a notion of what the current logging level is, and silently discards messages whose level is higher that the current level.  In this way, if you set the logging level to 0, 1, 2, 3 or 4, then you avoid seeing all the stuff that generateDebugInfo() blats all over your logfiles. But what if generateDebugInfo() is expensive?  I’ve seen code like this where generating the info is itself a significant overhead, maybe involving network accesses and suchlike.  You don’t want to go to all the trouble of generating that information only to throw it away when the logging level is low, do you?

The obvious “solution” is to enclose the maybePrintDebugInfo() call [and by the way, I am starting to wish I’d made up a shorter name now that I’ve typed it so many times] in a test, so that the info is never generated if the logging level is low:

if (getDebugLevel() >= 5) {
    maybePrintDebugInfo(5, generateDebugInfo(obj))
}

But that is all kinds of horrible.  It violates encapsulation by going off to the logging module to find out what logging level it’s running at, and it violates DRY by repeating the magic number that specifies the logging level.

Here is the Ruby approach:

maybePrintDebugInfo 5 { generateDebugInfo obj }

Got it?  We’re not passing in the result of calling of generateDebugInfo; we’re passing in a closure that says to invoke that function on obj.  (And obj is available later on, when that invocation is made from within maybePrintDebugInfo, thanks to the magic of closures).  Down in maybePrintDebugInfo, the closure never gets invoked at all if the current logging level is less than 5, so our expensive info-assembling procedure never wastes our time.

I just love this.  Yes, it’s trivial, but it’s so right.  It’s a significant growth in expressive power, and it leaps very readily to the fingertips once you’ve got the hang of it.

I could say more, but it looks like it’s quarter past three in the morning, so I think I’ll call it a night.

Advertisement