Closures, finally explained!

Programming language weirdos (by which I mostly mean people who program in Lisp) are forever telling you that you need closures, and that your language is unbearably primitive if it doesn’t support them.  Turns out that they’re right, but their rightness is completely obscured by the horrible, uninformative term “closure”, and even more so by the fact that the definition given for the term is usually some opaque hunk of verbiage like “a first-class function with free variables that are bound in the lexical environment” (from the Wikipedia article on the subject.)

It also doesn’t help that the examples people usually give are so lame: it’s all “Here’s a function that returns a function that yields the value of a hidden variables that is incremented every time it’s called”.  Whenever I see examples like that, I just think “Why would I ever want such a thing”?

Ruby to the rescue!  It’s my new Programming Language Of Choice (for reasons I’ll talk about in a future article), and one of the things it’s done for me is make me go “Ohhhhh … I get it!” about closures.  It did that by sneaking up on me.  In the Ruby world, no-one talks about closures, they talk about blocks, which sound much nicer and more solid.  When I started writing Ruby, I happily ploughed in with all the blocks that one uses all the time in that language without even thinking about it; and only after a few days of this did I realise I’d been had … WAITTA MINNIT — THESE “BLOCK” THINGS ARE CLOSURES!

Here is my definition of a closure, which I think should replace the boring version currently in the Wikipedia article.  Get ready:

Definition: a closure is a callback that Just Works.

That’s all.

I will now demonstrate, using a function in C and in Ruby.

Back in the 1990s, when I worked for System Simulation (great guys, by the way), we had a big in-house text-database system, written in C (hey, it was started in the early eighties, what else was it going to be written in?).  It was very nice, modular code, and one of the modules represented record lists stored on disk.  A function in that module allowed you to traverse such a list, sending each datum in turn to a callback function that you specified.  So that you could do what you wanted to with the datum, an additional piece of information, a typeless pointer, had to be passed into the traversal function, and was then passed into the callback.  We did this all the time — the same pattern in many modules. We conventionally called the additional-information pointer “udata”.  So we’d invoke the traverser with code like this (slightly simplified for clarity):

void print_list(LIST *l, FILE *fp) {
    list_traverse(l, printitem, (void*) fp);
}

static void printitem(LISTITEM item, void *udata) {
    fprintf((FILE*) udata, "%lu %s\n", item.address, item.val);
}

I’m sure you’ve seen similar code many times, in C, C++ and Java.  The udata pointer and the type-casts are necessary because the caller print_list() and the receiver printitem() both know that a file-pointer is needed, but list_traverse() quite properly knows nothing about file-pointers.

Now this is nicely designed C code; but it’s still C code, and it suffers accordingly from residual ugliness which comes from several sources:

  • Because C is not object-oriented, we need to fake OO with module-specific functions like list_traverse() that take the list as argument rather than a receiver.  OO languages — even primitive ones like C++ — let us fix this by calling l.traverse((void*) fp) rather than list_traverse(l, (void*) fp)
  • In C and C++, the callback code — in this case printitem() — has to be a separate function (so that it has an address which we can pass to list_traverse()).  More modern languages — even primitive ones like Java — have facilities for writing the callback code inline, which helps.
  • Although print_list() and printitem() both know that the extra information pointer being passed around really points to a FILE structure, the traverser doesn’t; so the poor pointer has to be cast to (void*) for the traverser’s benefit, and then cast back to (FILE*) when we want to actually use it.  This is not only ugly, it also drives a truck right through the static typing that these languages use.  It may be that Java has ways to avoid this, in these days of generics and suchlike — I can certainly imagine an ethically horrifying C++ “solution” using templates that would be much worse than the problem it solves.  Other languages, such as Ruby, just ignore the problem by not declaring the types of objects at all — an approach which, to be honest, I am still ambivalent about.  (Topic for a future post, I think.)
  • Finally, what the heck is that udata doing in there at all?  It’s ugly, it’s dumb, we surely want to just not have to mention it — have our callback Do The Right Thing all on its own.  To get rid of the udata, you need — get ready for the actual point of this article — a closure.

Putting it all together, here’s how the print_list code looks in Ruby:

def print_list(l, fp)
    list.traverse { |item| fp.printf "%lu %s\N, item.address, item.val }
end

That’s it — one line of actual code, wrapped into a function.  This is much more compact and readable than the C version, and rather better than the C++ or Java versions would be — note the specific improvements (based on the bullets above):

  • The trivial one: OO syntax lets us call list.traverse()
  • The callback code is inline — it’s a closure^H^H^H^H^H^H^Hblock.
  • We don’t have to cast the type of fp (in fact we don’t even state its type)
  • In fact, we don’t even pass it into the callback!

That last point is the magic that make it a closure.  Remember, Wikipedia defined a closure as “a first-class function with free variables that are bound in the lexical environment” and noted its intoxicating effect on certain carbon-based life-forms.  That “bound in the lexical environment” bit just means that from within the closure, you can still see the local variables that you had when you defined it — in this case, the file-pointer fp.

Folks, that’s all a closure is.  It’s a callback that can see the variables you had when you defined it.  And it turns out that when you have simple syntax for them, you use them all the time without even noticing.  Like in this trivial Ruby program for printing the first five integers to a specific file-pointer:

(1..5).each { |i| fp.puts i }

Who would even stop to look at that code and think about what it’s doing?  It’s so natural that it just doesn’t register.  But the fact that the code inside the curlies knows what fp is, is thanks to the magic of closures.

Here is another neat but trivial example.  Have you ever written logging code like this?

maybePrintDebugInfo(5, generateDebugInfo(obj))

The 5 in there is the debugging level of the output that might get emitted — maybePrintDebugInfo() has a notion of what the current logging level is, and silently discards messages whose level is higher that the current level.  In this way, if you set the logging level to 0, 1, 2, 3 or 4, then you avoid seeing all the stuff that generateDebugInfo() blats all over your logfiles. But what if generateDebugInfo() is expensive?  I’ve seen code like this where generating the info is itself a significant overhead, maybe involving network accesses and suchlike.  You don’t want to go to all the trouble of generating that information only to throw it away when the logging level is low, do you?

The obvious “solution” is to enclose the maybePrintDebugInfo() call [and by the way, I am starting to wish I’d made up a shorter name now that I’ve typed it so many times] in a test, so that the info is never generated if the logging level is low:

if (getDebugLevel() >= 5) {
    maybePrintDebugInfo(5, generateDebugInfo(obj))
}

But that is all kinds of horrible.  It violates encapsulation by going off to the logging module to find out what logging level it’s running at, and it violates DRY by repeating the magic number that specifies the logging level.

Here is the Ruby approach:

maybePrintDebugInfo 5 { generateDebugInfo obj }

Got it?  We’re not passing in the result of calling of generateDebugInfo; we’re passing in a closure that says to invoke that function on obj.  (And obj is available later on, when that invocation is made from within maybePrintDebugInfo, thanks to the magic of closures).  Down in maybePrintDebugInfo, the closure never gets invoked at all if the current logging level is less than 5, so our expensive info-assembling procedure never wastes our time.

I just love this.  Yes, it’s trivial, but it’s so right.  It’s a significant growth in expressive power, and it leaps very readily to the fingertips once you’ve got the hang of it.

I could say more, but it looks like it’s quarter past three in the morning, so I think I’ll call it a night.

About these ads

55 responses to “Closures, finally explained!

  1. Nice blog, great pic of Veronica and Buffy (you have a thing for petite blondes perhaps…) And a good subject to kick off with. In fact a colleague of mine was only asking me about this just a couple of days ago as it happens. I tried to explain but I could tell from his glazed expression that there was no mind meld going on. In fact his main question was “so why would I ever need one?” Well now I can point him here for the definitive answer: so’s you can print the numbers 1 to 5! Just kidding….
    Actually, I found this quite enlightening: http://www.artima.com/intv/closures2.html
    I particularly like the bit where Yukihiro Matsumoto (the inventor of Ruby) is asked what are the benefits of closures over blocks, and he kind of hints that he’s not too interested in the technicalities, he just wanted to do what Lisp does!

  2. Thanks, Vince. It’s not the petite blondes (though I can see how it might look that way), it’s the sensational construction and execution of both shows, and a huge part of that is the performances of Kristin Bell and Sarah Michelle Gellar. I think they are both stellar actresses. (Not that in saying that I intend for a moment to deny that they are, also, easy on the eye.)

    I think that the trick with closures, in the end, is not to think about them. You just find that, without meaning to, you’re using them. It doesn’t occur to you that they might not work once you get into the habit. No doubt VSL will support them :-)

  3. futilelaneswapper

    I guess VSL will have to support closures now :) Maybe something like this, where the lambda function is indicated by a block preceded by ^

    func (Number x)
    console.println(x.str)
    ^ {
    x += 1
    console.println(x.str)
    }
    }

  4. Well, I urge you to use something spiritually equivalent to Ruby’s closure syntax, i.e. one where you don’t have to think “Oh, oh, oh, I’m using a closure!” but where you just do it without noticing.

    (I have a post in the pipeline about the huge difference syntax makes, so that for example JavaScript’s closures, which are equally powerful as Ruby’s feel like a relatively Big Deal to use.)

  5. Just surfing your articles and I was wondering: why didn’t you use closures in Perl? They’re pretty much the same as Ruby’s, and they’ve been there for the past 10 years or so.

    sub maybePrint($&) {…}

    maybePrint 5, sub{ generateDebugInfo() };

    maybePrint 5, \&generateDebugInfo;

  6. If you haven’t, you might want to take a look at Smalltalk’s blocks/closures, which were, of course, the inspiration for Ruby’s.

    It’s not only relevant to your discussion of closures but also to the upcoming discussion of Syntax to which you allude in your last comment. This is because Smalltalk has one of the most minimal syntaxes I’ve ever seen. Even conditional and looping syntax is non-existant and blocks are used instead. For example (here, ifTrue: and ifFalse: are methods implemented on the class Boolean and [] are the syntax for a block):

    (a < 5) ifTrue: [ a := 0 ]

    or a loop (the first block is executed on each iteration and the second whenever the result is true):

    [ a < 5 ] whileTrue: [ a := a + 1 ]

    Because, as you mention, execution of the blocks is delayed, you can safely write:

    (a < 5) ifTrue: [ "compute something expensive" ] ifFalse: [ "compute something else expensive" ]

    or even short-circuit with:

    ((a < 5) or: [ "something expensive" ]) ifTrue: [ … ]

    That's just the tip of the iceberg, of course. :)

  7. Thanks, Julian. You’re right I should look into SmallTalk before I go spouting off much more :-)

  8. Kyzer, you are of course right that Perl’s closures are equivalent to Ruby’s. The reasons I don’t use them are interesting because of how trivial they are: 1, the syntax is not as nice; and 2, there is not a culture of doing things that way in Perl. Why should such trivial reasons have such profound consequences? I hope to talk about that in a forthcoming entry. I may have to quote Wittgenstein, if that’s not too pretentious :-)

  9. I was lucky enough to come at this the other way around.
    I first used closures in Javascript where they were just a way of implementing event handlers.
    I was able to sidestep the theory and just see them in action.

    When I came to Ruby I was familiar with the concept but not the name.
    At some point it all came together in my head; that blocks, closures and that strange thing I was doing in Javascript were all one and the same thing.

    Saying that, I was with Vince (see first comment above) when he tried to explain closures to our mystified colleague and I couldn’t manage it either. :-)

  10. Nice discussion of closures. You are perfectly right that the name “closure” is not appealing. But also, closures were historically expensive enough to implement that they didn’t “fit” in languages like C. However, this has recently been done, and done “right” in my opinion, with Apples… “blocks” extension to C. See http://en.wikipedia.org/wiki/Blocks_(C_language_extension).

    Also, if you are interested in syntax, may I invite you to look at my own programming language, XL (http://xlr.sf.net)? It has a rather novel approach to the problem, I think. For instance, here is how ‘if-then-else” is implemented in XL using… closures:

    if true then X else Y -> X
    if false then X else Y -> Y

    This can obviously only be done if X and Y are closures (otherwise you would not be able to use in the ‘if’ statement variables from the surrounding context).

  11. I first encountered closures in Smalltalk’s blocks and it was amazing. One of my favorite things about Ruby is that they just reused the syntax without trying to—for lack of a better phrase—rebrand it. There’s something to be said for not messing with a good thing.

  12. Re: Smalltalk “Closures”. Not quite. For single-threaded callbacks that “just work” maybeso. But Smalltalk (as implemented by Digitalk as Smalltalk/V) turned out to violate a principle that made it impossible to launch multiple copies of a context because its local variables were defined to be part of the block, not part of its runtime representation. So launching two instances to run concurrently would have an assignment in one instance change the value in all instances because there was in fact only one copy of it.

    I complained to Digitalk that this was a bug. Their response was that it was a feature of the language definition so live with it.

    This was in 1990 or so. Maybe things changed since in the Smalltalk world, but since I needed to run multi-channel datacomm sessions, and closures looked like the most natural way to implement the receiver callbacks, I had to drop Smalltalk and resort to C and lots of ugly custom infrastructure.

    Positive thing about Smalltalk? It really established a ground plane for my mind to grok OO and MVC paradigms.

    Why not C++ instead of C? Not available on the Mac OS at the time. AT&T advertised it, but when I called to order a copy they said “interesting. You are the 6th person to ask for this. When the count hits 10000 we might consider actually implementing it”.

  13. i’ve edited the wikipedia entry to include your version (“a callback that can see the variables you had when you defined it”). more of the CS and math related entries on wikipedia could do with some nice clear sentence stating it in simple terms like this..

  14. Ha, I got this thing about closures when I read “Pro Javascript Techniques” by John Resig.

  15. Pingback: Slicing your 2D class/function matrix vertically or horizontally « The Reinvigorated Programmer

  16. Pingback: The long-overdue serious attempt at Lisp, part 2: is Lisp just too hard? « The Reinvigorated Programmer

  17. Thought I’d mention the language (and explanation) that made closures work for me.

    http://www.lua.org/pil/6.1.html

  18. I was really glad you posted this. It finally got me off my ass and starting to write for blocks, and that’s really, really upped my game in just a couple of days.

    Thanks!

  19. Thanks for that James, it’s a great encouragement!

  20. (I realize I’m late to the party, but some of us have to get work done and kids to raise and only get to read nicely written blog posts at insanely late hours…)
    The (traditional) C++ way to do this is to pass a function object which captures the environment (‘udata’) transparent to the function called and visible to the function object’s call operator.
    In C++0x (or 1x, or 11, or whatever it should be called today) of course we have lambda functions, which are closures.

  21. Pingback: We can’t afford to write safe software « The Reinvigorated Programmer

  22. As a “weirdo programmer” by you definition I’m glad you’ve seen the light in terms of closures ;). There are some other bits of Lisp that are pretty cool but also a bit hard to approach at first. You might consider learning Common Lisp- it is a bit crufty in some ways, and not as approachable as Ruby if your experience is mostly with more mainstream languages, but you might eventually find that there’s a lt in it as interesting and useful as closures are.

    Anyway I know I am commenting a long time after your post, but i wanted to point something out. There’s no real advantage to writing someobj.somemethod() over writing somemethod(someobject) if the semantics are the same, in a language with single dispatch. The problem with that in C isn’t the syntax- it’s that C doesn’t (easily) give you polymorphism.

    On the other hand somemethod(someobject, someotherobject) is a more natural syntax in an object system that has multiple dispatch, and multiple dispatch is a really good thing, IMHO. A lot of the more difficult problems in Object-Oriented modeling just go away with multiple dispatch. CLOS, the object system in Common Lisp has multimethods. Clojure is a newer Lisp that isn’t really natively OO, but does have multimethods that can dispatch on a wide variety of criteria.

    Languages without closures make you use a lot of superfluous machinery to achieve effects that are trivial with them. The situation is very similar with multiple dispatch versus single dispatch, and maybe worse, as having to decide which method belongs on which type makes modeling harder than it has to be, which can mean worse models. Check out CLOS and Clojure . You might find that you get as much from them as you have gotten from learning Ruby.

  23. Tagore, thanks for your comment. (Yes, it’s late, but I see all comments, so no problem.)

    As I mentioned in a later post, I do think there is an intrinsic advantage to the object.method notation, albeit not a game-changer: that you can read from left to right. So

    “”.methods.sort.grep /index/i
    

    is nicer to read than

    grep(sort(methods(“”)), /index/i)
    

    which has to be read from the inside out.

    In other later posts, I started to chronicle my attempts to properly grok functional languages, through learning Scheme. That effort has stalled, more than anything due to other calls on my time. I will return to it when I can. Although I am very suspicious of fundamentalist functionalism (Haskell and suchlike), it’s very clear to me that there are other functional techniques that, like closures, I could really benefit from learning.

  24. Well, perhaps “no real advantage” is too strong. I’m pretty used to reading things in that order, so it doesn’t bother me much, but I can see why you would prefer the other order. I do think though that having multimethods is actually a pretty big deal- they are something I really miss when I do OO in languages that lack them.

    I think that for problems where functional programming fits well it has a lot of advantages, particularly in the face of concurrency. And pure functional languages have the advantage that they can make guarantees about immutability. But I also find some things quite unpleasant to program in a purely functional style, so I tend to prefer languages that are more multi-paradigm, or that at least give you an easy escape hatch.

    CL is not particularly a functional language, though it supports FP pretty well. It has a pretty complex stateful object system called CLOS that has a lot of interesting features. I have to admit that I’m growing a bit suspicious of OO in general, but CLOS is my favorite OO system.

    Special variables are also worth a mention- they are something like “global variables that just work.”

  25. Yes, concurrency is one of the main areas where the advantage of a functional approach is obvious even to an Imperative Jockey such as myself. It’s very easy to see why you might want specific routines of your program to be stateless, and it’s great when a language not only helps you write them that way but even allows you to enforce that property. But I am not able to shake off the feeling that many or most of FP’s problems come from the idea that all state is always bad, and the mentality that can come with it that programming not only can but should be like puzzle-solving. Which is a long way around to saying that, I agree, multi-paradigm languages are the ones that appeal to me most.

  26. Well, at the risk of outstaying my welcome on such an old post…

    I do think that there are things that are tricky enough to do purely that as a practical matter they shouldn’t be done that way. But I think there is a great deal of value in trying to do things purely even when you find them uncomfortable.

    When I first learned Scheme I was pretty much a C programmer (albeit a fairly green C programmer,) and we were required to avoid imperative constructs for a lot of our exercises (this was in a really wonderful undergrad class taught by the late, great Robin Popplestone.)

    I found a lot of the exercises really uncomfortable, and was annoyed by them. I could have done them very easily iteratively- I remember it taking me a day or so to solve an exercise that I could have written in C in 5 minutes, including debugging.

    Later in the semester we were given an exercise that required translating a subset of Pascal to Scheme (or was it the other way around… this was a long time ago,) which was pretty heady stuff in my second semester. I realized after doing it that I just would not have been able to do it in C in the amount of time we were given, without it being riddled with bugs- not that it was that hard, but I was pretty inexperienced at C too at that time. A lot of the things I had thought were really awkward to write functionally had become idioms that I now found no harder than the iterative equivalents.

    And I could compose these idioms in ways that made things that would have been very hard to get right using imperative idioms run correctly on the first or second try if I understood the problem well enough. In a way, I had pushed the work of correctness to an earlier point in the process- correctness was now mostly determined by how well I had thought through the problem at the outset. If only real-world programming were always like that ;).

    A Chinese friend recently asked me to help him understand a line by Blake: “The road of excess leads to the palace of wisdom.” This is an odd sentiment if you come from a culture strongly influenced by Confucianism. This led to a discussion of the idea that you must go too far to know that you have gone far enough, and what that means.

    I am convinced that absolutely pure FP is more trouble than it’s worth (leaving aside issues of necessary side-effects like I/O.) There is a line, somewhere, where even the best conceivable (human) FP programmer ought to resort to state. But I am also pretty sure that I still find things awkward to do purely that are better done purely. As a pragmatic matter I should certainly not try to do them purely- I have work to do. But I do still work on learning functional idioms for things I would rather do iteratively. I haven’t gone too far yet so I am not sure if I have gone far enough.

    That said, if you want a language that is really unbiased take a look at Common Lisp. It does imperative, OO, and FP (with qualifications) natively and I’d argue that it has the best class-based OO system out there. On top of that it is very able to support ideas that it doesn’t do natively.

    I wanted to comment on your static vs. dynamic typing post, but thought that maybe I should only necro-post to one ;). You can build a language on top of CL with any type system you want, and close to any syntax you want, and have it interoperate seamlessly with the rest of CL, if you have a few years to devote to the task- it just takes that long to build a robust language.

    I wonder if you are aware of Qi (http://www.lambdassociates.org/qilisp.htm)? It’s an extension to CL with a rich type system (I am really skeptical that crude type systems like those of C++ and Java add safety, but I might be willing to believe that more sophisticated type systems do,) and type inference, etc. I haven’t used Qi, so I don’t know if it is practical or not, but I do know that whatever your dream language is it can be built on top of CL as long as you can stomach a leading (or was that trailing…) parenthesis- hard to get rid of that.

    It’s too bad that CL has so much historical cruft- much of it doesn’t matter, once you understand CL, but it still makes it annoying to get started with, and these days that is death for a programming language (see the success of PHP, the best web programming language to write Hello World in, and close to the worst for almost any other task.) And some of the cruft really does remain annoying.

    Clojure gets rid of a lot of the cruft of CL, but it is not an unbiased language in the way that CL is (not a criticism of Clojure- Clojure is very nice for what it is, but it is not a better CL.) It is opinionated and prefers a functional style. What I would really like for (some) Christmas is a better CL, but there are a lot of reasons that that is not likely to happen- if not for those reasons I would (at least try to) start something like that myself.

    Anyway, sorry for the very long comment on an old post. Maybe I should start a programming blog.

  27. In a way, CL reminds me of your posts on the Silmarillion. It is hard enough to get to the meat of it that a lot of people will just abandon the whole thing at the beginning. If it were in my power to republish and re-edit the Silmarillion I would (hypothetically if I were a great editor or a great language designer) fix it, so that the bits most likely to draw you in were at the start, and the bits most likely to send you away were optional. I can’t redo CL or the Silmarillion though- all I can say is that CL is an imperfect and sometimes annoying condensation of a great work, just as the Silmarillion is. I like you, do not use the word “great” lightly here.

  28. Pingback: Tagore Smith on functional programming | The Reinvigorated Programmer

  29. A note to anyone coming to the party late and reading Tagore’s comment: I liked them so much that I posted them as their own article (with his permission). So I think further discussion belongs over there.

  30. Another Attempt on Closures

    (Posted here since it’s not a comment on Tagore Smith’s excellent posting.)

    Another attempt to explain closures, here with the help of world’s most used programming language, JavaScript. Stop laughing. JavaScript was heavily influenced by Scheme since its early beginnings (even if some of those features only showed up with the 3rd edition of the ECM-262 standard). The use of closures with callbacks is quite familiar in JS, but can be explained better while looking at a trivial construct.

    Let’s take a simple example, summing up squares, first done the traditional way:

    var sumSquares = function(n) {
        var total = 0;
        function square(r) {
            return r * r;
        }
        total += square(n);
        console.log("sumSquares total: " + total);
    };
    
    sumSquares(2);
    sumSquares(3);
    sumSquares(4);
    
    /*
    logs:
        sumSquares total: 4
        sumSquares total: 9
        sumSquares total: 16
    */

    Obviously this doesn’t work as expected: The total isn’t stored or static in any way. The var total is part of the scope of the function referred to as sumSquares and just expires after the function has been executed. So it is initialized every time we call this function. (Please mind that JS uses a function-based scope, not a block-based one.)

    So let’s have a look at the magic of closures:

    // get a closure by executing an anonymous function
    
    var sumSquares2 = ( function() {
        // anonymous function provides a private scope
        var total = 0;
        function square(r) {
            return r * r;
        }
        // return the actual function
        return function(n) {
            total += square(n);
            console.log("sumSquares2 total: " + total);
        };
    } )();
    
    sumSquares2(2);
    sumSquares2(3);
    sumSquares2(4);
    
    /*
    logs:
        sumSquares2 total: 4
        sumSquares2 total: 13
        sumSquares2 total: 29
    */

    Now, this time we get the cumulative sum of each of the squares. Why?
    total and square are still part of the private scope, they are not a property of anything we could refer to in dot-notation, nor are they part of the global scope. They should expire – together with the scope they belong to – after the execution of the outer anonymous function. So why?
    This is due to the anonymous function returned by the outer function on execution, which is stored in the var sumSquares2. By the return of the inner lambda function not only executable code is returned and stored in a variable, also the execution context of the outer lambda function is stored as the execution context of this code and such prevented from expiring. This way the scope of the outer function survives and its execution is provided to the inner function.

    This is even better illustrated by a common pitfall:

    var sumSquares3 = ( function() {
        var total = 0;
        var stack = new Array();
        function square(r) {
            return r * r;
        }
        // loop to store some closures in a stack to be returned
        for (var i = 2; i < 5; i++) {
            stack.push(
                function() {
                    total += square(i);
                    console.log("sumSquares3 total: " + total + ", i: " + i);
                } 
            );
        }
        return stack;
    } )();
    
    sumSquares3[0]();
    sumSquares3[1]();
    sumSquares3[2]();
    
    /*
    logs:
        sumSquares3 total: 25, i: 5
        sumSquares3 total: 50, i: 5
        sumSquares3 total: 75, i: 5
    */

    So what the *** is going on here? Shouldn’t var i be stored in a closure for any of the instances of the lambda created inside the loop? Yes and no. It’s obviously there, since we don’t get any error on using a unknown variable. So why is it 5 and why is it 5 each time? Remember: it’s not the value of the variables that survives, it’s just the scope. And after passing the loop var i is obviously set to a value of 5. So when we call the instances of the inner function from the stack (which happens after passing the loop) the value of i is 5 – and this for each of the instances since they share the same execution context and scope (which is created on the execution of the outer function).

  31. @ ideal Lisp
    Maybe we should consider JavaScript a bit more as a serious programming language. With the rise of highly optimized compilers/VMs like V8 and according server-based solutions like Node.js JavaScript might have a brilliant future. You can treat it as some kind of Scheme with Java/C-style syntax, as a procedual language, or as an OO language, and may mix these approches like you want or need. So it has much in common with the use-it-like-you-want style of Perl and shares (with server-side extensions) at least some of its powers with real UniCode-support. As long as you avoid cluttering the global scope, its a respectable scripting language featuring many of Scheme’s functionallity with a commonly known syntax. (And the next version will feature a “strict” mode like Perl as well.)

  32. @Norbert: Javascript should be taken more seriously than it is, both as a language and as a warning. Javascript has some basis in Scheme, but no one teaching or programming using Scheme would ever describe the mess (which I know you provided as an example of what not to do) you gave in your last code sample as a common pitfall, in much the same way that most Christian denominations would not consider killing and eating your neighbors an expected and trifling sin.

    Brendan Eich is a brilliant guy, but he was very young and under a lot of pressure when he slapped Javascript together. In retrospect it is clear that his brilliance far eclipsed his judgment. We would all have been better off (to the tune of billions, maybe hundreds of billions, unless I miss my mark) if he had just embedded Scheme in Netscape and called it a day. But it’s hard to blame him. In his position, at his age, I would have done _exactly_ the same thing.

    That said, Javascript is actually a fascinating abomination of a language. If you subset it properly it’s not a bad language at all- I would far rather program in JS than Java as long as I had a stick with which to hit fellow programmers who deviated from my idea of acceptable JS ( it’s not surprising that the best book about Javascript is subtitled “the good parts.”) And as long as (given current implementations) I didn’t care much about performance.

    On the other hand, these issues seem to have been solved by Lisp implementors a long time ago, in a more disciplined fashion….

  33. I’ve to admit that the last posting (“ideal Lisp”) didn’t address that much Lisp as I had in mind your article on looking for a replacement for Perl. I’m quite in the same position, as I do love Perl 5, but I think it’s time to change (the whole paradigm is showing age, the missing unicode-support for reg-exps, … and Perl 6 neither ready, nor am I feeling like becoming warm to it …). Personally I’m not that much in love with Ruby or Python (letting PHP as a serious language out of the account here). And there’s a whole generation of programmers who grew up under the C/Java-paradigm, for whom the syntax of Lisp requires a dramatic twist in thinking.
    To be honest as a confessing Perl-user: Most, if not all, of the atrocities that can be committed in JS can be made in Perl too. But there’s strict-mode, and I’m counting on strict-mode for the sake of JS.
    (There it is: The Stick! Grab them; hit them; they do deserve it! Maybe it has also an elvish name … Some used to call it the “Strict-dagger” and it’s said to start glowing if there’s a “with” or an undeclared variable nearby … :-) )

    On the closure examples: Yes, the last example is a dramatic mess (as a movie it would be rated adults only), but in real life it’s not that obvious in case you would loop around some UI-elements to create a bunch of callbacks as long as you haven’t a clear idea of how closures work. But again, I’ve seen similar things in Perl too …
    So the comment is not on JS vs Lisp, but on looking for a replacement for Perl and investigating in Scheme’s ways.

    Ad Brendan Eich: I haven’t read Douglas Crockford’s book yet, but I remember him stressing in a video that Eich wasn’t allowed to deliver much more than a prototype and that the whole culture of Netscape didn’t allow for “doing things right”. Nevertheless I still do think that the choice of the C/Java-paradigm was an interesting one. In the last 15 years most of the atrocities have been fixed and initially missing features have been added making up a mature language. It’s just that JS isn’t part of any formal education, it’s just out there in the wild (but this should be also common to Perl-users).

  34. Norbert,

    No time to reply properly, but let me just say this: if you are Perl programmer looking for a language that is going to be “a better Perl” — i.e. that will let you do all the same stuff, but in a much nicer way — then the language you are looking for is definitely Ruby. I know you say you’re “not that much in love with it”, but I think you will be.

  35. Norbert: I regretted my most recent comment after submitting it. I have a tendency to write for effect, and I like hyperbole as a humorous device. That can both obscure my meaning and offend. I think that the points I was making are important, but… I could have been gentler in making them.

    Like Mike I just don’t have the time to really respond (NE US here, and my commute today was hours longer than usual, because of snow) though I would like to.

    When I get a chance I will comment again- Perl is actually the first lispy language I was really good at, but I abandoned it a few year ago.

  36. Back in the day (by which I mean the mid-nineties) I made a rash decision. Actually, I made a lot of rash decisions. My grades in University were very uneven, but I took the honors section of every course I thought interesting, despite the fact that in many cases my grades in the main courses weren’t going to be high enough for me to actually get the honors on my transcript. I never cared much about my transcript, but I was sure that there were secrets that were being kept from me- some set of magic words that would make all of the murky bits of what I was studying clear. So I took the honors sections of every course I thought important because I didn’t want to miss out on the moment in which those secrets were imparted.

    In retrospect this was a wise decision- it turned out that a lot of what was murky remained murky. There is no silver bullet, and there are no magic words that make the murky clear. But there is the search for such, and that search is sometimes fruitful. My current job is basically about finding silver bullets that everyone else has given up on finding. It turns out that while there is no universal silver bullet there are lots of local silver bullets.

    It is very hard to make predictions, especially about the future. That said, I think that the work I’ve done in the last year might turn out to be important. It is a bit embarrassing to admit that I have spent a year writing what amounts to a few dozens of lines of Python in a prototype, and several hundred lines of optimized C that is still not fast enough. But if we succeed in finding the local silver bullet, and making it run fast enough, we will save so many thousands of hours of other people’s time that I will not be at all embarrassed.

    Most of what I know about programming, and about seeking local silver bullets I learned from Robin Popplestone. I took the honors section of his course and that wound up being me and Robin sitting in a room most of the time (after I found him in the robotics lab and reminded him that we had this section.) He would start talking (usually about ML,) and do so for our full hour, and I would fail to understand what he meant. My subsequent career in computing could be described as a series of insights- most of the time I think “Oh, that’s what Robin meant.”

    Robin was the father of functional language implementations (and should better known for that,) but he was not religious about this- if anything he was a scheme guy.

    What Robin really taught me was a simple thing, though I didn’t understand it until after he had died: “be disciplined in how you think about your problem.”

  37. @Tagore Smith
    Never mind, I didn’t take your comment in any way wrong or embarrassing.
    That said, I totally agree with your general judgement, and I’m also getting into functional programming more and more. (But I also think it doesn’t make to much sense to do it pure only.) I didn’t mean my posting as a comment to yours’ since my favour to JS had nothing to add to them. Even if functional programming has lots to add to client-side JavaScript-programming, like providing real private data and methods by the use of closures (a feature that might be missing in JS at first sight and which is adding much to client-side security) and easiness for callback-programming.

    I’m a bit in the silver-bullet-industry too, mainly on the client-side. Maybe this adds to my favour to JS (and offsprings like AS3, introducing compatibility to Java-API and optional strict typing). But I mainly advocated for JS having in mind the old idea of a lingua franca, which would be easy to learn and to employ for most programmers. It’s the most used language in the world and many programs (e.g., the Adobe family) already have a JS-API.
    It’s not that JavaScript would be the perfect language (but I don’t think that there is the perfect language at all; and: even Pascal had the using operator). It would be just a programmatic decision for a language that does employ most paradigms and that can be handled economically and quite safe, at least with some guide-lines. This would be exactly the profile of a handy scripting language.

    “be disciplined in how you think about your problem.”

    I’m very much with that! Imho programming is most about categorial thinking. (But again this might be a biographical approach on my side, since I do not have a degree in CS but in philosophy. Admitting this, it was one of the most reoccurring experiences — and sometimes the most embarrassing ones — in my career having to observe, how disciplined programmers — here in the sense of CS – would lack this discipline to quite an extent.) Maybe the last sample-script did look that odd because I tried to show the ambivalence of some of JavaScript’s entites. But it’s mostly for this categorial ambivalence of some of it’s features that does make it a powerful scripting language.

  38. I’m a weirdo programmer because I hate closures. I said it. They’re difficult to debug and maintain; I’ve ended up unwinding several clever ones in our code recently just to figure out what’s going wrong.

  39. It is true enough that programs written with shedloads of closures and callbacks can be difficult to understand. But then what you’re describing there sounds more like an abuse of the technique than a use of it.

  40. Maybe. I admire their cleverness but have yet to figure out how they can be effectively logged, for instance, without making them a lot less clever, and then not worth the trouble. I also have yet to find a case where a slightly more verbose but FAR more maintanable idiom didn’t do the trick (i.e. there has not yet been a need for them).

    I like Java too. When you’re dealing with fixing bugs in somebody else’s code, and remember your own code from a year ago might as well be somebody else’s, it’s nice to have them forced into verbosity sometimes.

  41. I’ve been super-busy lately, so it has taken me a long time to check back here- not sure if my response will be seen by anyone except for Mike, but…

    First off, @Norbert, your point (about categorical thinking) is well taken, and I agree with you. There are various kinds of discipline involved in programming. Some are mechanistic, but nonetheless important. Things like not only using version control (where I work now I had to introduce version control when I got there- I was a bit surprised to find that the source code for a number of successful commercial products was mostly versioned by commenting the older code out… I actually wound up having a lot of respect for the guy who did that actually, but not in regard to his source control skills ;) ) but being disciplined about how and when you commit. I switched to git a while ago, and like Mike had a hard time understanding it at first. I’ve gotten better at using git, but I had to track down a bug the other day that was both subtle and had been introduced a long time ago- I was cursing myself for the way I had been doing commits before, as it would have been much easier to find if I had been more disciplined about using git back then.

    I actually think that CS students should probably get more education in some things that are taught in Philosophy departments. I went to a reasonably good CS department, and they did try- but there’s only so much you can cram into an undergrad CS curriculum, particularly when the actual CS courses wind up being only about 1/3 of the courses taken. I’ve said (as a joke, in company where it was understood) that my profession is applied ontology ;) .) But of course a lot of the time my job is actually knowing a lot of ephemeral and arbitrary details about certain systems, or at least knowing where to look them up and how to skim the docs efficiently.

    It is precisely because JS is so widely used that I think it is a bit tragic that it wound up being so flawed. You may be right that this was more a matter of Netscape messing things up than a problem with Eich’s initial design- as I said I have a lot of respect for him. I am hard on JS because it has so many good bits, IMHO. But it also has a lot of fail, and it took a long time for a JS community to arise that had much of a clue.

    Lisp is not a popular language these days, but there has been a strong trend for languages to incorporate Lispy features. Even Java, which started off as the anti-lisp, intentionally I think, has thrown in the towel to a large degree. But the communities around these languages are often very slow to pick up on the lessons lispers have painfully acquired over four decades. I initially commented here mainly because of that- yes, you can understand closures by programming in Ruby. But there is a lot of knowledge about closures and other lispy features in Ruby bound up in the Lisp community, and it diffuses into the Ruby community slowly and imperfectly.

    If JS had been Scheme (and if browsers hadn’t been so fundamentally broken for such a long time) JS programmers would have had the experience of the Scheme community to draw on, and that would have been no bad thing, IMHO, as among other things it is one of the best languages out there in terms of the pedagogical texts that use it.

    This leads into what I would say to M1EK. It’s true that closures can be hard to debug in some cases, but this is mainly a matter of 1) being disciplined in how they are used, 2) coherent language design, and 3) having tools that take closures into account. Good Lisp programmers don’t find closures hard to debug, but that is because there is, as Kent Pitman pointed out, an ecosystem around Lisps that takes their features into account. When closures are grafted willy-nilly onto other languages the results vary considerably.

  42. @Tagore Smith
    You’re starting a nice virtaul history theme here:
    What, if Netscape had implemented Scheme?
    Some ideas on this scenario:

    * Scheme wouldn’t be a language quite arcane, but since the all web logics would be driven by Lisp (both server-side and client-side) the Lisp-community would be by far the largest programming community in the world. Scheme would be the lingua franca by definition, allowing anyone to “reach” through the layers of the entire system, regardless of where he started. The world might be better to a degree or even two and probably there would be an extra sunny day in the year. ;-)

    * Client-side web programming would have stayed a very arcane thing, related to the arcane language it used. The user-community of NS-Scheme hadn’t reached the spread of the client-side VB-Script-community, making VB-Script the lingua franca both client-side and server-side. The term “enterprise” would finally translate to MS or licensed libraries.

    * Same as above, but client-side web-scripting wouldn’t have become a success at all. Maybe even a better world than this (with a rainy day less every year, but still a cloudy day).

    Ad “my profession is applied ontology” – liked that very much! (I was once thinking of founding an Institute of Symbolic Techologies – in terms of applied ontology -, and once, some years later I caught myself thinking, hey – that’s still what I’m doing now. ;-))

    Be assured that your comments are real appetizers for going Lisp-wise.

  43. Well, that sort of parallel history is common enough in the Lisp world that I find it unremarkable ;). The truth is that almost every important development in computing in the last 40 years came from Lispers, in one way or another. If you find that a strong claim I suggest that you have a look at The Mother of All Demos: http://en.wikipedia.org/wiki/The_Mother_of_All_Demos.

    To quote wiki: “The Mother of All Demos is a name given retrospectively to Douglas Engelbart’s December 9, 1968, demonstration of experimental computer technologies that are now commonplace. The live demonstration featured the introduction of the computer mouse, video conferencing, teleconferencing, email, hypertext, word processing, hypermedia, object addressing and dynamic file linking, bootstrapping, and a collaborative real-time editor.”

    To draw up a really good alternate history here we should assume a bit more: we should assume that the browser makers actually gave us something reasonable to do with JS from the outset. I was head of tech for a pretty successful web company in the early aughties and my rules were simple: no JS for important things, and the minute it looks like your JS degrades badly it is taken out. As a result we had very little JS in our pages for a long time, which is one of the reasons we were successful, I think. Our pages loaded fast and didn’t fail, no matter your browser. And I mean no matter- if it didn’t work in Lynx it didn’t go live. Worse, it had to work in NS4.x and IE. As a result it took me a long time to understand that JS was interesting.

    I think that an interesting alternate history here wouldn’t demand actual Scheme. It might just ask for something that used M-expressions with Scheme semantics. In fact no one thought, at first, that people would keep programming Lisp using all those awful parens- it was assumed that there would be a layer over them that shielded you from them. But that never worked out well, mainly because people got attached to the parens.

  44. The truth is that almost every important development in computing in the last 40 years came from Lispers, in one way or another. If you find that a strong claim I suggest that you have a look at The Mother of All Demo

    Do you mean that Tree Meta / MOL is related to Lisp or that the NSL-Team (Bill English et al) where trained on Lisp?

    we should assume that the browser makers actually gave us something reasonable to do with JS from the outset

    Fully ackowledged! The implementation of JavaScript1.0 was pittysome (no array or object constructors, nothing more to access but forms and location properties). But you could do quite a lot with 1.1 soon. (Even, if the implementation in MSIE 3 was, er, …., *better-don’t-try-to-find-a-word-for-it*.) E.g., http://www.masswerk.at/demospace/relayWeb_en/welcome.htm (Example of an early web-app from mid 1998, MSIE 3.0-compliant, JS only, still working today without changes. Please note that MSIE 3 didn’t allow for external scripts, could be even more lean else.)
    For computer history interests: Note the use of “welcome.html”, there was a distinction in the use of index and welcome once …

    Ad parens: I think this is a main issue. The Lisp-syntax is quite arcane for any outsiders, but users identify with it and so the community seems to see no need in a change, such keeping Lisp a cause for insiders.

  45. I just want to say, Tagore and Norbert, that I am seeing your new comments, and finding them fascinating. It’s a shame in a way that they’re hidden away on this old page, where not many people are likely to see them, but at least you both have an audience of one. Keep it up!

    What was welcome.html about?

  46. @Mike Taylor We are having fun …

    @welcome.xml
    In the days of the early HTML and the late index-tag (rember this one? Same as some restfull-protocols that are now the latest cry …) an “index.html” was meant to give an index of the directory or site. (So index.html would have been a desigened version of an automatically generated directory-listing).
    Oposed to this there was the convention to use “welcome.html” as the default entrance page (which wouldn’t be a table of contents). MS IIS introduced “default.htm” for the very same purpose later.

    Note on the index-tag:
    The index-tag just provided am input field associated with a “search” button (no form required) and was meant to search the site. Clicking the “search”-button would result in a GET-request with just the search term as the only part of the query string.
    E.g.: Enter “cookie recipe”, click “search”
    http://example.com/?cookie+recipe

  47. Sorry for lack of concentration and/or typing skills :-)
    *welcome.html, *remember, *designed, *opposed &c, – oops!
    (Most of the times I can type but when the wind is nort by northwest. Some times I’m a *small village* of the keyboard.)

  48. Norbert: trust me, I am king of the typo, so no worries ;). More worryingly I think I expressed myself badly above. I don’t actually mean that everything important done in the last few decades was done by Lispers. What I mean is that in most cases when you see interesting things being done they either have some roots in Lisp (even when the people doing them are not aware of those roots) or they involve re-inventing a wheel that was first worked out in some Lisp.

    I’m also not claiming that Lisp is unique in this respect (though I think it has, in some ways at least, pride of place here when it comes to programming languages.) A distant cousin of mine, several times removed, fled Scotland for the Colonies after coming off rather the worse in the Jacobite uprisings. It turns out that more than 50% of living Cherokee are descended from him. I wouldn’t take this to mean that the Cherokee nation is mainly Scots ;). Just so, most developments in computing have roots in a number of older traditions. But Lisp shows up, one way or another, in the family trees of a suspiciously large number of important innovations.

    Now, I wouldn’t argue that this is because of some inherent superiority that Lisp has over every other conceivable or existing programming language. Instead I think it is because Lisp was the first language to come along that offered a certain kind of flexibility that turns out to be very important in doing exploratory programming. This led to a lot of very bright people (among them a few who I would consider really visionary, like Englebert) who wanted to push the state of the art along adopting Lisp as their main language. It also led to people like Alan Kay (someone I would certainly consider a visionary) developing new languages that were strongly influenced by Lisp, and doing visionary things with _them_.

    But sometime in the 80s, maybe as a result of the AI winter, maybe for other reasons, Lisp stopped being a mainstream language and became the preserve of weirdo programmers. In response, after a while, people started trying to reinvent Lisp. In some cases they did that in C, which is why we have Greenspun’s 10th law. In other cases they developed new languages that had some features of Lisp. Larry Wall never liked Lisp very much (I think he described Lisp as, and I paraphrase, a bowl of oatmeal full of fingernail clippings.) But an awful lot of Perl is clearly derived from Lisp.

    I’m not arguing that everyone should write Lisp all the time, but I am saying that a tremendous amount of knowledge was “lost” when Lisp became a niche language. Most of it wasn’t actually lost, of course (though some of it is really gone, and more of it goes away as old tapes decay, and people die.) But it isn’t being transmitted well. I’m actually all for reinventing the wheel- progress is mainly a matter of refinement, and a modern steel-belted radial tire is not the same as a wooden cart-wheel.

    I’m close to 40 now, and I suppose it’s natural that I find myself reflecting a bit on what I’ve achieved so far. I started programming seriously a bit later than most programmers do, but I’ve been programming professionally for more than a decade now. And I have to admit that I don’t think I know as much as I should (the understatement of the year here.) This is entirely my own fault, of course, but I do think that the way that we pay so little attention to the past makes it very easy to coast on things like being up on the details of the currently hot framework, no matter how arbitrary those details are. In turn, this leads to people re-inventing the wheel without knowing much of what people have learned about wheels before.

    This is why I think that JS was a bad choice for the scripting language in NS. Even the best language designers get things wrong at first, and the best languages are very unstable in their first few iterations. JS incorporated a number of features that hadn’t been seen in mainstream languages before, but since it went out as the scripting language in the browser those features were practically set in stone. It is actually a wonder that JS wound up working out as well as it did, I guess.

    As for the parens, I have a lot of thoughts on that, but they might have to wait for another comment.

  49. Also, looking over my last comment I see that I failed to make a point that I really meant to. Another problem with JS as the browser scripting language is that there was no history behind it. There was no JS community for people to go to to learn how to program in it. There was only a very small community of people who understood prototype-based class systems at the time. And the result of that was that very few JS programmers understood JS very well for years.

  50. Tagore: Mike Taylor has a point here – we’re monopolizing this thread :-)

    There’s on major point I’m quite unsure about: Is there anything like a perfect language, might there ever be one, and, would I like to use it? If this is true for natural languages, so even more for programming languages: I don’t think there is one, that there ever might be one, or that this would even might be a want-have-to.

    Side step to natural languages: There’s a theory (I’ve forgotten by whom) that some notable occurrences from German native speakers derived from a systematic lack of abstract concepts in the language itself. (E.g.: “Geist” is both mind, [a] spirit, a ghost. So abstract concepts and narratives begin to dance.) While this might be to the uttermost disturbance for a German native speaker (poets and thinkers!) there was an exemplification of this on Freud – who’s texts I happen to know to some extend – with some real insight. What we can learn from this: Every language provides a set of unique approaches, while it might not so good at covering other domains.

    I think these kind of shifts is very important. Any programmer should know and practice at least two or three languages! (I don’t think that PHP or even VB are that awful, it’s just that these languages are usually used as the single tool. Otherwise there would have been a culture of best practice preventing the worst case scenarios we can encounter all over.)

    Therefore I’m very much for the transition of smart concepts from one tool to another. I’m very much with your statement of re-inventing the wheel as a refinement process.

    And it’s also an issue of generation. IMHO, what’s so interesting and even important about Mike Taylor’s blog is the whole theme of reconsideration.

    I’m myself now in my mid 40s and also started a programming career late. (Started out just before home computers hit the market. Algol first language. Then some C64 stuff, followed by 12 years of total absence from computers. Stunned to see Pascal unified in the ’90s …) In the ’80s Lisp was the mystic stuff, legendary for its record in AI. But, yes, it looked like a thing of the past very much. Personally I was really impressed by Perl, but it’s also a piece of an era – and of a generation again. Things are changing and they should. So it’s also about: What was it exactly which made **** that revealing, and where would I like it to show up again, in an updated fashion? Maybe this shift might even be of great productivity, giving rise to a new paradigm …

    (That’s exactly were I started the argument about JS, with things like NodeJS on the horizon. –– Wow, 10K+ concurrent requests, forget about servers, a single tool – in two flavors – for both ends merging server- and client-side to a single distributed application … The hidden totality might even not proof as wise – see above – but it’s impressing to some extent.)

    P.S.: Mind the lessons learned and lost from RPG2. (There was for sure some wisdom in a large community that’s now gone forever.) Things to be learned from MUMPS – for all that I’ve read about it: probably not. ;-)

  51. Ad “welcome.html”:
    Just found an amazing piece of web-history: Netscape before it even was Netscape, still online! http://home.mcom.com/

    This site features a typical example of the ancient art of welcome.html:
    http://home.mcom.com/home/welcome.html.

  52. Norbert, that is beautiful!

  53. Trifting away to web-history …
    That site (home.mcom.com) made me wonder: Looking at the pages’ source there isn’t any html, head, or body container. Mind that these pages are by Netscape (or to be specific: the Mosaic Communications Corporation)! May it be that these constructs were introduced just later (after 1994)?

    (While having a look at the site, have in mind that the pages were originally rendered with a grey background. For an example of this see also – probably updated later –: http://mosaic.mcom.com/. These pages already feature a body tag, but no html or head.)

    http://mosaic.mcom.com/

  54. Tagore Smith

    I’m sorry that it has taken me so long to get back here to respond to the excellent points you two have raised. I can only plead that I have had a very difficult few weeks, culminating in getting some very bad news that took a little while to adjust to.

    First off, Norbert, I would have to go look it up to be completely sure, but I’m pretty sure that Tim Berners-Lee’s original HTML spec required the html, head, and body elements. HTML was originally defined using SGML and I am pretty sure that you cannot have a legal SGML document without a root element. But Mosaic and Netscape were very permissive in what they accepted, which led to a lot of trouble later, IMHO. As you might have surmised from my earlier posts I am still a bit bitter about the sloppiness of the early web browsers. I guess I made a fair amount of money navigating their problems, but I would have preferred to have been solving real problems instead of solving the artificial problems they introduced. I only have so many hours of work in me and I’d rather that they counted for something.

    Still, those pages are a great find. Nostalgia factor = very high. I just wish I could view them on the DEC systems I first browsed the web with. The monitors on those systems had a very strange color response that gave everything a sort of “dusty” look. It was actually a good thing at the time- my memories of the very early web are all pastel, dusty rose, and a better version of Netscape gray than I think most people got.

    As to the “perfect” language, I’m tempted to dispose of that question by noting that it is not well-formed. If any notion requires formalism(s) it is that of perfection. I recently watched a very interesting talk given at the very end of the Clojure conference held last year: http://clojure.blip.tv/file/4824610/

    Interestingly, it quotes our own Mike Taylor ;). I’m afraid that the presenter does so in order to disagree with him (and I will note that I also disagree with Mike about this,) but the presenter is disagreeing with him in order to highlight the subjective nature of a lot of what we programmers talk about and in order to ask questions about the degree to which we can turn some of our normative arguments into positive arguments. It’s good stuff and it is the kind of good stuff that a philosopher turned programmer might find interesting (who knew C. S. Lewis was a programmer at heart ;)?) Highly recommended.

    I’m going to have to disagree with you when you say that PHP is not that bad. I do so with a heavy heart because so much of the work out there is PHP programming and I am perfectly willing to program PHP for money. And in fact I think that understanding just how messed up PHP is is something that makes me better at doing PHP. But… PHP is simply awful, and always has been. (Here I go with the unjustified statements that Halloway is discussing in the presentation I linked above.) The best thing I have to say about PHP is that it is not nearly as bad as it was when it was PHP 3.

    Of course a number of very successful businesses have been launched using PHP. Facebook comes to mind, as does Etsy. I have a good friend who took on a management position at Etsy recently. I hadn’t realized that they were a PHP shop until I talked to him about his work. His take on it is that everyone there knows that using PHP sucks, but that the company was started by peo0ple who were better at business than at programming, and they picked the thing that was easiest to get started with.

    Frankly, that sounds like a recipe for success, even if PHP sucks, if you are in a domain where the programming challenges are easy, but the business challenges are hard. And Etsy was initially just such a beast, as are many web businesses in their early stages.

    Of course the technical challenges eventually become hard as a business like that scales. According to my friend Etsy has some really brilliant technical people now. The core of the site is still PHP, because it is so hard to do ground-up rewrites. But my impression is that (and note that this is based on a couple of conversations with someone there and may be very inaccurate) Etsy now does as much as possible in other languages.

    You can see the same pattern with Facebook, actually. PHP is very viable if you want to do something simple. Facebook is pretty simple, but making it scale is not. In the beginning Facebook was a very simple site, from a programmer’s point of view. Facebook didn’t become successful because it was technically innovative. But once it became successful it ceased to be technically easy, and you will note that nothing new at Facebook is done in PHP unless it has to be to fit with the pre-existing structure.

    My current job is basically solving a number of long-standing unsolved problems in animation software. It is not hard from a business point of view. It is hard from an algorithmic point of view. If we can successfully crack the problem (we had better, and I think we are 90% of the way there now) we are pretty confident that animators will eventually realize that our software will pay for itself in a week of use and that is a selling point.

    I could write a long comment just about our travails with C/C++/Python. I’ve learned a _great_ deal about how language choice affects how you go about solving hard problems in the last year. But this comment is already very long.

    Let me put it this way- the flexibility of the language we use is tremendously important to us. We have been prototyping in Python, and then implementing in straight C (performance is really important for us.) Python’s flexibility has been enormously important, but it is not flexible enough. It is more flexible than C, but I pine for the fjords- err Lisp- every day. If I had realized at the outset how hard the problem would turn out to be I would have used a Lisp for prototyping from the outset.

  55. @Tagore Smith
    On Netscape/mcom: Best seen on a 12″ screen!
    On HTML syntax: Early HTML has always been meant to be permissive regarding the structures used. I remember that when I hit the scene in 1996 the use of the body and head containers was regarded as some kind of insiders’ stuff. I loosely associated these with the introduction of the paragraph as a container (which was quite new then, initially there wasn’t a closing part of the “p” tag). But since we were doing “serious” stuff I haven’t wrote a single page without these constructs myself … :-)

    I really enjoyed to watch the talk Clojure conference and it it made me think (once again) about simplicity/complexity/handiness.

    But before I dwell on this, please let me state that I fully second that PHP is a bad language. (I just wanted to make a point about tradition and discipline in the sense that, provided the right discipline, you could handle PHP like Algol or any other beauty of a language. – But in fact you can’t. More on this below.) I really wanted to second your point on traditions. And I still think that Perl did become such a great language because it came with a style guide and the perlfaq and such provided an instant start into a programming tradition.

    But back to simplicity vs complexity and/or handiness regarding language designs.
    One of the popular misunderstandings about data is its aspect as a representation. E.g.: Binary encoding is the universal representation of any data. Sorry, it’s not. Because it’s not a representation at all. A representation always incorporates some concepts, which are part of the representation’s context. Binary data is just point zero, the nadir of any representation. There’s no way to hint any context. I really love the scene in the movie “Contact”, where they detect an alien binary encoded signal, have a look at it, and just know that these are some compressed video frames of a distinctive size. Just think of the amount of context needed to come to this conclusion!

    But what may this contribute to our discussion on languages? I do think that these are related to the strictness and simplicity of the underlying concepts. Let me take JS a last time as an example:
    There are two basic types, which can be addressed in pretty much the same way: Simple values and complex (objects). If I’m passing a simple value (to the left side, as an argument, etc) it will be copied. If I pass a complex value, it will be passed as a reference. That’s all. This simple principle governs how data may flow in any application. There’s no call by value or call by name. No explicit casting to reference, no pointers, etc. There may be secondary concepts like that I could even address a simple value as an object because the language will provide an automatic wrapper object, but these are clean and simple concepts too.

    In a greater context this brings us to the notion of programming languages being – like natural languages – cultural constructs. There is no way that a programming language could be simple or natural in the common sense of words. Both natural and programming languages incorporate a broad context of concepts which are part of the language and its cultural use. There’s no way to step over/around/beside/under them. There’s no way to look at a binary digit and know that it’s representing a video stream of any unknown encoding. So any language is complex, in the sense that it incorporates more than it provides in the ways of syntax and semantics. These puts some emphasis on this more that is in languages: Concepts as the context of any of its productions.

    So the key to simplicity and handiness in use might be the simplicity and strictness of the concepts. If there are simple and strict concepts even abstractions (which are contributing to the handiness; think of socket/select in Perl vs according C-code) won’t hide the complexity of the underlying structures, but will “re-package” them to conform to the general concepts, such making them simple to use to any acquainted user of the language (because the user may remain in the same concept-space).

    Maybe this is the point in all this liking and not-liking of Lisp: Lisp doesn’t have much concepts, in fact only one: The s-expression. Lisp is just a single concept. This might be sheer beauty and essence of design to one person and a dramatic lack to another. (But in fact it refers any user to the cultural tradition of the language, the Lisp community, in terms of context for high level approaches.)

    And this brings us back to the PHP again: Why PHP is a bad language – finally explained. It’s obvious that there’s some lack of concepts on the user-side, but there’s also a dramatic lack of concept in the language itself. P.e.: In one of the latest editions they changed the behavior of arrays passed as a value from copying to reference. If there’s any concept behind data, you can’t change this in a higher iteration of the language and just mumble something about everything remaining the same. That’s not concept space, it’s just getting around.
    If ther’s any relationship between the concepts of a language and any of its productions, PHP is all about getting around.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s