The collapse of complex computer programs

I’ve just read Clay Shirky’s new article, The Collapse of Complex Business Models, which is in turn based on Joseph Tainter’s 1990 book The Collapse of Complex Societies [,].  Shirky summarises Tainter’s analysis of why ancient cultures such as the Romans and Maya collapsed so catastrophically:

A group of people, through a combination of social organization and environmental luck, finds itself with a surplus of resources. Managing this surplus makes society more complex — agriculture rewards mathematical skill, granaries require new forms of construction, and so on.  Early on, the marginal value of this complexity is positive — each additional bit of complexity more than pays for itself in improved output — but over time, the law of diminishing returns reduces the marginal value, until it disappears completely. At this point, any additional complexity is pure cost.

Tainter’s thesis is that when society’s elite members add one layer of bureaucracy or demand one tribute too many, they end up extracting all the value from their environment.  […]  Complex societies collapse because, when some stress comes, those societies have become too inflexible to respond.  […]  In such systems, there is no way to make things a little bit simpler — the whole edifice becomes a huge, interlocking system not readily amenable to change.

When the value of complexity turns negative, a society plagued by an inability to react remains as complex as ever, right up to the moment where it becomes suddenly and dramatically simpler, which is to say right up to the moment of collapse. Collapse is simply the last remaining method of simplification.

Stop me if this seems too obvious to be worth saying, but isn’t this exactly what happens to big programs?  “When society’s elite members add one layer of bureaucracy or demand one tribute too many” sounds disturbingly like “When the framework introduces a notion of a connection factory manager builder”.

This is what happened to Multics (and UNIX was born from its ashes).  It’s what happened to Netscape (and Mozilla was born from its ashes, and died in turn of the same disease; and Firefox was born from its ashes).  And — dare I say it? — it’s the natural death that comes to over-Fowlered programs.

Please note, by the way, Fowler fans, that I am talking about over-Fowlered programs.  And, dear reader, it hardly need be said that I am not talking about the programs that you write.  Heaven forfend.  Your programs are tastefully designed, superbly executed, and engineered for a bright, shiny, future!  It’s your colleagues I’m worried about.  You’d better tell them about this important lesson!

16 responses to “The collapse of complex computer programs

  1. Wow. I’m not saying that I disagree, but that is possibly as politically incorrect one can go when it comes to programming, these days. I actually heard a priest call PoEAA for a bible once…

  2. “Complex societies collapse because, when some stress comes, those societies have become too inflexible to respond.”
    Sounds like global warming

  3. Michael Kohne

    You know, I’m not at all sure I don’t work on an over-Fowlerized program. My bits aren’t, but some of the stuff I’ve had to try to understand…

  4. Yeah, covercomplexifabulocation!!

    It’s happening. AutoCAD 2009 is a good example, I acutally have been saving sample files of things that just don’t always work right. This version of AutoCAD is most certainly feature creep. I sometimes revert back to AutoCAD 2005 to get things done with ease, fewer steps and with certainty. (as opposed to trial and error ware).

    And then there is Window downgrade options (oh my why?)…. and comments coming from someone at Microsoft about how OS need to become more supportive of multi-core CPU’s where it seems that its just MS that is lagging behind.

    Sometimes I joke that teh reason I have to select AutoCAD tools twice, before they work is because the dual CPU’s think its the other one gonna do it. Now I finding out it not a joke.

    But like the outdating of the roman numeral system used in accounting, something better comes along that is simpler and more powerful, more user empowering. Here it was the decimal system with its “nothing” can have value place holder “zero”. But only a fool would think nothing can have value. Don’t mess with the roman numeral elite accountants…. shrug… kinda sounds like some of the things I’ve heard come from programmer gods. But it all comes down to bit flipping with abstractions. We just need a better, easier and more empowering way to do it.

  5. While I’m more than willing to acknowledge that ‘over-Fowlered’ applications exist and are problematic, I’d assert that, in the aggregate of software that exist in our industry, ‘under-Fowlered’ software is much more prevalent and therefore more worthy of attention as a problem.

  6. Totally agree. We’re currently refactoring a solution which, in order to move items between three listboxes on a single screen, contains 27 different files, including abstract base classes, factories, data carriers with only three data items that are only used once, etc., etc.

    Making changes in this area is so hard to understand, error-prone and likely to introduce defects that management have finally allowed us to scrape off all the code and rewrite it. It’s now about 1/5 of the size, fits in three classes, and can be understood by anyone.

  7. Phillip Howell

    And it’s the natural death that comes to “under-Fowlered” programs, too: good design is about realizing when is the best time to break a piece off, and how.

  8. The only way truly complex software can continue to evolve to meet users’ needs is if it is adaptive. That is to say, it can alter its own behavior to suit changing environmental conditions and user requirements. There is very little (if any) software today that fits that description, but some of that philosophy has been part of my software design processes for many years, which is possibly why software I designed and implemented over 20 years ago is till in active use running major enterprise manufacturing systems – users are able to adapt the system to their changing needs without additional programming.

  9. Multics died because it was too far ahead of its time. No one really needed the level of security it provided. Of course, nowadays no one could get by for a minute without at least the level of security it provided forty years ago. The biggest strike against it was that it was a proprietary system. When Honeywell ditched it, it was gone. UNIX and Multics shared over a decade, but UNIX was given away free to universities and had the backing of DARPA for further development. It survives today because it was moved into the public domain, thanks to DARPA which funded Mach and BSD, and thanks to Linux.

    I don’t buy the societal complexity argument. Rome had a surprising ability to reinvent itself. It moved from a republic to an empire, then from a western to an eastern power. It centralized, then decentralized. Its core survived into the 15th century. Whenever someone calls something a dinosaur, I always ask if that means it’s going to dominate the earth for 50,000,000 years. I think Rome did pretty well with its 2,000 year run.

  10. Kaleberg, you make a good point regarding Rome. On dinosaurs, however, I am going to pull rank and point out that dinosaurs ruled the Earth for more like 150 million years — three times as long as the feeble duration you suggested :-)

  11. I must admit, this is a topic that has been troubling me for a while. I would say I tend to be guilty of overfowlerisation sometimes.

    I have noticed that OF (sorry, just getting too lazy to type it fullhand), and duck typing can lead to som seriously bloated designs as it is often simpler (quicker) to shoehorn a new abstraction into the system by making the new abstraction fit the old design by duck typing to a current interface. This can lead to some truely horrible sins.

  12. I think your favicon needs simplification. A sushi roll, perhaps?

  13. I don’t know if it’s even possible to change the favicon of a WordPress site. I’ve taken a stab at it, but the manuals are not clear … we’ll see whether it starts showing up in the next day or two.

  14. Wonderful thank you for your explanation

  15. Language is not logical in that it is learned before the brain is capable of logic which means the rules (logic) of a language
    are always imposed after the fact as a mechanism to effect cooperative behavior. Codified language allows for the creation of
    codified rules for other behaviors. Computer languages closest kin outside of computing would be the language of law.

    These two language domains, computer languages and the language of law are close enough in their functioning that if there was
    such a category as species in the world of abstraction, we might few them despite their seemingly different
    appearances as being of the same species in the same way that all breeds of dogs are of the same species.

    To understand what goes on in the world of computer languages one should look at the world of law. In both cases, human
    beings seek to use rationality to codify operation. Just as one can cross breed dogs, we can already see that law and computing
    are merging to create a new breed of codification that will be a combination of both. This new breed will dictate the structure of
    cooperative behavior for all societies.

    Individual programmers and computer scientists may have their ideals but these will come to nought. The forces in the world
    we seek to change are the same forces that will shape the future and result in a world that those who have yet to be born will
    imagine they can change.

    Change is never change as it is imagined. If this was not the case, we wouldn’t be disatisfied with the world as it is today.

    One thing is certain, nothing lasts. All we can do is make sure the brick we are putting in the wall doesn’t fall out and hit the
    foreman on the head and hope that when the wall does fall which it surely will, we are not crushed.

    Hubris is nothing new. We just continually forget the nature of its endgame.

  16. Pingback: Weekend miscellany | John D. Cook

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.