Last rites for Software Engineering?

Last rites for Software Engineering? Finished Ed Yourdon's Death March, a book about 'impossible mission' software projects.

Yourdon is one of the founding fathers of Software Engineering, along with Barry Boehm and Tom DeMarco. Their ideas had a profound effect on me and many other programmers when they appeared in the 1980s. Alarmed by projects that had slid relentlessly over time and budget, I was enchanted by the prospect of formal methods bringing certainty to software development.

Death March was published in 1996, the same year as Kent Beck's Extreme Programming Explained: Embrace Change. It's interesting to compare the two. The sloppy typography of Yourdon's book is the perfect complement to its content. Yourdon's long writing career seems to have enabled him to knock out a new book without very much trouble. He draws heavily on his correspondents, as Darwin did, which is reassuring: we have had a great deal of theory and should distrust ideas not grounded in a natural history of software development. We know how close to his sources Yourdon stays because much of his material is repeated word for word in extensive chapter notes, as if the author doubted his ability to paraphrase his correspondents.

In contrast, Beck's book is a model of lucidity and vigour, and follows the author's recommended practice: he wrote the simplest book that could possibly work.

Yourdon's book should be read though, for it is a testament to the failure of formal methods in software development, by one of SE's strongest advocates.

Yourdon says death-march projects have become the rule, not the exception. So what he has to say about them is what he now thinks about normal software development. This alone is extraordinarily interesting. In his view, normal software development is in a permanent state of emergency.

How does this emergency relate to Software Engineering's formal methods? What Yourdon does not say is at least as interesting as what he does. His book is largely about the dialogue between the project manager and his 'customers'. He says that the predictions of formal methods will be of little value to the project manager, because the customers reject them. Someone is out of touch with reality here. If SE's methods predict the right answers, then the customers are a pack of moon-crazed wolves.

And that's how they appear in Death March, which is mostly about how to negotiate with them, and when to resign. It's a desperate tale.

But what if the customer is right? Software Engineering promised to reduce uncertainty in software development. It was devised to make workable the dialogue between developer and customer. The breakdown of this dialogue that Yourdon testifies to is the breakdown of Software Engineering itself. A quarter century of SE has instilled in IT's customers a wary distrust of its predictions. The customers have every reason to howl.

SE did reduce uncertainty in software development — some, but at huge cost in resources and flexibility. The customers might be unreasonable, but they're not stupid. They can see when lots of work is being done and not much delivered. SE advocates might insist theirs is the only way to produce software, but customers are desperate for a better way to have software developed.

Death March is a tombstone for Software Engineering. SE was devised to make software development predictable and reliable. It failed, and the 'normal' state of software development projects is a doomed death march. This, mind, not an attack from one SE's critics, but from one of its founders.

To steal from Tom Wolfe: Software Engineering is dead; call in the cops.