In my last post on the Blogging Alliance I discussed information theory, the physics behind software development, the economics of new technology, and the intrinsic obsolescence that exists as a result. Dave Gordon in his regular blog described this work as laying “the groundwork for a generalized theory of managing software development and acquisition.” Dave has a habit of inspiring further thought, and his observation has helped me focus on where my inquiries are headed.
In developing an IT management and acquisition strategy there is one more concept that we need to visit. Back in September 2011, Bent Flyvbjerg and Alexander Budzier published an article in Harvard Business Review stating that, while the average cost overrun of IT projects was 27%, this masked the fact that one in six projects experienced average cost overruns of 200% and schedule slippages of 70%. They identified these large overruns as “Black Swan” events. To run the analogy further, it can be said that this “Fat Tail” of the overrun distribution is wagging the swan.
Lest one get the impression that this is old news or study bias, there are many published results since that time that confirm these results. For example, Michael Bloch, Sven Blumberg, and Jürgen Laartz published a paper for McKinsey in October 2012 that showed that the average cost overrun was actually much worse for the 5,400 IT projects they surveyed: 45%. The percentage of projects that threatened the existence of the organization was pegged at 17%—about one in six.
Flyvbjerg and Budzier’s prescription was for companies involved in IT upgrades to undertake a “stress test” to measure their risk of experiencing a Black Swan event. Bloch, Blumberg, and Laartz meanwhile seek solutions in systems, procedures, and team building. Given that this latter group is in consulting, their conclusion is to be expected. The reality is that many problems result when authors who are not exhaustive in the literature of project failure make recommendations for how to avoid catastrophic events.
The first of these is that, for example, consulting and team building have been around for a long time—and applied as process improvement—yet projects still fail at a largely set rate. This is apart from the various schools of thought regarding Waterfall, Spiral, and Agile development. The same goes with risk handling and mitigation techniques. One can argue correctly that the existence of these techniques does not guarantee their proper application, but given the consistency of results regarding project failure, there are likely other factors that we have been missing.
Furthermore, the definition of what these authors call a Black Swan seems overly broad. One in six is more than a rare event, regardless of the fact that it is catastrophic. We need to reevaluate our terms, and then, in understanding the phenomenon, explore strategies for what measures should be taken to minimize the probability of such an event.
The Concept and Criticism
The modern use of the analogy to the Black Swan event was coined by Nassim Nicholas Taleb in his 2001 book, Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets. According to Taleb, a Black Swan event is first “an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”
The last characteristic he describes is somewhat unnecessary to his thesis and seems to be advocating in opposition to human learning from experience. After all, without reflection and investigation after a disastrous event—such as a transportation mishap—we would never learn how to minimize such events from repeating themselves. When I was on active duty in the Navy working in Naval Aviation, the adage was that the NATOPS (Naval Air Training and Operating Procedures Manual) was written with the blood of the injured and the dead.
Among the events that Taleb describes as Black Swan events are World War I, the rise of Hitler and World War II, the market crash of 1987, the fall of the Soviet Union, the spread of the Internet, and the events of 9/11. But this definition would describe just about every important event and invention.
An effective takedown of Talib’s Black Swan event thesis was published by Glyn Holton on his Risk Management Lab website in the article entitled “Fooled by Black Swans.” Holton finds that there are generally three types of important events: gathering storm events, déjà vu events, and true Black Swan events that are so improbable that there are no signs of what is to come. He finds that none of Taleb’s examples fit this revised latter definition.
Thus, Taleb’s definition is so broad as to provide no insight except to say, as attributed to Arnold Toynbee in a bit of irony, that “history is one damned thing after another.” If every important event or invention that has widespread implications is a Black Swan event, then how are they extraordinary in any sense, given the frequency in which they occur?
Anyone who has participated in research and development understands that, with almost no exceptions, new developments are built on the hard work and failures of those who came before, requiring persistence, organization, and method. Those of us who worked within ARPANET prior to 1995 can tell you that the Internet and its impact was not an accident; it was intended to be, as then-Senator Al Gore coined it at the time, “the information superhighway.”
More damning is that Taleb’s definition fails basic physics: the determinism that is baked into the universe modified by uncertainty and probability. Thus, as a critic of risk management and probabilities based on statistical methods, he ignores not only stochastic processes and methodologies for measuring them, but also misses the most valuable insight of his concept: the role of the fat tail and why it exists.
Wagging the Swan…and Baseball
It appears that the fat tail of project failure—seemingly consistent at about 17% frequency—occurs often enough to fall into what Holton describes as a “déjà vu” event, rather than some unanticipated and anticipatable outlier. As such, counter to Taleb, fat tails can be handled through risk management.
In thinking of this, perhaps a more immediate example would help given the time of the year: American baseball. Of all of the possible game-changing events or developments that have occurred, the one that stands out is the confluence of live ball era in 1920 and the ability of Babe Ruth to hit it out of the park. So dominant was Babe Ruth that he hit more home runs than entire teams. For example, in 1920 the Babe hit 54 home runs, which was more than any team in the American League and more than all but one team in the National League. By most measures—and counter to Taleb’s most prominent examples—the entry and impact of Babe Ruth is probably one of the best candidates of a Black Swan if there ever was one. But let’s test out this hypothesis.
First of all, Ruth’s dominance didn’t last long. It is true that he continued to dominate well into 1933 in hitting more home runs than many teams, yet it wasn’t long until many other players caught up with him. By 1934 most teams had at least one power hitter. Despite the excitement engendered by the home run, other teams also garnered success during this era using alternative models. As evolutionary biologist Stephen Jay Gould once wrote in explaining about the decline of the .400 hitter in baseball, hitting and pitching is an arm’s race. Eventually the tails on distributions shorten and flatten. Not only are there more power hitters today than in the 1920s, even dominant players tend not to deviate too far from the pack.
Still, the impact of 1920 on the game remains to this day, so let’s look at whether it was a phenomenon that was unpredictable, as a Black Swan should be. If we were standing at 1919 could we have predicted Babe Ruth? Well, there were solid indicators. Ruth had been known to hit home runs even in the dead ball era, which is what prompted the Red Sox to convert him from his role of a dominant pitcher to an everyday player. He led the American League in home runs in 1918 and 1919.
While it is true that the impact of the live ball was not completely known, it was hoped and expected to make the game more exciting. Additionally in 1920, baseball adopted a rule to replace the ball during the game; previously the same ball was used throughout a game, where it became tattered, dirty, and increasingly hard to see. Thus, the landscape had changed to facilitate more easily seeing and hitting the ball.
The change was so rapid that pitchers did not have time to adjust. We have seen this kind of thing in the game at other times. In 1961 for example with league expansion, the more established teams in the American League dominated the new clubs, which inflated batting averages. Some players had offensive output that they would never duplicate, such as the one put on by Norm Cash of the Detroit Tigers in that same year.
All this back-of-the-envelope analysis required was an understanding of the changing landscape—the framing assumptions for hitting a baseball—and the probability that the existing home run leader—who was exceptional even antecedent to the change—would continue to be exceptional after the change. The conclusion is that Babe Ruth as home run king was not a Black Swan event; he was not part of a change or event that was unknowable, though it was unexpected. But even being unexpected is not significant. It falls into the logical fallacy based on an argument of ignorance. Lawyers often apply such arguments, especially when pressing a weak or fabricated case.
Thus, to return to the consistent rate of failure in project management that constitutes a fat tail, what we are looking at is not a Black Swan as defined by its author, but a phenomenon that probably represents a type of organizational or institutional failure. Using Gould’s example of the disappearance of the .400 hitter in baseball—and the plethora of home run hitters today with few outliers like Babe Ruth—the existence of a consistent fat tail points to a lack of learning and adapting.
It is almost as if project management professionals are continually borne anew, unburdened by the inconvenient facts and lessons of the past, the kind that informs NATOPS. Given that these events can have an existential impact on the organization, it is essential to identify their nature.
Reducing the Probability of Failure
Reducing the probability of failure requires the ingredients of sufficient data, proper analytical techniques, and knowing how to apply amelioration once the issues are found. Going back to the McKinsey paper, the main characteristics that contributed to overruns and failure were list as the following:
- Unclear objectives and lack of business focus
- Shifting requirements
- Technical complexity
- Unrealistic schedule
- Reactive planning
- Unaligned team and lack of skills
I would add to this list, found elsewhere in the data, both the strategic importance of the effort and relative financial investment of the effort vis-à-vis the resources of the organization.
The HBR study reveals other characteristics of failed projects as well as those that parallel the McKinsey paper. These included excessive reliance on customization, blindness to sustainability issues, and lack of configuration control.
I have written previously in this blog and elsewhere about the need, first identified in a RAND study, to develop framing assumptions. What is apparent by the identified characteristics of failure from the studies is that the acquisition strategy and execution was at the center of the cause. With the background provided in my previous post regarding the physics and economics of project management, particularly as it applies to technology, there are some basic principles that need to be applied that I will summarize below.
- Have a strategic plan. Ad hoc acquisition is a recipe for failure. An overarching vision of the performance goals of a notional solution should be first on your list.
- Avoid “too big to fail.” The latest financial crisis taught us on a macro scale what can occur when one puts too many eggs in one basket (or a small set of baskets). Furthermore, basic information economics identifies that risk is baked into the process, since investment in the technology is required as part of understanding its usefulness. To handle this risk, break down efforts into more manageable segments. Smaller proofs of concept broken into goal-centered efforts is an effective means of achieving this goal.
- Define operational command and control for the eventual solution. Identify the target consumer to handle the risk associated with the psychological defense mechanics that often arise with the introduction of new technology. Ensure that proper systems analysis is conducted to understand the organization’s needs, and the stakeholders to whom hand-off will be made. During this phase ensure that there is an effective organization in place to receive the solution and nurture it in its infancy, thus tying the organization’s success to the success of the new technology.
- Define sustainment and life-cycle issues up front. Failure to apply a cradle-to-grave perspective can bake in single points of failure. Knowing that a technological generation is 18 months to two years should inform your perspective on COTS vs. internal development, acquisition costs, life-cycle sustainability, capital investment, contract type and duration, and the ability to leverage new advances through regular major upgrades.
- Understand the technological market by performing a market survey. This is opposed to going through the motions of a market survey to justify a sole source acquisition. The purpose of a complete market survey is to overcome the asymmetrical relationship in information about the product and its capabilities.
- Write a focused performance work statement defining the required framing assumptions of the required technology, the measures of effectiveness, and the measures of performance. Establish cost and schedule measures that bound the effort. Establish tripwires in the measures that identify major breaches that may require that the effort be abandoned, including any change orders that will significantly alter the framing assumptions. New framing assumptions require a new acquisition action.
- Develop realistic qualitative and quantitative measures of success in terms of payback, improvements in productivity, business operations, cost avoidance, or cost reduction.
- Have an effective risk handling plan.
- Clearly identify up front what constitutes “done” if development is involved. If version 1.0 is good enough, it is good enough. Changes that are not essential to meeting the measures of effectiveness or measures of performance can wait until the next release.
For more brilliant insights, check out Nick’s blog: Life, Project Management, and Everything