How to distinguish managerial excellence from luck? It's a lot harder than you'd think, new research shows
October 1, 2009
For more information, contact: Benjamin Haimowitz, HHaimowitz@aol.com
"I'd rather be lucky than good."
-- Lefty Gomez, New York Yankee pitcher and Hall of Famer
Nobody has ever taken the late Lefty Gomez for a management guru. But at a time when the corporate world is groping for lessons in the midst of an economic debacle, a new skepticism is emerging about the way both scholars and popular authors distinguish true managerial excellence from just plain luck, whether in such popular blockbusters as In Search of Excellence or Good to Great or in strategy courses in the world's business schools.
Reports presented at the recent annual meeting of the Academy of Management or published in the journal Academy of Management Perspectives find that so-called "success studies" suffer in general from a lack of scientific rigor and in particular from a failure to take sufficiently into account the considerable role luck can play in engendering superior corporate results.
"Many of us who teach strategy cases have sometimes thought: 'I wonder if the firm we're talking about today...is truly skilled or just lucky?' write the authors of a paper presented at AOM's annual meeting in August. The paper was presented at the Chicago meeting by Andrew D. Henderson of the University of Texas at Austin, who wrote it with Michael E. Raynor and Mumtaz Ahmed of Deloitte Consulting LLP.
"The central issue," the authors note, "is not the number of sustained superior performers that one observes but rather the ratio, among top performers, of the exceptionally skilled to exceptionally lucky...A number of studies have assessed the degree to which firm performance, both good and bad, persists across time. But none, to our knowledge, has compared the observed number of sustained superior performers to the number that we would expect by chance. Doing so is vital because research...suggests that luck strongly affects firm performance in a world in which boundedly rational decision makers face strategic choices that involve high levels of causal ambiguity, complexity, and uncertainty."
By this standard, the three researchers show, virtually all of the most popular bestsellers on corporate success come up short. In a monograph published by Deloitte, they analyze 13 bestselling books on what it takes for companies to achieve sustained superlative performance, including such blockbusters as In Search of Excellence, Good to Great, and Built to Last, and show that in 12 of them a large majority of the firms identified as consistent winners fail to measure up convincingly. The findings parallel those of two recent in-depth studies of Good to Great in Academy of Management Perspectives, which find, in the words of one of the reports, that "the 11 Good to Great firms do not differ significantly from the average company on the S&P 500...[They] may be good, they aren't great."
Or, in the words of the presentation at this year's Academy of Management meeting, "It is easy to be fooled by randomness...The associated benchmarks are more stringent than many might imagine."
How do Henderson, Raynor, and Ahmed decide whether companies truly measure up? It requires what they call "a pinch of math."
Their basic research strategy was to fashion a kind of conceptual pliers out of two massive sets of data on corporate performance, one based purely on chance and the other derived from real-world results. Using a database of 41 years' worth of corporate-level performance, spanning 1965 through 2005, they analyzed the return on assets of more than 20,000 companies, constituting approximately 85% of companies traded during that time on U.S. equity markets. In doing this, they controlled for an array of factors that can affect performance quite apart from management -- for example, a firm's industry or its size or the market share it inherits.
Each firm was tracked in terms of its performance for every year during the 41-year period (most companies were not extant as independent firms for that entire time) and was assigned to one of 10 deciles, ranging from the bottom 10 percent to the top 10 percent, for each year. Thus was it found, for example, that among the 856 firms for which there was 11 years' worth of data, only 45 ranked in the top decile five or more years.
What did that mean? Did five years in the top rank demonstrate managerial excellence or was it just a matter of luck? To determine that, Henderson, Raynor and Ahmed employed the other arm of their research pliers, the one based on pure chance. They ran a thousand computer simulations for each firm in their database, giving each company the life span it actually had and starting in the same decile in which it actually started out. Year-to-year changes in performance were randomly drawn from a distribution of probabilities that took into account the inherent "stickiness" of rankings -- that is, the greater likelihood in the real world for firms to remain in the same decile than to move to another.
On the basis of this analysis, one of many findings was that, among firms with 11 years' worth of data, 4.3% would rank in the top decile five or more years by pure luck. Thus, among the 856 companies with 11-year records, 37 would be expected to achieve that number of top rankings merely by chance, meaning that any of the 45 firms in that select group would present an 82 percent likelihood of being a false positive.
What would it take to get the number of false positives down to an acceptably low proportion, say about 10 percent, not only for that group of firms but in general? That would require that a firm be in the top 0.2 percent -- in other words, that it rank above 99.8% of all other firms over the same span of years. In all, 150 firms achieved that rank out of the more than 20,000 companies in the database.
In their Academy of Management presentation, the authors offer a tentative essay at a performance ranking of their own, and come up with an intriguing mix of the celebrated and the obscure. Ranking first is Microsoft, achieving the top performance decile in 20 of 21 years; then comes Landauer Inc, a company specializing in measuring radiation exposure, which was in the top decile in 18 years out of 19; followed by Tambrands, Inc., the manufacturer of Tampax tampons, which achieved the top decile in 29 of 31 years before being acquired by Procter & Gamble in 1997. Rounding out the top ten are Premier Industrial Corp, an electronics distribution firm acquired in 1996; Campbell Red Lakes Mines, a Canadian firm merged in 1987; Total System Services Inc, an electronic payment service headquartered in Columbus, Georgia; Adobe System Inc, the software company; Heartland Express Inc, a commercial transport firm; Apco Argentina Inc, a company in oil and gas exploration; and Diagnostic Products Corp., part of Siemens since 2004.
Even at this exalted level, where the chance of false positives is at its lowest, luck needs to be reckoned with. Says Prof. Henderson: "I think there is a very high likelihood that Microsoft is among the lucky rather than the truly great. Campbell Red Lake Mines might be another example. If you dig a hole and hit gold, you can look great for a long time despite just being lucky. Among our highly ranked firms, Bandag Inc. and Bob Evans Farms are probably better examples of sustained excellence than those two firms."
The paper, entitled "How Long Must a Firm be Great to Rule Out Luck? Benchmarking Sustained Superior Performance," was among several thousand research reports at the Academy of Management annual meeting, held in Chicago from August 9 to 11th. Founded in 1936, the Academy is the largest organization in the world devoted to management research and teaching. It has about 19,000 members in 102 countries, including about 11,000 in the United States. In addition to Academy of Management Perspectives, it publishes three other scholarly journals, the Academy of Management Journal, Academy of Management Review, and Academy of Management Learning and Education.
- Media Coverage:
- The Wall Street Journal. When Bad Luck Is a Crime. (Wednesday, October 21, 2009).