I’ve been reading a book I had seen over the years, but had ignored until a friend recommended it. It is The Wisdom of Crowds by James Surowiecki. I guess I didn’t pay it much heed because of an assumption that it was likely lightweight pop psychology. I was wrong.
It is an intriguing book, although more of an introductory text and so can be not quite fully satisfying in that it entices the reader’s curiosity to learn more. The ideas in the book are quite radical, but the author doesn’t emphasize that aspect. There are so many directions that could be taken and that would require many other books. This particular book, however, accomplished what it sets out to do which was ambitious enough.
A number of things are covered in the text. It’s not really about crowds, the title being unfortunate. Rather, it’s about group psychology and all that relates to that.
The author discusses the conditions under which any group will operate well or not. It could be a group of average citizens, a school board, or a grassroots organization. But it also could be a group of scientists, investors, or politicians. All humans are social animals, and so the same group psychology applies to all people and all groups. Even the paternalistic elites of Scandinavian countries, for example, are dependent on and interdependent with, built upon and inseparable from the constructive group dynamics of a culture of trust.
This would apply even to effectively ruling an authoritarian regime. That said, the conditions when fulfilled to their utmost would inevitably tend toward democratization, no matter the intent. This is true in all spheres to which these conditions could be applied. It isn’t just about politics. Far from it.
In reading reviews of the book, I realized many people didn’t understand the central point the author was communicating. He isn’t opposing crowds against experts. As I said, group psychology is a universal human reality, even for experts. Much of what the author discusses is experts themselves, when they are useful and when they are not. I found it odd that some reviewers thought the book had nothing to do about experts. Either they didn’t actually read the book or the unfortunate title biased their reading experience.
The term ‘expert’ was mentioned 92 times in The Wisdom of Crowds. Expertise is discussed in numerous chapters throughout the book. I’ll offer a few examples of it being discussed, but other examples not included below go into much more detail about examples and research.
(I’d also suggest the even more recent book, The Smart Swarm by Peter Miller. He also has useful discussion of groups and experts, both separately and as they interact, including some fascinating research. There are some other books I’m reading that may be relevant. I’ll probably write more about this topic later. I specifically have in mind a post about democracy in the city I live in, as analyzed according to Surowieki’s conditions of a wise crowd.)
* * * *
The fact that cognitive diversity matters does not mean that if you assemble a group of diverse but thoroughly uninformed people, their collective wisdom will be smarter than an expert’s. But if you can assemble a diverse group of people who possess varying degrees of knowledge and insight, you’re better off entrusting it with major decisions rather than leaving them in the hands of one or two people, no matter how smart those people are. If this is difficult to believe—in the same way that March’s assertions are hard to believe—it’s because it runs counter to our basic intuitions about intelligence and business. Suggesting that the organization with the smartest people may not be the best organization is heretical, particularly in a business world caught up in a ceaseless “war for talent” and governed by the assumption that a few superstars can make the difference between an excellent and a mediocre company. Heretical or not, it’s the truth: the value of expertise is, in many contexts, overrated.
Now, experts obviously exist. The play of a great chess player is qualitatively different from the play of a merely accomplished one. The great player sees the board differently, he processes information differently, and he recognizes meaningful patterns almost instantly. As Herbert A. Simon and W. G. Chase demonstrated in the 1970s, if you show a chess expert and an amateur a board with a chess game in progress on it, the expert will be able to re-create from memory the layout of the entire game. The amateur won’t. Yet if you show that same expert a board with chess pieces irregularly and haphazardly placed on it, he will not be able to re-create the layout. This is impressive testimony to how thoroughly chess is imprinted on the minds of successful players. But it also demonstrates how limited the scope of their expertise is. A chess expert knows about chess, and that’s it. We intuitively assume that intelligence is fungible, and that people who are excellent at one intellectual pursuit would be excellent at another. But this is not the case with experts. Instead, the fundamental truth about expertise is that it is, as Chase has said, “spectacularly narrow.”
More important, there’s no real evidence that one can become expert in something as broad as “decision making” or “policy” or “strategy.” Auto repair, piloting, skiing, perhaps even management: these are skills that yield to application, hard work, and native talent. But forecasting an uncertain future and deciding the best course of action in the face of that future are much less likely to do so. And much of what we’ve seen so far suggests that a large group of diverse individuals will come up with better and more robust forecasts and make more intelligent decisions than even the most skilled “decision maker.”
We’re all familiar with the absurd predictions that business titans have made: Harry Warner of Warner Bros. pronouncing in 1927, “Who the hell wants to hear actors talk?,” or Thomas Watson of IBM declaring in 1943, “I think there is a world market for maybe five computers.” These can be written off as amusing anomalies, since over the course of a century, some smart people are bound to say some dumb things. What can’t be written off, though, is the dismal performance record of most experts.
Between 1984 and 1999, for instance, almost 90 percent of mutual-fund managers underperformed the Wilshire 5000 Index, a relatively low bar. The numbers for bond-fund managers are similar: in the most recent five-year period, more than 95 percent of all managed bond funds underperformed the market. After a survey of expert forecasts and analyses in a wide variety of fields, Wharton professor J. Scott Armstrong wrote, “I could find no studies that showed an important advantage for expertise.” Experts, in some cases, were a little better at forecasting than laypeople (although a number of studies have concluded that nonpsychologists, for instance, are actually better at predicting people’s behavior than psychologists are), but above a low level, Armstrong concluded, “expertise and accuracy are unrelated.” James Shanteau is one of the country’s leading thinkers on the nature of expertise, and has spent a great deal of time coming up with a method for estimating just how expert someone is. Yet even he suggests that “experts’ decisions are seriously flawed.”
Shanteau recounts a series of studies that have found experts’ judgments to be neither consistent with the judgments of other experts in the field nor internally consistent. For instance, the between-expert agreement in a host of fields, including stock picking, livestock judging, and clinical psychology, is below 50 percent, meaning that experts are as likely to disagree as to agree. More disconcertingly, one study found that the internal consistency of medical pathologists’ judgments was just 0.5, meaning that a pathologist presented with the same evidence would, half the time, offer a different opinion. Experts are also surprisingly bad at what social scientists call “calibrating” their judgments. If your judgments are well calibrated, then you have a sense of how likely it is that your judgment is correct. But experts are much like normal people: they routinely overestimate the likelihood that they’re right. A survey on the question of overconfidence by economist Terrance Odean found that physicians, nurses, lawyers, engineers, entrepreneurs, and investment bankers all believed that they knew more than they did. Similarly, a recent study of foreign-exchange traders found that 70 percent of the time, the traders overestimated the accuracy of their exchange-rate predictions. In other words, it wasn’t just that they were wrong; they also didn’t have any idea how wrong they were. And that seems to be the rule among experts. The only forecasters whose judgments are routinely well calibrated are expert bridge players and weathermen. It rains on 30 percent of the days when weathermen have predicted a 30 percent chance of rain.
Armstrong, who studies expertise and forecasting, summarized the case this way: “One would expect experts to have reliable information for predicting change and to be able to utilize the information effectively. However, expertise beyond a minimal level is of little value in forecasting change.” Nor was there evidence that even if most experts were not very good at forecasting, a few titans were excellent. Instead, Armstrong wrote, “claims of accuracy by a single expert would seem to be of no practical value.” This was the origin of Armstrong’s “seer-sucker theory”: “No matter how much evidence exists that seers do not exist, suckers will pay for the existence of seers.”
Again, this doesn’t mean that well-informed, sophisticated analysts are of no use in making good decisions. (And it certainly doesn’t mean that you want crowds of amateurs trying to collectively perform surgery or fly planes.) It does mean that however well-informed and sophisticated an expert is, his advice and predictions should be pooled with those of others to get the most out of him. (The larger the group, the more reliable its judgment will be.) And it means that attempting to “chase the expert,” looking for the one man who will have the answers to an organization’s problem, is a waste of time. We know that the group’s decision will consistently be better than most of the people in the group, and that it will be better decision after decision, while the performance of human experts will vary dramatically depending on the problem they’re asked to solve. So it is unlikely that one person, over time, will do better than the group.
Now, it’s possible that a small number of genuine experts—that is, people who can consistently offer better judgments than those of a diverse, informed group—do exist. The investor Warren Buffett, who has consistently outperformed the S&P 500 Index since the 1960s, is certainly someone who comes to mind. The problem is that even if these superior beings do exist, there is no easy way to identify them. Past performance, as we are often told, is no guarantee of future results. And there are so many would-be experts out there that distinguishing between those who are lucky and those who are genuinely good is often a near-impossible task. At the very least, it’s a job that requires considerable patience: if you wanted to be sure that a successful money manager was beating the market because of his superior skill, and not because of luck or measurement error, you’d need many years, if not decades, of data. And if a group is so unintelligent that it will flounder without the right expert, it’s not clear why the group would be intelligent enough to recognize an expert when it found him.
We think that experts will, in some sense, identify themselves, announcing their presence and demonstrating their expertise by their level of confidence. But it doesn’t work that way. Strangely, experts are no more confident in their abilities than average people are, which is to say that they are overconfident like everyone else, but no more so. Similarly, there is very little correlation between experts’ self-assessment and their performance. Knowing and knowing that you know are apparently two very different skills.
If this is the case, then why do we cling so tightly to the idea that the right expert will save us? And why do we ignore the fact that simply averaging a group’s estimates will produce a very good result? Richard Larrick and Jack B. Soll suggest that the answer is that we have bad intuitions about averaging. We assume averaging means dumbing down or compromising. When people are faced with the choice of picking one expert or picking pieces of advice from a number of experts, they try to pick the best expert rather than simply average across the group. Another reason, surely, is our assumption that true intelligence resides only in individuals, so that finding the right person—the right consultant, the right CEO—will make all the difference. In a sense, the crowd is blind to its own wisdom. Finally, we seek out experts because we get, as the writer Nassim Taleb asserts, “fooled by randomness.” If there are enough people out there making predictions, a few of them are going to compile an impressive record over time. That does not mean that the record was the product of skill, nor does it mean that the record will continue into the future. Again, trying to find smart people will not lead you astray. Trying to find the smartest person will.
Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 29-34). Knopf Doubleday Publishing Group. Kindle Edition.
This might be okay if people only spoke when they had expertise in a particular matter. And in many cases, if someone’s talking a lot, it’s a good sign that they have something valuable to add. But the truth is that there is no clear correlation between talkativeness and expertise. In fact, as the military-flier studies suggest, people who imagine themselves as leaders will often overestimate their own knowledge and project an air of confidence and expertise that is unjustified. And since, as political scientists Brock Blomberg and Joseph Harrington suggest, extremists tend to be more rigid and more convinced of their own rightness than moderates, discussion tends to pull groups away from the middle. Of course, sometimes truth lies at the extreme. And if the people who spoke first and most often were consistently the people with the best information or the keenest analysis, then polarization might not be much of a problem. But it is.
Surowiecki, James (2005-08-16). The Wisdom of Crowds (p. 186). Knopf Doubleday Publishing Group. Kindle Edition.
So what would the wider distribution of real decision-making power look like? To begin with, decisions about local problems should be made, as much as possible, by people close to the problem. Friedrich Hayek, as we’ve seen, emphasized that tacit knowledge—knowledge that emerged only from experience—was crucial to the efficiency of markets. It is just as important to the efficiency of organizations. Instead of assuming that all problems need to be filtered up the hierarchy and every solution filtered back down again, companies should start with the assumption that, just as in the marketplace, people with local knowledge are often best positioned to come up with a workable and efficient solution. The virtues of specialization and local knowledge often outweigh managerial expertise in decision making.
Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 209-210). Knopf Doubleday Publishing Group. Kindle Edition.
The best CEOs, of course, recognize the limits of their own knowledge and of individual decision making. That’s why important decisions at GM, in the days when it was the most successful corporation in the world, were made by what Alfred Sloan called “group management.” And it’s why legendary business thinker Peter Drucker has said, “The smart CEOs methodically build a management team around them.” The lesson of Richard Larrick and Jack Soll’s work applies to business as much as it does to other fields: chasing the expert is a mistake. The Federal Reserve’s decisions, after all, aren’t made by Alan Greenspan. They’re made by the board as a whole. In the face of uncertainty, the collective judgment of a group of executives will trump that of even the smartest executive. Think about John Craven’s work in finding the Scorpion. A relatively small group of diversely informed individuals making guesses about the likelihood of uncertain events produced, when their judgments had been aggregated, an essentially perfect decision. What more could a company want?
Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 220-221). Knopf Doubleday Publishing Group. Kindle Edition.
In January of 2003, 343 people, carefully chosen so that they represented an almost perfect cross-section of the American population, gathered in Philadelphia for a weekend of political debate. The topic was American foreign policy, with the issues ranging from the impending conflict with Iraq to nuclear proliferation to the global AIDS epidemic. Before the weekend, the participants were polled to get a sense of their positions on the issues. They were then sent a set of briefing materials that, in a deliberately evenhanded fashion, tried to lay out relevant facts and provide some sense of the ongoing debate about the issues. Once they arrived, they were divided up into small groups led by trained moderators, and went on to spend the weekend deliberating. Along the way, they were given the chance to interrogate panels of competing experts and political figures. At the end of the weekend, the participants were polled again, to see what difference their deliberations had made.
The entire event, which bore the unwieldy name of the National Issues Convention Deliberative Poll, was the brainchild of a political scientist at the University of Texas named James Fishkin. Fishkin invented the deliberative poll out of frustration with the limitations of traditional polling data and out of a sense that Americans were not being given either the information or the opportunity to make intelligent political choices. The idea behind deliberative polls—which have now been run in hundreds of cities across the world—is that political debate should not be, and doesn’t need to be, confined to experts and policy elites. Given enough information and the chance to talk things over with peers, ordinary people are more than capable of understanding complex issues and making meaningful choices about them. In that sense, Fishkin’s project is a profoundly optimistic one, predicated on a kind of deep faith in both the virtue of informed debate and the ability of ordinary people to govern themselves.
Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 257-258). Knopf Doubleday Publishing Group. Kindle Edition.
One knee-jerk reaction to the evidence of democracy’s failings is to insist that we would be better off ruled by a technocratic elite, which could make decisions with dispassion and attention to the public interest. To some extent, of course, we already are ruled by a technocratic elite, what with our republican form of government and the importance of unelected officials—for instance, Donald Rumsfeld or Colin Powell—in political life. But one would be hard-pressed to argue that most elites are able to see past their ideological blinders and uncover the imaginary public interest. And trusting an insulated, unelected elite to make the right decisions is a foolish strategy, given all we now know about small-group dynamics, groupthink, and the failure of diversity.
In any case, the idea that the right answer to complex problems is simply “ask the experts” assumes that experts agree on the answers. But they don’t, and if they did, it’s hard to believe that the public would simply ignore their advice. Elites are just as partisan and no more devoted to the public interest than the average voter. More important, as you shrink the size of a decision-making body, you also shrink the likelihood that the final answer is right. Finally, most political decisions are not simply decisions about how to do something. They are decisions about what to do, decisions that involve values, trade-offs, and choices about what kind of society people should live in. There is no reason to think that experts are better at making those decisions than the average voter. Thomas Jefferson, for one, thought it likely that they might be worse. “State a moral case to a ploughman and a professor,” he wrote. “The former will decide it as well and often better than the latter because he has not been led astray by artificial rules.”
It’s also the case that democracy allows for the persistent injection into the system of what I called earlier “local knowledge.” Politics is ultimately about the impact of government on the everyday lives of citizens. It seems strange, then, to think that the way to do politics well is to distance yourself as much as possible from citizens’ everyday lives. In the same way that a healthy market needs the constant flow of localized information that it gets from prices, a healthy democracy needs the constant flow of information it gets from people’s votes. That is information that experts cannot get because it is not part of the world they live in. And that keeps the system more diverse than it would otherwise be. As Richard Posner puts it: “Experts constitute a distinct class in society, with values and perspectives that differ systematically from those of ‘ordinary’ people. Without supposing that the man in the street has any penetrating insights denied the expert, or is immune from demagoguery, we may nevertheless think it reassuring that political power is shared between experts and nonexperts rather than being a monopoly of the former.”
Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 265-266). Knopf Doubleday Publishing Group. Kindle Edition.
Filed under: Psychology | Tagged: crowds, experts, group intelligence, group psychology, groups, intelligence, James Surowiecki, The Wisdom of Crowds | 35 Comments »