Eliminating Dietary Dissent

There was a hit piece in the Daily Mail that targeted three experts in the field, all doctors who are involved in research. It’s not exactly a respectable publication, but it does have a large mainstream readership and so its influence is immense, at least within the UK (even as an American, I occasionally come across Daily Mail articles). Here is the response by Dr. Malcolm Kendrick’s (Scottish). And by Dr. Zoe Harcombe’s (Welsh). Both responses were sent to the Daily Mail. The hit piece was published in timing with her planned speech before the UK Parliament, an attempt to discredit her and to distract from debate of the evidence. The third target of attack, Dr Aseem Malhotra (British) who also spoke to the UK Parliament (and the European Parliament as well), chose not to respond as he concluded it would be futile and it appears he was correct, in that the Daily Mail chose not to alter its message in the least because of what Kendrick and Harcombe wrote.

This is the same basic battle that I’ve mentioned previously, the conflict between two prestigious British medical journals, the BMJ and the Lancet. It has developed into full ideological warfare. But those defending the status quo are being forced to acknowledge their detractors, which is an improvement over silencing.

In the failed attacks on Robert Atkins (American), Annika Dahlqvist (Swedish), Gary Taubes (American), Tim Noakes (South African), Gary Fettke (Australian), Peter C.Gøtzsche (Danish), Maryanne Demasi (Australia), and Shawn Baker (American) over similar disputes, and among others who have felt the politically correct wrath of conventional and corporatist authority (I could mention Uffe Ravnskov, Nina Teicholz, etc; there is Malcolm Kendrick, Zoe Harcombe, and Aseem Malhotra as well; and, as I’ve discussed before, Adelle Davis, Carlton Fredericks, Gayelord Hauser, and Herman Taller), we see how the powers that be use mainstream institutions (private and public) as weapons. But that isn’t to ignore that there are also some successful examples of silencing such as John Yudkins (British), Jen Elliott (Australian), Maryanne Demasi (Australian), etc. In The Big Fat Lie that is soon to be a documentary, Nina Teicholz discusses other major figures in the healthcare field and research community that were effectively silenced in being discredited and excluded, in that they couldn’t get funding and were no longer invited to speak at scientific conferences; and Gary Taubes earlier discussed the same territory in Good Calories, Bad Calories; but if you prefer a detailed personal account of how a systematic attack is done, read Tim Noakes’ Lore of Nutrition. Anyways, failed or successful, these attacks are cautionary tales in setting examples of what the authorities can and will do to you if you step out of line. It creates a stultifying atmosphere and a sense of wariness among researchers, healthcare professionals, science writers, journalists, and public intellectuals — hence encouraging people to censor themselves.

In a similar area of dispute, there is another ongoing fight where an individual, Diana Rodgers (American), like the others has been targeted. Attacking individuals in trying to destroy their careers or authority seems to be the standard tactic. Fortunately, social media sheds light on this dark practice and brings out the support for these doctors, dieticians, researchers, etc who in the past would’ve felt isolated. It’s one of the positives of the internet.

Yet again, here is an example of conventional idiocy in its attempt to use a mainstream platform to spread disinfo and enforce conformity. Consider Newsweek that, like the Daily Mail, is a low quality but widely read mainstream publication. They decided to do a piece critical of the carnivore diet. And the writer they assigned to do it normally writes about video games and pop culture. Unsurprisingly, written by someone with no knowledge or expertise, the article was predictably misinformed. Every single comment in the comments section was critical (nearly the same in the comments of Nina Teicholz’s tweet), including comments by doctors and other experts. It’s less to do with a specific diet. This same kind of backlash is seen toward every variety of low-carb diet, whether plant-based paleo or plant-free carnivore, whether high-(healthy)fat or moderate, whether ketogenic or not. The reason is that there is no way to have a low-carb diet while maintaining large profits for the present model of the big biz food system of heavily-subsidized, chemical-drenched, and genetically-modified surplus grains as used to produce shelf-stable processed foods.

And it is far from limited to trashy popular media, as the same kinds of dismissive articles are found in higher quality publications like the Guardian, along with major medical organizations such as Harvard and the Mayo Clinic (although there is increasing positive press as the scientific research and popular support becomes overwhelming). Harvard, for example, is closely tied to the EAT-Lancet agenda (by way of Walter Willett, the ideological heir of Ancel Keys and, as I recall, involved in the leak of Robert Atkins’ medical records in a failed attempt to smear his reputation after his death) and the corporations behind it (Harvard, like other universities, have become heavily funded by corporations, as government funding has dried up; the Koch brothers have been key figures in the corporate takeover of universities with influence over hiring and firing of faculty and, by the way, the Koch brothers are heavily invested in big ag which is to say they are financially connected to the government-subsidized “green revolution” and the processed food industry).

Yet a growing movement is emerging from below, not only seen in comments sections and social media, but also in forming new organizations to demand accountability; for example, Gary Taubes’ Nutrition Science Initiative (NuSI) that is promoting much needed research. In reaction, the self-proclaimed authority figures in the mainstream are trying to enforce dietary conformity. I suspect the fact that so many people are questioning, doubting, and experimenting is precisely the reason elites all of a sudden are pushing even harder for basically the old views they’ve been pushing for decades. They sense the respect for their position is slipping and are in damage control mode. This isn’t only about statins, LCHF diet, or whatever else. It indicates a deeper shift going on (with low-carb diets on the rise) and those who are resisting it because of vested interests. What’s at stake is a paradigm change and the consequences of the status quo remaining in place are dire for public health.

* * *

On a related note, there is also a dark side to how the internet has been wielded as propaganda network. We know how effectively social media can be used to spread disinfo — yes, by whackos and controlled opposition like Alex Jones but even more powerfully by governments and corporations, think tanks and lobbyist groups, astroturf operations and paid trolls.

Wikipedia and Rational Wikipedia seem to have been taken over by defenders of the establishment, a sad fate for both of them. Many Wikipedia pages related to low-carb diets and alternative health (including tame criticism of statins by world reknown scientists) have been heavily slanted or deleted on Wikipedia. This agenda of censorship goes straight to the top — Jimmy Wales, the founder of Wikipedia, has called all critics of conventional medicine “lunatic charlatans” and demands that they be eliminated from Wikipedia, as if they never existed. This is a major change from earlier Wikipedia policy that promoted articles showing multiple viewpoints, but the reason for the change is that Wikipedia is being pressured to be an authoritative source as with traditional encyclopedias since Wikipedia is now used by services like Apple’s Siri.

Rational Wikipedia labels as “statin denier” anyone who is skeptical of highly profitable and corporate-promoted overprescription of statins, including critics who are practicing doctors and peer-reviewed researchers (the same false accusation is made by other pseudo-skeptical organizations such as CSICOP) — according to this logic, one of the most well-respected medical journals in the world, the BMJ, are “statin denialists” for being skeptical of the overuse of statins that the scientific research shows can cause much harm. Meanwhile, Rational Wikipedia rationalizes away this concerted effort of propaganda, probably because it’s the same people behind both operations, by way of hard-to-track sock puppets (I know from personal experience and research how deep the hole can go in trying to track down the identity of a disinfo agent, be they paid troll or merely the mentally disturbed). Pseudo-skepticism has come to rule the internet —- some of it as mentally disturbed true-believers but it also includes organizations that are astroturf. And so be skeptical most of all of anyone who poses as a skeptic.

Fortunately, alternatives are emerging such as Infogalactic as a non-censored, balanced, and independent version of Wikipedia. Unlike Wikipedia, an editor or group of editors can’t monopolize or delete a page simply because they ideologically disagree with it. And unlike Rational Wikpedia, there is no narrow institutional ideology informing what is allowable.

This is partly why it is so hard for the average person to find good info. Not only are we being lied to by big gov and big biz by way of big media for the same powerful interests are co-opting the new media as well. The purge and demonetizing of alternative voices, left and right, on YouTube was a great example of this. A similar purge has happened on Pinterest, generally censoring alternative health views and specifically targeting low-carb diets using centralized propaganda as the justification: “Keto doesn’t conform to CDC dietary guidelines” — despite the fact that ketogenic diets are among the most widely and longest researched with massive amount of data supporting numerous areas of benefit: longevity, cancer, epilepsy, autism, insulin resistance, autoimmune conditions, Alzheimer’s, etc. If the CDC is anti-science when particular science opposes highly profitable corporate interests, that is a major problem — but it shouldn’t be surprising that Pinterest, a highly profitable corporation (likely owned by a parent company that also owns other companies involved in agriculture, food production, pharmaceuticals, etc), defends the interests of big biz in collusion with big gov.

There is a struggle by the powerful to regain control of all potential avenues of propaganda and perception management. In terms of public debate, it’s always a matter of the perception of who wins. This is why propagandists, as with advertisers, have long understood that repetition of claims or ideas will make them so familiar as to feel true — what is called cognitive ease. That is why it is so important to silence opponents and make them invisible. Repetition requires total control, as the other side will also attempt to repeat their views. But it doesn’t matter how often alternative views are repeated if they are effectively erased from public view and from public forums. Look widely for info and scrutinize everything carefully. Find the few experts that are genuine honest actors and follow what they put out.

* * *

The Dark Side of Wikipedia
from Full Measure

Astroturfing Revealed–the Ruining of Wikipedia
by Angela A. Stanton

Wikipedia Declares War on Low Carb Diet Experts
by Aarn

Jimmy Wales Admits Free Access To Health Knowledge Has Strict Limits On Wikipedia
by Paul Anthony Taylor

Wikipedia: Cementing The Power Of The Status Quo
from Dr. Rath Health Foundation

Let me tell you a little bit about how the @Wikipedia farce works from someone who spent a lot of time battling there as an editor.
by Mike Carrato

Wikipedia Captured by Skeptics
from Skeptics about Skeptics

The Philip Cross Affair
by Craig Murray

Wikipedia censorship of natural, non-drug therapies
from Alliance for Natural Health

Kendrick, Wikipedia and ‘Dark Forces’ Waging War on Science
by Marika Sboros

Dr Malcolm Kendrick – deletion from Wikipedia
by Malcolm Kendrick

Wikipedia a parable for our times
by Malcolm Kendrick

Who Deserves to be a Wikipedia Article?: The Deletion of Dr. Malcolm Kendrick
by Anthony Pearson

‘Fat Head’ Targeted For Deletion By The Weenie At Wikipedia
by Tom Naughton

Follow-Up On The Weenie Wiki Editor
by Tom Naughton

BEWARE: New Plan to Censor Health Websites
by Joseph Mercola

Reddit discussions:
Doctors who are against statin are being removed from Wikipedia
Fat Head movie Wikipedia article up for deletion next !
Malcolm Kendrick and other low-carb and keto advocates are being attacked at Rationalwiki as pseudoscientists

Failure of Public Intellectuals

Over at Teeming Brain, Matt Cardin mentioned a book. It’s The Ideas Industry by Daniel Drezner. There is an initial response I gave in a comment to Cardin. I turned that comment into a post I made earlier, Public Intellectuals As Thought Leaders. And I added to that with another post, Thoughts on Inequality and the Elite. In a second comment to Cardin, I sought to put it into further context:

This is an important topic and this book being far from the only example of it being discussed. There is also The Death of Expertise by Thomas Nichols, another book I haven’t read. There are many other similar books as well, such as Rigor Mortis by Richard Harris where is discussed the damaging failure of expertise in a particular field.

As I thought more about it, I realized this should be put into a larger context. The whole issue of “fake news” has received focus as of late. But who determines what is fake?

It was quite shocking see what was in some of the leaked emails, that those in the mainstream media were working close with party insiders, even to the point of secretly sharing debate questions prior to the debate and sending articles to them for editing before publishing. Yet this same corporate media wants to judge alternative media, one of the last bastions of honest discussion of important issues. There is a fight going on right now between old media and new media, such as what is going on with YouTube and AdSense, a fight that could shut down the growing voices outside of the establishment.

It is all very concerning.

There are other books that I could point to. Some of them are listed below, along with a few reviews and articles.

I’m not a big fan of blaming the public in a society that gives so little voice and power to the public, such as calling the public stupid. It would be a fair criticism if this was a functioning democracy, but the fact of the matter is that this is a banana republic. The real power is some combination of neoconservatism, neoliberalism, military-industrial complex, deep state, corporatism, inverted totalitarianism, plutocracy, kleptocracy, oligrachy, and I’m sure others could add a few to the list. I’ve often prefer the lens of corporatism with its long history in progressivism, fascism, colonialism, and earlier ideological systems. Corporations have become the dominant institution of our age.

Here is another angle. The pseudo-meritocracy, despite the liberal and progressive rhetoric, is actually a rigidly stratified system of concentrated wealth and power that tends toward authoritarian expressions of technocracy and scientific management (see an earlier discussion). Those with power and privilege love to wield the authority of expertise. But who determines who gets to be called and perceived as an expert in the corporate media, corporatist political system, and increasingly corporate-funded academia and scientific research?

The simple fact is that public trust has been lost. In many cases, it’s uncertain that it was ever deserved. Consider the authority of our criminal system, as assessed by the National Academy of Sciences:

Rigorous and mandatory certification programs for forensic scientists are currently lacking, the report says, as are strong standards and protocols for analyzing and reporting on evidence. And there is a dearth of peer-reviewed, published studies establishing the scientific bases and reliability of many forensic methods. Moreover, many forensic science labs are underfunded, understaffed, and have no effective oversight.

Forensic evidence is often offered in criminal prosecutions and civil litigation to support conclusions about individualization — in other words, to “match” a piece of evidence to a particular person, weapon, or other source. But with the exception of nuclear DNA analysis, the report says, no forensic method has been rigorously shown able to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.

Now consider an estimated 2-5% of prisoners and 4% sentenced to die are innocent of all criminal charges. It was a public legal system built on professional expertise that led to these sad results. It’s sadder still when one looks at the racial biases. And the very public intellectuals getting promoted the most are often those, like Charles Murray, who preach a racial narrative and so offer justifications for prejudice.

We can’t simply turn to public intellectuals in the hope they’ll sort it all out. They are often part of the problem. And it isn’t public intellectuals who are most harmed in the process. When even public debate among public intellectuals fails to lead to public good, where does that leave the general public that has little voice at all, specifically those among us who suffer the worst consequences?

The failure isn’t intellectuals as a broad category. It’s a minority of intellectuals who become members of the affluent and influential intelligentsia, often working for special interest organizations, lobbyist groups, and think tanks. This is what being a public intellectual has come to mean, at least as it gets presented in corporate media and corporatist politics. What we need is more public intellectuals from more sectors and levels of society, in order to have genuine public debate.

A technocratic ruling elite is not going to save us.

* * * *

Flawed Scientific Research

Twilight of the Elites:
America After Meritocracy
by Chris Hayes

Failed:
What the “Experts” Got Wrong about the Global Economy
by Mark Weisbrot

Experts and Epistemic Monopolies: 17
by Roger Koppl, Steve Horwitz, & Laurent Dobuzinskis

Escape from Democracy:
The Role of Experts and the Public in Economic Policy
by David M. Levy &Sandra J. Peart

Scientism and Technocracy in the Twentieth Century:
The Legacy of Scientific Management

by Richard G. Olson

Beyond Technocracy:
Science, Politics and Citizens
by Massimiano Bucchi

The Revolt of the Public and the Crisis of Authority in the New Millennium
by Martin Gurri

Type of expertise and their goal matters
by d. doyle

The problem today is not necessarily a lack of experts as it is how to determine what is relevant and what the goal is behind any expert’s pronouncement.

The Limits of Expertise
A defense of experts exhibits the very problems it complains about.

by Noah Berlatsky

Believe the experts! Experts are not perfect, but they are more likely than non-experts to be right. Experts know what they do not know, and are therefore more cautious and better able to self-correct. Sometimes, in small ways, non-experts may outperform experts. But in general, America and the world need more respect for expertise.

That is the thesis of Tom Nichols’ The Death of Expertise: The Campaign Against Established Knowledge and Why It Matters. It is also, as it turns out, a critique of the book itself. Nichols, a professor at the U.S. Naval War College, is an expert on Russia and national security; he is not, however, an expert on expertise.* His hand wringing about kids today is not grounded in a scholarly background in education policy or the history of student activism. He is a generalist dilettante writing a polemic against generalist dilettantes. As such, the best support for his argument is his own failure to prove it.

There are two central flaws in The Death of Expertise. The first is temporal. As the title implies, the book is written as though there were once a golden age when expertise was widely valued—and when the democratic polity was well-informed and took its duty to understand foreign and domestic affairs seriously. “The foundational knowledge of the average American is now so low that it has crashed through the floor of ‘uninformed,’ passed ‘misinformed’ on the way down, and finally is now plummeting to ‘aggressively wrong,'” Nichols declares. His proof for this statement is that “within my living memory I’ve never seen anything like it.”

As Nichols would ordinarily be the first to point out, the vague common-sense intuitions and memories of non-experts are not a good foundation for a sweeping theory of social change. Nichols admits that Americans are not actually any more ignorant than they were 50 years ago. But he quickly pivots to insist that “holding the line [of ignorance] isn’t good enough” and then spends the rest of the book writing as if he didn’t know that Americans are not getting more ignorant. […]

The balance between trusting experts and challenging conventional wisdom is always difficult. How do you create discussions online where folks who have been traditionally marginalized are welcome without empowering bad actors determined to harass them or spread disinformation? How can political parties encourage participation and democratic engagement without opening themselves up to opportunists and quacks? Those are questions worth asking, but Nichols, alas, is not the writer to answer them. Someone with more expertise is needed. Or, possibly, with less.

Comment to above article
by VG Zaytsev

“Believe the experts! Experts are not perfect, but they are more likely than non-experts to be right. Experts know what they do not know, and are therefore more cautious and better able to self-correct.”

That is 100% wrong in two ways. First as the breadth of knowledge continually increases, the scope of expertise shrinks. Attaining and maintaining expertise requires an ever greater focus on an ever narrower field, which necessarily means less knowledge in other areas, getting progressive lower as the distance form their narrow specialty increases. Which is fine in itself, but it is not how humans perceive the world and their social groups. Instead we believe that wisdom, cast as expertise, is wide – to universal. So that an “expert”‘s opinion is valued on a wide range of issues, most of which he has less information and experience dealing with than a generalist. Experts themselves are prone to this flaw.

Secondly the trust in experts and the narrow scope of actual expertise creates,the opportunity for faux experts to claim a level of authority and deference that they have no legitimate claim to. We see this repeatedly with “experts” put forward by the media to push a pre determined agenda.

‘The Death of Expertise’
by Scott McLemee

A survey of 7,000 freshmen at colleges and universities around the country found just 6 percent of them able to name the 13 colonies that founded the United States. Many students thought the first president was Abraham Lincoln, also known for “emaciating the slaves.” Par for the course these days, right?

It happens that the study in question was reported in The New York Times in 1943. The paper conducted the survey again during the Bicentennial, using more up-to-date methods, and found no improvement. “Two‐thirds [of students] do not have the foggiest notion of Jacksonian democracy,” one history professor told the Times in 1976. “Less than half even know that Woodrow Wilson was president during World War I.”

Reading the remark now, it’s shocking that he was shocked. After 40 years, our skins are thicker. (They have to be: asking the current resident of the White House about Jacksonian democracy would surely be taken as an invitation to reminisce about his “good friend,” Michael.)

The problem with narratives of decline is that they almost always imply, if not a golden age, then at least that things were once much better than they are now. The hard truth in this case is that they weren’t. On the average, the greatest generation didn’t know any more about why The Federalist Papers were written, much less what they said, than millennials do now. The important difference is that today students can reach into their pockets and, after some quick thumb typing and a minute or two of reading, know at least something on the topic.

Beware: the experts are usually poor forecasters
by Allister Heath

To say that experts often get it wrong is an understatement.

Philip Tetlock, a brilliant US academic who has studied this phenomenon in detail, once concluded that the average “expert” was in fact “roughly as accurate as a dart-throwing chimpanzee”. Consumers of expert advice should thus always heed the old adage of caveat emptor, or “let the buyer beware”.

The record of private and public sector forecasters is all too often abysmal, and in some cases almost a counter-indicator. The world craves certainty, even though no such thing can possibly exist. Pollsters thought that Labour would win the last election, have miscalled many others around the world and didn’t originally foresee the rise of Donald Trump. Most economists and large companies supported the UK’s membership of the euro, for example, which would have been a complete disaster. With a few heroic exceptions, hardly any economists saw the financial crisis and Great Recession coming, and of the very few who did spot that something was amiss hardly any worked out how the collapse would unfold.

So much for the big calls; the smaller ones tend to be equally wrong. We tend to see a strong bias towards over-optimism at the top of a boom and towards excessive pessimism at the trough of a recession. GDP numbers are always at least a little incorrect, and nobody predicted the employment bonanza of the past few years or the disappointing productivity performance. Even the Bank of England cannot correctly predict its own actions. As to most active fund managers, again with a number of brilliant exceptions, they aren’t worth the money: they cannot consistently deliver above-market returns after costs, even though that is their job. It gets worse: even oil companies cannot accurately work out what’s going to happen to the price of oil. […]

The problem is that it is impossible to know from the outset which so-called expert is actually a superforecaster and who will turn out to be no better than a random prediction machine. We therefore need to be very careful when listening to the expert consensus.

What if Elite Experts are Wrong About What They Supposedly are Experts About?
by Peter Boettke

Ever since the Wilsonian period, the progressive agenda has come with trained experts who by design immune from direct democratic pressures.  This is most evident in the Independent Regulatory Agencies — CPSC, EPA, FTC, FAA, FCC, FERC, Fed Reserve System, FDA, ICC, NLRB, NRC, OSHA, SEC — but it is an embedded attitude in our universities, our legal system, our politics, our media.  Experts are expected to lead the way based on their expertise in the policy sciences. […]

The problem with experts isn’t that individuals can have superior judgement to others, or that one can earn authority through judicious study and successful action.  The problem is an institutional one, and institutional problems demand institutional solutions.  In the case of the Levy/Peart and Koppl stories, the problem results from monopoly expertise that produce systemic incentives and social epistemology which is distortionary from the perspective of correct policy response.  […]

In fact, this focus on institutions of governance, and the fragility or robustness of these institutions, has been a focus […] Our knavery comes in the form of arrogance and opportunism, and if we construct institutions of governance that fail to check our knavery, and instead unleashes experts immune from democratic pressures, we get expert failure.

Tremendous power and authority has been entrusted in these experts.  Yet, there are serious issues that potentially delegitimize large segments of the establishment in: education from primary to secondary to higher, media from traditional print to radio, TV and even the echo-chamber of social media, public services from police to infrastructure to public pensions, and government from local to state to federal.  One way to “read” the election results is that this was an indictment of the establishment of experts.

Comment to above article
by arun

I think experts who serve an ‘elite’ aren’t going to be objective because an elite, by definition, believes that it’s values and preferences are ‘hegemonic’ in the Gramscian sense- i.e. they are prescriptive because of some obvious virtue which everybody recognizes as attaching itself to the ‘elite’.

In other words, the elite has an incentive to employ an expert who predicts that which is in their narrow interest and tries to pass it off as a ‘Muth Rational’ solution.

If Elites are insecure or subject to rent-contestation, sure, they may consult ‘expert cognition’ mavens so as to hedge their bets but they still have an interest in supporting official ‘experts’ who either predict what they want them to predict or who make a policy space multidimensional in a manner that gives the Elite ‘agenda control’and thus the ability to rig the outcome in their favor.

Comment to above article
by BenK

This comes back to the local knowledge problem; that experts may indeed have general knowledge about class of problem abstracted from its setting, but that only works for problems that are truly able to be abstracted. As a result, effective experts usually need to embed, or ‘condescend’ to understand local conditions when addressing a problem in the specific. However, when community problems are fundamentally about the ‘community,’ the experts are likely to favor being ‘objective’ and ‘distant’ rather than ‘involved’ and perhaps compromised. As a result, there is a conundrum. They cannot sit on high in judgement on the community and still understand it; but if they become involved, the problem will not appear the same. It’s a kind of relativity, particularly well known in families.

The answer is not to have contests among the experts to see who is more frequently right. This favors cherry picking and all sorts of bad strategies. The answer is to have experts as local as feasible; and keep them local, not giving them broad authorities. They can learn from each other but not subsume each other. There are costs to this approach, but it will be more robust than the current brittle strategy.

Group Psychology For All, Experts And Non-Experts Alike

I’ve been reading a book I had seen over the years, but had ignored until a friend recommended it. It is The Wisdom of Crowds by James Surowiecki. I guess I didn’t pay it much heed because of an assumption that it was likely lightweight pop psychology. I was wrong.

It is an intriguing book, although more of an introductory text and so can be not quite fully satisfying in that it entices the reader’s curiosity to learn more. The ideas in the book are quite radical, but the author doesn’t emphasize that aspect. There are so many directions that could be taken and that would require many other books. This particular book, however, accomplished what it sets out to do which was ambitious enough.

A number of things are covered in the text. It’s not really about crowds, the title being unfortunate. Rather, it’s about group psychology and all that relates to that.

The author discusses the conditions under which any group will operate well or not. It could be a group of average citizens, a school board, or a grassroots organization. But it also could be a group of scientists, investors, or politicians. All humans are social animals, and so the same group psychology applies to all people and all groups. Even the paternalistic elites of Scandinavian countries, for example, are dependent on and interdependent with, built upon and inseparable from the constructive group dynamics of a culture of trust.

This would apply even to effectively ruling an authoritarian regime. That said, the conditions when fulfilled to their utmost would inevitably tend toward democratization, no matter the intent. This is true in all spheres to which these conditions could be applied. It isn’t just about politics. Far from it.

In reading reviews of the book, I realized many people didn’t understand the central point the author was communicating. He isn’t opposing crowds against experts. As I said, group psychology is a universal human reality, even for experts. Much of what the author discusses is experts themselves, when they are useful and when they are not. I found it odd that some reviewers thought the book had nothing to do about experts. Either they didn’t actually read the book or the unfortunate title biased their reading experience.

The term ‘expert’ was mentioned 92 times in The Wisdom of Crowds. Expertise is discussed in numerous chapters throughout the book. I’ll offer a few examples of it being discussed, but other examples not included below go into much more detail about examples and research.

(I’d also suggest the even more recent book, The Smart Swarm by Peter Miller. He also has useful discussion of groups and experts, both separately and as they interact, including some fascinating research. There are some other books I’m reading that may be relevant. I’ll probably write more about this topic later. I specifically have in mind a post about democracy in the city I live in, as analyzed according to Surowieki’s conditions of a wise crowd.)

 * * * *

The fact that cognitive diversity matters does not mean that if you assemble a group of diverse but thoroughly uninformed people, their collective wisdom will be smarter than an expert’s. But if you can assemble a diverse group of people who possess varying degrees of knowledge and insight, you’re better off entrusting it with major decisions rather than leaving them in the hands of one or two people, no matter how smart those people are. If this is difficult to believe—in the same way that March’s assertions are hard to believe—it’s because it runs counter to our basic intuitions about intelligence and business. Suggesting that the organization with the smartest people may not be the best organization is heretical, particularly in a business world caught up in a ceaseless “war for talent” and governed by the assumption that a few superstars can make the difference between an excellent and a mediocre company. Heretical or not, it’s the truth: the value of expertise is, in many contexts, overrated.

Now, experts obviously exist. The play of a great chess player is qualitatively different from the play of a merely accomplished one. The great player sees the board differently, he processes information differently, and he recognizes meaningful patterns almost instantly. As Herbert A. Simon and W. G. Chase demonstrated in the 1970s, if you show a chess expert and an amateur a board with a chess game in progress on it, the expert will be able to re-create from memory the layout of the entire game. The amateur won’t. Yet if you show that same expert a board with chess pieces irregularly and haphazardly placed on it, he will not be able to re-create the layout. This is impressive testimony to how thoroughly chess is imprinted on the minds of successful players. But it also demonstrates how limited the scope of their expertise is. A chess expert knows about chess, and that’s it. We intuitively assume that intelligence is fungible, and that people who are excellent at one intellectual pursuit would be excellent at another. But this is not the case with experts. Instead, the fundamental truth about expertise is that it is, as Chase has said, “spectacularly narrow.”

More important, there’s no real evidence that one can become expert in something as broad as “decision making” or “policy” or “strategy.” Auto repair, piloting, skiing, perhaps even management: these are skills that yield to application, hard work, and native talent. But forecasting an uncertain future and deciding the best course of action in the face of that future are much less likely to do so. And much of what we’ve seen so far suggests that a large group of diverse individuals will come up with better and more robust forecasts and make more intelligent decisions than even the most skilled “decision maker.”

We’re all familiar with the absurd predictions that business titans have made: Harry Warner of Warner Bros. pronouncing in 1927, “Who the hell wants to hear actors talk?,” or Thomas Watson of IBM declaring in 1943, “I think there is a world market for maybe five computers.” These can be written off as amusing anomalies, since over the course of a century, some smart people are bound to say some dumb things. What can’t be written off, though, is the dismal performance record of most experts.

Between 1984 and 1999, for instance, almost 90 percent of mutual-fund managers underperformed the Wilshire 5000 Index, a relatively low bar. The numbers for bond-fund managers are similar: in the most recent five-year period, more than 95 percent of all managed bond funds underperformed the market. After a survey of expert forecasts and analyses in a wide variety of fields, Wharton professor J. Scott Armstrong wrote, “I could find no studies that showed an important advantage for expertise.” Experts, in some cases, were a little better at forecasting than laypeople (although a number of studies have concluded that nonpsychologists, for instance, are actually better at predicting people’s behavior than psychologists are), but above a low level, Armstrong concluded, “expertise and accuracy are unrelated.” James Shanteau is one of the country’s leading thinkers on the nature of expertise, and has spent a great deal of time coming up with a method for estimating just how expert someone is. Yet even he suggests that “experts’ decisions are seriously flawed.”

Shanteau recounts a series of studies that have found experts’ judgments to be neither consistent with the judgments of other experts in the field nor internally consistent. For instance, the between-expert agreement in a host of fields, including stock picking, livestock judging, and clinical psychology, is below 50 percent, meaning that experts are as likely to disagree as to agree. More disconcertingly, one study found that the internal consistency of medical pathologists’ judgments was just 0.5, meaning that a pathologist presented with the same evidence would, half the time, offer a different opinion. Experts are also surprisingly bad at what social scientists call “calibrating” their judgments. If your judgments are well calibrated, then you have a sense of how likely it is that your judgment is correct. But experts are much like normal people: they routinely overestimate the likelihood that they’re right. A survey on the question of overconfidence by economist Terrance Odean found that physicians, nurses, lawyers, engineers, entrepreneurs, and investment bankers all believed that they knew more than they did. Similarly, a recent study of foreign-exchange traders found that 70 percent of the time, the traders overestimated the accuracy of their exchange-rate predictions. In other words, it wasn’t just that they were wrong; they also didn’t have any idea how wrong they were. And that seems to be the rule among experts. The only forecasters whose judgments are routinely well calibrated are expert bridge players and weathermen. It rains on 30 percent of the days when weathermen have predicted a 30 percent chance of rain.

Armstrong, who studies expertise and forecasting, summarized the case this way: “One would expect experts to have reliable information for predicting change and to be able to utilize the information effectively. However, expertise beyond a minimal level is of little value in forecasting change.” Nor was there evidence that even if most experts were not very good at forecasting, a few titans were excellent. Instead, Armstrong wrote, “claims of accuracy by a single expert would seem to be of no practical value.” This was the origin of Armstrong’s “seer-sucker theory”: “No matter how much evidence exists that seers do not exist, suckers will pay for the existence of seers.”

Again, this doesn’t mean that well-informed, sophisticated analysts are of no use in making good decisions. (And it certainly doesn’t mean that you want crowds of amateurs trying to collectively perform surgery or fly planes.) It does mean that however well-informed and sophisticated an expert is, his advice and predictions should be pooled with those of others to get the most out of him. (The larger the group, the more reliable its judgment will be.) And it means that attempting to “chase the expert,” looking for the one man who will have the answers to an organization’s problem, is a waste of time. We know that the group’s decision will consistently be better than most of the people in the group, and that it will be better decision after decision, while the performance of human experts will vary dramatically depending on the problem they’re asked to solve. So it is unlikely that one person, over time, will do better than the group.

Now, it’s possible that a small number of genuine experts—that is, people who can consistently offer better judgments than those of a diverse, informed group—do exist. The investor Warren Buffett, who has consistently outperformed the S&P 500 Index since the 1960s, is certainly someone who comes to mind. The problem is that even if these superior beings do exist, there is no easy way to identify them. Past performance, as we are often told, is no guarantee of future results. And there are so many would-be experts out there that distinguishing between those who are lucky and those who are genuinely good is often a near-impossible task. At the very least, it’s a job that requires considerable patience: if you wanted to be sure that a successful money manager was beating the market because of his superior skill, and not because of luck or measurement error, you’d need many years, if not decades, of data. And if a group is so unintelligent that it will flounder without the right expert, it’s not clear why the group would be intelligent enough to recognize an expert when it found him.

We think that experts will, in some sense, identify themselves, announcing their presence and demonstrating their expertise by their level of confidence. But it doesn’t work that way. Strangely, experts are no more confident in their abilities than average people are, which is to say that they are overconfident like everyone else, but no more so. Similarly, there is very little correlation between experts’ self-assessment and their performance. Knowing and knowing that you know are apparently two very different skills.

If this is the case, then why do we cling so tightly to the idea that the right expert will save us? And why do we ignore the fact that simply averaging a group’s estimates will produce a very good result? Richard Larrick and Jack B. Soll suggest that the answer is that we have bad intuitions about averaging. We assume averaging means dumbing down or compromising. When people are faced with the choice of picking one expert or picking pieces of advice from a number of experts, they try to pick the best expert rather than simply average across the group. Another reason, surely, is our assumption that true intelligence resides only in individuals, so that finding the right person—the right consultant, the right CEO—will make all the difference. In a sense, the crowd is blind to its own wisdom. Finally, we seek out experts because we get, as the writer Nassim Taleb asserts, “fooled by randomness.” If there are enough people out there making predictions, a few of them are going to compile an impressive record over time. That does not mean that the record was the product of skill, nor does it mean that the record will continue into the future. Again, trying to find smart people will not lead you astray. Trying to find the smartest person will.

Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 29-34). Knopf Doubleday Publishing Group. Kindle Edition.

This might be okay if people only spoke when they had expertise in a particular matter. And in many cases, if someone’s talking a lot, it’s a good sign that they have something valuable to add. But the truth is that there is no clear correlation between talkativeness and expertise. In fact, as the military-flier studies suggest, people who imagine themselves as leaders will often overestimate their own knowledge and project an air of confidence and expertise that is unjustified. And since, as political scientists Brock Blomberg and Joseph Harrington suggest, extremists tend to be more rigid and more convinced of their own rightness than moderates, discussion tends to pull groups away from the middle. Of course, sometimes truth lies at the extreme. And if the people who spoke first and most often were consistently the people with the best information or the keenest analysis, then polarization might not be much of a problem. But it is.

Surowiecki, James (2005-08-16). The Wisdom of Crowds (p. 186). Knopf Doubleday Publishing Group. Kindle Edition.

So what would the wider distribution of real decision-making power look like? To begin with, decisions about local problems should be made, as much as possible, by people close to the problem. Friedrich Hayek, as we’ve seen, emphasized that tacit knowledge—knowledge that emerged only from experience—was crucial to the efficiency of markets. It is just as important to the efficiency of organizations. Instead of assuming that all problems need to be filtered up the hierarchy and every solution filtered back down again, companies should start with the assumption that, just as in the marketplace, people with local knowledge are often best positioned to come up with a workable and efficient solution. The virtues of specialization and local knowledge often outweigh managerial expertise in decision making.

Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 209-210). Knopf Doubleday Publishing Group. Kindle Edition.

The best CEOs, of course, recognize the limits of their own knowledge and of individual decision making. That’s why important decisions at GM, in the days when it was the most successful corporation in the world, were made by what Alfred Sloan called “group management.” And it’s why legendary business thinker Peter Drucker has said, “The smart CEOs methodically build a management team around them.” The lesson of Richard Larrick and Jack Soll’s work applies to business as much as it does to other fields: chasing the expert is a mistake. The Federal Reserve’s decisions, after all, aren’t made by Alan Greenspan. They’re made by the board as a whole. In the face of uncertainty, the collective judgment of a group of executives will trump that of even the smartest executive. Think about John Craven’s work in finding the Scorpion. A relatively small group of diversely informed individuals making guesses about the likelihood of uncertain events produced, when their judgments had been aggregated, an essentially perfect decision. What more could a company want?

Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 220-221). Knopf Doubleday Publishing Group. Kindle Edition.

In January of 2003, 343 people, carefully chosen so that they represented an almost perfect cross-section of the American population, gathered in Philadelphia for a weekend of political debate. The topic was American foreign policy, with the issues ranging from the impending conflict with Iraq to nuclear proliferation to the global AIDS epidemic. Before the weekend, the participants were polled to get a sense of their positions on the issues. They were then sent a set of briefing materials that, in a deliberately evenhanded fashion, tried to lay out relevant facts and provide some sense of the ongoing debate about the issues. Once they arrived, they were divided up into small groups led by trained moderators, and went on to spend the weekend deliberating. Along the way, they were given the chance to interrogate panels of competing experts and political figures. At the end of the weekend, the participants were polled again, to see what difference their deliberations had made.

The entire event, which bore the unwieldy name of the National Issues Convention Deliberative Poll, was the brainchild of a political scientist at the University of Texas named James Fishkin. Fishkin invented the deliberative poll out of frustration with the limitations of traditional polling data and out of a sense that Americans were not being given either the information or the opportunity to make intelligent political choices. The idea behind deliberative polls—which have now been run in hundreds of cities across the world—is that political debate should not be, and doesn’t need to be, confined to experts and policy elites. Given enough information and the chance to talk things over with peers, ordinary people are more than capable of understanding complex issues and making meaningful choices about them. In that sense, Fishkin’s project is a profoundly optimistic one, predicated on a kind of deep faith in both the virtue of informed debate and the ability of ordinary people to govern themselves.

Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 257-258). Knopf Doubleday Publishing Group. Kindle Edition.

One knee-jerk reaction to the evidence of democracy’s failings is to insist that we would be better off ruled by a technocratic elite, which could make decisions with dispassion and attention to the public interest. To some extent, of course, we already are ruled by a technocratic elite, what with our republican form of government and the importance of unelected officials—for instance, Donald Rumsfeld or Colin Powell—in political life. But one would be hard-pressed to argue that most elites are able to see past their ideological blinders and uncover the imaginary public interest. And trusting an insulated, unelected elite to make the right decisions is a foolish strategy, given all we now know about small-group dynamics, groupthink, and the failure of diversity.

In any case, the idea that the right answer to complex problems is simply “ask the experts” assumes that experts agree on the answers. But they don’t, and if they did, it’s hard to believe that the public would simply ignore their advice. Elites are just as partisan and no more devoted to the public interest than the average voter. More important, as you shrink the size of a decision-making body, you also shrink the likelihood that the final answer is right. Finally, most political decisions are not simply decisions about how to do something. They are decisions about what to do, decisions that involve values, trade-offs, and choices about what kind of society people should live in. There is no reason to think that experts are better at making those decisions than the average voter. Thomas Jefferson, for one, thought it likely that they might be worse. “State a moral case to a ploughman and a professor,” he wrote. “The former will decide it as well and often better than the latter because he has not been led astray by artificial rules.”

It’s also the case that democracy allows for the persistent injection into the system of what I called earlier “local knowledge.” Politics is ultimately about the impact of government on the everyday lives of citizens. It seems strange, then, to think that the way to do politics well is to distance yourself as much as possible from citizens’ everyday lives. In the same way that a healthy market needs the constant flow of localized information that it gets from prices, a healthy democracy needs the constant flow of information it gets from people’s votes. That is information that experts cannot get because it is not part of the world they live in. And that keeps the system more diverse than it would otherwise be. As Richard Posner puts it: “Experts constitute a distinct class in society, with values and perspectives that differ systematically from those of ‘ordinary’ people. Without supposing that the man in the street has any penetrating insights denied the expert, or is immune from demagoguery, we may nevertheless think it reassuring that political power is shared between experts and nonexperts rather than being a monopoly of the former.”

Surowiecki, James (2005-08-16). The Wisdom of Crowds (pp. 265-266). Knopf Doubleday Publishing Group. Kindle Edition.

Knowledge and Wisdom in the Information Age

The Glass Bead Game from the Red Star Cafe blog

Magister LudiI suddenly realized that in the language, or at any rate in the spirit of the Glass Bead Game, everything actually was all-meaningful, that every symbol and combination of symbol led not hither and yon, not to single examples, experiments, and proofs, but into the center, the mystery and innermost heart of the world, into primal knowledge. Every transition from major to minor in a sonata, every transformation of a myth or a religious cult, every classical or artistic formulation was, I realized in that flashing moment, if seen with truly a meditative mind, nothing but a direct route into the interior of the cosmic mystery, where in the alternation between inhaling and exhaling, between heaven and earth, between Yin and Yang holiness is forever being created.

Joseph Knecht, Master of The Glass Bead Game by Hermann Hesse

 – – –

It’s been a long time since I read this book by Hesse, but I remember enjoying it.  I read a lot of Hesse in highschool and was highly impressed at the time.  This quote reminds me of a passage from Philip K. Dick’s Exegesis.  PKD was describing a mystical interaction with divine information.  Every thought, every question, every possibility led to infinity.  There was no final conclusion.  To read the this PKD passage, see my blog post PKD on God as Infinity.

 – – –

The Glass Bead Game Redux from the Red Star Cafe blog

Borg CubeReaders of blogs like this are witnessing a shift of intellectual authority from the traditional “expert” to the broader public. This is nowhere more tellingly illustrated than by Wikipedia, which has roughly 300,000 volunteer contributors every month.

What makes the mobilization of “crowd wisdom” intellectually powerful is that the technology of the Web makes it so easy for even amateurs to access a growing fraction of the body of human knowledge. The value of traditional expert authority is itself being diluted by the new incentive structure created by information technology that militates against what is deep and nuanced in favour of what is fast and stripped-down.

The result is the growing disintermediation of experts and gatekeepers of virtually all kinds. The irony is that experts have been the source of most of the nuggets of knowledge that the crowd now draws upon – for example, news and political bloggers depend heavily on a relatively small number of sources of professional journalism, just as many Wikipedia articles assimilate prior scholarship. The system works because it is able to mine intellectual capital. This suggests that today’s cult of the amateur will ultimately be self-limiting and will require continuous fresh infusions of more traditional forms of expert knowledge.

 – – –

I would point out that the intelligence of the internet age isn’t merely parasitic, but rather is a levelling of the playing field.  Instead of being passive receivers, people now interact with their media.

Two examples.

First, news media follows closely twitter and the blogosphere to catch new trends and breaking news.  Reporters aren’t usually the first people to be on a scene and with cellphones firsthand reporting can potentially come from anyone.

Second, bloggers often are very dedicated researchers who aren’t limited by the financial obligations of working for a media company.  Many bloggers are highly educated and trained in various fields.  Even if they don’t have the title of expert, they may act in that capacity.  Bloggers often do original analysis and uncover new data, and mainstream reporters do sometimes cite bloggers.  Bloggers don’t often get much respectability, but neither did the early muckrakers who were the earliest investigative reporters.

By being outside of the mainstream, bloggers have a different perspective.  Sometimes bloggers are reporting on issues and events that get almost entirely ignored by the mainstream media.

The value of traditional expert authority isn’t being diluted, but it is being challenged.  I would, however, argue that this strengthens expert authority by holding it to an even higher standard.

Objective analysis shows that Wikipedia articles on science and history are as reliable as encyclopedias (I would argue that they may be more reliable in some ways as they’re constantly being updated).  Also, Wikipedia cites many external sources that often are directly linked and so one can judge for themselves rather than solely relying on an expert.  In the long run, Wikipedia will on average become more reliable than a traditional printed encyclopedia.  Furthermore, Wikipedia has stringent standards and so acts as a training ground for any person to learn how to determine the validity of information.

So, the web doesn’t result in “the growing disintermediation of experts and gatekeepers”.  Rather, it increases mediation and creates better methods of gatekeeping.  Traditional experts still play a part, but they no longer dominate the discussion.

The above blog linked to an article by Peter Nicholson.  The following blog is a response to that article.  The opinion of stated below resonates with my own sense of this emerging information age.

 – – –

Reducing Life to a Formula from the Ooops…I’m still here blog

What has led me to rant about this pet peeve of mine,  is Peter Nicholson’s, Globe and Mail article, “Information – rich and attention – poor” (09-09-12).  Blaming the digital age he declares,

“In becoming information rich, we have become attention poor… [E]conomics teaches that the counterpart of every new abundance is a new scarcity – in this case the scarcity of human time and attention.”

[…]There is nothing wrong with the abundance of information created by digital technology.  Yes, I realise some of it is slim, but that’s okay, because there are ways of accessing deeper knowledge as well.  I personally have not experienced an attention deficit as a result of the “knowledge abundance”.  What I have experienced is a thrill at being able to access so much information in such a short time.  I do not fear what Nicholson refers to as the “24-hour knowledge cycle”, the ability to access news 24/7.  I relish in it.

                    Nicholson writes about the changing market for knowledge.  He states:

“When the effective shelf life of a document (or any information product) shrinks, fewer resources will be invested in its creation.  This is because the period during which the product is likely to be read or referred to is too short to repay a large allocation of scarce time and skill in its production.  As a result the ‘market’ for depth is narrowing.”

When you look at what is happening in the publishing world you have to agree with the first part of his comment, that because a “news product” has a short life it’s not financially feasible to invest heavily in it.  However, I disagree with his conclusion, that the result is  that the market for depth is narrowing.  Hey,  I’m part of the market and I’m not narrowing, nor are my eleven year old students who’s thirst for knowledge is unquenchable.  The desire for “depth” is not diminished by the abundance of knowledge.  In fact, it is enriched by it.