Another Example of the Replication Crisis

A Waste of 1,000 Research Papers
by Ed Yong

Between them, these 18 genes have been the subject of more than 1,000 research papers, on depression alone. And for what? If the new study is right, these genes have nothing to do with depression. “This should be a real cautionary tale,” Keller adds. “How on Earth could we have spent 20 years and hundreds of millions of dollars studying pure noise?”

“What bothers me isn’t just that people said [the gene] mattered and it didn’t,” wrote the psychiatrist Scott Alexander in a widely shared blog post. “It’s that we built whole imaginary edifices on top of this idea of [it] mattering.” Researchers studied how SLC6A4 affects emotion centers in the brain, how its influence varies in different countries and demographics, and how it interacts with other genes. It’s as if they’d been “describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot,” Alexander wrote. […]

“We’re told that science self-corrects, but what the candidate gene literature demonstrates is that it often self-corrects very slowly, and very wastefully, even when the writing has been on the wall for a very long time,” Munafo adds.

Many fields of science, from psychology to cancer biology, have been dealing with similar problems: Entire lines of research may be based on faulty results. The reasons for this so-called “reproducibility crisis” are manifold. Sometimes, researchers futz with their data until they get something interesting, or retrofit their questions to match their answers. Other times, they selectively publish positive results while sweeping negative ones under the rug, creating a false impression of building evidence.

Beyond a few cases of outright misconduct, these practices are rarely done to deceive. They’re an almost inevitable product of an academic world that rewards scientists, above all else, for publishing papers in high-profile journals—journals that prefer flashy studies that make new discoveries over duller ones that check existing work. People are rewarded for being productive rather than being right, for building ever upward instead of checking the foundations. These incentives allow weak studies to be published. And once enough have amassed, they create a collective perception of strength that can be hard to pierce. […]

Similar debates have played out in other fields. When one group of psychologists started trying to reproduce classic results in much larger studies, their peers argued that any failures might simply be due to differences between the new groups of volunteers and the originals. This excuse has eroded with time, but to Border, it feels familiar. “There’s an unwillingness to part with a previous hypothesis,” he says. “It’s hard to wrap your head around the fact that maybe you were on a wild goose chase for years.”

Keller worries that these problems will be used as ammunition to distrust science as a whole. “People ask, ‘Well, if scientists are publishing crap, why should we believe global warming and evolution,’” he says. “But there’s a real difference: Some people were skeptical about candidate genes even back in the 1990s. There was never unanimity or consensus in the way there is for human-made global warming and the theory of evolution.”

(Credit to Nina Teicholz for bringing my attention to this article.)

2 thoughts on “Another Example of the Replication Crisis

  1. I take this debacle to be evidence for my conjecture that the bureaucratic [B] organisation of corporate entities is a very big part of the dysfunction of most modern corporate entities. I see a clear analogy with the fundamental, proplematic, dynamic described by Norman Dixon in his ‘On the Psychology of Military Incompetence’ [a very worthwhile read].
    IMO two elements of the cure for our current cultural miasma are on one hand to transform all B onto sustainable and robust democratic entities [SARDE] and on the other to impose a significant tax upon all who publish statements that can be shown as false – with the exception of course of properly formed and documented conjecturrd hypotheses which can be refuted by clear and unambiguous empirical

    • I haven’t read Norman Dixon, but I understand your point and agree with you. I’ve long argued that corporations are as bureaucratic as governments. The largest and wealthiest corporations in the world are more centralized and in many ways more powerful than most of the countries in the world. The average country is small with little reach beyond a local population. But there are thousands of transnational corporations that have direct or indirect influence over numerous governments worldwide, far more influence than do the local populations that these governments theoretically represent.

      This relates to research. For decades, the government funding to research and to universities has been declining. This has meant that corporations and corporate-funded organizations now do most of the funding in major areas of science. And their funding to universities has helped determine entire education programs and what students are taught. The corporatist vision has infected every part of our society and so everything is for sale.

      On reading the piece again, the last part stood out to me: “But there’s a real difference: Some people were skeptical about candidate genes even back in the 1990s. There was never unanimity or consensus in the way there is for human-made global warming and the theory of evolution.” There is another example of this seen in diet and nutrition. The present dominant paradigm was promoted through the charisma of Ancel Keys who accumulated immense power around him. Both big gov and big food aligned themselves with this new ideology, despite the fact that the science was always weak and there were plenty of scientific critics at the time. Consensus only came after the critics were silenced and not because the evidence ever improved.

      If you’re interested, any of the books by Gary Taubes and Nina Teicholz offer great insight, and also good is Tim Noakes’ Lore of Nutrition. You’ll be shocked by how unscientific the scientific process can be at times. Once powerful interests are involved, scientific debate becomes all but impossible. Marion Nestle also writes about the influence of corporations over research in this area. I’d also recommend Joanna Blythman and Kristin Lawless. There is another angle as well with the power of big religion in dietary ideology, as explored by Belinda Fettke — the Seventh Day Adventists fund a lot of research and over the past century have been one of the leading forces in pushing a particular agenda that was in line with Ancel Keys.

Please read Comment Policy before commenting.

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s