Another Example of the Replication Crisis

A Waste of 1,000 Research Papers
by Ed Yong

Between them, these 18 genes have been the subject of more than 1,000 research papers, on depression alone. And for what? If the new study is right, these genes have nothing to do with depression. “This should be a real cautionary tale,” Keller adds. “How on Earth could we have spent 20 years and hundreds of millions of dollars studying pure noise?”

“What bothers me isn’t just that people said [the gene] mattered and it didn’t,” wrote the psychiatrist Scott Alexander in a widely shared blog post. “It’s that we built whole imaginary edifices on top of this idea of [it] mattering.” Researchers studied how SLC6A4 affects emotion centers in the brain, how its influence varies in different countries and demographics, and how it interacts with other genes. It’s as if they’d been “describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot,” Alexander wrote. […]

“We’re told that science self-corrects, but what the candidate gene literature demonstrates is that it often self-corrects very slowly, and very wastefully, even when the writing has been on the wall for a very long time,” Munafo adds.

Many fields of science, from psychology to cancer biology, have been dealing with similar problems: Entire lines of research may be based on faulty results. The reasons for this so-called “reproducibility crisis” are manifold. Sometimes, researchers futz with their data until they get something interesting, or retrofit their questions to match their answers. Other times, they selectively publish positive results while sweeping negative ones under the rug, creating a false impression of building evidence.

Beyond a few cases of outright misconduct, these practices are rarely done to deceive. They’re an almost inevitable product of an academic world that rewards scientists, above all else, for publishing papers in high-profile journals—journals that prefer flashy studies that make new discoveries over duller ones that check existing work. People are rewarded for being productive rather than being right, for building ever upward instead of checking the foundations. These incentives allow weak studies to be published. And once enough have amassed, they create a collective perception of strength that can be hard to pierce. […]

Similar debates have played out in other fields. When one group of psychologists started trying to reproduce classic results in much larger studies, their peers argued that any failures might simply be due to differences between the new groups of volunteers and the originals. This excuse has eroded with time, but to Border, it feels familiar. “There’s an unwillingness to part with a previous hypothesis,” he says. “It’s hard to wrap your head around the fact that maybe you were on a wild goose chase for years.”

Keller worries that these problems will be used as ammunition to distrust science as a whole. “People ask, ‘Well, if scientists are publishing crap, why should we believe global warming and evolution,’” he says. “But there’s a real difference: Some people were skeptical about candidate genes even back in the 1990s. There was never unanimity or consensus in the way there is for human-made global warming and the theory of evolution.”

(Credit to Nina Teicholz for bringing my attention to this article.)