Clearing Away the Rubbish

“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue”
~Richard Horton, editor in chief of The Lancet

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor”
~Dr. Marcia Angell, former editor in chief of NEJM

Back in September, there was a scientific paper published in Clinical Cardiology, a peer reviewed medical journal that is “an official journal of the American Society for Preventive Cardiology” (Wikipedia). It got a ton of attention from news media, social media, and the blogosphere. The reason for all the attention is that, in the conclusion, the authors claimed that low-carb diets had proven the least healthy over a year period:

“One-year lowered-carbohydrate diet significantly increases cardiovascular risks, while a low-to-moderate-fat diet significantly reduces cardiovascular risk factors. Vegan diets were intermediate. Lowered-carbohydrate dieters were least inclined to continue dieting after conclusion of the study. Reductions in coronary blood flow reversed with appropriate dietary intervention. The major dietary effect on atherosclerotic coronary artery disease is inflammation and not weight loss.”

It has recently been retracted and it has come out that the lead author, Richard M. Fleming, has a long history of fraud going back to 2002 with two FBI convictions of fraud in 2009, following his self-confession. He has also since been debarred by the U.S. Food and Drug Administration. (But his closest brush with fame or infamy was his leaking the medical records of Dr. Robert Atkins, a leak that was behind a smear campaign.) As for his co-authors: “Three of the authors work at Fleming’s medical imaging company in California, one is a deceased psychologist from Iowa, another is a pediatric nutritionist from New York and one is a Kellogg’s employee from Illinois. How this group was able to run a 12-month diet trial in 120 subjects is something of a mystery” (George Henderson). Even before the retraction, many wondered how it ever passed peer-review considering the low quality of the study: “This study has so many methodological holes in it that it has no real value.” (Low Carb Studies BLOG).

But of course, none of that has been reported as widely as the paper originally was. So, most people who read about it still assume it is valid evidence. This is related to the replication crisis, as even researchers are often unaware of retractions, that is when journals will allow retractions to be published at all, something they are reluctant to do because it delegitimizes their authority. So, a lot of low quality or in some cases deceptive research goes unchallenged and unverified, neither confirmed nor disconfirmed. It’s rare when any study falls under the scrutiny of replication. If not for the lead author’s criminal background in the Fleming case, this probably would have been another paper that could have slipped past and been forgotten or else, without replication, repeatedly cited in future research. As such, bad research builds on bad research, creating the appearance of mounting evidence, but in reality it is a house of cards (consider the takedown of Ancel Keys and gang in the work by numerous authors: Gary Taubes’ Good Calories, Bad Calories; Nina Tiecholz’s The Big Fat Surprise; Sally Fallon Morrell’s Nourishing Diets; et cetera).

This is why the systemic problem and failure is referred to as a crisis. Fairly or unfairly, the legitimacy of entire fields of science are being questioned. Even scientists no longer are certain which research is valid or not. The few attempts at determining the seriousness of the situation by replicating studies has found a surprisingly low replication rate. And this problem is worse in the medical field than in many other fields, partly because of the kind of funding involved and more importantly because of how few doctors are educated in statistics or trained in research methodology. It is even worse with nutrition, as the average doctor gets about half the questions wrong when asked about this topic, and keep in mind that so much of the nutritional research is done by doctors. An example of problematic dietary study is that of Dr. Fleming himself. We’d be better off letting physicists and geologists do nutritional research.

There is more than a half century of research that conventional medical and dietary opinions are based upon. In some major cases, re-analysis of data has shown completely opposite conclusions. For example, the most famous study by Ancel Keys blamed saturated fat for heart disease, while recent reappraisal has shown the data actually shows a stronger link to sugar as the culprit. Meanwhile, no study has ever directly linked saturated fat to heart disease. The confusion has come because, in the Standard American Diet (SAD), saturated fat and sugar have been conflated in the population under study. Yet, even in cases like that of Keys when we now know what the data shows, Keys’ original misleading conclusions are still referenced as authoritative.

The only time this crisis comes to attention is when the researcher gets attention. If Keys wasn’t famous and Fleming wasn’t criminal, no one would have bothered with their research. Lots of research gets continually cited without much thought, as the authority of research accumulates over time by being cited which encourages further citation. It’s similar to how legal precedents can get set, even when the initial precedent was intentionally misinterpreted for that very purpose.

To dig through the original data, assuming it is available and one knows where to find it, is more work than most are willing to do. There is no glory or praise to be gained in doing it, nor will it promote one’s career or profit one’s bank account. If anything, there are plenty of disincentives in place, as academic careers in science are dependent on original research. Furthermore, private researchers working in corporations, for obvious reasons, tend to be even less open about their data and that makes scrutiny even more difficult. If a company found their own research didn’t replicate, they would be the last in line to announce it to the world and instead would likely bury it where it never would be found.

There is no system put into place to guard against the flaws of the system itself. And the news media is in an almost continual state of failure when it comes to scientific reporting. The crisis has been stewing for decades, occasionally being mentioned, but mostly suppressed, until now when it has gotten so bad as to be undeniable. The internet has created alternative flows of information and so much of the scrutiny, delayed for too long, is now coming from below. If this had happened at an earlier time, Fleming might have gotten away with it. But times have changed. And in crisis, there is opportunity or at very least there is hope for open debate. So bring on the debate, just as soon as we clear away some of the rubbish.

* * *

Retracted: Long‐term health effects of the three major diets under self‐management with advice, yields high adherence and equal weight loss, but very different long‐term cardiovascular health effects as measured by myocardial perfusion imaging and specific markers of inflammatory coronary artery disease

The above article, published online on 27 September 2018 in Wiley Online Library (wileyonlinelibrary.com), has been withdrawn by agreement between the journal Editor in Chief, A. John Camm and Wiley Periodicals, Inc. The article has been withdrawn due to concerns with data integrity and an undisclosed conflict of interest by the lead author.

A convicted felon writes a paper on hotly debated diets. What could go wrong?
by Ivan Oransky, Retraction Watch

Pro-tip for journals and publishers: When you decide to publish a paper about a subject — say, diets — that you know will draw a great deal of scrutiny from vocal proponents of alternatives, make sure it’s as close to airtight as possible.

And in the event that the paper turns out not to be so airtight, write a retraction notice that’s not vague and useless.

Oh, and make sure the lead author of said study isn’t a convicted felon who pleaded guilty to healthcare fraud.

“If only we were describing a hypothetical.

On second thought: A man of many talents — with a spotty scientific record
by Adam Marcus, Boston Globe

Richard M. Fleming may be a man of many talents, but his record as a scientist has been spotty. Fleming, who bills himself on Twitter as “PhD, MD, JD AND NOW Actor-Singer!!!”, was a co-author of short-lived paper in the journal Clinical Cardiology purporting to find health benefits from a diet with low or modest amounts of fat. The paper came out in late September — just a day before the Food and Drug Administration banned Fleming from participating in any drug studies. Why? Two prior convictions for fraud in 2009.

It didn’t take long for others to begin poking holes in the new article. One researcher found multiple errors in the data and noted that the study evidently had been completed in 2002. The journal ultimately retracted the article, citing “concerns with data integrity and an undisclosed conflict of interest by the lead author.” But Fleming, who objected to the retraction, persevered. On Nov. 5, he republished the study in another journal — proving that grit, determination, and a receptive publisher are more important than a spotless resume.

Scientific Failure and Self Experimentation

In 2005, John P. A. Ioannidis wrote “Why Most Published Research Findings Are False” that was published in PloS journal. It is the most cited paper in that journal’s history and it has led to much discussion in the media. That paper was a theoretical model but has since been well supported — as Ioannidis explained in an interview with Julia Belluz:

“There are now tons of empirical studies on this. One field that probably attracted a lot of attention is preclinical research on drug targets, for example, research done in academic labs on cell cultures, trying to propose a mechanism of action for drugs that can be developed. There are papers showing that, if you look at a large number of these studies, only about 10 to 25 percent of them could be reproduced by other investigators. Animal research has also attracted a lot of attention and has had a number of empirical evaluations, many of them showing that almost everything that gets published is claimed to be “significant”. Nevertheless, there are big problems in the designs of these studies, and there’s very little reproducibility of results. Most of these studies don’t pan out when you try to move forward to human experimentation.

“Even for randomized controlled trials [considered the gold standard of evidence in medicine and beyond] we have empirical evidence about their modest replication. We have data suggesting only about half of the trials registered [on public databases so people know they were done] are published in journals. Among those published, only about half of the outcomes the researchers set out to study are actually reported. Then half — or more — of the results that are published are interpreted inappropriately, with spin favoring preconceptions of sponsors’ agendas. If you multiply these levels of loss or distortion, even for randomized trials, it’s only a modest fraction of the evidence that is going to be credible.”

This is part of the replication crisis that has been known about for decades, although rarely acknowledged or taken seriously. And it is a crisis that isn’t limited to single studies —- Ioannidis wrote that, “Possibly, the large majority of produced systematic reviews and meta-analyses are unnecessary, misleading, and/or conflicted” (from a paper reported in the Pacific Standard). The crisis cuts across numerous fields, from economics and genetics to neuroscience and psychology. But to my mind, medical research stands out. Evidence-based medicine is only as good as the available evidence — it has been “hijacked to serve agendas different from what it originally aimed for,” as stated by Ioannidis. (A great book on this topic, by the way, is Richard Harris’ Rigor Mortis.) Studies done by or funded by drug companies, for example, are more likely to come to positive results for efficacy and negative results for side effects. And because the government has severely decreased public funding since the Reagan administration, so much of research is now linked to big pharma. From a Retraction Watch interview, Ioannidis says:

“Since clinical research that can generate useful clinical evidence has fallen off the radar screen of many/most public funders, it is largely left up to the industry to support it. The sales and marketing departments in most companies are more powerful than their R&D departments. Hence, the design, conduct, reporting, and dissemination of this clinical evidence becomes an advertisement tool. As for “basic” research, as I explain in the paper, the current system favors PIs who make a primary focus of their career how to absorb more money. Success in obtaining (more) funding in a fiercely competitive world is what counts the most. Given that much “basic” research is justifiably unpredictable in terms of its yield, we are encouraging aggressive gamblers. Unfortunately, it is not gambling for getting major, high-risk discoveries (which would have been nice), it is gambling for simply getting more money.”

I’ve become familiar with this collective failure through reading on diet and nutrition. Some of the key figures in that field, specifically Ancel Keys, were either intentionally fraudulent or really bad at science. Yet the basic paradigm of dietary recommendations that was instituted by Keys remains in place. The fact that Keys was so influential demonstrates the sad state of affairs. Ioannidis has also covered this area and come to similar dire conclusions. Along with Jonathan Schoenfeld, he considered the question “Is everything we eat associated with cancer?”

“After choosing fifty common ingredients out of a cookbook, they set out to find studies linking them to cancer rates – and found 216 studies on forty different ingredients. Of course, most of the studies disagreed with each other. Most ingredients had multiple studies claiming they increased and decreased the risk of getting cancer. Most of the statistical evidence was weak, and meta-analyses usually showed much smaller effects on cancer rates than the original studies.”
(Alex Reinhart, What have we wrought?)

That is a serious and rather personal issue, not an academic exercise. There is so much bad research out there or else confused and conflicting. It’s about impossible for the average person to wade through it all and come to a certain conclusion. Researchers and doctors are as mired in it as the rest of us. Doctors, in particular, are busy people and don’t typically read anything beyond short articles and literature reviews, and even those they likely only skim in spare moments. Besides, most doctors aren’t trained in research and statistics, anyhow. Even if they were better educated and informed, the science itself is in a far from optimal state and one can find all kinds of conclusions. Take the conflict between two prestigious British journals, the Lancet and the BMJ, the former arguing for statin use and the latter more circumspect. In the context of efficacy and side effects, the disagreement is over diverse issues and confounders of cholesterol, inflammation, artherosclerosis, heart disease, etc — all overlapping.

Recently, my dad went to his doctor who said that research in respectable journals strongly supported statin use. Sure, that is true. But the opposite is equally true, in that there are also respectable journals that don’t support wide use of statins. It depends on which journals one chooses to read. My dad’s doctor didn’t have the time to discuss the issue, as that is the nature of the US medical system. So, probably in not wanting to get caught up in fruitless debate, the doctor agreed to my dad stopping statins and seeing what happens. With failure among researchers to come to consensus, it leaves the patient to be a guinea pig in his own personal experiment. Because of the lack of good data, self-experimentation has become a central practice in diet and nutrition. There are so many opinions out there that, if one cares about one’s health, one is forced to try different approaches and find out what seems to work, even as this methodology is open to many pitfalls and hardy guarantees success. But the individual person dealing with a major health concern often has no other choice, at least not until the science improves.

This isn’t necessarily a reason for despair. At least, a public debate is now happening. Ioannidis, among others, sees the solution as not difficult (psychology, despite its own failings, might end up being key in improving research standards; and also organizations are being set up to promote better standards, including The Nutrition Science Initiative started by the science journalist Gary Taubes, someone often cited by those interested in alternative health views). We simply need to require greater transparency and accountability in the scientific process. That is to say science should be democratic. The failure of science is directly related to the failure seen in politics and economics, related to powerful forces of big money and other systemic biases. It is not so much a failure as it is a success toward ulterior motives. That needs to change.

* * *

Many scientific “truths” are, in fact, false
by Olivia Goldhill

Are most published research findings false?
by Erica Seigneur

The Decline Effect – Why Most Published Research Findings are False
by Paul Crichton

Beware those scientific studies—most are wrong, researcher warns
by Ivan Couronne

The Truthiness Of Scientific Research
by Judith Rich Harris

Is most published research really wrong?
by Geoffrey P Webb

Are Scientists Doing Too Much Research?
by Peter Bruce