Published medical research is a failure – not what you think

Yes, you read that right. This so-called shill for the pharmaceutical industry is calling them as I see them – most published medical research is a failure –  but stay tuned for the full story.

Now by failure I mean that more often than not, claims that are supported by one or two published articles, rarely lead to a clinically significant product (such as a pharmaceutical). Of course, I don’t mean that the research is fraudulent, although some are, especially in low level journals frequented by pseudoscience pushers.

And I don’t mean it’s bad science, although there’s evidence of that, which I’ll discuss below.

And I don’t mean that there’s some grand conspiracy between Big Pharma and everyone else (again, no evidence to support that nonsense), although there is some evidence that research sponsored by Big Pharma is poorly done.

So what do I mean? Results from lot of medical research that get splashed in the news rarely, and I mean rarely, end up having any clinical utility. Rarely, but not never.

This does not mean that medical procedures, pharmaceuticals and devices that have been vetted through lots of research that repeat and confirm the original data and that form the basis of a scientific consensus are bad research. Almost everything that passes by the FDA and other regulatory agencies in other countries meets high standards for risk and benefit analysis.

Finally, arriving at a scientific consensus is a brutal, time-consuming process. It means that the theory or idea has been repeated many times, and the analysis and data are solid. So even though “most” research ends up in a failure, that’s because science is harsh to research that can’t be repeated, or was badly designed.

The best research isn’t a failure, even if it finds negative results. And the best ideas in medicine, let’s say vaccines, have been so thoroughly vetted that the consensus is nearly unassailable. Though people try with their poorly designed, unrepeatable research.

Medical research to clinical applicability

 

In 2003, researchers reviewed 101 studies published in  respected, high impact factor scientific and medical journals between 1979 and 1983 that claimed a new therapy or medical technology was clinically promising. What was shocking was that they found only five, of those 101, made it to market within a decade of publication.

Why did they choose a decade? That’s a good period of time for further research, clinical trials, and finally FDA approval.

So that’s just 5%. But what’s even more ironic is that only one, ACE inhibitors which are used to treat hypertension, was still used at the time of publication (2003). Of course, it’s still used today as one of the basic tools in treating high blood pressure.

Think about that. There were 101 breathless claims of some miraculous medical breakthrough (if the PR departments of the various universities and startup companies did their jobs), and only one is on the market today.

 

Cancer research

 

I often make a pointed joke about research into cancer. “We have cured cancer in mice over the past decade.” Setting aside the fact that there will never be “one cure to cure them all,” given that there are over 200 cancers, each so different that it would be impossible to imagine one drug.

But when you read an article that eating blueberries will prevent breast cancer based on one mouse study, you need to consider it very skeptically. The gold standard of medical research is a double blind clinical trial, repeated several times, and then rolled up into a systematic review–the basis for a scientific consensus. At that point, if it showed significant clinical differences, I’ll be eating blueberries 24/7.

The marijuana activists love to tell you that cannabis can treat cancer. Yet, the only evidence of its ability to treat cancer is in one poorly designed clinical trial for one specific cancer, and a few animal studies. Of course, the level of the active ingredient that might attack some cancers would require a blood level of THC that would kill the smoker. Well, one couldn’t smoke the hundreds of joints necessary to kill this cancer–a rate that would have to be sustained every day until the cancer was “cured.”

 
 

Matthew Herper, writing for Forbes, talked with one pharmaceutical industry consultant that there were over 200 failures of cancer treatment drugs over the past few years. Failure in cancer treatment is much more common than success. And this is why accepting the hype of a treatment that is successful in mice and rats, or in one or just a handful of real patients, is going to lead you down the wrong path of cancer treatment.

There are so many drugs that appear to work with one person, but that could be just random. Sometimes people get better from cancers because of unknown reasons (and don’t take that to mean that obviously it’s blueberries, it’s because sometimes the body gets it right, for reasons we will understand eventually).  So if you base your hype on ONE study with ONE patient, you will be mislead.

But as Herper’s article says,

 

[infobox icon=”quote-left”]There are two reasons to be wildly optimistic about the state of cancer research. One is that new treatments that harness the immune system to attack tumors do lead, in some cases, to long-lasting remissions, even cures. This includes the treatments Vice profiles, but also drugs that make tumor cells visible to our bodies so they can be destroyed.

The second is that new genetic technologies are allowing doctors to start to look at tumors based on what mutations have caused the cells in them to go haywire. That means targeted drugs like Gleevec, Rituxan, and Herceptin that add years to patients’ lives. It may also mean new types of diagnostic tests that can detect cancer early, when genetic material from tumors first starts to circulate in the blood. I think we truly are at a moment of revolution.[/infobox]

This is truly a wonderful technological advance in treating cancers. But, we’ve heard this before. Remember, there were 101 medical advances that were hyped just like this, and only one ended up having any clinical relevance today.

One more point. Over 90% of cancer drugs that enter Phase 1 trials never get FDA approval to be marketed (pdf). It more or less debunks that myth that the FDA is in the pocket of Big Pharma. Or someone needs to up the payments or something, because a 10% success rate probably indicates that the FDA  is pretty tough on these applications

Evaluating research

 

I’ve dealt with this subject in much more detail. The most important thing to note is that the further up the hierarchy of scientific evidence a claim sits, the more likely the evidence will support (or refute) a clinical claim.

At the bottom of the hierarchy are animal or cell culture studies. At the top, are systematic reviews or, even better, meta analyses. These large studies roll up data from dozens or hundreds of other studies to determine what all of the research says.

Too often university public relations departments or alternative “medicine” websites will tout an animal study. But it may be 10-20 years before that study will be shown to have any clinical usefulness. If you’re trying to convince someone that vitamin D reverses diabetic neuropathy, then you need to bring clinical trials and meta analyses. And right now, you don’t have that.

There’s that old saying that was popularized by Carl Sagan–”extraordinary claims demand extraordinary evidence.” Extraordinary evidence could be meta analyses, and until that is made available, then an extraordinary claim is just a claim. Or a belief. Or just a myth.

 

What does this all mean, the TL;DR version?

 

  1. Be skeptical of any medical research until it exceeds the standard of double blind clinical trials, is repeated by other researchers, and a meta analyses of all the results supports the hypothesis.
  2. If the whole basis of your medical claim is a poorly designed study, includes mice, or is published in a bad journal, it little validity in making a claim about it’s clinical utility.
  3. The FDA and Big Pharma are not in any kind of conspiracy given that most drugs fail to get approved. This is one of the most laughable conspiracies ever.
  4. Cherry picking research to support your a priori beliefs? That’s really not going to work.

Editor’s note: This article was originally published in March 2015. It has been revised and updated to include more comprehensive information, to improve readability and to add current research.

Key citations:

 

The Original Skeptical Raptor
Chief Executive Officer at SkepticalRaptor

Lifetime lover of science, especially biomedical research. Spent years in academics, business development, research, and traveling the world shilling for Big Pharma. I love sports, mostly college basketball and football, hockey, and baseball. I enjoy great food and intelligent conversation. And a delicious morning coffee!


  • Pingback: Scientific consensus – collective opinion of scientists()

  • Pingback: Acidic blood causes cancer – time to demolish a myth()

  • Pingback: Gardasil causes behavioral issues – more myth debunking()

  • Pingback: It's difficult to be a real scientific skeptic – let's make it easier()

  • Pingback: A failed vaccine clinical trial – it happens()

  • Pingback: The hierarchy of scientific evidence – keys to skepticism()

  • Pingback: Review of Trace Amounts – bad science and conspiracies()

  • Dylan Fazel

    Thanks for this useful articles. To know about the medical research is of extreme useful for all the people.

  • Pingback: Big Pharma vaccine profits conspiracy – Skeptical Raptor()

  • Pingback: Glyphosate causes cancer – Skeptical Raptor()

  • Pingback: Marijuana and cancer – facts and smoke – Skeptical Raptor()

  • Skinnydipper

    One thing I took from this but didn’t see in your article is that the cost to bring a new treatment into clinical usage is based not just on the cost of the new drug or treatment and its testing. You have to pay for the 99 other investigations that, while they probably produced valuable knowledge, didn’t result in a clinical treatment. In other words, there is a cost just to test ideas. And people wonder why new drugs are expensive and clamor to shorten the patent period.

    • Yes, I’m glad someone understood that without my having to beat it to death. There’s this myth amongst the Big Pharma haters that somehow it’s a huge scam. But the reality is that most medical advances are miserable failures. And that’s a good thing, because I know that real science can detect the difference between a high benefit:risk product versus a low one.

      And yes, it is frighteningly expensive. Just to get one drug to the end of Phase 3 clinical trials can cost up to $1 billion. And even then, after all that investment, around 45% fail. If someone is bribing someone to get drugs approved, apparently it’s not enough.

      • Diddly doo

        “Of course, I don’t mean that the research is fraudulent, although some are,

        especially in low level journals frequented by pseudoscience pushers.” Desperate smokescreen

        I think you have missed the elephant in the room here. The Lancet, the BMJ and the NEJM are not low level. The big problem with medical peer reviewed research is that it is not independently done.

        http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/

        Medical peer review – a flawed process
        “That is why Robbie Fox, the great 20th century editor of the Lancet,
        who was no admirer of peer review, wondered whether anybody would
        notice if he were to swap the piles marked `publish’ and `reject’. He
        also joked that the Lancet had a system of throwing a pile of papers down the stairs and publishing those that reached the bottom.”

        “But does peer review `work’ at all? A systematic review of all the
        available evidence on peer review concluded that `the practice of peer
        review is based on faith in its effects, rather than on facts'”

        http://www.collective-evolution.com/2015/05/16/editor-in-chief-of-worlds-best-known-medical-journal-half-of-all-the-literature-is-false/

        You need to start reading these critiques of medical peer review, because your opinion is way out of kilter with the damning evidence from the top journals.

        • Diddly doo

          You cannot dismiss the total disaster in medical peer review as a few minor issues septical rantor – that is one crazy denial system you have there.

          • I guess your anecdotes and misinformation and cherry picking and ad hominem and various other nonsense qualifies as a better system.

            And good for you to cherry pick one whining ignorant douche as a source to condemn all science. Wow, I’m impressed.

            What a maroon you are.

            • Diddly doo

              Interesting that you think medical peer reviewed papers are misinformation and anecdotes – so now you are joining the campaign against medical peer review as evidence too. What leg are you standing on?

              The BMJ, NEJM, the Lancet, NCBI, wow, that’s quite a large cherry tree douche then, there are not many others. Who is condemning all science?, the problem is that medicine sees itself as a science when in fact it isn’t really. I mean how can arthritis be treated by turning off the immune system? and acid reflux be treated by proton pump inhibitors – if we maintained cars like this we would all be walking.

            • Diddly doo

              Interesting that you think medical peer reviewed papers are misinformation and anecdotes – so now you are joining the campaign against medical peer review as evidence too. What leg are you standing on?

              The BMJ, NEJM, the Lancet, NCBI, wow, that’s quite a large cherry tree douche then, there are not many others. Who is condemning all science?, the problem is that medicine sees itself as a science when in fact it isn’t really. I mean how can arthritis be treated by turning off the immune system? and acid reflux be treated by proton pump inhibitors – if we maintained cars like this we would all be walking.