There is a common misunderstanding of clinical research design that confounds many, especially anti-vaccine advocates, who seem to think that only a placebo-controlled clinical trial provides useful data about the safety and effectiveness of drugs (especially vaccines).

Before anyone should take on the scientific consensus on a topic, like vaccines, research must include an understanding of what is called the hierarchy of biomedical research. It describes what gold (or even platinum) standards of research are. And which of them are nearly worthless?

For most drugs, including vaccines, research on safety and effectiveness continues for years after a drug is launched. These studies can provide additional data to governmental regulatory agencies, such as the FDA, that may cause changes to the package insert. And anti-vaccine activists love to misuse the package insert, but that’s another story.

This post will describe the hierarchy of clinical research so you can understand which type of study is being conducted. I often discuss studies on vaccines that employ one of the medical research designs critical to vaccine and drug development.

The hierarchy of clinical research

I’m going to list out the various types of clinical research in reverse order of usefulness. I’m only going to focus on types of studies that are published in real medical journals, not wild claims by internet influencers who have no knowledge of scientific research.

Case report

A case report is a detailed account of an individual patient’s clinical presentation, diagnosis, treatment, and follow-up, often highlighting unusual or educational aspects of the case.

Strengths: It can suggest further research.

Weaknesses: It is nothing more than an anecdote published in a medical journal.

Case-control study

A case-control study is an observational study comparing individuals with a specific condition (cases) to similar individuals without the condition (controls) to identify factors associated with the disease or outcome.

Strengths: Very powerful for studying rare outcomes. Low cost.

Weaknesses: It can show correlation or association, but it cannot establish causality

Retrospective cohort study

A retrospective cohort study is an observational study that looks back in time to compare outcomes between groups with and without a specific exposure. Medical records are often used in these types of studies.

Strengths: Large sample sizes are often possible, sometimes up to millions of patients. It is also low-cost.

Weaknesses: It can show correlation or association, but it can only suggest causality.

Randomized controlled trial

A randomized controlled trial (RCT) is a scientific experiment that evaluates the effectiveness or safety of an intervention by randomly assigning participants to experimental or control groups to minimize bias. The control group is often called the placebo group.

Strengths: Because of the design of the RCT, it can show causation.

Weaknesses: It is limited to smaller sample sizes and cost. In fact, the costs rise substantially with larger groups.

Note: Anti-vaxxers love to point to these type of studies as the gold standard of clinical research and insist that all vaccines should be compared to a placebo group. They insist that vaccines aren’t tested against placebos when that is not true. Moreover, if there is a predecessor vaccine, like with measles, it would be unethical to run a placebo-controlled study. This type of study would expose one group to measles which would never be undertaken by any ethical researchers.

Meta-analysis or systematic review

Meta-analysis is a statistical method used to combine and synthesize findings from multiple independent studies to provide a more accurate estimate of the effect size for a specific research question. Similarly, a systematic review is a rigorous, reproducible method for synthesizing all available evidence on a clearly defined research question using explicit, pre-specified methods.

A systematic review rigorously collects and synthesizes all relevant studies on a topic, while a meta-analysis statistically combines quantitative results from those studies to produce a precise overall effect estimate.

Strengths: Very powerful as it can show trends instead of statistical noise. It is also a low cost method of gathering a lot of data, and it is one of the primary data sets for evidence-based medicine.

Weaknesses: If the underlying clinical and observational trials were poorly designed, the results of a meta-analysis or systematic review may not show accurate results.

Summary

Although I discuss clinical research in the form of case-control, cohort, and randomized controlled studies, I have always preferred data from meta-analyses and systematic reviews since they provide the most robost clinical evidence.

But the most important point to make is that placebo controlled randomized clinical trials are not the only way to determine safety and effectiveness of vaccines. There are larger, statistically powerful methods that can tell us how any drug, including vaccines, are working.

Anti-vaccine activists love to cherry-pick research that seems to support (but rarely does) their desire to say “ah-ha” about vaccines. Unfortunately for them, research is complicated and there are lots of different types of studies that indicate vaccines are very safe and very effective.

Citations

• Spieth PM, Kubasch AS, Penzlin AI, Illigens BM, Barlinn K, Siepmann T. Randomized controlled trials – a matter of design. Neuropsychiatr Dis Treat. 2016 Jun 10;12:1341-9. doi: 10.2147/NDT.S101938. PMID: 27354804; PMCID: PMC4910682.

Help the Skeptical Raptor keep writing about vaccine research

There are four ways you can help contribute to keeping this website running at full speed:

1. GoFundMe.
2. Subscribing via Patreon.
3. Buying your favorite stuff on Amazon.
4. Sharing this article on Social Media.