Vaccine scientific knowledge – how to review and understand papers

This article about vaccine scientific knowledge is from a series of tweets from Lucian DiPeso, a Ph.D. candidate in the Hatch Lab at the Fred Hutchinson Cancer Research Center where he studies chromothripsis and micronuclei.

Below is a slightly edited version (with links) of the thread.


Time for another dive into a favorite anti-vaxxer canard—the demand for more double-blind, randomized placebo-controlled trials. There is much wrong here, and much to unpack.

Rather than focus on arguments over what kind of placebos to use or which study designs are better for what circumstance—which others have done splendidly — I want to talk about the nature of science and how scientists actually interpret studies. What anti-vaxxers are clinging to is the very real truth that any given study has weaknesses.

More broadly, how do we “know” something to be true or not? How do we know H. pylori infection causes stomach ulcers or that smoking leads to lung cancer?

Examining vaccine scientific knowledge

This seems straightforward at first. Let’s take a simple example: does treating mice with chemical X cause cancer? Simple, easy-peasy – let’s take 20 mice, give the drug to 10, not give the drug to the other 10, then compare them.

A few weeks later, what do we find? Eight of the treated mice have cancer. What can we conclude from this? Obviously: chemical X causes cancer!

But, does it? Do you know what would have happened to those eight mice if you hadn’t have given them chemical X? There is some random chance that any mouse will get cancer, so there is a possibility that the eight mice who got cancer would have anyways, even if you hadn’t treated them.

But, you protest, how likely is that? Not very! But, thousands of studies like this are done every day. Even if there was only a 1/1,000 chance that you’d randomly sort all the unlucky mice that were destined to get cancer into the treatment group, if there are 10,000 scientists all around the world doing similar experiments, you’d expect this to happen 10 times. In other words, you expect to see 10 cases where it looked, quite convincingly, like chemical X caused cancer.

And, someone could point to those 10 studies and exclaim that, of course, it causes cancer — just look! But, knowing as you do that 9,990 other studies did not find this effect, you could very confidently say it does not. And this is how science is actually done, how it is actually interpreted: any individual study must be evaluated in the context of what we already know.

Vaccine science papers 

Every study has weaknesses. Every study has flaws. Every couple of weeks, my lab will pour over a single article: does the data say what the authors claim? Do they use proper controls? What could they have done better? Are we convinced by the data we see?

There is not a single study that doesn’t have things it could or should have done better. Some are certainly better than others, but none are perfect. So how do we really know anything at all?

Replication and complimentary studies! Any one study will have its flaws. But, if 10 different studies by 10 different groups perform experiments 10 different ways that all point to the same conclusion, you should start to accept it.

Here’s the thing with vaccines: there haven’t been 10s of studies, or 100s, but 1000s, over decades, all over the world. Billions of vaccines are given every year. In short, vaccines are one of the most studied medical interventions ever.

So why isn’t this good enough for an anti-vaxxer?

What they’re demanding is the perfect study, the irrefutable study. But, that study doesn’t exist. That study can’t exist. Every individual study is going to have its drawbacks – too few people or too little time or too little money or what have you.

If we had to wait for the Ur-study to know if smoking caused cancer, we’d still be waiting. This was a tactic that cigarette companies exploited – they could always find or fund a study that found smoking was safe, because, by chance, some smokers will be fine.

If you happen to have a small study of, say, 20 people comparing smokers vs not, you might end up with the smokers living long, healthy lives — just by chance. The chance is small, but if you run this experiment enough times, it’s eventually going to happen.

The flip side is when someone shows me some study finding aluminum, one of the great Boogeymen of the anti-vax movement, does something bad (usually in mice). That one study is presented as proof, PROOF, of harm. And, if that one study were the only study, it would be interesting. But, it’s not. There are dozens if not hundreds of studies on the safety of aluminum adjuvants finding they are safe (it’s in baking powder, people—it’s fine).

Why, then, would I ever be convinced of the evils of aluminum by one study that cites known frauds, lacks any controls, and was published in a journal run by the author (really, true story), when there are hundreds more that show it’s safe?

You cannot look at scientific evidence as a collection of siloed, individual studies. Focusing on aluminum – it’s been used in food for decades, it’s been heavily studied, and the body of evidence as a whole says it’s safe. Maybe it’s not actually safe. Maybe pancakes are killing us all. But, I need more than a bad study to convince me.

So, no, there is no one perfect safety study of a vaccine, and there never will be. There are thousands (literally!) of good safety studies. That is something you can trust.

Please comment below, positive or negative. Of course, if you find spelling errors, tell me! And share this article.

There are two ways you can help me out to keep this website awesome. First, you can make a monthly contribution through Patreon:

Become a Patron!

Buy ANYTHING from Amazon.

Best Medical Blogs Worth Following -

Don’t miss each new article!

We don’t spam! Read our privacy policy for more info.

The Original Skeptical Raptor
Chief Executive Officer at SkepticalRaptor
Lifetime lover of science, especially biomedical research. Spent years in academics, business development, research, and traveling the world shilling for Big Pharma. I love sports, mostly college basketball and football, hockey, and baseball. I enjoy great food and intelligent conversation. And a delicious morning coffee!