Bad vaccine statistics – real math is hard

Science deniers are an awfully frustrating lot. Statistical evidence seems so cut and dried to me. Unfortunately, the anti-science crowd prefers anecdotes to data. And the abuse of vaccine statistics are the worse.

Generally, science is based on evidence that is obtained through the scientific method. It is not magic. It is not a religion. Evidence is the fundamental basis of all sciences.

If science is evidence, then the basis of evidence is statistical reliability. I don’t want to oversimplify statistics, but this branch of mathematics identifies random events. Once again, statistics is difficult to grasp, yet every single paper I’ve ever read (except for the very worst) has fairly easy-to-understand statistics, once you know the lingo.

For example, most papers with vaccine statistics use a term called “relative risk,” or RR. Relative risk is the ratio of the probability of an event occurring in one group (say vaccinated) compared to a control group (not vaccinated).

An RR less than 1.0 implies that the the vaccinated group has a lower risk of an event than the control group. An RR=1.0 means that the risk is the same in both groups. Of course, an RR greater than 1.0 indicates that the vaccinated group has a higher risk than the control group.

And the size of the risk changes as numbers grow much larger or smaller than 1.0.

That statistical measurement seems easy. Undoubtedly, the calculations to reach the RR value are complex, but the top line number is fairly easy to grasp.

Nevertheless, vaccine statistics, despite being fairly straightforward, are often misinterpreted and ignored. Maybe there’s a reason for it? Let’s look.

Humans suck at vaccine statistics

 

Steven Novella, MD, recently wrote a great article about statistics in his NeuroLogica Blog. It got me thinking about why vaccine deniers ignore the best vaccine statistics. Those individuals will use anecdotes and VAERS data (see Note 1) to “prove” that vaccines are dangerous.

In science, personal anecdotes are, at best, an observation that could lead to a hypothesis which would need to be examined with scientific method. However, at worst, anecdotes are nothing more than cognitive biases such as confirmation or selection biases. In other words, it is not evidence at all.

As Dr. Novella says about these biases,

Confirmation bias is part of the process also – we notice, accept, and remember data points which support the pattern, and we ignore, reject, or forget those data points that contradict the pattern.

Eventually we are absolutely certain the pattern is real, because we think we have experienced it first hand. The data seems overwhelming, because it has been systematically and unconsciously mined from a vast set of potential data.

In other words, a typical science denier will find patterns in numbers that support their biases. The problem with anecdotes is that they select for data that support our a priori beliefs while ignoring the huge population that may not have shown any effects.

I have recently seen this with respect to climate change. There is a heat wave on the west coast of the USA, and climate change believers are stating that this is evidence of climate change. Ironically, climate change deniers will use a very cold weather pattern to claim that climate change doesn’t exist.

The problem is that they are selecting personal observations of weather (which is not climate) to confirm their beliefs. The real evidence of climate change is found across numerous lines of evidence.

Vaccine statistics – huge numbers

 

Dr. Novella then describes the “Law of Large Numbers,” a mathematical theorem. It states that the larger the sample size, and number of experiments, the more closely the average of all the results will reach the expected value.

A large sample size, say the whole population of the USA, will yield rare events. However, the problem occurs is that we look at the rare event as highly probable, especially if you have a personal relationship to the individual.

In the written history of mankind, meteors have caused no known deaths. Of course, with 7 billion people on earth today, it might happen. And if you knew a person who was killed by a meteor, you’d think it was a frequent occurrence. Which it isn’t.

There have been over 2.5 billion doses of vaccines given since 2006. That many doses is obviously a “large number.” And given that many doses, there will be all kinds of “adverse events” that will appear. Auto accidents, broken arms, infections, and yes, even autism may be observed after vaccines.

However, if those events have an RR (remember, relative risk) that is almost the same as the general population, then we just see these adverse events because of large numbers.

And, according to Novella, “because of how connected the world is, and the amount of mass and social media, you are very likely to hear about the unlikely things that happen to people.” One event in Finland or Alaska may appear to be like it happened next door because of how information passes so quickly these days.

However, without a real statistical analysis, we cannot determine whether the “unlikely” event is related to vaccines.

Vaccine statistics – Data mining

 

Chance happens a lot in life. Many years ago, I happen to run into a coworker, whom I had not seen in 20 years, on a busy street in London, UK. London is a long way from where we worked together, and it was just a random event. I happen to be walking to a restaurant, while he and his daughter were completely lost looking for a West End theater.

I just happen to hear their accents, and I knew they were Americans who needed some directions to the theater. Since I had lived in London for quite a bit of time, I pretty much knew how to get around the city. So I stopped and helped. And then figured out who he was.

Now, some of you might think this is some psychic connection. I didn’t. Since I spent a lot of time in London, and it is a tourist location for lots of Americans, a rare probability event could happen. Moreover, I picked out that one event, ignoring the millions of moments where I never met anyone I knew in London (which would make quite a boring story).

It’s possible that there is a 1 in 10 million chance that I will run into someone I knew in my life in London. And it is completely possible that running into that guy and his daughter could be statistically predicted.

Rare random events happen a lot, because even though they are not common, they do happen, especially if we have large numbers. VAERS is a perfect opportunity for vaccine data mining. If you examine everything contributed there (if we can assume it’s accurate), you’d think “wow, lots of problems with vaccines.”

The problem is that you might find 1,000 adverse events, while ignoring the other 2.5 billion doses that lead to nothing. Generally, people do not write a report on VAERS to say “nothing happened, thanks.” (See Note 2)

Real vaccine statistics

 

If you set aside data mining, anecdotes, and cognitive biases, then it becomes fairly easy to read the vaccine statistics. And once you get those statistics, you will find out that vaccines are relatively safe and very effective.

Vaccine statistics tell us that vaccines are unrelated to autism.

Vaccine statistics, using extremely large sample sizes, tell us that HPV vaccines are extremely safe, possibly one of the safest vaccines ever.

But humans just prefer their personal experiences. And those personal observations are overstated as statistically powerful. Maybe the complex descriptions of statistical models in vaccine papers overwhelms some people. Maybe they just believe that vaccines are harmful, statistical evidence be damned.

But statistics allows us to separate the random event from the related event. I wish more people understood that.

Notes

 

  1. For those of you don’t follow these things, VAERS, the Vaccine Adverse Event Reporting System, is a program to self-report adverse events from vaccines. It is run by the Centers for Disease Control and Prevention (CDC) and the US Food and Drug Administration (FDA).
  2. Maybe we should write reports that “nothing happened” after a vaccination.
 
 
The Original Skeptical Raptor
Chief Executive Officer at SkepticalRaptor
Lifetime lover of science, especially biomedical research. Spent years in academics, business development, research, and traveling the world shilling for Big Pharma. I love sports, mostly college basketball and football, hockey, and baseball. I enjoy great food and intelligent conversation. And a delicious morning coffee!