One of the hallmarks of pseudoscience is an over-reliance on confirmation rather than refutation of a hypothesis.  The anti-vaccine crowd are well-known for this particular violation of the scientific method.  As discussed previously, science works on refutation–creating experiments that might actually disprove a hypothesis as a method to develop evidence in support of it.  The anti-vaccination crowd actually hypothesizes (but not in a scientific sense) that a vaccine or set of vaccines was the causal factor in some side effect (autism, death, or whatever else), then they should establish an experiment (double-blinded of course) that would refute that hypothesis.  If at some point, the data cannot refute it, then the anti-vaccinationists would have supporting data for their particular supposition. 

But instead of actually performing experiments (which cost money, which may show that they are wrong, or which might not be ethical), they resort to mining data to prove their point.  Data mining is dangerous, because confirmation bias, that is, finding information or data that supports a belief while ignoring all other data that does not, makes the data suspect or even useless. So, in that vein, the anti-vaccinationists often mine data from any database they can find, such as the Vaccine Adverse Event Reporting System (VAERS),  which is a program for vaccine safety, managed by the Centers for Disease Control and Prevention (CDC) and the Food and Drug Administration (FDA). VAERS functions as a post-marketing safety surveillance program (similar to other programs for almost every regulated medical device and pharmaceutical) which collects information about adverse events (whether related or unrelated to the vaccine) that occur after administration of vaccines. VAERS has numerous limitations, including lack of scientifically designed questions, unverified reports, underreporting, inconsistent data quality, and absence of an unvaccinated control group.  VAERS is basically a collector of information, but has limited value in making conclusions since it does not provide information that is obtained in a controlled manner.  However, it does have some usefulness, in that certain trends may be spotted given enough time and data points.

A few weeks ago, Matthew Herper, Forbes magazine’s medical editor, wrote an article about how Americans were rejecting the human papillomavirus (HPV) vaccine, Gardasil.  He has followed up that article with a another article about the vaccine, entitled Here Is How We Know Gardasil Has Not Killed 100 People.  You can read the whole article at the link, but there are a few key points that should be highlighted. 

For example, VAERS has reported over 20,000 adverse events and 100 deaths after the Gardasil injection.  But Herper has a incredibly perceptive analysis of the data:

…let’s take a look at those 20,000 adverse events and 100 deaths and figure out what they mean. It’s absolutely clear that these are for the most part not side effects from Gardasil. Nor is the vaccine, which has been given to more than 10 million people, likely responsible for those deaths.

The Vaccine Adverse Event Reporting System was put in place in 1990 as a result of a 1986 law that requires health providers to report harm that comes to patients within a specific time period after vaccination. (The same law also limits liability for vaccine manufacturers to prevent them from abandoning the category.) A great many of these reports can come from sales reps for drug manufacturers who hear about the incidents.

Unfortunately, VAERS data is notoriously spotty – better than nothing, but there’s no way to insure that potential side effects are reported. When a product gets bad press, the number of reported “adverse events” goes up. And there is no way to tell if a particular side effect is linked to the vaccine. Some people will die after any vaccination, not because vaccines cause death but because people, even babies and adolescents, die with terrible regularity.

It’s true that there have been 24,000 reports of adverse events with Gardasil. (All of these numbers come from the VAERS database, which you can searchhere.) There have also been 60,000 reports of death with the mumps, measles, and rubella vaccine, and 26,000 following vaccination with Pfizer’s Prevnar, for pneumococcus bacteria. And yes, it’s true that there have been 106 deaths reported after Gardasil vaccination. There have also been 101 deaths reported after vaccination with Prevnar 13, a new version of Prevnar introduced in 2010.. It’s normal for these reports to pour in for safe vaccines.

You can’t directly link any of those adverse events or deaths directly to the vaccines, any more than you could blame it on my morning coffee if I got hit by a truck later today. So to try to make use of this data, researchers compare the rates at which negative side effects are reported for different vaccines. The CDC and FDA did this for HPV vaccines in 2009, looking at the first 12,424 reports to VAERS and publishing the result in the Journal of the American Medical Association. They did note 2 cases of unusual neurological symptoms similar to Lou Gehrig’s disease, and there was an increase in patients who had potentially dangerous blood clots, although 90% of those patients had a risk factor for those clots, such as taking birth control pills, that might explain the increase. The researchers specifically looked at Guillain-Barré Syndrome, a neurological disorder that had been linked to a bad batch of flu shots; there wasn’t a signal that this was a problem with Gardasil.  The study did result in the FDA advising doctors to watch adolescents after they get their shots, because some faint.

Based on that analysis, it seems that of those dozens of deaths, only a handful could possibly be linked to Gardasil. And based on the data available, it is unlikely (though not impossible) that even those deaths were caused by the vaccine. The risks from the vaccine are very small and may be limited to headaches and fainting caused by the needle, not the vaccine itself. Gardasil has been studied in clinical trials of more than 30,000 people; Cervarix, the competitor vaccine, has run a similar gantlet.

It’s really important not to rely too much on VAERS, because it can lead to some totally Alice-in-Wonderland-type conclusions – fantastical results that we can tell aren’t true. Forbes commenter Rick Wobbe, a former pharmaceutical exec who is rarely shy about being critical of pharma, responded to my story by plunging bravely into the VAERS himself.

In summary, anyone reading that VAERS data as conclusive proof of the dangers of Gardasil would be engaging in a couple of major logical fallacies: the first, confirmation bias, is discussed above.  The data miners are looking at the data as if they deserve no criticism, no analysis, while ignoring the 100 million people who took the vaccine with no adverse event.  So using this data alone (without any further analysis), less than .02% of vaccinations have an adverse event, and less than 0.0001% would die.  On the other hand, about 150,000 Americans per year contract some type of cancer directly attributable to HPV, and about half of cervical cancers (almost always due to HPV) lead to death. Therefore, if we actually accepted the anti-vaccinationists analysis of this data, we would still conclude the risk was worth the incredible benefit.

The second logical fallacy is post hoc ergo propter hoc, or using correlation to presume some sort of biological causation.  Conflating correlation with causation leads to major errors in creating a conclusion.  If someone is to propose that there is correlation, the first step to finding causation is to actually provide a logical and plausible pathway between presumptive cause and the presumptive effect.  

Herper continues with this line of ridiculousness from the anti-Gardasil crowd:

Thus far, none of these adverse events appear to occur any more frequently among HPV vaccine recipients than they do in the population at large, or in the population of females in the age group that are eligible to receive the vaccine, even if I assume that VAERS misses 80% of potentially reportable adverse events. I can’t say this proves the HPV vaccine does not cause some adverse reactions, any more than I can say the statistics show that any one adverse event IS caused by the vaccine. But it does suggest that the frequency of these events is so close to the statistical noise that it is very difficult to even identify outliers that could be a “smoking gun” of causality. Is there some other way in which one should analyze the VAERS data that clearly indicates a higher-than-expected (i.e. greater than random chance) incidence of these adverse events?

This is one of the favorite ploys of the data-miners.  Out of context, a set of data may be scary.  But if you compare the data with a population at large, then it is “noise”, meaning no statistical difference between it and just pure randomness.  As sad as it is, a certain number of people die after receiving the vaccine, just because there is some background death rate from all sorts of causes.  For example, it’s quite possible that there are 10 automobile deaths of teenagers who receive the vaccine (and given how many teenagers die from text messaging while driving, it might not be that much of a stretch); but no one is going to state that those deaths are attributable to the vaccine (though probably the text messaging).

In fact, some of the data from VAERS show side effects that are LOWER than what is found in the general population.  For example, Herper found that in the VAERS database one of the frequent adverse events reported for HPV vaccine is cervical dysplasia, a pre-cancerous condition also related to HPV infection.  A researcher then found that the rate of cervical dysplasia was orders of magnitude lower in individuals who have had vaccinations for flu, hepatitis A, hepatitis B, rabies and meningococcus.  We could conclude that those vaccines prevented cervical dysplasia, but that would be incredibly silly.  The point being is that these data can be misinterpreted either for or against vaccines.  But if we are to accept that the VAERS data were absolutely an indication of causality, then the anti-vaccine believers ought to be pushing everyone to get flu, hep A, hep B, rabies and meningococcus vaccines.  But of course, that’s not going to happen.

But the probable reason that VAERS shows low adverse events for cervical dysplasia, may be individuals misusing the VAERS online reporting system to spread fear and uncertainty about Gardasil.  For the same reason that online political polls are inaccurate and, oftentimes, gamed by one side or another, we should be skeptical of the data from VAERS.  In a real scientific study of adverse events, a team of medical professionals would investigate each report, determine if it is plausible that the vaccine was related, and then issue a peer-reviewed report.  For the CDC and FDA, that probably exceeds their budget, so they do the best they can.  For the anti-vaccine conspiracy, they would never support further investigation of the study, because they don’t accept real science.  Or they would just accuse the CDC and FDA of being tools of the pharmaceutical industry.  

It’s more useful to use lies, more lies and statistics to make a case than real science.  Just be skeptical to outlandish pseudoscientific claims.

Tags: confirmation bias | Gardasil | HPV vaccine | influenza | Matthew Herper | post hoc ergo propter hoc | vaxfax