I am a scientific skeptic. It means that I pursue published scientific evidence to support or refute a scientific or medical principle. I am not a cynic, often conflated with skepticism. I don’t have an opinion about these ideas. Scientific skepticism depends on the quality and quantity of evidence that supports a scientific idea. And examining the hierarchy of scientific evidence can be helpful in deciding what is good data and what is bad. What can be used to form a conclusion, and what is useless.
That’s how science is done. And I use the hierarchy of scientific evidence to weigh the quality along with the quantity of evidence in reaching a conclusion. I am generally offended by those who push pseudoscience – they generally try to find evidence that supports their predetermined beliefs. That’s not science, that’s the opposite of good science.
Unfortunately, today’s world of instant news, with memes and 140 character analyses flying across social media, can be overwhelming. Sometimes we create an internal false balance, assuming that headlines (often written to be clickbait) on one side are somehow equivalent to another side. So, we think there’s a scientific debate when there isn’t one.
I attempt to write detailed, thoughtful and nuanced articles about scientific ideas. I know they can be complex and long-winded, but I also know science is hard. It’s difficult. Sorry about that, but if it were so easy, everyone on the internet would be doing science. Unfortunately, there are too many people writing on the internet who think they are talking science, but they fail to differentiate between good and bad evidence.
But there is a way to make this easier. Not easy, just easier. This is my guide to amateur (and if I do a good job, professional) method to evaluating scientific research quality across the internet.
Continue reading “Hierarchy of scientific evidence – keys to scientific skepticism”
Late last year, I spent a couple of thousand words critiquing an awful anti-vaccine paper that attempted to claim that vaccinated kids were sicker than unvaccinated kids. Soon after the paper was published, a huge kerfuffle arose on Twitter, which pointed out the false conclusions, bad experimental design, and high levels of bias made by the authors. Within days of publishing the article, the journal retracted it (see Note 1).
Tell me if you’ve heard this before.
Not two days ago, I spent a couple of thousand words taking down an anti-vaccine paper that tried to claim that vaccinated kids were sicker than unvaccinated kids. Soon after the paper was published, the journal retracted it (see Note 2).
Oh wait, you had heard it before. The papers were the same one, where the primary author went shopping for journal after journal to get it published. Why so persistent? The primary author, Anthony R Mawson, from Jackson State University, has a long, established background of supporting the Andrew Wakefield along with anti-vaccine research. In 2011, Mawson had actually filed a lawsuit against the Mississippi State Department of Health in which he claims that there is a need for more studies on vaccines (including his own design, apparently), and that the Department of Health interfered with his getting a contract renewal at the University of Mississippi Medical center. The suit was dismissed in 2012.
Mawson actually has a Master’s and Doctorate in Public Health, specializing in epidemiology. You’d think he’d know better about vaccines, but I guess being a sycophant of Mr. Andrew Wakefield makes him deny his scientific background. Then, he goes out and invents a bad experimental design just to support his nonsense beliefs. And to be more endearing to Wakefield I suppose.
Now, you, my reader, are probably thinking, “OK, got it, let’s talk about GMOs.” Except, I have to drag you back in, and remind you that myths, tropes and memes in the anti-vaccine world are like zombies. They never quite die, biting one anti-vaccine cheerleader after another. Even a dead paper, like Mawson’s, will arise again somewhere.
As the curmudgeonly Orac just wrote,
One of the most frequent topics for posts like this is a pseudoscientific or just plain bad study that, despite being retracted, keeps rising from the grave, like the proverbial zombie. I call them, appropriately enough, zombie studies. Depending on my mood when I write posts like this, I often add imagery featuring zombies (or, if you’re into The Walking Dead, walkers).
I can’t resist a good Walking Dead metaphor.
Let’s look at this zombie/walker anti-vaccine paper. Because you just know, like the end of the zombie movie, the story drops hints for Part II. Or Part II. Or Part XXI.
How did an anti-vaccine paper get retracted twice?
The simple answer is that it was published twice in predatory journals, whose standard of publication is not much more than if the author can pay the fees to publish their paper. Mawson’s article was published for the first time in Frontiers in Public Health, a part of the Frontiers Media empire. It was published for the second time in the Journal of Translational Science, a part of OAT. Both Frontiers Media and OAT are predatory publishers according to the definitive Beall’s List of predatory publishers. A predatory publisher is an “exploitative open-access publishing business model that involves charging publication fees to authors without providing the editorial and publishing services associated with legitimate journals.”
In case you missed my articles about this before, let’s look at Frontiers Media and OAT:
- Frontiers in Public Health (FPH) is not indexed in PubMed. The Journal of Translational Science (JTS) is not indexed in PubMed. PubMed is one of the most valued resources in searching for biomedical articles, even in obscure journals. Without PubMed, it is extraordinarily difficult to find the article. Of course, there’s no reason given for it’s exclusion from PubMed, but we can speculate that possibly the science is bad.
- Both FPH and JTS lack an impact factor. The impact factor gives us a relative understanding of the quality of a journal – generally, the impact factor tells us approximately how many times an average article published in a journal is cited per year. A journal like Nature has an impact factor of 36, which means the average article is cited 36 times a year. Why is this important? We know that more pivotal and valuable research is cited many times and often repeated by other scientists, especially in new publications. This is how science develops solid evidence, by examining and repeating the findings of others.
- As we mentioned above, both FPH and JTS are published by predatory publishers (one of the possible reasons that the journals are not indexed).
- Low impact factor, predatory journals are generally dismissed by scientific researchers because the peer-review is weak. These journals are often abused by researchers whose data has been rejected by one or more of the respected journals, so they settle for these low quality versions. In the academic community, publications such as these often cannot be used as evidence of qualification for tenure.
Clearly, neither FPH or JTS were actually interested in what Mawson had written. If they were, either or both of them would have rejected the articles in the first place. But here we are, both articles were “published” (or whatever verbiage they use), and both were retracted (or scrubbed from the internet).
Remember, this anti-vaccine paper was garbage
Just in case you’re anti-vaccine, and you think that there’s some conspiracy against “evidence” that vaccines are dangerous, get over yourself. These two journals are awful, and it’s entertaining that Mawson’s article was so bad that it fell beneath the journal’s miserable standards.
Let me remind you of how terrible this article was (before being retracted twice).
- This study intended to “compare vaccinated and unvaccinated children on a broad range of health outcomes, and to determine whether an association found between vaccination and neurodevelopmental disorders (NDD), if any, remains significant after adjustment for other measured factors.” This is a good start, and if done well, it could provide us with more information about any potential links between vaccines and NDDs such as autism spectrum disorder. But this is not a well-designed study to answer that question.
- The study surveyed 415 mothers of 666 children educated at home. And the paper goes downhill quickly. First of all, we are relying on data from a group, which may or may not be biased versus a random sampling, that home schools children. There are lot of reasons why parents do this, but are they representative of the population at large? Second, it relies upon the memories and, frankly, biases of the mother being surveyed – it doesn’t rely upon data in actual medical files of each of these children, which can be considered to be somewhat “scientific.”
- Parents who do vaccinate may be less reluctant to visit their physician to get diagnosed for diseases, while a non-vaccinating parent may choose to ignore any health issues as minor. Thus, there may be an unintentional bias towards health care issues for vaccinated children.
- The groups (vaccinated vs. unvaccinated) are not randomized which will lead to observational and confirmation bias.
- The population for the study is quite small to tease out data on very rare conditions. A meta-review that included nearly 1.3 million children, using real medical records to establish both vaccination status and ASD status, found that there was no difference between vaccinated and unvaccinated groups with respect to incidence of autism.
- Questionnaires are not the best way to gather data about medical issues – in fact, the use of questionnaires for epidemiological studies is frowned upon by many researchers. The reasons for this are many, but they include a reliance upon the memory of the participant for events that may be not be recalled correctly. In addition, there might be intentional or unintentional bias of the survey participants towards relating a condition to vaccines or ignoring one in unvaccinated kids. Better done, and frankly, more expensive, epidemiological studies rely upon medical records to reduce errors in memory and bias. This is a critical point – the best studies that show no link between vaccines and autism rely upon actual medical data for each child, not on a questionnaire which may or may not be tracking accurate information.
- At best, this is observational data, which are not the type of data used to reject or accept an hypothesis. The CDC uses surveys to determine vaccination uptake, for example, but they used a non-biased methodology to gather data. They surveyed nearly 15,000 children, and were given permission to contact the health care provider to get more information.
- Mawson’s study found that 39% of children surveyed were not vaccinated. Now if we accepted this data without analysis, we’d think that the vaccination rate has dropped. However, that’s not what we’ve seen in other, better-designed surveys, such as the CDC’s massive ones, which showed that vaccine uptake was between 85 and 95% depending on the vaccine and the area. With such a difference between this study and better designed ones, one would have to believe that the study with a much smaller number and an apparently biased population would be the outlier.
- There was a lack of analysis of potential cofounders which might bias the results. Were there other differences between he vaccinated and non-vaccinated groups that could have biased the results one way or another?
- The use of odds ratio by the researchers is curious and may have biased the results. The only reason to use OR is to inflate the difference between both groups. Real epidemiologists prefer relative risk (RR) which provides us with a much more sensitive understanding of the incidence of a condition in a vaccinated group vs. an unvaccinated group. For example, the RR for ASD from this study (if the data actually were useful, which it isn’t) is 3.1, whereas the OR is 4.2.
- The over reliance on p-values is always a red flag to researchers. With so many p-values, the probability of false positives is fairly large. Because the population for the study was so small, it’s also worth nothing that the OR had relatively large error brackets, most of which crossed near unity, that is, 1.0, indicating no difference between the groups.
- The authors seem to overlook or ignore the point that the vaccinated children had much lower risk of diseases like chickenpox and measles, which we know bring with them significant risks of long-term health issues. Even if we accept this poorly designed study as providing us with evidence of harm from vaccines, one would have to balance it against the benefits of avoiding vaccine preventable diseases. Of course, this study is so bad, it’s hard to accept anything useful from it.
There’s more, I’m sure. I just get tired of tearing apart these bogus articles, when it’s really hard to get over the fact that it’s been retracted twice by awful predatory publishers. Seriously, that should be the end of the story.
As the crotchety Orac wrote recently about Mawson’s article,
I’ll conclude by pointing out yet again that it is a myth that there are no studies comparing the health of vaccinated children compared to unvaccinated children. In fact, there have been several. It turns out that they don’t show what antivaxers think a vaxed/unvaxed study will show. Basically, all of the vaxed/unvaxed studies not done by antivaccine-friendly scientists or quacks have shown either no differences in the prevalence of neurodevelopmental or chronic diseases between vaccinated children and unvaccinated children or have actually found better health outcomes in the vaccinated population. Mawson concludes by arguing that further “research involving larger, independent samples and stronger research designs is needed to verify and understand these unexpected findings in order to optimize the impact of vaccines on children’s health.” Mawson’s study is so biased, flawed, and incompetently carried out and analyzed that its results can be discounted as almost certainly worthless. It doesn’t provide the rationale for “more studies.” Quite the contrary.
Yet, that’s how antivaxers are spinning it, as they always do.
Same as it ever was.
There really are plenty of good articles, published in real, indexed, high impact factor journals that show that there is little or no difference between the health of vaccinated versus unvaccinated children. We should also include that nagging point that vaccinated children don’t generally catch deadly, disabling vaccine preventable diseases.
We also have boatloads of evidence, again, published in major journals, that dismiss any link between vaccines and autism, which feels like the basis of Mawson’s junk article. So, if you want to use that rubbish anti-vaccine article as the basis of your “evidence” that vaccines are dangerous – well, you really need something more convincing. Mawson’s article was retracted twice by the detritus of predatory publishers, so it would be laughable to use it as real evidence.
But we all know that this zombie article will rise again. They eat brains, and apparently they’re looking for nutritious anti-vaxxer brains. But zombies aren’t real, except on TV and movies. So the anti-vaccine world keeps pushing zombie tales – because they think that by Part XXI, we will be bored with the zombie myth and not try to take it down. Don’t count on it.
- Actually Frontiers Media noted that the study was only “provisionally accepted but not published.” That might be what they think, but the article appeared on their website, then disappeared.
- Technically, we can’t tell that the article was retracted. It’s just been scrubbed from JTS’ website, as if it never existed. But the internet never forgets.
In my writing, I often refer to the scientific consensus, which is the collective opinion and judgement of scientists in a particular field of study. This consensus implies general agreement, though disagreement is limited and generally insignificant.
The major difference between a scientific theory and a scientific consensus is that the theory is essentially fact. It is so predictive, it is supported by so much evidence, and it is so well accepted, it takes an almost ridiculous amount of data to refute it, though it is possible.
In the hierarchy of scientific principles, we often mention scientific theories which “are large bodies of work that are a culmination or a composite of the products of many contributors over time and are substantiated by vast bodies of converging evidence. They unify and synchronize the scientific community’s view and approach to a particular scientific field.” A scientific theory is not a wild and arbitrary guess, but it is built upon a foundation of scientific knowledge that itself is based on evidence accumulated from data that resulted from scientific experimentation.
We want to focus on the scientific consensus, describing what it is. Take a deep breath, because this is a complicated one.
Continue reading “Developing and supporting a scientific consensus”
Here we go again. An anti-vaccine article was published in a journal, and now every vaccine denier will use it as absolute “proof” that vaccines are evil, bad, and useless. And that means one of the pro-science community has to provide a critical analysis so that those on the fence know what is supported and not supported by real science.
The article, “New Quality-Control Investigations on Vaccines: Micro- and Nanocontamination,” published in the International Journal of Vaccines and Vaccination on 23 January 2017. I want to examine this from the meta level, discussing the quality of the journal, down to the actual data. Spoiler alert – it’s bad. Continue reading “Another anti-vaccine article – bad journal, bad data”
The ignorance of the anti-vaccine cult is so annoying, because I have to spend valuable time trying to clear up their myths, tropes, and outright lies. An anti-science website, Health Impact News, recently published an article discussing a flu vaccine study from 2012. I don’t know why it took them this long to use this as an “anti-vaccine” study, but I don’t think keeping up with primary research is high on their list of goals.
I’m going to look at a few of the points the anti-vaccine crowd tried to make, refuting them one by one, by looking at the original article published in the relatively high impact factor journal, Clinical Infectious Diseases. The TL;DR version is that they didn’t get anything right. so, here we go into the fray.
Continue reading “Flu vaccine study does not show it causes 5.5X more respiratory infections”
This article has been updated and can be found here. The comments for this article have been closed permanently.
I could have a full-time job just debunking the rumors and myths about the HPV cancer-preventing vaccine, Gardasil. I’d bet one year of my Big Pharma Shill Income™ that the anti-vaccination gangsters make up more junk science about Gardasil than all other vaccines put together. And now, bogus claims that Gardasil causes behavior issues – time for a critical analysis.
This new claim about Gardasil arises from an article, “Behavioral abnormalities in young female mice following administration of aluminum adjuvants and the human papillomavirus (HPV) vaccine Gardasil,” published in the well respected, relatively high impact factor, journal Vaccine. When it was first published, my thoughts were that the editors of Vaccine missed something. Given that it’s been “temporarily removed,” I guess they did.
But let’s look at this claim with our critical thinking skills, which most of the readers here have.
Continue reading “Gardasil causes behavioral issues – more myth debunking”
This article has been substantially updated, and can be read here. Please read and comment at the newer article.
One of the most tiresome discussions that a scientific skeptic has when debunking and refuting pseudoscience or junk science (slightly different variations of the same theme) is what constitutes real evidence. You’d think that would be easy, “scientific evidence” should be the gold standard, but really, there is a range of evidence from garbage to convincing.
So this is my guide to amateur (and if I do a good job, professional) method to evaluating scientific research quality across the internet. This is a major update of my original article on this topic, with less emphasis on Wikipedia, and more detail about scientific authority and hierarchy of evidence.
In today’s world of instant news, with memes and 140 character analyses flying across social media pretending to present knowledge in a manner that makes it appear authoritative. Even detailed, 2000 word articles that I write are often considered to be too long, and people only read the title or the concluding paragraph. This happens all the time in the amateur science circles specifically. For example, many people only read the abstract and, even there, only the conclusion of the abstract for scientific articles.
Continue reading “Evaluating scientific research quality for better skeptical analysis”
The science deniers of the world, whether they deny evolution, global warming, vaccines, or GMO safety, spend their time inventing pseudoscience to support their beliefs and claims. As I have written previously, “Pseudoscience is easy. It doesn’t take work. It’s the lazy man’s (or woman’s) “science.” But it has no value, and because it lacks high quality evidence in support of it, it should be dismissed, and it should not be a part of the conversation.”
Alternatively, real science is really hard. And it takes time. And it’s based on high quality evidence. And it is repeated. And it is almost always published in high quality journals. As I’ve said a thousand times, real science takes hard work and is intellectually challenging. You just don’t wake up one day and say “I’m a scientist.” No, it requires college, graduate school, teaching, working in world class laboratories, publishing, defending your ideas to your peers, and one day, if you don’t stop, you will be an authority in your little field of science.
The anti-GMO crowd is mostly lazy. They have this luddite belief that all technology is bad, but have absolutely no evidence to support it. Sure, they pick out one or two poorly done articles and then shout for all the world to hear “GMO’s are dangerous to…bees, humans, babies, whales, trees” over and over and over again. Yet what do the GMO refusers really bring to the table? Continue reading “What does science say about GMO’s–they’re safe”
Here we go again. “Researchers” trying to show that a pseudoscientific concept is real medicine, but failing so badly that only true believers would qualify it as real “evidence.”
In this case, homeopaths from the Department of Homeopathy at the University of Johannesburg (seriously, a Department of Homeopathy?) in South Africa recently published a study that claimed a concoction of homeopathic potions, in pill form, treats tonsillitis, an infection and inflammation of a set of lymph nodes called tonsils in the back of the throat, better than a placebo. The researchers concluded that “the homeopathic complex used in this study exhibited significant anti-inflammatory and pain-relieving qualities in children with acute viral tonsillitis.”
Convincing conclusion. That’s it, next time my children have tonsillitis, I’m going to run down to my local homeopathic lotion and potion magician, and I’ll buy out the store.
Or maybe not. I’ll probably save my money from lining the pockets of that homeopathic wizard, and I’ll send the kids to a real physician who practices evidence-based medicine. And get real treatment.
Why am I so negative about a real peer-reviewed study published in a medical journal? For lots of reasons. Continue reading “Study concludes that homeopathy cures tonsillitis–probably not”