Since I started this website, nearly four years ago (that’s 50 internet years), I’ve noticed a serious problem – some skeptics, even scientific skeptics, are lazy, trying to take the easiest way to accept or refute a claim. It is difficult to be a real scientific skeptic (see Note 1). It’s not impossible, but it takes more than using some intentional or unintentional bias or fallacy.
This lack of real skepticism has manifested itself in some incredible meta memes: “vaccines are dangerous,” “gluten is dangerous,” and “GMOs are dangerous,” even in groups who are ostensibly “pro-science.” However, if I replied to them, “wow, you must be a creationist too,” generally, their indignant reply would make a sailor blush.
Scientific skepticism is hard, not because of the complex science (even though, that is part of the issue), it’s because searching for the evidence that supports or refutes some claim is often nuanced, and contradictory. And researching it isn’t easy.
Thus, as a self-proclaimed scientific skeptic, I thought I’d write an article to help anyone learn more about what interests them, how to discriminate between bad and good research, and where to find good information. Sit down with your favorite internet consuming device, grab your favorite snack and drink, and enjoy.
Let’s define a real scientific skeptic
This is an important step to being a real skeptic. As I’ve written before, the common definition of skeptic is nothing more than a cynic – a person inclined to question or doubt all accepted opinions. That’s not very informed nor very scientific. These kinds of skeptics just doubt everything, whether that doubt is backed by evidence, or the lack thereof.
In ordinary usage, this type of skepticism has one of three meanings:
- an attitude of doubt or a disposition to incredulity either in general or toward a particular object;
- the doctrine that true knowledge or knowledge in a particular area is uncertain; or
- the method of suspended judgment, systematic doubt, or criticism that is characteristic of skeptics.
Many scientific skeptics, myself included, get accused of not being a real skeptic, because we refuse to question everything. That’s just not how it works.
And it is much much more than that.
Scientific skepticism is the noble pursuit and accumulation of evidence, based on the scientific method, which is used to question and doubt claims and assertions. A scientific skeptic will hold the accumulation of evidence as fundamentally critical to the examination of claims. Moreover, a true scientific skeptic does not accept all evidence as being equal in quality, but, in fact, will give more weight to evidence which is derived from the scientific method and less weight to poorly obtained and poorly scrutinized evidence.
But, and this is a key point, a real scientific skeptic doesn’t doubt a claim forever. If the quality and quantity of evidence supports the claim, a valid skeptic will accept the claim. Until such point, better and more evidence appears, then the process starts all over.
Scientific skepticism is absolutely not dogmatic. It generally comes to a provisional conclusion, subject to change with new data. For example, I generally say “vaccines are relatively safe and effective.” An unsophisticated, who lacks real science knowledge, would interpret that as “you mean it’s 100% unsafe, and 100% ineffective.”
No, what I really mean is that absent a lot of high quality contradictory evidence, vaccines are safe, probably safer than most medical procedures, and its effectiveness in preventing disease is so high, that the cost of the vaccine (in terms of economic and possible adverse events) is far far lower than the costs of vaccine preventable diseases. I know, it doesn’t roll off the tongue (or keyboard), but it is factually supported by mountains of evidence.
Pseudoskepticism
The philosophical and scientific opposite of real scientific skepticism is pseudoskepticism. These individuals pretend to be skeptics but are really pseudoskeptics. They pretend to use scientific skepticism, but in reality they wouldn’t change their minds with any amount of evidence.
It is useful to note that these pseudoskeptics irrationally attack any evidence that rejects their preordained beliefs, but refuse to critically review any evidence that they think supports their point of view. They cherry pick evidence, seemingly from top level sources, to support their pre-existing bias.
Let’s be clear. Pseudoskeptics are nothing more than deniers – individuals who deny the scientific consensus for any number of reasons: conspiracies, ignorance, belief in a priori conclusions, political expediency, and probably another few dozen reasons.
A climate change “skeptic” is not a skeptic. They are climate change deniers. Period. They wouldn’t know real scientific skepticism if it proclaimed true love and wanted to marry them.
A vaccine skeptic is a vaccine denier. Again, period, no further discussion needed. Vaccine deniers rely on a handful of badly designed (or let’s face it, retracted) published articles, anecdotes, and internet websites whose idea of sourcing their information would have them flunked out of biology 101 at almost every university in the world.
So if you base your “skepticism” without mountains of high quality evidence replacing it with logical fallacies – pick up that denier card and hold it out for all to see.
The hard way to be a scientific skeptic
You have been warned, more than once. It is extremely difficult to be a real scientific evidence. It’s not easy. Therefore, the list of things you need to do to move beyond a cynic, and become a real skeptic, is long and complicated.
But most of my readers are smart and good thinkers, regardless of education or career. Well, there are a few deniers that troll around here, but we all know how to refute them.
So here’s some of the ways you can go about being a proper scientific skeptic:
1 – Published scientific evidence
The most important information you can use to refute or confirm a claim is to look for evidence that is actually published and peer reviewed.
Oh, but if it were just that easy. Because not all published evidence is equal. You actually have to review the data, review the methods, and review the authors. So-called skeptics who throw one published paper in my face, almost always get mocked and ridiculed.
There is a hierarchy of published evidence, from becoming the basis of all scientific-based medicine to being a long way to becoming a part of the medical discussion.
At the top of the list are systematic reviews (more specifically, meta-analyses). A systematic review is a critical assessment and evaluation of all research studies that address the effect of pharmaceuticals, devices, standards of care, and other issues related to a particular clinical condition. Researcher who produce systematic reviews use an organized method of collecting and analyzing a body of literature on a particular topic using a set of specific criteria.
A systematic review may examine the quality of research in each of the papers, describe the results qualitatively, and find bias and errors.
Although I, and others, consider systematic reviews to be the pinnacle of scientific research, it is not perfect. Cochrane publishes some of the most important systematic reviews, but they have occasionally published some awful ones. For example, one of their articles relied upon authors who have axes to grind about their personal beliefs. The worst thing that can happen is when biased researchers have a predetermined conclusion, then use a systematic review to confirm that conclusion.
So those of you who jump right to a meta-analysis, and then loudly proclaim “this is the truth.” Take a deep breath and wait a minute. It has to be reviewed for bias and expertise of the authors. But the vast majority of the time, systematic reviews are the pinnacle of great science and often are the basis of a scientific consensus, but that does not mean that they get to skate by without a critique.
Published scientific evidence takes precedence over everything else – over websites, government documents, news articles, press releases, anecdotes from your mother, or whatever else.
2 – Not all research matters
Here’s an important issue to remember – most research is a failure. That’s why quality AND quantity matter. Too often, I see a meme or trope tries to convince me that marijuana cures cancer. Well, there is no clinical evidence that smoking cannabis cures any cancer. The memes that marijuana has any positive health effect is based on animal studies – which less than 10% of the time end up being clinically important.
On Facebook, I see so-called “skeptics” pushing some news release from a university (even good ones) that eating XYZ will cure cancer. Setting aside the fact that there are over 250 cancers, all of them with different pathophysiology, if you read the original research, it’ll state that “blueberries may have an anti-cancer effect. More research is necessary to confirm if this has any clinical applicability.”
Yeah, that. There are years between preliminary results and clinical trials, and until you see published clinical trials, in great numbers, be scientifically skeptical.
3 – Really, not all research matters
The quality of journal in which an article published matters. If you find a study published in Medical Hypotheses, a journal that lacks credibility, lacks peer review, and is probably predatory, one has to dismiss it.
It’s easy to check the quality. Look at its impact factor, although not a perfect tool, does give an indication, broadly, of how respected a journal is. If the impact factor is above 3-5 (Nature is in the 30s), it’s probably pretty good, though an impact factor greater than 10 is the goal of most research scientists.
It’s also easy to see if the article is the basis of a sea-change in the scientific consensus. If you look at the PubMed (we’ll talk about it more) listing of an article, along the right hand side is a current listing of all other articles that cite the one you are examining:
The article I was checking, “Long-term persistence of vaccine-derived aluminum hydroxide is associated with chronic cognitive dysfunction,” an anti-vaccine article oft-quoted by vaccine deniers, was cited eight times in six years. What does that mean? Not much, because half of the cites were from other anti-vaccination “researchers,” who published it in really bad journals.
And more than that, many of the articles were just citing this article within their own discussion for much better research.
4 – I’m serious, not all research matters
If you’re trying to be a scientific skeptic, and you’ve followed my advice, then one last issue matters – the quality of the researchers (and the author). I am not saying that the appeal to false authority should override anything else.
But if Neil deGrasse Tyson publishes an article that he has evidence of aliens living on Pluto (not a planet), I’m going to take that with a lot more credibility than if Donald Trump said that there isn’t.
I’ve criticized so-called meta-analyses of vaccines because the author was biased, and showed extreme bias in the analysis. Others have criticized published systematic reviews of acupuncture, where most of the reviews were just wrong, and highly biased. And the authors affiliations betrayed that bias.
5 – Logical fallacies are just wrong
Any evidence that relies upon any logical fallacy is not evidence. Period. Logical fallacies exist because the claimant lacks real evidence.
Accusing a scientist of being a paid shill of Big Pharma, Big Agra, or some mysterious worldwide conspiracy is a logical fallacy. Refuting their science, especially if it is part of a huge scientific consensus, requires more scientific evidence than what supports the consensus.
Choosing published articles that support your point-of-view while ignoring the vast majority of evidence that support the opposite is a logical fallacy.
Picking one or two articles and then claiming it represents the best scientific information about a claim is a logical fallacy.
Avoiding logical fallacies are easy. I’ve published a list of my favorites, but there are many more.
6 – Plausibility matters
Scientific plausibility is one of the foundations of good science. If you’re saying you have the miracle food that cures all cancer, I’m going to be highly skeptical. Why? Because it’s implausible that there can be “one food” that can “cure” (lots of scare quotes) all 200 or so cancers. Each of these cancers are different cells with different causes and different pathophysiology. It’s inconceivable (yes, I know what that word means).
Also Occam’s razor should be a principle of scientific skepticism. If someone making a claim is making an incredible amount of assumptions, sometimes called the Gish Gallop, most of which are not plausible themselves, or twisting the logic into a series of pretzels to get to the claim, it’s hard to take it seriously.
7 – The scientific consensus matters
The scientific consensus is one of the most powerful statements on scientific facts that we have. A scientific theory is the pinnacle of scientific principles, they represent essentially a fact. The theory of evolution is a fact – it would take millions, literally, pieces of published evidence before anyone could move off of that fact.
A scientific consensus is one step in the hierarchy of scientific principles, a critical step.
The scientific consensus is the collective opinion and judgement of scientists in a particular field of study. This consensus implies general agreement, though disagreement is limited and generally insignificant.
Dismissing the consensus by inventing conspiracies, claiming that your Google search invalidates the consensus, or referring to some political expedient point. Those are just boring.
The consensus is always based on the preponderance of evidence, again, quality and quantity matter. It does not come as a result of a formal vote, but it can, especially when a majority of the experts in the field agree. A consensus generally is part of the development of important scientific principles.
The scientific consensus, if it exists for a particular biomedical topic, is your starting point. I defer to the the hundreds or thousands of scientific experts who developed that consensus.
If you think they were all bribed to come to that consensus, please show it with court documents or other evidence that your conspiracy theory is right. Referring to Natural News as your source is bogus.
And if you really want to overturn the scientific consensus then bring it. And by it, I mean mountains of evidence from many diverse research centers published in real journals written by other qualified researchers. Then you have an argument. Then you can change the consensus. Otherwise, you’ve got nothing. I mean it, you’ve got nothing.
I can’t say this enough – evidence matters.
Convinced? Where to find evidence.
At this point, I’m assuming the reader is now, officially, willing to be a scientific skeptic. You are now willing to look at all of the evidence, trusting sources that provide the best evidence, weighing the quality and quantity of evidence before forming a conclusion that refutes or accepts a claim. It’s not easy, but if you’re going to make a decision that effects the lives of children, then you damn well better have the highest quality and highest quantity of evidence
1 – Beware of satire
I just want to remind everyone that fake new and satire websites are really funny and really well done. Just make sure you don’t confuse an article in the Onion with a real scientific article. Of course, sometimes their satire is more scientifically based than the junk science pushed by Natural News and their lying ilk.
If you’re going to use a satire site as your primary source of a claim, you might just be laughed off the internet.
2 – Your first stops for information
Although accepting or refuting a claim should require the reader to read through 200 published articles, there are many websites that can get you started in the right direction, many written by experts with hundreds of man (and woman)-years of total experience in biomedical and scientific fields of study.
Snopes – I think that Snopes has become a verb in skeptic-talk. “I snoped that claim, and it’s wrong.” I’d almost recommend going to Snopes before anything else, because they really refute the most popular science and medicine myths. Although Snopes isn’t strictly about science, they have a lot of it there. They actually reference this website to debunk myths about Gardasil.
What Was Fake on the Internet This Week – Published by the Washington Post, which has a good record for science journalism, it covers more than science and medicine, but it debunks a lot of the silly stuff.
Science-Based Medicine (SBM)– This is my favorite website to start research on a topic. They have one major advantage that I don’t – they have a team of researchers in a large number of fields, so they cover topics that I don’t have the time or expertise to explore. So if I want to read up on the junk science on acupuncture, I start at SBM. Their search function is robust, so if you type in a few keywords, you’ll get a list of articles that should help the reader debunk some pseudoscience.
Bad Astronomy – This website is similar to SBM, although it generally goes beyond debunking junk biomedical science (hence the name). If you want to find out if scientists really uncovered an ancient alien civilization out there in space, you should start at this website. My only complaint is that it’s hosted on Slate, so there’s a lot of intrusive advertising.
Bad Science – This website is run by Ben Goldacre, a strong critic of both pseudoscience and the pharmaceutical industry. There is no way anyone could claim that Dr. Goldacre is a Big Pharma shill, because he goes after any research that doesn’t meet his exacting standards. He doesn’t quite say it, but I get the impression that if it’s not a meta-analysis, it’s worthless. I tend to agree. By the way, Dr. Goldacre is solidly pro-vaccine – ironic that a brilliant physician like him is both against Big Pharma and pro-vaccine.
SciCheck – this website really goes after the junk science spouted off by our politicians. Given that politicians on both the left and right haven’t a clue about real science, they must have a staff of 10,000 researchers just keeping up with Republican climate change denial.
Quackwatch – I’m not absolutely sure, but I think that Quackwatch is one of the oldest websites on the internet, starting up in January 1997. That’s like 10,000 internet years. And they use simple coding, no advertising, so you get what you want quickly. Quackwatch debunks everything from acupuncture to junk science about water.
Skeptical Science – If you want to quickly and easily debunk any claims about climate change, this should be your go-to website. But it’s more than just climate, it frequently has tremendous, detailed articles about related science, anything from evolution to geology. I actually read it for pleasure. Yeah, I know, strange.
Health News Review – This is like the Snopes of biomedical news. They review articles, mostly on websites or the news (or I suppose they can be both), for both quality and veracity of information presented. So if you read an article in a respected newspaper that claims drinking wine could help your type 2 diabetes, treat with a great amount of skepticism.
3 – Be brave. Read published research.
Yes, I know that most published research seems unintelligible, with technical terms that make it difficult to read. Outside of a few areas of my specific field of research, I require a medical dictionary app live at all times to read most articles. This slows down my reading speed to a crawl, but I muscle through it. You can too.
Many people just read the abstract of an article. That mostly works, but reading the full article can tell you so many things – the quality of statistics, the number of test subjects, the potential conflicts of interest, the plausibility of the conclusions, and so much else. But if it’s just reading the abstract, which are mostly eminently readable, that’s better than reading junk science websites like GreenMedInfo, which passes off misinformation and logical fallacies as real science.
But again, it requires open-mindedness to be a real scientific skeptic. I know I’m getting repetitive, but the only thing that matters is the quality and quantity of evidence. Open-mindedness isn’t claiming “this article supports my claim,” it’s “out of 100 articles on this claim, 98 support it.” It takes work.
So here are some tools to help you.
Trip database – This website is almost as old as Quackwatch, and it’s just as valuable. If you search some topic in evidence-based medicine, it sends back all of the research on that topic. However, it does one thing different – it tags the article with a useful tool to categorize that article.
For example, an article may be tagged as “primary research.” In the hierarchy of research studies, primary research is not very important. Why? Because primary research has generally not been confirmed by a other researchers – we cannot be certain that it has any meaningful clinical data attached to it.
On the other hand, the database tags certain articles as “systematic reviews,” which can be useful in finding the best of the best. The Trip database should be one of your key tools in researching medicine.
So, if you’re trying to find out if acupuncture works, you’ll find data that supports the conclusion that it probably doesn’t. Just in case you were wondering.
PubMed – This is an NIH administered database for biomedical research. It links to published research across the world, without any commentary or review. It’s just there.
The search function is very robust, so you can enter fairly specific terms to get what you want. For example, I can enter “HPV vaccine adverse effect” in the search, and receive back 62 hits. Just in case you are wondering, nearly all of the 62 said “HPV vaccine is relatively safe.” But if you’re a vaccine denier, you probably are going to look for the two that might not actually say that.
There are a couple of issues with PubMed. It really does stick with biomedical research, so if you’re researching “dinosaurs and asteroids,” there will be few hits, unless the journal is indexed in PubMed (and many broad scientific journals are, like PNAS). The other issue is that if you’re really interested in a specific topic trying to ascertain the quality and quantity of published research, you’re going to have a lot of reading. Again, being a scientific skeptic is hard work.
One more thing – this is not Google, it doesn’t rank the papers by popularity, or clicks or anything. It ranks them in reverse order of publication, nothing more, nothing less (though you can change the parameters easily). So, the 100th article in the list may or may not be as important as the first one. If you only read the first 5 articles, you may be cherry-picking, unintentionally. The 37th article may be the most important, most relevant one, but if you only scan the first few hits, you’ll be missing the actual scientific consensus.
Did I mention that being a skeptic is really hard work?
Cochrane Library – I am a big fan of Cochrane, which publishes systematic reviews of medical research. They generally roll up clinical research from a large number of studies, look for biases, review quality, analyze the data, and come to a conclusion.
From a skeptical standpoint, they occasionally produce some head-scratching studies, that make no sense. Some of their published research can be highly biased, with some authors trying to establish a conclusion by eliminating data that doesn’t conform to their preconceived conclusions, or using occasional logical fallacies to decide which is good or bad data. I would hope their peer-review would eliminate this issue, but I tend to review their review before I refer to it.
Google scholar – Run by Google itself, it is a valuable resource to find articles on a particular scientific topic. Most of my resources are biomedical-focused, so if I wanted to research “climate change” I need a different kind of database. Google scholar meets that need, but it has limitations.
The biggest problem is that it’s default response is to list the articles in order of “relevance.” So that’s dependent on Google’s algorithms to determine which is the best article, some of which is based on its popularity. That causes an unintentional cherry-picking that is useless to real science. Just because a scientific article is “popular,” doesn’t mean it is actually relevant to your research.
The second problem is that we tend to use the results like we do with any Google search. We quit reading after about 10 results (my default Google search returns 100 hits). But if the 11th article is the most important, you’ll miss that.
Everything else – The Committee for Skeptical Inquiry, a group that promotes scientific skepticism across all fields of science, published this article recently which lists a huge number of resources, some of which I also use, that can be useful to you.
For example, they include ways to debunk images, something that is not a goal of this website. If a website isn’t included in their list, it doesn’t exist. Wait. They didn’t include mine. Well, that hurts.
The real scientific skeptic
Well, I just wrote over 4000 words trying to provide help in being a good skeptic. So, here are the key points:
- Scientific skepticism requires you to find the best quality and largest quantity of published evidence to support or refute a claim.
- Quality and quantity of evidence is all that matters.
- Get all of your sources. Make sure it’s not satire. Make sure it’s secondary, not primary research. Make sure you look at everything.
- Use well-established scientific websites as your source. Ignore the junk science pushing ones.
- And most importantly, it’s difficult to be a real scientific skeptic. I’ve given you some information to make it somewhat easier, but that’s relative.
- And the Princess Bride is the most important movie of all time for skeptics. So sayeth me.
Notes
- Being an American, I’ll stick with our spelling, skeptic. For those English speakers outside of the USA, “sceptic” is the accepted spelling. Blame Noah Webster.
[wp_ad_camp_5]
