Since the start of the COVID-19 pandemic, I’ve noticed an epidemic of cherry-picking by people trying to prove this or that about face masks, vaccines, treatments, and mortality. If you don’t spend a lot of time reading the scientific literature on these points, you’d think that there was some sort of scientific debate on everything to do with COVID-19.

Even though some people will try to show that science is all over the place about this pandemic, it really isn’t. We know that facemasks worked, and probably helped reduce the infection rate. And it helped crush the seasonal flu across the world.

We know that the COVID-19 vaccines are very safe and very effective.

We know that all kinds of treatments don’t work from hydroxychloroquine to ivermectin to quack remedies from internet grifters.

And we know that the CDC isn’t intentionally inventing mortality numbers because of…reasons!

So, why does it seem like there are scientific debates about all of these? It’s because we seem to be in a world of false equivalence where cherry-picking one “science” article, irrespective of its merits, can “prove” a contradictory point. But this is not how science is done.

Not to be repetitive, but real science requires one to review all of the published evidence, giving more weight to published studies in respected journals, written by respected scientists, using respected methodologies and analyses, with respected conclusions. It is absolutely not cherry-picking those studies, irrespective of their quality (and they usually have no quality), just to support one’s pre-ordained conclusions. That’s pseudoscience.

I hate cherry-picking unless it’s gathering that delicious fruit. I can get behind that kind of cherry-picking.

What is cherry-picking?

Cherry-picking refers to the selective presentation of evidence in an argument in order to refute or affirm a point while ignoring other evidence which will not support the point(s) being made. It derives from the obvious reluctance to harvest unripe, or overripe, fruit and to select only those which you will consume.

Often, cherry-picked factoids or references will be over-extrapolated and oversold to give the impression that they are representative when they are not.

Cherry-picking often relies upon anecdotal evidence, because it only uses one or two examples to make a point. On the other hand, statistical cherry-picking essentially use larger-scale anecdotes, by ignoring the broader evidence on an issue.

Often, cherry-picked factoids or references will be over-extrapolated and oversold to give the impression that they are representative when they are not.

COVID-19 vaccine facts and myths – UPDATED info about the new vaccines

Why is cherry-picking bad?

Again, real science requires one to look at all of the evidence then generate a conclusion from it. Cherry-picking is always choosing the evidence that supports the pre-determined conclusion while ignore any contradictory evidence.

You might think that this is an acceptable way to frame a scientific argument, but it’s far from it.

It’s ironic how, for example, anti-vaccine activists will criticize every single paper that refutes the belief that vaccines cause autism, but they will jump on a retracted article that claims that two people will die from the COVID-19 vaccine for every three lives saved. In fact, the anti-vaccine crowd loves retracted papers.

Of course, retracted papers by journals is one of the methods by which science self-corrects. Unfortunately, retractions often take time (although the recent article referred to above was retracted within hours).

If only there was a logical way to examine the quality of papers.

So, how do I give weight to a paper?

One of the best methods is the so-called SMELL test. This test was created to help students figure out how to separate good from bad science (or fact from fiction): 

• S stands for Source. Who is providing the information?   
• M is for Motivation. Why are they telling me this?
• E represents Evidence. What evidence is provided for generalizations?
• L is for Logic. Do the facts logically compel the conclusions?
• L is for Left out. What’s missing that might change our interpretation of the information?

For example, I start with the source. Are the authors respected? Have they published a lot of papers in the field? Do they have credentials that support whether they are authoritative? Is the journal respected, or is it a low quality predatory journal?

I also examine the quality of the evidence. Are the methods sound? Is the statistical analysis strong? Does the discussion including citations to other respected articles in the field.

Yes, this is hard work. That’s what frustrates me about the anti-vaccine crowd – they look at the abstract on PubMed and take the one-line conclusion as fact. However, if you actually read the article you might find out their materials and methods are amateurish or laughably incompetent. Or you might look at the statistical analysis and find p-hacking.

Cherry-picking is bad enough, but cherry-picking a bad article to establish a conclusion is the worst.

I want to improve my cherry-picking

Ok, I will admit I cherry-pick. Yes, I do. Except, I really don’t.

I use the hierarchy of scientific evidence, focusing on research that’s near the top, and pretty much ignoring anything below that.

So, here’s the list of types of published scientific research from the best to the worst:

Systematic reviews or meta-analyses.

A systematic review may examine the quality of research in each of the papers, describe the results qualitatively, and find bias and errors. A published systematic review usually includes a description of the findings of the collection of research studies. Many systematic reviews also include a quantitative pooling of data, which is called a meta-analysis.

All meta-analyses are systematic reviews, but not all systematic reviews contain meta-analyses. These reviews do all of the hard work and sum up the good, the bad, and the ugly of research for a particular topic.

If I find a good systematic review, I’m nearly done with my work.

Systematic reviews are the pinnacle of great biomedical science, the top of the hierarchy of scientific evidence, and often are the basis of a scientific consensus, but that does not mean that they get to skate by without a critique.

Large clinical trials

These are randomized clinical trials that include fairly large numbers (in general, I like to see >1,000 subjects in each arm), with confidence intervals (errors) that do not overlap and show a clinically significant effect. The results are definitive and are published in high-quality journals.

Cohort studies (retrospective studies)

These studies utilize one or more samples (called cohorts) which are followed prospectively over time. Subsequent status evaluations with respect to a disease or outcome are conducted to determine which initial participants’ exposure characteristics (risk factors) are associated with it.

In more simple terms, the researchers follow one group who may be exposed to something and compare them to a group that does not. From this data, the study can tell us what the absolute risk may be from exposure to certain diseases.  A cohort study is often undertaken to obtain evidence to try to refute the existence of a suspected association between cause and effect.

Case-control studies

This is a type of analytical study which compares individuals who have a specific disease (“cases”) with a group of individuals without the disease (“controls”). The proportion of each group having a history of a particular exposure or characteristic of interest is then compared. An association between hypothesized exposure and the disease being studied will be reflected in a greater proportion of the cases being exposed to the factor of interest.

For example, one could take a group of individuals that have lung cancer and compare them to a group that does not. The researchers would pick a hypothesized factor, say smoking, and determine the rate of smoking in each group. From that data, the researchers could get a determination of the differences in risk between smokers and non-smokers for lung cancer.

Everything else

What I won’t do is cherry-pick any of the following:

• Cross-sectional studies. Nothing more than surveys that are filled with biases and lack of control for confounding factors. Plus, they usually rely upon the memories of participants which is just another route for bias.
• Animal or cell culture studies. Though they can be interesting, less than 1% of these “pre-clinical studies” ever end up having clinical importance. I make this sincere joke about animal studies – call me when the phase 3 clinical trials are completed and published. Seriously, these studies have little relevance except to help formulate a hypothesis about a new drug.
• Case reports. These are abused by cherry-pickers because they are often published in respected journals and are peer-reviewed. However, they are just observations and may have no relevance. If I had my way, I’d get rid of them, but they serve a purpose – they can give warnings to other physicians about potential issues that arise.
• Anecdotes. Let’s make this clear – anecdotes ≠ data. If you’re cherry-picking anecdotes, you have lost the argument twice.
• Meeting abstract or poster presentation. I do occasionally write about them if there’s something compelling, but I make sure the caveats, that these have not been peer-reviewed, are in bold print. In an article in JAMA, the authors found that within 3 years after abstracts were presented at meetings, only about 50% were published in high-impact journals, 25% in low-impact journals, and 25% remained unpublished
• Press releases. These are nothing more than universities patting themselves on the back about research. Many of them do link to the published article, but it is hysterical how often the press release misinterprets the data.
• Pseudoscience websites. Natural News, Joe Mercola, and dozens of others are worthless. If you’re using them to cherry-pick, give up and go home.

One important footnote. A lot of cherry-picking, especially during this time of COVID-19, is from pre-print servers, especially bioRxiv, which is run by the prestigious Cold Spring Harbor Laboratory. It has served a relatively important purpose during the COVID-19 pandemic, as it has hosted nearly 20,000 pre-print articles.

However, as the name implies, pre-prints have not been peer reviewed. They have not been accepted for publication anywhere. And their quality is all over the place.

One day, someone will publish some statistics as to what percentage of these pre-print articles eventually made their way into a journal somewhere, but too many people are cherry-picking these articles as if they are facts. They aren’t. Not even close.

One last thing – even the best research can be bad. There are lots of systematic reviews which are filled with bias and poor statistics.

Although I, and others, consider meta or systematic reviews to be the pinnacle of biomedical research, they are not perfect and are not above criticism. For example, Cochrane produces outstanding systematic reviews and meta-analyses, but they have occasionally published some awful reviews. One article relied upon authors who have axes to grind about their personal beliefs.

The worst thing that can happen is when biased researchers have a predetermined conclusion, then use a systematic review to confirm their conclusions. We see that frequently with Cochrane’s reviews of acupuncture, which seem to act as cheerleaders to the pseudoscience of acupuncture.

So, when I cherry-pick top quality research, that doesn’t mean I cut and paste the conclusions and say “the Skeptical Raptor has spoken.” No, I still read the article with an open mind, trying to find errors and bad conclusions.

I’m trying to make this easier. But I don’t think it’s easy.

Why is this bad?

The problem with cherry-picking is that it’s allowing bad science to appear like it’s good science. Too many people, including so-called science journalists, creating a false equivalence between good and bad science by allowing cherry-picking to become rampant.

One recent example, which is driving a lot of my peers in skepticism right up the wall, is the drug ivermectin to treat COVID-19. As we saw with hydroxychloroquine, ivermectin gets a lot of press and social media support, but the science just isn’t there.

My fellow Apple aficionado and Walking Dead fanboy, David Gorski writes about ivermectin:

A couple of months ago Scott Gavura explained why the veterinary deworming drug ivermectin is the new hydroxychloroquine, a repurposed drug touted as a “miracle cure” for COVID-19 despite evidence that is, at best, very weak and, at worst, supportive of the conclusion that ivermectin is ineffective against COVID-19. Then, two weeks ago, I posted a typically lengthy, detailed, and snarky article about how ivermectin is the new hydroxychloroquine. What I meant was that, just as 12-15 months ago the antimalarial drug hydroxychloroquine was the repurposed drug touted as a “miracle cure” for COVID-19 that fizzled when tested with rigorous clinical trials, over the first half of 2021 ivermectin has become the repurposed drug touted as a “miracle cure” for COVID-19. Like hydroxychloroquine, which by the end of last summer I was describing as the Black Knight of COVID-19 treatments, an homage to (of course) the Black Knight in Monty Python and the Holy Grail, belief in ivermectin as a highly effective treatment for COVID-19—that will eliminate the need for vaccines, too!—seems similarly immune to having its limbs hacked off by science, the way that they were for hydroxychloroquine. This post won’t be as long—although it might be as snarky—and will deal more with the conspiracy theories that have cropped up around ivermectin. Unsurprisingly, they’re very similar to the conspiracy theories that cropped up around hydroxychloroquine. Many of these conspiracy theories are being promoted by a group of doctors who bill themselves as the Front Line COVID-19 Critical Care Alliance (FLCCC).

I mentioned the FLCCC in my last post about ivermectin. The reason was because of its role in producing the latest “meta-analysis” of ivermectin clinical trials. Basically, Pierre Kory, one of the founders of the FLCCC, collaborated with Tess Lawrie, the founder of the British equivalent of the FLCCC, the BIRD Group. Both are groups that promote ivermectin, although the FLCCC promotes more than just ivermectin. For instance, FLCCC promotes protocols such as the I-MASS protocol, touted as an “in-home” treatment protocol for COVID-19 that involves vitamin D3, melatonin, aspirin, a multivitamin, a thermometer, and an antiseptic mouthwash. Another FLCCC protocol is I-MASK, which is promoted as an outpatient treatment protocol and involves ivermectin, zinc, melatonin, various vitamins, and fluvoxamine. The FLCCC’s most “advanced” protocol is MATH+, a hospital treatment protocol that involves—of course!—ivermectin, plus zinc, fluvoxamine, and a bunch of other vitamins and supplements, along with steroids and anticoagulants. None of these protocols has anything resembling solid evidence from randomized clinical trials to support it.

This is the epitome of cherry-picking. Pro-ivermectin activists out in the world are proclaiming it is a great drug for COVID-19 despite awful “clinical trials” will not provide us with even weak evidence.

I’ve seen people, who are otherwise reasonable about science, who are quick to say “seems like there’s some good preliminary evidence.” Uh, that would be wrong.

A review, published in the highly respected BMJ Evidence Based Medicine, of the clinical evidence and so-called systematic reviews on ivermectin, came to this conclusion:

assessments of ivermectin as prophylaxis or treatment for mild to severe COVID-19 continue being published in preprints and protocol repositories, which do not follow the recommended process to ensure quality standards in publications; whereas peer-reviewed reports (both observational and experimental studies) are slowly emerging, yet methodologically limited by heterogeneity in population receiving ivermectin, dosis applied and uncontrolled cointerventions. Similarly, other studies that can be rapidly retrieved in ClinicalTrials.gov, medRxiv and MEDLINE make up a quite heterogeneous body of evidence (including ivermectin as intervention, but with different underlying clinical questions), among other issues that do not contribute to the certainty of evidence—according to the systematic reviews that we comment on below.

Concluding, research related to ivermectin in COVID-19 has serious methodological limitations resulting in very low certainty of the evidence, and continues to grow. The use of ivermectin, among others repurposed drugs for prophylaxis or treatment for COVID-19, should be done based on trustable evidence, without conflicts of interest, with proven safety and efficacy in patient-consented, ethically approved, randomised clinical trials.

The purpose of my article isn’t to get into ivermectin – but it is a cautionary tale of cherry-picking to find results that support a pre-determined conclusion. There is little unbiased, strong methodological, and well-analyzed research that shows it works.

Cherry-picking does not give you scientific conclusions. Unless, it’s about picking delicious cherries.

Citations

• Garegnani LI, Madrid E, Meza N. Misleading clinical evidence and systematic reviews on ivermectin for COVID-19. BMJ Evid Based Med. 2021 Apr 22:bmjebm-2021-111678. doi: 10.1136/bmjebm-2021-111678. Epub ahead of print. PMID: 33888547.

[wp_ad_camp_5]