Last updated on September 9th, 2020 at 10:51 am
Over the past few weeks, I’ve been observing new coronavirus research peer review being done through Big Pharma and university press releases, preprint articles, and science journalism. It’s like watching a ping pong ball bounce back and forth, and I swear my neck is getting strained.
This is not how science should be done. It does a disservice to how science should be done. Science, especially with regards to coronavirus, must be done with careful analysis and critiques.
Now, there is one good thing about how we’re evaluating coronavirus research – peer review is beginning to be “crowd-sourced,” and that may be better than the old system.
This article will try to point out the good, the bad, and the damn ugly coronavirus research that we’re seeing these days.
OLYMPUS DIGITAL CAMERA
What is peer review?
Peer review is kind of simple. It is a system where an article written by a scientist or, more likely a team of scientists, is sent to usually a few other scientists who have similar expertise to review for everything from grammar to statistics to whatever.
The peer reviewers are anonymous and the authors of the paper they’re reviewing are removed. However, the reviewers can frequently guess who wrote the article, but it’s only a guess.
The journals employ the peer-review process to weed out bad science and to maintain high quality and provide credibility to the work being published. In science, papers published in peer-reviewed journals are considered the highest quality.
In academia, faculty who are attempting to get tenure are judged by the number of articles published in high-quality, peer-reviewed journals.
It’s not a perfect process. It’s slow – before the internet, papers were sent to the reviewers who took their time to review and edit, by literally using a red pen. It could take months or even years to get a paper published. Of course, this process is speeded up today, but it’s still not fast.
Let’s be clear – it is not a perfect process. There are predatory journals that use the term “peer-review,” but it’s not. There was an anti-vaccine paper in which one of the peer-reviewers was a chiropractor – it’s pseudoscience. That’s why anything published in a predatory journal should be dismissed – and most scientists ignore these types of papers.
And peer-review can fail even in the best journals. The Lancet, one of the top medical journals in the world, published the fraudulent study by Mr. Andrew Wakefield that led to the “vaccines cause autism” inanity. Of course, the journal retracted the paper
So, peer-review is slow and it sometimes, though very rarely, makes mistakes. Maybe there are better ways? Let’s take a look.
Coronavirus research peer review – the good
Recently, a paper was published in The Lancet that raised concerns about the safety of the experimental Covid-19 treatments chloroquine and hydroxychloroquine. Now, there is overwhelming evidence (or a lack of evidence period) that hydroxychloroquine lacks any effectiveness in treating COVID-19. So, getting evidence that it was also unsafe seemed to finally close the door on Donald Trump’s favorite coronavirus research.
At about the same time, the New England Journal of Medicine published an article about the safety and effectiveness of blood pressure medications to treat COVID-19.
They both relied upon data from Surgisphere, a company that produced dubious data that they claimed came from 1200 hospitals across the world.
All seemed good until blogs, Twitter, and PubPeer began critiquing one or both of the articles.
https://twitter.com/mikejohansenmd/status/1267673812402868224
Some excellent work here as more people pry into the @Surgisphere papers. I'm going to try to build on this a bit further… https://t.co/OAsVkO060h
— Andrew Althouse (@ADAlthousePhD) June 2, 2020
It looked liked Surgisphere basically invented the data. Scientists across the world noticed all kinds of anomalies from the number of hospitalizations in Africa to the ethnic makeup of the patients.
And when each journal requested the underlying data, Surgisphere refused to provide using the old canard, “patient privacy.” In fact, almost always this type of data can be anonymized so that no one knows who is who.
Of course, both The Lancet (dudes, have you not learned your lessons yet?) and the New England Journal of Medicine retracted the articles.
Peer review utterly failed in both of these cases. The dataset issues were so obvious, I’m wondering if, in the rush to publish coronavirus research, both journals tried to push the peer review too quickly.
However, what I call “Twitter Peer Review” worked perfectly. And instead of three sets of eyes, there were hundreds looking at the data and quickly publishing criticisms all over the internet.
Thes speed of these retractions probably set the record for the fastest retractions ever, but I’m not an expert on retractions, so maybe there were faster ones.
So, the system works, but it’s not a system that was invented by the journals, it was by real scientists. And no, for those anti-vaxxers who troll this blog, your whiny comments on Twitter about every vaccine article is not going to work this way, because real scientists don’t actually listen to your junk science.
Coronavirus peer review – the ugly
What has become annoying lately is coronavirus research that isn’t peer-reviewed at all. It’s either dueling press releases from pharmaceutical companies looking to boost their stock prices. Or hyping pre-print articles by universities trying to show off their work.
Either way, it’s not helpful.
I’ve already discussed how Moderna is overhyping its coronavirus vaccine without providing actually peer-reviewed data that establishes it actually showed anything in its phase 1 clinical trials. They’re pushing press releases that breathlessly claim that they’re going to have a vaccine out by the end of the year, boosting their stock prices.
Of course, one of the former executives is on Trump’s “Operation Warp Speed,” so I’m more than a bit skeptical.
Just in case you’re wondering, press releases, especially from pharmaceutical companies, rank right near the bottom of the hierarchy of clinical research. Only those pseudoscience morons at Natural News would rank lower.
We’re also seeing universities tout research that is “published” on a preprint server. In case you don’t know what a preprint is, they are scientific papers that have been submitted to a journal, but they have not been peer-reviewed, spell-checked, or anything. There are nearly 5600 articles on just coronavirus research that are listed on the largest preprint server, medRxiv.
There are some good reasons to have these coronavirus research articles available. COVID-19 is a deadly disease, and anything that can help physicians and scientists attack it quickly will be useful. Moreover, there are comment sections for each of these articles which are also a kind of a crowd-sourced peer review.
Unfortunately, many institutions are “advertising” research published on these preprint servers as real science. And they use those annoying press releases.
In an article in JAMA, the authors found that within 3 years after abstracts were presented at meetings, only about 50% were published in high-impact journals, 25% in low-impact journals, and 25% remained unpublished.
Interestingly, the 39 abstracts that received front-page coverage in newspapers had a publication rate almost identical to the overall publication rate. The authors concluded that “abstracts at scientific meetings receive substantial attention in the high-profile media.
A substantial number of the studies remain unpublished, precluding evaluation in the scientific community.” In other words, until the research is published in peer-reviewed journals, it’s hard to evaluate their quality and importance.
Recently, Oxford University touted a not-yet-published or peer-reviewed article where the steroid dexamethasone may reduce mortality from COVID-19. Except, Oxford refused to release the data. And after the hydroxychloroquine debacle, many physicians are reluctant to jump on another bandwagon. And they shouldn’t.
What’s more, Jones pointed out, “the death rate even with dexamethasone is still 28%. So, it’s still massively high; this is still a very nasty virus.” The clinical trial, he added, will need to “show real benefit over risk.”
And dexamethasone, like all steroids, has one serious side effect – it can suppress the immune system, something you really don’t want with COVID-19.
Will dexamethasone do anything? I don’t know, and I will not know until such time as the data is published and peer-reviewed, especially by the science crowd on Twitter.
Moreover, physicians have been debating (even arguing) about steroids and the treatment of these types of diseases for over 40 years. And yet, we have no consensus.
Summary
Coronavirus research is a fast-moving ship that has a lot of momentum, so it’s hard to stop, steer, or jump off. But here’s my recommendation:
- Ignore all coronavirus research that isn’t in the form of phase 3 clinical trial published in a highly respected, peer-reviewed journal.
- Even then, wait until the scientists on Twitter or PubPeer weigh-in with their thoughts, criticisms, and angry commentary.
- Ignore dueling press releases. They are just not helpful in what we know or don’t know about coronavirus research.
- And with respect to The Lancet – really people, what are you doing there? To be fair, 99.99% of their articles are really good and have not been retracted. It’s just that one from 1998 that gets us all steamed.
So, be careful out there. Don’t jump on that ship because you will get burned. Hey, your cute, but ancient, raptor got roasted for jumping on the wrong ship after reading an article on the old preprint server. Never again.
Citations
- Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005 Jul 13;294(2):218-28. PubMed PMID: 16014596.
- Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020 May 22:S0140-6736(20)31180-6. doi: 10.1016/S0140-6736(20)31180-6. Epub ahead of print. Retraction in: Lancet. 2020 Jun 5;:null. Erratum in: Lancet. 2020 May 30;: PMID: 32450107; PMCID: PMC7255293.
- Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19. N Engl J Med. 2020 Jun 18;382(25):e102. doi: 10.1056/NEJMoa2007621. Epub 2020 May 1. Retraction in: N Engl J Med. 2020 Jun 4;: PMID: 32356626; PMCID: PMC7206931.
- Wakefield AJ, Murch SH, Anthony A, Linnell J, Casson DM, Malik M, Berelowitz M, Dhillon AP, Thomson MA, Harvey P, Valentine A, Davies SE, Walker-Smith JA. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet. 1998 Feb 28;351(9103):637-41. doi: 10.1016/s0140-6736(97)11096-0. Retraction in: Lancet. 2010 Feb 6;375(9713):445. Erratum in: Lancet. 2004 Mar 6;363(9411):750. PMID: 9500320.
- Schwartz LM, Woloshin S, Baczek L. Media coverage of scientific meetings: too much, too soon? JAMA. 2002 Jun 5;287(21):2859-63. PubMed PMID: 12038934.
