As many of the regular readers of this website know, I put a lot of value in the quality of evidence for a scientific or medical claim, usually in the form of a paper published in a peer reviewed, high impact factor journal. But to be honest, I prefer a paper published in a peer reviewed, high impact fact journal that presents a systematic review of all of the evidence available to provide us with a broad statistical analysis over tens or hundreds of studies. It’s like a peer reviewed paper analyzing peer reviewed papers.
What is peer review?
Peer review is a system whereby independent researchers anonymously review other papers, almost always in the same field of science as the writer, to ascertain the quality of writing, research, analysis and conclusions. It actually works quite well, though there are some concerns. It is supposed to clean up most of the worst mistakes. But peer review failure could undermine our confidence in scientific and medical research.
Many researchers are under pressure to publish frequently to maintain their academic standing, or to get promoted to tenure positions, so many scientific publishers can actually charge scientists to publish their research. Publishers are pressured to get the most important research out into the public quickly, sometimes making the peer review process too short, because they want the papers out there, to build their own credibility and, let’s be honest, to make profits.
It’s important to understand that peer review is probably the most fundamental aspect of determine high from low quality science. The process of self-correction in the peer review system is literally one of the key foundations of the scientific method, that is, a literal review of the science before publication.
Furthermore, peer review is an ongoing process. I find it amusing that there are individuals who use peer reviewed articles as the final word on a topic, assuming that the article itself is some central dogmatic word from a higher power. In fact, most medical research is flawed and much of it never amounts to much.
In the real world of science, peer review doesn’t stop with the publication of the paper. There has been many papers that have been published, sometimes by some of the top scientific and medical journals in the world, but were subsequently withdrawn because of flaws.
One need only point to Mr. Andy Wakefield’s fraudulent paper published in one of the most prestigious medical journals, Lancet, as an example of where everything went horribly wrong in a publication. But it did get corrected, and the paper was eventually retracted.
Peer review failure at BioMed Central
Recently, open-access publisher BioMed Central has withdrawn 43 papers published over the past few years. BioMed Central is investigating many more papers, so this problem may expand in the near future. According to their original statement,
…the spectrum of ‘fakery’ has ranged from authors suggesting their friends who agree in advance to provide a positive review, to elaborate peer review circles where a group of authors agree to peer review each others’ manuscripts, to impersonating real people, and to generating completely fictitious characters.
I have occasionally become concerned about the quality of papers being published at some of the 277 journals owned by BioMed Central. It’s not that the research published there is shoddy or bad, it’s that sometimes the writing is poor, and the analysis weak.
It is also important to note that its journals have a moderately low impact factor on average (around 2.5), but ranging from below 1 (awful) to above 10 (outstanding).
Impact factors, though flawed, help us understand whether a particular journal is cited by other publications, an indication of the average importance of articles published by one journal. Some tenure committees, who review academic faculty for promotion to tenure track, exclude any articles published in lower ranked journal in examining the body of work of the applicant.
Also, BioMed Central charges authors to publish–I am not opposed to researchers paying fees to publish their work. In the open-access world of the internet, it’s annoying that great science is hidden behind paywalls. Charging the author removes those paywalls, and may speed the time for good research to be available publicly.
However, it is worrisome that the journal now might be incentivized to publish quickly, to get those fees, making the peer review process a sham. There’s probably some fine line between open-access and having too much money flow between the various constituents of this process, but it’s going to be hard to find.
At this point, the best research is still published in a handful of medical and science journals, some of which have impact factors into the 30’s and 40’s, substantially higher than BioMed Central. To be published in one of those journals is still a prestigious award.
So, a publisher like BioMed Central performs an invaluable service to science and medicine. There is just limited space for articles in the largest and most prestigious journals, and frankly, to defend their “market share,” those coveted journals only publish the most prestigious work. So BioMed Central, and other publishers of narrow focused journals, provide an outlet for new ideas and hypotheses, which may provide more data to emerging scientific consensuses.
Still, if we are to have confidence in the peer review process and scientific publications, then peer review has got to be above reproach. It must be trusted.
The Committee on Publication Ethics (COPE) issued a statement saying they have,
…become aware of systematic, inappropriate attempts to manipulate the peer review processes of several journals across different publishers. These manipulations appear to have been orchestrated by a number of third party agencies offering services to authors.
COPE was formed to define best practices in the ethics of scholarly publishing and to assist editors, publishers, and others to achieve this goal. It started out as an effort by a small group of medical journal editors to raise the standards of academic publication. It now has a membership of approximately 9000 editors from across all academic fields. Part of its growth has resulted from the growing concerns about the scientific peer review process.
Concerns about peer review have been growing for decades. Reviewers are almost always already struggling under the burden of their own research and teaching load. Few are paid for their efforts, and even fewer get credit from their employers for this vital contribution to the advancement of science. Many admit off the record to not giving papers the attention they deserve.
Although bad papers can send scientific research off down blind alleys, but bad work more often gets ignored by others in the field. However, authors can be rewarded with funding that should have gone to someone else. Media outlets, whether mainstream or science-specific, usually have no choice but to rely on publication in a peer reviewed journal as the test of whether work justifies publicity.
When I see a statement like that, I become more convinced that we should be very skeptical of scientific or medical research that show statistically small (or even clinically insignificant results) from a tiny sample population. Further research will either confirm or refute the initial publication, and eventually, we will have enough published results to form a well-accepted conclusion.
Conclusion, the TL;DR version
Yes, this makes science incredibly conservative. But a strong peer review process allows new scientific data to be considered credible until more data supports it. New ideas aren’t suppressed by such a system, it gives more credence to the new ideas when they are published.
It’s possible that this peer review failure at BioMed Central will go beyond 43 papers. But they publish tens of thousands of articles every year, so the peer review failure is probably not endemic, although it shakes our confidence in the publisher.
And when you look at the totality of all scientific publications, this represents a tiny number of problems relative to the whole universe of scientific and medical research. Unless you’re looking at this issue with the sunglasses of confirmation bias, and conclude every scientific article, but the ones that support your beliefs, is corrupt, it appears the system works. It is self-correcting. Abuse of peer review was found, because science is extremely self critical. That’s why some of us love it.
