Skeptical Raptor's Blog hunting pseudoscience in the internet jungle

Developing and supporting a scientific consensus

evidence-based-pyramidIn the hierarchy of scientific principles, we often mention scientific theories which “are large bodies of work that are a culmination or a composite of the products of many contributors over time and are substantiated by vast bodies of converging evidence. They unify and synchronize the scientific community’s view and approach to a particular scientific field.” A scientific theory is not a wild and arbitrary guess, but it is built upon a foundation of scientific knowledge that itself is based on evidence accumulated from data that resulted from scientific experimentation. 

Although Charles Darwin described the essentials of the Scientific Theory of Evolution in his 1859 book, On the Origin of Species, he was unaware of the existence of DNA, much of the fossil record, viruses, and other information which might have made an impact on his theory of evolution (probably by making it easier to describe). After 154 years since Darwin published his book on evolution, not one single piece of scientific evidence has been able to refute Darwin’s original theory. In fact, during the past 154 years, the scientific evidence thoroughly supported the powerful Darwinian theory of evolution.

In my writing, I often refer to the scientific consensus, which is the collective opinion and judgement of scientists in a particular field of study. This consensus implies general agreement, though disagreement is limited and generally insignificant. 

The major difference between a scientific theory and a scientific consensus is that the theory is essentially fact. It is so predictive, it is supported by so much evidence, and it is so well accepted, it takes an almost ridiculous amount of data to refute it, though it is possible. 

Evolving a scientific consensus

A scientific consensus, on the other hand, is still based on the consensus of experts in the field based on what is published. However, the current state of research is so new or based on small, but growing, number of studies, that we cannot be certain that future data will or will not overturn the consensus.

It’s important to note that there isn’t a distinct line between a consensus and a theory. There is a continuum from an early consensus to a solid theory. And sometimes it’s when a new idea is introduced, the actual consensus is against it, but enough data builds up over time to move some obscure crazy idea to a widely accepted scientific theory. 

I often talk about one of the great stories in modern science about how a crazy idea became the basis of a major scientific theory. In 1980, Luis Alvarez, a brilliant theoretical physicist whose career was extraordinary (he was part of the Manhattan Project that developed the atomic bomb), along with his son, Walter Alvarez (a geologist), and Frank Asaro, published an article that describe a sedimentary layer across the world that included a rare element, iridium, usually found in extraterrestrial objects like meteors and comets. The layer that the Alvarez’s and Asaro discovered was found right at the Cretaceous-Tertiary extinction event, which happened 65.5 million years ago, when all the dinosaurs (well, not all, since technically modern birds are surviving dinosaurs from a cladistic point of view) and some 75% of all extant genera died out or were killed.

Based on this data, they hypothesized that a meteor or comet impact event caused the demise of the dinosaurs. Acrimonious debate surrounded this idea, but ten years later, and after Luis Alvarez’s death, evidence of a large impact crater called Chicxulub was found off the coast of Mexico, providing support for the theory. It is now accepted as the cause of the downfall of the non-avian dinosaurs.

Alvarez’s theory of a an impact event causing the extinction of the dinosaurs did not happen because of arguments, debates, or a democratic vote. The transition from a simple observation, that dinosaurs suddenly went extinct around 65.5 million years, to a theory that an extraterrestrial impact caused it took nearly 30 years, which was based on the accumulation of more and more data until a consensus formed then a scientific theory.

All scientific consensus is based on evidence and data. And this evidence isn’t accumulated by using Google, but it’s actually formed by experts, who are substantially more knowledgeable than most of us in particular fields. As Steven Novella, MD, writes in his NeuroLogica blog:

For anyone trying to take a scientific approach to knowledge about the world, we must rely heavily upon experts, or those who are more knowledgable than we are. There is no choice – there is simply too much specialized scientific knowledge for anyone to be an expert in everything, or even a significant portion of scientific disciplines.

Further, being an educated layperson is usually not enough to form your own opinions on specific scientific questions. Forming a reliable opinion often requires a level of detailed knowledge that only an expert in the field can obtain. Even experts can be wrong, of course, and since lay opinions are likely to span all possibilities, some are bound to be correct. Experts, however, are far more likely to have an opinion that accurately reflects the evidence and to understand how to incorporate new evidence as it comes in.

Let’s take vaccines as an example. To understand just a part of the story of vaccinations requires research into the fields of immunology, virology, epidemiology, infectious diseases, physiology, and possibly dozens of other areas of the biomedical sciences. It is simply impossible for one person to be an expert across all of those fields. In fact, to become a competent immunologist, it might take eight years of formal education, plus another decade of research, writing, presentation, criticism, and publication. I wrote an article attempting to simply explain the human immune system, and it took me several months of research, contacting other researchers for information, and yet, I’m not sure I adequately described the nearly infinite complexity of the immune system. It was difficult despite my having graduate degrees in a specialized field of biomedical science and substantial clinical background in medicine.

That’s why, as Dr. Novella states, we rely upon the expert opinion, though as skeptics, we still have to weigh the evidence, even if presented by experts. As I wrote in articles on skepticism and debunking pseudoscience, large amounts of high quality evidence must be used to form a consensus, or, alternatively, shift us away from the consensus. Simply declaring that the consensus exists or doesn’t exist based on a few hours of Google research is ridiculous. 

Big Pharma and the scientific consensus

Anyone who reads my blog might come to the impression that I think that world’s pharmaceutical companies are simply wonderful unicorns surrounded by beautiful rainbows. Unfortunately, many people, even those who are “pro-science”, believe that Big Pharma is made up of evil aliens reading to destroy humanity.

Ignoring the false dichotomy, the truth is somewhere in between, probably closer to the “morally good” side. Big Pharma is generally, but not exclusively, made up of well-meaning, ethical individuals. There are countless medications, devices, and equipment invented and marketed by Big Pharma (they make more than just drugs) that have saved millions of lives, provided a better quality of life, and allowed us to live longer and be more productive. 

That being said, they are, in general, public corporations with shareholders that demand higher and higher profits. There are vast swaths of classes of medications where Big Pharma has overtly influenced or suppressed data that allows them to market medications that have marginal benefit. Statins, a class of drugs that are used to lower cholesterol levels, are one of the most profitable drugs on the market (approximately the same dollar value worldwide as vaccines, except at nearly 50% higher profits). The scientific consensus supports the evidence that statins are safe and effective in reducing cholesterol, potentially reducing mortality from cardiovascular events.

But Big Pharma, in the lust for making money, has developed several types of statins, each with specious claims that one is better than another. These claims are based on studies published in peer-reviewed journals, but usually just one or two; the various competitors use these marginal claims to try to get higher prices and switch customers from generic statins. The scientific consensus on the value of statins is based on a systematic reviews, but there is nothing but Big Pharma-supported research that states one individual statin brand is better than another. 

I could bring you similar stories about medications for depression, cardiovascular disease, and other chronic conditions where the drug’s safety and efficacy is supported only by company sponsored trials without the systematic review that is now the gold standard for science based medicine. If you desire a comprehensive condemnation of Big Pharma, read Dr. Ben Goldacre’s book Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients

But Dr. Goldacre says, near the beginning of his book, “drug companies are not withholding the secret to curing cancer, nor are they killing us all with vaccines.” Yes, even though I agree with Dr. Goldacre that the use of many new drugs are not supported by any sort of scientific consensus, there are several important exclusions: cancer treatments and vaccines. The former because much of the research is government sponsored and because oncology drugs are really targeted to each patient, so oncologists almost “design” the treatment. The latter, vaccines, are excluded because so much research has been done in huge prospective and retrospective epidemiological studies, it’s moved beyond the bias of the vaccine companies.

Of course, if you’re going to paint Big Pharma as an evil, profit driven, human health be damned entity, then vaccines don’t make sense. The profits that Big Pharma would derive from sending millions of kids who get vaccine preventable disease to the ER and ICU would far surpass, by hundreds of billions of dollars, the amount gained from marketing vaccines. If you’re going to claim that Big Pharma is coldly profit-driven, then you can’t stop short by claiming saying “they don’t want to go that far.” You kind of surrender the argument that vaccines work if you say that.

Systematic reviews and the scientific consensus

In some cases there are independent scientific groups that proclaim the scientific consensus. The American Association for the Advancement of Sciences is an international non-profit organization that has as its stated goals to promote cooperation among scientists, to defend scientific freedom, to encourage scientific responsibility, and to support scientific education and science outreach for the betterment of all humanity. It is the world’s largest and most prestigious general scientific society, and is the publisher of the well-known scientific journal ScienceIt occasionally makes statements on the scientific consensus.

For example, here is the consensus position of the American Association for the Advancement of Sciences on climate change:

The scientific evidence is clear: global climate change caused by human activities is occurring now, and it is a growing threat to society. Accumulating data from across the globe reveal a wide array of effects: rapidly melting glaciers, destabilization of major ice sheets, increases in extreme weather, rising sea level, shifts in species ranges, and more. The pace of change and the evidence of harm have increased markedly over the last five years. The time to control greenhouse gas emissions is now.

The AAAS has also released another statement of consensus science on genetically modified foods (pdf):

The science is quite clear: crop improvement by the modern molecular techniques of biotechnology is safe … The World Health Organization, the American Medical Association, the U.S. National Academy of Sciences, the British Royal Society, and every other respected organization that has examined the evidence has come to the same conclusion: consuming foods containing ingredients derived from GM crops is no riskier than consuming the same foods containing ingredients from crop plants modified by conventional plant improvement techniques.

There’s even a scientific consensus on vaccines, written by the the most prestigious scientific academy in the world, the National Academy of Sciences:

Vaccines offer the promise of protection against a variety of infectious diseases. Despite much media attention and strong opinions from many quarters, vaccines remain one of the greatest tools in the public health arsenal. Certainly, some vaccines result in adverse effects that must be acknowledged. But the latest evidence shows that few adverse effects are caused by the vaccines reviewed in this report.

Of course, someone will invent some conspiracy involving these two esteemed institutions to besmirch their reputation in providing us laypeople with a scientific consensus, what can we do? For example, in the field of vaccine research there are literally thousands of articles published. And one could cherry pick the results that give only a partial view, usually the one that supports the view that you want. Ironically, vaccine deniers cherry pick research that supports their view, while ignoring all of the data that does not, then turn around in nearly the next breath (or sentence, if they can write) claim that’s what Big Pharma does.

To prevent this cherry picking, systematic reviews were invented, and they allow us to logically build and support a consensus. Here’s how a systematic review works: instead of consciously or unconsciously (or deviously or innocently) choosing the papers that support your point of view, you use science on science. The systematic review examines all of the evidence, attempts to determine which evidence is weaker (based on statistical results and experimental design which might influence the results), and finally assembles it all into one study of studies that is complete and representative of all the research that has been done in one specialized field.

This isn’t easy work (as is everything else in science, which is something that I believe is always ignore by science deniers). Each study ever published in the field must be closely reviewed. Data that might not have been published needs to be tracked down. Authors of previous research might be contacted to explain information. Then all of the data is assembled into one huge statistical analysis. So instead of a clinical trial containing only 250 or 2500 patients, if you take all the trials ever done, you might have one of hundreds of thousands of patients.

blobbogramWhen the authors of the systematic analysis assemble all of the data, they find the summary effect. If it is near the line of “no effect”, the researchers can conclude that the analysis shows no effect from a drug.  On the left is an example of a systematic analysis in a visual form called a “blobbogram,” although more formally as a Forest plot. Each of the horizontal lines represents an individual study, with the solid dot equal to the mean, and the horizontal line itself the error around the mean. The one vertical line, at 1, is the line of no effect. In this graph, the diamond point is the summary effect, which implies that it this imaginary drug has some effect.

One of the best organizations to produce systematic reviews is the Cochrane Collaboration, a non-profit academic organization that has produced meta-reviews of evidence in important questions in medicine for over 30 years. I consider their reviews to be the gold standard of research, though I think they have some tarnish on the gold at some times. A few of their reviews have tended to show strong selection bias of their authors.

Cochrane has published a few articles that seem to indicate that acupuncture is clinically useful, but the research itself seems to be filled with bias, sometimes excluding trials that don’t support their findings and not adequately explaining the ineffectiveness of placebo controls that might actually show us some value to the procedures. Also, the conclusions written in Cochrane can be confusing and filled with the scientifically objective language that makes it seems like they’re taking two sides of an issue.

Not all systematic reviews are published by Cochrane. There are many other published in major medical and scientific journals which help establish a scientific consensus. And just as one can cherry-pick individual articles, without critical analysis, to support or refute a point of view, one can do the same with these larger reviews. 

As I’ve written before, there is a hierarchy of evidence, with systematic reviews at the very top:

  1. Systematic reviews published in peer-reviewed, high-impact journals. As I stated above, these studies are supposed to give us clinical data on a drug or procedure by removing bias and cherry-pciking. They are the gold standard, but still must analyzed critically. 
  2. High quality clinical trial reports. Again, if they’re published in high impact journals, and include fairly large numbers (in general, I like to see >1,000 subjects in each arm, but even those in the hundreds can be useful), they can provide evidence to support a hypothesis. However, and this is important, bad study design is bad study design. That’s why the peer-review process is so important, and that’s why we add weight to it. The expectation is that other experts can sort out statistical and design problems before the paper is published. Moreover, even high impact journal studies get it wrong–MrAndy Wakefield, whose fraudulent paper, alleging a connection between MMR and autism, was published and subsequently retracted by the Lancet, a very high impact factor journal, and one of the top medical journals in the world. 
  3. Small clinical trials. At this point, they border on observational studies that need to be confirmed (or refuted) by larger studies. In the world of clinical trials for pharmaceuticals, research starts at the small size (less than a 100 patients) at Phase I, progresses through several hundred patients for Phase II,  and finally ending with several thousand in Phase III. At each phase more and more data is accumulated, but at Phase I, little can be concluded from the data except for basic safety.
  4. Animal or cell culture studies. This is basic primary research which may or may not be applicable to human health issues. Too much credence is given to studies that may be 20 years away from having any applicability, if any at all. Someone publishes that cannabis “cures” breast cancer in mice, and that is used to support the smoking of pot to “cure cancer”. Except there’s no real evidence that it can. And the same issues apply. Is the research in a high quality journal. There is an old joke in medical research circles: science has cured cancer in mice for decades.
  5. Meeting abstract or poster presentation. These are presentations made at meetings/conventions of a scientific society. There are hundreds made every year, and preliminary research is presented in poster sessions or formal presentations. Usually, there is a lot of question and answers, that of course doesn’t show up in a link to the abstract or poster, that can help explain data. Moreover, these types of presentations vary from nearly ready for publication to pure speculation. They have not been peer-reviewed (although peer review can unintentionally happen through tough questions). They are not published formally. And they often do not contain enough explanation of methods, data, analysis, and other issues to evaluate properly. I would not consider this type of information as anything but “observational” in a scientific sense, though, as mentioned above, only around half are published.
  6. Press releases or news reports. Do not accept the conclusions stated by a press release from a major research university, they haven’t been peer reviewed. I have a feed filled with press releases from major research universities, and I’ve found errors in interpretation from the university’s public relations office relative to the real research. However, it is possible to use a press release to chase down the actual published, and peer reviewed study.
  7. Studies published in low impact factor journals (those with ratings less than 1-2). I am openminded that some article published in a bad journal with low quality peer review might actually be good research, so the line I draw at an impact factor of 5 is purely arbitrary. In the world of academic research, job promotions and tenure depend on publishing in high impact factor journals (almost always greater than 10), so impact factor has been correlated to quality of the research. In general, the scientific consensus is supported by lots of publications in high impact (and many times, low impact) factor journals. Good research, that goes against the scientific consensus, can make it into high impact factor journals. But only high quality research, with strong supporting evidence, methods, and analysis. If you’re going to say that all immunology is wrong, and vaccines don’t work, a point of view that is definitely against the scientific consensus, then publishing in insignificant, non-peer-reviewed journals will have little meaning to anyone. Well, as far as I can tell, pseudoscience pushing individuals love these articles published in insignificant journals.. 
  8. Medical case reports. I despise these types of articles. They often show up in high quality medical journals, but they are usually observations of one or two patients. Those in medicine know their purpose, which is to give a “heads up” to an observation. They have no scientific validity beyond observational. Unfortunately, science deniers will pull a case report published, and use it to condemn a whole field of medicine. Don’t use them to support your argument, one way or another.
  9. Natural News from Mike Adams, the Health Ranger. Whale.to. Let me make this clear, Natural News is a foul, fetid, putrid sewer of information, which befouls any real science with lies. The antisemitic, anti-science, anti-knowledge website, Whale.to, similarly reeks of malodorous smells of pure garbage. Anyone who uses either site as a source for anything in medicine loses the discussion without any further debate. The pseudoscience pushers who claim that they’ve done extensive “research” into a subject, and quote either of these websites are beneath contempt, and supports the hypothesis that those who support anti-science are intellectually lazy charlatans.

It’s not easy to determine the scientific consensus on a particular field. Sometimes the leading scientists in the world will make a statement that makes the consensus obvious (like with global warming, safety of GMO crops and vaccines). Sometimes you have to dig through systematic reviews, read them, understand them, critically analyze them, then use them to support your own opinion as to whether there is a consensus or not. But if you’re going to deny the consensus, if you’re going to set it aside, rhetoric and debate count for nothing.

Only evidence matters, and only evidence that is published in respected, high-impact journals. That’s it. You cannot play a game by pretending that you’re a credible scientific expert by trolling a few comment sections of vaccine articles, dropping a few bits of research you dig up on the internet. You cannot have a science or medical degree, and claim you have the authority to contradict and refute the established scientific consensus, unless you provide scientific evidence accumulated in the same manner as all other scientists. Just because you want to refute the consensus, just because you believe the consensus is wrong, does not discharge your obligation to provide the world with peer-reviewed evidence that supports your claim. 

Yes, I know it’s hard work. But it shouldn’t be easy to develop or refute a scientific consensus. It should be a gigantic pain in the brain. It should require all of your intellectual prowess. It should demand that you withstand withering criticism.  

But if you want to convince the world that vaccines don’t protect children, and then therefore put those children at risk, then your evidence should be impossibly strong, should be resistant to all criticism, should not be reliant on logical fallacies and whining about conspiracies. If you want to put children at risk of vaccine preventable diseases, then the consensus demands that you give us evidence that far exceeds, in quality and quantity, that which has been used to form a powerful scientific consensus on the safety and effectiveness of vaccines. That’s your job, and until you do so, all I can assume is that you have nothing, and you want to intentionally endanger the children of the world. Cause you’ve got nothing so far.

The scientific consensus on any number of ideas was not derived easily. We know that science supports the fact of anthropogenic global warming. We know that sciences shows that genetically modified foods are safe. And we know that vaccines are safe and effective. Because we have the evidence.

Key citations:

 

Comments (10)
Powered by WordPress 4.0.1
Don't forget to subscribe to this blog through any the services in the right sidebar.