Individual research studies volley across the landscape of any debate like tennis balls at Wimbledon. Opposing positions inevitably find credible studies supporting their point of view…on the exact same topic! If you follow any debate on social media sites, you will remark how comments quickly degenerate into verbal melees and slugfests. It’s not only unproductive…it’s nuts!
If you think scientists are somehow better equipped to understand individual studies outside their area of expertise, you’re mistaken. Specialization being what it is, a PhD in Biochemistry may be no better at judging a study on global warming than a well-educated layperson. Casadevall and Fang’s article Specialized Science[1] looks pessimistically at the impact of specialization on scientists’ ability to judge research outside their field of expertise:
“The current tendency to judge scientific work by the impact factor of the journal in which it is published may have roots in overspecialization, as scientists are less able to critically evaluate work outside their field than before.”
We seem to be drowning in a sea of individual studies with often conflicting conclusions about the same topic. News-thirsty media present these to the public as the latest truth. Is it any wonder that the Pew Research Centre[2] found major gaps between scientists’ and the public’s beliefs on critical societal issues? See graphic below.
Dietary movements and entire industries can be created by a sole study showing a positive health benefit for a single food fed to rats. They can equally be destroyed by another study showing the detrimental effect of the same food in hamsters! Single studies can be a marketer’s dream and rapidly devolve into a nightmare as trends collapse.
Motivated Reasoning
Worst of all, focusing on individual studies fuels Motivated Reasoning[3], the most pernicious of human cognitive processes. Motivated Reasoning is the tendency to develop elaborate rationalizations in support of a given point of view, and then seek out sources to support that point of view while ignoring contrary evidence. It’s confirmation bias on steroids.
So what’s to be done? Meta-analysis[4] is good but often very challenging because it’s so tough to harmonize methodologies across multiple studies, especially sampling, reporting tools, etc.
Systematic review[5] is the gold standard but very expensive and time-consuming to do properly. It requires convening expert panels that can assess studies for scientific quality and rigor. Even then, competent interpretation is essential. To get a sense of just how hair-raising this process is, look at the 2013 Systematic Evidence Review for Obesity[6]. 501 pages. It took a panel of some of the world’s leading experts on obesity 5 years to complete. Its results are now finding their way into the guidelines for various health organizations.
Systematic review can lead to scientific consensus, which in principle should carry more weight than individual opinion. For example, Linden and Leiserowitz[7] found that increased public awareness of the expert consensus on climate change led to greater opinion change among non-believers. On the other hand, Nyhan et al.[8] studied if intensive education about the scientific consensus could change the attitudes of anti-vaxxers. They found that refuting claims of a link between vaccines and autism actually decreased intent to vaccinate among strong believers! Clearly, there’s little consensus on the value of consensus!
Relying on single studies is a poor basis for arriving at conclusions and making important decisions! Look for systematic literature reviews and consensus panels. Vet the panels and their methods rather than the individual studies, especially in areas outside your expertise. And above all, remember Nobel laureate Konrad Lorenz’s famous dictum:
“Truth in science can be defined as the working hypothesis best suited to open the way to the next better one”
By Steve Courmanopoulos, Ph.D. (Psych)
Dr. Courmanopoulos is the Senior Partner and CEO of Medius International Inc, a global consulting firm providing expertise in three areas: Intelligence, Strategy, and Organizational Development.
[1] Arturo Casadevall, Ferric C. Fang. Specialized Science. Infect Immun. 2014 Apr; 82(4): 1355–1360.
[2] Pew Research Centre. Attitudes and Beliefs on Science and Technology Topics. Retrieved January 16, 2017 from: http://www.pewinternet.org/2015/01/29/chapter-3-attitudes-and-beliefs-on-science-and-technology-topics/#vaccines-and-access-to-experimental-treatments-18-point-gap
[3]Motivated Reasoning: The Most Dangerous Defense. Leadership Insights, November 2015. Retrieved from: https://mediusinternational.com/index.php/2015/11/19/motivated-reasoning-the-most-dangerous-defense/
[4] https://en.wikipedia.org/wiki/Meta-analysis
[5] https://en.wikipedia.org/wiki/Systematic_review
[6] U.S. National Institutes of Health (2013). Systematic Evidence Review for Obesity. Retrived January 16, 2017 from: https://www.nhlbi.nih.gov/sites/www.nhlbi.nih.gov/files/obesity-evidence-review.pdf
[7] Van der Linden SL, Leiserowitz AA, Feinberg GD, Maibach EW (2015) The Scientific Consensus on Climate Change as a Gateway Belief: Experimental Evidence. PLoS ONE 10(2): e0118489. doi:10.1371/journal.pone.0118489
[8] Brendan Nyhan, Jason Reifler, Sean Richey, Gary L. Freed (2014. Effective Messages in Vaccine Promotion: A Randomized Trial. Pediatrics Feb 2014, peds.2013-2365.