In these contrasting pieces, two authors provide differing argumentation regarding whether the replication crisis, p-hacking, and similar well-documented issues indicate science is broken or working as intended in a self-correcting process.
Both authors note that recent years have brought to light numerous examples of fraud, methodological failure, and other forms of scientific misconduct. For example:
1. Journals accept trivial papers
A 2014 paper that was accepted for publication contained nothing but the phrase “get me off your fucking mailing list” repeated for 10 straight pages. Predatory journals allow any author to get their work published by paying a fee, regardless of the merits of their research or data.
2. P-hacking allows subjective inference
P-hacking — that is, analyzing a single data set in different ways until a statistically significant correlation emerges — allows researchers to spin the data to reach their desired conclusion. For example, a single data set can be used to show either Republican or Democratic (American) legislatures are associated with economic growth. As another example, Brian Nosek had 29 statistician teams analyze a single data set to determine whether soccer referees give more red cards to dark-skinned than light-skinned players. The 29 teams found 29 different results: 20 found a significant correlation, and nine did not. All these results were technically valid, but if a single team had run this study, only one would have been published.
3. A nontrivial share of papers are retracted
The “replication crisis” made the front page of The New York Times in 2015 when an analysis in Science found 61 out of 100 findings in psychology could not be replicated. The Retraction Watch blog was started in 2010 to find and highlight these sorts of issues, with the goal of publishing a few articles per month. In short order, it was highlighting multiple incidences of misconduct per day and hired full-time staff to keep up.
4. Scientists fail to adjust their beliefs when the evidence shifts
Epidemiological evidence initially suggested higher doses of vitamin E reduced heart disease risk. Subsequent randomized controlled trials failed to support this conclusion, and some meta-analyses suggested vitamin E may in fact be harmful in high doses. Despite this, John Ioannidis found the initial epidemiological evidence continued to be cited even years after its findings had been disproven.
5. Fraud can persist for years
Diederik Stapel published 58 fraudulent papers before being found out in 2011. Many of his papers have been cited hundreds of times.
6. Known methodological issues can persist for years
The psychology community knew as far back as the 1960s that the majority of research in the field was underpowered — that is, it was being performed with too few subjects to generate reliable results. Despite repeated calls for course correction since this time, recent analyses show power has not increased, and the majority of psychological studies remain too small.
7. Core scientific methodologies can yield questionable results
Standard scientific practices have been used to support the existence of ESP and time travel. It is possible these observations are correct, but it may be more likely that they instead speak to an underlying defect in scientific methodology and an inability to distinguish truth from falsehood.
Writing for FiveThirtyEight, Christie Aschwanden argues these and similar issues do not indicate science is broken but instead demonstrate the self-corrective nature of science. Science, in other words, cannot be expected to proceed directly from a question to a clear answer. Instead, she argues, science proceeds in fits and starts whereby observations are made and then supported or refuted by a scientist’s current and future peers.
Daniel Engber, writing for Slate, takes on a more critical tone. He argues that while corrective mechanisms are in place, the frequency and degree with which we observe scientific misconduct suggests science, in its current form, is ineffective. If misconduct is as pervasive as these analyses suggest, a significant share of the time, effort, and money invested into research is either without value or is being expended only because of observations that will be found in the future to be fraudulent. He asks, “If the replication crisis doesn’t mean science is broken, what does broken even mean?”
Is Science Broken?