In these contrasting pieces, two authors provide differing argumentation regarding whether the replication crisis, p-hacking, and similar well-documented issues indicate science is broken or working as intended in a self-correcting process.
Both authors note that recent years have brought to light numerous examples of fraud, methodological failure, and other forms of scientific misconduct. For example:
1. Journals accept trivial papers
A 2014 paper that was accepted for publication contained nothing but the phrase “get me off your fucking mailing list” repeated for 10 straight pages. Predatory journals allow any author to get their work published by paying a fee, regardless of the merits of their research or data.
2. P-hacking allows subjective inference
P-hacking — that is, analyzing a single data set in different ways until a statistically significant correlation emerges — allows researchers to spin the data to reach their desired conclusion. For example, a single data set can be used to show either Republican or Democratic (American) legislatures are associated with economic growth. As another example, Brian Nosek had 29 statistician teams analyze a single data set to determine whether soccer referees give more red cards to dark-skinned than light-skinned players. The 29 teams found 29 different results: 20 found a significant correlation, and nine did not. All these results were technically valid, but if a single team had run this study, only one would have been published.
3. A nontrivial share of papers are retracted
The “replication crisis” made the front page of The New York Times in 2015 when an analysis in Science found 61 out of 100 findings in psychology could not be replicated. The Retraction Watch blog was started in 2010 to find and highlight these sorts of issues, with the goal of publishing a few articles per month. In short order, it was highlighting multiple incidences of misconduct per day and hired full-time staff to keep up.
4. Scientists fail to adjust their beliefs when the evidence shifts
Epidemiological evidence initially suggested higher doses of vitamin E reduced heart disease risk. Subsequent randomized controlled trials failed to support this conclusion, and some meta-analyses suggested vitamin E may in fact be harmful in high doses. Despite this, John Ioannidis found the initial epidemiological evidence continued to be cited even years after its findings had been disproven.
5. Fraud can persist for years
Diederik Stapel published 58 fraudulent papers before being found out in 2011. Many of his papers have been cited hundreds of times.
6. Known methodological issues can persist for years
The psychology community knew as far back as the 1960s that the majority of research in the field was underpowered — that is, it was being performed with too few subjects to generate reliable results. Despite repeated calls for course correction since this time, recent analyses show power has not increased, and the majority of psychological studies remain too small.
7. Core scientific methodologies can yield questionable results
Standard scientific practices have been used to support the existence of ESP and time travel. It is possible these observations are correct, but it may be more likely that they instead speak to an underlying defect in scientific methodology and an inability to distinguish truth from falsehood.
Writing for FiveThirtyEight, Christie Aschwanden argues these and similar issues do not indicate science is broken but instead demonstrate the self-corrective nature of science. Science, in other words, cannot be expected to proceed directly from a question to a clear answer. Instead, she argues, science proceeds in fits and starts whereby observations are made and then supported or refuted by a scientist’s current and future peers.
Daniel Engber, writing for Slate, takes on a more critical tone. He argues that while corrective mechanisms are in place, the frequency and degree with which we observe scientific misconduct suggests science, in its current form, is ineffective. If misconduct is as pervasive as these analyses suggest, a significant share of the time, effort, and money invested into research is either without value or is being expended only because of observations that will be found in the future to be fraudulent. He asks, “If the replication crisis doesn’t mean science is broken, what does broken even mean?”
Comments on Is Science Broken?
Bad science is a big problem esp if money and greed overrides safety of a population. Manipulating research data even before it goes to stats is nothing new. They control and approve everything. The chief scientist is the one in charge...not the data.
Scientific misconduct is nothing new. Modern analysis has speculated that Johannes Kepler fudged data to bolster his case that planets have elliptical, not circular, orbits. This was in the early 1600s. What is relatively new is that science is not just a hobby of wealthy noblemen, but a career and an industry. Science is a source of fame, job security and corporate profits now. The magnitude of the corruption of science has been a frightening theme explored here on the main page.
Society has taken for granted that scientific knowledge increases at a steady rate. This has been true for centuries. Scientists rely on observational tools to make discoveries and scientific discoveries have led to improved tools (computers, lasers, gravitational wave detectors). This feedback loop has kept the forward march of progress going unabated.
However, much of the low hanging fruit has been picked off and what remains will require more investment than what came before. For example, the Higgs boson was predicted to exist in the 1960s, but it wasn’t experimentally detected until 2012 by the Large Hadron Collider. This was a multi-billion dollar effort requiring a gigantic particle collider. Future discoveries won’t be any easier.
When you have more and more scientists trying to discover ever more elusive knowledge, there’s bound to be honest mistakes made and papers retracted. Add on all the perverse incentives of academia/industry and you get fraud, too. It’s encouraging to read in these articles that efforts are being made within the scientific community to improve the process of scientific discovery. As science gets harder, it will need to get better.
Not sure about the purpose of this analysis. Is CrossFit departing from science?
On the contrary. This is CrossFit identifying "bad science".
In 2007, Greg Glass published, "Understanding CrossFit" and describes CrossFit as relying on empirical data that is observable, measurable and repeatable.
The argument against CrossFit, it's nutritional recommendations and the methodology have uncovered academic fraud. Sadly, as we are learning from the likes of Uffe Ravnskov, Timothy Noakes, and many others, this problem is systemic.
So, CrossFit is not saying science is bad, the methodology is built upon it, but that there is bad science.
*Greg Glassman
I believe the purpose of this is to show how easy it is to publish biased information. Lets say for example a supplement company says you need 2g of protein per body weight and 2.5g to gain muscle. They can fork out a lot of money to a physician and get a study published to prove that.
Then what happens is everyone thinks they NEED 2g of protein and its hard to eat that much protein so they need.....supplements.
Also, its really easy for companies with lots of money to prove their product not harmful.
I think we all know that a honest mistake can and will always happen. Through some of those honest mistakes its also proven that a flat out lie or payment can cause skewed research.
Is Science Broken?
6