The Cochrane Collaboration, founded in 1993, purportedly exists to “produce high-quality, relevant, up-to-date systematic reviews and other synthesized research to inform health decision making.” Over the past 25 years, Cochrane has produced 7,500 such reviews and influenced the standards and practices by which systematic reviews are done by the academic community at large.
In “The Crisis in Cochrane: Evidence Debased Medicine,” psychiatrist Dr. David Healy reviews the circumstances leading to the removal of one of Cochrane’s co-founders, Peter Gøtzsche, from Cochrane’s board in late 2018.
In 2004, an FDA review found that many published trials of pediatric antidepressant drugs were “either ghost or company written, in all cases the data were inaccessible, and in the case of published studies, the publications were at odds with the data regulators revealed.” These problems fundamentally undermine the idea of systematic review, which implicitly assumes the data presented in published trials reflects experimental reality (Gøtzsche would later write “Deadly Psychiatry and Organised Denial” on the lack of evidence supporting pediatric antidepressants).
In 2007, Gøtzsche asked the European Medicines Agency to release clinical study reports on two slimming pills. EMA refused, arguing that such data was commercially confidential. Gøtzsche complained to the European Ombudsman, resulting in a release of the relevant data in 2010. This was a breakthrough for healthcare research and patient safety. Throughout this period, Gøtzsche further alienated Cochrane leadership with “his forthright manner.” In July 2018, Gøtzsche and two other authors published a critical review of Cochrane’s HPV vaccine review (published in May 2018), in which he criticized a variety of biases and oversights in its construction. Soon after, Gøtzsche was expelled from Cochrane, and nearly half the board resigned in solidarity (Gøtzsche wrote his own editorial on the expulsion for the BMJ). On Nov. 6, 2018, Denmark’s Rigshospitalet, host of the Nordic Cochrane Center, announced that Gøtzsche was suspended from the medical center.
Healy argues that Gøtzsche’s expulsion and the fundamental issues leading to the division represent a critical point for Cochrane and all regulatory bodies purporting to oversee evidence-based medicine. For their review to be seriously considered, they must contend with the problem at hand: A significant share of the evidence supporting pharmaceutical interventions is ghost-written and based on inaccessible data that cannot be scrutinized. Healy further argues that these trends have led to pervasive overmedication without sufficient evidence, and that this overmedication may in fact be shortening life expectancies.
Comments on The Crisis in Cochrane
It is hard to believe that the organisations that claim to be safeguarding our health are the one that fall for the easy profit and $ trap.
Thanks CrossFit for making sure that we know! I wasn't aware of the problem of ghostwritten studies and inaccessible data.
The crisis in Cochrane is the falling of another pillar in medicine. For those of you not familiar with Cochrane, it has been the trusted, gold standard reference for us physicians seeking a review on a medical topic. Cochrane conducts THE systematic reviews of medical interventions. We expect that Cochrane has reviewed the studies in detail, scrutinizing and bringing forth a thorough and unbiased conclusion.
I have read Gotzsche’s criticism of the Cochrane HPV review. I am a Pediatrician and I think it’s exactly the questions we should be asking. He addresses important points that are lacking in the review (proper control arm, inclusion of side effects, and bias). Physicians have been cautious and slow at jumping on this vaccine. HPV administration rates are 49%. By comparison, the Tdap is almost 90%. A thorough, transparent review would be of great value.
One of the biggest concerns for the HPV vaccine has been safety. The Cochrane Review concluded “we did not find an increase risk of serious adverse effects”. Great! Oh wait, except as Gotzsche points out, it’s because no one looked. When searching the database, the search criteria were limited. This is critical when doing a review.
There has been ongoing concern about the risks for POTS and CFS. This is something I counsel patients on. In fact there have multiple studies and reports about the possibility of increased incidences related to the HPV vaccine. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5209415/
So Gotzsche’s criticism is spot on.
When the Cochrane Review was released, it gave a false sense of security. As if all the side effects we were hearing about were just rumors....because Cochrane reviewed it, and if they said it’s not increased....
I’m not opposed to the HPV vaccine. I don’t find Gotzsche’s criticism antivax. I find his criticism is against false reviews. There needs to be checks and balances to ensure the highest quality care. Not just a poorly done review to tell us what we want to hear.
Ghostwritten studies, inaccessible data, cherry-picked results–just a few of the flaws of the research system.
This reminds me of former BMJ editor Richard Smith's writing on peer review, another flaw in the process: "In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused."
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
Smith–who at least at one point was the chair of the Cochrane Library's oversight committee–once compared Gí¸tzsche to the "boy who sees the emperor has no clothes" (https://richardswsmith.wordpress.com/2018/09/17/is-peter-gotzsche-the-boy-who-sees-that-the-emperor-has-no-clothes-and-says-so/) and seems to be on Gí¸tzsche's side (https://twitter.com/Richard56/status/1061349355561594880).
Thanks for linking to Smith's paper. First time I'd read it, and it's fascinating. Might warrant its own post in the future.
There’s a ton to chew on in here, and this passage stood out to me in particular:
~~~
The Lancet has tried to get round the problem by agreeing to consider the protocols (plans) for studies yet to be done. If it thinks the protocol sound and if the protocol is followed, the Lancet will publish the final results regardless of whether they are positive or negative. Such a system also has the advantage of stopping resources being spent on poor studies. The main disadvantage is that it increases the sum of peer reviewing–because most protocols will need to be reviewed in order to get funding to perform the study.
~~~
This seems to echo some of the same foundational principles as efforts like the Open Science Framework:
https://www.osf.io
That is, we would be better insulated from bias (and more able to highlight legitimately excellent studies) if we made that determination based on each study’s methodology, prior to its implementation, rather than on its results. After all, if we assume researchers implement their studies competently, the study’s design, methodology, and the significance of the hypothesis it is testing should tell us all we need to know about how seriously to take its results. Right?
I’m also very curious if his description of peer review as the “least bad option” has evolved over the decade since this was written. Has peer review become less bad? Have alternatives been seriously discussed? Have any journals tested alternative methodologies, for better or worse? Is our endpoint to continue improving peer review or replace it? (Do things like PubMed Commons play a role here, too, or could they be making the problem even worse?)
For many of the same reasons discussed in the original post’s articles on Cochrane, the answers to these questions have an outsize impact on what our scientific and medical communities will subsequently believe and implement.
"ghost-writing has become commonplace because it provides substantial benefits to three parties: drug companies, researchers and medical journals. By managing the publication of articles about their products, it is easier for drug companies to spread positive results and bury negative results. This often provides a big boost to drug sales."
The lack of transparency in such trials is dumbfounding. How it has become possible to release a study but not the data upon which the articles is based is more than frightening.
It's fantastic to see Crossfit diving into this stuff around medical research, and hopefully working its magic of bringing it to public consciousness. The complexity and fragility of the evidence on which many of our health decisions are based is huge and rarely acknowledged.
Some related pieces that interested readers might enjoy include Ionnadis's classic "Why most published research is false" and John Worrall's paper "What Evidence in Evidence Based Medicine?".
There's a big debate among physicians and philosophers about evidence-based medicine. On the one hand its goals seem laudable. On the other, it invokes ideas that are just way too simple, e.g. "evidence hiearchy" where some kinds of studies trump others. The problem with this hierarchy is that it always favours pharmaceutical interventions over lifestyle ones such as diet and exercise. So - don't believe all the rhetoric - but it's a fascinating movement in recent medical thinking, and the general area of how we decide what to believe about these things is fascinating. (I did write a book about it but hesitate to link it in case I'm perceived as pushing for personal profit... which, by the way, is miniscule from an academic book!).
Great stuff and I'm interested to see how Crossfit's move into health develops.
Alex, once again thanks for your comment.
I see you and Shaka may be getting at an underlying point from different angles. It's practically impossible for any individual (certainly not a physician, and likely not even a full-time researcher) to read all the literature relevant to their field with a fraction of the time and diligence required to do it justice. So it seems a modern (maybe pre-modern) necessity for us to put our trust in some sort of external authority to help make these judgments where we lack the time to make them ourselves - whether that's an entity like Cochrane or a methodology that implies certain expectations like EBM. If anything, these are only going to become more necessary over time as the volume of published research increases.
Your statement on the complexity and fragility of these evidence sources resonates. A lot of this discussion is bringing to light how assumptions or choices in research summarization methodologies systematically bias their outputs - which should influence how we treat those outputs.
Obviously we can't throw them out entirely. And there are some equally obvious improvements, as Gí¸tzsche and his colleagues have highlighted. But a key question, which I don't have an answer to, is how we handle conclusions knowing they might be biased or misguided, including in some ways we don't yet appreciate. How do we prevent the sort of errors Shaka referred to with the HPV vaccine without being equivocal to the point of ineffectiveness? If we can't trust Cochrane, where do we go from here?
Motivated individuals can do their own work and synthesize broad guidelines and individual (supportive and discordant) pieces of evidence to get a sense of what we know and how confident we are that we know it. What do we do about the rest?
The Crisis in Cochrane
7