Evidence-Based Medicine, Part 2: Unpublished Evidence

It is essential that all clinical trials carried out are then published, whether positive or negative. Otherwise, the data will be biased. As a simple analogy, imagine a scenario whereby you flip a coin 100 times and only keep a record when the coin comes up heads, discarding every tail event. The recorded results would be completely one-sided. It would appear as though you have achieved a 100% probability of heads vs. tails.

Clearly these results could not be relied on, and anyone wishing to wager on the next flip of the coin would have been completely misled. The reality is that there is not a 100% chance of the coin coming up heads. The chances of it coming up heads vs. tails is 50-50.

In the same way, evidence on medical interventions only has value when all the data can be seen. If only positive results are published while the negative remain hidden, then we have a distorted and flawed evidence base. The inevitable result is that the clinical guidelines and clinical decisions based upon them will also be fundamentally flawed — with extremely damaging consequences.

Unfortunately, in many cases this appears to be what has happened, and it has been going on for many years. Thirty years ago, the issue of “missing data” was highlighted in the paper “Underreporting Research Is Scientific Misconduct”:

Substantial numbers of clinical trials are never reported in print, and among those that are, many are not reported in sufficient detail to enable judgments to be made about the validity of their results. Failure to publish an adequate account of a well-designed clinical trial is a form of scientific misconduct that can lead those caring for patients to make inappropriate treatment decisions. Investigators, research ethics committees, funding bodies, and scientific editors all have responsibilities to reduce underreporting of clinical trials. An extended use of prospective registration of trials at inception, as well as benefiting clinical research in other ways, could help people to play their respective roles in reducing underreporting of clinical trials. (1, my emphasis)

That was written in 1990 in the Journal of the American Medical Association (JAMA). Twenty-four years later, the Journal of the British Medical Association (BMJ) published an article called “Medical Research—Still a Scandal”:

[R]esearch that is completed is not made fully accessible. Half of studies are never published at all, and there is a bias in what is published, meaning that treatments may seem to be more effective and safer than they actually are. Then not all outcome measures are reported, again with a bias towards those are positive. (2, my emphasis)

Even more recently (May 7, 2019), an article was published in the Annals of Internal Medicine entitled “Not Reporting Results of a Clinical Trial Is Academic Misconduct,” highlighting the fact, once more, that many clinical trials are still not reported:

Failure to report the results of clinical trials threatens the public’s trust in research and the integrity of the medical literature, and should be considered academic misconduct at the individual and institutional levels. (3)

As is made clear in this paper, the underreporting of clinical trials continues despite the fact that, in 2006, new rules were introduced to ensure all clinical trials be registered (4). Clearly, this legislation has had little or no effect.

Examples of withholding data

Perhaps the most widely known example of “hidden” data is in the field of psychiatry, specifically with the newer antidepressants. These are among the most widely prescribed medications in the Western world.

Unusually, following from a legal challenge, all the data (published and unpublished) were made available for review. The issues with the previously unpublished data were discussed by John Ioannidis in the paper “Effectiveness of Antidepressants: An Evidence Myth Constructed From a Thousand Randomized Trials?”:

I discuss how the use of many small randomized trials with clinically non-relevant outcomes, improper interpretation of statistical significance, manipulated study design, biased selection of study populations, short follow-up, and selective and distorted reporting of results has built and nourished a seemingly evidence-based myth on antidepressant effectiveness. (5, my emphasis)

Ioannidis then reviewed the larger trials used for FDA registration. Of the 74 trials including 12,564 patients, he found:

• 38 out of 74 trials were considered positive, with benefits for the antidepressant
• 37 of these trials were published
• 3 negative studies were published as negative
• 11 negative trials were published, but the results were presented in such a way as to make them seem positive
• 22 negative trials were silenced and never appeared in the literature

“These data suggest that antidepressants may be less effective than their wide marketing suggests. Short-term benefits are small and long-term balance of benefits and harms is understudied,” Ioannidis concluded.

Another drug that has become very widely prescribed is gabapentin, which has many different indications and has been the subject of litigation and a whistleblowing lawsuit by Dr. David Franklin, as outlined in the article “Pfizer Pleads Guilty, But Drug Sales Continue to Soar”:

Dr. Franklin’s suit detailed how the company suppressed study results, planted people in medical audiences to ask questions intended to put gabapentin in a good light, lavished perks on doctors, used ghostwriters, gave generous “consultation fees” to “thought leaders,” and used psychological profiling of doctors in its successful bid to move gabapentin to so-called blockbuster status (annual sales in excess of $1bn). Dr. Franklin said that off-label uses accounted for more than 90% of the drug’s $2.7bn sales worldwide last year. (6)

Another paper, “Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use,” analyzed the evidence used to promote the off-label prescribing of gabapentin.

The authors found that pre-specified outcomes were changed in the majority of studies, as were protocols and, once again, most negative studies were never published at all:

More than half the clinical trials that we included in our analysis (11 of 20) were not published as full-length research articles. For 7 of the 9 trials that were published as full-length research articles, a statistically significant primary outcome was reported, and for more than half these trials, the outcome specified in the published report differed from the outcome originally described in the protocol. (7, my emphasis)

The issue of unpublished data is a major unresolved problem in medical research that has not been adequately tackled. Guidelines are written and clinical decisions made based on a database that is missing an unknown number of unpublished studies and negative data.

It can therefore be argued that evidence-based medicine, and therefore the entire medical research database, has been corrupted to the point that it is not just unhelpful but potentially extremely damaging to health.

Additional Reading

• Evidence-Based Medicine, Part 1: Industry Distorts Clinical Priorities

Malcolm Kendrick is a family practitioner working near Manchester in England. He has a special interest in cardiovascular disease, what causes it, and what may prevent it. He has written three books: The Great Cholesterol Con, Doctoring Data, and A Statin Nation. He has authored several papers in this area and lectures on the subject around the world. He also has a blog, drmalcolmkendrick.org, which stimulates lively debate on a number of different areas of medicine, mainly heart disease.

He is a member of THINCS (The International Network of Cholesterol Sceptics), which is a network of doctors and scientists who believe that cholesterol is not the main underlying cause of heart disease. He remains a proud Scotsman, whisky drinker, and failed fitness fanatic who loves a good scientific debate — in the bar.

References

1. Chalmers I. Underreporting research is scientific misconduct. JAMA 263.10(March 1990): 1405-8. Available here.
2. Smith R. Medical research—Still a scandal. BMJ Opinion (Jan. 31, 2014). Available here.
3. Wallach JD, Krumholz HM. Not reporting results of a clinical trial is academic misconduct. Ann. of Int. Med. 171.4(2019): 293-294. Available here.
4. Miossec M, Miossec P. New regulatory rules for clinical trials in the United States and the European Union. Arth. & Rheum. 54.12(Dec. 2006): 3735–3740. Available here.
5. Ioannidis J. Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials? Philos. Ethics Humanit. Med. 3(2008): 14. Available here.
6. Lenzer J. Pfizer pleads guilty, but drug sales continue to soar. BMJ. 328.7450(May 2004): 1217. Available here.
7. Vedula SS, et al. Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use. N. Engl. J. Med. 361(2009): 1963-1971. Available here.