For many years, there has been concern about potential bias in clinical research. Many trials are never published (1), and this is significantly more likely if a trial has negative results.
To try to prevent the silencing of negative studies, the AllTrials campaign was established to ensure that, first, all clinical trials were registered, and second, that they be published along with all their data, whether positive or negative. As part of its work, the AllTrials group identified that fewer than half of clinical trials are published (1).
In 2008, John Ioannidis analyzed the impact of not publishing negative data in a review of antidepressants with newer agents (2). Seventy-four trials were identified and categorized as follows:
- 38 of the 74 trials were considered positive, finding benefits for the antidepressant.
- 37 of the positive trials were published.
- 3 negative studies were published as negative.
- 11 negative trials were published with the results presented in such a way as to make them seem positive.
- 22 negative trials were effectively silenced and never appeared in the literature.
Clearly, if the vast majority of positive studies are published while negative trials are not, this will seriously bias the research database and inevitably have an impact on any treatment guidelines produced.
Even if a trial is published, there are additional sources of bias. For example, a significant percentage of studies have had their endpoints altered while the trial is ongoing. This is more likely to occur if it appears that a primary outcome is not going to reach statistical significance.
In order to try to address both problems, in the 1990s, the U.S. authorities made attempts to ensure that clinical trials were fully registered. This requirement made it more difficult to silence research data and/or adjust outcomes mid-trial.
The Consolidated Standards of Reporting Trials (CONSORT) group was introduced in 1996 and declared that studies must provide their purpose, recruitment status, designs, eligibility criteria, locations, and pre-specified primary and secondary outcomes. These standards were expanded in 2001, journal editors began to require CONSORT reporting in 2004, and the full guidelines were introduced in 2006.
The National Heart, Lung, and Blood Institute (NHLBI) was an early adopter of trial registration, and all large trials funded by the NHLBI published after 2000 were to be pre-registered and transparently reported.
A study was done in 2015 to establish whether the introduction of the NHLBI guidelines in 2000 had made a difference to the reported outcomes. It was called “Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased Over Time.”
The study presented the following findings:
17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2 =12.2, df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. (3)
In effect, prior to the introduction of new and more wide-ranging clinical trial regulations, the proportion of (published) clinical trials reporting a positive result was 57%. After this time, the proportion of positive clinical trials fell to 8% — a seven-fold reduction.
It was not until 2006 that new guidelines that mirrored the CONSORT trial reporting standard were introduced for pharmaceutical trials.
As Marie and Pierre Miossec explain, “In both Europe and the U.S., regulations concerning the conduct of clinical trials have recently changed. Every report on a clinical trial must now show verification of registration of that trial in a recognized database” (4).
In the case of statins, the vast majority of randomized controlled trials (RCTs) used to create clinical guidelines were performed before 2006. They all reported positive outcomes, apart from a single non-industry-sponsored trial called the “Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial” (ALLHAT-LLP). This trial was sponsored by the NHLBI. The main findings were as follows:
No benefit was found when pravastatin was given for primary prevention to older adults with moderate hyperlipidemia and hypertension, and a nonsignificant direction toward increased all-cause mortality with pravastatin was observed among adults 75 years and older. (5)
In addition to the single negative (non-pharmaceutical-company-funded) trial, could there have been negative statin studies that were not published? This is difficult to establish. A trial that was never registered and then not published could not easily be found. However, it is clearly possible such studies may have been produced.
Leaving this issue to one side, it has been clearly established that clinical trials carried out prior to the introduction of CONSORT standards were significantly more likely to report positive results. We also know that almost all the major statin trials were done before 2006.
Have there been changes in the reported outcomes of statin trials since that date? This question was addressed by Michel de Lorgeril and Mikael Rabaeus in a review called “Beyond Confusion and Controversy, Can We Evaluate the Real Efficacy and Safety of Cholesterol-Lowering With Statins?” The two reviewers explained their impetus:
A strong controversy has emerged about the reality of safety and efficacy of statins as stated by company-sponsored reports. However, physicians need credible data to make medical decisions, in particular about the benefit/harm balance of any prescription. This study aimed to test the validity of data on the company-sponsored statin trial by comparing them over time and then comparing statins with each other.
Around the years 2005/2006, new stricter regulations were introduced in the conduct and publication of randomized controlled trials (RCTs). This would imply that RCTs were less reliable before 2006 than they were later on. To evaluate this, we first reviewed RCTs testing the efficacy of statins versus placebo in preventing cardiovascular complications and published after 2006. Our systematic review thereby identified four major RCTs, all testing rosuvastatin. (6)
De Lorgeril and Rabaeus found:
[The trials after 2006] unambiguously showed that rosuvastatin is not effective in secondary prevention, while the results are highly debatable in primary prevention. Because of the striking clinical heterogeneity and the inconsistency of the published data in certain RCTs, meta-analysis was not feasible. We then examined the most recent RCTs comparing statins to each other: all showed that no statin is more effective than any other, including rosuvastatin.
Furthermore, recent RCTs clearly indicate that intense cholesterol-lowering (including those with statins) does not protect high-risk patients any better than less-intense statin regimens. As for specific patient subgroups, statins appear ineffective in chronic heart failure and chronic kidney failure patients. (6)
They concluded:
The studies published before 2005/2006 were probably flawed, and this concerned in particular the safety issue. A complete reassessment is mandatory. Until then, physicians should be aware that the present claims about the efficacy and safety of statins are not evidence based. (6)
If this is correct, it is of great concern. Statins are the most widely prescribed medications worldwide. Guidelines have consistently recommended their use in increasingly wide populations. In the U.S., around 40 million people take statins.
With many medications, a clinician can observe if a medication is providing benefits. An antibiotic will cure an infection, a painkiller reduces pain, a skin cream can have a positive impact on irritation and rash, and suchlike. However, with statins, it is impossible to know if there is any benefit for any individual patient. The only sign that the statins are “working” is through lowering a “surrogate” endpoint: the LDL level.
With statins, doctors are entirely reliant on clinical trials to guide their prescribing. If those trials have been biased or silenced, the database that underpins the clinical guidelines will inevitably be wrong and needs to be reviewed.
As De Lorgeril and Rabaeus conclude: “A complete reassessment is mandatory. Until then, physicians should be aware that the present claims about the efficacy and safety of statins are not evidence based.”
Additional Reading
Malcolm Kendrick is a family practitioner working near Manchester in England. He has a special interest in cardiovascular disease, what causes it, and what may prevent it. He has written three books: The Great Cholesterol Con, Doctoring Data, and A Statin Nation. He has authored several papers in this area and lectures on the subject around the world. He also has a blog, drmalcolmkendrick.org, which stimulates lively debate on a number of different areas of medicine, mainly heart disease.
He is a member of THINCS (The International Network of Cholesterol Sceptics), which is a network of doctors and scientists who believe that cholesterol is not the main underlying cause of heart disease. He remains a proud Scotsman, whisky drinker, and failed fitness fanatic who loves a good scientific debate — in the bar.
References
- All trials registered | All results reported. 20 Aug. 2015. Available here.
- Ioannidis J. Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials? Philos Ethics Humanit Med. 3.14(2008). Available here.
-
Kaplan RM, Irvin VL. Likelihood of null effects of large NHLBI clinical trials have increased over time. PLoS ONE. 10.8(2015). Available here.
- Miossec M, Miossec P. New regulatory rules for clinical trials in the United States and the European Union. Arthritis and Rheumatism. 54.12(Dec. 2006): 3735-3740. Available here.
- Han BH, et al. Effect of statin therapy vs. usual care on primary cardiovascular prevention among older adults. JAMA Intern Med. 177.7(July 2017): 955-965. Available here.
- De Lorgeril M, Rabaeus M. Beyond confusion and controversy, can we evaluate the real efficacy and safety of cholesterol-lowering with statins? J. of Controversies in Biomed. Research. 1.1(2015): 67-92. Available here.
Statins — Are the Data Free From Bias?