The Challenge of Reforming Nutritional Epidemiologic Research

Question
How significant are the observable flaws in the design, analysis, and reporting of nutritional epidemiology?
Takeaway
Looking solely at whether the analysis and reporting of nutritional epidemiology are supported by “good scientific principles,” John Ioannidis argues the field is fundamentally flawed. This form of research regularly and predictably produces implausible correlations, which are then described using causal language that influences public thought and policy. Given the complexity of the modern diet and the many correlations between different dietary components, it is difficult if not impossible to isolate the health benefit (or harm) of any individual dietary factor through observational analysis.

In this 2018 commentary, John Ioannidis argues the reporting and analysis of nutritional epidemiology are fundamentally flawed. He writes, “The emerging picture of nutritional epidemiology is difficult to reconcile with good scientific principles. The field needs radical reform.”

Ioannidis reviews some of the dramatic — even implausible — claims that have emerged from epidemiological research including:

• Eating 12 hazelnuts per day will extend lifespan by 12 years; drinking three cups of coffee per day will have the same effect.
• Eating one mandarin orange each day will extend lifespan by five years.
• Consuming one egg daily will shorten lifespan by six years, and two daily strips of bacon will shorten lifespan by a decade. The latter effect is greater than that of smoking.

The authors of such studies generally describe these correlations using causal language, such as  “optimal consumption of risk-decreasing foods results in a 56% reduction in all-cause mortality.” Media reports often use causal language even when the authors do not.

Ioannidis argues these effects do not represent the true impact of individual food items but rather the cumulative biases present in this sort of research. Consumption of each food item is correlated, positively or negatively, with hundreds of others, and with an unknown number of nondietary factors that affect health (e.g., geography, education, income, smoking habits). We do not have a sufficient enough understanding of how these factors correlate with one another to correct for their impact, and given that there are over 250,000 foods in the U.S. food supply, fully understanding these correlations may be impossible. As a result, a single correlation drawn from nutritional epidemiology reflects any true effect that food item may have and the various confounders it is correlated with to an unknown extent.

Given this ambiguity and the thousands of possible correlations drawn from any epidemiological data set, it is easy for researchers to favor the correlations most consistent with their existing beliefs about diet and disease. Meta-analyses become “weighted averages of expert opinions” rather than objective assessments of the data. This is exacerbated by the fact that analyses of a single data set can be split across hundreds or thousands of papers; the Nurses’ Health Study alone has yielded more than 1,000 publications. When each association is reported in isolation, it is given exaggerated significance and not seen for what it is: one of thousands of possible associations, each confounded, which could have been drawn from the same data

At minimum, improving the quality of nutritional epidemiology will require publicizing the data from large trials and discarding any past claims in public health, media, or policy drawn from epidemiological studies’ causal claims. Future reform will require greater data transparency, data sharing, and a shift in how this research is presented and analyzed within and beyond the research community.