\nOnline Courses: Contact us at seminars@crossfit.com \nCrossFit Games Support: Reach out to support@crossfitgames.com \nFor T-Shirt Purchase: Check your inbox for instructions from customer@trackorder.site
\n","orderHistoryHeadlineText":"Order History","orderHistoryLabelText":"Order History","viewOrderDetailsText":"View order details"},"resourcesKickerText":"Resources","rulebookLabelText":"Rulebook","rulebookLinkUrl":"https://games.crossfit.com/rules","preferenceCenter":{"checkAllText":"Check All","emailCommunicationBodyText":"Select which lists you would like to receive communication from","emailCommunicationHeadlineText":"Email Communication","newsletters":[{"bodyText":"Every day since 2001, CrossFit has published a Workout of the Day for new CrossFit athletes, seasoned veterans, and anyone in between. Subscribe to Workout of the Day emails, and you'll receive CrossFit programming in a three-days-on, one-day-off, two-days-on, one-day-off cadence.","brazeGroupId":"d68551b9-ac7e-437e-bfd2-d225702413cc","headlineText":"Workout of the Day"},{"bodyText":"Get inspired with the latest and/or trending articles, stories, videos, and podcasts focused on the CrossFit community and methodology.","brazeGroupId":"1bfd806e-efe3-404d-9980-c8f38c39490b","headlineText":"What's Trending"},{"bodyText":"Be the first to know when official updates and announcements are released from CrossFit HQ.","brazeGroupId":"cb28cb07-ca09-4739-b73b-7413f95a4892","headlineText":"Updates and Announcements from CrossFit HQ"},{"bodyText":"This subscription is for those interested in announcements, updates, and stories about the CrossFit Games season and its competitors. ","brazeGroupId":"e56abfcc-c940-482d-879c-eed93958cbbe","headlineText":"CrossFit Games"},{"bodyText":"This monthly newsletter includes important CrossFit Games competition season updates, resources, and tips. It is intended for competitive athletes but is open to anyone interested in learning more about the CrossFit Games season.","brazeGroupId":"fce49890-9b3c-4e1f-9e06-1bd06cd696a3","headlineText":"The Hopper"},{"bodyText":"The Professional Coach provides a wealth of articles, media, and webinar opportunities that cover all aspects of coaching, including best practices, real-world coaching application, scaling, methodology, nutrition, CEU opportunities, and more. With contributions from experts inside and outside the CrossFit space, this twice-monthly newsletter is the ultimate tool for any coach looking to improve their skills and knowledge. This newsletter is exclusively available to those who hold a Level 1 Certificate or higher. ","brazeGroupId":"653207fb-9734-44c9-a988-23eb18142529","headlineText":"The Professional Coach"},{"bodyText":"The Affiliate Update is a newsletter for CrossFit affiliate owners and includes upcoming event details, business and marketing resources, and general updates from CrossFit HQ. The Affiliate Update is only available to licensees of record (LOR).","brazeGroupId":"1d3e4c63-f03f-4b61-8453-1fbb186ae27e","headlineText":"CrossFit Affiliate Update"},{"bodyText":"An email series for learning what it takes to open and run a successful CrossFit gym","brazeGroupId":"6ab8a53a-167c-4c74-b5ea-3a06cb857068","headlineText":"Interested in Affiliation"},{"bodyText":"CAP is a weekly email available exclusively to affiliate owners that provides comprehensive class plans, resources, and educational tips to help coaches deliver an exceptional experience to their members.","brazeGroupId":"ba7e977e-ec76-41e3-8465-158d38167ed6","headlineText":"CrossFit Affiliate Programming (CAP)"},{"bodyText":"CrossFit Health serves as a vital bridge connecting the realms of fitness and healthcare. Committed to delivering valuable insights, CrossFit Health harnesses the expertise of renowned professionals from the health, wellness, and CrossFit communities. Our mission is to educate individuals on the transformative potential of CrossFit as a powerful lifestyle intervention, ultimately optimizing health outcomes.","brazeGroupId":"a9c5a8a4-df0a-4b0c-acb6-e9b4631fc00c","headlineText":"CrossFit Health"},{"bodyText":"Be the first to know when new products are added to the Official CrossFit Store and receive special offers and discounts conveniently delivered directly to your inbox.","brazeGroupId":"905db0be-a3c8-4f5b-8903-536a04cb0a40","headlineText":"The CrossFit Store"},{"bodyText":"This subscription is for companies interested in on-site event activations and vendor booths at the CrossFit Games and other events hosted by CrossFit.","brazeGroupId":"92fa7fe8-262a-4d6a-9b0d-9fcc8cc17e7d","headlineText":"Interested in On-Site Event Activations / Vendor Booths"},{"bodyText":"Get alerted for upcoming courses. CrossFit courses and certifications are open to individuals and trainers seeking to improve their health and fitness through effective training and nutritional strategies.","brazeGroupId":"dd1ee7a4-2ef5-477b-affb-77347ce81684","headlineText":"Interested in Finding a Course Near You"}],"preferenceCenterHeadlineText":"Newsletter Preferences","preferenceCenterLabelText":"Newsletter Preferences","uncheckAllText":"Uncheck All"}}},"pages":{"breadcrumbs":{"links":[{"text":"Health","url":"/health"},{"text":"Deming, Data and Observational Studies","url":"#"}]},"contentPublishDate":"20200118","contentType":"article","commentTopics":[{"title":"Deming, Data and Observational Studies","topicId":"article.20200102101644834"}],"title":"Deming, Data and Observational Studies","topicId":"article.20200102101644834","socialMetaData":{"title":"Deming, Data and Observational Studies","image":"https://www.crossfit.com/wp-content/uploads/2020/01/13124608/Deming-Significant-16x9.png"},"path":"/health/deming-data-and-observational-studies","components":[{"name":"ArticleHeader","props":{"topicId":"article.20200102101644834","date":"20200118","articleDate":"20200118","authorName":"CrossFit","bylineText":"By","headlineText":"Deming, Data and Observational Studies"}},{"name":"TextBlock","props":{"children":"
In this 2011 editorial, S. Stanley Young and Alan Karr, of Bioinformatics and the National Institute of Statistical Sciences, respectively, present evidence that the results of observational trials (1) are more likely to be false than true.
\n
The validity of observational evidence has been questioned since at least 1988, when L. C. Mayes et al. found that across 56 topic areas, an average of 2.4 studies could be found to support a particular observational claim alongside 2.3 studies that did not (2). In the case of the antidepressant reserpine, for example, three studies indicated the drug increased cancer risk, while eight did not.
\n
More recently, an analysis of 49 claims from highly cited studies found five of six claims drawn from observational studies — 83% — failed to replicate (3). The authors of the analysis gathered 12 randomized controlled trials (RCTs), which collectively tested the claims of 52 observational trials. Zero of the 52 observational claims were supported by the subsequent RCTs, with the RCTs finding significant effects in the opposite direction of the observational evidence (i.e., an RCT indicated harm when the observational study indicated health, or vice versa) in five out of 52 claims. Young and Karr conclude:
\n
There is now enough evidence to say what many have long thought: that any claim coming from an observational study is most likely to be wrong — wrong in the sense that it will not replicate if tested rigorously.
\n
The authors cite W. Edwards Deming, who introduced the concept of quality control, to argue this scale of failure suggests the process by which observational studies are generated is fundamentally flawed. They specifically point to three factors that consistently lead to unreliable observational evidence:
\n
Multiple testing – A single observational study can test many associations simultaneously. For example, as mocked by the xkcd webcomic below, a single study can test for correlations between dozens of dietary and demographic inputs and clinical outcomes. Mere chance indicates some of these correlations will appear statistically significant, even if they lack any biological basis.
\n
\n
Bias – Data is often confounded by unobserved distortions. For example, the HIV drug abacavir was primarily given to patients at high risk of cardiovascular disease. Observational evidence concluded abacavir increased risk of heart disease when, in fact, abacavir was simply being given to patients who were more likely to have a heart attack, with or without the drug (4).
\n
Multiple modeling – As with multiple testing, the regression models used to uncover correlations between variables in observational data can be manipulated until significant associations fall out. Here, the authors note the supposed harms of BPA were derived from an observational study that tested the urine of 1,000 people for 275 chemicals, then compared urine levels of these chemicals to 32 medical outcomes and 10 demographic variables. From this data set, 9 million possible regression models can be constructed, many of which will show significant associations as a result of chance alone (5).
\n
These problems are exacerbated by the widespread reliance on p-values (i.e., statistical significance) to identify meaningful findings. Authors, journal editors, and those who read and interpret science treat statistically significant associations as meaningful and real, even when these significant associations may entirely result from the distortions above.
\n
The authors propose a fix to this process that involves researchers splitting their data prior to analysis, analyzing part using a published statistical methodology, publishing the paper using only this data, and then analyzing the remaining data using the same methodology in an addendum. This method provides a sort of internal replication: If the addendum data informs the same conclusions as the main body of data, these conclusions are more likely to be valid, and if not, the authors’ analytical methods can be questioned.
\n
The takeaway: This paper highlights the extreme degree to which observational evidence is unreliable. Based on the data these authors present, observational evidence is no more likely to be right than wrong, and in fact, it may be more likely to be wrong than right. In other words, any health claim based on observational evidence is only valuable according to the extent to which it informs subsequent controlled trials to test its validity.
\n"}},{"name":"Button","props":{"variant":"secondaryBlue","children":"Read the editorial","url":"https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.1740-9713.2011.00506.x"}},{"name":"TextBlock","props":{"children":"\n
Notes
\n\n
That is, trials in which the authors observe whether particular inputs (i.e., patterns in diet, behavior, demography, etc.) correlate with increased or decreased risk of disease, without attempting to affect those inputs.
\n","hideText":"Hide","status":"active","type":"alert_banner","userCanHide":"no"},"header":{"desktopLinks":[{"below":[{"id":93961,"parent":"93960","target":"_self","text":"What is CrossFit?","url":"/what-is-crossfit"},{"id":93962,"parent":"93960","target":"_self","text":"Find a Gym","url":"/map"}],"id":93960,"parent":"0","target":"_self","text":"Getting Started","url":"/what-is-crossfit"},{"below":[{"id":93964,"parent":"93963","target":"_self","text":"About","url":"/education/about"},{"id":93966,"parent":"93963","target":"_self","text":"Level 1 Certificate Course","url":"/certificate-courses/level-1"},{"id":93965,"parent":"93963","target":"_self","text":"Explore All Courses","url":"/education/explore-courses"},{"id":109033,"parent":"93963","target":"_self","text":"Private Courses","url":"/private-course/"},{"id":109034,"parent":"93963","target":"_self","text":"Military Resources","url":"/army-ignited"},{"id":93967,"parent":"93963","target":"_self","text":"Coaching Resources","url":"/education/resources"}],"id":93963,"parent":"0","target":"_self","text":"Courses","url":"/education/about"},{"below":[{"id":95951,"parent":"93968","target":"_self","text":"Workout of the Day","url":"/wod"},{"id":93969,"parent":"93968","target":"_self","text":"Explore Workouts","url":"https://www.crossfit.com/workout"},{"id":101892,"parent":"93968","target":"_self","text":"Hero Workouts","url":"https://www.crossfit.com/heroes/"},{"id":93970,"parent":"93968","target":"_self","text":"Movements","url":"https://www.crossfit.com/crossfit-movements"}],"id":93968,"parent":"0","target":"_self","text":"Workouts","url":"https://www.crossfit.com/workout"},{"id":93971,"parent":"0","target":"_self","text":"Learn","url":"/media"},{"id":93972,"parent":"0","target":"_blank","text":"Games","url":"https://games.crossfit.com"},{"id":105467,"parent":"0","target":"_blank","text":"Store","url":"https://store.crossfit.com/?utm_source=owned&utm_medium=nav&utm_campaign=main"},{"id":106137,"parent":"0","target":"_self","text":"Open A Gym","url":"/open-crossfit-gym"}],"mobileLinks":[{"text":"Open a CrossFit Gym"},{"text":"Browse","links":[{"below":[{"id":93961,"parent":"93960","target":"_self","text":"What is CrossFit?","url":"/what-is-crossfit"},{"id":93962,"parent":"93960","target":"_self","text":"Find a Gym","url":"/map"}],"id":93960,"parent":"0","target":"_self","text":"Getting Started","url":"/what-is-crossfit"},{"below":[{"id":93964,"parent":"93963","target":"_self","text":"About","url":"/education/about"},{"id":93966,"parent":"93963","target":"_self","text":"Level 1 Certificate Course","url":"/certificate-courses/level-1"},{"id":93965,"parent":"93963","target":"_self","text":"Explore All Courses","url":"/education/explore-courses"},{"id":109033,"parent":"93963","target":"_self","text":"Private Courses","url":"/private-course/"},{"id":109034,"parent":"93963","target":"_self","text":"Military Resources","url":"/army-ignited"},{"id":93967,"parent":"93963","target":"_self","text":"Coaching Resources","url":"/education/resources"}],"id":93963,"parent":"0","target":"_self","text":"Courses","url":"/education/about"},{"below":[{"id":95951,"parent":"93968","target":"_self","text":"Workout of the Day","url":"/wod"},{"id":93969,"parent":"93968","target":"_self","text":"Explore Workouts","url":"https://www.crossfit.com/workout"},{"id":101892,"parent":"93968","target":"_self","text":"Hero Workouts","url":"https://www.crossfit.com/heroes/"},{"id":93970,"parent":"93968","target":"_self","text":"Movements","url":"https://www.crossfit.com/crossfit-movements"}],"id":93968,"parent":"0","target":"_self","text":"Workouts","url":"https://www.crossfit.com/workout"},{"id":93971,"parent":"0","target":"_self","text":"Learn","url":"/media"},{"id":93972,"parent":"0","target":"_blank","text":"Games","url":"https://games.crossfit.com"},{"id":105467,"parent":"0","target":"_blank","text":"Store","url":"https://store.crossfit.com/?utm_source=owned&utm_medium=nav&utm_campaign=main"},{"id":106137,"parent":"0","target":"_self","text":"Open A Gym","url":"/open-crossfit-gym"}]}],"brandLinks":[{"id":93974,"parent":"0","target":"_self","text":"Open a CrossFit Gym","url":"/open-crossfit-gym"}],"loggedInMenu":[{"id":72863,"parent":"0","target":"_self","text":"Manage Account","url":"https://www.crossfit.com/dashboard"},{"id":72864,"parent":"0","target":"_self","text":"Competition Dashboard","url":"https://games.crossfit.com/manage-competition/athlete"}]},"footer":{"mainLinks":[{"below":[{"id":9764,"parent":"511","target":"_self","text":"What Is CrossFit?","url":"/what-is-crossfit"},{"id":9765,"parent":"511","target":"_self","text":"Get Started","url":"/get-started"},{"id":45861,"parent":"511","target":"_self","text":"Workouts","url":"/workout/"},{"id":45862,"parent":"511","target":"_self","text":"Movements","url":"/essentials/movements"},{"id":1741,"parent":"511","target":"_self","text":"FAQ","url":"/faq"},{"id":90183,"parent":"511","target":"_self","text":"Help Center","url":"https://crossfit.my.site.com/Support/s/"},{"id":45103,"parent":"511","target":"_self","text":"Careers","url":"/careers/"}],"id":511,"parent":"0","target":"_self","text":"About CrossFit","url":"#"},{"below":[{"id":45864,"parent":"45863","target":"_self","text":"Courses Near You","url":"/courses-near-you/"},{"id":45865,"parent":"45863","target":"_self","text":"Certificate Courses","url":"/certificate-courses/"},{"id":45866,"parent":"45863","target":"_self","text":"Certifications","url":"/certifications/"},{"id":45868,"parent":"45863","target":"_self","text":"Online Courses","url":"https://www.crossfit.com/online-courses"}],"id":45863,"parent":"0","target":"_self","text":"Education","url":"#"},{"below":[{"id":45870,"parent":"45871","target":"_self","text":"Open a CrossFit Gym","url":"/open-crossfit-gym"},{"id":45884,"parent":"45871","target":"_self","text":"Field Leaders","url":"/field-leaders"},{"id":90935,"parent":"45871","target":"_self","text":"Global Mentor Program","url":"/global-mentor-program/"},{"id":93982,"parent":"45871","target":"_self","text":"Affiliate Portal","url":"https://affiliate.crossfit.com/tools/"}],"id":45871,"parent":"0","target":"_self","text":"Affiliates","url":"#"},{"below":[{"id":16638,"parent":"45875","target":"_self","text":"Find a Trainer","url":"https://trainerdirectory.crossfit.com/"},{"id":45876,"parent":"45875","target":"_self","text":"Scholarship Program","url":"/scholarship-program-inquiry"},{"id":45877,"parent":"45875","target":"_self","text":"Foundation","url":"/foundation"},{"id":104122,"parent":"45875","target":"_self","text":"CrossFit Medical Society","url":"https://www.crossfitmedicalsociety.com/"}],"id":45875,"parent":"0","target":"_self","text":"Community","url":"#"},{"below":[{"id":45881,"parent":"45879","target":"_self","text":"About the Games","url":"https://games.crossfit.com/"},{"id":45880,"parent":"45879","target":"_self","text":"Leaderboard","url":"https://games.crossfit.com/leaderboard/games/2022?division=2"},{"id":45882,"parent":"45879","target":"_self","text":"Schedule","url":"https://games.crossfit.com/games/schedule?_ga=2.142929340.802559833.1690817966-401013740.1685640771"},{"id":91727,"parent":"45879","target":"_self","text":"Workouts","url":"https://games.crossfit.com/workouts/games/2023"}],"id":45879,"parent":"0","target":"_self","text":"The CrossFit Games","url":"#"}],"brandLinks":[],"socialLinks":[{"id":66715,"parent":"0","target":"_self","text":"facebook","url":"https://www.facebook.com/crossfit"},{"id":66716,"parent":"0","target":"_self","text":"twitter","url":"https://twitter.com/CrossFit"},{"id":66717,"parent":"0","target":"_self","text":"instagram","url":"https://www.instagram.com/crossfit/"},{"id":66718,"parent":"0","target":"_self","text":"youtube","url":"https://www.youtube.com/channel/UCtcQ6TPwXAYgZ1Mcl3M1vng"}],"contactLinks":[{"id":531,"target":"_self","text":"Careers","url":"https://www.crossfit.com/jobs"},{"id":532,"target":"_self","text":"Contact Us","url":"https://www.crossfit.com/contact-us"}],"legalLinks":[{"id":533,"parent":"0","target":"_self","text":"Terms & Conditions","url":"https://www.crossfit.com/terms-and-conditions"},{"id":534,"parent":"0","target":"_self","text":"Privacy Policy","url":"https://www.crossfit.com/privacy-policy"},{"id":10717,"parent":"0","target":"_self","text":"Cookie Policy","url":"https://www.crossfit.com/cookie-policy"},{"id":57312,"parent":"0","target":"_self","text":"Disclaimer","url":"https://www.crossfit.com/disclaimer"},{"id":10883,"parent":"0","target":"_self","text":"Contact Us","url":"https://www.crossfit.com/contact-us"},{"id":45883,"parent":"0","target":"_self","text":"Report IP Theft","url":"https://www.crossfit.com/iptheft"}]},"locationPages":{"simpleTiles":{"cards":[{"bodyText":"CrossFit offers a results-based, community-driven approach to functional training that helps you build strength and improve your health—over your lifetime.","headlineText":"Functional Fitness for Everyday Life","icon":{"alt":"Lifting Icon","url":"https://www.crossfit.com/wp-content/uploads/2024/04/26123658/crossfit-icon-workout-barbell-overhead-press.svg"}},{"bodyText":"Stay motivated with our community of coaches and members, who support your goals through approachable group training classes, helping you stay accountable and excited for each session.","headlineText":"Community Support and Motivation","icon":{"alt":"Heart Icon","url":"https://www.crossfit.com/wp-content/uploads/2024/04/26123700/crossfit-icon-heart.svg"}},{"bodyText":"Coaches scale class workouts to your fitness level to help you achieve your strength building goals.","headlineText":"Tailored to Your Fitness Levels","icon":{"alt":"Graph Icon","url":"https://www.crossfit.com/wp-content/uploads/2024/04/26123649/crossfit-icon-levels.svg"}},{"bodyText":"CrossFit coaches are experienced and trained to provide well-rounded and personalized guidance to individuals with varying fitness levels and goals.","headlineText":"Industry Certified Coaches","icon":{"alt":"Certification Icon","url":"https://www.crossfit.com/wp-content/uploads/2024/04/26123656/crossfit-icon-check-badge.svg"}}],"headlineText":"Build Strength for a Fuller Life"},"gymFinderPromo":{"bodyText":"Contact CrossFit to start your fitness journey. We’ll connect you to the closest gym in your area.","headlineText":"Get Started today","image":{"alt":"Man Lifting Barbell","sizes":{"1536x1536":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1.jpeg","1536x1536Height":512,"1536x1536Width":768,"2048x2048":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1.jpeg","2048x2048Height":512,"2048x2048Width":768,"hero":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1.jpeg","heroHeight":512,"heroWidth":768,"heroLarge":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1.jpeg","heroLargeHeight":512,"heroLargeWidth":768,"large":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1.jpeg","largeHeight":512,"largeWidth":768,"medium":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1.jpeg","mediumHeight":512,"mediumWidth":768,"mediumLarge":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1.jpeg","mediumLargeHeight":512,"mediumLargeWidth":768,"small":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1-300x200.jpeg","smallHeight":200,"smallWidth":300,"thumbnail":"https://www.crossfit.com/wp-content/uploads/2018/12/13135656/aa6daff2983e555e3688654515568d25-1-768x512-1-150x150.jpeg","thumbnailHeight":150,"thumbnailWidth":150}},"links":[{"link":{"priority":"primary","text":"Find a Gym","url":"/map"}},{"link":{"priority":"secondary","text":"What Is CrossFit?","url":"/what-is-crossfit"}}]},"locationCourseCards":{"courseLinkText":"Register Now","findCourseLink":{"text":"Find More Courses","url":"https://www.crossfit.com/courses-near-you"},"headerText":"Courses Near You","seminarType":["level-one","level-two"]},"socialSharing":{"facebook":{"description":"Jump into CrossFit in [City Name]! Our certified coach led group training and fitness classes feature strength, conditioning, and functional workouts.","title":"CrossFit Gyms in [State] | Functional Fitness Classes & Group Training"}}}},"user":{},"coachCertifications":{}}
In this 2011 editorial, S. Stanley Young and Alan Karr, of Bioinformatics and the National Institute of Statistical Sciences, respectively, present evidence that the results of observational trials (1) are more likely to be false than true.
The validity of observational evidence has been questioned since at least 1988, when L. C. Mayes et al. found that across 56 topic areas, an average of 2.4 studies could be found to support a particular observational claim alongside 2.3 studies that did not (2). In the case of the antidepressant reserpine, for example, three studies indicated the drug increased cancer risk, while eight did not.
More recently, an analysis of 49 claims from highly cited studies found five of six claims drawn from observational studies — 83% — failed to replicate (3). The authors of the analysis gathered 12 randomized controlled trials (RCTs), which collectively tested the claims of 52 observational trials. Zero of the 52 observational claims were supported by the subsequent RCTs, with the RCTs finding significant effects in the opposite direction of the observational evidence (i.e., an RCT indicated harm when the observational study indicated health, or vice versa) in five out of 52 claims. Young and Karr conclude:
There is now enough evidence to say what many have long thought: that any claim coming from an observational study is most likely to be wrong — wrong in the sense that it will not replicate if tested rigorously.
The authors cite W. Edwards Deming, who introduced the concept of quality control, to argue this scale of failure suggests the process by which observational studies are generated is fundamentally flawed. They specifically point to three factors that consistently lead to unreliable observational evidence:
Multiple testing – A single observational study can test many associations simultaneously. For example, as mocked by the xkcd webcomic below, a single study can test for correlations between dozens of dietary and demographic inputs and clinical outcomes. Mere chance indicates some of these correlations will appear statistically significant, even if they lack any biological basis.
Bias – Data is often confounded by unobserved distortions. For example, the HIV drug abacavir was primarily given to patients at high risk of cardiovascular disease. Observational evidence concluded abacavir increased risk of heart disease when, in fact, abacavir was simply being given to patients who were more likely to have a heart attack, with or without the drug (4).
Multiple modeling – As with multiple testing, the regression models used to uncover correlations between variables in observational data can be manipulated until significant associations fall out. Here, the authors note the supposed harms of BPA were derived from an observational study that tested the urine of 1,000 people for 275 chemicals, then compared urine levels of these chemicals to 32 medical outcomes and 10 demographic variables. From this data set, 9 million possible regression models can be constructed, many of which will show significant associations as a result of chance alone (5).
These problems are exacerbated by the widespread reliance on p-values (i.e., statistical significance) to identify meaningful findings. Authors, journal editors, and those who read and interpret science treat statistically significant associations as meaningful and real, even when these significant associations may entirely result from the distortions above.
The authors propose a fix to this process that involves researchers splitting their data prior to analysis, analyzing part using a published statistical methodology, publishing the paper using only this data, and then analyzing the remaining data using the same methodology in an addendum. This method provides a sort of internal replication: If the addendum data informs the same conclusions as the main body of data, these conclusions are more likely to be valid, and if not, the authors’ analytical methods can be questioned.
The takeaway: This paper highlights the extreme degree to which observational evidence is unreliable. Based on the data these authors present, observational evidence is no more likely to be right than wrong, and in fact, it may be more likely to be wrong than right. In other words, any health claim based on observational evidence is only valuable according to the extent to which it informs subsequent controlled trials to test its validity.
That is, trials in which the authors observe whether particular inputs (i.e., patterns in diet, behavior, demography, etc.) correlate with increased or decreased risk of disease, without attempting to affect those inputs.
Deming, Data and Observational Studies