Please join us in-person or online for a Queensland Branch Meeting. The seminar will start at 5:00 pm. Details for the seminar are provided below.
TITLE: Evidence of statistical "hacking" in clinical prediction models
SPEAKER: Prof Adrian Barnett, Queensland University of Technology
TIME: 5:00 pm - 6:30 pm (AEST), Wednesday 2nd August 2023
VENUE: 50-N202 - Hawken Engineering Building, Learning Theatre, UQ St Lucia Campus or Online (Zoom link will be sent with registration)
Please note that the seminar will be recorded and might be put on YouTube or similar platform.
ABSTRACT:
Clinical prediction models are widely used in health and medical research. The area under the receiving operating curve (AUC) is a frequently used estimate to describe the utility of a clinical prediction model. The AUC is often interpreted relative to thresholds, with “good” or “excellent” models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in “hacking”, where researchers are spurred on to re-analyze their data until they achieve a “good” result.
We extracted AUC values from PubMed abstracts to look for evidence of hacking. The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds. The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.
*This is joint work with Nicole White (QUT), Rex Parsons (QUT), and Gary Collins (Oxford)
SPEAKER'S BIO:
Prof Adrian Barnett
Adrian Barnett is a statistician who has over 28 years of applied experience, specialising in medical statistics, health services research and research funding. Adrian is a founder member of AusHSI, where he has evaluated large and complex changes to the health system. He is a fellow of the Academy of Social Science in Australia, a member of the NHMRC Research Committee, and member of Science and Technology Australia’s policy committee.
Adrian’s current focus is on improving the quality of health and medical research, and to improve how researchers apply and describe their statistical analyses. He is passionate about the application of statistics to health and the use of data analysis to develop cost-effective interventions in our health system. Adrian has worked with clinicians, researchers and policy makers in almost all areas of the health system.
@StatSocAus
