Using leave-one-out cross-validation in a multilevel regression and poststratification workflow: A cautionary tale 

Abstract:

In recent decades, multilevel regression and poststratification (MRP) has surged in popularity for population inference. However, the validity of the estimates can depend on details of the model, and there is currently little research on validation. We explore how two approximate calculations of leave-one-out cross-validation (LOO) — the Pareto smoothed importance sampling (PSIS-LOO) and a survey-weighted alternative (WTD-PSIS-LOO) can be used to compare Bayesian models for MRP. Using two simulation designs, we examine how accurately these two criteria recover the correct ordering of model goodness at predicting population and small area level estimands. Our findings suggest that while not terrible, PSIS-LOO-based ranking techniques may not be suitable to evaluate MRP as a method. The results show that in practice, PSIS-LOO-based model validation tools need to be used with caution and might not convey the full story when validating MRP as a method. All results are obtained using Stan in R and the brms package. This is joint work with Dr. Lauren Kennedy, Assoc. Prof. Qixuan Chen and Prof. Andrew Gelman.

Bio:

Dr. Fui Swen Kuh (Swen) was recently a Research Fellow at the University of Adelaide and currently a teaching associate at Monash University. She graduated with an Honours degree in Statistics from the University of Auckland and then obtained her PhD from the Australian National University in 2022. She is especially interested in various social science applications with Bayesian methods and inference.

Her previous projects include developing a novel Bayesian methodology to produce countries’ socio-economic health index using World Bank and United Nations data and involvement in New Zealand’s longitudinal census data in comparing sibling mortality risks.