The pollsters did not perform well in the lead up to the 2019 May Federal election. All predicted a win by the Labour party (ALP) with margins from 2% to 4% of the two party preferred vote. In the event, the Liberal National Party (LNP) coalition won by 3.0%. Election and political polling are consequential and influential and play an important role in the democratic process. In recent times, at least one Prime Minister lost his job as a result of poor polling. Furthermore, policy decisions are influenced by opinion polls.

Both the Statistical Society and the Australian Market and Social Research Organisations (AMSRO) proposed reviews of the performance to determine why all the published polls incorrectly called the outcome at 2019’s Federal election and how methods can be improved in the future.

It was agreed that AMSRO would take the main responsibility for the review but that the Statistical Society would be closely involved.

Modalities

A panel was established under the leadership of Darren Pennay. Darren was the Founder and past CEO of the Social Research Centre. the other panel members were a mixture of political scientists, statisticians and polling experts. Several were Statistical Society members. They were assisted by an Advisory Board. John Henstridge was the Statistical Society nominee for the Board. The Chair was Dennis Trewin who is also a Statistical Society member.

AMSRO invited pollsters, media organisations and others who commission election and political polling to contribute to the inquiry. While the Inquiry Panel was grateful for the cooperation it received from the pollsters, the lack of access to data sets (which have been routinely provided to similar reviews overseas) and to detailed descriptions of the survey methods and statistical techniques used affected their ability to conclusively identify all the specific factors that contributed to the relative inaccuracy of the 2019 polls.

The Inquiry sought cooperation from the four pollsters responsible for these polls: Essential Research, Ipsos, Roy Morgan Research and YouGov (responsible for the YouGov Galaxy and Newspoll polls). Roy Morgan Research refused outright to co-operate.

Despite the limited information, a quite substantial report was prepared. It was released on 11 November and it and further information on the AMSRO Polling Inquiry can be found at: https://www.amsro.com.au/amsro-polling-inquiry/.

Findings

The report said that the polls erred in their estimate of the vote in a manner that was statistically significant and erred in the same direction and at a similar level. It also found that the source of errors lie in the polls themselves and was not a result of a last-minute shift in preferences among voters.

The first preference votes were either underestimated (LNP) or overestimated (ALP) because of inadequately adjusted, unrepresentative samples. The report said that it was very likely the polls were skewed towards over-representing more politically engaged and better educated voters and this bias was not corrected and as a result, the polls over-represented Labor voters.

The Inquiry Panel could not rule out the possibility that the uncommon convergence of the polls in 2019 was due to ‘herding’ although it may be due, in part, to their methodologies leading to similar weaknesses in the representativeness of their samples.  The absence of access to the data or the detailed methods means that this this remains a conjecture.

The Inquiry Panel also found that the reporting of the polls failed to meet the basic disclosure guidelines for editors and journalists set out by the Australian Press Council.

Recommendations

The Inquiry report has made 10 key recommendations aimed at improving polling methods and the disclosure thereof.  Recommendations include (1) calling for an enforceable Code of Conduct for Election Polling developed in consultation with experts including statistical experts; (2) improvements and greater transparency in polling methods, (3) better measures of uncertainty and (4) better education of the media and public on polling.

We believe there is an important role for statisticians and members of the SSA in particular in improving the quality of polls, both in the design of the surveys and in generating a productive debate about their performance. 

Postscript - 2020 US election polling

The accuracy of the US election polls has come under criticism.  Most of the criticism came early when it appeared that Biden might lose the election. As the count progressed with the inclusion of mail-in votes the difference between the poll predictions and the actual vote narrowed. In the end, how did the polls perform?

The following analysis is based on data from Real Clear Politics, one of the main poll aggregators in the US.

Our [Real Clear Politics] conclusions are:

• The performance of the polls was not great but better than generally perceived.
• At the national level, the poll average was a 7.2% margin to Biden compared with an expected final outcome of about 4.5%. At least 4 of the polls predicted the result within the margin of error.
• There was only one State where there was a polling miss – FLORIDA – where the average of the polls suggested a small win to Biden rather than a comfortable win to Trump.
• The average of the polls was very good in many of the Swing States and predicted the correct outcome in PENNSLYVANIA, MICHIGAN, MINNESOTA, NORTH CAROLINA, ARIZONA, and NEVADA, mostly quite close to the actual margin.  We are treating GEORGIA as an effective dead heat, but the average of the polls suggested a small win to Trump.
• The average of the polls was outside the margin of error in several of the rust belt States – OHIO, WISCONSIN, IOWA.

Like Australia and in the 2016 US elections, we expect the main reason for any lack of performance will be unrepresentative samples but that is conjecture at this stage. Polling is more difficult in the US because they have to predict election turnout which may vary between the candidates.

Dennis Trewin and John Henstridge