Michael Crowe wrote:

Nothing like p-values to get an argument discussion going! Here are some thoughts from Megan Higgs.

Megan Higgs' thoughts are very much my thoughts. Note especially her comment that "Scientific integrity is far larger and more important than anything statistical methods can insert into the process. It doesn’t help to simply state the potential positives and ignore the negatives."

The editorial, and the accompanying statement, strike me as an exercise in avoiding "any ruffling of feathers".  P-values are particularly dangerous when they are used uncritically in a court of law.  A classic (and horrible) case is that of the unfortunate Sally Clark, accused of murdering her two children.  See Misuse of statistics in the courtroom. The probability that two children in the same family would die of natural causes ought to have been (and was not) set against the probability (even if one could only at best make a guess) that a  woman who otherwise appeared to be well-meaning and caring would commit such acts.  Worse, the probability that that two children in the same family would die of natural causes was, because the two deaths were assumed independent, grossly underestimated by the leading witness for the prosecution.  This is a striking example of the "prosecutors fallacy".  Some years ago, I reviewed a book on probability, written by a computer scientist, where all probabilities were assumed independent.  Maybe we have not moved as far as we imagine from the days when "witchcraft" was a crime that played out in the courts.

If p-values are to be used, it is important to understand what they do, and do not mean.  The "under the NULL, occurs with probability α" is correct only for the class of p-values for which p<=α, i.e., it applies when s decision has been made, in advance, to use α as a threshold.  The expected value of such p-values, under the NULL, is α/2.  Interpreting p=0.05 as occuring with a probability of 0.05 makes as much sense as interpreting it as having a probability of 0.95 (the probability with which one will get a greater value).  It may reasonably be seen a lying in the middle of a range that runs from 0.10 to 0, and that, under the NULL, occurs with probability 0.1. 

If one wants to attach a probability interpretation to an individual p-value, a way to set an upper limit on what this means for a prior 1:1 odds ratio is to look at the maximum likelihood ratio under the alternative.  For 6 or more degrees of freedom p = 0.05 translates to a ratio that is less than 5.0, while it is less than 4.5 for 10 or more degrees of freedom, for comparisons based on a t-statistic.  The R package tTOlr that I have placed on CRAN has two vignettes that explore such comparisons in more detail.  The BayesFactor package provides a ready means to translate p-values into what, under the "uninformative" prior used, is the factor by which the prior odds ratio should be multiplied, with ~2.5 as the upper bound for p-0.05.

The American Statistical Association (ASA) set up a task force on statistical significance and reproducibility in late 2019. Its two-page statement has now been released in the Annals of Applied Statistics here. The road to release seems to have been a bumpy one. Unlike the earlier ASA Statement on P-values, this one is not an official release of the ASA.

The Statement contains some conclusions that are quite supportive of P-values and significance testing, such as: “P-values are valid statistical measures that provide convenient conventions for communicating uncertainty inherent in quantitative results. Indeed, P-values and significance tests are … important tools that have advanced science through their proper application.” It also has a section titled: “Thresholds are helpful when actions are required”. It also notes that there are many other valid ways to describe uncertainty, including various Bayesian tools.

One of the messages I take away is that there are many valid ways to do statistical analysis, and the choice is often not a matter of right and wrong, but rather preference.