How Could a Group of Psychologists Be so Wrong?

I’ll carry a single tome in our tour: Morrison and Henkel’s Reference Morrison and Henkel1970 classic, The Significance Test Controversy. Some abuses of the proper interpretation of significance tests were deemed so surprising even back then that researchers in psychology conducted studies to try to understand how this could be. Notably, Rosenthal and Gaito (Reference Rosenthal and Gaito1963) discovered that statistical significance at a given level was often fallaciously taken as evidence of a greater discrepancy from the null hypothesis the larger the sample size n. In fact, it is indicative of less of a discrepancy from the null than if it resulted from a smaller sample size.

(Our convention is for “discrepancy” to refer to the parametric, not the observed, difference. Their use of “deviation” from the null alludes to our “discrepancy.”)

As statistician John Pratt notes, “the more powerful the test, the more a just significant result favors the null hypothesis” (Reference Pratt1961, p. 166). Yet we still often hear: “The thesis implicit in the [N-P] approach, [is] that a hypothesis may be rejected with increasing confidence or reasonableness as the power of the test increases” (Howson and Urbach Reference Howson and Urbach1993, p. 209). In fact, the thesis implicit in the N-P approach, as Bakan remarks, is the opposite! The fallacy is akin to making mountains out of molehills according to severity (Section 3.2):

Consider an analogy with two fire alarms: The first goes off with a sensor liable to pick up on burnt toast; the second is so insensitive it doesn’t kick in until your house is fully ablaze. You’re in another state, but you get a signal when the alarm goes off. Which fire alarm indicates the greater extent of fire? Answer: the second, less sensitive one. When the sample size increases it alters what counts as a single sample. It is like increasing the sensitivity of your fire alarm. It is true that a large enough sample size triggers the alarm with an observed mean that is quite “close” to the null hypothesis. But, if the test rings the alarm (i.e., rejects H₀) even for tiny discrepancies from the null value, then the alarm is poor grounds for inferring larger discrepancies. Now this is an analogy, you may poke holes in it. For instance, a test must have a large enough sample to satisfy model assumptions. True, but our interpretive question can’t even get started without taking the P-values as legitimate and not spurious.

Exhibit (v): Responding to a Familiar Chestnut

I follow the treatment in Elliott Sober (Reference Sober2008, p. 56), who is echoing Howson and Urbach (Reference Howson and Urbach1993, pp. 208–9), who are echoing Lindley (Reference Lindley1957). The only difference is that I will allude to a variant on test T+: H₀: μ = 0 vs. H₁: μ > 0 with σ = 1. “Another odd property of significance tests,” says Sober, “concerns the way in which they are sensitive to sample size.” Suppose you are applying test T+ with null H₀: μ = 0. If your sample size is n = 25, and you choose α = 0.025, you will reject H₀ whenever $\overline{x}\ge 0.4$. If you examine n = 100, and choose the same value for α, you will reject H₀ whenever $\overline{x}\ge 0.2$. And if you examine n = 400, again with α = 0.025, you will reject H₀ whenever $\overline{x}\ge 0.1$. “As sample size increases” the sample mean $\overline{x}$ must be closer and closer to 0 for you not to reject H₀. “This may not seem strange until you add the following detail. Suppose the alternative to H₀ is the hypothesis” H₁: μ = 0.24. “The Law of Likelihood now entails that observing” $\overline{x}<0.12$ favors H₀ over H₁, so in particular $\overline{x}=0.1$ favors H₀ over H₁ (Section 1.4).

Your reply: Hold it right at “add the following detail.” You’re observing that the significance test disagrees with a Law of Likelihood appraisal to a point vs. point test: H₀: µ = 0 vs. H₁: µ = 0.24. We require the null and alternative hypotheses to exhaust the space of parameters, and these don’t. Nor are our inferences to points, but rather to inequalities about discrepancies. That said, we’re prepared to consider your example, and make short work of it. We’re testing H₀: μ ≤ 0 vs. H₁: μ > 0 (one could equally view it as testing H₀: µ = 0 vs. H₁: µ > 0). The outcome $\overline{x}=0.1$, while indicating some positive discrepancy from 0, offers bad evidence and an insevere test for inferring μ as great as 0.24. Since $\overline{x}=0.1$ rejects H₀, we say the result accords with H₁. The severity associated with inference µ ≥ 0.24 asks: what’s the probability of observing $\overline{X}\le 0.1$ – i.e., a result more discordant with H₁ assuming µ = 0.24.

SEV(µ ≥ 0.24) with $\overline{x}=0.1$ and n = 400 is computed as $\mathrm{Pr}(\overline{X}$ < 0.1; µ = 0.24). Standardizing $\overline{X}$ yields Z = √400 (0.1 − 0.24)/1 = 20(−0.14) = −2.8. So SEV(µ ≥ 0.24) = 0.003! Were µ as large as 0.24, we’d have observed a larger observed mean than we did with 0.997 probability! It’s terrible evidence for H₁: µ = 0.24.

This is redolent of the Binomial example in discussing Royall (Section 1.4). To underscore the difference between the Likelihoodist’s comparative appraisal and the significance tester, you might go further. Consider an alternative that the Likelihoodist takes as favored over H₀: µ = 0 with $\overline{x}=0.1$, namely, the maximum likely alternative H₁: µ = 0.1. This is one of our key benchmarks for a discrepancy that’s poorly indicated. To the Likelihoodist, inferring that H₁: µ = 0.1 “is favored” over H₀: µ = 0 makes sense, whereas to infer a discrepancy of 0.1 from H₀ is highly unwarranted for a significance tester.Footnote ¹ Our aims are very different.

We can grant this: starting with any value for $\overline{x}$, however close to 0, there’s an n such that $\overline{x}$ is statistically significantly greater than 0 at a chosen level. If one understands the test’s intended task, this is precisely what is wanted. How large would n need to be so that 0.02 is statistically significant at the 0.025 level (still retaining σ = l)?

Statistics won’t tell you what magnitudes are of relevance to you. No matter, we can critique results and purported inferences.

Exhibit (vi): Reforming the Reformers on Confidence Intervals.

You will be right to wonder why some of the same tribes who raise a ruckus over P-values – to the extent, in some cases, of calling for a “test ban” – are cheerleading for confidence intervals (CIs), given there is a clear duality between the two (Section 3.7). What they’re really objecting to is a dichotomous use of significance tests where the report is “significant” or not at a predesignated significance level. I completely agree with this objection, and reject the dichotomous use of tests (which isn’t to say there are no contexts where an “up/down” indication is apt). We should reject Unitarianism where a single method with a single interpretation must be chosen. Ironically, some of the most outspoken CI leaders use them in the dichotomous fashion (rightly) deplored when it comes to testing.

Geoffrey Cumming, an acknowledged tribal leader on CIs, tells us that “One-sided CIs are analogous to one-tailed tests but, as usual, the estimation approach is better” (Reference Cumming2012, p. 109). Well, it might be better, but like hypothesis testing, it calls for supplements and reinterpretations as begun in Section 3.7.

Our one-sided test T+ (H₀: µ ≤ 0 vs. H₁: µ > 0, and σ = 1) at α = 0.025 has as its dual the one-sided (lower) 97.5% general confidence interval: $\mu >\overline{X}-2(1/\surd n)$ – rounding to 2 from 1.96. So you won’t have to flip back pages, here’s a quick review of the notation we developed to avoid the common slipperiness with confidence intervals. We abbreviate the generic lower limit of a (1 − α) confidence interval as ${\widehat{\mu }}_{1-\alpha }(\overline{X})$ and the particular limit as ${\widehat{\mu }}_{1-\alpha }\left(\overline{x}\right)$. The general estimating procedure is: Infer $\mu >{\widehat{\mu }}_{1-\alpha }(\overline{X})$. The particular estimate is $\mu >{\widehat{\mu }}_{1-\alpha }\left(\overline{x}\right)$. Letting α = 0.025 we have: $\mu >\overline{x}-2(1/\surd n)$. With α = 0.05, we have $\mu >\overline{x}-1.65(1/\surd n)$.

Cumming’s interpretation of CIs and confidence levels points to their performance-oriented construal: “In the long run 95% of one-sided CIs will include the population mean … We can say we’re 95% confident our one-sided interval includes the true value … meaning that for 5% of replications the [lower limit] will exceed the true value” (Cumming Reference Cumming2012, p. 112). What does it mean to be 95% confident in the particular interval estimate for Cumming? “It means that the values in the interval are plausible as true values for μ, and that values outside the interval are relatively implausible – though not impossible” (ibid., p. 79). The performance properties of the method rub off in a plausibility assessment of some sort.

The test that’s dual to the CI would “accept” those parameter values within the corresponding interval, and reject those outside, all at a single predesignated confidence level 1 − α. Our main objection to this is it gives the misleading idea that there’s evidence for each value in the interval, whereas, in fact, the interval simply consists of values that aren’t rejectable, were one testing at the α level. Not being a rejectable value isn’t the same as having evidence for that value. Some values are close to being rejectable, and we should convey this. Standard CIs do not.

To focus on how CIs deal with distinguishing sample sizes, consider again the three instances of test T+ with (i) n = 25, (ii) n = 100, and (iii) n = 400. Imagine the observed mean $\overline{x}$ from each test just hits the significance level 0.025. That is, (i) $\overline{x}=0.4$, (ii) $\overline{x}=0.2$, and (iii) $\overline{x}=0.1$. Form 0.975 confidence interval estimates for each:

1. (i) for n = 25, the inferred estimate is $\mu >\widehat{\mu }{}_{0.975}$, that is, $\mu >\overline{x}-2(1/5)$;

2. (ii) for n = 100, the inferred estimate is $\mu >\widehat{\mu }{}_{0.975}$, that is, $\mu >\overline{x}-2(1/10)$;

3. (iii) for n = 400, the inferred estimate is $\mu >\widehat{\mu }{}_{0.975}$, that is, $\mu >\overline{x}-2(1/20)$.

Substituting $\overline{x}$ in all cases, we get the same one-sided confidence interval:

Cumming writes them as [0, infinity). How are the CIs distinguishing them?

They are not. The construal is dichotomous: in or out, plausible or not. Would we really want to say “the values in the interval are plausible as true values for µ”? Clearly not, since that includes values to infinity. I don’t want to step too hard on the CI champion’s toes, since CIs are in the frequentist, error statistical tribe. Yet, to avoid fallacies, this standard use of CIs won’t suffice. Severity directs you to avoid taking your result as indicating a discrepancy beyond what’s warranted. For an example, we can show the same inference is poorly indicated with n = 400, while fairly well indicated when n = 100. For a poorly indicated claim, take our benchmark for severity of 0.5; for fairly well, 0.84:

$\text{For}n=400,{\overline{x}}_{0.025}=0.1,\text{so}\mu >0.1\text{is}\text{poorly indicated};$

$\text{For}n=100,{\overline{x}}_{0.025}=0.2,\text{and}\mu >0.1\text{is fairly well indicated}.$

The reasoning based on severity is counterfactual: were µ less than or equal to 0.1, it is fairly probable, 0.84, that a smaller $\overline{X}$ would have occurred. This is not part of the standard CI account, but enables the distinction we want. Another move would be for a CI advocate to require we always compute a two-sided interval. The upper 0.975 bound would reflect the greater sensitivity with increasing sample sizes:

But we cannot just deny one-sided tests, nor does Cumming. In fact, he encourages their use: “it’s unfortunate they are usually ignored” (Reference Cumming2012, p. 113). (He also says he is happy for people to decide afterwards whether to report it as a one- or two-sided interval (ibid., p. 112), only doubling α, which I do not mind.) Still needed is a justification for bringing in the upper limit when applying a one-sided estimator, and severity supplies it. You should always be interested in at least two benchmarks: discrepancies well warranted and those terribly warranted. In test T+, our handy benchmark for the terrible is to set the lower limit to $\overline{x}$. The severity for $(\mu >\overline{x})\text{is}0.5$. Two side notes:

First I grant it would be wrong to charge Cumming with treating all parameter values within the confidence interval on par, because he does suggest distinguishing them by their likelihoods (by how probable each renders the outcome). Take just the single 0.975 lower CI bound with n = 100 and $\overline{x}=0.2$. A µ value closer to the observed 0.2 has higher likelihood (in the technical sense) than ones close to the 0.975 lower limit 0. For example, µ = 0.15 is more likely than µ = 0.05. However, this moves away from CI reasoning (toward likelihood comparisons). The claim µ > 0.05 has a higher confidence level (0.93) than does µ > 0.15 (0.7)Footnote ³ even though the point hypothesis µ = 0.05 is less likely than µ = 0.15 (the latter is closer to $\overline{\text{x}}=0.2$ than is the former). Each point in the lower CI corresponds to a different lower bound, each associated with a different confidence level, and corresponding severity assessment. That’s how to distinguish them.

Second there’s an equivocation, or at least a potential equivocation, in Cumming’s assertion “that for [2.5%] of replications the [lower limit] will exceed the true value” (Cumming Reference Cumming2012, p. 112 replacing 5% with 2.5%). This is not a true claim if “lower limit” is replaced by a particular lower limit: $\widehat{\mu }{}_{0.025}(\overline{x})$, it holds only for the generic lower limit $\widehat{\mu }{}_{0.025}(\overline{X})$. That is, we can’t say µ exceeds zero 2.5% of the time, which would be to assign a probability of 0.975 to µ > 0. Yet this misinterpretation of CIs is legion, as we’ll see in a historical battle about fiducial intervals (Section 5.8).

Getting Beyond “I’m Rubber and You’re Glue”.

The danger in critiquing statistical method X from the standpoint of the goals and measures of a distinct school Y, is that of falling into begging the question. If the P-value is exaggerating evidence against a null, meaning it seems too small from the perspective of school Y, then Y’s numbers are too big, or just irrelevant, from the perspective of school X. Whatever you say about me bounces off and sticks to you. This is a genuine worry, but it’s not fatal. The goal of this journey is to identify minimal theses about “bad evidence, no test (BENT)” that enable some degree of scrutiny of any statistical inference account – at least on the meta-level. Why assume all schools of statistical inference embrace the minimum severity principle? I don’t, and they don’t. But by identifying when methods violate severity, we can pull back the veil on at least one source of disagreement behind the battles.

Thus, in tackling this latest canard, let’s resist depicting the critics as committing a gross blunder of confusing a P-value with a posterior probability in a null. We resist, as well, merely denying we care about their measure of support. I say we should look at exactly what the critics are on about. When we do, we will have gleaned some short-cuts for grasping a plethora of critical debates. We may even wind up with new respect for what a P-value, the least popular girl in the class, really does.

To visit the core arguments, we travel to 1987 to papers by J. Berger and Sellke, and Casella and R. Berger. These, in turn, are based on a handful of older ones (Cox Reference Cox1977, E, L, & S Reference Edwards, Lindman and Savage1963, Pratt Reference Pratt1965), and current discussions invariably revert back to them. Our struggles through quicksand of Excursion 3, Tour II, are about to pay large dividends.

J. Berger and Sellke, and Casella and R. Berger.

Berger and Sellke (1987a) make out the conflict between P-values and Bayesian posteriors by considering the two-sided test of the Normal mean, H₀: μ = 0 vs. H₁: μ ≠ 0. “Suppose that X = (X₁, …, X_n), where the X_i are IID N(μ, σ²), σ² known” (p. 112). Then the test statistic $\text{d}(\mathit{X})=\surd n\left|\overline{X}-{\mu }_0\right|/\sigma$, and the P-value will be twice the P-value of the corresponding one-sided test.

Starting with a lump of prior, generally 0.5, on the point hypothesis H₀, they find the posterior probability in H₀ is larger than the P-value for a variety of different priors on the alternative. However, the result depends entirely on how the remaining 0.5 is allocated or smeared over the alternative (a move dubbed spike and smear). Using what they call a Jeffreys-type prior, the 0.5 is spread out over the alternative parameter values as if the parameter is itself distributed N(µ₀, σ). Now Harold Jeffreys recommends the lump prior only to capture cases where a special value of a parameter is deemed plausible, for instance, the GTR deflection effect λ = 1.75″, after about 1960. The rationale is to avoid a 0 prior on H₀ and enable it to receive a reasonable posterior probability .

By subtitling their paper “The irreconcilability of P-values and evidence,” Berger and Sellke imply that if P-values disagree with posterior assessments, they can’t be measures of evidence at all. Casella and R. Berger (1987) retort that “reconciling” is at hand, if you move away from the lump prior. So let’s see how this unfolds. I assume throughout, as do the critics, that the P-values are “audited,” so that neither selection effects nor violated model assumptions are in question at this stage. I see no other way to engage their arguments.

Table 4.1 gives the values of Pr(H₀|x). We see that we would declare no evidence against the null, and even evidence for it (to the degree indicated by the posterior) whenever d(x) fails to reach a 2.5 or 3 standard error difference. With n = 50, “one can classically ‘reject H₀ at significance level p = 0.05,’ although Pr(H₀|x) = 0.52 (which would actually indicate that the evidence favors H₀)” (J. Berger and Sellke Reference Berger and Sellke1987, p. 113).

If n = 1000, a result statistically significant at the 0.05 level results in the posterior probability to μ = 0 going up from 0.5 (the lump prior) to 0.82! From their Bayesian perspective, this suggests P-values are exaggerating evidence against H₀. Error statistical testers, on the other hand, balk at the fact that using the recommended priors allows statistically significant results to be interpreted as no evidence against H₀ – or even evidence for it. They point out that 0 is excluded from the two-sided confidence interval at level 0.95. Although a posterior probability doesn’t have an error probability attached, a tester can evaluate the error probability credentials of these inferences. Here we’d be concerned with a Type II error: failing to find evidence against the null, and providing a fairly high posterior for it, when it’s false (Souvenir I).

Let’s use a less extreme example where we have some numbers handy: our water-plant accident. We had σ = 10, n = 100 leading to the nice (σ/√n) value of 1. Here it would be two-sided, to match their example: H₀: μ = 150 vs. H₁: μ ≠ 150. Look at the second entry of the 100 column, the posterior when z_α = 1.96. With the Jeffreys prior, perhaps championed by the water coolant company, J. Berger and Sellke assign a posterior of 0.6 to H₀: μ = 150 degrees when a mean temperature of 152 (151.96) degrees is observed – reporting decent evidence the cooling mechanism is working just fine. How often would this occur even if the actual underlying mean temperature is, say, 151 degrees? With a two-sided test, cutting off 2 standard errors on either side, we’d reject whenever either $\overline{X}\ge 152$ or $\overline{X}\le 148$. The probability of the second is negligible under µ = 151, so the probability we want is $\mathrm{Pr}(\overline{X}<152;\mu =151)=0.84(Z=(152-151)=1).$ The probability of declaring evidence for 150 degrees (with posterior of 0.6 to H₀) even if the true increase is actually 151 degrees is around 0.84; 84% of the time they erroneously fail to ring the alarm, and would boost their probability of μ = 150 from 0.5 to 0.6. Thus, from our minimal severity principle, the statistically significant result can’t even be taken as evidence for compliance with 151 degrees, let alone as evidence for the null of 150 (Table 3.1).

Is this a problem for them? It depends what you think of that prior. The N-P test, of course, does not use a prior, although, as noted earlier, one needn’t rule out a frequentist prior on mean water temperature after an accident (Section 3.2). For now our goal is making out the criticism.

A Dialogue at the Water Plant Accident

This quote from J. Berger and Sellke is peculiar. They go on to add: “We emphasize this obvious point because some react to the Bayesian–classical conflict by attempting to argue that [prior] π₀ should be made small in the Bayesian analysis so as to force agreement” (ibid.). We should not force agreement. But it’s scarcely an obvious justification for a lump of prior on the null H₀ – one which results in a low capability to detect discrepancies – that it ensures, if they do reject H₀, there will be a meaningful drop in its probability. Let’s listen to the pushback from Casella and R. Berger (Reference Casella and Berger1987a), the Berger being Roger now (I use initials to distinguish them).

The Cult of the Holy Spike and Slab.

Casella and R. Berger (Reference Casella and Berger1987a) charge that the problem is not P-values but the high prior, and that “concentrating mass on the point null hypothesis is biasing the prior in favor of H₀ as much as possible” (p. 111) whether in one- or two-sided tests. According to them:

Most of the time “there is a direction of interest in many experiments, and saddling an experimenter with a two-sided test would not be appropriate” (ibid.). The “cult of the holy spike” is an expression I owe to Sander Greenland (personal communication).

By contrast, we can reconcile P-values and posteriors in one-sided tests if we use more diffuse priors. (e.g., Cox and Hinkley Reference Cox and Hinkley1974, Jeffreys Reference Jeffreys1939/1961, Pratt Reference Pratt1965). In fact, Casella and Berger show that for sensible priors in that case, the P-value is at least as big as the minimum value of the posterior probability on the null, again contradicting claims that P-values exaggerate the evidence.Footnote ⁴

J. Berger and Sellke (Reference Berger and Sellke1987) adhere to the spikey priors, but following E, L, & S (Reference Edwards, Lindman and Savage1963), they’re keen to show that P-values exaggerate evidence even in cases less extreme than the Jeffreys posteriors in Table 4.1. Consider the likelihood ratio of the null hypothesis over the hypothesis most generous to the alternative, they say. This is the point alternative with maximum likelihood, H_max – arrived at by setting $\mu =\overline{x}$. Through their tunnel, it’s disturbing that even using this likelihood ratio, the posterior for H₀ is still larger than 0.05 – when they give a 0.5 spike to both H₀ and H_max. Some recent authors see this as the key to explain today’s lack of replication of significant results. Through the testing tunnel, things look different (Section 4.5).

Why Blame Us Because You Can’t Agree on Your Posterior?

Stephen Senn argues that the reason for the wide range of variation of the posterior is the fact that it depends radically on the choice of alternative to the null and its prior.Footnote ⁵ According to Senn, “… the reason that Bayesians can regard P-values as overstating the evidence against the null is simply a reflection of the fact that Bayesians can disagree sharply with each other” (Senn Reference Senn2002, p. 2442). Senn illustrates how “two Bayesians having the same prior probability that a hypothesis is true and having seen the same data can come to radically different conclusions because they differ regarding the alternative hypothesis” (Senn Reference Senn2001b, p. 195). One of them views the problem as a one-sided test and gets a posterior on the null that matches the P-value; a second chooses a Jeffreys-type prior in a two-sided test, and winds up with a posterior to the null of 1 − p!

Here’s a recap of Senn’s example (ibid., p. 200): Two scientists A and B are testing a new drug to establish its treatment effect, δ, where positive values of δ are good. Scientist A has a vague prior whereas B, while sharing the same distribution about the probability of positive values of δ, is less pessimistic than A regarding the effect of the drug. If it’s not useful, B believes it will have no effect. They “share the same belief that the drug has a positive effect. Given that it has a positive effect, they share the same belief regarding its effect. … They differ only in belief as to how harmful it might be.” A clinical trial yields a difference of 1.65 standard units, a one-sided P-value of 0.05. The result is that A gives 1/20 posterior probability to H₀: the drug does not have a positive effect, while B gives a probability of 19/20 to H₀. B is using the two-sided test with a lump of prior on the null (H₀: μ = 0 vs. H₁: μ ≠ 0), while A is using a one-sided test T+ (H₀: μ ≤ 0 vs. H₁: μ > 0). The contrast, Senn observes, is that of Cox’s distinction between “precise and dividing hypothesis” (Section 3.3). “[F]rom a common belief in the drug’s efficacy they have moved in opposite directions” (ibid., pp. 200–201). Senn riffs on Jeffreys’ well-known joke that we heard in Section 3.4:

In other words, if Bayesians disagree with each other even when they’re measuring the same thing – posterior probabilities – why be surprised that disagreement is found between posteriors and P-values? The most common argument behind the “P-values exaggerate evidence” appears not to hold water. Yet it won’t be zapped quite so easily, and will reappear in different forms.

Exhibit (vii): Contrasting Bayes Factors and Jeffreys–Lindley Paradox.

We’ve uncovered some interesting bones in our dig. Some lead to seductive arguments purporting to absolve the latitude in assigning priors in Bayesian tests. Take Wagenmakers and Grünwald (Reference Wagenmakers and Grünwald2006, p. 642): “Bayesian hypothesis tests are often criticized because of their dependence on prior distributions … [yet] no matter what prior is used, the Bayesian test provides substantially less evidence against H₀ than” P-values, in the examples we’ve considered. Be careful in translating this. We’ve seen that what counts as “less” evidence runs from seriously underestimating to overestimating the discrepancy we are entitled to infer with severity. Begin with three types of priors appealed to in some prominent criticisms revolving around the Fisher – Jeffreys disagreement.

1. 1. Jeffreys-type prior with the “spike and slab” in a two-sided test. Here, with large enough n, a statistically significant result becomes evidence for the null; the posterior to H₀ exceeds the lump prior.

2. 2. Likelihood ratio most generous to the alternative. Here, there’s a spike to a point null, H₀: θ = θ₀ to be compared to the point alternative that’s maximally likely θ_max. Often, both H₀ and H_max are given 0.5 priors.

3. 3. Matching. Instead of a spike prior on the null, it uses a smooth diffuse prior, as in the “dividing” case. Here, the P-value “is an approximation to the posterior probability that θ < 0” (Pratt Reference Pratt1965, p. 182).

In sync with our attention to high-energy particle physics (HEP) in Section 3.6, consider an example that Aris Spanos (Reference Spanos2013b) explores in relation to the Jeffreys–Lindley paradox. The example is briefly noted in Stone (Reference Stone1997).

A large number (n = 527,135) of independent collisions that can be of either type A or type B are used to test if the proportion of type A collisions is exactly 0.2, as opposed to any other value. It’s modeled as n Bernoulli trials testing H₀: θ = 0.2 vs. H₁: θ ≠ 0.2. The observed proportion of type A collisions is scarcely greater than the point null of 0.2:

$\overline{x}=k/n=\mathrm{0.20165233,}\text{where}n=527,135,k=106,298.$

The significance level against H₀ is small (so there’s evidence against H₀)

The test statistic $\text{d}(\mathit{X})=[\surd n(\overline{X}-0.2)/\sigma ]=3$, $\sigma =\sqrt{\left[\theta (1-\theta )\right]}$, which under the null is $\sqrt{\left[0.2\left(0.8\right)\right]}=0.4$. The significance level associated with d(x₀) in this two-sided test is

Pr(|d(X)| > |d(x₀)|;H₀) = 0.0027.

So the result $\overline{x}$ is highly significant, even though it’s scarcely different from the point null.

The Bayes factor in favor of H₀ is high

H₀ is given the spiked prior of 0.5, and the remaining 0.5 is spread equally among the values in H₁. I follow Spanos’ computations:Footnote ⁶

$\mathrm{Pr}\left(k|H_0\right)=\left(\begin{array}{c}n\\ k\end{array}\right){0.2}^k{\left(0.8\right)}^{n-k},$

$\mathrm{Pr}\left(k|H_1\right)={\displaystyle \underset{1}{\overset{0}{\int }}}\left(\begin{array}{c}n\\ k\end{array}\right){\theta }^k{\left(1-\theta \right)}^{n-k}\text{d}\theta =1/\left(n+1\right),$

where n = 527,135 and k = 106,298.

$\text{The Bayes factor}{B}_{01}=\mathrm{Pr}\left(k|H_0\right)/\mathrm{Pr}\left(k|H_1\right)=0.000015394/0.000001897=8.115.$

While the likelihood of H₀ in the numerator is tiny, the likelihood of H₁ is even tinier. Since B₀₁ in favor of H₀ is 8, which is greater than 1, the posterior for H₀ goes up, even though the outcome is statistically significantly greater than the null.

There’s no surprise once you consider the Bayesian question here: compare the likelihood of a result scarcely different from 0.2 being produced by a universe where θ = 0.2 – where this has been given a spiked prior of 0.5 under H₀ – with the likelihood of that result being produced by any θ in a small band of θ values, which have been given a very low prior under H₁. Clearly, θ = 0.2 is more likely, and we have an example of the Jeffreys–Fisher disagreement.

Who should be afraid of this disagreement (to echo the title of Spanos’ paper)? Many tribes, including some Bayesians, think it only goes to cast doubt on this particular Bayes factor. Compare it with proposal 2 in Exhibit (vii): the Likelihood ratio most generous to the alternative: Lik(0.2)/Lik(θ_max). We know the maximally likely value for θ, ${\theta }_{\text{max}}=\overline{x}$:

$\overline{x}=k/n=0.20165233={\theta }_{\text{max}},$

$\text{Pr}\left(k|H_{\text{max}}\right)=\left(\begin{array}{c}n\\ k\end{array}\right){0.20165233}^k{\left(1-0.20165233\right)}^{n-k}=0.0013694656,$

$\text{Lik}(0.2)=0.000015394,\text{and}\text{Lik}({\theta }_{\text{max}})=0.0013694656.$

Now B₀₁ is 0.01 and B₁₀, Lik(θ_max)/Lik(0.2) = 89.

Why should a result 89 times more likely under alternative θ_max than under θ = 0.2 be taken as strong evidence for θ = 0.2? It shouldn’t, according to some, including Lindley’s own student, default Bayesian José Bernardo (Reference Bernardo2010). Presumably, the Likelihoodist concurs. There are family feuds within and between the diverse tribes of probabilisms.Footnote ⁷

Greenland and Poole

Given how often spiked priors arise in foundational arguments, it’s worth noting that even Bayesians Edwards, Lindman, and Savage (Reference Edwards, Lindman and Savage1963, p. 235), despite raising the “P-values exaggerate” argument, aver that for Bayesian statisticians, “no procedure for testing a sharp null hypothesis is likely to be appropriate unless the null hypothesis deserves special initial credence.” Epidemiologists Sander Greenland and Charles Poole, who claim not to identify with any one statistical tribe, but who often lead critics of significance tests, say:

They angle to reconcile P-values and posteriors, and to this end they invoke the matching result in # 3, Exhibit (vii). An uninformative prior, assigning equal probability to all values of the parameter, allows the P-value to approximate the posterior probability that θ < 0 in one-sided testing (θ ≤ 0 vs. θ > 0). In two-sided testing, the posterior probability that θ is on the opposite side of 0 than the observed is P/2. They proffer this as a way “to live with” P-values. Commenting on them, Andrew Gelman (Reference Gelman2013, p. 72) raises this objection:

(The P-value is 0.16.) Rather than relying on non-informative priors, Gelman prefers to use prior information that leans towards the null. This avoids as high a posterior to H₁ as when using the matching result.

Greenland and Poole respond that Gelman is overlooking the hazard of “strong priors that are not well founded. … Whatever our prior opinion and its foundation, we still need reference analyses with weakly informative priors to alert us to how much our prior probabilities are driving our posterior probabilities” (Reference Greenland and Poole2013, p. 76). They rightly point out that, in some circles, giving weight to the null can be the outgrowth of some ill-grounded metaphysics about “simplicity.” Or it may be seen as an assumption akin to a presumption of innocence in law. So the question turns on the appropriate prior on the null.

Look what has happened! The problem was simply to express “I’m not impressed” with a result reaching a P-value of 0.16: Differences even larger than 1 standard error are not so very infrequent – they occur 16% of the time – even if there’s zero effect. So I’m not convinced of the reality of the effect, based on this result. P-values did their job, reflecting as they do the severity requirement. H₁ has passed a lousy test. That’s that. No prior probability assignment to H₀ is needed. Problem solved.

But there’s a predilection for changing the problem (if you’re a probabilist). Greenland and Poole feel they’re helping us to live with P-values without misinterpretation. By choosing the prior so that the P-value matches the posterior on H₀, they supply us “with correct interpretations” (ibid., p. 77) where “correct interpretations” are those where the misinterpretation (of a P-value as a posterior in the null) is not a misinterpretation. To a severe tester, this results in completely changing the problem from an assessment of how well tested the reality of the effect is, with the given data, to what odds I would give in betting, or the like. We land in the same uncharted waters as with other attempts to fix P-values, when we could have stayed on the cruise ship, interpreting P-values as intended.

Exhibit (viii): Whether P-values Exaggerate Depends on Philosophy.

When a group of authors holding rather different perspectives get together to examine a position, the upshot can take them out of their usual comfort zones. We need more of that. (See also the survey in Hurlbert and Lombardi Reference Hurlbert and Lombardi2009, and Haig 2016.) Here’s an exhibit from Greenland et al. (Reference Greenland, Senn and Rothman2016). They greet each member of a list of incorrect interpretations of P-values with “No!”, but then make this exciting remark:

It’s erroneous to fault one statistical philosophy from the perspective of a philosophy with a different and incompatible conception of evidence or inference. The severity principle always evaluates a claim as against its denial within the framework set. In N-P tests, the frame is within a model, and the hypotheses exhaust the parameter space. Part of the problem may stem from supposing N-P tests infer a point alternative, and then seeking that point. Whether you agree with the error statistical form of inference, you can use the severity principle to get beyond this particular statistics battle.