Two Points Stephen Senn Correctly Dubs Nonsense and Ludicrous

Start with an extreme example: Suppose someone is testing H₀: the drug cures no one. An alternative H₁ is it cures nearly everyone. Clearly these are not the only possibilities. Say the test is practically guaranteed to reject H₀, if in fact H₁, the drug cures practically everyone. The test has high power to detect H₁. You wouldn’t say that its rejecting H₀ is evidence of H₁. H₁ entails it’s very probable you’ll reject H₀; but rejecting H₀ doesn’t warrant H_1. To think otherwise is to allow problematic statistical affirming the consequent – the basis for the MM fallacy (Section 2.1). This obvious point lets you zero in on some confusions about power.

Stephen Senn’s contributions to statistical foundations are once again spot on. In drug development, it is typical to require a high power of 0.8 or 0.9 to detect effects deemed of clinical relevance. The clinically relevant discrepancy, as Senn sees it, is the discrepancy “one should not like to miss” (Reference Senn2007, p. 196). Senn labels this delta Δ. He is considering a difference between means, so the null hypothesis is typically 0. We’ll apply severity to his example in Exhibit (iv) of this tour. Here the same points will be made with respect to our one-sided Normal test T+: H₀: μ ≤ μ₀ vs. H₁: μ > μ₀, letting μ₀ = 0, σ known. We may view Δ as the value of μ of clinical relevance. (Nothing changes in this discussion if it’s estimated as s.) The test takes the form

“Reject H₀” is the shorthand for “infer a statistically significant difference” at the level of the test. Though Z is the test statistic, it makes for a simpler presentation to use the cut-off for rejection in terms of $\overline{x}$_α: Reject H₀ iff $\overline{X}\ge {\overline{x}}_{\alpha }=({\mu }_0+z_{\alpha }\sigma /\surd n)$.

Let’s abbreviate the alternative against which test T+ has 0.8 power by μ^.8, when it’s clear what test we’re talking about. So POW(μ^.8) = 0.8, and let’s suppose μ^.8 is the clinically relevant difference Δ. Senn asks, what does μ^.8 mean in relation to what we are entitled to infer when we obtain statistical significance? Can we say, upon rejecting the null hypothesis, that the treatment has a clinically relevant effect, i.e., μ ≥ μ^.8 (or μ > μ^.8)?

“This is a surprisingly widespread piece of nonsense which has even made its way into one book on drug industry trials” (ibid., p. 201). The reason it is nonsense, Senn explains, is that μ^.8 must be in excess of the cut-off for rejection, in particular, ${\mu }^{.8}={\overline{x}}_{\alpha }+0.85{\sigma }_{\overline{X}}$ (where ${\sigma }_{\overline{X}}=\sigma /\surd n$). We know we are only entitled to infer μ exceeds the lower bound of the confidence interval at a reasonable level; whereas, μ^.8 is actually the upper bound of a 0.8 (one-sided) confidence interval, formed having observed $\overline{x}={\overline{x}}_{\alpha }.$ All we are to infer, officially, from just reaching the cut-off ${\overline{x}}_{\alpha }$, is that μ > 0.

Granted, as Senn admits, the test “lacks ambition” (ibid., p. 202), but with more data and with results surpassing the minimal cut-off, we may uncover a clinically relevant discrepancy. Why not just set up the test to enable the clinically relevant discrepancy to be inferred whenever the null is rejected?

This requires redefining Δ. “It is no longer ‘the difference we should not like to miss’ but instead becomes ‘the difference we should like to prove obtains’” (ibid.). Some call this the “clinically irrelevant difference” (ibid.). But then we can’t also have high power to detect H₁: μ > Δ.

Indeed, it will be approximately 1 − α. So the power – the probability the observed difference exceeds the critical value under H₁ – is, in this case, around α. The researcher is free to specify the null as H₀: μ ≤ Δ, but Senn argues against doing so, at least in drug testing, because “a nonsignificant result will often mean the end of the road for a treatment. It will be lost forever. However, a treatment which shows a ‘significant’ effect will be studied further” (ibid.). This goes beyond issues of interest now. The point is: Δ cannot be the value in H₀ and also the value against which we want 0.8 power to detect, i.e., μ^.8.

If testing H₀: μ ≤ 0 vs. H₁: μ > 0, then a just α-significant result is poor evidence for μ ≥ μ^.8 (or other alternative with high power). To think it’s good evidence is nonsense. Senn’s related point is that it is ludicrous to assume the effect is either 0 or a clinically relevant difference, as if we are testing

“But where we are unsure whether a drug works or not, it would be ludicrous to maintain that it cannot have an effect which, while greater than nothing, is less than the clinically relevant difference” (ibid., p. 201). That is, it is ludicrous to cut out everything in between 0 and Δ. By the same token, it would seem odd to give a 0.5 prior probability to H₀, and the remaining 0.5 to H₁. We will have plenty of occasions to return to Senn’s points about what’s nonsensical and ludicrous.

Trade-offs and Benchmarks

Between H₀ and ${\overline{\mathit{x}}}_{\mathbf{\alpha }}$ the power goes from α to 0.5.

Keeping to our simple test T+ will amply reward us here.

a. The power against H₀ is α. We can use the power function to define the probability of a Type I error or the significance level of the test:

$\text{POW}(\text{T}+,{\mu }_0)=\text{Pr}(\overline{X}\ge {\overline{x}}_{\alpha };{\mu }_0),\text{where}{\overline{x}}_{\alpha }={\mu }_0+z_{\alpha }{\sigma }_{\overline{X}}$

Standardizing $\overline{X}$, we get $Z=[({\mu }_0+z_{\alpha }{\sigma }_{\overline{X}})-{\mu }_0]/{\sigma }_{\overline{X}}$.

The power at the null is Pr(Z ≥ z_α;μ₀) = α.

It’s the low power against H₀ that warrants taking a rejection as evidence that μ >μ₀. This is desirable: we infer an indication of discrepancy from H₀ because a null world would probably have resulted in a smaller difference than we observed.

b. The power of T+ for ${\mu }_1={\overline{x}}_{\alpha }$ is 0.5. In that case, Z = 0, and Pr(Z ≥ 0) = 0.5, so

$\text{POW}(\text{T}+,{\mu }_1={\overline{x}}_{\alpha })=\mathrm{0.5.}$

The power only gets to be greater than 0.5 for alternatives that exceed the cut-off ${\overline{x}}_{\alpha }$, whatever it is. As noted, ${\mu }^{.8}={\overline{x}}_{\alpha }+0.85{\sigma }_{\overline{X}}$ since $\text{POW}(\text{T}+,{\overline{x}}_{\alpha }+0.85{\sigma }_{\overline{X}})=0.8$. Tests ensuring 0.9 power are also often of interest: ${\mu }^{.9}={\overline{x}}_{\alpha }+1.28{\sigma }_{\overline{X}}$. We get these shortcuts:

• Case 1: POW(T+, μ) for μ between H₀ and $\mu ={\overline{x}}_{\alpha }$:

• If ${\mu }_1={\overline{x}}_{\alpha }-k{\sigma }_{\overline{X}}$ then POW(T+, μ₁) = area to the right of k under N(0,1) (≤ 0.5).

• Case 2: POW(T+, μ) for μ greater than ${\overline{x}}_{\alpha }$:

• If ${\mu }_1={\overline{x}}_{\alpha }+k{\sigma }_{\overline{X}}$ then POW(T+, μ₁) = area to the right of −k under N (0,1) (> 0.5).

• Remember ${\overline{x}}_{\alpha }$ is ${\mu }_0+z_{\alpha }{\sigma }_{\overline{X}}$.

Trade-offs Between the Type I and Type II Error Probability

We know that, for a given test, as the probability of a Type I error goes down the probability of a Type II error goes up (and power goes down). And as the probability of a Type II error goes down (and power goes up), the probability of a Type I error goes up, assuming we leave everything else the same. There’s a trade-off between the two error probabilities. (No free lunch.) So if someone said: As the power increases, the probability of a Type I error decreases, they’d be saying, as the Type II error decreases, the probability of a Type I error decreases. That’s the opposite of a trade-off! You’d know automatically they had made a mistake or were simply defining things in a way that differs from standard N-P statistical tests. Now you may say, “I don’t care about Type I and II errors, I’m interested in inferring estimated effect sizes.” I too want to infer magnitudes. But those will be ready to hand once we tell what’s true about the existing concepts.

While μ^.8 is obtained by adding 0.85 ${\sigma }_{\overline{X}}$ to ${\overline{x}}_{\alpha }$, in day-to-day rounding, if you’re like me, you’re more likely to remember the result of adding 1${\sigma }_{\overline{X}}$ to ${\overline{x}}_{\alpha }$. That takes us to a value of μ against which the test has 0.84 power, μ^.84:

In test T+ the range of possible values of $\overline{X}$ and µ are the same, so we are able to set µ values this way, without confusing the parameter and sample spaces.

Exhibit (i).

Let test T+ (α = 0.025) be H₀: μ = 0 vs. H₁: μ ≥ 0, α = 0.025, n = 25, σ = 1. Using the 2 ${\sigma }_{\overline{X}}$ cut-off: ${\overline{x}}_{0.025}=2(1)/\surd 25=0.4$ (using 1.96 it’s 3.92). Suppose you are instructed to decrease the Type I error probability α to 0.001 but it’s impossible to get more samples. This requires the hurdle for rejection to be higher than in our original test. The new cut-off for test T+ will be ${\overline{x}}_{0.001}$. It must be 3${\sigma }_{\overline{X}}$ greater than 0 rather than only 2${\sigma }_{\overline{X}}$: ${\overline{x}}_{0.001}=0+3(1)/\surd 25=0.6$. We decrease α (the Type I error probability) from 0.025 to 0.001 by moving the hurdle over to the right by 1${\sigma }_{\overline{X}}$ unit. But we’ve just made the power lower for any discrepancy or alternative. For what value of µ does this new test have 0.84 power?

POW(T+, α = 0.001, µ^.84 = ?) = 0.84.

We know: µ^.84 = 0.6 + (0.2) = 0.8. So, POW(T+, α = 0.001, µ = 0.8) = 0.84. Decreasing the Type I error by moving the hurdle over to the right by 1 ${\sigma }_{\overline{X}}$ unit results in the alternative against which we have 0.84 power µ^.84 also moving over to the right by 1${\sigma }_{\overline{X}}$ (Figure 5.1). We see the trade-off very neatly, at least in one direction.

Figure 5.1 POW(T+, α = 0.001, µ = 0.8) = 0.84.

Consider the discrepancy of μ = 0.6 (Figure 5.2). The power to detect 0.6 in test T+ (α = 0.001) is now only 0.5! In test T+ (α = 0.025) it is 0.84. Test T+ (α = 0.001) is less powerful than T+ (α = 0.025).

Figure 5.2 POW(T+, α = 0.001, µ = 0.6) = 0.5.

Should you hear someone say that the higher the power, the higher the hurdle for rejection, you’d know they are confused or using terms in an incorrect way. (The hurdle is how large the cut-off must be before rejecting at the given level.) Why then do Ziliak and McCloskey, popular critics of significance tests, announce: “refutations of the null are trivially easy to achieve if power is low enough or the sample is large enough” (Reference Ziliak and McCloskey2008a, p. 152)? Increasing sample size means increased power, so the second disjunct is correct. The first disjunct is not. One might be tempted to suppose they mean “power is high enough,” but one would be mistaken. They mean what they wrote. Aris Spanos (2008a) points this out (in a review of their book), and I can’t figure out why they dismiss such corrections as “a lot of technical smoke” (Reference Spanos, Durlauf and Blume2008b, p. 166).

Ziliak and McCloskey Get Their Hurdles in a Twist

Still, their slippery slides are quite illuminating.

With a wink and a nod, the first sentence isn’t too bad, even though, at the very least, it is mandatory to specify a particular “another hypothesis,” µ′. But what about the statement: if the power of a test is high, then a rejection of the null is probably correct?

We follow our rule of generous interpretation to try to see it as true. Let’s allow the “;” in the first premise to be a conditional probability “|”, using μ^0.84:

1. 1. Pr(Test T+ rejects the null | μ^.84) = 0.84.

2. 2. Test T+ rejects the null hypothesis.

   Therefore, the rejection is correct with probability 0.84.

Oops. The premises are true, but the conclusion fallaciously transposes premise 1 to obtain conditional probability Pr(μ^.84 | test T+ rejects the null) = 0.84. What I think they want to say, or at any rate what would be correct, is

So the Type II error probability is 0.16. Looking at it this way, the flaw is in computing the complement of premise 1 by transposing (as we saw in the Higgs example, Section 3.8). Let’s be clear about significance levels and hurdles. According to Ziliak and McCloskey:

They construe “little significance” as little hurdles! It explains how they wound up supposing high power translates into high hurdles. It’s the opposite. The higher the hurdle, the more difficult it is to reject, and the lower the power. High hurdles correspond to insensitive tests, like insensitive fire alarms. It might be that using “sensitivity” rather than power would make this abundantly clear. We may coin: The high power = high hurdle (for rejection) fallacy. A powerful test does give the null hypothesis a harder time in the sense that it’s more probable that discrepancies from it are detected. That makes it easier for H₁. Z & M have their hurdles in a twist.

Three Steps to the Argument

Goodman’s context involves statistically significant results – he writes them as z values, as it is a case of Normal testing. We’re not given the sample size or the precise test, but it won’t matter for the key argument. He gives the two-sided α value, although “[w]e assume that we know the direction of the effect” (ibid., p. 491). I am not objecting. Even if we run a two-sided test, once we see the direction, it makes sense to look at the power of the relevant one-sided test, but double the α value. There are three steps. First step: form the likelihood ratio of the statistically significant outcome z_0.025 (i.e., 1.96) under the null hypothesis and some alternative (where Pr is the density):

But which alternative H′? Alternatives against which the test has high power, say 0.8, 0.84, or 0.9, are of interest, and he chooses μ^.9. He writes this alternative as μ = Δ_{0.05, 0.9}, “the difference against which the hypothesis test has two-sided α = 0.05 and one-sided β = 0.10 (power = 0.90)” (ibid., p. 496). We know from our benchmarks that μ^.9 is approximately 1.28 standard errors from the one-sided cut-off: $(1.96+1.28){\sigma }_{\overline{X}}=3.24{\sigma }_{\overline{X}}$. The likelihood for alternative μ^.9 is smaller than if one had used the maximum likely alternative (as in Johnson).Footnote ¹

I’ll follow Goodman in computing the likelihood of the null over the alternative, although most of the authors we’ve considered do the reverse. Not that it matters so long as you keep them straight.

“Two likelihood ratios are compared, one for a ‘precise’ p value, e.g., p = 0.03, and one for … p ≤ α = 0.03. The alternative hypothesis used here is the one against which the hypothesis test has 90 percent power (two-sided α = 0.05) …” (ibid., pp. 490–1). The precise and imprecise P-values correspond to reporting z = 1.96 and z ≥ 1.96, respectively.Footnote ² The likelihood ratio for the precise P-value “corresponds to the ratio of heights of the two probability densities” (ibid., p. 491): Number this as (1):

These are the ordinates not the tail areas of the Normal distribution.

That’s the first step. The second step is to consider the likelihood ratio for the imprecise P-value, where the result is described coarsely as {z ≥ 1.96} rather than z = 1.96 (or equivalently p ≤ 0.025 rather than p = 0.025):

We see at once that the value of (2) is

The comparative evidence for the null using (1) is considerably larger (0.33) than using (2) where it’s only 0.03. Figure 5.3 shows what’s being compared.

Figure 5.3 Comparing precise and imprecise P-values. The likelihood ratio is A/B, the ratio of the curve heights at the observed data. The likelihood ratio associated with the imprecise P-value (p ≤ α) is the ratio of the small darkly shaded area to the total shaded area.

(adapted from Goodman Reference Goodman1993, Figure 1 p. 492)

The difference Goodman wishes to highlight looks even more impressive if we flip the ratios in (1) and (2) to get the comparative evidence for the alternative compared to the null. We get 0.176/0.058 = 3.03 using z = 1.96, and 0.9/0.025 = 36 using z ≥ 1.96. Either way you look at it, using the tail areas to compare support exaggerates the evidence against the null in favor of μ^.9. Or so it seems.

Now for the third step: Assign priors of 0.5 to μ₀ and to μ^.9:

He concludes:

The posterior probabilities, he says, are assessments of credibility in the hypothesis. Join me for a break at the coffeehouse, called “Take Type II”, where we’ll engage a live exhibit.

Live Exhibit (ii): Drill Prompt.

Using any relevant past travels, tours, and souvenirs, critically appraise his argument. Resist the impulse to simply ask, “where do significance tests recommend using error probabilities to form a likelihood ratio, and treat the result as a relative support measure?” Give him the most generous interpretation in order to see what’s being claimed.

How do you grade the following test taker?

Goodman actually runs two distinct issues together: The first issue contrasts a report of the observed P-values with a report of whether or not a predesignated cut-off is met (his “imprecise” P-value); the second issue is using likelihood ratios (as in (1)) as opposed to using tail areas. As I understand him, Goodman treats a report of the precise P-value as calling for a likelihood analysis as in (1), supplemented in the third step to get a posterior probability. But, even a reported P-value will lead to the use of tail areas in Fisherian statistics. This might not be a major point, but it is worth noting. For the error statistician, use of the tail area isn’t to throw away the particular data and lump together all z ≥ z_α. It’s to signal an interest in the probability the method would produce z ≥ z_α under various hypotheses, to determine its capabilities (Translation Guide, Souvenir C). Goodman’s hypothesis tester only reports z ≥ z_α and so he portrays the Bayesian (or Likelihoodist) as forced to compute (2). The error statistical tester doesn’t advocate this.

Let’s have a look at (1): comparing the likelihoods of μ₀ and μ^.9 given z = 1.96. Why look at an alternative so far away from μ₀? The value μ^.9 gets a low likelihood, even given statistically significant z_α supplying a small denominator in (1). A classic issue with Bayes factors or likelihood ratios is the ease of finding little evidence against a null, and even evidence for it, by choosing an appropriately distant alternative. But μ^.9 is a “typical choice” in epidemiology, Goodman notes (ibid., p. 491). Sure it’s a discrepancy we want high power to detect. That’s different from using it to form a comparative likelihood. As Senn remarks, it’s “ludicrous” to consider just the null and μ^.9, which scarcely exhaust the parameter space, and “nonsense” to take a single statistically significant result as even decent evidence for the discrepancy we should not like to miss. μ^.9 is around 1.28 standard errors from the one-sided cut-off: $(1.96+1.28){\sigma }_{\overline{X}}=3.24{\sigma }_{\overline{X}}$. We know right away that any μ value in excess of the observed statistically significant z_α is one we cannot have good evidence for. We can only have decent evidence that μ exceeds values that would form the CI lower bound for a confidence (or severity) level at least 0.7, 0.8, 0.9, etc. That requires subtracting from the observed data, not adding to it. Were the data generated by μ^.9, then 90% of the time we’d have gotten a larger difference than we observed. Goodman might reply that I am using error probability criteria to judge his Bayesian analysis; but his Bayesian analysis was to show the significance test exaggerates evidence against the test hypothesis H₀. To the tester, it’s his analysis that’s exaggerating by giving credibility 0.75 to μ^.9. Perhaps it makes sense with the 0.5 prior, but does that make sense?

Assigning 0.75 to the alternative μ^.9 (using the likelihood ratio in (1)) does not convey that this is terrible evidence for a discrepancy that large. Granted, the posterior for μ^.9 would be even higher using the tail areas in (2), namely, 0.97 – and that is his real point, which I’ve yet to consider. He’s right that using (2) in his Bayesian computation gives a whopping 0.97 posterior probability to μ^.9 (instead of merely 0.75, as on the analysis he endorses). Yet a significance tester wouldn’t compute (2), it’s outside significance testing. Considering the tails makes it harder, not easier, to find evidence against a null – when properly used in a significance test.

He’s right that using α/POW(μ^.9) as a likelihood ratio in computing a posterior probability for the point alternative μ^.9 (using spiked priors of 0.5) gives some very strange answers. The tester isn’t looking to compare point values, but to infer discrepancies. I’m doing the best I can to relate his criticism to what significance testers do.

The error statistician deservedly conveys the lack of stringency owed to any inference to μ^.9. She finds the data give fairly good grounds that μ is less than μ^.9 (confidence level 0.9 if one-sided, 0.8 if two-sided). Statistical quantities are like stew ingredients that you can jumble together into a farrago of computations, but there’s a danger it produces an inference at odds with error statistical principles. For a significance tester, the alleged criticism falls apart; but it also leads me to question Goodman’s posterior in (1).

What’s Wrong with Using (1 − β)/α (or α/(1 − β)) to Compare Evidence?

I think the test taker from last year’s cruise did pretty well, don’t you? But her “farrago” remark leads to a general point about the ills of using (1 − β)/α and α/(1 − β) to compare the evidence in favor of H₁ over H₀ or H₀ over H₁, respectively. Let’s focus on the (1 − β)/α formulation. Benjamin and J. Berger (Reference Benjamin and Berger2016) call it the pre-data rejection ratio:

It is the probability of rejection when the alternative hypothesis is true, divided by the probability of rejection when the null hypothesis is true, i.e., the ratio of the power of the experiment to the Type I error of the experiment. The rejection ratio has a straightforward interpretation as quantifying the strength of evidence about the alternative hypothesis relative to the null hypothesis conveyed by the experimental result being statistically significant.

(p. 1)

But does it? I say no (and J. Berger says he concursFootnote ³). Let’s illustrate. Looking at Figure 5.3, we can select an alternative associated with power as high as we like by dragging the curve representing H₁ to the right.

Imagine pulling it even further than alternative μ^.9. How about μ^.999? If we consider the alternative $\mu ={\overline{x}}_{0.025}+3{\sigma }_{\overline{X}}$ then POW(T+, μ₁) = area to the right of −3 under the Normal curve, which is a whopping 0.999. For some numbers, use our familiar T+: H₀: μ ≤ 0 vs. H₁: μ > 0, α = 0.025, n = 25, σ = 1, ${\sigma }_{\overline{X}}=0.2$. So the cut-off ${\overline{x}}_{0.025}={\mu }_0+1.96{\sigma }_{\overline{X}}=(0+1.96(0.2))=0.392$. Thus,

${\mu }^{.999}=4.96{\sigma }_{\overline{X}}=4.96(0.2)=0.99\simeq 1.$

Let the observed outcome just reach the cut-off to reject H₀, z₀ = 0.392. The rejection ratio is

POW(T+, μ = 1)/α = 40 (i.e., 0.999/0.025)!

Would even a Likelihoodist wish to say the strength of evidence for μ = 1 is 40 times that of H₀? The data $\overline{x}=0.392$ are even closer to 0 than to 1.

How then can it seem plausible, for comparativists, to compute a relative likelihood this way?

We can view it through their eyes as follows: take Z ≥ 1.96 as the lump outcome and reason along Likelihoodist lines. The probability is very high that Z ≥ 1.96 under the assumption that μ =1:

Pr(Z ≥ 1.96; μ^0.999) = 0.999.

The probability is low that Z ≥ 1.96 under the assumption that μ = μ₀ = 0:

Pr(Z ≥ 1.96; μ = μ₀) = 0.025.

We’ve observed z₀ = 1.96 (so Z ≥ 1.96).

Therefore, μ^0.999 (i.e., 1) makes the result Z ≥ 1.96 more probable than does μ = 0.

Therefore, the result is better evidence that μ =1 than it is for μ = 0. But this likelihood reasoning only holds for the specific value of z. Granted, Bayarri, Benjamin, Berger, and Sellke (Reference Bayarri, Benjamin, Berger and Sellke2016) recommend the prerejection ratio before the data are in, and “the ‘post-experimental rejection ratio’ (or Bayes factor) when presenting their experimental results” (p. 91). The authors regard the pre-data rejection ratio as frequentist, but it turns out they’re using Berger’s “frequentist principle,” which, you will recall, is in terms of error probability₂ (Section 3.6). A creation built on frequentist measures doesn’t mean the result captures frequentist error statistical₁ reasoning. It might be a kind of Frequentstein entity!

Notably, power works in the opposite way. If there’s a high probability you should have observed a larger difference than you did, assuming the data came from a world where μ = μ₁, then the data indicate you’re not in a world where μ> μ₁.

If Pr(Z > z₀; μ = μ₁) = high, then Z = z₀ is strong evidence that μ ≤ μ₁!

Rather than being evidence for μ₁, the just statistically significant result, or one that just misses, is evidence against μ being as high as μ₁. POW(μ₁) is not a measure of how well the data fit μ₁, but rather a measure of a test’s capability to detect μ₁ by setting off the significance alarm (at size α). Having set off the alarm, you’re not entitled to infer μ₁, but only discrepancies that have passed with severity (SIR). Else you’re making mountains out of molehills.

Power Analysis Follows Significance Test Reasoning

Early proponents of power analysis that I’m aware of include Cohen (Reference Cohen1962), Gibbons and Pratt (Reference Gibbons and Pratt1975), and Neyman (Reference Neyman1955). It was the basis for my introducing severity, Mayo (Reference Mayo1983). Both Neyman and Cohen make it clear that power analysis uses the same reasoning as does significance testing.Footnote ⁴ First Cohen:

Here Cohen imagines the researcher sets the size of a negligible discrepancy ahead of time – something not always available. Even where a negligible i may be specified, it’s rare that the power to detect it is high. Two important points can still be made: First, Cohen doesn’t instruct you to infer there’s no discrepancy from H₀, merely that it’s “no more than i.” Second, even if your test doesn’t have high power to detect negligible i, you can infer the population discrepancy is less than whatever γ your test does have high power to detect. Some call this its detectable discrepancy size.

A little note on language. Cohen distinguishes the population ES and the observed ES, both in σ units. Keeping to Cohen’s ES_s for “the effect size in the sample” (Reference Cohen1988, p. 17) prevents a tendency to run them together. I continue to use “discrepancy” and “difference” for the population and observed differences, respectively, indicating the units being used.

Neyman Chides Carnap, Again

Neyman was an early power analyst? Yes, it’s in his “The Problem of Inductive Inference” (Reference Neyman1955) where we heard Neyman chide Carnap for ignoring the statistical model (Section 2.7). Neyman says:

Ironically, Neyman also criticizes Fisher’s move from a large P-value to inferring the null hypothesis as:

Typically, the hypothesis tested, [H₀] in the N-P context, could be swapped with the alternative. Let’s leave the museum where a leader of the severe testing tribe makes some comparisons.

Attained Power Π

So power analysis is in the spirit of severe testing. Still, power analysis is calculated relative to an outcome just missing the cut-off c_α. This corresponds to an observed difference whose P-value just exceeds α. This is, in effect, the worst case of a negative result. What if the actual outcome yields an even smaller difference (larger P-value)?

Consider test T+ (α = 0.025) above. No one wants to turn pages so here it is: H₀: μ ≤ 0 vs. H₁: μ > 0, α = 0.025, n = 25, σ = 1, ${\sigma }_{\overline{X}}=0.2$. So the cut-off ${\overline{x}}_{0.025}={\mu }_0+1.96{\sigma }_{\overline{X}}=(0+1.96(0.2))=0.392$, or, with the 2 ${\sigma }_{\overline{X}}$ cut-off, ${\overline{x}}_{0.025}=0.4$. Consider an arbitrary inference μ ≤ 0.2. We know POW(T+, μ = 0.2) = 0.16 (1 ${\sigma }_{\overline{X}}$ is subtracted from 0.4). A value of 0.16 is quite lousy power. It follows that no statistically insignificant result can warrant μ ≤ 0.2 for the power analyst. Power analysis only allows ruling out values as high as μ^.8, μ^.84, μ^.9, and so on. The power of a test is fixed once and for all and doesn’t change with the observed mean $\overline{x}$. Why consider every non-significant result as if it just barely missed the cut-off? Suppose, $\overline{x}=-0.2$. This is 2 ${\sigma }_{\overline{X}}$ lower than 0.2. Surely that should be taken into account? It is: 0.2 is the upper 0.975 confidence bound and $\text{SEV}(\text{T}+,{\overline{x}}_0=-0.2,\mu <0.2)=0.975$.Footnote ⁵

What enables substituting the observed value of the test statistic, d(x₀), is the counterfactual reasoning of severity:

That is, if the attained power (att-power) of T+ against μ ≤ μ₀ + γ is very high, the inference to μ ≤ μ₀ + γ is warranted with severity. (As always, whether it’s a mere indication or genuine evidence depends on whether it passes an audit.) If you elect to use the term attained power, you’ll have to avoid confusing it with animals given similar names; I’ll introduce you to them shortly.

Compare power analytic reasoning with severity (or att-power) reasoning from a negative or insignificant result from T+.

Severity replaces the predesignated cut-off c_α with the observed d(x₀). Thus we obtain the same result if we choose to remain in the Fisherian tribe, as seen in the Frequentist Evidential Principle FEV(ii) (Section 3.3).

We still abide by the logic of power analysis, since if Pr(d(X) ≥ d(x₀); µ₁) = high, then Pr(d(X) ≥ c_α; µ₁) = high, at least in a test with a sensible distance measure like T+. In other words, power analysis is conservative. It gives a sufficient but not a necessary condition for warranting an upper bound: µ ≤ µ₁. But it can be way too conservative as we just saw.

Ordinary power analysis requires (1) to be high (for non-significance to warrant μ ≤ μ₀ + γ).

Just missing the cut-off c_α is the worst case. It is more informative to look at (2):

(1) can be low while (2) is high. The computation in (2) measures the severity (or degree of corroboration) for the inference μ ≤ μ₀ + γ. The analysis with Cox kept to Π(γ) (or “attained sensitivity”), keeping “power” out of it.

As an entrée to Exhibit (iv): Isn’t severity just power? This is to compare apples and frogs. The power of a test to detect an alternative is an error probability of a method (one minus the probability of the corresponding Type II error). Power analysis is a way of using power to assess a statistical inference in the case of a negative result. Severity, by contrast, is always to assess a statistical inference. Severity is always in relation to a particular claim or inference C, from a test T and an outcome x. So with that out of the way, what if the question is put properly thus: If a result from test T+ is just statistically insignificant at level α, then is the test’s power to detect μ₁ equal to the severity for inference C: µ> µ₁? The answer is no. It would be equal to the severity for inferring the denial of C! See Figure 5.4 comparing SEV(µ> µ₁) and POW(µ₁).

Figure 5.4 Severity for (µ> µ₁) vs power (µ₁).

Figure 5.5 The observed difference is 90, each group has n = 200 patients, and the standard deviation is 450.

Exhibit (iv): Difference Between Means Illustration.

I’ve been making use of Senn’s points about the nonsensical and the ludicrous in motivating the severity assessment in relation to power. I want to show you severity for a difference between means, and fortuitously Senn (2019) alludes to severity in the latest edition of his textbook to make the same point. An example is a placebo-controlled trial in asthma where the amount of air a person can exhale in one second, the forced expiratory volume, is measured. “The clinically relevant difference is presumed to be 200 ml and the standard deviation 450 ml” (p. 197). It’s a test of

H₀: δ = μ₁ − μ₀ ≤ 0 vs. H₁: δ > 0.

He will use a one-sided test at the 2.5% level, using 200 patients in each group yielding the standard error (SE) for the difference between two means equal to $(450\surd 2/\surd \mathrm{n})$.Footnote ⁶ The test has over 0.99 power for detecting δ = 200. The observed difference d = $\left(\overline{X}-\overline{Y}\right)=90$, which is statistically significant at the 0.025 level (90/SE = 2), but it wouldn’t warrant inferring δ > 200, and we can see the severity for δ > 200 is extremely low. The observed difference is statistically significantly different from H₀: in accord with (or in the direction of) H₁, so severity computes the probability of a worse fit with H₁ under δ = 200:

SEV(δ > 200) = Pr(d < 90; δ = 200).

While this is strictly a Student’s t distribution, given the large sample size, we can use the Normal distribution:Footnote ⁷

$Z=\left\{\frac{\surd 200\left(90-200\right)}{450\surd 2}\right\}=\left(90–200\right)/45=-\mathrm{2.44.}$

SEV(δ > 200) = the area to the left of −2.44 (See Figure 5.5) which is 0.007,

For a few examples, SEV(δ > 100) = 0.412 the area to the left of z = (90 − 100)/45 = −0.222. SEV(δ > 50) = 0.813 the area to the left of z = (90 − 50)/ 45 = 0.889.

SEV(δ > 10) = 0.962 the area to the left of z = (90 − 10)/45 = 1.778.

Senn gives another way to view the severity assessment of δ > δ′, namely “adopt [δ = δ′], as a null hypothesis and then turn the significance test machinery on it (2019). In the case of testing δ = 200, the P-value would be 0.999. Scarcely evidence against it. We first visited this mathematical link in touring the connection between severity and confidence intervals in Section 3.8. As noted, the error statistician is loath to advocate modifying the null hypothesis because the point of a severity assessment is to supply a basis for interpreting tests that is absent in existing tests. Since significance tests aren’t explicit about assessing discrepancies, and since the rationale for P-values is questioned in all the ways we’ve espied, it’s best to supply a fresh rationale. I have offered the severity rationale as a basis for understanding, if not buying, error statistical reasoning. The severity computation might be seen as a rule of thumb to avoid misinterpretations; it could be arrived at through other means, including varying the null hypotheses. It’s the idea of viewing statistical inference as severe testing that invites a non-trivial difference with probabilism.

Souvenir W: The Severity Interpretation of Negative Results (SIN) for Test T+

Applying our general abbreviation: SEV(test T+, outcome x, inference H), we get “the severity with which μ ≤ μ₁ passes test T+, with data x₀”:

where μ₁ = (μ₀ + γ), for some γ ≥ 0. If it’s clear which test we’re discussing, we use our abbreviation: SEV(μ ≤ μ₁). We obtain a companion to the severity interpretation of rejection (SIR), Section 4.4, Souvenir R:

To break it down, in the case of a statistically insignificant result:

We look at {d(X) > d(x₀)} because severity directs us to consider a “worse fit” with the claim of interest. That μ ≤ μ₁ is false within our model means that μ > μ₁. Thus:

Now μ > μ₁ is a composite hypothesis, containing all the values in excess of μ₁. How can we compute it? As with power calculations, we evaluate severity at a point μ₁ = (μ₀ + γ), for some γ ≥ 0, because for values μ ≥ μ₁ the severity increases. So we need only to compute

To compute SEV we compute Pr(d(X) > d(x₀); μ = μ₁) for any μ₁ of interest. Swapping out the claims of interest (in significant and insignificant results), gives us a single criterion of a good test, severity.

Exhibit(v): Severity Curves.

The severity tribes want to present severity using a standard Normal example, one where ${\sigma }_{\overline{X}}=1$ (as in the water plant accident). For this illustration:

$\text{Test T}+:H_0:\mu \le 0\text{vs}.H_1:\mu >0,\sigma =10,n=100,\sigma /\surd n={\sigma }_{\overline{X}}=1.$

If α = 0.025, we reject H₀ iff d(X) ≥ c_0.025 =1.96.

Suppose test T+ yields the statistically insignificant result d(x₀) = 1.5. Under the alternative d(X) is N(δ, 1) where $\delta =(\mu -{\mu }_0)/{\sigma }_{\overline{X}}$.

Even without identifying a discrepancy of importance ahead of time, the severity associated with various inferences can be evaluated.

The severity curves (Figure 5.6) show d(x₀) = 0.5, 1, 1.5, and 1.96.

$\text{How severely does}\mu \le 0.5\text{pass the test with}\overline{X}=1.5(d({\mathit{x}}_0)=1.5)?$

The easiest way to compute it is to go back to the observed ${\overline{x}}_0$, which would be 1.5.

$\text{SEV}(\mu \le 0.5)=\text{Pr}(\overline{X}>1.5;\mu =0.5)=\mathrm{0.16.}$

Here, Z = [(1.5 − 0.5)/1] = 1, and the area under the standard Normal distribution to the right of 1 is 0.16. Lousy. We can read it off the curve, looking at where the d(x) = 1.5 curve hits the bottom-most dotted line. The severity (vertical) axis hits 0.16, and the corresponding value on the μ axis is 0.5. This could be used more generally as a discrepancy axis, as I’ll show.

Figure 5.6 Severity curves.

We can find some discrepancy from the null that this statistically insignificant result warrants ruling out at a reasonable level – one that very probably would have produced a more significant result than was observed. The value d(x₀) = 1.5 yields a severity of 0.84 for a discrepancy of 2.5. SEV(μ ≤ 2.5) = 0.84 with d(x₀) = 1.5. Compare this to d(x) = 1.95, failing to trigger the significance alarm. Now a larger upper bound is needed for severity 0.84, namely, μ ≤ 2.95. If we have discrepancies of interest, by setting a high power to detect them, we ensure – ahead of time – that any insignificant result entitles us to infer “it’s not that high.” Power against μ₁ evaluates the worst (i.e., lowest) severity for values μ ≤ μ₁ for any outcome that leads to non-rejection. This test ensures any insignificant result entitles us to infer μ ≤ 2.95, call it μ ≤ 3. But we can determine discrepancies that pass with severity, post-data, without setting them at the outset. Compare four different outcomes:

d(x₀) = 0.5, SEV(μ ≤ 1.5) = 0.84; d(x₀) = 1, SEV(μ ≤ 2) = 0.84;

d(x₀) = 1.5, SEV(μ ≤ 2.5) = 0.84; d(x₀) = 1.95, SEV(μ ≤ 2.95) = 0.84.

In relation to test T+ (standard Normal): If you add 1 ${\sigma }_{\overline{X}}$ to d(x₀), the result being μ₁, then SEV(μ ≤ μ₁) = 0.84.Footnote ⁸

We can also use severity curves to compare the severity for a given claim, say μ ≤ 1.5:

d(x₀) = 0.5, SEV(μ ≤ 1.5) = 0.84; d(x₀) = 1, SEV(μ ≤ 1.5) = 0.7;

d(x₀) = 1.5, SEV(μ ≤ 1.5) = 0.5; d(x₀) = 1.95, SEV(μ ≤ 1.5) = 0.3.

Low and high benchmarks convey what is and is not licensed, and suffice for avoiding fallacies of acceptance. We can deduce SIN from the case where T+ has led to a statistically significant result, SIR. In that case, the inference that passes the test is of form μ > μ₁, where μ₁ = μ₀ + γ. Because (μ > μ₁) and (μ ≤ μ₁) partition the parameter space of μ, we get SEV(μ > μ₁) = 1 − SEV(μ ≤ μ₁).

The more devoted amongst you will want to improve and generalize my severity curves. Some of you are staying the night at Confidence Court Inn, others at Best Bet and Breakfast. We meet at the shared lounge, Calibation. Here’s a souvenir of SIR, and SIN.

Souvenir X: Power and Severity Analysis

Let’s record some highlights from Tour I:

First, ordinary power analysis versus severity analysis for Test T+:

It can happen that claim µ ≤ μ₁ is warranted by severity analysis but not by power analysis.

Now an overview of severity for test T+: Normal testing: H₀: μ ≤ μ₀ vs. H₁: μ > μ₀ with σ known. The severity reinterpretation is set out using discrepancy parameter γ. We often use μ₁ where μ₁ = μ₀ + γ.

The severe tester reports the attained significance levels and at least two other benchmarks: claims warranted with severity, and ones that are poorly warranted.

Talking through SIN and SIR. Let d₀ = d(x₀).