The logic underlying the statistical tests described in this book is simple. A statistical test produces a test statistic of which the distribution is known. What we want to know is whether the test statistic has a value that is extreme, so extreme that it is unlikely to be attributable to chance. In the traditional terminology, we pit a null-hypothesis, actually a straw man, that the test statistic does not have an extreme value, against an alternative hypothesis according to which its value is indeed extreme. Whether a test statistic has an extreme value is evaluated by calculating how far out it is in one of the tails of the distribution. Functions like pt (), pf (), and pchisq () tell us how far out we are in a tail by means of p-values, which assess what proportion of the population has even more extreme values. The smaller this proportion is, the more reason we have for surprise that our test statistic is as extreme as it actually is.

However, the fuzzy notion of what counts as extreme needs to be made more precise. It is generally assumed that a probability begins to count as extreme by the time it drops below 0.05. However, opinions differ with respect to how significance should be assessed.

One tradition holds that the researcher should begin by defining what counts as extreme, before gathering and analyzing data.