Introduction

Many statistical applications involve significance tests to assess the plausibility of scientific hypotheses. Resampling methods are not new to significance testing, since randomization tests and permutation tests have long been used to provide nonparametric tests. Also Monte Carlo tests, which use simulated datasets, are quite commonly used in certain areas of application. In this chapter we describe how resampling methods can be used to produce significance tests, in both parametric and nonparametric settings. The range of ideas is somewhat wider than the direct bootstrap approach introduced in the preceding two chapters. To begin with, we summarize some of the key ideas of significance testing.

The simplest situation involves a simple null hypothesis H0 which completely specifies the probability distribution of the data. Thus, if we are dealing with a single sample y1, …, yn from a population with CDF F, then H0 specifies that F = F0, where F0 contains no unknown parameters. An example would be “exponential with mean 1”. The more usual situation in practice is that H0 is a composite null hypothesis, which means that some aspects of F are not determined and remain unknown when H0 is true. An example would be “normal with mean 1”, the variance of the normal distribution being unspecified.

P-values

A statistical test is based on a test statistic T which measures the discrepancy between the data and the null hypothesis.