In late 1982, Ted Hannan discussed with me a question he had been asked by some astronomers – how could you estimate the two frequencies in two sinusoids when the frequencies were so close together that you could not tell, by looking at the periodogram, that there were two frequencies? He asked me if I would like to work with him on the problem and gave me a reprint of his paper (Hannan 1973) on the estimation of frequency. Together we wrote a paper (Hannan and Quinn 1989) which derived the regression sum of squares estimators of the frequencies, and showed that the estimators were strongly consistent and satisfied a central limit theorem. It was clear that there were no problems asymptotically if the two frequencies were fixed, so Ted's idea was to fix one frequency, and let the other converge to it at a certain rate, in much the same way as the alternative hypothesis is constructed to calculate the asymptotic power of a test. Since then, I have devoted much of my research to sinusoidal models. In particular, I have spent a lot of time constructing algorithms for the estimation of parameters in these models, to implementing the algorithms in practice and, for me perhaps the most challenging, establishing the asymptotic (large sample) properties of the estimators.

Periodic functions

We encounter periodic: phenomena every day of our lives. Those of us who still use analogue clocks are acutely aware of the 60 second, 60 minute and 12 hour periods associated with the sweeps of the second, minute and hour hands. We are conscious of the fact that the Earth rotates on its axis roughly every 24 hours and that it completes a revolution of the Sun roughly every 365 days. These periodicities are reasonably accurate. The quantities we are interested in measuring are not precisely periodic and there will also be error associated with their measurement. Indeed, some phenomena only seem periodic. For example, some biological population sizes appear to fluctuate regularly over a long period of time, but it is hard to justify using common sense any periodicity other than that associated with the annual cycle. It has been argued in the past that some cycles occur because of predator-prey interaction, while in other cases there is no obvious reason. On the other hand, the sound associated with musical instruments can reasonably be thought of as periodic, locally in time, since musical notes are produced by regular vibration and propagated through the air via the regular compression and expansion of the air. The ‘signal’ will not be exactly periodic, since there are errors associated with the production of the sound, with its transmission through the air (since the air is not a uniform medium) and because the ear is not a perfect receiver.

Introduction

We introduce in this chapter those statistical and probability techniques that underlie what is presented later. Few proofs will be given because a complete treatment of even a small part of what is dealt with here would require a book in itself. We do not intend to bother the reader with too formal a presentation. We shall be concerned with a sample space, Ω, which can be thought of as the set of all conceivable realisations of the random processes with which we are concerned. If A is a subset of Ω, then P(A) is the probability that the realisation is in A. Because we deal with discrete time series almost exclusively, questions of ‘measurability’, i.e. to which sets A can P(·) be applied, do not arise and will never be mentioned. We say this once and for all so that the text will not be filled with requirements that this or that set be measurable or that this or that function be a measurable function. Of course we shall see only (part of) one realisation, {x(t), t = 0, ±1, ±2,…} and are calling into being in our mind's eye, so to say, a whole family of such realisations. Thus we might write ω (t; ω) where ω ∈ Ω is the point corresponding to a particular realisation and, as ω varies for given t, we get a random variable, i.e. function defined on the sample space Ω.

Introduction

There are several types of frequency estimation techniques which we have not yet discussed. In particular, we have not paid any attention to those based on autocovariances, such as Pisarenko's technique (Pisarenko 1973), or those based on phase differences, for complex time series, such as two techniques due to Kay (1989). We have not spent much effort on these for the very reason that we have been concerned with asymptotic theory and asymptotic optimality. That is, for fixed system parameters, we have been interested in the behaviour of frequency estimators as the sample size T increases, with the hope that the sample size we have is large enough for the asymptotic theory to hold well enough. Moreover, we have not wished to impose conditions such as Gaussianity or whiteness on the noise process, as the latter in particular is rarely met in practice. Engineers, however, are often interested in the behaviour of estimators for fixed values of T, and decreasing SNR. The usual measure of this behaviour is mean square error, which may be estimated via simulations. Such properties, however, may rarely be justified theoretically, as there is no statistical limit theory which allows the mean square errors of nonlinear estimators to be calculated using what are essentially limiting distribution results. Although the methods mentioned above are computationally simple and computationally efficient, we shall see that they cannot be statistically asymptotically efficient and may even be inconsistent, i.e., actually converge to the wrong value.

Introduction

In this chapter, we apply the theory of Chapter 2 to sinusoidal models with fixed frequencies. In Section 3.2, the likelihood function under Gaussian noise assumptions is derived, for both white and coloured noise cases, and the relationships between the resulting maximum likelihood estimators and local maximisers of the periodogram is explored. The problem of estimating the fundamental frequency of a periodic signal in additive noise is also discussed. The asymptotic properties of these estimators are derived in Section 3.3. The results of a number of simulations are then used to judge the accuracy of the asymptotic theory in ‘small samples’.

The exact CRB for the single sinusoid case is computed in Section 3.4 and this is used in Section 3.5 to obtain accurate asymptotic theory for two special cases. In the first case, we assume that there are two sinusoids, with frequencies very close together. In fact, we assume that they are so close together that we expect sidelobe interference, and that the periodogram will not resolve the frequencies accurately. Although the difference between the frequencies is taken to be of the form, where T is the sample size, we show that the maximum likelihood estimators of the two frequencies still have the usual orders of accuracy.

Introduction

The wavelet analysis of a time series can be defined in terms of an orthonormal transform, sohere we briefly review the key ideas behind such transforms. We first review the basic theory fororthonormal transforms in Section 3.1. Section 3.2 discusses the important projection theorem, while3.3 considers complex-valued transforms. Prior to introducing the discrete wavelet transform (DWT)in Chapter 4, we discuss the orthonormal discrete Fourier transform (ODFT) in Section 3.4 because itparallels and contrasts the DWT in a number of interesting ways. We summarize the key points of thischapter in Section 3.5 - readers who are already comfortable with orthonormal transforms can readthis section simply to become familiar with our notation and conventions.

Basic Theory for Orthonormal Transforms

Orthonormal transforms are of interest because they can be used to re-express a time series insuch a way that we can easily reconstruct the series from its transform. In a loose sense, the‘information’ in the transform is thus equivalent to the ‘information’in the original series; to put it another way, the series and its transform can be considered to betwo representations of the same mathematical entity. Orthonormal transforms can be used tore-express a series in a standardized form (e.g., a Fourier series) for further manipulation, toreduce a series to a few values summarizing its salient features (compression), and to analyze aseries to search for particular patterns of interest (e.g., analysis of variance).

Introduction

In subsequent chapters we will make substantial use of some basic results from the Fourier theoryof sequences and – to a lesser extent – functions, and we will find that filters playa central role in the application of wavelets. This chapter is intended as a self-contained guide tosome key results from Fourier and filtering theory. Our selection of material is intentionallylimited to just what we will use later on. For a more thorough discussion employing the samenotation and conventions adopted here, see Percival and Walden (1993). We also recommend Briggs andHenson (1995) and Hamming (1989) as complementary sources for further study.

Readers who have extensive experience with Fourier analysis and filters can just quickly scanthis chapter to become familiar with our notation and conventions. We encourage others to study thematerial carefully and to work through as many of the embedded exercises as possible (answers areprovided in the appendix). It is particularly important that readers understand the concept ofperiodized filters presented in Section 2.6 since we use this idea repeatedly in Chapters 4 and5.

Complex Variables and Complex Exponentials

The most elegant version of Fourier theory for sequences and functions involves the use ofcomplex variables, so here we review a few key concepts regarding them (see, for example, Brown andChurchill, 1995, for a thorough treatment). Let i ≡ √–1 sothat i2 = –1 (throughout the book, we take‘≡’ to mean ‘equal by definition’).

Introduction

As discussed in Chapter 4, the discrete wavelet transform (DWT) allows us to analyze (decompose) a time series X into DWT coefficients W, from which we can then synthesize (reconstruct) our original series. We have already noted that the synthesis phase can be used, for example, to construct a multiresolution analysis of a time series (see Equation (64) or (104a)) and to simulate long memory processes (see Section 9.2). In this chapter we study another important use for the synthesis phase that provides an answer to the signal estimation (or function estimation, or denoising) problem, in which we want to estimate a signal hidden by noise within an observed time series. The basic idea here is to modify the elements of W to produce, say, W′, from which an estimate of the signal can be synthesized. With the exception of methods briefly discussed in Section 10.8, once certain parameters have been estimated, the elements Wn of W are treated one at a time; i.e., how we modify Wn is not directly influenced by the remaining DWT coefficients. The wavelet-based techniques that we concentrate on here are thus conceptually very simple, yet they are remarkably adaptive to a wide variety of signals.

Introduction

Wavelets are mathematical tools for analyzing time series or images (although not exclusively so:for examples of usage in other applications, see Stollnitz et al., 1996, andSweldens, 1996). Our discussion of wavelets in this book focuses on their use with time series,which we take to be any sequence of observations associated with an ordered independent variablet (the variable t can assume either a discrete set of values suchas the integers or a continuum of values such as the entire real axis - examples of both typesinclude time, depth or distance along a line, so a time series need not actually involve time).Wavelets are a relatively new way of analyzing time series in that the formal subject dates back tothe 1980s, but in many aspects wavelets are a synthesis of older ideas with new elegant mathematicalresults and efficient computational algorithms. Wavelet analysis is in some cases complementary toexisting analysis techniques (e.g., correlation and spectral analysis) and in other cases capable ofsolving problems for which little progress had been made prior to the introduction of wavelets.

Broadly speaking (and with apologies for the play on words!), there have been two main waves ofwavelets. The first wave resulted in what is known as the continuous wavelet transform (CWT), whichis designed to work with time series defined over the entire real axis; the second, in the discretewavelet transform (DWT), which deals with series defined essentially over a range of integers(usually t = 0, 1,…,N – 1, where Ndenotes the number of values in the time series). In this chapter we introduce and motivate waveletsvia the CWT.