Introduction

This chapter serves as an appetizer to the main course, maximum likelihood and nonlinear least squares. This is stated so boldly because many statistical problems of this type originate in estimation problems with maximum likelihood (or a similar criterion) as the goal. Our discussion begins with some of the background calculus and definitions. Next, the discussion turns to the safe and slow methods for optimization in a single variable, for which the statistical term “nonparametric” has the correct connotations. Next, the root-finding problem is addressed with the standard techniques, Newton and secant methods, followed by a brief presentation of convergence rates. After a short digression on stopping and condition, the multivariate problem is first approached with Newton's methods. After a second digression on numerical differentiation, quasi-Newton methods are discussed for optimization and nonlinear equations. Discussions of condition, scaling, and implementation conclude the chapter.

Some topics are not addressed in this discussion. One problem is the solution of polynomial equations, which arise rarely in an isolated form in statistics. Constrained optimization can often be avoided through reparameterization. The specialized problem of nonlinear regression is postponed until the next chapter, to be treated as a special topic in maximum likelihood.

Before attacking the problems at hand, it is wise to review some foundations to gain a clearer perspective of the situation. The cornerstone for everything are the first results of calculus, the primary tools in applied mathematics. These results will first be stated in their univariate form.

Introduction

In this chapter, we apply the theory of Chapter 2 to sinusoidal models with fixed frequencies. In Section 3.2, the likelihood function under Gaussian noise assumptions is derived, for both white and coloured noise cases, and the relationships between the resulting maximum likelihood estimators and local maximisers of the periodogram is explored. The problem of estimating the fundamental frequency of a periodic signal in additive noise is also discussed. The asymptotic properties of these estimators are derived in Section 3.3. The results of a number of simulations are then used to judge the accuracy of the asymptotic theory in ‘small samples’.

The exact CRB for the single sinusoid case is computed in Section 3.4 and this is used in Section 3.5 to obtain accurate asymptotic theory for two special cases. In the first case, we assume that there are two sinusoids, with frequencies very close together. In fact, we assume that they are so close together that we expect sidelobe interference, and that the periodogram will not resolve the frequencies accurately. Although the difference between the frequencies is taken to be of the form, where T is the sample size, we show that the maximum likelihood estimators of the two frequencies still have the usual orders of accuracy.

Introduction

The wavelet analysis of a time series can be defined in terms of an orthonormal transform, sohere we briefly review the key ideas behind such transforms. We first review the basic theory fororthonormal transforms in Section 3.1. Section 3.2 discusses the important projection theorem, while3.3 considers complex-valued transforms. Prior to introducing the discrete wavelet transform (DWT)in Chapter 4, we discuss the orthonormal discrete Fourier transform (ODFT) in Section 3.4 because itparallels and contrasts the DWT in a number of interesting ways. We summarize the key points of thischapter in Section 3.5 - readers who are already comfortable with orthonormal transforms can readthis section simply to become familiar with our notation and conventions.

Basic Theory for Orthonormal Transforms

Orthonormal transforms are of interest because they can be used to re-express a time series insuch a way that we can easily reconstruct the series from its transform. In a loose sense, the‘information’ in the transform is thus equivalent to the ‘information’in the original series; to put it another way, the series and its transform can be considered to betwo representations of the same mathematical entity. Orthonormal transforms can be used tore-express a series in a standardized form (e.g., a Fourier series) for further manipulation, toreduce a series to a few values summarizing its salient features (compression), and to analyze aseries to search for particular patterns of interest (e.g., analysis of variance).

Introduction

In subsequent chapters we will make substantial use of some basic results from the Fourier theoryof sequences and – to a lesser extent – functions, and we will find that filters playa central role in the application of wavelets. This chapter is intended as a self-contained guide tosome key results from Fourier and filtering theory. Our selection of material is intentionallylimited to just what we will use later on. For a more thorough discussion employing the samenotation and conventions adopted here, see Percival and Walden (1993). We also recommend Briggs andHenson (1995) and Hamming (1989) as complementary sources for further study.

Readers who have extensive experience with Fourier analysis and filters can just quickly scanthis chapter to become familiar with our notation and conventions. We encourage others to study thematerial carefully and to work through as many of the embedded exercises as possible (answers areprovided in the appendix). It is particularly important that readers understand the concept ofperiodized filters presented in Section 2.6 since we use this idea repeatedly in Chapters 4 and5.

Complex Variables and Complex Exponentials

The most elegant version of Fourier theory for sequences and functions involves the use ofcomplex variables, so here we review a few key concepts regarding them (see, for example, Brown andChurchill, 1995, for a thorough treatment). Let i ≡ √–1 sothat i2 = –1 (throughout the book, we take‘≡’ to mean ‘equal by definition’).

Introduction

As discussed in Chapter 4, the discrete wavelet transform (DWT) allows us to analyze (decompose) a time series X into DWT coefficients W, from which we can then synthesize (reconstruct) our original series. We have already noted that the synthesis phase can be used, for example, to construct a multiresolution analysis of a time series (see Equation (64) or (104a)) and to simulate long memory processes (see Section 9.2). In this chapter we study another important use for the synthesis phase that provides an answer to the signal estimation (or function estimation, or denoising) problem, in which we want to estimate a signal hidden by noise within an observed time series. The basic idea here is to modify the elements of W to produce, say, W′, from which an estimate of the signal can be synthesized. With the exception of methods briefly discussed in Section 10.8, once certain parameters have been estimated, the elements Wn of W are treated one at a time; i.e., how we modify Wn is not directly influenced by the remaining DWT coefficients. The wavelet-based techniques that we concentrate on here are thus conceptually very simple, yet they are remarkably adaptive to a wide variety of signals.

Introduction

Wavelets are mathematical tools for analyzing time series or images (although not exclusively so:for examples of usage in other applications, see Stollnitz et al., 1996, andSweldens, 1996). Our discussion of wavelets in this book focuses on their use with time series,which we take to be any sequence of observations associated with an ordered independent variablet (the variable t can assume either a discrete set of values suchas the integers or a continuum of values such as the entire real axis - examples of both typesinclude time, depth or distance along a line, so a time series need not actually involve time).Wavelets are a relatively new way of analyzing time series in that the formal subject dates back tothe 1980s, but in many aspects wavelets are a synthesis of older ideas with new elegant mathematicalresults and efficient computational algorithms. Wavelet analysis is in some cases complementary toexisting analysis techniques (e.g., correlation and spectral analysis) and in other cases capable ofsolving problems for which little progress had been made prior to the introduction of wavelets.

Broadly speaking (and with apologies for the play on words!), there have been two main waves ofwavelets. The first wave resulted in what is known as the continuous wavelet transform (CWT), whichis designed to work with time series defined over the entire real axis; the second, in the discretewavelet transform (DWT), which deals with series defined essentially over a range of integers(usually t = 0, 1,…,N – 1, where Ndenotes the number of values in the time series). In this chapter we introduce and motivate waveletsvia the CWT.

Introduction

Here we introduce the discrete wavelet transform (DWT), which is the basic tool needed forstudying time series via wavelets and plays a role analogous to that of the discrete Fouriertransform in spectral analysis. We assume only that the reader is familiar with the basic ideas fromlinear filtering theory and linear algebra presented in Chapters 2 and 3. Our exposition buildsslowly upon these ideas and hence is more detailed than necessary for readers with strongbackgrounds in these areas. We encourage such readers just to use the Key Facts and Definitions ineach section or to skip directly to Section 4.12 – this has a concise self-containeddevelopment of the DWT. For complementary introductions to the DWT, see Strang (1989, 1993), Riouland Vetterli (1991), Press et al. (1992) and Mulcahy (1996).

The remainder of this chapter is organized as follows. Section 4.1 gives a qualitativedescription of the DWT using primarily the Haar and D(4) wavelets as examples. The formalmathematical development of the DWT begins in Section 4.2, which defines the wavelet filter anddiscusses some basic conditions that a filter must satisfy to qualify as a wavelet filter. Section4.3 presents the scaling filter, which is constructed in a simple manner from the wavelet filter.The wavelet and scaling filters are used in parallel to define the pyramid algorithm for computing(and precisely defining) the DWT – various aspects of this algorithm are presented inSections 4.4, 4.5 and 4.6.

The last decade has seen an explosion of interest in wavelets, a subject area that has coalescedfrom roots in mathematics, physics, electrical engineering and other disciplines. As a result,wavelet methodology has had a significant impact in areas as diverse as differential equations,image processing and statistics. This book is an introduction to wavelets and their application inthe analysis of discrete time series typical of those acquired in the physical sciences. While wepresent a thorough introduction to the basic theory behind the discrete wavelet transform (DWT), ourgoal is to bridge the gap between theory and practice by

1. • emphasizing what the DWT actually means in practical terms;

2. • showing how the DWT can be used to create informative descriptive statistics fortime series analysts;

3. • discussing how stochastic models can be used to assess the statisticalproperties of quantities computed from the DWT; and

4. • presenting substantive examples of wavelet analysis of time seriesrepresentative of those encountered in the physical sciences.

To date, most books on wavelets describe them in terms of continuous functions and oftenintroduce the reader to a plethora of different types of wavelets. We concentrate on developingwavelet methods in discrete time via standard filtering and matrix transformation ideas.