The correlation coefficient measures the linear relation between scalar X and scalar Y. How can the linear relation between vector X and vector Y be measured?Canonical Correlation Analysis (CCA) provides a way. CCA finds a linear combination of X, and a (separate) linear combination of Y, that maximizes the correlation. The resulting maximized correlation is called a canonical correlation. More generally, CCA decomposes two sets of variables into an ordered sequence of component pairs ordered such that the first pair has maximum correlation, the second has maximum correlation subject to being uncorrelated with the first, and so on. The entire decomposition can be derived from a Singular Value Decomposition of a suitable matrix. If the dimension of the X and Y vectors is too large, overfitting becomes a problem. In this case, CCA often is computed using a few principal components of X and Y. The criterion for selecting the number of principal components is not standard. The Mutual Information Criterion (MIC) introduced in Chapter 14 is used in this chapter.

The previous chapter discussed data assimilation for the case in which the variables have known Gaussian distributions. However, in atmospheric and oceanic data assimilation, the distributions are neither Gaussian nor known, and the large number of state variables creates numerical challenges. This chapter discusses a class of algorithms, called Ensemble Square Root Filters, for performing data assimilation with high-dimensional, nonlinear systems. The basic idea is to use a collection of forecasts (called an ensemble) to estimate the statistics of the background distribution. In addition, observational information is incorporated by adjusting individual ensemble members (i.e., forecasts) rather than computing an entire distribution. This chapter discusses three standard filters: the Ensemble Transform Kalman Filter (ETKF), the Ensemble Square Root Filter (EnSRF), and the Ensemble Adjustment Kalman Filter (EAKF). However, ensemble filters often experience filter divergence, in which the analysis no longer tracks the truth. This chapter discusses standard approaches to mitigating filter divergence, namely covariance inflation and covariance localization.

Some variables can be modeled by a linear combination of other random variables, plus random noise. Such models are used to quantify the relation between variables, to make predictions, and to test hypotheses about the relation between variables. After identifying the variables to include in a model, the next step is to estimate the coefficients that multiply them, called the regression parameters. This chapter discusses the least squares method for estimating regression parameters. The least squares method estimates the parameters by minimizing the sum of squared differences between the fitted model and the data. This chapter also describes measures for the goodness of fit and an illuminating geometric interpretation of least squares fitting. The least squares method is illustrated on various routine calculations in weather and climate analysis (e.g., fitting a trend). Procedures for testing hypotheses about linear models are discussed in the next chapter.

Multivariate linear regression is a method for modeling linear relations between two random vectors, say X and Y. Common reasons for using multivariate regression include (1) to predicting Y given X, (2) to testing hypotheses about the relation between X and Y, and (3) to projecting Y onto prescribed time series or spatial patterns. Special cases of multivariate regression models include Linear Inverse Models (LIMs) and Vector Autoregressive Models. Multivariate regression also is fundamental to other statistical techniques, including canonical correlation analysis, discriminant analysis, and predictable component analysis. This chapter introduces multivariate linear regression and discusses estimation, measures of association, hypothesis testing, and model selection. In climate studies, model selection often involves selecting Y as well as X. For instance, Y may be a set of principal components that need to be chosen, which is not a standard selection problem. This chapter introduces a criterion for selecting X and Y simultaneously called Mutual Information Criterion (MIC).

The method of least squares will fit any model to a data set, but is the resulting model "good"?One criterion is that the model should fit the data significantly better than a simpler model with fewer predictors. After all, if the fit is not significantly better, then the model with fewer predictors is almost as good. For linear models, this approach is equivalent to testing if selected regression parameters vanish. This chapter discusses procedures for testing such hypotheses. In interpreting such hypotheses, it is important to recognize that a regression parameter for a given predictor quantifies the expected rate of change of the predict and while holding the other predictors constant. Equivalently, the regression parameter quantifies the dependence between two variables after controlling or regressing out other predictors. These concepts are important for identifying a confounding variable, which is a third variable that influences two variables to produce a correlation between those two variables. This chapter also discusses how detection and attribution of climate change can be framed in a regression model framework.

Large data sets are difficult to grasp. To make progress, we often seek a few quantities that capture as much of the information in the data as possible. In this chapter, we discuss a procedure called Principal Component Analysis (PCA), also called Empirical Orthogonal Function (EOF) analysis, which finds the components that minimizes the sum square difference between the components and the data. The components are ordered such that the first approximates the data the best (in a least squares sense), the second approximates the data the best among all components orthogonal to the first, and so on. In typical climate applications, a principal component consists of two parts: (1) a fixed spatial structure, called an Empirical Orthogonal Function (EOF), and (2) its time-dependent amplitude, called a PC time series. The EOFs are orthogonal and the PC time series are uncorrelated. Principal components often are used as input to other analyses, such as linear regression, canonical correlation analysis, predictable components analysis, or discriminant analysis. The procedure for performing area-weighted PCA is discussed in detail in this chapter.

This chapter introduces the power spectrum. The power spectrum is the Fourier Transform of the autocovariance function, and the autocovariance function is the (inverse) Fourier Transform of the power spectrum. As such, the power spectrum and autocovariance function offer two complementary but mathematically equivalent descriptions of a stochastic process. The power spectrum quantifies how variance is distributed over frequencies and is useful for identifying periodic behavior in time series. The discrete Fourier transform of a time series can be summarized in a periodogram, which provides a starting point for estimating power spectra. Estimation of the power spectrum can be counterintuitive because the uncertainty in periodogram elements does not decrease with increasing sample size. To reduce uncertainty, periodogram estimates are averaged over a frequency interval called the bandwidth. Trends and discontinuities in time series can lead to similar low-frequency structure despite very different temporal characteristics. Spectral analysis provides a particularly insightful way to understand the behavior of linear filters.