In the previous chapter, we discussed the L-96 model and showed the skill of two ensemble square-root filters, ETKF and EAKF, on filtering this model in various turbulent regimes. As we discussed in Chapter 1, with the ensemble approach, there is an inherently difficult practical issue of small ensemble size in filtering statistical solutions of these complex problems due to the large computational overload in generating individual ensemble members through the forward dynamical operator (Haven et al., 2005). Furthermore, as we have seen in Chapter 11, the ensemble square-root filters (ETKF, EAKF) on the L-96 model suffer from severe catastrophic filter divergence, where solutions diverge beyond machine infinity in finite time in many chaotic regimes, when the observations are partially available. For the two-layer QG model, we also found that extensive calibration is needed on the EAKF with the local least-squares framework to avoid catastrophic filter divergence (see chapter 11 and Harlim and Majda (2010b)); yet, the filtered solutions are not accurate in the numerically stiff “oceanic” regime. Naturally one would ask whether there is any skillful reduced filtering strategy that can overcome these challenges of computational overhead.

In this chapter, we discuss a radical approach for filtering nonlinear systems which has several desirable features including high computational efficiency and robust skill under variation of parameters. In particular, we implement an analogue of the Fourier diagonal filter as developed in Chapters 6–8 on the nonlinear L-96 model discussed in Chapter 11 with varying degrees of nonlinearity in the true dynamics ranging from weakly chaotic to fully turbulent.

The difficulties in filtering turbulent complex systems are largely due to our incomplete understanding of the dynamical system that underlies the observed signals, which have many spatio-temporal scales and rough turbulent energy spectra near the resolved mesh scale. In this chapter, we develop theoretical criteria as guidelines to address issues for filtering turbulent signals in an idealized context. In particular, we consider the simplest turbulent model discussed in Chapter 5 with plentiful observations, that is, the observations are available at every model grid point.

In this idealized context, we will provide a useful insight into answering several practical issues, including:

• As the model resolution is increased, there is typically a large computational overhead in propagating the dynamical operator and this restricts the predictions to relatively small ensemble sizes. When is it possible to filter using standard explicit and implicit solvers for the original dynamic equations by using a large time step equal to the observation time (even violating the CFL stability condition with standard explicit schemes) to increase ensemble size, yet still retain statistical accuracy?

• If plentiful observations are available on refined meshes, what is gained by increasing the resolution of the operational model? How does this depend on the nature of the turbulent spectrum?

As discussed in the introductory chapter, the scientific issue in real-time prediction problems is to provide a statistical estimate of a true state given that the nature of the physical process is chaotic and given the fact that the measurements (observations) are inaccurate or sometimes even unavailable. Essentially, at every time when observations become available, one chooses the best estimate of the true state by accounting for the prior forecasts and these observations. There are two challenges for improving the real-time prediction of a turbulent signal with multiple scales: the first is to improve the model which suffers from model error since we don't yet understand the underlying physical processes. Even if we do, we cannot realize these processes at every temporal or spatial scale with our limited computing power. Thus, the second challenge is to provide efficient and accurate strategies that meet this practical constraint.

In this chapter, we derive the one-dimensional Kalman filter formula which is a specific analytical solution of the Bayesian update in a simplified setting. As mentioned in the introductory chapter, this is an important test problem for filtering multi-scale turbulent systems and this point of view is emphasized here. We show the numerical results of filtering a one-dimensional complex Ornstein–Uhlenbeck process. We then discuss the conditions for filtering stability and compute the asymptotic off-line variables in closed form for the one-dimensional Kalman filter.

This book is an outgrowth of lectures by both authors in the graduate course of the first author at the Courant Institute during spring 2008 and 2010 on the topic of filtering turbulent dynamical systems as well as lectures by the second author at the North Carolina State University in a graduate course in fall 2009. The material is based on the authors' joint research as well as collaborations with Marcus Grote and Boris Gershgorin; the authors thank these colleagues for their explicit and implicit contributions to this material. Chapter 1 presents a detailed overview and summary of the viewpoint and material in the book. This book is designed for applied mathematicians, scientists and engineers, ranging from first- and second-year graduate students to senior researchers interested in filtering large-dimensional complex nonlinear systems.

The first author acknowledges the generous support of DARPA through Ben Mann and ONR through Reza Malek-Madani which funded the research on these topics and helped make this book a reality.

A major difficulty in accurate filtering of noisy turbulent signals with many degrees of freedom is model error; the true signal from nature is processed for filtering and prediction through an imperfect model in which numerical approximations are imposed and important physical processes are parametrized due to inadequate resolution or incomplete physical understanding. In Chapters 2 and 6, we discussed the filtering performance on a canonical linear stochastic PDE model for turbulence in the presence of model error through discrete numerical solvers. Subsequently, in Chapter 7 we assessed the filtering performance with model error that arises from reduced filtering strategies that are used when the observations are sparse.

In this chapter, we discuss filtering with model error which arises from unstable turbulent processes that are often observed in nature such as the intermittent burst of baroclinic instability in the mid-latitude atmosphere and ocean turbulent dynamics and which might be hidden from the forecast model. In the study below, we will simulate an intermittent burst of instability with two-state Markov damping coefficients; we allow the damping coefficients of some Fourier modes in the turbulent model (SPDE in Chapter 5) to be negative (or unstable) on some random occasions with a positive long-time average damping (i.e. an overall stable dynamics). We will prescribe the switching times between the two (stable and unstable) states to be randomly drawn from exponential distributions.

In this chapter, we review the L-96 model (Lorenz, 1996) and its dynamical properties. This model is a 40-dimensional nonlinear chaotic dynamical system mimicking the large-scale behavior of the mid-latitude atmosphere. This model has a wide variety of different chaotic regimes as the external forcing varies, ranging from weakly chaotic to strongly chaotic to fully turbulent. We check the performance of the ensemble square-root filters described in Chapter 9 on filtering this model for various chaotic regimes, observation times, observation errors for plentiful and regularly spaced sparse observations, and we discuss the phenomenon of “catastrophic filter divergence” for the sparsely observed L-96 model. As a more realistic example of geophysical turbulence, we also review the two-layer quasi-geostrophic (QG) model (Salmon, 1998; Vallis, 2006) that mimics the waves and turbulence of both the mid-latitude atmosphere and the ocean in suitable parameter regimes. We present numerical results of filtering true signals from the QG model with the ensemble adjustment Kalman filter (EAKF), implemented sequentially with the local least-squares framework (Anderson, 2003). In Chapters 12 and 13, we will revisit these models and study much cheaper filters with high skill based on suitable linear stochastic models (Chapter 12) combined with the stochastic parametrized extended Kalman filter (SPEKF) in Chapter 13.

The L-96 model

The L-96 model was introduced by Lorenz (1996) to represent an “atmospheric variable” u at 2N equally spaced points around a circle of constant latitude; thus, it is natural for u to be solved with a periodic boundary condition.

Many contemporary problems in science and engineering involve large-dimensional turbulent nonlinear systems with multiple time-scales, i.e. slow-fast systems. Here we introduce a prototype nonlinear slow-fast system with exactly solvable first- and second-order statistics which despite its simplicity, mimics crucial features of realistic vastly more complex physical systems. The exactly solvable features facilitate new nonlinear filtering algorithms (see Chapter 13 for another use in stochastic parametrizations) and allow for unambiguous comparison of the true signal with an approximate filtering strategy to understand model error. First we present an overview of the models, the main issues in filtering slow–fast systems, and filter performance. In the second part of this chapter we give more pedagogical details.

The nonlinear test model for filtering slow–fast systems with strong fast forcing: An overview

The dynamic models for the coupled atmosphere–ocean system are prototype examples of slow–fast systems where the slow modes are advective vortical modes and the fast modes are gravity waves (Salmon, 1998; Embid and Majda, 1998; Majda, 2003). Depending on the spatio-temporal scale, one might need only a statistical estimate through filtering of the slow modes, as in synoptic scales in the atmosphere (Daley, 1991) or both slow and fast modes such as for squall lines on mesoscales due to the impact of moist convection (Khouider and Majda, 2006).