Filtering is the process of obtaining the best statistical estimate of a natural system from partial observations of the true signal from nature. In many contemporary applications in science and engineering, real-time filtering of a turbulent signal from nature involving many degrees of freedom is needed to make accurate predictions of the future state. This is obviously a problem with significant practical impact. Important contemporary examples involve the real-time filtering and prediction of weather and climate as well as the spread of hazardous plumes or pollutants. Thus, an important emerging scientific issue is the real-time filtering through observations of noisy signals for turbulent nonlinear dynamical systems as well as the statistical accuracy of spatio-temporal discretizations for filtering such systems. From the practical standpoint, the demand for operationally practical filtering methods escalates as the model resolution is significantly increased. In the coupled atmosphere–ocean system, the current practical models for prediction of both weather and climate involve general circulation models where the physical equations for these extremely complex flows are discretized in space and time and the effects of unresolved processes are parametrized according to various recipes; the result of this process involves a model for the prediction of weather and climate from partial observations of an extremely unstable, chaotic dynamical system with several billion degrees of freedom. These problems typically have many spatio-temporal scales, rough turbulent energy spectra in the solutions near the mesh scale, and a very large-dimensional state space, yet real-time predictions are needed.

In Chapters 9 and 11 we discussed finite ensemble filtering of nonlinear dynamical systems with Kalman-based (or linear) techniques and we demonstrated their advantages in the less turbulent regime and their limitation in the sparsely observed fully turbulent regime. We also showed their sensitivity toward variations of filtering parameters such as the variance inflation and ensemble size. Subsequently in Chapters 12–14, we discussed alternative strategies based on cheap reduced stochastic models to avoid all the disadvantages of ensemble Kalman filtering strategies and demonstrated their skill for filtering various turbulent nonlinear systems ranging from the idealized toy models like L-96 to quasi-geostrophic turbulence to turbulent diffusion models with non-Gaussian statistics.

Recently, there is an emerging need to develop nonlinear filtering methods beyond the ensemble Kalman filters since most physical problems are nonlinear and sometimes have highly non-Gaussian distributions. One of the most difficult problems is to devise a practically useful ensemble method based on the theoretically well-established particle filtering (or sequential Monte Carlo) methods (Del Moral, 1996; Del Moral and Jacod, 2001; Doucet et al., 2008). The major challenge is to have a particle filter that is practically skillful for high-dimensional complex turbulent systems with only small ensemble size (small number of particles); this is very important because of the large overload in generating individual ensemble members through the forward dynamical operator (Haven et al., 2005).

Throughout the book, we have stressed that a central issue in practical filtering of turbulent signals is model error. Naively, one might think that model errors always have a negative effect on filter performance and indeed this was illustrated in Chapters 2 and 3 in simple examples with simple time differencing methods like backward or forward Euler with associated natural time discrete noise. However, a central issue of this book is to emphasize that judicious model errors in the forward operator, guided by mathematical theory, can both ameliorate the effect of the curse of ensemble size for turbulent dynamical systems and retain high filtering skill, remarkably, often exceeding that with the perfect model! In particular, we have illustrated these principles with various model errors arising from using approximate numerical solvers (Chapter 2), reduced strategies for filtering dynamical systems with instability (Chapters 3 and 8), reduced strategies for filtering sparsely observed signals (Chapter 7) and simple linear stochastic models for filtering turbulent signals from nonlinear dynamical systems (Chapters 5, 10, and 12). In our earlier discussion (see Chapters 8, 10, and 12), we demonstrated that the off-line strategy accounting for model errors, the mean stochastic model (MSM), under some circumstances produces reasonably accurate filtered solutions. However, this off-line strategy often has limited skill for estimating real-time prediction problems with rapid fluctuations that are often observed in nature since the off-line strategy (MSM) relies heavily on a fixed parameter set that is extracted from long-time or climatological statistical quantities such as the energy spectrum and correlation time.

As motivated in Chapter 1, one goal of the present book is to develop an explicit off-line test criterion for stable accurate time filtering of turbulent signals which is akin to the classical frozen linear constant stability test for finite difference schemes for systems of nonlinear partial differential equations as presented in chapter 4 of Richtmeyer and Morton (1967). In applications for complex turbulent spatially extended systems, the actual dynamics is typically turbulent and energetic at the smallest mesh scales but the climatological spectrum of the turbulent modes is known; for example, a mesh truncation of the compressible primitive equations with a fine mesh spacing of 10–50 kilometers still has substantial random and chaotic energy on the smallest 10-kilometer scales due to chaotic motion of clouds, topography and boundary layer turbulence which are not resolved. Similar unresolved features occur in many engineering problems with turbulence. Thus, the first step is the development of an appropriate constant-coefficient stochastic PDE test problem.

The simplest models for representing turbulent fluctuations involve replacing nonlinear interaction by additional linear damping and stochastic white noise forcing in time which incorporates the observed climatological spectrum and decorrelation time for the turbulent field (Majda et al., 2005; Majda and Wang, 2006). Thus, the first step in developing analogous off-line test criteria is to utilize the above approximations. This approach is developed in this chapter and builds on earlier material from Section 2.1.1 and Chapter 4.

Turbulent diffusion is a physical process that describes the transport of a tracer in a turbulent velocity field. Very often the tracer itself has very little or no influence on the background flow, in which case it is referred to as a passive tracer. Practically important examples include engineering problems such as the spread of hazardous plumes or pollutants in the atmosphere and contaminants in the ocean. Another class of problems that involve turbulent diffusion are climate science problems concerning the transport of greenhouse gases such as carbon dioxide and others. One of the characteristics of these systems is their complex multi-scale structure in both time and space. For example, the spatial scales of atmospheric flows span from planetary-scale Rossby waves to local weather patterns with the size of kilometers. Similarly, temporal scales involve both slow dynamics on the scales of decades as well as the fast dynamics on the scales of hours. Another remarkable property of many tracers in the atmosphere is their highly intermittent probability distributions with long exponential tails (Neelin et al., 2011). Many contemporary applications in science and engineering involve real-time filtering of such turbulent non-Gaussian signals from nature with multiple scales in time and space.

Real-time tracking of a chemical plume released into the atmosphere or a contaminant injected into the ocean is another extremely important and practical example where real-time data assimilation plays a crucial role.