Mathematical background and formulation of numerical minimization process are described in terms of gradient-based methods, whose ingredients include gradient, Hessian, directional derivatives, optimality conditions for minimization, Hessian eigensystem, conjugate number of Hessian, and conjugate vectors. Various minimization algorithms, such as the steepest descent method, Newton’s method, conjugate gradient method, and quasi-Newton’s method, are introduced along with practical examples.

The estimation task is classified as filtering, smoothing, and prediction, depending on when the estimation and the observation incorporation are made. Basic techniques of filtering and smoothing are introduced. Characteristics and formulations of various filters and smoothers are discussed, including the Kalman filter, extended Kalman filter, fixed-point smoother, fixed-lag smoother, and fixed-interval smoother. Bayesian perspectives of filtering and smoothing are also discussed, especially on joint smoother and marginal smoother.

A standard probability formalism is introduced, including a definition of the probability density function (PDF) and its first four moments. Most basic PDFs such as the uniform and Gaussian PDFs, are defined. The fundamentals of the Bayes’ formula derivation and its formulation in terms of PDFs are also presented. More importantly, data assimilation is described as a recursive Bayes’ formula, which connects the standard Bayes’ formula from different analysis times by using transition PDFs. A basic introduction to Shannon information theory is presented, followed by a definition of uncertainty in terms of entropy, and therefore establishing a mathematical basis for interpreting data assimilation in terms of information processing that is used throughout this book. The multivariate Gaussian data assimilation framework, most often used in practice, is described. Common analysis solutions that include maximum a posteriori and minimum variance methods are derived, which include a formulation of the cost function and posterior probability.

Probabilistic prediction in terms of the probability density function and using Kolmogorov equation is introduced. The error of probabilistic prediction is defined, and its growth further analyzed using normed measures. Error growth is also connected with the Lyapunov exponent to underline its relevance to chaotic dynamics. Furthermore, the forecast and analysis errors of recursive data assimilation are mathematically related to the Lyapunov exponent, with implications for the control of errors of dynamical imbalances. It is shown how errors propagate in a data assimilation system and that the control of unbalanced errors is critical for successful data assimilation. In addition, Bayesian inference was identified as a mechanism that can help in implicitly controlling the growth of errors. A practical approach of dealing with dynamical imbalances in data assimilation using penalty function is also presented and briefly discussed.

The role of forecast error covariance in practical ensemble and variational data assimilation is described following algebraic and dynamical views. This is used to introduce a motivation for ensemble data assimilation. It is shown how a dynamically induced and anisotropic ensemble error covariance can benefit data assimilation, compared to climatological (static) and isotropic error covariance used in variational methods. In addition to the standard ensemble Kalman filter (EnKF), more practical square root EnKF equations are also presented. Direct transform ensemble methods are also introduced and their connection with both ensemble and variational methods described. Error covariance localization in terms of the Schur product, a standard component of any realistic ensemble-based data assimilation, is also introduced and discussed. Following that, hybrid data assimilation and in particular the ensemble-variational (EnVar) methods are introduced and presented in relation to pure ensemble and variational methods. As a particular example of hybrid methods the maximum likelihood ensemble filter (MLEF) is introduced.

Various experiments for data assimilation are suggested using the codes of one-dimensional Burgers’ equation for practical experience.

Hands-on experiments of constructing tangent linear and adjoint model are given along with the practical generation of their codes by both hands-on work and an automatic differentiation tool (Tapenade).

This chapter focuses on assimilation of observations from satellites, which is a dominant source of observation information in weather and climate. This includes satellite radiances, both clear sky and all-sky. The most important challenges of all-sky radiances come from their connection to cloud microphysics, which potentially implies nonlinear, non-Gaussian, and nondifferentiable processes that are difficult for data assimilation. The complexity of error covariance with microphysical variables is illustrated in a few real-world examples. An additional difficulty with assimilating all-sky radiances comes from correlated observation errors that require special attention in data assimilation. Practical ways to deal with correlated observation errors are described. Nonlinearity and nondifferentiability of observation operators for all-sky radiances is also briefly explained. Since satellite radiance observations and observation operators generally contain bias, a common formulation of radiance bias correction methods is also presented. The observations from satellites also include radio occultation and lightning observations, as well as satellite products.

Generating the adjoint model (ADJM) by hand is tedious, time-consuming, and error prone. In most practical applications of data assimilation these days, the derivative codes, including the ADJM, are generated by the automatic differentiation (AD) tools, which evaluate the exact derivative information of a function in terms of a program. Terminologies and methods in AD are introduced, including the practical exclusion of the forward and reverse modes of differentiation. Various AD tools based on two major AD approaches, source transformation and operator overloading, are compiled with their webpages.