This chapter demonstrates the use of optimization, namely the 3DVAR and 4DVAR methodologies, to obtain information from the filtering and smoothing distributions. We emphasize that the methods we present in this chapter do not provide approximations of the filtering and smoothing distributions; they simply provide estimates of the signal, given data, in the filtering (on-line) and smoothing (off-line) data scenarios.

This chapter is devoted to the particle filter, a method that approximates the filtering distribution by a sum of Dirac masses. Particle filters provably converge to the filtering distribution as the number of particles, and hence the number of Dirac masses, approaches infinity. We focus on the bootstrap particle filter (BPF), also known as sequential importance resampling; it is linked to the material on Monte Carlo and importance sampling described in Chapter 5.

In this chapter we study Markov chain Monte Carlo (MCMC), a methodology that delivers approximate samples from a given target distribution π. The methodology applies to settings in which π is the posterior distribution in (1.2), but it is also widely used in numerous applications beyond Bayesian inference. As with Monte Carlo and importance sampling, MCMC may be viewed as approximating the target distribution by a sum of Dirac masses, thus allowing the approximation of expectations with respect to the target. Implementation of Monte Carlo presupposes that independent samples from the target can be obtained. Importance sampling and MCMC bypass this restrictive assumption: importance sampling by appropriately weighting independent samples from a proposal distribution, and MCMC by drawing correlated samples from a Markov kernel that has the target as invariant distribution.

In this chapter we again adopt an optimization approach to the problem of Bayesian inference, but instead seek a Gaussian distribution 𝑝 = N(μ, Σ) that minimizes some distance-like measure from the posterior 𝜋𝑦 (u). However, rather than using a metric to define the distance, we use the Kullback–Leibler divergence introduced in Section 4.1.

This chapter demonstrates the use of optimization, namely the 3DVAR and 4DVAR methodologies, to obtain information from the filtering and smoothing distributions. We emphasize that the methods we present in this chapter do not provide approximations of the filtering and smoothing distributions; they simply provide estimates of the signal, given data, in the filtering (on-line) and smoothing (off-line) data scenarios.

This chapter brings together the material in the first two parts of these notes, demonstrating how the principles and ideas underpinning the derivation of extended and ensemble Kalman filters for data assimilation can be used to design ensemble Kalman methods for inverse problems.

Chapter 5 starts out with a physics motivation, as well as a mathematical statement of the problems that will be tackled in later sections. Several methods are introduced to solve a single nonlinear equation in one variable: fixed-point iteration, the bisection method, Newton’s method, the secant method, and Ridders’ method. After providing some advice about advantages and disadvantages of each approach, the text then studies how to find zeros of polynomials, employing two different techniques. The sophistication is then increased, by tackling systems of nonlinear equations and examining the corresponding challenges; in addition to Newton’s method, the text derives the equations behind Broyden’s method. A related subject is then broached, minimization in one or several dimensions; this includes the gradient-descent method, as well as detailed analysis of critical points; the second edition includes extensive new material on derivative-free optimization (golden-section search and Powell’s method).The chapter is rounded out by a physics project, the extremization of the action in classical mechanics, and a problem set. The physics project shows Hamilton’s principle in... action, translated into a multidimensional minimization problem.

Chapter 7 starts out with a physics motivation, as well as a mathematical statement of the problems that will be tackled in later sections. Newton-Cotes integration methods are first studied ad hoc, via Taylor expansions and, second, building on the interpolation machinery of the previous chapter. Standard techniques like the trapezoid rule and Simpson’s rule are introduced, including the Euler-Maclaurin summation formula. The error behavior is employed to produce an adaptive-integration routine and also, separately, to introduce the topic of Romberg integration. The theme of integration from interpolation continues, when Gauss-Legendre quadrature is explicitly derived, including the integration abscissas, weights, and error behavior. Emphasis is placed on analytic manipulations that can help the numerical evaluation of integrals. The chapter then turns to Monte Carlo, namely stochastic integration: this is painstakingly introduced for one-dimensional problems, and then generalized to the real-world problem of multidimensional integration. The chapter is rounded out by a physics project, on variational Monte Carlo for many-particle quantum mechanics, and a problem set.

Chapter 3 starts out with a physics motivation, as well as a mathematical statement of the problem that will be tackled in later sections. After a brief discussion of analytical differentiation, the bulk of the chapter is devoted to increasingly better finite-difference approximations, like the forward difference and the central difference. These are explicitly derived using Taylor expansions, and also applied to second derivatives and to points on a grid. A section introduces the useful tool of Richardson extrapolation, which reappears in later chapters. The chapter also includes an original discussion of automatic differentiation, which is built up from the concept of dual numbers. The chapter is rounded out by a physics project, which studies the kinetic energy in single-particle quantum mechanics, and a problem set. The physics project involves different wave functions and provides the groundwork for the project in the integrals chapter.

Chapter 2 starts out with a physics motivation, chosen to be the experimental study of the photoelectric effect by Millikan. It then discusses the concepts of absolute and relative error, as well as eror propagation in general. It then goes on to discuss the representation of real numbers on the computer, touching on topics like overflow, machine precision, and catastrophic cancellation. It then addresses a variety of problems where rounding error becomes significant: compensated summation, analytic manipulation, Taylor series, and recursion. The chapter is rounded out by a physics project, which studies the multipole expansion in electromagnetism, and a problem set. The physics project provides an opportunity to introduce Legendre polynomials, which play an important role in the following chapters.

Chapter 8 starts out with a physics motivation, as well as a mathematical statement of the problems that will be tackled in later sections. Starting from differential-equation initial-value problems, the text introduces both explicit and implicit methods, like backward Euler and the fourth-order Runge-Kutta method. Emphasis is placed on the interplay between method stability and problem conditioning (stiffness). The chapter then discusses boundary-value problems, first, via a combination of the earlier machinery on initial-value problems along with root-finding techniques and, second, via a finite-difference/matrix approach, which converts the problem to a linear system of equations. Next, the chapter tackles eigenvalue problems, again, via either rootfinding plus earlier tools, or a finite-difference approach; this time, the latter turns into a matrix eigenvalue problem. The second edition discussesfinite-difference approaches to solving the diffusion equation. The chapter is rounded out by a physics project, on Poisson’s equation in two dimensions, and a problem set. The physics project introduces and uses the two-dimensional fast Fourier transform, as part of a spectral method applied to the solution of a partial differential equation.

Chapter 6 starts out with a physics motivation, as well as a mathematical statement of the problems that will be tackled in later sections. First, polynomial interpolation is carried out using both the monomial basis and the Lagrange-interpolation formalism, sped up via the barycentric formula. This includes a derivation of the error and an emphasis on using unequally spaced points (Chebyshev nodes). Second, cubic-spline interpolation is introduced. Third, a section is dedicated to trigonometric interpolation, carefully working through the conventions and formalism needed to implement one of the most successful algorithms ever, the fast Fourier transform (FFT). Fourth, the topic of linear least-squares fitting is tackled, including the general formalism of the normal equations. The second edition includes a substantive new section on statistical inference, covering both frequentist and Bayesian approaches to linear regression. Nonlinear least-squares fitting is covered next, including the Gauss-Newton method and artificial neural networks. The chapter is rounded out by a physics project, on the experimental verification of the Stefan-Boltzmann law, and a problem set. In addition to providing a historical background on black-body radiation, the physics project shows an example of nonlinear least-squares fitting.

Chapter 4 starts out with a physics motivation, as well as a mathematical statement of the problems that will be tackled in later sections. An extensive section discusses the conditioning of linear-algebra problems: borrowing ideas and examples from matrix perturbation theory, this teaches students what to look out for. Roughly half of the remaining chapter is dedicated to the solution of linear systems of equations, employing methods of varying sophistication: Gaussian elimination, LU decomposiion, pivoting, and the Jacobi iterative method. The second half addresses the eigenvalue problem, again with a variety of methods, including the power method, eigenvalue shifting, and the QR method. Crucially, this includes explicit mathematical derivations of these approaches. A brief introduction to the singular value decomposition is also given, including both an existence proof and a programming implementation. The chapter is rounded out by an extensive physics project, which studies the eigenvalue problem of interacting spins, and a problem set. The physics project patiently builds up matrix quantum mechanics, allowing students to tackle problems of increasing difficulty.