We develop a sequential version of the importance sampling technique from Chapter 33 in order to respond to streaming data, thus leading to a sequential Monte Carlo solution. The algorithm will lead to the important class of particle filters. This chapter presents the basic data model and the main construction that enables recursive inference. Many of the inference and learning methods in subsequent chapters will possess a recursive structure, which is a fundamental property to enable them to continually learn in response to the arrival of sequential data measurements. Particle filters are particularly well suited for scenarios involving nonlinear models and non-Gaussian signals, and they have found applications in a wide range of areas where these two features (nonlinearity and non-Gaussianity) are prevalent, including in guidance and control, robot localization, visual tracking of objects, and finance.

The optimal Bayes classifier (52.8) requires knowledge of the conditional probability distribution $\mathbb{P}\left(\mathit{r}=r|\mathit{h}=h\right)$, which is generally unavailable. In this and the next few chapters, we describe data‐based generative methods that approximate the joint probability distribution $f_{\mathit{r},\mathit{h}}\left(r,h\right)$, or its components $\mathbb{P}\left(r=r\right)$ and $f_{h|r}\left(h|r\right)$, directly from the data.

We described several data-based methods for inference and learning in the previous chapters. These methods operate directly on the data to arrive at classification or inference decisions. One key challenge these methods face is that the available training data need not provide sufficient representation for the sample space.

We indicated in the concluding remarks of the previous chapter that feedforward neural networks have powerful modeling capabilities, as reflected by the universal approximation theorem. In one of its versions, the theorem asserts that networks with a single hidden layer are rich enough to model almost any arbitrary function.

We encountered one instance of Bayesian inference in Chapter 50, based on the quadratic loss in the context of mean-square-error (MSE) estimation. We explained there that the optimal solution for inferring a hidden zero-mean random variable $x$ from observations of another zero-mean random variable $y$ is given by the conditional estimator, $\mathbb{E}\left(x|y\right)$, whose computation requires knowledge of the conditional distribution, $f_{x|y}\left(x|y\right)$.

In supervised methods, learning is attained by training on a sufficient amount of labeled data in order to deliver reliable levels of classification. However, there are important situations in practice where data is scarce because it is either difficult or expensive to collect. This scenario leads to few-shot learning, where it is desired to train a classifier by using only a few training samples for each class.

We illustrated in Example 63.2 one limitation of linear separation surfaces by considering the XOR mapping (63.11). The example showed that certain feature spaces are not linearly separable and cannot be resolved by the perceptron algorithm. The result in the example was used to motivate one powerful approach to nonlinear separation surfaces by means of kernel methods.

In the immediate past chapters we developed several techniques for the design of linear classifiers, such as logistic regression, perceptron, and support vector machines (SVM). These algorithms are suitable for data that are linearly separable; otherwise, their performance degrades significantly. In this chapter we explain how the methods can be adjusted to determine nonlinear separation surfaces.

In most multistage decision problems, we are interested in determining the optimal strategy, ${\pi }^{\star }\left(a\right|s)$ (i.e., the optimal actions to follow in the state–action space). Most of the algorithms described in the previous chapters focused on evaluating the state and state–action value functions, ${\upsilon }^{\pi }\left(s\right)$ and $q^{\pi }(s,a)$, for a given policy $\pi \left(a\right|s)$. More is needed to learn the optimal policy.

We derived in the previous two chapters procedures for assessing the performance of strategies used by agents interacting with a Markov decision process (MDP), including obtaining optimal policies. Among other methods, we discussed the policy evaluation algorithm (44.116) and the value and policy iterations (45.23) and (45.43), respectively.

Principal component analysis (PCA) is a formidable tool for dimensionality reduction. Given feature vectors $\left\{h_n\right\}$ in $M$‐dimensional space, PCA replaces them by lower‐dimensional vectors $\left\{h_n^{\prime }\right\}$ of size $M^{\prime }\ll M$ each.