Convolutional neural networks (CNNs) are prevalent in computer vision, image, speech, and language processing applications, where they have been successfully applied to perform classification tasks at high accuracy rates. One of their main attractions is the ability to operate directly on raw input signals, such as images, and to extract salient features automatically from the raw data. The designer does not need to worry about which features to select to drive the classification process.

When the training data $\{\text{γ}\left(n\right),h_n\}$ is linearly separable, there will exist many separating hyperplanes that can discriminate the data into two classes. Some of the techniques we described in the previous chapters, such as logistic regression and perceptron, are able to find such separating hyperplanes.

We develop a sequential version of the importance sampling technique from Chapter 33 in order to respond to streaming data, thus leading to a sequential Monte Carlo solution. The algorithm will lead to the important class of particle filters. This chapter presents the basic data model and the main construction that enables recursive inference. Many of the inference and learning methods in subsequent chapters will possess a recursive structure, which is a fundamental property to enable them to continually learn in response to the arrival of sequential data measurements. Particle filters are particularly well suited for scenarios involving nonlinear models and non-Gaussian signals, and they have found applications in a wide range of areas where these two features (nonlinearity and non-Gaussianity) are prevalent, including in guidance and control, robot localization, visual tracking of objects, and finance.

The optimal Bayes classifier (52.8) requires knowledge of the conditional probability distribution $\mathbb{P}\left(\mathit{r}=r|\mathit{h}=h\right)$, which is generally unavailable. In this and the next few chapters, we describe data‐based generative methods that approximate the joint probability distribution $f_{\mathit{r},\mathit{h}}\left(r,h\right)$, or its components $\mathbb{P}\left(r=r\right)$ and $f_{h|r}\left(h|r\right)$, directly from the data.

We described several data-based methods for inference and learning in the previous chapters. These methods operate directly on the data to arrive at classification or inference decisions. One key challenge these methods face is that the available training data need not provide sufficient representation for the sample space.

In this chapter we describe scattering representations, a signal representation built using wavelet multiscale decompositions with a deep convolutional architecture. Its construction highlights the fundamental role of geometric stability in deep learning representations, and provides a mathematical basis to study CNNs. We describe its main mathematical properties, its applications to computer vision, speech recognition and physical sciences, as well as its extensions to Lie Groups and non-Euclidean domains. Finally, we discuss recent applications to modeling high-dimensional probability densities.

We indicated in the concluding remarks of the previous chapter that feedforward neural networks have powerful modeling capabilities, as reflected by the universal approximation theorem. In one of its versions, the theorem asserts that networks with a single hidden layer are rich enough to model almost any arbitrary function.

We encountered one instance of Bayesian inference in Chapter 50, based on the quadratic loss in the context of mean-square-error (MSE) estimation. We explained there that the optimal solution for inferring a hidden zero-mean random variable $x$ from observations of another zero-mean random variable $y$ is given by the conditional estimator, $\mathbb{E}\left(x|y\right)$, whose computation requires knowledge of the conditional distribution, $f_{x|y}\left(x|y\right)$.

In supervised methods, learning is attained by training on a sufficient amount of labeled data in order to deliver reliable levels of classification. However, there are important situations in practice where data is scarce because it is either difficult or expensive to collect. This scenario leads to few-shot learning, where it is desired to train a classifier by using only a few training samples for each class.