In the feedforward networks and convolutional neural networks (CNNs) studied in the previous chapters, the training data was assumed to be static, with no sequential relation among the samples. Using the data, we were able to train the networks to perform reliable classification tasks. There are many applications, however, where the input data will be sequential in nature, with one sample following another in some ordered manner, as happens with words in a sentence.

The material in the last three chapters focused on the use of neural network structures for the solution of inference (regression and classification) problems. In this chapter, we use the same networks to develop two generative methods whose purpose is to generate samples from the same underlying distribution as the training data.

We studied in Chapters 29 and 30 the mean‐square error (MSE) criterion in some detail, and applied it to the problem of inferring an unknown (or hidden) variable $x$ from the observation of another variable $x$ when $\left\{x,y\right\}$ are related by means of a linear regression model or a state‐space model.

The mean-square-error (MSE) criterion (27.17) is one notable example of the Bayesian approach to statistical inference. In the Bayesian approach, both the unknown quantity, $\mathit{x}$, and the observation, $\mathit{y}$, are treated as random variables and an estimator $\widehat{\mathit{x}}$ for $\mathit{x}$ is sought by minimizing the expected value of some loss function denoted by $Q\left(\mathit{x},\widehat{\mathit{x}}\right)$. In the previous chapter, we focused exclusively on the quadratic loss $Q\left(\mathit{x},\widehat{\mathit{x}}\right)={(\mathit{x}-\widehat{\mathit{x}})}^2$ for scalar $\mathit{x}$. In this chapter, we consider more general loss functions, which will lead to other types of inference solutions such as the mean-absolute error (MAE) and the maximum a-posteriori (MAP) estimators. We will also derive the famed Bayes classifier as a special case when the realizations for $\mathit{x}$ are limited to the discrete values $\mathit{x}\in \left\{\pm 1\right\}$.

In this chapter, we give a comprehensive overview of the large variety of approximation results for neural networks. Approximation rates for classical function spaces as well as the benefits of deep neural networks over shallow ones for specifically structured function classes are discussed. While the main body of existing results is for general feedforward architectures, we also review approximation results for convolutional, residual and recurrent neural networks.

The chapter discusses how to process data from irregular discrete domains, an emerging area called graph signal processing (GSP). Basically, the type of graph we deal with consists of a network with distributed vertices and weighted edges defining the neighborhood and the connections among the nodes. As such, the graph signals are collected in a vector whose entries represent the values of the signal nodes at a given time. A common issue related to GSP is the sampling problem, given the irregular structure of the data, where some sort of interpolation is possible whenever the graph signals are bandlimited or nearly bandlimited. These interpolations can be performed through the extension of the conventional adaptive filtering to signals distributed on graphs where there is no traditional data structure. The chapter presents the LMS, NLMS, and RLS algorithms for GSP along with their analyses and application to estimate bandlimited signals defined on graphs. In addition, the chapter presents a general framework for data-selective adaptive algorithms for GSP.

The chapter briefly introduces the main concepts of array signal processing, emphasizing those related to adaptive beamforming, and discusses how to impose linear constraints to adaptive filtering algorithms to achieve the beamforming effect. Adaptive beamforming, emphasizing the incoming signal impinging from a known direction by means of an adaptive filter, is the primary objective of the array signal processing addressed in this chapter. We start this study with the narrowband beamformer. The constrained LMS, RLS, conjugate gradient, and SMAP algorithms are introduced along with the generalized sidelobe canceller, and the Householder constrained structures; sparse promoting adaptive beamforming algorithms are also addressed in this chapter. In the following, it introduces the concepts of frequency-domain and time-domain broadband adaptive beamforming and shows their equivalence. The chapter wraps up with brief discussions and reference suggestions on essential topics related to adaptive beamforming, including the numerical robustness of adaptive beamforming algorithms.

This chapter explains the basic concepts of kernel-based methods, a widely used tool in machine learning. The idea is to present online parameter estimation of nonlinear models using kernel-based tools. The chapters aim is to introduce the kernel version of classical algorithms such as least mean square (LMS), recursive least squares (RLS), affine projection (AP), and set membership affine projection (SM-AP). In particular, we will discuss how to keep the dictionary of the kernel finite through a series of model reduction strategies. This way, all discussed kernel algorithms are tailored for online implementation.

It provides a brief description of the classical adaptive filtering algorithms, starting with defining the actual objective function each algorithm minimizes. It also includes a summary of the expected performance according to available results from the literature.