Markov decision processes (MDPs) are at the core of reinforcement learning theory. Similar to Markov chains, MDPs involve an underlying Markovian process that evolves from one state to another, with the probability of visiting a new state being dependent on the most recent state. Different from Markov chains, MDPs involve both agents and actions taken by these agents. As a result, the next state is dependent on which action was chosen at the state preceding it. MDPs therefore provide a powerful framework to explore state spaces and to learn from actions and rewards.

In the feedforward networks and convolutional neural networks (CNNs) studied in the previous chapters, the training data was assumed to be static, with no sequential relation among the samples. Using the data, we were able to train the networks to perform reliable classification tasks. There are many applications, however, where the input data will be sequential in nature, with one sample following another in some ordered manner, as happens with words in a sentence.

The material in the last three chapters focused on the use of neural network structures for the solution of inference (regression and classification) problems. In this chapter, we use the same networks to develop two generative methods whose purpose is to generate samples from the same underlying distribution as the training data.

We studied in Chapters 29 and 30 the mean‐square error (MSE) criterion in some detail, and applied it to the problem of inferring an unknown (or hidden) variable $x$ from the observation of another variable $x$ when $\left\{x,y\right\}$ are related by means of a linear regression model or a state‐space model.

The mean-square-error (MSE) criterion (27.17) is one notable example of the Bayesian approach to statistical inference. In the Bayesian approach, both the unknown quantity, $\mathit{x}$, and the observation, $\mathit{y}$, are treated as random variables and an estimator $\widehat{\mathit{x}}$ for $\mathit{x}$ is sought by minimizing the expected value of some loss function denoted by $Q\left(\mathit{x},\widehat{\mathit{x}}\right)$. In the previous chapter, we focused exclusively on the quadratic loss $Q\left(\mathit{x},\widehat{\mathit{x}}\right)={(\mathit{x}-\widehat{\mathit{x}})}^2$ for scalar $\mathit{x}$. In this chapter, we consider more general loss functions, which will lead to other types of inference solutions such as the mean-absolute error (MAE) and the maximum a-posteriori (MAP) estimators. We will also derive the famed Bayes classifier as a special case when the realizations for $\mathit{x}$ are limited to the discrete values $\mathit{x}\in \left\{\pm 1\right\}$.

In this chapter, we give a comprehensive overview of the large variety of approximation results for neural networks. Approximation rates for classical function spaces as well as the benefits of deep neural networks over shallow ones for specifically structured function classes are discussed. While the main body of existing results is for general feedforward architectures, we also review approximation results for convolutional, residual and recurrent neural networks.

The chapter discusses how to process data from irregular discrete domains, an emerging area called graph signal processing (GSP). Basically, the type of graph we deal with consists of a network with distributed vertices and weighted edges defining the neighborhood and the connections among the nodes. As such, the graph signals are collected in a vector whose entries represent the values of the signal nodes at a given time. A common issue related to GSP is the sampling problem, given the irregular structure of the data, where some sort of interpolation is possible whenever the graph signals are bandlimited or nearly bandlimited. These interpolations can be performed through the extension of the conventional adaptive filtering to signals distributed on graphs where there is no traditional data structure. The chapter presents the LMS, NLMS, and RLS algorithms for GSP along with their analyses and application to estimate bandlimited signals defined on graphs. In addition, the chapter presents a general framework for data-selective adaptive algorithms for GSP.

The chapter briefly introduces the main concepts of array signal processing, emphasizing those related to adaptive beamforming, and discusses how to impose linear constraints to adaptive filtering algorithms to achieve the beamforming effect. Adaptive beamforming, emphasizing the incoming signal impinging from a known direction by means of an adaptive filter, is the primary objective of the array signal processing addressed in this chapter. We start this study with the narrowband beamformer. The constrained LMS, RLS, conjugate gradient, and SMAP algorithms are introduced along with the generalized sidelobe canceller, and the Householder constrained structures; sparse promoting adaptive beamforming algorithms are also addressed in this chapter. In the following, it introduces the concepts of frequency-domain and time-domain broadband adaptive beamforming and shows their equivalence. The chapter wraps up with brief discussions and reference suggestions on essential topics related to adaptive beamforming, including the numerical robustness of adaptive beamforming algorithms.