The $k$-nearest neighbor ($k$-NN) rule is appealing. However, each new feature $h\in {\mathbb{\text{R}}}^M$ requires searching over the entire training set of size $N$ to determine the neighborhood around $h$.

In this chapter, we describe three other data-based generative methods that approximate the solution to the optimal Bayes classifier (52.8) in the absence of knowledge of the conditional probabilities $\mathbb{P}\left(r=r|h=h\right)$. The methods estimate the prior probabilities $\mathbb{P}\left(r=r\right)$ for the classes and, in some cases, assume a Gaussian form for the reverse conditional distribution, $f_{h|r}\left(h|r\right)$. The training data is used to estimate the priors and the first-and second-order moments of $f_{h|r}\left(h|r\right)$.

This chapter provides theoreticalinsights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, non-robustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuousgeneralization guarantees for deep learning. On the basis of the theoreticalobservations, wepropose new open problems.

The discussion in the last two chapters focused on directed graphical models or Bayesian networks, where a directed link from a variable ${\mathit{x}}_1$ toward another variable ${\mathit{x}}_2$ carries with it an implicit connotation of “causal effect” by ${\mathit{x}}_1$ on ${\mathit{x}}_2$. In many instances, this implication need not be appropriate or can even be limiting. For example, there are cases where conditional independence relations cannot be represented by a directed graph. One such example is provided in Prob. 43.1. In this chapter, we examine another form of graphical representations where the links are not required to be directed anymore, and the probability distributions are replaced by potential functions. These are strictly positive functions defined over sets of connected nodes; they broaden the level of representation by graphical models. The potential functions carry with them a connotation of “similarity” or “affinity” among the variables, but can also be rolled back to represent probability distributions. Over undirected graphs, edges linking nodes will continue to reflect pairwise relationship between the variables but will lead to a fundamental factorization result in terms of the product of clique potential functions. We will show that these functions play a prominent role in the development of message-passing algorithms for the solution of inference problems.

The inference of a random variable $\mathit{x}$ from observations $\{{\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_N\}$ requires that we evaluate the posterior distribution $f_{\mathit{x}\mid {\mathit{y}}_{1:N}}(x\mid y_1,\dots ,y_N)$ as happens, for example, in inference formulations based on mean-square-error (MSE), maximum a-posteriori (MAP), or probability of error metrics. In previous chapters, we described several techniques to facilitate the computation or approximation of such posterior distributions using Monte Carlo or variational inference methods. We will encounter other types of approximations in later chapters. For example, in the context of naïve Bayes classifiers in Chapter 55, we will assume that, conditioned on the latent variable $\mathit{x}$, the observations are independent of each other in order to write

We give a short and concise review about the dynamical system and the control theory approach to deep learning. From the viewpoint of the dynamical systems, the back-propagation algorithm in deep learning becomes a simple consequence of the variational equations in ODEs. From the viewpoint of control theory, deep learning is a case of mean-field control in that all the agents share the same control. As an application, we discuss a new class of algorithms for deep learning based on Pontryagin’s maximum principle in control theory.

We continue our treatment of Markov decision processes (MDPs) and focus in this chapter on methods for determining optimal actions or policies. We derive two popular methods known as value iteration and policy iteration, and establish their convergence properties. We also examine the Bellman optimality principle in the context of value and policy learning. In a later section, we extend the discussion to the more challenging case of partially observable MDPs (POMDPs), where the successive states of the MDP are unobservable to the agent, and the agent is only able to sense measurements emitted randomly by the MDP from the various states. We will define POMDPs and explain that they can be reformulated as belief‐MDPs with continuous (rather than discrete) states. This fact complicates the solution of the value iteration. Nevertheless, we will show that the successive value iterates share a useful property, namely, that they are piecewise linear and convex. This property can be exploited by computational methods to reduce the complexity of solving the value iteration for POMDPs.

We take a look at the universal approximation question for stochastic feedforward neural networks. In contrast with deterministic networks, which represent mappings from inputs to outputs, stochastic networks represent mappings from inputs to probability distributions over outputs. Even if the sets of inputs and outputs are finite, the set of stochastic mappings is continuous. Moreover, the values of the output variables may be correlated, which requires that their values are computed jointly. A prominent class of stochastic feedforward networks are deep belief networks. We discuss the representational power in terms of compositions of Markov kernels expressed by the layers of the network. We investigate different types of shallow and deep architectures, and the minimal number of layers and units that are necessary and sufficient in order for the network to be able to approximate any stochastic mapping arbitrarily well. The discussion builds on notions of probability sharing, focusing on the case of binary variables and sigmoid units. After reviewing existing results, we present a detailed analysis of shallow networks and a unified analysis for a variety of deep networks.

In this chapter, we describe a popular discriminative approach for classification problems, known as logistic regression. Assuming binary classification with labels $\text{γ}\in \left\{\pm 1\right\}$ and features $h\in {\mathbb{\text{R}}}^M$, we explained earlier in expression (28.85) that the optimal Bayes classifier for predicting $\text{γ}$ is given by

We mentioned earlier in Section 52.3 that the nearest-neighbor (NN) rule for classification and clustering treats equally all attributes within each feature vector, $h_n\in {\mathbb{\text{R}}}^M$. If, for example, some attributes are more relevant to the classification task than other attributes, then this aspect is ignored by the NN classifier because all entries of the feature vector will contribute similarly to the calculation of Euclidean distances and the determination of neighborhoods.

Deep generative models have been recently proposed as modular datadriven priors to solve inverse problems. Linear inverse problems involve the reconstruction of an unknown signal (e.g. a tomographic image) from an underdetermined system of noisy linear measurements. Most results in the literature require that the reconstructed signal has some known structure, e.g. it is sparse in some known basis (usually Fourier or wavelet). Such prior assumptions can be replaced with pre-trained deep generative models (e.g. generative adversarial getworks (GANs) and variational autoencoders (VAEs)) with significant performance gains. This chapter surveys this rapidly evolving research area and includes empirical and theoretical results in compressed sensing for deep generative models.

We describe the new field of the mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail.