In this chapter, we present the notions of Markov decision problem, the T-stage evaluation and the discounted evaluation. We introduce and study contracting mappings, and use such mappings to show that the decision maker has a stationary discounted optimal strategy. We also define the concept of uniform optimality, and show that the decision maker has a stationary uniformly optimal strategy.

In Chapter~\ref{section:blackwell epsilon}, we studied uniform $\ep$-optimality in Markov decision problems.

In this chapter, we define the analogous concept for two-player zero-sum stochastic games and for multiplayer stochastic games.

In this chapter, we define infinite orbits and show how to approximate such orbits. We prove that for every function that has no fixed point there is an approximate infinite orbit with unbounded variation, and we use this result to show that a certain class of quitting games admits undiscounted $\ep$-equilibria.

In this chapter, we define the model of stochastic games.

In this chapter, we extend the notion of discounted payoff to the model of stochastic games, and we define the concept of discounted equilibrium. We then prove that every two-player zero-sum stochastic game admits a discounted value, and that each player has a stationary discounted optimal strategy. The proof uses the same tools we employed in Chapter~\ref{section:mdp} to prove that in Markov decision problems the decision maker has a stationary discounted optimal strategy.

We finally prove that the discounted value is continuous in the parameters of the game, namely the payoff function, the transition function, and the discount factor.

In this chapter, we proveKakutani's Fixed Point Theorem, which is an extension of Brouwer's Fixed Point Theoremto correspondences (set-valued mappings). We then define the concept of $\lambda$-discounted equilibrium, and using Kakutani's Fixed Point Theorem we prove that every multiplayer stochastic game admits a stationary $\lambda$-discounted equilibrium, for every discount factor $\lambda \in (0,1]$.

This chapter describes methods based on gradient information that achieve faster rates than basic algorithms such as those described in Chapter 3. These accelerated gradient methods, most notably the heavy-ball method and Nesterov’s optimal method, use the concept of momentum which means that each step combines information from recent gradient values but also earlier steps. These methods are described and analyzed using an analysis based on Lyapunov functions. The cases of convex and strongly convex functions are analyzed separately. We motivate these methods using continuous-time limits, which link gradient methods to dynamical systems described by differential equations. We mention also the conjugate gradient method, which was developed separately from the other method but which also makes use of momentum. Finally, we discuss the concept of lower bounds on algorithmic complexity, introducing a function on which no method based on gradients can attain convergence faster than a certain given rate.

Here, we describe methods for minimizing a smooth function over a closed convex set, using gradient information. We first state results that characterize optimality of points in a way that can be checked, and describe the vital operation of projection onto the feasible set. We next describe the projected gradient algorithm, which is in a sense the extension of the steepest-descent method to the constrained case, analyze its convergence, and describe several extensions. We next analyze the conditional-gradient method (also known as “Frank-Wolfe”) for the case in which the feasible set is compact and demonstrate sublinear convergence of this approach when the objective function is convex.

Here, we discuss concepts of duality for convex optimization problems, and algorithms that make use of these concepts. We define the Lagrangian function and its augmented Lagrangian counterpart. We use the Lagrangian to derive optimality conditions for constrained optimization problems in which the constraints are expressed as linear algebraic conditions. We introduce the dual problem, and discuss the concepts of weak and strong duality, and show the existence of positive duality gaps in certain settings. Next, we discuss the dual subgradient method, the augmented Lagrangian method, and the alternating direction method of multipliers (ADMM), which are useful for several types of data science problems.

In this introductory chapter, we outline the ways in which various problems in data analysis can be formulated as optimization problems. Specifically, we discuss least squares problems, problems in matrix optimization (particularly those involving low-rank matrices), linear and kernel support vector machines, binary and multiclass logistic regression, and deep learning. We also outline the scope of the remainder of the book.

We describe the stochastic gradient method, the fundamental algorithm for several important problems in data science, including deep learning. We give several example problems for which this method is suitable, then described its operation for the simple problem of computing a mean of a collection of values. We related it to a classical method, the Kaczmarz method for solving a system of linear equalities and inequalities. Next, we describe the key assumptions to be used in convergence analysis, then describe the convergence rates attainable by several variants of stochastic gradient under several scenarios. Finally, we discuss several aspects of practical implementation of stochastic gradient, including minibatching and acceleration.