Parts II and III of the book discussed dynamic programming algorithms for solving MDPs and POMDPs. Dynamic programming assumes that the MDP or POMDP model is completely specified. This chapter presents simulation-based stochastic approximation algorithms for estimating the optimal policy of MDPs when the transition probabilities are not known. Simulation-based means that although the transition probabilities are unknown, the decision-maker can observe the system trajectory under any choice of control actions. The simulation-based algorithms given in this chapter also apply as suboptimal methods for solving POMDPs.

The following algorithms are presented in this chapter:

1. The Q-learning algorithm is described in §16.1. It uses the Robbins–Monro algorithm (described in Chapter 15) to estimate the value function for an unconstrained MDP. It is also shown how a primal-dual Q-learning algorithm can be used for MDPs with monotone optimal policies. The Q-learning algorithm also applies as a suboptimal method for POMDPs.

2. Policy gradient algorithms are presented in §16.2 to §16.7. These use gradient estimation (described in Chapter 15) of the cost function together with a stochastic gradient algorithm to estimate the optimal policy. The policy gradient algorithms apply to MDPs and constrained MDPs. They also yield a suboptimal policy search method for POMDPs.

Some terminology: Determining the optimal policy of an MDP or POMDP when the parameters are not known falls under the class of stochastic adaptive control problems. Stochastic adaptive control algorithms are of two types: direct methods, where the unknown transition probabilities Pij(u) are estimated simultaneously while updating the control policy, and implicit methods (such as simulation-based methods), where the transition probabilities are not directly estimated in order to compute the control policy. In this chapter, we focus on implicit simulation-based algorithms for solving MDPs and POMDPs. These are also called reinforcement learning algorithms. One motivation for such implicit methods is that, since they are simulation based, only regions of the state space visited by the simulated sample path are used to determine the controller. Effort is not wasted on determining parameters for low probability regions which are rarely or never visited.

Q-learning algorithm

The Q-learning algorithm is a widely used reinforcement learning algorithm. As described below, the Q-learning algorithm is the simply the Robbins–Monro stochastic approximation algorithm (15.5) of Chapter 15 applied to estimate the value function of Bellman's dynamic programming equation.