In this chapter, we present linear complementarity problems, and use them to provide sufficient conditions that guarantee the existence of an undiscounted $\ep$-equilibrium in quitting games.

In this chapter, we present a technique to study uniform equilibria in stochastic games, called the \emph{vanishing discount factorapproach}.

This approach was developed to prove the existence of a uniform $\ep$-equilibrium in two-player nonzero-sum absorbing games using a function $\lambda \mapsto_\lambda$, which assigns a stationary $\lambda$-discounted equilibrium $x_\lambda$ to every\lambda \in (0,1]$, and analyzing the asymptotic properties of this function as $\lambda$ goes to 0.

We will use this approach to show that every absorbing game in which the probability of absorption is positive whatever the players play has a stationary uniform 0-equilibrium,and that every two-player absorbing game has a uniform $\ep$-equilibrium, which need not be stationary, for every $\ep > 0$.

To prove the second result, we will show how statistical tests are used in the construction of uniform $\ep$-equilibria.

In this chapter, we review material on strategic-form games thatwill be needed in the sequel. Readers who are interested in expanding their knowledge of strategic-form games are referred to Maschler, Solan, and Zamir (2013, Chapters 4 and 5).

In Section~\ref{continuity} we proved that the discounted value is continuous in the parameters of the game, see Theorem~\ref{theorem7}.

One weakness of this result is that it does not bound the Lipschitz constant of the value function $(\lambda,q,r) \mapsto v_\lambda(s;q,r)$.

In this chapter, we will strengthen Theorem~\ref{theorem7}, and, using the concept of $B$-graphs, develop a bound on the Lipschitz constant of the value function.

Our technique will allow us to study the continuityof the limit $\lim_{\lambda \to 0} v_\lambda(s;q,r)$ as a function of $q$ and $r$.

In this chapter, we prove a Tauberian Theorem regarding the relation between the Abel limit and the Ces`aro limit of a sequence of real numbers, and apply it to prove that a uniformly $\ep$-optimal strategy exists in Hidden Markov decision problems.

In this chapter, we prove Ramsey's Theorem, which states that for every coloring of the complete infinite graph by finitely many colors there is an infinite complete onochromatic subgraph.

We then define the notion of undiscounted $\ep$-equilibrium, and show that every two-player deterministic stopping game admits an undiscounted $\ep$-equilibrium.

In this chapter we define semi-algebraic sets and study their basic properties. We then apply our findings to prove that for every initial state s the limit $\lim_{\lambda \to 0} v_\lambda(s)$ exists.

In this chapter, we present the notions of Markov decision problem, the T-stage evaluation and the discounted evaluation. We introduce and study contracting mappings, and use such mappings to show that the decision maker has a stationary discounted optimal strategy. We also define the concept of uniform optimality, and show that the decision maker has a stationary uniformly optimal strategy.

In Chapter~\ref{section:blackwell epsilon}, we studied uniform $\ep$-optimality in Markov decision problems.

In this chapter, we define the analogous concept for two-player zero-sum stochastic games and for multiplayer stochastic games.

In this chapter, we define infinite orbits and show how to approximate such orbits. We prove that for every function that has no fixed point there is an approximate infinite orbit with unbounded variation, and we use this result to show that a certain class of quitting games admits undiscounted $\ep$-equilibria.

In this chapter, we define the model of stochastic games.

In this chapter, we extend the notion of discounted payoff to the model of stochastic games, and we define the concept of discounted equilibrium. We then prove that every two-player zero-sum stochastic game admits a discounted value, and that each player has a stationary discounted optimal strategy. The proof uses the same tools we employed in Chapter~\ref{section:mdp} to prove that in Markov decision problems the decision maker has a stationary discounted optimal strategy.

We finally prove that the discounted value is continuous in the parameters of the game, namely the payoff function, the transition function, and the discount factor.

In this chapter, we proveKakutani's Fixed Point Theorem, which is an extension of Brouwer's Fixed Point Theoremto correspondences (set-valued mappings). We then define the concept of $\lambda$-discounted equilibrium, and using Kakutani's Fixed Point Theorem we prove that every multiplayer stochastic game admits a stationary $\lambda$-discounted equilibrium, for every discount factor $\lambda \in (0,1]$.