Skip to main content
    • Aa
    • Aa
  • Probability in the Engineering and Informational Sciences, Volume 25, Issue 3
  • July 2011, pp. 307-342


  • Dinard van der Laan (a1)
  • DOI:
  • Published online: 17 May 2011

In this article we study Markov decision process (MDP) problems with the restriction that at decision epochs, only a finite number of given Markov decision rules are admissible. For example, the set of admissible Markov decision rules could consist of some easy-implementable decision rules. Additionally, many open-loop control problems can be modeled as an MDP with such a restriction on the admissible decision rules. Within the class of available policies, optimal policies are generally nonstationary and it is difficult to prove that some policy is optimal. We give an example with two admissible decision rules—={d1, d2} —for which we conjecture that the nonstationary periodic Markov policy determined by its period cycle (d1, d1, d2, d1, d2, d1, d2, d1, d2) is optimal. This conjecture is supported by results that we obtain on the structure of optimal Markov policies in general. We also present some numerical results that give additional confirmation for the conjecture for the particular example we consider.

Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

2.E. Altman , B. Gaujal & A. Hordijk (2000). Balanced sequences and optimal routing. Journal of the ACM 47: 752775.

3.E. Altman , B. Gaujal & A. Hordijk (2000). Multimodularity, convexity and optimization properties. Mathematics of Operations Research 25: 324347.

5.E. Altman & A. Shwartz (1991). Markov decision problems and state-action frequencies. SIAM Journal on Control and Optimization 29: 786809.

7.X.R. Cao (1998). The MacLaurin series for performance functions of Markov chains. Advances in Applied Probability 30: 676692.

8.E. Fernández-Gaucherand , A. Araposthathis & S.I. Marcus (1991). On the average cost optimality equation and the structure of optimal policies for partially observable Markov decision processes. Annals of Operations Research 29: 439470.

9.E. Fernández-Gaucherand , A. Araposthathis & S.I. Marcus (1991). Remarks on the existence of solutions to the average cost optimality equation in Markov decision processes. Systems and Control Letters 15: 425432.

11.B. Hajek (1985). Extremal splittings of point processes. Mathematics of Operations Research 10(4): 543556.

12.B. Heidergott & A. Hordijk (2003). Taylor series expansions for stationary Markov chains. Advances in Applied Probability 35: 10461070.

13.B. Heidergott & F. Vázquez-Abad (2008). Measure valued differentiation for Markov chains. Journal of Optimization and Applications 136: 187209.

14.B. Heidergott , F.J. Vázquez-Abad , G. Pflug & T. Farenhorst-Yuan (2010). Gradient estimation for discrete-event systems by measure-valued differentiation. ACM Transactions on Modeling and Computer Simulation (TOMACS) 20: 128.

15.O. Hernández-Lerma & J.B. Lasserre (1996). Discrete-time Markov control processes: Basic optimality criteria. New York: Springer.

16.A. Hordijk & D.A. van der Laan (2005). On the average waiting time for regular routing to deterministic queues. Mathematics of Operations Research 30: 521544.

17.G. Koole (1999). On the static assignment to parallel servers. IEEE Transactions on Automatic Control 44: 15881592.

20.M. Morse & G.A. Hedlund (1940). Symbolic dynamics II — sturmian trajectories. American Journal of Mathematics 62: 142.

21.G.C. Pflug (1996). Optimization of stochastic models. Amsterdam: Kluwer Academic.

22.M. Puterman (1994). Markov decision processes: Discrete stochastic dynamic programming. New York: John Wiley and Sons.

23.K.W. Ross (1989). Randomized and past-dependent policies for Markov decision processes with multiple constraints. Operations Research 37: 474477.

25.R. Tijdeman (2000). Fraenkel's conjecture for six sequences. Discrete Mathematics 222: 223234.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Probability in the Engineering and Informational Sciences
  • ISSN: 0269-9648
  • EISSN: 1469-8951
  • URL: /core/journals/probability-in-the-engineering-and-informational-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *