Skip to main content
    • Aa
    • Aa


  • Dinard van der Laan (a1)

In this article we study Markov decision process (MDP) problems with the restriction that at decision epochs, only a finite number of given Markov decision rules are admissible. For example, the set of admissible Markov decision rules could consist of some easy-implementable decision rules. Additionally, many open-loop control problems can be modeled as an MDP with such a restriction on the admissible decision rules. Within the class of available policies, optimal policies are generally nonstationary and it is difficult to prove that some policy is optimal. We give an example with two admissible decision rules—={d1, d2} —for which we conjecture that the nonstationary periodic Markov policy determined by its period cycle (d1, d1, d2, d1, d2, d1, d2, d1, d2) is optimal. This conjecture is supported by results that we obtain on the structure of optimal Markov policies in general. We also present some numerical results that give additional confirmation for the conjecture for the particular example we consider.

Hide All
1. B. Altman , E. Gaujal & A. Hordijk (2003). Discrete-event control of stochastic networks: Multimodularity and regularity. Lecture Notes in Mathematics. New York: Springer Verlag.

15. O. Hernández-Lerma & J.B. Lasserre (1996). Discrete-time Markov control processes: Basic optimality criteria. New York: Springer.

18. M. Lothaire (2002). Algebraic combinatorics on words. Cambridge: Cambridge University Press.

21. G.C. Pflug (1996). Optimization of stochastic models. Amsterdam: Kluwer Academic.

22. M. Puterman (1994). Markov decision processes: Discrete stochastic dynamic programming. New York: John Wiley and Sons.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Probability in the Engineering and Informational Sciences
  • ISSN: 0269-9648
  • EISSN: 1469-8951
  • URL: /core/journals/probability-in-the-engineering-and-informational-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 8 *
Loading metrics...

Abstract views

Total abstract views: 62 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 18th October 2017. This data will be updated every 24 hours.