Skip to main content

Strong consistency of a modified maximum likelihood estimator for controlled Markov chains

  • Bharat Doshi (a1) and Steven E. Shreve (a2)

A controlled Markov chain with finite state space has transition probabilities which depend on an unknown parameter α lying in a known finite set A. For each α, a stationary control law ϕ α is given. This paper develops a control scheme whereby at each stage t a parameter α t is chosen at random from among those parameters which nearly maximize the log likelihood function, and the control ut is chosen according to the control law ϕ αt. It is proved that this algorithm leads to identification of the true α under conditions weaker than any previously considered.

Corresponding author
Postal address: HP1B323, Holmdel, NJ 07733, U.S.A. Research carried out when the author was at Rutgers University.
∗∗ Postal address: Department of Mathematics, Carnegie-Mellon University, Pittsburgh, PA 15213, U.S.A.
Hide All

Research sponsored in part by the Air Force Office of Scientific Research (AFSC), United States Air Force, under Contract F-49620–79-C-0165.

Hide All
[1] Borkar, V. and Varaiya, P. (1979) Adaptive control of Markov chains, I: Finite parameter set. IEEE Trans. Auto. Control 24, 953957.
[2] Loève, M. (1960) Probability Theory Van Nostrand, Princeton, NJ.
[3] Mandl, P. (1974) Estimation and control in Markov chains. Adv. Appl. Prob. 6, 4060.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Applied Probability
  • ISSN: 0021-9002
  • EISSN: 1475-6072
  • URL: /core/journals/journal-of-applied-probability
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed