Skip to main content Accessibility help
×
Home

Analysis of a stochastic approximation algorithm for computing quasi-stationary distributions

  • J. Blanchet (a1), P. Glynn (a2) and S. Zheng (a1)

Abstract

We study the convergence properties of a Monte Carlo estimator proposed in the physics literature to compute the quasi-stationary distribution on a transient set of a Markov chain (see De Oliveira and Dickman (2005), (2006), and Dickman and Vidigal (2002)). Using the theory of stochastic approximations we verify the consistency of the estimator and obtain an associated central limit theorem. We provide an example showing that convergence might occur very slowly if a certain eigenvalue condition is violated. We alleviate this problem using an easy-to-implement projection step combined with averaging.

Copyright

Corresponding author

* Postal address: Department of Industrial Engineering & Operations Research, Columbia University, 500 West 120th Street, New York, NY 10027, USA.
** Email address: jose.blanchet@columbia.edu
*** Postal address: Department of Management Science & Engineering, Stanford University, Huang Engineering Center, 475 Via Ortega, Stanford, CA 94035-4121, USA. Email address: glynn@stanford.edu
**** Email address: johnz622@gmail.com

References

Hide All
[1] Aldous, D.,Flannery, B. and Palacios, J. L. (1988).Two applications of urn processes to the fringe analysis of search trees and the simulation of quasi-stationary distributions of Markov chains. Prob. Eng. Inf. Sci. 2,293307.
[2] Athreya, K. B. and Karlin, S. (1968).Embedding of urn schemes into continuous time Markov branching processes and related limit theorems.Ann. Math. Statist. 39,18011817.
[3] Benaïm, M. and Cloez, B. (2015).A stochastic approximation approach to quasi-stationary distributions on finite spaces.Electron. Commun. Prob. 20, 14 pp.
[4] Blanchet, J.,Glynn, P. and Zheng, S. (2013).Empirical analysis of a stochastic approximation approach for computing quasi-stationary distributions. In EVOLVE,Springer,Berlin,1937.
[5] Boyd, S. and Vandenberghe, L. (2004).Convex Optimization.Cambridge University Press.
[6] Burzdy, K.,Hołyst, R. and March, P. (2000).A Fleming‒Viot particle representation of the Dirichlet Laplacian.Commun. Math. Phys. 214,679703.
[7] Darroch, J. N. and Seneta, E. (1965).On quasi-stationary distributions in absorbing discrete-time finite Markov chains.J. Appl. Prob. 2,88100.
[8]}Darroch, J. N. and Seneta, E. (1967).On quasi-stationary distributions in absorbing continuous-time finite Markov chains.J. Appl. Prob. 4,192196.
[9] De Oliveira, M. M. and Dickman, R. (2005).How to simulate the quasistationary state.Phys. Rev. E 71, 016129.
[10] De Oliveira, M. M. and Dickman, R. (2006).Quasi-stationary simulation: the subcritical contact process.Brazilian J. Phys. 36,685689.
[11] Del Moral, P. and Miclo, L. (2006).Self-interacting Markov chains.Stoch. Anal. Appl. 24,615660.
[12] Dickman, R. and Vidigal, R. (2002).Quasi-stationary distributions for stochastic processes with an absorbing state.J. Phys. A 35,11471166.
[13] Dimov, I. T.,Karaivanova, A. N. and Yordanova, P. I. (1998).Monte Carlo algorithms for calculating eigenvalues. In Monte Carlo and Quasi-Monte Carlo Methods 1996(Salzburg; Lecture Notes Statist. 127),Springer,New York, pp. 205220.
[14] Ferrari, P. A. and Marić, N. (2007).Quasistationary distributions and Fleming‒Viot processes in countable spaces.Electron. J. Prob. 12,684702.
[15] Golub, G. H. and Van Loan, C. F. (1996).Matrix Computations,3rd edn.Johns Hopkins University Press,Baltimore, MD.
[16] Groisman, P. and Jonckheere, M. (2013).Simulation of quasi-stationary distributions on countable spaces.{Markov Process. Relat. Fields 19,521542.
[17] Karlin, S. and Taylor, H. M. (1975).A First Course in Stochastic Processes,2nd edn.Academic Press,New York.
[18] Krasulina, T. P. (2013).The method of stochastic approximation for the determination of the least eigenvalue of a symmetrical matrix.{USSR Comput. Math. Math. Phys. 9,189195.
[19] Krasulina, T. P. (1970).Method of stochastic approximation in the determination of the largest eigenvalue of the mathematical expectation of random matrices.Automat. Rem. Contr. 2,215221.
[20] Kushner, H. J. and Yin, G. (2003).Stochastic Approximation and Recursive Algorithms and Applications,2nd edn.Springer,New York.
[21] Méléard, S. and Villemonais, D. (2012).Quasi-stationary distributions and population processes.Prob. Surveys 9,340410.
[22] Oja, E. and Karhunen, J. (1985).On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix.J. Math. Anal. Appl. 106,6984.
[23] Permantle, R. (2007).A survey of random processes with reinforcement.Prob. Surveys 4,179.
[24] Polyak, B. T. and Juditsky, A. B. (1992).Acceleration of stochastic approximation by averaging.SIAM J. Control Optimization 30,838855.
[25] Robbins, H. and Monro, S. (1951).A stochastic approximation method.Ann. Math. Statist. 22,400407.
[26] Zheng, S. (2014).Stochastic approximation algorithms in the estimation of quasi-stationary distribution of finite and general state space Markov chains. Doctoral Thesis, Columbia University.

Keywords

MSC classification

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed