Hostname: page-component-89b8bd64d-x2lbr Total loading time: 0 Render date: 2026-05-09T04:13:01.499Z Has data issue: false hasContentIssue false
  • English
  • Français

Perturbation analysis for denumerable Markov chains with application to queueing models

Part of: Markov processes

Published online by Cambridge University Press:  01 July 2016

Eitan Altman ,
Konstantin E. Avrachenkov  and
Rudesindo Núñez-Queija
Eitan Altman*
Affiliation:
INRIA
Konstantin E. Avrachenkov*
Affiliation:
CWI and Eindhoven University of Technology
Rudesindo Núñez-Queija*
Affiliation:
CWI and Eindhoven University of Technology
*
∗ Postal address: INRIA Sophia Antipolis, 2004 Route des Lucioles, BP 93, Sophia Antipolis Cedex 06902, France.
∗ Postal address: INRIA Sophia Antipolis, 2004 Route des Lucioles, BP 93, Sophia Antipolis Cedex 06902, France.
∗∗∗∗ CWI, PO Box 94079, Amsterdam, 1090 GB, The Netherlands. Email address: sindo@cwi.nl
Get access Rights & Permissions [Opens in a new window]

Abstract

We study the parametric perturbation of Markov chains with denumerable state spaces. We consider both regular and singular perturbations. By the latter we mean that transition probabilities of a Markov chain, with several ergodic classes, are perturbed such that (rare) transitions among the different ergodic classes of the unperturbed chain are allowed. Singularly perturbed Markov chains have been studied in the literature under more restrictive assumptions such as strong recurrence ergodicity or Doeblin conditions. We relax these conditions so that our results can be applied to queueing models (where the conditions mentioned above typically fail to hold). Assuming ν-geometric ergodicity, we are able to explicitly express the steady-state distribution of the perturbed Markov chain as a Taylor series in the perturbation parameter. We apply our results to quasi-birth-and-death processes and queueing models.

Keywords

Denumerable Markov chainperturbation analysisgeometric ergodicityquasi-birth-and-death processqueueing model

MSC classification

Primary: 60J10: Markov chains (discrete-time Markov processes on discrete state spaces)
Secondary: 60J22: Computational methods in Markov chains 60J27: Continuous-time Markov processes on discrete state spaces

Information

Type
General Applied Probability
Information
Advances in Applied Probability , Volume 36 , Issue 3 , September 2004 , pp. 839 - 853
Copyright
Copyright © Applied Probability Trust 2004 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

[1] Altman, E., Artiges, D. and Traore, K. (1999). On the integration of best-effort and guaranteed performance services. Europ. Trans. Telecommun. 10, 125–134.CrossRefGoogle Scholar
[2] Anisimov, V. V. (2001). Asymptotic analysis of some type optimal maintenance policies in multicomponent systems with Markov switches. In Proc. 10th Internat. Symp. Appl. Stoch. Models Data Anal. (Compiegne, June 2001), pp. 112–117. Available at http://www.hds.utc.fr/asmda2001/.Google Scholar
[3] Avrachenkov, K. E. (1999). Analytic perturbation theory and its applications. , University of South Australia. Available at http://www-sop.inria.fr/mistral/personnel/K.Avrachenkov/moi.html Google Scholar
[4] Avrachenkov, K. E., Filar, J. A. and Haviv, M. (2002). Singular perturbations of Markov chains and decision processes. In Handbook of Markov Decision Processes: Methods and Applications, eds Feinberg, E. A. and Shwartz, A., Kluwer, Boston, MA, pp. 113–150.CrossRefGoogle Scholar
[5] Bielecki, T. R. and Stettner, L. (1998). Ergodic control of singularly perturbed Markov process in discrete time with general state and compact action spaces. Appl. Math. Optimization 38, 261–281.CrossRefGoogle Scholar
[6] Cao, X.-R. and Chen, H.-F. (1997). Perturbation realization, potentials, and sensitivity analysis of Markov processes. IEEE Trans. Automatic Control 42, 1382–1393.Google Scholar
[7] Chang, C.-S. and Nelson, R. (1993). Perturbation analysis of the M/M/1 queue in a Markovian environment via the matrix-geometric method. Commun. Statist. Stoch. Models 9, 233–246.CrossRefGoogle Scholar
[8] Courtois, P. J. (1977). Decomposability: Queueing and Computer System Applications. Academic Press, New York.Google Scholar
[9] Dekker, R., Hordijk, A. and Spieksma, F. M. (1994). On the relation between recurrence and ergodicity properties in denumerable Markov decision chains. Math. Operat. Res. 19, 539–559.CrossRefGoogle Scholar
[10] Delebecque, F. (1983). A reduction process for perturbed Markov chains. SIAM J. Appl. Math. 43, 325–350.CrossRefGoogle Scholar
[11] Gelenbe, E. and Rosenberg, C. (1990). Queues with slowly varying arrival and service processes. Management Sci. 36, 928–937.CrossRefGoogle Scholar
[12] Glasserman, P. (1991). Gradient Estimation via Perturbation Analysis. Kluwer, Boston, MA.Google Scholar
[13] Heidergott, B. and Cao, X.-R. (2002). A note on the relation between weak derivatives and perturbation realization. IEEE Trans. Automatic Control 47, 1112–1115.CrossRefGoogle Scholar
[14] Heidergott, B., Hordijk, A. and Weisshaupt, H. (2002). Measure-valued differentiation for stationary Markov chains. Res. Rep. 2002–27, EURANDOM. Available at http://staff.feweb.vu.nl/bheidergott/.Google Scholar
[15] Kolmogorov, A. N. and Fomīn, S. V. (1975). Introductory Real Analysis. Dover, New York.Google Scholar
[16] Kontoyiannis, I. and Meyn, S. P. (2003). Spectral theory and limit theory for geometrically ergodic Markov processes. Ann. Appl. Prob. 13, 304–362.CrossRefGoogle Scholar
[17] Koole, G. (1998). The deviation matrix of the M/M/1/∞ and M/M/1/N queue with applications to controlled queueing models. In Proc. IEEE CDC'98 (Tampa, FL), pp. 56–59.CrossRefGoogle Scholar
[18] Korolyuk, V. S. and Turbin, A. F. (1993). Mathematical Foundations of The State Lumping of Large Systems. Kluwer, Dordrecht.CrossRefGoogle Scholar
[19] Latouche, G. and Ramaswami, V. (1999). Introduction to Matrix Analytic Methods in Stochastic Modeling. SIAM, Philadelphia, PA.CrossRefGoogle Scholar
[20] Latouche, G. and Schweitzer, P. G. (1995). A Markov modulated, nearly completely decomposable M/M/1 queue. In Computations with Markov chains, ed. Stewart, W. J., Kluwer, Boston, MA.Google Scholar
[21] Meyn, S. P. and Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. Springer, London.CrossRefGoogle Scholar
[22] Meyn, S. P. and Tweedie, R. L. (1994). Computable bounds for geometric convergence rates of Markov chains. Ann. Appl. Prob. 4, 981–1012.CrossRefGoogle Scholar
[23] Neuts, M. F. (1981). Matrix-Geometric Solutions in Stochastic Models: An Algorithmic Approach. Johns Hopkins University Press, Baltimore, MD.Google Scholar
[24] Núñez-Queija, R. (1999). Processor-sharing models for integrated-services networks. , Eind-hoven University of Technology. Available at http://www.cwi.nl/∼sindo/.Google Scholar
[25] Pervozvanski, A. A. and Gaitsgori, V. G. (1988). Theory of Suboptimal Decisions. Kluwer, Dordrecht. Russian original: Decomposition, Aggregation and Approximate Optimization, Nauka, Moscow, 1979.CrossRefGoogle Scholar
[26] Pflug, G. (1992). Gradient estimates for the performance of Markov chains and discrete event processes. Ann. Operat. Res. 39, 173–194.CrossRefGoogle Scholar
[27] Spieksma, F. M. (1990). Geometrically ergodic Markov chains and the optimal control of queues. , Leiden University.Google Scholar
[28] Spieksma, F. M. and Tweedie, R. L. (1994). Strengthening ergodicity to geometric ergodicity for Markov chains. Commun. Statist. Stoch. Models 10, 45–74.CrossRefGoogle Scholar
[29] Yin, G. and Zhang, H. (2002). Countable-state-space Markov chains with two time scales and applications to queueing systems. Adv. Appl. Prob. 34, 662–688.CrossRefGoogle Scholar
[30] Yin, G. and Zhang, Q. (1998). Continuous-Time Markov Chains and Applications. A Singular Perturbation Approach (Appl. Math. 37). Springer, New York.Google Scholar