Skip to main content

Forward sensitivity analysis for contracting stochastic systems

  • Thomas Flynn (a1)

In this paper we investigate gradient estimation for a class of contracting stochastic systems on a continuous state space. We find conditions on the one-step transitions, namely differentiability and contraction in a Wasserstein distance, that guarantee differentiability of stationary costs. Then we show how to estimate the derivatives, deriving an estimator that can be seen as a generalization of the forward sensitivity analysis method used in deterministic systems. We apply the results to examples, including a neural network model.

Corresponding author
* Current address: Computational Science Initiative, Brookhaven National Laboratory, P.O. Box 5000, Upton, NY 11973, USA. Email address:
Hide All
[1] Borovkov, A. A. and Hordijk, A. (2004). Characterization and sufficient conditions for normed ergodicity of Markov chains. Adv. Appl. Prob. 36, 227242.
[2] Burago, D., Burago, Y. and Ivanov, S. (2001). A Course in Metric Geometry (Graduate Stud. Math. 33). American Mathematical Society, Providence, RI.
[3] Flynn, T. (2015). Timescale separation in recurrent neural networks. Neural Comput. 27, 13211344.
[4] Flynn, T. (2016). Convergence of one-step adjoint methods. In Proceedings of the 22nd International Symposium on Mathematical Theory of Networks and Systems.
[5] Forni, F. and Sepulchre, R. (2014). A differential Lyapunov framework for contraction analysis. IEEE Trans. Automatic Control 59, 614628.
[6] Griewank, A. and Walther, A. (2008). Evaluating Derivatives, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia, PA.
[7] Hairer, M. (2006). Ergodic properties of Markov processes. Lecture given at the University of Warwick. Available at
[8] Hairer, M. and Mattingly, J. C. (2008). Spectral gaps in Wasserstein distances and the 2D stochastic Navier–Stokes equations. Ann. Prob. 36, 20502091.
[9] Heidergott, B. and Hordijk, A. (2003). Taylor series expansions for stationary Markov chains. Adv. Appl. Prob. 35, 10461070. (Correction: 36 (2004), 1300.)
[10] Heidergott, B., Hordijk, A. and Weisshaupt, H. (2006). Measure-valued differentiation for stationary Markov chains. Math. Operat. Res. 31, 154172.
[11] Joulin, A. and Ollivier, Y. (2010). Curvature, concentration and error estimates for Markov chain Monte Carlo. Ann. Prob. 38, 24182442.
[12] Lohmiller, W. and Slotine, J.-J. E. (1998). On contraction analysis for non-linear systems. Automatica J. 34, 683696.
[13] Pflug, G. C. (1992). Gradient estimates for the performance of Markov chains and discrete event processes. Ann. Operat. Res. 39, 173194.
[14] Pflug, G. C. (1996). Optimization of Stochastic Models: The Interface Between Simulation and Optimization. Kluwer, Boston, MA.
[15] Pineda, F. J. (1988). Dynamics and architecture for neural computation. J. Complexity 4, 216245.
[16] Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533536.
[17] Russo, G., di Bernardo, M. and Sontag, E. D. (2010). Global entrainment of transcriptional systems to periodic inputs. PLoS Comput. Biol. 6, e1000739.
[18] Simpson-Porco, J. W. and Bullo, F. (2014). Contraction theory on Riemannian manifolds. Systems Control Lett. 65, 7480.
[19] Steinsaltz, D. (1999). Locally contractive iterated function systems. Ann. Prob. 27, 19521979.
[20] Stenflo, Ö. (2012). A survey of average contractive iterated function systems. J. Difference Equat. Appl. 18, 13551380.
[21] Vázquez-Abad, F. J. and Kushner, H. J. (1992). Estimation of the derivative of a stationary measure with respect to a control parameter. J. Appl. Prob. 29, 343352.
[22] Villani, C. (2009). Optimal Transport: Old and New. Springer, Berlin.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Advances in Applied Probability
  • ISSN: 0001-8678
  • EISSN: 1475-6064
  • URL: /core/journals/advances-in-applied-probability
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


MSC classification


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed