Hostname: page-component-68c7f8b79f-xmwfq Total loading time: 0 Render date: 2025-12-15T11:56:34.340Z Has data issue: false hasContentIssue false
  • English
  • Français

Numerical method for solving impulse control problems in partially observed piecewise deterministic Markov processes

Part of: Stochastic systems and control Markov processes

Published online by Cambridge University Press:  15 December 2025

Alice Cleynen  and
Benoîte De Saporta Open the ORCID record for Benoîte De Saporta [Opens in a new window]
Alice Cleynen*
Affiliation:
CNRS, The Australian National University
Benoîte De Saporta*
Affiliation:
University of Montpellier
*
*Postal address: IMAG, University of Montpellier, CNRS, Montpellier, France and John Curtin School of Medical Research, The Australian National University, Canberra, ACT, Australia. Email: alice.cleynen@umontpellier.fr
**Postal address: IMAG, University of Montpellier, CNRS, Montpellier, France. Email: benoite.de-saporta@umontpellier.fr
Get access Rights & Permissions [Opens in a new window]

Abstract

Designing efficient and rigorous numerical methods for sequential decision-making under uncertainty is a difficult problem that arises in many applications frameworks. In this paper we focus on the numerical solution of a subclass of impulse control problems for the piecewise deterministic Markov process (PDMP) when the jump times are hidden. We first state the problem as a partially observed Markov decision process (POMDP) on a continuous state space and with controlled transition kernels corresponding to some specific skeleton chains of the PDMP. We then proceed to build a numerically tractable approximation of the POMDP by tailor-made discretizations of the state spaces. The main difficulty in evaluating the discretization error comes from the possible random jumps of the PDMP between consecutive epochs of the POMDP and requires special care. Finally, we discuss the practical construction of discretization grids and illustrate our method on simulations.

Keywords

Continuous time Markov processdynamic programminghidden processnumerical approximationpartially observed Markov decision process

MSC classification

Primary: 60J25: Continuous-time Markov processes on general state spaces 93E20: Optimal stochastic control
Secondary: 60J05: Discrete-time Markov processes on general state spaces 93E11: Filtering

Information

Type
Original Article
Information
Advances in Applied Probability , First View , pp. 1 - 32
Check for updates
Copyright
© The Author(s), 2025. Published by Cambridge University Press on behalf of Applied Probability Trust

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Almudevar, A. (2001). A dynamic programming algorithm for the optimal control of piecewise deterministic markov processes. SIAM J. Control Optim. 40, 525–539.CrossRefGoogle Scholar
Bäuerle, N. and Lange, D. (2018). Optimal control of partially observable piecewise deterministic Markov processes. SIAM J. Control Optim. 56, 1441–1462.CrossRefGoogle Scholar
Bäuerle, N. and Rieder, U. (2011). Markov Decision Processes with Applications to Finance . Universitext. Springer, Berlin.Google Scholar
Baysse, C., Bihannic, D., Gégout-Petit, A., Prenat, M., de Saporta, B. and Saracco, J. (2013). Maintenance optimisation of optronic equipment. Chem. Eng. Trans. 33, 709–714.Google Scholar
Baysse, C., Bihannic, D., Gégout-Petit, A., Prenat, M. and Saracco, J. (2014). Hidden Markov model for the detection of a degraded operating mode of optronic equipment. J. Soc. Fr. Stat. 155, 48–61.Google Scholar
Brandejsky, A., de Saporta, B. and Dufour, F. (2013). Optimal stopping for partially observed piecewise-deterministic Markov processes. Stochastic Process. Appl. 123, 3201–3238.CrossRefGoogle Scholar
Calvia, A. (2020). Stochastic filtering and optimal control of pure jump markov processes with noise-free partial observation. ESAIM: Control Optim. Calc. Var. 26, 25.Google Scholar
Cleynen, A. and de Saporta, B. (2018). Change-point detection for piecewise deterministic Markov processes. Automatica 97, 234–247.CrossRefGoogle Scholar
Cocozza-Thivent, C. et al. (2021). Markov Renewal and Piecewise Deterministic Processes. Springer, Switzerland.CrossRefGoogle Scholar
Costa, M. (2016). A piecewise deterministic model for a prey-predator community. Ann. Appl. Probab. 26, 3491–3530.CrossRefGoogle Scholar
Costa, O. and Dufour, F. (2013). Continuous Average Control of Piecewise Deterministic Markov Processes . Springer Briefs in Mathematics. Springer, New York.Google Scholar
Costa, O. and Raymundo, C. (2000). Impulse and continuous control of piecewise deterministic Markov processes. Stochastics Stochastic Rep. 70, 75–107.CrossRefGoogle Scholar
Costa, O. L. and Davis, M. (1989). Impulse control of piecewise-deterministic processes. Math. Control Signals Syst. 2, 187–206.CrossRefGoogle Scholar
Davis, M. (1984). Piecewise-deterministic Markov processes: a general class of nondiffusion stochastic models. J. Roy. Statist. Soc. Ser. B 46, 353–388. With discussion.CrossRefGoogle Scholar
Davis, M. (1993). Markov Models and Optimization . Monographs on Statistics and Applied Probability, Vol. 49. Chapman & Hall, London.Google Scholar
de Saporta, B. and Dufour, F. (2012). Numerical method for impulse control of piecewise deterministic Markov processes. Automatica 48, 779–793.CrossRefGoogle Scholar
de Saporta, B., Dufour, F. and Geeraert, A. (2017). Optimal strategies for impulse control of piecewise deterministic Markov processes. Automatica 77, 219–229.CrossRefGoogle Scholar
de Saporta, B., Dufour, F. and Zhang, H. (2016). Numerical Methods for Simulation and Optimization of Piecewise Deterministic Markov Processes. Mathematics and Statistics Series. ISTE, London; John Wiley & Sons, Hoboken, NJ. Application to reliability.Google Scholar
de Saporta, B., Dufour, F. and Zhang, H. (2019). Dynamic optimization of maintenance policies for multistate system. In ESREL 2019: 29th European Safety and Reliability Conference.Google Scholar
Dempster, M. and Ye, J. (1995). Impulse control of piecewise deterministic Markov processes. Ann. Appl. Probab. 5, 399–423.CrossRefGoogle Scholar
Devooght, J. (1997). Dynamic reliability. In Advances in Nuclear Science and Technology, pp. 215–278. Springer, Boston.Google Scholar
Doumic, M., Hoffmann, M.and Krell, N. and Robert, L. (2015). Statistical estimation of a growth-fragmentation model observed on a genealogical tree. Bernoulli 21, 1760–1799.CrossRefGoogle Scholar
Dufour, F., Horiguchi, M. and Piunovskiy, A. (2016). Optimal impulsive control of piecewise deterministic Markov processes. Stochastics 88, 1073–1098.CrossRefGoogle Scholar
Fritsch, C., Harmand, J. and Campillo, F. (2015). A modeling approach of the chemostat. Ecol. Model. 299, 1–13.CrossRefGoogle Scholar
Guo, X., Hernández-Lerma, O., Guo, X. and Hernández-Lerma, O. (2009). Continuous-Time Markov Decision Processes. Springer, Berlin.CrossRefGoogle Scholar
Herbach, U., Bonnaffoux, A., Espinasse, T. and Gandrillon, O. (2017). Inferring gene regulatory networks from single-cell data: a mechanistic approach. BMC Syst. Biol. 11, 105.CrossRefGoogle Scholar
Lange, D. K. (2017). Cost optimal control of piecewise deterministic Markov processes under partial observation. PhD thesis, Karlsruher Institut für Technologie (KIT), Karlsruhe.Google Scholar
Makis, V. and Jiang, X. (2003). Optimal replacement under partial observations. Math. Oper. Res. 28, 382–394.CrossRefGoogle Scholar
Marseguerra, M. and Zio, E. (1996). Monte Carlo approach to PSA for dynamic process systems. Reliab. Eng. Syst. Safety 52, 227–241.CrossRefGoogle Scholar
Pasin, C., Dufour, F., Villain, L., Zhang, H. and Thiébaut, R. (2018). Controlling IL-7 injections in HIV-infected patients. Bull. Math. Biol. 80, 2349–2377.CrossRefGoogle ScholarPubMed
Piunovskiy, A., Plakhov, A., Torres, D. F. and Zhang, Y. (2019). Optimal impulse control of dynamical systems. SIAM J. Control Optim. 57, 2720–2752.CrossRefGoogle Scholar
Riedler, M. and Thieullen, M. (2015). Spatio-temporal hybrid (PDMP) models: central limit theorem and Langevin approximation for global fluctuations. Application to electrophysiology. Bernoulli 21, 647–696.CrossRefGoogle Scholar
Riedler, M., Thieullen, M. and Wainrib, G. (2012). Limit theorems for infinite-dimensional piecewise deterministic Markov processes. Applications to stochastic excitable membrane models. Electron. J. Probab. 17(55), 48.CrossRefGoogle Scholar
Soleimani, M., Campean, F. and Neagu, D. (2021). Integration of hidden Markov modelling and Bayesian network for fault detection and prediction of complex engineered systems. Reliab. Eng. Syst. Safety 215, 107808.CrossRefGoogle Scholar
Zhang, H., Dufour, F., Dutuit, Y. and Gonzalez, K. (2008). Piecewise deterministic Markov processes and dynamic reliability. Proc. Inst. Mech. Eng. Part O: J. Risk Reliab. 222, 545–551.Google Scholar
Supplementary material: File

Cleynen and De Saporta supplementary material

Cleynen and De Saporta supplementary material
Download Cleynen and De Saporta supplementary material(File)
File 268.1 KB