×

Control of continuous-time Markov chains with safety constraints. (English) Zbl 1273.93145

Summary: In this work the controlled continuous-time finite-state Markov chain with safety constraints is studied. The constraints are expressed as a finite number of inequalities, whose intersection forms a polyhedron. A probability distribution vector is called safe if it is in the polyhedron. Under the assumptions that the controlled Markov chain is completely observable and the controller induces a unique stationary distribution in the interior of the polyhedron, the author identifies the Supreme Invariant Safety Set (SISS) where a set is called an invariant safety set if any probability distribution in the set is initially safe and remains safe as time evolves. In particular, a necessary and sufficient condition for the SISS to be the polyhedron itself is given via linear programming formulations. A closed-form expression for the condition is also derived as the constraints impose only upper and/or lower bounds on the components of the distribution vectors. If the condition is not satisfied, a finite time bound is identified and used to characterize the SISS. Numerical examples are provided to illustrate the results.

MSC:

93E03 Stochastic systems in control theory (general)
60J27 Continuous-time Markov processes on discrete state spaces
90C05 Linear programming
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Puterman, Markov Decision Process: Discrete Stochastic Dynamic Programming (1994) · Zbl 0829.90134
[2] Altman, Constrained Markov Decision Processes (1999) · Zbl 0963.90068
[3] Ramadge, Supervisory control of a class of discrete event processes, SIAM Journal on Control and Optimization 25 (1) pp 206– (1987) · Zbl 0618.93033 · doi:10.1137/0325013
[4] Arapostathis A Kumar R Hsu SP State-feedback control of Markov chains with safety bounds 6283 6288
[5] Arapostathis, Control of Markov chains with safety bounds, IEEE Transactions on Automation Science and Engineering 2 (4) pp 333– (2005) · doi:10.1109/TASE.2005.853392
[6] Arapostathis, Controlled Markov chains with safety upper bounds, IEEE Transactions on Automatic Control 48 (7) pp 1230– (2003) · Zbl 1364.93744 · doi:10.1109/TAC.2003.814267
[7] Guo, Continuous-time controlled Markov chains, The Annals of Applied Probability 13 (1) pp 363– (2003) · Zbl 1049.60067 · doi:10.1214/aoap/1042765671
[8] Grimmett, Probability and Random Processes (2001)
[9] Ross, Stochastic Processes (1995)
[10] Feinberg EA Constrained finite continuous-time Markov decision processes with average rewards 3805 3810
[11] Peng, Spectral representations of the transition probability matrices for continuous time finite Markov chains, Journal of Applied Probability 33 (1) pp 28– (1996) · Zbl 0847.60060 · doi:10.2307/3215261
[12] Seneta, Non-negative Matrices and Markov Chains (1981) · Zbl 1099.60004 · doi:10.1007/0-387-32792-4
[13] Blanchini, Set invariance in control, Automatica 35 (11) pp 1747– (1999) · Zbl 0935.93005 · doi:10.1016/S0005-1098(99)00113-2
[14] Bemporad, The explicit linear quadratic regulator for constrained systems, Automatica 38 (1) pp 3– (2002) · Zbl 0999.93018 · doi:10.1016/S0005-1098(01)00174-1
[15] Kerrigan EC Maciejowski JM Invariant sets for constrained nonlinear discrete-time systems with application to feasibility in model predictive control 4951 4956
[16] Liu, State estimation for discrete-time Markovian jumping neural networks with mixed mode-dependent delays, Physics Letters A 372 (48) pp 7147– (2008) · Zbl 1227.92002 · doi:10.1016/j.physleta.2008.10.045
[17] Wang, State estimation for jumping recurrent neural networks with discrete and distributed delays, Neural Networks 22 (1) pp 41– (2009) · Zbl 1335.93125 · doi:10.1016/j.neunet.2008.09.015
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.