×

On risk-sensitive piecewise deterministic Markov decision processes. (English) Zbl 1467.90086

This paper is devoted to a risk-sensitive piecewise deterministic Markov process (PDMDP) in Borel state and an action space with nonnegative cost rate. The transition and cost rates are assumed to be weakly integrable along the drift and the exponential utility of the total cost has to be minimized. The authors show that the value function is a solution to the optimality equation, justify the value iteration algorithm and prove the existence of the deterministic stationary policy. It should be stressed that a PDMDP, not systematically earlier studied in the literature, is an extention of a continuous-time Markov decision process (CTMDP), where between consecutive jumps, the process evolves according to a deterministic Markov process. The obtained results are further applied to improve the known results for finite horizon undisconected and infinite horizon disconected risk-sensitive CTMDP presented in: [M. K. Ghosh and S. Saha, Stochastics 86, No. 4, 655–675 (2014; Zbl 1337.49046)] and [Q. Wei, Math. Methods Oper. Res. 84, No. 3, 461–487 (2016; Zbl 1354.93179)].

MSC:

90C40 Markov and semi-Markov decision processes
93C20 Control/observation systems governed by partial differential equations
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Bäuerle, N.; Jaśkiewicz, A., Risk-sensitive Divident problems, Eur. J. Oper. Res., 242, 161-171 (2015) · Zbl 1341.91087 · doi:10.1016/j.ejor.2014.10.046
[2] Bäuerle, N.; Rieder, U., MDP algorithms for portfolio optimization problems in pure jump markets, Financ. Stoch., 13, 591-611 (2009) · Zbl 1199.91169 · doi:10.1007/s00780-009-0093-0
[3] Bäuerle, N.; Rieder, U., Markov Decision Processes with Applications to Finance (2011), Berlin: Springer, Berlin · Zbl 1236.90004
[4] Bäuerle, N.; Rieder, U., More risk-sensitive Markov decision processes, Math. Oper. Res., 39, 105-120 (2014) · Zbl 1291.90289 · doi:10.1287/moor.2013.0601
[5] Bertsekas, D.; Shreve, S., Stochastic Optimal Control (1978), New York: Academic Press, New York · Zbl 0471.93002
[6] Cavazos-Cadena, R.; Montes-de-Oca, R., Optimal stationary policies in risk-sensitive dynamic programs with finite state space and nonnegative rewards, Appl. Math. (Warsaw), 27, 167-185 (2000) · Zbl 1006.93070 · doi:10.4064/am-27-2-167-185
[7] Chung, K.; Sobel, M., Discounted MDP’s: distribution functions and exponential utility maximization, SIAM J Control Optim., 25, 49-62 (1987) · Zbl 0617.90085 · doi:10.1137/0325004
[8] Coraluppi, S., Marcus, S.: Risk-sensitive queueing. In: Proceedings of the 35th Annual Allerton Conference on Communication Control and Computing, 943-952 (1997)
[9] Costa, O.; Dufour, F., Continuous Average Control of Piecewise Deterministic Markov Processes (2013), New York: Springer, New York · Zbl 1272.93002
[10] Davis, M., Markov Models and Optimization (1993), London: Chapman and Hall, London · Zbl 0780.60002
[11] Di Masi, G.; Stettner, L., Risk-sensitive control of discrete-time Markov processes with infinite horizon, SIAM J. Control Optim., 38, 61-78 (1999) · Zbl 0946.93043 · doi:10.1137/S0363012997320614
[12] Fainberg, E., Controlled Markov processes with arbitrary numerical criteria, Theory Probab. Appl., 27, 486-503 (1982) · Zbl 0515.90076 · doi:10.1137/1127058
[13] Forwick, L.; Schäl, M.; Schmitz, M., Piecewise deterministic Markov control processes with feedback controls and unbounded costs, Acta Appl. Math., 82, 239-267 (2004) · Zbl 1084.49027 · doi:10.1023/B:ACAP.0000031200.76583.75
[14] Ghosh, M.; Saha, S., Risk-sensitive control of continuous time Markov chains, Stochastics, 86, 655-675 (2014) · Zbl 1337.49046 · doi:10.1080/17442508.2013.872644
[15] Hernández-Lerma, O.; Lasserre, J., Discrete-Time Markov Control Processes (1996), New York: Springer, New York
[16] Howard, R.; Matheson, J., Risk-sensitive Markov decision proceses, Manag. Sci., 18, 356-369 (1972) · Zbl 0238.90007 · doi:10.1287/mnsc.18.7.356
[17] Jaquette, S., A utility criterion for Markov decision processes, Manag. Sci., 23, 43-49 (1976) · Zbl 0337.90053 · doi:10.1287/mnsc.23.1.43
[18] Jaśkiewicz, A., A note on negative dynamic programming for risk-sensitive control, Oper. Res. Lett., 36, 531-534 (2008) · Zbl 1210.90171 · doi:10.1016/j.orl.2008.03.003
[19] Kitaev, M.; Rykov, V., Controlled Queueing Systems (1995), Boca Raton: CRC Press, Boca Raton · Zbl 0876.60077
[20] Kumar, S.; Pal, C., Risk-sensitive control of pure jump process on countable space with near monotone cost, Appl. Math. Optim., 68, 311-331 (2013) · Zbl 1282.93278 · doi:10.1007/s00245-013-9208-2
[21] Piunovski, A.; Khametov, V., New effective solutions of optimality equations for the controlled Markov chains with continuous parameter (the unbounded price-function), Probl. Control Inf. Theory, 14, 303-318 (1985) · Zbl 0591.93064
[22] Piunovskiy, A., Optimal Control of Random Sequences in Problems with Constraints (1997), Dordrecht: Kluwer, Dordrecht · Zbl 0894.93001
[23] Schäl, M., On piecewise deterministic Markov control processes: control of jumps and of risk processes in insurance, Insur. Math. Econ., 22, 75-91 (1998) · Zbl 0906.90170 · doi:10.1016/S0167-6687(98)00010-9
[24] Wei, Q., Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion, Math. Methods Oper. Res., 84, 461-487 (2016) · Zbl 1354.93179 · doi:10.1007/s00186-016-0550-4
[25] Wei, Q.; Chen, X., Continuous-time Markov decision processes under the risk-sensitive average cost criterion, Oper. Res. Lett., 44, 457-462 (2016) · Zbl 1380.90279 · doi:10.1016/j.orl.2016.04.010
[26] Yushkevich, A., On reducing a jump controllable Markov model to a model with discrete time, Theory Probab. Appl., 25, 58-68 (1980) · Zbl 0456.90086 · doi:10.1137/1125005
[27] Zhang, Y., Continuous-time Markov decision processes with exponential utility, SIAM J. Control Optim., 55, 2636-2660 (2017) · Zbl 1370.90288 · doi:10.1137/16M1086261
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.