Optimal control of diffusion processes and Hamilton-Jacobi-Bellman equations. I: The dynamic programming principle and applications.

*(English)*Zbl 0716.49022[Parts II and III, which are also covered by this review, were published in ibid. 8, No.11, 1229-1276 (1983; Zbl 0716.49023) and in “Nonlinear partial differential equations and their applications”, Res. Notes Math. 93, 95-205 (1983; Zbl 0716.49024).]

This set of papers deals with the optimal control of the Itô system \[ dX_ t=\sigma (X_ t,a_ t)dB_ t+b(X_ t,a_ t)dt,\quad X_ 0=x, \] x in an open domain \({\mathcal G}\) of \({\mathbb{R}}^ N\). Here \(a_ t\) is the control, \(B_ t\) is N-dimensional Brownian motion, \(\sigma\) is the given diffusion coefficient and b is the given drift. Notice that the controls appear in the diffusion coefficient. In the simplest problem we are given the cost functional \[ J(x,a)=E_ x\int^{\tau}_{0}f(X_ t,a_ t)\exp (-\lambda t)dt, \] where \(\tau\) is the first exit time of \(X_ t\) from \({\mathcal G}\). The controls are chosen to minimize J and the value function is defined by \(u(x)=\inf_{a}J(x,a)\). Formally, the Hamilton- Jacobi-Bellman (HJB) equation satisfied by u is \[ (*)\quad \sup_{a}(A_ au(x)-f(x,a))=0\text{ in } {\mathcal G}, \] with \(u=0\) on some portion of \(bdy({\mathcal G}).\) \(A_ a\) here is a second-order elliptic operator. No assumptions are made of the diffusion guaranteeing nondegeneracy or uniform ellipticity of the operator in (*). The formalism of dynamic programming which is used to derive (*) can be made rigorous if either we know that u is a priori \(C^ 2\) or we have a \(C^ 2\) solution of (*). This second criterion cannot be assured in the presence of degeneracy or nonuniform ellipticity.

Then, since “weak” solutions must be considered, how does one define the “right” concept of “weak”? The “right” weak solution concept must provide for existence, uniqueness, stability and reduction to the smooth solution when one exists. This same problem is present in first- order nonlinear p.d.e.’s for which \(C^ 1\) solutions do not usually exist and for which uniqueness in the class of Lipschitz solutions is generally false. This problem was solved in the seminal paper of M. G. Crandall and the author [Trans. Am. Math. Soc. 277, 1-42 (1983; Zbl 0599.35024)] by the introduction of the notion of viscosity solutions, which need only be continuous.

In the second paper under review the author extends the idea of viscosity solution to fully nonlinear, second-order equations \(F(D^ 2u,Du,u,x)=0\), assuming F is continuous and F(A,p,t,x)\(\geq F(B,p,t,x)\) if A and B are symmetric \(N\times N\) matrices with \(B\geq A\). These conditions include the HJB equation (*) under consideration. A viscosity solution u, of \(F(D^ 2u,Du,u,x)=0\) exists if for each g in \(C^ 2\), (i) when u-q has a local min at \(x_ 0\), \(F(D^ 2g,Dg,u,x_ 0)\geq 0\) at \(x_ 0\) and (ii) when u-g has a local max at \(x_ 0\), the inequality is reversed. In this second paper the author develops the necessary theory to show that if the value is continuous, then it is the unique viscosity solution of HJB(*). Further, if \(u^*\) is any viscosity solution (satisfying appropriate boundary conditions), then \(u^*=u\). Further properties of viscosity solutions are given and the “right” concept of “weak” solution is established.

Now the problem of control becomes that of determining when the value u is continuous. The author attacks this problem in part I. First, u is additionally characterized as the maximum subsolution of (*); then under additional assumptions u is successively shown to be upper semicontinuous, and continuous and Hölder continuous. The technical assumptions are shown to hold in a wide class of problems without introducing nondegeneracy or uniform ellipticity conditions.

The third paper of this set is the completion of the study. Regularity properties of u are developed here. For example, under some assumptions involving the discount coefficient and \(\sigma\) on the boundary of \({\mathcal G}\), the author proves that u is \(W^{1,\infty}({\mathcal G})\) and semiconcave in \({\mathcal G}\). This implies that \(\sup_{a}\| A_ au\| <\infty\) and also that u satisfies HJB(*) almost everywhere.

The three papers contain many more important results than could be reviewed here. Extensions to other problems (reflecting boundaries, optimal stopping, etc.) are also presented here. Together, these papers are a major contribution to the theory of optimal stochastic control and to the theory of fully nonlinear second-order p.d.e.’s.

This set of papers deals with the optimal control of the Itô system \[ dX_ t=\sigma (X_ t,a_ t)dB_ t+b(X_ t,a_ t)dt,\quad X_ 0=x, \] x in an open domain \({\mathcal G}\) of \({\mathbb{R}}^ N\). Here \(a_ t\) is the control, \(B_ t\) is N-dimensional Brownian motion, \(\sigma\) is the given diffusion coefficient and b is the given drift. Notice that the controls appear in the diffusion coefficient. In the simplest problem we are given the cost functional \[ J(x,a)=E_ x\int^{\tau}_{0}f(X_ t,a_ t)\exp (-\lambda t)dt, \] where \(\tau\) is the first exit time of \(X_ t\) from \({\mathcal G}\). The controls are chosen to minimize J and the value function is defined by \(u(x)=\inf_{a}J(x,a)\). Formally, the Hamilton- Jacobi-Bellman (HJB) equation satisfied by u is \[ (*)\quad \sup_{a}(A_ au(x)-f(x,a))=0\text{ in } {\mathcal G}, \] with \(u=0\) on some portion of \(bdy({\mathcal G}).\) \(A_ a\) here is a second-order elliptic operator. No assumptions are made of the diffusion guaranteeing nondegeneracy or uniform ellipticity of the operator in (*). The formalism of dynamic programming which is used to derive (*) can be made rigorous if either we know that u is a priori \(C^ 2\) or we have a \(C^ 2\) solution of (*). This second criterion cannot be assured in the presence of degeneracy or nonuniform ellipticity.

Then, since “weak” solutions must be considered, how does one define the “right” concept of “weak”? The “right” weak solution concept must provide for existence, uniqueness, stability and reduction to the smooth solution when one exists. This same problem is present in first- order nonlinear p.d.e.’s for which \(C^ 1\) solutions do not usually exist and for which uniqueness in the class of Lipschitz solutions is generally false. This problem was solved in the seminal paper of M. G. Crandall and the author [Trans. Am. Math. Soc. 277, 1-42 (1983; Zbl 0599.35024)] by the introduction of the notion of viscosity solutions, which need only be continuous.

In the second paper under review the author extends the idea of viscosity solution to fully nonlinear, second-order equations \(F(D^ 2u,Du,u,x)=0\), assuming F is continuous and F(A,p,t,x)\(\geq F(B,p,t,x)\) if A and B are symmetric \(N\times N\) matrices with \(B\geq A\). These conditions include the HJB equation (*) under consideration. A viscosity solution u, of \(F(D^ 2u,Du,u,x)=0\) exists if for each g in \(C^ 2\), (i) when u-q has a local min at \(x_ 0\), \(F(D^ 2g,Dg,u,x_ 0)\geq 0\) at \(x_ 0\) and (ii) when u-g has a local max at \(x_ 0\), the inequality is reversed. In this second paper the author develops the necessary theory to show that if the value is continuous, then it is the unique viscosity solution of HJB(*). Further, if \(u^*\) is any viscosity solution (satisfying appropriate boundary conditions), then \(u^*=u\). Further properties of viscosity solutions are given and the “right” concept of “weak” solution is established.

Now the problem of control becomes that of determining when the value u is continuous. The author attacks this problem in part I. First, u is additionally characterized as the maximum subsolution of (*); then under additional assumptions u is successively shown to be upper semicontinuous, and continuous and Hölder continuous. The technical assumptions are shown to hold in a wide class of problems without introducing nondegeneracy or uniform ellipticity conditions.

The third paper of this set is the completion of the study. Regularity properties of u are developed here. For example, under some assumptions involving the discount coefficient and \(\sigma\) on the boundary of \({\mathcal G}\), the author proves that u is \(W^{1,\infty}({\mathcal G})\) and semiconcave in \({\mathcal G}\). This implies that \(\sup_{a}\| A_ au\| <\infty\) and also that u satisfies HJB(*) almost everywhere.

The three papers contain many more important results than could be reviewed here. Extensions to other problems (reflecting boundaries, optimal stopping, etc.) are also presented here. Together, these papers are a major contribution to the theory of optimal stochastic control and to the theory of fully nonlinear second-order p.d.e.’s.

##### MSC:

49K45 | Optimality conditions for problems involving randomness |

93E20 | Optimal stochastic control |

49L20 | Dynamic programming in optimal control and differential games |

49L25 | Viscosity solutions to Hamilton-Jacobi equations in optimal control and differential games |

35R60 | PDEs with randomness, stochastic partial differential equations |

60J60 | Diffusion processes |

##### Keywords:

Itô system; diffusion; Hamilton-Jacobi-Bellman; second-order elliptic operator; viscosity solution
PDF
BibTeX
XML
Cite

\textit{P. L. Lions}, Commun. Partial Differ. Equations 8, 1101--1174 (1983; Zbl 0716.49022)

Full Text:
DOI

**OpenURL**

##### References:

[1] | Bellman R., Dynamic Programming (1957) |

[2] | Bensoussan A, Applications des inéquations varationnells en contröle stochastique. (1978) |

[3] | Bensoussan A, Contröle impulsionnel er inéquations quasi-variationnells (1982) |

[4] | Crandall M. G., To apper in Trans. Amer. Math. Soc. (1982) |

[5] | Crandall M. G., Trans. Math. Soc., (1982) |

[6] | Evans L.C., Trans. Amer. Math. Soc., 253 pp 365– (1979) |

[7] | Fleming W. H., Deterministic and stochastic optimal control (1975) · Zbl 0323.49001 |

[8] | Gilbarg D, Elliptic partial differential equations of second order (1977) · Zbl 0361.35003 |

[9] | Ikeda N, Stochastic differential equations and diffusion processes (1981) |

[10] | Jensen, R. and Lions, P. L. ”Some asymptotic problems in fully nonlinear equations and optimal stochastic control”. To apper · Zbl 0552.35027 |

[11] | Krylon, N. V. 1980. ”Controlled diffusion processes.”. Berlin: Springer. |

[12] | Th. Proba. Appl., 17 pp 114– (1972) · Zbl 0265.60055 |

[13] | Krylov N. V., Math. User Izv., 6 pp 249– (1972) · Zbl 0265.60056 |

[14] | Krylov N. V., Control of the diffusion type processes (1978) |

[15] | Krylov N. V, Liet. Mat. Rink., 21 pp 101– (1981) |

[16] | Krylov N. V., Soviet. Math. Dokl., 20 pp 253– (1979) |

[17] | Lions, P. L. 1982. ”Generalized solutions of Hamilton-Jacobi equations”. London: pitman. · Zbl 0497.35001 |

[18] | Lions P. L., In proceedings IFIP Conf. on Optimal Stochastic control and Filtering in Cocoyoc (1982) |

[19] | Lions P. L, Proceedings ”Function Spaces and Applications”Conf. in pisek. (1972) |

[20] | Lions P. L, C. R. Acad. Sc. Paris 289 pp 329– (1979) |

[21] | In Mathematical Optimal Control Theory |

[22] | Lions P. L., Acta Mathematica 146 pp 151– (1981) · Zbl 0467.49016 |

[23] | Lions P. L., Comm. Pure. Appl. Math., 34 pp 121– (1981) |

[24] | Lions P. L., To apper in Manuscripta Math.. |

[25] | Lions P. L, To apper in Arch. Rat. Mech. Anal.. 34 (1981) |

[26] | Lions P. L, To appear in Ricerche Mat. 34 (1981) |

[27] | Lions P. L, To appear in Nonlinear Anal. T.M.A. 34 (1981) |

[28] | Lions P. L, Siam J. Control. Optim., 20 pp 58– (1982) · Zbl 0478.93069 |

[29] | Lions P. L, Siam J. Control. Optim., 20 pp 82– (1982) · Zbl 0478.93070 |

[30] | Lions P. L, R.A.I.R.O., 14 pp 369– (1980) |

[31] | Lions P. L. Sznitman A. S. To appear |

[32] | Menaldi, J. L. ”Sur les problémes de temp d’arrët, Contröle impulsionnel et continu correspondant à des opérateurs dégénérés.”. 1980: Thèse d’Etat. |

[33] | Nisio M., Proc. Third USSR-Japan Sympos.Prona. Theory (1976) |

[34] | Nisio M., Jap.J. Math., 1 pp 159– (1975) |

[35] | Nisio, M. 1981. ”ISI Lecture Notes”. Bombay: Macmillan India Ltd, Lecture on stochastic control theory |

[36] | Nisio M, publ. R.I.M.S. Kyoto Univ., 12 pp 513– (1976) · Zbl 0364.93039 |

[37] | Nisio, M. 1978. ”on stochastic optimal controls and envelope of Markovian semi-groups.”. Tokyo: Kinokuniya. Proc. Intern. Symp. Sde Kyoto 1976 · Zbl 0418.49031 |

[38] | Oleinik O, Rend. Classe Sci. Fis. Mat. Nat. Acad.Naz.Linei, 40 pp 775– (1966) |

[39] | Perthame, B. ”Thèse de 3ecycle”. Paris |

[40] | Quadrat J. P, Siam J. Control Opt., 18 pp 199– (1980) · Zbl 0439.93057 |

[41] | SAfonov M. V, Math. User Sbornik 31 pp 231– (1977) · Zbl 0386.93059 |

[42] | Safonov M. V, Math. User Sbornik, 34 pp 521– (1978) · Zbl 0442.35036 |

[43] | Schwartz, L. 1950. ”Théorie des distribution”. Paris: Hermann. |

[44] | Serrin J, Phil. Trans. Soc. London A 264 pp 413– (1969) · Zbl 0181.38003 |

[45] | Stroock, D.W and Varadman, S.R.S. 1979. ”Multidimensionnal diffusion processes”. Berlin: Springer. |

[46] | Stroock D.W, Comm.Pure Appl. Math., 25 pp 651– (1972) · Zbl 0344.35041 |

[47] | Stroock D. W, Comm. Pure Appl. Math., 24 pp 147– (1971) · Zbl 0227.76131 |

[48] | Tanaka H, Hiroshima Math. J., 9 pp 63– (1979) |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.