# zbMATH — the first resource for mathematics

Optimal control of diffusion processes and Hamilton-Jacobi-Bellman equations. I: The dynamic programming principle and applications. (English) Zbl 0716.49022
[Parts II and III, which are also covered by this review, were published in ibid. 8, No.11, 1229-1276 (1983; Zbl 0716.49023) and in “Nonlinear partial differential equations and their applications”, Res. Notes Math. 93, 95-205 (1983; Zbl 0716.49024).]
This set of papers deals with the optimal control of the Itô system $dX_ t=\sigma (X_ t,a_ t)dB_ t+b(X_ t,a_ t)dt,\quad X_ 0=x,$ x in an open domain $${\mathcal G}$$ of $${\mathbb{R}}^ N$$. Here $$a_ t$$ is the control, $$B_ t$$ is N-dimensional Brownian motion, $$\sigma$$ is the given diffusion coefficient and b is the given drift. Notice that the controls appear in the diffusion coefficient. In the simplest problem we are given the cost functional $J(x,a)=E_ x\int^{\tau}_{0}f(X_ t,a_ t)\exp (-\lambda t)dt,$ where $$\tau$$ is the first exit time of $$X_ t$$ from $${\mathcal G}$$. The controls are chosen to minimize J and the value function is defined by $$u(x)=\inf_{a}J(x,a)$$. Formally, the Hamilton- Jacobi-Bellman (HJB) equation satisfied by u is $(*)\quad \sup_{a}(A_ au(x)-f(x,a))=0\text{ in } {\mathcal G},$ with $$u=0$$ on some portion of $$bdy({\mathcal G}).$$ $$A_ a$$ here is a second-order elliptic operator. No assumptions are made of the diffusion guaranteeing nondegeneracy or uniform ellipticity of the operator in (*). The formalism of dynamic programming which is used to derive (*) can be made rigorous if either we know that u is a priori $$C^ 2$$ or we have a $$C^ 2$$ solution of (*). This second criterion cannot be assured in the presence of degeneracy or nonuniform ellipticity.
Then, since “weak” solutions must be considered, how does one define the “right” concept of “weak”? The “right” weak solution concept must provide for existence, uniqueness, stability and reduction to the smooth solution when one exists. This same problem is present in first- order nonlinear p.d.e.’s for which $$C^ 1$$ solutions do not usually exist and for which uniqueness in the class of Lipschitz solutions is generally false. This problem was solved in the seminal paper of M. G. Crandall and the author [Trans. Am. Math. Soc. 277, 1-42 (1983; Zbl 0599.35024)] by the introduction of the notion of viscosity solutions, which need only be continuous.
In the second paper under review the author extends the idea of viscosity solution to fully nonlinear, second-order equations $$F(D^ 2u,Du,u,x)=0$$, assuming F is continuous and F(A,p,t,x)$$\geq F(B,p,t,x)$$ if A and B are symmetric $$N\times N$$ matrices with $$B\geq A$$. These conditions include the HJB equation (*) under consideration. A viscosity solution u, of $$F(D^ 2u,Du,u,x)=0$$ exists if for each g in $$C^ 2$$, (i) when u-q has a local min at $$x_ 0$$, $$F(D^ 2g,Dg,u,x_ 0)\geq 0$$ at $$x_ 0$$ and (ii) when u-g has a local max at $$x_ 0$$, the inequality is reversed. In this second paper the author develops the necessary theory to show that if the value is continuous, then it is the unique viscosity solution of HJB(*). Further, if $$u^*$$ is any viscosity solution (satisfying appropriate boundary conditions), then $$u^*=u$$. Further properties of viscosity solutions are given and the “right” concept of “weak” solution is established.
Now the problem of control becomes that of determining when the value u is continuous. The author attacks this problem in part I. First, u is additionally characterized as the maximum subsolution of (*); then under additional assumptions u is successively shown to be upper semicontinuous, and continuous and Hölder continuous. The technical assumptions are shown to hold in a wide class of problems without introducing nondegeneracy or uniform ellipticity conditions.
The third paper of this set is the completion of the study. Regularity properties of u are developed here. For example, under some assumptions involving the discount coefficient and $$\sigma$$ on the boundary of $${\mathcal G}$$, the author proves that u is $$W^{1,\infty}({\mathcal G})$$ and semiconcave in $${\mathcal G}$$. This implies that $$\sup_{a}\| A_ au\| <\infty$$ and also that u satisfies HJB(*) almost everywhere.
The three papers contain many more important results than could be reviewed here. Extensions to other problems (reflecting boundaries, optimal stopping, etc.) are also presented here. Together, these papers are a major contribution to the theory of optimal stochastic control and to the theory of fully nonlinear second-order p.d.e.’s.

##### MSC:
 49K45 Optimality conditions for problems involving randomness 93E20 Optimal stochastic control 49L20 Dynamic programming in optimal control and differential games 49L25 Viscosity solutions to Hamilton-Jacobi equations in optimal control and differential games 35R60 PDEs with randomness, stochastic partial differential equations 60J60 Diffusion processes
Full Text:
##### References:
  Bellman R., Dynamic Programming (1957)  Bensoussan A, Applications des inéquations varationnells en contröle stochastique. (1978)  Bensoussan A, Contröle impulsionnel er inéquations quasi-variationnells (1982)  Crandall M. G., To apper in Trans. Amer. Math. Soc. (1982)  Crandall M. G., Trans. Math. Soc., (1982)  Evans L.C., Trans. Amer. Math. Soc., 253 pp 365– (1979)  Fleming W. H., Deterministic and stochastic optimal control (1975) · Zbl 0323.49001  Gilbarg D, Elliptic partial differential equations of second order (1977) · Zbl 0361.35003  Ikeda N, Stochastic differential equations and diffusion processes (1981)  Jensen, R. and Lions, P. L. ”Some asymptotic problems in fully nonlinear equations and optimal stochastic control”. To apper · Zbl 0552.35027  Krylon, N. V. 1980. ”Controlled diffusion processes.”. Berlin: Springer.  Th. Proba. Appl., 17 pp 114– (1972) · Zbl 0265.60055  Krylov N. V., Math. User Izv., 6 pp 249– (1972) · Zbl 0265.60056  Krylov N. V., Control of the diffusion type processes (1978)  Krylov N. V, Liet. Mat. Rink., 21 pp 101– (1981)  Krylov N. V., Soviet. Math. Dokl., 20 pp 253– (1979)  Lions, P. L. 1982. ”Generalized solutions of Hamilton-Jacobi equations”. London: pitman. · Zbl 0497.35001  Lions P. L., In proceedings IFIP Conf. on Optimal Stochastic control and Filtering in Cocoyoc (1982)  Lions P. L, Proceedings ”Function Spaces and Applications”Conf. in pisek. (1972)  Lions P. L, C. R. Acad. Sc. Paris 289 pp 329– (1979)  In Mathematical Optimal Control Theory  Lions P. L., Acta Mathematica 146 pp 151– (1981) · Zbl 0467.49016  Lions P. L., Comm. Pure. Appl. Math., 34 pp 121– (1981)  Lions P. L., To apper in Manuscripta Math..  Lions P. L, To apper in Arch. Rat. Mech. Anal.. 34 (1981)  Lions P. L, To appear in Ricerche Mat. 34 (1981)  Lions P. L, To appear in Nonlinear Anal. T.M.A. 34 (1981)  Lions P. L, Siam J. Control. Optim., 20 pp 58– (1982) · Zbl 0478.93069  Lions P. L, Siam J. Control. Optim., 20 pp 82– (1982) · Zbl 0478.93070  Lions P. L, R.A.I.R.O., 14 pp 369– (1980)  Lions P. L. Sznitman A. S. To appear  Menaldi, J. L. ”Sur les problémes de temp d’arrët, Contröle impulsionnel et continu correspondant à des opérateurs dégénérés.”. 1980: Thèse d’Etat.  Nisio M., Proc. Third USSR-Japan Sympos.Prona. Theory (1976)  Nisio M., Jap.J. Math., 1 pp 159– (1975)  Nisio, M. 1981. ”ISI Lecture Notes”. Bombay: Macmillan India Ltd, Lecture on stochastic control theory  Nisio M, publ. R.I.M.S. Kyoto Univ., 12 pp 513– (1976) · Zbl 0364.93039  Nisio, M. 1978. ”on stochastic optimal controls and envelope of Markovian semi-groups.”. Tokyo: Kinokuniya. Proc. Intern. Symp. Sde Kyoto 1976 · Zbl 0418.49031  Oleinik O, Rend. Classe Sci. Fis. Mat. Nat. Acad.Naz.Linei, 40 pp 775– (1966)  Perthame, B. ”Thèse de 3ecycle”. Paris  Quadrat J. P, Siam J. Control Opt., 18 pp 199– (1980) · Zbl 0439.93057  SAfonov M. V, Math. User Sbornik 31 pp 231– (1977) · Zbl 0386.93059  Safonov M. V, Math. User Sbornik, 34 pp 521– (1978) · Zbl 0442.35036  Schwartz, L. 1950. ”Théorie des distribution”. Paris: Hermann.  Serrin J, Phil. Trans. Soc. London A 264 pp 413– (1969) · Zbl 0181.38003  Stroock, D.W and Varadman, S.R.S. 1979. ”Multidimensionnal diffusion processes”. Berlin: Springer.  Stroock D.W, Comm.Pure Appl. Math., 25 pp 651– (1972) · Zbl 0344.35041  Stroock D. W, Comm. Pure Appl. Math., 24 pp 147– (1971) · Zbl 0227.76131  Tanaka H, Hiroshima Math. J., 9 pp 63– (1979)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.