×

Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ. (English) Zbl 1478.68309

Summary: We estimate the expressive power of certain deep neural networks (DNNs for short) on a class of countably-parametric, holomorphic maps \(u : U \rightarrow \mathbb{R}\) on the parameter domain \(U = [- 1, 1]^{\mathbb{N}}\). Dimension-independent rates of best \(n\)-term truncations of generalized polynomial chaos (gpc for short) approximations depend only on the summability exponent of the sequence of their gpc expansion coefficients. So-called \((\mathbf{b}, \varepsilon)\)-holomorphic maps \(u\), with \(\mathbf{b} \in \ell^p\) for some \(p \in(0, 1)\), are known to allow gpc expansions with coefficient sequences in \(\ell^p\). Such maps arise for example as response surfaces of parametric PDEs, with applications in PDE uncertainty quantification (UQ) for many mathematical models in engineering and the sciences. Up to logarithmic terms, we establish the dimension independent approximation rate \(s = 1 / p - 1\) for these functions in terms of the total number \(N\) of units and weights in the DNN. It follows that certain DNN architectures can overcome the curse of dimensionality when expressing possibly countably-parametric, real-valued maps with a certain degree of sparsity in the sequences of their gpc expansion coefficients. We also obtain rates of expressive power of DNNs for countably-parametric maps \(u : U \rightarrow V\), where \(V\) is the Hilbert space \(H_0^1([0, 1])\).

MSC:

68T07 Artificial neural networks and deep learning
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Bachmayr, M., Cohen, A., Dũng, D. and Schwab, C., Fully discrete approximation of parametric and stochastic elliptic PDEs, SIAM J. Numer. Anal.55(5) (2017) 2151-2186. · Zbl 1377.65005
[2] Bachmayr, M., Cohen, A. and Migliorati, G., Sparse polynomial approximation of parametric elliptic PDEs. Part I: Affine coefficients, ESAIM Math. Model. Numer. Anal.51(1) (2017) 321-339. · Zbl 1365.41003
[3] Băcuţă, C., Li, H. and Nistor, V., Differential operators on domains with conical points: Precise uniform regularity estimates, Rev. Roumaine Math. Pures Appl.62(3) (2017) 383-411. · Zbl 1399.35171
[4] Barron, A. R., Complexity regularization with application to artificial neural networks, in Nonparametric Functional Estimation and Related Topics, , Vol. 335 (Kluwer Academic Publisher, Dordrecht, 1991), pp. 561-576. · Zbl 0739.62001
[5] Barron, A. R., Universal approximation bounds for superpositions of a sigmoidal function, IEEE Trans. Inform. Theory39(3) (1993) 930-945. · Zbl 0818.68126
[6] C. Beck, E. Weinan and A. Jentzen, Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations, Technical Report 2017-49, Seminar for Applied Mathematics, ETH Zürich (2017). · Zbl 1442.91116
[7] Y. Bengio, Practical recommendations for gradient-based training of deep architectures, preprint (2012), arXiv:1206.5533v2.
[8] H. Bölcskei, P. Grohs, G. Kutyniok and P. Petersen, Optimal approximation with sparsely connected deep neural networks, preprint (2017), arXiv:1705.01714. · Zbl 1499.41029
[9] Chkifa, A., Cohen, A. and Schwab, C., High-dimensional adaptive sparse polynomial interpolation and applications to parametric PDEs, J. Found. Comput. Math.14(4) (2013) 601-633. · Zbl 1298.65022
[10] Cohen, A., Chkifa, A. and Schwab, C., Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs, J. Math. Pures Appl.103(2) (2015) 400-428. · Zbl 1327.65251
[11] Cohen, A. and DeVore, R., Approximation of high-dimensional parametric PDEs, Acta Numer.24 (2015) 1-159. · Zbl 1320.65016
[12] Cohen, A., DeVore, R. and Schwab, C., Convergence rates of best \(N\)-term Galerkin approximations for a class of elliptic sPDEs, Found. Comput. Math.10(6) (2010) 615-646. · Zbl 1206.60064
[13] Cohen, A., Devore, R. and Schwab, C., Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE’s, Anal. Appl. (Singap.)9(1) (2011) 11-47. · Zbl 1219.35379
[14] Cohen, A., Schwab, C. and Zech, J., Shape holomorphy of the stationary Navier-Stokes equations, SIAM J. Math. Anal.50(2) (2018) 1720-1752. · Zbl 1390.35227
[15] Cohen, N., Sharir, O. and Shashua, A., On the expressive power of deep learning: A tensor analysis, in Proc. of 29th Annual Conf. Learning Theory, New York, USA, (2016), pp. 698-728.
[16] Dick, J., Gantner, R. N., Le Gia, Q. T. and Schwab, C., Multilevel higher-order quasi-Monte Carlo Bayesian estimation, Math. Models Methods Appl. Sci.27(5) (2017) 953-995. · Zbl 1375.65169
[17] Funahashi, K.-I., Approximate realization of identity mappings by three-layer neural networks, Electron. Comm. Japan Part III Fund. Electron. Sci.73(11) (1991) 61-68.
[18] Gittelson, C. J., Convergence rates of multilevel and sparse tensor approximations for a random elliptic PDE, SIAM J. Numer. Anal.51(4) (2013) 2426-2447. · Zbl 1274.35438
[19] Grohs, P., Wiatowski, T. and Boelcskei, H. H., Deep convolutional neural networks on cartoon functions, in Proc. 2016 IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain, (2016), pp. 1163-1167.
[20] Hervé, M., Analyticity in Infinite-Dimensional Spaces, , Vol. 10 (Walter de Gruyter & Co., Berlin, 1989). · Zbl 0666.58008
[21] Hornik, K., Approximation capabilities of multilayer feedforward networks, Neural Net.4(2) (1991) 251-257.
[22] Hornik, K., Stinchcombe, M. and White, H., Multilayer feedforward networks are universal approximators, Neural Net.2(5) (1989) 359-366. · Zbl 1383.92015
[23] Jerez-Hanckes, C., Schwab, C. and Zech, J., Electromagnetic wave scattering by random surfaces: Shape holomorphy, Math. Models Methods Appl. Sci.27(12) (2017) 2229-2259. · Zbl 1381.35170
[24] Liang, L., Kong, F., Martin, C., Pham, T., Wang, Q., Duncan, J. and Sun, W., Machine learning-based 3-D geometry reconstruction and modeling of aortic valve deformation using 3-D computed tomography images, Int. J. Numer. Methods Biomed. Eng.33(5) (2017) e2827.
[25] Liang, L., Liu, M., Martin, C. and Sun, W., A deep learning approach to estimate stress distribution: A fast and accurate surrogate of finite-element analysis, J. R. Soc. Interface15 (2018) 20170844.
[26] Liang, S. and Srikant, R., Why deep neural networks for function approximation? in Proc. of ICLR 2017, Toulon, France, (2017), pp. 1-17.
[27] Mhaskar, H. N., Neural networks for optimal approximation of smooth and analytic functions, Neural Comput.8 (1996) 164-177.
[28] Mhaskar, H. N. and Poggio, T., Deep vs. shallow networks: An approximation theory perspective, Anal. Appl. (Singap.)14(6) (2016) 829-848. · Zbl 1355.68233
[29] Muñoz, G. A., Sarantopoulos, Y. and Tonge, A., Complexifications of real Banach spaces, polynomials and multilinear maps, Studia Math.134(1) (1999) 1-33. · Zbl 0945.46010
[30] Pinkus, A., Approximation theory of the MLP model in neural networks, Acta Numer.8 (1999) 143-195. · Zbl 0959.68109
[31] Poon, H. and Domingos, P., Sum-product networks: A new deep architecture, in 2011 IEEE Int. Conf. Computer Vision Workshops (ICCV Workshops) (IEEE, 2011), pp. 689-690.
[32] Quarteroni, A. and Rozza, G., Numerical solution of parametrized Navier-Stokes equations by reduced basis methods, Numer. Methods Partial Differential Equations23(4) (2007) 923-948. · Zbl 1178.76238
[33] D. Rolnik and M. Tegmark, The power of deeper networks for expressing natural functions, Technical Report (2017).
[34] Schwab, C. and Stuart, A. M., Sparse deterministic approximation of Bayesian inverse problems, Inv. Prob.28(4) (2012) 045003. · Zbl 1236.62014
[35] Stuart, A. M., Inverse problems: A Bayesian perspective, Acta Numer.19 (2010) 451-559. · Zbl 1242.65142
[36] R. Tripathy and I. Bilionis, Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification, Technical Report (2018), arXiv:1705.05502. · Zbl 1419.68084
[37] Yarotsky, D., Error bounds for approximations with deep ReLU networks, Neural Net.94 (2017) 103-114. · Zbl 1429.68260
[38] J. Zech, D. Dung and C. Schwab, Multilevel approximation of parametric and stochastic pdes, Technical Report 2018-05, Seminar for Applied Mathematics, ETH Zürich (2018), (in review). · Zbl 1427.65386
[39] J. Zech and C. Schwab, Convergence rates of high dimensional Smolyak quadrature, Technical Report 2017-27, Seminar for Applied Mathematics, ETH Zürich (2017), (in review). · Zbl 07240638
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.