×

Hidden physics models: machine learning of nonlinear partial differential equations. (English) Zbl 1381.68248

Summary: While there is currently a lot of enthusiasm about “big data”, useful data is usually “small” and expensive to acquire. In this paper, we present a new paradigm of learning partial differential equations from small data. In particular, we introduce hidden physics models, which are essentially data-efficient learning machines capable of leveraging the underlying laws of physics, expressed by time dependent and nonlinear partial differential equations, to extract patterns from high-dimensional data generated from experiments. The proposed methodology may be applied to the problem of learning, system identification, or data-driven discovery of partial differential equations. Our framework relies on Gaussian processes, a powerful tool for probabilistic inference over functions, that enables us to strike a balance between model complexity and data fitting. The effectiveness of the proposed approach is demonstrated through a variety of canonical problems, spanning a number of scientific domains, including the Navier-Stokes, Schrödinger, Kuramoto-Sivashinsky, and time dependent linear fractional equations. The methodology provides a promising new direction for harnessing the long-standing developments of classical methods in applied mathematics and mathematical physics to design learning machines with the ability to operate in complex domains without requiring large quantities of data.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
35G50 Systems of nonlinear higher-order PDEs
35Q30 Navier-Stokes equations
35Q41 Time-dependent Schrödinger equations and Dirac equations
35R11 Fractional partial differential equations
60G15 Gaussian processes
68T37 Reasoning under uncertainty in the context of artificial intelligence

Software:

L-BFGS; PMTK
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Numerical gaussian processes for time-dependent and non-linear partial differential equations (2017), arXiv preprint
[2] Rudy, S. H.; Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Data-driven discovery of partial differential equations, Sci. Adv., 3 (2017)
[3] Crutchfield, J. P.; McNamara, B. S., Equations of motion from a data series, Complex Syst., 1, 121 (1987)
[4] Kevrekidis, I. G.; Gear, C. W.; Hyman, J. M.; Kevrekidid, P. G.; Runborg, O.; Theodoropoulos, C., Equation-free, coarse-grained multiscale computation: enabling microscopic simulators to perform system-level analysis, Commun. Math. Sci., 1, 715-762 (2003) · Zbl 1086.65066
[5] Gonzalez-Garcia, R.; Rico-Martinez, R.; Kevrekidis, I., Identification of distributed parameter systems: a neural net based approach, Comput. Chem. Eng., 22, S965-S968 (1998)
[6] Voss, H. U.; Kolodner, P.; Abel, M.; Kurths, J., Amplitude equations from spatiotemporal binary-fluid convection data, Phys. Rev. Lett., 83, 3422 (1999)
[7] Sugihara, G.; May, R.; Ye, H.; Hsieh, C.-h.; Deyle, E.; Fogarty, M.; Munch, S., Detecting causality in complex ecosystems, Science, 338, 496-500 (2012) · Zbl 1355.92144
[8] Ye, H.; Beamish, R. J.; Glaser, S. M.; Grant, S. C.; Hsieh, C.-h.; Richards, L. J.; Schnute, J. T.; Sugihara, G., Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling, Proc. Natl. Acad. Sci., 112, E1569-E1576 (2015)
[9] Roberts, A. J., Model Emergent Dynamics in Complex Systems (2014), SIAM
[10] Schmidt, M. D.; Vallabhajosyula, R. R.; Jenkins, J. W.; Hood, J. E.; Soni, A. S.; Wikswo, J. P.; Lipson, H., Automated refinement and inference of analytical models for metabolic networks, Phys. Biol., 8, Article 055011 pp. (2011)
[11] Daniels, B. C.; Nemenman, I., Automated adaptive inference of phenomenological dynamical models, Nat. Commun., 6 (2015)
[12] Daniels, B. C.; Nemenman, I., Efficient inference of parsimonious phenomenological models of cellular dynamics using s-systems and alternating regression, PLoS ONE, 10, Article e0119821 pp. (2015)
[13] Majda, A. J.; Franzke, C.; Crommelin, D., Normal forms for reduced stochastic climate models, Proc. Natl. Acad. Sci., 106, 3649-3653 (2009) · Zbl 1202.86011
[14] Giannakis, D.; Majda, A. J., Nonlinear laplacian spectral analysis for time series with intermittency and low-frequency variability, Proc. Natl. Acad. Sci., 109, 2222-2227 (2012) · Zbl 1256.62053
[15] Mezić, I., Spectral properties of dynamical systems, model reduction and decompositions, Nonlinear Dyn., 41, 309-325 (2005) · Zbl 1098.37023
[16] Budišić, M.; Mohr, R.; Mezić, I., Applied koopmanism a, Chaos, 22, Article 047510 pp. (2012) · Zbl 1319.37013
[17] Mezić, I., Analysis of fluid flows via spectral properties of the Koopman operator, Annu. Rev. Fluid Mech., 45, 357-378 (2013) · Zbl 1359.76271
[18] Brunton, S. L.; Brunton, B. W.; Proctor, J. L.; Kaiser, E.; Kutz, J. N., Chaos as an intermittently forced linear system, Nat. Commun., 8 (2017)
[19] Bongard, J.; Lipson, H., Automated reverse engineering of nonlinear dynamical systems, Proc. Natl. Acad. Sci., 104, 9943-9948 (2007) · Zbl 1155.37044
[20] Schmidt, M.; Lipson, H., Distilling free-form natural laws from experimental data, Science, 324, 81-85 (2009)
[21] Tibshirani, R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. B, 267-288 (1996) · Zbl 0850.62538
[22] Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci., 113, 3932-3937 (2016) · Zbl 1355.94013
[23] Mangan, N. M.; Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Inferring biological networks by sparse identification of nonlinear dynamics, IEEE Trans. Mol. Biol. Multi-Scale Commun., 2, 52-63 (2016)
[24] Wang, W.-X.; Yang, R.; Lai, Y.-C.; Kovanis, V.; Grebogi, C., Predicting catastrophes in nonlinear dynamical systems by compressive sensing, Phys. Rev. Lett., 106, Article 154101 pp. (2011)
[25] Schaeffer, H.; Caflisch, R.; Hauck, C. D.; Osher, S., Sparse dynamics for partial differential equations, Proc. Natl. Acad. Sci., 110, 6634-6639 (2013) · Zbl 1292.35012
[26] Ozoliņš, V.; Lai, R.; Caflisch, R.; Osher, S., Compressed modes for variational problems in mathematics and physics, Proc. Natl. Acad. Sci., 110, 18368-18373 (2013) · Zbl 1292.81024
[27] Mackey, A.; Schaeffer, H.; Osher, S., On the compressive spectral method, Multiscale Model. Simul., 12, 1800-1827 (2014) · Zbl 1314.65152
[28] Brunton, S. L.; Tu, J. H.; Bright, I.; Kutz, J. N., Compressive sensing and low-rank libraries for classification of bifurcation regimes in nonlinear dynamical systems, SIAM J. Appl. Dyn. Syst., 13, 1716-1732 (2014) · Zbl 1354.37078
[29] Proctor, J. L.; Brunton, S. L.; Brunton, B. W.; Kutz, J., Exploiting sparsity and equation-free architectures in complex systems, Eur. Phys. J. Spec. Top., 223, 2665-2684 (2014)
[30] Bai, Z.; Wimalajeewa, T.; Berger, Z.; Wang, G.; Glauser, M.; Varshney, P. K., Low-dimensional approach for reconstruction of airfoil data via compressive sensing, AIAA J., 53, 4, 920-933 (2015)
[31] Tran, G.; Ward, R., Exact recovery of chaotic systems from highly corrupted data (2016), arXiv preprint
[32] Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Inferring solutions of differential equations using noisy multi-fidelity data, J. Comput. Phys., 335, 736-746 (2017) · Zbl 1382.65229
[33] Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Machine learning of linear differential equations using gaussian processes, J. Comput. Phys., 348, 683-693 (2017) · Zbl 1380.68339
[34] Rasmussen, C. E.; Williams, C. K., Gaussian Processes for Machine Learning, vol. 1 (2006), MIT Press: MIT Press Cambridge
[35] Murphy, K. P., Machine Learning: A Probabilistic Perspective (2012), MIT Press · Zbl 1295.68003
[36] Neal, R. M., Bayesian Learning for Neural Networks, vol. 118 (2012), Springer Science & Business Media
[37] Vapnik, V., The Nature of Statistical Learning Theory (2013), Springer Science & Business Media · Zbl 0934.62009
[38] Schölkopf, B.; Smola, A. J., Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (2002), MIT Press
[39] Tipping, M. E., Sparse Bayesian learning and the relevance vector machine, J. Mach. Learn. Res., 1, 211-244 (2001) · Zbl 0997.68109
[40] Tikhonov, A., Solution of incorrectly formulated problems and the regularization method, Soviet Math. Dokl., 4, 1035-1038 (1963) · Zbl 0141.11001
[41] Tikhonov, A. N.; Arsenin, V. Y., Solutions of Ill-Posed Problems (1977), W.H. Winston · Zbl 0354.65028
[42] Poggio, T.; Girosi, F., Networks for approximation and learning, Proc. IEEE, 78, 1481-1497 (1990) · Zbl 1226.92005
[43] Aronszajn, N., Theory of reproducing kernels, Trans. Am. Math. Soc., 68, 337-404 (1950) · Zbl 0037.20701
[44] Saitoh, S., Theory of Reproducing Kernels and Its Applications, vol. 189 (1988), Longman
[45] Berlinet, A.; Thomas-Agnan, C., Reproducing Kernel Hilbert Spaces in Probability and Statistics (2011), Springer Science & Business Media
[46] Duvenaud, D.; Lloyd, J. R.; Grosse, R.; Tenenbaum, J. B.; Ghahramani, Z., Structure discovery in nonparametric regression through compositional kernel search (2013), arXiv preprint
[47] Grosse, R.; Salakhutdinov, R. R.; Freeman, W. T.; Tenenbaum, J. B., Exploiting compositionality to explore a large space of model structures (2012), arXiv preprint
[48] G. Malkomes, C. Schaff, R. Garnett, Bayesian optimization for automated model selection, in: Advances in Neural Information Processing Systems, pp. 2900-2908.; G. Malkomes, C. Schaff, R. Garnett, Bayesian optimization for automated model selection, in: Advances in Neural Information Processing Systems, pp. 2900-2908.
[49] R. Calandra, J. Peters, C.E. Rasmussen, M.P. Deisenroth, Manifold Gaussian processes for regression, in: 2016 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 3338-3345.; R. Calandra, J. Peters, C.E. Rasmussen, M.P. Deisenroth, Manifold Gaussian processes for regression, in: 2016 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 3338-3345.
[50] Raissi, M.; Karniadakis, G., Deep multi-fidelity Gaussian processes (2016), arXiv preprint
[51] Liu, D. C.; Nocedal, J., On the limited memory BFGS method for large scale optimization, Math. Program., 45, 503-528 (1989) · Zbl 0696.90048
[52] Rasmussen, C. E.; Ghahramani, Z., Occam’s razor, Adv. Neural Inf. Process. Syst., 294-300 (2001)
[53] Stuart, A. M., Inverse problems: a Bayesian perspective, Acta Numer., 19, 451-559 (2010) · Zbl 1242.65142
[54] E. Snelson, Z. Ghahramani, Sparse Gaussian processes using pseudo-inputs, in: Advances in Neural Information Processing Systems, pp. 1257-1264.; E. Snelson, Z. Ghahramani, Sparse Gaussian processes using pseudo-inputs, in: Advances in Neural Information Processing Systems, pp. 1257-1264.
[55] Hensman, J.; Fusi, N.; Lawrence, N. D., Gaussian processes for big data, (Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence. Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI 2013, Bellevue, WA, USA, August 11-15 (2013))
[56] Raissi, M., Parametric gaussian process regression for big data (2017), arXiv preprint
[57] Basdevant, C.; Deville, M.; Haldenwang, P.; Lacroix, J.; Ouazzani, J.; Peyret, R.; Orlandi, P.; Patera, A., Spectral and finite difference solutions of the Burgers equation, Comput. Fluids, 14, 23-41 (1986) · Zbl 0612.76031
[58] Dauxois, T., Fermi, Pasta, Ulam and a mysterious lady (2008), arXiv preprint
[59] Hyman, J. M.; Nicolaenko, B., The Kuramoto-Sivashinsky equation: a bridge between pde’s and dynamical systems, Physica D, 18, 113-126 (1986) · Zbl 0602.58033
[60] Shraiman, B. I., Order, disorder, and phase turbulence, Phys. Rev. Lett., 57, 325 (1986)
[61] Nicolaenko, B.; Scheurer, B.; Temam, R., Some global dynamical properties of the Kuramoto-Sivashinsky equations: nonlinear stability and attractors, Physica D, 16, 155-183 (1985) · Zbl 0592.35013
[62] Kutz, J. N.; Brunton, S. L.; Brunton, B. W.; Proctor, J. L., Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems, vol. 149 (2016), SIAM · Zbl 1365.65009
[63] Taira, K.; Colonius, T., The immersed boundary method: a projection approach, J. Comput. Phys., 225, 2118-2137 (2007) · Zbl 1343.76027
[64] Colonius, T.; Taira, K., A fast immersed boundary method using a nullspace approach and multi-domain far-field boundary conditions, Comput. Methods Appl. Mech. Eng., 197, 2131-2146 (2008) · Zbl 1158.76395
[65] Podlubny, I., Fractional Differential Equations: An Introduction to Fractional Derivatives, Fractional Differential Equations, to Methods of Their Solution and Some of Their Applications, vol. 198 (1998), Academic Press
[66] Nolan, J., Stable Distributions: Models for Heavy-Tailed Data (2003), Birkhauser: Birkhauser New York
[67] Chambers, J. M.; Mallows, C. L.; Stuck, B., A method for simulating stable random variables, J. Am. Stat. Assoc., 71, 340-344 (1976) · Zbl 0341.65003
[68] Weron, A.; Weron, R., Computer simulation of Lévy \(α\)-stable variables and processes, (Chaos—The Interplay Between Stochastic and Deterministic Behaviour (1995)), 379-392 · Zbl 0835.60009
[69] J. Hartikainen, S. Särkkä, Kalman filtering and smoothing solutions to temporal Gaussian process regression models, in: 2010 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), IEEE, pp. 379-384.; J. Hartikainen, S. Särkkä, Kalman filtering and smoothing solutions to temporal Gaussian process regression models, in: 2010 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), IEEE, pp. 379-384.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.