×

Machine learning of linear differential equations using Gaussian processes. (English) Zbl 1380.68339

Summary: This work leverages recent advances in probabilistic machine learning to discover governing equations expressed by parametric linear operators. Such equations involve, but are not limited to, ordinary and partial differential, integro-differential, and fractional order operators. Here, Gaussian process priors are modified according to the particular form of such operators and are employed to infer parameters of the linear equations from scarce and possibly noisy observations. Such observations may come from experiments or “black-box” computer simulations, as demonstrated in several synthetic examples and a realistic application in functional genomics.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
34K37 Functional-differential equations with fractional derivatives
35K57 Reaction-diffusion equations
45J05 Integro-ordinary differential equations
60G15 Gaussian processes
92D10 Genetics and epigenetics

Software:

deGradInfer; PMTK; L-BFGS
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Alvarez, M.; Lawrence, N. D., Sparse convolved Gaussian processes for multi-output regression, (Advances in Neural Information Processing Systems (2009)), 57-64
[2] Alvarez, M. A.; Luengo, D.; Lawrence, N. D., Latent force models, (Aistats, vol. 12 (2009)), 9-16
[3] Álvarez, M. A.; Luengo, D.; Lawrence, N. D., Linear latent force models using Gaussian processes, IEEE Trans. Pattern Anal. Mach. Intell., 35, 11, 2693-2705 (2013)
[4] Aronszajn, N., Theory of reproducing kernels, Trans. Am. Math. Soc., 68, 3, 337-404 (1950) · Zbl 0037.20701
[5] Berlinet, A.; Thomas-Agnan, C., Reproducing Kernel Hilbert Spaces in Probability and Statistics (2011), Springer Science & Business Media
[6] Boyle, P.; Frean, M., Dependent Gaussian processes, (Advances in Neural Information Processing Systems (2004)), 217-224
[7] Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci., 113, 15, 3932-3937 (2016) · Zbl 1355.94013
[8] Calandra, R.; Peters, J.; Rasmussen, C. E.; Deisenroth, M. P., Manifold Gaussian processes for regression, (2016 International Joint Conference on Neural Networks (2016)), 3338-3345
[9] Chorin, A. J.; Hald, O. H.; Kupferman, R., Optimal prediction and the Mori-Zwanzig representation of irreversible processes, Proc. Natl. Acad. Sci., 97, 7, 2968-2973 (2000) · Zbl 0968.60036
[10] Cockayne, J.; Oates, C.; Sullivan, T.; Girolami, M., Probabilistic meshless methods for partial differential equations and Bayesian inverse problems (2016), arXiv preprint
[11] Cockayne, J.; Oates, C.; Sullivan, T.; Girolami, M., Probabilistic numerical methods for PDE-constrained Bayesian inverse problems, (Proceedings of the 36th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering (2016))
[12] Cohn, D. A.; Ghahramani, Z.; Jordan, M. I., Active learning with statistical models, J. Artif. Intell. Res. (1996) · Zbl 0900.68366
[13] Diaconis, P., Bayesian numerical analysis, (Statistical Decision Theory and Related Topics IV, Vol. 1 (1988)), 163-175
[14] Dondelinger, F.; Husmeier, D.; Rogers, S.; Filippone, M., ODE parameter inference using adaptive gradient matching with Gaussian processes, (Aistats (2013)), 216-228
[15] Duvenaud, D.; Lloyd, J. R.; Grosse, R.; Tenenbaum, J. B.; Ghahramani, Z., Structure discovery in nonparametric regression through compositional kernel search (2013), arXiv preprint
[16] Fasshauer, G. E.; Ye, Q., A kernel-based collocation method for elliptic partial differential equations with random coefficients, (Monte Carlo and Quasi-Monte Carlo Methods 2012 (2013), Springer), 331-347 · Zbl 1302.65025
[17] Franke, C.; Schaback, R., Solving partial differential equations by collocation using radial basis functions, Appl. Math. Comput., 93, 1, 73-82 (1998) · Zbl 0943.65133
[18] Graepel, T., Solving noisy linear operator equations by Gaussian processes: application to ordinary and partial differential equations, (ICML (2003)), 234-241
[19] Hensman, J.; Fusi, N.; Lawrence, N. D., Gaussian processes for big data (2013), arXiv preprint
[20] Kaipio, J.; Somersalo, E., Statistical and Computational Inverse Problems, Vol. 160 (2006), Springer Science & Business Media
[21] Kennedy, M. C.; O’Hagan, A., Predicting the output from a complex computer code when fast approximations are available, Biometrika, 87, 1, 1-13 (2000) · Zbl 0974.62024
[22] Kharazmi, E.; Zayernouri, M.; Karniadakis, G. E., Petrov-Galerkin and spectral collocation methods for distributed order differential equations (2016), arXiv preprint
[23] Krause, A.; Guestrin, C., Nonmyopic active learning of Gaussian processes: an exploration-exploitation approach, (Proceedings of the 24th International Conference on Machine Learning (2007), ACM), 449-456
[24] Lawrence, N., Probabilistic non-linear principal component analysis with Gaussian process latent variable models, J. Mach. Learn. Res., 6, Nov, 1783-1816 (2005) · Zbl 1222.68247
[25] Lawrence, N. D., Gaussian process latent variable models for visualisation of high dimensional data, Adv. Neural Inf. Process. Syst., 16, 3, 329-336 (2004)
[26] Lawrence, N. D.; Sanguinetti, G.; Rattray, M., Modelling transcriptional regulation using Gaussian processes, (Advances in Neural Information Processing Systems (2006)), 785-792
[27] Liu, D. C.; Nocedal, J., On the limited memory BFGS method for large scale optimization, Math. Program., 45, 1-3, 503-528 (1989) · Zbl 0696.90048
[28] MacKay, D. J., Information-based objective functions for active data selection, Neural Comput., 4, 4, 590-604 (1992)
[29] Melkumyan, A., Operator induced multi-task Gaussian processes for solving differential equations, (Neural Information Processing Systems (NIPS) Workshop: New Directions in Multiple Kernel Learning (2012))
[30] Murphy, K. P., Machine Learning: A Probabilistic Perspective (2012), MIT Press · Zbl 1295.68003
[31] Neal, R. M., Bayesian Learning for Neural Networks, Vol. 118 (2012), Springer Science & Business Media
[32] Osborne, M. A.; Roberts, S. J.; Rogers, A.; Ramchurn, S. D.; Jennings, N. R., Towards real-time information processing of sensor network data using computationally efficient multi-output Gaussian processes, (Proceedings of the 7th International Conference on Information Processing in Sensor Networks (2008), IEEE Computer Society), 109-120
[33] Owhadi, H., Bayesian numerical homogenization, Multiscale Model. Simul., 13, 3, 812-828 (2015) · Zbl 1322.35002
[34] Perkins, T. J.; Jaeger, J.; Reinitz, J.; Glass, L., Reverse engineering the gap gene network of drosophila melanogaster, PLoS Comput. Biol., 2, 5, e51 (2006)
[35] Podlubny, I., Fractional Differential Equations: An Introduction to Fractional Derivatives, Fractional Differential Equations, to Methods of Their Solution and Some of Their Applications, vol. 198 (1998), Academic Press
[36] Poggio, T.; Girosi, F., Networks for approximation and learning, Proc. IEEE, 78, 9, 1481-1497 (1990) · Zbl 1226.92005
[37] Poustelnikova, E.; Pisarev, A.; Blagov, M.; Samsonova, M.; Reinitz, J., A database for management of gene expression data in situ, Bioinformatics, 20, 14, 2212-2221 (2004)
[38] Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Inferring solutions of differential equations using noisy multi-fidelity data, J. Comput. Phys., 335, 736-746 (2017) · Zbl 1382.65229
[39] Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Numerical Gaussian processes for time-dependent and non-linear partial differential equations (2017), arXiv preprint
[40] Rasmussen, C. E.; Ghahramani, Z., Occam’s razor, (Advances in Neural Information Processing Systems (2001)), 294-300
[41] Rudy, S. H.; Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Data-driven discovery of partial differential equations, Sci. Adv., 3, 4, Article e1602614 pp. (2017)
[42] Saitoh, S., Theory of Reproducing Kernels and Its Applications, vol. 189 (1988), Longman
[43] Särkkä, S., Linear operators and stochastic partial differential equations in Gaussian process regression, (International Conference on Artificial Neural Networks (2011), Springer), 151-158
[44] Schölkopf, B.; Smola, A. J., Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (2002), MIT Press
[45] Snelson, E.; Ghahramani, Z., Sparse Gaussian processes using pseudo-inputs, (Advances in Neural Information Processing Systems (2005)), 1257-1264
[46] Stuart, A. M., Inverse problems: a Bayesian perspective, Acta Numer., 19, 451-559 (2010) · Zbl 1242.65142
[47] Tikhonov, A., Solution of incorrectly formulated problems and the regularization method, Sov. Math. Dokl., 5, 1035-1038 (1963) · Zbl 0141.11001
[48] Tikhonov, A. N.; Arsenin, V. Y., Solutions of Ill-Posed Problems (1977), W.H. Winston · Zbl 0354.65028
[49] Tipping, M. E., Sparse Bayesian learning and the relevance vector machine, J. Mach. Learn. Res., 1, 211-244 (2001) · Zbl 0997.68109
[50] Titsias, M. K.; Lawrence, N. D., Bayesian Gaussian process latent variable model, (Aistats (2010)), 844-851
[51] Vapnik, V., The Nature of Statistical Learning Theory (2013), Springer Science & Business Media · Zbl 0934.62009
[52] Williams, C. K.; Rasmussen, C. E., Gaussian Processes for Machine Learning (2006), The MIT Press, 2(3), 4 · Zbl 1177.68165
[53] Zwanzig, R., Ensemble method in the theory of irreversibility, J. Chem. Phys., 33, 5, 1338-1341 (1960)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.