×

Parameter and uncertainty estimation for dynamical systems using surrogate stochastic processes. (English) Zbl 1420.60047

Summary: Inference on unknown quantities in dynamical systems via observational data is essential for providing meaningful insight, furnishing accurate predictions, enabling robust control, and establishing appropriate designs for future experiments. Merging mathematical theory with empirical measurements in a statistically coherent way is critical and challenges abound, e.g., ill-posedness of the parameter estimation problem, proper regularization and incorporation of prior knowledge, and computational limitations. To address these issues, we propose a new method for learning parameterized dynamical systems from data. We first customize and fit a surrogate stochastic process directly to observational data, front-loading with statistical learning to respect prior knowledge (e.g., smoothness), cope with challenging data features like heteroskedasticity, heavy tails, and censoring. Then, samples of the stochastic process are used as “surrogate data” and point estimates are computed via ordinary point estimation methods in a modular fashion. Attractive features of this two-step approach include modularity and trivial parallelizability. We demonstrate its advantages on a predator-prey simulation study and on a real-world application involving within-host influenza virus infection data paired with a viral kinetic model, with comparisons to a more conventional Markov chain Monte Carlo (MCMC) based Bayesian approach.

MSC:

60G15 Gaussian processes
62F10 Point estimation
62F15 Bayesian inference
65L09 Numerical solution of inverse problems involving ordinary differential equations
65L05 Numerical methods for initial value problems involving ordinary differential equations
92-08 Computational methods for problems pertaining to biology
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] B. Ankenman, B. L. Nelson, and J. Staum, Stochastic kriging for simulation metamodeling, Oper. Res., 58 (2010), pp. 371-382. · Zbl 1342.62134
[2] M. Asch, M. Bocquet, and M. Nodet, Data Assimilation: Methods, Algorithms, and Applications, Fundam. Algorithms 11, SIAM, Philadelphia, 2016, https://doi.org/10.1137/1.9781611974546. · Zbl 1361.93001
[3] R. Aster, B. Borchers, and C. Thurber, Parameter Estimation and Inverse Problems, 2nd ed., Academic Press, Waltham, MA, 2012. · Zbl 1088.35081
[4] E. Baake, M. Baake, H. G. Bock, and K. M. Briggs, Fitting ordinary differential equations to chaotic data, Phys. Rev. A, 45 (1992), 5524.
[5] S. Bandara, J. Schlöder, R. Eils, H. Bock, and T. Meyer, Optimal experimental design for parameter estimation of a cell signaling model, PLoS Comput. Biol., 5 (2009), e1000558.
[6] J. M. Bardsley, A. Solonen, H. Haario, and M. Laine, Randomize-then-optimize: A method for sampling from posterior distributions in nonlinear inverse problems, SIAM J. Sci. Comput., 36 (2014), pp. A1895-A1910, https://doi.org/10.1137/140964023. · Zbl 1303.65003
[7] M. Binois and R. B. Gramacy, hetGP: Heteroskedastic Gaussian Process Modeling and Design under Replication, R package version 1.0.1, 2017.
[8] M. Binois, R. B. Gramacy, and M. Ludkovski, Practical Heteroskedastic Gaussian Process Modeling for Large Simulation Experiments, preprint, https://arxiv.org/abs/1611.05902, 2016. · Zbl 07498993
[9] H. Bock and K. Plitt, A multiple shooting algorithm for direct solution of optimal control problems, in Proceedings of the 9th IFAC World Congress, Budapest, Hungary, 1984, pp. 243-247.
[10] H. G. Bock, Recent advances in parameter identification techniques for O.D.E., in Numerical Treatment of Inverse Problems in Differential and Integral Equations, Progr. Sci. Comput. 2, Birkhäuser Boston, Boston, MA, 1983, pp. 95-121. · Zbl 0516.65067
[11] A. Boukouvalas, D. Cornford, and M. Stehlik, Optimal design for correlated processes with input-dependent noise, Comput. Statist. Data Anal., 71 (2014), pp. 1088-1102. · Zbl 1471.62031
[12] G. Burgers, P. Jan van Leeuwen, and G. Evensen, Analysis scheme in the ensemble Kalman filter, Monthly Weather Rev., 126 (1998), pp. 1719-1724.
[13] B. Calderhead, M. Girolami, and N. D. Lawrence, Accelerating Bayesian inference over nonlinear differential equations with Gaussian processes, in Proceedings of the International Conference on Advances in Neural Information Processing Systems, 2009, pp. 217-224.
[14] D. Calvetti and E. Somersalo, An Introduction to Bayesian Scientific Computing, Springer-Verlag, New York, 2007. · Zbl 1137.65010
[15] D. Calvetti and E. Somersalo, Computational Mathematical Modeling: An Integrated Approach Across Scales, Math Model. Comput. 17, SIAM, Philadelphia, 2012. · Zbl 1285.00035
[16] M. Caracotsios and W. E. Stewart, Sensitivity analysis of initial value problems with mixed ODEs and algebraic equations, Comput. Chem. Engrg., 9 (1985), pp. 359-365.
[17] Y. Chen and D. S. Oliver, Ensemble randomized maximum likelihood method as an iterative ensemble smoother, Math. Geosci., 44 (2012), pp. 1-26.
[18] M. Chung and E. Haber, Experimental design for biological systems, SIAM J. Control Optim., 50 (2012), pp. 471-489, https://doi.org/10.1137/100791063. · Zbl 1243.93130
[19] M. Chung, J. Krueger, and M. Pop, Robust parameter estimation for biological systems: A study on the dynamics of microbial communities, Math. Biosci., 294 (2017), pp. 71-84. · Zbl 1380.92044
[20] M. Conrad, C. Hubold, B. Fischer, and A. Peters, Modeling the hypothalamus-pituitary-adrenal system: Homeostasis by interacting positive and negative feedback, J. Biolog. Phys., 35 (2009), pp. 149-162.
[21] E. M. Constantinescu, A. Sandu, T. Chai, and G. R. Carmichael, Ensemble-based chemical data assimilation. I: General approach, Quart. J. Roy. Meteorolog. Soc., 133 (2007), pp. 1229-1243.
[22] P. Courtier, J.-N. Thépaut, and A. Hollingsworth, A strategy for operational implementation of \(4\) D-var, using an incremental approach, Quart. J. Roy. Meteorolog. Soc., 120 (1994), pp. 1367-1387.
[23] F. Dondelinger, D. Husmeier, S. Rogers, and M. Filippone, ODE parameter inference using adaptive gradient matching with Gaussian processes, in Proceedings of the 16th International Conferenceon Artificial Intelligence and Statistics, Scottsdale, AZ, 2013, pp. 216-228.
[24] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap, CRC Press, Boca Raton, FL, 1994.
[25] H. W. Engl, M. Hanke, and A. Neubauer, Regularization of Inverse Problems, Math. Appl. 375, Springer, Cham, 1996. · Zbl 0859.65054
[26] B. L. Fridley and P. Dixon, Data augmentation for a Bayesian spatial model involving censored observations, Environmetrics, 18 (2007), pp. 107-123.
[27] B. Göbel, K. M. Oltmanns, and M. Chung, Linking neuronal brain activity to the glucose metabolism, Theor. Biol. Med. Model., 10 (2013), 50.
[28] G. Goel, I.-C. Chou, and E. O. Voit, System estimation from metabolic time-series data, Bioinformatics, 24 (2008), pp. 2505-2511.
[29] P. W. Goldberg, C. K. Williams, and C. M. Bishop, Regression with input-dependent noise: A Gaussian process treatment, in Advances in Neural Information Processing Systems, Vol. 10, MIT Press, Cambridge, MA, 1998, pp. 493-499.
[30] J. Gong, G. Wahba, D. R. Johnson, and J. Tribbia, Adaptive tuning of numerical weather prediction models: Simultaneous estimation of weighting, smoothing, and physical parameters, Monthly Weather Rev., 126 (1998), pp. 210-231.
[31] N. S. Gorbach, S. Bauer, and J. M. Buhmann, Mean-Field Variational Inference for Gradient Matching with Gaussian Processes, preprint, https://arxiv.org/abs/1610.06949, 2016.
[32] J. Hadamard, Lectures on Cauchy’s Problem in Linear Differential Equations, Yale University Press, New Haven, CT, 1923. · JFM 49.0725.04
[33] G. Hairer, S. Nørsett, and E. Wanner, Solving Ordinary Differential Equations I: Nonstiff Problems, 2nd ed., Springer, Berlin, 1993. · Zbl 0789.65048
[34] G. Hairer and E. Wanner, Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, 2nd ed., Springer, Berlin, 1996. · Zbl 0859.65067
[35] P. C. Hansen, Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion, Math. Model. Comput. 4, SIAM, Philadelphia, 1998, https://doi.org/10.1137/1.9780898719697. · Zbl 0890.65037
[36] W. K. Hastings, Monte Carlo sampling methods using Markov chains and their applications, Biometrika, 57 (1970), pp. 97-109. · Zbl 0219.65008
[37] D. Higdon, M. Kennedy, J. C. Cavendish, J. A. Cafeo, and R. D. Ryne, Combining field data and computer simulations for calibration and prediction, SIAM J. Sci. Comput., 26 (2004), pp. 448-466, https://doi.org/10.1137/S1064827503426693. · Zbl 1072.62018
[38] J. Kennedy and R. Eberhart, Particle swarm optimization, in Proceedings of the 1995 IEEE International Conference on Neural Networks, Vol. 4, IEEE, Washington, DC, 1995, pp. 1942-1948.
[39] M. Kennedy and A. O’Hagan, Bayesian calibration of computer models, J. R. Stat. Soc. Ser. B Stat. Methodol., 63 (2001), pp. 425-464. · Zbl 1007.62021
[40] K. Kersting, C. Plagemann, P. Pfaff, and W. Burgard, Most likely heteroscedastic Gaussian process regression, in Proceedings of the International Conference on Machine Learning, ACM, New York, 2007, pp. 393-400.
[41] S. Kirkpatrick, C. Gelatt, Jr., and M. Vecchi, Optimization by simulated annealing, Science, 220 (1983), pp. 671-680. · Zbl 1225.90162
[42] K. Law, A. Stuart, and K. Zygalakis, Data Assimilation: A Mathematical Introduction, Springer, Cham, 2015. · Zbl 1353.60002
[43] B. A. J. Lawson, C. C. Drovandi, N. Cusimano, P. Burrage, B. Rodriguez, and K. Burrage, Unlocking data sets by calibrating populations of models to data density: A study in atrial electrophysiology, Sci. Adv., 4 (2018), e1701676.
[44] F.-X. Le Dimet and O. Talagrand, Variational algorithms for analysis and assimilation of meteorological observations: Theoretical aspects, Tellus A Dynam. Meteorol. Oceanogr., 38 (1986), pp. 97-110.
[45] A. C. Lorenc, The potential of the ensemble Kalman filter for NWP–a comparison with \(4\) D-var, Quart. J. Roy. Meteorolog. Soc., 129 (2003), pp. 3183-3203.
[46] S. Marino, I. B. Hogue, C. J. Ray, and D. E. Kirschner, A methodology for performing global uncertainty and sensitivity analysis in systems biology, J. Theoret. Biol., 254 (2008), pp. 178-196. · Zbl 1400.92013
[47] D. E. Morris, J. E. Oakley, and J. A. Crowe, A web-based tool for eliciting probability distributions from experts, Environ. Model. Softw., 52 (2014), pp. 1-4.
[48] J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed., Springer-Verlag, New York, 2006. · Zbl 1104.65059
[49] V. Picheny and D. Ginsbourger, A nonstationary space-time Gaussian process model for partially converged simulations, SIAM/ASA J. Uncertain. Quantif., 1 (2013), pp. 57-78, https://doi.org/10.1137/120882834. · Zbl 1291.60075
[50] J. Ramsay, Principal differential analysis: Data reduction by differential operators, J. Roy. Statist. Soc. Ser. B, 58 (1996), pp. 495-508. · Zbl 0853.62043
[51] J. O. Ramsay, G. Hooker, D. Campbell, and J. Cao, Parameter estimation for differential equations: A generalized smoothing approach, J. R. Stat. Soc. Ser. B Stat. Methodol., 69 (2007), pp. 741-796. · Zbl 07555374
[52] C. E. Rasmussen and C. Williams, Gaussian Processes for Machine Learning, MIT Press, Cambridge, MA, 2006, http://www.gaussianprocess.org/gpml/. · Zbl 1177.68165
[53] A. Saltelli, M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, and S. Tarantola, Global Sensitivity Analysis: The Primer, John Wiley & Sons, Chichester, UK, 2008. · Zbl 1161.00304
[54] T. J. Santner, B. J. Williams, and W. I. Notz, The Design and Analysis of Computer Experiments, Springer, New York, 2013.
[55] A. Shah, A. Wilson, and Z. Ghahramani, Student-t processes as alternatives to Gaussian processes, in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2014, pp. 877-885.
[56] B. W. Silverman, Some aspects of the spline smoothing approach to nonparametric curve fitting, J. Roy. Statist. Soc. Ser. B, 47 (1985), pp. 1-52. · Zbl 0606.62038
[57] D. Simon, Evolutionary Optimization Algorithms, John Wiley & Sons, Hoboken, NJ, 2013. · Zbl 1280.68008
[58] A. Smith, F. Adler, J. McAuley, R. Gutenkunst, R. Ribeiro, J. McCullers, and A. Perelson, Effect of 1918 PB1-F2 expression on influenza A virus infection kinetics, PLoS Comput. Biol., 7 (2011), e1001081.
[59] A. Smith, F. Adler, and A. Perelson, An accurate two-phase approximate solution to an acute viral infection model, J. Math. Biol., 60 (2010), pp. 711-726. · Zbl 1198.92029
[60] A. Smith, F. Adler, R. Ribeiro, R. Gutenkunst, J. McAuley, J. McCullers, and A. Perelson, Kinetics of coinfection with influenza A virus and Streptococcus pneumoniae, PLoS Pathog., 9 (2013), e1003238.
[61] A. Smith and A. Smith, A critical, nonlinear threshold dictates bacterial invasion and initial kinetics during influenza, Sci. Rep., 6 (2016), 38703.
[62] A. M. Smith, Host-pathogen kinetics during influenza infection and coinfection: Insights from predictive modeling, Immuno. Rev., 285 (2018), pp. 97-112.
[63] A. P. Smith, D. J. Moquin, V. Bernhauerova, and A. M. Smith, Influenza virus infection model with density dependence supports biphasic viral decay, Front. Microbiol., 9 (2018), 1554.
[64] R. C. Smith, Uncertainty Quantification: Theory, Implementation, and Applications, Comput. Sci. Engrg. 12, SIAM, Philadelphia, 2013.
[65] J. Stoer and R. Bulirsch, Introduction to Numerical Analysis, 3rd ed., Springer, New York, 2002. · Zbl 1004.65001
[66] D. S. Stoffer and K. D. Wall, Bootstrapping state-space models: Gaussian maximum likelihood estimation and the Kalman filter, J. Amer. Statist. Assoc., 86 (1991), pp. 1024-1033. · Zbl 0850.62693
[67] J. Taubenberger and D. Morens, The pathology of influenza virus infections, Annu. Rev. Pathol., 3 (2008), pp. 499-522.
[68] L. Tenorio, An Introduction to Data Analysis and Uncertainty Quantification for Inverse Problems, Math. Industry 3, SIAM, Philadelphia, 2017, https://doi.org/10.1137/1.9781611974928. · Zbl 1435.65003
[69] W. Thompson, D. Shay, E. Weintraub, L. Brammer, N. Bridges, C.B. Cox, and K. Fukuda, Influenza-associated hospitalizations in the United States, J. Amer. Med. Assoc., 292 (2004), pp. 1333-1340.
[70] R. Torrence, Bayesian Parameter Estimation on Three Models of Influenza, Master’s thesis, Virgina Tech, Blacksburg, VA, 2017.
[71] C. R. Vogel, Computational Methods for Inverse Problems, Front. Appl. Math. 23, SIAM, Philadelphia, 2002, https://doi.org/10.1137/1.9780898717570. · Zbl 1008.65103
[72] E. O. Voit, A First Course in Systems Biology, Garland Science, New York, 2013.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.