×

Gaussian process hyper-parameter estimation using parallel asymptotically independent Markov sampling. (English) Zbl 1466.62073

Summary: Gaussian process emulators of computationally expensive computer codes provide fast statistical approximations to model physical processes. The training of these surrogates depends on the set of design points chosen to run the simulator. Due to computational cost, such training set is bound to be limited and quantifying the resulting uncertainty in the hyper-parameters of the emulator by uni-modal distributions is likely to induce bias. In order to quantify this uncertainty, this paper proposes a computationally efficient sampler based on an extension of Asymptotically Independent Markov Sampling, a recently developed algorithm for Bayesian inference. Structural uncertainty of the emulator is obtained as a by-product of the Bayesian treatment of the hyper-parameters. Additionally, the user can choose to perform stochastic optimisation to sample from a neighbourhood of the Maximum a Posteriori estimate, even in the presence of multimodality. Model uncertainty is also acknowledged through numerical stabilisation measures by including a nugget term in the formulation of the probability model. The efficiency of the proposed sampler is illustrated in examples where multi-modal distributions are encountered. For the purpose of reproducibility, further development, and use in other applications the code used to generate the examples is freely available for download at https://github.com/agarbuno/paims_codes.

MSC:

62-08 Computational methods for problems pertaining to statistics

Software:

gprege; GitHub
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Andrianakis, Y.; Challenor, P. G., Parameter estimation for Gaussian process emulators. technical report, (2011), Managing Uncertainty in Complex Models
[2] Andrianakis, I.; Challenor, P. G., The effect of the nugget on Gaussian process emulators of computer models, Comput. Statist. Data Anal., 56, 12, 4215-4228, (2012) · Zbl 1255.62306
[3] Andrieu, C.; Doucet, A.; Holenstein, R., Particle Markov chain Monte Carlo methods, J. R. Stat. Soc. Ser. B Stat. Methodol., 72, 3, 269-342, (2010)
[4] Bastos, L. S.; O’Hagan, A., Diagnostics for Gaussian process emulators, Technometrics, 51, 4, 425-438, (2009)
[5] Beck, J.; Zuev, K. M., Asymptotically independent Markov sampling: a new MCMC scheme for Bayesian inference, Int. J. Uncertain. Quant., 3, 5, (2013)
[6] Berger, J. O.; Bernardo, J. M., On the development of reference priors, Bayesian Stat., 4, 4, (1992)
[7] Berger, J. O.; Bernardo, J. M.; Sun, D., The formal definition of reference priors, Ann. Statist., 37, 2, 905-938, (2009) · Zbl 1162.62013
[8] Berger, J. O.; Liseo, B.; Wolpert, R. L., Integrated likelihood methods for eliminating nuisance parameters, Statist. Sci., 14, 1, 1-28, (1999) · Zbl 1059.62521
[9] Ching, J.; Chen, Y.-C., Transitional Markov chain Monte Carlo method for Bayesian model updating, model class selection and model averaging, J. Eng. Mech., 133, 7, 816-832, (2007)
[10] Cressie, N., (Statistics for Spatial Data, Wiley Series in Probability and Statistics, (1993), Wiley)
[11] Del Moral, P.; Doucet, A.; Jasra, A., Sequential Monte Carlo samplers, J. R. Stat. Soc. Ser. B, 68, 3, 411-436, (2006) · Zbl 1105.62034
[12] Del Moral, P.; Doucet, A.; Jasra, A., On adaptive resampling strategies for sequential Monte Carlo methods, Bernoulli, 18, 1, 252-278, (2012) · Zbl 1236.60072
[13] De Oliveira, V., Objective Bayesian analysis of spatial data with measurement error, Canad. J. Statist., 35, 2, 283-301, (2007) · Zbl 1129.62086
[14] Draper, D., Assessment and propagation of model uncertainty, J. R. Stat. Soc. Ser. B, 57, 1, 45-97, (1995) · Zbl 0812.62001
[15] Fearnhead, P.; Taylor, B. M., An adaptive sequential Monte Carlo sampler, Bayesian Anal., 8, 2, 411-438, (2013) · Zbl 1329.62055
[16] Forrester, A. I.J.; Sóbester, A.; Keane, A. J., Engineering design via surrogate modelling, (2008)
[17] Gibbs, M. N., Bayesian Gaussian processes for regression and classification, (1998), Citeseer, (Ph.D. thesis)
[18] Golub, G.; Van Loan, C., (Matrix Computations, Johns Hopkins Studies in the Mathematical Sciences, (1996), Johns Hopkins University Press)
[19] Gramacy, R. B.; Polson, N. G., Particle learning of Gaussian process models for sequential design and optimization, J. Comput. Graph. Statist., 20, 1, 18, (2009)
[20] Haario, H.; Saksman, E.; Tamminen, J., An adaptive metropolis algorithm, Bernoulli, 7, 2, 223-242, (2001) · Zbl 0989.65004
[21] Hankin, R., Introducing BACCO, an R bundle for Bayesian analysis of computer code output, J. Stat. Softw., 14, 16, (2005)
[22] Kalaitzis, A. A.; Lawrence, N. D., A simple approach to ranking differentially expressed gene expression time courses through Gaussian process regression, BMC Bioinformatics, 12, 180, (2011)
[23] Kennedy, M. C.; O’Hagan, A., Bayesian calibration of computer models, J. R. Stat. Soc. Ser. B Stat. Methodol., 63, 3, 425-464, (2001) · Zbl 1007.62021
[24] Kennedy, M. C.; O’Hagan, A., Supplementary details on Bayesian calibration of computer models. technical report, (2001)
[25] Kirkpatrick, S., Optimization by simulated annealing, J. Stat. Phys., 34, 5-6, 975-986, (1983)
[26] Liu, J., (Monte Carlo Strategies in Scientific Computing, Springer Series in Statistics, (2008), Springer) · Zbl 1132.65003
[27] MacKay, D. J.C., Hyperparameters: optimize, or integrate out?, (Maximum Entropy and Bayesian Methods, (1996)), 43-59 · Zbl 0895.62030
[28] MacKay, D. J., Introduction to Monte Carlo methods, (Learning in Graphical Models, (1998), Springer), 175-204 · Zbl 0911.65004
[29] Mira, A., On metropolis-Hastings algorithms with delayed rejection, Metron, 59, 3-4, 231-241, (2001) · Zbl 0998.65502
[30] Neal, R. M., Probabilistic inference using Markov chain Monte Carlo methods. technical report CRG-TR-93-1, 1-144, (1993)
[31] Neal, R. M., Sampling from multimodal distributions using tempered transitions, Stat. Comput., 6, 4, 353-366, (1996)
[32] Neal, R. M., Monte Carlo implementation of gaussian process models for Bayesian regression and classification. technical report 9702, (1997)
[33] Neal, R. M., Regression and classification using Gaussian process priors, Bayesian Stat., 6, (1998)
[34] Neal, R. M., Annealed importance sampling, Stat. Comput., 11, 2, 125-139, (2001)
[35] Neal, R. M., Slice sampling, Ann. Statist., 31, 3, (2003) · Zbl 1051.65007
[36] Neal, R. M., MCMC using Hamiltonian dynamics, (Handbook of Markov Chain Monte Carlo, (2011)) · Zbl 1229.65018
[37] Nilson, T.; Kuusk, A., A reflectance model for the homogeneous plant canopy and its inversion, Remote Sens. Environ., 27, 2, 157-167, (1989)
[38] Nocedal, J.; Wright, S. J., Numerical optimization, (2004), Springer
[39] Oakley, J., Bayesian uncertainty analysis for complex computer codes, (1999), (Ph.D. thesis)
[40] Oakley, J., Eliciting Gaussian process priors for complex computer codes, J. R. Stat. Soc. Ser. D: Stat., 51, 1, 81-97, (2002)
[41] Paulo, R., Default priors for Gaussian processes, Ann. Statist., 33, 2, 556-582, (2005) · Zbl 1069.62030
[42] Ranjan, P.; Haynes, R.; Karsten, R., A computationally stable approach to Gaussian process interpolation of deterministic computer simulation data, Technometrics, 53, 4, 366-378, (2011)
[43] Rasmussen, C. E.; Williams, C. K.I., Gaussian processes for machine learning, (2006), MIT Press · Zbl 1177.68165
[44] Robert, C.; Casella, G., Monte Carlo statistical methods, (2004), Springer Science & Business Media · Zbl 1096.62003
[45] Schneider, J.; Kirkpatrick, S., (Stochastic Optimization, Scientific Computation, (2007), Springer Berlin, Heidelberg)
[46] Taflanidis, A. A.; Beck, J. L., Stochastic subset optimization for optimal reliability problems, Probab. Eng. Mech., 23, 2-3, 324-338, (2008)
[47] Taflanidis, A. A.; Beck, J. L., An efficient framework for optimal robust stochastic system design using stochastic simulation, Comput. Methods Appl. Mech. Engrg., 198, 1, 88-101, (2008) · Zbl 1194.74260
[48] Vernon, I.; Goldstein, M.; Bower, R. G., Galaxy formation: a Bayesian uncertainty analysis, Bayesian Anal., 5, 4, 619-669, (2010) · Zbl 1330.85005
[49] Wilkinson, R.D., 2014. Accelerating ABC methods using Gaussian processes. arXiv preprint.
[50] Williams, C. K.I.; Rasmussen, C. E., Gaussian processes for regression, Adv. Neural Inf. Process. Syst., 514-520, (1996)
[51] Zuev, K. M.; Beck, J. L., Global optimization using the asymptotically independent Markov sampling method, Comput. Struct., 126, 107-119, (2013)
[52] Zuev, K. M.; Katafygiotis, L. S., Modified metropolis-Hastings algorithm with delayed rejection, Probab. Eng. Mech., 26, 3, 405-412, (2011)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.