×

Workshop on statistical approaches for the evaluation of complex computer models. (English) Zbl 1032.62102

Summary: As decision- and policy-makers come to rely increasingly on estimates and simulations produced by computerized models of the world, in areas as diverse as climate prediction, transportation planning, economic policy and civil engineering, the need for objective evaluation of the accuracy and utility of such models likewise becomes more urgent.
This article summarizes a two-day workshop that took place in Santa Fe, New Mexico, in December 1999, whose focus was the evaluation of complex computer models. Approximately half of the workshop was taken up with formal presentation of four computer models by their creators, each paired with an initial assessment by a statistician. These prepared papers are presented, in shortened form, in Section 3 of this paper. The remainder of the workshop was devoted to introductory and summary comments, short contributed descriptions of related models and a great deal of floor discussion, which was recorded by assigned rapporteurs. These are presented in Sections 2 and 4 in the paper. In the introductory and concluding sections we attempt to summarize the progress made by the workshop and suggest next steps.

MSC:

62Pxx Applications of statistics

Software:

bootstrap
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] BAy ARRI, M. J. and BERGER, J. O. (1999). Quantifying surprise in the data and model verification. In Bayesian Statistics VI (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 53-82. Oxford Univ. Press. · Zbl 0974.62021
[2] BEy ER, W. E. P., DEBRUIJN, I. A., PALACHE, A. M., WESTEN · Zbl 1014.68134 · doi:10.1023/A:1015059928466
[3] DORP, R. G. J. and OSTERHAUS, A. D. M. E. (1998). The plea against annual influenza vaccination? ‘The Hoskins’ Paradox’ revisited. Vaccine 16 1929-1932.
[4] BOSSERT, J., REISNER, J. M., LINN R. R., WINTERKAMP, · Zbl 1072.91583 · doi:10.1007/s355-002-8328-x
[5] J. L., SCHAUB, R. and RIGGAN, P. J. (1998). Validation of coupled atmosphere-fire behavior models. Preprints of the 14th Conference on Fire and Forest Meteorology, November 16-20, 1998, Luso-Coimbra, Portugal.
[6] BOX, G. E. P. and DRAPER, N. (1987). Empirical Model Building and Response Surfaces. Wiley, New York. · Zbl 0614.62104
[7] BRIEMAN, L. (2001). Statistical modeling: The two cultures. Statist. Sci. 16 199-231. · Zbl 1059.62505 · doi:10.1214/ss/1009213726
[8] CLARK, T. L., RADKE, L., COEN, J. and MIDDLETON, D. (1999). Analy sis of small-scale convective dy namics in a crown fire using infrared video camera imagery. J. Appl. Meteor. 38 1401-1420.
[9] CLICK, S. and ROUPHAIL, N. (1999). Lane group level field evaluation of computer-based signal timing models. Paper presented at the 78th Annual Meeting of the Transportation Research Board, Washington, DC, January 1999.
[10] CORSIM USER’S MANUAL. (1997). FHWA, U.S. Department of Transportation, Office of Safety and Traffic Operation R&D, Intelligent Sy stems and Technology Division, McLean, VA.
[11] EFRON, B. and TIBSHIRANI, R. (1993). An Introduction to the Bootstrap. Chapman and Hall, London. · Zbl 0835.62038
[12] GOLDBERG, D. E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading, MA. · Zbl 0721.68056
[13] HOSKINS, T. W., DAVIS, J. R., SMITH, A. J., MILLER, C. L. and · Zbl 1030.46054
[14] ALLCHIN, A. (1979). Assessment of inactivated influenza-A vaccine after three outbreaks of influenza-A at Christ’s Hospital. The Lancet 1 (8106) 33-35.
[15] KEITEL, W. A., CATE, T. R., COUCH, R. B., HUGGINS, L. · Zbl 1016.81051 · doi:10.1023/A:1010976119400
[16] L. and HESS, K. R. (1997). Continued efficacy of annual immunization with inactivated influenza virus vaccine over a five year period: A placebo-controlled trial in healthy adults. Vaccine 15 1114-1122.
[17] KENNEDY, M. and O’HAGAN, A. (2000). Bayesian calibration of computer models. J. Roy. Statist. Soc. Ser. B 63 425-464. JSTOR: · Zbl 1007.62021 · doi:10.1111/1467-9868.00294
[18] KIEFER, J. and WOLFOWITZ, J. (1952). Stochastic estimation of the maximum in a regression. Ann. Math. Statist. 23 462-466. · Zbl 0049.36601 · doi:10.1214/aoms/1177729392
[19] KUSHNER, H. and YIN, G. (1997). Stochastic Approximation Algorithms and Applications. Springer, New York. · Zbl 0914.60006
[20] LINN, R. R. (1997). Transport model for prediction of wildfire behavior. Los Alamos National Laboratory Scientific Report LA13334-T.
[21] MILLER, R. G. (1966). Simultaneous Statistical Inference. McGraw Hill, New York. · Zbl 0192.25702
[22] PERELSON, A. S. and OSTER, G. F. (1979). Theoretical studies of clonal selection: minimal antibody repertoire size and reliability of self-nonself discrimination. J. Theoret. Biol. 81 645-670. · doi:10.1016/0022-5193(79)90275-3
[23] SCHOENBERG, F., BERK, R., FOVELL, R., LI, C., LU, R. and · Zbl 1037.62083 · doi:10.1023/A:1022494523519
[24] WEISS, R. (2001). Approximation and inversion of a complex meteorological sy stem via local linear filters. J. Appl. Meteor. 40 446-458.
[25] SMITH, D. J., FORREST, S., ACKLEY, D. H. and PERELSON, A. S. · Zbl 0995.62057 · doi:10.1080/02331880210932
[26] . Variable efficacy of repeated annual influenza vaccination. Proc. Natl. Acad. Sci. USA 96 14001-14006.
[27] VON STORCH, H. and NAVARRA, A., eds. (1999). Analy sis of Climate Variability: Applications of Statistical Techniques. Springer, New York.
[28] ZEIGLER, B. (1976). Theory of Modelling and Simulation. Wiley, New York. · Zbl 0352.68122
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.