×

zbMATH — the first resource for mathematics

Sensitivity analysis of model output. An investigation of new techniques. (English) Zbl 0875.62200
Summary: Sensitivity Analysis (SA) of model output investigates the relationship between the predictions of a model, possibly implemented in a computer program, and its input parameters. Such an analysis is relevant for a number of practices, including quality assurance of models and codes, and the identification of crucial regions in the parameter space. This note compares established techniques with variants, such as a modified version of the Hora and Iman importance measure (SANDIA Laboratory Report SAND85-2839, 1989), or new methods, such as the iterated fractional factorial design (Andres, Hajas, Report in prep. for AECL, Pinawa, Canada, 1991). Comparison is made on the basis of method reproducibility and of method accuracy. The former is a measure of how well SA predictions are replicated when repeating the analysis on different samples taken from the same input parameters space. The latter deals with the physical correctness of the SA results. The present article is a sequel to an earlier study in this journal (Saltelli, Homma, Comp. Stat. and Data Anal. 13 (1), 73-94 (1992)) on limitations in existing SA techniques, where the inadequacy of existing schemes to deal with non-monotonic relationships within the model was pointed out. International benchmark test models were taken from the field of performance analysis of nuclear waste disposal in geological formations. The results based on these models show that the modified version of the Hora and Iman method proposed in this paper is extremely robust, when compared with the other existing statistics, even in the presence of model non-monotonicities. This importatice measure is accurate, although its application is demanding - in terms of computer time - for system with very large numbers of input parameters. Its predictions are also among the most reproducible. The newly proposed iterated fractional factorial design appears to score the best in reproducibility. The accuracy of this latter method demands further investigation.

MSC:
62G99 Nonparametric inference
65C99 Probabilistic methods, stochastic differential equations
Software:
LISA
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Andres, T. H.: Statistical sampling strategies. Proceedings of uncertainty analysis for performance assessments of radioactive waste disposal systems (February 24–26, 1987)
[2] T.H. Andres. User manual for SAMPLE. In preparation as a report for AECL, Pinawa, Canada R0E 1L0
[3] T.H. Andres and W.C. Hajas. Sensitivity analysis of the SYVAC3-CC3 model of a nuclear fuel waste disposal using iterated fractional factorial design. In preparation as a report for AECL, Pinawa, Canada R0E 1L0.
[4] Carlyle, S. G.: The activities, objectives and recent achievements of the NEA probabilistic system assessment codes user group. Proceedings of waste management ’87 (1987)
[5] Conover, W. J.: Practical non-parametric statistics. (1980)
[6] Goodwin, B. W.; Andres, T. H.; Davis, P. A.; Leneveu, D. M.; Melnyk, T. W.; Sherman, G. R.; Wuschke, D. M.: Post-closure environmental assessment for the canadian nuclear fuel waste management program. Radioactive waste management and the nuclear fuel cycle 8, No. 2–3, 241-272 (1987)
[7] Helton, J. C.; Iman, R. L.; Johnson, J. D.; Leigh, C. D.: Uncertainty and sensitivity analysis of a dry containment test problem for the MAEROS aerosol model. Nucl. sci. Eng. 102, 22-42 (1989)
[8] Hora, S. C.; Iman, R. L.: A comparison of maximum/bounding and Bayesian/Monte Carlo for fault tree uncertainty analysis. SANDIA laboratory report SAND85-2839 (1989)
[9] Homma, T.; Saltelli, A.: PREP (Statistical pre-processor) preparation of input sample for Monte Carlo simulation. Program description and user guide (1991)
[10] Iman, R. L.; Conover, W. J.: The use of rank transform in regression. Technometrics 21, No. 4, 499-509 (1979)
[11] Iman, R. L.; Davenport, J. M.; Frost, E. L.; Shortnecarier, M. J.: Stepwise regression with PRESS and rank regression. (1980)
[12] Iman, R. L.; Helton, J. C.; Campbell, J. E.: An approach to sensitivity analysis of computer models, parts I and II. Journal of quality technology 13, No. 3,4, 232-240 (1981)
[13] Iman, R. L.; Davenport, J. M.: Rank correlation plots for use with correlated input variables. Comm. statist. Simulation comput. 11, No. 3, 335-360 (1982)
[14] Iman, R. L.; Helton, J. C.: An investigation of uncertainty and sensitivity analysis techniques for computer models. Risk analysis 8, 71-90 (1988)
[15] Iman, R. L.; Helton, J. C.: A comparison of uncertainty and sensitivity analysis techniques for computer models. Sandia natl. Laboratories report NUREG/CR-3904, SAND, 84-1461 (1985)
[16] Iman, R. L.; Shortencarier, M. J.; Johnson, J. D.: A Fortran 77 program and user’s guide for the calculation of partial correlation and standardized regression coefficients. (1985)
[17] Ishigami, T.; Homma, T.: An importance quantification technique in uncertainty analysis. Japan atomic energy research institute report JAERI-M, 89-111 (1989)
[18] . Dictionary of scientific and technical terms (1978)
[19] . (1987)
[20] . (1989)
[21] Prado, P.; Saltelli, A.; Homma, T.: LISA package user guide. Part II. LISA. Program description and user guide. CEC/JRC nuclear science and technology report EUR 13923/EN (1991)
[22] Robinson, P.: Probabilistic system assessment codes group. (February 1989)
[23] Saltelli, A.; Marivoet, J.: Performances of nonparametric statistics in sensitivity analysis and parameter ranking. CEC/JRC nuclear science and technology report EUR 10851 EN (1986)
[24] Saltelli, A.; Marivoet, J.: Safety assessment for nuclear waste disposal. Some observations about actual risk calculations. Radioactive waste management and the nuclear fuel cycle 9, No. 4 (1988)
[25] Saltelli, A.: The role of the code intercomparison exercise: activities of the probabilistic system assessment codes group. Proceedings of the ispra course on risk analysis in nuclear waste management, 69-95 (1989)
[26] Saltelli, A.: Techniques for uncertainty and sensitivity analyses. Proceedings of the ispra course on risk analysis in nuclear waste management, 129-160 (1989)
[27] Saltelli, A.; Homma, T.: LISA package user guide. Part III. SPOP. Uncertainty and sensitivity analysis for model output. Program description and user guide. CEC/JRC nuclear science and technology report EUR 13924/EN (1991)
[28] Saltelli, A.; Andres, T. H.; Goodwin, B. W.; Sartori, E.; Carlyle, S. G.; Cronhjort, B.: PSACOIN level 0 intercomparison; an international verification exercise on a hypothetical safety assessment case study. Proceedings of the twenty-second annual hawaii conference on system sciences (January 3–6 1989)
[29] Saltelli, A.; Homma, T.: Sensitivity analysis for model output. Performance of black box techniques on three international benchmark exercises. Computational statistics and data analysis 13, No. 1, 73-94 (1992)
[30] Saltelli, A.; Marivoet, J.: Nonparametric statistics in sensitivity analysis for model output; a comparison of selected techniques. Reliability engineering and system safety 28, 229-253 (1990)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.