Abstract
Addressing safety concerns in commercial nuclear power plants (NPPs) often requires the use of advanced modeling and simulation (M&S) in association with the probabilistic risk assessment (PRA). Advanced M&S are also needed to accelerate the analysis, design, licensing, and operationalization of advanced nuclear reactors. However, before a simulation model can be used for PRA, its validity must be adequately established. The objective of this research is to develop a systematic and scientifically justifiable validation methodology, namely, probabilistic validation (PV), to facilitate the validity evaluation (especially when validation data are not sufficiently available) of advanced simulation models that are used for PRA in support of risk-informed decision-making and regulation. This paper is the first in a series of two papers related to PV that provides the theoretical foundation and methodological platform. The second paper applies the PV methodological platform for a case study of fire PRA of NPPs. Although the PV methodology is explained in the context of PRA of the nuclear industry, it is grounded on a cross-disciplinary review of literature and so applicable to validation of simulation models, in general, not necessarily associated with PRA or nuclear applications.
Issue Section:
Research Papers
References
1.
Bui
,
H.
,
Sakurahara
,
T.
,
Pence
,
J.
,
Reihani
,
S.
,
Kee
,
E.
, and
Mohaghegh
,
Z.
, 2019
, “
An Algorithm for Enhancing Spatiotemporal Resolution of Probabilistic Risk Assessment to Address Emergent Safety Concerns in Nuclear Power Plants
,” Reliab. Eng. Syst. Saf.
,
185
, pp. 405
–28
.10.1016/j.ress.2019.01.0042.
Oberkampf
,
W. L.
, and
Smith
,
B. L.
, 2017
, “
Assessment Criteria for Computational Fluid Dynamics Model Validation Experiments
,” ASME J. Verif. Valid. Uncertainty Quantif.
,
2
(3
), 031002.10.1115/1.40378873.
Oberkampf
,
W. L.
, 2019
, “
Simulation Accuracy, Uncertainty, and Predictive Capability: A Physical Sciences Perspective
,” Computer Simulation Validation
,
Springer
, Cham, pp. 69
–97
.4.
National Research Council,
2012
, Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification
,
National Academies Press
, Washington, DC.5.
Sakurahara
,
T.
,
Reihani
,
S.
,
Kee
,
E.
, and
Mohaghegh
,
Z.
, 2019
, “
Global Importance Measure Methodology for Integrated Probabilistic Risk Assessment
,” Proc. Inst. Mech. Eng., Part O J. Risk Reliab.
,
234
(2
), pp. 377
–396
.10.1177/1748006X198793166.
Biersdorf
,
J.
,
Bui
,
H.
,
Sakurahara
,
T.
,
Reihani
,
S.
,
LaFleur
,
C.
,
Luxat
,
D.
, Prescott, S., and Mohaghegh, Z., 2020
, Risk Importance Ranking of Fire Data Parameters to Enhance Fire PRA Model Realism
,
Idaho National Laboratory
, Idaho Falls, ID.7.
Bui
,
H.
,
Sakurahara
,
T.
,
Reihani
,
S.
,
Biersdorf
,
J.
, and
Mohaghegh
,
Z.
, 2020
, “
I-PRA Uncertainty Importance Ranking to Enhance Fire PRA Realism for Nuclear Power Plants
,” Transactions of American Nuclear Society (ANS) Winter Meeting and Nuclear Technology Expo
,
Virtual Meeting
, Nov. 16–19, 2020.8.
Sakurahara
,
T.
,
Mohaghegh
,
Z.
,
Reihani
,
S.
,
Kee
,
E.
,
Brandyberry
,
M.
, and
Rodgers
,
S.
, 2018
, “
An Integrated Methodology for Spatio-Temporal Incorporation of Underlying Failure Mechanisms Into Fire Probabilistic Risk Assessment of Nuclear Power Plants
,” Reliab. Eng. Syst. Saf.
,
169
, pp. 242
–257
.10.1016/j.ress.2017.09.0019.
Mohaghegh
,
Z.
,
Kee
,
E.
,
Reihani
,
S.
,
Kazemi
,
R.
,
Johnson
,
D.
,
Grantom
,
R.
, Fleming, K., et al., 2013
, “
Risk-Informed Resolution of Generic Safety Issue 191
,” International Topical Meeting on Probabilistic Safety Assessment and Analysis
, Columbia, SC, Sept. 22
–26
.10.
Sakurahara
,
T.
,
Schumock
,
G.
,
Reihani
,
S.
,
Kee
,
E.
, and
Mohaghegh
,
Z.
, 2019
, “
Simulation-Informed Probabilistic Methodology for Common Cause Failure Analysis
,” Reliab. Eng. Syst. Saf.
,
185
, pp. 84
–99
.10.1016/j.ress.2018.12.00711.
Sakurahara
,
T.
,
Mohaghegh
,
Z.
,
Reihani
,
S.
, and
Kee
,
E.
, 2018
, “
Methodological and Practical Comparison of Integrated Probabilistic Risk Assessment (I-PRA) With the Existing Fire PRA of Nuclear Power Plants
,” Nucl. Technol.
,
204
, pp. 354
–77
.10.1080/00295450.2018.148615912.
Bui
,
H.
,
Sakurahara
,
T.
,
Reihani
,
S.
,
Kee
,
E.
, and
Mohaghegh
,
Z.
, 2020
, “
Spatiotemporal Integration of an Agent-Based First Responder Performance Model With a Fire Hazard Propagation Model for Probabilistic Risk Assessment of Nuclear Power Plants
,” ASCE-ASME J. Risk Uncertainty Eng. Syst.: Part B Mech. Eng.
,
6
(1
), p. 011011
.https://ascelibrary.org/doi/abs/10.1115/1.404479313.
Lin
,
S.
,
Barson
,
S.
, and
Sindir
,
M.
, 1992
, “
Development of Evaluation Criteria and a Procedure for Assessing Predictive Capability and Code Performance
,” Advanced Earth-to-Orbit Propulsion Technology Conference
, NASA George C. Marshall Space Flight Center, AL, May 19–21.14.
Cosner
,
R.
, 1995
, “
CFD Validation Requirements for Technology Transition
,” Fluid Dynamics Conference
, San Diego, CA, June 19–22, p. 2227.10.2514/6.1995-222715.
Marvin
,
J. G.
, 1995
, “
Perspective on Computational Fluid Dynamics Validation
,” AIAA J.
,
33
, pp. 1778
–1787
.10.2514/3.1272716.
Oberkampf
,
W. L.
,
Sindir
,
M.
, and
Conlisk
,
A.
, 1998
, Guide for the Verification and Validation of Computational Fluid Dynamics Simulations
,
American Institute of Aeronautics and Astronautics
,
Reston, VA
.17.
Urbina
,
A.
,
Mahadevan
,
S.
, and
Paez
,
T. L.
, 2011
, “
Quantification of Margins and Uncertainties of Complex Systems in the Presence of Aleatoric and Epistemic Uncertainty
,” Reliab. Eng. Syst. Saf.
,
96
, pp. 1114
–1125
.10.1016/j.ress.2010.08.01018.
Sankararaman
,
S.
, and
Mahadevan
,
S.
, 2015
, “
Integration of Model Verification, Validation, and Calibration for Uncertainty Quantification in Engineering Systems
,” Reliab. Eng. Syst. Saf.
,
138
, pp. 194
–209
.10.1016/j.ress.2015.01.02319.
Roy
,
C. J.
, and
Oberkampf
,
W. L.
, 2011
, “
A Comprehensive Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing
,” Comput. Methods Appl. Mech. Eng.
,
200
, pp. 2131
–2144
.10.1016/j.cma.2011.03.01620.
Choudhary
,
A.
,
Voyles
,
I. T.
,
Roy
,
C. J.
,
Oberkampf
,
W. L.
, and
Patil
,
M.
, 2016
, “
Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem
,” ASME J. Verif. Valid. Uncertainty Quantif.
,
1
(1
), p. 011003
.10.1115/1.403128521.
Schlesinger
,
S.
,
Crosbie
,
R.
,
Gagne
,
R.
,
Innis
,
G.
,
Lalwani
,
C. S.
,
Loch
,
J.
, Sylvester, R., et al., 1979
, “
Terminology for Model Credibility
,” Simulation
,
32
, pp. 103
–104
.10.1177/00375497790320030422.
Institute of Electrical and Electronics Engineers,
1991
, “
IEEE Standard Glossary of Software Engineering Terminology
,” Institute of Electrical and Electronics Engineers, New York, Standard No. IEEE STD 610.12-1990.23.
Institute of Electrical and Electronics Engineers,
1984
, “
IEEE Standard Dictionary of Electrical and Electronics Terms
,” IEEE, New York, Standard No. ANSI/IEEE STD 100
–1984
.24.
U.S. Department of Defense,
1994
, “
Directive No. 5000.59: Modeling and Simulation Management
,”
U.S. Department of Defense
, VA.25.
American Society of Mechanical Engineers
, 2006
, “Guide for Verification and Validation in Computational Solid Mechanics
,”
American Society of Mechanical Engineers
, New York, Standard No. ASME V&V 10
–2006
.26.
American Society of Mechanical Engineers,
2009
, “Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer
,”
American Society of Mechanical Engineers
, New York, Standard No. ASME V&V 20
–2009
.27.
American Society of Mechanical Engineers
, 2019
, “Standard for Verification and Validation in Computational Solid Mechanics
,”
American Society of Mechanical Engineers
, New York, Standard No. ASME V&V 10
–2019
.28.
Oberkampf
,
W. L.
, and
Roy
,
C. J.
, 2010
, Verification and Validation in Scientific Computing
,
Cambridge University Press
, New York.29.
Beisbart
,
C.
, 2019
, “
What is Validation of Computer Simulations? Toward a Clarification of the Concept of Validation and of Related Notions
,” Computer Simulation Validation
,
Springer
, Cham, Switzerland, pp. 35
–67
.30.
Roache
,
P. J.
, 2009
, “
Perspective: Validation—What Does It Mean?
” ASME J. Fluids Eng.
,
131
(3
), 034503.10.1115/1.307713431.
Danquah
,
B.
,
Riedmaier
,
S.
, and
Lienkamp
,
M.
, 2020
, “
Potential of Statistical Model Verification, Validation and Uncertainty Quantification in Automotive Vehicle Dynamics Simulations: A Review
,” Veh. Syst. Dyn., Int. J. Vehicle Mech. Mobility
, 60(4), pp. 1
–30
.10.1080/00423114.2020.185431732.
Trivedi
,
C.
, and
Dahlhaug
,
O. G.
, 2019
, “
A Comprehensive Review of Verification and Validation Techniques Applied to Hydraulic Turbines
,” Int. J. Fluid Mach. Syst.
,
12
, pp. 345
–367
.10.5293/IJFMS.2019.12.4.34533.
Lee
,
G.
,
Kim
,
W.
,
Oh
,
H.
,
Youn
,
B. D.
, and
Kim
,
N. H.
, 2019
, “
Review of Statistical Model Calibration and Validation—From the Perspective of Uncertainty Structures
,” Struct. Multidiscip. Optim.
,
60
, pp. 1619
–1644
.10.1007/s00158-019-02270-234.
Borg
,
M.
,
Englund
,
C.
,
Wnuk
,
K.
,
Duran
,
B.
,
Levandowski
,
C.
,
Gao
,
S.
, Tan, Y., et al., 2018
, “
Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry
,” arXiv eprint arXiv:1812.05389
.10.48550/arXiv.1812.0538935.
Goerlandt
,
F.
,
Khakzad
,
N.
, and
Reniers
,
G.
, 2017
, “
Validity and Validation of Safety-Related Quantitative Risk Analysis: A Review
,” Saf. Sci.
,
99
, pp. 127
–39
.10.1016/j.ssci.2016.08.02336.
Durst
,
P. J.
,
Anderson
,
D. T.
, and
Bethel
,
C. L.
, 2017
, “
A Historical Review of the Development of Verification and Validation Theories for Simulation Models
,” Int. J. Model., Simul., Sci. Comput.
,
8
, p. 1730001
.10.1142/S179396231730001137.
Tsioptsias
,
N.
,
Tako
,
A.
, and
Robinson
,
S.
, 2016
, “
Model Validation and Testing in Simulation: A Literature Review
,” Fifth Student Conference on Operational Research (SCOR 2016): Schloss Dagstuhl-Leibniz-Zentrum Fuer Informatik
, Open Access Series in Informatics, Dagstuhl Publishing, Germany.38.
Kutluay
,
E.
, and
Winner
,
H.
, 2014
, “
Validation of Vehicle Dynamics Simulation Models – A Review
,” Veh. Syst. Dyn.
,
52
, pp. 186
–200
.10.1080/00423114.2013.86850039.
Franke
,
J.
, 2010
, “
A Review of Verification and Validation in Relation to CWE
,” Proceedings Fifth International Symposium on Computational Wind Engineering (CWE2010)
,
Chapel Hill
,
NC
, May 23
–27
.40.
Yeo
,
D.
, and
Yeo
,
D.
, 2020
, “
A Summary of Industrial Verification, Validation, and Uncertainty Quantification Procedures in Computational Fluid Dynamics
,” US Department of Commerce
,
National Institute of Standards and Technology
, Gaithersburg, MD.41.
Sargent
,
R. G.
, and
Balci
,
O.
, 2017
, “
History of Verification and Validation of Simulation Models
,” Winter Simulation Conference (WSC
), Las Vegas, NV, Dec. 3–6, pp. 292
–307
.http://simulation.su/uploads/files/default/2017-sargentbalci.pdf42.
Mehta
,
U. B.
,
Eklund
,
D. R.
,
Romero
,
V. J.
,
Pearce
,
J. A.
, and
Keim
,
N. S.
, 2016
, “
Simulation Credibility: Advances in Verification, Validation, and Uncertainty Quantification
,” NASA/TP—2016–219422, National Aeronautics and Space Administration, Ames Research Center, Moffett Field, CA.43.
Sargent
,
R. G.
, 2010
, “
Verification and Validation of Simulation Models
,” Proceedings of the Winter Simulation Conference,
Baltimore, MD, Dec. 5–8, pp. 166
–183
.44.
Freitas
,
C. J.
, 2020
, “
Standards and Methods for Verification, Validation, and Uncertainty Assessments in Modeling and Simulation
,” ASME J. Verif. Valid. Uncertainty Quantif.
,
5
(2
), p. 021001
.10.1115/1.404727445.
American Society of Mechanical Engineers,
2012
, “
An Illustration of the Concepts of Verification and Validation in Computational Solid Mechanics
,”
American Society of Mechanical Engineers
,
New York
, Standard No. ASME V&V 10.1-2012.46.
Coleman
,
H.
, and
Stern
,
F.
, 1997
, “
Uncertainties and CFD Code Validation
,” ASME J. Fluids Eng.
,
119
(4
), pp. 795
–803
.10.1115/1.281950047.
Oberkampf
,
W. L.
, and
Ferson
,
S.
, 2007
, “
Model Validation Under Both Aleatory and Epistemic Uncertainty
,” NATO/RTO Applied Vehicle Technology Panel, Symposium on Computational Uncertainty in Military Vehicle Design,
Athens, Greece
, Dec 3–6, 2007.48.
Ferson
,
S.
,
Oberkampf
,
W. L.
, and
Ginzburg
,
L.
, 2008
, “
Model Validation and Predictive Capability for the Thermal Challenge Problem
,” Comput. Methods Appl. Mech. Eng.
,
197
, pp. 2408
–2430
.10.1016/j.cma.2007.07.03049.
Roy
,
C. J.
, 2019
, “
Errors and Uncertainties: Their Sources and Treatment
,” Computer Simulation Validation
,
Springer
, Cham, Switzerland, pp. 119
–141
.50.
Kennedy
,
M. C.
, and
O'Hagan
,
A.
, 2001
, “
Bayesian Calibration of Computer Models
,” J. R. Stat. Soc. Ser. B (Stat. Methodol.)
,
63
, pp. 425
–464
.10.1111/1467-9868.0029451.
Subramanian
,
A.
, and
Mahadevan
,
S.
, 2020
, “
Model Error Propagation in Coupled Multiphysics Systems
,” AIAA J.
,
58
, pp. 2236
–2245
.10.2514/1.J05849652.
Li
,
W.
,
Chen
,
S.
,
Jiang
,
Z.
,
Apley
,
D. W.
,
Lu
,
Z.
, and
Chen
,
W.
, 2016
, “
Integrating Bayesian Calibration, Bias Correction, and Machine Learning for the 2014 Sandia Verification and Validation Challenge Problem
,” ASME J. Verif. Valid. Uncertainty Quantif.
,
1
(1
), p. 011004
.10.1115/1.403198353.
Mullins
,
J.
, and
Mahadevan
,
S.
, 2016
, “
Bayesian Uncertainty Integration for Model Calibration, Validation, and Prediction
,” ASME J. Verif. Valid. Uncertainty Quantif.
,
1
(1
), p. 011006
.10.1115/1.403237154.
Gray
,
A.
, and
Patelli
,
E.
, 2019
, “
Towards a Bayesian Framework for Model Validation
,” 13th International Conference on Applications ofStatistics and Probability in Civil Engineering (ICASP13
), Seoul, South Korea, May 26
–30
.https://www.researchgate.net/publication/333001516_Towards_a_Bayesian_framework_for_model_validation55.
Bhattacharyya
,
R.
, and
Mahadevan
,
S.
, 2022
, “
Calibration and Validation of Multiscale Model for Ultimate Strength Prediction of Composite Laminates Under Uncertainty
,” ASCE-ASME J. Risk Uncert. Eng. Syst. Part B Mech. Eng.
,
8
(2
), p. 021205
.10.1115/1.405306056.
Wang
,
J.-X.
,
Roy
,
C. J.
, and
Xiao
,
H.
, 2017
, “Propagation of Input Uncertainty in Presence of Model-Form
Uncertainty: A Multifidelity Approach for Computational Fluid Dynamics Applications
,” ASCE-ASME J. Risk Uncertainty Eng. Syst. Part B Mech. Eng.
,
4
(1
), p. 011002
.10.1115/1.403745257.
Kwag
,
S.
,
Gupta
,
A.
, and
Dinh
,
N.
, 2018
, “
Probabilistic Risk Assessment Based Model Validation Method Using Bayesian Network
,” Reliab. Eng. Syst. Saf.
,
169
, pp. 380
–393
.10.1016/j.ress.2017.09.01358.
Bodda
,
S. S.
,
Gupta
,
A.
, and
Dinh
,
N.
, 2020
, “
Enhancement of Risk Informed Validation Framework for External Hazard Scenario
,” Reliab. Eng. Syst. Saf.
,
204
(2020
), p. 107140
.10.1016/j.ress.2020.10714059.
Beisbart
,
C.
, and
Saam
,
N. J.
, 2019
, Computer Simulation Validation: Fundamental Concepts, Methodological Frameworks, and Philosophical Perspectives
,
Springer
, Cham, Switzerland.60.
Jiang
,
X.
,
Mahadevan
,
S.
, and
Urbina
,
A.
, 2010
, “
Bayesian Nonlinear Structural Equation Modeling for Hierarchical Validation of Dynamical Systems
,” Mech. Syst. Signal Process.
,
24
, pp. 957
–75
.10.1016/j.ymssp.2009.10.00261.
U.S. Nuclear Regulatory Commission,
2017
, Guidance on the Treatment of Uncertainties Associated With PRAs in Risk-Informed Decisionmaking (NUREG-1855, Revision 1)
,
Office of Nuclear Regulatory Research, U.S. Nuclear Regulatory Commission
, Washington, DC.62.
Crespo
,
L. G.
, and
Kenny
,
S. P.
, 2022
, “
Synthetic Validation of Responses to the NASA Langley Challenge on Optimization Under Uncertainty
,” Mech. Syst. Signal Process.
,
164
, p. 108253
.10.1016/j.ymssp.2021.10825363.
Droguett
,
E. L.
, 1999
, “
Methodology for the Treatment of Model Uncertainty
,” Doctoral dissertation,
University of Maryland
,
College Park, MD
.64.
Zhou
,
T.
,
Modarres
,
M.
, and
Droguett
,
E. L.
, 2019
, “
Multi-Unit Risk Aggregation With Consideration of Uncertainty and Bias in Risk Metrics
,” Reliab. Eng. Syst. Saf.
,
188
, pp. 473
–482
.10.1016/j.ress.2019.04.00165.
American Society for Testing and Materials,
2011
, “
ASTM Standard Guide for Evaluating the Predictive Capability of Deterministic Fire Models
,” ASTM, New York, Standard No. E1355-11.66.
U.S. Nuclear Regulatory Commission,
2018
, “
Regulatory Guide 1.174 (Revision 3): An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis
,” U.S. Nuclear Regulatory Commission, Washington, DC.67.
Morris
,
M. D.
, 1991
, “
Factorial Sampling Plans for Preliminary Computational Experiments
,” Technometrics
,
33
, pp. 161
–174
.10.1080/00401706.1991.1048480468.
Bossel
,
H.
, 1994
, Modeling and Simulation
,
A.K. Peters
,
Wellesley, MA
.69.
Hill, D.R.C.
, 2002, “
Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems
Second Edition by B. P. Zeigler, H. Praehofer, T. G. Kim, Academic Press, San Diego, CA, 2000,” Int. J. Robust Nonlinear Control
, 12, pp. 91
–92
.https://doi.org/10.1002/rnc.61070.
Law
,
A. M.
, 2007
, Simulation Modeling and Analysis
, 4th ed., McGraw-Hill New York.71.
Alvin
,
K.
,
Oberkampf
,
W.
,
Diegert
,
K.
, and
Rutherford
,
B.
, 1998
, “
Uncertainty Quantification in Computational Structural Dynamics: A New Paradigm for Model Validation
,” 16th International Modal Analysis Conference
,
Society for Experimental Mechanics
, Santa Barbara, CA, Feb 2–5, pp. 1191
–1198
.72.
Oberkampf
,
W. L.
,
DeLand
,
S. M.
,
Rutherford
,
B. M.
,
Diegert
,
K. V.
, and
Alvin
,
K. F.
, 2000
, “
Estimation of Total Uncertainty in Modeling and Simulation
,” Sandia Albuquerque, NM, Report No. SAND2000-0824.73.
Oberkampf
,
W. L.
,
DeLand
,
S. M.
,
Rutherford
,
B. M.
,
Diegert
,
K. V.
, and
Alvin
,
K. F.
, 2002
, “
Error and Uncertainty in Modeling and Simulation
,” Reliab. Eng. Syst. Saf.
,
75
, pp. 333
–357
.10.1016/S0951-8320(01)00120-X74.
Zhu
,
T.
, 2016
, Model Uncertainty Analysis in Operational Risk Analysis
, Master thesis, Department of Production and Quality Engineering, Norwegian University of Science and Technology (
NTNU)
, Norway.75.
Isukapalli
,
S.
, 1999
, “
Uncertainty Analysis of Transport-Transformation Models
,” Doctoral dissertation, Graduate Program in Chemical and Biochemical Engineering, Graduate School—New Brunswick Rutgers, The State University of New Jersey.76.
Refsgaard
,
J. C.
,
van der Sluijs
,
J. P.
,
Højberg
,
A. L.
, and
Vanrolleghem
,
P. A.
, 2007
, “
Uncertainty in the Environmental Modelling Process – A Framework and Guidance
,” Environ. Modell. Software
,
22
, pp. 1543
–1556
.10.1016/j.envsoft.2007.02.00477.
Bock
,
C.
,
Dandashi
,
F.
,
Friedenthal
,
S.
,
Harrison
,
N.
,
Jenkins
,
S.
,
McGinnis
,
L.
, Sztipanovits, J., et al., 2017
, “
Conceptual Modeling,” in Research Challenges in Modeling and Simulation for Engineering Complex Systems
, Simulation Foundations, Methods and Applications, Springer International Publishing AG, pp. 23
–44
.78.
Gallegos
,
D. P.
, and
Bonano
,
E. J.
, 1993
, “
Consideration of Uncertainty in the Performance Assessment of Radioactive Waste Disposal From an International Regulatory Perspective
,” Reliab. Eng. Syst. Saf.
,
42
, pp. 111
–123
.10.1016/0951-8320(93)90085-D79.
Wang
,
W.
, and
Brooks
,
R. J.
, 2007
, “
Improving the Understanding of Conceptual Modelling
,” J. Simul.
,
1
, pp. 153
–158
.10.1057/palgrave.jos.425002480.
Robinson
,
S.
, 2006
, “
Conceptual Modeling for Simulation: Issues and Research Requirements
,” Proceedings of the Winter Simulation Conference
, Monterey, CA, Dec 3–6, pp. 792
–800
.10.1109/WSC.2006.32316081.
Robinson
,
S.
,
Arbez
,
G.
,
Birta
,
L. G.
,
Tolk
,
A.
, and
Wagner
,
G.
, 2015
, “
Conceptual Modeling: Definition, Purpose and Benefits
,” Winter Simulation Conference
, Huntington Beach, CA, Dec. 6–9, pp. 2015
–2026
.10.1109/WSC.2015.740838682.
McGrattan
,
K.
,
Hostikka
,
S.
,
McDermott
,
R.
,
Floyd
,
J.
,
Weinschenk
,
C.
, and
Overholt
,
K.
, 2013
, “
Fire Dynamics Simulator Technical Reference Guide Volume 1: Mathematical Model
,” NIST Special Publication
, 1018, 6th Ed., Gaithersburg, MD.83.
U.S. Nuclear Regulatory Commission,
2008
, A Phenomena Identification and Ranking Table (PIRT) Exercise for Nuclear Power Plant Fire Modeling Applications
,
U.S. Nuclear Regulatory Commission
,
Washington, DC
, Standard No. NUREG/CR-6978.84.
Pourgol-Mohammad
,
M.
,
Modarres
,
M.
, and
Mosleh
,
A.
, 2009
, “
Integrated Methodology for Thermal-Hydraulic Code Uncertainty Analysis With Application
,” Nucl. Technol.
,
165
, pp. 333
–359
.10.13182/NT165-33385.
Ontiveros
,
V.
,
Cartillier
,
A.
, and
Modarres
,
M.
, 2010
, “
An Integrated Methodology for Assessing Fire Simulation Code Uncertainty
,” Nucl. Sci. Eng.
,
166
, pp. 179
–201
.10.13182/NSE10-0586.
Gideon
,
S.
, 1978
, “
Estimating the Dimension of a Model
,” Ann. Stat.
,
6
(2
), pp. 461
–464
.https://www.jstor.org/stable/295888987.
Neath
,
A. A.
, and
Cavanaugh
,
J. E.
, 2011
, “
The Bayesian Information Criterion: Background, Derivation, and Applications
,” WIREs Comput. Stat.
,
4
(2
), pp. 199
–203
.https://doi.org/10.1002/wics.19988.
Ayyub
,
B. M.
, 2001
, Elicitation of Expert Opinions for Uncertainty and Risks
,
CRC Press
, Boca Raton, FL.89.
Ouchi
,
F.
, 2004
, “
A Literature Review on the Use of Expert Opinion in Probabilistic Risk Analysis
,” World Bank Policy Research Working Paper 3201, February 2004.90.
Kraan
,
B.
, and
Bedford
,
T.
, 2005
, “
Probabilistic Inversion of Expert Judgments in the Quantification of Model Uncertainty
,” Manage. Sci.
,
51
, pp. 995
–1006
.10.1287/mnsc.1050.037091.
Werner
,
C.
,
Bedford
,
T.
,
Cooke
,
R. M.
,
Hanea
,
A. M.
, and
Morales-Nápoles
,
O.
, 2017
, “
Expert Judgment for Dependence in Probabilistic Modelling: A Systematic Literature Review and Future Research Directions
,” Eur. J. Oper. Res.
,
258
, pp. 801
–819
.10.1016/j.ejor.2016.10.01892.
Jcgm
,
J.
, 2008
, “
Evaluation of Measurement Data—Guide to the Expression of Uncertainty in Measurement
,”
International Organization for Standardization
,
Geneva
, Switzerland, p. 134
.93.
Coleman
,
H. W.
, and
Steele
,
W. G.
, 2018
, “Experimentation, Validation, and Uncertainty Analysis for Engineers
, 4th ed.,
Wiley
, Hoboken, NJ.94.
Smith
,
R. C.
, 2013
, Uncertainty Quantification: Theory, Implementation, and Applications
,
Siam
, Philadelphia, PA.95.
Campolongo
,
F.
,
Cariboni
,
J.
, and
Saltelli
,
A.
, 2007
, “
An Effective Screening Design for Sensitivity Analysis of Large Models
,” Environ. Modell. Software
,
22
, pp. 1509
–1518
.10.1016/j.envsoft.2006.10.00496.
Saltelli
,
A.
,
Ratto
,
M.
,
Andres
,
T.
,
Campolongo
,
F.
,
Cariboni
,
J.
,
Gatelli
,
D.
, Saisana, M., et al., 2008
, Global Sensitivity Analysis: The Primer
,
Wiley
, West Sussex, UK.97.
Ge
,
Q.
, and
Menendez
,
M.
, 2017
, “
Extending Morris Method for Qualitative Global Sensitivity Analysis of Models With Dependent Inputs
,” Reliab. Eng. Syst. Saf.
,
162
, pp. 28
–39
.10.1016/j.ress.2017.01.01098.
Helton
,
J. C.
, and
Johnson
,
J. D.
, 2011
, “
Quantification of Margins and Uncertainties: Alternative Representations of Epistemic Uncertainty
,” Reliab. Eng. Syst. Saf.
,
96
, pp. 1034
–1052
.10.1016/j.ress.2011.02.01399.
Helton
,
J. C.
,
Johnson
,
J. D.
,
Oberkampf
,
W. L.
, and
Sallaberry
,
C. J.
, 2010
, “
Representation of Analysis Results Involving Aleatory and Epistemic Uncertainty
,” Int. J. Gen. Syst.
,
39
, pp. 605
–646
.10.1080/03081079.2010.486664100.
Helton
,
J. C.
,
Johnson
,
J. D.
, and
Oberkampf
,
W. L.
, 2004
, “
An Exploration of Alternative Approaches to the Representation of Uncertainty in Model Predictions
,” Reliab. Eng. Syst. Saf.
,
85
, pp. 39
–71
.10.1016/j.ress.2004.03.025101.
Electric Power Research Institute,
2008
, Treatment of Parameter and Model Uncertainty for Probabilistic Risk Assessments
,
Electric Power Research Institute
,
Palo Alto, CA
.102.
Knupp
,
P. M.
, and
Salari
,
K.
, 2003
, Verification of Computer Codes in Computational Science and Engineering
,
Chapman & Hall/CRC
,
Boca Raton, FL
.103.
Roache
,
P.
, 2003,
“
Error Bars for CFD
,” 41st Aerospace Sciences Meeting and Exhibit
,
Reno, NV
, Jan. 6–9.104.
Hoffman
,
F. O.
, and
Hammonds
,
J. S.
, 1994
, “
Propagation of Uncertainty in Risk Assessments: The Need to Distinguish Between Uncertainty Due to Lack of Knowledge and Uncertainty Due to Variability
,” Risk Analysis
,
14
, pp. 707
–712
.10.1111/j.1539-6924.1994.tb00281.x105.
Helton
,
J. C.
, and
Davis
,
F. J.
, 2003
, “
Latin Hypercube Sampling and the Propagation of Uncertainty in Analyses of Complex Systems
,” Reliab. Eng. Syst. Saf.
,
81
, pp. 23
–69
.10.1016/S0951-8320(03)00058-9106.
Rochman
,
D.
,
Koning
,
A. J.
,
van der Marck
,
S. C.
,
Hogenbirk
,
A.
, and
Sciolla
,
C. M.
, 2011
, “
Nuclear Data Uncertainty Propagation: Perturbation Vs. Monte Carlo
,” Ann. Nucl. Energy
,
38
, pp. 942
–952
.10.1016/j.anucene.2011.01.026107.
Hosder
,
S.
,
Walters
,
R.
, and
Perez
,
R.
, 2006
, “
A Non-Intrusive Polynomial Chaos Method for Uncertainty Propagation in CFD Simulations
,” AIAA
2006
–891
.10.2514/6.2006-891108.
Najm
,
H. N.
, 2009
, “
Uncertainty Quantification and Polynomial Chaos Techniques in Computational Fluid Dynamics
,” Annu. Rev. Fluid Mech.
,
41
, pp. 35
–52
.10.1146/annurev.fluid.010908.165248109.
Janssen
,
H.
, 2013
, “
Monte Carlo Based Uncertainty Analysis: Sampling Efficiency and Sampling Convergence
,” Reliab. Eng. Syst. Saf.
,
109
, pp. 123
–132
.10.1016/j.ress.2012.08.003110.
Faes
,
M.
,
Daub
,
M.
,
Marelli
,
S.
,
Patelli
,
E.
, and
Beer
,
M.
, 2021
, “
Engineering Analysis With Probability Boxes: A Review on Computational Methods
,” Structural Safety
. 93, p. 102092
.10.1016/j.strusafe.2021.102092111.
Liu
,
Y.
,
Chen
,
W.
,
Arendt
,
P.
, and
Huang
,
H.-Z.
, 2011
, “
Toward a Better Understanding of Model Validation Metrics
,” ASME J. Mech. Des.
,
133
(7
), p. 071005
.10.1115/1.4004223112.
Oberkampf
,
W. L.
, and
Barone
,
M. F.
, 2006
, “
Measures of Agreement Between Computation and Experiment: Validation Metrics
,” J. Comput. Phys.
,
217
, pp. 5
–36
.10.1016/j.jcp.2006.03.037113.
Chen
,
W.
,
Baghdasaryan
,
L.
,
Buranathiti
,
T.
, and
Cao
,
J.
, 2004
, “
Model Validation Via Uncertainty Propagation and Data Transformations
,” AIAA J.
,
42
, pp. 1406
–1415
.10.2514/1.491114.
Hills
,
R. G.
, and
Trucano
,
T. G.
, 2002
, “
Statistical Validation of Engineering and Scientific Models: A Maximum Likelihood Based Metric
,” Sandia National Laboratories, Albuquerque, NM/Livermore, CA, Report No. SAND2001-1783.115.
Hills
,
R. G.
, and
Leslie
,
I. H.
, 2003
, “
Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application
,” Sandia National Laboratories, Albuquerque, NM/Livermore, CA, Report No. SAND2003-0706.116.
Rebba
,
R.
, and
Mahadevan
,
S.
, 2008
, “
Computational Methods for Model Reliability Assessment
,” Reliab. Eng. Syst. Saf.
,
93
, pp. 1197
–1207
.10.1016/j.ress.2007.08.001117.
Rebba
,
R.
, and
Mahadevan
,
S.
, 2006
, “
Model Predictive Capability Assessment Under Uncertainty
,” AIAA J.
,
44
, pp. 2376
–2384
.10.2514/1.19103118.
Rebba
,
R.
,
Huang
,
S.
,
Liu
,
Y.
, and
Mahadevan
,
S.
, 2006
, “
Statistical Validation of Simulation Models
,” Int. J. Mater. Prod. Technol.
,
25
, pp. 164
–181
.10.1504/IJMPT.2006.008280119.
Mahadevan
,
S.
, and
Rebba
,
R.
, 2005
, “
Validation of Reliability Computational Models Using Bayes Networks
,” Reliab. Eng. Syst. Saf.
,
87
, pp. 223
–232
.10.1016/j.ress.2004.05.001120.
Ferson
,
S.
, and
Oberkampf
,
W. L.
, 2009
, “
Validation of Imprecise Probability Models
,” Int. J. Reliab. Saf.
,
3
, pp. 3
–22
.10.1504/IJRS.2009.026832121.
Mullins
,
J.
,
Ling
,
Y.
,
Mahadevan
,
S.
,
Sun
,
L.
, and
Strachan
,
A.
, 2016
, “
Separation of Aleatory and Epistemic Uncertainty in Probabilistic Model Validation
,” Reliab. Eng. Syst. Saf.
,
147
, pp. 49
–59
.10.1016/j.ress.2015.10.003122.
Li
,
W.
,
Chen
,
W.
,
Jiang
,
Z.
,
Lu
,
Z.
, and
Liu
,
Y.
, 2014
, “
New Validation Metrics for Models With Multiple Correlated Responses
,” Reliab. Eng. Syst. Saf.
,
127
, pp. 1
–11
.10.1016/j.ress.2014.02.002123.
Voyles
,
I. T.
, and
Roy
,
C. J.
, 2015
, “
Evaluation of Model Validation Techniques in the Presence of Aleatory and Epistemic Input Uncertainties
,” AIAA
Paper No. 2015
–1374
.10.2514/6.2015-1374124.
Pourgol-Mohammad
,
M.
,
Mosleh
,
A.
, and
Modarres
,
M.
, 2010
, “
Methodology for the Use of Experimental Data to Enhance Model Output Uncertainty Assessment in Thermal Hydraulics Codes
,” Reliab. Eng. Syst. Saf.
,
95
, pp. 77
–86
.10.1016/j.ress.2009.08.003125.
Lopez Droguett
,
E.
, and
Mosleh
,
A.
, 2014
, “
Bayesian Treatment of Model Uncertainty for Partially Applicable Models
,” Risk Anal.
,
34
, pp. 252
–270
.10.1111/risa.12121126.
Ferson
,
S.
,
Kreinovich
,
V.
,
Grinzburg
,
L.
,
Myers
,
D.
, and
Sentz
,
K.
, 2015
, “
Constructing Probability Boxes and Dempster-Shafer Structures
,” Report No. SAND2015-4166J, Origninally Published by Sandia National Laboratories as SAND2002-4015, January 2003.127.
Reinert
,
J. M.
, and
Apostolakis
,
G. E.
, 2006
, “
Including Model Uncertainty in Risk-Informed Decision Making
,” Ann. Nucl. Energy
,
33
, pp. 354
–369
.10.1016/j.anucene.2005.11.010128.
Iman
,
R. L.
, and
Conover
,
W. J.
, 1987
, “
A Measure of Top-Down Correlation
,” Technometrics
,
29
, pp. 351
–357
.129.
Saltelli
,
A.
, and
Marivoet
,
J.
, 1990
, “
Non-Parametric Statistics in Sensitivity Analysis for Model Output: A Comparison of Selected Techniques
,” Reliab. Eng. Syst. Saf.
,
28
, pp. 229
–253
.10.1016/0951-8320(90)90065-U130.
Helton
,
J. C.
, 1993
, “
Uncertainty and Sensitivity Analysis Techniques for Use in Performance Assessment for Radioactive Waste Disposal
,” Reliab. Eng. Syst. Saf.
,
42
, pp. 327
–367
.10.1016/0951-8320(93)90097-I131.
Hamby
,
D. M.
, 1994
, “
A Review of Techniques for Parameter Sensitivity Analysis of Environmental Models
,” Environ. Monit. Assess.
,
32
, pp. 135
–154
.10.1007/BF00547132132.
Borgonovo
,
E.
, 2006
, “
Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches
,” Risk Anal.
,
26
, pp. 1349
–1361
.10.1111/j.1539-6924.2006.00806.x133.
Helton
,
J. C.
,
Johnson
,
J. D.
,
Sallaberry
,
C. J.
, and
Storlie
,
C. B.
, 2006
, “
Survey of Sampling-Based Methods for Uncertainty and Sensitivity Analysis
,” Reliab. Eng. Syst. Saf.
,
91
, pp. 1175
–1209
.10.1016/j.ress.2005.11.017134.
Ikonen
,
T.
, and
Tulkki
,
V.
, 2014
, “
The Importance of Input Interactions in the Uncertainty and Sensitivity Analysis of Nuclear Fuel Behavior
,” Nucl. Eng. Des.
,
275
, pp. 229
–241
.10.1016/j.nucengdes.2014.05.015135.
Ikonen
,
T.
, 2016
, “
Comparison of Global Sensitivity Analysis Methods – Application to Fuel Behavior Modeling
,” Nucl. Eng. Des.
,
297
, pp. 72
–80
.10.1016/j.nucengdes.2015.11.025136.
Sobol′
,
I. M.
, 1990
, “
On Sensitivity Estimation for Nonlinear Mathematical Models
,” Mat. Model.
,
2
(1
), pp. 112
–118
.137.
Sobol
,
I. M.
, 1993
, “
Sensitivity Estimates for Nonlinear Mathematical Models
, Math. Model. Comput. Exp.
,
1
(4
), p. 407
.138.
Saltelli
,
A.
,
Tarantola
,
S.
, and
Campolongo
,
F.
, 2000
, “
Sensitivity Analysis as an Ingredient of Modeling
,” Stat. Sci.
,
15
(4
), pp. 377
–395
.https://www.jstor.org/stable/2676831139.
Saltelli
,
A.
, 2002
, “
Sensitivity Analysis for Importance Assessment
,” Risk Anal.
,
22
, pp. 579
–590
.10.1111/0272-4332.00040140.
Homma
,
T.
, and
Saltelli
,
A.
, 1996
, “
Importance Measures in Global Sensitivity Analysis of Nonlinear Models
,” Reliab. Eng. Syst. Saf.
,
52
, pp. 1
–17
.10.1016/0951-8320(96)00002-6141.
Borgonovo
,
E.
, 2007
, “
A New Uncertainty Importance Measure
,” Reliab. Eng. Syst. Saf.
,
92
, pp. 771
–784
.10.1016/j.ress.2006.04.015142.
Liu
,
Q.
, and
Homma
,
T.
, 2010
, “
A New Importance Measure for Sensitivity Analysis
,” J. Nucl. Sci. Technol.
,
47
, pp. 53
–61
.10.1080/18811248.2010.9711927143.
Pianosi
,
F.
, and
Wagener
,
T.
, 2015
, “
A Simple and Efficient Method for Global Sensitivity Analysis Based on Cumulative Distribution Functions
,” Environ. Modell. Software
,
67
, pp. 1
–11
.10.1016/j.envsoft.2015.01.004144.
Pianosi
,
F.
, and
Wagener
,
T.
, 2018
, “
Distribution-Based Sensitivity Analysis From a Generic Input-Output Sample
,” Environ. Modell. Software
,
108
, pp. 197
–207
.10.1016/j.envsoft.2018.07.019145.
Borgonovo
,
E.
,
Aliee
,
H.
,
Glaß
,
M.
, and
Teich
,
J.
, 2016
, “
A New Time-Independent Reliability Importance Measure
,” Eur. J. Oper. Res.
,
254
, pp. 427
–442
.10.1016/j.ejor.2016.03.054146.
Cui
,
L.
,
Lu
,
Z.
, and
Wang
,
Q.
, 2012
, “
Parametric Sensitivity Analysis of the Importance Measure
,” Mech. Syst. Signal Process.
,
28
, pp. 482
–491
.10.1016/j.ymssp.2011.10.015147.
Borgonovo
,
E.
,
Zentner
,
I.
,
Pellegri
,
A.
,
Tarantola
,
S.
, and
de Rocquigny
,
E.
, 2013
, “
On the Importance of Uncertain Factors in Seismic Fragility Assessment
,” Reliab. Eng. Syst. Saf.
,
109
, pp. 66
–76
.10.1016/j.ress.2012.08.007148.
Sankararaman
,
S.
, and
Mahadevan
,
S.
, 2013
, “
Separating the Contributions of Variability and Parameter Uncertainty in Probability Distributions
,” Reliab. Eng. Syst. Saf.
,
112
, pp. 187
–199
.10.1016/j.ress.2012.11.024149.
Schöbi
,
R.
, and
Sudret
,
B.
, 2019
, “
Global Sensitivity Analysis in the Context of Imprecise Probabilities (p-Boxes) Using Sparse Polynomial Chaos Expansions
,” Reliab. Eng. Syst. Saf.
,
187
, pp. 129
–141
.10.1016/j.ress.2018.11.021150.
Ferson
,
S.
, and
Troy Tucker
,
W.
, 2006
, “
Sensitivity Analysis Using Probability Bounding
,” Reliab. Eng. Syst. Saf.
,
91
, pp. 1435
–1442
.10.1016/j.ress.2005.11.052151.
Feng
,
G.
, 2018
, “
Sensitivity Analysis for Systems Under Epistemic Uncertainty With Probability Bounds Analysis
,” Int. J. Comput. Appl.
,
179
, pp. 1
–6
.10.5120/ijca2018915892152.
Oberguggenberger
,
M.
,
King
,
J.
, and
Schmelzer
,
B.
, 2009
, “
Classical and Imprecise Probability Methods for Sensitivity Analysis in Engineering: A Case Study
,” Int. J. Approx. Reason.
,
50
, pp. 680
–693
.10.1016/j.ijar.2008.09.004153.
Zhang
,
J.
,
TerMaath
,
S.
, and
Shields
,
M. D.
, 2021
, “
Imprecise Global Sensitivity Analysis Using Bayesian Multimodel Inference and Importance Sampling
,” Mech. Syst. Signal Process.
,
148
, p. 107162
.10.1016/j.ymssp.2020.107162154.
Pence
,
J.
,
Sakurahara
,
T.
,
Zhu
,
X.
,
Mohaghegh
,
Z.
,
Ertem
,
M.
,
Ostroff
,
C.
, Kee, E., et al., 2019
, “
Data-Theoretic Methodology and Computational Platform to Quantify Organizational Factors in Socio-Technical Risk Analysis
,” Reliab. Eng. Syst. Saf.
,
185
, pp. 240
–60
.10.1016/j.ress.2018.12.020Copyright © 2023 by ASME
You do not currently have access to this content.
