Principles and methods of model validation for model risk reduction

Abstract

Models, being incomplete and simplified representations of complex, real-world systems, contain simplifications and abstractions. However, models with errors or incorrect assumptions can suggest misleading results from the perspective of the decision-maker. Model risk arises from errors within models, or the incorrect use of models, and if undetected, can have significant impacts on businesses and organizations whose decisions and business processes depend on the model outputs. Thus, there is a need for model validation to reduce model risk. Validation is a process of comparing the correspondence between model outputs and system behavior. Systems engineering, given its primary focus on the theory and methodology of systems modeling, is a valuable source for principles and best practices in model validation, even for applications outside of systems engineering. In this paper, we discuss how model validation principles and methods developed in systems engineering research are applicable across a variety of modeling methodologies and domain areas for the reduction of model risk. Specific open challenges are discussed, including dealing with the subjectivity in models, the validation of decision models, and determining what level of validation is sufficient.