A Quasi-Experimental Model of Complex Human–Machine System Validation

Abstract:

Complex systems, such as power plants and commercial aircraft, are designed to operate safely under normal and disturbance conditions. Integrated-system validation is one means to evaluate whether safety and productivity goals can be achieved prior to actual operation. While some types of systems can be tested in an actual operational environment, validation is especially difficult when safety or economic considerations preclude testing in the disturbance conditions they are designed to handle. This paper proposes a conceptual model for integrated-system validation based on an integration of current approaches to design validation and concepts adopted from quasi-experimental research methodology. The model identifies important validation principles and the methodological considerations needed to support a logical and defensible inference to be made from validation tests to system performance under actual operating conditions. Central to the model are four general forms of validity: system representation, performance representation, test design and statistical conclusion validity. Threats to the inference process and weakness of the model are also discussed.