Abstract
There is a fundamental misalignment between current approaches to designing and executing verification and validation (V&V) strategies and the nature of AI-enabled systems. Current V&V approaches rely on the assumption that system behavior is preserved during a system’s lifetime. However, AI-enabled systems are developed so that they evolve their own behavior during their lifetime; this is the consequence of learning by the AI-enabled system. This misalignment makes existing approaches to designing and executing V&V strategies ineffective. In this chapter, we will provide a systems-theoretic explanation for (1) why learning capabilities originate a unique and unprecedented family of systems, and (2) why current V&V methods and processes are not fit for purpose. AI-enabled systems necessitate a paradigm shift in V&V activities. To enable this shift, we will delineate a set of theoretical advances and process transformations that could support such shift.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arifin, S. N., & Madey, G. R. (2015). Verification, validation, and replication methods for agent-based modeling and simulation: Lessons learned the hard way! In In Concepts and Methodologies for Modeling and Simulation (pp. 217–242). Springer.
Bertalanffy, L. v. (1969). General system theory: Foundations, development, applications.
Chollet, F. (2019). On the measure of intelligence. cs.
Engel, A. (2010). Verification, validation, and testing of engineered systems (Vol. 73). Wiley.
Felder, W. N. (2018). Addressing the complexity challenge with adaptive verification and validation.
Finn, C., Abbeel, P., Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (pp. 1126–1135). PMLR.
Hoppe, M., Engel, A., & Shachar, S. (2007). Systest: Improving the verification, validation, and testing process-assessing six industrial pilot projects. Systems Engineering, 10(4), 323–347.
Hunt, E. B. (1962). Concept learning: An information processing problem.
INCOSE, D. D. W. (2015). Systems engineering handbook: A guide for system life cycle processes and activities. San Diego, US-CA: International Council on Systems Engineering.
Isele, D., Rahimi, R., Cosgun, A., Subramanian, K., & Fujimura, K. (2018). Navigating occluded intersections with autonomous vehicles using deep reinforcement learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2034–2039). IEEE.
Johnson, M. (2013). The body in the mind: The bodily basis of meaning, imagination, and reason. University of Chicago Press.
Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles for autonomous vehicles from demonstration. In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2641–2646). IEEE.
Lakoff, G., & Johnson, M. (2008). Metaphors we live by. University of Chicago press.
Legg, S., Hutter, M., et al. (2007). A collection of definitions of intelligence. Frontiers in Artificial Intelligence and applications, 157, 17.
Mercier, H., & Sperber, D. (2017). The enigma of reason. Harvard University Press.
Murphy, G. L. (1996). On metaphoric representation. Cognition, 60(2), 173–204.
Nguyen, A., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 427–436).
Rendell, L. A., Sheshu, R., & Tcheng, D. K. (1987). Layered concept-learning and dynamically variable bias management. In IJCAI (pp. 308–314).
Salado, A. A (2021). Systems-theoretic articulation of stakeholder needs and system requirements. Systems Engineering, 24, 83–99. https://doi.org/10.1002/sys.21568.
Salado, A., & Kannan, H. (2018). A mathematical model of verification strategies. Systems Engineering, 21(6), 593–608.
Salado, A., & Kannan, H. (2019). Elemental patterns of verification strategies. Systems Engineering, 22(5), 370–388.
Salado, A., & Nilchiani, R. (2014). A categorization model of requirements based on max-neef’s model of human needs. Systems Engineering, 17(3), 348–360.
Schroyens, W. . J., Schaeken, W., & d’Ydewalle, G. (2001). The processing of negations in conditional reasoning: A meta-analytic case study in mental model and/or mental logic theory. Thinking and Reasoning, 7(2), 121–172.
Sengupta, S., Chakraborti, T., & Kambhampati, S. (2019). Mtdeep: boosting the security of deep neural nets against adversarial attacks with moving target defense. In International Conference on Decision and Game Theory for Security (pp. 479–491). Springer.
Suto, I. (2012). What are the impacts of qualifications for 16 to 19 year olds on higher education? a survey of 633 university lecturers. Cambridge Assessment.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013). Intriguing properties of neural networks. arXiv:1312.6199.
Thrun, S., & Pratt, L. (1998). Learning to learn: Introduction and overview. In Learning to learn (pp. 3–17). Springer.
Vilalta, R., & Drissi, Y. (2002). A perspective view and survey of meta-learning. Artificial Intelligence Review, 18(2), 77–95.
Wang, X., Li, J., Kuang, X., Tan, Y.-A., & Li, J. (2019). The security of machine learning in an adversarial setting: A survey. Journal of Parallel and Distributed Computing, 130, 12–23.
Wymore, A. W. (2018). Model-based systems engineering (Vol. 3). CRC Press.
Xiang, W., Musau, P., Wild, A. A., Lopez, D. M., Hamilton, N., Yang, X., Rosenfeld, J., & Johnson, T. T. (2018). Verification for machine learning, autonomy, and neural networks survey. arXiv:1810.01989.
Yilmaz, L. (2006). Validation and verification of social processes within agent-based computational organization models. Computational & Mathematical Organization Theory, 12(4), 283–312.
Yilmaz, L. (2015). Concepts and methodologies for modeling and simulation. Springer.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Shadab, N., Kulkarni, A.U., Salado, A. (2021). Shifting Paradigms in Verification and Validation of AI-Enabled Systems: A Systems-Theoretic Perspective. In: Lawless, W.F., Mittu, R., Sofge, D.A., Shortell, T., McDermott, T.A. (eds) Systems Engineering and Artificial Intelligence . Springer, Cham. https://doi.org/10.1007/978-3-030-77283-3_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-77283-3_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77282-6
Online ISBN: 978-3-030-77283-3
eBook Packages: Computer ScienceComputer Science (R0)