Software Quality Journal

, Volume 15, Issue 3, pp 309–326 | Cite as

Validating neural network-based online adaptive systems: a case study

  • Yan LiuEmail author
  • Bojan Cukic
  • Srikanth Gururajan


Biologically inspired soft computing paradigms such as neural networks are popular learning models adopted in online adaptive systems for their ability to cope with the demands of a changing environment. However, continual changes induce uncertainty that limits the applicability of conventional validation techniques to assure the reliable performance of such systems. In this paper, we discuss a dynamic approach to validate the adaptive system component. Our approach consists of two run-time techniques: (1) a statistical learning tool that detects unforeseen data; and (2) a reliability measure of the neural network output after it accommodates the environmental changes. A case study on NASA F-15 flight control system demonstrates that our techniques effectively detect unusual events and provide validation inferences in a real-time manner.


Validation Online adaptive system Novelty detection Support vector data description Validity index 


  1. Boyd, M. A., Schumann, J., Brat, G., Giannakopoulou, D., Cukic, B.,& Mili, A. (2001). Validation and verification process guide for software and neural nets, Technical report, NASA Ames Research Center, September.Google Scholar
  2. Bruske, J. & Sommer, G. (1995). Dynamic cell structure learns perfectly topology preserving map. Neural Computation, 7(4) 845–865.CrossRefGoogle Scholar
  3. Darrah, M., Taylor, B., & Skias, S. (2004). Rule extraction from dynamic CEll structure neural networks used in a safety critical application. In Proc. of the Seventeenth International Conference of the Florida Artificial Intelligence Research Society, Miami Beach, FL, USA, September 2004.Google Scholar
  4. Del Gobbo, D., & Cukic, B. (2001). Validating on-line neural networks. Technical report, Lane department of computer science and electrical engineering, West Virginia University, December 2001.Google Scholar
  5. Institute of Software Reseach. (2001). Dynamic cell structure neural network report for the intelligent flight control system, Technical report, Document ID: IFC-DCSR-D002-UNCLASS-010401, January 2001.Google Scholar
  6. Jorgensen, C. C. (1991). Feedback linearized aircraft control using dynamic cell structures. InWorld Automation Congress (ISSCI) (pp. 050.1-050.6). Alaska.Google Scholar
  7. Kohonen T. (1990). The self-organizing map. Proceedings. of the IEEE, 78(9), 1464–1480.CrossRefGoogle Scholar
  8. Leonard, J. A., Kramer, M. A., & Ungar, L. H. (1992). Using radial bais functions to approximate a function and its error bounds. IEEE Transactions on Neural Networks, 3(4), 624–627.CrossRefGoogle Scholar
  9. Liu, Y., Cukic, B., Menzies, T., Gururajan, S., Napolitano, M. (2003). “Validating an on-line adaptive system using support vector data description”. In Proceedings of fifteenth international conference on tools with artificial intelligence, Sacramento, CA.Google Scholar
  10. Mackall, D., Nelson, S., & Schumann, J. (2002). Verification and validation of neural networks of aerospace applications, Technical report, CR-211409, NASA.Google Scholar
  11. Martinetz, T., & Schulten, K. (1994). Topology representing networks. Neural Networks, 7(3), 507–522.CrossRefGoogle Scholar
  12. Mili, A., Cukic, B., Liu, Y., & Ben Ayed, R. (2003). Towards the verification and validation of on-line learning adaptive systems. In Computational methods in software engineering. Kluwer Scientific Publishing.Google Scholar
  13. Napolitano, M., Molinaro, G., Innocenti, M., & Martinelli, D. (1998). A complete hardware package for a fault tolerant flight control system using online learning neural networks. Appears in Proc. of the 1999 American Control Conference (Vol. 4, pp. 2615–2619). San Diego, CA, USA, 1999.Google Scholar
  14. Raz, O. (2000). Validation of online artificial neural networks–an informal classification of related approaches. Technical report, NASA Ames Research Center, Moffet Field, CA.Google Scholar
  15. Schumann, J., & Nelson, S. (2002). Towards V&V of neural network based controllers. Workshop on Self-Healing Systems.Google Scholar
  16. Schumann, J., & Gupta, P. Monitoring the Performance of a neuro-adaptive Controller. In Proc. of the twentyfourth international workshop on Bayesian inference and maximum entropy methods in Sscience and engineering, Garching bei München, Germany, 2004.Google Scholar
  17. Tax, D. M. J., & Duin, R. P. W. (1999a). Data domain description using support vectors. In Proc. european symposium on artificial neural networks, Bruges, April 21–23, 1999 (pp. 251–257). Brussels: D-Facto.Google Scholar
  18. Tax, D. M. J., & Duin, R. P. W. (1999b). Support vector domain description. Pattern Recognition Letters, 20(11–13), 1191–1199.CrossRefGoogle Scholar
  19. Tax, D. M. J. (2001). “One-class classification,” Dissertation, ISBN: 90-75691-05-x.Google Scholar
  20. Vapnik, V. N. (1998). Statistical learning theory. NY: Wiley.Google Scholar
  21. Yerramalla, S., Cukic, B., & Fuller, E. (2003a). Lyapunov stability analysis of quantization error for DCS neural networks. International joint conference on neural networks (IJCNN’03), Oregon.Google Scholar
  22. Yerramalla, S., Fuller, E., & Cukic, B. (2003b). Lyapunov analysis of neural network stability in an adaptive flight control system. Sixth symposium on self stabilizing systems (SSS-03). San Francisco, CA, June 2003.Google Scholar
  23. Yerramalla, S., Liu, Y., Fuller, E., Cukic B., & Gururajan, S. (2004). An approach to V&V of embedded adaptive systems. In Lecture notes in computer science (LNCS) Proceeding of third NASA-Goddard/IEEE workshop on formal approaches to agent-based systems, Springer-Verlag.Google Scholar
  24. Yerramalla, S. (2005). Tability monitoring and analysis of online learning neural networks. Doctoral dissertation. WV: West Virginia University.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Motorola LabsMotorola IncSchaumburgUSA
  2. 2.Lane Department of Computer Science and Electrical EngineeringWest Virginia UniversityMorgantownUSA
  3. 3.Mechanical and Aerospace Engineering DepartmentWest Virginia UniversityMorgantownUSA

Personalised recommendations