Evaluating Verification and Validation Methods in Knowledge Engineering

  • Alun Preece


Verification and validation (V&V) techniques have always been an essential part of the knowledge engineering process, because they offer the only way to judge the success (or otherwise) of a knowledge base development project. This remains true in the context of knowledge management: V&V techniques provide ways to measure the quality of knowledge in a knowledge base, and to indicate where work needs to be done to rectify anomalous knowledge. This paper provides a critical assessment of the state of the practice in knowledge base V&V, including a survey of available evidence as to the effectiveness of various V&V techniques in real-world knowledge base development projects. For the knowledge management practitioner, this paper offers guidance and recommendations for the use of V&V techniques; for researchers in knowledge management, the paper offers pointers to areas where further work needs to be done on developing more effective V&V techniques.




Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Liebowitz, J., and Wilcox, L., 1997, Knowledge Management and Its Integrative Elements. CRC Press, New York.MATHGoogle Scholar
  2. [2]
    Rushby, J., 1990, Validation and testing of Knowledge-Based Systems: how bad can it get? In (Gupta, 1990).Google Scholar
  3. [3]
    Batarekh, A., Preece, A., Bennett. A., and Grogono, P., 1996, Specifying an expert system. Expert Systems with Applications, Vol. 2(4).Google Scholar
  4. [4]
    Boehm, B., 1984, Verifying and validating software requirements and design specifications. IEEE Software, Vol. 1(1).Google Scholar
  5. [5]
    Meseguer, P. and Preece, A., 1995, Verification and Validation of Knowledge-Based Systems with Formal Specifications. Knowledge Engineering Review, Vol. 10(4).Google Scholar
  6. [6]
    Preece, A., 1990, Towards a methodology for evaluating expert systems. Expert Systems, Vol. 7(4).Google Scholar
  7. [7]
    Preece, A., 1995, Towards a Quality Assessment Framework for Knowledge-Based Systems. Journal of Systems and Software, Vol. 29(3).Google Scholar
  8. [8]
    Wielinga, B.J., Schreiber, A.Th., and Breuker, J.A., 1992, KADS: a modelling approach to knowledge engineering. Knowledge Acquisition, Vol. 4(1).Google Scholar
  9. [9]
    Brazier, F., Keplics, B., Jennings, N. and Treur, J., 1997, DESIRE: Modelling multi-agent systems in a compositional formal framework. International Journal of Cooperative Information Systems, Vol. 6(1).Google Scholar
  10. [10]
    Kingston, J., and Macintosh, A., 1999, Knowledge Management through Multi-Perspective Modelling, In Research and Development in Intelligent Systems XVI, pages 221–239, Springer-Verlag, Berlin.Google Scholar
  11. [11]
    Gupta, U.G., 1990, Validating and Verifying Knowledge-based Systems. IEEE Press, Los Alamitos, CA.Google Scholar
  12. [12]
    Ayel, M. and Laurent, J-P., Eds., 1991, Verification, Validation and Test of Knowledge-based Systems. John Wiley & Sons, New York.Google Scholar
  13. [13]
    O’Leary, D.E., 1991, Design, development and validation of expert systems: a survey of developers. In (Ayel and Laurent, 1991), pages 3–20.Google Scholar
  14. [14]
    Preece, A., Shinghal, R. and Batarekh, A., 1992, Principles and practice in verifying rule-based systems. Knowledge Engineering Review, Vol. 7(2).Google Scholar
  15. [15]
    O’Leary, D.E., 1998, On a common language for a “best practices” knowledge base. In Using AI for Knowledge Management and Business Process Reengineering: Papers from the AAAI-98 Workshop, AAAI Press, 1998..Google Scholar
  16. [16]
    Kirani, S., Zualkernan, I.A., and Tsai, W.T., 1992, Comparative Evaluation of Expert System Testing Methods. Technical report TR 92–30, Computer Science Department, University of Minnesota, Minneapolis.Google Scholar
  17. [17]
    Rushby, J. and J Crow, J., 1990, Evaluation of an Expert System for Fault Detection, Isolation, and Recovery in the Manned Maneuvering Unit. NASA Contractor Report CR-187466, SRI International, Menlo Park CA.Google Scholar
  18. [18]
    Preece, A. and Shinghal, R., 1994, Foundation and application of knowledge base verification. International Journal of Intelligent Systems, Vol. 9(8).Google Scholar
  19. [19]
    Miller, L., Hayes, J., and Mirsky, S., 1993, Evaluation of Knowledge Base Certification Methods. SAIC Report for U.S. Nuclear Regulatory Commission and Electrical Power Research Institute NUREG/CR-6316 SAIC-95/1028 Vol. 4.Google Scholar
  20. [20]
    Preece, A., Talbot, S. and Vignollet, L., 1997, Evaluation of Verification Tools for Knowledge-Based Systems. International Journal of Human-Computer Studies, Vol. 47.Google Scholar
  21. [21]
    Ayel, M. and Vignollet, L., 1993, SYCOJET and SACCO, two tools for verifying expert systems. International Journal of Expert Systems: Research and Applications, Vol. 6(2).Google Scholar
  22. [22]
    Fisher, M. and Wooldridge, M., 1993, Specifying and verifying distributed artificial intelligent systems. In Progress in Artificial Intelligence-Sixth Portugese Conference on Artificial Intelligence (LNAI Volume 727), pages 13–28, Springer-Verlag, Berlin.Google Scholar

Copyright information

© Springer-Verlag London 2001

Authors and Affiliations

  • Alun Preece
    • 1
  1. 1.Computing Science DepartmentUniversity of AberdeenAberdeenScotland

Personalised recommendations