Evaluating Verification and Validation Methods in Knowledge Engineering

  • Alun Preece


Verification and validation (V&V) techniques have always been an essential part of the knowledge engineering process, because they offer the only way to judge the success (or otherwise) of a knowledge base development project. This remains true in the context of knowledge management: V&V techniques provide ways to measure the quality of knowledge in a knowledge base, and to indicate where work needs to be done to rectify anomalous knowledge. This paper provides a critical assessment of the state of the practice in knowledge base V&V, including a survey of available evidence as to the effectiveness of various V&V techniques in real-world knowledge base development projects. For the knowledge management practitioner, this paper offers guidance and recommendations for the use of V&V techniques; for researchers in knowledge management, the paper offers pointers to areas where further work needs to be done on developing more effective V&V techniques.


Knowledge Management Design Model Knowledge Engineering Knowledge Engineer Static Verification 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Liebowitz, J., and Wilcox, L., 1997, Knowledge Management and Its Integrative Elements. CRC Press, New York.zbMATHGoogle Scholar
  2. [2]
    Rushby, J., 1990, Validation and testing of Knowledge-Based Systems: how bad can it get? In (Gupta, 1990).Google Scholar
  3. [3]
    Batarekh, A., Preece, A., Bennett. A., and Grogono, P., 1996, Specifying an expert system. Expert Systems with Applications, Vol. 2(4).Google Scholar
  4. [4]
    Boehm, B., 1984, Verifying and validating software requirements and design specifications. IEEE Software, Vol. 1(1).Google Scholar
  5. [5]
    Meseguer, P. and Preece, A., 1995, Verification and Validation of Knowledge-Based Systems with Formal Specifications. Knowledge Engineering Review, Vol. 10(4).Google Scholar
  6. [6]
    Preece, A., 1990, Towards a methodology for evaluating expert systems. Expert Systems, Vol. 7(4).Google Scholar
  7. [7]
    Preece, A., 1995, Towards a Quality Assessment Framework for Knowledge-Based Systems. Journal of Systems and Software, Vol. 29(3).Google Scholar
  8. [8]
    Wielinga, B.J., Schreiber, A.Th., and Breuker, J.A., 1992, KADS: a modelling approach to knowledge engineering. Knowledge Acquisition, Vol. 4(1).Google Scholar
  9. [9]
    Brazier, F., Keplics, B., Jennings, N. and Treur, J., 1997, DESIRE: Modelling multi-agent systems in a compositional formal framework. International Journal of Cooperative Information Systems, Vol. 6(1).Google Scholar
  10. [10]
    Kingston, J., and Macintosh, A., 1999, Knowledge Management through Multi-Perspective Modelling, In Research and Development in Intelligent Systems XVI, pages 221–239, Springer-Verlag, Berlin.Google Scholar
  11. [11]
    Gupta, U.G., 1990, Validating and Verifying Knowledge-based Systems. IEEE Press, Los Alamitos, CA.Google Scholar
  12. [12]
    Ayel, M. and Laurent, J-P., Eds., 1991, Verification, Validation and Test of Knowledge-based Systems. John Wiley & Sons, New York.Google Scholar
  13. [13]
    O’Leary, D.E., 1991, Design, development and validation of expert systems: a survey of developers. In (Ayel and Laurent, 1991), pages 3–20.Google Scholar
  14. [14]
    Preece, A., Shinghal, R. and Batarekh, A., 1992, Principles and practice in verifying rule-based systems. Knowledge Engineering Review, Vol. 7(2).Google Scholar
  15. [15]
    O’Leary, D.E., 1998, On a common language for a “best practices” knowledge base. In Using AI for Knowledge Management and Business Process Reengineering: Papers from the AAAI-98 Workshop, AAAI Press, 1998..Google Scholar
  16. [16]
    Kirani, S., Zualkernan, I.A., and Tsai, W.T., 1992, Comparative Evaluation of Expert System Testing Methods. Technical report TR 92–30, Computer Science Department, University of Minnesota, Minneapolis.Google Scholar
  17. [17]
    Rushby, J. and J Crow, J., 1990, Evaluation of an Expert System for Fault Detection, Isolation, and Recovery in the Manned Maneuvering Unit. NASA Contractor Report CR-187466, SRI International, Menlo Park CA.Google Scholar
  18. [18]
    Preece, A. and Shinghal, R., 1994, Foundation and application of knowledge base verification. International Journal of Intelligent Systems, Vol. 9(8).Google Scholar
  19. [19]
    Miller, L., Hayes, J., and Mirsky, S., 1993, Evaluation of Knowledge Base Certification Methods. SAIC Report for U.S. Nuclear Regulatory Commission and Electrical Power Research Institute NUREG/CR-6316 SAIC-95/1028 Vol. 4.Google Scholar
  20. [20]
    Preece, A., Talbot, S. and Vignollet, L., 1997, Evaluation of Verification Tools for Knowledge-Based Systems. International Journal of Human-Computer Studies, Vol. 47.Google Scholar
  21. [21]
    Ayel, M. and Vignollet, L., 1993, SYCOJET and SACCO, two tools for verifying expert systems. International Journal of Expert Systems: Research and Applications, Vol. 6(2).Google Scholar
  22. [22]
    Fisher, M. and Wooldridge, M., 1993, Specifying and verifying distributed artificial intelligent systems. In Progress in Artificial Intelligence-Sixth Portugese Conference on Artificial Intelligence (LNAI Volume 727), pages 13–28, Springer-Verlag, Berlin.Google Scholar

Copyright information

© Springer-Verlag London 2001

Authors and Affiliations

  • Alun Preece
    • 1
  1. 1.Computing Science DepartmentUniversity of AberdeenAberdeenScotland

Personalised recommendations