Knowledge and Information Systems

, Volume 7, Issue 2, pp 224–245 | Cite as

A quantitative analysis of the robustness of knowledge-based systems through degradation studies

  • Perry GrootEmail author
  • Annette ten Teije
  • Frank van Harmelen


The overall aim of this paper is to provide a general setting for quantitative quality measures of knowledge-based system behaviour that is widely applicable to many knowledge-based systems. We propose a general approach that we call degradation studies: an analysis of how system output changes as a function of degrading system input, such as incomplete or incorrect data or knowledge. To show the feasibility of our approach, we have applied it in a case study. We have taken a large and realistic vegetation-classification system, and have analysed its behaviour under various varieties of incomplete and incorrect input. This case study shows that degradation studies can reveal interesting and surprising properties of the system under study.


Quantitative analysis Knowledge-based systems Robust behaviour Validation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Cohen PR (1995) Empirical methods for artificial intelligence. MIT Press, Cambridge, MAGoogle Scholar
  2. Dean T, Boddy M (1988) An analysis of time-dependent planning. In Proceedings of the 7th national conference on artificial intelligence. Kaufmann, Saint Paul, MN, August 1988, pp 49–54Google Scholar
  3. Hayes-Roth F (1984) Knowledge-based expert systems—the state of the art in the US. In Fox J (ed) Expert systems: state of the art report. Pergamon, Oxford, UKGoogle Scholar
  4. IEEE (1990) IEEE standard glossary of software engineering terminology IEEE standard 610.12-1990, ISBN 1-55937-067-XGoogle Scholar
  5. Menzies T, van Harmelen F (1999) Evaluating knowledge-engineering techniques. Int J Hum Comput Stud 51(4):715–727CrossRefGoogle Scholar
  6. Preece AD, Talbot S, Vignollet L (1997) Evaluation of verification tools for knowledge-based systems. Int J Hum Comput Stud 47(5):629–658CrossRefGoogle Scholar
  7. Puppe F, Poeck K, Gappa U, et al (1994) Wiederverwendbare bausteine für eine konfigurierbare Diagnostik-shell. Künstliche Intelligenz 94(2):13–18Google Scholar
  8. Puppe F, Gappa U, Poeck K, et al (1996) Wissenbasierte Diagnose- und Informationssysteme: mit Anwendungen des Expertensystem-Shell-Baukastens D3. Springer, Berlin Heidelberg New YorkGoogle Scholar
  9. Salton G, McGill MJ (1983) Introduction to modern information retrieval. McGraw-Hill, New YorkGoogle Scholar
  10. Schumann J, Fischer B (1997) NORA/HAMMR: making deduction-based software component retrieval practical. In: Proceedings of the 12th international conference on automated software engineering, IEEE Press, Lake Tahoe, CA, November 1997, pp 246–254Google Scholar
  11. Shadbolt NR, O’Hara K, Crow L (1999) The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions. Int J Hum Comput Stud 51(4):729–755CrossRefGoogle Scholar
  12. ten Teije A, van Harmelen F (1996) Computing approximate diagnoses by using approximate entailment. In Aiello G, Doyle J (eds). Proceedings of the 5th international conference on principles of knowledge representation and reasoning. Kaufmann, Cambridge, MA, November 1996, pp 256–265Google Scholar
  13. ten Teije A, van Harmelen F (1997) Exploiting domain knowledge for approximate diagnosis (1997) In: Pollack ME (ed) Proceedings of the 15th international joint conference on artificial intelligence. Kaufmann, Nogoya, Japan, August 1997, pp 454–459Google Scholar
  14. van Harmelen F, ten Teije A (1998) Characterising approximate problem-solving by partial pre- and postconditions. In: Prade H (ed) Proceedings of the 13th European conference on artificial intelligence. Wiley, Brighton, UK, August 1998, pp 78–82Google Scholar
  15. Zilberstein S (1996) Using anytime algorithms in intelligent systems. Artif Intell Mag 17(3):73–83Google Scholar

Copyright information

© Springer-Verlag 2004

Authors and Affiliations

  • Perry Groot
    • 1
    Email author
  • Annette ten Teije
    • 1
  • Frank van Harmelen
    • 1
  1. 1.Division of Mathematics and Computer Science, Faculty of SciencesVrije UniversiteitAmsterdamThe Netherlands

Personalised recommendations