Abstract
The overall aim of this paper is to provide a general setting for quantitative quality measures of Knowledge-Based System behavior which is widely applicable to many Knowledge-Based Systems. We propose a general approach that we call “degradation studies”: an analysis of how system output degrades as a function of degrading system input, such as incomplete or incorrect inputs. Such degradation studies avoid a number of problems that have plagued earlier attempts at defining such quality measures because they do not require a comparison between different (and often incomparable) systems, and they are entirely independent of the internal workings of the particular Knowledge-Based System at hand.
To show the feasibility of our approach, we have applied it in a specific case-study. We have taken a large and realistic vegetation-classification system, and have analyzed its behavior under various varieties of missing input. This case-study shows that degradation studies can reveal interesting and surprising properties of the system under study.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
P. R. Cohen. Empirical methods for artificial intelligence. MIT Press, 1995.
T. Dean and M. Boddy.An analysis of time-dependent planning. In Proceedings of the seventh National conference on artificial intelligence AAAI-88, pages 49–54, Saint Paul, Minnesota, 1988.
B. Fischer and J. Schumann. NORA/HAMMR:Making deduction-based software component retrieval practical. InAutomated Software Engineering (ASE)’97, pages 246–254. IEEE, 1997.
F. Hayes-Roth. Knowledge-based expert systems-the state of the art in the US. In J. Fox, ed., Expert Systems: state of the art report. Pergamon Infotech, Oxford, 1984.
IEEE. IEEE standard glossary of software engineering terminology, 1990. IEEE Standard 610.12-1990, ISBN 1-55937-067-X.
Tim Menzies and Frank van Harmelen. Evaluating Knowledge-Engineering Techniques. International Journal of Human-Computer Studies, 51(4):715–727, October 1999.
A. Preece, S. Talbot, and L. Vignollet. Evaluation of verification tools for knowledge-based systems. Internationl Journal of Human-Computer Studies, 47:629–658, 1997.
F. Puppe, U. gappa, K. Poeck, and S. Bamberger. Wissensbasierte Diagnose-und Informationssysteme. Springer-Verlag, Juli 1996.
F. Puppe, K. Poeck, U. Gappa, S. Bamberger, and K. Goos. Wiederverwendbare Bausteine für eine konfigurierbare Diagnostik-shell. Künstliche Intelligenz, 94(2):13–18, 1994.
G. Salton and M. J. McGill. Introduction to Modern Information Retrieval. McGraw-Hill, NewYork, 1983.
Nigel Shadbolt, Kieron O’Hara, and Louise Crow. The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions. International Journal of Human-Computer Studies, 51(4):729–755, October 1999.
A. ten Teije and F. van Harmelen. Computing approximate diagnoses by using approximate entailment. In Proceedings of the Fifth International Conference on Principles of Knowledge Representation and Reasoning (KR’96), pages 265–256, Boston, Massachusetts, November 1996.
A. ten Teije and F. van Harmelen. Exploiting domain knowledge for approximate diagnosis. In M. Pollack, ed., Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI’97), pages 454–459, Nagoya, Japan, August 1997.
F. van Harmelen and A. ten Teije. Characterising approximate problem-solving by partial pre-and postconditions. In Proceedings of ECAI’98, pages 78–82, Brighton, August 1998.
S. Zilberstein. Using anytime algorithms in intelligent systems. Artificial Intelligence Magazine, fall:73–83, 1996.
S. Zilberstein and S. J. Russell. Approximate Reasoning Using Anytime Algorithms. In S. Natarajan, ed., Imprecise and Approximate Computation. Kluwer Academic Publishers, 1995.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Groot, P., Van Harmelen, F., Teije, A.T. (2000). Torture Tests: A Quantitative Analysis for the Robustness of Knowledge-Based Systems. In: Dieng, R., Corby, O. (eds) Knowledge Engineering and Knowledge Management Methods, Models, and Tools. EKAW 2000. Lecture Notes in Computer Science(), vol 1937. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-39967-4_31
Download citation
DOI: https://doi.org/10.1007/3-540-39967-4_31
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-41119-2
Online ISBN: 978-3-540-39967-4
eBook Packages: Springer Book Archive