Abstract
Although various methods for the evaluation of intelligent systems have been proposed in the past, almost no techniques are present that support the manual inspection of knowledge bases by the domain specialist. Manual knowledge base inspection is an important and frequently applied method in knowledge engineering. Since it can hardly be performed in an automated manner, it is a time-consuming and costly task. In this paper, we discuss a collection of appropriate visualization techniques that help developers to interactively browse and analyze the knowledge base in order to find deficiencies and semantic errors in their implementation. We describe standard visualization methods adapted to specifically support the analysis of the static knowledge base structure, but also of the usage of knowledge base objects such as questions or solutions. Additionally, we introduce a novel visualization technique that supports the validation of the derivation and interview behavior of a knowledge system in a semi-automatic manner. The application of the presented methods was motivated by the daily practice of knowledge base development.
Similar content being viewed by others
References
Aamodt A, Plaza E (1994) Case-based reasoning: foundational issues, methodological variations, and system approaches. AI Commun 7(1): 39–59
Adeli H (1990) Knowledge engineering, vol I: fundamentals. McGraw-Hill Inc., New York
Akers S (1978) Binary decision diagrams. IEEE Trans Comput 27(6): 509–516
Ayel M, Laurent JP (1991) Validation, verification and test of knowledge-based systems. Wiley, London
Baumeister J (2009) Advanced measures for empirical testing. In: FLAIRS’09: proceedings of the 22th international Florida artificial intelligence research society conference. AAAI Press, pp 378–383
Baumeister J, Bregenzer J, Puppe F (2006) Gray box robustness testing of rule systems. In: KI’06: proceedings of the 29th annual German conference on artificial intelligence. LNAI 4314. Springer, pp 346–360
Baumeister J, Menge M, Puppe F (2008) Visualization techniques for the evaluation of knowledge systems. In: FLAIRS’08: proceedings of the 21th international Florida artificial intelligence research society conference. AAAI Press, pp 329–334
Baumeister J, Reutelshoefer J, Puppe F (2010) KnowWE: a semantic wiki for knowledge engineering. Appl Intell. doi:10.1007/s10489-010-0224-5
Baumeister J, Seipel D (2005) Smelly owls—design anomalies in ontologies. In: FLAIRS’05: proceedings of the 18th international Florida artificial intelligence research society conference. AAAI Press, pp 215–220
Baumeister J, Seipel D (2006) Verification and refactoring of ontologies with rules. In: EKAW’06: proceedings of the 15th international conference on knowledge engineering and knowledge management, LNAI 4248. Springer, Berlin, pp 82–95
Baumeister J, Seipel D (2010) Anomalies in ontologies with rules. Web Semant Sci Serv Agents World Wide Web 8(1): 55–68
Baumeister J, Seipel D, Puppe F (2003) Incremental development of diagnostic set-covering models with therapy effects. Int J Uncertain Fuzziness Knowl Based Syst 11(Suppl Issue 2): 25–49
Baumeister J et al (2008) The knowledge modeling environment d3web.KnowME, open-source at: http://d3web.sourceforge.net
Bederson BB, Shneiderman B, Wattenberg M (2002) Ordered and quantum treemaps: making effective use of 2D space to display hierarchies. ACM Trans Graph (TOG) 21(4): 833–854
Blanchard J, Guillet F, Briand H (2007) Interactive visual exploration of association rules with rule- focusing methodology. Knowl Inf Syst 13(1): 43–75
Bresciani S, Eppler MJ (2008) Do visualizations foster experience sharing and retention in groups? towards an experimental validation. In: Proceedings of I-KNOW ’08 and I-MEDIA ’08
Cai Y, Stumpf R, Wynne T, Tomlinson M, Chung DSH, Boutonnier X, Ihmig M, Franco R, Bauernfeind N (2007) Visual transformation for interactive spatiotemporal data mining. Knowl Inf Syst 13(2): 119–142
Clancey WJ (1985) Heuristic classification. Artif Intell 27(3): 289–350
Ensan F, Du W (2010) A knowledge encapsulation approach to ontology modularization. Knowl Inf Syst. doi:10.1007/s10115-009-0279-y
Fluit C, Sabou M, van Harmelen F (2006) Ontology-based information visualization. In: Visualizing the semantic web. Springer, pp 36–48
Groot P, ten Teije A, van Harmelen F (2003) A quantitative analysis of the robustness of knowledge-based systems through degradation studies. Knowl Inf Syst 7(2): 224–245
Hartung RL, Håkansson A (2007) Automated testing for knowledge based systems. In: KES 2007: proceedings of the 11th international conference on knowledge-based intelligent information and engineering systems, part I, LNCS 4693. Springer, pp 270–278
Helfman R, Dumer J, Hanratty T (1995) TED-turbine engine diagnostics: an expert diagnostic system for the M1 Abrams AGT1500 turbine engine. In: 11th conference on artificial intelligence for applications. IEEE Computer Society, Los Alamitos, CA, USA, pp 1032–1038
Horiyama T, Ibaraki T (2002) Ordered binary decision diagrams as knowledge-bases. Artif Intell 136(2): 189–213
Hu B, Shadbolt N (2003) Visualising a DL knowledge base with delogviz. In: DL2003: proceedings of the international workshop on description logics (CEUR 81)
Hüttig M, Buscher G, Menzel T, Scheppach W, Puppe F, Buscher H-P (2004) A diagnostic expert system for structured reports, quality assessment, and training of residents in sonography. Med Klin 3: 117–122
Iglezakis I, Reinartz T, Roth-Berghofer T (2004) Maintenance memories: beyond concepts and techniques for case base maintenance. In: Advances in case-based reasoning. Springer, LNCS 3155, pp 227–241
Ji S, Matsumura Y, Kuwata S, Nakano H, Chen Y, Teratani T, Zhang Q, Mineno T, Takeda H (2004) Creation of a master table for checking indication and contraindication of medicine from a knowledge base linked with a thesaurus. J Med Syst 28(6): 561–573
JSPWiki (n.d.) The open-source wiki clone JSPWiki. http://www.jspwiki.org
Jurisica I, Mylopoulos J, Yu E (2004) Ontologies for knowledge management: an information systems perspective. Knowl Inf Syst 6(4): 380–401
Kimura M, Sakamoto M, Adachi T, Sagara H (2005) Diagnosis of febrile illnesses in returned travelers using the PC software GIDEON. Travel Med Infect Dis 3(3): 157–160
Knauf R, Gonzalez AJ, Abel T (2002) A framework for validation of rule-based systems. IEEE Trans Syst Man Cybern B Cybern 32(3): 281–295
Kotis K, Vouros A (2006) Human-centered ontology engineering: the hcome methodology. Knowl Inf Syst 10(1): 109–131
Lancaster FW, Ulvila JW, Humphrey SM, Smith LC, Allen B, Herner S (1996) Evaluation of interactive knowledge-based systems: overview and design for empirical testing. J Am Soc Inf Sci 47(1): 57–69
Ligeza A (2006) Logical foundations for rule-based systems, studies in computational intelligence, vol 11. Springer, Berlin
Mazza R (2009) Introduction to information visualization. Springer, Berlin
Mersmann S, Dojat M (2004) SmartCaretm—automated clinical guidelines in critical care. In: ECAI’04/PAIS’04: proceedings of the 16th European conference on artificial intelligence, including prestigious applications of intelligent systems. IOS Press, Valencia, Spain, pp 745–749
Miah SJ, Kerr D, Gammack J, Cowan T (2008) A generic design environment for the rural industry knowledge acquisition. Knowl Based Syst 21(8): 892–899
Middelfart M, Pedersen TB (2009) Discovering sentinel rules for business intelligence. In: DEXA ’09: proceedings of the 20th international conference on database and expert systems applications. pp 592–602
Milne R, Nicol C (2000) TIGER: continuous diagnosis of gas turbines. In: ECAI’00: proceedings of the 14th European conference on artificial intelligence. Berlin, Germany
Nalepa GJ (2007) Proposal of business process and rules modeling with the XTT method. In: SYNASC ’07: proceedings of the ninth international symposium on symbolic and numeric algorithms for scientific computing. IEEE Computer Society, Washington, DC, USA, pp 500–506
Nalepa GJ, Ligeza A (2005) A graphical tabular model for rule-based logic programming and verification. Syst Sci 31(2): 89–95
Nghia DD, Puppe F (2009) Hybrides, skalierbares Diagnosesystem für freie Kfz-Werkstätten[Hybrid and scalable diagnosis system for car garages]. KI Ger AI Mag 23(2): 31–37
Nilsson NJ (1982) Principles of artificial intelligence. Springer, Berlin
Noy NF, Klein M (2004) Ontology evolution: not the same as schema evolution. Knowl Inf Syst 6(4): 428–440
O’Leary DE (1991) Design, development and validation of expert systems: a survey of developers. In: Ayel M, Laurent JP (eds) Validation, verification and test of knowledge-based systems. John Wiley & Sons Inc., London, pp 3–19
Ou MH, West GAW, Lazarescu M, Clay C (2007) Dynamic knowledge validation and verification for CBR teledermatology system. Art Intell Med 39(1): 79–96
Padma T, Balasubramanie P (2009) Knowledge based decision support system to assist work-related risk analysis in musculoskeletal disorder. Knowle Based Syst 22(1): 72–78
Pearl J (1988) Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann Publisher, San Mateo
Pinto HS, Martins JP (2004) Ontologies: how can they be built?. Knowl Inf Syst 6(4): 441–464
Plaisant C, Grosjean J, Bederson BB (2002) Spacetree: supporting exploration in large node link tree, design evolution and empirical evaluation. In: INFOVIS ’02: proceedings of the IEEE symposium on information visualization (InfoVis’02). pp 57–64
Preece A (1998) Building the right system right. In: Evaluating V&V methods in knowledge engineering, verification and validation of knowledge-based systems: papers from the AAAI-98 Workshop. AAAI Press, pp 38–45
Preece A (2001) Evaluating verification and validation methods in knowledge engineering. In: Micro-level knowledge management. Morgan-Kaufman, pp 123–145
Preece A, Shinghal R (1994) Foundation and application of knowledge base verification. Int J Intell Syst 9: 683–702
Puppe F (2000) Knowledge formalization patterns. In: PKAW 2000: proceedings of the Pacific rim knowledge acquisition workshop. Sydney, Australia
Puppe F, Atzmueller M, Buscher G, Hüttig M, Luehrs H, Buscher H-P (2008) Application and evaluation of a medical knowledge system in sonography (SONOCONSULT). In: ECAI’08/PAIS’08: proceedings of the 18th European conference on artificial intelligence, including prestigious applications of intelligent systems. IOS Press, Amsterdam, The Netherlands, pp 683–687
Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1): 81–106
Reggia JA, Nau DS, Wang PY (1983) Diagnostic expert systems based on a set covering model. J Man Mach Stud 19(5): 437–460
Richards D (2004) Addressing the ontology acquisition bottleneck through reverse ontological engineering. Knowl Inf Syst 6(4): 402–427
Roth-Berghofer T (2003) Developing maintainable case-based reasoning systems: applying SIAM to empolis orenge. In: GWEM’03: German workshop on experience management, CEUR vol 67
Seipel D, Hopfner M, Baumeister J (2005) Declarative querying and visualizing knowledge bases in XML. In: INAP/WLP’04: applications of declarative programming and knowledge management (selected papers), LNAI 3392. Springer, Berlin, pp 16–31
Shneiderman B (1991) Tree visualization with tree-maps: a 2-d space-filling approach. In: ACM transactions on graphics, vol 11. pp 92–99
Soman KP, Diwakar S, Ajay V (2006) Insight into data mining: theory and practice. Prentice-Hall, Englewood Cliffs
Spence R (2007) Information visualization. Design for interaction. Prentice-Hall, Upper Saddle River
Subasic I, Berendt B (2009) Discovery of interactive graphs for understanding and searching time-indexed corpora. Knowl Inf Syst 23(3): 293–319
Vermesan A, Coenen F (1999) Validation and verification of knowledge based systems. Theory, tools and practice. Kluwer Academic Publisher, Dordrecht
Wang S, Wang H (2002) Knowledge discovery through self-organizing maps: data visualization and query processing. Knowl Inf Syst 4(1): 31–45
Yi JS, Kang Ya, Stasko J, Jacko J (2007) Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans Vis Comput Graph 13(6): 1224–1231
Zacharias V (2007) Visualization of rule bases—the overall structure. In: I-KNOW ’07: proceedings of international conference on knowledge management
Zacharias V (2008) Development and verification of rule based systems—a survey of developers. In: RuleML ’08: proceedings of the international symposium on rule representation, interchange and reasoning on the Web, LNCS 5321. Springer-Verlag, Berlin, Heidelberg, pp 6–16
Zlatareva NP (1997) Verification of non-monotonic knowledge bases. Decis Support Syst 21(4): 253–261
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Baumeister, J., Freiberg, M. Knowledge visualization for evaluation tasks. Knowl Inf Syst 29, 349–378 (2011). https://doi.org/10.1007/s10115-010-0350-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-010-0350-8