Abstract
Textual-case based reasoning (TCBR) systems where the problem and solution are in free text form are hard to evaluate. In the absence of class information, domain experts are needed to evaluate solution quality, and provide relevance information. This approach is costly and time consuming. We propose three measures that can be used to compare alternate TCBR system configurations, in the absence of class information. The main idea is to quantify alignment as the degree to which similar problems have similar solutions. Two local measures capture this information by analysing similarity between problem and solution neighbourhoods at different levels of granularity, whilst a global measure achieves the same by analyzing similarity between problem and solution clusters. We determine the suitability of the proposed measures by studying their correlation with classifier accuracy on a health and safety incident reporting task. Strong correlation is observed with all three approaches with local measures being slightly superior over the global one.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Weber, R., Ashley, K., Bruninghaus, S.: Textual CBR. Knowledge Engineering Review (2006)
Wiratunga, N., Craw, S., Rowe, R.: Learning to adapt for case based design. In: Proc. of the 6th European Conf. on CBR, pp. 421–435 (2002)
Bruninghaus, S., Ashley, K.: Evaluation of Textual CBR Approaches. In: AAAI 1998 workshop on TCBR, pp. 30–34 (1998)
Joachims, T.: Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In: Proc. of European Conf. on ML, pp. 137–142 (1998)
Richter, M.: Introduction. In: Case-Based Reasoning Technology: From Foundations to Applications, pp. 1–15 (1998)
Glick, N.: Separation and probability of correct classification among two or more distributions. Annals of the Institute of Statistical Mathematics 25, 373–383 (1973)
Wallace, S., Boulton, D.M.: An information theoretic measure for classification. Computer Journal 11(2), 185–194 (1968)
Marchette, D.J.: Random Graphs for Statistical Pattern Recognition. Wiley Series in Probability and Statistics (2004)
Singh, S.: Prism, Cells and Hypercuboids. Pattern Analysis & Applications 5 (2002)
Vinay, V., Cox, J., Milic-Fralyling, N., Wood, K.: Measuring the Complexity of a Collection of Documents. In: Proc of 28th European Conf on Information Retrieval, pp. 107–118 (2006)
Lamontagne, L.: Textual CBR Authoring using Case Cohesion. In: 3rd TCBR 2006 - Reasoning with Text, Proceedings of the ECCBR 2006 Workshops, pp. 33–43 (2006)
Massie, S., Craw, S., Wiratunga, N.: Complexity profiling for informed case-base editing. In: Proc. of the 8th European Conf. on Case-Based Reasoning, pp. 325–339 (2006)
Chakraborti, S., Beresi, U., Wiratunga, N., Massie, S., Lothian, R., Watt, S.: A Simple Approach towards Visualizing and Evaluating Complexity of Textual Case Bases. In: Proc. of the ICCBR 2007 Workshops (2007)
Massie, S., Wiratunga, N., Craw, S., Donati, A., Vicari, E.: From Anomaly Reports to Cases. In: Proc. of the 7th International Conf. on Case-Based Reasoning, pp. 359–373 (2007)
Deerwester, S., Dumais, S., Landauer, T., Furnas, G., Harshman, R.: Indexing by Latent Semantic Analysis. JASIST 41(6), 391–407 (1990)
JCOLIBRI Framework, Group for Artificial Intelligence Applications, Complutense University of Madrid, http://gaia.fdi.ucm.es/projects/jcolibri/jcolibri2/index.html
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Raghunandan, M.A., Wiratunga, N., Chakraborti, S., Massie, S., Khemani, D. (2008). Evaluation Measures for TCBR Systems. In: Althoff, KD., Bergmann, R., Minor, M., Hanft, A. (eds) Advances in Case-Based Reasoning. ECCBR 2008. Lecture Notes in Computer Science(), vol 5239. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-85502-6_30
Download citation
DOI: https://doi.org/10.1007/978-3-540-85502-6_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-85501-9
Online ISBN: 978-3-540-85502-6
eBook Packages: Computer ScienceComputer Science (R0)