Evaluation Measures for TCBR Systems
Textual-case based reasoning (TCBR) systems where the problem and solution are in free text form are hard to evaluate. In the absence of class information, domain experts are needed to evaluate solution quality, and provide relevance information. This approach is costly and time consuming. We propose three measures that can be used to compare alternate TCBR system configurations, in the absence of class information. The main idea is to quantify alignment as the degree to which similar problems have similar solutions. Two local measures capture this information by analysing similarity between problem and solution neighbourhoods at different levels of granularity, whilst a global measure achieves the same by analyzing similarity between problem and solution clusters. We determine the suitability of the proposed measures by studying their correlation with classifier accuracy on a health and safety incident reporting task. Strong correlation is observed with all three approaches with local measures being slightly superior over the global one.
Unable to display preview. Download preview PDF.
- 1.Weber, R., Ashley, K., Bruninghaus, S.: Textual CBR. Knowledge Engineering Review (2006)Google Scholar
- 2.Wiratunga, N., Craw, S., Rowe, R.: Learning to adapt for case based design. In: Proc. of the 6th European Conf. on CBR, pp. 421–435 (2002)Google Scholar
- 3.Bruninghaus, S., Ashley, K.: Evaluation of Textual CBR Approaches. In: AAAI 1998 workshop on TCBR, pp. 30–34 (1998)Google Scholar
- 4.Joachims, T.: Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In: Proc. of European Conf. on ML, pp. 137–142 (1998)Google Scholar
- 5.Richter, M.: Introduction. In: Case-Based Reasoning Technology: From Foundations to Applications, pp. 1–15 (1998)Google Scholar
- 8.Marchette, D.J.: Random Graphs for Statistical Pattern Recognition. Wiley Series in Probability and Statistics (2004)Google Scholar
- 9.Singh, S.: Prism, Cells and Hypercuboids. Pattern Analysis & Applications 5 (2002)Google Scholar
- 10.Vinay, V., Cox, J., Milic-Fralyling, N., Wood, K.: Measuring the Complexity of a Collection of Documents. In: Proc of 28th European Conf on Information Retrieval, pp. 107–118 (2006)Google Scholar
- 11.Lamontagne, L.: Textual CBR Authoring using Case Cohesion. In: 3rd TCBR 2006 - Reasoning with Text, Proceedings of the ECCBR 2006 Workshops, pp. 33–43 (2006)Google Scholar
- 12.Massie, S., Craw, S., Wiratunga, N.: Complexity profiling for informed case-base editing. In: Proc. of the 8th European Conf. on Case-Based Reasoning, pp. 325–339 (2006)Google Scholar
- 13.Chakraborti, S., Beresi, U., Wiratunga, N., Massie, S., Lothian, R., Watt, S.: A Simple Approach towards Visualizing and Evaluating Complexity of Textual Case Bases. In: Proc. of the ICCBR 2007 Workshops (2007)Google Scholar
- 14.Massie, S., Wiratunga, N., Craw, S., Donati, A., Vicari, E.: From Anomaly Reports to Cases. In: Proc. of the 7th International Conf. on Case-Based Reasoning, pp. 359–373 (2007)Google Scholar
- 16.JCOLIBRI Framework, Group for Artificial Intelligence Applications, Complutense University of Madrid, http://gaia.fdi.ucm.es/projects/jcolibri/jcolibri2/index.html