Evaluation Measures for TCBR Systems

  • M. A. Raghunandan
  • Nirmalie Wiratunga
  • Sutanu Chakraborti
  • Stewart Massie
  • Deepak Khemani
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5239)

Abstract

Textual-case based reasoning (TCBR) systems where the problem and solution are in free text form are hard to evaluate. In the absence of class information, domain experts are needed to evaluate solution quality, and provide relevance information. This approach is costly and time consuming. We propose three measures that can be used to compare alternate TCBR system configurations, in the absence of class information. The main idea is to quantify alignment as the degree to which similar problems have similar solutions. Two local measures capture this information by analysing similarity between problem and solution neighbourhoods at different levels of granularity, whilst a global measure achieves the same by analyzing similarity between problem and solution clusters. We determine the suitability of the proposed measures by studying their correlation with classifier accuracy on a health and safety incident reporting task. Strong correlation is observed with all three approaches with local measures being slightly superior over the global one.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Weber, R., Ashley, K., Bruninghaus, S.: Textual CBR. Knowledge Engineering Review (2006)Google Scholar
  2. 2.
    Wiratunga, N., Craw, S., Rowe, R.: Learning to adapt for case based design. In: Proc. of the 6th European Conf. on CBR, pp. 421–435 (2002)Google Scholar
  3. 3.
    Bruninghaus, S., Ashley, K.: Evaluation of Textual CBR Approaches. In: AAAI 1998 workshop on TCBR, pp. 30–34 (1998)Google Scholar
  4. 4.
    Joachims, T.: Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In: Proc. of European Conf. on ML, pp. 137–142 (1998)Google Scholar
  5. 5.
    Richter, M.: Introduction. In: Case-Based Reasoning Technology: From Foundations to Applications, pp. 1–15 (1998)Google Scholar
  6. 6.
    Glick, N.: Separation and probability of correct classification among two or more distributions. Annals of the Institute of Statistical Mathematics 25, 373–383 (1973)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Wallace, S., Boulton, D.M.: An information theoretic measure for classification. Computer Journal 11(2), 185–194 (1968)MATHGoogle Scholar
  8. 8.
    Marchette, D.J.: Random Graphs for Statistical Pattern Recognition. Wiley Series in Probability and Statistics (2004)Google Scholar
  9. 9.
    Singh, S.: Prism, Cells and Hypercuboids. Pattern Analysis & Applications 5 (2002)Google Scholar
  10. 10.
    Vinay, V., Cox, J., Milic-Fralyling, N., Wood, K.: Measuring the Complexity of a Collection of Documents. In: Proc of 28th European Conf on Information Retrieval, pp. 107–118 (2006)Google Scholar
  11. 11.
    Lamontagne, L.: Textual CBR Authoring using Case Cohesion. In: 3rd TCBR 2006 - Reasoning with Text, Proceedings of the ECCBR 2006 Workshops, pp. 33–43 (2006)Google Scholar
  12. 12.
    Massie, S., Craw, S., Wiratunga, N.: Complexity profiling for informed case-base editing. In: Proc. of the 8th European Conf. on Case-Based Reasoning, pp. 325–339 (2006)Google Scholar
  13. 13.
    Chakraborti, S., Beresi, U., Wiratunga, N., Massie, S., Lothian, R., Watt, S.: A Simple Approach towards Visualizing and Evaluating Complexity of Textual Case Bases. In: Proc. of the ICCBR 2007 Workshops (2007)Google Scholar
  14. 14.
    Massie, S., Wiratunga, N., Craw, S., Donati, A., Vicari, E.: From Anomaly Reports to Cases. In: Proc. of the 7th International Conf. on Case-Based Reasoning, pp. 359–373 (2007)Google Scholar
  15. 15.
    Deerwester, S., Dumais, S., Landauer, T., Furnas, G., Harshman, R.: Indexing by Latent Semantic Analysis. JASIST 41(6), 391–407 (1990)CrossRefGoogle Scholar
  16. 16.
    JCOLIBRI Framework, Group for Artificial Intelligence Applications, Complutense University of Madrid, http://gaia.fdi.ucm.es/projects/jcolibri/jcolibri2/index.html

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • M. A. Raghunandan
    • 1
  • Nirmalie Wiratunga
    • 2
  • Sutanu Chakraborti
    • 3
  • Stewart Massie
    • 2
  • Deepak Khemani
    • 1
  1. 1.Department of Computer Science and EngineeringIndian Institute of TechnologyMadrasIndia
  2. 2.School of ComputingThe Robert Gordon UniversityScotlandUK
  3. 3.Tata Research Development and Design CentrePuneIndia

Personalised recommendations