Evaluating CBR Systems Using Different Data Sources: A Case Study

  • Mingyang Gu
  • Agnar Aamodt
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4106)


The complexity and high construction cost of case bases make it very difficult, if not impossible, to evaluate a CBR system, especially a knowledge-intensive CBR system, using statistical evaluation methods on many case bases. In this paper, we propose an evaluation strategy, which uses both many simple case bases and a few complex case bases to evaluate a CBR system, and show how this strategy may satisfy different evaluation goals. The identified evaluation goals are classified into two categories: domain-independent and domain-dependent. For the evaluation goals in the first category, we apply the statistical evaluation method using many simple case bases (for example, UCI data sets); for evaluation goals in the second category, we apply different, relatively weak, evaluation methods on a few complex domain-specific case bases. We apply this combined evaluation strategy to evaluate our knowledge-intensive conversational CBR method as a case study.


Case Base Evaluation Goal Target Case Case Retrieval General Domain Knowledge 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Simon, H.A.: Artificial Intelligence: an Empirical Science. Artif. Intell. 77, 95–127 (1995)CrossRefGoogle Scholar
  2. 2.
    Cohen, P.R., Howe, A.E.: How Evaluation Guides AI Research. AI Mag. 9, 35–43 (1988)Google Scholar
  3. 3.
    Cohen, P., Howe, A.: Toward AI Research Methodology: Three Case Studies in Evaluation. Systems, Man and Cybernetics, IEEE Transactions 19, 634–646 (1989)CrossRefGoogle Scholar
  4. 4.
    Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT Press, Cambridge (1995)MATHGoogle Scholar
  5. 5.
    McSherry, D.: Interactive Case-Based Reasoning in Sequential Diagnosis. Applied Intelligence 14, 65–76 (2001)MATHCrossRefGoogle Scholar
  6. 6.
    Aamodt, A., Plaza, E.: Case-Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches. AI Communications 7, 39–59 (1994)Google Scholar
  7. 7.
    Kolodner, J.: Case-Based Reasoning. Morgan Kaufmann, San Francisco (1993)Google Scholar
  8. 8.
    Santamaria, J.C., Ram, A.: Systematic Evaluation of Design Decisions in CBR Systems. In: Proceedings of the AAAI Case-Based Reasoning Workshop, Seattle, Washington, pp. 23–29 (1994)Google Scholar
  9. 9.
    Díaz-Agudo, B., González-Calero, P.A.: An Architecture for Knowledge Intensive CBR Systems. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 37–48. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  10. 10.
    Aha, D.W.: Generalizing from Case Studies: A Case Study. In: Sleeman, D.H., Edwards, P. (eds.) Proceedings of the Ninth International Workshop on Machine Learning, Aberdeen, Scotland, UK, pp. 1–10. Morgan Kaufmann, San Francisco (1992)Google Scholar
  11. 11.
    Aamodt, A.: Knowledge-Intensive Case-Based Reasoning in Creek. In: Funk, P., González-Calero, P.A. (eds.) 7th European Conference on Case-Based Reasoning, Madrid, Spain, pp. 1–15 (2004)Google Scholar
  12. 12.
    Sørmo, F., Cassens, J., Aamodt, A.: Explanation in Case-Based Reasoning-Perspectives and Goals. Artificial Intelligence Review 24, 109–143 (2005)CrossRefGoogle Scholar
  13. 13.
    Newman, D., Hettich, S., Blake, C., Merz, C.: UCI Repository of Machine Learning Databases (1998), http://www.ics.uci.edu/~mlearn/mlrepository.html
  14. 14.
    Doyle, M., Cunningham, P.: A Dynamic Approach to Reducing Dialog in On-line Decision Guides. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 49–60. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  15. 15.
    Tong, X., Öztürk, P., Gu, M.: Dynamic Feature Weighting in Nearest Neighbor Classifiers. In: Proceedings of the 3rd International Conference on Machine Learning and Cybe (ICMLC 2004), Shanghai, China, vol. 4, pp. 2406–2411 (2004)Google Scholar
  16. 16.
    Yang, Q., Wu, J.: Enhancing the Effectiveness of Interactive Case-Based Reasoning with Clustering and Decision Forests. Applied Intelligence 12, 49–64 (2001)CrossRefGoogle Scholar
  17. 17.
    Bogaerts, S., Leake, D.: Facilitating CBR for Incompletely-Described Cases: Distance Metrics for Partial Problem Descriptions. In: Funk, P., González Calero, P.A. (eds.) ECCBR 2004. LNCS, vol. 3155, pp. 62–76. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  18. 18.
    Bareiss, R.: The Experimental Evaluation of a Case-Based Learning Apprentice. In: The proceedings of the Case-Based Reasoning Workshop, Pensacola Beach, Florida, pp. 162–167 (1989)Google Scholar
  19. 19.
    McLaren, B.M.: Extensionally Defining Principles and Cases in Ethics: an AI Model. Artificial Intelligence Journal 150, 145–181 (2003)MATHCrossRefGoogle Scholar
  20. 20.
    Aha, D.W., Breslow, L., Muñoz-Avila, H.: Conversational Case-Based Reasoning. Applied Intelligence: The International Journal of Artificial Intelligence, Neural Networks, and Complex Problem-Solving Technologies 14, 9 (2001)MATHGoogle Scholar
  21. 21.
    McSherry, D.: Minimizing Dialog Length in Interactive Case-Based Reasoning. In: International Joint Conferences on Artificial Intelligence, pp. 993–998 (2001)Google Scholar
  22. 22.
    Gupta, K.M., Aha, D.W., Sandhu, N.: Exploiting Taxonomic and Causal Relations in Conversational Case Retrieval. In: European Conference on Case- Based Reasoning, Aberdeen, Scotland, UK, pp. 133–147 (2002)Google Scholar
  23. 23.
    Gu, M., Aamodt, A.: A Knowledge-Intensive Method for Conversational CBR. In: Muñoz-Ávila, H., Ricci, F. (eds.) ICCBR 2005. LNCS, vol. 3620, pp. 296–311. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  24. 24.
    Gu, M., Tong, X., Aamodt, A.: Comparing Similarity Calculation Methods in Conversational CBR. In: Zhang, D., Khoshgoftaar, T.M., Shyu, M.L. (eds.) Proceedings of the 2005 IEEE International Conference on Information Reuse and Integration, Hilton, Las Vegas, Nevada, USA, pp. 427–432 (2005)Google Scholar
  25. 25.
    Gu, M., Aamodt, A.: Dialog Learning in Conversational CBR. In: Proceedings of the 19th International FLAIRS Conference, Melbourne Beach, Florida. AAAI Press (to appear, 2006)Google Scholar
  26. 26.
    Gu, M., Bø, K.: Component retrieval using knowledge-intensive conversational CBR. In: Ali, M., Dapoigny, R. (eds.) IEA/AIE 2006. LNCS, vol. 4031, pp. 554–563. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  27. 27.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann Series in Data Management Systems. Morgan Kaufmann, San Francisco (2005)MATHGoogle Scholar
  28. 28.
    Aha, D.W., McSherry, D., Yang, Q.: Advances in Conversational Case-Based Reasoning. Knowledge Engineering Review 20, 7 (2006)Google Scholar
  29. 29.
    Göker, M.H., Thompson, C.A.: Personalized Conversational Case-Based Recommendation. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 99–111. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  30. 30.
    Shimazu, H.: Expertclerk: A Conversational Case-Based Reasoning Tool for Developing Salesclerk Agents in E-Commerce Webshops. Artificial Intelligence Review 18, 223–244 (2002)CrossRefGoogle Scholar
  31. 31.
    Ferguson, A., Bridge, D.G.: Partial Orders and Indifference Relations: Being Purposefully Vague in Case-Based Retrieval. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS (LNAI), vol. 1898, pp. 74–85. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  32. 32.
    Gu, M., Aamodt, A., Tong, X.: Component Retrieval Using Conversational Case-Based Reasoning. In: Shi, Z., He, Q. (eds.) Intelligent Information Processing II. IFIP International Federation for Information Processing, vol. 163. Springer Science + Business Media Inc. (2004) (2004)Google Scholar
  33. 33.
    McSherry, D.: Explanation in Recommender Systems. Artificial Intelligence Review 24, 179–197 (2005)MATHCrossRefGoogle Scholar
  34. 34.
    Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Explaining compound critiques. Artificial Intelligence Review 24, 199–220 (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Mingyang Gu
    • 1
  • Agnar Aamodt
    • 1
  1. 1.Department of Computer and Information ScienceNorwegian University of Science and TechnologyTrondheimNorway

Personalised recommendations