Skip to main content
SpringerLink
Log in

Evaluating CBR Systems Using Different Data Sources: A Case Study

  • Conference paper
Advances in Case-Based Reasoning (ECCBR 2006)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4106))

Included in the following conference series:

Abstract

The complexity and high construction cost of case bases make it very difficult, if not impossible, to evaluate a CBR system, especially a knowledge-intensive CBR system, using statistical evaluation methods on many case bases. In this paper, we propose an evaluation strategy, which uses both many simple case bases and a few complex case bases to evaluate a CBR system, and show how this strategy may satisfy different evaluation goals. The identified evaluation goals are classified into two categories: domain-independent and domain-dependent. For the evaluation goals in the first category, we apply the statistical evaluation method using many simple case bases (for example, UCI data sets); for evaluation goals in the second category, we apply different, relatively weak, evaluation methods on a few complex domain-specific case bases. We apply this combined evaluation strategy to evaluate our knowledge-intensive conversational CBR method as a case study.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Simon, H.A.: Artificial Intelligence: an Empirical Science. Artif. Intell. 77, 95–127 (1995)

    Article  Google Scholar 

  2. Cohen, P.R., Howe, A.E.: How Evaluation Guides AI Research. AI Mag. 9, 35–43 (1988)

    Google Scholar 

  3. Cohen, P., Howe, A.: Toward AI Research Methodology: Three Case Studies in Evaluation. Systems, Man and Cybernetics, IEEE Transactions 19, 634–646 (1989)

    Article  Google Scholar 

  4. Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT Press, Cambridge (1995)

    MATH  Google Scholar 

  5. McSherry, D.: Interactive Case-Based Reasoning in Sequential Diagnosis. Applied Intelligence 14, 65–76 (2001)

    Article  MATH  Google Scholar 

  6. Aamodt, A., Plaza, E.: Case-Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches. AI Communications 7, 39–59 (1994)

    Google Scholar 

  7. Kolodner, J.: Case-Based Reasoning. Morgan Kaufmann, San Francisco (1993)

    Google Scholar 

  8. Santamaria, J.C., Ram, A.: Systematic Evaluation of Design Decisions in CBR Systems. In: Proceedings of the AAAI Case-Based Reasoning Workshop, Seattle, Washington, pp. 23–29 (1994)

    Google Scholar 

  9. Díaz-Agudo, B., González-Calero, P.A.: An Architecture for Knowledge Intensive CBR Systems. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 37–48. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  10. Aha, D.W.: Generalizing from Case Studies: A Case Study. In: Sleeman, D.H., Edwards, P. (eds.) Proceedings of the Ninth International Workshop on Machine Learning, Aberdeen, Scotland, UK, pp. 1–10. Morgan Kaufmann, San Francisco (1992)

    Google Scholar 

  11. Aamodt, A.: Knowledge-Intensive Case-Based Reasoning in Creek. In: Funk, P., González-Calero, P.A. (eds.) 7th European Conference on Case-Based Reasoning, Madrid, Spain, pp. 1–15 (2004)

    Google Scholar 

  12. Sørmo, F., Cassens, J., Aamodt, A.: Explanation in Case-Based Reasoning-Perspectives and Goals. Artificial Intelligence Review 24, 109–143 (2005)

    Article  Google Scholar 

  13. Newman, D., Hettich, S., Blake, C., Merz, C.: UCI Repository of Machine Learning Databases (1998), http://www.ics.uci.edu/~mlearn/mlrepository.html

  14. Doyle, M., Cunningham, P.: A Dynamic Approach to Reducing Dialog in On-line Decision Guides. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 49–60. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  15. Tong, X., Öztürk, P., Gu, M.: Dynamic Feature Weighting in Nearest Neighbor Classifiers. In: Proceedings of the 3rd International Conference on Machine Learning and Cybe (ICMLC 2004), Shanghai, China, vol. 4, pp. 2406–2411 (2004)

    Google Scholar 

  16. Yang, Q., Wu, J.: Enhancing the Effectiveness of Interactive Case-Based Reasoning with Clustering and Decision Forests. Applied Intelligence 12, 49–64 (2001)

    Article  Google Scholar 

  17. Bogaerts, S., Leake, D.: Facilitating CBR for Incompletely-Described Cases: Distance Metrics for Partial Problem Descriptions. In: Funk, P., González Calero, P.A. (eds.) ECCBR 2004. LNCS, vol. 3155, pp. 62–76. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  18. Bareiss, R.: The Experimental Evaluation of a Case-Based Learning Apprentice. In: The proceedings of the Case-Based Reasoning Workshop, Pensacola Beach, Florida, pp. 162–167 (1989)

    Google Scholar 

  19. McLaren, B.M.: Extensionally Defining Principles and Cases in Ethics: an AI Model. Artificial Intelligence Journal 150, 145–181 (2003)

    Article  MATH  Google Scholar 

  20. Aha, D.W., Breslow, L., Muñoz-Avila, H.: Conversational Case-Based Reasoning. Applied Intelligence: The International Journal of Artificial Intelligence, Neural Networks, and Complex Problem-Solving Technologies 14, 9 (2001)

    MATH  Google Scholar 

  21. McSherry, D.: Minimizing Dialog Length in Interactive Case-Based Reasoning. In: International Joint Conferences on Artificial Intelligence, pp. 993–998 (2001)

    Google Scholar 

  22. Gupta, K.M., Aha, D.W., Sandhu, N.: Exploiting Taxonomic and Causal Relations in Conversational Case Retrieval. In: European Conference on Case- Based Reasoning, Aberdeen, Scotland, UK, pp. 133–147 (2002)

    Google Scholar 

  23. Gu, M., Aamodt, A.: A Knowledge-Intensive Method for Conversational CBR. In: Muñoz-Ávila, H., Ricci, F. (eds.) ICCBR 2005. LNCS, vol. 3620, pp. 296–311. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  24. Gu, M., Tong, X., Aamodt, A.: Comparing Similarity Calculation Methods in Conversational CBR. In: Zhang, D., Khoshgoftaar, T.M., Shyu, M.L. (eds.) Proceedings of the 2005 IEEE International Conference on Information Reuse and Integration, Hilton, Las Vegas, Nevada, USA, pp. 427–432 (2005)

    Google Scholar 

  25. Gu, M., Aamodt, A.: Dialog Learning in Conversational CBR. In: Proceedings of the 19th International FLAIRS Conference, Melbourne Beach, Florida. AAAI Press (to appear, 2006)

    Google Scholar 

  26. Gu, M., Bø, K.: Component retrieval using knowledge-intensive conversational CBR. In: Ali, M., Dapoigny, R. (eds.) IEA/AIE 2006. LNCS, vol. 4031, pp. 554–563. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  27. Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann Series in Data Management Systems. Morgan Kaufmann, San Francisco (2005)

    MATH  Google Scholar 

  28. Aha, D.W., McSherry, D., Yang, Q.: Advances in Conversational Case-Based Reasoning. Knowledge Engineering Review 20, 7 (2006)

    Google Scholar 

  29. Göker, M.H., Thompson, C.A.: Personalized Conversational Case-Based Recommendation. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS, vol. 1898, pp. 99–111. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  30. Shimazu, H.: Expertclerk: A Conversational Case-Based Reasoning Tool for Developing Salesclerk Agents in E-Commerce Webshops. Artificial Intelligence Review 18, 223–244 (2002)

    Article  Google Scholar 

  31. Ferguson, A., Bridge, D.G.: Partial Orders and Indifference Relations: Being Purposefully Vague in Case-Based Retrieval. In: Blanzieri, E., Portinale, L. (eds.) EWCBR 2000. LNCS (LNAI), vol. 1898, pp. 74–85. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  32. Gu, M., Aamodt, A., Tong, X.: Component Retrieval Using Conversational Case-Based Reasoning. In: Shi, Z., He, Q. (eds.) Intelligent Information Processing II. IFIP International Federation for Information Processing, vol. 163. Springer Science + Business Media Inc. (2004) (2004)

    Google Scholar 

  33. McSherry, D.: Explanation in Recommender Systems. Artificial Intelligence Review 24, 179–197 (2005)

    Article  MATH  Google Scholar 

  34. Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Explaining compound critiques. Artificial Intelligence Review 24, 199–220 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, Norwegian University of Science and Technology, Sem Saelands vei 7-9, N-7491, Trondheim, Norway

    Mingyang Gu & Agnar Aamodt

Authors
  1. Mingyang Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Agnar Aamodt
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Knowledge Management Department, German Research Center for Artificial Intelligence (DFKI) GmbH, Trippstadter Straße 122, 67663, Kaiserslautern, Germany

    Thomas R. Roth-Berghofer

  2. PricewaterhouseCoopers LLP, Center for Advanced Research, Ten Almaden Blvd, Suite 1600, 95113, San Jose, CA

    Mehmet H. Göker

  3. Department of Computer Engineering, Bilkent University, 06800, Ankara, Turkey

    H. Altay Güvenir

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Gu, M., Aamodt, A. (2006). Evaluating CBR Systems Using Different Data Sources: A Case Study. In: Roth-Berghofer, T.R., Göker, M.H., Güvenir, H.A. (eds) Advances in Case-Based Reasoning. ECCBR 2006. Lecture Notes in Computer Science(), vol 4106. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11805816_11

Download citation

  • DOI: https://doi.org/10.1007/11805816_11

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-36843-4

  • Online ISBN: 978-3-540-36846-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation