Designing and Evaluating Explanations for Recommender Systems

  • Nava TintarevEmail author
  • Judith Masthoff


This chapter gives an overview of the area of explanations in recommender systems. We approach the literature from the angle of evaluation: that is, we are interested in what makes an explanation “good”, and suggest guidelines as how to best evaluate this. We identify seven benefits that explanations may contribute to a recommender system, and relate them to criteria used in evaluations of explanations in existing systems, and how these relate to evaluations with live recommender systems. We also discuss how explanations can be affected by how recommendations are presented, and the role the interaction with the recommender system plays w.r.t. explanations. Finally, we describe a number of explanation styles, and how they may be related to the underlying algorithms. Examples of explanations in existing systems are mentioned throughout.


Recommender System Recommendation Algorithm Style Explanation Recommendation Process Explanatory Criterion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pandora (2006).
  2. 2.
    Movielens dataset (2009).
  3. 3.
    Netflix dataset (2009).
  4. 4.
  5. 5.
    Nutking (2010). do?action=english
  6. 6.
    Adrissono, L., Goy, A., Petrone, G., Segnan, M., Torasso, P.: Intrigue: Personalized recommendation of tourist attractions for desktop and handheld devices. Applied Artificial Intelligence 17, 687–714 (2003)CrossRefGoogle Scholar
  7. 7.
    Ahn, J.W., Brusilovsky, P., Grady, J., He, D., Syn, S.Y.: Open user profiles for adaptive news systems: help or harm? In: WWW ’07: Proceedings of the 16th international conference on World Wide Web, pp. 11–20. ACM Press, New York, NY, USA (2007).CrossRefGoogle Scholar
  8. 8.
    Andersen, S.K., Olesen, K.G., Jensen, F.V.: HUGIN—a shell for building Bayesian belief universes for expert systems. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1990)Google Scholar
  9. 9.
    Bederson, B., Shneiderman, B., Wattenberg, M.: Ordered and quantum treemaps: Making effective use of 2d space to display hierarchies. ACM Transactions on Graphics 21(4), 833–854. (2002)CrossRefGoogle Scholar
  10. 10.
    Bennett, S.W., Scott., A.C.: The Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project, chap. 19 - Specialized Explanations for Dosage Selection, pp. 363–370. Addison-Wesley Publishing Company (1985)Google Scholar
  11. 11.
    Bilgic, M., Mooney, R.J.: Explaining recommendations: Satisfaction vs. promotion. In: Proceedings of the Wokshop Beyond Personalization, in conjunction with the International Conference on Intelligent User Interfaces, pp. 13–18 (2005)Google Scholar
  12. 12.
    Billsus, D., Pazzani, M.J.: A personal news agent that talks, learns, and explains. In: Proceedings of the Third International Conference on Autonomous Agents, pp. 268–275 (1999)Google Scholar
  13. 13.
    Burke, R.: Hybrid recommender systems: Survey and experiments. User Modeling and User- Adapted Interaction 12(4), 331–370 (2002)zbMATHCrossRefGoogle Scholar
  14. 14.
    Burke, R.D., Hammond, K.J., Young, B.C.: Knowledge-based navigation of complex information spaces. In: AAAI/IAAI, Vol. 1, pp. 462–468 (1996)Google Scholar
  15. 15.
    Carenini, G., Mittal, V., Moore, J.: Generating patient-specific interactive natural language explanations. Proc Annu Symp Comput Appl Med Care pp. 5–9 (1994)Google Scholar
  16. 16.
    Chen, L., Pu, P.: Trust building in recommender agents. In: WPRSIUI in conjunction with Intelligent User Interfaces, pp. 93–100 (2002)Google Scholar
  17. 17.
    Chen, L., Pu, P.: Hybrid critiquing-based recommender systems. In: Intelligent User Interfaces, pp. 22–31 (2007)Google Scholar
  18. 18.
    Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J.: Is seeing believing?: how recommender system interfaces affect users’ opinions. In: CHI, Recommender systems and social computing, vol. 1, pp. 585–592 (2003).Google Scholar
  19. 19.
    Cramer, H., Evers, V., Someren, M.V., Ramlal, S., Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.: The effects of transparency on perceived and actual competence of a contentbased recommender. In: Semantic Web User Interaction Workshop, CHI (2008)Google Scholar
  20. 20.
    Cramer, H.S.M., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.J.: The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User-Adapt. Interact 18(5), 455–496 (2008).CrossRefGoogle Scholar
  21. 21.
    Czarkowski, M.: A scrutable adaptive hypertext. Ph.D. thesis, University of Sydney (2006)Google Scholar
  22. 22.
    Doyle, D., Tsymbal, A., Cunningham, P.: A review of explanation and explanation in casebased reasoning. Tech. rep., Department of Computer Science, Trinity College, Dublin (2003)Google Scholar
  23. 23.
    Felfernig, A., Gula, B.: Consumer behavior in the interaction with knowledge-based recommender applications. In: ECAI 2006Workshop on Recommender Systems, pp. 37–41 (2006)Google Scholar
  24. 24.
    Fogg, B., Marshall, J., Kameda, T., Solomon, J., Rangnekar, A., Boyd, J., Brown, B.: Web credibility research: A method for online experiments and early study results. In: CHI 2001, pp. 295–296 (2001)Google Scholar
  25. 25.
    Fogg, B.J., Soohoo, C., Danielson, D.R., Marable, L., Stanford, J., Tauber, E.R.: How do users evaluate the credibility of web sites?: a study with over 2,500 participants. In: Proceedings of DUX’03: Designing for User Experiences, no. 15 in Focusing on user-to-product relationships, pp. 1–15 (2003). URL
  26. 26.
    Ginty, L.M., Smyth, B.: Comparison-based recommendation. Lecture Notes in Computer Science 2416, 731–737 (2002).Google Scholar
  27. 27.
    Hance, E., Buchanan, B.: Rule-based expert systems: the MYCIN experiments of the Stanford Heuristic Programming Project. Addison-Wesley (1984)Google Scholar
  28. 28.
    Häubl, G., Trifts, V.: Consumer decision making in online shopping environments: The effects of interactive decision aids. Marketing Science 19, 4–21 (2000)CrossRefGoogle Scholar
  29. 29.
    Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: ACM conference on Computer supported cooperative work, pp. 241–250 (2000)Google Scholar
  30. 30.
    Hingston, M.: User friendly recommender systems. Master’s thesis, Sydney University (2006)Google Scholar
  31. 31.
    Hu, Y., Koren, Y., Volinsky, C.: Collaborative filtering for implicit feedback datasets. In: ICDM (2008)Google Scholar
  32. 32.
    Hunt, J.E., Price, C.J.: Explaining qualitative diagnosis. Engineering Applications of Artificial Intelligence 1(3), Pages 161–169 (1988)CrossRefGoogle Scholar
  33. 33.
    Khan, O.Z., Poupart, P., Black, J.P.: Minimal sufficient explanations for mdps. In: Workshop on Explanation-Aware Computing associated with IJCAI (2009)Google Scholar
  34. 34.
    Krulwich, B.: The infofinder agent: Learning user interests through heuristic phrase extraction. IEEE Intelligent Systems 12, 22–27 (1997)Google Scholar
  35. 35.
    Lacave, C., Diéz, F.J.: A review of explanation methods for bayesian networks. The Knowledge Engineering Review 17:2, 107–127 (2002)Google Scholar
  36. 36.
    Lacave, C., Diéz, F.J.: A review of explanation methods for heuristic expert systems. The Knowledge Engineering Review 17:2, 107–127 (2004)Google Scholar
  37. 37.
    Lewis, C., Rieman, J.: Task-centered user interface design: a practical introduction. University of Colorado (1994)Google Scholar
  38. 38.
    Lopez-Suarez, A., Kamel, M.: Dykor: a method for generating the content of explanations in knowledge systems. Knowledge-based Systems 7(3), 177–188 (1994)CrossRefGoogle Scholar
  39. 39.
    McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Thinking positively - explanatory feedback for conversational recommender systems. In: Proceedings of the European Conference on Case-Based Reasoning (ECCBR-04) Explanation Workshop,, pp. 115–124 (2004)Google Scholar
  40. 40.
    McNee, S., Lam S.K.and Guetzlaff, C., Konstan J.A.and Riedl, J.: Confidence displays and training in recommender systems. In: INTERACT IFIP TC13 International Conference on Human-Computer Interaction, pp. 176–183 (2003)Google Scholar
  41. 41.
    McNee, S.M., Lam, S.K., Konstan, J.A., Riedl, J.: Interfaces for eliciting new user preferences in recommender systems. User Modeling pp. pp. 178–187 (2003)Google Scholar
  42. 42.
    McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: How accuracy metrics have hurt recommender systems. In: Extended Abstracts of the 2006 ACM Conference on Human Factors in Computing Systems (CHI 2006) (2006)Google Scholar
  43. 43.
    McSherry, D.: Explanation in recommender systems. Artificial Intelligence Review 24(2), 179 – 197 (2005)zbMATHCrossRefGoogle Scholar
  44. 44.
    Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: ACM CHI’90, pp. 249–256 (1990)Google Scholar
  45. 45.
    Ohanian, R.: Construction and validation of a scale to measure celebrity endorsers’ perceived expertise, trustworthiness, and attractiveness. Journal of Advertising 19:3, 39–52 (1990)Google Scholar
  46. 46.
    O’Sullivan, D., Smyth, B., Wilson, D.C., McDonald, K., Smeaton, A.: Improving the quality of the personalized electronic program guide. User Modeling and User-Adapted Interaction 14, pp. 5–36 (2004)CrossRefGoogle Scholar
  47. 47.
    Paramythis, A., Totter, A., Stephanidis, C.: A modular approach to the evaluation of adaptive user interfaces. In: S.Weibelzahl, D.N. Chin, G.Weber (eds.) Evaluation of Adaptive Systems in conjunction with UM’01, pp. 9–24 (2001)Google Scholar
  48. 48.
    Pazzani, M.J.: A framework for collaborative, content-based and demographic filtering. Artificial Intelligence Review 13, 393–408 (1999)CrossRefGoogle Scholar
  49. 49.
    Pu, P., Chen, L.: Trust building with explanation interfaces. In: IUI’06, Recommendations I, pp. 93–100 (2006).Google Scholar
  50. 50.
    Pu, P., Chen, L.: Trust-inspiring explanation interfaces for recommender systems. Knowledge-based Systems 20, 542–556 (2007)CrossRefMathSciNetGoogle Scholar
  51. 51.
    Rafter, R., Smyth, B.: Conversational collaborative recommendation - an experimental analysis. Artif. Intell. Rev 24(3-4), 301–318 (2005). URL
  52. 52.
    Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Dynamic critiquing. In: P. Funk, P.A. González-Calero (eds.) ECCBR, Lecture Notes in Computer Science, vol. 3155, pp. 763–777. Springer (2004)Google Scholar
  53. 53.
    Roth-Berghofer, T., Schulz, S., Leake, D.B., Bahls, D.:Workshop on explanation-aware computing. In: ECAI (2008)Google Scholar
  54. 54.
    Roth-Berghofer, T., Tintarev, N., Leake, D.B.: Workshop on explanation-aware computing. In: IJCAI (2009)Google Scholar
  55. 55.
    Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: Conference on Human Factors in Computing Systems, pp. 830–831 (2002)Google Scholar
  56. 56.
    Sørmo, F., Cassens, J., Aamodt, A.: Explanation in case-based reasoning perspectives and goals. Artificial Intelligence Review 24(2), 109 – 143 (2005)CrossRefGoogle Scholar
  57. 57.
    Swearingen, K., Sinha, R.: Interaction design for recommender systems. In: Designing Interactive Systems, pp. 25–28 (2002).Google Scholar
  58. 58.
    Symeonidis, P., Nanopoulos, A., Manolopoulos, Y.: Justified recommendations based on content and rating data. In:WebKDDWorkshop onWeb Mining andWeb Usage Analysis (2008)Google Scholar
  59. 59.
    Thompson, C.A., G¨oker, M.H., Langley, P.: A personalized system for conversational recommendations. J. Artif. Intell. Res. (JAIR) 21, 393–428 (2004).Google Scholar
  60. 60.
    Tintarev, N., Masthoff, J.: Over- and underestimation in different product domains. In:Workshop on Recommender Systems associated with ECAI (2008)Google Scholar
  61. 61.
    Tintarev, N., Masthoff, J.: Personalizing movie explanations using commercial meta-data. In: Adaptive Hypermedia (2008)Google Scholar
  62. 62.
    Vig, J., Sen, S., Riedl, J.: Tagsplanations: Explaining recommendations using tags. In: Intelligent User Interfaces (2009)Google Scholar
  63. 63.
    Wang, W., Benbasat, I.: Recommendation agents for electronic commerce: Effects of explanation facilities on trusting beliefs. Journal of Managment Information Systems 23, 217–246 (2007)CrossRefGoogle Scholar
  64. 64.
    Wärnestål, P.: Modeling a dialogue strategy for personalized movie recommendations. In: Beyond Personalization Workshop, pp. 77–82 (2005)Google Scholar
  65. 65.
    Wärnestål, P.: User evaluation of a conversational recommender system. In: Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pp. 32–39 (2005)Google Scholar
  66. 66.
    Wick, M.R., Thompson, W.B.: Reconstructive expert system explanation. Artif. Intell. 54(1- 2), 33–70 (1992).CrossRefGoogle Scholar
  67. 67.
    Ye, L., Johnson, P., Ye, L.R., Johnson, P.E.: The impact of explanation facilities on user acceptance of expert systems advice. MIS Quarterly 19(2), 157–172 (1995).CrossRefGoogle Scholar
  68. 68.
    Yee, K.P., Swearingen, K., Li, K., Hearst, M.: Faceted metadata for image search and browsing. In: ACM Conference on Computer-Human Interaction (2003)Google Scholar
  69. 69.
    Zaslow, J.: Oh no! My TiVo thinks I’m gay (2002).Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.University of AberdeenAberdeenU.K

Personalised recommendations