Explaining Recommendations: Design and Evaluation


This chapter gives an overview of the area of explanations in recommender systems. We approach the literature from the angle of evaluation: that is, we are interested in what makes an explanation “good”. The chapter starts by describing how explanations can be affected by how recommendations are presented, and the role the interaction with the recommender system plays w.r.t. explanations. Next, we introduce a number of explanation styles, and how they are related to the underlying algorithms. We identify seven benefits that explanations may contribute to a recommender system, and relate them to criteria used in evaluations of explanations in existing recommender systems. We conclude the chapter with outstanding research questions and future work, including current recommender systems topics such as social recommendations and serendipity. Examples of explanations in existing systems are mentioned throughout.


Recommender System Recommendation Algorithm Preference Elicitation Style Explanation User Trust 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Pandora (2006).
  2. 2.
  3. 3.
    Adamopoulos, P., Tuzhilin, A.: On unexpectedness in recommender systems: Or how to except the unexpected. In: Workshop on Novelty and Diversity in Recommender Systems in conjuction with Recsys (2011)Google Scholar
  4. 4.
    Adrissono, L., Goy, A., Petrone, G., Segnan, M., Torasso, P.: Intrigue: Personalized recommendation of tourist attractions for desktop and handheld devices. Applied Artificial Intelligence 17, 687–714 (2003)CrossRefGoogle Scholar
  5. 5.
    Ahn, J.W., Brusilovsky, P., Grady, J., He, D., Syn, S.Y.: Open user profiles for adaptive news systems: help or harm? In: World Wide Web (WWW), pp. 11–20. ACM Press, New York, NY, USA (2007)Google Scholar
  6. 6.
    Andersen, S.K., Olesen, K.G., Jensen, F.V.: HUGIN—a shell for building Bayesian belief universes for expert systems. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1990)Google Scholar
  7. 7.
    Backstrom, L., Sun, E., Marlow, C.: Find me if you can: Improving geographical prediction with social and spatial proximity. In: World Wide Web (WWW) (2010)Google Scholar
  8. 8.
    Bennett, S.W., Scott., A.C.: The Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project, chap. 19 - Specialized Explanations for Dosage Selection, pp. 363–370. Addison-Wesley Publishing Company (1985)Google Scholar
  9. 9.
    Bilgic, M., Mooney, R.J.: Explaining recommendations: Satisfaction vs. promotion. In: Proceedings of the Wokshop Beyond Personalization, in conjunction with the International Conference on Intelligent User Interfaces, pp. 13–18 (2005)Google Scholar
  10. 10.
    Billsus, D., Pazzani, M.J.: A personal news agent that talks, learns, and explains. In: Proceedings of the Third International Conference on Autonomous Agents, pp. 268–275 (1999)Google Scholar
  11. 11.
    Bridge, D., Kelly, J.P.: Ways of computing diverse collaborative recommendations. In: Adaptive Hypermedia and Adaptive Web-based Systems (2006)CrossRefGoogle Scholar
  12. 12.
    Burke, R.: Hybrid recommender systems: Survey and experiments. User Modeling and User-Adapted Interaction 12(4), 331–370 (2002)MATHCrossRefGoogle Scholar
  13. 13.
    Burke, R.D., Hammond, K.J., Young, B.C.: Knowledge-based navigation of complex information spaces. In: AAAI/IAAI, Vol. 1, pp. 462–468 (1996)Google Scholar
  14. 14.
    Carenini, G., Mittal, V., Moore, J.: Generating patient-specific interactive natural language explanations. Proc Annu Symp Comput Appl Med Care pp. 5–9 (1994)Google Scholar
  15. 15.
    Chen, L., Pu, P.: Trust building in recommender agents. In: WPRSIUI in conjunction with Intelligent User Interfaces, pp. 93–100 (2002)Google Scholar
  16. 16.
    Chen, L., Pu, P.: Hybrid critiquing-based recommender systems. In: Intelligent User Interfaces, pp. 22–31 (2007)Google Scholar
  17. 17.
    Chen, L., Pu, P.: Interaction design guidelines on critiquing-based recommender systems. User Modeling and User-Adapted Interaction 3, 167–206 (2009)MATHCrossRefGoogle Scholar
  18. 18.
    Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J.: Is seeing believing?: how recommender system interfaces affect users’ opinions. In: CHI, Recommender systems and social computing, vol. 1, pp. 585–592 (2003)Google Scholar
  19. 19.
    Cramer, H., Evers, V., Someren, M.V., Ramlal, S., Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.: The effects of transparency on perceived and actual competence of a content-based recommender. In: Semantic Web User Interaction Workshop, CHI (2008)Google Scholar
  20. 20.
    Cramer, H.S.M., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.J.: The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User-Adapt. Interact 18(5), 455–496 (2008)CrossRefGoogle Scholar
  21. 21.
    Czarkowski, M.: A scrutable adaptive hypertext. Ph.D. thesis, University of Sydney (2006)Google Scholar
  22. 22.
    Darlington, K.: Aspects of intelligent systems explanation. Universal Journal of Control and Automation 1, 40–51 (2013)CrossRefGoogle Scholar
  23. 23.
    Doyle, D., Tsymbal, A., Cunningham, P.: A review of explanation and explanation in case-based reasoning. Tech. rep., Department of Computer Science, Trinity College, Dublin (2003)Google Scholar
  24. 24.
    Erlich, K., Kirk, S., Patterson, J., Rasmussen, J., Ross, S., Gruen, D.: Taking advice from intelligent systems: The double-edged sword of explanations. In: Intelligent User Interfaces (2011)CrossRefGoogle Scholar
  25. 25.
    Felfernig, A., Gula, B.: Consumer behavior in the interaction with knowledge-based recommender applications. In: ECAI 2006 Workshop on Recommender Systems, pp. 37–41 (2006)Google Scholar
  26. 26.
    Felfernig, A., Teppan, E., Gula, B.: Knowledge-based recommender technologies for marketing and sales. Int. J. Patt. Recogn. Artif. Intell. 21, 333–355 (2007)CrossRefGoogle Scholar
  27. 27.
    Fogg, B., Marshall, J., Kameda, T., Solomon, J., Rangnekar, A., Boyd, J., Brown, B.: Web credibility research: A method for online experiments and early study results. In: CHI 2001, pp. 295–296 (2001)Google Scholar
  28. 28.
    Fogg, B.J., Soohoo, C., Danielson, D.R., Marable, L., Stanford, J., Tauber, E.R.: How do users evaluate the credibility of web sites?: a study with over 2,500 participants. In: Designing for User Experiences (DUX), no. 15 in Focusing on user-to-product relationships, pp. 1–15 (2003)Google Scholar
  29. 29.
    Gedikli, F., Jannach, D., Ge, M.: How should I explain? A comparison of different explanation types of recommender systems. International Journal of Human-Computer Studies 72(4), 367–382 (2014)CrossRefGoogle Scholar
  30. 30.
    Green, S., Lamere, P., Alexander, J., Maillet, F.: Generating transparent, steerable recommendations from textual descriptions of items. In: Recommender Systems Conference (2009)CrossRefGoogle Scholar
  31. 31.
    Gretarsson, B., O’Donovan, J., Bostandjiev, S., Hall, C., Höllerer, T.: Smallworlds: Visualizing social recommendations. Computer Graphics Forum 29(3), 833–842 (2010)CrossRefGoogle Scholar
  32. 32.
    Guy, I., Ronen, I., Wilcox, E.: Do you know? recommending people to invite into your social network. In: International Conference on Intelligent User Interfaces, pp. 77–86 (2009)Google Scholar
  33. 33.
    Hance, E., Buchanan, B.: Rule-based expert systems: the MYCIN experiments of the Stanford Heuristic Programming Project. Addison-Wesley (1984)Google Scholar
  34. 34.
    Häubl, G., Trifts, V.: Consumer decision making in online shopping environments: The effects of interactive decision aids. Marketing Science 19, 4–21 (2000)CrossRefGoogle Scholar
  35. 35.
    Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: ACM conference on Computer supported cooperative work, pp. 241–250 (2000)Google Scholar
  36. 36.
    Hingston, M.: User friendly recommender systems. Master’s thesis, Sydney University (2006)Google Scholar
  37. 37.
    Hu, R., Pu, P.: Helping users perceive recommendation diversity. In: Workshop on Novelty and Diversity in Recommender Systems in conjunction with Recsys (2011)Google Scholar
  38. 38.
    Hunt, J.E., Price, C.J.: Explaining qualitative diagnosis. Engineering Applications of Artificial Intelligence 1(3), Pages 161–169 (1988)CrossRefGoogle Scholar
  39. 39.
    Krulwich, B.: The infofinder agent: Learning user interests through heuristic phrase extraction. IEEE Intelligent Systems 12, 22–27 (1997)Google Scholar
  40. 40.
    Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., Wong, W.K.: Conference on visual languages and human-centric computing. In: Too Much, Too Little, or Just Right? Ways Explanations Impact End Users Mental Models (2013)Google Scholar
  41. 41.
    Lacave, C., Diéz, F.J.: A review of explanation methods for bayesian networks. The Knowledge Engineering Review 17:2, 107–127 (2002)Google Scholar
  42. 42.
    Lacave, C., Diéz, F.J.: A review of explanation methods for heuristic expert systems. The Knowledge Engineering Review 17:2, 107–127 (2004)Google Scholar
  43. 43.
    Lewis, C., Rieman, J.: Task-centered user interface design: a practical introduction. University of Colorado (1994)Google Scholar
  44. 44.
    Lopez-Suarez, A., Kamel, M.: Dykor: a method for generating the content of explanations in knowledge systems. Knowledge-based Systems 7(3), 177–188 (1994)CrossRefGoogle Scholar
  45. 45.
    McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Thinking positively - explanatory feedback for conversational recommender systems. In: Proceedings of the European Conference on Case-Based Reasoning (ECCBR-04) Explanation Workshop, pp. 115–124 (2004)Google Scholar
  46. 46.
    McGinty, L., Reilly, J.: On the evolution of critiquing recommenders. In: F. Ricci, L. Rokach, B. Shapira, P.B. Kantor (eds.) Recommender Systems Handbook, pp. 547–576. Springer US (2011)Google Scholar
  47. 47.
    McGinty, L., Smyth, B.: Comparison-based recommendation. Lecture Notes in Computer Science 2416, 575–589 (2002)CrossRefGoogle Scholar
  48. 48.
    McNee, S.M., Lam, S.K., Konstan, J.A., Riedl, J.: Interfaces for eliciting new user preferences in recommender systems. User Modeling pp. pp. 178–187 (2003)Google Scholar
  49. 49.
    McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: How accuracy metrics have hurt recommender systems. In: Extended Abstracts of the 2006 ACM Conference on Human Factors in Computing Systems (CHI 2006) (2006)Google Scholar
  50. 50.
    McSherry, D.: Explanation in recommender systems. Artificial Intelligence Review 24(2), 179–197 (2005)MATHCrossRefGoogle Scholar
  51. 51.
    Nagulendra, S., Vassileva, J.: Providing awareness, understanding and control of personalized stream filtering in a p2p social network. In: Conference on Collaboration and Technology (CRIWG) (2013)Google Scholar
  52. 52.
    Nguyen, T.T., Kluver, D., Wang, T.Y., Hui, P.M., Ekstrand, M.D., Willemsen, M.C., Rield, J.: Rating support interfaces to improve user experience and recommender accuracy. In: Recommender Systems Conference (2013)CrossRefGoogle Scholar
  53. 53.
    Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: ACM CHI’90, pp. 249–256 (1990)Google Scholar
  54. 54.
    Ohanian, R.: Construction and validation of a scale to measure celebrity endorsers’ perceived expertise, trustworthiness, and attractiveness. Journal of Advertising 19:3, 39–52 (1990)Google Scholar
  55. 55.
    O’Sullivan, D., Smyth, B., Wilson, D.C., McDonald, K., Smeaton, A.: Improving the quality of the personalized electronic program guide. User Modeling and User-Adapted Interaction 14, pp. 5–36 (2004)CrossRefGoogle Scholar
  56. 56.
    Papadimitriou, A., Symeonidis, P., Manolopoulos, Y.: A generalized taxonomy of explanation styles for traditional and social recommender systems. Data Mining and Knowledge Discovery 24, 555–583 (2012)CrossRefGoogle Scholar
  57. 57.
    Pazzani, M.J.: A framework for collaborative, content-based and demographic filtering. Artificial Intelligence Review 13, 393–408 (1999)CrossRefGoogle Scholar
  58. 58.
    Pizzato, L., Rej, T., Akehurst, J., Koprinska, I., Yacef, K., Kay, J.: Recommending people to people: the nature of reciprocal recommenders with a case study in online dating. User Modeling and User-Adapted Interaction 23, 447–488 (2013)CrossRefGoogle Scholar
  59. 59.
    Pu, P., Chen, L.: Trust building with explanation interfaces. In: IUI’06, Recommendations I, pp. 93–100 (2006)Google Scholar
  60. 60.
    Pu, P., Chen, L.: Trust-inspiring explanation interfaces for recommender systems. Knowledge-based Systems 20, 542–556 (2007)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Pu, P., Faltings, B., Chen, L., Zhang, J., Viappiani, P.: Usability guidelines for product recommenders based on example critiquing research. In: F. Ricci, L. Rokach, B. Shapira, P.B. Kantor (eds.) Recommender Systems Handbook, pp. 547–576. Springer US (2011)Google Scholar
  62. 62.
    Rafter, R., Smyth, B.: Conversational collaborative recommendation - an experimental analysis. Artif. Intell. Rev 24(3–4), 301–318 (2005)CrossRefGoogle Scholar
  63. 63.
    Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Dynamic critiquing. In: P. Funk, P.A. González-Calero (eds.) ECCBR, Lecture Notes in Computer Science, vol. 3155, pp. 763–777. Springer (2004)Google Scholar
  64. 64.
    Ricci, F.: Mobile recommender systems. Information Technology & Tourism 12.3, 205–231 (2010)CrossRefGoogle Scholar
  65. 65.
    Said, A., Bellogin Kouki, A., de Vries, A.P., Kille, B.: Information Retrieval And User-Centric Recommender System Evaluation. In: Extended Proceedings of The 21st Conference on User Modeling, Adaptation and Personalization (UMAP’13), CEUR (2013). URL
  66. 66.
    Shani, G., Rokach, L., Shapira, B., Hadash, S., Tangi, M.: Investigating confidence displays for top-n recommendations. Journal of the American Society for Information Science and Technology 64, 2548–2563 (2013)CrossRefGoogle Scholar
  67. 67.
    Sharma, A., Cosley, D.: Do social explanations work? studying and modeling the effects of social explanations in recommender systems. In: World Wide Web (WWW) (2013)Google Scholar
  68. 68.
    Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: Conference on Human Factors in Computing Systems, pp. 830–831 (2002)Google Scholar
  69. 69.
    Swearingen, K., Sinha, R.: Interaction design for recommender systems. In: Designing Interactive Systems, pp. 25–28 (2002)Google Scholar
  70. 70.
    Symeonidis, P., Nanopoulos, A., Manolopoulos, Y.: Justified recommendations based on content and rating data. In: WebKDD Workshop on Web Mining and Web Usage Analysis (2008)Google Scholar
  71. 71.
    Tanaka-Ishii, K., Frank, I.: Multi-agent explanation strategies in real-time domains. In: 38th Annual Meeting on Association for Computational Linguistics, pp. 158–165 (2000)Google Scholar
  72. 72.
    Thompson, C.A., Göker, M.H., Langley, P.: A personalized system for conversational recommendations. J. Artif. Intell. Res. (JAIR) 21, 393–428 (2004)Google Scholar
  73. 73.
    Tintarev, N.: Explaining recommendations. In: User Modeling, pp. 470–474 (2007)Google Scholar
  74. 74.
    Tintarev, N., Dennis, M., Masthoff, J.: Adapting recommendation diversity to openness to experience: A study of human behaviour. In: UMAP (2013)Google Scholar
  75. 75.
    Tintarev, N., Masthoff, J.: Effective explanations of recommendations: User-centered design. In: Recommender Systems, pp. 153–156 (2007)Google Scholar
  76. 76.
    Tintarev, N., Masthoff, J.: Over- and underestimation in different product domains. In: Workshop on Recommender Systems associated with ECAI (2008)Google Scholar
  77. 77.
    Tintarev, N., Masthoff, J.: Personalizing movie explanations using commercial meta-data. In: Adaptive Hypermedia (2008)Google Scholar
  78. 78.
    Tintarev, N., Masthoff, J.: Evaluating the effectiveness of explanations for recommender systems: Methodological issues and empirical studies on the impact of personalization. User Modeling and User-Adapted Interaction 22, 399–439 (2012)CrossRefGoogle Scholar
  79. 79.
    Verbert, K., Parra, D., Brusilovsky, P., Duval, E.: Visualizing recommendations to support exploration, transparency and controllability. In: Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ‘13, pp. 351–362. ACM, New York, NY, USA (2013). DOI 10.1145/2449396.2449442
  80. 80.
    Victor, P., Cock, M.D., Cornelis, C.: Trust and recommendations. In: F. Ricci, L. Rokach, B. Shapira, P.B. Kantor (eds.) Recommender Systems Handbook, pp. 547–576. Springer US (2011)Google Scholar
  81. 81.
    Vig, J., Sen, S., Riedl, J.: Tagsplanations: Explaining recommendations using tags. In: Intelligent User Interfaces (2009)Google Scholar
  82. 82.
    Wang, W., Benbasat, I.: Recommendation agents for electronic commerce: Effects of explanation facilities on trusting beliefs. Journal of Management Information Systems 23, 217–246 (2007)CrossRefGoogle Scholar
  83. 83.
    Wärnestål, P.: Modeling a dialogue strategy for personalized movie recommendations. In: Beyond Personalization Workshop, pp. 77–82 (2005)Google Scholar
  84. 84.
    Wärnestål, P.: User evaluation of a conversational recommender system. In: Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pp. 32–39 (2005)Google Scholar
  85. 85.
    Webster, A., Vassileva, J.: The keeup recommender system. In: Recsys (2007)Google Scholar
  86. 86.
    Wick, M.R., Thompson, W.B.: Reconstructive expert system explanation. Artif. Intell. 54(1–2), 33–70 (1992)CrossRefGoogle Scholar
  87. 87.
    Ye, L., Johnson, P., Ye, L.R., Johnson, P.E.: The impact of explanation facilities on user acceptance of expert systems advice. MIS Quarterly 19(2), 157–172 (1995)CrossRefGoogle Scholar
  88. 88.
    Yee, K.P., Swearingen, K., Li, K., Hearst, M.: Faceted metadata for image search and browsing. In: ACM Conference on Computer-Human Interaction (2003)CrossRefGoogle Scholar
  89. 89.
    Zheng, V.W., Zheng, Y., Xie, X., Yang, Q.: Collaborative location and activity recommendations with GPS history data. In: World Wide Web (WWW) (2010)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.University of AberdeenAberdeenUK

Personalised recommendations