Explaining the user experience of recommender systems

Abstract

Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems.

References

  1. Ackerman, M., Cranor, L., Reagle, J.: Privacy in e-commerce: examining user scenarios and privacy preferences. In: Conference on Electronic Commerce, pp. 1–8. Denver, CO (1999)

  2. Adomavicius G., Tuzhilin A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 734–749 (2005)

    Article  Google Scholar 

  3. Adomavicius G., Sankaranarayanan R., Sen S., Tuzhilin A.: Incorporating contextual information in recommender systems using a multidimensional approach. ACM Trans. Inf. Syst. 23, 103–145 (2005)

    Article  Google Scholar 

  4. Anderson J.C., Gerbing D.W.: Structural equation modeling in practice: a review and recommended two-step approach. Psychol. Bull. 103, 411–423 (1988)

    Article  Google Scholar 

  5. Bagozzi R., Yi Y.: On the evaluation of structural equation models. J. Acad. Market. Sci. 16, 74–94 (1988)

    Article  Google Scholar 

  6. Baudisch, P., Terveen, L.: Interacting with recommender systems. In: SIGCHI Conference on Human Factors in Computing Systems, p. 164. Pittsburgh, PA (1999)

  7. Bechwati N., Xia L.: Do computers sweat? The impact of perceived effort of online decision aids on consumers’ satisfaction with the decision process. J. Consum. Psychol. 13, 139–148 (2003)

    Google Scholar 

  8. Bentler P.M., Bonett D.G.: Significance tests and goodness of fit in the analysis of covariance structures. Psychol. Bull. 88, 588–606 (1980)

    Article  Google Scholar 

  9. Berendt, B., Teltzrow, M.: Addressing users’ privacy concerns for improving personalization quality: towards an integration of user studies and algorithm evaluation. In: IJCAI 2003 Workshop on Intelligent Techniques for Web Personalization, LNAI, vol. 3169, pp. 69–88. Acapulco, Mexico (2005)

  10. Bharati P., Chaudhury A.: An empirical investigation of decision-making satisfaction in web-based decision support systems. Decis. Support Syst. 37, 187–197 (2004)

    Article  Google Scholar 

  11. Bhatnagar A., Ghose S.: Online information search termination patterns across product categories and consumer demographics. J. Retail. 80, 221–228 (2004)

    Article  Google Scholar 

  12. Bollen, D., Knijnenburg, B., Willemsen, M., Graus, M.: Understanding choice overload in recommender systems. In: Fourth ACM Conference on Recommender systems, pp. 63–70. Barcelona, Spain (2010)

  13. Bradley, K., Smyth, B.: Improving recommendation diversity. In: Twelfth Irish Conference on Artificial Intelligence and Cognitive Science, pp. 85–94. Maynooth, Ireland (2001)

  14. Brodie C., Karat C., Karat J.: Creating an E-commerce environment where consumers are willing to share personal information. In: Karat, C.-M., Blom, J.O., Karat, J. (eds.) Designing Personalized User Experiences in eCommerce, pp. 185–206. Kluwer, Dordrecht (2004)

    Google Scholar 

  15. Burke R.: Hybrid recommender systems: survey and experiments. User Model. User-Adap. Int. 12, 331–370 (2002)

    MATH  Article  Google Scholar 

  16. Cena, F., Vernero, F., Gena, C.: Towards a customization of rating scales in adaptive systems. In: 18th International Conference on User Modeling, Adaptation, and Personalization, vol. 6075, pp. 369–374. Big Island, HI, LNCS (2010)

  17. Chellappa R., Sin R.: Personalization versus privacy: an empirical examination of the online consumer’s dilemma. Inf. Technol. Manag. 6, 181–202 (2005)

    Article  Google Scholar 

  18. Chen L., Pu P.: Interaction design guidelines on critiquing-based recommender systems. User Model. User-Adap. Inter. 19, 167–206 (2009)

    Article  Google Scholar 

  19. Chen, L., Pu, P.: Critiquing-based recommenders: survey and emerging trends. User. Model. User-Adap. Inter. 22(1–2), 125–150 (2012)

    Google Scholar 

  20. Chin D.: Empirical evaluation of user models and user-adapted systems. User Model. User-Adap. Inter. 11, 181–194 (2001)

    MATH  Article  Google Scholar 

  21. Cooke A., Sujan H., Sujan M., Weitz B.A.: Marketing the unfamiliar: the role of context and item-specific information in electronic agent recommendations. J. Market. Res. 39, 488–497 (2002)

    Article  Google Scholar 

  22. Cosley, D., Lam, S., Albert, I., Konstan, J., Riedl, J.: Is seeing believing?: how recommender system interfaces affect users’ opinions. In: SIGCHI Conference on Human Factors in Computing Systems, pp. 585–592. Ft. Lauderdale, FL (2003)

  23. Cramer H., Evers V., van Someren M., Ramlal S., Rutledge L., Stash N., Aroyo L., Wielinga B.: The effects of transparency on trust and acceptance in interaction with a content-based art recommender. User Model. User-Adap. Inter. 18, 455–496 (2008a)

    Article  Google Scholar 

  24. Cramer, H., Evers, V., van Someren, M., Ramlal, S., Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.: The effects of transparency on perceived and actual competence of a content-based recommender. In: CHI’08 Semantic Web User Interaction Workshop. Florence, Italy (2008b)

  25. Csikszentmihalyi M.: Beyond Boredom and Anxiety. Jossey-Bass Publishers, San Fransisco (1975)

    Google Scholar 

  26. Davis F.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–340 (1989)

    Article  Google Scholar 

  27. Davis F., Bagozzi R., Warshaw P.: User acceptance of computer technology: a comparison of two theoretical models. Manag. Sci. 35, 982–1003 (1989)

    Article  Google Scholar 

  28. Diehl K., Kornish L., Lynch J. Jr: Smart agents: when lower search costs for quality information increase price sensitivity. J. Consum. Res. 30, 56–71 (2003)

    Article  Google Scholar 

  29. Felfernig A., Teppan E., Gula B.: Knowledge-based recommender technologies for marketing and sales. Int. J. Pattern Recog. Artif. Intell. 21, 333–354 (2007)

    Article  Google Scholar 

  30. Felix, D., Niederberger, C., Steiger, P., Stolze, M.: Feature-oriented vs. needs-oriented product access for non-expert online shoppers. In: IFIP Conference on Towards the E-Society: E-commerce, E-business, and E-government, pp. 399–406. Zürich, Switzerland (2001)

  31. Fishbein M., Ajzen I.: Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Addison-Wesley, Reading (1975)

    Google Scholar 

  32. Gena, C., Brogi, R., Cena, F., Vernero, F.: Impact of rating scales on user’s rating behavior. In: 19th International Conference on User Modeling, Adaptation, and Personalization, LNCS, vol. 6787, pp. 123–134. Girona, Spain (2011)

  33. Harper F., Li X., Chen Y., Konstan J.: An economic model of user rating in an online recommender system. In: 10th International Conference on User Modeling, LNCS, vol. 3538, pp. 307–316. Edinburgh, UK (2005)

  34. Hassenzahl M.: The thing and I: understanding the relationship between user and product. In: Blythe, M.A., Monk, A.F., Overbeeke, K., Wright, P.C. (eds.) Funology, pp. 31–42. Kluwer, Dordrecht (2005)

    Google Scholar 

  35. Hassenzahl, M.: User experience (UX): towards an experiential perspective on product quality. In: 20th International Conference of the Association Francophone d’Interaction Homme-Machine, pp. 11–15. Metz, France (2008)

  36. Häubl G., Trifts V.: Consumer decision making in online shopping environments: the effects of interactive decision aids. Market. Sci. 19, 4–21 (2000)

    Article  Google Scholar 

  37. Häubl G., Dellaert B., Murray K., Trifts V.: Buyer behavior in personalized shopping environments. In: Karat, C.-M., Blom, J.O., Karat, J. (eds.) Designing Personalized User Experiences in eCommerce, pp. 207–229. Kluwer, Dordrecht (2004)

    Google Scholar 

  38. Hauser J., Urban G., Liberali G., Braun M.: Website morphing. Market. Sci. 28, 202–223 (2009)

    Article  Google Scholar 

  39. Hayes, C., Massa, P., Avesani, P., Cunningham, P.: An on-line evaluation framework for recommender systems. In: AH’2002 Workshop on Recommendation and Personalization in E-Commerce, pp. 50–59. Málaga, Spain (2002)

  40. Herlocker, J., Konstan, J., Riedl, J.: Explaining Collaborative filtering recommendations. In: 2000 ACM Conference on Computer Supported Cooperative Work, pp. 241–250. Philadelphia, PA (2000)

  41. Herlocker J., Konstan J., Terveen L., Riedl J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 5–53 (2004)

    Article  Google Scholar 

  42. Ho S.Y., Tam K.Y.: An empirical examination of the effects of web personalization at different stages of decision making. Int. J. Human-Comput. Interact. 19, 95–112 (2005)

    Article  Google Scholar 

  43. Hostler R., Yoon V., Guimaraes T.: Assessing the impact of internet agent on end users’ performance. Decis. Supp. Syst. 41, 313–323 (2005)

    Article  Google Scholar 

  44. Hsu C., Lu H.: Why do people play on-line games? An extended TAM with social influences and flow experience. Inf. Manag. 41, 853–868 (2004)

    Article  Google Scholar 

  45. Hu L.-T., Bentler P.: Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equ. Model: A Multidiscip. J. 6, 1–55 (1999)

    Article  Google Scholar 

  46. Hu, R., Pu, P.: A comparative user study on rating vs. personality quiz based preference elicitation methods. In: 14th International Conference on Intelligent User Interfaces, pp. 367–371. Sanibel Island, FL (2009)

  47. Hu, R., Pu, P.: A study on user perception of personality-based recommender systems. In: 18th International Conference on User Modeling, Adaptation, and Personalization, LNCS, vol. 6075. pp. 291–302. Big Island, HI (2010)

  48. Hu, R., Pu., P.: Enhancing recommendation diversity with organization interfaces. In: 16th International Conference on Intelligent User Interfaces, pp. 347–350. Palo Alto, CA (2011)

  49. Hu, Y., Koren, Y., Volinsky, C.: Collaborative filtering for implicit feedback datasets. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 263–272. Pisa, Italy (2008)

  50. Huang P., Lurie N.H., Mitra S.: Searching for experience on the web: an empirical examination of consumer behavior for search and experience goods. J. Market. 73(2), 55–69 (2009)

    Article  Google Scholar 

  51. Iyengar S., Lepper M.: When choice is demotivating: can one desire too much of a good thing?. J. Pers. Soci. Psychol. 79, 995–1006 (2000)

    Article  Google Scholar 

  52. Jones, N., Pu, P., Chen, L.: How users perceive and appraise personalized recommendations. In: 17th International Conference on User Modeling, Adaptation, and Personalization Conference, vol. 5535, pp. 461–466. Trento, Italy, LNCS (2009)

  53. Kamis, A., Davern, M.J.: Personalizing to product category knowledge: exploring the mediating effect of shopping tools on decision confidence. In: 37th Annual Hawaii International Conference on System Sciences. Big Island, HI (2004)

  54. Kaplan B., Duchon D.: Combining qualitative and quantitative methods in information systems research: a case study. Mis Q. 12, 571–586 (1988)

    Article  Google Scholar 

  55. Kautz H., Selman B., Shah M.: Referral Web: combining social networks and collaborative filtering. Commun. ACM 40, 63–65 (1997)

    Article  Google Scholar 

  56. Knijnenburg, B.P., Reijmer, N.J.M., Willemsen, M.C.: Each to his own: how different users call for different interaction methods in recommender systems. In: 5th ACM Conference on Recommender Systems, pp.141–148. Chicago, IL (2011)

  57. Knijnenburg, B.P., Willemsen, M.C.: Understanding the effect of adaptive preference elicitation methods on user satisfaction of a recommender system. In: Third ACM Conference on Recommender Systems, pp. 381–384. New York, NY (2009)

  58. Knijnenburg, B.P., Willemsen, M.C.: The effect of preference elicitation methods on the user experience of a recommender system. In: 28th International Conference on Human Factors in Computing Systems, pp. 3457–3462. Atlanta, GA (2010)

  59. Knijnenburg, B.P., Willemsen, M.C., Kobsa, A.: A pragmatic procedure to support the user-centric evaluation of recommender systems. In: 5th ACM Conference on Recommender Systems, pp. 321–324. Chicago, IL (2011a)

  60. Knijnenburg, B.P., Meesters, L., Marrow, P., Bouwhuis, D.: User-centric evaluation framework for multimedia recommender systems. In: First International Conference on User Centric Media, LNICST, vol. 40, pp. 366–369. Venice, Spain (2010a)

  61. Knijnenburg, B.P., Willemsen, M.C., Hirtbach, S.: Receiving recommendations and providing feedback: The user-experience of a recommender system. In: 11th International Conference on Electronic Commerce and Web Technologies, LNBIP, vol. 61, pp. 207–216. Bilbao, Spain (2010b)

  62. Kobsa, A. Teltzrow, M.: Contextualized communication of privacy practices and personalization benefits: impacts on users’ data sharing and purchase behavior. In: Workshop on Privacy Enhancing Technologies, LNCS, vol. 3424, pp. 329–343. Toronto, Canada (2005)

  63. Komiak S.Y.X., Benbasat I.: The effects of personalization and familiarity on trust and adoption of recommendation agents. Mis Q. 30, 941–960 (2006)

    Google Scholar 

  64. Konstan J.A., Riedl J.: Recommender Systems: From Algorithms to User Experience. User Model. User-Adap. Inter. 22(1–2), 101–123 (2012)

  65. Koren Y., Bell R., Volinsky C.: Matrix factorization techniques for recommender systems. IEEE Comput. 42, 30–37 (2009)

    Article  Google Scholar 

  66. Koren Y.: Factor in the neighbors: Scalable and accurate collaborative filtering. Trans. Knowl. Discov. Data 4, 1–24 (2010)

    Article  Google Scholar 

  67. Koufaris M.: Applying the technology acceptance model and flow theory to online consumer behavior. Inf. Syst. Res. 13, 205–223 (2003)

    Article  Google Scholar 

  68. Kramer T.: The effect of measurement task transparency on preference construction and evaluations of personalized recommendations. J. Market. Res. 44, 224–233 (2007)

    Article  Google Scholar 

  69. Krishnan, V., Narayanashetty, P., Nathan, M., Davies, R., Konstan, J.: Who predicts better?: Results from an online study comparing humans and an online recommender system. In: 2008 ACM Conference on Recommender Systems, pp. 211–218. Lausanne, Switzerland (2008)

  70. Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., Kort, J.: Understanding, scoping and defining user experience: a survey approach. In: 27th International Conference on Human Factors in Computing Systems, pp. 719–728. Boston, MA (2009)

  71. Lynch J.G. Jr, Ariely D.: Wine online: search cost and competition on price, quality, and distribution. Market. Sci. 19, 83–103 (2000)

    Article  Google Scholar 

  72. Marrow, P., Hanbidge, R., Rendle, S., Wartena, C., Freudenthaler, C.: MyMedia: producing an extensible framework for recommendation. In: Networked Electronic Media Summit 2009. Saint-Malo, France (2009)

  73. McNamara N., Kirakowski J.: Functionality, usability, and user experience: three areas of concern. ACM Interact. 13, 26–28 (2006)

    Article  Google Scholar 

  74. McNee, S., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S., Rashid, A., Konstan, J., Riedl, J.: On the recommending of citations for research papers. In: 2002 ACM Conference on Computer Supported Cooperative Work, pp. 116–125. New Orleans, LA (2002)

  75. McNee, S., Riedl, J., Konstan, J.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: 24th International Conference Human factors in computing systems, pp. 1097–1101. Montréal, Canada (2006a)

  76. McNee, S., Riedl, J., Konstan, J.: Making recommendations better: an analytic model for human-recommender interaction. In: 24th International Conference Human factors in computing systems, pp. 1103–1108. Montréal, Canada (2006b)

  77. Meesters, L., Marrow, P., Knijnenburg, B.P., Bouwhuis, D., Glancy, M.: MyMedia Deliverable 1.5 End-user Recommendation Evaluation Metrics (2008) http://www.mymediaproject.org/Publications/WP1/MyMedia_D1.5.pdf

  78. Murray K., Häubl G.: Interactive consumer decision aids. In: Wierenga, B. (ed.) Handbook of Marketing Decision Models, pp. 55–77. Springer, Heidelberg (2008)

    Google Scholar 

  79. Murray K., Häubl G.: Personalization without interrogation: towards more effective interactions between consumers and feature-based recommendation agents. J. Interact. Market. 23, 138–146 (2009)

    Article  Google Scholar 

  80. Muthen B.: A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika 49, 115–132 (1984)

    Article  Google Scholar 

  81. Nelson P.: Information and consumer behavior. J. Polit. Econ. 78, 311–329 (1970)

    Article  Google Scholar 

  82. Netemeyer R., Bentler P.: Structural equations modeling and statements regarding causality. J. Consum. Psychol. 10, 83–85 (2001)

    Article  Google Scholar 

  83. Ochi P., Rao S., Takayama L., Nass C.: Predictors of user perceptions of web recommender systems: how the basis for generating experience and search product recommendations affects user responses. Int. J. Hum.-Comput. Stud. 68, 472–482 (2010)

    Article  Google Scholar 

  84. Ozok A.A., Fan Q., Norcio A.F.: Design guidelines for e?ective recommender system interfaces based on a usability criteria conceptual model: results from a college student population. Behav. Inf. Technol. 29, 57–83 (2010)

    Article  Google Scholar 

  85. Paramythis A., Weibelzahl S., Masthoff J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User-Adap. Inter. 20, 383–453 (2010)

    Article  Google Scholar 

  86. Pathak B., Garfinkel R., Gopal R.D., Venkatesan R., Yin F.: Empirical analysis of the impact of recommender systems on sales. J. Manag. Inf. Syst. 27, 159–188 (2010)

    Article  Google Scholar 

  87. Pedersen P.: Behavioral effects of using software agents for product and merchant brokering: an experimental study of consumer decision-making. Int. J. Electron. Commer. 5, 125–141 (2000)

    Google Scholar 

  88. Pommeranz, A., Broekens, J., Wiggers, P., Brinkman, W.-P., Jonker, C. M.: Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process. User Model. User-Adap. Inter. 22 (2012). doi:10.1007/s11257-011-9116-6

  89. Preece J., Rogers Y., Sharp H.: Interaction Design: Beyond Human-Computer Interaction. Wiley, New York (2002)

    Google Scholar 

  90. Pu, P., Chen, L.: Trust building with explanation interfaces. In: 11th International Conference on Intelligent User Interfaces, pp. 93–100. Sydney, Australia (2006)

  91. Pu P., Chen L.: Trust-inspiring explanation interfaces for recommender systems. Knowl.-Based Syst. 20, 542–556 (2007)

    MathSciNet  Article  Google Scholar 

  92. Pu, P., Chen, L.: A user-centric evaluation framework of recommender systems. In: ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces, pp. 14–21. Barcelona, Spain (2010)

  93. Pu, P., Chen, L., Hu, R.: Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Model. User-Adap. Inter. 22 (2012). doi:10.1007/s11257-011-9115-7

  94. Pu P., Chen L., Kumar P.: Evaluating product search and recommender systems for E-commerce environments. Electron. Commer. Res. 8, 1–27 (2008)

    MATH  Article  Google Scholar 

  95. Rendle, S., Schmidt-Thieme, L.: Online-updating regularized kernel matrix factorization models for large-scale recommender systems. In: 2008 ACM Conference on Recommender systems, pp. 251–258. Lausanne, Switzerland (2008)

  96. Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: BPR: Bayesian personalized ranking from implicit feedback. In: Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 452–461. Montreal, Canada (2009)

  97. Resnick P., Varian H.: Recommender systems. Commun. ACM 40, 56–58 (1997)

    Article  Google Scholar 

  98. Scheibehenne B., Greifeneder R., Todd P.: Can there ever be too many options? A meta-analytic review of choice overload. J. Consum. Res. 37, 409–425 (2010)

    Article  Google Scholar 

  99. Schwartz B.: The Paradox of Choice: Why More Is Less. HarperCollins, New York (2004)

    Google Scholar 

  100. Senecal S., Nantel J.: The influence of online product recommendations on consumers’ online choices. J. Retail. 80, 159–169 (2004)

    Article  Google Scholar 

  101. Sheeran P.: Intention-behavior relations: a conceptual and empirical review. Eur. Rev. Soc Psychol. 12, 1–36 (2002)

    Article  Google Scholar 

  102. Simonson I., Tversky A.: Choice in context: tradeoff contrast and extremeness aversion. J. Market. Res. 29, 281–295 (1992)

    Article  Google Scholar 

  103. Spiekermann, S., Grossklags, J., Berendt, B.: E-privacy in 2nd generation E-commerce: privacy preferences versus actual behavior. In: 3rd ACM Conference on Electronic Commerce, pp. 38–47. Tampa, FL (2001)

  104. Stolze, M., Nart, F.: Well-integrated needs-oriented recommender components regarded as helpful. In: 22nd International Conference on Human Factors in Computing Systems, p. 1571. Vienna, Austria (2004)

  105. Tam K.Y., Ho S.Y.: Web personalization as a Persuasion strategy: an elaboration likelihood model perspective. Inf. Syst. Res. 16, 271–291 (2005)

    Article  Google Scholar 

  106. Teltzrow M., Kobsa A.: Impacts of user privacy preferences on personalized systems. Hum.-Comput. Interact. Ser. 5, 315–332 (2004)

    Article  Google Scholar 

  107. Tintarev, N., Masthoff, J.: Evaluating the Effectiveness of Explanations for Recommender Systems. User Model. User-Adap. Inter. 22 (2012). doi:10.1007/s11257-011-9117-5

  108. Torres, R., McNee, S., Abel, M., Konstan, J., Riedl, J.: Enhancing digital libraries with TechLens+. In: 4th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 228–236. Tucson, AZ (2004)

  109. Van Velsen L., VanDer Geest T., Klaassen R., Steehouder M.: User-centered evaluation of adaptive and adaptable systems: a literature review. Knowl. Eng. Rev. 23, 261–281 (2008)

    Article  Google Scholar 

  110. Venkatesh V., Morris M., Davis G., Davis F.: User acceptance of information technology: toward a unified view. Mis Q. 27, 425–478 (2003)

    Google Scholar 

  111. Viappiani P., Faltings B., Pu P.: Preference-based search using example-critiquing with suggestions. J. Artif. Intell. Res. 27, 465–503 (2006)

    MATH  Google Scholar 

  112. Viappiani P., Pu P., Faltings B.: Preference-based search with adaptive recommendations. AI Commun. 21, 155–175 (2008)

    MathSciNet  MATH  Google Scholar 

  113. Vijayasarathy L.R., Jones J.M.: Do internet shopping aids make a difference? an empirical investigation. Electron. Markets 11, 75–83 (2001)

    Article  Google Scholar 

  114. Wang W., Benbasat I.: Recommendation agents for electronic commerce: effects of explanation facilities on trusting beliefs. J. Manag. Inf. Syst. 23, 217–246 (2007)

    Article  Google Scholar 

  115. Willemsen, M.C., Knijnenburg, B.P., Graus, M.P., Velter-Bremmers, L.C.M., Fu, K.: Using latent features diversification to reduce choice difficulty in recommendation lists. In: RecSys’11 Workshop on Human Decision Making in Recommender Systems, CEUR-WS, vol. 811, pp. 14–20. Chicago, IL (2011)

  116. Xiao B., Benbasat I.: ECommerce product recommendation agents: use, characteristics, and impact. Mis Q. 31, 137–209 (2007)

    Google Scholar 

  117. Yu J., Ha I., Choi M., Rho J.: Extending the TAM for a t-commerce. Inf. Manag. 42, 965–976 (2005)

    Article  Google Scholar 

  118. Ziegler, C., McNee, S., Konstan, J., Lausen, G.: Improving recommendation lists through topic diversification. In: 14th international World Wide Web Conference, pp. 22–32. Chiba, Japan (2005)

  119. Zins, A., Bauernfeind, U.: Explaining online purchase planning experiences with recommender websites. In: International Conference on Information and Communication Technologies in Tourism, pp. 137–148. Innsbruck, Austria (2005)

Download references

Acknowledgements

We would like to thank Mark Graus for programming the recommender systems used in EX1 and EX2, Steffen Rendle for implementing the explicit feedback MF algorithm, Niels Reijmer, Yunan Chen and Alfred Kobsa for their comments at several stages of this paper, and Dirk Bollen for allowing us to incorporate the results of his choice overload experiment (EX1) in this paper. We also thank the three anonymous reviewers for their extensive comments on the initial submission. We gratefully acknowledge the funding of our work through the European Commission FP7 project MyMedia (www.mymediaproject.org) under the grant agreement no. 215006. For inquiries please contact info@mymediaproject.org.

Open Access

This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Bart P. Knijnenburg.

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Knijnenburg, B.P., Willemsen, M.C., Gantner, Z. et al. Explaining the user experience of recommender systems. User Model User-Adap Inter 22, 441–504 (2012). https://doi.org/10.1007/s11257-011-9118-4

Download citation

Keywords

  • Recommender systems
  • Decision support systems
  • User experience
  • User-centric evaluation
  • Decision-making
  • Human-computer interaction
  • User testing
  • Preference elicitation
  • Privacy