User Modeling and User-Adapted Interaction

, Volume 22, Issue 4, pp 441–504

Explaining the user experience of recommender systems


    • Department of Informatics, Donald Bren School of Information and Computer SciencesUniversity of California
    • Human-Technology Interaction Group, School of Innovation SciencesEindhoven University of Technology (TU/e)
  • Martijn C. Willemsen
    • Human-Technology Interaction Group, School of Innovation SciencesEindhoven University of Technology (TU/e)
  • Zeno Gantner
    • Information Systems and Machine Learning Lab (ISMLL)University of Hildesheim
  • Hakan Soncu
    • European Microsoft Innovation Center GmbH
  • Chris Newell
    • BBC Research & Development, Centre House
Open AccessOriginal Paper

DOI: 10.1007/s11257-011-9118-4


Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems.


Recommender systems Decision support systems User experience User-centric evaluation Decision-making Human-computer interaction User testing Preference elicitation Privacy


We would like to thank Mark Graus for programming the recommender systems used in EX1 and EX2, Steffen Rendle for implementing the explicit feedback MF algorithm, Niels Reijmer, Yunan Chen and Alfred Kobsa for their comments at several stages of this paper, and Dirk Bollen for allowing us to incorporate the results of his choice overload experiment (EX1) in this paper. We also thank the three anonymous reviewers for their extensive comments on the initial submission. We gratefully acknowledge the funding of our work through the European Commission FP7 project MyMedia ( under the grant agreement no. 215006. For inquiries please contact

Open Access

This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.

Copyright information

© The Author(s) 2012