User Modeling and User-Adapted Interaction

, Volume 22, Issue 4–5, pp 399–439

Evaluating the effectiveness of explanations for recommender systems

Methodological issues and empirical studies on the impact of personalization
Original Paper

DOI: 10.1007/s11257-011-9117-5

Cite this article as:
Tintarev, N. & Masthoff, J. User Model User-Adap Inter (2012) 22: 399. doi:10.1007/s11257-011-9117-5


When recommender systems present items, these can be accompanied by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under- and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distribution when evaluating effectiveness.


Recommender systems Metrics Item descriptions Explanations Empirical studies 

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.Department of Computing ScienceUniversity of AberdeenAberdeenScotland, UK

Personalised recommendations