Looking for “Good” Recommendations: A Comparative Evaluation of Recommender Systems

  • Paolo Cremonesi
  • Franca Garzotto
  • Sara Negro
  • Alessandro Vittorio Papadopoulos
  • Roberto Turrin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6948)


A number of researches in the Recommender Systems (RSs) domain suggest that the recommendations that are “best” according to objective metrics are sometimes not the ones that are most satisfactory or useful to the users. The paper investigates the quality of RSs from a user-centric perspective. We discuss an empirical study that involved 210 users and considered seven RSs on the same dataset that use different baseline and state-of-the-art recommendation algorithms. We measured the user’s perceived quality of each of them, focusing on accuracy and novelty of recommended items, and on overall users’ satisfaction. We ranked the considered recommenders with respect to these attributes, and compared these results against measures of statistical quality of the considered algorithms as they have been assessed by past studies in the field using information retrieval and machine learning algorithms.


Recommender systems quality metrics user study 


  1. 1.
    Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. on Knowledge and Data Engineering 17(6), 734–749 (2005)CrossRefGoogle Scholar
  2. 2.
    Bennett, J., Lanning, S.: The Netix Prize. In: Proceedings of KDD Cup and Workshop, pp. 3–6 (2007)Google Scholar
  3. 3.
    Celma, Ò., Herrera, P.: A new approach to evaluating novel recommendations. In: RecSys 2008: Proc. of the 2008 ACM Conf. on Recommender Systems, pp. 179–186. ACM, New York (2008)CrossRefGoogle Scholar
  4. 4.
    Chen, L., Pu, P.: A cross-cultural user evaluation of product recommender interfaces. In: Proc. of the 2008 ACM Conf. on Recommender Systems, RecSys 2008, pp. 75–82. ACM, New York (2008)CrossRefGoogle Scholar
  5. 5.
    Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-N recommendation tasks. In: RecSys 2010: Proc. of the Fourth ACM Conf. on Recommender Systems, pp. 39–46. ACM, Barcelona (2010)CrossRefGoogle Scholar
  6. 6.
    Cremonesi, P., Turrin, R.: Analysis of cold-start recommendations in IPTV systems. In: RecSys 2009: Proc. ACM Conf. on Recommender Systems, pp. 233–236. ACM, New York (2009)CrossRefGoogle Scholar
  7. 7.
    Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Information Systems (TOIS) 22(1), 143–177 (2004)CrossRefGoogle Scholar
  8. 8.
    Fleder, D., Hosanagar, K.: Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity. Management Science 55(5), 697–712 (2009)CrossRefGoogle Scholar
  9. 9.
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. on Information Systems (TOIS) 22(1), 5–53 (2004)CrossRefGoogle Scholar
  10. 10.
    Hu, R., Pu, P.: Acceptance issues of personality-based recommender systems. In: Proc. of the Third ACM Conf. on Recommender Systems, pp. 221–224. ACM, New York City (2009)CrossRefGoogle Scholar
  11. 11.
    Husbands, P., Simon, H., Ding, C.H.Q.: On the use of the singular value decomposition for text retrieval. Computational Information Retrieval, 145–156 (2001)Google Scholar
  12. 12.
    Jones, N., Pu, P.: User Technology Adoption Issues in Recommender Systems. In: Proc. of the 2007 Networking and Electronic Commerce Research Conf., Riva del Garda, Italy, pp. 379–394 (2007)Google Scholar
  13. 13.
    Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: KDD 2008: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 426–434. ACM, New York (2008)CrossRefGoogle Scholar
  14. 14.
    McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI 2006 Extended Abstracts on Human Factors in Computing Systems, pp. 1097–1101. ACM, New York City (2006)CrossRefGoogle Scholar
  15. 15.
    Pu, P., Chen, L.: Trust building with explanation interfaces. In: Proc. of the 11th int. Conf. on Intelligent User Interfaces, IUI 2006, pp. 93–100. ACM, New York (2006)CrossRefGoogle Scholar
  16. 16.
    Pu, P., Chen, L., Kumar, P.: Evaluating product search and recommender systems for e-commerce environments. Electric Commerce Research Journal 8(1-2), 27 (2008)Google Scholar
  17. 17.
    Pu, P., Zhou, M., Castagnos, S.: Critiquing recommenders for public taste products. In: Proc. of the Third ACM Conf. on Recommender Systems, RecSys 2009, pp. 249–252. ACM, New York (2009)CrossRefGoogle Scholar
  18. 18.
    Pu, P., Chen, L.: A User-Centric Evaluation Framework of Recommender Systems. In: Proc. of the ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI), Barcelona, Spain (September 2010)Google Scholar
  19. 19.
    Raghavan, V., Bollmann, P., Jung, G.S.: A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst. 7, 205–229 (1989)CrossRefGoogle Scholar
  20. 20.
    Sarwar, B., Karypis, G., Konstan, J., Reidl, J.: Item-based collaborative filtering recommendation algorithms. In: 10th Int. Conf. on World Wide Web, pp. 285–295 (2001)Google Scholar
  21. 21.
    Shearer, A.W.: User response to two algorithms as a test of collaborative filtering. In: CHI 2001 Extended Abstracts on Human Factors in Computing Systems, pp. 451–452. ACM, New York (2001)CrossRefGoogle Scholar
  22. 22.
    Takács, G., Pilászy, I., Németh, B., Tikk, D.: Scalable collaborative filtering approaches for large recommender systems. The J. of Machine Learning Research 10, 623–656 (2009)Google Scholar
  23. 23.
    Weng, L., Xu, Y., Li, Y., Nayak, R.: Improving recommendation novelty based on topic taxonomy. In: Proc. of the 2007 IEEE/WIC/ACM International Conf. on Web Intelligence and Intelligent Agent Technology - Workshops, WI-IATW 2007, pp. 115–118. IEEE Computer Society, Washington, DC, USA (2007)CrossRefGoogle Scholar
  24. 24.
    Zhang, Y., Callan, J., Minka, T.: Novelty and redundancy detection in adaptive filtering. In: Proc. of the 25nd ACM SIGIR Conf. on R&D in Information Retrieval, pp. 81–88. ACM, New York City (2002)Google Scholar
  25. 25.
    Ziegler, C.N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: Proc. of the 14th International Conf. on World Wide Web, pp. 22–32. ACM, New York (2005)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Paolo Cremonesi
    • 1
  • Franca Garzotto
    • 1
  • Sara Negro
    • 1
  • Alessandro Vittorio Papadopoulos
    • 1
  • Roberto Turrin
    • 2
  1. 1.Department of Electronics and InformationPolitecnico di MilanoMilanoItaly
  2. 2.Moviri srl, R&DItaly

Personalised recommendations