Abstract
A number of researches in the Recommender Systems (RSs) domain suggest that the recommendations that are “best” according to objective metrics are sometimes not the ones that are most satisfactory or useful to the users. The paper investigates the quality of RSs from a user-centric perspective. We discuss an empirical study that involved 210 users and considered seven RSs on the same dataset that use different baseline and state-of-the-art recommendation algorithms. We measured the user’s perceived quality of each of them, focusing on accuracy and novelty of recommended items, and on overall users’ satisfaction. We ranked the considered recommenders with respect to these attributes, and compared these results against measures of statistical quality of the considered algorithms as they have been assessed by past studies in the field using information retrieval and machine learning algorithms.
Chapter PDF
Similar content being viewed by others
References
Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. on Knowledge and Data Engineering 17(6), 734–749 (2005)
Bennett, J., Lanning, S.: The Netix Prize. In: Proceedings of KDD Cup and Workshop, pp. 3–6 (2007)
Celma, Ò., Herrera, P.: A new approach to evaluating novel recommendations. In: RecSys 2008: Proc. of the 2008 ACM Conf. on Recommender Systems, pp. 179–186. ACM, New York (2008)
Chen, L., Pu, P.: A cross-cultural user evaluation of product recommender interfaces. In: Proc. of the 2008 ACM Conf. on Recommender Systems, RecSys 2008, pp. 75–82. ACM, New York (2008)
Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-N recommendation tasks. In: RecSys 2010: Proc. of the Fourth ACM Conf. on Recommender Systems, pp. 39–46. ACM, Barcelona (2010)
Cremonesi, P., Turrin, R.: Analysis of cold-start recommendations in IPTV systems. In: RecSys 2009: Proc. ACM Conf. on Recommender Systems, pp. 233–236. ACM, New York (2009)
Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Information Systems (TOIS) 22(1), 143–177 (2004)
Fleder, D., Hosanagar, K.: Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity. Management Science 55(5), 697–712 (2009)
Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. on Information Systems (TOIS) 22(1), 5–53 (2004)
Hu, R., Pu, P.: Acceptance issues of personality-based recommender systems. In: Proc. of the Third ACM Conf. on Recommender Systems, pp. 221–224. ACM, New York City (2009)
Husbands, P., Simon, H., Ding, C.H.Q.: On the use of the singular value decomposition for text retrieval. Computational Information Retrieval, 145–156 (2001)
Jones, N., Pu, P.: User Technology Adoption Issues in Recommender Systems. In: Proc. of the 2007 Networking and Electronic Commerce Research Conf., Riva del Garda, Italy, pp. 379–394 (2007)
Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: KDD 2008: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 426–434. ACM, New York (2008)
McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI 2006 Extended Abstracts on Human Factors in Computing Systems, pp. 1097–1101. ACM, New York City (2006)
Pu, P., Chen, L.: Trust building with explanation interfaces. In: Proc. of the 11th int. Conf. on Intelligent User Interfaces, IUI 2006, pp. 93–100. ACM, New York (2006)
Pu, P., Chen, L., Kumar, P.: Evaluating product search and recommender systems for e-commerce environments. Electric Commerce Research Journal 8(1-2), 27 (2008)
Pu, P., Zhou, M., Castagnos, S.: Critiquing recommenders for public taste products. In: Proc. of the Third ACM Conf. on Recommender Systems, RecSys 2009, pp. 249–252. ACM, New York (2009)
Pu, P., Chen, L.: A User-Centric Evaluation Framework of Recommender Systems. In: Proc. of the ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI), Barcelona, Spain (September 2010)
Raghavan, V., Bollmann, P., Jung, G.S.: A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst. 7, 205–229 (1989)
Sarwar, B., Karypis, G., Konstan, J., Reidl, J.: Item-based collaborative filtering recommendation algorithms. In: 10th Int. Conf. on World Wide Web, pp. 285–295 (2001)
Shearer, A.W.: User response to two algorithms as a test of collaborative filtering. In: CHI 2001 Extended Abstracts on Human Factors in Computing Systems, pp. 451–452. ACM, New York (2001)
Takács, G., Pilászy, I., Németh, B., Tikk, D.: Scalable collaborative filtering approaches for large recommender systems. The J. of Machine Learning Research 10, 623–656 (2009)
Weng, L., Xu, Y., Li, Y., Nayak, R.: Improving recommendation novelty based on topic taxonomy. In: Proc. of the 2007 IEEE/WIC/ACM International Conf. on Web Intelligence and Intelligent Agent Technology - Workshops, WI-IATW 2007, pp. 115–118. IEEE Computer Society, Washington, DC, USA (2007)
Zhang, Y., Callan, J., Minka, T.: Novelty and redundancy detection in adaptive filtering. In: Proc. of the 25nd ACM SIGIR Conf. on R&D in Information Retrieval, pp. 81–88. ACM, New York City (2002)
Ziegler, C.N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: Proc. of the 14th International Conf. on World Wide Web, pp. 22–32. ACM, New York (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 IFIP International Federation for Information Processing
About this paper
Cite this paper
Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A.V., Turrin, R. (2011). Looking for “Good” Recommendations: A Comparative Evaluation of Recommender Systems. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2011. INTERACT 2011. Lecture Notes in Computer Science, vol 6948. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23765-2_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-23765-2_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23764-5
Online ISBN: 978-3-642-23765-2
eBook Packages: Computer ScienceComputer Science (R0)