Collaborative Filtering: The Aim of Recommender Systems and the Significance of User Ratings

  • Jennifer Redpath
  • David H. Glass
  • Sally McClean
  • Luke Chen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5993)


This paper investigates the significance of numeric user ratings in recommender systems by considering their inclusion / exclusion in both the generation and evaluation of recommendations. When standard evaluation metrics are used, experimental results show that inclusion of numeric rating values in the recommendation process does not enhance the results. However, evaluating the accuracy of a recommender algorithm requires identifying the aim of the system. Evaluation metrics such as precision and recall evaluate how well a system performs at recommending items that have been previously rated by the user. By contrast, a new metric, known as Approval Rate, is intended to evaluate how well a system performs at recommending items that would be rated highly by the user. Experimental results demonstrate that these two aims are not synonymous and that for an algorithm to attempt both obscures the investigation. The results also show that appropriate use of numeric rating valuesin the process of calculating user similarity can enhance the performance when Approval Rate is used.


Recommender Systems Collaborative Filtering Information Retrieval 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahn, H.J.: A New Similarity Measure for Collaborative Filtering to Alleviate the New User Cold-starting Problem. Inf. Sci. 178(1), 37–51 (2008)CrossRefGoogle Scholar
  2. 2.
    Breese, J., Heckerman, D., Kadie, C.: Empirical Analysis of Predictive Algorithms for Collaborative Filtering. In: UAI 1998, pp. 43–52 (1998)Google Scholar
  3. 3.
    Goldberg, D., Nichols, D., Oki, B.M., Terry, D.: Using Collaborative Filtering to Weave an Information Tapestry. Commun. ACM 35(12), 61–70 (1992)CrossRefGoogle Scholar
  4. 4.
    Hayes, C., Massa, P., Avesani, P., Cunningham, P.: An On-line Evaluation Framework for Recommender Systems. In: Workshop on Personalization and Recommendation in E-Commerce (2002)Google Scholar
  5. 5.
    Herlocker, J., Konstan, J., Terveen, L., Riedl, J.: Evaluating Collaborative Filtering Recommender Systems. ACM TOIS 22(1), 5–53 (2004)CrossRefGoogle Scholar
  6. 6.
    Huang, Z., Zeng, D., Chen, H.: A Comparison of Collaborative-Filtering Recommendation Algorithms for E-commerce. IEEE IS 22(5), 68–78 (2007)Google Scholar
  7. 7.
    Konstan, J.A., Riedl, J.: Research Resources for Recommender Systems. In: CHI 1999 (1999)Google Scholar
  8. 8.
    McLaughlin, M.R., Herlocker, J.L.: A Collaborative Filtering Algorithm and Evaluation Metric that Accurately Model the User Experience. In: SIGIR 2004, pp. 329–336. ACM, New York (2004)CrossRefGoogle Scholar
  9. 9.
    McNee, S.M., Riedl, J., Konstan, J.A.: Being Accurate is Not Enough: How Accuracy Metrics have hurt Recommender Systems. In: CHI 2006, pp. 1097–1101. ACM, New York (2006)CrossRefGoogle Scholar
  10. 10.
    McNee, S.M., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S.K., Rashid, A.M., Konstan, J.A., Riedl, J.: On the Recommending of Citations for Research Papers. In: ACM CSCW 2002, pp. 116–125. ACM Press, New York (2002)CrossRefGoogle Scholar
  11. 11.
    MovieLens Dataset (2006),
  12. 12.
    Resnick, P., Varian, H.: Recommender Systems. CACM 40(3), 56–58 (1997)Google Scholar
  13. 13.
    Resnick, P., Iacovou, N., Sushak, M., Bergstrom, P., Riedl, J.: GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In: ACM CSCW 1994, pp. 175–186. ACM Press, New York (1994)CrossRefGoogle Scholar
  14. 14.
    Riedl, J., Jameson, A.: Advanced topics in recommendation. In: The 12th IUI 2007, pp. 11–11. ACM, New York (2007)CrossRefGoogle Scholar
  15. 15.
    Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Analysis of Recommendation Algorithms for E-commerce. In: EC 2000, pp. 158–167. ACM, New York (2000)Google Scholar
  16. 16.
    Shardanand, U., Maes, P.: Social Information Filtering: Algorithms for Automating ‘Word of Mouth’. In: SIGCHI 1995 Conference on Human Factors in Computing Systems, pp. 210–217. ACM Press, New York (1995)Google Scholar
  17. 17.
    Swearingen, K., Sinha, R.: Interaction Design for Recommender Systems. In: Designing Interactive Systems. ACM Press, New York (2002)Google Scholar
  18. 18.
    Symeonidis, P., Nanopoulos, A., Papadopoulos, A., Manolopoulos, Y.: Collaborative Filtering Process in a Whole New Light. In: IDEAS 2006, pp. 29–36. IEEE Computer Society, Washington (2006)Google Scholar
  19. 19.
    Ziegler, C., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: WWW 2005, pp. 22–32. ACM, New York (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Jennifer Redpath
    • 1
  • David H. Glass
    • 1
  • Sally McClean
    • 1
  • Luke Chen
    • 1
  1. 1.School of Computing and MathematicsUniversity of Ulster, Newtownabbey, Co. AntrimUK

Personalised recommendations