What Recommenders Recommend – An Analysis of Accuracy, Popularity, and Sales Diversity Effects

  • Dietmar Jannach
  • Lukas Lerche
  • Fatih Gedikli
  • Geoffray Bonnin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7899)


In academic studies, the evaluation of recommender system (RS) algorithms is often limited to offline experimental designs based on historical data sets and metrics from the fields of Machine Learning or Information Retrieval. In real-world settings, however, other business-oriented metrics such as click-through-rates, customer retention or effects on the sales spectrum might be the true evaluation criteria for RS effectiveness. In this paper, we compare different RS algorithms with respect to their tendency of focusing on certain parts of the product spectrum. Our first analysis on different data sets shows that some algorithms – while able to generate highly accurate predictions – concentrate their top 10 recommendations on a very small fraction of the product catalog or have a strong bias to recommending only relatively popular items than others. We see our work as a further step toward multiple-metric offline evaluation and to help service providers make better-informed decisions when looking for a recommendation strategy that is in line with the overall goals of the recommendation service.


Recommender System Gini Index Product Spectrum Recommendation List Accuracy Metrics 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jannach, D., Zanker, M., Ge, M., Gröning, M.: Recommender systems in computer science and information systems - a landscape of research. In: Huemer, C., Lops, P. (eds.) EC-Web 2012. LNBIP, vol. 123, pp. 76–87. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  2. 2.
    McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: Proceedings of the 2006 Conference on Human Factors in Computing Systems (CHI 2006), pp. 1097–1101 (2006)Google Scholar
  3. 3.
    Adomavicius, G., Kwon, Y.: Improving aggregate recommendation diversity using ranking-based techniques. IEEE Transactions on Knowledge and Data Engineering 24(5), 896–911 (2012)CrossRefGoogle Scholar
  4. 4.
    Steck, H.: Item popularity and recommendation accuracy. In: Proceedings of the 2011 ACM Conference on Recommender Systems, Chicago, Illinois, USA, pp. 125–132 (2011)Google Scholar
  5. 5.
    Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A.V., Turrin, R.: Looking for “good” recommendations: A comparative evaluation of recommender systems. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part III. LNCS, vol. 6948, pp. 152–168. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    Jannach, D., Hegelich, K.: A case study on the effectiveness of recommendations in the mobile internet. In: Proceedings of the 2009 ACM Conference on Recommender Systems, New York, pp. 41–50 (2009)Google Scholar
  7. 7.
    Kirshenbaum, E., Forman, G., Dugan, M.: A live comparison of methods for personalized article recommendation at In: Flach, P.A., De Bie, T., Cristianini, N. (eds.) ECML PKDD 2012, Part II. LNCS, vol. 7524, pp. 51–66. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Cremonesi, P., Koren, Y., Turrin, R.: Algorithms on top-n recommendation tasks. In: Proceedings of the 2010 ACM Conference on Recommender Systems, Barcelona, pp. 39–46 (2010)Google Scholar
  9. 9.
    Fleder, D., Hosanagar, K.: Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity. Management Science 55(5), 205–208 (2009)CrossRefGoogle Scholar
  10. 10.
    Prawesh, S., Padmanabhan, B.: The “top N” news recommender: count distortion and manipulation resistance. In: Proceedings of the 2011 ACM Conference on Recommender Systems, Chicago, USA, pp. 237–244 (2011)Google Scholar
  11. 11.
    Zhang, M.: Enhancing the diversity of collaborative filtering recommender systems. PhD Thesis. Univ. College Dublin (2010)Google Scholar
  12. 12.
    Said, A., Tikk, D., Shi, Y.: Recommender Systems Evaluation: A 3D Benchmark. In: ACM RecSys 2012 Workshop on Recommendation Utility Evaluation: Beyond RMSE, Dublin, Ireland, pp. 21–23 (2012)Google Scholar
  13. 13.
    Ekstrand, M.D., Ludwig, M., Konstan, J.A., Riedl, J.T.: Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit. In: Proceedings of the 2011 ACM Conference on Recommender Systems, Chicago, Illinois, USA, pp. 133–140 (2011)Google Scholar
  14. 14.
    Meyer, F., Fessant, F., Clérot, F., Gaussier, E.: Toward a new protocol to evaluate recommender systems. In: ACM RecSys 2012 Workshop on Recommendation Utility Evaluation: Beyond RMSE, Dublin, Ireland, pp. 9–14 (2012)Google Scholar
  15. 15.
    Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, USA, pp. 426–434 (2008)Google Scholar
  16. 16.
    Gedikli, F., Bagdat, F., Ge, M., Jannach, D.: RF-REC: Fast and accurate computation of recommendations based on rating frequencies. In: 13th IEEE Conference on Commerce and Enterprise Computing, CEC 2011, Luxembourg, pp. 50–57 (2011)Google Scholar
  17. 17.
    Lemire, D., Maclachlan, A.: Slope one predictors for online rating-based collaborative filtering. In: SIAM Conference on Data Mining, Newport Beach, pp. 471–480 (2005)Google Scholar
  18. 18.
    (2006), (last accessed March 2013)
  19. 19.
    Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: BPR: Bayesian Personalized Ranking from Implicit Feedback. In: Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, Canada, pp. 452–461 (2009)Google Scholar
  20. 20.
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems 22(1), 5–53 (2004)CrossRefGoogle Scholar
  21. 21.
    Castagnos, S., Jones, N., Pu, P.: Eye-Tracking Product Recommenders’ Usage. In: Proceedings of the 2010 ACM Conference on Recommender Systems, Barcelona, Spain, pp. 29–36 (2010)Google Scholar
  22. 22.
    Dias, M.B., Locher, D., Li, M., El-Deredy, W., Lisboa, P.J.: The value of personalised recommender systems to e-business: A case study. In: Proceedings of the 2008 ACM Conference on Recommender Systems, Lausanne, Switzerland, pp. 291–294 (2008)Google Scholar
  23. 23.
    Zanker, M., Bricman, M., Gordea, S., Jannach, D., Jessenitschnig, M.: Persuasive online-selling in quality & taste domains. In: Bauknecht, K., Pröll, B., Werthner, H. (eds.) EC-Web 2006. LNCS, vol. 4082, pp. 51–60. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  24. 24.
    Jannach, D., Karakaya, Z., Gedikli, F.: Accuracy improvements for multi-criteria recommender systems. In: Proceedings of the 13th ACM Conference on Electronic Commerce, EC 2012, Valencia, Spain, pp. 674–689 (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Dietmar Jannach
    • 1
  • Lukas Lerche
    • 1
  • Fatih Gedikli
    • 1
  • Geoffray Bonnin
    • 1
  1. 1.TU DortmundDortmundGermany

Personalised recommendations