The Magic Barrier of Recommender Systems – No Magic, Just Ratings

  • Alejandro Bellogín
  • Alan Said
  • Arjen P. de Vries
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8538)


Recommender Systems need to deal with different types of users who represent their preferences in various ways. This difference in user behaviour has a deep impact on the final performance of the recommender system, where some users may receive either better or worse recommendations depending, mostly, on the quantity and the quality of the information the system knows about the user. Specifically, the inconsistencies of the user impose a lower bound on the error the system may achieve when predicting ratings for that particular user.

In this work, we analyse how the consistency of user ratings (coherence) may predict the performance of recommendation methods. More specifically, our results show that our definition of coherence is correlated with the so-called magic barrier of recommender systems, and thus, it could be used to discriminate between easy users (those with a low magic barrier) and difficult ones (those with a high magic barrier). We report experiments where the rating prediction error for the more coherent users is lower than that of the less coherent ones. We further validate these results by using a public dataset, where the magic barrier is not available, in which we obtain similar performance improvements.


Root Mean Square Error Feature Space Recommender System Coherence Function Average Root Mean Square Error 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amatriain, X., Pujol, J.M., Oliver, N.: I like it... I like it not: Evaluating user ratings noise in recommender systems. In: UMAP, pp. 247–258 (2009)Google Scholar
  2. 2.
    Amatriain, X., Pujol, J.M., Tintarev, N., Oliver, N.: Rate it again: increasing recommendation accuracy by user re-rating. In: RecSys, pp. 173–180 (2009)Google Scholar
  3. 3.
    Bellogín, A.: Predicting performance in recommender systems. In: RecSys, pp. 371–374 (2011)Google Scholar
  4. 4.
    Bellogín, A., Castells, P., Cantador, I.: Predicting the performance of recommender systems: An information theoretic approach. In: Amati, G., Crestani, F. (eds.) ICTIR 2011. LNCS, vol. 6931, pp. 27–39. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  5. 5.
    Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J.: Is seeing believing?: how recommender system interfaces affect users’ opinions. In: CHI, pp. 585–592 (2003)Google Scholar
  6. 6.
    Ekstrand, M.D., Riedl, J.: When recommenders fail: predicting recommender failure for algorithm selection and combination. In: RecSys, pp. 233–236 (2012)Google Scholar
  7. 7.
    Griffith, J., O’Riordan, C., Sorensen, H.: Investigations into user rating information and predictive accuracy in a collaborative filtering domain. In: SAC, pp. 937–942 (2012)Google Scholar
  8. 8.
    He, J., Larson, M., de Rijke, M.: Using coherence-based measures to predict query difficulty. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 689–694. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)CrossRefGoogle Scholar
  10. 10.
    Hill, W.C., Stead, L., Rosenstein, M., Furnas, G.W.: Recommending and evaluating choices in a virtual community of use. In: CHI, pp. 194–201 (1995)Google Scholar
  11. 11.
    Kille, B.: Modeling difficulty in recommender systems. In: RUE 2012. RecSys, pp. 30–32 (2012)Google Scholar
  12. 12.
    Kluver, D., Nguyen, T.T., Ekstrand, M.D., Sen, S., Riedl, J.: How many bits per rating? In: RecSys, pp. 99–106 (2012)Google Scholar
  13. 13.
    Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning 51(2), 181–207 (2003)CrossRefzbMATHGoogle Scholar
  14. 14.
    Misra, H., Cappé, O., Yvon, F.: Using lda to detect semantically incoherent documents. In: CoNLL, Stroudsburg, PA, USA, pp. 41–48 (2008)Google Scholar
  15. 15.
    Newman, D., Bonilla, E.V., Buntine, W.L.: Improving topic coherence with regularized topic models. In: NIPS, pp. 496–504 (2011)Google Scholar
  16. 16.
    Rashid, A.M., Karypis, G., Riedl, J.: Influence in ratings-based recommender systems: An algorithm-independent approach. In: SDM (2005)Google Scholar
  17. 17.
    Said, A., Berkovsky, S., Luca, E.W.D.: Movie recommendation in context. ACM TIST 4(1), 13 (2013)Google Scholar
  18. 18.
    Said, A., Jain, B.J., Narr, S., Plumbaum, T.: Users and noise: The magic barrier of recommender systems. In: Masthoff, J., Mobasher, B., Desmarais, M.C., Nkambou, R. (eds.) UMAP 2012. LNCS, vol. 7379, pp. 237–248. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  19. 19.
    Said, A., Jain, B.J., Narr, S., Plumbaum, T., Albayrak, S., Scheel, C.: Estimating the magic barrier of recommender systems: a user study. In: SIGIR, pp. 1061–1062 (2012)Google Scholar
  20. 20.
    Said, A., Kille, B., De Luca, E.W., Albayrak, S.: Personalizing tags: a folksonomy-like approach for recommending movies. In: HetRec 2011, New York, NY, USA. RecSys, pp. 53–56 (2011)Google Scholar
  21. 21.
    Shardanand, U., Maes, P.: Social information filtering: Algorithms for automating “word of mouth”. In: Katz, I.R., Mack, R.L., Marks, L., Rosson, M.B., Nielsen, J. (eds.) CHI, pp. 210–217. ACM/Addison-Wesley (1995)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Alejandro Bellogín
    • 1
  • Alan Said
    • 2
  • Arjen P. de Vries
    • 3
  1. 1.Ciudad Universitaria de CantoblancoUniversidad Autónoma de MadridMadridSpain
  2. 2.Delft University of TechnologyDelftThe Netherlands
  3. 3.Centrum Wiskunde & InformaticaAmsterdamThe Netherlands

Personalised recommendations