Evaluation Framework for Statistical User Models

  • Javier Calle
  • Leonardo Castaño
  • Elena Castro
  • Dolores Cuadra
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 217)


This paper analyzes the main barriers that user model developers have to face when evaluating a statistical user model. Main techniques used to evaluate statistical user models, mostly borrowed from the areas of Machine Learning and Information Retrieval, are examined. Then an evaluation methodology for statistical user models is proposed together with a set of metrics to specifically evaluate statistical user models. Finally, a benchmark for statistical user models is proposed, thus making possible to compare and replicate the evaluations. Thus, main contribution of this paper is to enable that several user model evaluations were comparable.


Statistical user model evaluation methodology benchmarking 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Amento, B., Terveen, L., Hix, D., Ju, P.: An empirical evaluation of user interfaces for topic management of web sites. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI 1999), pp. 552–559. ACM, New York (1999)Google Scholar
  2. 2.
    Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley (1999)Google Scholar
  3. 3.
    Balabanovic, M., Shoham, Y.: Fab: Content-based, collaborative recommendation. Commun. ACM 40, 66–72 (1997)CrossRefGoogle Scholar
  4. 4.
    Basu, C., Hirsh, H., Cohen, W.: Recommendation as Classification: Using Social and Content-Based Information in Recommendation. In: Proceedings of the 15th National Conference on Artificial Intelligence, Madison, WI, pp. 714–720 (1998)Google Scholar
  5. 5.
    Breese, J., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for collaborative clustering. In: Proc. of the Fourteenth Annual Conference on Uncertainty in Artificial Intelligence, San Francisco, vol. 43 (1998)Google Scholar
  6. 6.
    Bollen, D., Knijnenburg, B.P., Willemsen, M.C., Graus, M.: Understanding choice overload in recommender systems. In: Proceedings of the Fourth ACM Conference on Recommender Systems, pp. 63–70. ACM, New York (2010),, doi:10.1145/1864708.1864724CrossRefGoogle Scholar
  7. 7.
    Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J.: Is seeing believing? How recommender interfaces affect users’ opinions. CHI Lett. 5 (2003)Google Scholar
  8. 8.
    Heckerman, D., Chickering, D.M., Meek, C., Rounthwaite, R., Kadie, C.: Dependency networks for inference, collaborative filtering, and data visualization. J. Mach. Learn. Res. 1, 49–75 (2000)Google Scholar
  9. 9.
    Herlocker, J.L., Konstan, J.A., Riedl, J.: An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Inf. Retr. 5, 287–310 (2002)CrossRefGoogle Scholar
  10. 10.
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems 22(1), 5–53 (2004)CrossRefGoogle Scholar
  11. 11.
    Hernández, F.O., Gaudioso, E.: Evaluation of recommender systems: A new approach. Expert Syst. 35(3), 790–804 (2008),, doi:10.1016/j.eswa.2007.07.047CrossRefGoogle Scholar
  12. 12.
    Larson, H.J.: Introduction to Probability Theory and Statistical Inference. Wiley International Edition (1969)Google Scholar
  13. 13.
    Miller, B., Albert, I., Lam, S.K., Konstan, J., Riedl, J.: MovieLens Unplugged: Experiences with a Recommender System on Four Mobile Devices. In: Proceedings of the 17th Annual Human-Computer Interaction Conference (HCI). British HCI Group (September 2003)Google Scholar
  14. 14.
    Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems. In: Proceedings of the Fifth ACM Conference on Recommender Systems, pp. 157–164. ACM, New York (2011),, doi:10.1145/2043932.2043962CrossRefGoogle Scholar
  15. 15.
    Wade, W.: A grocery cart that holds bread, butter and preferences. NY Times (January 16, 2003)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Javier Calle
    • 1
  • Leonardo Castaño
    • 1
  • Elena Castro
    • 1
  • Dolores Cuadra
    • 1
  1. 1.Computer Science DepartmentCarlos III University of MadridMadridSpain

Personalised recommendations