Predicting the Performance of Recommender Systems: An Information Theoretic Approach

  • Alejandro Bellogín
  • Pablo Castells
  • Iván Cantador
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6931)

Abstract

Performance prediction is an appealing problem in Recommender Systems, as it enables an array of strategies for deciding when to deliver or hold back recommendations based on their foreseen accuracy. The problem, however, has been barely addressed explicitly in the area. In this paper, we propose adaptations of query clarity techniques from ad-hoc Information Retrieval to define performance predictors in the context of Recommender Systems, which we refer to as user clarity. Our experiments show positive results with different user clarity models in terms of the correlation with single recommender’s performance. Empiric results show significant dependency between this correlation and the recommendation method at hand, as well as competitive results in terms of average correlation.

Keywords

performance prediction recommender systems language models 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Adomavicius, G., Tuzhilin, T.: Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. on Knowl. and Data Eng. 17(6), 734–749 (2005)CrossRefGoogle Scholar
  2. 2.
    Amati, G., Carpineto, C., Romano, G.: Query difficulty, robustness, and selective application of query expansion. In: McDonald, S., Tait, J.I. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 127–137. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  3. 3.
    Bellogín, A., Castells, P.: A performance prediction approach to enhance collaborative filtering performance. In: Gurrin, C., He, Y., Kazai, G., Kruschwitz, U., Little, S., Roelleke, T., Rüger, S., van Rijsbergen, K. (eds.) ECIR 2010. LNCS, vol. 5993, pp. 382–393. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  4. 4.
    Bellogín, A., Wang, J., Castells, P.: Text retrieval methods for item ranking in collaborative filtering. In: Clough, P., Foley, C., Gurrin, C., Jones, G.J.F., Kraaij, W., Lee, H., Mudoch, V. (eds.) ECIR 2011. LNCS, vol. 6611, pp. 301–306. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  5. 5.
    Buckley, C.: Topic prediction based on comparative retrieval rankings. In: 27th ACM Conference on Research and Development in Information Retrieval, pp. 506–507. ACM Press, New York (2004)Google Scholar
  6. 6.
    Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. In: 25th ACM Conference on Research and Development in Information Retrieval, pp. 299–306. ACM Press, New York (2002)Google Scholar
  7. 7.
    Cronen-Townsend, S., Zhou, Y., Croft, W.B.: A framework for selective query expansion. In: 13th ACM Conference on Information and Knowledge Management, pp. 236–237. ACM Press, New York (2004)Google Scholar
  8. 8.
    Dang, V., Bendersky, M., Croft, W.B.: Learning to rank query reformulations. In: 33rd ACM Conference on Research and Development in Information Retrieval, pp. 807–808. ACM Press, New York (2010)Google Scholar
  9. 9.
    Hauff, C.: Predicting the Effectiveness of Queries and Retrieval Systems. PhD thesis, University of Twente, Enschede (2010)Google Scholar
  10. 10.
    Hauff, C., Hiemstra, D., de Jong, F.: A Survey of Pre-Retrieval Query Performance Predictors. In: 17th ACM Conference on Information and Knowledge Management, pp. 439–448. ACM Press, New York (2008)Google Scholar
  11. 11.
    He, B., Ounis, I.: Inferring query performance using pre-retrieval predictors. In: Apostolico, A., Melucci, M. (eds.) SPIRE 2004. LNCS, vol. 3246, pp. 43–54. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  12. 12.
    Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: 22nd ACM Conference on Research and Development in Information Retrieval, pp. 230–237. ACM Press, New York (1999)Google Scholar
  13. 13.
    Hofmann, T.: Latent semantic models for Collaborative filtering. ACM Trans. Inf. Syst. 22(1), 89–115 (2004)CrossRefGoogle Scholar
  14. 14.
    Mothe, J., Tanguy, L.: Linguistic features to predict query difficulty. In: ACM SIGIR Workshop on Predicting Query Difficulty – Methods and Applications (2005)Google Scholar
  15. 15.
    Pérez-Iglesias, J., Araujo, L.: Ranking List Dispersion as a Query Performance Predictor. In: Azzopardi, L., Kazai, G., Robertson, S., Rüger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 371–374. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  16. 16.
    Plachouras, V., He, B., Ounis, I.: University of Glasgow at TREC2004: Experiments in Web, Robust and Terabyte tracks with Terrier. In: 13th Text Retrieval Conference, Gaithesburg, Maryland (2004)Google Scholar
  17. 17.
    Shtok, A., Kurland, O., Carmel, D.: Predicting Query Performance by Query-Drift Estimation. In: Azzopardi, L., Kazai, G., Robertson, S., Rüger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 305–312. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  18. 18.
    Wang, J., de Vries, A.P., Reinders, M.J.T.: Unified relevance models for rating prediction in collaborative filtering. ACM Trans. Inf. Syst. 26(3), 1–42 (2008)CrossRefGoogle Scholar
  19. 19.
    Yom-Tov, E., Fine, S., Carmel, D., Darlow, A.: Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In: 28th ACM Conference on Research and Development in Information Retrieval, pp. 512–519. ACM Press, New York (2005)Google Scholar
  20. 20.
    Zhou, Y., Croft, B.W.: Query performance prediction in web search environments. In: 30th ACM Conference on Research and Development in Information Retrieval, pp. 543–550. ACM Press, New York (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Alejandro Bellogín
    • 1
  • Pablo Castells
    • 1
  • Iván Cantador
    • 1
  1. 1.Departamento de Ingeniería InformáticaUniversidad Autónoma de Madrid Escuela Politécnica SuperiorMadridSpain

Personalised recommendations