Join the Living Lab: Evaluating News Recommendations in Real-Time

  • Frank Hopfgartner
  • Torben Brodt
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9022)


Participants of this tutorial learnt how to participate in CLEF NEWSREEL, a living lab for the evaluation of news recommender algorithms. Various research challenges can be addressed within NEWSREEL, such as the development and evaluation of collaborative filtering or content-based filtering strategies. Satisfying information needs by techniques including preference elicitation, pattern recognition, and prediction, recommender systems connect the research areas information retrieval and machine learning.


Recommender systems living lab user-centric evaluation large-scale evaluation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kille, B., Hopfgartner, F., Brodt, T., Heintz, T.: The plista dataset. In: NRS 2013: Proceedings of the International Workshop and Challenge on News Recommender Systems, pp. 14–21. ACM (October 2013)Google Scholar
  2. 2.
    Hopfgartner, F., Jose, J.M.: Semantic user modelling for personal news video retrieval. In: Boll, S., Tian, Q., Zhang, L., Zhang, Z., Chen, Y.-P.P. (eds.) MMM 2010. LNCS, vol. 5916, pp. 336–346. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  3. 3.
    Hopfgartner, F., Urban, J., Villa, R., Jose, J.M.: Simulated testing of an adaptive multimedia information retrieval system. In: International Workshop on Content-Based Multimedia Indexing, CBMI 2007, Bordeaux, France, June 25-27, pp. 328–335 (2007)Google Scholar
  4. 4.
    Schoeffmann, K., Ahlström, D., Bailer, W., Cobârzan, C., Hopfgartner, F., McGuinness, K., Gurrin, C., Frisson, C., Le, D., del Fabro, M., Bai, H., Weiss, W.: The video browser showdown: a live evaluation of interactive video search tools. IJMIR 3(2), 113–127 (2014)Google Scholar
  5. 5.
    White, R.W., Ruthven, I., Jose, J.M., van Rijsbergen, C.J.: Evaluating implicit feedback models using searcher simulations. ACM Trans. Inf. Syst. 23(3), 325–361 (2005)CrossRefGoogle Scholar
  6. 6.
    Kamps, J., Geva, S., Peters, C., Sakai, T., Trotman, A., Voorhees, E.M.: Report on the SIGIR 2009 workshop on the future of IR evaluation. SIGIR Forum 43(2), 13–23 (2009)CrossRefGoogle Scholar
  7. 7.
    Azzopardi, L., Balog, K.: Towards a living lab for information retrieval research and development - a proposal for a living lab for product search tasks. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 26–37. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  8. 8.
    Kelly, D., Dumais, S.T., Pedersen, J.O.: Evaluation challenges and directions for information-seeking support systems. IEEE Computer 42(3), 60–66 (2009)CrossRefGoogle Scholar
  9. 9.
    Hopfgartner, F., Kille, B., Lommatzsch, A., Plumbaum, T., Brodt, T., Heintz, T.: Benchmarking news recommendations in a living lab. In: Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., Toms, E. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 250–267. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  10. 10.
    Kille, B., Brodt, T., Heintz, T., Hopfgartner, F., Lommatzsch, A., Seiler, J.: NEWSREEL 2014: Summary of the news recommendation evaluation lab. In: Working Notes for CLEF 2014 Conference, Sheffield, UK, September 15-18, pp. 790–801 (2014)Google Scholar
  11. 11.
    Brodt, T., Hopfgartner, F.: Shedding Light on a Living Lab: The CLEF NEWSREEL Open Recommendation Platform. In: Proceedings of the Information Interaction in Context conference, pp. 223–226. Springer, Heidelberg (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Frank Hopfgartner
    • 1
  • Torben Brodt
    • 2
  1. 1.University of GlasgowGlasgowUnited Kingdom
  2. 2.plista GmbHBerlinGermany

Personalised recommendations