Stream-Based Recommendations: Online and Offline Evaluation as a Service

  • Benjamin Kille
  • Andreas Lommatzsch
  • Roberto Turrin
  • András Serény
  • Martha Larson
  • Torben Brodt
  • Jonas Seiler
  • Frank HopfgartnerEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9283)


Providing high-quality news recommendations is a challenging task because the set of potentially relevant news items changes continuously, the relevance of news highly depends on the context, and there are tight time constraints for computing recommendations. The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms online and offline. In this paper, we discuss the objectives and challenges of the NewsREEL lab. We motivate the metrics used for benchmarking the recommender algorithms and explain the challenge dataset. In addition, we introduce the evaluation framework that we have developed. The framework makes possible the reproducible evaluation of recommender algorithms for stream data, taking into account recommender precision as well as the technical complexity of the recommender algorithms.


Recommender systems News Evaluation Living lab Stream-based recommender 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Balog, K., Kelly, L., Schuth, A.: Head first: Living labs for ad-hoc search evaluation. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, New York, NY, USA, pp. 1815–1818. ACM (2014)Google Scholar
  2. 2.
    Blomo, J., Ester, M., Field, M.: Recsys challenge 2013. In: Proceedings of the 7th ACM Conference on Recommender Systems, RecSys 2013, pp. 489–490 (2013)Google Scholar
  3. 3.
    Brodt, T., Hopfgartner, F.: Shedding Light on a living lab: the CLEF NEWSREEL open recommendation platform. In: Proceedings of the Information Interaction in Context conference, IIiX 2014, pp. 223–226. Springer-Verlag (2014)Google Scholar
  4. 4.
    Campos, P.G., Díez, F., Cantador, I.: Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Model. User-Adapt. Interact. 24(1–2), 67–119 (2014)CrossRefGoogle Scholar
  5. 5.
    Chen, J., Nairn, R., Nelson, L., Bernstein, M.S., Chi, E.H.: Short and tweet: experiments on recommending content from information streams. In: Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Atlanta, Georgia, USA, April 10–15, 2010, pp. 1185–1194 (2010)Google Scholar
  6. 6.
    Chen, Y., Berkhin, P., Anderson, B., Devanur, N.R.: Real-time bidding algorithms for performance-based display ad allocation. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2011, pp. 1307–1315 (2011)Google Scholar
  7. 7.
    Diaz-Aviles, E., Drumond, L., Schmidt-Thieme, L., Nejdl, W.: Real-time top-n recommendation in social streams. In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys 2012, pp. 59–66 (2012)Google Scholar
  8. 8.
    Doychev, D., Lawlor, A., Rafter, R.: An analysis of recommender algorithms for online news. In: Working Notes for CLEF 2014 Conference, Sheffield, UK, September 15–18, 2014, pp. 825–836 (2014)Google Scholar
  9. 9.
    Garcin, F., Faltings, B., Donatsch, O., Alazzawi, A., Bruttin, C., Huber, A.: Offline and online evaluation of news recommender systems at In: Eighth ACM Conference on Recommender Systems, RecSys 2014, Foster City, Silicon Valley, CA, USA - October 06–10, 2014, pp. 169–176 (2014)Google Scholar
  10. 10.
    Goldberg, K., Roeder, T., Gupta, D., Perkins, C.: Eigentaste: A constant time collaborative filtering algorithm. Inf. Retr., 4(2), July 2001Google Scholar
  11. 11.
    Gonzáles, Á.C., García-Serrano, A.M., Cigarrán, J.: UNED @ clef-newsreel 2014. In: Working Notes for CLEF 2014 Conference, Sheffield, UK, September 15–18, 2014, pp. 802–812 (2014)Google Scholar
  12. 12.
    GroupLens Research. MovieLens data sets, October 2006.
  13. 13.
    Hopfgartner, F., Hanbury, A., Mueller, H., Kando, N., Mercer, S., Kalpathy-Cramer, J., Potthast, M., Gollup, T., Krithara, A., Lin, J., Balog, K., Eggel, I.: Report of the evaluation-as-a-service (eaas) expert workshop. SIGIR Forum 49(1), 57–65 (2015)CrossRefGoogle Scholar
  14. 14.
    Hopfgartner, F., Kille, B., Lommatzsch, A., Plumbaum, T., Brodt, T., Heintz, T.: Benchmarking news recommendations in a living lab. In: Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., Toms, E. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 250–267. Springer, Heidelberg (2014) Google Scholar
  15. 15.
    Kille, B., Hopfgartner, F., Brodt, T., Heintz, T.: The plista dataset. In: NRS 2013: Proceedings of the International Workshop and Challenge on News Recommender Systems, pp. 14–21. ACM, 10 2013Google Scholar
  16. 16.
    Kille, B., Lommatzsch, A., Turrin, R., Serny, A., Larson, M., Brodt, T., Seiler, J., Hopfgartner, F.: Overview of clef newsreel 2015: News recommendation evaluation labs. In: 6th International Conference of the CLEF Initiative (2015)Google Scholar
  17. 17.
    Kuchar, J., Kliegr, T.: Inbeat: Recommender system as a service. In: Working Notes for CLEF 2014 Conference, Sheffield, UK, September 15–18, 2014, pp. 837–844 (2014)Google Scholar
  18. 18.
    Li, L., Chu, W., Langford, J., Wang, X.: Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In: Proceedings of the Forth International Conference on Web Search and Web Data Mining, WSDM 2011, Hong Kong, China, February 9–12, 2011, pp. 297–306 (2011)Google Scholar
  19. 19.
    Lin, J., Efron, M.: Evaluation as a service for information retrieval. SIGIR Forum 47(2), 8–14 (2013)CrossRefGoogle Scholar
  20. 20.
    Lommatzsch, A., Albayrak, S.: Real-time recommendations for user-item streams. In: Proc. of the 30th Symposium On Applied Computing, SAC 2015, New York, NY, USA, pp. 1039–1046. ACM (2015) Google Scholar
  21. 21.
    McFee, B., Bertin-Mahieux, T., Ellis, D.P., Lanckriet, G.R.: The million song dataset challenge. In: Proceedings of the 21st International Conference Companion on World Wide Web, WWW 2012 Companion, pp. 909–916 (2012)Google Scholar
  22. 22.
    Di Noia, T., Cantador, I., Ostuni, V.C.: Linked open data-enabled recommender systems: ESWC 2014 challenge on book recommendation. In: Presutti, V., Stankovic, M., Cambria, E., Cantador, I., Di Iorio, A., Di Noia, T., Lange, C., Reforgiato Recupero, D., Tordai, A. (eds.) SemWebEval 2014. CCIS, vol. 475, pp. 129–143. Springer, Heidelberg (2014) Google Scholar
  23. 23.
    Said, A., Loni, B., Turrin, R., Lommatzsch, A.: An extended data model format for composite recommendation. In: Poster Proceedings of the 8th ACM Conference on Recommender Systems, RecSys 2014, Foster City, Silicon Valley, CA, USA, October 6–10, 2014 (2014)Google Scholar
  24. 24.
    Said, A., Tikk, D., Stumpf, K., Shi, Y., Larson, M., Cremonesi, P.: Recommender systems evaluation: A 3D benchmark. In: Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE, RUE 2012, Dublin, Ireland, September 9, 2012, pp. 21–23. CEUR (2012) Google Scholar
  25. 25.
    Tavakolifard, M., Gulla, J.A., Almeroth, K.C., Hopfgartner, F., Kille, B., Plumbaum, T., Lommatzsch, A., Brodt, T., Bucko, A., Heintz, T.: Workshop and challenge on news recommender systems. In: Seventh ACM Conference on Recommender Systems, RecSys 2013, Hong Kong, China, October 12–16, 2013, pp. 481–482 (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Benjamin Kille
    • 1
  • Andreas Lommatzsch
    • 1
  • Roberto Turrin
    • 2
  • András Serény
    • 3
  • Martha Larson
    • 4
  • Torben Brodt
    • 5
  • Jonas Seiler
    • 5
  • Frank Hopfgartner
    • 6
    Email author
  1. 1.TU BerlinBerlinGermany
  2. 2.ContentWise R&D - MoviriMilanItaly
  3. 3.Gravity R&DBudapestHungary
  4. 4.TU DelftDelftThe Netherlands
  5. 5.Plista GmbHBerlinGermany
  6. 6.University of GlasgowGlasgowUK

Personalised recommendations