Overview of NewsREEL’16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms

  • Benjamin Kille
  • Andreas Lommatzsch
  • Gebrekirstos G. Gebremeskel
  • Frank Hopfgartner
  • Martha Larson
  • Jonas Seiler
  • Davide Malagoli
  • András Serény
  • Torben Brodt
  • Arjen P. de Vries
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9822)

Abstract

Successful news recommendation requires facing the challenges of dynamic item sets, contextual item relevance, and of fulfilling non-functional requirements, such as response time. The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to tackle news recommendation and to optimize and evaluate their recommender algorithms both online and offline. In this paper, we summarize the objectives and challenges of NewsREEL 2016. We cover two contrasting perspectives on the challenge: that of the operator (the business providing recommendations) and that of the challenge participant (the researchers developing recommender algorithms). In the intersection of these perspectives, new insights can be gained on how to effectively evaluate real-time stream recommendation algorithms.

Keywords

Recommender Systems News Multi-dimensional Evaluation Living Lab Stream-based Recommender 

Notes

Acknowledgments

The research leading to these results was performed in the CrowdRec project, which has received funding from the European Union Seventh Framework Program FP7/2007–2013 under grant agreement No. 610594.

References

  1. 1.
    Balog, K., Kelly, L., Schuth, A.: Head first: living labs for ad-hoc search evaluation. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, pp. 1815–1818. ACM, New York, NY, USA (2014)Google Scholar
  2. 2.
    Blomo, J., Ester, M., Field, M.: RecSys challenge 2013. In: Proceedings of the 7th ACM Conference on Recommender Systems, RecSys 2013, pp. 489–490 (2013)Google Scholar
  3. 3.
    Brodt, T., Hopfgartner, F.: Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform. In: IIiX 2014, pp. 223–226 (2014)Google Scholar
  4. 4.
    Chen, J., Nairn, R., Nelson, L., Bernstein, M.S., Chi, E.H.: Short and tweet: experiments on recommending content from information streams. In: Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Atlanta, Georgia, USA, April 10–15, 2010, pp. 1185–1194 (2010)Google Scholar
  5. 5.
    Chen, Y., Berkhin, P., Anderson, B., Devanur, N.R.: Real-time bidding algorithms for performance-based display ad allocation. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2011, pp. 1307–1315 (2011)Google Scholar
  6. 6.
    Diaz-Aviles, E., Drumond, L., Schmidt-Thieme, L., Nejdl, W.: Real-time top-n recommendation in social streams. In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys 2012, pp. 59–66 (2012)Google Scholar
  7. 7.
    Ekstrand, M.D., Ludwig, M., Konstan, J.A., Riedl, J.T.: Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit. In: RecSys 2011, pp. 133–140. ACM (2011)Google Scholar
  8. 8.
    Gantner, Z., Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: MyMediaLite: a free recommender system library. In: RecSys 2011, pp. 305–308. ACM (2011)Google Scholar
  9. 9.
    Garcin, F., Faltings, B., Donatsch, O., Alazzawi, A., Bruttin, C., Huber, A.: Offline and online evaluation of news recommender systems at swissinfo.ch. In: Eighth ACM Conference on Recommender Systems, RecSys 2014, Foster City, Silicon Valley, CA, USA - October 6–10, 2014, pp. 169–176 (2014)Google Scholar
  10. 10.
    Gebremeskel, G., de Vries, A.: Random performance differences between online recommender systems algorithms. (Manuscript submitted for publication) (2016)Google Scholar
  11. 11.
    Harper, F.M., Konstan, J.A.: The Movielens datasets: history and context. ACM Trans. Interact. Intell. Syst. 5(4), 19: 1–19: 19 (2015)CrossRefGoogle Scholar
  12. 12.
    Hopfgartner, F., Brodt, T., Seiler, J., Kille, B., Lommatzsch, A., Larson, M., Turrin, R., Serény, A.: Benchmarking news recommendations: The CLEF NewsREEL use case. SIGIR Forum 49(2), 129–136 (2015)CrossRefGoogle Scholar
  13. 13.
    Hopfgartner, F., Kille, B., Lommatzsch, A., Plumbaum, T., Brodt, T., Heintz, T.: Benchmarking news recommendations in a living lab. In: Kanoulas, E., Lupu, M., Clough, P., Sanderson, M., Hall, M., Hanbury, A., Toms, E. (eds.) CLEF 2014. LNCS, vol. 8685, pp. 250–267. Springer, Heidelberg (2014)Google Scholar
  14. 14.
    Kille, B., Hopfgartner, F., Brodt, T., Heintz, T.: The plista dataset. In: Proceedings of the 2013 International News Recommender Systems Workshop and Challenge, NRS 2013, pp. 16–23. ACM, New York, (2013)Google Scholar
  15. 15.
    Kille, B., Lommatzsch, A., Turrin, R., Serény, A., Larson, M., Brodt, T., Seiler, J., Hopfgartner, F.: Stream-based recommendations: online and offline evaluation as a service. In: Mothe, J., et al. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 497–517. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  16. 16.
    Lommatzsch, A., Albayrak, S.: Real-time recommendations for user-item streams. In: Proceedings of the 30th Symposium On Applied Computing, SAC 2015, SAC 2015, pp. 1039–1046. ACM, New York, (2015)Google Scholar
  17. 17.
    Maksai, A., Garcin, F., Faltings, B.: Predicting online performance of news recommender systems through richer evaluation metrics. In: Proceedings of the 9th ACM Conference on Recommender Systems, RecSys 2015, pp. 179–186. ACM, New York, (2015)Google Scholar
  18. 18.
    McFee, B., Bertin-Mahieux, T., Ellis, D.P., Lanckriet, G.R.: The million song dataset challenge. In: Proceedings of the 21st International Conference Companion on World Wide Web, WWW 2012 Companion, pp. 909–916 (2012)Google Scholar
  19. 19.
    Di Noia, T., Cantador, I., Ostuni, V.C.: Linked open data-enabled recommender systems: ESWC 2014 challenge on book recommendation. In: Presutti, V., et al. (eds.) SemWebEval 2014. CCIS, vol. 475, pp. 129–143. Springer, Heidelberg (2014)Google Scholar
  20. 20.
    Said, A., Bellogín, A.: Rival: a toolkit to foster reproducibility in recommender system evaluation. In: RecSys 2014, pp. 371–372. ACM, New York, (2014)Google Scholar
  21. 21.
    Tavakolifard, M., Gulla, J.A., Almeroth, K.C., Hopfgartner, F., Kille, B., Plumbaum, T., Lommatzsch, A., Brodt, T., Bucko, A., Heintz, T.: Workshop and challenge on news recommender systems. In: Seventh ACM Conference on Recommender Systems, RecSys 2013, Hong Kong, China, October 12–16, 2013, pp. 481–482 (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Benjamin Kille
    • 1
  • Andreas Lommatzsch
    • 1
  • Gebrekirstos G. Gebremeskel
    • 2
  • Frank Hopfgartner
    • 3
  • Martha Larson
    • 4
    • 7
  • Jonas Seiler
    • 5
  • Davide Malagoli
    • 6
  • András Serény
    • 8
  • Torben Brodt
    • 5
  • Arjen P. de Vries
    • 7
  1. 1.TU BerlinBerlinGermany
  2. 2.CWIAmsterdamThe Netherlands
  3. 3.University of GlasgowGlasgowUK
  4. 4.TU DelftDelftThe Netherlands
  5. 5.Plista GmbHBerlinGermany
  6. 6.ContentWise R&D — MoviriMilanItaly
  7. 7.Radboud University NijmegenNijmegenThe Netherlands
  8. 8.Gravity ResearchBudapestHungary

Personalised recommendations