Online Evaluations for Everyone: Mr. DLib’s Living Lab for Scholarly Recommendations

  • Joeran BeelEmail author
  • Andrew Collins
  • Oliver Kopp
  • Linus W. Dietz
  • Petr Knoth
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11438)


We introduce the first ‘living lab’ for scholarly recommender systems. This lab allows recommender-system researchers to conduct online evaluations of their novel algorithms for scholarly recommendations, i.e., recommendations for research papers, citations, conferences, research grants, etc. Recommendations are delivered through the living lab’s API to platforms such as reference management software and digital libraries. The living lab is built on top of the recommender-system as-a-service Mr. DLib. Current partners are the reference management software JabRef and the CORE research team. We present the architecture of Mr. DLib’s living lab as well as usage statistics on the first sixteen months of operating it. During this time, 1,826,643 recommendations were delivered with an average click-through rate of 0.21%.


Recommender system evaluation Living lab Online evaluation 


  1. 1.
    Hanbury, A., et al.: Evaluation-as-a-service: overview and outlook. arXiv preprint arXiv:1512.07454 (2015)
  2. 2.
    Hopfgartner, F., et al.: Report on the evaluation-as-a-service (EaaS) expert workshop. In: ACM SIGIR Forum, pp. 57–65. ACM (2015)Google Scholar
  3. 3.
    Hopfgartner, F., et al.: Evaluation-as-a-service for the computational sciences: overview and outlook. J. Data Inf. Qual. (JDIQ) 10, 15 (2018)Google Scholar
  4. 4.
    Brodt, T., Hopfgartner, F.: Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform. In: Proceedings of the 5th Information Interaction in Context Symposium, pp. 223–226. ACM (2014)Google Scholar
  5. 5.
    Hopfgartner, F., et al.: Benchmarking news recommendations: the clef newsreel use case. In: ACM SIGIR Forum, pp. 129–136. ACM (2016)Google Scholar
  6. 6.
    Kille, B., et al.: Overview of NewsREEL’16: multi-dimensional evaluation of real-time stream-recommendation algorithms. In: Fuhr, N., et al. (eds.) CLEF 2016. LNCS, vol. 9822, pp. 311–331. Springer, Cham (2016). Scholar
  7. 7.
    Balog, K., Elsweiler, D., Kanoulas, E., Kelly, L., Smucker, M.D.: Report on the CIKM workshop on living labs for information retrieval evaluation. In: ACM SIGIR Forum, pp. 21–28. ACM (2014)Google Scholar
  8. 8.
    Carevic, Z., Schüller, S., Mayr, P., Fuhr, N.: Contextualised browsing in a digital library’s living lab. In: Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries, Fort Worth, Texas, USA, pp. 89–98. ACM (2018)Google Scholar
  9. 9.
    Li, S., Brusilovsky, P., Su, S., Cheng, X.: Conference paper recommendation for academic conferences. IEEE Access 6, 17153–17164 (2018)CrossRefGoogle Scholar
  10. 10.
    Vargas, S., Hristakeva, M., Jack, K.: Mendeley: recommendations for researchers. In: Proceedings of the 10th ACM Conference on Recommender Systems, Boston, Massachusetts, USA, pp. 365–365. ACM (2016)Google Scholar
  11. 11.
    Färber, M., Thiemann, A., Jatowt, A.: CITEWERTs: a system combining cite-worthiness with citation recommendation. In: Pasi, G., Piwowarski, B., Azzopardi, L., Hanbury, A. (eds.) ECIR 2018. LNCS, vol. 10772, pp. 815–819. Springer, Cham (2018). Scholar
  12. 12.
    Jia, H., Saule, E.: Graph Embedding for Citation Recommendation, arXiv preprint arXiv:1812.03835 (2018)
  13. 13.
    Beierle, F., Tan, J., Grunert, K.: Analyzing social relations for recommending academic conferences. In: Proceedings of the 8th ACM International Workshop on Hot Topics in Planet-scale mObile computing and online Social neTworking, pp. 37–42. ACM (2016)Google Scholar
  14. 14.
    Yu, S., Liu, J., Yang, Z., Chen, Z., Jiang, H., Tolba, A., Xia, F.: PAVE: personalized academic venue recommendation exploiting co-publication networks. J. Netw. Comput. Appl. 104, 38–47 (2018)CrossRefGoogle Scholar
  15. 15.
    Kou, N.M., Mamoulis, N., Li, Y., Li, Y., Gong, Z., et al.: A topic-based reviewer assignment system. Proc. VLDB Endowment 8, 1852–1855 (2015)CrossRefGoogle Scholar
  16. 16.
    Lian, J.W., Mattei, N., Noble, R., Walsh, T.: The conference paper assignment problem: using order weighted averages to assign indivisible goods. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)Google Scholar
  17. 17.
    Tan, J., Wan, X., Liu, H., Xiao, J.: QuoteRec: toward quote recommendation for writing. ACM Trans. Inf. Syst. (TOIS) 36, 34 (2018)CrossRefGoogle Scholar
  18. 18.
    Kong, X., Jiang, H., Wang, W., Bekele, T.M., Xu, Z., Wang, M.: Exploring dynamic research interest and academic influence for scientific collaborator recommendation. Scientometrics 113, 369–385 (2017)CrossRefGoogle Scholar
  19. 19.
    Moreira, G.S.P., de Souza, G.A., da Cunha, A.M.: Comparing offline and online recommender system evaluations on long-tail distributions. In: Proceedings of the ACM Recommender Systems Conference RecSys (2015)Google Scholar
  20. 20.
    Rossetti, M., Stella, F., Zanker, M.: Contrasting offline and online results when evaluating recommendation algorithms. In: Proceedings of the 10th ACM Conference on Recommender Systems, Boston, Massachusetts, USA, pp. 31–34. ACM (2016)Google Scholar
  21. 21.
    Beel, J., Aizawa, A., Breitinger, C., Gipp, B.: Mr. DLib: recommendations-as-a-service (RaaS) for academia. In: Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries, Toronto, Ontario, Canada, pp. 313–314. IEEE Press (2017)Google Scholar
  22. 22.
    Beel, J., Collins, A., Aizawa, A.: The architecture of Mr. DLib’s scientific recommender-system API. In: Proceedings of the 26th Irish Conference on Artificial Intelligence and Cognitive Science (AICS), CEUR-WS, pp. 78–89 (2018)Google Scholar
  23. 23.
    Feyer, S., Siebert, S., Gipp, B., Aizawa, A., Beel, J.: Integration of the scientific recommender system Mr. DLib into the reference manager JabRef. In: Jose, Joemon M., et al. (eds.) ECIR 2017. LNCS, vol. 10193, pp. 770–774. Springer, Cham (2017). Scholar
  24. 24.
    Kopp, O., Breitenbuecher, U., Mueller, T.: CloudRef - towards collaborative reference management in the cloud. In: Proceedings of the 10th Central European Workshop on Services and their Composition (2018)Google Scholar
  25. 25.
    Hristakeva, M., et al.: Building recommender systems for scholarly information. In: Proceedings of the 1st Workshop on Scholarly Web Mining, pp. 25–32. ACM (2017)Google Scholar
  26. 26.
    Knoth, P., et al.: Towards effective research recommender systems for repositories. In: Proceedings of the Open Repositories Conference (2017)Google Scholar
  27. 27.
    Pontika, N., Anastasiou, L., Charalampous, A., Cancellieri, M., Pearce, S., Knoth, P.: CORE recommender: a plug in suggesting open access content (2017).
  28. 28.
    Beel, J., Dinesh, S., Mayr, P., Carevic, Z., Raghvendra, J.: Stereotype and most-popular recommendations in the digital library sowiport. In: Proceedings of the 15th International Symposium of Information Science (ISI), pp. 96–108 (2017)Google Scholar
  29. 29.
    Beel, J., Smyth, B., Collins, A.: RARD II: The 2nd Related-Article Recommendation Dataset. arXiv:1807.06918 [cs.IR] (2018)
  30. 30.
    Beel, J., Langer, S., Gipp, B., Nuernberger, A.: The architecture and datasets of Docear’s research paper recommender system. D-Lib Mag. 20 (2014)Google Scholar
  31. 31.
    Hienert, D., Sawitzki, F., Mayr, P.: Digital library research in action-supporting information retrieval in sowiport. D-Lib Mag. 21 (2015)Google Scholar
  32. 32.
    Mayr, P.: Sowiport User Search Sessions Data Set (SUSS). GESIS Datorium (2016)Google Scholar
  33. 33.
    Stempfhuber, M., Schaer, P., Shen, W.: Enhancing visibility: integrating grey literature in the SOWIPORT information cycle. In: International Conference on Grey Literature, pp. 23–29 (2008)Google Scholar
  34. 34.
    Beel, J., Breitinger, C., Langer, S., Lommatzsch, A., Gipp, B.: Towards reproducibility in recommender-systems research. User Model. User-Adap. Inter. (UMUAI) 26, 69–101 (2016)CrossRefGoogle Scholar
  35. 35.
    Ferro, N., Fuhr, N., Rauber, A.: Introduction to the special issue on reproducibility in information retrieval: tools and infrastructures. J. Data Inf. Qual. (JDIQ) 10, 14 (2018)Google Scholar
  36. 36.
    Collins, A., Tkaczyk, D., Beel, J.: A novel approach to recommendation algorithm selection using meta-learning. In: Proceedings of the 26th Irish Conference on Artificial Intelligence and Cognitive Science (AICS). CEUR-WS, pp. 210–219 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Joeran Beel
    • 1
    • 2
    Email author
  • Andrew Collins
    • 1
  • Oliver Kopp
    • 3
  • Linus W. Dietz
    • 4
  • Petr Knoth
    • 5
  1. 1.School of Computer Science and Statistics, ADAPT CentreTrinity College DublinDublinIreland
  2. 2.Digital Content and Media Sciences DivisionNational Institute of InformaticsTokyoJapan
  3. 3.IPVS, University of StuttgartStuttgartGermany
  4. 4.Department of InformaticsTechnical University of MunichGarchingGermany
  5. 5.Knowledge Media InstituteThe Open UniversityLondonUK

Personalised recommendations