Evaluating Group Recommender Systems

Part of the SpringerBriefs in Electrical and Computer Engineering book series (BRIEFSELECTRIC)


In the previous chapters, we have learned how to design group recommender systems but did not explicitly discuss how to evaluate them. The evaluation techniques for group recommender systems are often the same or similar to those that are used for single user recommenders. We show how to apply these techniques on the basis of examples and introduce evaluation approaches that are specifically useful in group recommendation scenarios.


  1. 1.
    M. Allahbakhsh, B. Benatallah, A. Ignjatovic, H. Motahari-Nezhad, E. Bertino, S. Dustdar, Quality control in crowdsourcing systems: issues and directions. IEEE Int. Comput. 17(12), 76–81 (2013)CrossRefGoogle Scholar
  2. 2.
    L. Baltrunas, T. Makcinskas, F. Ricci, Group recommendations with rank aggregation and collaborative filtering, in 4th ACM Conference on Recommender Systems, Barcelona, 2010, pp. 119–126Google Scholar
  3. 3.
    R. Burke, Multisided fairness for recommendation, in 2017 Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2017), 2017, pp. 1–5Google Scholar
  4. 4.
    P. Campos, F. Díez, I. Cantador, Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Model. User Adap. Inter. 24(1–2), 67–119 (2014)CrossRefGoogle Scholar
  5. 5.
    J. Castro, F. Quesada, I. Palomares, L. Martínez, A consensus-driven group recommender system. Intell. Syst. 30(8), 887–906 (2015)Google Scholar
  6. 6.
    S. Chang, F. Harper, L. He, L. Terveen, CrowdLens: experimenting with crowd-powered recommendation and explanation, in 10th International AAAI Conference on Web and Social Media (ICWSM’16) (AAAI, Menlo Park, 2016), pp. 52–61Google Scholar
  7. 7.
    T. DePessemier, J. Dhondt, L. Martens, Hybrid group recommendations for a travel service. Multimedia Tools Appl. 76(2), 2787–2811 (2017)CrossRefGoogle Scholar
  8. 8.
    W. Dubitzky, M. Granzow, D. Berrar, Fundamentals of Data Mining in Genomics and Proteomics (Springer, Berlin, 2007)CrossRefzbMATHGoogle Scholar
  9. 9.
    A. Felfernig, M. Schubert, C. Zehentner, An efficient diagnosis algorithm for inconsistent constraint sets. Artif. Intell. Eng. Design Anal. Manufact. 26(1), 53–62 (2012)CrossRefGoogle Scholar
  10. 10.
    A. Felfernig, M. Jeran, G. Ninaus, F. Reinfrank, S. Reiterer, M. Stettinger, Basic approaches in recommendation systems, in Recommendation Systems in Software Engineering (Springer, Berlin, 2013), pp. 15–37Google Scholar
  11. 11.
    A. Felfernig, L. Hotz, C. Bagley, J. Tiihonen, Knowledge-Based Configuration: From Research to Business Cases, 1st edn. (Elsevier/Morgan Kaufmann, Amsterdam/Waltham, 2014)Google Scholar
  12. 12.
    A. Felfernig, M. Atas, T.N. Trang Tran, M. Stettinger, Towards group-based configuration, in International Workshop on Configuration 2016 (ConfWS’16) (2016), pp. 69–72Google Scholar
  13. 13.
    M. Ge, C. Delgado-Battenfeld, D. Jannach, Beyond accuracy: evaluating recommender systems by coverage and serendipity, in 4th ACM Conference on Recommender Systems, Barcelona, 2010, pp. 257–260Google Scholar
  14. 14.
    F. Harper, J. Konstan, The MovieLens Datasets: history and context. ACM Trans. Interactive Intell. Syst. 5(4), 19 (2015)Google Scholar
  15. 15.
    J. Herlocker, J. Konstan, L. Terveen, J. Riedl, Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)CrossRefGoogle Scholar
  16. 16.
    B. Knijnenburg, M. Willemsen, Evaluating recommender systems with user experiments, in Recommender Systems Handbook, ed. by F. Ricci, L. Rokach, B. Shapira (Springer, Berlin, 2015), pp. 309–352CrossRefGoogle Scholar
  17. 17.
    J. Masthoff, A. Gatt, In pursuit of satisfaction and the prevention of embarrassment: affective state in group recommender systems. User Model. User Adap. Inter. 16(3–4), 281–319 (2006)CrossRefGoogle Scholar
  18. 18.
    A. Mislove, B. Viswanath, K. Gummadi, P. Druschel, You are who you know: inferring user profiles in online social networks, in 3rd ACM Conference on Web Search and Data Mining (WSDM’10), New York, 2010, pp. 251–260Google Scholar
  19. 19.
    D. Parra, S. Sahebi, Recommender systems: sources of knowledge and evaluation metrics, in Advanced Techniques in Web Intelligence-2: Web User Browsing Behaviour and Preference Analysis (Springer, Berlin, 2013), pp. 149–175Google Scholar
  20. 20.
    L. Quijano-Sanchez, J. Recio-García, B. Díaz-Agudo, G. Jiménez-Díaz, Happy Movie: a group recommender application in Facebook, in 24th International Florida Artificial Intelligence Research Society Conference, Palm Beach, FL, 2011, pp. 419–420Google Scholar
  21. 21.
    A. Said, D. Tikk, P. Cremonesi, Benchmarking – a methodology for ensuring the relative quality of recommendation systems in software engineering, in Recommender Systems in Software Engineering (Springer, Berlin, 2014), pp. 275–300Google Scholar
  22. 22.
    H. Schaefer, S. Hors-Fraile, R. Karumur, A. Valdez, A. Said, H. Torkamaan, H. Ulmer, C. Trattner, Towards health (aware) recommender systems, in DH 2017 (2017)Google Scholar
  23. 23.
    D. Serbos, S. Qi, N. Mamoulis, E. Pitoura, P. Tsaparas, Fairness in package-to-group recommendations, in WWW’17 (ACM, New York, 2017), pp. 371–379Google Scholar
  24. 24.
    G. Shani, A. Gunawardana, Evaluating recommender systems, in Recommender Systems Handbook, ed. by F. Ricci, L. Rokach, B. Shapira, P. Kantor (Springer, Berlin, 2011), pp. 257–297CrossRefGoogle Scholar
  25. 25.
    M. Stettinger, A. Felfernig, G. Leitner, S. Reiterer, Counteracting anchoring effects in group decision making, in 23rd Conference on User Modeling, Adaptation, and Personalization (UMAP’15), Dublin. Lecture Notes in Computer Science, vol. 9146 (2015), pp. 118–130CrossRefGoogle Scholar
  26. 26.
    J. Tiihonen, A. Felfernig, Towards recommending configurable offerings. Int. J. Mass Custom. 3(4), 389–406 (2010)CrossRefGoogle Scholar
  27. 27.
    C. Trattner, D. Elsweiler, Investigating the healthiness of internet-sourced recipes: implications for mean planning and recommender systems, in 26th International Conference on the World Wide Web (2017), pp. 489–498Google Scholar
  28. 28.
    T. Ulz, M. Schwarz, A. Felfernig, S. Haas, A. Shehadeh, S. Reiterer, M. Stettinger, Human computation for constraint-based recommenders. J. Intell. Inf. Syst. 49(1), 37–57 (2017)CrossRefGoogle Scholar
  29. 29.
    L. Xiao, Z. Min, Z. Yongfeng, G. Zhaoquan, L. Yiqun, M. Shaoping, Fairness-aware group recommendation with pareto-efficiency, in ACM Conference on Recommender Systems (RecSys’17) (ACM, New York, 2017), pp. 107–115Google Scholar
  30. 30.
    A. Zapata, V. Menéndez, M. Prieto, C. Romero, Evaluation and selection of group recommendation strategies for collaborative searching of learning objects. Int. J. Hum. Comput. Stud. 76, 22–39 (2015)CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Authors and Affiliations

  1. 1.Institute for Software TechnologyGraz University of TechnologyGrazAustria
  2. 2.EURECATCentre Tecnológico de CatalunyaBarcelonaSpain
  3. 3.Faculty of Computer ScienceFree University of Bozen-BolzanoBolzanoItaly

Personalised recommendations