The Impact of Demographics (Age and Gender) and Other User-Characteristics on Evaluating Recommender Systems
In this paper we show the importance of considering demographics and other user characteristics when evaluating (research paper) recommender systems. We analyzed 37,572 recommendations delivered to 1,028 users and found that elderly users clicked more often on recommendations than younger ones. For instance, 20-24 years old users achieved click-through rates (CTR) of 2.73% on average while CTR for users between 50 and 54 years was 9.26%. Gender only had a marginal impact (CTR males 6.88%; females 6.67%) but other user characteristics such as whether a user was registered (CTR: 6.95%) or not (4.97%) had a strong impact. Due to the results we argue that future research articles on recommender systems should report detailed data on their users to make results better comparable.
Keywordsrecommender systems demographics evaluation research paper
Unable to display preview. Download preview PDF.
- 1.Naak, A., Hage, H., Almeur, E.: A multi-criteria collaborative filtering approach for research paper recommendation in papyres. E-Technologies: Innovation in an Open World (2009)Google Scholar
- 3.Jomsri, P., Sanguansintukul, S., Choochaiwattana, W.: A framework for tag-based research paper recommender system: an IR approach. In: 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops, WAINA (2010)Google Scholar
- 4.Bonhard, P., Harries, C., McCarthy, J., Sasse, M.A.: Accounting for taste: using profile similarity to improve recommender systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1057–1066. ACM (2006)Google Scholar
- 5.Parsons, J., Ralph, P., Gallagher, K.: Using viewing time to infer user preference in recommender systems. In: Proceedings of the AAAI Workshop on Semantic Web Personalization Held in Conjunction with the 9th National Conference on Artificial Intelligence (2004)Google Scholar
- 6.Beel, J., Langer, S., Genzmehr, M., Nürnberger, A.: Introducing Docear’s Research Paper Recommender System. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, JCDL (2013)Google Scholar
- 7.Stober, S., Steinbrecher, M., Nürnberger, A.: A Survey on the Acceptance of Listening Context Logging for MIR Applications. In: Proceedings of the 3rd Workshop on Learning the Semantics of Audio Signals (LSAS), pp. 45–57 (2009)Google Scholar