Metrics for Evaluating the Serendipity of Recommendation Lists
In this paper we propose metrics unexpectedness and unexpectedness_r for measuring the serendipity of recommendation lists produced by recommender systems. Recommender systems have been evaluated in many ways. Although prediction quality is frequently measured by various accuracy metrics, recommender systems must be not only accurate but also useful. A few researchers have argued that the bottom-line measure of the success of a recommender system should be user satisfaction. The basic idea of our metrics is that unexpectedness is the distance between the results produced by the method to be evaluated and those produced by a primitive prediction method. Here, unexpectedness is a metric for a whole recommendation list, while unexpectedness_r is that taking into account the ranking in the list. From the viewpoints of both accuracy and serendipity, we evaluated the results obtained by three prediction methods in experimental studies on television program recommendations.
KeywordsPrediction Method Recommender System User Satisfaction Collaborative Filter Recommendation List
Unable to display preview. Download preview PDF.
- [Breese 98]Breese, J., Herlocker, J., Kadie, C.: Empirical analysis of predictive algorithms for collaborative filtering. In: UAI 1998. Proc. of the 14th Conference on Uncertainty in Artificial Intelligence, pp. 43–52 (1998)Google Scholar
- [Cleverdon 68]Cleverdon, C., Kean, M.: Factors Determining the Performance of Indexing Systems. In: Aslib Cranfield Research Project, Cranfield, England (1968)Google Scholar
- [Billsus 98]Billsus, D., Pazzani, M.: Learning collaborative information filters. In: Proc. of the 15th National Conference on Artificial Intelligence(AAAI), pp. 46–53 (1998)Google Scholar
- [Sarwar 00]Sarwar, B., et al.: Analysis of recommendation algorithms for E-commerce. In: EC 2000. Proc. of the 2nd ACM Conference on Electronic Commerce, pp. 285–295 (2000)Google Scholar
- [Shardanand 95]Shardanand, U., Maes, P.: Social Information Filtering: Algorithms for Automating ”Word of Mouth”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(ACM SIGCHI), pp. 210–217. ACM Press, New York (1995)Google Scholar
- [Swets 69]
- [Swearingen 01]Swearingen, K., Sinha, R.: Beyond Algorithms: An HCI Perspective on Recommender Systems. In: ACM SIGIR Workshop on Recommender Systems (2001)Google Scholar
- [Herlocker 04]
- [Ziegler 05]Ziegler, C.-N, et al.: Improving Recommendation Lists Through Topic Diversification. In: Proc. of WWW 2005, pp. 22–32 (2005)Google Scholar
- [Graham 02]Graham, P.: A plan for spam (August 2002), http://www.paulgraham.com/spam.html