Skip to main content
SpringerLink
Log in

Beyond accuracy measures: the effect of diversity, novelty and serendipity in recommender systems on user engagement

  • Published:
Electronic Commerce Research Aims and scope Submit manuscript

Abstract

The quality of recommender systems (RS) is typically measured by their predictive accuracy. There is an emerging understanding that RS must provide not just accuracy, but also usefulness and enhanced user engagement, where diversity, novelty, and serendipity have been identified as the most common quality features to improve the RS beyond accuracy measures. This research investigates how diversity, novelty and serendipity of the recommended items as well as user’s prosumer behavior affect user engagement dynamically. We formulate a dynamic panel data model using the data collected from NetEase Cloud Music, one of China’s largest music streaming platforms. The findings indicate that both novelty and serendipity of the recommended items have positive impact on user engagement while a more diversified recommendation list could hurt user engagement. Our findings also suggest being a prosumer who also creates videos instead of a pure consumer of music videos will make the user more engaged with the platform in the long run. In addition, our findings clarify the relationship between prosumer behavior and the impact of diversity, novelty and serendipity on user engagement. Being a prosumer alters the effect of diversity on user engagement from negative to positive. Also, creators are drawn to unpopular and unexpected videos as they serve as a source of inspiration for their creative endeavors. The findings of this study have substantial implications for music streaming platforms and other social media and e-commerce platforms to leverage long-term customer engagement through the improvement of recommender systems. For example, a targeted 90-2-20 rule can be implemented to balance the diversity, novelty and serendipity of the recommended items, which prioritizes the selection of 90% of recommended items from the user’s top 2 preferred genres, the remaining 10% from unrecommended genres, and includes 20% of unpopular items within each genre. To encourage the users to create contents, various means can be applied by the platforms such as bestowing a creator badge, offering reward cashback and subscription discounts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Adamopoulos, P., & Tuzhilin, A. (2015). On unexpectedness in recommender systems. ACM Transactions on Intelligent Systems and Technology, 5(4), 1–32. https://doi.org/10.1145/2559952

    Article  Google Scholar 

  2. Aivazoglou, M., Roussos, A. O., Margaris, D., Vassilakis, C., Ioannidis, S., Polakis, J., & Spiliotopoulos, D. (2020). A fine-grained social network recommender system. Social Network Analysis and Mining, 10(1), 8. https://doi.org/10.1007/s13278-019-0621-7

    Article  Google Scholar 

  3. Baier, D., & Stüber, E. (2010). Acceptance of recommendations to buy in online retailing. Journal of Retailing and Consumer Services, 17, 173–180. https://doi.org/10.1016/j.jretconser.2010.03.005

    Article  Google Scholar 

  4. Brodie, R. J., Hollebeek, L. D., Jurić, B., & Ilić, A. (2011). Customer engagement: conceptual domain, fundamental propositions, and implications for research. Journal of Service Research, 14(3), 252–271. https://doi.org/10.1177/1094670511411703

    Article  Google Scholar 

  5. Brodie, R. J., Ilic, A., Juric, B., & Hollebeek, L. (2013). Consumer engagement in a virtual brand community: An exploratory analysis. Journal of Business Research, 66(1), 105–114. https://doi.org/10.1016/j.jbusres.2011.07.029

    Article  Google Scholar 

  6. Calder, B. J., Isaac, M. S., & Malthouse, E. C. (2016). How to capture consumer experiences: A context-specific approach to measuring engagement. Journal of Advertising Research, 56(1), 39–52. https://doi.org/10.2501/JAR-2015-028

    Article  Google Scholar 

  7. Castells, Pablo, Hurley, Neil J., & Vargas, Saul. (2015). Novelty and diversity in recommender systems. Recommender systems handbook (pp. 881–918). Springer, US.

    Chapter  Google Scholar 

  8. Chen, A., Yaobin, Lu., Wang, B., Zhao, L., & Li, M. (2013). What drives content creation behavior on SNSs? A commitment perspective. Journal of Business Research, 66(12), 2529–2535. https://doi.org/10.1016/j.jbusres.2013.05.045

    Article  Google Scholar 

  9. Cheng, P., Wang, S., Ma, J., Sun, J., & Xiong, H. (2017). Learning to recommend accurate and diverse items. In Proceedings of the 26th international conference on world wide web (pp. 183–192). ACM. https://doi.org/10.1145/3038912.3052585

  10. Choi, K., & Suh, Y. (2013). A new similarity function for selecting neighbors for each target item in collaborative filtering. Knowledge-Based Systems, 37, 146–153. https://doi.org/10.1016/j.knosys.2012.07.019

    Article  Google Scholar 

  11. Choi, S.-M., Ko, S.-K., & Han, Y.-S. (2012). A movie recommendation algorithm based on genre correlations. Expert Systems with Applications, 39(9), 8079–8085. https://doi.org/10.1016/j.eswa.2012.01.132

    Article  Google Scholar 

  12. van Doorn, J., Lemon, K. N., Mittal, V., Nass, S., Pick, D., Pirner, P., & Verhoef, P. C. (2010). Customer engagement behavior: Theoretical foundations and research directions. Journal of Service Research, 13(3), 253–266. https://doi.org/10.1177/1094670510375599

    Article  Google Scholar 

  13. Eady, G., Nagler, J., Guess, A., Zilinsky, J., & Tucker, J. A. (2019). How Many people live in political bubbles on social media? Evidence from linked survey and twitter data. SAGE Open, 9(1), 215824401983270. https://doi.org/10.1177/2158244019832705

    Article  Google Scholar 

  14. Fleder, D., & Hosanagar, K. (2009). Blockbuster culture’s next rise or fall: The Impact of recommender systems on sales diversity. Management Science, 55(5), 697–712. https://doi.org/10.1287/mnsc.1080.0974

    Article  Google Scholar 

  15. de Gemmis, M., Lops, P., Semeraro, G., & Musto, C. (2015). An investigation on the serendipity problem in recommender systems. Information Processing & Management, 51(5), 695–717. https://doi.org/10.1016/j.ipm.2015.06.008

    Article  Google Scholar 

  16. Giurgiu, L., Bârsan, G., Academia Forrelor Terestre, & Bălcescu, N. (2008). The prosumer-core and consequence of the web 2.0 era.

  17. Gunawardana, A., & Shani, G. (2015). Evaluating recommender systems. In Recommender systems handbook (pp. 265–308). Boston, MA: Springer US.

  18. Hamilton, M., & Singal, R. (2021). Churning While learning: maximizing user engagement in a recommendation system. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3871915

    Article  Google Scholar 

  19. Helberger, N., Karppinen, K., & D’Acunto, L. (2018). Exposure diversity as a design principle for recommender systems. Information, Communication & Society, 21(2), 191–207. https://doi.org/10.1080/1369118X.2016.1271900

    Article  Google Scholar 

  20. Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1), 5–53. https://doi.org/10.1145/963770.963772

    Article  Google Scholar 

  21. Hollebeek, L. D., Glynn, M. S., & Brodie, R. J. (2014). Consumer brand engagement in social media: Conceptualization, scale development and validation. Journal of Interactive Marketing, 28(2), 149–165. https://doi.org/10.1016/j.intmar.2013.12.002

    Article  Google Scholar 

  22. Hollebeek, L. D., Srivastava, R. K., & Chen, T. (2019). S-D logic-informed customer engagement: Integrative framework, revised fundamental propositions, and application to CRM. Journal of the Academy of Marketing Science, 47(1), 161–185. https://doi.org/10.1007/s11747-016-0494-5

    Article  Google Scholar 

  23. Hong, L., & Lalmas, M. (2019). Tutorial on online user engagement: Metrics and optimization. In Companion proceedings of the 2019 World Wide Web conference (pp. 1303–1305). ACM.

  24. Hurley, N., & Zhang, Mi. (2011). Novelty and diversity in top-N recommendation – Analysis and evaluation. ACM Transactions on Internet Technology, 10(4), 1–30. https://doi.org/10.1145/1944339.1944341

    Article  Google Scholar 

  25. Kaminskas, M., & Bridge, D. (2017). Diversity, serendipity, novelty, and coverage. ACM Transactions on Interactive Intelligent Systems, 7(1), 1–42. https://doi.org/10.1145/2926720

    Article  Google Scholar 

  26. Karimi, M., Jannach, D., & Jugovac, M. (2018). News Recommender systems – Survey and roads ahead. Information Processing & Management, 54(6), 1203–1227. https://doi.org/10.1016/j.ipm.2018.04.008

    Article  Google Scholar 

  27. Kim, K.-J., & Ahn, H. (2008). A Recommender system using GA K-means clustering in an online shopping market. Expert Systems with Applications, 34(2), 1200–1209. https://doi.org/10.1016/j.eswa.2006.12.025

    Article  Google Scholar 

  28. Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8), 30–37. https://doi.org/10.1109/MC.2009.263

    Article  Google Scholar 

  29. Kotkov, D., Veijalainen, J., & Wang, S. (2020). How does serendipity affect diversity in recommender systems? A serendipity-oriented greedy algorithm. Computing, 102(2), 393–411. https://doi.org/10.1007/s00607-018-0687-5

    Article  MathSciNet  Google Scholar 

  30. Kotkov, D., Wang, S., & Veijalainen, J. (2016). A survey of serendipity in recommender systems. Knowledge-Based Systems, 111, 180–192. https://doi.org/10.1016/j.knosys.2016.08.014

    Article  Google Scholar 

  31. Lehmann, J., Lalmas, M., Yom-Tov, E., & Dupret, G. (2012). Models of user engagement. In User modeling, adaptation, and personalization (pp. 164–75). Springer.

  32. Li, Y., Ping, Y., Zhong, Y., & Misra, R. (2023). Learning-by-doing in non-homogeneous tasks: An empirical study of content creator performance on a music streaming platform. Electronic Commerce Research and Applications, 58, 101241. https://doi.org/10.1016/j.elerap.2023.101241

    Article  Google Scholar 

  33. Lu, J., Dianshuang, Wu., Mao, M., Wang, W., & Zhang, G. (2015). Recommender system application developments: A survey. Decision Support Systems, 74, 12–32. https://doi.org/10.1016/j.dss.2015.03.008

    Article  Google Scholar 

  34. Lunardi, G. M., Machado, G. M., Maran, V., & de Oliveira., J. P. (2020). A metric for filter bubble measurement in recommender algorithms considering the news domain. Applied Soft Computing, 97, 106771. https://doi.org/10.1016/j.asoc.2020.106771

    Article  Google Scholar 

  35. Lundquist, E. (2020, June 28). Factorization machines for item recommendation with implicit feedback data. Towards Data Science.

  36. Maslowska, E., Malthouse, E. C., & Hollebeek, L. D. (2022). The role of recommender systems in fostering consumers’ long-term platform engagement. Journal of Service Management, 33(4/5), 721–732. https://doi.org/10.1108/JOSM-12-2021-0487

    Article  Google Scholar 

  37. Nakatsuji, M., Fujiwara, Y., Tanaka, A., Uchiyama, T., Fujimura, K., & Ishida, T. (2010). Classical music for rock fans? In Proceedings of the 19th ACM international conference on Information and knowledge management (pp. 949–958). ACM.

  38. Nguyen, T. T., Hui, P.-M., Maxwell Harper, F., Terveen, L., & Konstan, J. A. (2014). Exploring the filter bubble. In Proceedings of the 23rd international conference on World Wide Web (pp. 677–686). ACM.

  39. Pariser, E. (2011). The filter bubble: What the internet is hiding from you. KG: Carl Hanser Verlag GmbH & Co.

    Google Scholar 

  40. Plümper, T., & Troeger, V. E. (2007). Efficient estimation of time-invariant and rarely changing variables in finite sample panel analyses with unit fixed effects. Political Analysis, 15(2), 124–139. https://doi.org/10.1093/pan/mpm002

    Article  Google Scholar 

  41. Preece, J., Nonnecke, B., & Andrews, D. (2004). The Top five reasons for lurking: Improving community experiences for everyone. Computers in Human Behavior, 20(2), 201–223. https://doi.org/10.1016/j.chb.2003.10.015

    Article  Google Scholar 

  42. Ricci, F., Rokach, L., & Shapira, B. (2011). Introduction to recommender systems handbook. In Recommender systems handbook (pp. 1–35). Springer US.

  43. Roodman, D. (2020). Xtabond2: Stata module to extend xtabond dynamic panel data estimator.

  44. Roy, D., & Dutta, M. (2022). A systematic review and research perspective on recommender systems. Journal of Big Data, 9(1), 59. https://doi.org/10.1186/s40537-022-00592-5

    Article  Google Scholar 

  45. Salvador, F., Alba, C., Madiedo, J. P., Tenhiälä, A., & Bendoly, E. (2021). Project managers’ breadth of experience, project complexity, and project performance. Journal of Operations Management, 67(6), 729–754. https://doi.org/10.1002/joom.1140

    Article  Google Scholar 

  46. Schomakers, E.-M., Lidynia, C., & Ziefle, M. (2020). All of me? Users’ Preferences for privacy-preserving data markets and the importance of anonymity. Electronic Markets, 30(3), 649–665. https://doi.org/10.1007/s12525-020-00404-9

    Article  Google Scholar 

  47. Seo, Y.-D., Kim, Y.-G., Lee, E., & Kim, H. (2021). Group recommender system based on genre preference focusing on reducing the clustering cost. Expert Systems with Applications, 183, 115396. https://doi.org/10.1016/j.eswa.2021.115396

    Article  Google Scholar 

  48. Silveira, T., Zhang, M., Lin, X., Liu, Y., & Ma, S. (2019). How Good your recommender system is? A survey on evaluations in recommendation. International Journal of Machine Learning and Cybernetics, 10(5), 813–831. https://doi.org/10.1007/s13042-017-0762-9

    Article  Google Scholar 

  49. Srivastava, A., Bala, P. K., & Kumar, B. (2020). New perspectives on gray sheep behavior in E-commerce recommendations. Journal of Retailing and Consumer Services 53. https://doi.org/10.1016/j.jretconser.2019.02.018

  50. Sun, J., Song, J., Jiang, Y., Liu, Y., & Li, J. (2022). Prick the filter bubble: A novel cross domain recommendation model with adaptive diversity regularization. Electronic Markets, 32(1), 101–121. https://doi.org/10.1007/s12525-021-00492-1

    Article  Google Scholar 

  51. Toffler, A. (1980). The Third wave. William Morrow & Company Inc.

    Google Scholar 

  52. Wu, W., Chen, L., & Zhao, Y. (2018). Personalizing recommendation diversity based on user personality. User Modeling & User-Adapted Interaction, 28(3), 237–276. https://doi.org/10.1007/s11257-018-9205-x

  53. Yoganarasimhan, H. (2012). Impact of social network structure on content propagation: A study using YouTube data. Quantitative Marketing and Economics, 10(1), 111–150. https://doi.org/10.1007/s11129-011-9105-4

  54. Ziemba, W., Ewa, M. E., Mullins, R., & Dettmer, S. (2019). Prosumers’ engagement in business process innovation – The case of Poland and the UK. Interdisciplinary Journal of Information, Knowledge, and Management, 14, 119–143. https://doi.org/10.28945/4320

    Article  Google Scholar 

  55. Zhang, D. J., Ming, Hu., Liu, X., Yuxiang, Wu., & Li, Y. (2022). NetEase cloud music data. Manufacturing & Service Operations Management, 24(1), 275–284. https://doi.org/10.1287/msom.2020.0923

    Article  Google Scholar 

  56. Zhang, Y. C., Séaghdha, D. Ó., Quercia, D., Jambor, T. (2012). Auralist: Introducing serendipity into music recommendation. In Proceedings of the fifth ACM international conference on web search and data mining (pp 13–22). ACM.

  57. Zheng, Q., Chan, C.-K., & Ip, H. H. S. (2015). An unexpectedness-augmented utility model for making serendipitous recommendation. In Advances in data mining: Applications and theoretical aspects (pp. 216–30). Springer.

  58. Zhou, T., Kuscsik, Z., Liu, J.-G., Medo, M., Wakeling, J. R., & Zhang, Y.-C. (2010). Solving the apparent diversity-accuracy dilemma of recommender systems. Proceedings of the National Academy of Sciences, 107(10), 4511–4515. https://doi.org/10.1073/pnas.1000488107

    Article  ADS  Google Scholar 

  59. Ziegler, C.-N., McNee, S. M., Konstan, J. A., & Lausen, G. (2005). Improving recommendation lists through topic diversification. In Proceedings of the 14th international conference on World Wide Web – WWW ’05 (p. 22). ACM Press.

  60. Zou, L., Xia, L., Ding, Z., Song, J., Liu, W., & Yin, D. (2019). Reinforcement learning to optimize long-term user engagement in recommender systems. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2810–2818). New York, NY, USA, ACM.

Download references

Author information

Authors and Affiliations

  1. Department of Business Analytics and Information Systems, The Peter J. Tobin College of Business, St. John’s University, Queens, NY, 11439, USA

    Yanni Ping

  2. Department of Information Management and Business Analytics, Feliciano School of Business, Montclair State University, Montclair, NJ, 07043, USA

    Yang Li

  3. Prudential Financial, 213 Washington St, Newark, NJ, 07102, USA

    Jiaxin Zhu

Authors
  1. Yanni Ping
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiaxin Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanni Ping.

Ethics declarations

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ping, Y., Li, Y. & Zhu, J. Beyond accuracy measures: the effect of diversity, novelty and serendipity in recommender systems on user engagement. Electron Commer Res (2024). https://doi.org/10.1007/s10660-024-09813-w

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10660-024-09813-w

Keywords

Search

Navigation