Skip to main content
SpringerLink
Log in

Revealing the Hidden Impact of Top-N Metrics on Optimization in Recommender Systems

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14608))

Included in the following conference series:

  • 413 Accesses

Abstract

The hyperparameters of recommender systems for top-n predictions are typically optimized to enhance the predictive performance of algorithms. Thereby, the optimization algorithm, e.g., grid search or random search, searches for the best hyperparameter configuration according to an optimization-target metric, like nDCG or Precision. In contrast, the optimized algorithm, e.g., Alternating Least Squares Matrix Factorization or Bayesian Personalized Ranking, internally optimizes a different loss function during training, like squared error or cross-entropy. To tackle this discrepancy, recent work focused on generating loss functions better suited for recommender systems. Yet, when evaluating an algorithm using a top-n metric during optimization, another discrepancy between the optimization-target metric and the training loss has so far been ignored. During optimization, the top-n items are selected for computing a top-n metric; ignoring that the top-n items are selected from the recommendations of a model trained with an entirely different loss function. Item recommendations suitable for optimization-target metrics could be outside the top-n recommended items; hiddenly impacting the optimization performance. Therefore, we were motivated to analyze whether the top-n items are optimal for optimization-target top-n metrics. In pursuit of an answer, we exhaustively evaluate the predictive performance of 250 selection strategies besides selecting the top-n. We extensively evaluate each selection strategy over twelve implicit feedback and eight explicit feedback data sets with eleven recommender systems algorithms. Our results show that there exist selection strategies other than top-n that increase predictive performance for various algorithms and recommendation domains. However, the performance of the top \(\sim 43\%\) of selection strategies is not significantly different. We discuss the impact of our findings on optimization and re-ranking in recommender systems and feasible solutions. The implementation of our study is publicly available.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://code.isg.beel.org/scoring-optimizer.

  2. 2.

    https://rees46.com/.

  3. 3.

    https://www.kaggle.com/datasets/retailrocket/ecommerce-dataset.

  4. 4.

    https://github.com/EthanRosenthal/rec-a-sketch.

  5. 5.

    https://www.yelp.com/dataset.

  6. 6.

    https://www.kaggle.com/datasets/chadgostopp/recsys-challenge-2015.

  7. 7.

    https://guoguibing.github.io/librec/datasets.html.

References

  1. Abdollahpouri, H., Burke, R., Mobasher, B.: Managing popularity bias in recommender systems with personalized re-ranking. arXiv preprint arXiv:1901.07555 (2019)

  2. Anand, R., Beel, J.: Auto-surprise: an automated recommender-system (autorecsys) library with tree of parzens estimator (TPE) optimization. In: Proceedings of the 14th ACM Conference on Recommender Systems (RecSys 2020), pp. 585–587. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3383313.3411467

  3. Anelli, V.W., et al.: Elliot: a comprehensive and rigorous framework for reproducible recommender systems evaluation. In: Diaz, F., Shah, C., Suel, T., Castells, P., Jones, R., Sakai, T. (eds.) The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event (SIGIR 2021), Canada, 11–15 July 2021, pp. 2405–2414. ACM (2021). https://doi.org/10.1145/3404835.3463245

  4. Barkan, O., Hirsch, R., Katz, O., Caciularu, A., Koenigstein, N.: Anchor-based collaborative filtering. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM 2021), pp. 2877–2881. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3459637.3482056

  5. Beel, J., Dinesh, S.: Real-world recommender systems for academia: the pain and gain in building, operating, and researching them. In: Mayr, P., Frommholz, I., Cabanac, G. (eds.) Proceedings of the Fifth Workshop on Bibliometric-enhanced Information Retrieval (BIR) Co-located with the 39th European Conference on Information Retrieval (ECIR 2017), Aberdeen, UK, 9th April 2017. CEUR Workshop Proceedings, vol. 1823, pp. 6–17. CEUR-WS.org (2017). https://ceur-ws.org/Vol-1823/paper1.pdf

  6. Bruch, S., Wang, X., Bendersky, M., Najork, M.: An analysis of the softmax cross entropy loss for learning-to-rank with binary relevance. In: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR 2019), pp. 75–78. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3341981.3344221

  7. Cantador, I., Brusilovsky, P., Kuflik, T.: Second workshop on information heterogeneity and fusion in recommender systems (hetrec2011). In: Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys 2011), pp. 387–388. Association for Computing Machinery, New York (2011). https://doi.org/10.1145/2043932.2044016

  8. Chen, B., Zhao, X., Wang, Y., Fan, W., Guo, H., Tang, R.: A comprehensive survey on automated machine learning for recommendations. ACM Trans. Recomm. Syst. (2023). https://doi.org/10.1145/3630104

  9. Chen, H., et al.: Denoising self-attentive sequential recommendation. In: Proceedings of the 16th ACM Conference on Recommender Systems (RecSys 2022). pp. 92–101. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3523227.3546788

  10. Cho, E., Myers, S.A., Leskovec, J.: Friendship and mobility: user movement in location-based social networks. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2011), pp. 1082–1090. Association for Computing Machinery, New York (2011). https://doi.org/10.1145/2020408.2020579

  11. Dooms, S., De Pessemier, T., Martens, L.: MovieTweetings: a movie rating dataset collected from twitter. In: Workshop on Crowdsourcing and Human Computation for Recommender Systems, Held in Conjunction with the 7th ACM Conference on Recommender Systems, p. 2 (2013)

    Google Scholar 

  12. Ekstrand, M.D.: Lenskit for python: next-generation software for recommender systems experiments. In: Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM 2020), pp. 2999–3006. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3340531.3412778

  13. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc. (2015)

    Google Scholar 

  14. Frederickson, B.: Fast python collaborative filtering for implicit datasets (2018). https://githubcom/benfred/implicit

    Google Scholar 

  15. Ge, Y., et al.: Understanding echo chambers in e-commerce recommender systems. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), pp. 2261–2270. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3397271.3401431

  16. Goldberg, K., Roeder, T., Gupta, D., Perkins, C.: Eigentaste: a constant time collaborative filtering algorithm. Inf. Retr. 4(2), 133–151 (2001). https://doi.org/10.1023/A:1011419012209

  17. Gulla, J.A., Zhang, L., Liu, P., Özgöbek, O., Su, X.: The adressa dataset for news recommendation. In: Proceedings of the International Conference on Web Intelligence (WI 2017), pp. 1042–1048. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3106426.3109436

  18. Harper, F.M., Konstan, J.A.: The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5(4), 1–19 (2015). https://doi.org/10.1145/2827872

  19. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). https://doi.org/10.1145/963770.963772

  20. Hernández del Olmo, F., Gaudioso, E.: Evaluation of recommender systems: a new approach. Exp. Syst. Appl. 35(3), 790–804 (2008). https://doi.org/10.1016/j.eswa.2007.07.047

  21. Jankiewicz, P., Kyrashchuk, L., Sienkowski, P., Wójcik, M.: Boosting algorithms for a session-based, context-aware recommender system in an online travel domain. In: Proceedings of the Workshop on ACM Recommender Systems Challenge (RecSys Challenge 2019). Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3359555.3359557

  22. Jannach, D., Adomavicius, G.: Price and profit awareness in recommender systems. arXiv preprint arXiv:1707.08029 (2017)

  23. Joglekar, M.R., et al.: Neural input search for large scale recommendation models. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2020), pp. 2387–2397. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3394486.3403288

  24. Langer, S., Beel, J.: Apache lucene as content-based-filtering recommender system: 3 lessons learned. In: Mayr, P., Frommholz, I., Cabanac, G. (eds.) Proceedings of the Fifth Workshop on Bibliometric-enhanced Information Retrieval (BIR) co-located with the 39th European Conference on Information Retrieval (ECIR 2017), Aberdeen, 9 April 2017. CEUR Workshop Proceedings, vol. 1823, pp. 85–92. CEUR-WS.org (2017). https://ceur-ws.org/Vol-1823/paper8.pdf

  25. Li, Z., Ji, J., Ge, Y., Zhang, Y.: Autolossgen: automatic loss function generation for recommender systems. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022), pp. 1304–1315. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3477495.3531941

  26. Liang, D., Krishnan, R.G., Hoffman, M.D., Jebara, T.: Variational autoencoders for collaborative filtering. In: Proceedings of the 2018 World Wide Web Conference (WWW 2018), pp. 689–698. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE (2018). https://doi.org/10.1145/3178876.3186150

  27. Liu, H., Zhao, X., Wang, C., Liu, X., Tang, J.: Automated embedding size search in deep recommender systems. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), pp. 2307–2316. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3397271.3401436

  28. Liu, W., et al.: Neural re-ranking in multi-stage recommender systems: a review. In: Raedt, L.D. (ed.) Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22), pp. 5512–5520. International Joint Conferences on Artificial Intelligence Organization (2022). https://doi.org/10.24963/ijcai.2022/771

  29. Melchiorre, A.B., Rekabsaz, N., Ganhör, C., Schedl, M.: Protomf: prototype-based matrix factorization for effective and explainable recommendations. In: Proceedings of the 16th ACM Conference on Recommender Systems (RecSys 2022), pp. 246–256. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3523227.3546756

  30. Michiels, L., Verachtert, R., Goethals, B.: Recpack: an(other) experimentation toolkit for top-n recommendation using implicit feedback data. In: Proceedings of the 16th ACM Conference on Recommender Systems (RecSys 2022). pp. 648–651. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3523227.3551472

  31. Moreira, G.D.S.P., Jannach, D., Cunha, A.M.D.: Contextual hybrid session-based news recommendation with recurrent neural networks. IEEE Access 7, 169185–169203 (2019)

    Google Scholar 

  32. Ni, J., Li, J., McAuley, J.: Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 188–197. Association for Computational Linguistics, Hong Kong (2019). https://doi.org/10.18653/v1/D19-1018

  33. Pang, L., Ai, Q., Xu, J.: Beyond probability ranking principle: modeling the dependencies among documents. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM 2021), pp. 1137–1140. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3437963.3441662

  34. Pei, C., et al.: Personalized re-ranking for recommendation. In: Proceedings of the 13th ACM Conference on Recommender Systems (RecSys 2019), pp. 3–11. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3298689.3347000

  35. Pichl, M., Zangerle, E., Specht, G.: Towards a context-aware music recommendation approach: what is hidden in the playlist name? In: 2015 IEEE International Conference on Data Mining Workshop (ICDMW), pp. 1360–1365 (2015). https://doi.org/10.1109/ICDMW.2015.145

  36. Poddar, A., Zangerle, E., Yang, Y.H.: #nowplaying-rs: a new benchmark dataset for building context-aware music recommender systems. In: Proceedings of the 15th Sound and Music Computing Conference. Limassol, Cyprus (2018). https://mac.citi.sinica.edu.tw/~yang/pub/poddar18smc.pdf, code at https://github.com/asmitapoddar/nowplaying-RS-Music-Reco-FM

  37. Portugal, I., Alencar, P., Cowan, D.: The use of machine learning algorithms in recommender systems: a systematic review. Expert Syst. Appl. 97, 205–227 (2018)

    Google Scholar 

  38. Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI 2009), pp. 452–461. AUAI Press, Arlington (2009)

    Google Scholar 

  39. Rendle, S., Krichene, W., Zhang, L., Koren, Y.: Revisiting the performance of ials on item recommendation benchmarks. In: Proceedings of the 16th ACM Conference on Recommender Systems (RecSys 2022), pp. 427–435. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3523227.3548486

  40. Robertson, S.: The probability ranking principle in IR. J. Document. 33(4), 294–304 (1977). https://doi.org/10.1108/eb026647

  41. de Souza Pereira Moreira, G., Ferreira, F., da Cunha, A.M.: News session-based recommendations using deep neural networks. In: Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems. ACM (2018). https://doi.org/10.1145/3270323.3270328

  42. Sun, B., Wu, D., Shang, M., He, Y.: Toward auto-learning hyperparameters for deep learning-based recommender systems. In: Bhattacharya, A., et al. (eds.) Database Systems for Advanced Applications, pp. 323–331. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-00126-0_25

  43. Sun, F., et al.: Bert4rec: sequential recommendation with bidirectional encoder representations from transformer. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM 2019), pp. 1441–1450. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3357384.3357895

  44. Tang, H., Liu, J., Zhao, M., Gong, X.: Progressive layered extraction (PLE): a novel multi-task learning (MTL) model for personalized recommendations. In: Proceedings of the 14th ACM Conference on Recommender Systems (RecSys 2020), pp. 269–278. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3383313.3412236

  45. Vente, T., Ekstrand, M., Beel, J.: Introducing lenskit-auto, an experimental automated recommender system (autorecsys) toolkit. In: Proceedings of the 17th ACM Conference on Recommender Systems (RecSys 2023), pp. 1212–1216. Association for Computing Machinery, New York (2023). https://doi.org/10.1145/3604915.3610656

  46. Wang, H., Chen, B., Li, W.J.: Collaborative topic regression with social regularization for tag recommendation. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence (IJCAI 2013), pp. 2719–2725. AAAI Press (2013)

    Google Scholar 

  47. Wang, X., Li, C., Golbandi, N., Bendersky, M., Najork, M.: The lambdaloss framework for ranking metric optimization. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM 2018), pp. 1313–1322. Association for Computing Machinery, New York (2018)

    Google Scholar 

  48. Weston, J., Yee, H., Weiss, R.J.: Learning to rank recommendations with the k-order statistic loss. In: Proceedings of the 7th ACM Conference on Recommender Systems (RecSys 2013), pp. 245–248. Association for Computing Machinery, New York (2013). https://doi.org/10.1145/2507157.2507210

  49. Xu, L., et al.: Recent advances in RecBole: extensions with more practical considerations. arXiv preprints arXiv:2211.15148 (2022)

  50. Yang, C., Hou, Y., Song, Y., Zhang, T., Wen, J.R., Zhao, W.X.: Modeling two-way selection preference for person-job fit. In: Proceedings of the 16th ACM Conference on Recommender Systems (RecSys 2022), pp. 102–112. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3523227.3546752

  51. Yue, Z., He, Z., Zeng, H., McAuley, J.: black-box attacks on sequential recommenders via data-free model extraction. In: Proceedings of the 15th ACM Conference on Recommender Systems (RecSys 2021), pp. 44–54. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3460231.3474275

  52. Yue, Z., Zeng, H., Kou, Z., Shang, L., Wang, D.: Defending substitution-based profile pollution attacks on sequential recommenders. In: Proceedings of the 16th ACM Conference on Recommender Systems (RecSys 2022), pp. 59–70. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3523227.3546770

  53. Zangerle, E., Bauer, C.: Evaluating recommender systems: survey and framework. ACM Comput. Surv. 55(8) (2022). https://doi.org/10.1145/3556536

  54. Zhang, R., Bao, H., Sun, H., Wang, Y., Liu, X.: Recommender systems based on ranking performance optimization. Front. Comp. Sci. 10(2), 270–280 (2015)

    Google Scholar 

  55. Zhao, W.X., et al.: Recbole 2.0: towards a more up-to-date recommendation library. In: Proceedings of the 31st ACM International Conference on Information and Knowledge Management, pp. 4722–4726 (2022)

    Google Scholar 

  56. Zhao, W.X., et al.: Recbole: towards a unified, comprehensive and efficient framework for recommendation algorithms. In: CIKM, pp. 4653–4664. ACM (2021)

    Google Scholar 

  57. Zheng, R., Qu, L., Cui, B., Shi, Y., Yin, H.: Automl for deep recommender systems: a survey. ACM Trans. Inf. Syst. 41(4) (2023). https://doi.org/10.1145/3579355

Download references

Acknowledgement

The OMNI cluster of the University of Siegen was used to compute the results presented in this paper.

Author information

Authors and Affiliations

  1. Intelligent Systems Group, University of Siegen, Siegen, Germany

    Lukas Wegmeth, Tobias Vente & Lennart Purucker

Authors
  1. Lukas Wegmeth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tobias Vente
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lennart Purucker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lukas Wegmeth .

Editor information

Editors and Affiliations

  1. Georgetown University, Washington, WA, USA

    Nazli Goharian

  2. University of Pisa, Pisa, Italy

    Nicola Tonellotto

  3. King's College London, London, UK

    Yulan He

  4. University College London, London, UK

    Aldo Lipani

  5. University of Glasgow, Glasgow, UK

    Graham McDonald

  6. University of Glasgow, Glasgow, UK

    Craig Macdonald

  7. University of Glasgow, Glasgow, UK

    Iadh Ounis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wegmeth, L., Vente, T., Purucker, L. (2024). Revealing the Hidden Impact of Top-N Metrics on Optimization in Recommender Systems. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14608. Springer, Cham. https://doi.org/10.1007/978-3-031-56027-9_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56027-9_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56026-2

  • Online ISBN: 978-3-031-56027-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation