Skip to main content
SpringerLink
Log in

Event-Based Data Pipelines in Recommender Systems: The Data Engineering Perspective

  • Conference paper
  • First Online:
Emerging Technologies in Computing (iCETiC 2023)

Included in the following conference series:

  • 123 Accesses

Abstract

Recommender Systems (RS) are information retrieval systems that can be used for serving personalized content to online users. Most industrial recommendation systems utilize a large amount of online data to generate personalized recommendations for users. The quality of the data plays an important role in the performance of the RS. The majority of the RS data is generated from event data that are stored in data lakes through multiple data pipelines. Event-based data pipelines have emerged as a popular approach to handle the massive amount of data generated by modern applications. In this paper, we explore the impact of event-based data pipelines on recommendation systems. We discuss how these pipelines enable efficient data ingestion, real-time processing, and low-latency recommendations.

D. Reddy, U. Sinha, and R. S. Rajput—Contributed equally to this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005). https://doi.org/10.1109/TKDE.2005.99

    Article  Google Scholar 

  2. Afchar, A., Epure, E.V., Mille, A., Moussallam, M.: Explainability in music recommender systems. arXiv preprint arXiv:2201.10528 (2022)

  3. Ahmad, S., Lavin, A., Purdy, S., Agha, Z.: Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262, 134–147 (2017)

    Article  Google Scholar 

  4. Bobadilla, J., Ortega, F., Hernando, A., Gutiérrez, A.: Recommender systems survey. Knowl.-Based Syst. 46, 109–132 (2013)

    Article  Google Scholar 

  5. Bouneffouf, D., Rish, I., Aggarwal, C.: Survey on applications of multi-armed and contextual bandits. IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2020). https://doi.org/10.1109/CEC48606.2020.9185782

  6. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: Stream and batch processing in a single engine. Bull. Tech. Committee Data Eng. 38(4), 13–26 (2015)

    Google Scholar 

  7. Chen, J., Gao, Z.: A comprehensive pipeline for hotel recommendation system. arXiv preprint arXiv:2009.01860 (2020)

  8. Choi, S.M., Lee, D., Jang, K., Park, C., Lee, S.: Improving data sparsity in recommender systems using matrix regeneration with item features. Mathematics 11(2), 292 (2023). https://doi.org/10.3390/math11020292

    Article  Google Scholar 

  9. Covington, P., Adams, J., Sargin, E.: Deep neural networks for youtube recommendations. In: Proceedings of the 10th ACM Conference on Recommender Systems, pp. 191–198 (2016)

    Google Scholar 

  10. Deng, Y.: Recommender systems based on graph embedding techniques: a comprehensive review. arXiv preprint arXiv:2109.09587 (2021)

  11. Gautham, S., Bakirtzis, G., Will, A., Jayakumar, A.V., Elks, C.R.: Stpadriven multilevel runtime monitoring for in-time hazard detection. arXiv preprint arXiv:2204.08999 (2022)

  12. Gomes-Ju´nior, A.R., Santana, T., Winter, O.C., Sfair, R.: The main perturbing objects on the orbits of (616) prometheus and (617) pandora. arXiv preprint arXiv:2202.01617 (2022)

  13. Kohar, M., Rana, C.: Survey paper on recommendation system. Int. J. Comput. Sci. Inf. Technol. 3(2), 3460–3462 (2012)

    Google Scholar 

  14. Kumar, B., Sharma, N.: Approaches, issues and challenges in recommender systems: a systematic review. Indian J. Sci. Tech. 9(47), 94892 (2016). https://doi.org/10.17485/ijst/2016/v9i47/94892

    Article  Google Scholar 

  15. Kunaver, M., Požrl, T.: Diversity in recommender systems a survey. Knowl.-Based Syst. 123, 154–162 (2017). https://doi.org/10.1016/j.knosys.2017.02.009, https://www.sciencedirect.com/science/article/pii/S0950705117300680

  16. Lika, B., Kolomvatsos, K., Hadjiefthymiades, S.: Facing the cold start problem in recommender systems. Expert Syst. Appl.: Int. J. 41, 2065–2073 (2014). https://doi.org/10.1016/j.eswa.2013.09.005

    Article  Google Scholar 

  17. Mazaheri, M., Kiar, G., Glatard, T.: A recommender system for scientific datasets and analysis pipelines. arXiv preprint arXiv:2108.09275 (2021)

  18. Mishra, N., et al.: Research problems in recommender systems. J. Phys.: Conf. Ser. 1717(1), 012002 (2021). https://doi.org/10.1088/1742-6596/1717/1/012002

    Article  Google Scholar 

  19. Mohamed, M.H., Khafagy, M.H., Ibrahim, M.H.: Recommender systems challenges and solutions. In: 2019 International Conference on Innovative Trends in Computer Engineering (ITCE’2019), pp. 1–6 (2019)

    Google Scholar 

  20. Ngoc, T.V., Thi, H.T.: Techniques, benefits, and challenges of recommendation system in e-commerce: a literature review. In: Proceedings of the International Conference on Industrial Engineering and Operations Management, pp. 107–114 (2021)

    Google Scholar 

  21. Ponnuswami, G., Kailasam, S., Dinesh, D.A.: Event-driven data pipeline for network management systems. In: 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–6 (2020). https://doi.org/10.1109/ICCCNT49239.2020.9225344

  22. Roa, J., Farnocchia, D., Chesley, S.R.: A novel approach to asteroid impact monitoring and hazard assessment. arXiv preprint arXiv:2108.03201 (2021)

  23. Roy, D., Dutta, M.: A systematic review and research perspective on recommender systems. J. Big Data 9(1), 59 (2022). https://doi.org/10.1186/s40537-022-00414-2

    Article  Google Scholar 

  24. Sharma, L., Gera, A.: A survey of recommendation system: research challenges. Int. J. Eng. Trends Technol. 4(5), 1989–1995 (2013)

    Google Scholar 

  25. Tagliabue, J.: You do not need a bigger boat recommendations at reasonable scale in a (mostly) serverless and openstack. In: Woodstock ‘18: ACM Symposium on Neural Gaze Detection, p. 6. ACM (2018)

    Google Scholar 

  26. Tiwalola, A.B., Asafe, Y.N.: A comprehensive study of recommender systems: prospects and challenges. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 9(5), 224–230 (2019)

    Google Scholar 

  27. Vrijenhoek, S.: Do you mind? Reflections on the mind dataset for research on diversity in new recommendations. arXiv preprint arXiv:2304.08253 (2023)

  28. Wang, H.: Dotmat: solving cold-start problem and alleviating sparsity problem for recommender systems. arXiv preprint arXiv:2303.14419 (2023)

  29. Wang, H.: Evolution of the online rating platform data structures and its implications for recommender systems. arXiv preprint arXiv:2303.14419 (2023)

  30. Wang, Y., et al.: Experimental comparison of various techniques for spot size measurement of high-energy x-ray source. arXiv preprint arXiv:1511.07668 (2015)

  31. Wu, F., et al.: Mind: a large-scale dataset for news recommendation. In: Proceedings of the Association for Computational Linguistics (ACL) (2020)

    Google Scholar 

  32. Zhang, M., Jansen, B.J.: The effect of missing click-through data on click through rate estimation in sponsored search. Inf. Process. Manage. 47(4), 671–688 (2011)

    Google Scholar 

Download references

Acknowledgements

While the original ideas, study, findings, and interpretations expressed in this paper are our own, the clarity of the presentation in specific sub-sections was achieved with the assistance of ChatGPT which helped us in enhancing the readability of the paper.

Author information

Authors and Affiliations

  1. University of Connecticut, Storrs, Connecticut, USA

    Deexith Reddy

  2. Iowa State University, Ames, Iowa, USA

    Urjoshi Sinha

  3. Headspace, Los Angeles, California, USA

    Rohan Singh Rajput

Authors
  1. Deexith Reddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Urjoshi Sinha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rohan Singh Rajput
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deexith Reddy .

Editor information

Editors and Affiliations

  1. Xiamen University Malaysia, Selangor, Malaysia

    Mahdi H. Miraz

  2. University of Chester, Chester, UK

    Garfield Southall

  3. Universiteti Metropolitan Tirana, Tirana, Albania

    Maaruf Ali

  4. University of South Wales, Pontypridd, UK

    Andrew Ware

Rights and permissions

Reprints and permissions

Copyright information

© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Reddy, D., Sinha, U., Rajput, R.S. (2024). Event-Based Data Pipelines in Recommender Systems: The Data Engineering Perspective. In: Miraz, M.H., Southall, G., Ali, M., Ware, A. (eds) Emerging Technologies in Computing. iCETiC 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 538. Springer, Cham. https://doi.org/10.1007/978-3-031-50215-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50215-6_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50214-9

  • Online ISBN: 978-3-031-50215-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation