Skip to main content
SpringerLink
Log in

Session-aware recommender system using double deep reinforcement learning

  • Research
  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Session-aware recommender systems capture user-specific preferences that emerge within multiple user sessions by leveraging the sequential nature of user interactions. Existing session-aware recommendation methods face challenges in finding the right balance between exploration and exploitation leading to less diverse recommendations and also suffering from overestimation bias. This bias problem refers to the tendency for value estimates to be higher than their true values resulting in slower convergence, suboptimal, and less diverse recommendations. This paper proposes a Double Deep Q-network based session-aware recommender system, DDQN-SaRS, which takes care of the overestimation bias and generates diverse recommendations capturing the user’s dynamic interests. The proposed system works in two phases. The first phase generates embedding for users, items, and sessions using a Graph Convolutional Network (GCN). The obtained embedding vectors are then given to Double Deep Q-Network (DDQN), a Double Deep Reinforcement Learning (DDRL) technique for suggesting items of interest to the user(s) in the second phase. DDQN decouples the task of action selection and action evaluation by utilizing two networks viz. main and target networks and resolves the overestimation bias problem while maintaining diversity in recommendations. The proposed system learns recommendation policies and corresponding rewards from a pure offline setting. It is validated on two real-world datasets: Diginetica from the CIKM Cup Challenge 2016 and Retailrocket from the Kaggle competition. Experimental results show that our proposed system, DDQN-SaRS outperformed various baseline algorithms viz. S-POP, Item-KNN, GRU4Rec, STAMP, HRNN, NSAR,IDSR, and GNN-GNF.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Algorithm 3
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of data and materials

No datasets were generated. Datasets used in the current study are publicly available.

Notes

  1. https://competitions.codalab.org/competitions/11161

  2. https://www.kaggle.com/datasets/retailrocket/ecommerce-dataset

References

  • Aggarwal, C. C. (2016). Content-Based Recommender Systems. Recommender Systems: The Textbook (pp. 139–166). Cham: Springer International Publishing.

    Chapter  Google Scholar 

  • Bedi, P., Khurana, P., & Sharma, R. (2021). Session based recommendations using CNN-LSTM with fuzzy time series. In International conference on artificial intelligence and speech technology (pp. 432–446). https://doi.org/10.1007/978-3-030-95711-7_36

  • Cai, Q., Zhao, & Hu, H. (2018). Session-aware music recommendation via a generative model approach. Soft Computing, 22, 1023–1031. https://doi.org/10.1007/s00500-016-2407-4

  • Chen, H. (2021). A DQN-based Recommender system for item-list recommendation. In IEEE International conference on big data (big data) (pp. 5699–5702). https://doi.org/10.1109/BigData52589.2021.9671947

  • Chen, W., Ren, P., & Cai, F., et al. (2020). Improving end-to-end sequential recommendations with intent-aware diversication. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 175–184). https://doi.org/10.1145/3340531.3411897

  • Ding, Z., Huang, Y., & Yuan, H., et. al. (2020). Introduction to reinforcement learning. In Deep reinforcement learning: fundamentals, research and applications (pp. 47–123). Springer Singapore

  • Feng, L., Cai, Y., Wei, E., et al. (2022). Graph neural networks with global noise ltering for session-based recommendation. Neurocomputing, 472, 113–123. https://doi.org/10.1016/j.neucom.2021.11.068

    Article  Google Scholar 

  • Garg, D., Gupta, P., & Malhotra, P., et al. (2020). Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation. arXiv preprint, 1–15. https://doi.org/10.48550/arXiv.2012.08984

  • Girmay, M., Maglogiannis, V., Naudts, D., et al. (2021). Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based QLearning. Sensors, 21(21), 6977. https://doi.org/10.3390/s21216977

    Article  Google Scholar 

  • Hidasi, B., Karatzoglou, A., & Baltrunas, L., et al. (2015). Session-based Recommendations with Recurrent Neural Networks. arXiv preprint, 1–10. https://doi.org/10.48550/arXiv.1511.06939

  • Hu, L., Chen, Q., Zhao, H., et al. (2018). Neural Cross-Session Filtering: Next-Item Prediction Under Intra and Inter-Session Context. IEEE Intelligent Systems, 33(6), 57–67. https://doi.org/10.1109/MIS.2018.2881516

    Article  Google Scholar 

  • Jannach, D., Quadrana, M., & Cremonesi, P. (2022). Session-Based Recommender Systems. Recommender Systems Handbook (pp. 301–334). Springer, US: New York, NY.

    Chapter  Google Scholar 

  • Jiang, L., Huang, H., & Ding, Z. (2019). Path planning for intelligent robots based on deep Q-learning with experience replay and heuristic knowledge. IEEE/CAA Journal of Automatica Sinica, 7(4), 1179–1189. https://doi.org/10.1109/JAS.2019.1911732

    Article  MathSciNet  Google Scholar 

  • Kumari, T., Sharma, R., & Bedi, P. (2022). A contextual-bandit approach for multifaceted reciprocal recommendations in online dating. Journal of Intelligent Information Systems, 59(3), 705–731. https://doi.org/10.1007/s10844-022-00708-6

    Article  Google Scholar 

  • Lei, Y., Pei, H., & Yan, H., et al. (2020). Reinforcement learning based recommendation with graph convolutional Q-network. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (pp. 1757–1760). https://doi.org/10.1145/3397271.3401237

  • Li, J., Ren, P., & Chen, Z., et al. (2017). Neural attentive session-based recommendation. In Proceedings of the 2017 ACM conference on information and knowledge management (pp .1419–1428). https://doi.org/10.1145/3132847.3132926

  • Linden, G., Smith, B., & York, J. (2003). Amazon.com recommendations: item-to-item collaborative ltering. IEEE Internet Computing, 7(1), 76–80. https://doi.org/10.1109/MIC.2003.1167344

  • Liu, C., Li, Y., Lin, H., et al. (2023). GNNRec: Gated graph neural network for session-based social recommendation model. Journal of Intelligent Information Systems, 60(1), 137–156. https://doi.org/10.1007/s10844-022-00733-5

    Article  Google Scholar 

  • Liu, Q., Zeng, Y., & Mokhosi, R., et al. (2018). STAMP: short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 1831–1839). https://doi.org/10.1145/3219819.3219950

  • Luo, X., Zhou, M., Xia, Y., et al. (2014). An Efficient Non-Negative Matrix- Factorization-Based Approach to Collaborative Filtering for Recommender Systems. IEEE Transactions on Industrial Informatics, 10(2), 1273–1284. https://doi.org/10.1109/TII.2014.2308433

    Article  Google Scholar 

  • Mollel, M. S., Abubakar, A. I., Ozturk, M., et al. (2020). Intelligent handover decision scheme using double deep reinforcement learning. Physical Communication, 42, 101–133. https://doi.org/10.1016/j.phycom.2020.101133

    Article  Google Scholar 

  • Muvunza, T., & Li, Y. (2023). Session-based recommendation with temporal dynamics for large volunteer networks. Journal of Intelligent Information Systems 1–22. https://doi.org/10.1007/s10844-023-00801-4

  • Nguyen, L. V., Hong, M. S., Jung, J. J., et al. (2020). Cognitive Similarity-Based Collaborative Filtering Recommendation System. Applied Science, 10(12), 4183. https://doi.org/10.3390/app10124183

    Article  Google Scholar 

  • Nguyen, L. V., Vo, Q. T., & Nguyen, T. H. (2023). Adaptive KNN-Based Extended Collaborative Filtering Recommendation Services. Big Data and Cognitive Computing, 7(2), 106. https://doi.org/10.3390/bdcc7020106

    Article  Google Scholar 

  • Pan, Z., Cai, F., & Chen, W., et al. (2020). Star graph neural networks for session-based recommendation. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 1195–1204). https://doi.org/10.1145/3340531.3412014

  • Pang, Y., Wu, L., Shen, Q., et al. (2022). Heterogeneous global graph neural networks for personalized session-based recommendation. In Proceedings of the fifteenth ACM international conference on web search and data mining (pp. 775–783). https://doi.org/10.1145/3488560.3498505

  • Phuong, T. M., Thanh, T. C., & Bach, N. X. (2019). Neural Session-Aware Recommendation. IEEE. Access, 7, 86884–86896. https://doi.org/10.1109/ACCESS.2019.2926074

    Article  Google Scholar 

  • Quadrana, M., Karatzoglou, A., Hidasi, B., et al. (2017). Personalizing session-based recommendations with hierarchical recurrent neural networks. In Proceedings of the eleventh ACM conference on recommender systems. (pp. 130–137). https://doi.org/10.1145/3109859.3109896

  • Rettig, L., Audiffren, J., & Cudre-Mauroux, P. (2019). Fusing vector space models for domain-specific applications. In IEEE 31st International conference on tools with artificial intelligence (ICTAI) (pp. 1110–1117). https://doi.org/10.1109/ICTAI.2019.00155

  • Richa, & Bedi, P. (2019). Parallel context-aware multi-agent tourism recommender system. International Journal of Computational Science and Engineering, 20(4), 536–549. https://doi.org/10.1504/IJCSE.2019.104440

  • Sewak, M. (2019). Deep Q network (DQN), double DQN, and dueling DQN: a step towards general artificial intelligence. In Deep reinforcement learning: frontiers of artificial intelligence (pp. 95–108). Springer Singapore

  • Sutton, R.S., & Barto, A.G. (2018). Reinforcement Learning: An Introduction. MIT Press

  • Tang, X., Chen, J., Pu, H., et al. (2021). Double Deep Reinforcement Learning- Based Energy Management for a Parallel Hybrid Electric Vehicle With Engine StartStop Strategy. IEEE Transactions on Transportation Electrification, 8(1), 1376–1388. https://doi.org/10.1109/TTE.2021.3101470

    Article  Google Scholar 

  • Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference on artificial intelligence (pp. 2094–2100). https://doi.org/10.1609/aaai.v30i1.10295

  • Veličković, P. (2023). Everything is connected: Graph neural networks. Current Opinion in Structural Biology, 79, 102538. https://doi.org/10.1016/j.sbi.2023.102538

  • Wallach, D., & Goffinet, B. (1989). Mean squared error of prediction as a criterion for evaluating and comparing system models. Ecological Modelling, 44(3–4), 299–306. https://doi.org/10.1016/0304-3800(89)90035-5

    Article  Google Scholar 

  • Wu, S., Tang, Y., & Zhu, Y., et al. (2019). Session-based recommendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence (pp. 346–353). https://doi.org/10.1609/aaai.v33i01.3301346

  • Xi, L., Yu, L., Xu, Y., et al. (2019). A Novel Multi-Agent DDQN-AD Method-Based Distributed Strategy for Automatic Generation Control of Integrated Energy Systems. IEEE Transactions on Sustainable Energy, 11(4), 2417–2426. https://doi.org/10.1109/TSTE.2019.2958361

    Article  Google Scholar 

  • Xia, X., Yin, H., Yu, J., et al. (2021). Self-supervised hypergraph convolutional networks for session-based recommendation. In Proceedings of the AAAI con-ference on artificial intelligence (pp. 4503–4511). https://doi.org/10.1609/aaai.v35i5.16578

  • Ying, H., Zhuang, F., & Zhang, F., et al. (2018). Sequential recommender system based on hierarchical attention network. In IJCAI international joint conference on artificial intelligence (pp. 3926–3932). https://doi.org/10.24963/ijcai.2018/546

  • Zhang, P., Guo, J., & Li, C., et al. (2023). Efficiently leveraging multi-level user intent for session-based recommendation via atten-mixer network. In Proceedings of the sixteenth ACM international conference on web search and data mining (pp. 168–176). https://doi.org/10.1145/3539597.3570445

  • Zhang, S., Tong, H., & Xu, J., et al. (2019). Graph convolutional networks: a comprehensive review. Computational Social Networks, 6(1), 1–23. https://doi.org/10.1186/s40649-019-0069-y

  • Zhao, J., Li, H., Qu, L., et al. (2022). DCFGAN: An adversarial deep reinforcement learning framework with improved negative sampling for session-based recommender systems. Information Sciences, 596, 222–235. https://doi.org/10.1016/j.ins.2022.02.04

    Article  Google Scholar 

  • Zhao, X., Gu, C., Zhang, H., et al. (2021). DEAR: deep reinforcement learning for online advertising impression in recommender systems. In Proceedings of the AAAI conference on artificial intelligence (750–758). https://doi.org/10.1609/aaai.v35i1.16156

  • Zhao, X., Xia, L., Zhang, L., et al. (2018). Deep reinforcement learning for page-wise recommendations. In Proceedings of the 12th ACM conference on recommender systems (pp. 95–103). https://doi.org/10.1145/3240323.3240374

  • Zhou, S., Dai, X., & Chen, H., et al. (2020). Interactive recommender system via knowledge graph-enhanced reinforcement learning. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (pp. 179–188). https://doi.org/10.1145/3397271.3401174

  • Zou, L., Xia, L., & Ding, Z., et al. (2019). Reinforcement learning to optimize long-term user engagement in recommender systems. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2810–2818). https://doi.org/10.1145/3292500.3330668

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Delhi, Delhi, 110007, India

    Purnima Khurana & Punam Bedi

  2. Keshav Mahavidyalaya College, University of Delhi, Pitampura, Delhi, 110034, Delhi, India

    Bhavna Gupta

  3. P.G.D.A.V. College, University of Delhi, Nehru Nagar, Delhi, 110065, India

    Ravish Sharma

Authors
  1. Purnima Khurana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bhavna Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ravish Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Punam Bedi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Sr. Prof. Punam Bedi: Supervision, Reviewing, Editing, Resources Dr. Bhavna Gupta: Supervision, Writing, Review, Editing, Project Administration, Resources, Validation Dr. Ravish Sharma: Supervision, Writing, Review, Editing, Project Administration, Resources, Validation Ms. Purnima Khurana: Conceptualization, Methodology, Software, Resources, Validation, Investigation, Data curation, Formal analysis. Writing-Original draft preparation, Visualization, Writing- Reviewing and Editing.

Corresponding author

Correspondence to Punam Bedi.

Ethics declarations

Ethics approval

‘Yes’

Consent to participate

‘Yes’

Consent for publication

‘Yes’

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khurana, P., Gupta, B., Sharma, R. et al. Session-aware recommender system using double deep reinforcement learning. J Intell Inf Syst (2023). https://doi.org/10.1007/s10844-023-00824-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10844-023-00824-x

Keywords

Search

Navigation