Skip to main content
SpringerLink
Log in

MhSa-GRU: combining user’s dynamic preferences and items’ correlation to augment sequence recommendation

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Product recommendation systems have become an effective tool to help users make choices under information overload. For sequence recommendation, the user’s dynamic preferences and the correlations between items are essential for exploring temporal information in hidden states and latent relationships, which previous research has rarely considered. This paper proposes a novel model named MhSa-GRU that combines Multi-head Self-attention with a Gated Recurrent Unit, and integrates the prices, user behavior, and category features of items to help generate the next recommendation. In this model, an improved self-attention layer captures the items’ correlations, and multiple heads learn thorough local information about the vector. The mask and attention threshold mechanisms can exclude disturbing information commonly observed in product recommendation systems. Moreover, the GRU module introduces users’ global and local preferences to capture their behavior. Universal experiments on three real-world datasets verify that MhSa-GRU outperforms competitive baseline models, such as HR@10, HR@20, MRR@10, and MRR@20, in evaluation metrics for prediction. MhSa-GRU improves performance by 5% on the Cosmetics 2019-Nov and Cosmetics 2020-Jan datasets, and by about 10% on the Ta Feng dataset. Ablation experiments help determine the interpretability and effectiveness of the multi-head self-attention module, and show this module can strengthen efficiency by about 5%–10%. We also find that the predicted efficacy is optimal when the attention-head dimension is roughly 10% to 20% of the embedding vector. In addition, the sparse dataset should use high-dimensional attention-head vectors since the local information is too sparse to capture the relevance between items.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Availability of supporting data

The datasets that support this study are available from https://www.kaggle.com/mkechinov/ecommerce-events-history-in-cosmetics-shop and https://www.kaggle.com/chiranjivdas09/ta-feng-grocery-dataset

Notes

  1. https://www.kaggle.com/mkechinov/ecommerce-events-history-in-cosmetics-shop.

  2. https://www.kaggle.com/chiranjivdas09/ta-feng-grocery-dataset.

  3. https://github.com/yuyu12shine/MhSa-GRU.

References

  • Ajaegbu, C. (2021). An optimized item-based collaborative filtering algorithm. Journal of Ambient Intelligence and Humanized Computing, 12(12), 10629–10636. https://doi.org/10.1007/s12652-020-02876-1

    Article  Google Scholar 

  • Ali, Z., Kefalas, P., Muhammad, K., et al. (2020). Deep learning in citation recommendation models survey. Expert Systems with Applications, 162, 113790. https://doi.org/10.1016/j.eswa.2020.113790

    Article  Google Scholar 

  • Bellogín, A., Castells, P., & Cantador, I. (2014). Neighbor selection and weighting in user-based collaborative filtering: A performance prediction approach. ACM Transactions on the Web, 8(2), 1–30. https://doi.org/10.1145/2579993

    Article  Google Scholar 

  • Boratto, L., Fenu, G., & Marras, M. (2021). Connecting user and item perspectives in popularity debiasing for collaborative recommendation. Information Processing and Management, 58(1), 102387. https://doi.org/10.1016/j.ipm.2020.102387

    Article  Google Scholar 

  • Cechinel, C., Sicilia, M. Á., Sánchez-Alonso, S., et al. (2013). Evaluating collaborative filtering recommendations inside large learning object repositories. Information Processing and Management, 49(1), 34–50. https://doi.org/10.1016/j.ipm.2012.07.004

    Article  Google Scholar 

  • Chen, C., Li, D., Yan, J., et al. (2021). Modeling dynamic user preference via dictionary learning for sequential recommendation. IEEE Transactions on Knowledge and Data Engineering. https://doi.org/10.1109/TKDE.2021.3050407

    Article  Google Scholar 

  • Chen, W., Ren, P., Cai, F., et al. (2022). Multi-interest diversification for end-to-end sequential recommendation. ACM Transactions on Information Systems, 40(1), 20:1-20:30.

    Article  Google Scholar 

  • Devlin, J., Chang, M., Lee, K., et al. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL 2019 (pp. 4171–4186). https://doi.org/10.48550/arXiv.1810.04805

  • Fusi, N., Sheth, R., & Elibol, M. (2018). Probabilistic matrix factorization for automated machine learning. In Proceedings of 32th annual conference on neural information processing systems (pp. 3348–3357). https://doi.org/10.48550/arXiv.1705.05355

  • Grbovic, M., Radosavljevic, V., Djuric, N., et al. (2015). E-commerce in your inbox: Product recommendations at scale. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1809–1818). https://doi.org/10.1145/2783258.2788627

  • Guan, Y., Wei, Q., & Chen, G. (2019). Deep learning based personalized recommendation with multi-view information integration. Decision Support Systems, 118, 58–69. https://doi.org/10.1016/j.dss.2019.01.003

    Article  Google Scholar 

  • Guo, T., Luo, J., Dong, K., et al. (2019). Locally differentially private item-based collaborative filtering. Information Sciences, 502, 229–246. https://doi.org/10.1016/j.ins.2019.06.021

    Article  MathSciNet  MATH  Google Scholar 

  • Ha, T., & Lee, S. (2017). Item-network-based collaborative filtering: A personalized recommendation method based on a user’s item network. Information Processing and Management, 53(5), 1171–1184. https://doi.org/10.1016/j.ipm.2017.05.003

    Article  Google Scholar 

  • Hao, J., Wang, X., Shi, S., et al. (2019). Multi-granularity self-attention for neural machine translation. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (pp. 887–897). https://doi.org/10.18653/v1/D19-1082

  • He, R., & McAuley, J. (2016). Fusing similarity models with markov chains for sparse sequential recommendation. In Proceedings of 2016 IEEE 16th international conference on data mining (pp. 191–200). https://doi.org/10.1109/ICDM.2016.0030

  • He, J., Li, X., & Liao, L. (2018). Next point-of-interest recommendation via a category-aware listwise bayesian personalized ranking. Journal of Computer Science, 28, 206–216. https://doi.org/10.1016/j.jocs.2017.09.014

    Article  Google Scholar 

  • Hidasi, B., & Karatzoglou, A. (2018). Recurrent neural networks with top-k gains for session-based recommendations. In Proceedings of the 27th ACM international conference on information and knowledge management (pp. 843–852). https://doi.org/10.1145/3269206.3271761

  • Huang, X., Qian, S., Fang, Q., et al. (2018). Csan: Contextual self-attention network for user sequential recommendation. In Proceedings of the 26th ACM international conference on multimedia (pp. 447–455). https://doi.org/10.1145/3240508.3240609

  • Huang, L., Ma, Y., Wang, S., et al. (2019). An attention-based spatiotemporal lstm network for next poi recommendation. IEEE Transactions on Services Computing, 14(6), 1585–1597. https://doi.org/10.1109/TSC.2019.2918310

    Article  Google Scholar 

  • Ji, Y., Yin, M., Fang, Y., et al. (2020). Temporal heterogeneous interaction graph embedding for next-item recommendation. In Proceedings of joint european conference on machine learning and knowledge discovery in databases (pp. 314–329). https://doi.org/10.1007/978-3-030-67664-3_19

  • Jia, Z., Yang, Y., Gao, W., et al. (2015). User-based collaborative filtering for tourist attraction recommendations. In: Proceedings of 2015 IEEE international conference on computational intelligence & communication technology (pp. 22–25). https://doi.org/10.1109/CICT.2015.20

  • Kang, W.C., & McAuley, J. (2018). Self-attentive sequential recommendation. In Proceedings of 2018 IEEE international conference on data mining (pp. 197–206). https://doi.org/10.1109/ICDM.2018.00035

  • Lei, J., Li, Y., Yang, S., et al. (2022). Two-stage sequential recommendation for side information fusion and long-term and short-term preferences modeling. Journal of Intelligent Information System. https://doi.org/10.1007/s10844-022-00723-7

    Article  Google Scholar 

  • Li, X., Song, J., Gao, L., et al. (2019). Beyond rnns: Positional self-attention with co-attention for video question answering. In Proceedings of the AAAI conference on artificial intelligence (pp. 8658–8665). https://doi.org/10.1609/aaai.v33i01.33018658

  • Li, C., Bao, Z., Li, L., et al. (2020). Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Information Processing and Management, 57(3), 102185. https://doi.org/10.1016/j.ipm.2019.102185

    Article  Google Scholar 

  • Liao, G., Deng, X., Wan, C., et al. (2022). Group event recommendation based on graph multi-head attention network combining explicit and implicit information. Information Processing and Management, 59(2), 102797. https://doi.org/10.1016/j.ipm.2021.102797

    Article  Google Scholar 

  • Linden, G., Smith, B., & York, J. (2003). Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Computing, 7(1), 76–80. https://doi.org/10.1109/MIC.2003.1167344

    Article  Google Scholar 

  • Liu, Q., Zeng, Y., Mokhosi, R., et al. (2018). STAMP: Short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1831–1839). https://doi.org/10.1145/3219819.3219950

  • Liu, L., Wang, L., & Lian, T. (2021). CaSe4SR: Using category sequence graph to augment session-based recommendation. Knowledge-Based Systems, 212, 106558. https://doi.org/10.1016/j.knosys.2020.106558

    Article  Google Scholar 

  • Liu, C., Li, Y., Lin, H., et al. (2022). GNNRec: Gated graph neural network for session-based social recommendation model. Journal of Intelligent Information System. https://doi.org/10.1007/s10844-022-00733-5

    Article  Google Scholar 

  • Lu, Y., & Zhang, J. (2021). Bibliometric analysis and critical review of the research on big data in the construction industry. Engineering, Construction and Architectural Management. https://doi.org/10.1108/ECAM-01-2021-0005

    Article  Google Scholar 

  • Ma, M., Ren, P., Chen, Z., et al. (2022). Improving transformer-based sequential recommenders through preference editing. ACM Transactions on Information Systems. https://doi.org/10.1145/3564282

    Article  Google Scholar 

  • Quadrana, M., Cremonesi, P., & Jannach, D. (2018). Sequence-aware recommender systems. ACM Computing Surveys, 51(4), 66:1-66:36. https://doi.org/10.1145/3190616

    Article  Google Scholar 

  • Qin, Y., Wang, P., & Li, C. (2021). The world is binary: Contrastive learning for denoising next basket recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval (pp. 859–868). https://doi.org/10.1145/3404835.3462836

  • Rappaz, J., McAuley, J., & Aberer, K. (2021). Recommendation on live-streaming platforms: Dynamic availability and repeat consumption. In Proceedings of 15th ACM conference on recommender systems (pp. 390–399). https://doi.org/10.1145/3460231.3474267

  • Rendle, S., Freudenthaler, C., & Schmidt-Thieme, L. (2010). Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on world wide web (pp. 811–820). https://doi.org/10.1145/1772690.1772773

  • Sarwar, B., Karypis, G., Konstan, J., et al. (2001). Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on world wide web (pp. 285–295). https://doi.org/10.1145/371920.372071

  • Smirnova, E., & Vasile, F. (2017). Contextual sequence modeling for recommendation with recurrent neural networks. In Proceedings of the 2nd workshop on deep learning for recommender systems (pp. 2–9). https://doi.org/10.1145/3125486.3125488

  • Stiff, A., Song, Q., & Fosler-Lussier, E. (2020). How self-attention improves rare class performance in a question-answering dialogue agent. In Proceedings of the 21th annual meeting of the special interest group on discourse and dialogue (pp. 196–202).

  • Stratigi, M., Pitoura, E., Nummenmaa, J., et al. (2022). Sequential group recommendations based on satisfaction and disagreement scores. Journal of Intelligent Information System, 58, 227–254. https://doi.org/10.1007/s10844-021-00652-x

    Article  Google Scholar 

  • Sun, F., Liu, J., Wu, J., et al. (2019). BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management (pp. 1441–1450). https://doi.org/10.1145/3357384.3357895

  • Tang, G., Müller, M., Rios, A., et al. (2018). Why self-attention? A targeted evaluation of neural machine translation architectures. In Proceedings of the 2018 conference on empirical methods in natural language processing (pp. 4263–4272). https://doi.org/10.18653/v1/D18-1458

  • Thakkar, P., Varma, K., Ukani, V., et al. (2018). Combining user-based and item-based collaborative filtering using machine learning. In Proceedings of information and communication technology for intelligent systems (pp. 173–180). https://doi.org/10.1007/978-981-13-1747-7_17

  • Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. In Proceedings of 31th annual conference on neural information processing systems (pp. 5998–6008). https://doi.org/10.48550/arXiv.1706.03762

  • Wang, J., Huang, P., Zhao, H., et al. (2018). Billion-scale commodity embedding for e-commerce recommendation in alibaba. In Proceedings of the 24th ACM SIGKDD conference on knowledge discovery and data mining (pp. 839–848). https://doi.org/10.1145/3219819.3219869

  • Wang, K., Wang, X., & Lu, X. (2021). POI recommendation method using LSTM-attention in LBSN considering privacy protection. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-021-00440-8

    Article  Google Scholar 

  • Wu, S., Tang, Y., Zhu, Y., et al. (2019). Session-based recommendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, 346–353. https://doi.org/10.1609/aaai.v33i01.3301346

  • Xu, C., Zhao, P., Liu, Y., et al. (2019). Recurrent convolutional neural network for sequential recommendation. In Proceedings of the 28th international conference on world wide web (pp. 3398–3404). https://doi.org/10.1145/3308558.3313408

  • Xu, C., Feng, J., Zhao, P., et al. (2021). Long- and short-term self-attention network for sequential recommendation. Neurocomputing, 423, 580–589. https://doi.org/10.1016/j.neucom.2020.10.066

    Article  Google Scholar 

  • Xue, L., Li, X., & Zhang, N.L. (2020). Not all attention is needed: gated attention network for sequence data. In Proceedings of the AAAI conference on artificial intelligence (pp. 6550–6557). https://doi.org/10.1609/aaai.v34i04.6129

  • Yan, A., Cheng, S., Kang, W. C., et al. (2019). CosRec: 2D convolutional neural networks for sequential recommendation. In Proceedings of the 28th ACM international conference on information and knowledge management (pp. 2173–2176). https://doi.org/10.1145/3357384.3358113

  • Yu, L., Zhang, C., Liang, S., et al. (2019). Multi-order attentive ranking model for sequential recommendation. In Proceedings of the AAAI conference on artificial intelligence (pp. 5709–5716). https://doi.org/10.1609/aaai.v33i01.33015709

  • Yuan, W., Wang, H., Yu, X., et al. (2020). Attention-based context-aware sequential recommendation model. Information Sciences, 510, 122–134. https://doi.org/10.1016/j.ins.2019.09.007

    Article  Google Scholar 

  • Zhang, S., Tay, Y., Yao, L., et al. (2019a). Next item recommendation with self-attentive metric learning. In Proceedings of the 33th AAAI conference on artificial intelligence (pp. 1–9).

  • Zhang, T., Zhao, P., Liu, Y., et al. (2019b). Feature-level deeper self-attention network for sequential recommendation. In Proceedings of the 28th international joint conference on artificial intelligence, (pp. 4320–4326). https://doi.org/10.24963/ijcai.2019b/600

  • Zhang, J., Ma, C., Zhong, C., et al. (2021). MBPI: Mixed behaviors and preference interaction for session-based recommendation. Applied Intelligence, 51(10), 7440–7452. https://doi.org/10.1007/s10489-021-02284-8

    Article  Google Scholar 

  • Zhao, Q., Zhang, Y., Friedman, D., et al. (2015). E-commerce recommendation with personalized promotion. In Proceedings of the 9th ACM conference on recommender systems (pp. 219–226). https://doi.org/10.1145/2792838.2800178

  • Zhao, C., You, J., Wen, X., et al. (2020). Deep Bi-LSTM networks for sequential recommendation. Entropy, 22(8), 870. https://doi.org/10.3390/e22080870

    Article  Google Scholar 

Download references

Acknowledgements

We appreciate Yixuan Feng, Mingzhou Chen, Hao Zhou for reviewing the manuscript. Moreover, the authors are grateful to the editors and referees for their constructive comments on the paper.

Funding

This work is financially supported by the National Natural Science Foundation of China under Projects 72171176, 71771179, 72021002.

Author information

Authors and Affiliations

  1. School of Economics and Management, Tongji University, 1239 Siping Road, Shanghai, 200092, People’s Republic of China

    Yongrui Duan, Peng Liu & Yusheng Lu

Authors
  1. Yongrui Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yusheng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yongrui Duan and Peng Liu designed the model. Yusheng Lu and Peng Liu conducted the experiments. All authors drafted, revised and approved the manuscript.

Corresponding author

Correspondence to Yusheng Lu.

Ethics declarations

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Human and animal ethics

Not applicable.

Competing interests

The authors have no conflicts of interest to declare.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Duan, Y., Liu, P. & Lu, Y. MhSa-GRU: combining user’s dynamic preferences and items’ correlation to augment sequence recommendation. J Intell Inf Syst 61, 225–248 (2023). https://doi.org/10.1007/s10844-022-00754-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-022-00754-0

Keywords

Search

Navigation