Skip to main content
SpringerLink
Log in

AUBRec: adaptive augmented self-attention via user behaviors for sequential recommendation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Recently, self-attention models, and especially BERT4Rec (Sun, in: Proceedings of the 28th ACM international conference on information and knowledge management, 2019), have demonstrated themselves to be power tools for sequential recommendation. At their core, these models take a sequence of historical user behaviors as a sequential input, and then learn user behavior embeddings through an attention network to make their next recommendation. Yet, although a traditional self-attention model can effectively mine the potential relationship between user behaviors, the deep inherent characteristics of a user behavior sequence are still ignored. This mainly includes: the short-term and long-term information of a user behavior sequence; its continuous and non-continuous information; its forward and reverse asymmetric information; and its non-strong order dependency information. To address these issues, we propose a sequential recommendation model called AUBRec that uses controlled bidirectional self-attention to model user behavior sequences in an augmented manner. Specifically, we construct item interaction patterns based on the above-mentioned user behavior characteristics and then use these interaction patterns to locally augment attention. The item interaction patterns are created from a set of trainable parameter pairs, so it is learnable and lightweight. To further improve the accuracy and robustness of the model, we propose a dual-channel and confrontational self-attention model based on AUBRec, called AUBRec+. Extensive experiments on four publicly available datasets show that our method outperforms the current states-of-the-art in sequential recommendation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://jmcauley.ucsd.edu/data/amazon/.

  2. https://cseweb.ucsd.edu/~jmcauley/datasets.html#steam_data.

  3. https://grouplens.org/datasets/movielens/1m/.

  4. https://grouplens.org/datasets/movielens/20m/.

References

  1. Mobasher B, Dai H, Luo T, Nakagawa M (2001) “Effective personalization based on association rule discovery from web usage data,” In: Proceedings of the 3rd international workshop on Web information and data management, pp 9–15

  2. Forsati R, Meybodi MR, Neiat AG (2009) “Web page personalization based on weighted association rules,” In: 2009 International Conference on Electronic Computer Technology. IEEE, pp 130–135

  3. Yap G-E, Li X-L, Philip SY (2012) “Effective next-items recommendation via personalized sequential pattern mining,” In: International conference on database systems for advanced applications. Springer, pp 48–64

  4. Rendle S, Freudenthaler C, Schmidt-Thieme L (2010) “Factorizing personalized markov chains for next-basket recommendation,” In: Proceedings of the 19th international conference on World wide web, pp 811–820

  5. Liu D-R, Lai C-H, Lee W-J (2009) A hybrid of sequential rules and collaborative filtering for product recommendation. Inf Sci 179(20):3505–3519

    Article  Google Scholar 

  6. Feng S, Li X, Zeng Y, Cong G, Chee YM, Yuan Q (2015) “Personalized ranking metric embedding for next new poi recommendation,” In: Twenty-Fourth International Joint Conference on Artificial Intelligence

  7. He R, Kang W-C, McAuley J (2017) “Translation-based recommendation,” In: Proceedings of the eleventh ACM conference on recommender systems, pp 161–169

  8. Rendle S, Freudenthaler C, Schmidt-Thieme L (2010) “Factorizing personalized markov chains for next-basket recommendation,”In: Proceedings of the 19th international conference on World wide web, pp 811–820

  9. He R, McAuley J (2016) “Fusing similarity models with markov chains for sparse sequential recommendation,” In: 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, pp 191–200

  10. Hidasi B, Karatzoglou A, Baltrunas L, Tikk D (2015) “Session-based recommendations with recurrent neural networks,” arXiv preprint arXiv:1511.06939

  11. Wu C-Y, Ahmed A, Beutel A, Smola AJ, Jing H (2017) “Recurrent recommender networks,” In: Proceedings of the tenth ACM international conference on web search and data mining, pp 495–503

  12. Tan YK, Xu X, Liu Y (2016) “Improved recurrent neural networks for session-based recommendations,” In: Proceedings of the 1st workshop on deep learning for recommender systems, pp 17–22

  13. Quadrana M, Karatzoglou A, Hidasi B, Cremonesi P (2017) “Personalizing session-based recommendations with hierarchical recurrent neural networks,” In: Proceedings of the Eleventh ACM Conference on Recommender Systems, pp 130–137

  14. Tang J, Wang K (2018) “Personalized top-n sequential recommendation via convolutional sequence embedding,” In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp 565–573

  15. Tuan TX, Phuong TM (2017) “3d convolutional networks for session-based recommendation with content features,” In: Proceedings of the eleventh ACM conference on recommender systems, pp 138–146

  16. Yuan F, Karatzoglou A, Arapakis I, Jose JM, He X (2019) “A simple convolutional generative network for next item recommendation,” In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp 582–590

  17. Bahdanau D, Cho K, Bengio Y (2014) “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473

  18. Anything AM (2015) “Dynamic memory networks for natural language processing,” Kumar et al. arXiv Pre-Print, vol 97,

  19. Guo Q, Qiu X, Liu P, Xue X, Zhang Z (2020) Multi-scale self-attention for text classification. Proc AAAI Conf Artif Intell 34(05):7847–7854

    Google Scholar 

  20. Kang W-C, McAuley J (2018) “Self-attentive sequential recommendation,” In: 2018 IEEE International Conference on Data Mining (ICDM). IEEE, pp 197–206

  21. Sun F, Liu J, Wu J, Pei C, Lin X, Ou W, Jiang P (2019) “Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,” In: Proceedings of the 28th ACM international conference on information and knowledge management, pp 1441–1450

  22. Beltagy I, Peters ME, Cohan A (2020) “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150

  23. Guo Q, Qiu X, Liu P, Shao Y, Xue X, Zhang Z (2019) “Star-transformer,” arXiv preprint arXiv:1902.09113

  24. Wu S, Tang Y, Zhu Y, Wang L, Xie X, Tan T (2019) Session-based recommendation with graph neural networks. Proc AAAI Conf Artif Intell 33(01):346–353

    Google Scholar 

  25. Xu C, Zhao P, Liu Y, Sheng VS, Xu J, Zhuang F, Fang J, Zhou X (2019) Graph contextualized self-attention network for session-based recommendation. IJCAI 19:3940–3946

    Google Scholar 

  26. Wu L, Li S, Hsieh C-J, Sharpnack J (2020) “Sse-pt: Sequential recommendation via personalized transformer,” In: Fourteenth ACM Conference on Recommender Systems, pp 328–337

  27. Li J, Wang Y, McAuley J (2020) “Time interval aware self-attention for sequential recommendation,” In: Proceedings of the 13th international conference on web search and data mining, pp 322–330

  28. Cho SM, Park E, Yoo S (2020) “Meantime: Mixture of attention mechanisms with multi-temporal embeddings for sequential recommendation,” In: Fourteenth ACM Conference on Recommender Systems, pp 515–520

  29. Liu C, Li X, Cai G, Dong Z, Zhu H, Shang L (2021) Noninvasive self-attention for side information fusion in sequential recommendation. Proc AAAI Conf Artif Intell 35(5):4249–4256

    Google Scholar 

  30. Qiu Z, Wu X, Gao J, Fan W (2021) “U-bert: Pre-training user representations for improved recommendation,” In: Proc of the AAAI Conference on Artificial Intelligence. Menlo Park, CA, AAAI, pp 1–8

  31. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078

  32. Liu Q, Zeng Y, Mokhosi R, Zhang H (2018) “Stamp: short-term attention/memory priority model for session-based recommendation,” In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1831–1839

  33. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) “Attention is all you need,” In: Advances in neural information processing systems, pp 5998–6008

  34. Devlin J, Chang M.-W, Lee K, Toutanova K (2018) “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805

  35. Zhang S, Tay Y, Yao L, Sun A, An J (2019) “Next item recommendation with self-attentive metric learning,” In: Thirty-Third AAAI Conference on Artificial Intelligence, vol 9

  36. Lin J, Pan W, Ming Z (2020) “Fissa: fusing item similarity models with self-attention networks for sequential recommendation,” In: Fourteenth ACM Conference on Recommender Systems, pp 130–139

  37. Fan X, Liu Z, Lian J, Zhao WX, Xie X, Wen J-R (2021) “Lighter and better: low-rank decomposed self-attention networks for next-item recommendation,” In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 1733–1737

  38. Yao T, Yi X, Cheng DZ, Yu F, Chen T, Menon A, Hong L, Chi EH, Tjoa S, Kang J et al. (2021) “Self-supervised learning for large-scale item recommendations,” In: Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pp 4321–4330

  39. Xie X, Sun F, Liu Z, Gao J, Ding B, Cui B (2020) “Contrastive pre-training for sequential recommendation,” arXiv e-prints, pp. arXiv–2010

  40. Wu J, Wang X, Feng F, He X, Chen L, Lian J, Xie X (2021) “Self-supervised graph learning for recommendation,” In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 726–735

  41. Liu Z, Fan Z, Wang Y, Yu PS (2021) “Augmenting sequential recommendation with pseudo-prior items via reversely pre-training transformer,” In: Proceedings of the 44th international ACM SIGIR conference on Research and development in information retrieval, pp 1608–1612

  42. Wang Z, Zhang J, Xu H, Chen X, Zhang Y, Zhao WX, Wen JR (2021) “Counterfactual data-augmented sequential recommendation,” In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 347–356

  43. Guo M, Zhang Y, Liu T (2019) Gaussian transformer: a lightweight approach for natural language inference. Proc AAAI Conf Artif Intell 33(01):6489–6496

    Google Scholar 

  44. Krichene W, Rendle S (2020) “On sampled metrics for item recommendation,” In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery and data mining, pp 1748–1757

  45. Gao T, Yao X, Chen D (2021) “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821

  46. Chen T, Kornblith S, Norouzi M, Hinton G (2020) “A simple framework for contrastive learning of visual representations,” In: International conference on machine learning. PMLR, pp 1597–1607

Download references

Acknowledgements

This work was supported by a Grant from The National Natural Science Foundation of China (No. U21A20484) and Science and Technology Program of Zhejiang Province (No.2021C01187)

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China

    Jin Fan, Xiaofeng Yu, Zehao Wang, Weijie Wang, Danfeng Sun & Huifeng Wu

Authors
  1. Jin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaofeng Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zehao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weijie Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Danfeng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Huifeng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Fan.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, J., Yu, X., Wang, Z. et al. AUBRec: adaptive augmented self-attention via user behaviors for sequential recommendation. Neural Comput & Applic 34, 21715–21728 (2022). https://doi.org/10.1007/s00521-022-07623-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07623-5

Keywords

Search

Navigation