Skip to main content
SpringerLink
Log in

Graphs get personal: learning representation with contextual pretraining for collaborative filtering

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The interactions of users and items in recommender systems can be naturally modeled as user-item bipartite graphs. The iterative propagation of graph neural networks (GNN) can explicitly exploit high-order connectivity from those user-item interactions. Apart from these advantages, there are two limitations in GNN-based recommendation systems that might lead to performance degradation: 1) Existing GNN methods only depend on the graph topology but ignore the insightful relationship between the connected nodes. The edge representation in GNNs cannot effectively express the personalized information of users and items, and can only represent the structural connection information of the graph. 2) The representations of nodes and edges were initialized randomly, these initial representations participate in subsequent node propagation and updating computations in the graph neural network. It directly affects the final representation of the user and item nodes in the network and result in bad recommendation performence. To address these issues, this study proposes a graph attention network with contextual pretraining (GAT-CP) for content-based collaborative filtering. It explicitly exploits the user-item graph structure twofold. First, an contextual personalized sentiment analysis task was applied by fine-tuning the BERT model to initialize the representations of nodes and edges by investigating the user preference for products based on the reviews of the users. Second, the obtained edge representations were used as the propagation constraints to assign different weights to the edges in GAT. Comparative results show the significant performance gains of GAT-CP and the necessity of node and edge initialization with contextual tasks. The code for this paper is available at: https://github.com/Yellow4Submarine7/GAT_AP

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://huggingface.co/docs/transformers/

References

  1. Das D, Sahoo L, Datta S (2017) A survey on recommendation system. Int J Comput Appl 160(7)

  2. Dang DL, Moughnyeh S, Stephens E, Convers V, Adkins–Jablonsky SJ, Raut S (2022) A pandemic pivot: podcast as an active engagement tool in the classroom and beyond. CourseSource

  3. Son J, Kim SB (2017) Content-based filtering for recommendation systems using multiattribute networks. Expert Syst Appl 89:404–412

    Article  Google Scholar 

  4. Shambour QY, Hussein A, Kharma Q, Abu–Alhaj MM (2022) Effective hybrid content-based collaborative filtering approach for requirements engineering. Comput Syst Sci Eng

  5. Ahmed NK, Rossi RA, Zhou R, Lee JB, Kong X, Willke TL, Eldardiry H (2017) Inductive representation learning in large attributed graphs. In: Advances in neural information processing systems (NeurIPS–2017), pp 1025–1035

  6. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th international conference on learning representations (ICLR–2017), pp 1–14

  7. Wang X, He X, Wang M, Feng F, Chua T-S (2019) Neural graph collaborative filtering. In: Proceedings of the 42nd international ACMSI37 GIR conference on research and development in information retrieval (SIGIR–2019), pp 165–174

  8. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv:1609.02907

  9. He X, Deng K, Wang X, Li Y, Zhang YD, Wang M (2020) Lightgcn: simplifying and powering graph convolution network for recommendation. In: SIGIR 2020 – Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pp 639–648

  10. Kumar I, Hu Y, Zhang Y (2022) Eflec: Efficient feature-leakage correction in gnn based recommendation systems. In: Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp 1885–1889

  11. Nguyen TT, Quach KND, Nguyen TT, Huynh TT, Vu VH, Le Nguyen P, Jo J, Nguyen QVH (2022) Poisoning gnn–based recommender systems with generative surrogate–based attacks. ACM Trans Inf Syst

  12. Wang X, Wang R, Shi C, Song G, Li Q. Multi-component graph convolutional collaborative filtering. Technical report (0)

  13. Ma J, Zhou C, Yang H, Cui P, Wang X, Zhu W (2020) Disentangled self-supervision in sequential recommenders. In: Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining (SIGKDD-2020), pp 483–491

  14. Veličković P, Casanova A, Lió P, Cucurull G, Romero A, Bengio Y (2018) Graph attention networks. In: Proceedings of the 6th international conference on learning representations (ICLR–2018), pp 1–12

  15. Nguyen Thanh T, Quach NDK, Nguyen TT, Huynh TT, Vu VH, Nguyen PL, Jo J, Nguyen QVH (2023) Poisoning gnn-based recommender systems with generative surrogate-based attacks. ACM Trans Inf Syst 41(3):1–24

    Article  Google Scholar 

  16. Yang L, Wang S, Tao Y, Sun J, Liu X, Yu PS, Wang T (2023) Dgrec: graph neural network for recommendation with diversified embedding generation. In: Proceedings of the sixteenth ACM international conference on web search and data mining, pp 661–669

  17. Devlin J, Chang M–W, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805

  18. Wu L, He X, Wang X, Zhang K, Wang M (2022) A survey on accuracy-oriented neural recommendation: from collaborative filtering to information-rich recommendation. IEEE Trans Knowledge Data Eng

  19. He X, Liao L, Zhang H, Nie L, Hu X, Chua T-S (2017) Neural collaborative filtering. In: Proceedings of the 26th international conference on world wide web, pp 173–182

  20. Xu R, Li J, Li G, Pan P, Zhou Q, Wang C (2022) Sdnn: symmetric deep neural networks with lateral connections for recommender systems. Inf Sci 595:217–230

    Article  Google Scholar 

  21. Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791

    Article  Google Scholar 

  22. Mnih A, Salakhutdinov RR (2007) Probabilistic matrix factorization. Adv Neural Inf Process Syst 20

  23. Liao M, Sundar SS, B Walther J (2022) User trust in recommendation systems: a comparison of content-based, collaborative and demographic filtering. In: Proceedings of the 2022 CHI conference on human factors in computing systems, pp 1–14

  24. Yera R, Alzahrani AA, Martínez L (2022) A fuzzy content-based group recommender system with dynamic selection of the aggregation functions. Int J Approx Reason 150:273–296

    Article  Google Scholar 

  25. Budhani S, Kataria R, Nagdev M, Niranjani S, Saindane P (2022) Moelleux–music recommendation system. In: Data intelligence and cognitive informatics, pp 443–457

  26. Wang X, Yu L, Ren K, Tao G, Zhang W, Yu Y, Wang J (2017) Dynamic attention deep model for article recommendation by learning human editors’ demonstration. In: Proceedings of the 23rd Acm Sigkdd international conference on knowledge discovery and data mining, pp 2051–2059

  27. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25

  28. Ma R, Zhang Q, Wang J, Cui L, Huang X (2018) Mention recommendation for multimodal microblog with cross-attention memory network. In: The 41st international ACM SIGIR conference on research & development in information retrieval, pp 195–204

  29. Lyu C, Yang L, Zhang Y, Graham Y, Foster J (2023) Exploiting rich textual user-product context for improving personalized sentiment analysis. Findings of the Association for Computational Linguistics: ACL 2023:1419–1429

    Google Scholar 

  30. Ezaldeen H, Misra R, Bisoy SK, Alatrash R, Priyadarshini R (2022) A hybrid e-learning recommendation integrating adaptive profiling and sentiment analysis. J Web Semantics 72:100700

  31. Quercia D, Kosinski M, Stillwell D, Crowcroft J (2011) Our twitter pro files, our selves: predicting personality with twitter. In: 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing, pp 180–185

  32. Peng K-H, Liou L-H, Chang C-S, Lee D-S (2015) Predicting personality traits of Chinese users based on facebook wall posts. In: 2015 24th wireless and optical communication conference (WOCC), pp 9–14

  33. Majumder N, Poria S, Gelbukh A, Cambria E (2017) Deep learning-based document modeling for personality detection from text. IEEE Intell Syst 32(2):74–79

    Article  Google Scholar 

  34. Wang P, Li L, Wang R, Zheng X, He J, Xu G (2022) Learning persona-driven personalized sentimental representation for review-based recommendation. Expert Syst Appl 203:117317

    Article  Google Scholar 

  35. Li M, Xu H, Tu Z, Su T, Xu X, Wang Z (2022) A deep learning based personalized qoe/qos correlation model for composite services. In: 2022 IEEE international conference on web services (ICWS). IEEE, pp 312–321

  36. Chen D, Gao D, Kuang W, Li Y, Ding B (2022) Pfl-bench: a comprehensive benchmark for personalized federated learning. Adv Neural Inf Process Syst 35:9344–9360

    Google Scholar 

  37. Asabere NY, Acakpovi A, Michael MB (2018) Improving socially-aware recommendation accuracy through personality. IEEE Trans Affect Comput 9(3):351–361

    Article  Google Scholar 

  38. Liu K, Xue F, Guo D, Wu L, Li S, Hong R (2023) Megcf: multimodal entity graph collaborative filtering for personalized recommendation. ACM Trans Inf Syst 41(2):1–27

    Article  Google Scholar 

  39. Scarselli F, Yong SL, Gori M, Hagenbuchner M, Tsoi AC, Maggini M (2005) Graph neural networks for ranking web pages. In: The 2005 IEEE/WIC/ACM international conference on web intelligence (WI’05), pp 666–672

  40. Zhang P, Chen J, Che C, Zhang L, Jin B, Zhu Y (2023) Iea-gnn: anchor-aware graph neural network fused with information entropy for node classification and link prediction. Inf Sci 634:665–676

    Article  Google Scholar 

  41. Wang K, Han SC, Poon J (2022) Induct-gcn: inductive graph convolutional networks for text classification. In: 2022 26th international conference on pattern recognition (ICPR). IEEE, pp 1243–1249

  42. Huang Y-H, Chen Y-H, Chen Y-S (2022) Contexting: granting document wise contextual embeddings to graph neural networks for inductive text classification. In: Proceedings of the 29th international conference on computational linguistics, pp 1163–1168

  43. Benslimane S, Azé J, Bringay S, Servajean M, Mollevi C (2023) A text and gnn based controversy detection method on social media. WorldWide Web 26(2):799–825

  44. Wang X, Jin H, Zhang A, He X, Xu T, Chua TS (2020) Disentan17 gled graph collaborative filtering. SIGIR 2020 – Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pp 1001–1010

  45. Quan Y, Ding J, Gao C, Yi L, Jin D, Li Y (2023) Robust preference guided denoising for graph based social recommendation. Proc ACM Web Conf 2023:1097–1108

    Google Scholar 

  46. Wei T, Chow TW, Ma J, Zhao M (2022) Expgcn: review-aware graph convolution network for explainable recommendation. Neural Networks

  47. Wang X, Ji H, Shi C, Wang B, Ye Y, Cui P, Yu PS (2019) Heterogeneous graph attention network. In: The world wide web conference, pp 2022–2032

  48. Wang X, Wang R, Shi C, Song G, Li Q Multi-component graph convolutional collaborative filtering. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 6267–6274

  49. Kulkarni S, Rodd SF (2020) Context aware recommendation systems: a review of the state of the art techniques. Comput Sci Rev 37:100255

  50. Wu Y, Schuster M, Chen Z, Le VQ, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J, Shah A, John son M, Liu X, Kaiser L, Gouws S, Kato Y, Kudo T, Kazawa H, Stevens K, Kurian G, Patil N, Wang W, Young C, Smith J, Riesa J, Rudnick A, Vinyals O, Corrado G, Hughes M, Dean J (2016) Google’s neural machine translation system: bridging the gap between human and machine translation. In: Proceedings of the conference of the association for machine translation in the Americans, pp 193–199

  51. Wu Z, Dai X-Y, Yin C, Huang S, Chen J (2018) Improving review representations with user attention and product attention for sentiment classification. In: Proceedings of the AAAI conference on artificial intelligence, vol 32

  52. Wang W, Pan SJ, Dahlmeier D, Xiao X (2017) Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In: Proceedings of the AAAI conference on artificial intelligence, vol 31

  53. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems (NIPS-2017), pp 5999–6009

  54. Rendle S, Freudenthaler C, Gantner Z, Schmidt–Thieme L (2009) Bpr: Bayesian personalized ranking from implicit feedback. Proceedings of the 25th conference on uncertainty in artificial intelligence, UAI 2009, pp 452–461

  55. McAuley J, Targett C, Shi Q, Van Den Hengel A (2015) Image-based recommendations on styles and substitutes. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp 43–52

  56. Berg vdR, Kipf TN, Welling M (2017) Graph convolutional matrix completion. arXiv:1706.02263

  57. Kim D, Park C, Oh J, Lee S, Yu H (2016) Convolutional matrix factorization for document context-aware recommendation. In: RecSys 2016 - Proceedings of the 10th ACM conference on recommender systems, pp 233–240

  58. Zheng L, Noroozi V, Yu PS (2017) Joint deep modeling of users and items using reviews for recommendation. In: Proceedings of the tenth ACM international conference on web search and data mining, pp 425–434

  59. Wang X, He X, Wang M, Feng F, Chua T-S (2019) Neural graph collaborative filtering. Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 165–174

  60. Mao K, Zhu J, Xiao X, Lu B, Wang Z, He X (2021) Ultragcn: ultra simplification of graph convolutional networks for recommendation. In: Proceedings of the 30th ACM international conference on information & knowledge management, pp 1253–1262

  61. Fan W, Liu X, Jin W, Zhao X, Tang J, Li Q (2022) Graph trend filtering networks for recommendation. In: Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp 112–121

  62. Liu K, Xue F, Hong R (2022) Rgcf: refined graph convolution collaborative filtering with concise and expressive embedding. Intell Data Anal 26(2):427–445

    Article  Google Scholar 

  63. Wang C, Yu Y, Ma W, Zhang M, Chen C, Liu Y, Ma S (2022) Towards representation alignment and uniformity in collaborative filtering. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp 1816–1825

  64. Wang Y, Zhao Y, Zhang Y, Derr T (2023) Collaboration-aware graph convolutional network for recommender systems. Proc ACM Web Conf 2023:91–101

    Google Scholar 

  65. Guo J, Du L, Chen X, Ma X, Fu Q, Han S, Zhang D, Zhang Y (2023) On manipulating signals of user–item graph: a Jacobi polynomial-based graph collaborative filtering. ACM Digital Library

  66. Kenton JDM-WC, Toutanova LK (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of naacL-HLT, vol 1, p 2

  67. Loshchilov I, Hutter F (2017) Decoupled weight decay regularization. arXiv:1711.05101

  68. Van der Maaten L, Hinton G (2008) Visualizing data using t–sne. J Mach Learn Res 9(11)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) under Grants Nos. 61966038 and 62266051. The authors would like to thank the anonymous reviewers for their constructive comments.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yunnan University, Kunming, China

    Tiesunlong Shen, You Zhang, Jin Wang & Xuejie Zhang

Authors
  1. Tiesunlong Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. You Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuejie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shen, T., Zhang, Y., Wang, J. et al. Graphs get personal: learning representation with contextual pretraining for collaborative filtering. Appl Intell 53, 30416–30430 (2023). https://doi.org/10.1007/s10489-023-05144-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05144-9

Keywords

Search

Navigation