Skip to main content
SpringerLink
Log in

Multimodal collaborative graph for image recommendation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Recent works for personalized recommendation typically emphasize their efforts on learning users’ interests from interactions. However, users make decisions depending on multiple factors, especially various attributes of items like appearance, reviews, price, etc. Therefore, in the case of image recommendation, we strive to unveil users’ interests in a multimodal manner. In this work, we propose a multimodal collaborative graph (MCG) model for image recommendation, which builds users’ interests in both visual and collaborative signals. On visual modality, visual interest filtering is designed to explore the interest non-linearity of users’ interacted images. In the pairwise collaborative module, multi-hop interactions are embedded elaborately to encode the heterogeneous structure of user-image interactions by deep interest propagation. Both visual and collaborative signals are aggregated to embed users and items and match pairwise user-item for the following personalized recommendation. Experiments are conducted on three public real-world datasets. Further analysis demonstrates the compensation capability of visual and collaborative signals in mining users’ interests and verifies the effectiveness of the proposed MCG for image recommendation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Eppler M, Jeanne M (2004) The concept of information overload: a review of literature from organization science, accounting, marketing, MIS, and related disciplines. Inf Soc 20(5):325–344

    Article  Google Scholar 

  2. Niu W, Caverlee J, Lu H, et al. (2018) Neural personalized ranking for image recommendation. In: ACM international conference on web search and data mining, pp 423–431

  3. Qian X, Feng H, Zhao G, Mei T (2014) Personalized recommendation combining user interest and social circle. IEEE Trans Knowl Data Eng 26(7):1763–1777

    Article  Google Scholar 

  4. Wang X, Ji H, Shi C, et al. (2019) Heterogeneous graph attention network. In: International conference on world wide web

  5. He X, Liao L, Zhang H, Nie L, Hu X, Chua T (2017) Neural collaborative filtering. In: International conference on world wide web, pp 173–182

  6. He X, Zhang H, Kan M, Chua T (2016) Fast matrix factorization for online recommendation with implicit feedback. In: International ACM SIGIR conference on research and development in information retrieval, pp 549–558

  7. Wang X, He X, Wang M, et al. (2019) Neural graph collaborative filtering. In: International ACM SIGIR conference on research and development in information retrieval, pp 165–174

  8. Wu Y, Cao N, Gotz D, et al. (2016) A survey on visual analytics of social media data. IEEE Transactions on Multimedia 18(11):2135–2148

    Article  Google Scholar 

  9. Zhang H, Shen F, Liu W, et al. (2016) Discrete collaborative filtering. In: International ACM SIGIR conference on research and development in information retrieval, pp 325–334

  10. Yang Y, Xu Y, Wang E, et al. (2018) Improving existing collaborative filtering recommendations via serendipity-based algorithm. IEEE Transactions on Multimedia 20(7):1888–1900

    Article  Google Scholar 

  11. Pan R, Zhou Y, Cao B, Liu NN (2008) One-class collaborative filtering. In: International conference on data mining, pp 502–511

  12. Xue F, He X, Wang X, et al. (2019) Deep item-based collaborative filtering for top-N recommendation. ACM Trans Inf Syst 37(3):1–25

    Article  Google Scholar 

  13. He X, He Z, Song J, et al. (2018) NAIS: Neural Attentive item similarity model for recommendation. IEEE Trans Knowl Data Eng 30(12):2354–2366

    Article  Google Scholar 

  14. Kabbur S, Ning X, Karypis G (2013) Fism: Factored item similarity models for top-N recommender systems. In: The 19th ACM SIGKDD international conference on knowledge discovery and data mining, pp 659–667

  15. Jiang M, Cui P, Wang F, et al. (2012) Social recommendation across multiple relational domains. In: Conference on information and knowledge management, pp 1422–1431

  16. Chen T, He X, Kan M (2016) Context-aware image tweet modelling and recommendation. In: Acm on Multimedia Conference, pp 1018–1027

  17. Lian D, Ge Y, Zhang F, et al. (2018) Scalable content-aware collaborative filtering for location recommendation. IEEE Trans Knowl Data Eng 30(6):1122–1135

    Article  Google Scholar 

  18. Du X, Yin H, Chen L, et al. (2020) Personalized video recommendation using rich contents from videos. IEEE Trans Knowl Data Eng 32(3):492–505

    Article  Google Scholar 

  19. Geng X, Zhang HW, Bian JW, Chua TS (2015) Learning image and user features for recommendation in social networks. In: International conference on computer vision, pp 4274–4282

  20. He R, Mcauley J (2016) VBPR: Visual Bayesian personalized ranking from implicit feedback. In: National conference on artificial intelligence, pp 144–150

  21. Rendle S, Freudenthaler C, Gantner Z, Schmidt T (2009) BPR: Bayesian Personalized ranking from implicit feedback. In: The 25th conference on uncertainty in artificial intelligence, pp 452–461

  22. Zhang J, Yang Y, Zhuo L, et al. (2019) Personalized recommendation of social images by constructing a user interest tree with deep features and tag trees. IEEE Transactions on Multimedia 21(11):2762–2775

    Article  Google Scholar 

  23. Jian M, Jia T, Yang X, Wu L, Huo L (2019) Cross-modal collaborative manifold propagation for image recommendation. In: ACM SIGMM international conference on multimedia retrieval, pp 344–348

  24. Jian M, Jia T, Wu L, Zhang L, Wang D (2020) Content-based bipartite user-image correlation for image recommendation. Neural Processing Letters

  25. Yuan Z, Zhang J (2016) Feature extraction and image retrieval based on AlexNet. In: International conference on digital image processing, pp 1–5

  26. Rendle S, Freudenthaler C, Gantner Z, et al. (2009) BPR: Bayesian Personalized ranking from implicit feedback. Uncertainty in Artificial Intelligence, pp 452–461

  27. Hsieh C-K, Yang L, Cui Y, Lin T-Y, Belongie S, Estrin D (2017) Collaborative metric learning. In: International conference on world wide web, pp 193–201

  28. Kim D, Park C, Oh J, et al. (2016) Convolutional matrix factorization for document context-aware recommendation. ACM Recommender Systems, pp 233–240

  29. He X, Du X, Wang X, et al. (2018) Outer product based neural collaborative filtering. In: International joint conferences on artificial intelligence, pp 2227–2233

  30. Wu L, Chen L, Hong R, Fu Y, Xie X, Wang M (2019) A hierarchical attention model for social contextual image recommendation. IEEE Trans Knowl Data Eng, pp 1–1

  31. Den Berg RV, Kipf T, Welling M (2017) Graph convolutional matrix completion. In: International conference on world wide web

  32. Niepert M, Ahmed M, Kutzkov K, et al. (2017) Learning convolutional neural networks for graphs. In: International conference on machine learning, pp 2014–2023

  33. Wang H, Wang Y, Zhang Z, Fu X, Zhuo L, Xu M, Wang M (2021) Kernelized multiview subspace analysis by Self-Weighted learning. IEEE Transactions on Multimedia 23:3828–3840

    Article  Google Scholar 

  34. Wang H, Peng J, Zhao Y, Fu X (2020) Multi-path Deep CNNs for Fine-Grained Car Recognition. IEEE Trans Veh Technol 69(10):10484–10493

    Article  Google Scholar 

  35. Wang H, Peng J, Chen D, Jiang G, Zhao T, Fu X (2020) Attribute-Guided Feature learning network for vehicle reidentification. IEEE MultiMedia 27(4):112–121

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant NO. 62176011, NO. 61802011, and NO. 61976010, Beijing Municipal Education Committee Science Foundation under Grant NO. KM201910005024, Inner Mongolia Autonomous Region Science and Technology Foundation under Grant NO. 2021GG0333, and Beijing Postdoctoral Research Foundation under Grant NO. Q6042001202101.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Meng Jian, Jingjing Guo, Ge Shi & Lifang Wu

  2. Inner Mongolia Aerospace Power Machinery Testing Institute, Huhhot, 010076, China

    Zhangquan Wang

Authors
  1. Meng Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingjing Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ge Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lifang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhangquan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lifang Wu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jian, M., Guo, J., Shi, G. et al. Multimodal collaborative graph for image recommendation. Appl Intell 53, 560–573 (2023). https://doi.org/10.1007/s10489-022-03304-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03304-x

Keywords

Search

Navigation