Skip to main content
SpringerLink
Log in

Neural attention model for recommendation based on factorization machines

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In recommendation systems, it is of vital importance to comprehensively consider various aspects of information to make accurate recommendations for users. When the low-order feature interactions between items are insufficient, it is necessary to mine information to learn higher-order feature interactions. In addition, to distinguish the different importance levels of feature interactions, larger weights should be assigned to features with larger contributions to predictions, and smaller weights to those with smaller contributions. Therefore, this paper proposes a neural attention model for recommendation (NAM), which deepens factorization machines (FMs) by adding an attention mechanism and fully connected layers. Through the attention mechanism, NAM can learn the different importance levels of low-order feature interactions. By adding fully connected layers on top of the attention component, NAM can model high-order feature interactions in a nonlinear way. Experiments on two real-world datasets demonstrate that NAM has excellent performance and is superior to FM and other state-of-the-art models. The results demonstrate the effectiveness of the proposed model and the potential of using neural networks for prediction under sparse data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Ricci F, Rokach L, Shapira B (2015) Recommender Systems Handbook. https://doi.org/10.1007/978-1-4899-7637-6

  2. Cheng HT, Koc L, Harmsen J, Shaked T, Chandra T, Aradhye H, Anderson G, Corrado G, Chai W, Ispir M (2016) Wide & Deep Learning for Recommender Systems

  3. Bobadilla J, Ortega F, Hernando A, Gutiérrez A (2013) Recommender systems survey. Knowledge Based Syst 46(0):109–132

    Article  Google Scholar 

  4. Lu J, Wu D, Mao M, Wang W, Zhang G (2015) Recommender system application developments: A survey. Decis Support Syst 74:12–32

    Article  Google Scholar 

  5. Liu G, Nguyen TT, Zhao G, Zha W, Yang J, Cao J, Wu M, Zhao P, Chen W (2016) Repeat buyer prediction for e-commerce. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp 155–164

  6. Rendle S (2010) Factorization machines. In: 2010 IEEE International Conference on Data Mining. IEEE, pp 995–1000

  7. Genzel M, Kutyniok G (2016) A mathematical framework for feature selection from real-world data with non-linear observations

  8. Lin Z, Feng M, Santos CNd, Yu M, Xiang B, Zhou B, Bengio Y (2017) A structured self-attentive sentence embedding. arXiv preprint arXiv:170303130

  9. He X, He Z, Song J, Liu Z, Jiang Y-G, Chua T-S (2018) Nais: neural attentive item similarity model for recommendation. IEEE Trans Knowl Data Eng 30(12):2354–2366

    Article  Google Scholar 

  10. Yuan W, Wang H, Yu X, Liu N, Li Z (2020) Attention-based context-aware sequential recommendation model. Inf Sci 510:122–134

    Article  Google Scholar 

  11. Xiao J, Ye H, He X, Zhang H, Wu F, Chua T-S (2017) Attentional factorization machines: learning the weight of feature interactions via attention networks. arXiv preprint arXiv:170804617

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778

  13. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. pp 1097–1105

  14. Hinton G, Deng L, Yu D, Dahl GE, A-r M, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Article  Google Scholar 

  15. Amodei D, Ananthanarayanan S, Anubhai R, Bai J, Battenberg E, Case C, Casper J, Catanzaro B, Cheng Q, Chen G (2016) Deep speech 2: end-to-end speech recognition in english and mandarin. In: International conference on machine learning. pp 173–182

  16. Guo H, Tang R, Ye Y, Li Z, He X (2017) DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:170304247

  17. He X, Chua T-S (2017) Neural factorization machines for sparse predictive analytics. In: Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. pp 355–364

  18. Shan Y, Hoens TR, Jiao J, Wang H, Yu D, Mao J (2016) Deep crossing: Web-scale modeling without manually crafted combinatorial features. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp 255–262

  19. Toledo RY, Martinez L (2017) Fuzzy tools in recommender systems: A survey. Int J Computation Intell Syst 10(1):776–803

    Article  Google Scholar 

  20. Zhang Z, Xu G, Zhang P, Wang Y (2017) Personalized recommendation algorithm for social networks based on comprehensive trust. Appl Intell 47(3):659–669

    Article  Google Scholar 

  21. Koren Y, Bell R, Volinsky C (2009) Matrix factorization techniques for recommender systems. Computer 42(8):30–37

    Article  Google Scholar 

  22. Zhang Z, Liu H (2015) Social recommendation model combining trust propagation and sequential behaviors. Appl Intell 43(3):695–706

    Article  Google Scholar 

  23. Zhang P, Zhang Z, Tian T, Wang Y (2019) Collaborative filtering recommendation algorithm integrating time windows and rating predictions. Appl Intell 49(8):3146–3157

    Article  Google Scholar 

  24. Juan Y, Zhuang Y, Chin W-S, Lin C-J (2016) Field-aware factorization machines for CTR prediction. In: Proceedings of the 10th ACM Conference on Recommender Systems, pp 43–50

  25. Hong L, Doumith AS, Davison BD (2013) Co-factorization machines: modeling user interests and predicting individual decisions in twitter. In: Proceedings of the sixth ACM international conference on Web search and data mining. pp 557–566

  26. He X, Liao L, Zhang H, Nie L, Hu X, Chua T-S (2017) Neural collaborative filtering. In: Proceedings of the 26th international conference on world wide web. pp 173–182

  27. Lian J, Zhou X, Zhang F, Chen Z, Xie X, Sun G (2018) xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp 1754–1763

  28. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:14090473

  29. Zhou G, Zhu X, Song C, Fan Y, Zhu H, Ma X, Yan Y, Jin J, Li H, Gai K (2018) Deep interest network for click-through rate prediction. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp 1059–1068

  30. Zhou C, Bai J, Song J, Liu X, Zhao Z, Chen X, Gao J (2017) ATRank: An attention-based user behavior modeling framework for recommendation

  31. Cheng Z, Ding Y, He X, Zhu L, Song X, Kankanhalli MS (2018) A3NCF: An adaptive aspect attention model for rating prediction. In: IJCAI. pp 3748–3754

  32. Song W, Shi C, Xiao Z, Duan Z, Xu Y, Zhang M, Tang J (2019) AutoInt: automatic feature interaction learning via self-attentive neural networks. Proceedings of the 28th ACM international conference on information and knowledge management. Association for Computing Machinery, Beijing. https://doi.org/10.1145/3357384.3357925

    Book  Google Scholar 

  33. Cao D, He X, Miao L, An Y, Yang C, Hong R (2018) Attentive group recommendation. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. pp 645–654

  34. Song X, Feng F, Han X, Yang X, Liu W, Nie L (2018) Neural compatibility modeling with attentive knowledge distillation. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. pp 5–14

  35. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:12070580

  36. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:150203167

  37. Baltrunas L, Church K, Karatzoglou A, Oliver N (2015) Frappe: understanding the usage and perception of mobile app recommendations in-the-wild. arXiv preprint arXiv:150503014

  38. Harper FM, Konstan JA (2015) The movielens datasets: history and context. Acm Trans Interact Intell Systems (tiis) 5(4):1–19

    Google Scholar 

  39. Rendle S (2012) Factorization machines with libfm. ACM Trans Intell Syst Technol (TIST) 3(3):1–22

    Article  Google Scholar 

  40. Blondel M, Fujino A, Ueda N, Ishihata M (2016) Higher-order factorization machines. In: Advances in Neural Information Processing Systems. pp 3351–3359

Download references

Acknowledgments

This paper is made possible thanks to the generous support from the Key Research and Development Program of Shandong Province (2018GGX106006, 2019GGX101068), Jinan Science and Technology Project (201704065), A Project of Shandong Province Higher Educational Science and Technology Program (J17KA070), PhD Funding Project of Shandong Jianzhu University (No. X19044Z), Science and Technology Program Project of Shandong Colleges and Universities (No.J17KA070), Undergraduate Education Reform Project of Shandong Province (No.Z2016M016, Z2016M014, M2018X197).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong Jianzhu University, Jinan, 250001, Shandong, China

    Peng Wen, Weihua Yuan, Qianqian Qin, Sheng Sang & Zhijun Zhang

Authors
  1. Peng Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weihua Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qianqian Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sheng Sang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhijun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhijun Zhang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, P., Yuan, W., Qin, Q. et al. Neural attention model for recommendation based on factorization machines. Appl Intell 51, 1829–1844 (2021). https://doi.org/10.1007/s10489-020-01921-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01921-y

Keywords

Search

Navigation