Skip to main content

Reinforced KGs reasoning for explainable sequential recommendation

World Wide Web volume 25, pages 631–654 (2022)Cite this article

Abstract

We explore the semantic-rich structured information derived from the knowledge graphs (KGs) associated with the user-item interactions and aim to reason out the motivations behind each successful purchase behavior. Existing works on KGs-based explainable recommendations focus purely on path reasoning based on current user-item interactions, which generally result in the incapability of conjecturing users’ subsequence preferences. Considering this, we attempt to model the KGs-based explainable recommendation in sequential settings. Specifically, we propose a novel architecture called Reinforced Sequential Learning with Gated Recurrent Unit (RSL-GRU), which is composed of a Reinforced Path Reasoning Network (RPRN) component and a GRU component. RSL-GRU takes users’ sequential behaviors and their associated KGs in chronological order as input and outputs potential top-N items for each user with appropriate reasoning paths from a global perspective. Our RPRN features a remarkable path reasoning capacity, which is regulated by a user-conditioned derivatively action pruning strategy, a soft reward strategy based on an improved multi-hop scoring function, and a policy-guided sequential path reasoning algorithm. Experimental results on four of Amazon’s large-scale datasets show that our method achieves excellent results compared with several state-of-the-art alternatives.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. https://nijianmo.github.io/amazon/index.html

References

  1. Ai, Q., Azizi, V., Chen, X., Zhang, Y.: Learning heterogeneous knowledge base embeddings for explainable recommendation. Algorithms 11(9), 137 (2018)

    MathSciNet  Article  Google Scholar 

  2. Alkawsi, G.A., Ali, N., Baashar, Y.: An empirical study of the acceptance of iot-based smart meter in malaysia: The effect of electricity-saving knowledge and environmental awareness. IEEE Access 8, 42794–42804 (2020)

    Article  Google Scholar 

  3. Bianchi, F., Rossiello, G., Costabello, L., Palmonari, M., Minervini, P.: Knowledge graphs for explainable artificial intelligence: Foundations, applications and challenges. In: Studies on the Semantic Web, vol. 47, pp. 49–72. IOS Press (2020)

  4. Chen, X., Huang, C., Yao, L., Wang, X., Liu, W., Zhang, W.: Knowledge-guided deep reinforcement learning for interactive recommendation. In: 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020, pp. 1–8 (2020)

  5. Cui, Q., Wu, S., Liu, Q., Zhong, W., Wang, L.: Mv-rnn: A multi-view recurrent neural network for sequential recommendation. IEEE Trans. Knowl. Data Eng. 32(2), 317–331 (2016)

    Article  Google Scholar 

  6. Cui, Q., Wu, S., Liu, Q., Zhong, W., Wang, L.: MV-RNN: A multi-view recurrent neural network for sequential recommendation. IEEE Trans. Knowl. Data Eng. 32(2), 317–331 (2020)

    Article  Google Scholar 

  7. Dai, F., Gu, X., Li, B., Zhang, J., Qian, M., Wang, W.: Meta-graph based attention-aware recommendation over heterogeneous information networks. In: Computational Science - ICCS 2019 - 19th International Conference, Faro, Portugal, June 12-14, 2019, Proceedings, Part II, pp. 580–594 (2019)

  8. Gong, J., Wang, S., Wang, J., Feng, W., Peng, H., Tang, J., Yu, P.S.: Attentional graph convolutional networks for knowledge concept recommendation in moocs in a heterogeneous view. In: Proceedings of the 43rd International Conference on Research and Development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 79–88 (2020)

  9. He, X., An, B., Li, Y., Chen, H., Wang, R., Wang, X., Yu, R., Li, X., Wang, Z.: Learning to collaborate in multi-module recommendation via multi-agent reinforcement learning without communication. In: RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020, pp. 210–219 (2020)

  10. Heinz, S., Bracher, C., Vollgraf, R.: An lstm-based dynamic customer model for fashion recommendation. CEUR-WS.org 1922, 45–49 (2017)

    Google Scholar 

  11. Hu, B., Shi, C., Zhao, W.X., Yu, P.S.: Leveraging meta-path based context for top- N recommendation with A neural co-attention model. In: Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 1531–1540 (2018)

  12. Huang, X., Fang, Q., Qian, S., Sang, J., Li, Y., Xu, C.: Explainable interaction-driven user modeling over knowledge graph for sequential recommendation. In: Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019, pp. 548–556 (2019)

  13. Kolahkaj, M., Harounabadi, A., Nikravanshalmani, A., Chinipardaz, R.: A hybrid context-aware approach for e-tourism package recommendation based on asymmetric similarity measurement and sequential pattern mining. Electron. Commer. Res. Appl. 42, 100978 (2020)

    Article  Google Scholar 

  14. Li, L., Wang, P., Wang, Y., Jiang, J., Tang, B., Yan, J., Wang, S., Liu, Y.: Prtransh: Embedding probabilistic medical knowledge from real world emr data. arXiv:1909.00672(8) (2019)

  15. Ma, C., Ma, L., Zhang, Y., Sun, J., Liu, X., Coates, M.: Memory augmented graph neural networks for sequential recommendation. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 5045–5052 (2020)

  16. Palumbo, E., Rizzo, G., Troncy, R., Baralis, E., Osella, M., Ferro, E.: Knowledge graph embeddings with node2vec for item recommendation. In: The Semantic Web: ESWC 2018 Satellite Events, Heraklion, Crete, Greece, June 3-7, 2018, Revised Selected Papers, Lecture Notes in Computer Science, vol. 11155, pp. 117–120. Springer (2018)

  17. Qiang, C., Shu, W., Yan, H., Liang, W.: A hierarchical contextual attention-based gru network for sequential recommendation. Neurocomputing arXiv:1711.05114(8) (2017)

  18. Saito, T., Watanobe, Y.: Learning path recommendation system for programming education based on neural networks. IJDET 18(1), 36–64 (2020)

    Google Scholar 

  19. Shi, D., Wang, T., Xing, H., Xu, H.: A learning path recommendation model based on a multidimensional knowledge graph framework for e-learning. Knowl. Based Syst. 195, 105618 (2020)

    Article  Google Scholar 

  20. Song, W., Duan, Z., Yang, Z., Zhu, H., Zhang, M., Tang, J.: Explainable knowledge graph-based recommendation via deep reinforcement learning. arXiv:1906.09506, 13 (2019)

  21. Sun, Z., Huang, J., Hu, W., Chen, M., Guo, L., Qu, Y.: Transedge: Translating relation-contextualized embeddings for knowledge graphs. arXiv:2004.13579(17) (2020)

  22. Suzuki, T., Oyama, S., Kurihara, M.: Explainable recommendation using review text and a knowledge graph. In: 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, December 9-12, 2019, pp. 4638–4643. IEEE (2019)

  23. Tang, J., Wang, K.: Personalized top-n sequential recommendation via convolutional sequence embedding. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pp. 565–573. ACM (2018)

  24. Wang, B., Cai, W.: Knowledge-enhanced graph neural networks for sequential recommendation. Inf. 11(8), 388 (2020)

    Google Scholar 

  25. Wang, L., Wang, Y., Liu, B., He, L., Liu, S., de Melo, G., Xu, Z.: Link prediction by exploiting network formation games in exchangeable graphs. In: 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, May 14-19, 2017, pp. 619–626. IEEE (2017)

  26. Wang, L., Zhang, W., He, X., Zha, H.: Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 2447–2456. ACM (2018)

  27. Wang, P., Fan, Y., Xia, L., Zhao, W.X., Niu, S., Huang, J.: KERL: A knowledge-guided reinforcement learning model for sequential recommendation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 209–218 (2020)

  28. Wang, T., Shi, D., Wang, Z., Xu, S., Xu, H.: Mrp2rec: Exploring multiple-step relation path semantics for knowledge graph-based recommendations. IEEE Access 8, 134817–134825 (2020)

    Article  Google Scholar 

  29. Wang, X., He, X., Cao, Y., Liu, M., Chua, T.S.: Kgat: Knowledge graph attention network for recommendation. ACM 9(9), 950–958 (2019)

    Google Scholar 

  30. Wang, X., Wang, D., Xu, C., He, X., Cao, Y., Chua, T.: Explainable reasoning over knowledge graphs for recommendation. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, 8, pp. 5329–5336. AAAI Press (2019)

  31. Wang, X., Wang, Y., Hsu, D., Wang, Y.: Exploration in interactive personalized music recommendation: A reinforcement learning approach. Acm Trans. Multimed. Comput. Commun. Appl. 11(1), 1–22 (2013)

    Google Scholar 

  32. Wang, Z., Li, Y., Fang, L., Chen, P.: Joint knowledge graph and user preference for explainable recommendation. In: 2019 IEEE 5th International Conference on Computer and Communications (ICCC), pp. 1338–1342. IEEE (2019)

  33. Xian, Y., Fu, Z., Muthukrishnan, S., de Melo, G., Zhang, Y.: Reinforcement knowledge graph reasoning for explainable recommendation. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pp. 285–294. ACM (2019)

  34. Yang, C., Sun, M., Zhao, W.X., Liu, Z., Chang, E.Y.: A neural network approach to joint modeling social networks and mobile trajectories. Acm Trans. Inf. Syst. 35(4), 36 (2016)

    Google Scholar 

  35. Yu, X., Ren, X., Sun, Y., Sturt, B., Khandelwal, U., Gu, Q., Norick, B., Han, J.: Recommendation in heterogeneous information networks with implicit user feedback. In: Seventh ACM Conference on Recommender Systems, RecSys ’13, Hong Kong, China, October 12-16, 2013, pp. 347–350 (2013)

  36. Yuan, W., Wang, H., Yu, X., Liu, N., Li, Z.: Attention-based context-aware sequential recommendation model. Inf. Sci. 510, 122–134 (2020)

    Article  Google Scholar 

  37. Zhang, F., Yuan, N.J., Lian, D., Xie, X., Ma, W.: Collaborative knowledge base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 353–362. ACM (2016)

  38. Zhang, J., Gao, M., Yu, J., Yang, L., Wang, Z., Xiong, Q.: Path-based reasoning over heterogeneous networks for recommendation via bidirectional modeling. arXiv:2008.04185 (2020)

  39. Zhao, B., Xu, Z., Tang, Y., Li, J., Liu, B., Tian, H.: Effective knowledge-aware recommendation via graph convolutional networks. In: Web Information Systems and Applications - 17th International Conference, WISA 2020, Guangzhou, China, September 23-25, 2020, Proceedings, pp. 96–107 (2020)

  40. Zhao, H., Yao, Q., Li, J., Song, Y., Lee, D.L.: Meta-graph based recommendation fusion over heterogeneous information networks. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13- 17, 2017, pp. 635–644. ACM (2017)

  41. Zhao, K., Wang, X., Zhang, Y., Zhao, L., Liu, Z., Xing, C., Xie, X.: Leveraging demonstrations for reinforcement recommendation reasoning over knowledge graphs. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 239–248 (2020)

  42. ZHAO, Z., ZHU, M., SHENG, Y., WANG, J.: A top-n-balanced sequential recommendation based on recurrent network. Ieice Trans. Inf. Syst. 102 (4), 737–744 (2019)

    Article  Google Scholar 

  43. Zheng, L., Tianlong, Z., Huijian, H., Caiming, Z.: Personalized tag recommendation based on convolution feature and weighted random walk. Int. J. Comput. Intell. Syst. 13(1), 24–35 (2020)

    Article  Google Scholar 

  44. Zhu, R., Yang, D., Li, Y.: Learning improved semantic representations with tree-structured lstm for hashtag recommendation: An experimental study. Information 10(4), 127 (2019)

    Article  Google Scholar 

  45. Zhu, H., Tian, F., Wu, K., Shah, N., Chen, Y., Ni, Y., Zhang, X., Chao, K.M., Zheng, Q.: A multi-constraint learning path recommendation algorithm based on knowledge map. Knowl. Based Syst. 143(MAR.1), 102–114 (2018)

    Article  Google Scholar 

  46. Zhu, Q., Zhou, X., Song, Z., Tan, J., Guo, L.: DAN: deep attention neural network for news recommendation. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, pp. 5973–5980. AAAI Press (2019)

  47. Zhu, Q., Zhou, X., Wu, J., andLi Guo, J.T.: A knowledge-aware attentional reasoning network for recommendation. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, 8, pp. 6999–7006. AAAI Press (2020)

Download references

Acknowledgements

The authors would like to acknowledge the support provided by the National Natural Science Foundation of China under Grant 61872222, the Natural Science Foundation of Shandong Province (ZR2020LZH011), the Young Scholars Program of Shandong University, and the ARC Discovery Project (Grant No. DP200101374, LP170100891, and DP190101985).

Author information

Authors and Affiliations

  1. School of Software, Shandong University, Jinan, China

    Zhihong Cui, Lizhen Cui & Shijun Liu

  2. University of Technology Sydney, Ultimo, Australia

    Hongxu Chen & Guandong Xu

  3. School of Computer Science and Technology, Jilin University, Changchun, 130012, China

    Xueyan Liu

  4. The University of Queensland, Brisbane, Australia

    Hongzhi Yin

Authors
  1. Zhihong Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongxu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lizhen Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shijun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xueyan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guandong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hongzhi Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hongxu Chen or Shijun Liu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on Large Scale Graph Data Analytics

Guest Editors: Xuemin Lin, Lu Qin, Wenjie Zhang, and Ying Zhang

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cui, Z., Chen, H., Cui, L. et al. Reinforced KGs reasoning for explainable sequential recommendation. World Wide Web 25, 631–654 (2022). https://doi.org/10.1007/s11280-021-00902-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-021-00902-6

Keywords

  • Reinforcement learning
  • Sequential recommendation
  • Path reasoning
  • Knowledge graphs