Counterfactual-Guided and Curiosity-Driven Multi-hop Reasoning over Knowledge Graph

Abstract

Recently, multi-hop reasoning over incomplete Knowledge Graphs (KGs) to predict missing facts has attracted widespread attention due to its desirable effectiveness and interpretability. It typically adopts the Reinforcement Learning (RL) framework and traverses over the KG to reach the target answer and find evidential paths. However, existing methods often give all reached paths equal hit rewards. Intuitively, not all paths have the same contribution to the proof of the reasoning process. Moreover, the severely sparse rewards obtained after a multi-step traversal are usually insufficient to encourage a sophisticated RL-based model to work well. In order to tackle the above two problems, we propose a novel Counterfactual-guided and Curiosity-driven Knowledge Graph multi-hop Reasoning model (CoCuKGR). CoCuKGR constructs counterfactual relation reasoning tasks to estimate the semantic contribution to the query relation of each path and give each arrival path a different soft reward that can distinguish its validity. In addition, our method leverages the curiosity mechanism to generate curiosity-driven intrinsic rewards, which can not only alleviate the reward sparsity issue but also drive the agent to explore the environment more thoroughly to find more abundant paths. Experimental results show that our proposed model outperforms existing multi-hop reasoning methods significantly.

Supported by the National Key R&D Program of China under Grant Nos. 2021ZD0112501 and 2021ZD0112502; the National Natural Science Foundation of China under Grant Nos. 62172185 and 61876069; Jilin Province Key Scientific and Technological Research and Development Project under Grant Nos. 20180201067GX and 20180201044GX; and Jilin Province Natural Science Foundation under Grant No. 20200201036JC.