Convolutional Complex Knowledge Graph Embeddings

Abstract

We investigate the problem of learning continuous vector representations of knowledge graphs for predicting missing links. Recent results suggest that using a Hermitian inner product on complex-valued embeddings or convolutions on real-valued embeddings can be effective means for predicting missing links. We bring these insights together and propose ConEx—a multiplicative composition of a 2D convolution with a Hermitian inner product on complex-valued embeddings. ConEx utilizes the Hadamard product to compose a 2D convolution followed by an affine transformation with a Hermitian inner product in $\mathbb {C}$. This combination endows ConEx with the capability of (1) controlling the impact of the convolution on the Hermitian inner product of embeddings, and (2) degenerating into ComplEx if such a degeneration is necessary to further minimize the incurred training loss. We evaluated our approach on five of the most commonly used benchmark datasets. Our experimental results suggest that ConEx outperforms state-of-the-art models on four of the five datasets w.r.t. Hits@1 and MRR even without extensive hyperparameter optimization. Our results also indicate that the generalization performance of state-of-the-art models can be further increased by applying ensemble learning. We provide an open-source implementation of our approach, including training and evaluation scripts as well as pretrained models (github.com/dice-group/Convolutional-Complex-Knowledge-Graph-Embeddings).

Appendix

Statistical Hypothesis Testing. We carried out a Wilcoxon signed-rank test to check whether our results are significant. Our null hypothesis was that the link prediction performances of ConEx, ComplEx and ConvE come from the same distribution. The alternative hypothesis was correspondingly that these results come from different distributions. To perform the Wilcoxon signed-rank test (two-sided), we used the differences of the MRR, Hits@1, Hits@3, and Hits@10 performances on WN18RR, FB15K-237 and YAGO3-10. We performed two hypothesis tests between ConEx and ComplEx as well as between ConEx and ConvE. In both tests, we were able to reject the null hypothesis with a p-value $< 1\%$. Ergo, the superior performance of ConEx is statistically significant.

Ablation Study. We conducted our ablation study in a fashion akin to [9]. Like [9], we evaluated 2 different parameter initialisations to compute confidence intervals that is defined as $\bar{x}\pm 1.96 \cdot \frac{s}{\sqrt{n}}$, where $\bar{x}= \frac{1}{n} \sum _i ^n x_i$ and $s=\sqrt{\frac{\sum _i ^n (x_i - \bar{x} )^2}{n}}$, respectively. Hence, the mean and the standard deviation are computed without Bessel’s correction. Our results suggest that the initialization of parameters does not play a significant role in the link performance of ConEx. The dropout technique is the most important component in the generalization performance of ConEx. This is also observed in [9]. Moreover, replacing the Adam optimizer with the RMSprop optimizer [32] leads to slight increases in the variance of the link prediction results. During our ablation experiments, we were also interested in decomposing ConEx through removing $\text {conv}(\cdot ,\cdot )$, after ConEx is trained with it on benchmark datasets. By doing so, we aim to observe the impact of a 2D convolution in the computation of scores. Table 9 indicates that the impact of $\text {conv}(\cdot ,\cdot )$ differs depending on the input knowledge graph. As the size of the input knowledge graph increases, the impact of $\text {conv}(\cdot ,\cdot )$ on the computation of scores of triples increases.

Table 8. Ablation study for ConEx on FB15K-237. dp and ls denote the dropout technique and the label smoothing technique, respectively.

Table 9. Link prediction results on benchmark datasets. $\textsc {ConEx}^-$ stands for removing $\text {conv}(\cdot ,\cdot )$ in ConEx during the evaluation.

Link Prediction Results on WN18 and FB15K. Table 10 reports link prediction results on the WN18 and FB15K benchmark datasets.

Table 10. Link prediction results on WN18 and FB15K obtained from [4, 38].