Simple and automated negative sampling for knowledge graph embedding

Abstract

Negative sampling, which samples negative triplets from non-observed ones in knowledge graph (KG), is an essential step in KG embedding. Recently, generative adversarial network (GAN) has been introduced in negative sampling. By sampling negative triplets with large gradients, these methods avoid the problem of vanishing gradient and thus obtain better performance. However, they make the original model more complex and harder to train. In this paper, motivated by the observation that negative triplets with large gradients are important but rare, we propose to directly keep track of them with the cache. In this way, our method acts as a “distilled” version of previous GAN-based methods, which does not waste training time on additional parameters to fit the full distribution of negative triplets. However, how to sample from and update the cache are two critical questions. We propose to solve these issues by automated machine learning techniques. The automated version also covers GAN-based methods as special cases. Theoretical explanation of NSCaching is also provided, justifying the superior over fixed sampling scheme. Besides, we further extend NSCaching with skip-gram model for graph embedding. Finally, extensive experiments show that our method can gain significant improvements on various KG embedding models and the skip-gram model and outperforms the state-of-the-art negative sampling methods.

A Appendix: Proof of Theorem 1

Proof

Following [68], we consider a more general optimization formulation as $\min _{\mathbf {w}} \bar{F}(\mathbf {w}) \equiv \frac{1}{n} \sum \nolimits _{i = 1}^n \phi _{i}(\mathbf {w}) + \lambda r(\mathbf {w})$, which covers (11) as a special case with $\lambda = 0$. Then, the stochastic training gives $\mathbf {w}^{t + 1}$ as

$$\begin{aligned} \mathbf {w}^{t + 1}&=\! \arg \min _{\mathbf {w}} \left[ (np_{i_t}^t)^{-1}\mathbf {w}^{\top } \nabla \phi _{i_t}(\mathbf {w}^t) \! + \! \lambda r(\mathbf {w}) \! \right. \nonumber \\&\left. + \! \frac{1}{\eta _t} \mathscr {B}_{\psi }(\mathbf {w}, \mathbf {w}^t) \right] \!,\! \end{aligned}$$

(18)

where $\mathscr {B}_{\psi }$ is a Bregman distance function measuring the difference between $\mathbf {w}$ and $\mathbf {w}^t$. Based on (18), we state Theorem 3 in [68] as

Theorem 2

([68]) Let $\mathbf {w}^t$ be generated from (18). Assume $\mathscr {B}_{\psi }$ is $\sigma$-strongly convex w.r.t. a norm $\Vert \cdot \Vert$, $\bar{F}$ and r are convex, if $\eta _t = \eta$, the following inequality holds for any $T \ge 1$

$$\begin{aligned}&\frac{1}{T}\sum \nolimits _{t =1}^T \mathbb {E}[\bar{F}(\mathbf {w}^t)] - \mathbb E[\bar{F}(\mathbf {w}^*)] \\&\le \frac{1}{T}\left[ \frac{1}{\eta } \mathscr {B}_{\psi }(\mathbf {w}^*, \mathbf {w}^1) + \frac{\eta }{2\sigma } \sum \nolimits _{t = 1}^T \mathbb {E} \Vert \nicefrac {\nabla \phi _{i_t}(\mathbf {w}^t)}{ n \mathbf {p}_{i_t}^t }\Vert ^2 \right] \end{aligned}$$

where the expectation is take with distribution $\mathbf {p}^t$.

In our Algorithm 5, we do not have r and thus $\lambda = 0$ in (11). Besides, $\mathscr {B}_{\psi }(\mathbf {w}, \mathbf {w}^t) = \frac{1}{2} \Vert \mathbf {w} - \mathbf {w}^t \Vert ^2$ in our case; thus, $\sigma = 1$. Above all, we have Theorem 1 which is derived from Theorem 2. $\square$