Synthetic Datasets and Evaluation Tools for Inductive Neural Reasoning

Abstract

Logical rules are a popular knowledge representation language in many domains. Recently, neural networks have been proposed to support the complex rule induction process. However, we argue that existing datasets and evaluation approaches are lacking in various dimensions; for example, different kinds of rules or dependencies between rules are neglected. Moreover, for the development of neural approaches, we need large amounts of data to learn from and adequate, approximate evaluation measures. In this paper, we provide a tool for generating diverse datasets and for evaluating neural rule learning systems, including novel performance metrics.

C. Cornelio and V. Thost: Equal contribution. The work was partly conducted while C.C. was at IBM Research.

Appendices

A Additional Preliminaries

Our measures are based on the concepts of Herbrand base and structure. Generally formal logic distinguishes between syntax (symbols) and its interpretation. Herbrand structures, which are used for interpretation, are however defined solely by the syntactical properties of the vocabulary. The idea is to directly take the symbols of terms as their interpretations which, for example, has proven useful in the analysis of logic programming.

The focus is on ground terms (e.g., atoms and rules), which are terms that do not contain variables.

The Herbrand base of a FOL vocabulary (of constants, predicates, etc.) is the set of all ground terms that can be formulated in the vocabulary. If the vocabulary does not contain constants, then the language is extended by adding an arbitrary new constant.

A Herbrand model interprets terms over a Herbrand base, hence it can be seen as a set of ground atoms. Let T be the set of all variable-free terms over a vocabulary V. A structure S over V and with base U is said to be a Herbrand structure iff $U=T_0$ and $c^S$ = c for every constant $c\in V$.¹³

B Dataset Generation

In this section, we describe the generation process of the rules and facts in detail, assuming the generator parameters (also configuration) listed in Sect. 3 to be set.

Preprocessing. As already mentioned, most parameters are determined randomly in a preprocessing step if they are not fixed in the configuration, such as the symbols that will be used, the numbers of DGs to be generated, and their depths. However, all random selections are within the bounds given in the configuration under consideration; for instance, we ensure that the symbols chosen suffice to generate rule graphs and fact sets of selected size and that at least one graph is of the given maximal depth.

Rule Generation. According to the rule set category specified and graph depths determined, rules (nodes in the graphs) of form (2) are generated top down breadth first, for each of the rule graphs to be constructed. The generation is largely at random, that is, w.r.t. the number of child nodes of a node and which body atom they relate to; the number of atoms in a rule; and the predicates within the latter, including the choice of the target predicate (i.e., the predicate in the head of the root) in the very first step. RuDaS also offers the option that all graphs have the same target predicate. To allow for more derivations, we currently only consider variables as terms in head atoms; the choice of the remaining terms is based on probabilities as described in the following. Given the atoms to be considered (in terms of their number and predicates) and an arbitrary choice of head variables, we first determine a position for each of the latter in the former. Then we populate the other positions one after the other: a head variable is chosen with probability $p_h=\frac{1}{5}$; for one of the variables introduced so far, we have probability $p_v=(1-p_h) * \frac{3}{4}$; for a constant, $p_c=(1-p_h) * (1-p_v) * \frac{1}{10}$; and, for a fresh variable, $p_f=(1-p_h) * (1-p_v) * (1-p_c)$. While this conditional scheme might seem rather complex, we found that it works best in terms of the variety it yields; also, these probabilities can be changed easily.

Fact Generation. The fact generation is done in three phases: we first construct a set $\mathbb {D}$ of relevant facts in a closed-world setting, consisting of support facts $\mathbb {S}$ and their consequences $\mathbb {C}$, and then adapt it according to $n_{\text {OW}}$ and $n_{\text {Noise*}}$.

As it is the (natural) idea, we generate facts by instantiating the rule graphs multiple times, based on the assumption that rule learning systems need positive examples for a rule to learn that rule, and stop the generation when the requested number of facts has been generated. We actually stop later because we need to account for the fact that we subsequently will delete some of them according to $n_{\text {OW}}$. More specifically, we continuously iterate over all rule graphs, for each, select an arbitrary but fresh variable assignment $\sigma$, and then iterate over the graph nodes as described in the following, in a bottom-up way. First, we consider each leaf n and corresponding rule of form (2) and generate support facts $\sigma (\alpha _1),\dots ,\sigma (\alpha _m)$. Then, we infer the consequences based on the rules and all facts generated so far. For every node n on the next level and corresponding rule of form (2), we only generate those of the facts $\sigma (\alpha _1),\dots ,\sigma (\alpha _m)$ as support facts which are not among the consequences inferred previously. We then again apply inference, possibly obtaining new consequences, and continue iterating over all nodes in the graph in this way. We further diversify the process based on two integer parameters, $n_{\text {DG}}$ and $n_{\text {Skip}}$: in every $n_{\text {DG}}$-th iteration the graph is instantiated exactly in the way described; in the other iterations, we skip the instantiation of a node with probability 1/$n_{\text {Skip}}$ and, in the case of DR-DGs, only instantiate a single branch below disjunctive nodes. We implemented this diversification to have more variability in the supports facts, avoiding to have only complete paths from the leaves to the root.

In the open-world setting, we subsequently construct a set $\mathbb {D} _{\text {OW}}$ by randomly deleting consequences from $\mathbb {D}$ according to the open-world degree given: assuming $\mathbb {T} \subseteq \mathbb {C}$ to be the set of target facts (i.e., consequences containing the target predicate), we remove $n_{\text {OW}} \%$ from $\mathbb {C} \setminus \mathbb {T}$, and similarly $n_{\text {OW}} \%$ from $\mathbb {T}$. In this way, we ensure that the open-world degree is reflected in the target facts. Though, there is the option to have it more arbitrary by removing $n_{\text {OW}} \%$ from $\mathbb {C}$ instead of splitting the deletion into two parts.

The noise generation is split similarly. Specifically, we construct a set $\mathbb {D} _{\text {OW+Noise}}$ based on $\mathbb {D} _{\text {OW}}$ by arbitrarily removing $n_{\text {Noise-}} \%$ from $\mathbb {S}$, and by adding arbitrary fresh facts that are neither in $\mathbb {C}$ (i.e., we do not add facts we have removed in the previous step) nor contain the target predicate such that $\mathbb {D} _{\text {OW+Noise}} \setminus \mathbb {T}$ contains $n_{\text {Noise+}} \%$ of noise. In addition, we add arbitrary fresh facts on the target predicate that are not in $\mathbb {T}$ already such that subset of $\mathbb {D} _{\text {OW+Noise}}$ on that predicate finally contains $n_{\text {Noise+}} \%$ of noise.

Output. The dataset generation produces: the rules; a training set ($\mathbb {D} _{\text {OW+Noise}}$), which is of the requested size, and fulfills $n_{\text {OW}}$, $n_{\text {Noise+}}$, and $n_{\text {Noise-}}$; and custom fact sets $\mathbb {S} '$ and $\mathbb {C} '$ for our evaluation tools generated in the same way as $\mathbb {S}$ and $\mathbb {C}$. For further experiments, RuDaS also outputs $\mathbb {D}$, $\mathbb {D} _{\text {Noise}}$ (an adaptation of $\mathbb {D}$ which contains noise but all of $\mathbb {C}$), $\mathbb {D} _{\text {OW}}$, $\mathbb {S}$, and $\mathbb {C}$ (see also the end of Sect. C).

Table 5. Overview of our generated datasets, altogether 196; column # is the count of datasets described in the corresponding row. All other numbers are averages. For Chain, we have $n_{\text {OW}} =0.3$, $n_{\text {Noise-}} =0.2$, and $n_{\text {Noise+}} =0.1$. For RDG and DRDG: $n_{\text {OW}} \in \{0.2,0.3,0.4\}$, $n_{\text {Noise-}} \in \{0.15,0.2,0.3\}$, and $n_{\text {Noise+}} \in \{0.1,0.2,0.3\}$. Note that the size bounds of our fact sets are not strict, some sizes are slightly larger than expected (e.g., 1065 for size M) because our initial generation needs to take into account that some facts, e.g., consequences, may be removed thereafter.

C Statistics of RuDaS .v0

Table 5 provides statistics regarding RuDaS .v0: the generated set of datasets available to the community in our repository.

D Approaches to Rule Learning

Classical ILP systems such as FOIL [24] and Progol [19] usually apply exhaustive algorithms to mine rules for the given data and either require false facts as counter-examples or assume a closed world (for an overview of classical ILP systems see Table 2 in [32]). The closed-world assumption (CWA) (vs. open world assumption or OWA) states that all facts that are not explicitly given as true are assumed to be false.

Today, however, knowledge graphs with their often incomplete, noisy, heterogeneous, and, especially, large amounts of data raise new problems and require new solutions. For instance, real data most often only partially satisfies the CWA and does not contain counter-examples. Moreover, in an open world, absent facts cannot be considered as counter-examples either, since they are not regarded as false. Therefore, successor systems, with AMIE+ [12] and RDF2Rules [34] as the most prominent representatives, assume the data to be only partially complete and focus on rule learning in the sense of mining patterns that occur frequently in the data. Furthermore, they implement advanced optimization approaches that make them applicable in wider scenarios. In this way, they address already many of the issues that arise with today’s knowledge graphs, still maintaining their processing exhaustive.

Recently, neural rule learning approaches have been proposed: [4, 10, 17, 22, 27, 35]. These methodologies seem a promising alternative considering that deep learning copes with vast amounts of noisy and heterogeneous data. The proposed solutions consider vector or matrix embeddings of symbols, facts and/or rules, and model inference using differentiable operations such as vector addition and matrix composition. However, they are still premature: they only learn certain kinds of rules or lack scalability (e.g., searching the entire rule space) and hence cannot compete with established rule mining systems such as AMIE+ yet, as shown in [22], for example.

E System Configurations

All the systems have the same computational restrictions (i.e. CPU, memory, time limit, etc.). The reader can find all the details (scripts etc.) in the RuDaS GitHub repository.

FOIL

• Paper: Learning logical definitions from relations. Machine Learning, 5:239-266, 1990.

  https://www.semanticscholar.org/paper/Learning-logical-definitions-from-relations-Quinlan/554f3b32b956035fbfabba730c6f0300d6955dce

• Source Code:

  http://www.rulequest.com/Personal/ or

  http://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/areas/learning/systems/foil/foil6/0.html, Version: 6

• Running configuration:

  

  -m 200000: used when the max tuples are exceeded

• Parameter for accepting the rules: NA – all the rules are accepted

Amie+

• Paper: Fast Rule Mining in Ontological Knowledge Bases with AMIE+. Luis Galárraga, Christina Teflioudi, Fabian Suchanek, Katja Hose. VLDB Journal 2015. https://suchanek.name/work/publications/vldbj2015.pdf

• Source Code:

  https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/amie/, Version of 2015-08-26

• Running configuration:

  

• Parameter for accepting the rules: learned using grid-search $=0.7$ – all the rules with PCA Confidence $> 0.7$ are accepted

Neural-LP

• Paper: Differentiable Learning of Logical Rules for Knowledge Base Reasoning.Fan Yang, Zhilin Yang, William W. Cohen. NIPS 2017. https://arxiv.org/abs/1702.08367

• Source Code:

  https://github.com/fanyangxyz/Neural-LP

• Running configuration:

  

• Parameter for accepting the rules: learned using grid-search $=0.6$ – all the rules with ri-normalized prob $> 0.6$ are accepted

Neural-theorem prover (ntp)

• Paper: End-to-end Differentiable Proving. Tim Rocktaeschel and Sebastian Riedel. NIPS 2017. http://papers.nips.cc/paper/6969-end-to-end-differentiable-proving

• Source Code:

  https://github.com/uclmr/ntp

• Running configuration:

  

• Parameter for accepting the rules: learned using grid-search $=0.0$ – all the rules are accepted