Reduced implication-bias logic loss for neuro-symbolic learning

Abstract

Integrating logical reasoning and machine learning by approximating logical inference with differentiable operators is a widely used technique in the field of Neuro-Symbolic Learning. However, some differentiable operators could introduce significant biases during backpropagation, which can degrade the performance of Neuro-Symbolic systems. In this paper, we demonstrate that the loss functions derived from fuzzy logic operators commonly exhibit a bias, referred to as Implication Bias. To mitigate this bias, we propose a simple yet efficient method to transform the biased loss functions into Reduced Implication-bias Logic Loss (RILL). Empirical studies demonstrate that RILL outperforms the biased logic loss functions, especially when the knowledge base is incomplete or the supervised training data is insufficient.

Appendices

Appendix A: Discussion with other kinds of fuzzy operators

In this section, we investigate different types of fuzzy operators and experimentally validate their effectiveness while maintaining the implication biased. When selecting fuzzy operators for NeSy, it is crucial to consider their smoothness and ease of optimization. Therefore, we concentrate on commonly used fuzzy operators that possess these desirable properties. Fuzzy operators that do not have a smooth gradient will be disregarded in our analysis.

1.1 A.1 Analysis

Sigmoidal van Krieken et al. (2022) proposed an operator which is smoothed Reichenbach operator by a sigmoid function. This implication likelihood function is defined as follows:

$$\begin{aligned} \sigma _{I}(p,q)= \frac{1+e^{-s\left( 1+b_{0}\right) }}{e^{-b_{0} s}-e^{-s\left( 1+b_{0}\right) }} \cdot \left( \left( 1+e^{-b_{0} s}\right) \cdot \sigma \left( s \cdot \left( I(p,q)+b_{0}\right) \right) -1\right) , \end{aligned}$$

where $s>0,b_0\in {\mathbb {R}},\sigma (\cdot ) = \frac{1}{1+e^x}$ denotes the sigmoid function. Substituting $d=\frac{1+e^{-s\left( 1+b_{0}\right) }}{e^{-b_{0} s}-e^{-s\left( 1+b_{0}\right) }}, h =(1+e^{-s\cdot b_0}),f=\sigma \left( s \cdot \left( I(p,q)+b_{0}\right) \right)$, we find:

$$\begin{aligned} \frac{\partial \sigma _{I}(p,q)}{\partial I(p,q)}=d \cdot h \cdot s \cdot f\cdot (1-f)>0. \end{aligned}$$

That is to say when I(p, q) is $\delta$-Confidence Monotonic, $\sigma _I(p,q)$ will be $\delta$-Confidence Monotonic too. This means logic loss derived from $\sigma _I$ will become implication biased.

Łukasiewicz Łukasiewicz implication likelihood (Cignoli, 2007) was defined as follows:

$$\begin{aligned} I_{LK}(p,q) = \min (1-p+q,1). \end{aligned}$$

It is easy to calculate the gradient of this likelihood, $\frac{\partial I_{LK}}{\partial p} = -1 \cdot {\mathbb {I}}[p>q]$. It turns out this is also implication biased.

G$\ddot{\text {o}}$ del G$\ddot{\text {o}}$del implication likelihood (Paad, 2016) was defined as follows:

$$\begin{aligned} I_{G}(p,q) = \max (1 - p, q). \end{aligned}$$

Also, the gradient of this likelihood $\frac{\partial I_{G}}{\partial p} = -1 \cdot {\mathbb {I}}[p+q<1]$ indicates this operator is implication biased.

Nilpotent Nilpotent implication likelihood (Gerla & Rovere, 2011) was defined as follows:

$$\begin{aligned} I_{N}(p,q) = \left\{ \begin{aligned}&1,&\textrm{if}\quad p\le q. \\&\max (1-p,q),&\textrm{otherwise}. \end{aligned} \right. \end{aligned}$$

Also, the gradient of this likelihood $\frac{\partial I_{N}}{\partial p} = -1 \cdot {\mathbb {I}}[(p+q<1) \& (p >q)]$ indicates this operator is implication biased.

1.2 A.2 Empirical study

Fig. 7

Verification of implication bias across various fuzzy operators is crucial. The empirical evidence illustrated in the preceding images substantiates our assertion that this phenomenon is pervasive

In this section, we present the empirical results of the above-analyzed operators for validation, with a particular emphasis on incomplete knowledge bases (especially Add-MNIST) and insufficient labeled data (especially Add-CIFAR10) scenarios. All experiments follow the same settings used in Sect. 6.1.1.

As shown in Fig. 7, implication bias significantly harms the performance of the model, especially when the knowledge base is incomplete or the amount of supervised information is not sufficient, which further supports our above analysis.

Appendix B: Discussion about Clark’s completion

RILL and Clark’s completion both aim to reduce the uncertainty of negative information. While RILL does not alter the information in the knowledge base, but instead reduces the importance of weak samples, causing the model to pay more attention to samples that are more relevant to the given rule.

Fig. 8

Performance of Clark’s Completion drops when the completeness of KB is decreasing

Surprisingly, explicitly applying Clark’s completion in a NeSy system may not be helpful. There are two reasons behind this claim.

First, a explicitly Clark’s completion need to replace $\rightarrow$ to $\leftrightarrow$. For example:

$$\begin{aligned} A\leftarrow A_1, A\leftarrow A_2,\cdots , A\leftarrow A_n, \end{aligned}$$

will be replaced as $A\leftrightarrow (A_1\vee A_2\vee \cdots \vee A_n)$.

However, the fuzzy operator is unsuitable for approximating a rule with many atoms (Marra et al., 2021). An example is the n-ary Łukasiewicz strong disjunction $F_{\vee }(x_1,\cdots ,x_n) = \min (1,x_1+\cdots +x_n)$. Although all $x_i$ can be very small, this approximation of disjunction will give a value near 1. Because Clark’s completion will increase the number of atoms in the logical rule, the knowledge base may suffer from this problem after completion.

Second, in a NeSy system, if the knowledge base is incomplete, replace $\rightarrow$ to $\leftrightarrow$ will change the information in the knowledge base, which may induce wrong information. In logic programming, Clark’s completion will not change the soundness of the system, while in the NeSy setting with a data-driven approach, it may not be promised.

Here we adopt the same experimental setting of incomplete knowledge base case in Sect. 6.1.1 to validate the performance of Clark’s Completion. As depicted in Fig. 8, when the knowledge base becomes incomplete, completion of the knowledge base will not help improve the model’s performance because it introduces wrong information.

Appendix C: Sensitivity analysis

Both $\text {RILL}_{hinge}$ and $\text {RILL}_{l2+hinge}$ contain a hyper-parameter $\epsilon$ in their definition. In this section, we investigate the sensitivity of $\epsilon$ and its impact on the model’s accuracy in the Add-MNIST experiment when the knowledge base incompleteness is 40%. Figure 9 displays the relationship between $\epsilon$ and the model’s accuracy.

The results indicate that when the threshold $\epsilon$ decreases to a certain value (in Fig. 9, it is 0.001), the model’s performance drops, suggesting the existence of a sensitive region that could be related to weak samples in addressing the problem of implication bias. When the threshold $\epsilon$ is above this value, the performance of RILL is stable but slightly decreasing. This decrease may be due to the increased number of weak samples, which results in the loss of some useful information.

Fig. 9

Sensitivity Analysis of the hyper-parameter $\epsilon$

Appendix D: Details of experiments

In this section, we will provide more details of our experiments.

Implementation of Semantic Loss The formal definition of Semantic Loss requires the conversion of CNF (conjunctive normal form) into SDD (sentential decision diagram), which can become impractical when dealing with knowledge bases that use a large number of predicates and complex rules due to the heavy computational burden. To address this issue, we propose an alternative approach where each rule in the knowledge base is converted separately into wmc (weighted model counting) models and then combined at the end. Although explicit repetitive terms are reduced during the combining phase, detecting implicit repetitive terms is computationally expensive, and thus they remain unchanged. This approach enables us to implement Semantic Loss efficiently while still accounting for the complexity of the knowledge base.

However, even with this optimization, the cost of using Semantic Loss is still much higher than that of using RILL, with a cost that is around three times higher. As a result, Semantic Loss may not be practical to use in many cases, particularly when dealing with large or complex knowledge bases.

Details of Task 1 The backbone for both MNIST and FashionMNIST datasets is a three-layer Multilayer Perceptron (MLP) with a width of each layer being [256,512,10], and the activation function is Rectified Linear Unit (ReLU) (Maas et al., 2013). In contrast, the backbone for CIFAR10 is ResNet9 (He et al., 2016). The value of $\lambda$ for both Fuzzy and RILL logic loss is 0.7. For Semantic Loss, we sample $\lambda$ from $\{0.001,0.005,0.01,0.05,0.1,0.5\}$ and choose the best one, which is 0.5 in this experiment. The learning rate is set to 0.0001, with a decay rate of 0.7. The learning rate scheduler is set to StepLR, with a decay step of 60. The optimizer used is AdamW as default, and the weight decay rate is set to 5e-4.

Details of Task 2 For this task, we select WideResNet-28-8 (Zagoruyko & Komodakis, 2016) as the backbone architecture with two classification heads, one for class classification, and the other for super-class classification. The value of $\lambda$ for both Fuzzy and RILL logic loss is 0.002, and for Semantic Loss, we still choose 0.5 as the default value. The learning rate is set to 0.005, with a decay rate of 0.9 and decay steps equal to 45. The learning rate scheduler is set to StepWithWarmUp, and the warm-up epoch is set as 5. The optimizer used is set as default, and momentum is set to 0.9.