Select and calibrate the low-confidence: dual-channel consistency based graph convolutional networks

Abstract

Although Graph Convolutional Networks (GCNs) have achieved excellent results in various graph-related tasks, their performance at low label rates is still unsatisfactory. Previous studies in Semi-Supervised Learning (SSL) for graph primarily focused on utilizing network predictions to generate pseudo-labels or instruct message propagation, often resulting in incorrect predictions owing to over-confidence. To address this issue, we propose a novel approach called Dual-Channel Consistency based Graph Convolutional Networks (DCC-GCN) for semi-supervised node classification. The key idea behind DCC-GCN is to leverage the extraction of embeddings from both node features and topological structures by employing GCN encoders in two separate channels. We consider samples with consistent predictions from these two channels as high-confidence samples, while those with differing predictions are labeled as low-confidence samples. DCC-GCN calibrate the feature embeddings of low-confidence samples by aggregating high-confidence samples from their respective neighborhoods. DCC-GCN can significantly improve the classification accuracy of low-confidence samples, thus improving the overall accuracy. Experiments on seven graph datasets demonstrate that DCC-GCN outperforms prior SSL methods, improving node classification accuracy by a considerable margin.

Appendix A: A.1 Theorem 1 in section III

Proof

The inputs for the two GCN models are graphs $\mathcal {G}$ and $\mathcal {G}^{\prime }$, respectively, and the average accuracy of classification is $p_{1}$ and $p_{2}$, respectively. There are N nodes in graph $\mathcal {G}$ and $\mathcal {G}^{\prime }$. The number of samples with the same classification result for both GCN models $N_{a}$ is the number of samples $N_{r}$ correctly classified by both GCN models added to the number of samples $N_{w}$ incorrectly classified as the same class by the two models.

Fig. 7

Variation of the upper bound of $p_{GAIN}$ with $p_{1}$, $p_{2}$ (c is 3, 7 and 70 respectively)

Table 7 Hype-parameter specifications

If the two GCN models are completely independent:

$$\begin{aligned} N_{r}= & {} p_{1} p_{2} N, N_{e}=C_{c-1}\left( \frac{1-p_{1}}{c-1}\right) \left( \frac{1-p_{2}}{c-1}\right) \nonumber \\ N= & {} \frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1} N, \end{aligned}$$

(A1)

$$\begin{aligned} N_{a}=N_{r}+N_{w}=\left[ p_{1} p_{2}+\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}\right] N. \end{aligned}$$

(A2)

In fact, due to the correlation between the two GCN models, the number of samples correctly classified by the two models $N_{r}=p_{1} p_{2} N+\alpha N$, and the number of samples incorrectly classified by the two models as the same category $N_{w}=\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1} N+\beta N$. We can get:

$$\begin{aligned} N_{a}\!=\!N_{r}+N_{w}=\left[ p_{1} p_{2}+\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}+\alpha +\beta \right] N. \end{aligned}$$

(A3)

The more significant the correlation between the two GCN models, the more excellent $\alpha$ and $\beta$ will be,and $\alpha >0$, $\beta >0$. For simplicity, let $\gamma =\alpha +\beta$. The average classification accuracy for low-confidence samples is $p_{low-conf}$. For the first GCN model, the number of correctly classified samples can be expressed as $p_{1}N$. The number of correctly classified samples can also be expressed as:

$$\begin{aligned} N_{a}+\left[ 1-p_{1} p_{2}-\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}-\gamma \right] N \cdot p_{low-conf}. \end{aligned}$$

(A4)

Equation can be obtained as:

$$\begin{aligned} N_{a}+ & {} \left[ 1-p_{1} p_{2}-\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}-\gamma \right] \nonumber \\\times & {} N \cdot p_{low-conf}=p_{1} N. \end{aligned}$$

(A5)

An upper bound for $p_{low-conf}$ can be obtained:

$$\begin{aligned} p_{low-conf}= & {} \frac{p_{1}-p_{1} p_{2}-\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}-\gamma }{1-p_{1} p_{2}-\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}-\gamma }\nonumber \\< & {} p_{1}\left( \frac{1-p_{2}}{1-p_{1} p_{2}}\right) . \end{aligned}$$

(A6)

1.1 A. 2 Theorem 2 in section III

Proof

The average classification accuracy of the low- confidence samples is $p_{low-conf}$, the average classification accuracy after the calibration is $p_{low-conf}^{\prime }$, and the performance improvement of the model is $p_{GAIN}$. After the low-confidence samples calibration, the (A5) can be rewritten as:

$$\begin{aligned} N_{a}+ & {} \left[ 1-p_{1} p_{2}-\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}-\gamma \right] \nonumber \\\times & {} N \cdot p_{low-conf}^{\prime }=\left( p_{1}+p_{GAIN}\right) N. \end{aligned}$$

(A7)

Substitute $N_{a}=\left[ p_{1} p_{2}+\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}+\gamma \right] N$ into the above expression:

$$\begin{aligned}{} & {} \left[ p_{1} p_{2}+\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}+\gamma \right] \nonumber \\{} & {} \quad +\left[ 1-p_{1} p_{2}-\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}-\gamma \right] p_{\text{ low-conf }}^{\prime }=p_{1}\nonumber \\{} & {} \quad +p_{GAIN}. \end{aligned}$$

(A8)

Since $p_{low-conf}^{\prime }<p_{1}$, we can obtain the inequality:

$$\begin{aligned} p_{1}+ & {} p_{G A I N}<\left[ p_{1} p_{2}+\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}+\gamma \right] \nonumber \\{} & {} +\left[ 1-p_{1} p_{2}-\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}-\gamma \right] p_{1}. \end{aligned}$$

(A9)

Simplify inequality,

$$\begin{aligned} p_{GAIN}<\left( 1-p_{1}\right) \left[ p_{1} p_{2}+\frac{\left( 1-p_{1}\right) \left( 1-p_{2}\right) }{c-1}+\gamma \right] . \end{aligned}$$

(A10)

1.2 A.3 Analysis 2 in section III

$\gamma$ is related to the correlation between the dual-channel models and is determined by the model and its parameters and data set. According to Assumption 2, $p_{low-conf}^{\prime }<p_{1}$. Similarly, we can obtain: $p_{low-conf}^{\prime }<p_{2}$. The upper bound of model improvement accuracy is determined by $p_{1}$ and $p_{2}$. For simplicity, let $\gamma =0$, using $p_{1}$ and $p_{2}$ as the X and Y axes respectively, draw the upper bound of the accuracy of the promotion when the category c is 3, 7 and 70 in Fig. 7 respectively.

We can observe that the bigger the difference between $p_{1}$ and $p_{2}$, the lower the upper bound of $p_{GAIN}$. When the value of $p_{1}$ is fixed and the value of $p_{2}$ is the same as $p_{1}$, the upper bound of $p_{GAIN}$ is the highest.

1.3 A.4 Implementation details

In this paper, all models use a 2-layer GCN with ReLU as the activation function. We train the model for a fixed number of epochs, specifically. 200, 200, 500, 1000 epochs for Cora, Citeseer, Pubmed and CoraFull, respectively, 300 for acm, 200 for Flickr, and 200 for UAI2010. All models along with $\varvec{\mu }$ are initialized with Xavier initialization, and matrix $\varvec{\Sigma }$ is initialized with identity. All models were trained using Adam optimizer on all datasets. Our models were implemented in PyTorch version 1.6.0. When constructing feature graph, $k\in \{2, 3,..., 10\}$ for k-nearest neighbor graph. All dataset-specific hyper-parameters are summarized in Table 7.