Bootstrap Knowledge Distillation for Chest X-ray Image Classification with Noisy Labelling

Abstract

Chest X-ray images classification provides an essential way for lung disease diagnosis. However, this task is challenging due to the lack of professional knowledge and high annotation cost on Chest X-ray images. A common solution for medical data annotation is to use Natural Language Processing (NLP) techniques to extract labels from radiology reports. However, due to the complex structure of radiology reports, NLP based annotation will inevitably bring noisy labels into data, making analysis very difficult. Most existing methods seek to train a classification model (such as convolutional neural network) directly on the original data and ignore the noisy labels, which, however, may lead to very limited diagnosis performance. In this work, we propose a novel Bootstrap Knowledge Distillation (BKD) method, which seeks to improve the label qualities gradually, thereby degrade the noise level of the whole dataset. We theoretically analyze that the distribution of distilled labels will gradually approach to the unseen real labels distribution. Extensive experimental results on real-world Chest X-ray datasets demonstrate the effectiveness of the proposed method.

A Theoretical Analysis

Proposition 1

At each time step t, we assume the variance term of $D_y$ and $D_{s_{t - 1}}$ are independent, leading to the distance $D_{y_t^\lambda }$ is smaller than both the distance $D_y$ and $D_{s_{t - 1}}$,

$$\begin{aligned} \min _\lambda D_{y_t^\lambda } < \min \{D_y, D_{s_{t - 1}}\} \end{aligned}$$

(6)

where $s_{t-1}$ and y are the labels output from model $f_{t - 1}$ and true labels respectively. $D_{y_t^\lambda }$ can reach its optimal

$$\begin{aligned} D_{y_t^{\lambda ^*}} = \min _\lambda D_{y_t^\lambda } = \frac{D_y \cdot D_{s_{t - 1}}}{D_y + D_{s_{t - 1}}}, \end{aligned}$$

(7)

if and only if $\lambda ^* = \frac{D_{s_{t - 1}}}{D_{s_{t - 1}} + D_y}$.

Proof

At time step $t=0$, we train a model $f_0$ from a clean dataset $\mathcal {D}_c$, the expected prediction error can be composed into the bias term and variance term.

$$\begin{aligned} \mathbb {E}_{\mathcal {D}_\text {test}}[\ell (s_0, y^*)] = \ell (\bar{s}_0, y^*) + \mathbb {E}_{\mathcal {D}_\text {test}}[\ell (\bar{s}_0, s_0)] \end{aligned}$$

(8)

where $\ell (\cdot , \cdot )$ is $\ell _2$ distance, $\bar{s}_0 = \mathbb {E}_{\mathcal {D}_\text {test}}[s_0]$. Since the high capacity of CNN model, we make a reasonable assumption that the bias term $\ell (s_0, y^*)$ is close to zero . Also, the variance term of $D_y$ and $D_{s_0}$ is independent. This leads to

$$\begin{aligned} \ell (\bar{s}_0, y^*) = 0 \Rightarrow \bar{s}_0 \approx y^* \end{aligned}$$

(9)

$$\begin{aligned} \mathbb {E}_{\mathcal {D}_\text {test}}[\ell (s_0, y^*)] \approx \mathbb {E}_{\mathcal {D}_\text {test}}[\ell (s_0, \bar{s}_0)] \triangleq D_{s_0} \end{aligned}$$

(10)

$$\begin{aligned} \mathbb {E}_{\mathcal {D}_\text {test}}[(y - y^*)^T(s_0 - y^*)]&= \mathbb {E}_{\mathcal {D}_\text {test}}[y-y^*]^T \mathbb {E}_{\mathcal {D}_\text {test}}[s_0 - y^*] \nonumber \\&= \mathbb {E}_{\mathcal {D}_\text {test}}[y-y^*]^T \mathbb {E}_{\mathcal {D}_\text {test}}[s_0 - \bar{s}_0] \nonumber \\&= \mathbb {E}_{\mathcal {D}_\text {test}}[y-y^*]^T \mathbf {0} = 0 \end{aligned}$$

(11)

Due to Eqs. (9)–(11), we have

$$\begin{aligned} D_{y_1^\lambda }&= \mathbb {E}_{\mathcal {D}_\text {test}}[\Vert y_1^\lambda - y^* \Vert ^2] \nonumber \\&= \mathbb {E}_{\mathcal {D}_\text {test}}[\Vert \lambda y + (1 - \lambda )s_0 - y^*\Vert ^2] \nonumber \\&= \mathbb {E}_{\mathcal {D}_\text {test}} [\Vert \lambda (y - y^*) + (1 - \lambda )(s_0 - y^*)\Vert ^2] \nonumber \\&= \lambda ^2 D_y + (1 - \lambda )^2 D_{s_0} \end{aligned}$$

(12)

Since to Eq. (12), when $\lambda = \frac{D_{_{t - 1}}}{D_{s_{t - 1}} + D_y}$, $D_{y_1^\lambda }$ reach its minimum,

$$\begin{aligned} D_{y_1^{\lambda ^*}} = \min _\lambda D_{y_1^\lambda } = \frac{D_y \cdot D_{s_0}}{D_y + D_{s_0}} \end{aligned}$$

(13)

At each time step t, we train model $f_t$ from a dataset $\mathcal {D}_t = \{(x, y_t^\lambda )\}$. We assume $\ell (\bar{s}_t, y_t^\lambda )$ is close to zero, where $s_t = f_t(x)$, $\bar{s}_t = \mathbb {E}_{\mathcal {D}_\text {test}}[s_t]$, this leads to $\mathbb {E}_{\mathcal {D}_\text {test}}[s_t] \approx y_t^\lambda$, so that the distance between $s_t$ and y is approximate to $D_{y_t^\lambda }$.

$$\begin{aligned} D_{s_t}&= \mathbb {E}_{\mathcal {D}_\text {test}}[\ell (s_t, y^*)] \nonumber \\&= \ell (\bar{s}_t, y_t^\lambda ) + \mathbb {E}_{\mathcal {D}_\text {test}}[\ell (s_t, y^*)] \nonumber \\&\approx \mathbb {E}_{\mathcal {D}_\text {test}}[\ell (y_t^\lambda , y^*)] \nonumber \\&= D_{y_t^\lambda } \end{aligned}$$

(14)

As Eqs. (9)–(12), we have

$$\begin{aligned} \min _\lambda D_{y_t^\lambda } < \min \{D_y, D_{s_{t - 1}}\} \end{aligned}$$

(15)

and

$$\begin{aligned} D_{y_t^{\lambda ^*}} = \min _\lambda D_{y_t^\lambda } = \frac{D_y \cdot D_{s_{t - 1}}}{D_y + D_{s_{t - 1}}} \end{aligned}$$

(16)

if and only if $\lambda ^* = \frac{D_{s_{t - 1}}}{D_{s_{t - 1}} + D_y}$.    $\square$