BUS-Net: Breast Tumour Detection Network for Ultrasound Images Using Bi-directional ConvLSTM and Dense Residual Connections

Abstract

Breast ultrasound (BUS) imaging has become one of the key imaging modalities for medical image diagnosis and prognosis. However, the manual process of lesion delineation from ultrasound images can incur various challenges concerning variable shape, size, intensity, curvature, or other medical priors of the lesion in the image. Therefore, computer-aided diagnostic (CADx) techniques incorporating deep learning–based neural networks are automatically used to segment the lesion from BUS images. This paper proposes an encoder-decoder-based architecture to recognize and accurately segment the lesion from two-dimensional BUS images. The architecture is utilized with the residual connection in both encoder and decoder paths; bi-directional ConvLSTM (BConvLSTM) units in the decoder extract the minute and detailed region of interest (ROI) information. BConvLSTM units and residual blocks help the network weigh ROI information more than the similar background region. Two public BUS image datasets, one with 163 images and the other with 42 images, are used. The proposed model is trained with the augmented images (ten forms) of dataset one (with 163 images), and test results are produced on the second dataset and the testing set of the first dataset—the segmentation performance yielding comparable results with the state-of-the-art segmentation methodologies. Similarly, the visual results show that the proposed approach for BUS image segmentation can accurately identify lesion contours and can potentially be applied for similar and larger datasets.

Appendices

Appendix

A. Depthwise Separable Convolution

Let an input to a convolution layer be represented as $W \epsilon \mathbb {R}^{n_i \times n_o \times f_h \times f_w}$, where $n_i$ and $n_o$ are the input and output number of channels, with $f_h$ and $f_w$ denoting the height and width of the used filter, respectively. When the filter is applied to x (an image patch) of image I with size $n_i \times f_h \times f_w$, a response $y \ \epsilon \ \mathbb {R}^{n_o}$ is obtained as:

$$\begin{aligned} y_o = W * x \end{aligned}$$

(10)

where $y_o = \sum _{i=1}^{n_i} W_{i,o} * x_i, \ o \ \epsilon \ [n_o]$ and $i \epsilon \ [n_i]$, and $*$ denotes the convolution operation. $W_{i,o} = W[i,o,:,:]$ is a slice of tensor along the $i^{th}$ input and $o^{th}$ output channels. The computational complexity of the concerned patch x is $O (n_i \times n_o \times f_h \times f_w)$.

In contrast to normal convolution, depthwise separable convolution contains depthwise convolution (DC) and pointwise convolution(PC). DC focuses on the spatial connection among pixels, with PC focusing on the cross-channel relationship with $1 \times 1$ convolution. In order to get the same output shape as that of the input with the application of depthwise separable convolution, the DC convolution kernel is set to $D \ \in \ \mathbb {R}^{n_i \times 1 \times f_h \times f_w}$, and PC convolution kernel is set to $P \in \mathbb {R}^{n_i \times n_o \times 1 \times 1}$. When it is applied to the input patch x of image I, the response vector $\widehat{y}$ is obtained as:

$$\begin{aligned} \widehat{y_o} = (P \circ D) * x \end{aligned}$$

(11)

where $\widehat{y} = \sum _{i=1}^{n_i} P_{i,o} (D_i * x_i)$, with o being the compound operation. $P_{i,o} = P[i,o,:,:]$ is the corresponding pointwise tensor slice. Also, the corresponding complexity for the depthwise separable operation is $O(H \times W \times (n_i \times f_h \times f_w + n_i \times n_o))$.

B. Residual Unit

The more a network becomes deep, the more it impedes convergence from the beginning. Later, it faces the problem of network degradation. To attenuate this, residual connections have come into existence and attempted to solve the problem of deep network gradient dispersion by stacking residual modules in the conventional deep networks [36]. The skip connection added in residual modules increases the network’s depth without adding any extra computation parameters. Moreover, this structure promotes training efficiency.

A pyramid form of the residual unit when used, increases the network’s ability to extricate features efficiently. The pyramidal form has the advantage of generalizability and linearly increases the size of output feature maps. Besides, it substantially reduced the network’s training parameters and computational cost. The pyramidal residual unit (in the bottom of Fig. 3), starts with LeakyReLU activation function [22] and ends with Batch Normalization (BN) [23], which is also added before first separable convolution operation. In Fig. 3 (residual block portion), $X_i$ is the input and $X_{i+1}$ is the output of i-th residual unit with F denoting the residual function. H signifies the shortcut (residual) connection: if this connections is replaced by identity mapping, then $H(X_i) = X_i$. Using these notations, the output $X_{i+1}$ for the basic residual unit is represented as:

$$\begin{aligned} X_{i+1} = F(X_i,W_i)+X_i \end{aligned}$$

(12)

C. Bi-directional ConvLSTM

ConvLSTM acts as a convolutional counterpart of conventional fully connected LSTMs [19]. They are introduced to exploit convolution operations into different state transitions, not in conventional LSTM networks that do not consider pixels’ spatial correlation. ConvLSTM consists of different gates for its operation: an input gate $i_t$, an output gate $o_t$, a memory cell $m_t$, and a forget gate $f_t$. All gates except a memory cell act as control gates for accessing, updating and clearing (forgetting) the memory cell information. With the above definitions, different gates in ConvLSTM are represented as:

$$\begin{aligned} \begin{aligned} i_t&= \sigma (W_{xi}*X_t + W_{hi}*H_{t-1} + W_{mi}*m_{t-1} + b_i)\\ f_t&= \sigma (W_{xf}*X_t + W_{hf}*H_{t-1} + W_{mf}*m_{t-1} + b_f)\\ o_t&= \sigma (W_{xo}*X_t + W_{ho}*H_{t-1} + W_{mo}\circ m_{t-1} + b_m)\\ m_t&= f_t \circ m_{t-1} + i_t \circ tanh(W_{xm}*X_t + W_{hm}*H_{t-1} + b_m)\\ H_t&= o_t \circ tanh(m_t) \end{aligned} \end{aligned}$$

(13)

where convolution operation and Hadamard product are represented by * and $\circ$, respectively. $X_t$ and $H_t$ are the input and hidden state tensor with $m_t$ being the memory cell tensor. $W_{xt}$ and $W_{ht}$ are the convolution filters for the input and hidden states, respectively. $b_i$, $b_f$, $b_o$, and $b_m$ are the bias terms.