Abstract
A neural network is one of the current trends in deep learning, which is increasingly gaining attention owing to its contribution in transforming the different facets of human life. It also paves a way to approach the current crisis caused by the coronavirus disease (COVID-19) from all scientific directions. Convolutional neural network (CNN), a type of neural network, is extensively applied in the medical field, and is particularly useful in the current COVID-19 pandemic. In this article, we present the application of CNNs for the diagnosis and prognosis of COVID-19 using X-ray and computed tomography (CT) images of COVID-19 patients. The CNN models discussed in this review were mainly developed for the detection, classification, and segmentation of COVID-19 images. The base models used for detection and classification were AlexNet, Visual Geometry Group Network with 16 layers, residual network, DensNet, GoogLeNet, MobileNet, Inception, and extreme Inception. U-Net and voxel-based broad learning network were used for segmentation. Even with limited datasets, these methods proved to be beneficial for efficiently identifying the occurrence of COVID-19. To further validate these observations, we conducted an experimental study using a simple CNN framework for the binary classification of COVID-19 CT images. We achieved an accuracy of 93% with an F1-score of 0.93. Thus, with the availability of improved medical image datasets, it is evident that CNNs are very useful for the efficient diagnosis and prognosis of COVID-19.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Introduction
Coronavirus disease (COVID-19), caused by the severe acute respiratory syndrome coronavirus-2, is a pathogenic viral infection [1]. The spread of the COVID-19 pandemic has become an issue of global concern [2], which has forced the World Health Organization to reassess the conventions of the healthcare system [3]. The rapid spread of COVID-19 has also hampered the supply chain of critical care equipment and medical products [1, 4,5,6]. The current crises has attracted the attention of academicians, researchers, and scientists from different backgrounds [7,8,9,10,11]. New technologies are being developed to provide intelligent solutions for complex treatments and procedures, leading to an industrial revolution [12,13,14]. Doctors, virologists, disease transmission experts, and phylogeneticists have teamed up with policymakers to incorporate the available knowledge regarding the disease pathogenesis and control the spread of the infection. Effective screening of patients is the fundamental step in fighting COVID-19 so that infected patients can be isolated and treated. Laboratory diagnosis through reverse transcription-polymerase chain reaction (RT-PCR) is considered as the reference standard for the screening of COVID-19. However, false-negative instances of the RT-PCR tests have been progressively reported [15]. Laboratory findings through RT-PCR testing alone cannot be used as a substantial basis to confirm COVID-19 diagnosis. Other diagnostic methods, along with clinical correlations, are essential to confirm the infection status. Medical imaging is being tested as a potential screening tool for the early diagnosis and prognosis of COVID-19 [16]. Studies suggest that medical imaging will perform a crucial role in validating RT-PCR tests [17].
The interpretation of medical images is by default performed by experts in the field, such as radiologists and physicians. As medical data vary significantly from one patient to another depending on the disease, the task of diagnosis becomes labor intensive. Further, the interpretation is highly dependent on clinicians who exhibit limitations in terms of experience, speed, and fatigue. With the current crisis, the ability of the healthcare system to cope with the pandemic has been tested. One way to achieve this is to conduct medical image analysis through an automated, precise, and efficient computational framework such as a neural network, which is capable of replicating the accuracies of a trained human brain. In fact, the quantum of medical image analysis that an efficient neural network can perform at any given time is significantly greater than that of a human cortex.
The application of neural networks to tackle COVID-19 is primarily achieved by providing meaningful insights to medical image data [18]. Driven by a combination of factors such as public health emergencies, availability of a large collection of data, and advances in technology, several neural network models have been constructed [19]. A convolutional neural network (CNN) is a class of deep neural networks that are primarily employed for medical image processing. These neural network models help extract specific findings from chest radiology images of COVID-19 patients. In this article, we discuss different types of CNN models that have been proposed to recognize the patterns in chest X-ray and computed tomography (CT) images of COVID-19 patients, enabling automatic detection, segmentation, and classification of images. Keywords such as COVID-19, RT-PCR, CT, X-ray, neural network, CNN, deep learning, and medical image analysis were used to search for articles through the websites of PubMed, Radiopaedia, and Google Scholar. Further, to gain an intuitive and simplified understanding of the CNNs for COVID-19 image classification, we conducted an experimental study using a simple CNN framework. The experiment was conducted to classify COVID-19 and non-COVID-19 CT images using a publicly available dataset.
Overview of the CNN
This section provides a formal introduction to the neural network approach and architecture. A neural network is a learning algorithm and is the primary element of most deep learning methods. Neural networks or artificial neural networks are capable of functioning like the human cerebrum. These networks can achieve faster and more reliable pattern recognition and aid in interpreting medical images. The neural network is an information processing paradigm designed to mimic the processes of the human brain and analyze the available information.
The neural network architecture consists of neurons along with the input layer, output layer, and one or more intermediate layers called hidden layers. Neurons are mathematical operations. The layers of neural networks are highly interconnected and coordinate in a distributed way to learn from the given input information and optimize its final output. Figure 1 shows a simple neural network architecture. First, the data enter the network through the input layer, followed by multiple hidden layers, where the data are transformed as they pass through each layer. Further, the resulting data from the previous layer are passed to the output layer, which produces the desired result. Neural networks that involve multiple hidden layers are considered as deep neural networks, resulting in the introduction of the term “deep learning” [20].
Deep learning techniques can be easily adapted for different applications and domains when compared to classical machine learning methods. Unlike traditional systems that use handcrafted features [21], these deep neural network models learn features from the data, i.e., they can completely discover the morphological patterns in an image from the data. The neural network technique does not require manual preprocessing of the raw data. The different types of neural networks include CNNs, recurrent neural networks, long short-term memory networks, and generative adversarial networks (GANs). The most popular model considered for medical image analysis is the CNN, also known as ConvNet.
The architecture of a CNN can help achieve multilevel hierarchical feature learning. The early layers of CNN can fetch the low-level features (texture and shape), while the deep layers can extract the high-level semantic features that are combined to locate the key points accurately. Because the input consists of an image, the CNN compels the architecture more sensibly. The ConvNet layers have neurons arranged in three dimensions, that is height, width, and depth. Figure 2 shows the basic CNN architecture used for image classification. The CNN architecture includes an input layer, convolutional layer, rectified linear unit (ReLu), pooling layer, fully connected layer, and output layer. These layers are stacked to build a full CNN architecture. Feature extraction is performed through the convolutional, ReLU, and pooling layers. The classification is performed in the fully connected layer. Recognizing the layers and their connectivity in a deep neural network is more complex than running a pretrained general architecture. Designing an end-to-end neural network model can improve the recognition of the relationship between layers. As it filters irrelevant features, the model becomes more sensitive to meaningful patterns and handles the task in a more significant way.
The key research areas and applications of neural networks in medical image analysis include classification, detection, segmentation, localization, and registration. The neural network can provide a method to augment the early detection of COVID-19. Transfer learning has facilitated the utilization of pretrained neural network models for processing, analysis, segmentation, and classification of radiology images of COVID-19 patients. Further, certain neural networks have been further expanded in accordance with the task to be accomplished.
Medical image analysis of the COVID-19
Effective screening is an important step in limiting the spread of COVID-19 and creating awareness so that people can self-isolate if they develop any symptoms. Laboratory diagnosis through RT-PCR is considered the standard screening method where the respiratory samples are tested, and the results are obtained within a few hours to two days. Several RT-PCR analyses were developed for the detection of COVID-19. However, because of suboptimal sensitivity, the RT-PCR tests may sometimes yield false-negative results. In one of the studies conducted [22], in the early stages with the first appearance of the symptoms, the CT scans of a few patients showed the presence of COVID-19 infection, whereas the RT-PCR test results of these patients were negative. Both tests were performed repeatedly for several days, and the RT-PCR results subsequently confirmed that the CT results were true. RT-PCR can yield a false-negative result if the viral load is below the detection limit in the early stages or if there is an error in sampling [23]. Radiological images, such as chest X-rays and CT scans, are considered as an alternative screening method for COVID-19. A large amount of research has been conducted to prove the convenience of thoracic radiology evaluation for the diagnosis of COVID-19. Research suggests that the primary findings of COVID-19 are pneumonia, as observed by examining the X-ray and CT images of the thorax. Early investigators discovered a few common imaging patterns in the X-ray and CT images [15, 24]. Depending on the time after the onset of the disease, image characteristics of the infected patients vary from normal to dispersed patchy consolidations. The image findings frequently observed in the CT scans of the infected patients were ground-glass opacities (GGOs), crazy-paving appearance, air-space consolidation, bronchovascular thickening, and traction bronchiectasis [25, 26]. Figure 3 shows these common image findings in lung CT images. Figure 4 shows the normal and affected X-ray images of COVID-19 patients [26].
Common image findings observed in lung CT scans of the patients infected with COVID-19. a Ground-glass opacity; b Crazy paving; c Traction bronchiectasis; d Vascular dilation [26]
X-rays of a patient with COVID-19. a Normal; b After 4 days with bilateral consolidations [26]
Chest radiology imaging for diagnosis has several advantages over the conventional RT-PCR method. Methods involving radiology imaging tend to be quick and can simultaneously analyze multiple cases. Given the significance of radiography in the modern health care system, a radiology framework is available in all hospitals, which is more convenient and easily accessible. This framework is significantly helpful in hospitals without any testing kits or with only a limited number of kits. Further, not all hospitals have the required laboratory setup to assist in a high rate of sample testing procedures. The imaging method appears to be a valuable tool for COVID-19 clinical management and therefore can be used for the diagnosis, detection, follow-up, and evaluation of the severity of the infection.
To conduct any research, datasets are significantly essential. Some of the COVID-19 image datasets were made publicly available to encourage institutes and organizations to conduct research and discover solutions. Table 1 lists datasets consisting of chest X-ray and CT images of COVID-19 patients.
CNN for COVID-19 medical image analysis
As CNNs are being widely used in medical image analysis, in this section, we list the major developments in neural network technology to gain an insight on how these networks are being employed for the detection, classification, and segmentation of COVID-19. Following an experimental study to obtain a better understanding of the CNNs for COVID-19 image classification, we discuss the advantages and challenges of using these methods.
Detection and classification
CNNs are the best frameworks for the detection and classification of COVID-19 as they are capable of identifying the patterns present in chest radiology images. The classification process extracts important features and inputs them to the deep layers and classifies the affected chest CTs and X-rays from other images. Table 2 lists the CNN models developed for the detection and classification of COVID-19.
Abbas et al. [49] developed a CNN that relies on a class decomposition approach called decompose, transfer, and compose (DeTraC) for classifying the chest X-rays of COVID-19 patients. DeTraC can manage any irregularities in an image dataset by examining class boundaries through the class decomposition mechanism. DeTraC showed an accuracy of 85.12% in detecting COVID-19 X-ray images from normal and severe acute respiratory syndrome cases. COVID-Net, a deep CNN model, was developed by Wang et al. [36]. The model achieved 93.3% accuracy while identifying the normal, COVID-19, and pneumonia-infected X-rays of patients. Hemdan et al. [48] proposed COVIDX-Net using X-ray images to identify the positive or negative status of COVID-19 infection in patients. COVIDX-Net involves seven deep learning classifiers, namely DenseNet201, Visual Geometry Group Network with 19-layered CNN (VGG19), InceptionV3, residual network V2 (ResNet-V2), Inception-ResNet-V2, extreme Inception (Xception), and MobileNetV2. VGG19 and DensNet201 showed the highest performance scores. The high computational speed of MobileNetV2 presents a scope for improvement; consequently, it can be integrated with smart devices.
A deep-transfer-learning-based approach was adapted by Narin et al. [39] using chest X-ray radiographs to predict normal and COVID-19 infected patients automatically. The pretrained models were tested for accuracy, where the ResNet50 model showed 98% accuracy, while InceptionV3 and Inception-ResNet-V2 achieved accuracies of 97% and 87%, respectively. Mahmud et al. [37] developed a deep CNN architecture called CovXNet, which uses depthwise convolution with varying dilation rates for extracting diversified features from chest X-rays. Different forms of CovXNets, which are highly scalable with a large receptive capacity, that can be employed to identify COVID-19 patients were designed. The abnormal regions of the X-ray images involving different types of pneumonia were identified by integrating gradient-based discriminative localization. By using the DarkNet model classifier as the basis of the you only look once object detection system, Ozturk et al. [50] developed a model to detect and classify the COVID-19 cases from X-ray images. The classification results of this algorithm were evaluated for both binary and triple classes, which demonstrated accuracies of 98.08% and 87.02%, respectively. Another study by Apostolopoulos and Mpeisiana [42] deployed transfer learning with a CNN for the automatic detection of COVID-19 from X-ray images.
For the automatic detection of COVID-19 positive cases using X-ray images, Mukherjee et al. [38] developed a tailored shallow architecture that consists of only four layers. The experiment was conducted with a smaller X-ray dataset with 130 COVID and 51 non-COVID cases for which the false-positive rate was zero. To avoid possible bias, five-fold cross-validation was performed. The accuracy of this lightweight CNN was 96.92%. Hall et al. [41] fine-tuned a pretrained ResNet50 on chest X-rays of patients with COVID-19 and pneumonia using 10-fold cross-validation. The ensemble of ResNet50, VGG16, and a CNN classifier developed by them showed an overall accuracy of 89.2%. Rahimzadeh and Attar [43] concatenated Xception and ResNet-50-V2 networks to develop a model for the classification of X-ray images into normal, pneumonia, and COVID-19 cases. The model achieved an overall accuracy of 91.4%. Kumar et al. [47] used ResNet152 and a machine learning classifier for the classification of COVID-19 and pneumonia patients using X-rays. The intraclass variations of the datasets were balanced using the synthetic minority oversampling technique or SMOTE. The random forest classifier achieved an accuracy of 97.3% and the extreme gradient boosting classifier achieved an accuracy of 97.7%.
Wang et al. [52] built a transfer learning neural network by modifying the Inception network, which was referred to as M-Inception, to extract graphical features and provide a clinical diagnosis ahead of a pathogenic test. The model achieved an accuracy of 82.9% while classifying the CT images of COVID-19 cases. Apostolopoulos et al. [44] trained MobileNetV2 from scratch to investigate the extracted features for classifying the seven classes of pulmonary diseases, including COVID-19, using X-ray images. The proposed model provided an overall classification accuracy of 87.66% for the seven classes and a specificity of 99.42% for detecting COVID-19.
Mishra et al. [46] employed a decision-fusion-based approach that combined the predictions of each baseline model, that is, VGG16, InceptionV3, ResNet50, DenseNet121, and DenseNet201 for detection of COVID-19 from chest CT images. The proposed model achieved an average accuracy of 88.34%, which outperformed the performance of each individual model; moreover, DenseNet121 performed the best while considering the individual models. The COVID-19 infected region appears in different levels of gray in the CT image of the lungs, which poses difficulty in the analysis of the image. Özkaya et al. [51] used these differences in the gray levels to generate two datasets: subset-1 (16 × 16) and subset-2 (32 × 32) by obtaining random patches from the CT images of COVID-19 patients. The pretrained networks, i.e., VGG16, GoogleNet, and ResNet50, were employed for feature extraction. Feature fusion was performed to obtain high-dimensional features, and feature ranking was performed. The support vector machine classifier was trained for classification purposes.
Segmentation
Segmentation can help to provide the quantitative information necessary to detect COVID-19, analyze the severity, and extract the region of interest. Only a few works have employed segmentation to enhance the performance of developed frameworks. Table 3 lists the CNN models used for COVID-19 image segmentation. Alom et al. [57] used an Inception Residual Recurrent CNN (IRRCNN) with a transfer learning approach for the detection of COVID-19. The model was evaluated for both chest X-ray and CT images, and a quantitative evaluation was performed to determine the severity of COVID-19. The IRRCNN model was used to classify COVID-19 cases. NABLA-3 was employed for the segmentation of the infected lung region. The accuracy of the proposed method was 84.67% and 98.78% for X-ray and CT images, respectively.
Rajaraman et al. [40] used an iteratively pruned deep learning model to detect the presence of COVID-19 using chest X-ray images. The best-performing models were pruned to reduce the complexity and improve the memory efficiency. Modality-specific transfer learning was performed by retraining the customized CNN and ImageNet models. These models were further combined with different ensemble strategies to improve the classification. U-Net-based semantic segmentation was performed to fetch the important lung features and separate them from the background. Weighted averaging of the best-performing pruned models demonstrated superior performance while classifying the X-rays into normal, bacterial pneumonia, or COVID-19 viral pneumonia cases with an accuracy of 99.01%. Yan et al. [56] evaluated a three-dimensional deep learning model called COVID-SegNet. The model was used to enhance the boundary of the COVID-19 pneumonia-infected lung CT images through segmentation. Feature variation and atrous spatial pyramid pooling blocks were used to highlight the infected lung area and capture global information for semantic segmentation. The proposed method achieved a dice similarity coefficient of 0.987 and 0.726 for lung and COVID-19 segmentation, respectively.
To expedite the process of manual delineation of the infected region in CT images, Shan et al. [58] developed a framework. The voxel-based broad learning network (VB-Net) was employed for segmenting the COVID-19 infected region from lung CT images. The human-in-the-loop (HITL) strategy was adapted where the radiologists monitored the proposed deep learning model to improve the automatic annotation of each case. The dice similarity coefficient of the proposed model was 91.6% ± 10% between automatic and manual segmentation. Chen et al. [59] proposed a modified U-Net model. Here, an aggregated residual transformation was used to enhance the feature extraction, which was combined with the soft attention mechanism to obtain high-quality multiclass segmentation of the infected region from the COVID-19 chest CT images.
Fan et al. [60] developed a COVID-19 lung infection segmentation deep network called Inf-Net to automatically delineate the infected region from CT images. A parallel partial decoder was used to extract the high-level features and produce a global map. To improve segmentation, implicit reverse attention and explicit edge attention were used to enhance the boundaries. Wang et al. [61] developed a noise-robust framework called COVID-19 pneumonia lesion segmentation network (COPLE-Net) to read noisy labeled CT images. The noise-robust dice loss and COPLE-Net were fused with an adaptive self-ensemble framework for training. Images were grouped based on the size of the lesion. COPLE-Net outperformed all other models when dealing with different lesion sizes.
Data augmentation using GAN
Smaller datasets are considered to achieve satisfactory accuracy rates. However, as the amount of data increases, the results are correspondingly improved. At the beginning of the pandemic, the amount of data available was insufficient to add more variability to the dataset. Hence, synthetic data augmentation was performed using the GAN. Waheed et al. [53] proposed an auxiliary classifier GAN called CovidGAN by generating synthetic chest X-ray images. VGG16 was used as the base network. The aim was to prove the advantage of using synthetic images produced from CovidGAN to enhance the performance of CNNs for COVID-19 diagnosis. Because of the synthetic images, the accuracy of the CNN increased from 85% to 95%.
Khalifa et al. [54] fine-tuned CNN by deep transfer learning along with GAN for the detection of COVID-19 using X-ray images. The models for deep transfer learning included AlexNet, GoogLeNet, SqueezeNet, and ResNet18. The research used only 10% of the dataset for training and generated 90% of images using GAN to prove the efficiency of the proposed model. With GAN as the image augmenter, ResNet18 outperformed the other models by showing 99% accuracy. Zebin and Rezvy [55] employed pretrained CNNs VGG16, ResNet50, and EfficientNetB0 for the classification of COVID-19, pneumonia, and normal X-ray images. They also employed CycleGAN for data augmentation. The overall accuracies achieved were 90%, 94.3%, and 96.8% for VGG16, ResNet50, and EfficientNetB0, respectively.
Experimental study
We attempted to gain an intuitive and simplified understanding of the CNNs for COVID-19 image classification. Consequently, we conducted a small-scale experiment using a simple CNN framework as a binary classifier for classifying the CT images into COVID-19 and non-COVID-19 classes [63]. The CNN framework was fine-tuned according to our requirements.
Dataset
The publicly available COVID-CT dataset was used [64]. This dataset was downloaded from Kaggle [27]. The practicality of this dataset was confirmed by radiologists. The COVID-CT dataset consists of 349 COVID-19 CT images from 216 patients and 397 non-COVID-19 CT images. Figure 5 shows the sample COVID-19 affected CT images from the COVID-CT dataset. By understanding the importance of images and labels for training and testing of the CNN classifier, the dataset was labeled as COVID and non-COVID images, which were later transformed into arrays.
Experimental setup
The CNN framework used consisted of convolutional, pooling, and fully connected layers. The convolutional and pooling layers perform feature extraction, whereas the fully connected layer maps the extracted features into the output, which, in this case, was classification. The CT images were resized to 200 × 200 pixels. A four-layered CNN with an Adam optimizer was employed to optimize the weights of the model. The learning rate was set to 0.001, the epoch value was three, and the softmax activation function was used. As CNN cannot directly work with categorical data, the images and labels are transformed into an array using one-hot encoder. Then, the dataset was split into a training set to train the network and a validation set for cross-validation of the model during the training phase. The validation set monitored the model performance based on which the parameters were fine-tuned, and a final model was selected. The CNN framework was implemented using Python, Keras with Tensorflow, and a graphical processing unit on an Intel Core i5 processor.
Performance analysis
To evaluate the performance of the proposed CNN model we used the confusion matrix, which gives a holistic view of how well the model is performing (Table 4). Using the confusion matrix, the precision, recall, F1-score, and accuracy values were computed as follows.
Results
To obtain a suitable result and for significant evaluation of the final model, the CNN was fed with a testing dataset consisting of 60 unlabeled, unseen COVID and non-COVID CT images. We used a straightforward approach without data augmentation. The classification results of the final CNN model are shown in Fig. 6. CT images that were wrongly classified are marked by a red square, as shown in Fig. 6b and c. We evaluated the performance of the classification model using the confusion matrix method. The above CNN model for the classification of COVID-19 CT images yielded an accuracy of 93% with a precision value of 0.91, a recall value of 0.96, and an F1-score of 0.93. This experiment was conducted to obtain a better understanding of the application of CNNs for COVID-19 image classification. The CNN model classifies the images based on valid information that is available in the lung images by ignoring the artifacts and false visual indicators. However, the model can be further improved by adding more CT images to the dataset, which can yield better results. Further, the CNN model should be examined by clinical experts to create a production-ready solution. The next section provides an overview of the protocols to be followed to design a CNN that can satisfy the clinical requirements.
Discussion
In this section, the advantages and challenges of the aforementioned different studies are discussed. Tables 2 and 3 list various CNN frameworks used for the detection, classification, and segmentation of COVID-19 radiology images. It can be noted from the tables that CNNs are most commonly used for automated detection and classification. While analyzing the modalities used for the experiment, chest X-rays are mainly used for detection, whereas CT scans are used for classification. The segmentation of COVID-19 findings is predominantly performed using CT scans. The models listed in Table 2 either perform binary classification, i.e., normal versus COVID-19, or triple class classification, i.e., COVID-19 versus normal versus bacterial or viral pneumonia.
Researchers have used transfer learning to accelerate the learning process and reduce the requirement for large training data by employing pretrained CNN models. These CNN models were trained on a dataset that included COVID-19 medical images of the patients. Here, the early layers of the CNN learned basic features, and the high-level features were learned by the fine-tuned upper layers. The base models of most of the proposed CNNs are AlexNet, VGG16, ResNet, DensNet, GoogLeNet, MobileNet, Inception, and Xception (Table 2). If we consider the evaluation matrix, many of the proposed models show promising results.
The given images of the training dataset act as visual cues that include a variety of features. Deep neural networks learn by adjusting the weights that are backpropagated. This requires the model to learn from a large dataset covering all the possibilities for obtaining the highest accuracy when compared to state-of-the-art methods. The lack of publicly available datasets hinders the use of large medical images for training. Most of the models were trained and tested on a small dataset. The studies conducted involved images of COVID-19, bacterial, and other viral pneumonia. However, the factors such as age, sex, and other primary health conditions, such as diabetes, hypertension, and cardiovascular diseases, were not considered, which does not create a well-defined dataset and might not yield good results.
A large clinical dataset is used in the CNN to improve generalizability and limit overfitting. Although few techniques have encouraged learning on small datasets, as depicted in the articles, a well-annotated large medical dataset is still required. The vast majority of the eminent achievements of CNNs are typically based on a large amount of data. Unfortunately, building such datasets in medicine is expensive and requires a large amount of work by specialists; moreover, ethical and privacy issues must also be considered.
Another challenge due to the limited accessibility of medical datasets is the classification of medical images for clinical diagnosis. Because of the commonality that exists between COVID-19 and other types of bacterial or viral pneumonia, it is difficult to focus on the contrast between them. Most of the experiments performed by the authors did not explicitly confirm the recognition of COVID-19 and other pneumonia from identical CT findings. Further, there is a contradiction in the suitability of the modalities. In a few of the early instances, the CT images of patients with mild or no symptoms posed a difficulty in early detection. This limits the use of CT scans in testing. Considering the cost, CT scans are also more expensive, and exposure to radiation is higher than that of X-rays. However, CT scans are more detailed than X-ray scans.
To overcome the issue of the database, few authors used GAN models for data augmentation. Even though no new information is added to the network, to compensate for the inefficiency of the training dataset, existing image data were augmented, which increased the size of the dataset. One advantage of data augmentation is that it can also make the model more robust to overfitting.
Pretrained networks can add bias to the network, often leading to a local minimum and suboptimal solution. Developing an efficient and robust computer-aided diagnosis system for the diagnosis of COVID-19 is a difficult task. These pretrained models learn from the ImageNet database, which is different from the radiology images.
Most of the CNN frameworks that are listed in Table 2 focus on detection and classification, whereas the CNN frameworks for segmentation that are listed in Table 3 are comparatively lesser in number. U-Net and VB-Net were used for the segmentation of the lesion area (Table 3). These frameworks have demonstrated good results. However, when the effectiveness of segmentation is evaluated, there is no standard well-annotated reference dataset that covers all the visual variations of the lesions. COVID-19 lesions such as GGO, crazy paving, air-space consolidation, bronchovascular thickening, and traction bronchiectasis are highly variable and have complex appearances. They differ in size, texture, shape, contrast, and location. Obtaining a rich manually annotated dataset is again a cause of concern as labeling a large number of CT images is time consuming, expensive, and highly dependent on the perspective and knowledge of the expert. To overcome these issues, few authors have used the HITL strategy for delineation. Even though this can accelerate the process, the annotations may be more inclined toward the algorithm results and can produce weak and noisy labels.
The implementation of these models for developing a clinical workstation for COVID-19 diagnosis requires appropriate hardware resources. Further, the application of CNN models in clinical practice requires careful assessment and confirmation from clinical experts regarding their usefulness in solving the issue. This is beyond the mathematical precision and accuracy of the model. The proposed CNN models should be able to understand and apply the right epidemiologic principles for disease surveillance and investigation activities. However, only a few researchers have used clinical data and conducted experiments under the supervision of doctors, which must be extended for the further diagnosis of COVID-19.
Researchers are consistently active in addressing the emerging challenges. The listed CNN models are still in the development stage, and their utility in COVID-19 diagnosis is unproven. The expectation is that promising outcomes can be accomplished using the proposed CNN models. The availability of the standard dataset along with the ground truth can help in the design of better models for the diagnosis and prognosis of COVID-19. A dedicated medical pretrained network can presumably be proposed once such a dataset becomes accessible, which may further encourage deep learning research on COVID-19. These models are plausible tools to manage and combat current and future public health emergencies in a more systematic, economic, and timely manner. While considering the pandemic, these tools can help bridge the gap between different fields and facilitate the discovery of a solution.
Conclusions
In this article, we presented a comprehensive review of various studies conducted using CNNs to effectively detect and diagnose COVID-19 patients. The CNN models listed performed the detection, classification, and segmentation of COVID-19 images. Transfer learning facilitated the utilization of pretrained neural network models for accomplishing the different tasks. Most of the models used smaller datasets. The lack of publicly available datasets hinders the use of large medical images for training. Despite these challenges, CNN models exhibit good performance accuracy.
The experimental study conducted using a CNN framework for the classification of chest CT images into COVID-19 and non-COVID-19 cases showed an accuracy of 93% and an F1-score of 0.93. The experiment was conducted using a small dataset, without data augmentation. Moreover, the training epoch value was maintained at a small value to avoid overfitting. The proposed CNN model was trained and tested only on 746 CT images, for which good accuracy was achieved. The model must be trained and tested with more samples to result in increased generalization, which will further improve the accuracy and robustness. Nevertheless, the study conducted showed that CNN can effectively contribute to the development of a COVID-19 automated diagnostic system. A well-defined dataset along with clinical supervision can help develop CNN models for implementation in real time.
Availability of data and materials
The dataset analyzed during the current study is publicly available (https://github.com/UCSD-AI4H/COVID-CT).
Abbreviations
- COVID-19:
-
Coronavirus disease
- CT:
-
Computed tomography
- ReLu:
-
Rectified linear unit
- VGG19:
-
Visual Geometry Group Network with 19-layered CNN
- ResNet-V2:
-
Residual network V2
- CNN:
-
Convolutional neural network
- RT-PCR:
-
Reverse transcription-polymerase chain reaction
- GAN:
-
Generative adversarial network
- GGO:
-
Ground-glass opacity
- DeTraC:
-
Decompose, transfer, and compose
- VGG:
-
Visual geometry group network
- IRRCNN:
-
Inception Residual Recurrent Convolutional Neural Network
- HITL:
-
Human in the loop
- COPLE-Net:
-
COVID-19 pneumonia lesion segmentation network
- VB-Net:
-
Voxel-based broad learning network
References
Suman R, Javaid M, Haleem A, Vaishya R, Bahl S, Nandan D (2020) Sustainability of coronavirus on different surfaces. J Clin Exp Hepatol 10(4):386–390. https://doi.org/10.1016/j.jceh.2020.04.020
WHO (2020) Coronavirus disease 2019 (COVID-19): situation report-51. https://apps.who.int/iris/handle/10665/331475. Accessed 20 Sept 2020
Haleem A, Javaid M, Vaishya R (2020) Effects of COVID-19 pandemic in daily life. Curr Med Res Pract 10(2):78–79. https://doi.org/10.1016/j.cmrp.2020.03.011
Irfan Ul Haq M, Khuroo S, Raina A, Khajuria S, Javaid M, Ul Haq MF et al (2020) 3D printing for development of medical equipment amidst coronavirus (COVID-19) pandemic-review and advancements. Res Biomed Eng. https://doi.org/10.1007/s42600-020-00098-0
Javaid M, Haleem A, Vaishya R, Bahl S, Suman R, Vaish A (2020) Industry 4.0 technologies and their applications in fighting COVID-19 pandemic. Diabetes Metab Syndr Clin Res Rev 14(4):419–422. https://doi.org/10.1016/j.dsx.2020.04.032
Iyengar K, Bahl S, Vaishya R, Vaish A (2020) Challenges and solutions in meeting up the urgent requirement of ventilators for COVID-19 patients. Diabetes Metab Syndr Clin Res Rev 14(4):499–501. https://doi.org/10.1016/j.dsx.2020.04.048
Haleem A, Javaid M, Vaishya R, Deshmukh SG (2020) Areas of academic research with the impact of COVID-19. Am J Emerg Med 38(7):1524–1526. https://doi.org/10.1016/j.ajem.2020.04.022
Singh RP, Javaid M, Haleem A, Suman R (2020) Internet of things (IoT) applications to fight against COVID-19 pandemic. Diabetes Metab Syndr Clin Res Rev 14(4):521–524. https://doi.org/10.1016/j.dsx.2020.04.041
Singh RP, Javaid M, Kataria R, Tyagi M, Haleem A, Suman R (2020) Significant applications of virtual reality for COVID-19 pandemic. Diabetes Metab Syndr Clin Res Rev 14(4):661–664. https://doi.org/10.1016/j.dsx.2020.05.011
Bahl S, Singh RP, Javaid M, Khan IH, Vaishya R, Suman R (2020) Telemedicine technologies for confronting COVID-19 pandemic: a review. J Ind Integr Manag 5(4):547–561. https://doi.org/10.1142/S2424862220300057
Singh RP, Javaid M, Haleem A, Vaishya R, Bahl S (2020) Significance of health information technology (HIT) in context to COVID-19 pandemic: potential roles and challenges. J Ind Integr Manag 5(4):427–440. https://doi.org/10.1142/S2424862220500232
Haleem A, Javaid M (2020) Medical 4.0 and its role in healthcare during covid-19 pandemic: a review. J Ind Integr Manag 5(4):531–545. https://doi.org/10.1142/S2424862220300045
Javaid M, Haleem A, Singh RP, Ul Haq MI, Raina A, Suman R (2020) Industry 5.0: potential applications in covid-19. J Ind Integr Manag 5(4):507–530. https://doi.org/10.1142/S2424862220500220
Kushwaha S, Bahl S, Bagha AK, Parmar KS, Javaid M, Haleem A, Singh RP (2020) Significant applications of machine learning for COVID-19 pandemic. J Ind Integr Manag 5(4):453–479. https://doi.org/10.1142/S2424862220500268
Xie XZ, Zhong Z, Liu JJ, Liu W, Zheng C, Wang F et al (2020) Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing. Radiology 296(2):E41–E45. https://doi.org/10.1148/radiol.2020200343
Bai HX, Hsieh B, Xiong Z, Halsey K, Choi JW, Tran TML, Pan I, Shi LB, Wang DC, Mei J, Jiang XL, Zeng QH, Egglin TK, Hu PF, Agarwal S, Xie FF, Li S, Healey T, Atalay MK, Liao WH (2020) Performance of radiologists in differentiating COVID-19 from non-COVID-19 viral pneumonia at chest CT. Radiology 296(2):E46–E54. https://doi.org/10.1148/radiol.2020200823
Kanne JP, Little BP, Chung JH, Elicker BM, Ketai LH (2020) Essentials for radiologists on COVID-19: an update-radiology scientific expert panel. Radiology 296(2):E113–E114. https://doi.org/10.1148/radiol.2020200527
Farhat H, Sakr GE, Kilany R (2020) Deep learning applications in pulmonary medical imaging: recent updates and insights on COVID-19. Mach Vis Appl 31(6):53. https://doi.org/10.1007/s00138-020-01101-5
Shoeibi A, Khodatars M, Alizadehsani R, Ghassemi N, Jafari M, Moridian P, et al (2020) Automated detection and forecasting of COVID-19 using deep learning techniques: a review. arXiv: 2007.10785v3
Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88. https://doi.org/10.1016/j.media.2017.07.005
Kugunavar S, Prabhakar CJ (2021) Content-based medical image retrieval using delaunay triangulation segmentation technique. J Inf Technol Res 14(2):48–66. https://doi.org/10.4018/JITR.2021040103
Ai T, Yang ZL, Hou HY, Zhan CN, Chen C, Lv WZ, et al (2020) Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology 296(2):E32–E40. https://doi.org/10.1148/radiol.2020200642
Gietema HA, Zelis N, Nobel JM, Lambriks LJG, van Alphen LB, Lashof AMLO et al (2020) CT in relation to RT-PCR in diagnosing COVID-19 in the Netherlands: a prospective study. PLoS One 15(7):e0235844. https://doi.org/10.1371/journal.pone.0235844
Pan F, Ye TH, Sun P, Gui S, Liang B, Li LL, et al (2020) Time course of lung changes at chest CT during recovery from coronavirus disease 2019 (COVID-19). Radiology 295(3):715–721. https://doi.org/10.1148/radiol.2020200370
Shi HS, Han XY, Jiang NC, Cao YK, Alwalid O, Gu J, et al (2020) Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study. Lancet Infect Dis 20(4):425–434. https://doi.org/10.1016/S1473-3099(20)30086-4
COVID-19 imaging findings, the Radiology Assistant (2020). https://radiologyassistant.nl/chest/covid-19/covid19-imaging-findings. Accessed 20 Sept 2020
COVID-CT Dataset, GitHub (2020). https://github.com/UCSD-AI4H/COVID-CT. Accessed 20 Sept 2020
COVID-19 clinical studies, Eurorads (2020). https://www.eurorad.org/advanced-search?search=COVID. Accessed 23 Sept 2020
COVID-19 Resources, British Society of Thoracic Imaging (2020). https://www.bsti.org.uk/covid-19-resources/. Accessed 21 Sept 2020
COVID-19 chest X-ray and CT datasets, GitHub (2020). https://github.com/ieee8023/covid-chestxray-dataset. Accessed 20 Sept 2020
COVID-19 CT segmentation dataset, MedSeg (2020). https://medicalsegmentation.com/covid19/. Accessed 20 Sept 2020
COVID-19 database, Italian Society of Medical Radiology (2020). https://www.sirm.org/category/senza-categoria/covid-19/. Accessed 21 Sept 2020
Actualmed COVID-19 Chest X-ray Dataset Initiative, GitHub (2020). https://github.com/agchung/Actualmed-COVID-chestxray-dataset. Accessed 23 Sept 2020
COVID-19 datasets, Kaggle (2020). https://www.kaggle.com/datasets?search=covid. Accessed 20 Sept 2020
COVID-Net open source initiative, GitHub (2020). https://github.com/lindawangg/COVID-Net. Accessed 20 Sept 2020
Wang LD, Lin ZQ, Wong A (2020) COVID-net: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images. Sci Rep 10(1):19549. https://doi.org/10.1038/s41598-020-76550-z
Mahmud T, Rahman MA, Fattah S (2020) CovXNet: a multi-dilation convolutional neural network for automatic COVID-19 and other pneumonia detection from chest X-ray images with transferable multi-receptive feature optimization. Comput Biol Med 122:103869. https://doi.org/10.1016/j.compbiomed.2020.103869
Mukherjee H, Ghosh S, Dhar A, Obaidullah SM, Santosh KC, Roy K (2021) Shallow convolutional neural network for COVID-19 outbreak screening using chest X-rays. Cognit Comput. https://doi.org/10.1007/s12559-020-09775-9
Narin A, Kaya C, Pamuk Z (2020) Automatic detection of coronavirus disease (COVID-19) using X-ray images and deep convolutional neural networks. arXiv: 2003.10849
Rajaraman S, Siegelman J, Alderson PO, Folio LS, Folio LR, Antani SK et al (2020) Iteratively pruned deep learning ensembles for COVID-19 detection in chest X-rays. IEEE Access 8:115041–115050. https://doi.org/10.1109/ACCESS.2020.3003810
Hall LO, Paul R, Goldgof DB, Goldgof GM (2020) Finding COVID-19 from chest X-rays using deep learning on a small dataset. arXiv: 2004.02060. https://doi.org/10.36227/techrxiv.12083964.v3
Apostolopoulos ID, Mpesiana TA (2020) Covid-19: automatic detection from X-ray images utilizing transfer learning with convolutional neural networks. Phys Eng Sci Med 43(2):635–640. https://doi.org/10.1007/s13246-020-00865-4
Rahimzadeh M, Attar A (2020) A new modified deep convolutional neural network for detecting Covid-19 from X-ray images. arXiv: 2004.08052. https://doi.org/10.1016/j.imu.2020.100360
Apostolopoulos ID, Aznaouridis SI, Tzani MA (2020) Extracting possibly representative COVID-19 biomarkers from X-ray images with deep learning approach and image data related to pulmonary diseases. J Med Biol Eng 40(3):462–469. https://doi.org/10.1007/s40846-020-00529-4
Rehman A, Naz S, Khan A, Zaib A, Razzak I (2020) Improving coronavirus (COVID-19) diagnosis using deep transfer learning. medRxiv Prepr. https://doi.org/10.1101/2020.04.11.20054643
Mishra AK, Das SK, Roy P, Bandyopadhyay S (2020) Identifying COVID19 from chest CT images: a deep convolutional neural networks based approach. J Healthc Eng 2020:8843664–8843667. https://doi.org/10.1155/2020/8843664
Kumar R, Arora R, Bansal V, Sahayasheela VJ (2020) Accurate prediction of COVID-19 using chest X-ray images through deep feature learning model with SMOTE and machine learning classifiers. medRxiv Prepr. https://doi.org/10.1101/2020.04.13.20063461
Hemdan EED, Shouman MA, Karar ME (2020) COVIDX-net: a framework of deep learning classifiers to diagnose COVID-19 in X-ray images. arXiv: 2003.11055v1
Abbas A, Abdelsamea MM, Gaber MM (2021) Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network. Appl Intell 51(2):854–864. https://doi.org/10.1007/s10489-020-01829-7
Ozturk T, Talo M, Yildirim EA, Baloglu UB, Yildirim O, Acharya UR (2020) Automated detection of COVID-19 cases using deep neural networks with X-ray images. Comput Biol Med 121:103792. https://doi.org/10.1016/j.compbiomed.2020.103792
Özkaya U, Öztürk Ş, Barstugan M (2020) Coronavirus (COVID-19) classification using deep features fusion and ranking technique. arXiv: 2004.03698. https://doi.org/10.1007/978-3-030-55258-9_17
Wang S, Kang B, Ma JL, Zeng XJ, Xiao MM, Guo J, Cai M, Yang J, Li Y, Meng X, Xu B (2021) A deep learning algorithm using CT images to screen for Corona virus disease (COVID-19). Eur Radiol. https://doi.org/10.1007/s00330-021-07715-1
Waheed A, Goyal M, Gupta D, Khanna A, Al-turjman F, Pinheiro PR (2020) CovidGAN: data augmentation using auxiliary classifier GAN for improved Covid-19 detection. IEEE Access 8:91916–91923. https://doi.org/10.1109/ACCESS.2020.2994762
Khalifa NEM, Taha MHN, Hassanien AE, Elghamrawy S (2020) Detection of coronavirus (COVID-19) associated pneumonia based on generative adversarial networks and a fine-tuned deep transfer learning model using chest X-ray dataset. arXiv: 2004.01184
Zebin T, Rezvy S (2021) COVID-19 detection and disease progression visualization: deep learning on chest X-rays for classification and coarse localization. Appl Intell 51(2):1010–1021. https://doi.org/10.1007/s10489-020-01867-1
Yan QS, Wang B, Gong D, Luo C, Zhao W, Shen JH, et al (2020) COVID-19 chest CT image segmentation--a deep convolutional neural network solution. arXiv: 2004.10987
Alom MZ, Rahman MMS, Nasrin MS, Taha TM, Asari VK (2020) COVID_MTNet: COVID-19 detection with multi-task deep learning approaches. arXiv: 2004.03747
Shan F, Gao YZ, Wang J, Shi WY, Shi NN, Han MF, et al (2020) Lung infection quantification of COVID-19 in CT images with deep learning. arXiv: 2003.04655v3
Chen XC, Yao L, Zhang Y (2020) Residual attention U-net for automated multi-class segmentation of COVID-19 chest CT images. arXiv: 2004.05645
Fan DP, Zhou T, Ji GP, Zhou Y, Chen G, Fu HZ, Shen J, Shao L (2020) Inf-net: automatic COVID-19 lung infection segmentation from CT images. IEEE Trans Med Imaging 39(8):2626–2637. https://doi.org/10.1109/TMI.2020.2996645
Wang GT, Liu XL, Li CP, Xu ZY, Ruan JG, Zhu HF, Meng T, Li K, Huang N, Zhang S (2020) A noise-robust framework for automatic segmentation of COVID-19 pneumonia lesions from CT images. IEEE Trans Med Imaging 39(8):2653–2663. https://doi.org/10.1109/TMI.2020.3000314
Voulodimos A, Protopapadakis E, Katsamenis I, Doulamis A, Doulamis N (2020) Deep learning models for COVID-19 infected area segmentation in CT images. medRxiv Prepr. https://doi.org/10.1101/2020.05.08.20094664
ConvNet, Kaggle (2017). https://www.kaggle.com/sentdex/full-classification-example-with-convnet. Accessed 2 Sept 2020
Zhao JY, He XH, Yang XY, Zhang YC, Zhang SH, Xie PT (2020) COVID-CT-Dataset: a CT scan dataset about COVID-19. arXiv: 2003.13865v3
Acknowledgements
Not applicable.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
CJP conceptualized and designed the study, interpreted the data and wrote the manuscript; SK conceptualized and designed the study, performed the experiments, data analysis and its interpretation and wrote the manuscript. The author(s) read and approved the final manuscript.
Authors’ information
Sneha Kugunavar received her M.Tech in Computer Cognition Technology from the University of Mysore, Karnataka, India. She is currently pursuing her final year PhD at Kuvempu University, Karnataka, India. Her research is focused on medical image processing. She has also taught information science and engineering for undergraduate and postgraduate students at Visvesvaraya Technological University, India.
CJ Prabhakar received his PhD degree in computer science and technology from Gulbarga University, Gulbarga, Karnataka, India in 2010. He is currently working as an Associate Professor in the Department of Computer Science and M.C.A, Kuvempu University, Karnataka, India. His research interests are pattern recognition, document image processing, computer vision and machine vision.
Corresponding author
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kugunavar, S., Prabhakar, C.J. Convolutional neural networks for the diagnosis and prognosis of the coronavirus disease pandemic. Vis. Comput. Ind. Biomed. Art 4, 12 (2021). https://doi.org/10.1186/s42492-021-00078-w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s42492-021-00078-w