Adaptive deep learning for head and neck cancer detection using hyperspectral imaging
- 128 Downloads
It can be challenging to detect tumor margins during surgery for complete resection. The purpose of this work is to develop a novel learning method that learns the difference between the tumor and benign tissue adaptively for cancer detection on hyperspectral images in an animal model. Specifically, an auto-encoder network is trained based on the wavelength bands on hyperspectral images to extract the deep information to create a pixel-wise prediction of cancerous and benign pixel. According to the output hypothesis of each pixel, the misclassified pixels would be reclassified in the right prediction direction based on their adaptive weights. The auto-encoder network is again trained based on these updated pixels. The learner can adaptively improve the ability to identify the cancer and benign tissue by focusing on the misclassified pixels, and thus can improve the detection performance. The adaptive deep learning method highlighting the tumor region proved to be accurate in detecting the tumor boundary on hyperspectral images and achieved a sensitivity of 92.32% and a specificity of 91.31% in our animal experiments. This adaptive learning method on hyperspectral imaging has the potential to provide a noninvasive tool for tumor detection, especially, for the tumor whose margin is indistinct and irregular.
KeywordsHyperspectral imaging Deep learning Adaptive learning Noninvasive cancer detection
Convolutional neural network
Fully connected networks
Green fluorescence protein
Orophary cancer is a common cancer worldwide and in recent years its incidence increased in a fast pace in both America and Europe . More than half a million patients receive the diagnosis of squamous-cell carcinoma of the head and neck worldwide each year . Survival rate of patients relates directly to the size of the primary tumor at first diagnosis, hence, early detection can be helpful in curing the disease completely. Squamous-cell carcinoma of the head and neck is a complex disease, which can be biopsied for histopathological assessment to make a definitive diagnosis traditionally. That is not only time consuming and invasive, but also subjective and inconsistent .
Hyperspectral imaging (HSI) is a technology that can acquire a series of images in many adjacent narrow spectral bands and reconstruct the reflectance spectrum for every pixel of the image . By measuring the reflection and absorption of the lights at different wavelengths, HSI has the ability to simultaneously provide information about different tissue constituents and their spatial distribution from the spectral signature of each pixel in the hyperspectral image . Hence, HSI technique can be applied in the noninvasive detection and diagnosis of cancer, such as breast cancer, gastric cancer, tongue cancer, and so on .
Hyperspectral images, known as hypercubes, contain rich information on a wide range of spectra with a high spectral resolution , hence, dimensionality reduction, image processing, and machine learning techniques are applied to extract the useful information from the vast amounts of HSI data, and have made many of the advancements in cancer identification: (1) Dimensionality reduction techniques. The principal component analysis [8, 9], tensor decompositions , and T-distributed stochastic neighbor approach [11, 12], were to reduce the dimensionality of features in hyperspectral images for compact expression; (2) Image processing techniques. Fourier coefficients , normalized difference nuclear index , sparse representation , box-plot and the watershed method , superpixel method , markov random fields [17, 18], and morphological method , were used for hyperspectral image processing and quantification analysis; (3) Machine learning techniques. Many of the advancements have been done in cancer identification using traditional machine learning classification models, such as linear discriminant analysis [20, 21, 22, 23, 24, 25, 26], quadratic discriminant analysis , support vector machine [12, 17, 20, 21, 22, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37], decision trees , k-nearest neighbors algorithm [22, 38], k-means [12, 19, 39], naïve bayes , random forests [21, 22, 34, 37], maximum likelihood , minimum spanning forest , gaussian mixture models , and semantic texton forest , and artificial neural network [33, 34, 35, 37], and so on.
However, these technologies require domain-specific knowledge to extract discriminant data to convert suitable features. In contrast to these conventional machine learning techniques, deep learning models can learn representations of data with multiple levels of abstraction, thereby can discover intricate structures in high-dimensional data with very little engineering by hand . Convolutional neural network (CNN) is a type of feed-forward artificial neural network, which has many successes in image recognition, natural language understanding, and medical image analysis . It also can improve the detection and classification performance on HSI . An HSI-based optical biopsy method was proposed using CNN, which could provide multi-category diagnostic information for normal head-and-neck tissue, squamous-cell carcinoma, and thyroid carcinomas [45, 46, 47]. A CNN-based modeling framework was introduced for the analysis of hyperspectral images for the detection of head and neck cancer in an animal model . A modified inception-v4 CNN architecture was used to detect the squamous cell carcinoma . In addition, several CNN-based architectures with pixel-wise prediction have shown their efficiency in the segmentation or detection task, such as fully connected networks (FCN) , SegNet , and U-Net . The U-Net deep neural network was used for the tumor segmentation  and the breast tumor detection  in hyperspectral images.
Since hyperspectral imagery has the system noise and image artifacts arising from uneven surface illumination, the tumor margin is irregular and unclear. So it is difficult to distinguish a tumor from surrounding normal tissue. In this study, we proposed an automated cancer detection algorithm for highlighting the tumor by adaptive auto-encoder network learning. Auto-encoder is an unsupervised deep neural network that can learn the inherent features and extract the suitable representation from complex data automatically. We involved the auto-encoder network to learn and recognize the depth features of pixels in hyperspectral imagery for the initial cancer detection. Each pixel is assigned a weight according to its classification result. The proposed adaptive auto-encoder learning method is performed on these weighted pixels and is trained to correct the misclassified pixels for the improvement of the detection performance. In this study, we demonstrate the efficiency and effectiveness of the auto-encoder and adaptive deep learning in HSI for head and neck cancer detection in an animal model. The method and experiments are described in the following sections.
Hyperspectral images were obtained by a wavelength-scanning CRI Maestro in vivo imaging system. This instrument mainly consists of a flexible fiber-optic lighting system, a solid-state liquid crystal filter, a spectrally optimized lens, and a 16-bit high-resolution charge-coupled device. For image acquisition, the wavelength setting can be defined within the range of 450 to 950 nm with 2-nm increments. Further details can be referred in our previous paper [10, 55].
The proposed adaptive deep learning method
Deep feature learning
The auto-encoder network can identify the cancer pixels and healthy pixels. Then we can obtain the initial detection result of cancer and the output hypothesis of pixels.
Adaptive weight learning
Eqs. (4) and (5) show that the initial detected tumor is refined in two ways. If the detected tumor lost some relevant regions, then weight_se() could assign the weights to those false negative pixels and the updated image could focus on those pixels. So the new auto-encoder learner on the updated images tends to expand the relevant pixels to improve the sensitivity of tumor detection. If the detected tumor contained some irrelevant regions, then weight_sp() could assign the weights to those false positive pixels, and the updated image could highlight those pixels. So the new learner tends to eliminate the irrelevant pixels to improve the specificity of tumor detection. Thus, the pixels classified correctly keep their correct prediction while the pixels misclassified change their values adaptively. Therefore, the adaptive learner heightens the ability to identify the tumor and healthy tissue.
In this paper, the refined type improving the sensitivity or specificity will be determined by experiments according to the detection performance of training images.
Since our method is based on the classification of each pixel, the detected tumor may contain some noise and holes. The flood-fill operation is used to fill holes in the segmented binary image and the biggest connected component is chosen as our detected cancer tissue.
All methods were carried out in accordance with the approved Institutional Animal Care and Use Committee protocol (YER-2003103-042918BN) and the relevant guidelines and regulations of Emory University. We acquired the hyperspectral reflectance images from 12 tumor-bearing mice approximately 2 weeks post-tumor cell injection. The reflectance images contained 251 spectral bands and the image size on each spectral band was 390 × 435. Therefore, the data cube collected was a three-dimensional array of the size 390 × 435 × 251. In this study, tumor cells had green fluorescence protein (GFP) signals and thus GFP images were also acquired as the reference standard to evaluate the proposed tumor detection algorithm.
We conducted leave-one-out cross-validation experiments for the tumor detection in hyperspectral images. We take each hyperspectral image as the testing sample in turn, and the 11 remaining samples as the training samples.
Advantage of auto-encoder
Advantage of adaptive weight learning
In Fig. 9, we can see that the initial cancer detection results are sensitive to the blood vessel and uneven surface. Since the improved cancer detection method focuses the misclassified pixels with the help of adaptive weight, it is robust to blood vessel, uneven surface and noise, like the detection results of the first and third mouse. Even it can achieve good performance when the tumor has the irregular or unclear margin, like the detection results of the second and fourth mouse. The final detection results refined by post-processing are satisfactory. However, we also can see the performance as shown in the last column is not satisfied. That because the difference between the tumor and normal tissue is too obscure, the intra-similarity between the tumor in the center and the tumor at the edge is much bigger than the inter-similarity between the tumor at the edge and the sounding normal tissue. But we can see that our adaptive auto-encoder method still works better than the auto-encoder method even though when an edge part of the tumor disappears. Hence, our adaptive weight learning is effective.
The performance of cancer detection for 12 mice
In this study, we proposed automated detection method for head and neck cancer using the adaptive deep learning on hyperspectral imagery in an animal model. Auto-encoder network model is involved to extract the deep features from a hyperspectral imagery with size of 390 × 435 × 251 and distinguish the cancerous tissue from its surrounding normal tissue. Because of the noise and uneven surface and so on, the detected cancer region is not satisfactory. To improve the initial performance and obtain a complete tumor, the adaptive auto-encoder network model is proposed, which focus on the misclassified pixels and enhance to learn for the misclassified pixels. The method is shown to classify the tumor region with high sensitivity, specificity, and accuracy.
Since reflectance hyperspectral images contain 251 spectral bands from 450 to 950 nm with 2 nm increments, and each hyperspectral image contains over millions of reflectance spectral signatures, it is difficult to extract discriminant features from the huge data by hand. Deep learning methods can learn features by building high-level features from low-level ones and automatically discover the features needed for cancer detection. The auto-encoder is an unsupervised deep neural network that tries to denoise the inputs automatically by finding the latent representation from which to reconstruct the original input, hence it is especially suitable for describing the hyperspectral data. As shown in the Fig. 6, the extracted features by auto-encoder method can better distinguish the cancerous tissue from the non-cancerous tissues. In addition, the auto-encoder can achieve higher accuracy compared with the other neural network models, hence, the auto-encoder is used as the learner for the initial cancer detection.
Although auto-encoder can extract the useful information for detecting the cancer tissue, the blood vessel or uneven surface make it difficult to identify the complete tumor from the normal tissue. Since blood vessel or uneven surface could misrepresent the tissue structures, neither the traditional classification methods nor the deep learning methods could detect the tumor with high accuracy by learning the intensity or the distribution of intensity. The proposed adaptive auto-encoder network can predict the misrepresented tissue structures into their true classes by adaptively weighting those misclassified pixels, and thus greatly improve the performance of tumor detection, as shown in Figs. 7, 8, 9 and Table 1. Under the same conditions, our proposed method performed better than the traditional classification methods [10, 31], and the CNN based deep learning method [45, 50, 51, 52]. However, we obtain unsatisfied performance on some mice. In the next work, we plan to improve the cancer detection performance on those special images. We will learn the deep feature by iteratively updating the sensitivity-weights and specificity-weights until convergence to overcome the effects of noise and artifacts.
The automatic detection algorithm was written and run in MATLAB on Intel Core 2.60GHz CPU with 16GB of RAM. The time for normalization, deep feature extraction, cancer detection, post-processing is about 0.1 s, 2.8 s, 3.2 s, and 0.02 s, respectively. The total running time is about 6 s for per hyperspectral image. It greatly improved the efficiency of cancer detection compared with the method  using 45 min. This automatic cancer detection method can be implemented in real time if involving the multi-thread, GPU acceleration or parallel programming.
In this study, an adaptive deep learning framework was proposed and validated for head and neck cancer detection using HSI in an animal model. This algorithm extracted the deep feature of hyperspectral images for reducing the dimensionality effectively and better characterizing the cancerous tissue. The adaptive weight learning could improve the cancer detection performance by focusing on harder-to-classify pixels. In the head and neck cancer mouse model, the proposed cancer detection method was able to obtain a high sensitivity and specificity. The results demonstrated that the HSI combined with deep learning technique may enable accurate and fast detection of cancers in a noninvasive manner and may provide a promising tool for future clinical applications.
The author thanks the members of the Quantitative Bioimaging Laboratory (https://www.fei-lab.org) for their helpful discussions.
All methods were carried out in accordance with the approved Institutional Animal Care and Use Committee (IACUC) protocol (YER-2003103-042918BN) and the relevant guidelines and regulations of Emory University.
LM, GL, XQ, BF participated in the literature search, data analysis, manuscript writing and editing; DW, ZGC participated in the experiment design; all the authors read and approved the final manuscript.
This work was supported in part by NIH grants (R01CA204254, R01HL140325, and R21CA231911).
The authors declare that they have no competing interests.
- 1.Mehanna H, Beech T, Nicholson T, El-Hariry I, McConkey C, Paleri V et al (2013) Prevalence of human papillomavirus in oropharyngeal and nonoropharyngeal head and neck cancer-systematic review and meta-analysis of trends by time and region. Head Neck 35(5):747–755 https://doi.org/10.1002/hed.22015 CrossRefGoogle Scholar
- 7.Borengasser M, Hungate WS, Watkins R (2007) Hyperspectral remote sensing: principles and applications. CRC Press, Boca Raton, FL, USA. DOI: https://doi.org/10.1201/9781420012606
- 9.Chung H, Lu GL, Tian ZQ, Wang DS, Chen ZG, Fei BW (2016) Superpixel-based spectral classification for the detection of head and neck cancer with hyperspectral imaging. In: abstracts of SPIE 9788, medical imaging 2016: biomedical applications in molecular, structural, and functional imaging, SPIE, San Diego, CA, USA, 29 March 2016, p 978813 DOI: https://doi.org/10.1117/12.2216559
- 17.Kho E, de Boer LL, Van de Vijver KK, Sterenborg HJCM, Ruers TJ (2018) Hyperspectral imaging for detection of breast cancer in resection margins using spectral-spatial classification. In: abstracts of SPIE 10472, diagnosis and treatment of diseases in the breast and reproductive system IV, SPIE, San Francisco, CA, USA, 14 March 2018, p 104720F DOI: https://doi.org/10.1117/12.2288367
- 18.Gopi A, Reshmi CS, Aneesh RP (2017) An effective segmentation algorithm for the hyperspectral cancer images. In: abstracts of 2017 international conference on networks & advances in computational technologies, IEEE, Thiruvanthapuram, India, 20 July 2017, pp 294-299 DOI: https://doi.org/10.1109/NETACT.2017.8076783
- 19.Zarei N, Bakhtiari A, Gallagher P, Keys M, MacAulay C (2017) Automated prostate glandular and nuclei detection using hyperspectral imaging. In: abstracts of the IEEE 14th international symposium on biomedical imaging, IEEE, Melbourne, VIC, Australia, 18 April 2017, pp 1028-1031 DOI: https://doi.org/10.1109/ISBI.2017.7950691
- 20.Fei BW, Lu GL, Halicek MT, Wang X, Zhang HZ, Little JV, et al (2017) Label-free hyperspectral imaging and quantification methods for surgical margin assessment of tissue specimens of cancer patients. In: abstracts of the 39th annual international conference of the IEEE engineering in medicine and biology society, IEEE, Seogwipo, South Korea, 11 July 2017, pp 4041-4045 DOI: https://doi.org/10.1109/EMBC.2017.8037743
- 22.Lu GL, Qin XL, Wang DS, Muller S, Zhang HZ, Chen A, et al (2016) Hyperspectral imaging of neoplastic progression in a mouse model of oral carcinogenesis. In: abstracts of SPIE 9788, medical imaging 2016: biomedical applications in molecular, structural, and functional imaging, SPIE, San Diego, CA, USA, 29 March 2016, p 978812 DOI: https://doi.org/10.1117/12.2216553
- 25.de Koning SG, Karakullukcu MB, Smit L, Baltussen EJM, Sterenborg HJCM, Ruers TJM (2018) Near infrared hyperspectral imaging to evaluate tongue tumor resection margins intraoperatively. In: abstracts of SPIE 10469, optical imaging, therapeutics, and advanced technology in head and neck surgery and otolaryngology 2018, SPIE, San Francisco, CA, USA, 14 March 2018, p 104690GGoogle Scholar
- 30.Akbari H, Halig LV, Zhang HZ, Wang DS, Chen ZG, Fei BW (2012) Detection of cancer metastasis using a novel macroscopic hyperspectral method. In: abstracts of SPIE 8317, medical imaging 2012: biomedical applications in molecular, structural, and functional imaging, SPIE, San Diego, CA, USA, 14 April 2012, p 831711 DOI: https://doi.org/10.1117/12.912026
- 33.Nathan M, Kabatznik AS, Mahmood A (2018) Hyperspectral imaging for cancer detection and classification. In: abstracts of the 3rd biennial south African biomedical engineering conference, IEEE, Stellenbosch, South Africa, 4 April 2018, pp 1-4 DOI: https://doi.org/10.1109/SAIBMEC.2018.8363180
- 35.Ortega S, Callicó GM, Plaza ML, Camacho R, Fabelo H, Sarmiento R (2016) Hyperspectral database of pathological in-vitro human brain samples to detect carcinogenic tissues. In: abstracts of the IEEE 13th international symposium on biomedical imaging, IEEE, Prague, Czech Republic, 13 April 2016, pp 369-372 DOI: https://doi.org/10.1109/ISBI.2016.7493285
- 36.Calin MA, Parasca Sr SV, Manea D (2018) Comparison of spectral angle mapper and support vector machine classification methods for mapping skin burn using hyperspectral imaging. In: abstracts of SPIE 10677, unconventional optical imaging, SPIE, Strasbourg, France, 13 August 2018, p 106773P DOI: https://doi.org/10.1117/12.2319267
- 40.Lall M, Deal J, Hill S, Rider P, Boudreaux C, Rich T, Leavesley S (2017) Classification of normal and Lesional colon tissue using fluorescence excitation-scanning hyperspectral imaging as a method for early diagnosis of colon cancer. In: abstracts of the national conference on undergraduate research, University of Memphis, Memphis, TN, USA, 6-8 April 2017, pp 1063-1073Google Scholar
- 44.Makantasis K, Karantzalos K, Doulamis A, Doulamis N. (2015) Deep supervised learning for hyperspectral data classification through convolutional neural networks. In: Abstracts of 2015 IEEE international geoscience and remote sensing symposium, IEEE, Milan, Italy, 26 July 2015, pp 4959–4962 DOI: https://doi.org/10.1109/IGARSS.2015.7326945
- 47.Halicek M, Little JV, Wang X, Patel M, Griffith CC, Chen AY, et al (2018) Tumor margin classification of head and neck cancer using hyperspectral imaging and convolutional neural networks. In: abstracts of SPIE 10576, medical imaging 2018: image-guided procedures, robotic interventions, and modeling, SPIE, Houston, TX, United States, 12 March 2018, p 1057605 DOI: https://doi.org/10.1117/12.2293167
- 48.Ma L, Lu GL, Wang DS, Wang X, Chen ZG, Muller S, et al (2017) Deep learning based classification for head and neck cancer detection with hyperspectral imaging in an animal model. In: abstracts of SPIE 10137, medical imaging 2017: biomedical applications in molecular, structural, and functional imaging, SPIE, Orlando, FL, USA, 13 March 2017, p 101372G DOI: https://doi.org/10.1117/12.2255562
- 49.Halicek M, Dormer JD, Little JV, Chen AY, Myers L, Sumer BD et al (2019) Hyperspectral imaging of head and neck squamous cell carcinoma for cancer margin detection in surgical specimens from 102 patients using deep learning. Cancers 11(9):1367 https://doi.org/10.3390/cancers11091367 CrossRefGoogle Scholar
- 50.Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: abstracts of 2015 IEEE conference on computer vision and pattern recognition, IEEE, Boston, MA, USA, 7-12 June 2015, pp 3431-3440 DOI: https://doi.org/10.1109/CVPR.2015.7298965
- 52.Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: abstracts of the 18th international conference on medical image computing and computer-assisted intervention, Springer, Munich, Germany, 5-9 October 2015, pp 234-241 DOI: https://doi.org/10.1007/978-3-319-24574-4_28
- 53.Trajanovski S, Shan CF, Weijtmans PJC, de Koning, SGB, Ruers TJM (2019) Tumor semantic segmentation in hyperspectral images using deep learning. In: Abstracts Proceedings of the 2nd international conference on medical imaging with deep learning, MIDL, London, UK, 7 July 2019, pp 8–10Google Scholar
- 55.Lu GL, Halig L, Wang DS, Chen ZG, Fei BW (2014) Spectral-spatial classification using tensor modeling for cancer detection with hyperspectral imaging. In: abstracts of SPIE 9034, medical imaging 2014: image processing, SPIE, San Diego, CA, United States. 21 March 2014, p 903413 DOI: https://doi.org/10.1117/12.2043796
- 56.Hinton GE, Zemel RS (1993) Autoencoders, minimum description length and Helmholtz free energy. In: abstracts of the 6th international conference on neural information processing systems, Morgan Kaufmann publishers Inc., Denver, CO, USA, 2 December 1993, pp 3-10Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.