Abstract
Measuring morphological and biochemical features of tissue is crucial for disease diagnosis and surgical guidance, providing clinically significant information related to pathophysiology. Hyperspectral imaging (HSI) techniques obtain both spatial and spectral features of tissue without labeling molecules such as fluorescent dyes, which provides rich information for improved disease diagnosis and treatment. Recent advances in HSI systems have demonstrated its potential for clinical applications, especially in disease diagnosis and image-guided surgery. This review summarizes the basic principle of HSI and optical systems, deep-learning-based image analysis, and clinical applications of HSI to provide insight into this rapidly growing field of research. In addition, the challenges facing the clinical implementation of HSI techniques are discussed.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
1 Introduction
Recent advances in optical imaging methods have improved the accuracy of disease diagnosis and surgical operation under clinical conditions [1]. Compared to other medical imaging technologies such as magnetic resonance imaging, computed tomography, and positron emission tomography, optical imaging methods are cost-effective, safe, and easy to use in clinics [1]. Moreover, optical imaging provides a high-resolution structural image with a compact optical system. Given these advantages, many optical imaging methods, including optical coherence tomography [2], holographic imaging [3, 4], and photoacoustic tomography [5, 6], have been developed for biomedical applications in the last few decades.
Among them, the spectrally resolved imaging method has recently demonstrated a wealth of possibilities for disease diagnostic and surgical tools in clinics [7, 8]. This technique could capture the morphological and biochemical characteristics required for accurate disease diagnosis or functional analysis during a surgical operation. The spectrally resolved imaging methods are classified into ‘multispectral’ and ‘hyperspectral’ imaging technologies based on the number of measured spectral channels. ‘Multispectral’ imaging captures a few spectral features (less than 30 channels). For example, a standard digital color camera, a simple multispectral imaging system, measures three colors (red, green, and blue) by replicating human vision (Fig. 1a). Although a color imaging method has been widely used in clinics due to its simple and powerful capabilities in examining tissue samples, it is unable to capture information beyond the naturally observable by the human eye. On the other hand, ‘Hyperspectral’ imaging (HSI) obtains spectral information over 30 channels, which allows capturing complex spectrally-resolved light-tissue interactions. HSI measures 2D structural and 1D spectral information in the form of a 3D dataset, called a ‘hypercube’ (Fig. 1b, c). The hypercube enables the discrimination or classification of types of a sample, which is unable to be achieved by conventional color imaging methods. Therefore, HSI techniques have been widely used in various applications, which includes remote sensing [9], food inspection [10], recycling [11], forensic science [12], forgery detection [13], and biomedical application [7, 8, 14–18].
Schematic of hyperspectral imaging. a A conventional color image consists of three spectral channels. b Hyperspectral image shows a 3D hypercube containing spectral and spatial information. The hyperspectral image is experimentally obtained via using a liquid crystal tunable filter. c Plots show spectral features of the four dashed square areas shown in (b). d Experimentally measured absorbance of oxy-/deoxy hemoglobin from mice blood via a spectrometer
This review focuses on and summarizes the recent progress of HSI technologies in clinics. Owing to rapid advances in the light source, detector, image processing method, and computing power, HSI methods have recently been translated from the research laboratory to the clinic. In addition, many spectral imaging devices are commercially available, which accelerates the clinical application of HSI methods. This review aims to mainly discuss the principle of various HSI systems and focus on ongoing clinical applications of HSI technologies. This review also discusses challenges in translating the spectral imaging system into routine clinical practice, and the perspectives for future developments and applications of clinical HSI techniques.
2 Principles of Hyperspectral Imaging Methods
2.1 Light-Tissue Interactions
HSI measures reflected light that interacts with tissue, containing information about structural and spectral features. When the light meets tissue, there are complicated light-tissue interactions, such as absorption, scattering, or transmission. Biomolecules show distinct absorption properties, which means spectrally-resolved imaging methods could predict biochemical features by measuring the amount of attenuated light signal as a function of wavelength [8]. For example, one of the primary attenuators in tissue is haemoglobin. This shows distinct absorption coefficients corresponding to oxygen binding (Fig. 1d). Therefore, HSI has shown great potential to detect blood vessels and quantify blood oxygen levels. In addition, some biomolecules, such as NADPH and FAD reradiate fluorescent light of longer wavelength, called autofluorescence [19, 20]. These biomolecules are known to be altered during disease progression, and thus autofluorescence signals are also good targets for HSI methods.
Elastic light scattering occurs owing to heterogeneous refractive index distribution in tissue [21]. Tissue is an ensemble of various substances ranging from nanometers to hundreds of microns. However, there are changes in tissue morphology during disease progression, resulting in varying overall scattering properties [22–24]; scattering features can be a potential target for disease diagnostic markers. Therefore, the HSI of reflected light includes rich information about the structural and biochemical features of tissue, which enables accurate disease diagnosis and functional analysis during surgical operations. Other light-tissue interactions, such as inelastic scattering (Raman and Brillouin scattering) and photoacoustic effect, are not discussed here because they are beyond the scope of this review.
2.2 Working Principle of Hyperspectral Imaging Systems
The fundamental working principle of hyperspectral imaging technology is to measure an image as a function of wavelength using spectroscopic techniques. Therefore HSI methods require optical components that enable the detection of spectral information from broadband light signals. HSI methods are mainly categorized into three methods: spectral scanning, spatial scanning, and snapshot methods. Each HSI method has a different spatial and spectral resolution and imaging speed; it is very important to select the method that is best suited to target and imaging conditions. Furthermore, as these methods could be implemented as illumination or detection units, the conventional microscopic and mesoscopic optical imaging systems could be transformed into HSI optical systems. This section summarizes three HSI methods with a discussion of the pros and cons.
2.2.1 Spectral Scanning HSI Method
The spectral scanning HSI method is to acquire images at different wavelengths sequentially. It measures a 2D structural image at specific wavelength ranges in a single measurement, and then wavelength ranges are changed to obtain the whole hyperspectral image. The spectral scanning method usually exploits spectral filters and a monochrome camera (Fig. 2a, b). As the filter only allows the transmission of light in a narrow spectral range, the image captured by a monochrome camera could be considered as a 2D structural image of the target wavelength. Spectral scanning could be implemented on either illumination and detection sides (Fig. 2d).
Representative spectral detection methods and hyperspectral imaging techniques. a Various filters transmit light with specific spectral ranges. b A monochromator selects illumination wavelength by scanning grating angles. A spectrometer allows the measurement of a spectral image via the camera. c A snapshot sensor consists of mosaic filter arrays placed in front of the camera sensor. d Spectral scanning methods can be implemented from the illumination and detection sides. e Spatial scanning methods measure spectral information via point and line scanning procedures. f Snapshot methods capture spectral-spatial image with a single measurement. Images at the bottom of (d–f) show representative spectral images captured by each imaging method, respectively
Optical bandpass filters allow the selection of illuminating wavelength from a broadband light source. According to the number of filters, multiple images at different wavelengths could be measured. Another approach is to use a tunable light source implemented by exploiting a thin-film tunable filter [25], acoustic optic tunable filter (AOTF) [26], and monochromator [27, 28]. A tunable light source could adjust a wavelength of illumination light with spectral resolution between a few and tens of nanometers. Recently, a digital micromirror device [29] or multiple light-emitting devices [30, 31] have been used to illuminate light with different wavelengths.
The spectral scanning method could also be implemented on the detection side using optical bandpass filters [8], liquid–crystal tunable filter [15, 32, 33], and AOTF [14, 34, 35]. These filters are placed in front of the camera, allowing transmission of a specific wavelength of the reflected light. A broadband light source is normally used as a light source, and the whole hyperspectral image is acquired by adjusting the wavelength ranges of a filter.
The spectral scanning method requires minimal modifications, such as adding a spectral scanning unit on the illumination or detection sides; it can readily transform conventional optical systems into a hyperspectral imaging system. Therefore, the spectral scanning method has been widely used in developing hyperspectral microscopy to examine tissue samples. Even though the spectral scanning method is simple to implement, there are a few limitations. Additional time for adjusting the wavelength range of the filters is inevitable to obtain a whole hyperspectral image, which causes image distortion among spectral images with different wavelengths. Thus, an image registration process might be necessary, limiting the application of the spectral scanning method under unstable clinical conditions. Another limitation is relatively low spectral resolution. Because of the nature of optical filters, there are varying spectral resolutions ranging from a few to tens of nanometers across the wavelength.
2.2.2 Spatial Scanning HSI Method
Dispersive optical elements such as prism and grating are key components for a spatial scanning HSI method (Fig. 2b). Reflected light from a single point (point scanning) or single line (line scanning) is measured via a spectrometer or spectrograph [7, 36], respectively. The measured signal shows the light intensity as a function of wavelength, but there is missing spatial information. To acquire the entire hyperspectral image of a target using the spatial scanning method, an optical system or target scanning procedure is necessary (Fig. 2e). For scanning positions of the imaging area, a motorized stage or Galvano mirrors are utilized. A narrow bandwidth linear variable filter also enables the measurement of a hyperspectral image by sliding the filter during measurements [37, 38]. Spectral properties of this filter vary continuously along one axis of the filter, allowing each row or column of the camera pixels to measure spectral information at a different wavelength. Thus, by sliding the filter laterally, each camera pixel captures both spatial and spectral information in the end [37].
The spatial scanning method usually provides higher spectral resolution than the spectral scanning method by exploiting dispersive materials [8]. However, when the target shows high motility, the spatial scanning procedure causes huge artifacts. Therefore, it requires image post-processing for hyperspectral image reconstruction from multiple spectral information measured at points or lines. To overcome these limitations, spectral and wide-field images are captured simultaneously to exploit wide-field images as guidance for hyperspectral image reconstruction [39]. But if there are huge motion artifacts, it is very challenging to implement a hyperspectral imaging system based on a spatial scanning method.
2.2.3 Snapshot Imaging Method
Snapshot method is a ‘non-scanning' method that acquires spatial and spectral information in a single measurement [40]. The snapshot method has been intensively reviewed in the previous review article [41]. One of the snapshot methods exploits multiple narrow-band bandpass filter arrays based on the Fabry-Perot interferometer (Fig. 2c) [42]. According to the length of an optical cavity made from two parallel reflecting mirrors, only light with a specific wavelength can be transmitted. Therefore, multiple filter arrays are placed in front of the camera sensor; it could measure both spatial and spectral information without scanning procedures (Fig. 2f). Even though snapshot methods compromise spatial and spectral resolution, they allow fast imaging and easy implementation of the spectral imaging method to the conventional optical imaging systems by adding filter arrays to the image sensor. As each camera pixel measures spectral information with a different wavelength, the ‘demosaicing’ process is necessary to obtain structural images that correspond to spectral channels. [42].
Moreover, compact spectroscopic techniques using random spectral filters and image processing enable the development of a snapshot hyperspectral imaging system. For example, photonic crystal (PC) slabs could produce spectral filters with random spectral profiles based on the sizes and structures of PC slabs [43]. These random spectral filters are placed in front of the camera sensor, and the original spectral features can be reconstructed using least-square methods or compressive sensing techniques. Even though a snapshot method with the random spectral filters-based snapshot methods requires image post-processing, it enables single-shot hyperspectral imaging with a spectral resolution of a few nanometers.
2.2.4 Medical Hyperspectral Imaging System
HSI methods could be implemented as microscopic and mesoscopic imaging systems to observe and examine various scales from cell to tissue. According to target and imaging conditions, HSI methods should be carefully selected. The strengths and weaknesses of each HSI method are summarized in Table 1. The line scanning method provides high spatial and spectral resolution; thus, it has been used to obtain a high-quality hyperspectral image. However, the mechanical scanning procedure limits the measurement of a target with high motility. Several approaches have been proposed to overcome this limitation, but the HSI of fast-moving tissue using spatial scanning methods remains very challenging. The spectral scanning method could sequentially capture spatial images at different wavelengths without mechanical movement of the optical system, allowing very stable imaging environments. Even though the spectral resolution of the spectral scanning method is poor compared to that of the spatial scanning method, the spectral scanning method has been widely used in clinics because the optical system is stable, and easy to build.
Non-scanning HSI methods, such as a snapshot method, have shown their capability as medical hyperspectral imaging techniques. The snapshot method could simultaneously capture both spatial and spectral information, which enables real-time spectral imaging. As real-time imaging is very important in medical imaging areas, the snapshot method has been continuously developed to improve spatial and spectral resolution, while maintaining imaging speed [44–47].
3 Deep-Learning-Based Hyperspectral Image Analysis Methods
Due to the high complexity of hyperspectral image data, there are many image analysis methods, including spectral unmixing, spectral angle mapper, principal component analysis, and statistical analysis methods, that have been developed to extract physiological and diagnostic features for clinical applications. These image analysis methods have been intensively overviewed in other review articles [7, 48]. This article focuses on convolutional neural networks (CNNs) based on hyperspectral image analysis methods. As described in the previous section, biomolecules show distinct spectral profiles, which means spectral information is a mixture of various spectral profiles of biomolecules. Therefore, hyperspectral image analysis methods try to decompose the spectral signal into a set of known spectral signatures (endmembers) for the quantitative analysis of biochemical properties in tissue. However, intra-/inter-patient variabilities make it difficult to accurately decompose the hyperspectral signals. To overcome these variabilities, data-driven hyperspectral image analysis methods have been suggested via exploiting statistical analysis methods such as machine learning [7]. Statistical machine learning methods, including k-nearest neighbors, support vector machine, and random forest, show promising clinical results, but these methods require a feature extraction process which makes it challenging to translate hyperspectral imaging techniques into clinical practice.
Recently, the CNN, one of the deep learning methods, has become the most successful and popular image analysis method [49–55]. Many CNN models have been proposed for hyperspectral data classification and regression. The architecture of a CNN model is composed of input, hidden, and output layers. Hidden layers consist of multiple stacks of convolution, activation, pooling, and fully connected layers, enabling automatic feature extraction, classification, and regression. Among various layers, a convolutional layer is a basic structural unit that convolves the input data with kernels (filters). During the training process of a CNN model, the weight values, including kernels and bias, are optimized to get the most accurate outputs. A CNN model could be trained via supervised or unsupervised learning. Supervised learning requires ground-truth data which are usually obtained by pathological examination or medical doctor’s opinion. On the other hand, unsupervised learning is applied when due to the complicated nature of the disease, ground-truth information is unable to be obtained.
CNN models for analyzing hyperspectral data could be categorized based on the input data dimension: spectral (1D) [52, 55], spatial (2D) [51, 54, 56], and spectral-spatial (3D) [53, 56] inputs (Fig. 3). A spectral model, called a pixel-wise model, analyses spectral profiles without considering spatial features. As spectral information is a mixture of various spectral profiles of biomolecules, clinically significant features can be directly extracted from spectral information. To detect features from a 1D spectral signal, convolution is performed via 1D kernels (Fig. 3a). Another approach to analyzing 1D spectral signal is to reshape a 1D spectral signal into a 2D image, then apply 2D convolution for the CNN model [52]. The advantage of using the spectral model is the collection of large data for training and validation of the CNN model. For example, if a hyperspectral image consists of a 1024 × 1024 pixels image with 100 spectral channels, then a 1024 × 1024 spectral dataset can be obtained from the single hyperspectral image. Moreover, 1D convolution is computationally very efficient and fast. Given these advantages, the pixel-wise CNN model has been widely used in clinical hyperspectral data analysis.
Three convolutional neural networks based hyperspectral image analysis methods. a Spectral (pixel-wise) methods analyze a single spectral feature independently via 1D convolutions. b The spatial method exploits 2D convolutions to extract spatial features for data analysis. c The spectral-spatial method performs 3D convolution to fully exploit the spatial and spectral information of a hypercube. Grey areas (a–c) indicated in 1D, 2D, and 3D kernels for convolution operation, respectively
Although the spectral method is promising, it ignores spatial features. Malignant or abnormal tissue usually has an irregular shape and unclear margin, which means tissue morphology is also a significant clinical feature. To obtain clinically meaningful output from tissue morphology, a spatial model has been used to train a CNN model using 2D images at different wavelengths as input data (Fig. 3b). To train the CNN model, 2D kernels are employed for convolution layers that extract spatial features. The trained 2D CNN model successfully demonstrated the potential for disease diagnosis or functional imaging under clinical conditions.
The spectral-spatial model exploits the full information of hyperspectral data via using 3D convolution (Fig. 3c). 3D convolution is usually employed in the reconstruction and analysis of 3D imaging techniques such as magnetic resonance imaging [57] and computed tomography [58]. 3D convolution of hyperspectral data can be done via employing 3D kernels scanned along three dimensions, including x, y, and spectral axes. As this approach trains the CNN model using full hyperspectral data, the output results involve all tissue morphological and biochemical features. However, there are challenges in the requirement of heavy computational power due to 3D convolution and large numbers of hyperspectral data compared to spectral and spatial models.
Given flexible architectures, CNNs are becoming powerful and accurate artificial intelligence (AI) models for hyperspectral data analysis. A CNN model could be trained and established via spectral, spatial, and spectral-spatial inputs. Although the CNN-based hyperspectral data analysis methods show great potential in clinical applications, there are still many challenges. To train the CNN model via supervised learning, accurate ground-truth data is necessary. Current ground-truth data in clinics can be obtained from pathological analysis. But performing pathological analysis in in-vivo patients is very challenging. One strategy to get hyperspectral information with the ground-truth data is to collect tissue from a patient, then perform hyperspectral imaging, followed by a pathological inspection. This provides spectral signals with the ground-truth labeling, but spectral features vary between in-vivo and ex-vivo tissue due to blood supply and oxygen levels that hamper clinical applications of the CNN-based hyperspectral image analysis. Another challenge is the standardization of hyperspectral imaging systems and data. Measured spectral information varies according to the imaging system and conditions, hampering the build of a standard CNN model for clinical applications.
3.1 Clinical Applications of HSI Technology
Due to the aforementioned advantages, an increasing number of studies employ HSI for clinical trials as disease diagnostic and surgical guidance tools. Commercially available HSI systems and new spectral imaging systems developed in laboratories have accelerated the translation of HSI techniques in clinics. The following section summarizes and highlights the representative clinical applications of HSI techniques performed in in-vivo and ex-vivo human tissue.
3.1.1 Disease Diagnosis
The rationale underlying HSI-based disease diagnosis is that morphological and biochemical alterations related to disease progression change the optical properties of tissue. [59] For example, malignant cells require huge amounts of nutrients and oxygen due to rapid cell division. Therefore, there are increased metabolic enzyme levels and newly formed vessels via the angiogenesis process to support oxygen and nutrients to malignant cells [60]. These changes allow HSI to detect lesions and abnormal tissue without histological examination, which has the advantages of saving time and improving treatment efficacy.
3.1.2 Ex-vivo Tissue Imaging
As the biopsy samples such as cells and tissue obtained from a patient are stable, the scanning HSI method could be easily implemented as microscopic and mesoscopic HSI systems that provide high spatial and spectral resolution. Many recent studies have focused on studying whether there are correlations between hyperspectral images and histological examination results. If HSI could accurately delineate lesions without histology, it would save the time required for histological examination and improve treatment efficacy, due to rapid diagnosis. Recently, many reports have validated the capability of HSI as disease diagnostic tools from various tissues, including breast [61–63], liver [64], brain [65], kidney [66], stomach [67, 68], head & neck [69, 70], and thyroid gland [71]. As a hyperspectral image is complicated, artificial intelligence is employed for accurate disease diagnosis, and compared to histology, it shows comparable diagnostic accuracy (Fig. 4a) [72].
Clinical applications of HSI. a Color image, pathology image, and corresponding hyperspectral image with a classification result from the resected colorectal tumor. HSI-based classification was performed via the trained artificial model. Reprinted from Ref. [72] with permission. b White-light endoscopic images and hyperspectral signals obtained from in-vivo human colon. Representative color images (top) of normal mucosa, poly, and post-resection tissue. Blue dashed lines indicate polyp annotated by a clinician. Bottom graphs show representative hyperspectral signals of each tissue type obtained from the grey areas above. Reprinted from Ref. [82] with permission
HSI-based disease diagnosis allows direct examination of biopsy tissue during surgery. Currently, histology is necessary to examine biopsy tissue, which takes at least a few hours. However, HSI could analyze the tissue within a few minutes, which enables checking whether there are residual tumor cells on the margin of resection areas. Kho et al. reported that HSI could identify breast cancer from the excised breast tissue during surgery with an accuracy of over 84% [62]. Another HSI application is the identification of blood cells [73]. White blood cells were classified based on rich spectral information. These applications demonstrate that HSI could be employed to delineate abnormal tissue without biochemical techniques, which helps make rapid and accurate decisions in clinics.
3.1.3 In-vivo Patient Imaging
Although HSI shows promising results as disease diagnostic tools, there are issues risen from the biopsy tissue, which due to lack of oxygen and blood supply, has different physiological conditions than normal in-vivo tissue. To overcome this limitation, HSI has been performed in an in-vivo patient, which allows the measurement of functional information such as blood perfusion and tissue oxygen level.
As human skin and retina are easily accessible by the optical system, many studies have applied HSI in these tissues. Melanoma, a type of skin cancer, develops in cells producing the pigment known as melanin. Therefore, HSI has been employed for skin cancer diagnosis by exploiting the spectral information of melanin [74–76]. Moreover, the functional imaging capability of the HSI method enables the characterization of blood perfusion and tissue oxygen levels in the skin [77, 78]; thus, HSI is used for the evaluation of skin graft quality. [79].
Recently, HSI has been applied to diagnose Alzheimer’s disease (AD) via the retina [80, 81]. Amyloid beta (Aβ) is a potential marker for AD, but to detect Aβ in the brain is very challenging. Recent reports have shown that Aβ accumulated in the retina could be detected via HSI. Spectral features measured from the retina with AD showed different spectral responses, compared to that measured from a healthy person. These findings suggest that retinal HSI could be a potential AD diagnostic tool.
Hyperspectral endoscopy has been developed for imaging tissue inside the body, such as the gastrointestinal tract, via scanning [39, 82], and snapshot methods [55]. Yoon et al. reported an HSI endoscopy system with a flexible endoscope using a spatial-scanning method [39, 82]. Spectral images have been captured together with wide-field images; then, the wide-area hyperspectral image has been reconstructed by exploiting wide-field images as guidance for image registration. The in-vivo colon has been measured via the hyperspectral endoscope, and normal, polyp and resected colon tissues have been segmented based on spectral features (Fig. 4b). Waterhouse et al. demonstrated that a snapshot spectral imaging system and spectrometer enable the early detection of esophageal cancer via exploiting a deep-learning method [55].
HSI has also shown potential as a surgical guidance tool. In brain surgery, minimally invasive surgery is critical to preserve brain function. HSI was employed to detect brain tumor margin under in-vivo imaging conditions, which helps surgeons determine optimal excision areas [53, 83, 84]. Functional imaging via HSI has been applied to assess physiological properties, such as blood and oxygen supply in anastomotic areas after oesophagectomy [85], liver resection [86], and colorectal resection [87]. In addition, HSI could be used as quality assessment in organ transplantation via the measurement of functional information. [88].
3.2 Challenges in Clinical HSI systems
Even though HSI methods have shown their potential through preclinical and clinical studies, key challenges still remain for HSI to be practical in clinics. First, optimization of the HSI optical system is necessary. Currently, HSI has been performed in various spectral regions, ranging from UV to NIR. The research work, exploiting different spectral windows, showed results with varying accuracy of disease diagnosis [7, 67, 89]. Moreover, experimental conditions and parameters also affect the results. Thus, optimization of the HSI imaging system and experimental parameters should be investigated for the acquisition of high-quality data.
Second, there is a lack of methods for evaluating different HSI modalities and systems. According to spectral imaging methods, the optical system shows different specifications, such as spatial, spectral, and temporal resolution, which hampers the development of standard HSI systems. Recently, standard tissue-mimicking phantoms have been suggested for the quantitative evaluation of optical systems [1, 90]. This will help to overcome issues related to the standardization of the HSI optical system.
Third, a consistent and robust HSI optical system with real-time imaging speed is required. As summarized the strengths and weaknesses of each HSI method in Table 1, there are different performance parameters among HSI methods. However, the ultimate goal of the clinical HSI system is to get real-time information for accurate and efficient disease diagnosis and treatment. Therefore, a real-time HSI method with high-quality information would allow the acquisition of reliable and clinically important data from a patient.
Lastly, it is very challenging to perform quantitative 3D hyperspectral imaging in clinics. The quantitative 3D imaging capability of the optical imaging techniques is significant to define a boundary of lesions or diagnose metastasis. However, conventional HSI methods measure broadband light signals via spectroscopic approaches; thus, the 3D depth information is scrambled. One solution for obtaining a 3D hyperspectral image is to cut tissue into thin slices similar to histological tissue preparation. But it is impractical for rapid image acquisition. To overcome these limitations, spectral optical coherence tomography [91] and multispectral photoacoustic tomography [92] have been suggested to obtain 3D spectral information. However, it requires advances in optical systems for practical clinical applications.
By addressing the aforementioned challenges, HSI will become a versatile clinical tool in disease diagnosis and surgical guidance in the near future.
4 Summary and Outlook
In this review, the principles of HSI, image analysis methods, and clinical applications for disease diagnosis and surgical guidance have been presented. The recent trends of clinical HSI methods suggest that morphological and biochemical information may play a significant role in clinical practice, saving the time and resources required for histological examination. For example, histological examination puts most of the time on preparing a sample with sectioning, staining, and imaging. Although HSI is challenging to provide 3D quantitative depth information, HSI could diagnose lesions from measured structural and biochemical information of superficial tissue without sectioning and staining. Therefore, HSI could save the time required for histological examination in the case of lesions located on tissue surfaces like adenocarcinoma or melanoma.
This review summarizes the conventional optical system-based HSI methods. Moreover, there is an important research field of deep-learning-based HSI techniques. This has been actively studied recently to overcome the limitations of optical components. Deep learning enables the reconstruction of multi / hyperspectral images from an RGB image [93–97]. RGB images have just three channels of spectral information, but recent progress in deep learning could train the AI model via the RGB image and its original spectral information. The AI model learns the correlation between RGB images and their original spectral profiles, and then it successfully produces a multi/hyperspectral image from an RGB image. This process is similar to an AI-based image enhancing technique that improves image quality for better spatial resolution [98, 99]. Instead of enhancing spatial resolution, the AI-based spectral imaging method retrieves high-dimensional spectral information from sampled spectral images, such as RGB images. This means that a spectral image can be obtained from conventional color imaging systems without any modifications to the optical system. That is why many new AI models have been proposed for clinical applications of AI-based hyperspectral imaging technologies.
In addition to the development of clinical HSI systems, image analysis and classification methods are very important for clinical applications of HSI technology. As discussed in the previous section, HSI data obtained from different optical systems show a lot of heterogeneities. Therefore, many image analysis methods, including deep-learning-based methods, have been intensively proposed.
Considering the recent exponential growth of the clinical HSI methods, HSI will become a versatile clinical tool in disease diagnosis and surgical guidance in the near future by addressing the aforementioned challenges.
References
Waterhouse, D.J., Fitzpatrick, C.R., Pogue, B.W., O’Connor, J.P., Bohndiek, S.E.: A roadmap for the clinical implementation of optical-imaging biomarkers. Nat. Biomed. Eng. 3, 339–353 (2019)
Prati, F., et al.: Expert review document on methodology, terminology, and clinical applications of optical coherence tomography: physical principles, methodology of image acquisition, and clinical application for assessment of coronary arteries and atherosclerosis. Eur. Heart J. 31, 401–415 (2010)
Jung, J., et al.: Biomedical applications of holographic microspectroscopy. Appl. Opt. 53, G111–G122 (2014)
Park, Y., Depeursinge, C., Popescu, G.: Quantitative phase imaging in biomedicine. Nat. Photonics 12, 578–589 (2018)
Wang, L.V., Yao, J.: A practical guide to photoacoustic tomography in the life sciences. Nat. Methods 13, 627–638 (2016)
Luke, G.P., Yeager, D., Emelianov, S.Y.: Biomedical applications of photoacoustic imaging with exogenous contrast agents. Ann. Biomed. Eng. 40, 422–437 (2012)
Lu, G., Fei, B.: Medical hyperspectral imaging: a review. J. Biomed. Opt. 19, 010901 (2014)
Clancy, N.T., Jones, G., Maier-Hein, L., Elson, D.S., Stoyanov, D.: Surgical spectral imaging. Med. Image Anal. 63, 101699 (2020)
Van der Meer, F.D., et al.: Multi-and hyperspectral geologic remote sensing: a review. Int. J. Appl. Earth Obs. Geoinf. 14, 112–128 (2012)
Feng, Y.-Z., Sun, D.-W.: Application of hyperspectral imaging in food safety inspection and control: a review. Crit. Rev. Food Sci. Nutr. 52, 1039–1058 (2012)
Singh, N., et al.: Recycling of plastic solid waste: a state of art review and future applications. Compos. B Eng. 115, 409–422 (2017)
Edelman, G., Gaston, E., Van Leeuwen, T., Cullen, P., Aalders, M.: Hyperspectral imaging for non-contact analysis of forensic traces. Forensic Sci. Int. 223, 28–39 (2012)
Polak, A., et al.: Hyperspectral imaging combined with data classification techniques as an aid for artwork authentication. J. Cult. Herit. 26, 1–11 (2017)
Vo-Dinh, T.: A hyperspectral imaging system for in vivo optical diagnostics. IEEE Eng. Med. Biol. Mag. 23, 40–49 (2004)
Martin, M.E., et al.: Development of an advanced hyperspectral imaging (HSI) system with applications for cancer detection. Ann. Biomed. Eng. 34, 1061–1068 (2006)
Johnson, W.R., Wilson, D.W., Fink, W., Humayun, M.S., Bearman, G.H.: Snapshot hyperspectral imaging in ophthalmology. J. Biomed. Opt. 12, 014036 (2007)
Goto, A., et al.: Use of hyperspectral imaging technology to develop a diagnostic support system for gastric cancer. J. Biomed. Opt. 20, 016017 (2015)
Fei, B., et al.: Label-free reflectance hyperspectral imaging for tumor margin assessment: a pilot study on surgical specimens of cancer patients. J. Biomed. Opt. 22, 086009 (2017)
Blacker, T.S., Duchen, M.R.: Investigating mitochondrial redox state using NADH and NADPH autofluorescence. Free Radic. Biol. Med. 100, 53–65 (2016)
Bartolomé, F., Abramov, A.Y.: Measurement of mitochondrial NADH and FAD autofluorescence in live cells. Mitochondrial Med. 1264, 263–270 (2015)
Yu, H., et al.: Recent advances in wavefront shaping techniques for biomedical applications. Curr. Appl. Phys. 15, 632–641 (2015)
Collier, T., Arifler, D., Malpica, A., Follen, M., Richards-Kortum, R.: Determination of epithelial tissue scattering coefficient using confocal microscopy. IEEE J. Sel. Top. Quantum Electron. 9, 307–313 (2003)
Nandy, S., et al.: Characterizing optical properties and spatial heterogeneity of human ovarian tissue using spatial frequency domain imaging. J. Biomed. Opt. 21, 101402 (2016)
Volynskaya, Z.I., et al.: Diagnosing breast cancer using diffuse reflectance spectroscopy and intrinsic fluorescence spectroscopy. J. Biomed. Opt. 13, 024012 (2008)
Favreau, P.F., et al.: Excitation-scanning hyperspectral imaging microscope. J. Biomed. Opt. 19, 046010 (2014)
Sheoran, G., Dubey, S., Anand, A., Mehta, D.S., Shakher, C.: Swept-source digital holography to reconstruct tomographic images. Opt. Lett. 34, 1879–1881 (2009)
Castellanos-Gomez, A., Quereda, J., van der Meulen, H.P., Agraït, N., Rubio-Bollinger, G.: Spatially resolved optical absorption spectroscopy of single-and few-layer MoS2 by hyperspectral imaging. Nanotechnology 27, 115705 (2016)
Torabzadeh, M., et al.: Hyperspectral imaging in the spatial frequency domain with a supercontinuum source. J. Biomed. Opt. 24, 071614 (2019)
Wood, T.C., Elson, D.S.: A tunable supercontinuum laser using a digital micromirror device. Meas. Sci. Technol. 23, 105204 (2012)
Islam, K., Ploschner, M., Goldys, E.M.: Multi-LED light source for hyperspectral imaging. Opt. Express 25, 32659–32668 (2017)
Wang, H., et al.: An active hyperspectral imaging system based on a multi-LED light source. Rev. Sci. Instrum. 90, 026107 (2019)
Keller, M.D., et al.: Autofluorescence and diffuse reflectance spectroscopy and spectral imaging for breast surgical margin analysis. Lasers Surg. Med. 42, 15–23 (2010). https://doi.org/10.1002/lsm.20865
Wirth, D., et al.: Hyperspectral imaging and spectral unmixing for improving whole-body fluorescence cryo-imaging. Biomed. Opt. Express 12, 395–408 (2021)
Zhang, C., et al.: Narrowband double-filtering hyperspectral imaging based on a single AOTF. Opt. Lett. 43, 2126–2129 (2018)
Yushkov, K.B., Champagne, J., Kastelik, J.-C., Makarov, O.Y., Molchanov, V.Y.: AOTF-based hyperspectral imaging phase microscopy. Biomed. Opt. Express 11, 7053–7061 (2020)
Abdo, M., Badilita, V., Korvink, J.: Spatial scanning hyperspectral imaging combining a rotating slit with a Dove prism. Opt. Express 27, 20290–20304 (2019)
Luthman, A.S., Dumitru, S., Quiros-Gonzalez, I., Joseph, J., Bohndiek, S.E.: Fluorescence hyperspectral imaging (fHSI) using a spectrally resolved detector array. J. Biophotonics 10, 840–853 (2017)
Renhorn, I.G., Bergström, D., Hedborg, J., Letalick, D., Möller, S.: High spatial resolution hyperspectral camera based on a linear variable filter. Opt. Eng. 55, 114105 (2016)
Yoon, J., et al.: A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract. Nat. Commun. 10, 1–13 (2019)
Johnson, W.R., Wilson, D.W., Fink, W., Humayun, M., Bearman, G.: Snapshot hyperspectral imaging in ophthalmology. J Biomed Opt 12, 014036 (2007). https://doi.org/10.1117/1.2434950
Hagen, N.A., Kudenov, M.W.: Review of snapshot spectral imaging technologies. Opt. Eng. 52, 090901 (2013)
Williams, C., Gordon, G.S.D., Wilkinson, T.D., Bohndiek, S.E.: Grayscale-to-color: scalable fabrication of custom multispectral filter arrays. ACS Photonics 6, 3132–3141 (2019). https://doi.org/10.1021/acsphotonics.9b01196
Wang, Z., et al.: Single-shot on-chip spectral sensors based on photonic crystal slabs. Nat. Commun. 10, 1020 (2019). https://doi.org/10.1038/s41467-019-08994-5
Sahoo, S.K., Tang, D., Dang, C.: Single-shot multispectral imaging with a monochromatic camera. Optica 4, 1209–1213 (2017)
Park, J., Feng, X., Liang, R., Gao, L.: Snapshot multidimensional photography through active optical mapping. Nat. Commun. 11, 1–13 (2020)
Pawlowski, M.E., Dwight, J.G., Nguyen, T.-U., Tkaczyk, T.S.: High performance image mapping spectrometer (IMS) for snapshot hyperspectral imaging applications. Opt. Express 27, 1597–1612 (2019)
Hedde, P.N., Cinco, R., Malacrida, L., Kamaid, A., Gratton, E.: Phasor-based hyperspectral snapshot microscopy allows fast imaging of live, three-dimensional tissues for biomedical applications. Commun. Biol. 4, 1–11 (2021)
Khan, M.J., Khan, H.S., Yousaf, A., Khurshid, K., Abbas, A.: Modern trends in hyperspectral image analysis: a review. IEEE Access 6, 14118–14129 (2018)
Paoletti, M., Haut, J., Plaza, J., Plaza, A.: Deep learning classifiers for hyperspectral imaging: a review. ISPRS J. Photogramm. Remote Sens. 158, 279–317 (2019)
Trajanovski, S., Shan, C., Weijtmans, P.J., de Koning, S.G.B., Ruers, T.J.: Tongue tumor detection in hyperspectral images using deep learning semantic segmentation. IEEE Trans. Biomed. Eng. 68, 1330–1340 (2020)
Yun, B., et al.: SpecTr: spectral transformer for hyperspectral pathology image segmentation (2021). arXiv:2103.03604
Grigoroiu, A., Yoon, J., Bohndiek, S.E.: Deep learning applied to hyperspectral endoscopy for online spectral classification. Sci. Rep. 10, 1–10 (2020)
Manni, F., et al.: Hyperspectral imaging for glioblastoma surgery: Improving tumor identification using a deep spectral-spatial approach. Sensors 20, 6955 (2020)
Fabelo, H., et al.: Deep learning-based framework for in vivo identification of glioblastoma tumor using hyperspectral images of human brain. Sensors 19, 920 (2019)
Waterhouse, D.J., et al.: Spectral endoscopy enhances contrast for neoplasia in surveillance of Barrett’s esophagus. Cancer Res. 81, 3415–3425 (2021)
Ortac, G., Ozcan, G.: Comparative study of hyperspectral image classification by multidimensional convolutional neural network approaches to improve accuracy. Expert Syst. Appl. 182, 115280 (2021)
Chen, L., et al.: MRI tumor segmentation with densely connected 3D CNN. Medical Imaging 2018: Image Processing, vol. 105741F, Houston, Texas, US (2018)
Ker, J., et al.: Image thresholding improves 3-dimensional convolutional neural network diagnosis of different acute brain hemorrhages on computed tomography scans. Sensors 19, 2167 (2019)
Bashkatov, A.N., Genina, E.A., Tuchin, V.V.: Optical properties of skin, subcutaneous, and muscle tissues: a review. J. Innov. Opt. Health Sci. 4, 9–38 (2011)
Kroemer, G., Pouyssegur, J.: Tumor cell metabolism: cancer’s Achilles’ heel. Cancer Cell 13, 472–482 (2008)
Aboughaleb, I.H., Aref, M.H., El-Sharkawy, Y.H.: Hyperspectral imaging for diagnosis and detection of ex-vivo breast cancer. Photodiagn. Photodyn. Ther. 31, 101922 (2020)
Kho, E., et al.: Hyperspectral imaging for resection margin assessment during cancer surgery. Clin. Cancer Res. 25, 3572–3580 (2019)
Kho, E., et al.: Broadband hyperspectral imaging for breast tumor detection using spectral and spatial information. Biomed. Opt. Express 10, 4496–4515 (2019)
Zhang, Y., et al.: Explainable liver tumor delineation in surgical specimens using hyperspectral imaging and deep learning. Biomed. Opt. Express 12, 4510–4529 (2021)
Ortega, S., et al.: Hyperspectral imaging for the detection of glioblastoma tumor cells in H&E slides using convolutional neural networks. Sensors 20, 1911 (2020)
Lv, M., et al.: Membranous nephropathy classification using microscopic hyperspectral imaging and tensor patch-based discriminative linear regression. Biomed. Opt. Express 12, 2968–2978 (2021)
Liu, N., Guo, Y., Jiang, H., Yi, W.: Gastric cancer diagnosis using hyperspectral imaging with principal component analysis and spectral angle mapper. J. Biomed. Opt. 25, 066005 (2020)
Li, Y., et al.: Diagnosis of early gastric cancer based on fluorescence hyperspectral imaging technology combined with partial-least-square discriminant analysis and support vector machine. J. Biophotonics 12, e201800324 (2019)
Halicek, M., Little, J.V., Wang, X., Chen, A.Y., Fei, B.: Optical biopsy of head and neck cancer using hyperspectral imaging and convolutional neural networks. J. Biomed. Opt. 24, 036007 (2019)
Halicek, M., et al.: Hyperspectral imaging of head and neck squamous cell carcinoma for cancer margin detection in surgical specimens from 102 patients using deep learning. Cancers 11, 1367 (2019)
Halicek, M., Dormer, J.D., Little, J.V., Chen, A.Y., Fei, B.: Tumor detection of the thyroid and salivary glands using hyperspectral imaging and deep learning. Biomed. Opt. Express 11, 1383–1400 (2020)
Baltussen, E.J., et al.: Hyperspectral imaging for tissue classification, a way toward smart laparoscopic colorectal surgery. J. Biomed. Opt. 24, 016002 (2019)
Huang, Q., et al.: Blood cell classification based on hyperspectral imaging with modulated Gabor and CNN. IEEE J. Biomed. Health Inform. 24, 160–170 (2019)
Fabelo, H., et al.: 2019 XXXIV Conference on Design of Circuits and Integrated Systems (DCIS), pp. 1–6. IEEE
Leon, R., et al.: Non-invasive skin cancer diagnosis using hyperspectral imaging for in-situ clinical support. J. Clin. Med. 9, 1662 (2020)
Hosking, A.M., et al.: Hyperspectral imaging in automated digital dermoscopy screening for melanoma. Lasers Surg. Med. 51, 214–222 (2019)
Zherebtsov, E., et al.: Hyperspectral imaging of human skin aided by artificial neural networks. Biomed. Opt. Express 10, 3545–3559 (2019)
He, Q., Wang, R.: Hyperspectral imaging enabled by an unmodified smartphone for analyzing skin morphological features and monitoring hemodynamics. Biomed. Opt. Express 11, 895–910 (2020)
Kohler, L.H., et al.: Hyperspectral imaging (HSI) as a new diagnostic tool in free flap monitoring for soft tissue reconstruction: a proof of concept study. BMC Surg. 21, 1–9 (2021)
More, S.S., Beach, J.M., McClelland, C., Mokhtarzadeh, A., Vince, R.: In vivo assessment of retinal biomarkers by hyperspectral imaging: early detection of Alzheimer’s disease. ACS Chem. Neurosci. 10, 4492–4501 (2019)
Hadoux, X., et al.: Non-invasive in vivo hyperspectral imaging of the retina for potential biomarker use in Alzheimer’s disease. Nat. Commun. 10, 1–12 (2019)
Yoon, J., et al.: First experience in clinical application of hyperspectral endoscopy for evaluation of colonic polyps. J. Biophotonics (2021). https://doi.org/10.1002/jbio.202100078
Fabelo, H., et al.: In-vivo hyperspectral human brain image database for brain cancer detection. IEEE Access 7, 39098–39116 (2019)
Florimbi, G., et al.: Towards real-time computing of intraoperative hyperspectral imaging for brain cancer detection using multi-GPU platforms. IEEE Access 8, 8485–8501 (2020)
Köhler, H., et al.: Evaluation of hyperspectral imaging (HSI) for the measurement of ischemic conditioning effects of the gastric conduit during esophagectomy. Surg. Endosc. 33, 3775–3782 (2019)
Sucher, R., et al.: Hyperspectral Imaging (HSI) in anatomic left liver resection. Int. J. Surg. Case Rep. 62, 108–111 (2019)
Jansen-Winkeln, B., et al.: Determination of the transection margin during colorectal resection with hyperspectral imaging (HSI). Int. J. Colorectal Dis. 34, 731–739 (2019)
Sucher, R., et al.: Hyperspectral imaging (HSI) of human kidney allografts. Ann. Surg. (2020). https://doi.org/10.1097/SLA.0000000000004429
Larsen, E.L., et al.: Hyperspectral imaging of atherosclerotic plaques in vitro. J. Biomed. Opt. 16, 026011 (2011)
Hacker, L., et al.: A copolymer-in-oil tissue-mimicking material with tuneable acoustic and optical characteristics for photoacoustic imaging phantoms. IEEE Trans. Medical Imaging 40(12), 3595–3603 (2021). https://doi.org/10.1109/TMI.2021.3090857
Harper, D.J., et al.: Hyperspectral optical coherence tomography for in vivo visualization of melanin in the retinal pigment epithelium. J. Biophotonics 12, e201900153 (2019)
Diot, G., et al.: Multispectral optoacoustic tomography (MSOT) of human breast cancer. Clin. Cancer Res. 23, 6912–6922 (2017)
Li, H., et al.: 25th IEEE International Conference on Image Processing (ICIP), pp. 3323–3327. IEEE (2018)
Zhang, J., Sun, Y., Chen, J., Yang, D., Liang, R.: Deep-learning-based hyperspectral recovery from a single RGB image. Opt. Lett. 45, 5676–5679 (2020)
Shi, Z., Chen, C., Xiong, Z., Liu, D., Wu, F.: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 939–947
Kaya, B., Can, Y.B., Timofte, R.: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3546–3555. IEEE
Zhang, W., et al.: Deeply learned broadband encoding stochastic hyperspectral imaging. Light Sci. Appl. 10, 1–7 (2021)
Liu, T., et al.: Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 1–13 (2019)
Fang, L., et al.: Deep learning-based point-scanning super-resolution imaging. Nat. Methods 18, 406–416 (2021)
Acknowledgements
This research was supported by Ajou University and the National Research Foundation (NRF) of Korea (No. 2021R1C1C1011047, 2021R1A4A5032470)
Funding
Funding was provided by National Research Foundation (NRF) of Korea (no. 2021R1C1C101104711, 2021R1A4A5032470).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author declares no competing financial interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Yoon, J. Hyperspectral Imaging for Clinical Applications. BioChip J 16, 1–12 (2022). https://doi.org/10.1007/s13206-021-00041-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13206-021-00041-0