Abstract
We propose ghost imaging (GI) with deep learning to improve detection speed. GI, which uses an illumination light with random patterns and a single-pixel detector, is correlation-based and thus suitable for detecting weak light. However, its detection time is too long for practical inspection. To overcome this problem, we applied a convolutional neural network that was constructed based on a classification of the causes of ghost image degradation. A feasibility experiment showed that when using a digital mirror device projector and a photodiode, the proposed method improved the quality of ghost images.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
The demand for fine processing has led to interest in the optical detection of micro/nano defects in a large area. Large fine optics for high-power lasers produce surfaces that are free from sub-micron defects [1,2,3]. Zero-defect manufacturing for realizing specular surfaces has been extensively studied [4]. Recent studies in the fields of additive manufacturing and semiconductor processing have shown that sub-micron defect or particle detection is critical for maintaining fine processing conditions [5]. However, it is difficult to detect sub-micron defects because the intensity of scattered light substantially decreases with decreasing defect size [6].
For the optical detection of weakly scattered light from small defects, studies have applied physical and machine vision approaches. For the physical approach, many studies have considered the dark-field condition. The structured light of a spatial or spectral region offers advantages for detecting oriented sub-micron defects [7, 8]. A line-scanning laser has been applied to improve detection speed [9, 10]. Both the intensity and phase information of scattered light can be utilized with good robustness [11,12,13]. Localized physical phenomena of evanescent waves or Raman scattering allow the detection of sub-micron defects [14, 15]. However, the above methods have poor sensitivity and efficiency. For the machine vision approach, methods that use a microscope have been developed to detect the position of small surface defects and classify them [16,17,18,19]. The numerical aperture of the optical system determines the defect detection limit.
To overcome this problem, we adopt ghost imaging (GI), which is a single-pixel imaging method based on the correlation between the intensity distribution of an illumination light and the light intensities detected by a bucket detector [20,21,22]. The high-sensitivity detector and correlation-based detection of GI make its sensitivity higher than that of general methods. GI is thus attractive for detecting weak light intensity [23]. However, obtaining ghost images is inefficient because many measurements are required to acquire clear images for correlation calculations. Therefore, GI has been limited to static objects. To overcome this problem, we apply deep learning (DL) to reduce the number of measurements required for correlation. DL is applied to predict true values using less information [24, 25]. Specifically, for the imaging field, DL can reconstruct high-resolution images using lower-resolution images. The image quality of GI with fewer measurements is low even though a GI image contains sample information. There are a few studies about DL combined with GI [26, 27]. However, these proposed papers were based on an analytical method, called compressive sensing, rather than using correlation. Therefore, it is difficult to apply for detecting weak light information. Accordingly, to recover an image for weak light intensity, we apply DL to GI (DLGI) with fewer measurements. In this paper, we develop a GI system and combine it with a convolutional neural network (CNN) based on noise analysis of GI towards fast defect-position-mapping of weakly scattered light sources.
2 Concept of Sub-Micron Defect Mapping Using Ghost Imaging with Deep Learning
The concept of sub-micron defect mapping using GI with DL is shown in Fig. 1. Computer-generated random patterns are illuminated onto a sample with small defects. Then, light scattered by the defects is detected by a single-pixel detector. GI is based on the second-order correlation of the fluctuation between illumination patterns and detected signals [20]. The distribution of correlation function \( G\left( {x,y} \right) \) is expressed as
where \( \Delta I_{n}\left( {x,y} \right) \) is the fluctuation of the spatial distribution of illumination patterns and \( \Delta B_{n} \) is the intensity fluctuation of time-dependent signals detected by the single-pixel detector with n measurements.
Figure 2 shows examples of GI images. A commercial projector was used as an illumination light (wavelength: 532 nm) and a photodiode was used as a single-pixel detector to image a ϕ10-μm pinhole. As shown in Fig. 2, although the difference in illumination intensity was large (12 vs. 0.3 mW), the reconstructed images had almost the same quality. The correlation-based imaging automatically reduced the noise signals in the imaging process. However, with GI, a sufficient number of measurements is required to obtain clear images. Therefore, the detection time is very long. To overcome this problem, a CNN was employed to improve ghost image quality with fewer measurements.
3 Design and Numerical Analysis of Convolutional Neural Network for Ghost Imaging
A CNN generally consists of convolutional layers and dense layers [25, 26]. Before designing the CNN for GI, we analyzed the degradation of reconstructed GI images. Figure 3 shows the causes of GI degradation, which can be divided into local and global causes. Local causes include the presence of an alternative hypothesis and a blurred pattern. Because GI is based on a null hypothesis analogy, GI with fewer measurements does not satisfy the condition of the null hypothesis. Therefore, some pixels have a high correlation value regardless of defect existence. The alternative hypothesis is independent of each pixel. Furthermore, the point spread function (PSF) of the optical setup depends on the projection setup. Therefore, the projected pattern is blurred on the sample surface. The blurred error does not depend on each pixel. A convolutional layer can reduce the effect of these local causes. Global causes include global noise around the GI system, which affects all pixels in the image. Environmental noise, such as atmospheric turbulence and stray light, and electrical noise such as shot noise, thermal noise, and dark current, are error sources that can cause GI image degradation when few measurements are used. The signal of scattered light affects the detected intensity. A dense layer can reduce the effect of these global causes.
Figure 4 shows the architecture of the CNN for GI. The CNN consists of an input layer, some convolutional and dense layers, and a softmax function for deriving probability distributions. The input layer uses a reconstructed image and its illumination patterns to avoid the alternative hypothesis. First, we reduced the effects of the local and global causes using convolutional and dense layers, respectively. Then, we obtained the existence probability of light at all pixels as an image. All layers used a rectified linear unit (ReLU) function as the activation function. A dataset of 4000 images, each of which was 16 × 16 pixels and generated a numerical simulation, was used for training.
We evaluated the performance of the proposed CNN using numerical calculations. Figures 5 and 6 show the simulation results for a CNN with convolutional layers and an input layer without and with illumination patterns, respectively. The sample size was 16 × 16 pixels with one bright pixel in the center. The prediction accuracy initially increased with the number of epochs and then remained at a constant value. The accuracy of the CNN with convolutional layers was higher than that without convolutional layers. Using illumination patterns in the input layer, with or without the convolution layers, resulted in a larger difference in accuracy compared to the case without illumination patterns. This large difference is attributed to the image degradation contributing less to the null hypothesis condition.
To confirm the validity of the built CNN, we have compared the predicted position of the conventional ghost imaging with that of the proposed method. Figure 7 shows simulated results of residual from set pixel between GI and DLGI. The numbers of measurements of GI and DLGI are from 0 to 16. The sample size was 16 × 16 pixels with one bright pixel in the center. It took less than a second to derive the expected images from the original GI images. Each residual was an average of the difference between the set position and the predicted position over 1000 repeated measurements. As the number of measurements was increased, the residuals decreased. In particular, the residuals are smaller than one pixel in four or more measurements. In our proposed method, it is important to improve not only the detection speed but also the residuals. In order to meet the two requirements, we have determined that the measurement numbers were four. In the case of using the conventional GI, the residuals also decreased as the number of measurements increased. However, the residual for the four measurements was 5.61, which was higher than the value of 0.61 measured using DLGI. From these results, we have determined the number of measurements and confirmed the effectiveness of the constructed CNN.
4 Experimental Results
The efficiency of GI with the CNN was experimentally evaluated. We used a digital mirror device (DMD) (Texas Instruments: DLP 3000) as a projector as shown in Fig. 8. Its wavelength (532 nm) was selected using an interferometric filter. Computer-generated random patterns (16 × 16 pixels) were illuminated onto a sample. The illumination light was scattered by the sample and detected by a photodiode (Hamamatsu: S6967). The detected signals were converted by an analogue-to-digital device and recorded on a personal computer. A GI image was obtained by calculating the correlation efficiency between the illumination patterns and detected signals. Figure 9 shows the experimental results of GI images obtained without the CNN. To confirm the proposed principle, we used a simple sample made of a black piece of paper with a square hole in the center. As shown in Fig. 9, the quality of the conventional GI images increased with increasing number of measurements. With 256 measurements, although the detection time was long, the position of the hole was clear, whereas with four measurements, the reconstructed image was noisy.
Figure 10 shows the experimental results of reconstructed DLGI images. The trained CNN improved the quality of the reconstructed images. However, as shown in Fig. 10a, the quality of some images was not improved. A histogram of the predicted hole position in images of 50 measurements is shown in Fig. 10b. The standard deviation of the predicted values was 7.11 pixels. These distributions are most likely due to reducing global noise imperfectly.
To overcome this problem, we analyzed the noise dependency of the predicted positions. Fewer measurements are not sufficient for noise reduction. In our experimental conditions, other than the detector, for example, fluctuations in air or light source intensity and so on, can also generate noise. Most of the fluctuations have a Gaussian distribution. Therefore, Gaussian distributions were used to improve CNNs. Figure 11 shows simulated results of residual from set pixel between GI and DLGI with 10% Gaussian noise signal derived from the experimental environment. The numbers of measurements of GI and DLGI are from 0 to 32 and 4, respectively. The sample size was 16 × 16 pixels with one bright pixel in the center. Before the calculation of Eq. (1), a 10% Gaussian noise was added to the detected signal B in Eq. (1). The CNN was also retrained with the new dataset for 200 epochs with a signal containing 10% Gaussian noise. Each residual was an average of the difference between the set position and the predicted position over 1000 repeated measurements. In the case of using the conventional GI, the residuals decreased as the number of measurements increased. However, the residual for the four measurements was 5.61, which was higher than the value of 3.66 measured using DLGI. From these results, we confirmed the effectiveness of the reconstructed CNN by using 10% Gaussian noise signals.
The residuals to the noise loading rate of the detection signal used for training were experimentally investigated by varying the noise loading rate of the detection signal used for training. The sample was the same as the one shown in Fig. 9. The amount of added noise was varied from 0 to 10%. We retrained the CNN in each noise level. Each residual was an average of the difference between the set position and the predicted position over 100 repeated measurements. As shown in Fig. 12, with increasing added noise, the residuals were decreased significantly.
Figure 13 shows the experimental results of reconstructed DLGI images with 10% Gaussian noise. As shown in Fig. 13a, the quality of DLGI images of four measurements was improved. A histogram of the predicted hole position in images of 50 measurements is shown in Fig. 13b. The standard deviation of the predicted values was 3.12 pixels. Although the cause of this Gaussian effect is not well understood at this time, the proposed CNN is demonstrated to increase the quality of GI images.
5 Conclusions
We proposed a method that uses GI combined with DL for fast imaging. We introduced a CNN to increase the quality and speed of GI detection. The CNN had convolutional and dense layers and the obtained image and illuminated random patterns were applied to its input layer. In a feasibility experiment that used a DMD projector and a photodiode, the proposed method was found to improve the quality of GI images. After experimental investigation, a noise-loaded dataset was found to be effective in reducing global noise. The results show that GI with a noise-trained CNN is suitable for detecting a position of scattered light. In future work, we will attempt to improve prediction accuracy by reconsidering the CNN architecture, and implement the method to detect sub-micron defects or particles on a Si wafer with a large inspection area.
Change history
13 December 2022
A Correction to this paper has been published: https://doi.org/10.1007/s41871-022-00160-8
References
Campbell J, Hayden J, Marker A (2011) High-power solid-state lasers: a laser glass perspective. Int J Appl Glass 2:3–29
Wang S, Liu D, Yang Y, Chen X, Cao P, Li L, Yan L, Cheng Z, Shen Y (2014) Distortion correction in surface defects evaluating system of large fine optics. Opt Commun 312:110–116
Brinksmeier E, Mutlugünes Y, Klocke F, Aurich JC, Shore P, Ohmori H (2010) Ultra-precision grinding. CIRP Ann Manuf Technol 59:652–671
Azorin-Lopez J, Fuster-Guillo A, Saval-Calvo M, Mora-Mora H, Garcia-Chamizo J (2017) A novel active imaging model to design visual systems: a case of inspection system for specular surfaces. Sensors 17:1466
Caggiano A, Zhang J, Alfieri V, Caiazzo F, Gao R, Teti R (2019) Machine learning-based image processing for on-line defect recognition in additive manufacturing. CIRP Ann Manuf Technol 68:451–454
Lilienfeld P (1986) Optical detection of particle contamination on surfaces: a review. Aerosol Sci Technol 5:145–165
Miyoshi T, Takahashi S, Takaya Y, Shimada S (2001) High sensitivity optical detection of oriented microdefects on silicon wafer surfaces using annular illumination. CIRP Ann Manuf Technol 50:389–392
Yazaki A, Kim C, Chan J, Mahjoubfar A, Goda K, Watanabe M, Jalali B (2014) Ultrafast dark-field surface inspection with hybrid-dispersion laser scanning. Appl Phys Lett 104:251106
Dong J (2017) Line-scanning laser scattering system for fast defect inspection of a large aperture surface. Appl Opt 56:7089–7098
Kim MS, Choi HS, Lee SH, Kim C (2015) A high-speed particle-detection in a large area using line-laser light scattering. Curr Appl Phys 15:930–937
Choi H, Kam JM, Berkson JD, Graves LR, Kim DW (2019) Modulated dark-field phasing detection for automatic optical inspection. Opt Eng 58:092603
Zhu J, Zhou R, Zhang L, Ge B, Luo C, Goddard LL (2019) Regularized pseudo-phase imaging for inspecting and sensing nanoscale features. Opt Express 27:6719–6733
Zhou R, Edwards C, Arbabi A, Popescu G, Goddard LL (2013) Detecting 20-nm-wide defects in large area nanopatterns using optical interferometric microscopy. Nano Lett 13:3716–3721
Takahashi S, Kudo R, Usuki S, Takamasu K (2011) Super resolution optical measurements of nanodefects on Si wafer surface using infrared standing evanescent wave. CIRP Ann Manuf Technol 60:523–526
Takaya Y, Michihata M, Hayashi T, Murai R, Kano K (2013) Surface analysis of the chemical polishing process using a fullerenol slurry by Raman spectroscopy under surface plasmon excitation. CIRP Ann Manuf Technol 62:571–574
Liu D, Yang Y, Wang L, Zhuo Y, Lu C, Yang L, Li R (2007) Microscopic scattering imaging measurement and digital evaluation system of defects for fine optical surface. Opt Commun 278:240–246
Scholz-Reiter B, Weimer D, Thamer H (2012) Automated surface inspection of cold-formed micro-parts. CIRP Ann Manuf Technol 61:531–534
Galantucci LM, Pesce M, Lavecchia F (2015) A stereo photogrammetry scanning methodology, for precise and accurate 3D digitization of small parts with sub-millimeter sized features. CIRP Ann Manuf Technol 64:507–510
Ye R, Chang M, Pan CS, Chiang CA, Gabayno J (2018) High-resolution optical inspection system for fast detection and classification of surface defects. Int J Optomech 12:1–10
Pittman TB, Shih YH, Strekalov DV, Sergienko AV (1995) Optical imaging by means of two-photon quantum entanglement. Phys Rev A 52:R3429–R3452
Gatti A, Brambilla E, Bache M, Lugiato LA (2004) Ghost imaging with thermal light: comparing entanglement and classical correlation. Phys Rev Lett 93:093602-1-4
Shapiro JH (2008) Computational ghost imaging. Phys Rev A 78:061802
Shibuya K, Nakae K, Mizutani Y, Iwata T (2015) Comparison of reconstructed images between ghost imaging and Hadamard transform imaging. Opt Rev 22:897–902
Fukushima K, Miyake S (1982) Neocognitron: a self-organizing neural network model for a mechanism of visual pattern recognition. Competition and cooperation in neural nets. Springer, Berlin, pp 267–285
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324
Wang HY, Dong G, Zhu G, Chen S, Zhang H, Xu Z (2018) Ghost imaging based on deep learning. Scientific Rep 8:1–7
Li F, Zhao M, Tian Z, Willomitzer F, Cossairt O (2020) Compressive ghost imaging through scattering media with deep learning. Opt Exp 28:17395–17408
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Yasuhiro Takaya is an editorial board member for “Nanomanfucturing and Metrology” and was not involved in the editorial review, or the decision to publish this article. All authors declare that there are no competing interests.
Additional information
The original online version of this article was revised: The declaration text was missing.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Mizutani, Y., Kataoka, S., Uenohara, T. et al. Ghost Imaging with Deep Learning for Position Mapping of Weakly Scattered Light Source. Nanomanuf Metrol 4, 37–45 (2021). https://doi.org/10.1007/s41871-020-00085-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41871-020-00085-0