Integrative Multi-Spectral Sensor Device for Far-Infrared and Visible Light Fusion

Infrared and visible light image fusion technology is a hot spot in the research of multi-sensor fusion technology in recent years. Existing infrared and visible light fusion technologies need to register before fusion because of using two cameras. However, the application effect of the registration technology has yet to be improved. Hence, a novel integrative multi-spectral sensor device is proposed for infrared and visible light fusion, and by using the beam splitter prism, the coaxial light incident from the same lens is projected to the infrared charge coupled device (CCD) and visible light CCD, respectively. In this paper, the imaging mechanism of the proposed sensor device is studied with the process of the signals acquisition and fusion. The simulation experiment, which involves the entire process of the optic system, signal acquisition, and signal fusion, is constructed based on imaging effect model. Additionally, the quality evaluation index is adopted to analyze the simulation result. The experimental results demonstrate that the proposed sensor device is effective and feasible.


Introduction
With the rapid development of the sensor technology, the infrared sensor and visible light sensor are widely used in many fields, such as fire detection [1,2], and intelligent transportation [3][4][5]. The visible light sensor [6,7] can provide an image signal with the characteristics of abundant digital object information and clear texture feature.
Furthermore, it is sensitive to capture the brightness information in the scene, and it is more in accordance with human visual characteristics. Nevertheless, its application is limited because that the visible light sensor only responses to a very narrow visible light band. In the faint light, the visible light sensor cannot obtain a clear image of the scene. The infrared sensor [8,9] can capture infrared radiation of the target, and any substance can produce infrared radiation when temperature is above absolute zero. Therefore, the infrared sensor can be well utilized in the low-light level (even in the dark) and has ability to preserve the contour feature of the object scene. Unfortunately, the infrared sensors have some drawbacks, such as scanty digital information and the overall dark of the captured infrared image. In the machine vision application, the requirement of the information richness of the image becomes higher and higher, thus using the single infrared sensor or visible light sensor is not able to meet the demand of the vision application due to the above characteristics of infrared and visible light sensors. Therefore, based on the feature of the infrared image and visible light image, the fusion technology [10] of the infrared and image visible light image is studied in order to obtain detailed target information and stronger scene understanding.
The image fusion technology can synthesize the image that is obtained by different kinds of vision sensors in the same scene or the image of the same scene and same time that is obtained by the same kind of vision sensors [11,12]. In recent years, the image fusion technology is used in many fields because they can take full advantage of the superiority of the different sensors to make up for being lack of a single sensor [13,14]. Nevertheless, currently existing infrared and visible light image fusion system exhibits a registration problem of the image fusion [15]. In order to resolve this problem, on the one hand, the binocular structure is selected as the auxiliary construction to complete the registration of the image fusion, i.e., the two cameras are utilized in the image fusion system, one is used in the infrared spectrum, and the other is used in the visible spectrum [16]. Unfortunately, the actual image will have discrepancy in the engineering application due to these phenomena, which involves the difference of the camera's view and focal length, the move of the camera, and so on. The real-time performance and the reliability of those inspection systems based on vision sensor are undoubtedly decreased [17,18]. On the other hand, based on the work above, the hot mirror is selected as a dichroic beam splitter, transmitting the visual part of the incoming radiation to the visible light camera while reflecting the infrared part via the infrared reflecting mirror into the infrared camera. Obviously, those fusion systems are costly and inefficient from the user's perspective for two cameras [19][20][21]. Furthermore, the complexity of the fusion system is increased twice, and the reliability of fusion system is reduced because of two sets of cameras, lens, and so on.
Solving the problem of the existing image fusion system, a novel integrative multi-spectral sensor device is proposed for infrared and visible fusion. In the proposed sensor device, through a single lens, the infrared and visible light is transmitted into two sensitive elements, one is the visible light imaging chip, and the other is the infrared imaging chip. Namely, infrared and visible light can be captured simultaneously through the same optical path. More importantly, the rectification is not needed before the infrared signal and visible image fuse, so the fusion image can be obtained immediately. Due to the selection of the lens, we cannot verify it via experiment. But the lens has been customized from Wuhan Guide Infrared Co., Ltd in China. The remainder of the paper is organized as follows. Section 2 presents the operational principle of beam splitter and the imaging theoretical of the sensor device. Then, the theoretical modeling of the integrative multi-spectral sensor device is described in Section 3. Both the simulation experiment and analysis are provided to verify the effectiveness and feasibility of the sensor device in Section 4. Finally, the overall conclusions are given in Section 5.

Imaging theory of the sensor device
The imaging theory is the key to study the integrative multi-spectral sensor device. In our proposed sensor device, there are two important parts: signal acquisition unit and signal fusion unit. Figure 1 illustrates the imaging process composition of the sensor device. The signals acquisition unit consists of a lens, a beam splitter prism, and two charge coupled devices (CCDs). The lens can transmit both visible light (380 nm-780 nm) and long-wave infrared (8 μm-14μm). Moreover, the beam splitter prism is made up of three identical plane mirrors of isosceles triangular shape. After the signal acquisition, the signal is fed to the signal fusion unit, which fuses the infrared signal and visible light signal via the pixel-level fusion method.

Infrared and visible signal acquisition
In the proposed sensor device, the beam splitter prism is selected as the light splitting element, which is made up of transparent materials, such as glass and crystal. Due to the outstanding advantages of splitting performance and cost efficiency, the beam splitter prism is usually used to divide the light. Hence, in this paper, the beam splitter prism is utilized to realize the synchronous sampling of the optic signal. Furthermore, the structure and the principle of the beam splitter prism are schematically shown in Fig. 2.
From Fig. 2(a), it can be observed that the beam splitter prism is composed of five planes: three rectangle shapes and two identical irregular triangles. Moreover, it can be seen that the beam-splitting film is used in two rectangle shapes, namely, GHCD and ABFE, which work in two wide spectral ranges (high-transmissivity at 400 nm -700 nm and high-reflectance at 8 μm -12 um). With the beam-splitting film, the two rectangle shapes (ABHG and ABFE) can transmit the visible bands and reflect the infrared bands.
The principle of the beam splitter prism is shown in Fig. 2(b). As it can be seen, the infrared and the visible radiations of the object are focused on the prism through a single lens. Then, the ABFE plane can transmit the visible and reflect the infrared because of the beam-splitting film. Afterward, the infrared is reflected once more by the GHCD plane. Finally, the infrared is received by the infrared sensing element, at the same time, the visible light is received by the visible sensing element. That is to say, the two CCDs are placed in one device through a single lens, and the visible light and infrared are transmitted into two sensitive elements, one is the visible light imaging chip, and the other is the infrared imaging chips. After reflecting or transmitting into the sensing element by the beam splitter prism, the infrared and visible lights are converted to corresponding electric signals respectively on the basis of photoelectric effect. Then, the electric signal, which is also called analog signal, is converted to the analog signal by making use of the analog-digital converter after a series of processing units, such as filtering and amplifier. That is to say, the infrared digital signal and the visible digital signal are captured respectively. In our proposed sensor device, the lens is an important device. Due to the selection of the lens, we cannot verify it via experiment. But the lens has been customized from Wuhan Guide Infrared Co., Ltd in China. And we simulate the lens in the following section. Figure 3 demonstrates the overall block diagram of the signal fusion process for the proposed sensor device. Before the signal fusion, the median filtering method is adopted to de-noise the infrared radiation of the heat object in environment and the noise data, which is caused by the lens and the beam splitter prism. Then, the infrared signal and the visible light signal are sent to the signal fusion chip.

Infrared and visible signals fusion
As shown in the signal fusion block diagram, the fusion process includes the flowing four steps: firstly, the infrared signal and the visible light signal are separately decomposed into multiscale and multidirection with nonsubsampled contourlet transform (NSCT), then, the coefficient of the infrared and the visible light signals are obtained, respectively. Secondly, the low-frequency sub-image signal is decomposed to the object region signal and the background region signal by the threshold segmentation based maximum entropy, then, the low-frequency sub-image signal of the fused image signal can be obtained by a certain fusion rules. Thirdly, the weighted average method is used to the high-frequency component fusion. Fourthly, with the inverse NSCT, the finally fused signal can be obtained.

Theoretical modeling of the sensor device
Based on the above imaging theory of the proposed sensor device, the physical effect, which is generated in the imaging process of every component unit of the sensor device, is analyzed.
The model of these effects is proposed by using the pixel processing method and the modulation transfer function (MTF) method [22,23]. In order to know the MTF effect of the optical system directly, these effects are simulated respectively by using the MATLAB. The effects are introduced in detail in the fowling sections, which involve the signal acquisition unit effect and signal fusion unit effect [24,25].

Signal acquisition unit effect
In this paper, by analyzing the signal acquisition of the sensor device, it can be seen that the image quality of the sensor device is directly influenced by the signal acquisition unit, which is the core part of the sensor device. To achieve the imaging process of the sensor device, the impact of the imaging process must be simulated. The effects of the signal acquisition unit, including the optic system effect, spatial filtering effect, temporal filtering effect, CCD transfer effect, low-pass filtering effect, and high-pass filtering effect, are taken into account here according to the above sensor device theory.

Optic system effect
The optic system is utilized to focus the light on the detector as much as possible by using the filter to carry out the spatial filter and temporal filter. However, because of the limit of the optical material or structure, the optical system will affect the image quality of each channel. The influence is mainly reflected in the effects of energy attenuation, diffraction effect, and aberration. The MTF method is utilized to simulate the effect of the spatial frequency domain, which involves the diffraction and aberration, and the pixel process method is utilized to simulate the effect of the space domain, which involves the energy attenuation A. Energy attenuation The light will be reflected and absorbed to some extent through the optic system, so it is not transmitted completely by the optical system. Additionally, the average transmittance coefficient of the optic system in every band is proposed to describe the model of the energy attenuation, that is to say every pixel value of the input image is multiplied by the average transmittance coefficient. In this paper, the energy attenuation is mainly derived from two aspects.
(1) The attenuation of infrared and visible light through lens. In the optic system, a lens is placed in front of the beam splitter prism. Moreover, there are some lenses utilized to focus the infrared and visible light on the sensing element and eliminate the aberration, which is placed below the beam splitter prism, and its number is equal to 2 in the proposed optic system. This optic attenuation rate of the optic element with the refection reducing coating is less than 0.1, and the maximum value is chosen. Therefore, the transmittance of the infrared or visible light through the lens is defined as The attenuation of infrared and visible lights through the beam splitter prism.
The beam splitter prism is utilized to divide the optical band to infrared band and visible band. The infrared light is reflected to the infrared path channel, and the visible light is transmitted to the visible light path channel. For the infrared band, the reflectance of the infrared band is equal to 0.95, and the transmissivity of the visible light is equal to 0.9. Thus, the transmittance of the infrared band channel is equal to

B. Diffraction effect
According to the diffraction theory, the high quality optic system should be close to the diffraction-limited optic system, of which the MTF depends on the effective lens aperture of the optic system and the average wavelength, as calculated by (1). The MTF of the diffraction-limited optic system is defined as the following equations.
where D is the diameter of the aperture,  is the average wavelength, c f is the diffraction-limited angular resolution for a circular aperture, and its value is connected with the value of D and , , and s f is the spatial angle frequency defined as (2).

C. Aberration effect
As it is well known, there are all sorts of aberrations in the actual optical system, and the image of a point is the comprehensive results of various aberrations. Additionally, the wave aberration is used to represent the aberration. In this paper, the approximate empirical correlation is adopted to describe the aberration effect of the optic system, as shown in (3):

Spatial filtering effect
Spatial filtering effect, which is generated by the limitations of the pixel size and the pixel interval, is a reduction phenomenon of an image's high frequency and the image digital is weakened by it. The MTF of the effect is shown in (4): where a b  is the pixel size.

Temporal filtering effect
The response of the detector to the incident infrared and visible lights is not instantaneous, and there is a process to reach a steady state. Therefore, the temporal filtering effect of the signal acquisition unit can combine into an equivalent low-pass, and its physics models are illustrated as where 0 f is the frequency of the time response at 3 dB, and t f is the temporal frequency which is connected with the spatial frequency s f . The relevant transformation formula is described as t s f sf f  (6) where s is the scan rate.

CCD transfer effect
Because of the impact of the CCD transfer efficiency, the signal will be attenuated when the signal is transmitted by the CCD. Thereby, the simulation of the CCD transfer is necessary, and its MTF is given by (7)  where N is the total charge, which is transmitted from sensing element to the amplifier,  is the transfer efficiency of the CCD, and ts f is the sampling frequency of the CCD signal.

Low-pass filtering effect and high-pass filtering effect
In the proposed sensor device, the amplifier is utilized to the signal acquisition unit because the output signal of the sensing element of the sensor is weak. There are two amplifiers in the sensor, pre-amplifier and post amplifier, which correspond to every sensing element in the infrared channel and the visible channel, respectively, and the uses of pre-amplifier and post amplifier are equivalent to the high-pass filtering and low-pass filtering, respectively. Therefore, the incoming physical effects mainly consist of the high-pass filtering and low-pass filtering. Furthermore, since the signal, which is transmitted by the signal acquisition unit of the proposed sensor, is the time-domain signal, the MTF is obtained in the temporal frequency domain, as show in (8)  f and l f are the frequencies of high-pass filtering and low-pass filtering at 3 dB, respectively.

Figures 4 (a)-4(g)
show the simulation images of every effect in signal acquisition unit, namely the MTF distribution maps, and taking infrared channel effect as an example, its model parameters, which are set according to the CCD sensor in our laboratory and the requirement of the belt tearing for the sensor device are list in Table 1.

Signal fusion unit effect
For signal fusion unit, the fusion principle in Section 2 is utilized as the model of signal fusion unit to simulate the sensor device. The fusion image is obtained by using the fusion principle to fuse the infrared signal and visible signal, which is gained by adding the above effect. In order to obtain better fusion effect, the image enhancement is necessary to the image before fusion. The concrete steps of the image fusion are introduced in Section 2.2.
By multiplying the above effects of the signal acquisition and fusion in the imaging process of the sensor device, the imaging process model of the proposed sensor device is developed. In the following section, there is a simulation of the proposed sensor device by applying the model to the input picture.

Simulation experiment and analysis
To verify the proposed sensor device, we conduct simulation experiments based on the model of the sensor device imaging effect with the entire process of the signal acquisition and fusion. The simulation experiments are conducted as the following steps. Firstly, we obtain the longitudinal tear images as the input images, which are collected in the same scene by high definition visible light and infrared sensor device in our laboratory. Secondly, the MTF effect of the proposed sensor device is added to the input images, then, the images with MTF effect is fused. Finally, the simulation results are analyzed by comparing the values of quality evaluation index. Figure 5 illustrates the simulation flow. The total MTF of the sensor device is obtained by multiplying the MTF effect of the optical system and the signal acquisition unit in the imaging process of the sensor device. Furthermore, the input images are fast Fourier transform (FFT) to get their spectrogram, then the spectrogram of fusion image is obtained by fusing the two spectrograms with MTF effect, which is obtained by multiplying the spectrogram and the total MTF together, respectively. Finally, the ultimate fused image can be obtained through the inverse fast Fourier transform (IFFT). Based on the above MTF models, the whole imaging process of the sensor device is realized in the imaging simulation platform, and the simulation program is written by using MATLAB. The simulation platform interface is designed through graphical user interface (GUI) in MATLAB, as shown in Fig. 6. The panel is labeled with input parameters, which involves effects of optic system unit and signal acquisition unit, and some function buttons such as imread, infrared image with the MTF effect, visible image with the MTF effect, and fusion image. In the panel labeled with optic system unit effect, parameters of the MTF module of the optic system unit can be set, such as average wavelength, diameter of the aperture, mean square of the wave aberration, and focal length. At the same time, the parameters of the MTF module of the signal acquisition unit can be set, such as pixel value and scan rate. Moreover, in the simulation platform interface, buttons are provided to manage operations, such as reading the image and displaying the infrared or visible image with MTF effect or the fusion image.

Simulation results and analysis
Because of the hostile coal mine environment, longitudinal tear inspection for conveyor belts is a very serious problem. In this paper, the longitudinal tear image of the conveyor belt is selected as the case to testify the effectiveness and feasibility of the proposed sensor device. As shown in Fig. 7, the simulation experiment is performed in Figs. 7(a) and 7(b), which is collected in the same scene by high definition visible light and far-infrared (7 μm-14 μm) digital camera in our laboratory and used as the input of the simulation. Figs. 7(c) and 7(d) are obtained by the infrared channel and visible channel respectively with the effect model of the optical system and the signal acquisition unit. Fig. 7(e) is the simulation result of the total sensor device model, namely the ultimate fused image.
In the visible light image, the background information is more abundant than that in the infrared image, and the infrared thermal sources of background can hardly be detected. On the contrary, the infrared image has clear contour information of belt tearing, but the information of the belt tearing is limited in the visible light image. On paralleling the visual effects observed in Fig. 7, it could be noted that compared with the visible light image shown in Fig. 7(b), the ultimate fused image of the proposed sensor device can preserve detailed information of the scene. Furthermore, Fig. 7(e) appears more natural and its visible effect is much better than that shown in Fig. 7(c).
In addition to the visual inspection, the characteristics of every image should be analyzed quantitatively. Therefore, four objective evaluation indexes, i.e. information entropy (IE), average gradient (AG), root mean square (RMS), and spatial frequency (SP), are adapted to evaluate the simulation results objectively. The analysis results are illustrated in Table 2 (IR: infrared image, VI: visible image, and FI: fusion image). From this evaluation values, we can see that the information entropy value of the fused image is higher than that of the visible light image and close to the value of infrared image. Meanwhile, the root mean square value of the fused image is higher than that of the infrared image and lower than the value of visible light image. This shows that, the detailed object information and the higher luminance information can be integrated into one image via the image fusion. Additionally, the average gradient value and the spatial frequency value of the fused image is the maximum in the image of Fig. 7, as shown in Table  2, which means that the fused image, which is obtained by the sensor device simulation, contains the largest amount of information, and its sharpness is better than those of the infrared and visible light images.  As shown in Fig. 7 and Table 2, the findings of the statistical assessments conform with the visual inspection. Visual and quantitative assessments show that the fused image obtained by the proposed sensor device can retain the strengths of the infrared and visible light image. Namely, the proposed sensor device is feasible and effective.
Though the proposed sensor device has not been made yet, the simulation experiment is constructed to verify the effectiveness and feasibility of the proposed sensor device. The experimental results demonstrate that the proposed integrative multi-spectral sensor device is effective and feasible. Next, we will customize the lens from Wuhan Guide Infrared Co., Ltd in China to develop the sensor device and apply it to the visual inspection.

Conclusions and future work
In this paper, the integrative multi-spectral sensor device is proposed, and the imaging process of the sensor device is successfully simulated, which utilizes a lens, a beam splitter prism, and two CCDs. The triple prism is chosen as the light splitting element, and the principle of the beam splitter prism is analyzed. Moreover, the imaging mechanism of the sensor device is studied with the process of the signals acquisition and fusion. Additionally, in order to verify the feasibility and effectiveness of the proposed sensor device, the simulation experiment, which involves the entire process of the optic system, signal acquisition, and signal fusion, is constructed based on the model of the imaging effect. Finally, the quality evaluation index is adopted to analyze the simulation result. The experimental results demonstrate that the proposed integrative multispectral sensor device is effective and feasible, so it can meet the requirements of the vision application and also can resolve the problems of the existing infrared and visible light fusion system.
Compared with the traditional fusion system, the proposed sensor device innovation is the use of a lens, a beam splitter prism, and two CCDs instead of two cameras, which makes the proposed sensor device suitable for the vision inspection in the hostile environment. The infrared and visible lights can be captured simultaneously through the same optical path, and the fusion image can be obtained immediately by the proposed integrative multispectral sensor device, namely that the captured images do not need registration before image fusion. The simulation experiment results show that the proposed sensor device is effective and feasible, and it can meet the requirement of the vision inspection in hostile environment. Moreover, small size, low cost, high efficiency, high real-time performance, and high reliability are the further advantages of the proposed sensor device based on infrared and visible lights.
However, further work should be conducted, such as how to collect the infrared and visible light signals synchronously, how to integrate the infrared CCD and the visible light CCD, and how to improve the quality of the fused image.