An HDR imaging method with DTDI technology for push-broom cameras

Conventionally, high dynamic-range (HDR) imaging is based on taking two or more pictures of the same scene with different exposure. However, due to a high-speed relative motion between the camera and the scene, it is hard for this technique to be applied to push-broom remote sensing cameras. For the sake of HDR imaging in push-broom remote sensing applications, the present paper proposes an innovative method which can generate HDR images without redundant image sensors or optical components. Specifically, this paper adopts an area array CMOS (complementary metal oxide semiconductor) with the digital domain time-delay-integration (DTDI) technology for imaging, instead of adopting more than one row of image sensors, thereby taking more than one picture with different exposure. And then a new HDR image by fusing two original images with a simple algorithm can be achieved. By conducting the experiment, the dynamic range (DR) of the image increases by 26.02 dB. The proposed method is proved to be effective and has potential in other imaging applications where there is a relative motion between the cameras and scenes.


Introduction
Remote sensing cameras, especially those on the satellite working as payloads for observing the earth, are widely used in the field of natural resources, tourism, agriculture, forestry, marine and so on. And most of high-resolution remote sensing cameras operate on low-earth orbit, in which satellites always go around the earth, because the resolution of a remote sensing camera is in inverse proportional to its orbital height. To name but a few, GeoEye-1, WorldView-2, QuickBird-2, and Pleiades-HR are all functioning in low-earth orbits, whose heights are less than 500 km from the surface of the earth [1]. For these applications, dynamic range (DR) is an important index to measure the quality of remote sensing images and high dynamic-range (HDR) imaging technology is in urgent demand: the DR for a single scene in these applications can sometimes exceed 10 6 (120 dB) and even reach up to 10 8 (160 dB) because there are backlit objects and objects under strong sunlight [2]. However, the DR of many push-broom remote sensing cameras is smaller than 85 dB (such as Worldview-4, Quickbird, GeoEye-1, IKONOS, and so forth) [1], which is hard to keep up with demand.
Many efforts have been made for the HDR imaging technology. For the present, there are three kinds of solutions to realize HDR imaging. For the first kind of solution, HDR images are acquired from a single image with an estimation algorithm, such as camera response function estimation [3]. The second kind of solution aims to improve the design and manufacturing process of image sensors. By improving the voltage swing at the pixel output and developing new readout architecture, the DR of the CMOS image sensor is optimized [4]. For time-delay-integration (TDI) charge coupled device (CCD) image sensor which can be used in space cameras [5], an extended dynamic range architecture was designed by P. A. Levine et al. [6]. As for the third kind of solution, they are all based on multi-exposure. In 1997, Paul E. Debevec and Jitendra Malik proposed a method to recover high dynamic range radiance maps from multiple photographs taken with different exposure [7]. Then many researches have been conducted to improve and perfect the method. For example, a fast exposure time decision in multi-exposure HDR imaging was proposed by Yongjie Piao and Guang Jin [8]. And researchers have studied minimal capture sets for multi-exposure enhanced-dynamicrange imaging [9], as well as the iterative exposure bracketing method for HDR imaging [10]. HDR microscope imaging based on exposure bracketing [11] also depends on the similar method mentioned above. Some methods or algorithms based on multi-exposure imaging have been put forward, and new techniques are applied in them, such as compressive imaging [12] and phase-shifting algorithm [13]. Other efforts are made in the aspect of tone mapping [14] and in making full use of the existing DR by conducting appropriate amount of exposure [15]. However, such efforts could not get the DR of cameras increased, but it can improve the imaging quality or image's appearance.
Nevertheless, there will always be something missing. The first solution for HDR imaging is inadequate for remote sensing because the estimation algorithm is only an artificial prediction without good accuracy. What's more, it could not be applied to a whole image due to its working mechanism. For the second kind of solution, due to the limitation of semiconductor manufacturing process, it could be questioned by having no significant effect in a short period.
When it comes to the third kind of solution, it doesn't work normally because of a knotty problem for the push-broom cameras to re-capture the same scene with different exposure for one time: there is a high-speed relative motion between the camera and the scene. The relative motion leads to a circumstance in which only when the satellite completes at least (N-1) revisit periods can the scene be captured repeatedly for N times in push-broom remote sensing applications. The revisit periods of many remote sensing satellites are always longer than 1 day [1]. Accordingly, the scene may vary after a revisit period due to the motion of the target, the difference in the atmosphere, the solar altitude and other factors, thereby impacting the fusion of the images. Sometimes scenes of the same location may vary a lot. Therefore, it is rather difficult to adopt a method or algorithm to improve the DR of these cameras.
To improve the DR of a push-broom remote sensing camera, Xue et al. proposed a scheme of using two-row TDI CCD sensors with different integral stages to take pictures of the same scene and then fused the original images into a new HDR image by a simple algorithm [16]. However, this method requires a lot of redundant optical equipment and complex circuit which brings quite high power-consumption and huge cost.
As scientific and effective as possible in trying to think rigorously, this paper proposes a new method to get an HDR image in the field of push-broom imaging. In more detail, by adopting the DTDI imaging technology, this method makes it possible to get two images of the same scene with different exposures at the same time. And then the imaging process for higher dynamic range can be conducted, which is similar to the imaging process of digital still cameras.
Here, by carrying out a thorough exploration and comparison, this paper proposes a new method to improve the DR of push-broom cameras. The new HDR imaging method with the DTDI technology and the analysis of DR will be introduced in Section 2. Section 3 presents an experiment designed to test and verify the new method. And Section 4 conducts the discussion of the method. Finally, we come to a conclusion in Section 5.

Principles of DTDI technology
For the application where there is a high-speed relative motion between the camera and the target, apparent radiance of the scenes is quite low. TDI technology is usually used in TDI CCD image sensor to overcome the limitation of weak illumination. Normally, the traditional TDI technology is regarded as the analog domain TDI technology because it relies on accumulating photoelectric charges from multiple lines of pixels. While the DTDI technology is a special operation mode of area array CMOS sensor which is increasingly used as an optical sensor [17]. The technology can also be used in the application where there is a high-speed relative motion between the camera and the target. And in the DTDI technology, charge is converted into digital numbers (DNs) in each pixel and only the DNs will be transferred and accumulated. Figure 1 shows a DTDI CMOS imaging process with M N  pixels. In the push broom remote sensing satellite application, the working mechanism of DTDI is stated as follows: as for a fixed point j a (1 j N   ) in the observation scene, the pixel 1, j v in the 1st line should aim at it; and with the camera moving forward, so does the pixel 2, j v in the 2nd pixels line, the pixel 3, j v in the 3rd pixels line, …, and the pixel , M j v in the last pixels line. Meanwhile, this process is known as the whole working cycle for imaging a point. During this process, M DN values for a certain target point j a can be collected. Accordingly, by virtue of its working pixel by pixel, partial or even total of M DN values are accumulated; and then the item is generated as the final DN value of the target point.
In more detail, as for of m is small, a relatively dark image will be presented, and vice versa. Besides, if the value of m ranges from 1 to M, M images of the observation scene with different integration stages ranging from 1 to M will be collected. With the camera continuing to move forward, pictures of all the scanned area can be taken. Here so far, the whole working mechanism of the DTDI technology is expounded at some length.
It is evident that the area array CMOS imaging with the DTDI technology has the same advantages with ordinary CMOS application, such as lower power consumption, simpler circuit, and less limitation of full well charge number. Moreover, the process of charge's vertical and horizontal transference, as well as charge's accumulation among pixels is not necessary for CMOS sensors. In this sense, the restriction of limited full well charge numbers will be smaller. And as long as the single pixel does not get saturated, so will the output of DTDI CMOS.
In a word, compared with the TDI CCD camera which has the same full well charge number, DTDI CMOS camera can take pictures of a brighter target and won't get saturated.

Method for HDR imaging and image fusion algorithm
The  is darker than 2  .
Next by fusing these two images with some algorithms, an image with a higher DR than those of the original images can be taken.
Considering that the image process should be conducted on the satellite, the algorithm ought to be fast and easy implemented. In line with the method of Xue et al. [15], 1 and by (3), it is easy to conclude that 2 ( , ) x y  is more likely to reach FS  than 1 ( , ) x y  . In an ideal condition where noises can be neglected, the camera can be regarded as a linear system. It is obvious that when x y  meets the following inequation: which is the same as which means that the corresponding pixels in 2  will get saturated in a faster speed than those in 1 and as for other pixels, which satisfies (9) and their values should be expressed as (10). In this way, these pixels can take advantage of the highstage image and get a larger value in the new image: According to the analyses above, the complete expression of the HDR ( , ) x y  can be written as follows: . / x y x y x y x y x y In the end, in order to obtain a better display effect, HDR  can be transformed into a new image TRA  with rich gray level by image enhancement. where E basic is the exposure of each frame in the DTDI imaging process. Combining (12) and (13), a new equation can be reached as follows:

Dynamic range analysis of this method
Besides, dB is used to be served as the unit of DR. Compared with the original two images, the increment of the DR can be obtained, and it can be expressed as follows:

Experiments and results
To verify the HDR imaging method, an experimental platform is set up and is illustrated in Fig. 2. In particular, the experimental platform consists of the following four parts: (1) optics lens; (2) a camera with an area array CMOS sensor and DTDI imaging technology; (3) a computer for image processing; (4) a roller with target pattern.  , which is shown in Fig. 4(b).
To evaluate the experimental result, there are two ways to compare the images. The first one is to analyze the gray level. To some degree, an image whose pixels tend to occupy the entire range of possible intensity levels will have an appearance of high contrast, and the histogram of the images can indicate the dynamic range [18]. The more uniformly the pixels are distributed, the higher the image's DR will be. And the histogram can be expressed in a discrete function shown in (16) as where k  is the kth DN value of an image, and k  is the number of pixels whose DN equals to k  .
Finally, a measurement coefficient ( )   can be introduced as follows:  Table 1. It is easy to get the comparison of the parameters shown in Table 1 as which means the HDR  is more likely to cover the highest DR among the three images.
The second way to compare 1  , 2  , and HDR  is to observe their appearances. It can be easily found that in the HDR image HDR  , details of the dark part can be recognized, and it has a better appearance than 1  ; and details of the bright part are clearer than those of 2  without saturation.
Obviously, most of the information from 1  and 2  can be observed from TRA  .
After a comprehensive consideration of the images' appearance and a relatively small HDR ( )   , such a fusion process has indeed increased the DR. Based on (15), the increment of DR is 26.02 dB.

Discussions
This novel method has proven effective in increasing the DR of push-broom space-borne cameras. Previous studies of HDR imaging more concentrated on the application of static cameras. Usually they adopt the process of imaging a certain scene with multiple exposure time, and then fuse the low-DR images to achieve an HDR image. However, all of these methods haven't been adopted in push-broom remote sensing cameras, because the relative motion between the camera and the scenes makes it difficult to capture a certain scene for more than one time.
Few studies of HDR imaging methods are conducted in the application of push-broom cameras. Xue et al. ever mentioned that the push-broom remote sensing cameras can use two-row TDI CCD sensors with different integral stages to take two pictures of the same scene [15]. The above method really helps to increase the DR, but it also enhances difficulty in power consumption, circuit, data transfer, optical structures, launch weight, and so forth. By contrast, the novel method adopted by the present paper has higher efficiency in the following two aspects. The first one is that there are no large increases in hardware circuit, optical structures, and launch weight, etc. The second one is that there is more increasing room for the dynamic range, because the integral stage of TDI CCD image sensors is limited to a small value. Meanwhile, with the application of DTDI, the largest integral stage can be the maximum number of CMOS pixels' rows.
And then the dynamic range can be significantly increased.
In the present paper, the HDR imaging method is employed to the push-broom remote sensing cameras. Moreover, this method can be applied for almost all kinds of motion imaging. Considering the limited data handling capability of a satellite and the limited data transmission capacity, this paper adopts the image fusion method which is simple and practical. If this method is applied in other fields, such as the industry field, the medical field and aerial remote sensing field, the effect of HDR images can be better with fine algorithms.

Conclusions
This paper proposes a new solution to increase the dynamic range of push-broom remote sensing cameras. As for a fixed scene, more than one picture with different exposure can be taken simultaneously by employing the DTDI technology. In this way, many traditional methods can be applied to push-broom remote sensing cameras because it can be treated as common static cameras.
In addition, based on the memory space, computing power, and data transmission bandwidth of the satellite, this paper adopts a simple and executable algorithm to increase the DR of the camera. With this algorithm explained in Section 2.2, two images taken of a same scene with different exposure are fused into an HDR image. By conducting the present research smoothly, the new method proposed by this paper gives a new view of obtaining HDR images and makes the increment of DR larger than 26 dB. And it is worthwhile to make further strenuous effort on the HDR imaging with push-broom remote sensing cameras.